STATISTICAL SIGNIFICANCE, THE SHOTGUN APPROACH, AND OTHER ISSUES

To finish this chapter, I want to deal with four thorny issues in social science data analysis: (1) measurement and statistical assumptions, (2) significance tests, (3) eliminating the outliers, and (4) the shotgun method of analysis.

Measurement and Statistical Assumptions

By now you are comfortable with the idea of nominal, ordinal, and interval-level measurement. This seminal notion was introduced into social science in a classic article by

S. S. Stevens in 1946. Stevens said that statistics like t and r, because of certain assumptions that they made, required interval-level data, and this became an almost magical prescription.

Thirty-four years later, Gaito (1980) surveyed the (by then voluminous) mathematical statistics literature and found no support for the idea that measurement properties have anything to do with the selection of statistical procedures. Social scientists, said Gaito, confuse measurement (which focuses on the meaning of numbers) with statistics (which doesn’t care about meaning at all) (p. 566). So, treating ordinal variables as if they were interval, for purposes of statistical analysis, is almost always a safe thing to do, especially with five or more ordinal categories (Boyle 1970; Labovitz 1971a).

The important thing is measurement, not statistics. As I pointed out in chapter 2, many concepts, such as gender, race, and class are much more subtle and complex than we give them credit for. Instead of measuring them qualitatively (remember that assignment of something to a nominal category is a qualitative act of measurement), we ought to be thinking hard about how to measure them ordinally.

Emile Durkheim was an astute theorist. He noted that the division of labor became more complex as the complexity of social organization increased (Durkheim 1933 [1893]). But he, like other theorists of his day, divided the world of social organization into a series of dichotomous categories (Gemeinschaft vs. Gesellschaft, or mechanical vs. organic solidarity).

Today, social theorists want to know how degrees of differences in aspects of social organization (like the division of labor in society) are related to social complexity. This requires some hard thinking about how to measure these two variables with more subtlety. The meaning of the measurements is crucial (Further Reading: measurement in anthropology).