Empirical Identification of Evidence

How do we empirically identify the causes of crime? This seems like a simple, basic question to ask of a discipline that takes this question as one of its core concerns. Surprisingly, there is no clear answer, and criminologists do not have a shared, publicly acknowledged definition of the empirical standards required for assessing the scientific standing or validity of an explanation of crime. This produces some confusion, especially because criminologists employ a variety of standards and tend to simply allow the peer review process and the competing arguments reviewers make on these points to control this determination. This is less than satisfactory and can often result in conflicting peer assessments of work under review.

Lacking a specified disciplinary standard, how do criminologists empirically identify the criteria for testing an explanation of crime? Historically, the most common method is the test statistic. The test of statistical significance is a simple, empirical test that allows for an assessment of the data. Specifically, that assessment examines whether a particular variable is able to distinguish between offenders and nonoffenders—that is, is the relationship between an indicator and outcome likely to be significantly different than zero in the population?

There is nothing magical about tests of statistical significance, since they are simply based on the probability associated with random selection. Moreover, tests of statistical significance tell us nothing about the strength of any relationship—only that a relationship is likely to exist in the population. In short, test statistics and their associated probabilities are rather straightforward. The test of significance is limited to the extent that they only tell us something about difference when considered in the form in which that explanation has been tested. As a result, we must rely on somewhat subjective interpretations of discovered relationships within the context in which they are assessed. As a result, it is entirely possible that variables are statistically significant and yet potentially irrelevant. This may happen when the empirical test itself provides a poor prediction of the outcome, a problem that is not uncommon in microlevel tests of explanations of crime. In order to know whether the test of statistical significance has meaning to the scientific determination being made, we need additional measures of the empirical test.

One of the reasons we need these additional measures is that the test of statistical significance is affected by the other factors being assessed in an analysis. When, for example, important control variables that might explain the outcome are omitted, the results of the test of significance are questionable because they are estimated in relation only to the variables included in the model being analyzed. Thus studies that omit important explanatory variables may yield statistically significant results when measured a certain way and in empirical models that are inefficient and exclude important explanatory variables but not when other variables are included.

Other factors may also impact the calculation of test statistics. For example, holding other factors that influence statistical significance tests constant, these tests are more likely to indicate that a relationship is statistically significant if a large, rather than a small, sample of cases is studied (Royall 1996). If, for example, we are studying the effect of X on Y in a multiple regression and have a large sample, the likelihood that the p value for this relationship attains significance increases with sample size. Thus the opposite condition may also be true. If a research endeavor starts with a relatively small sample (say, n = 50), the likelihood of rejecting a null hypothesis (or of having a confidence interval that encompasses positive and negative values) is less likely than if the researcher stated with a large sample. Of course, researchers understand this problem and often conduct estimates of statistical power to pick the most appropriate sample size to determine the desired level of impact.

Another problem with microlevel studies is that the vast majority of those studies produce weak statistical results when tested empirically. By “weak” we are referring to the ability of the selected variables to explain an adequate amount of the variation in the dependent variable (whether that variation is important is another matter). Of course, the problem here is defining what is meant by the term “low.” For example, one common (though much less common today) statistical measure that is used to make an assessment about the goodness of model fit in criminological studies is the coefficient of determination or R² (or some derivation of that measure). Many scholars interpret R²—which ranges between 0 and 1—as the proportion of variation in one variable accounted for by another variable (see Ozer 1985). When criminologists report R² for an equation in published research, they often report R² values that are rather low. For example, looking through the last two years of the journal Criminology, we can see that the coefficients of determination for individual-level studies range between 0.07 and 0.49. Research (e.g., Taylor 1990) suggests that weak or low coefficients of determination are usually less than 0.35; moderate are between 0.36 and 0.67; and strong are between 0.68 and 1.0. Thus most of these coefficients in criminology are weak to moderate in nature. Unfortunately these R² values are generally accepted as normal within social sciences and lead us to question the extent to which knowledge has advanced in criminology over the past century, since it is based on weak to moderate effects.

To illustrate this point, we refer to a study by Lober et al. (2005) that displays regression results for a series of models used to estimate violence and homicide by young men. In using this work, it is not our intention to focus criticism on these particular researchers— all are well respected—but to point out that criminologists tend to neglect the relevance of R² in the interpretation of empirical results.

In their article, Lober et al. (2005) report the results for seven regression models. Two of these models (five and six) do not report R² values and will hence be omitted from this discussion. The researchers start with a base model, and each of the additional models adds explanatory variables, and thus they have some use here in relation to the question concerning the effect of model fit on the test of significance for independent variables and how the estimate of significance changes as the form of the model changes and becomes more complete, and R² increases.

The number of variables in each model and the R² for each of the five models is shown here:

In Model 1, the three variables, all of which have test statistics large enough to suggest that the probability of observing an association between the variables at least that extreme, if there were no relationship in the population, is less than 0.001. As variables are added to the equation and the fit of the model is improved (i.e., R² increases by definition, Greene 2003), the test statistic for these three variables, however, is attenuated between Model 1 and Model 7, and the variables are now more likely to be related by chance (p < 0.05). However, there is still a significant amount of variation left to explain. What would happen to the regression coefficients for the three variables in Model 7 if 13 additional variables were added into that model? Would they then appear to be produced simply by chance? We do not know if that would happen, but the research as presented should cause us to pause and ask whether the results ought to be accepted as an accurate, scientifically acceptable description of the causes of violence across young men. In the current form, the hypothesized relationships cannot be dismissed. But it is hard to claim overwhelming support for the alternative hypothesis. From a scientific perspective, we need to be much more careful about the conclusions we reach from individual-level data. Should we base policy decisions on these data? Even if a researcher is careful with regard to policy suggestions and cautions that the estimates may be problematic, reviewers may require researchers to make those recommendations.

As an analogy, consider the difference between the science of climate change and the political and media manipulation of climate science research that has been occurring for the past 40 years (Boykoff 2007). While climate scientists were building a consensus on climate change, some members of the media and politicians were reporting the opposite. And the more scientists agreed, the more these politicians and the media disagreed. Criminology is essentially locked in the same battle with science. Unable to produce very efficient models of criminal behavior, criminologists adapted to this situation by employing more sophisticated empirical models in the hope that somehow doing so despite previous failures would provide better answers to the question, “What causes crime?” That problem, of course, is not that science is not useful, but that criminologists have adopted a political and not a scientific definition of crime.

This concern relates to a potentially serious problem in the discipline of criminology that occurs when predicting crime is affected by omitted-variable bias, which can alter the results of empirical studies (Berry and Feldman 1985). Omitted-variable bias often causes an equation to predict an outcome poorly; therefore, the results of that equation should be interpreted quite cautiously. One of the problems is that we typically have no idea how an omitted variable may impact the results of a study, since it is likely excluded because it is unknown to the researcher. Its impact could be quite large and change the correlations of the variables that are included in the analysis of crime. We must keep in mind that equations that suffer from omitted-variable bias produce biased results, and we must be willing to acknowledge that possibility and be cautious about generalizing the results of this kind of research outcome. The question is this: How serious of a problem is model misspecification due to omitted-variable bias in microlevel research? The answer to this question is not known. From the existing literature, however, we know that we can find numerous examples of poorly specified models of criminal behavior with low explained variance in which there are statistically significant relationships that the authors accept as evidence that the explanation is valid. As noted, such results should be viewed with caution, and few criminologists engage in such cautious interpretations. This condition may be a reason that tests of the same explanation of crime sometimes support and sometimes reject an explanation of crime.

Why do criminologists accept poor to moderate results produced by potentially misspecified statistical models as evidence of the scientific nature of the discipline of criminology? As we have noted, criminologists will often counter that it is difficult to explain crime, especially at the individual level. Criminologists are often satisfied with explaining a small portion of the variation in crime at the microlevel. Instead of being concerned with those outcomes, those kinds of results are treated as normal and expected. While we do not have an answer as to how criminologists should fix this problem, we suspect that acknowledging the bias in the definition of crime found in the criminal law and correcting for it will help. We believe that it is possible, perhaps, that the issues we have described might not be as prevalent in the discipline of criminology if crime were defined differently so that it could be studied more consistently across space and time—that is, if it were a scientific rather than a legally derived concept.