Statistical models represent the data-generation process.

A recent paper I read (Niemeyer et al., 2023), building on the work of John Ioannidis and others, suggested that most published findings in my particular slice of social science, criminology and criminal justice, were false. A claim like that is not as provocative as it once was, but what I did find provocative was that the authors primarily centered the blame on bad theory, rather than the now-classics like the file-drawer problem, small sample sizes, p-hacking, and so on.

It is all too easy to forget that theory, design, and analysis are integrated parts of a whole. The results of a sophisticated analysis of a set of observations might be meaningless if too far detached from aspects like conceptualization and operationalization. Moreover, an oft-ignored assumption is that our models are correct, and, as I began this blog post stating, models represent the data-generation process.

The cart (design), the horse (analysis), and the driver (theory).

A pitfall I find myself and others step into is putting the cart before the horse, but I think the more insidious, yet seemingly innocuous, pitfall is when the driver is separated from the horse and/or cart. According to Greenberg (1983), a theory has to meet three criteria in order to be a theory: 1) propositions specifying relationships among variables that can be deduced from assumptions, 2) variables can be measured and should be measured in certain ways, 3) the evidence required to test validity and implications of evidence. A lot of popular “theories” across the social sciences arguably fail these criteria.

When I or others I speak with “just can’t get this model to work”, often the root of the issue is that the horse is attempting to push the cart and the driver is sipping tea or aggressively whipping the horse to “encourage” them. Instead of taking a step back and considering whether the components are a coherent whole, we’re often narrowly focused on a single constituent part.

I plan for most of my content here to revolve around statistics, so on its face it seems tangential to discuss theory, and perhaps even design. However, using Greenberg’s criteria, theory influences design, such as a dependent variable’s measurement, and both theory and design influence analysis, such as including a particular covariate or “control variable”. In my view, a theoretical framework is explicit or implicit, but it is always there, and therefore, so too is a set of assumptions about the fundamental nature of something (e.g., a paradigm of human behavior), often among other assumptions.

No such thing as an atheoretical choice.

Ronald Fisher (1921), in a piece from over a century ago, demonstrated a technique that is common today (ANOVA), focused on the (exhilarating) topic of crop yields. Relatively early on, he makes the point that the variance of a dependent variable is equal to the sum of variances in that dependent variable explained by independent causes, and therefore it is possible to assign portions of variance to independent causes. I emphasize that particular word because it is a verb. It is an action that can be done with “more or less accuracy” (p. 111). It is possible to assign a portion of variance in pool drownings to Nicolas Cage movies.

To make this particularly droll–you can slice a pizza in a virtually infinite number of slices and still have a whole pizza, so it is up to you to decide how you want to slice it. Such profundity, right?

The big point here is that, even estimating a simple model like Y is an outcome X, is laden with theoretical choices. That simple model is built on presuppositions, such as: X and Y and are real-world “things” or tap into (or “correspond” to) real-world “things” to some extent; X and Y are continuous or categories; the relationship is positive or negative, linear or non-linear. Any given model is correct insofar as one can assume the the theoretical framework it is based on is correct. If you contracted me to build you a two-story house, and I built you a bungalow, you would have a bungalow, even if my craftmanship was immaculate. A bungalow might be an excellent fit to the property, just as a statistical model might be an excellent fit to observed data, but that does not make a two-story house.

Don’t forget the driver or the cart.

In sum, our knowledge of the social world based on empirical research might be in large part spurious. And if that is true, it is probably not because social scientists are overwhelmingly bad at math or fraudsters–though there is some of that, to be sure–but rather probably because of a mismatch between theory, design, and analysis. Mathematical properties can be very useful, but we have to make choices in how we use them. Those choices should be driven by theory and design because our goal is to approximate a real-world process. Think carefully about the cart, the horse, and the driver independently, but also think carefully about the big picture. You may find that what seems like problems with your horse are actually problems with your cart and/or driver.

References

Fisher, R. A. (1921). Studies in crop variation. Journal of Agricultural Sciences, 11, 107-135.

Greenberg, D. V. (1983). Donald Black’s Sociology of Law: A critique. Law & Society Review, 17(2), 337-368.

Niemeyer, R. E., Proctor, K. R., Schwartz, J. A., & Niemeyer, R. G. (2023). Are most published criminological research findings wrong? Taking stock of criminological research using a Bayesian simulation approach. International Journal of Offender Therapy and Comparative Criminology. https://doi.org/10.1177/0306624X221132997