Is it a Control Variable, Mediator or Moderator?

It's complicated and often a source of bad science.

As a basic rule, bivariate correlations are a poor test of hypotheses. There is a kind of general agreement if one is looking to examine the correlation between X and Y, it makes sense to control for theoretically relevant control variables A, B and C¹. Otherwise, those bivariate correlations will give you an inflated sense of support for the hypothesis. As such, it’s a data tragedy that so much of social science still relies on bivariate correlations, which grossly misinform our impression of whether a correlation exists or not. This is particularly true for meta-analyses of correlational studies which often stubbornly adhere to a failed culture of using bivariate correlations, despite their blatant heterogeneity and unsuitability for testing hypotheses². This, I would argue, renders meta-analyses too often a significant source of misinformation.

Nonetheless, there are legitimate questions about how to handle any given potential “third” variable. In the examination of whether X predicts Y, should we think of A (the third variable) best as a control variable, mediator or moderator? Unfortunately, there are no clear guidelines, often leaving scholars to make decisions that simply confirm their beloved hypotheses (or moral advocacy, or best-selling book, etc.) And often the answer is nuanced…a given variable may be potentially two or even all three of these at the same time. But here are some thoughts on to how to best consider third variables. Let’s start with defining terms.

Control Variables:

Control variables are variables that theoretically, based on prior science, are underlying predictors of both X and Y in our basic correlational equation. In this sense, the question we are asking is not just does X and Y correlate (they might) but does X provide unique predictive value above and beyond what we already know predicts Y which would be variable A. Or is the relationship between X and Y basically explained away by A? In which case X is likely an artefact of A and tells us nothing new of particular importance.

For example, consider the case of violent video games and physical aggression. In this case, we already know some things that are important. Particularly, boys are more physically aggressive than girls. Also, boys tend to play more violent video games than girls. So, if there is a correlation between violent video games and aggression, this may merely be an artefact of boyness. As such, sex is an important control variable. Other theoretically relevant controls such as trait aggressiveness and family conflict function in similar ways such that they are “basic” control variables that should always be controlled when considered correlations between violent games and aggression. The raw bivariate correlation is likely to mislead us, giving us greater confidence in a correlation than we should have.

Of course, it’s entirely possible to control the wrong variables, so some degree of critical thinking and theoretical rationale is required. First, a model can be undersaturated by throwing in theoretically irrelevant control variables. For example, if in the video game example, we control for astrological sign, this is useless. It gives us the illusion of a controlled analysis when, for all intents and purposes, nothing useful has been controlled and we still have a bivariate correlation.

It’s also possible to oversaturate an analysis by including a control variable that, in essence, occupies the same constructural space as the predictor. For instance, if we examine the correlation between race and socioeconomic status, but then control for skin color, I’m likely to underestimate the predictive value of race. This is because, like it or not, race is still often defined by skin color or other crude physical features³.

Mediators and Moderators

I sometimes call mediator and moderator analyses the “devil’s work” because they are so often misused, typically in an attempt to rescue a hypothesis from null results. The main problem is, without preregistration, mediation and moderation analyses open up multiple opportunities for capitalizing on chance and post-hoc reasoning. For this reason, they are often suspect.

However, with an a priori hypothesis, and preregistered analyses, they can absolutely be legitimate. Here are some thoughts on them.

Mediation

Mediators imply a temporal order where in the relationship between X and Y, A is a mid-point in the relationship. So, X leads to A, then A leads to Y. We can think of this like genetic inheritance leads to trait neuroticism, then trait neuroticism leads to a diagnosis of major depression. There’s a temporal chain and implied causality.

Such analyses are most appropriate to longitudinal designs with multiple time points where temporal order can be established. Granted, unless this is a longitudinal experiment, the data remains correlational, so definitive causal statements can’t be made. But at least the analysis makes sense. Mediation analyses make less sense for cross-sectional data where temporal order can’t be established, and the “mediator” variable might be better construed as a control variable. I’m also skeptical that the statistics that evaluate indirect pathways are all that reliable. So, we may have a good sense that X leads to A, and A leads to Y, but saying therefor X causes Y indirectly through A feels a bit more tenuous. After all, lots of things presumably lead to A and, particularly where effect sizes are generally pretty weak, trying to degrees of Kevin Bacon an indirect pathway into causal certainly feels like it lacks rigor.

But generally a mediator variable comes in the middle, which is the easiest way to remember it.

Moderation

A moderator variable is a variable that changes the strength or direction of a relationship. So, for example, we might say for X correlates with Y, that reading books is associated with better mental health. Reasonable hypothesis, I guess? But maybe that’s not true for everyone? Maybe reading books is associated with better mental health for women but has no predictive relationship for men. In that case sex would be a moderator. The strength of the X and Y correlation is different for different groups.

The problem with moderator analyses is, like with mediators, obvious. Post hoc, there are dozens and dozens of potential moderators in many datasets, and people can go fishing through them, capitalizing on chance, in order to rescue a hypothesis from null results by focusing on moderators. “Hah, alright, so my hypothesis that reading books is associated with schizophrenia is still valid, because I find a weak correlation between books and psychosis for Sagittariuses. Therefor Sagittariuses are a vulnerable group!” Yes, this is the dreaded “vulnerable group” argument which, upon basic failure of a hypothesis, opens up literally unlimited opportunities to rescue it as there are virtually unbounded potential vulnerable groups. We could run through the sexes, LGBT individuals, various ethnic groups, people with various preexisting diagnoses, on and on.

Again, it’s not that moderation analyses are always bad, it’s just that they often are, particularly when conducted post-hoc without a limited number of preregistered analyses. In that sense, indeed, they are the devil’s work and often offered as a self-defensive rationalization to preserve a failing hypothesis.

So, my argument is not that mediation or moderation analyses should never be done, but rather that they shouldn’t be done willy nilly which is how they often are done. Good, theoretically relevant analyses that are limited in number, and preregistered can absolutely be useful.

But then, how do we know whether a variable is a control variable, or a mediator, or moderator. Unfortunately, for a given variable, often an argument can be made for any of the three. And scholars will tend to interpret a variable in accordance with their preexisting beliefs, as scholars are human. Skeptical of the relationship between X and Y, a variable might look like a control variable. Believe the relationship between X and Y is an important public health policy concern: well, then almost nothing looks like a control variable.

Even trickier, variables don’t always fit into clear boxes. The same variable may explain away some of the initial relationship between X and Y (a control variable), but also function as a moderator because, down the road, X influences individuals differently. For instance, it may be entirely valid to use sex as a control variable for social media use and mental health, given girls both use more social media and express more anxiety and depression, but at the same time, social media may or may not have different impacts on girls and boys (ultimately, though I don’t think the data say this). But even that might be a post-hoc rationalization. The idea that social media impacts girls more than boys seems to be a backwards facing theory based on a few initial study results that haven’t always held up, rather than an a priori hypothesis. But if it had been theorized a priori, it could be valid. Point being: it’s complicated.

I’ve definitely seen scholars argue for, basically, never having theoretically relevant control variables but I feel this is horrible advice and often self-serving for personal or advocacy goals. Instead, we should subject our hypotheses to rigorous evaluations and be willing to admit perhaps they’re weaker than we thought or able to be explained by other things.

Basic Guidelines to Live By⁴

· If there’s a reasonable theoretical reason why A is likely a cause of both X and Y, it should be treated as a control variable, unless there’s rigorous data to suggest otherwise. Non-causal control variables can still be relevant for correlational analyses, particularly when we’re interested specifically in unique predictive value beyond already known predictors. For instance, if we are trying to predict job performance, and we already have instruments A and B, knowing that new instrument C doesn’t improve the overall predictive model even if it does predict job performance independently is still useful information.

· Mediation analyses should only be done when temporal order can be established in longitudinal designs or perhaps in some controlled experimental process.

· If there’s a good a priori theoretical reason that a variable is not a cause of X and Y but may change the direction of a relationship between X and Y, it should be treated as a moderator.

· If something could be reasonably be argued to be either a control variable or moderator or mediator, consider running the results each way and interpreting them conservatively. Because the weight of evidence is on the causal model, if it can be explained as a control variable, that should take precedence, but at least you’re letting the reader decide for themselves, reporting all analyses.

· If the moderator is a simple categorical moderator like biological sex, run separate regression analyses for each category, rather than one of those fing-fangled moderator “model” analyses. My suspicion is the reliability of those is iffy.

· Though mediation models imply causality, if your data is correlational, the old maxim of “correlation doesn’t equal causation” still takes precedence. No, longitudinal data is not causal.

· Moderator and mediator analyses should be preregistered prior to data collection. Any exploratory mediation or moderation models should not be central to hypothesis interpretation or override preregistered null results. Likewise, control variables should be specified during preregistration.

· It should go without saying that doing mediation and moderation analyses, or adding or removing control variables post-hoc in order to achieve support for any hypotheses could constitute p-hacking. It may be understandable to change models under some circumstances (collinearity for instance) but this should be documented if differing from the original plan.

· If you’re model is getting super complicated with multiple mediators and or moderators, you’re likely getting further away from anything that can actually be reliably useful or understood in the real world.

· There are probably exceptions to everything I’ve said above…just be cautious and use critical thinking before giving yourself free reign to not be rigorous.

I hope these thoughts are helpful, and good luck with those analyses!

1. This seems to be one of those things everybody both knows and simultaneously will argue against once it’s in their best interest to do so.

2. Seriously, that this is simultaneously so obviously bad for science and yet so difficult for people to not do despite being able to not do it, apparently, that I fear for even basic critical reasoning in social science sometimes.

3. Here again, though there are many nuances and few clear paths. Concept Balkanization is a big problem in social science. If depression is our predictor, can we use anxiety as a control variable? Or do they occupy too close a conceptual space? There aren’t always easy answers. Multicollinearity analyses will help weed out some issues, but not necessarily all of them.

4. As with all things human nature, I’m sure I’ve violated my own advice here plenty of times in the past.