In the last 2 years we’ve seen a massive push across nations to ban youth access to social media. This is despite the lack of evidence that social media is associated with youth mental health, and experimental studies which find no or “weak” evidence that reducing social media time is helpful. Further, such bans have been promoted despite no evidence they might work, and increasing evidence from Australia that its ban is failing. If governments are going to lurch into clumsy and free-speech harming government surveillance of the internet, we’d hope they’d at least objectively analyze whether this is likely to be successful.
So far…the signs aren’t good. I’ve been pretty vocal that Australia’s effort to “prove” its ban is helpful will be high pressure, trying to squeeze soup from a stone1. Mike Males has an excellent essay on the US Centers for Disease Control’s inability to look at its own data. As Males says, I think we’ll be seeing a lot of misleading claims about the harms of social media and the “success” of these government surveillance programs under conditions of extreme political pressure. But maybe the UK would be different?
I’ll say at the upfront that writing this essay brings me no pleasure. My impression is that the team heading up the UK’s analyses of social media has integrity and means well. Thus, I was shocked to discover it’s plan for analyzing the impact of social media reduction is as bad as it is: essentially doubling down on a heavily biased strategy I noted is insufficient back in my 2025 meta-analysis of such studies.
Put simply, the randomized controlled trials (RCT) in this realm have a shared fatal flaw: it’s obvious to the participants what the hypothesis is and which condition they are in. Particularly in an environment where news, schools, parents, etc., are constantly hectoring kids about their social media use, participants, both kids and parents, will know what responses they are supposed to give. Under these circumstances, when participants can guess a study’s hypothesis, they’ll tend to give the researchers the behavior they want2.
As I said two years ago, these types of studies are completely incapable of answering any useful questions due to this fatal flaw. And yet, for some reason, researchers keep doing them. This UK study is even more egregious. They advertise not only their hypotheses but the desired outcome on their public website saying, “We hope this trial will support teenagers in focusing more on their offline, in‑person activities — which is why we’ve named the study The IRL (In Real Life) Trial.” They’re even having a public information session on it.
Unfortunately, if I were to design a study that put a thumb on the scale in favor of the UK’s apparently preferred outcome, this is how I would do it. It is an entirely biased approach, even worse than most studies in this realm due to the public proclamation of the results they want. This will bias participant responses but also hints toward researcher expectancy effects which can influence research results as well as interpretations of weak data. They’re basically naming the destination before the map is even drawn. I don’t at all believe that this is a group that would purposefully falsify data, but having staked such a public position, it’s just human nature to unconsciously nudge results or the interpretation of results in the direction of that public position.
Given that social science already has a big problem with naïve interpretation of “statistical significance” and a tendency to grossly misinterpret weak, likely noise effects, this is not encouraging. With a planned 4000 student participants, this study is all but guaranteed to get false positive results due to noise/crud. This is a common problem in social science, particularly for bigger samples, where everything basically correlates with everything else a little bit, but doesn’t necessarily reflect real relationships. This is a common source of significant misinformation, as social science researchers commonly neglect this issue.
It’s easy to tear apart an unliked study post-hoc. I once heard from a researcher who said he could tear apart any published study if he was so inclined. The problem is that this leads to selective enforcement of rigor. Due to confirmation bias, we humans including researchers, selectively criticize studies that produce results we don’t like, while giving a pass to studies whose results confirm our personal beliefs.
As such, I am putting down my marker before this study is run. It’s useless, entirely without value. It doesn’t matter if it confirms my personal beliefs or challenges them. Like all studies of this particular kind it has no internal validity. Unfortunately, it will also waste the time of the youth and families participating in the study.
This seems to be an example of the “more studies” problem I recently wrote about. In effect, given underwhelming evidence and poor quality studies, researchers often double down on just doing more of the same, flawed studies. But such a flawed study is worse than no study at all, as it runs a real danger of misinforming policy one way or another. Whether from the UK or Australia we’re going to need to buckle down for a lot of bad social science propping up failing policies.
Given Australia’s youth mental health was already improving before the ban, any pre-post analysis runs a serious risk of misattribution. Simply comparing Australia to another country or two like New Zealand isn’t going to fix that.
This is a long-standing and well-known problem in social science which, nonetheless, many social scientists simply ignore when convenient.
They really seem to luuurrrve junk science!