Did a New Study Show that a Norwegian "Ban" on Smartphones Helped Kids?
Nope.
A recent study made a small splash for claiming that a ban on cell phones in Norway resulted in better mental health and grades for teens. I first heard of this study was in a Twitter post from psychologist Jay Van Bavel, who was not involved with the study. Van Bavel highlighted that the study claimed the ban resulted in a 29% decrease in psychological symptoms, a 43 % reduction in bullying and improved grades by .08 standard deviations. But…did Norway actually ban phones in schools? And are the results really as positive as some suggest?
Anybody, even the statistically naïve1 might raise an eyebrow at that last result. Even if you don’t know what a standard deviation is, doesn’t .08 sound rather small? That’s because it is. So small it’s a rounding error (or more likely simply statistical noise). The percentage figures sound more impressive, but they’re also tiny, statistically speaking, still within noise/rounding error range2. That’s one reason I hate when research papers talk in terms of percentages…they can sound more impressive to lay people than they actually are (a lot of medical research is guilty of this3).
The paper itself appears to be unpublished in any peer-reviewed outlet. This isn’t necessarily a death knell as I think it’s safe to say peer-review has hardly been the gatekeeper of scientific rigor and objectivity we might hope it would be. But this paper adds to my nervousness about the whole preprint trend in science where rubbish papers get posted online and get picked up by the popular press (or Twitter) without much fact checking.
The first thing that’s worth noting is that the whole claim about “banning” smartphones is misleading. Neither the country of Norway nor most schools did any such thing, at least at the time of this study. Most simply required students to set their phone to silent mode during class…a very reasonable request and hardly a ban4:
So, we’re mostly talking about the effects of asking students to silence their phones, not schools banning them5. Unfortunately, the author herself sets up this misleading conversation by referring to all of these as “type of smartphone ban”…perhaps something is lost in translation, but that’s simply not what happened in most schools.
As is often the case with these hot-button moral panic issues, the narrative around the evidence focused exclusively on results that trended in the direction the author wanted (whether they were statistically significant or not), and ignored results that did not. As the author says early on in her results regarding mental health “Analysis of the full sample indicates no effect of banning smartphones on student’s likelihood of being treated, or on the intensity of treatment, related to psychological symptoms and diseases…” That pretty obviously conflicts with claims about a 29% reduction in symptoms.
So where does the 29% come from? The author then split analyses by sex. She finds that girls experienced a 29% reduction in mental health visits to specialists…but not to general practitioners. So one out of two outcomes…hey maybe that’s not bad, right…but what she fails to focus on is by that same metric, boys got worse!
What about the bullying? The author explains…”Bullying incidents decline by 0.25–0.35 standard deviation (sic) two to four years after a smartphone ban is implemented (p-values of 0.067 and 0.094)…” Hey, that sounds pretty good…but wait a minute! Record scratch!
Those results are actually not statistically significant! And those effect sizes, though better than the laughably tiny .08 for GPA, are still smaller than the noise threshold for garbage results. She goes on to do the usual split by sex (boys don’t get much further mention because, again, there was no trend at all), and once again relies on a mixture of non-significant results (p = .058 for the statistically minded) and threshold significant (p = .039) results for girls to make her case.
Put simply, even if what Norway is doing is a ban, which it is not, this study provides very weak evidence for any benefits for these policies. It mainy relies on tiny effect sizes and often non-significant results. I blame this on the academic field of economics…the author appears to be from an academic department of economics. Don’t get me wrong, economists do smart and important work…on economics. But they have a marked tendency to blow it with human behavior, often engaging poor quality studies and exaggerating the meaningfulness of trivial effect sizes. Ok, that actually sounds a lot like psychologists, so perhaps I should be kinder to my economist friends.
In short, there is very little to see here. There’s nothing in this to support any kind of “ban” as educational policy (in fairness, the study so poorly defines banning that arguably there’s not much here to refute bans either). Yet it is a shame that some smart people have promoted it in such a misleading way. This is further evidence of how low the bar is for evidence as regards to our current social media panic.
No offense, you benighted souls.
As we’re going to see in a minute…
Particularly diet advice, most of which is worth ignoring.
Note: Copyrights for these graphs (but not the funny AI picture) belong to: Sara Abrahamsson, the original study author. I reproduce them here as a.) the purpose of reproduction is educational and, as this is a free post, results in no commercial gain for myself, b.) is the reproduction of a freely available work of research data, c.) represents only a small portion of the research article and d.) as the original work is freely and publicly available, reproduction has no impact on Dr. Abrahamsson’s commercial profits. Thus, the reproduction falls under my understanding of US Fair Use Doctrine and I include it here in good faith.
In fairness to Dr. Abrahamsson, the Norwegian government use the term “ban” themselves, though that mainly doesn’t seem to be what’s happening (or at least wasn’t in this report).