A New Study Spells Bad News for Cellphone Bans
The study everyone's talking about (last week) but how conclusive is it?
The latest study of cellphone bans caused shockwaves as it appeared to be really bad news for advocates of the bans. This study basically compared a specific type of ban, lockable pouches (such as Yondr pouches) with schools that did not use them. Across the board this was basically a null study…no benefits to standardized testing scores, classroom attention, bullying, and the mental health outcomes were “statistically significant”, though wobbling in both directions, and so close to zero in effect size to be more likely due to methodological noise than “real” effects (which would also explain the wobble)1.
Predictably, these results sent advocates for bans trying to squeeze soup from a stone…particularly around those near-zero mental health effects, but at least on the surface, this study is very bad news, indeed for cellphone ban fans. It supports the narrative that they are largely ineffective.
There are some reasonable criticisms of the study though. First, it’s unpublished and I really don’t like this trend of unpublished economic studies getting widespread “cold fusion” style news coverage without peer review (not that peer review is great at stopping bad studies from being published). Second, the best critique I’ve seen of the study from cellphone ban fans is that it only examines one policy…Youndr type pouches2, not bans more generally. I think that’s a fair observation. Third, the study appears to pull data from a lot of different sources, which I think can be a significant source of error. Fourth, I think the difference-in-differences method the authors use is vastly oversold and too fragile to researcher choices and assumptions. And, last, the authors too often fall for a naïve interpretation of p-values when the underlying effect sizes such as for mental health, suggest a null interpretation would be more appropriate.
This is actually one of the surprising things about the paper though. Speaking frankly, I perceive these authors as being in a habit of misrepresenting weak data inappropriately. One of their studies on social media restriction was too often sold as evidence for benefits of such an approach, when the effect size was so close to zero as actually to better support the opposite conclusion3. But that’s what’s rather remarkable here…even with authors who, and I’m not implying bad faith, have some evidence biases that tend to put the thumb on the scale in favor of anti-tech narratives, this is a big bunch of nothing.
That didn’t stop at least one of them from still defending cellphone bans. Study coauthor Thomas Dee told journalist Alexander Russo he was concerned the coverage might cause people to “jump to the conclusion that phone bans aren’t a good idea.” He also told journalists it would be “wrong for policymakers to see the results as a reason to shy away from restrictions.” Where else do you find researchers run a study that finds a policy doesn’t work yet still stump for that policy? During moral panics, critical thinking can be pretty low.
I don’t think this one study will be the end of the debate of course. I have it on good authority another economist paper that made some headlines last fall in support of bans when it was unpublished is now searching for more press attention as it gets published. As I wrote, that study actually provided better evidence against bans than for them, due to near-zero effect sizes, but in this climate, watch for advocates to ignore that in favor of more naïve interpretation of “statistical significance” from p-values rather than effect sizes.
Meanwhile, more studies are finding such bans don’t work, yet those studies tend to get less news attention. Be alert for more moral panic shenanigans before people come to their senses.
This is a tragically common problem of social science as I’ve covered elsewhere, one that keeps social science more or less a pseudoscience as scholars refuse to correct it.
Yondr pouches are an excellent example of how industries will try to capitalize on moral panics by profiting from people’s fear.
Again, a common problem in social science, in all fairness. Almost certainly the result of bad training being passed down from generation to generation.
Effect sizes aren't taken seriously in many fields. And the problem becomes even worse when scientists, instead of returning to the drawing board to create better studies, decrease the quality criteria. Like the issue you have talked about before. The sprinkle dust fallacy of small, practically insignificant effect sizes being treated as somehow useful when applied over millions of people, without that idea ever being tested as being true.
Indeed, it's what I like to call "The Incredible Shrinking Effect Size", where the more studies are done over time, the smaller the effect size becomes over time, once the "early adopter effect" wears off. And in this case, it was probably just noise all along.