A New Study Found that Cellphone Bans in Schools Don't Work, but Was the Study Any Good?
The answer to that probably depends on what you thought of cellphone bans in schools in the first place....
A few weeks ago, a study was released which found that cellphone bans in schools don’t work. The study examined 30 schools in the UK, 20 of which engaged in some form of cellphone restriction, 10 which did not. The study found no evidence that restricting student access to cellphones improved their wellbeing nor grades in reading and math.
Naturally, such a study landed like a lead balloon among individuals for whom cellphone bans are treated like received wisdom. The whole issue of cellphones/smartphones/social media has become very strange. Unlike prior media moral panics in which the scientific community often played a big and unhelpful role, current concerns about kids and phones has largely charged ahead of anything like a scientific consensus. Indeed, this recent study has brought out plenty of “We don’t need science to tell us phones are bad!” comments, so I at least appreciate the clarity in perspective. Also there are big, messianic personalities involved, and opinions seem to be partially fanboyed1, and this issue appears to be wrapped into other social debates such as the trans youth medicine debate. All my way of saying, it feels like a lot of people are struggling to keep two thoughts in their head at once and how people feel about this new study likely reflects their a priori position on cellphones more generally2.
But a few people have been asking my thoughts about it. Here goes.
First though, I want to draw people to a somewhat different opinion on the study. Here’s a radio interview featuring Pete Etchells, who’s an excellent researcher with a sharp eye. I have a somewhat different opinion on the study than Pete…I am a bit more sanguine about it and think Pete is overgenerous with teachers’ ad hoc observations, but I’ll get to that. But I think his opinion is worth heeding, so I offer this as a reputable contrast to my own.
As noted, the original article examined the wellbeing of 1227 British schoolchildren aged 12-15 across schools in the UK. 20 of these schools had restrictive cellphone policies which the authors describe as, “In restrictive schools, phones were not allowed to be used during the school day for recreational purposes, and were required to be kept off inside bags, stored in lockers, kept in a pouch, handed into the school reception, or phones were not allowed onto the school premises altogether. In permissive schools, phones were permitted to be used at any time or at certain times (e.g., breaks/lunch) and/or in certain zones (e.g., outside).” This distinction is part of the subsequent debate, so I’ll return to that.
First, reading the literature review, I didn’t have the impression that the authors are on the “skeptical” side of the smartphone/social media debate. If anything, they appear to be the opposite, albeit maybe not with the same fervor as some. They do allow that moderate use of smartphones could be good, but then add “At higher levels of use the reverse effect tends to be seen, with increasing time spent on phones and social media associated with decreasing levels of mental wellbeing and higher levels of anxiety, and depression.” This would, in my opinion, be an overstatement at best…an overstatement, like so many in the social science, largely bred from an ignorance of effect sizes (and failure to mention them at all). In fact, evidence increasingly finds that time spent on smartphones and social media play little role in youth mental health, particularly once other factors are controlled in appropriate statistical analysis3. I wish scholars wrote more honest literature reviews but point being here…these folks aren’t on the skeptical side. I didn’t get the impression they’re necessarily rabidly anti-phone people either, but reading this I’d guess their “researcher expectancy effects” would have been to find evidence in support of cellphone bans.
The main critique I’ve found compelling for this study is that the study created a binary split of restrictive/no restrictive schools when, in fact, the schools have a wide range of policies. Now, it’s important to note that even schools in the lower-to-mid range of restrictive policies have taken to claiming immediate and dramatic positive outcomes (albeit without providing any evidence for those claims beyond the anecdotal). For instance, the local school district in Orange County, FL, restricts youth from taking phones out of their bags at any point during the day, including lunch or other outside-of-class activities. Since students aren’t required to keep their phones in special locked pouches, it could be argued this is less restrictive than ideal, but Orange County Schools claims this policy was an incredible success within mere months. If they’re telling the truth, it should reflect in this kind of data.
As such, the argument goes, if only schools were maximally draconian, we’d see clearer evidence for effects. Hey, why stop at locked pouches: after all, kids could just smuggle phones in without putting them in the pouches4. Let’s pat down every student or send them through daily x-ray scanners. Howabout full body cavity searches for the particularly shifty looking students? This feels a bit like a moving goalposts argument. One could always argue that even more restrictions could have produce clearer results. But maybe at some point those restrictions themselves are starting to seem obsessive, weird, and creepy?
I’m burying the lede a bit too. The authors actually addressed this concern anyway with a sensitivity analysis. They reanalyzed the data using only those schools (n = 4) with the most restrictive policies. As they say, “In the sensitivity analysis using only restrictive schools where phones were inaccessible to pupils, there were no significant differences with permissive schools across all outcomes.”
I guess some might argue that 4 is a fairly small number of schools to generalize from, and fair enough. But it also feels like the anti-phone side are grasping at straws a bit here too. It doesn’t feel like there’s much “there” there.
Even without the sensitivity analysis though…I dunno, I think it’s fair to say we’d see higher effect sizes from more restrictive policies if this kind of thing worked. But even with the lesser policies or mid-range ones like at Orange County Schools, I think we’d see some difference, just less of a difference. Outcomes aren’t necessarily binary.
The authors do find some correlational analyses finding correlation between smartphone use outside of schools and negative wellbeing outcomes. However, these are only bivariate correlations, with no controls for other child/family variables (which would be the scientific standard)5. Some of those correlations I’d argue are trivial, likely noise, but a few of them even exceed the level I’d argue is clinically significant (r = .20). But, given that they don’t employ very good controls, I’m not sure how much we can make of them.
In his interview, Pete and the reporter brought up the issue of teachers often reporting positive outcomes, even though these claims are not being supported in research studies. Pete and the interviewer came to the very generous conclusion that maybe the teachers are seeing different things than what is being measured in research studies. I didn’t find this line of argument compelling, however.
Most things related to “wellbeing”…anxiety, depression, self-esteem, happiness, life satisfaction, etc., tend to intercorrelate very highly, so much so they’re probably all facets of a single blob-like construct. Psychology/Psychiatry have an unfortunate tendency to treat concepts as discrete and clearly defined when they are neither of these things and I think this sometimes results in concept Balkanization and that, in turn, leads to cherry-picking. I note that people happily conflate these concepts when convenient, but they then employ selective rigor when otherwise convenient.
More likely, the teachers are simply full of crap. Not in an intentional way, but people tend to see what they want to see or expect to see. This kind of anecdotal/observational impression stuff is notoriously unreliable. Testimonials from older adults have been common to moral panics since the time of the ancient Athenians. People engage in confirmation bias…it’s part of the human condition. If the testimonials from teachers are not supported by research data, the most prosaic/parsimonious explanation is that the teacher testimonials are nonsense.
Indeed, if this study tells us one thing for sure, it’s that the kinds of dramatic claims of success such as we saw from Orange County Schools, ought to be taken with a pretty large shaker of salt. That’s true even if you think more restrictive policies might work (you can’t have it both ways…arguing this study is flawed on this detail, but Orange County Schools’ claims are still somehow meaningful).
One further comment on the study…some people have criticized it for using causal language when the study is not a randomized controlled trial. I think that’s probably the fairest criticism. Students were not assigned to conditions, nor were schools. Granted, given how policy-makers charged ahead so callously on cellphone bans, I’m not sure a RCT is even possible now. This concern also applies to their second set of “time spent on smartphones” correlations which weren’t very good to begin with, but from which they do appear to make causal assertions. Granted, some of the people criticizing this study have done the exact same thing but it’s still a fair critique.
Overall, I think the study is “fine” but not groundbreaking. Some of the criticism of flaws is fair, but I still think this is one piece of a larger puzzle. I also find some of the criticisms to have the air of motivated reasoning (but hey we’re all human, myself included), as well as selective demands for rigor. Some of the folks criticizing this study (fair or not) I’ve seen turn around and uncritically proclaim the virtues of some pretty weak evidence so long as it happens to fall on “their” side of this debate. Alas, though, once again…such is human nature.
It’s worth noting cellphone bans never were based in good science. At this point, I think the evidence that smartphones and social media are not a primary driver of what’s going on with the US’ mental health crisis (which doesn’t only or even primarily affect teens) is pretty solid. Whether cellphones harm class performance/grades…I think the evidence there is less clear either way, although some adjacent literature has not been promising vis-a-vis cellphone bans.
So, do I think this study is perfect? Far from it. Do I think it is fatally flawed as some suggest? Also, no. Will it end these debates? Certainly not. I don’t even think an RCT would. No single study should.
Perhaps more depressing is that these types of debates don’t really operate on evidence anyway. Moral panics thrive off of “feels” with evidence accepted so long as it supports the panic, with people rejecting even the need for science when it does not. At least people have been more open about that latter issue this time around. Once again, opinions will change once the old people die.
This seems particularly true among the “anti-woke” side of this very tri-partisan issue, albeit maybe that’s only my perspective because I know more people in that community.
My general sense is nobody is immune to this, scholars included. I’m doing my best to take a fresh eye on this study, but I’m obviously a skeptic of technopanics and I think the most transparent thing I can do is acknowledge that upfront and let you, dear reader, decide how to interpret that.
Part of the tragedy of this whole field is how people became obsessed with “time spent” and abstinence only approaches, rather than considering how people used smartphones and social media which might open us up to some kind of media literacy approach rather than bans. Alas, that moment slipped away from us around 2018 or so…
This is also one reason I’m skeptical of cellphone bans…I think they mainly set of challenges for youth to circumvent them. But maybe like hiding food for animals living in zoo cages to enrich their minds, maybe giving school kids a challenge too is good! What’s more fun than pulling one over on teachers, after all?
To be fair, they do control for some school variables, but I didn’t think these were terribly theoretically relevant…what I’d call an undersaturated model. I don’t think things like school religious affiliation matter much as a control variable. They do include an income deprivation index which appears to be at the school level. But I’d want to see child-level variables controlled…family environment, bullying victimization, neurotic personality traits, etc. Ideally, we’d also control for genetics given studies that now find that media habits and behavioral/mental health outcomes are often explained by underlying genetic variance, not causal effects from media itself.