Those were heady days.
Of course you’d be hard-pressed to find many Pokémon hunters at the local PokéStop nowadays. And yet in the summer of 2016, we tracked all sorts of health claims related to the app, including one that was based on a “study” that didn’t even exist.
Monday’s reporting on a new video game that aims to become a treatment for Attention Deficit Hyperactivity Disorder (ADHD) brings to mind similar concerns about a phantom study.
To be clear, a study about this video game most certainly exists and it’s a randomized controlled trial (RCT). That’s the good news — and I share enthusiasm for a non-drug approach to ADHD that’s being tested rigorously by researchers.
A screenshot from the Akili video game.
The bad news is that nobody outside of the company sponsoring the research has seen the results of the study.
And yet some news outlets are reporting on those unseen results in a way that makes the product sound like it will be a big help for children with ADHD. We don’t have enough information to draw that conclusion.
What we do have is a company news release that reports the video game produced “a statistically significant improvement compared to an active control” on one measure of attention in children who used it.
But how big was that improvement actually? And will it translate to noticeable and meaningful benefits on children’s’ behavior and ability to learn? The news release doesn’t tell us. It says the full results are to be presented at an upcoming meeting.
Boston Business Journal didn’t need to see the data in order to declare the study a “landmark” in its headline.
Reuters poured on the accolades and informed us that the company developing the treatment, Akili, “now plans to file for regulatory approval with the U.S. Food and Drug Administration in the first half of 2018, paving the way for what would be the first such ‘digital’ prescription product.”
STAT‘s only quoted source was Alkili’s CEO, Eddie Martucci, who said: “We have something that looks and feels and is delivered through a video game, but when someone’s using it, they’re getting a direct physiological activation that will lead hopefully — and we have a nice glimpse of data now — to cognitive and general clinical improvement.”
Actually, though, readers haven’t had a glimpse of any of that data. Nor have external experts. We’re being asked to take the company CEO’s word that these data suggest “general clinical improvement.”
The trial data haven’t been presented, published, or peer-reviewed, something none of these outlets cautioned against in their coverage. Nor did they attempt to compensate for this by seeking out an independent perspective on the findings.
Susan Molchan, MD
And STAT does a nice job of providing important context when it warns, for example, “it remains to be seen whether clinicians and insurers will embrace [the video game].”
But incomplete reporting based on news releases is a recurring issue and it’s worth noting that there are remedies. In this case, had any of these outlets reached out to an independent source, they might have heard concerns similar to what psychiatrist and former NIH clinical researcher Susan Molchan, MD, told me.
She says the primary outcome touted in the news release appears to be “a simple computer-generated neuropsychological task” and that the FDA would likely want to see a much broader range of assessments — including “more general scales (clinical global impression scales) that…try to capture whether there is an overall general improvement in what is going on in someone’s life.” She adds that it is premature to describe these results as “paving the way” for an FDA-approved digital prescription product as Reuters puts it.
Alan Schroeder, MD
“There’s the question of ‘statistically significant’ as found here, vs clinically significant—does this make a difference in behavior in the classroom, which the scales try to get at, or ultimately grades? We have no idea,” Molchan says.
Alan Schroeder, MD, the associate chief for research in the division of pediatric hospital medicine at Lucile Packard Children’s Hospital Stanford, expressed enthusiasm for the research in general and said he was pleased to see a company investigating non-drug approaches to ADHD. But he also raised concerns about reporting on the findings before peer review and said there were questions about the results. For example, “while the choice of the control group is understandable, is it possible that that video game used in the control arm may have been counterproductive? It would have been interesting to see a 3rd comparator arm of ‘usual care.'”
A final concern brings us back to Pokémon Go and its ultimately fleeting health impact. The new study lasted just four weeks, and that’s not enough time to tell if this product is going to achieve lasting and meaningful benefits for children with ADHD.
“There is the question of whether the kids would need to keep playing the game, and how often, to sustain the response (if there is one that matters),” Molchan says.
People get bored with video games. Even one that’s engaging enough to become a global sensation may quickly lose favor.
It’s a perspective worth keeping in mind as we assess this exciting but uncertain new area of psychological research.
“It’s great that non-drug interventions are being tested in RCTs,” Molchan says, “but there’s a lot of work left to do.”