Studies that show a statistical association between two things do not necessarily prove that one thing causes another to occur.
We saw that principle violated several times in news coverage of several different studies today – regarding coffee and sleep.
We will have a more in-depth systematic criteria-driven review of one of the coffee stories, but here’s a quick overview. As usual, the problem resided in the headlines of most stories, with some half-hearted attempts in some stories to include the limitations of two observational studies, published in the Annals of Internal Medicine. The conclusion of one study emphasized association, not causation:
Higher consumption of coffee was associated with lower risk for death in African Americans, Japanese Americans, Latinos, and whites.
Coffee drinking was associated with reduced risk for death from various causes.
What were the limitations of the studies?These studies were based on a single report of how often the participants drank coffee, which may have changed over the years they were followed and might not have been accurate. In addition, although the researchers tried to account for this, people who say they drink a lot of coffee may differ from those who do not in other ways that may affect their health.
An editorial accompanying the published results of the studies in the Annals stated that the protective qualities of coffee was “biologically plausible” but “the association of coffee intake with mortality, however, was modest and sensitive to confounding.” In other words, other things may account for the statistical association. Cause and effect was not established. So any headline you see like the following is wrong:
CNN: “Drinking more coffee leads to a longer life, two studies say.” (“Leads to” implies cause and effect.)
ABC News: “Java drinkers, rejoice: Coffee may help you live longer” (Or it may not. This study didn’t prove anything either way.)
The Guardian: “Coffee cuts risk of dying from stroke and heart disease, study suggests.” (Cutting risk implies that cause-and-effect has been established. It has not.)
How could it have been stated better? Read our primer, “Observational studies: Does the language fit the evidence?”
Our associate editor, Jill Adams, who does the daily searches for stories eligible for our review, wrote to our team this morning:
Four of our regularly reviewed outlets failed to make the point that association does not equal causation in covering a recent study that found people with a sense of purpose in life sleep better. The lead researcher certainly didn’t help, saying things like: “Helping people cultivate a purpose in life could be an effective drug-free strategy to improve sleep quality, particularly for a population that is facing more insomnia.”
The study found purposeful subjects reported less sleep-associated problems such as apnea and restless legs. Well geez, perhaps sleep problems lead to difficulties maintaining a sense of purpose.
The stories she found were:
US News & World Report was one of the news organizations using the researcher quote: “”Helping people cultivate a purpose in life could be an effective drug-free strategy to improve sleep quality, particularly for a population that is facing more insomnia,” (the researcher) added. “Purpose in life is something that can be cultivated and enhanced through mindfulness therapies.”
The Guardian failed to explore the inherent limitations of relying on recall and questionnaires, when it reported:
“More than 800 people aged 60 to 100 took part in the study and answered questions on their sleep quality and motivations in life. To assess their sense of purpose, the participants were asked to rate statements such as: ‘I feel good when I think of what I’ve done in the past and what I hope to do in the future’.”
And TIME’s headline, “Having a sense of purpose may help you sleep at night,” must be countered with the possibly-equally-true statement, “Or not.”
So, today’s news readers, move on. There’s probably something in political news today that’s worth more of your time.