Are 90% of all medical studies wrong — including nearly half of those claimed to be the most reliable? That’s the provocative claim made by researcher John Ioannidis, profiled in this month’s issue of The Atlantic.
Atlantic contributor David Freedman writes:
[Ioannidis] zoomed in on 49 of the most highly regarded research findings in medicine over the previous 13 years, as judged by the science community’s two standard measures: the papers had appeared in the journals most widely cited in research articles, and the 49 articles themselves were the most widely cited articles in these journals. These were articles that helped lead to the widespread popularity of treatments such as the use of hormone-replacement therapy for menopausal women, vitamin E to reduce the risk of heart disease, coronary stents to ward off heart attacks, and daily low-dose aspirin to control blood pressure and prevent heart attacks and strokes.
Ioannidis was putting his contentions to the test not against run-of-the-mill research, or even merely well-accepted research, but against the absolute tip of the research pyramid. Of the 49 articles, 45 claimed to have uncovered effective interventions. Thirty-four of these claims had been retested, and 14 of these, or 41 percent, had been convincingly shown to be wrong or significantly exaggerated. If between a third and a half of the most acclaimed research in medicine was proving untrustworthy, the scope and impact of the problem were undeniable. That article was published in the Journal of the American Medical Association.
What are doctors and patients supposed to do with this information? Toss darts to determine what treatments to use? (More on Time.com: An Interview with David Freedman)
All is not lost. Fortunately, there is a major group of researchers who have been addressing this very issue for nearly two decades. Known as the Cochrane Collaboration, its members produce rigorous reviews of multiple studies, excluding those with severe methodological problems and summing up the results of the rest. The outcome is powerful meta-analyses whose findings elucidate what the weight of the best evidence shows when it comes to medical treatments.
It was Cochrane’s research that showed that doing breast self-exams don’t reduce breast-cancer death, and may even do harm. Cochrane reviewers have also suggested that there isn’t enough evidence to show that getting a flu vaccine reduces infection or flu-related death (though government and public-health experts might disagree).
Funded by governments, universities and donations, Cochrane researchers do not take pharmaceutical company money. Ioannidis himself has participated in developing some of the Collaboration’s review methods and his work is utilized by the group. (More on Time.com: How Not to Get Sick.)
I think the Atlantic article is exceptional journalism, and it illustrates the importance of examining research carefully and not accept anything uncritically — as either a doctor or a consumer. But it would be nice if such stories would at least mention that there are major organizations that are currently trying and succeeding at addressing the problem.
More on Time.com:
Health Checkup: How to Live to 100 Years Old