The reliability of science is increasingly under fire. We all know that media often gives a distorted picture of scientific findings (i.e. Hot news: Curry, Curcumin, Cancer & cure). But there is also an ever growing number of scientific misreports or even fraud (see bmj editorial announcing retraction of the Wakefield paper about causal relation beteen MMR vaccination and autism). Apart from real scientific misconduct there are Ghost Marketing and “Publication Bias”, that makes (large) positive studies easier to find than those with negative or non-significant result.
Then there are also the ever growing contradictions, that makes the public sigh: what IS true in science?
Indeed according to doctor John Ioannidis “Much of what medical researchers conclude in their studies is misleading, exaggerated, or flat-out wrong. (see “Lies, Damned Lies, and Medical Science” in the Atlantic (2010). In 2005 he wrote the famous PLOS-article “Why most published research findings are false”  .
With Iaonnides as an editor, a new PLOS-one paper has recently been published on the topic . The authors Gonon, Bezard and Boraud state that there is often a huge gap between neurobiological facts and firm conclusions stated by the media. They suggest that the misrepresentation often starts in the scientific papers, and is echoed by the media.
Although this article has already been reviewed by another researchblogger (Hadas Shema), I would like to give my own views on this paper
Gonon et al found 3 types of misrepresentation.*
1. Internal inconsistencies (between results and claimed conclusions).
In a (non-systematic) review of 360 ADHD articles Gonon et al.  found two studies with “obvious” discrepancies between results and claimed conclusions. One paper claimed that dopamine is depressed in the brain of ADHD patient. Mitigations were only mentioned in the results section and of course only the positive message was resonated by the media without further questioning any alternative explanation (in this case a high baseline dopamine tone). The other paper  claimed that treatment with stimulant medications was associated with more favorite long-term school outcomes. However the average reading score and the school drop-outs did not differ significantly between treatment and control group. The newspapers also trumpeted that ”ADHD drugs help boost children’s grades” .
2. Fact Omission
To quantify fact omission in the scientific literature, Gonon et al systematically searched for ADHD articles mentioning the the D4 dopamine receptor (DRD4) gene. Among the 117 primary human studies with actual data (like odds ratios), 74 articles state in their summary that alleles of the DRD4 genes are significantly associated with ADHD but only 19 summaries mentioned that the risk was small. Fact omission was even more preponderant in articles, that only cite studies about DRD4. Not surprisingly, 82% of the media articles didn’t report that the DRD4 only confers a small risk either.
In accordance with Ioannidis findings  Gonon et al found that the most robust effects were reported in initial studies: odds-ratios decreased from 2.4 in the oldest study in 1996 to 1.27 in the most recent meta-analysis.
3. Extrapolating basic and pre-clinical findings to new therapeutic prospects
Animal ADHD models have their limitations because investigations based on mouse behavior cannot capture the ADHD complexity. Analysis of all ADHD-related studies in mice showed that 23% of the conclusions were overstated. The frequency of this overstatement was positively related with the impact factor of the journal.
Again, the positive message was copied by the press. (see Figure below)Discussion
The article by Gonon et al is another example that “published research findings are false” [ 2], or at least not completely true. The authors show that the press isn’t culprit number one, but that it “just” copies the overstatements in the scientific abstracts.
The merit of Gonon et al is that they have extensively looked at a great number of articles and at press articles citing those articles.
The first type of misrepresentation wasn’t systematically studied, but types 2 and 3 misrepresentations were studied by analyzing papers on a specific ADHD topic obtained by a systematic search.
One of the solutions the authors propose is that “journal editors collectively reject sensationalism and clearly condemn data misrepresentation”. I agree and would like to add that the reviewers should check that the summary actual reflects the data. Some journals already have strict criteria in this respect. It striked me that the few summaries I checked were very unstructured and short, unlike most summaries I see. Possibly, unstructured abstracts are more typically for journals about neuroscience and animal research.
The choice of the ADHD-topics investigated doesn’t seem random. A previous review, written by Francois Gonon deals entirely with “the need to reexamine the dopaminergic hypothesis of ADHD” . The type 1 misrepresentation data stem from this opinion piece.
The putative ADHD-DRD4 gene association and the animal studies, taken as examples for type 2 and type 3 misrepresentations respectively, can also be seen as topics of the “ADHD is a genetic disease” -kind.
Gonon et al clearly favor the hypothesis that ADHD is primarily caused by environmental factors . In his opinion piece he starts with saying:
This dopamine-deﬁcit theory of ADHD is often based upon an overly simplistic dopaminergic theory of reward. Here, I question the relevance of this theory regarding ADHD. I underline the weaknesses of the neurochemical, genetic, neuropharmacological and imaging data put forward to support the dopamine-deﬁcit hypothesis of ADHD. Therefore, this hypothesis should not be put forward to bias ADHD management towards psychostimulants.
I wonder whether it is fair of the authors to limit the study to ADHD topics they oppose to in order to (indirectly) confirm their “ADHD has a social origin” hypothesis. Indeed in the paragraph “social and public health consequences” Gonon et al state:
Unfortunately, data misrepresentation biases the scientific evidence in favor of the first position stating that ADHD is primarily caused by biological factors.
I do not think that this conclusion is justified by their findings, since similar data misrepresentation might also occur in papers investigating social causes or treatments, but this was not investigated. (mmm, a misrepresentation of the third kind??)
I also wonder why impact factor data were only given for the animal studies.
Gonon et al interpret a lot, also in their results section. For instance, they mention that 2 out of 60 articles show obvious discrepancies between results and claimed conclusions. This is not much. Then they reason:
Our observation that only two articles among 360 show obvious internal inconsistencies must be considered with caution however. First, our review of the ADHD literature was not a systematic one and was not aimed at pointing out internal inconsistencies. Second, generalization to other fields of the neuroscience literature would be unjustified
But this is what they do. See title:
” Misrepresentation of Neuroscience Data Might Give Rise to Misleading Conclusions in the Media.”
Furthermore they selectively report themselves. The Barbaresi paper , a large retrospective cohort, did not find an effect on average reading score and school drop-outs, but it did find a significantly lowered grade retention, which is -after all- an important long-term school outcome.
Misrepresentation type 2 (“omission”) I would say.*
- Gonon, F., Bezard, E., & Boraud, T. (2011). Misrepresentation of Neuroscience Data Might Give Rise to Misleading Conclusions in the Media: The Case of Attention Deficit Hyperactivity Disorder PLoS ONE, 6 (1) DOI: 10.1371/journal.pone.0014618
- Ioannidis, J. (2005). Why Most Published Research Findings Are False PLoS Medicine, 2 (8) DOI: 10.1371/journal.pmed.0020124
- Barbaresi, W., Katusic, S., Colligan, R., Weaver, A., & Jacobsen, S. (2007). Modifiers of Long-Term School Outcomes for Children with Attention-Deficit/Hyperactivity Disorder: Does Treatment with Stimulant Medication Make a Difference? Results from a Population-Based Study Journal of Developmental & Behavioral Pediatrics, 28 (4), 274-287 DOI: 10.1097/DBP.0b013e3180cabc28
- GONON, F. (2009). The dopaminergic hypothesis of attention-deficit/hyperactivity disorder needs re-examining Trends in Neurosciences, 32 (1), 2-8 DOI: 10.1016/j.tins.2008.09.010
- Hot news: curry, curcumin, cancer, cure/ (http://laikaspoetnik.wordpress.com: 2009/11/03/)
- Mercks ghostwriters, haunted papers and fake elsevier journals (laikaspoetnik.wordpress.com: 2009/05/08/)
- Kids With ADHD Often Have Other Problems (webmd.com)
- Wakefield’s MMR work takes another kicking (blogs.nature.com)
- MMR scandal: Wakefield accused of ‘deliberate fraud’ (newscientist.com)
- “Piltdown medicine” and Andrew Wakefield’s MMR vaccine fraud (sciencebasedmedicine.org)
- Study: ADHD Diet Helps Reduce Symptoms (webmd.com)
- Most ADHD Kids Have Multiple Conditions (nlm.nih.gov)
- Childhood ADHD and Food “Sensitivity” (brainposts.blogspot.com)
[*A short comment in the NRC Handelsblad (Febr 5th) comes to a similar conclusion]