In a previous post  I reviewed a recent Dutch study published in the New England Journal of Medicine (NEJM  about the effects of sugary drinks on the body mass index of school children.
The study got widely covered by the media. The NRC, for which the main author Martijn Katan works as a science columnist, columnist, spent two full (!) pages on the topic -with no single critical comment-.
As if this wasn’t enough, the latest column of Katan again dealt with his article (text freely available at mkatan.nl).
I found Katan’s column “Col hors Catégorie”  quite arrogant, especially because he tried to belittle a (as he called it) “know-it-all” journalist who criticized his work in a rivaling newspaper. This wasn’t fair, because the journalist had raised important points [5, 1] about the work.
The piece focussed on the long road of getting papers published in a top journal like the NEJM.
Katan considers the NEJM as the ”Tour de France” among medical journals: it is a top achievement to publish in this paper.
Katan also states that “publishing in the NEJM is the best guarantee something is true”.
I think the latter statement is wrong for a number of reasons.*
- First, most published findings are false . Thus journals can never “guarantee” that published research is true.
Factors that make it less likely that research findings are true include a small effect size, a greater number and lesser preselection of tested relationships, selective outcome reporting, the “hotness” of the field (all applying more or less to Katan’s study, he also changed the primary outcomes during the trial), a small study, a great financial interest and a low pre-study probability (not applicable) .
- It is true that NEJM has a very high impact factor. This is a measure for how often a paper in that journal is cited by others. Of course researchers want to get their paper published in a high impact journal. But journals with high impact factors often go for trendy topics and positive results. In other words it is far more difficult to publish a good quality study with negative results, and certainly in an English high impact journal. This is called publication bias (and language bias) . Positive studies will also be more frequently cited (citation bias) and will more likely be published more than once (multiple publication bias) (indeed, Katan et al already published about the trial , and have not presented all their data yet [1,7]). All forms of bias are a distortion of the “truth”.
(This is the reason why the search for a (Cochrane) systematic review must be very sensitive  and not restricted to core clinical journals, but even include non-published studies: for these studies might be “true”, but have failed to get published).
- Indeed, the group of Ioannidis just published a large-scale statistical analysis showing that medical studies revealing “very large effects” seldom stand up when other researchers try to replicate them. Often studies with large effects measure laboratory and/or surrogate markers (like BMI) instead of really clinically relevant outcomes (diabetes, cardiovascular complications, death)
- More specifically, the NEJM does regularly publish studies about pseudoscience or bogus treatments. See for instance this blog post  of ScienceBased Medicine on Acupuncture Pseudoscience in the New England Journal of Medicine (which by the way is just a review). A publication in the NEJM doesn’t guarantee it isn’t rubbish.
- Importantly, the NEJM has the highest proportion of trials (RCTs) with sole industry support (35% compared to 7% in the BMJ)  . On several occasions I have discussed these conflicts of interests and their impact on the outcome of studies ([13, 14; see also [15,16] In their study, Gøtzsche and his colleagues from the Nordic Cochrane Centre  also showed that industry-supported trials were more frequently cited than trials with other types of support, and that omitting them from the impact factor calculation decreased journal impact factors. The impact factor decrease was even 15% for NEJM (versus 1% for BMJ in 2007)! For the journals who provided data, income from the sales of reprints contributed to 3% and 41% of the total income for BMJ and The Lancet.
A recent study, co-authored by Ben Goldacre (MD & science writer)  confirms that funding by the pharmaceutical industry is associated with high numbers of reprint orders. Again only the BMJ and the Lancet provided all necessary data.
- Finally and most relevant to the topic is a study , also discussed at Retractionwatch, showing that articles in journals with higher impact factors are more likely to be retracted and surprise surprise, the NEJM clearly stands on top. Although other reasons like higher readership and scrutiny may also play a role , it conflicts with Katan’s idea that ”publishing in the NEJM is the best guarantee something is true”.
I wasn’t aware of the latter study and would like to thank drVes and Ivan Oranski for responding to my crowdsourcing at Twitter.
@laikas re: "NEJM publishing is the best guarantee something is true" - Incorrect, of course. Just look at the retractions, etc.—
Ves Dimov, M.D. (@DrVes) October 20, 2012
Ivan Oransky (@ivanoransky) October 20, 2012
- Sugary Drinks as the Culprit in Childhood Obesity? a RCT among Primary School Children (laikaspoetnik.wordpress.com)
- de Ruyter JC, Olthof MR, Seidell JC, & Katan MB (2012). A trial of sugar-free or sugar-sweetened beverages and body weight in children. The New England journal of medicine, 367 (15), 1397-406 PMID: 22998340
- NRC Wim Köhler Eén kilo lichter.NRC | Zaterdag 22-09-2012 (http://archief.nrc.nl/)
- Martijn Katan. Col hors Catégorie [Dutch], (published in de NRC, (20 oktober)(www.mkatan.nl)
- Hans van Maanen. Suiker uit fris, De Volkskrant, 29 september 2012 (freely accessible at http://www.vanmaanen.org/)
- Ioannidis, J. (2005). Why Most Published Research Findings Are False PLoS Medicine, 2 (8) DOI: 10.1371/journal.pmed.0020124
- Changes to the protocol http://clinicaltrials.gov/archive/NCT00893529/2011_02_24/changes
- Publication Bias. The Cochrane Collaboration open learning material (www.cochrane-net.org)
- de Ruyter JC, Olthof MR, Kuijper LD, & Katan MB (2012). Effect of sugar-sweetened beverages on body weight in children: design and baseline characteristics of the Double-blind, Randomized INtervention study in Kids. Contemporary clinical trials, 33 (1), 247-57 PMID: 22056980
- Pereira, T., Horwitz, R.I., & Ioannidis, J.P.A. (2012). Empirical Evaluation of Very Large Treatment Effects of Medical InterventionsEvaluation of Very Large Treatment Effects JAMA: The Journal of the American Medical Association, 308 (16) DOI: 10.1001/jama.2012.13444
- Acupuncture Pseudoscience in the New England Journal of Medicine (sciencebasedmedicine.org)
- Lundh, A., Barbateskovic, M., Hróbjartsson, A., & Gøtzsche, P. (2010). Conflicts of Interest at Medical Journals: The Influence of Industry-Supported Randomised Trials on Journal Impact Factors and Revenue – Cohort Study PLoS Medicine, 7 (10) DOI: 10.1371/journal.pmed.1000354
- One Third of the Clinical Cancer Studies Report Conflict of Interest (laikaspoetnik.wordpress.com)
- Merck’s Ghostwriters, Haunted Papers and Fake Elsevier Journals (laikaspoetnik.wordpress.com)
- Lexchin, J. (2003). Pharmaceutical industry sponsorship and research outcome and quality: systematic review BMJ, 326 (7400), 1167-1170 DOI: 10.1136/bmj.326.7400.1167
- Smith R (2005). Medical journals are an extension of the marketing arm of pharmaceutical companies. PLoS medicine, 2 (5) PMID: 15916457 (free full text at PLOS)
- Handel, A., Patel, S., Pakpoor, J., Ebers, G., Goldacre, B., & Ramagopalan, S. (2012). High reprint orders in medical journals and pharmaceutical industry funding: case-control study BMJ, 344 (jun28 1) DOI: 10.1136/bmj.e4212
- Fang, F., & Casadevall, A. (2011). Retracted Science and the Retraction Index Infection and Immunity, 79 (10), 3855-3859 DOI: 10.1128/IAI.05661-11
- Is it time for a Retraction Index? (retractionwatch.wordpress.com)
- Agrawal A, & Sharma A (2012). Likelihood of false-positive results in high-impact journals publishing groundbreaking research. Infection and immunity, 80 (3) PMID: 22338040
* Addendum: my (unpublished) letter to the NRC