P C R : Publish, Cite Cyagen and get a Reward of $100 or more !?

Thursday I got an odd mail from Cyagen.

Cyagen offered me $100 or more in reward for citing their animal model services in an article. Even more surprising the reward was dependent on the impact factor of the journal I was publishing in.

Cyagen gives the following example:

If you published a paper in Science (IF = 30) and cite Cyagen Biosciences, you will be entitled to a voucher with a face value of $3,000 upon notification of the publication (PMID).

Thus the higher the impact factor the higher the reward.

Click to enlarge

First I thought this was link bait, because the mail was send from vresp.com. (spam reputation) and not cyagen.com.

But tweets from @doctorsdilemma and @PhytophthoraLab pointed me to this advertorial at Cyagen’s website.

It was real.

This doesn’t feel right, because it “smells like a conflict of interest”.

In clinical research we all know where conflicts of interest can lead to: biased results. Or according to Corollary 5 in the famous Iaonnides paper “Why Most Published Research Findings Are False”:

“The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.”

We expressed our concerns on Twitter but it took a while for Cyagen to respond to our concern (see Facebook-Q&A *).

@laikas @doctorsdilemma @PhytophthoraLab @bengoldacre @rvosa Thank you for being patient while we respond. https://t.co/ieQvO3yCCe

— Cyagen Biosciences (@CyagenBio) August 14, 2015

Meanwhile Ben Goldacre from BadScience and Retraction Watch wrote a post about Cyagen’s “pay-for-citation program”.

Ben Goldacre emphasizes the COI-problem:

I would imagine that this is something journal editors will be interested in, and concerned by. We worry about “conflict of interest” a lot in science, and especially medicine: if someone has shares in a company, or receives money from it, and their publication discusses that company’s products, then this needs to be declared in the paper. If you get funding for a study then, again, you must declare it. If you have received payment for making an academic citation, then in my view this is clearly a source of funds that should be declared.

The most problematic point, according to Goldacre, is that (part of) the researchers “have received funds from Cyagen, in exchange for an academic citation.”  He keeps an archive of the 164 papers that cite Cyagen, so one can always check whether “any one of them has declared receiving money from the company”.
(because in his opinion they should)

Some of the commenters to Goldacres’ post go even further and state:

“It’s one thing to fund a study. It’s another thing entirely to pay someone to write what you want — that’s not science, it’s PR. You’re not being puritanical at all, this is corruption, plain and simple.”

and

“This is an old and very well researched problem. TV makers get paid to have the popular detective drive a certain make of car, scoff a certain brand of drink and so on. That this kind of advertising is not innocent and that it works is beyond doubt. The media have established firm rules about this, science journals need just copy them.”

Although Retraction Watch lets both sides be heard, the title of its post “Researchers, need $100? Just mention Cyagen in your paper!” suggests researchers are being bribed by Cyagen to mention its product anywhere in the paper.

I agree that the Cyagen incentive is a (very curious) marketing trick, but  it has nothing to do with ghostwriting or corruption and IMHO the authors of the 164 papers are not to blame and need not declare anything.

Let me explain why.

• First, as everybody familiar with benchwork knows, authors are required to mention the brand and source of the used reagents and tools (including strains of mice). This is for transparency and reproducibility. There can be huge differences between products and it is important to know which enzyme, primer or mouse strain someone has used.
• Authors only MENTION the products, reagents etc. that they have used. Mentioning a brand doesn’t mean an endorsement, although you probably wouldn’t choose a product if you thought it worked suboptimal.
  Indeed the first two papers on the Cyagen citations list both mentionCyagen only ONCE in the materials and methods where it belongs
• The product itself is not being tested, it is just being used, something else is being tested. (it would be different if the discount was for a diet supplement that was being tested)
• The authors don’t receive money, they receive a discount on their next purchase. This is common practice in the lab when you are a regular customer.
• You don’t write COI’s for getting (normal) discounts.

Still, I do think that this incentive is rather inappropriate and not a very wise thing to do (even from a marketing point of view).

It is not the discount that is problematic, neither is the mention of the product’s name and brand in the articles (as this is required).

It is the coupling of the discount (erroneously called “reward”) to “a mention” (erroneously called “citation”) that is not a proper thing to do and even more so, the coupling of the height of the discounts to the impact factor of a journal.

The idea behind it is that Cyagen not only wants to thank its customers, it wants to “increase the number of publications featuring Cyagen as a product or service provider to add strength to our company’s reputation and visible experience”. 

Of course, Cyagen’s reputation is not linearly dependent on the number of the publications and certainly not the citation score. Reputation depends on other things, like the quality of your product, client satisfaction and …  your reputatIon.

It now seems that the entire marketing campaign will just have the opposite effect: it will harm Cyagen’s reputation (and perhaps also that of the researchers).

My advice to Cyagen is to directly remove the incentive from their website and stop the unsolicited mailing.

And please, Cyagen, stop to defend your P(C)R-tactics, just admit it was a wrong approach and learn from it.

* Later I found out that the a Q&A on Facebook had been extracted from an interview a blogger was still in the process of doing with a product manager of Cyagen. Apparently social media etiquette is another thing Cyagen doesn’t master very well.

Update post:2015-08-16

Social Media in Clinical Practice by Bertalan Meskó [Book Review]

How to review a book on Medical Social Media written by an author, who has learned you many Social Media skills himself?

Thanks to people like Bertalan Meskó, the author of the book concerned,  I am not a novice in the field of Medical Social Media.

But wouldn’t it be great if all newcomers in the medical social media field could benefit from Bertalan’s knowledge and expertise? Bertalan Meskó, a MD with a  Summa Cum Laude PhD degree in clinical genomics, has already shared his insights by posts on award-winning blog ScienceRoll, via Twitter and Webicina.com (an online service that curates health-related social media resources), by giving presentations and social media classes to medical students and physicians.

But many of his students rather read (or reread) the topics in a book instead of e-learning materials. Therefore Bertalan decided to write a handbook entitled “Social Media in Clinical Practice”.

This is the table of contents (for more complete overview see Amazon):

1. Social media is transforming medicine and healthcare
2. Using medical search engines with a special focus on Google
3. Being up-to-date in medicine
4. Community sites Facebook, Google+ and medical social networks
5. The world of e-patients
6. Establishing a medical blog
7. The role of Twitter and microblogging in medicine
8. Collaboration online
9. Wikipedia and Medical Wikis
10. Organizing medical events in virtual environments
11. Medical smartphone and tablet applications
12. Use of social media by hospitals and medical practices
13. Medical video and podcast
14. Creating presentations and slideshows
15. E-mails and privacy concerns
16. Social bookmarking
17. Conclusions

As you can see, many social media tools are covered and in this respect the book is useful for everyone, including patients and consumers.

But what makes “Social Media in Clinical Practice” especially valuable for medical students and clinicians?

First, specific medical search engines/social media sites/tools are discussed, like (Pubmed [medical database, search engine], Sermo [Community site for US physicians], Medworm [aggregator of RSS feeds], medical smartphone apps and sources where to find them, Medical Wiki’s like Radiopaedia.
Scientific Social media sites, with possible relevance to physicians are also discussed, like Google Scholar and Wolphram Alpha.

Second, numerous medical examples are given (with links and descriptions). Often, examples are summarized in tables in the individual chapters (see Fig 1 for a random example 😉 ). Links can also be found at the end of the book, organized per chapter.

Fig 1. Examples represented in a Table

Third, community sites and non-medical social media tools are discussed from the medical prespective. With regard to community sites and tools like Facebook, Twitter, Blogs and Email special emphasis is placed on (for clinicians very important) quality, privacy and legacy concerns, for instance the compliance of websites and blogs with the HONcode (HON=The Health On the Net Foundation) and HIPAA (Health Insurance Portability and Accountability Act), the privacy settings in Facebook and Social Media Etiquette (see Fig 2).

Fig. 2 Table from “Social Media in Clinical Practice” p 42

The chapters are succinctly written, well organized and replete with numerous examples. I specifically like the practical examples (see for instance Example #4).

Fig 3 Example of Smartphone App for consumers

Some tools are explained in more detail, i.e. the anatomy of a tweet or a stepwise description how to launch a WordPress blog.
Most chapters end with a self test (questions),  next steps (encouraging to put the theory into practice) and key points.

Thus in many ways a very useful book for clinical practice (also see the positive reviews on Amazon and the review of Dean Giustini at his blog).

Are there any shortcomings, apart from the minimal language-shortcomings, mentioned by Dean?

Personally I find that discussions of the quality of websites concentrate a bit too much on the formal quality (contact info, title, subtitle etc)). True, it is of utmost importance, but quality is also determined by  content and clinical usefulness. Not all websites that are formally ok deliver good content and vice versa.

As a medical  librarian I pay particular attention to the search part, discussed in chapter 3 and 4.
Emphasis is put on how to create alerts in PubMed and Google Scholar, thus on the social media aspects. However searches are shown, that wouldn’t make physicians very happy, even if used as an alert: who wants a PubMed-alert for cardiovascular disease retrieving 1870195 hits? This is even more true for a the PubMed search “genetics” (rather meaningless yet non-comprehensive term).
More importantly, it is not explained when to use which search engine.  I understand that a search course is beyond the scope of this book, but a subtitle like “How to Get Better at Searching Online?” suggests otherwise. At least there should be hints that searching might be more complicated in practice, preferably with link to sources and online courses.  Getting too much hits or the wrong ones will only frustrate physicians (also to use the socia media tools, that are otherwise helpful).

But overall I find it a useful, clearly written and well structured practical handbook. “Social Media in Clinical Practice” is unique in his kind – I know of no other book that is alike-. Therefore I recommend it to all medical students and health care experts who are interested in digital medicine and social media.

This book will also be very useful to clinicians who are not very fond of social media. Their reluctance may change and their understanding of social medicine developed or enhanced.

Lets face it: a good clinician can’t do without digital knowledge. At the very least his patients use the internet and he must be able to act as a gatekeeper identifying and filtering thrustworty, credible and understandable information. Indeed, as Berci writes in his conclusion:

“it obviously is not a goal to transform all physicians into bloggers and Twitter users, but (..) each physician should find the platforms, tools and solutions that can assist them in their workflow.”

If not convinced I would recommend clinicians to read the blog post written at the the Fauquier ENT-blog (refererred to by Bertalan in chapter 6, #story 5) entiteld: As A Busy Physician, Why Do I Even Bother Blogging?

SM in Practice (AMAZON)

Book information: (also see Amazon):

• Title: Social Media in Clinical Practice
• Author: Bertalan Meskó
• Publisher: Springer London Heidelberg New York Dordrecht
• 155 pages
• ISBN 978-1-4471-4305-5
• ISBN 978-1-4471-4306-2 (eBook)
• ISBN-10: 1447143051
• DOI 10.1007/978-1-4471-4306-2
• $37.99 (Sept 2013) (pocket at Amazon)

Related articles

• “Social Media in Clinical Practice” Handbook Is Published! (scienceroll.com)
• Social Media in Clinical Practice: Chapter 1, The Introduction (scienceroll.com)
• Here comes Bertalan Meskó’s book on ‘social media’ (aplaceofmind.ubc.ca)
• My Book, Social Media in Clinical Practice, To Be Released on the 8th of August! (scienceroll.com)

• Berci’s New Book Is A Winner (blogs.ubc.ca/dean/)

• Social Media in Clinical Practice: Chapter 3, Being Up-to-Date in Medicine (scienceroll.com)

Friday Foolery #55 Entrance to the Maternity Ward

This Picture speaks for itself, I suppose

Source: Facebook page of George Takei (link to picture)

Related articles

• Leave it to Takei! (juliannrich.wordpress.com)
• George Takei’s Facebook Adventure Now Includes An Apologizing Ghostwriter (webpronews.com)
• Journalist apologizes to George Takei for revealing that he contributes Facebook items (jimromenesko.com)
• Friday Foolery #54 The Best 404 Message ever? (laikaspoetnik.wordpress.com)
• Friday Foolery #53 : Variations on the Cochrane Logo & Farewell from the #Cochraneauckland Colloquium. (laikaspoetnik.wordpress.com)

Between the Lines. Finding the Truth in Medical Literature [Book Review]

In the 1970s a study was conducted among 60 physicians and physicians-in-training. They had to solve a simple problem:

“If a test to detect a disease whose prevalence is 1/1000 has a false positive rate of 5 %, what is the chance that a person found to have a positive result actually has the disease, assuming that you know nothing about the person’s symptoms or signs?” 

Half of the “medical experts” thought the answer was 95%.
Only a small proportion, 18%, of the doctors arrived at the right answer of 2%.

If you are a medical expert who comes the same faulty conclusion -or need a refresher how to arrive at the right answer- you might benefit from the book written by Marya Zilberberg: “Between the Lines. Finding the Truth in Medical Literature”.

The same is true for a patient whose doctor thinks he/she is among the 95% to benefit form such a test…
Or for journalists who translate medical news to the public…
Or for peer reviewers or editors who have to assess biomedical papers…

In other words, this book is useful for everyone who wants to be able to read “between the lines”. For everyone who needs to examine medical literature critically from time to time and doesn’t want to rely solely on the interpretation of others.

I hope that I didn’t scare you off with the abovementioned example. Between the Lines surely is NOT a complicated epidemiology textbook, nor a dull studybook where you have to struggle through a lot of definitions, difficult tables and statistic formulas and where each chapter is followed by a set of review questions that test what you learned.

This example is presented half way the book, at the end of Part I. By then you have enough tools to solve the question yourself. But even if you don’t feel like doing the exact calculation at that moment, you have a solid basis to understand the bottomline: the (enormous) 93% gap (95% vs 2% of the people with a positive test are considered truly positive) serves as the pool for overdiagnosis and overtreatment.

In the previous chapters of Part I (“Context”), you have learned about the scientific methods in clinical research, uncertainty as the only certain feature of science, the importance of denominators, outcomes that matter and outcomes that don’t, Bayesian probability, evidence hierarchies, heterogeneous treatment effects (does the evidence apply to this particular patient?) and all kinds of biases.

Most reviewers prefer part I of the book. Personally I find part II (“Evaluation”) as interesting.

Part II deals with the study question, and study design, pros and cons of observational and interventional studies, validity, hypothesis testing and statistics.

Perhaps part II  is somewhat less narrative. Furthermore, it deals with tougher topics like statistics. But I find it very valuable for being able to critically appraise a study. I have never seen a better description of “ODDs”: somehow ODDs it is better to grasp if you substitute “treatment A” and “treatment B” for “horse A” and “horse B”, and substitute “death” for “loss of a race”.
I knew the basic differences between cohort studies, case control studies and so on, but I kind of never realized before that ODDs Ratio is the only measure of association available in a case-control study and that case control studies cannot estimate incidence or prevalence (as shown in a nice overview in table 4).

Unlike many other books about “the art of reading of medical articles”, “study designs” or “Evidence Based Medicine”, Marya’s book is easy to read. It is written at a conversational tone and statements are illustrated by means of current, appealing examples, like the overestimation of risk of death from the H1N1 virus, breast cancer screening and hormone replacement therapy.

Although I had printed this book in a wrong order (page 136 next to 13 etc), I was able to read (and understand) 1/3 of the book (the more difficult part II) during a 2 hour car trip….

Because this book is comprehensive, yet accessible, I recommend it highly to everyone, including fellow librarians.

Marya even mentions medical librarians as a separate target audience:

Medical librarians may find this book particularly helpful: Being at the forefront of evidence dissemination, they can lead the charge of separating credible science from rubbish.

(thanks Marya!)

In addition, this book may be indirectly useful to librarians as it may help to choose appropriate methodological filters and search terms for certain EBM-questions. In case of etiology questions words like “cohort”, “case-control”, “odds”, “risk” and “regression” might help to find the “right” studies.

By the way Marya Ziberberg @murzee at Twitter and she writes at her blog Healthcare etc.

p.s. 1 I want to apologize to Marya for writing this review more than a year after the book was published. For personal reasons I found little time to read and blog. Luckily the book lost none of its topicality.

p.s. 2 patients who are not very familiar with critical reading of medical papers might benefit from reading “your medical mind” first [1]. 

Amazon Product Details

• Paperback: 170 pages
• Publisher: EviMed Research Press; 1 edition (May 17, 2012)
• Language: English
• ISBN-10: 0985456205; ISBN-13: 978-0985456207
• Average Customer Review: 4.9 out of 5 stars  See all reviews (13 customer reviews)

Related articles

• Your Medical Mind. How to Decide What is Right for You [Book Review] (laikaspoetnik.wordpress.com)
  
• The Best Study Design… For Dummies. (laikaspoetnik.wordpress.com)
• When doctor’s do harm: understanding pre-test probability (prudentlayperson.com)
• New test spots more lung clots but seems to result in overdiagnosis (eurekalert.org)
• Long-term psychosocial consequences of false-positive screening mammography. (rinf.com)

Medpedia, the Medical Wikipedia, is Dead. And we Missed its Funeral…

In a post about Wikipedia in 2009 I suggested that initiatives like Ganfyd or Medpedia, might be a solution to Wikipedia’s accuracy and credibility problems, because only health experts are allowed to edit or contribute to the content of these knowledge bases.

MedPedia is a more sophisticated platform than Ganfyd, which looks more like a simple medical encyclopedia. A similar online encyclopedia project with many medical topics, Google Knol, was discontinued by Google as of May 1, 2012.

But now it appears Medpedia may have followed Google KNOL into the same blind alley.

Medpedia was founded in 2007 [2a] by James Currier, an entrepreneur and investor [2b], and an early proponent of social media. He founded the successful Tickle in 1999, when the term Web 2.0 was coined, but not yet mainstream. And his list of  investments is impressive: Flickr, Branchout and Goodreads for instance.

On its homepage Medpedia was described as a “long term, worldwide project to evolve a new model for sharing and advancing knowledge about health, medicine and the body.”
It was developed in association with top medical schools and organizations such as Harvard, Stanford, American College of Physicians, and the NHS. Medpedia was running on the same software and under the same license as Wikipedia and aimed both at the public and  the experts. Contrary to Wikipedia only experts were qualified to contribute to the main content (although others could suggest changes and new topics). [3, 4 , 5, 6] In contrast to many other medical wikis, Medpedia featured a directory of medical editor profiles with general and Medpedia-specific information. This is far more transparent than wikis without individual author recognition [5].

Although promising, Medpedia never became a real success. Von Muhlen wrote in 1999 [4] that there were no articles reporting success metrics for Medpedia or similar projects. In contrast, Wikipedia remains immensely popular among patients and doctors.

Health 2.0 pioneers like E-Patient Dave (@ePatientDave) and Bertalan Meskó (@berci) saw Medpedia’s Achilles heel right from the start:

Bertalan Meskó at his blog Science Roll [7]:

We need Medpedia to provide reliable medical content? That’s what we are working on in Wikipedia.

I believe elitism kills content. Only the power of masses controlled by well-designed editing guidelines can lead to a comprehensive encyclopaedia.

E-patient Dave (who is a fierce proponent of participatory medicine where everyone, medical expert or not, works in partnership to produce accurate information), addresses his concern in his post

“Medpedia: Who gets to say what info is reliable?” [8]

The title says it all. In Dave’s opinion it is “an error to presume that doctors inherently have the best answer” or as Dave summarizes his concern: “who will vet the vetters?”

In addition, Clay Shirky noted that some Wikipedia entries like the biopsy-entry were far more robust than the Medpedia entries [9,10 ].

Ben Toth on the other hand found the Atrial Fibrillation-Medpedia item better than the corresponding Wikipedia page in some respects, but less up-to-date [11].

In her Medpedia review in the JMLA medical librarian Melissa Rethlefsen [5] concludes that “the content of Medpedia is varied and not clearly developed, lacks topical breadth and depth and that it is more a set of ideals than a workable reference source. Another issue is that Medpedia pages never ranked high, which means its content was hardly findable in today’s Google-centric world.

She concludes that for now (2009) “it means that Wikipedia will continue to be the medical wiki of choice”.

I fear that this will be forever, for Medpedia ceased to exist.

I noticed it yesterday totally by coincidence: both my Medpedia blog badge  and Mesko’s Webicina-“Medical Librarianship in Social Medicine”-wiki page were redirected to a faulty page.

I checked the Internet, but all I could find was a message at Wikipedia:

‘It appears that Medpedia is now closed but there is no information about it closing. Their Facebook and Twitter feeds are still open but they have not been updated in a few years. Their webpage now goes to a spam site.

I checked the Waybackmachine and found the “last sparks of life” at January 2013:

This morning I contacted Medpedia’s founder James Currier, who kindly and almost instantly replied to all my questions.

These are shown (with permission) in entirety below.

=============================================================================

[me: ] I hope that you don’t mind that I use LinkedIn to ask you some questions about Medpedia.

{James:] I don’t mind at all!

Is Medpedia dead? And if so, why was it discontinued?

For now it is. We worked on it for 6 years, had a fantastic team of developers, had fantastic partners who supported us, had a fantastic core group of contributors like yourself, and I personally spent millions of dollars on it. In other words, we gave it a really good effort. But it never got the sort of scale it needed to become something important. So for the last two years, we kept looking for a new vision of what it could become, a new mission. We never found one, and it was expensive to keep running.
In the meantime, we had found a new mission that Medpedia could not be converted into, so we started a new company, Jiff, to pursue it. “Health Care in a Jiff” is the motto. Jiff continues the idea of digitizing healthcare, and making it simple and transparent for the individual, but goes after it in a very different way. More info about Jiff here: https://www.jiff.com and here https://www.jiff.com/static/news. Jiff has taken our time and attention, and hopefully will produce the kinds of benefits we were hoping to see from Medpedia.

Why weren’t people informed and  was Medpedia quietly shut down?

We definitely could have done a better job with that! I apologize. We were under a tight time frame due to several things, such as people leaving the effort, technical issues around where the site was being hosted, and corporate and tax issues after 6 years of operating. So it was rushed, and we should have figured out a way to do a better job of communicating.

Couldn’t the redirection to the spam-site be prevented? And can you do something about it?

I didn’t know about that! I’ll look into it and find out what’s going on.*

Your LinkedIn profile says you’re still working for MedPedia. Why is that? Are there plans to make a new start, perhaps? And how?

Yes, I haven’t updated my LinkedIn profile in a while. I just made that change. We have no current plans to restart Medpedia. But we’re always looking for a new mission that can be self sustaining! Let me know if you have one.

And/or do you have (plans for) other health 2.0 initiatives?

Jiff is our main effort now, and there’s a wonderful CEO, Derek Newell running it.

I know you are a busy man, but I think it is important to inform all people who thought that Medpedia was a good initiative.

Thank you for saying you thought it was a good initiative. I did too! I just wish it had gotten bigger. I really appreciate your questions, and your involvement. Not all projects flourish, but we’ll all keep trying new ideas, and hopefully one will break out and make the big difference we hope for.

*somewhat later James gave an update about the redirection:

By the way, I asked about the redirect, and found out that that that page is produced by our registrar that holds the URL medpedia.com.

We wanted to put up the following message and I thought it was up:

“Medpedia was a great experiment begun in 2007.
Unfortunately, it never reached the size to be self sustaining, and it ceased operations in early 2013.
Thank you to all who contributed!”

I’m going to work again on getting that up!

============================================================================

I have one question left : what happened with all the materials the experts produced? Google Knol gave people time to export their contributions. Perhaps James Currier can answer that question too.

I also wonder why nobody noticed that Medpedia was shut down. Apparently it isn’t missed.

Finally I would like to thank all wo have contributed to this “experiment”. As a medical librarian, who is committed to providing reliable medical information, I still find it a shame that Medpedia didn’t work.

I wish James Currier all the best with his new initiatives.

References

1. The Trouble with Wikipedia as a Source for Medical Information
   (https://laikaspoetnik.wordpress.com) (2009/09/14)
2. [a] Medpedia and [b] James Currier , last edited at 6/30/13*  and 7/12/13 respectively (crunchbase.com)
3. Laurent M.R. & Vickers T.J. (2009). Seeking Health Information Online: Does Wikipedia Matter?, Journal of the American Medical Informatics Association, 16 (4) 471-479. DOI: 10.1197/jamia.M3059
4. von Muhlen M. & Ohno-Machado L. (2012). Reviewing social media use by clinicians, Journal of the American Medical Informatics Association, 19 (5) 777-781. DOI: 10.1136/amiajnl-2012-000990
5. Rethlefsen M.L. (2009). Medpedia, Journal of the Medical Library Association : JMLA, 97 (4) 325-326. DOI: 10.3163/1536-5050.97.4.024
6. Medpedia: Reliable Crowdsourcing of Health and Medical Information (highlighthealth.com) (2009/7/24)
7. Launching MedPedia: From the perspective of a Wikipedia administrator (scienceroll.com) (2009/2/20)
8. Medpedia: Who gets to say what info is reliable? (e-patients.net/) (2009/2/20)
9. Clay Shirky at MLA ’11 – On the Need for Health Sciences Librarians to Rock the Boat (mbanks.typepad.com) (2011
10. Wikipedia vs Medpedia: The Crowd beats the Experts (http://blog.lib.uiowa.edu/hardinmd/2011/05/31
11. Medpedia and Wikipedia (nelh.blogspot.nl) (2009/10/08)
12. Jiff wants to do for employer wellness programs what WordPress did for blogs (medcitynews.com)
13. Jiff Unveils Health App Development Platform, Wellness Marketplace (eweek.com)

No, Google Scholar Shouldn’t be Used Alone for Systematic Review Searching

Several papers have addressed the usefulness of Google Scholar as a source for systematic review searching. Unfortunately the quality of those papers is often well below the mark.

In 2010 I already [1]  (in the words of Isla Kuhn [2]) “robustly rebutted” the Anders’ paper “PubMed versus Google Scholar for Retrieving Evidence” [3] at this blog.

But earlier this year another controversial paper was published [4]:

“Is the coverage of google scholar enough to be used alone for systematic reviews?“

It is one of the highly accessed papers of BMC Medical Informatics and Decision Making and has been welcomed in (for instance) the Twittosphere.

Researchers seem  to blindly accept the conclusions of the paper:

Is the coverage of Google Scholar enough to be used alone for systematic reviews? So long PubMed and Cochrane! biomedcentral.com/1472-6947/13/7

—
Jeff Vallance, PhD (@jeffvallance) May 31, 2013

But don’t rush  and assume you can now forget about PubMed, MEDLINE, Cochrane and EMBASE for your systematic review search and just do a simple Google Scholar (GS) search instead.

You might  throw the baby out with the bath water….

… As has been immediately recognized by many librarians, either at their blogs (see blogs of Dean Giustini [5], Patricia Anderson [6] and Isla Kuhn [1]) or as direct comments to the paper (by Tuulevi Ovaska, Michelle Fiander and Alison Weightman [7].

In their paper, Jean-François Gehanno et al examined whether GS was able to retrieve all the 738 original studies included in 29 Cochrane and JAMA systematic reviews.

And YES! GS had a coverage of 100%!

WOW!

All those fools at the Cochrane who do exhaustive searches in multiple databases using controlled vocabulary and a lot of synonyms when a simple search in GS could have sufficed…

But it is a logical fallacy to conclude from their findings that GS alone will suffice for SR-searching.

Firstly, as Tuulevi [7] rightly points out :

“Of course GS will find what you already know exists”

Or in the words of one of the official reviewers [8]:

What the authors show is only that if one knows what studies should be identified, then one can go to GS, search for them one by one, and find out that they are indexed. But, if a researcher already knows the studies that should be included in a systematic review, why bother to also check whether those studies are indexed in GS?

Right!

Secondly, it is also the precision that counts.

As Dean explains at his blog a 100% recall with a precision of 0,1% (and it can be worse!) means that in order to find 36 relevant papers you have to go through  ~36,700 items.

Dean:

Are the authors suggesting that researchers consider a precision level of 0.1% acceptable for the SR? Who has time to sift through that amount of information?

It is like searching for needles in a haystack.  Correction: It is like searching for particular hay stalks in a hay stack. It is very difficult to find them if they are hidden among other hay stalks. Suppose the hay stalks were all labeled (title), and I would have a powerful haystalk magnet (“title search”)  it would be a piece of cake to retrieve them. This is what we call “known item search”. But would you even consider going through the haystack and check the stalks one by one? Because that is what we have to do if we use Google Scholar as a one stop search tool for systematic reviews.

Another main point of criticism is that the authors have a grave and worrisome lack of understanding of the systematic review methodology [6] and don’t grasp the importance of the search interface and knowledge of indexing which are both integral to searching for systematic reviews.[7]

One wonders why the paper even passed the peer review, as one of the two reviewers (Miguel Garcia-Perez [8]) already smashed the paper to pieces.

The authors’ method is inadequate and their conclusion is not logically connected to their results. No revision (major, minor, or discretionary) will save this work. (…)

Miguel’s well funded criticism was not well addressed by the authors [9]. Apparently the editors didn’t see through and relied on the second peer reviewer [10], who merely said it was a “great job” etcetera, but that recall should not be written with a capital R.
(and that was about the only revision the authors made)

Perhaps it needs another paper to convince Gehanno et al and the uncritical readers of their manuscript.

Such a paper might just have been published [11]. It is written by Dean Giustini and Maged Kamel Boulos and is entitled:

“Google Scholar is not enough to be used alone for systematic reviews“

It is a simple and straightforward paper, but it makes its points clearly.

Giustini and Kamel Boulos looked for a recent SR in their own area of expertise (Chou et al [12]), that included a comparable number of references as that of Gehanno et al. Next they test GS’ ability to locate these references.

Although most papers cited by Chou et al. (n=476/506;  ~95%) were ultimately found in GS, numerous iterative searches were required to find the references and each citation had to be managed once at a time. Thus GS was not able to locate all references found by Chou et al. and the whole exercise was rather cumbersome.

As expected, trying to find the papers by a “real-life” GS search was almost impossible. Because due to its rudimentary structure, GS did not understand the expert search strings and was unable to translate them. Thus using Chou et al.’s original search strategy and keywords yielded unmanageable results of approximately >750,000 items.

Giustini and Kamel Boulos note that GS’ ability to search into the full-text of papers combined with its PageRank’s algorithm can be useful.

On the other hand GS’ changing content, unknown updating practices and poor reliability make it an inappropriate sole choice for systematic reviewers:

As searchers, we were often uncertain that results found one day in GS had not changed a day later and trying to replicate searches with date delimiters in GS did not help. Papers found today in GS did not mean they would be there tomorrow.

But most importantly, not all known items could be found and the search process and selection are too cumbersome.

Thus shall we now for once and for all conclude that GS is NOT sufficient to be used alone for SR searching?

We don’t need another bad paper addressing this.

But I would really welcome a well performed paper looking at the additional value of a GS in SR-searching. For I am sure that GS may be valuable for some questions and some topics in some respects. We have to find out which.

References

1. PubMed versus Google Scholar for Retrieving Evidence 2010/06 (laikaspoetnik.wordpress.com)
2. Google scholar for systematic reviews…. hmmmm  2013/01 (ilk21.wordpress.com)
3. Anders M.E. & Evans D.P. (2010) Comparison of PubMed and Google Scholar literature searches, Respiratory care, May;55(5):578-83  PMID: 20420728
4. Gehanno J.F., Rollin L. & Darmoni S. (2013). Is the coverage of Google Scholar enough to be used alone for systematic reviews., BMC medical informatics and decision making, 13:7  PMID: 23302542 (open access)
5. Is Google scholar enough for SR searching? No. 2013/01 (blogs.ubc.ca/dean)
6. What’s Wrong With Google Scholar for “Systematic” Review 2013/01 (etechlib.wordpress.com)
7. Comments at Gehanno’s paper (www.biomedcentral.com)
8. Official Reviewer’s report of Gehanno’s paper [1]: Miguel Garcia-Perez, 2012/09
9. Authors response to comments  (www.biomedcentral.com)
10. Official Reviewer’s report of Gehanno’s paper [2]: Henrik von Wehrden, 2012/10
11. Giustini D. & Kamel Boulos M.N. (2013). Google Scholar is not enough to be used alone for systematic reviews, Online Journal of Public Health Informatics, 5 (2) DOI: 10.5210/ojphi.v5i2.4623
12. Chou W.Y.S., Prestin A., Lyons C. & Wen K.Y. (2013). Web 2.0 for Health Promotion: Reviewing the Current Evidence, American Journal of Public Health, 103 (1) e9-e18. DOI: 10.2105/AJPH.2012.301071

Diane-35: Geen Reden tot Paniek!

Dear english-speaking readers of this blog.

This post is about the anti-acne drug Diane-35 that (with other 3rd and 4th generation combined oral contraceptives (COCs)) has been linked to the deaths of several women in Canada, France and the Netherlands. Since there is a lot of media attention (and panic) in the Netherlands, the remainder of this post is in Dutch. Please write in the comments (or tweet) if you would like me to summarize the health concerns of these COCs in a separate English post.

————————-

Mediaophef 

Er is de laatste tijd nogal veel media-aandacht voor Diane-35. Het begon allemaal in Frankrijk, waar de Franse toezichthouder op geneesmiddelen ANSM Diane-35* eind januari van de markt wilde halen omdat in de afgelopen 25 jaar 4 vrouwen na het gebruik ervan waren overleden. In beroep werd dit afgewezen, waarna de ANSM de EMA (European Medicines Agency) verzocht om de veiligheid van Diane en 3e/4e generatie orale combinatie anticonceptiepillen (OAC) nog eens te onderzoeken.

In januari overleed ook een 21-jarige gebruikster van Diane-35 in Nederland. Met terugwerkende kracht ontving het Nederlandse Bijwerkingscentrum Lareb 97 meldingen van bijwerkingen van Diane-35. Hieronder waren 9 sterfgevallen uit 2011 en eerder.** Overigens werden ook sterfgevallen gemeld van vrouwen die vergelijkbare (3e en 4e generatie) orale anticonceptiemiddelen hadden gebruikt, zoals Yasmin.

Alle vrouwen zijn overleden aan bloedproppen in bloedvaten (trombose dan wel longembolie).  Totaal waren er  89 meldingen van bloedstolsels, wat altijd (ook zonder dodelijke afloop) een ernstige bijwerking is.

Aanleiding voor Canada en België, om ook in de statistieken te duiken. In Canada bleken sinds 2000 11 vrouwen die Diane-35 slikten te zijn overleden en in België zijn er sinds 2008 29 meldingen van trombose door het gebruik van de 3e/4e generatiepil (5 door Diane-35, geen doden)

Dit nieuws sloeg in als een bom. Veel mensen raakten in paniek. Of zijn boos op Bayer*, het CBG (College ter Beoordeling van Geneesmiddelen) en/of minister Schippers die het naar hun idee laten afweten. Op Twitter zie ik aan de lopende band tweets voorbijkomen als:

“Diane-35 pil: heet deze zo omdat je er niet altijd de 35 jaar mee haalt?“.

“Tonnen #rundvlees halen we uit de handel om wat #paardenvlees. Maar doden door de Diane 35 #pil doet de regering niks mee.

Oud Nieuws

Dergelijke reacties zijn sterk overdreven. Er is absoluut geen reden tot paniek.

Echter waar rook is, is vuur. Ook al gaat het hier om een klein brandje.

Maar wat betreft Diane-35 is die rook er al jaren…. Waarom roept men nu ineens: “Brand!”?

De meeste sterfgevallen zijn van vòòr dit jaar. Dat de Fransen authoriteiten nu zoveel daadkracht tonen komt waarschijnlijk omdat hen laksheid verweten werd bij recente schandalen met PIP-borstimplantaten en Mediator, dat meer dan 500 sterfgevallen veroorzaakt heeft. [Reuters, zie ook het blog van Henk Jan Out]

Verder was allang bekend dat Diane-35 de kans op bloedstolsels verhoogde.

Niet alleen Diane-35

Men kan de risico’s van Diane-35 niet los zien van de risico’s van orale anticonceptiemiddelen (OAC’s) in het algemeen.

Diane-35 lijkt qua samenstelling erg op de 3e generatie OAC.  Het is echter uniek omdat het in plaats van een 3e generatie progestogeen cyproteronacetaat bevat. ‘De pil’ bevat levonorgestrel, dit is een 2e generatie progestogeen. Al de OAC combinatiepillen bevatten daarnaast (tegenwoordig) een lage dosering ethinylestradiol.

Zoals gezegd, is al jaren bekend dat alle OAC’s, dus ook ‘de pil’, de kans op bloedstolsels in bloedvaten licht verhogen[1,2,3]. Op zijn hoogst verhogen 2e generatie OAC’s (met levonorgestrel) die kans met een factor 4. Derde generatie pillen lijken die kans verder te verhogen. Met hoeveel precies, daarover verschillen de meningen. Voor wat beteft Diane-35, ziet de een géén tot nauwelijks effect [4], de ander een 1,5 [5]  tot 2 x [3] sterker effect.  Het totaalplaatje ziet er ongeveer als volgt uit:

Absolute en relatieve kans op VTE (Veneuze trombo-embolie).
Uit: http://www.anticonceptie-online.nl/pil.htm

Risico’s in Perspectief

Een 1,5-2 x groter risico vergeleken met de “gewone pil”, lijkt een enorm groot risico. Dit zou ook een groot effect zijn als trombo-embolie vaak voorkwam. Stel dat 1 op de 100 mensen trombose krijgt per jaar, dan zouden op jaarbasis 2-4 op de 100 mensen trombose krijgen na de ‘pil’ en 3-8 mensen na Diane-35 of een 3e of 4e generatiepil. Dit is een groot absoluut risico. Dat risico zou je normaal niet nemen.

Maar trombo-embolie is zeldzaam. Het komt voor bij 5-10 op de 100.000 vrouwen per jaar. En totaal zal 1 op miljoen vrouwen daaraan overlijden. Dat is een heel minieme kans.Vier tot zes keer een kans van iets meer dan 0 blijft een kans van bijna 0. Dus in absolute zin, brengen Diane-35 en OAC’s weinig risico met zich mee.

Daarbij komt dat trombose niet direct door de pil veroorzaakt hoeft te zijn. Roken, leeftijd, (over)gewicht, erfelijke aanleg voor stollingsproblemen kunnen ook een (grote) rol spelen. Verder kunnen deze factoren samenspelen. Om deze reden worden OAC’s (ook de pil) afgeraden aan risicogroepen (oudere vrouwen die veel roken, aanleg voor trombose e.d.)

Het aantal bijwerkingen, dat mogelijk samenhangt met het gebruik van Diane-35, geeft eigenlijk aan dat dit een relatief veilig middel is.

@Net_Arts @medicalfacts @laikas @jopinie Zijn er überhaupt medicijnen waar de afgelopen 25 jaar minder dan 10 mensen aan zijn overleden?

—
Hugo van der Wedden (@hugovdwedden) March 07, 2013

Aanvaardbaar risico?

Om het nog meer in perspectief te plaatsen: zwangerschap geeft een 2x hoger risico op trombose dan Diane-35, en in de 12 weken na de bevalling is de kans nog weer 4-8 keer hoger dan in de zwangerschap (FDA). Toch zullen vrouwen het daarvoor niet laten om zwanger te worden. Het krijgen van een kind weegt meestal (impliciet) op tegen(kleine) risico’s (waarvan trombose er één is).

Men kan de (kans op) risico’s dus niet los zien van de voordelen. Als het voordeel hoog is zal men zelfs een zeker risico op de koop toe willen nemen (afhankelijk van de ernst van de aandoening en eht nut). Aan de andere kant wil je zelfs een heel klein risico niet lopen, als je geen baat hebt bij een middel of als er even goede, maar veiliger middelen zijn.

Maar mag de patiënte die overweging niet zelf met haar arts maken?

Geen plaats voor Diane-35  als anticonceptiemiddel

Diane-35 heeft een anticonceptiewerking, maar het is hiervoor niet (langer) geregistreerd. De laatste anticonceptie-richtlijn van de nederlandse huisartsen (NHG) uit 2010 [6] zegt expliciet dat er geen plaats meer is voor de pil met cyproteronacetaat. Dit omdat de gewone ‘pil’ even goed zwangerschap voorkomt én (iets) minder kans geeft op trombose als bijwerking. Dus waarom zou je een een potentieel hoger risico lopen, als dat niet nodig is? Helaas is de NHG-standaard minder expliciet over 2e en 3e generatie OAC’s.

In andere landen denkt men vaak net zo (In de VS is Diane-35 echter niet geregistreerd).

Dit zegt bijv de RCOG (Royal College of Obstetricians and Gynaecologist, UK) in hun evidence-based richtlijn die specifiek gaat over OAC’s en kans op trombose [1]

Diane-35 als middel tegen ernstige acne en overbeharing.

Omdat het cyproteron acetaat in Diane-35 een sterk anti-androgene werking heeft kan het worden ingezet  bij ernstige acné en overbeharing (dat laatste met name bij vrouwen met PCOS, een gynecologische aandoening). Desgewenst kan het dan tevens dienst doen als anticonceptiemiddel: 2 vliegen in één klap.

Clinical Evidence, dat een heel mooie evidence based bron is die voor-en nadelen van behandelingen tegen elkaar afzet, concludeert dat middelen met cyproteron acetaat ondanks hun prima werking, bij ernstige overbeharing (bij PCOS) niet de voorkeur verdienen boven middelen als metformine. Het risico op trombose is in deze overweging meegenomen.[7]

Volgens een Cochrane Systematisch Review hielpen alle OAC’s wel bij acné, maar OAC’s met cyproteron leken wat effectiever dan pillen met 2e of 3e generatie progestogeen. De resultaten waren echter tegenstrijdig en de studies niet zo erg sterk.[8]

Sommigen concluderen op basis van dit Cochrane Review dat alle OAC’s even goed helpen en dat de gewone pil dus voorkeur verdient (zie bijv. dit recente artikel van Helmerhorst in de BMJ [2], en de NHG standaard acne [9]

Maar in de meest recente Richtlijn Acneïforme Dermatosen [10] van de Nederlandse Vereniging voor Dermatologie en Venereologie (NVDV) wordt er op basis van dezelfde evidence iets anders geconcludeerd:

De Nederlandse dermatologen komen dus met een positieve aanbeveling van Diane-35 ten opzichte van andere anticonceptiemiddelen bij vrouwen die ook anticonceptie wensen. Nergens in deze richtlijn wordt expliciet gerefereerd aan trombose als mogelijke bijwerking.

Het voorschrijfbeleid in de praktijk.

Als Diane-35 niet als anticonceptiemiddel voorgeschreven wordt, en het wordt slechts bij ernstige vormen van acne of overbeharing gebruikt, hoe kan dit middel met een zo’n laag risico dan zo’n omvangrijk probleem worden? De doelgroep èn de kans op bijwerkingen is immers heel klein. En hoe zit het met 3e en 4e generatie OAC’s die niet eens bij acné voorgeschreven zullen worden? Daar zou de doelgroep nog kleiner moeten zijn.

De realiteit is dat de omvang van het probleem niet zozeer door het on-label gebruik komt maar, zoals Janine Budding al aangaf op haar blog Medical Facts door off-label voorschrijfgedrag, dus voor een  andere indicatie dan waarvoor het geneesmiddel is geregistreerd. In Frankrijk gebruikt de helft van de vrouwen die OAC’s gebruiken, de 3e en 4e generatie OAC: dat is ronduit buitensporig, en niet volgens de richtlijnen.

In Nederland slikkten ruim 161.000 vrouwen Diane-35 of een generieke variant met exact dezelfde werking. Ook veel Nederlandse en Canadese gebruiken Diane-35 en andere 3e en 4e generatie OAC’s puur anticonceptiemiddel. Voor een deel, omdat sommige huisartsen het ‘in de pen’ hebben of denken dat een meisje dan gelijk van haar puistjes afgeholpen is. Voor een deel omdat, in  Nederland en  Frankrijk, Diane-35 vergoed wordt en de gewone pil niet. Er is, zeker in Frankrijk, een run op een ‘gratis’ pil.

Online bedrijven spelen mogelijk ook een rol. Deze lichten vaak niet goed voor. Eén zo’n bedrijf (met gebrekkige info over Diane op hun website) gaat zelfs zover het twitter account @diane35nieuws te creeeren als dekmantel voor online pillenverkoop.

Wat nu?

Hoewel de risico’s van Diane-35 allang bekend waren en gering lijken te zijn, en bovendien vergelijkbaar met die van de 3e en 4e generatie  OAC’s, is er een massaal verzet tegen Diane-35 op gang gekomen, die niet meer te stuiten lijkt. Niet de experts, maar de media en de politiek lijken de discussie te voeren.Erg verwarrend en soms misleidend voor de patiënt.

Mijn inziens is het besluit van de Nederlandse huisartsen, gynecologen en recent ook de dermatologen om Diane-35 voorlopig niet voor te schrijven aan nieuwe patiënten tot de autoriteiten een uitspraak hebben gedaan over de veiligheid***, gezien de huidige onrust, een verstandige.

Wat niet verstandig is om zomaar met de Diane-35 pil te stoppen. Overleg altijd eerst met uw arts wat voor u de beste optie is.

In 1995 heeft een vergelijkbare reactie op waarschuwingen over de tromboserisico’s van bepaalde OAC’s geleid tot een ware “pil scare”: vrouwen gingen massaal over op een andere pil of stopten er in het geheel mee. Gevolg: een piek aan ongewenste zwangerschappen (met overigens een veel hogere kans op trombose) en abortussen. Conclusie destijds [10]:

“The level of risk should, in future, be more carefully assessed and advice more carefully presented in the interests of public health.”

Kennelijk is deze les aan Nederland en Frankrijk voorbijgegaan.

Hoewel ik denk dat Diane-35 maar voor een beperkte groep echt zinvol is boven de bestaande middelen, is het te betreuren dat op basis van ongefundeerde reacties, patiënten straks mogelijk zelf geen keuzevrijheid meer hebben. Mogen zij zelf de balans tussen voor-en nadelen bepalen?

Het is begrijpelijk (maar misschien niet zo heel professioneel), dat dermatologen nogal gefrustreerd reageren, nu een bepaalde groep patienten tussen wal en schip raakt. Tevens moet men niet op basis van evidence en argumenten, maar onder druk van media en politiek, tot een ander beleid overgaan.

Want laten we wel wezen, sommige dermatologische en gynecologische patiënten hebben wel baat bij Diane-35.

Ik kreeg hem op mijn 14e voorgeschreven, acnévrij. RT @volkskrant: Waarom wordt #Diane35 niet in Nederland verboden? bit.ly/Z11dqE

—
Kiind Magazine (@KiindNL) March 07, 2013

en

@laikas great, completely agree (being on Diane-35 for 20 years 😉 )

—
Esther van Zuuren (@Ezzoef) March 08, 2013

En tot slot een prachtige reactie van een acne-patiënte op een  blog post van Ivan Wolfers. Zij vat de essentie in enkele zinnen samen. Net als bovenstaande dames, een patient die zeer weloverwogen met haar arts beslissingen neemt op basis van de bestaande info.

Zoals het zou moeten…

Noten

* Diane-35 wordt geproduceerd door Bayer. Het staat ook bekend als Minerva, Elisa en in buitenland bijvoorbeels als Dianette. Er zijn verder ook veel merkloze preparaten met dezelfde samenstelling.

**Inmiddels heeft zijn er nog 4 dodelijke gevallen na gebruik van Diane-35 in Nederland bijgekomen (Artsennet, 2013-03-11)

***Hopelijk wordt de gewone pil dan ook in de vergelijking meegenomen. Dit is wel zo eerlijk: het gaat immers om een vergelijking.

Referenties 

• Venous Thromboembolism and Hormone Replacement Therapy – Green-top Guide line 40 (2010) Royal College of Obstetricians and Gynaecologists, 

  2011

• Helmerhorst F.M. & Rosendaal F.R. (2013). Is an EMA review on hormonal contraception and thrombosis needed?, BMJ (Clinical research ed.), PMID: 23471363
• van Hylckama Vlieg A., Helmerhorst F.M., Vandenbroucke J.P., Doggen C.J.M. & Rosendaal F.R. (2009). The venous thrombotic risk of oral contraceptives, effects of oestrogen dose and progestogen type: results of the MEGA case-control study., BMJ (Clinical research ed.), PMID: 19679614
• Spitzer W.O. (2003) Cyproterone acetate with ethinylestradiol as a risk factor for venous thromboembolism: an epidemiological evaluation., Journal of obstetrics and gynaecology Canada : JOGC = Journal d’obstétrique et gynécologie du Canada : JOGC, PMID: 14663535
• Martínez F., Ramírez I., Pérez-Campos E., Latorre K. & Lete I. (2012) Venous and pulmonary thromboembolism and combined hormonal contraceptives. Systematic review and meta-analysis., The European journal of contraception & reproductive health care : the official journal of the European Society of Contraception, PMID: 22239262

• NHG-Standaard Anticonceptie 2010 Anke Brand, Anita Bruinsma, Kitty van Groeningen, Sandra Kalmijn, Ineke Kardolus, Monique Peerden, Rob Smeenk, Suzy de Swart, Miranda Kurver, Lex Goudswaard.

• Cahill D. (2009). PCOS., Clinical evidence, PMID: 19445767
• Arowojolu A.O., Gallo M.F., Lopez L.M. & Grimes D.A. (2012). Combined oral contraceptive pills for treatment of acne., Cochrane database of systematic reviews (Online), PMID: 22786490
• Kertzman M.G.M., Smeets J.G.E., Boukes F.S. & Goudswaard A.N. [Summary of the practice guideline ‘Acne’ (second revision) from the Dutch College of General Practitioners]., Nederlands tijdschrift voor geneeskunde, PMID: 18590061
• Richtlijn Acneïforme Dermatosen, © 2010, Nederlandse Vereniging voor Dermatologie en Venereologie (NVDV)
• Furedi A. The public health implications of the 1995 ‘pill scare’., Human reproduction update, PMID: 10652971

Of Mice and Men Again: New Genomic Study Helps Explain why Mouse Models of Acute Inflammation do not Work in Men

This post is update after a discussion at Twitter with @animalevidence who pointed me at a great blog post at Speaking of Research ([19], a repost of [20], highlighting the shortcomings of the current study using just one single inbred strain of mice (C57Bl6)  [2013-02-26]. Main changes are in blue

A recent paper published in PNAS [1] caused quite a stir both inside and outside the scientific community. The study challenges the validity of using mouse models to test what works as a treatment in humans. At least this is what many online news sources seem to conclude: “drug testing may be a waste of time”[2], “we are not mice” [3, 4], or a bit more to the point: mouse models of inflammation are worthless [5, 6, 7].

But basically the current study looks only at one specific area, the area of inflammatory responses that occur in critically ill patients after severe trauma and burns (SIRS, Systemic Inflammatory Response Syndrome). In these patients a storm of events may eventually lead to organ failure and death. It is similar to what may occur after sepsis (but here the cause is a systemic infection).

Furthermore the study only uses one single approach: it compares the gene response patterns in serious human injuries (burns, trauma) and a human model partially mimicking these inflammatory diseases (human healthy volunteers receiving  a low dose endotoxin) with the corresponding three animal models (burns, trauma, endotoxin).

And, as highlighted by Bill Barrington of “Understand Nutrition” [8], the researchers have only tested the gene profiles in one single strain of mice: C57Bl6 (B6 for short). If B6 was the only model used in practice this would be less of a problem. But according to Mark Wanner of the Jackson Laboratory [19, 20]:

 It is now well known that some inbred mouse strains, such as the C57BL/6J (B6 for short) strain used, are resistant to septic shock. Other strains, such as BALB and A/J, are much more susceptible, however. So use of a single strain will not provide representative results.

The results in itself are very clear. The figures show at a glance that there is no correlation whatsoever between the human and B6 mouse expression data.

Seok and 36 other researchers from across the USA  looked at approximately 5500 human genes and their mouse analogs. In humans, burns and traumatic injuries (and to a certain extent the human endotoxin model) triggered the activation of a vast number of genes, that were not triggered in the present C57Bl6 mouse models. In addition the genomic response is longer lasting in human injuries. Furthermore, the top 5 most activated and most suppressed pathways in human burns and trauma had no correlates in mice. Finally, analysis of existing data in the Gene Expression (GEO) Database showed that the lack of correlation between mouse and human studies was also true for other acute inflammatory responses, like sepsis and acute infection.

This is a high quality study with interesting results. However, the results are not as groundbreaking as some media suggest.

As discussed by the authors [1], mice are known to be far more resilient to inflammatory challenge than humans*: a million fold higher dose of endotoxin than the dose causing shock in humans is lethal to mice.* This, and the fact that “none of the 150  candidate agents that progressed to human trials has proved successful in critically ill patients” already indicates that the current approach fails.

[This is not entirely correct the endotoxin/LPS dose in mice is 1000–10,000 times the dose required to induce severe disease with shock in humans [20] and mice that are resilient to endotoxin may still be susceptible to infection. It may well be that the endotoxin response is not a good model for the late effects of  sepsis]

The disappointing trial results have forced many researchers to question not only the usefulness of the current mouse models for acute inflammation [9,10; refs from 11], but also to rethink the key aspects of the human response itself and the way these clinical trials are performed [12, 13, 14]. For instance, emphasis has always been on the exuberant inflammatory reaction, but the subsequent immunosuppression may also be a major contributor to the disease. There is also substantial heterogeneity among patients [13–14] that may explain why some patients have a good prognosis and others haven’t. And some of the initially positive results in human trials have not been reproduced in later studies either (benefit of intense glucose control and corticosteroid treatment) [12]. Thus is it fair to blame only the mouse studies?

dick mouse (Photo credit: Wikipedia)

The coverage by some media is grist to the mill of people who think animal studies are worthless anyway. But one cannot extrapolate these findings to other diseases. Furthermore, as referred to above, the researchers have only tested the gene profiles in one single strain of mice: C57Bl6, meaning that “The findings of Seok et al. are solely applicable to the B6 strain of mice in the three models of inflammation they tested. They unduly generalize these findings to mouse models of inflammation in general. [8]”

It is true that animal studies, including rodent studies, have their limitations. But what are the alternatives? In vitro studies are often even more artificial, and direct clinical testing of new compounds in humans is not ethical.

Obviously, the final proof of effectiveness and safety of new treatments can only be established in human trials. No one will question that.

A lot can be said about why animal studies often fail to directly translate to the clinic [15]. Clinical disparities between the animal models and the clinical trials testing the treatment (like in sepsis) are one reason. Other important reasons may be methodological flaws in animal studies (i.e. no randomization, wrong statistics) and publication bias: non-publication of “negative” results appears to be prevalent in laboratory animal research.[15–16]. Despite their shortcomings, animal studies and in vitro studies offer a way to examine certain aspects of a process, disease or treatment.

In summary, this study confirms that the existing (C57Bl6) mouse model doesn’t resemble the human situation in the systemic response following acute traumatic injury or sepsis: the genomic response is entirely different, in magnitude, duration and types of changes in expression.

The findings are not new: the shortcomings of the mouse model(s) were long known. It remains enigmatic why the researchers chose only one inbred strain of mice, and of all mice only the B6-strain, which is less sensitive to endotoxin, and only develop acute kidney injury (part of organ failure) at old age (young mice were used) [21]. In this paper from 2009 (!) various reasons are given why the animal models didn’t properly mimic the human disease and how this can be improved. The authors stress that:

“the genetically heterogeneous human population should be more accurately represented by outbred mice, reducing the bias found in inbred strains that might contain or lack recessive disease susceptibility loci, depending on selective pressures.” 

Both Bill Barrington [8] and Mark Wanner [18,19] propose the use of “diversity outbred cross or collaborative cross mice that  provide additional diversity.” Indeed, “replicating genetic heterogeneity and critical clinical risk factors such as advanced age and comorbid conditions (..) led to improved models of sepsis and sepsis-induced AKI (acute kidney injury). 

The authors of the PNAS paper suggest that genomic analysis can aid further in revealing which genes play a role in the perturbed immune response in acute inflammation, but it remains to be seen whether this will ultimately lead to effective treatments of sepsis and other forms of acute inflammation.

It also remains to be seen whether comprehensive genomic characterization will be useful in other disease models. The authors suggest for instance,  that genetic profiling may serve as a guide to develop animal models. A shotgun analyses of gene expression of thousands of genes was useful in the present situation, because “the severe inflammatory stress produced a genomic storm affecting all major cellular functions and pathways in humans which led to sufficient perturbations to allow comparisons between the genes in the human conditions and their analogs in the murine models”. But rough analysis of overall expression profiles may give little insight in the usefulness of other animal models, where genetic responses are more subtle.

And predicting what will happen is far less easy that to confirm what is already known….

NOTE: as said the coverage in news and blogs is again quite biased. The conclusion of a generally good Dutch science  news site (the headline and lead suggested that animal models of immune diseases are crap [6]) was adapted after a critical discussion at Twitter (see here and here), and a link was added to this blog post). I wished this occurred more often….
In my opinion the most balanced summaries can be found at the science-based blogs: ScienceBased Medicine [11] and NIH’s Director’s Blog [17], whereas “Understand Nutrition” [8] has an original point of view, which is further elaborated by Mark Wanner at Speaking of Research [19] and Genetics and your health Blog [20]

References

1. Seok, J., Warren, H., Cuenca, A., Mindrinos, M., Baker, H., Xu, W., Richards, D., McDonald-Smith, G., Gao, H., Hennessy, L., Finnerty, C., Lopez, C., Honari, S., Moore, E., Minei, J., Cuschieri, J., Bankey, P., Johnson, J., Sperry, J., Nathens, A., Billiar, T., West, M., Jeschke, M., Klein, M., Gamelli, R., Gibran, N., Brownstein, B., Miller-Graziano, C., Calvano, S., Mason, P., Cobb, J., Rahme, L., Lowry, S., Maier, R., Moldawer, L., Herndon, D., Davis, R., Xiao, W., Tompkins, R., , ., Abouhamze, A., Balis, U., Camp, D., De, A., Harbrecht, B., Hayden, D., Kaushal, A., O’Keefe, G., Kotz, K., Qian, W., Schoenfeld, D., Shapiro, M., Silver, G., Smith, R., Storey, J., Tibshirani, R., Toner, M., Wilhelmy, J., Wispelwey, B., & Wong, W. (2013). Genomic responses in mouse models poorly mimic human inflammatory diseases Proceedings of the National Academy of Sciences DOI: 10.1073/pnas.1222878110
2. Drug Testing In Mice May Be a Waste of Time, Researchers Warn 2013-02-12 (science.slashdot.org)
3. Susan M Love We are not mice 2013-02-14 (Huffingtonpost.com)
4. Elbert Chu  This Is Why It’s A Mistake To Cure Mice Instead Of Humans 2012-12-20(richarddawkins.net)
5. Derek Low. Mouse Models of Inflammation Are Basically Worthless. Now We Know. 2013-02-12 (pipeline.corante.com)
6. Elmar Veerman. Waardeloos onderzoek. Proeven met muizen zeggen vrijwel niets over ontstekingen bij mensen. 2013-02-12 (wetenschap24.nl)

7. Gina Kolata. Mice Fall Short as Test Subjects for Humans’ Deadly Ills. 2013-02-12 (nytimes.com)

8. Bill Barrington. Are Mice Reliable Models for Human Disease Studies? 2013-02-14 (understandnutrition.com)
9. Raven, K. (2012). Rodent models of sepsis found shockingly lacking Nature Medicine, 18 (7), 998-998 DOI: 10.1038/nm0712-998a
10. Nemzek JA, Hugunin KM, & Opp MR (2008). Modeling sepsis in the laboratory: merging sound science with animal well-being. Comparative medicine, 58 (2), 120-8 PMID: 18524169
11. Steven Novella. Mouse Model of Sepsis Challenged 2013-02-13 (http://www.sciencebasedmedicine.org/index.php/mouse-model-of-sepsis-challenged/)
12. Wiersinga WJ (2011). Current insights in sepsis: from pathogenesis to new treatment targets. Current opinion in critical care, 17 (5), 480-6 PMID: 21900767
13. Khamsi R (2012). Execution of sepsis trials needs an overhaul, experts say. Nature medicine, 18 (7), 998-9 PMID: 22772540
14. Hotchkiss RS, Coopersmith CM, McDunn JE, & Ferguson TA (2009). The sepsis seesaw: tilting toward immunosuppression. Nature medicine, 15 (5), 496-7 PMID: 19424209
15. van der Worp, H., Howells, D., Sena, E., Porritt, M., Rewell, S., O’Collins, V., & Macleod, M. (2010). Can Animal Models of Disease Reliably Inform Human Studies? PLoS Medicine, 7 (3) DOI: 10.1371/journal.pmed.1000245
16. ter Riet, G., Korevaar, D., Leenaars, M., Sterk, P., Van Noorden, C., Bouter, L., Lutter, R., Elferink, R., & Hooft, L. (2012). Publication Bias in Laboratory Animal Research: A Survey on Magnitude, Drivers, Consequences and Potential Solutions PLoS ONE, 7 (9) DOI: 10.1371/journal.pone.0043404
17. Dr. Francis Collins. Of Mice, Men and Medicine 2013-02-19 (directorsblog.nih.gov)
18. Tom/ Mark Wanner Why mice may succeed in research when a single mouse falls short (2013-02-15) (speakingofresearch.com) [repost, with introduction]
19. Mark Wanner Why mice may succeed in research when a single mouse falls short (2013-02-13/) (http://community.jax.org) %5Boriginal post]
20. Warren, H. (2009). Editorial: Mouse models to study sepsis syndrome in humans Journal of Leukocyte Biology, 86 (2), 199-201 DOI: 10.1189/jlb.0309210
21. Doi, K., Leelahavanichkul, A., Yuen, P., & Star, R. (2009). Animal models of sepsis and sepsis-induced kidney injury Journal of Clinical Investigation, 119 (10), 2868-2878 DOI: 10.1172/JCI39421

BAD Science or BAD Science Journalism? – A Response to Daniel Lakens

Two weeks ago  there was a hot debate among Dutch Tweeps on “bad science, bad science journalism and bad science communication“. This debate was started and fueled by different Dutch blog posts on this topic.[1,4-6]

A controversial post, with both fierce proponents and fierce opposition was the post by Daniel Lakens [1], an assistant professor in Applied Cognitive Psychology.

I was among the opponents. Not because I don’t like a new fresh point of view, but because of a wrong reasoning and because Daniel continuously compares apples and oranges.

Since Twitter debates can’t go in-depth and lack structure and since I cannot comment to his Google sites blog, I pursue my discussion here.

The title of Daniels post is (freely translated, like the rest of his post):

“Is this what one calls good science?” 

In his post he criticizes a Dutch science journalist, Hans van Maanen, and specifically his recent column [2], where Hans discusses a paper published in Pediatrics [3].

This longitudinal study tested the Music Marker theory among 309 Dutch kids. The researchers gathered information about the kids’ favorite types of music and tracked incidents of “minor delinquency”, such as shoplifting or vandalism, from the time they were 12 until they reached age 16 [4]. The researchers conclude that liking music that goes against the mainstream (rock, heavy metal, gothic, punk, African American music, and electronic dance music) at age 12 is a strong predictor of future minor delinquency at 16, in contrast to chart pop, classic music, jazz.

The University press office send out a press release [5 ], which was picked up by news media [4,6] and one of the Dutch authors of this study,  Loes Keijsers,  tweeted enthusiastically: “Want to know whether a 16 year old adult will suffer from delinquency, than look at his music taste at age 12!”

Weten of een jongeren met 16 jaar aan winkeldiefstal doet? Kijk dan naar muziekvoorkeur op 12 jaar! Nieuwe studie @FSW_UU @UniUtrecht

—
Loes Keijsers (@LoesKeijsers) January 08, 2013

According to Hans, Loes could have easily broadcasted (more) balanced tweets, like “Music preference doesn’t predict shoplifting” or “12 year olds who like Bach keep quiet about shoplifting when 16.” But even then, Hans argues, the tweets wouldn’t have been scientifically underpinned either.

In column style Hans explains why he thinks that the study isn’t methodologically strong: no absolute numbers are given; 7 out of 11 (!) music styles are positively associated with delinquency, but these correlations are not impressive: the strongest predictor (Gothic music preference) can explain no more than 9%  of the variance in delinquent behaviour, which can include anything from shoplifting, vandalism, fighting, graffiti spraying, switching price tags.  Furthermore the risks of later “delinquent” behavior are small:  on a scale 1 (never) to 4 (4 times or more) the mean risk was 1,12. Hans also wonders whether it is a good idea to monitor kids with a certain music taste.

Thus Hans concludes “this study isn’t good science”. Daniel, however, concludes that Hans’ writing is not good science journalism.

First Daniel recalls he and other PhD’s took a course on how to peer review scientific papers. On basis of their peer review of a (published) article 90% of the students decided to reject it. The two main lessons learned by Daniel were:

• It is easy to critize a scientific paper and grind it down. No single contribution to science (no single article) is perfect.
• New scientific insights, although imperfect, are worth sharing, because they help to evolve science. *¹

According to Daniel science jounalists often make the same mistakes as the peer reviewing PhD-students: critisizing the individuel studies without a “meta-view” on science.

Peer review and journalism however are different things (apples and oranges if you like).

Peer review (with all its imperfections) serves to filter, check and to improve the quality of individual scientific papers (usually) before they are published  [10]. My papers that passed peer review, were generally accepted. Of course there were the negative reviewers, often  the ignorant ones, and the naggers, but many reviewers had critique that helped to improve my paper, sometimes substantially. As a peer reviewer myself I only try to separate the wheat from the chaff and to enhance the quality of the papers that pass.

Science journalism also has a filter function: it filters already peer reviewed scientific papers* for its readership, “the public” by selecting novel relevant science and translating the scientific, jargon-laded language, into language readers can understand and appreciate. Of course science journalists should put the publication into perspective (call it “meta”).

Surely the PhD-students finger exercise resembles the normal peer review process as much as peer review resembles science journalism.

I understand that pure nitpicking seldom serves a goal, but this rarely occurs in science journalism. The opposite, however, is commonplace.

Daniel disapproves Hans van Maanen’s criticism, because Hans isn’t “meta” enough. Daniel: “Arguing whether an effect size is small or mediocre is nonsense, because no individual study gives a good estimate of the effect size. You need to do more research and combine the results in a meta-analysis”.

Apples and oranges again.

Being “meta” has little to do with meta-analysis. Being meta is … uh … pretty meta. You could think of it as seeing beyond (meta) the findings of one single study*.

A meta-analysis, however, is a statistical technique for combining the findings from independent, but comparable (homogeneous) studies in order to more powerfully estimate the true effect size (pretty exact). This is an important, but difficult methodological task for a scientist, not a journalist. If a meta-analysis on the topic exist, journalists should take this into account, of course (and so should the researchers). If not, they should put the single study in broader perspective (what does the study add to existing knowledge?) and show why this single study is or is not well done?

Daniel takes this further by stating that “One study is no study” and that journalists who simply echo the press releases of a study ànd journalists who just amply criticizes only single publication (like Hans) are clueless about science.

Apples and oranges! How can one lump science communicators (“media releases”), echoing journalists (“the media”) and critical journalists together?

I see more value in a critical analysis than a blind rejoicing of hot air. As long as the criticism guides the reader to appreciate the study.

And if there is just one single novel study, that seems important enough to get media attention, shouldn’t we judge the research on its own merits?

Then Daniel asks himself: “If I do criticize those journalists, shouldn’t I criticize those scientists who published just a single study and wrote a press release about it? “

His conclusion? “No”.

Daniel explains: science never provides absolute certainty, at the most the evidence is strong enough to state what is likely true. This can only be achieved by a lot of research by different investigators. 

Therefore you should believe in your ideas and encourage other scientists to pursue your findings. It doesn’t help when you say that music preference doesn’t predict shoplifting. It does help when you use the media to draw attention to your research. Many researchers are now aware of the “Music Marker Theory”. Thus the press release had its desired effect. By expressing a firm belief in their conclusions, they encourage other scientists to spend their sparse time on this topic. These scientists will try to repeat and falsify the study, an essential step in Cumulative Science. At a time when science is under pressure, scientists shouldn’t stop writing enthusiastic press releases or tweets. 

The latter paragraph is sheer nonsense!

Critical analysis of one study by a journalist isn’t what undermines the  public confidence in science. Rather it’s the media circus, that blows the implications of scientific findings out of proportion.

As exemplified by the hilarious PhD Comic below research results are propagated by PR (science communication), picked up by media, broadcasted, spread via the internet. At the end of the cycle conclusions are reached, that are not backed up by (sufficient) evidence.

PhD Comics – The news Cycle

Daniel is right about some things. First one study is indeed no study, in the sense that concepts are continuously tested and corrected: falsification is a central property of science (Popper). He is also right that science doesn’t offer absolute certainty (an aspect that is often not understood by the public). And yes, researchers should believe in their findings and encourage other scientists to check and repeat their experiments.

Though not primarily via the media. But via the normal scientific route. Good scientists will keep track of new findings in their field anyway. Suppose that only findings that are trumpeted in the media would be pursued by other scientists?

And authors shouldn’t make overstatements. They shouldn’t raise expectations to a level which cannot be met. The Dutch study only shows weak associations. It simply isn’t true that the Dutch study allows us to “predict” at an individual level if a 12 year old will “act out” at 16.

This doesn’t help lay-people to understand the findings and to appreciate science.

The idea that media should just serve to spotlight a paper, seems objectionable to me.

Going back to the meta-level: what about the role of science communicators, media, science journalists and researchers?

According to Maarten Keulemans, journalist, we should just get rid of all science communicators as a layer between scientists and journalists [7]. But Michel van Baal [9] and Roy Meijer[8] have a point when they say that  journalists do a lot PR-ing too and they should do better than to rehash news releases.*²

Now what about Daniel criticism of van Maanen? In my opinion, van Maanen is one of those rare critical journalists who serve as an antidote against uncritical media diarrhea (see Fig above). Comparable to another lone voice in the media: Ben Goldacre. It didn’t surprise me that Daniel didn’t approve of him (and his book Bad Science) either [11]. 

Does this mean that I find Hans van Maanen a terrific science journalist? No, not really. I often agree with him (i.e. see this post [12]). He is one of those rare journalists who has real expertise in research methodology . However, his columns don’t seem to be written for a large audience: they seem too complex for most lay people. One thing I learned during a scientific journalism course, is that one should explain all jargon to one’s audience.

Personally I find this critical Dutch blog post[13] about the Music Marker Theory far more balanced. After a clear description of the study, Linda Duits concludes that the results of the study are pretty obvious, but that the the mini-hype surrounding this research is caused by the positive tone of the press release. She stresses that prediction is not predetermination and that the musical genres are not important: hiphop doesn’t lead to criminal activity and metal not to vandalism.

And this critical piece in Jezebel [14],  reaches far more people by talking in plain, colourful language, hilarious at times.

It also a swell title: “Delinquents Have the Best Taste in Music”. Now that is an apt conclusion!

———————-

*¹ Since Daniel doesn’t refer to  open (trial) data access nor the fact that peer review may , I ignore these aspects for the sake of the discussion.

*² Coincidence? Keulemans has covered  the music marker study quite uncritically (positive).

Photo Credits

http://www.phdcomics.com/comics/archive.php?comicid=1174

References

1. Daniel Lakens: Is dit nou goede Wetenschap? – Jan 24, 2013 (sites.google.com/site/lakens2/blog)
2. Hans van Maanen: De smaak van boefjes in de dop,De Volkskrant, Jan 12, 2013 (vanmaanen.org/hans/columns/)
3. ter Bogt, T., Keijsers, L., & Meeus, W. (2013). Early Adolescent Music Preferences and Minor Delinquency PEDIATRICS DOI: 10.1542/peds.2012-0708
4. Lindsay Abrams: Kids Who Like ‘Unconventional Music’ More Likely to Become Delinquent, the Atlantic, Jan 18, 2013
5. Muziekvoorkeur belangrijke voorspeller voor kleine criminaliteit. Jan 8, 2013 (pers.uu.nl)
6. Maarten Keulemans: Muziek is goede graadmeter voor puberaal wangedrag – De Volkskrant, 12 januari 2013  (volkskrant.nl)
7. Maarten Keulemans: Als we nou eens alle wetenschapscommunicatie afschaffen? – Jan 23, 2013 (denieuwereporter.nl)
8. Roy Meijer: Wetenschapscommunicatie afschaffen, en dan? – Jan 24, 2013 (denieuwereporter.nl)
9. Michel van Baal. Wetenschapsjournalisten doen ook aan PR – Jan 25, 2013 ((denieuwereporter.nl)
10. What peer review means for science (guardian.co.uk)
11. Daniel Lakens. Waarom raadde Maarten Keulemans me Bad Science van Goldacre aan? Oct 25, 2012
12. Why Publishing in the NEJM is not the Best Guarantee that Something is True: a Response to Katan – Sept 27, 2012 (laikaspoetnik.wordpress.com)
13. Linda Duits: Debunk: worden pubers crimineel van muziek? (dieponderzoek.nl)
14. Lindy west: Science: “Delinquents Have the Best Taste in Music” (jezebel.com)

Related articles

• Sceptical humility and peer review in science (openparachute.wordpress.com)
• Putting science journalism under the microscope (crikey.com.au)
• Peer review matters to the public (eurekalert.org)

Friday Foolery #54 The Best 404 Message ever?

claimtoken-510ebd2ada419

Somebody send me a direct message via Twitter, asking me if he had missed any posts. Sorting his Google Reader feeds, he saw this blog was last updated October.

And he is right :(.

Just to assure you that this blog is not dead, but hibernating*, I would like to link to perhaps the BEST 404 message ever.

This 404 message aptly shows where you can turn to when you “Lost your sense of direction” at the ASRM (American Society for Reproductive Medicine) website.

http://www.asrm.org/error.aspx?aspxerrorpath=/Media/Ethics/childrearing.pdf

——————————

Hattip: Bora Zivkovic (@BoraZ), @palmd) and Rebecca Weinberg (@sciliz)

* I have little spare time (and energy) at the moment to write my “usual” long exhaustive posts. Sorry. But I will come back!

Why Publishing in the NEJM is not the Best Guarantee that Something is True: a Response to Katan

In a previous post [1] I reviewed a recent  Dutch study published in the New England Journal of Medicine (NEJM [2] about the effects of sugary drinks on the body mass index of school children.

The study got widely covered by the media. The NRC, for which the main author Martijn Katan works as a science columnist,  columnist, spent  two full (!) pages on the topic -with no single critical comment-[3].
As if this wasn’t enough, the latest column of Katan again dealt with his article (text freely available at mkatan.nl)[4].

I found Katan’s column “Col hors Catégorie” [4] quite arrogant, especially because he tried to belittle a (as he called it) “know-it-all” journalist who criticized his work  in a rivaling newspaper. This wasn’t fair, because the journalist had raised important points [5, 1] about the work.

The piece focussed on the long road of getting papers published in a top journal like the NEJM.
Katan considers the NEJM as the “Tour de France” among  medical journals: it is a top achievement to publish in this paper.

Katan also states that “publishing in the NEJM is the best guarantee something is true”.

I think the latter statement is wrong for a number of reasons.*

1. First, most published findings are false [6]. Thus journals can never “guarantee”  that published research is true.
   Factors that  make it less likely that research findings are true include a small effect size,  a greater number and lesser preselection of tested relationships, selective outcome reporting, the “hotness” of the field (all applying more or less to Katan’s study, he also changed the primary outcomes during the trial[7]), a small study, a great financial interest and a low pre-study probability (not applicable) .
2. It is true that NEJM has a very high impact factor. This is  a measure for how often a paper in that journal is cited by others. Of course researchers want to get their paper published in a high impact journal. But journals with high impact factors often go for trendy topics and positive results. In other words it is far more difficult to publish a good quality study with negative results, and certainly in an English high impact journal. This is called publication bias (and language bias) [8]. Positive studies will also be more frequently cited (citation bias) and will more likely be published more than once (multiple publication bias) (indeed, Katan et al already published about the trial [9], and have not presented all their data yet [1,7]). All forms of bias are a distortion of the “truth”.
   (This is the reason why the search for a (Cochrane) systematic review must be very sensitive [8] and not restricted to core clinical journals, but even include non-published studies: for these studies might be “true”, but have failed to get published).
3. Indeed, the group of Ioannidis  just published a large-scale statistical analysis[10] showing that medical studies revealing “very large effects” seldom stand up when other researchers try to replicate them. Often studies with large effects measure laboratory and/or surrogate markers (like BMI) instead of really clinically relevant outcomes (diabetes, cardiovascular complications, death)
4. More specifically, the NEJM does regularly publish studies about pseudoscience or bogus treatments. See for instance this blog post [11] of ScienceBased Medicine on Acupuncture Pseudoscience in the New England Journal of Medicine (which by the way is just a review). A publication in the NEJM doesn’t guarantee it isn’t rubbish.
5. Importantly, the NEJM has the highest proportion of trials (RCTs) with sole industry support (35% compared to 7% in the BMJ) [12] . On several occasions I have discussed these conflicts of interests and their impact on the outcome of studies ([13, 14; see also [15,16] In their study, Gøtzsche and his colleagues from the Nordic Cochrane Centre [12] also showed that industry-supported trials were more frequently cited than trials with other types of support, and that omitting them from the impact factor calculation decreased journal impact factors. The impact factor decrease was even 15% for NEJM (versus 1% for BMJ in 2007)! For the journals who provided data, income from the sales of reprints contributed to 3% and 41% of the total income for BMJ and The Lancet.
   A recent study, co-authored by Ben Goldacre (MD & science writer) [17] confirms that  funding by the pharmaceutical industry is associated with high numbers of reprint orders. Again only the BMJ and the Lancet provided all necessary data.
6. Finally and most relevant to the topic is a study [18], also discussed at Retractionwatch[19], showing that  articles in journals with higher impact factors are more likely to be retracted and surprise surprise, the NEJM clearly stands on top. Although other reasons like higher readership and scrutiny may also play a role [20], it conflicts with Katan’s idea that  “publishing in the NEJM is the best guarantee something is true”.

I wasn’t aware of the latter study and would like to thank drVes and Ivan Oranski for responding to my crowdsourcing at Twitter.

@laikas re: "NEJM publishing is the best guarantee something is true" - Incorrect, of course. Just look at the retractions, etc.

—
Ves Dimov, M.D. (@DrVes) October 20, 2012

@laikas The NEJM has the highest Retraction Index, which has caveats but may be useful for your purposes: bit.ly/n8Iq3a

—
Ivan Oransky (@ivanoransky) October 20, 2012

References

1. Sugary Drinks as the Culprit in Childhood Obesity? a RCT among Primary School Children (laikaspoetnik.wordpress.com)
2. de Ruyter JC, Olthof MR, Seidell JC, & Katan MB (2012). A trial of sugar-free or sugar-sweetened beverages and body weight in children. The New England journal of medicine, 367 (15), 1397-406 PMID: 22998340
3. NRC Wim Köhler Eén kilo lichter.NRC | Zaterdag 22-09-2012 (http://archief.nrc.nl/)
4. Martijn Katan. Col hors Catégorie [Dutch], (published in de NRC,  (20 oktober)(www.mkatan.nl)
5. Hans van Maanen. Suiker uit fris, De Volkskrant, 29 september 2012 (freely accessible at http://www.vanmaanen.org/)
6. Ioannidis, J. (2005). Why Most Published Research Findings Are False PLoS Medicine, 2 (8) DOI: 10.1371/journal.pmed.0020124
7. Changes to the protocol http://clinicaltrials.gov/archive/NCT00893529/2011_02_24/changes
8. Publication Bias. The Cochrane Collaboration open learning material (www.cochrane-net.org)
9. de Ruyter JC, Olthof MR, Kuijper LD, & Katan MB (2012). Effect of sugar-sweetened beverages on body weight in children: design and baseline characteristics of the Double-blind, Randomized INtervention study in Kids. Contemporary clinical trials, 33 (1), 247-57 PMID: 22056980
10. Pereira, T., Horwitz, R.I., & Ioannidis, J.P.A. (2012). Empirical Evaluation of Very Large Treatment Effects of Medical InterventionsEvaluation of Very Large Treatment Effects JAMA: The Journal of the American Medical Association, 308 (16) DOI: 10.1001/jama.2012.13444
11. Acupuncture Pseudoscience in the New England Journal of Medicine (sciencebasedmedicine.org)
12. Lundh, A., Barbateskovic, M., Hróbjartsson, A., & Gøtzsche, P. (2010). Conflicts of Interest at Medical Journals: The Influence of Industry-Supported Randomised Trials on Journal Impact Factors and Revenue – Cohort Study PLoS Medicine, 7 (10) DOI: 10.1371/journal.pmed.1000354
13. One Third of the Clinical Cancer Studies Report Conflict of Interest (laikaspoetnik.wordpress.com)
14. Merck’s Ghostwriters, Haunted Papers and Fake Elsevier Journals (laikaspoetnik.wordpress.com)
15. Lexchin, J. (2003). Pharmaceutical industry sponsorship and research outcome and quality: systematic review BMJ, 326 (7400), 1167-1170 DOI: 10.1136/bmj.326.7400.1167
16. Smith R (2005). Medical journals are an extension of the marketing arm of pharmaceutical companies. PLoS medicine, 2 (5) PMID: 15916457 (free full text at PLOS)
17. Handel, A., Patel, S., Pakpoor, J., Ebers, G., Goldacre, B., & Ramagopalan, S. (2012). High reprint orders in medical journals and pharmaceutical industry funding: case-control study BMJ, 344 (jun28 1) DOI: 10.1136/bmj.e4212
18. Fang, F., & Casadevall, A. (2011). Retracted Science and the Retraction Index Infection and Immunity, 79 (10), 3855-3859 DOI: 10.1128/IAI.05661-11
19. Is it time for a Retraction Index? (retractionwatch.wordpress.com)
20. Agrawal A, & Sharma A (2012). Likelihood of false-positive results in high-impact journals publishing groundbreaking research. Infection and immunity, 80 (3) PMID: 22338040

——————————————–

* Addendum: my (unpublished) letter to the NRC

Tour de France.

Nadat het NRC eerder 2 pagina’ s de loftrompet over Katan’s nieuwe studie had afgestoken, vond Katan het nodig om dit in zijn eigen column dunnetjes over te doen. Verwijzen naar je eigen werk mag, ook in een column, maar dan moeten wij daar als lezer wel wijzer van worden. Wat is nu de boodschap van dit stuk “Col hors Catégorie“? Het beschrijft vooral de lange weg om een wetenschappelijke studie gepubliceerd te krijgen in een toptijdschrift, in dit geval de New England Journal of Medicine (NEJM), “de Tour de France onder de medische tijdschriften”. Het stuk eindigt met een tackle naar een journalist “die dacht dat hij het beter wist”. Maar ach, wat geeft dat als de hele wereld staat te jubelen? Erg onsportief, omdat die journalist (van Maanen, Volkskrant) wel degelijk op een aantal punten scoorde. Ook op Katan’s kernpunt dat een NEJM-publicatie “de beste garantie is dat iets waar is” valt veel af te dingen. De NEJM heeft inderdaad een hoge impactfactor, een maat voor hoe vaak artikelen geciteerd worden. De NEJM heeft echter ook de hoogste ‘artikelterugtrekkings’ index. Tevens heeft de NEJM het hoogste percentage door de industrie gesponsorde klinische trials, die de totale impactfactor opkrikken. Daarnaast gaan toptijdschriften vooral voor “positieve resultaten” en “trendy onderwerpen”, wat publicatiebias in de hand werkt. Als we de vergelijking met de Tour de France doortrekken: het volbrengen van deze prestigieuze wedstrijd garandeert nog niet dat deelnemers geen verboden middelen gebruikt hebben. Ondanks de strenge dopingcontroles.

Friday Foolery #53 : Variations on the Cochrane Logo & Farewell from the #Cochraneauckland Colloquium.

Two weeks ago the annual Cochrane Colloquium was held in Auckland, New Zealand.

Unfortunately I wasn’t there, but I followed the tweets (see Storify for a selection). And for a more in depth coverage: the webcasts of the plenary sessions are still online here.

The Cochrane Logo is very recognizable and stereotype and therefore well chosen. I‘ve explained the details of the logo in a previous post.

In the Cochrane Colloquium Announcement the blue logo is replaced by a Cochrane logo in “New-Zealand” style (or whatever).

A few entities have permanently adapted the Cochrane Logo.
Below are the logos of the Canadian and the Brazilian Cochrane Centres and the Cochrane Public Health group

The trigger for  this Friday Foolery post is the “souvenir of the #cochraneauckland Colloquium“, the “last”  photo taken by Chris Mavergames on Pukatea Bay, South Island, NZ.

It illustrates very well the Farewell-feeling….. a memorable (Cochrane) meeting in a fabulous country (although these memories will slowly fade away, taken by the sea)

The Diamond in the Cochrane Logo (the result of the metaanalysis) is standardly called Wybertje by the director of the Dutch Cochrane Centre, because its format resembles a Dutch liquorice with that name. (if you don’t believe me, then read the text of his oration speech: “Hoe sterk is het eenzame bewijs” (“how strong is the solitary evidence”, which alludes to a Dutch song)

Perhaps because he is fond of liquorice and sweets, coworkers once gave him M &M’s with the Cochrane logo on it (below is a photo-shopped version).

I also like to play with the Cochrane Logo. For instance, when I gave a presentation at the OLVG, a Dutch hospital, I changed the O into the Cochrane logo (left: the OLVG logo, right: the logo in my presentation)

Silly Sunday #52 Online Education Sites: and the Spam Goes on.

On many occasions  (here, here, here and here [1-4), I have warned against top 50 and 100 lists made by online education sites, like  accreditedonlinecolleges.com, onlinecolleges.com.

They are no more than splogs and link bait scams. Thus please don’t give them credit by linking to their sites.

I have also mentioned that people affiliated with these sites sometimes offer to write guest posts. Or they ask me to place an infographic.

Apparently they don’t do a lot of research. The post don’t really fit the topic of this blog and the writers don’t seem aware of my critical posts in the pasts.

Nevertheless, the number of requests keeps on growing. Sometimes I get 4-5 a day. Really ridiculous…

They don’t seem discouraged by my lack of response.

The letters are usually quite impersonal (they just found a wordpress-tag for instance).

———————–

Hey ,

Re:  laikaspoetnik.wordpress.com/tag/medicine-20/

While doing research  for an online educational resource I write for, I ran across your blog and thought you may be interested in an idea for a post I have been thinking about.

The fate of schools in California is tied to the financial health of the state and because of years of economic downturn and recession, the state can no longer support the schools and the price of tuition is skyrocketing. This is making attending college considerably more difficult for many qualified applicants.

I would love to write about this for your blog. Let me know if you’re interested and I will send you a full outline.

Thanks!

———————-

Lately I’m also informed about dead links at my blog. How kind. Three guesses which link is offered instead…..

——————————-

Hi Laika Spoetnik,

I came across your website and wanted to notify you about a broken link on your page in case you weren’t aware of it. The link on http://laikaspoetnik.wordpress.com/2009/06 which links to http://www.visi.com/juan/congress is no longer working. I’ve included a link to a useful page on Members of Congress that you could replace the broken link with if you’re interested in updating your website. Thanks for providing a great resource!

Link: http://www. onlinebachelordegreeprograms . com / resources / bachelor-of-arts-in-political-science-congress /
(spaces added)

Best,
Alexandra Sawyer

—————————-

p.s. ( as far as I know I never linked to visi com, and 2009/6 is not a single post, but many..)

References

1. Vanity is the Quicksand of Reasoning: Beware of Top 100 and 50 lists! (laikaspoetnik.wordpress.com)
2. Beware of Top 50 “Great Tools to Double Check your Doctor” or whatever Lists. ((laikaspoetnik.wordpress.com)
3. Even the Scientific American Blog Links to Spammy Online Education Affiliate Sites… (laikaspoetnik.wordpress.com)
4. Health and Science Twitter & Blog Top 50 and 100 Lists. How to Separate the Wheat from the Chaff. (laikaspoetnik.wordpress.com)

 

Sugary Drinks as the Culprit in Childhood Obesity? a RCT among Primary School Children

Childhood obesity is a growing health problem. Since 1980, the proportion of overweighted children has almost tripled in the USA:  nowadays approximately 17% of children and adolescents are obese.  (Source: cdc.gov [6])

Common sense tells me that obesity is the result of too high calory intake without sufficient physical activity.” – which is just what the CDC states. I’m not surprised that the CDC also mentions the greater availability of high-energy-dense foods and sugary drinks at home and at school as main reasons for the increased intake of calories among children.

In my teens I already realized that sugar in sodas were just “empty calories” and I replaced tonic and cola by low calory  Rivella (and omitted sugar from tea). When my children were young I urged the day care to restrain from routinely giving lemonade (often in vain).

I was therefore a bit surprised to notice all the fuss in the Dutch newspapers [NRC] [7] about a new Dutch study [1] showing that sugary drinks contributed to obesity. My first reaction was “Duhhh?!…. so what?”.

Also, it bothered me that the researchers had performed a RCT (randomized controlled trial) in kids giving one half of them sugar-sweetened drinks and the other half sugar-free drinks. “Is it ethical to perform such a scientific “experiment” in healthy kids?”, I wondered, “giving more than 300 kids 14 kilo sugar over 18 months, without them knowing it?”

But reading the newspaper and the actual paper[1], I found that the study was very well thought out. Also ethically.

It is true that the association between sodas and weight gain has been shown before. But these studies were either observational studies, where one cannot look at the effect of sodas in isolation (kids who drink a lot of sodas often eat more junk food and watch more television: so these other life style aspects may be the real culprit) or inconclusive RCT’s (i.e. because of low sample size). Weak studies and inconclusive evidence will not convince policy makers, organizations and beverage companies (nor schools) to take action.

As explained previously in The Best Study Design… For Dummies [8] the best way to test whether an intervention has a health effect is to do a  double blind RCT, where the intervention (in this case: sugary drinks) is compared to a control (drinks with artificial sweetener instead of sugar) and where the study participants, and direct researchers do not now who receives the  actual intervention and who the phony one.

The study of Katan and his group[1] was a large, double blinded RCT with a long follow-up (18 months). The researchers recruited 641 normal-weight schoolchildren from 8 primary schools.

Importantly, only children were included in the study that normally drank sugared drinks at school (see announcement in Dutch). Thus participation in the trial only meant that half of the children received less sugar during the study-period. The researchers would have preferred drinking water as a control, but to ensure that the sugar-free and sugar-containing drinks tasted and looked essentially the same they used an artificial sweetener as a control.

The children drank 8 ounces (250 ml) of a 104-calorie sugar-sweetened or no-calorie sugar-free fruit-flavoured drink every day during 18 months.  Compliance was good as children who drank the artificially sweetened beverages had the expected level of urinary sucralose (sweetener).

At the end of the study the kids in the sugar-free group gained a kilo less weight than their peers. They also had a significant lower BMI-increase and gained less body fat.

Thus, according to Katan in the Dutch newspaper NRC[7], “it is time to get rid of the beverage vending machines”. (see NRC [6]).

But does this research really support that conclusion and does it, as some headlines state [9]: “powerfully strengthen the case against soda and other sugary drinks as culprits in the obesity epidemic?”

Rereading the paper I wondered as to the reasons why this study was performed.

If the trial was meant to find out whether putting children on artificially sweetened beverages (instead of sugary drinks) would lead to less fat gain, then why didn’t the researchers do an  intention to treat (ITT) analysis? In an ITT analysis trial participants are compared–in terms of their final results–within the groups to which they were initially randomized. This permits the pragmatic evaluation of the benefit of a treatment policy.
Suppose there were more dropouts in the intervention group, that might indicate that people had a reason not to adhere to the treatment. Indeed there were many dropouts overall: 26% of the children had stopped consuming the drinks, 29% from the sugar-free group, and 22% from the sugar group.
Interestingly, the majority of the children who stopped drinking the cans because they no longer liked the drink (68/94 versus 45/70 dropouts in the sugar-free versus the sugar group).
Ànd children who correctly assumed that the sweetened drinks were “artificially sweetened” was 21% higher than expected by chance (correct identification was 3% lower in the sugar group).
Did some children stop using the non-sugary drinks because they found the taste less nice than usual or artificial? Perhaps.

This  might indicate that replacing sugar-drinks by artificially sweetened drinks might not be as effective in “practice”.

Indeed most of the effect on the main outcome, the differences in BMI-Z score (the number of standard deviations by which a child differs from the mean in the Netherland for his or her age or sex) was “strongest” after 6 months and faded after 12 months.

Mind you, the researchers did neatly correct for the missing data by multiple imputation. As long as the children participated in the study, their changes in body weight and fat paralleled those of children who finished the study. However, the positive effect of the earlier use of non-sugary drinks faded in children who went back to drinking sugary drinks. This is not unexpected, but it underlines the point I raised above: the effect may be less drastic in the “real world”.

Another (smaller) RCT, published in the same issue of the NEJM [2](editorial in[4]), aimed to test the effect of an intervention to cut the intake of sugary drinks in obese adolescents. The intervention (home deliveries of bottled water and diet drinks for one year) led to a significant reduction in mean BMI (body mass index), but not in percentage body fat, especially in Hispanic adolescents. However at one year follow up (thus one year after the intervention had stopped) the differences between the groups evaporated again.

But perhaps the trial was “just” meant as a biological-fysiological experiment, as Hans van Maanen suggested in his critical response in de Volkskrant[10].

Indeed, the data actually show that sugar in drinks can lead to a greater increase in obesity-related parameters (and vice versa). [avoiding the endless fructose-glucose debate [11].

In the media, Katan stresses the mechanistic aspects too. He claims that children who drank the sweetened drinks, didn’t compensate for the lower intake of sugars by eating more. In the NY-times he is cited as follows[12]: “When you change the intake of liquid calories, you don’t get the effect that you get when you skip breakfast and then compensate with a larger lunch…”

This seems a logic explanation, but I can’t find any substatation in the article.

Still “food intake of the children at lunch time, shortly after the morning break when the children have consumed the study drinks”, was a secondary outcome in the original protocol!! (see the nice comparison of the two most disparate descriptions of the trial design at clinicaltrials.gov [5], partly shown in the figure below).

“Energy intake during lunchtime” was later replaced by a “sensory evaluation” (with questions like: “How satiated do you feel?”). The results, however were not reported in their current paper. That is also true for a questionnaire about dental health.

Looking at the two protocol versions I saw other striking differences. At 2009_05_28, the primary outcomes of the study are the children’s body weight (BMI z-score),waist circumference (replaced by waist to height), skin folds and bioelectrical impedance.
The latter three become secondary outcomes in the final draft. Why?

Click to enlarge (source Clinicaltrials.gov [5])

It is funny that although the main outcome is the BMI z score, the authors mainly discuss the effects on body weight and body fat in the media (but perhaps this is better understood by the audience).

Furthermore, the effect on weight is less then expected: 1 kilo instead of 2,3 kilo. And only a part is accounted for by loss in body fat: -0,55 kilo fat as measured by electrical impedance and -0,35 kilo as measured by changes in skinfold thickness. The standard deviations are enormous.

Look for instance at the primary end point (BMI z score) at 0 and 18 months in both groups. The change in this period is what counts. The difference in change between both groups from baseline is -0,13, with a P value of 0.001.

(data are based on the full cohort, with imputed data, taken from Table 2)

Sugar-free group : 0.06±1.00  [0 Mo]  –> 0.08±0.99 [18 Mo] : change = 0.02±0.41  

Sugar-group: 0.01±1.04  [0 Mo]  –> 0.15±1.06 [18 Mo] : change = 0.15±0.42 

Difference in change from baseline: −0.13 (−0.21 to −0.05) P = 0.001

Looking at these data I’m impressed by the standard deviations (replaced by standard errors in the somewhat nicer looking fig 3). What does a value of 0.01 ±1.04 represent? There is a looooot of variation (even though BMI z is corrected for age and sex). Although no statistical differences were found for baseline values between the groups the “eyeball test” tells me the sugar- group has a slight “advantage”. They seem to start with slightly lower baseline values (overall, except for body weight).

Anyway, the changes are significant….. But significance isn’t identical to relevant.

At a second look the data look less impressive than the media reports.

Another important point, raised by van Maanen[10], is that the children’s weight increases more in this study than in the normal Dutch population. 6-7 kilo instead of 3 kilo.

In conclusion, the study by the group of Katan et al is a large, unique, randomized trial, that looked at the effects of replacement of sugar by artificial sweeteners in drinks consumed by healthy school children. An effect was noticed on several “obesity-related parameters”, but the effects were not large and possibly don’t last after discontinuation of the trial.

It is important that a single factor, the sugar component in beverages is tested in isolation. This shows that sugar itself “does matter”. However, the trial does not show that sugary drinks are the main obesity  factor in childhood (as suggested in some media reports).

It is clear that the investigators feel very engaged, they really want to tackle the childhood obesity problem. But they should separate the scientific findings from common sense.

The cans fabricated for this trial were registered under the trade name Blikkie (Dutch for “Little Can”). This was to make sure that the drinks would never be sold by smart business guys using the slogan: “cans which have scientifically been proven to help to keep your child lean and healthy”.[NRC]

Still soft drink stakeholders may well argue that low calory drinks are just fine and that curbing sodas is not the magic bullet.

But it is a good start, I think.

Photo credits Cola & Obesity:  Melliegrunt Flikr [CC]

References

1. de Ruyter JC, Olthof MR, Seidell JC, & Katan MB (2012). A Trial of Sugar-free or Sugar-Sweetened Beverages and Body Weight in Children. The New England journal of medicine PMID: 22998340
2. Ebbeling CB, Feldman HA, Chomitz VR, Antonelli TA, Gortmaker SL, Osganian SK, & Ludwig DS (2012). A Randomized Trial of Sugar-Sweetened Beverages and Adolescent Body Weight. The New England journal of medicine PMID: 22998339
3. Qi Q, Chu AY, Kang JH, Jensen MK, Curhan GC, Pasquale LR, Ridker PM, Hunter DJ, Willett WC, Rimm EB, Chasman DI, Hu FB, & Qi L (2012). Sugar-Sweetened Beverages and Genetic Risk of Obesity. The New England journal of medicine PMID: 22998338
4. Caprio S (2012). Calories from Soft Drinks – Do They Matter? The New England journal of medicine PMID: 22998341
5. Changes to the protocol http://clinicaltrials.gov/archive/NCT00893529/2011_02_24/changes
6. Overweight and Obesity: Childhood obesity facts  and A growing problem (www.cdc.gov)
7. NRC Wim Köhler Eén kilo lichter.NRC | Zaterdag 22-09-2012 (http://archief.nrc.nl/)
8.  The Best Study Design… For Dummies (https://laikaspoetnik.wordpress.com)
9. Studies point to sugary drinks as culprits in childhood obesity – CTV News (ctvnews.ca)
10. Hans van Maanen. Suiker uit fris, De Volkskrant, 29 september 2012 (freely accessible at http://www.vanmaanen.org/)
11. Sugar-Sweetened Beverages, Diet Coke & Health. Part I. (https://laikaspoetnik.wordpress.com)
12. Roni Caryn Rabina. Avoiding Sugared Drinks Limits Weight Gain in Two Studies. New York Times, September 21, 2012

Related articles

• Studies more firmly tie sugary drinks to obesity (news.yahoo.com)
• Teens who cut out sugary drinks slowed weight gain (usatoday.com)
• Evidence against soda is piling up – Philadelphia Inquirer (philly.com)
• Can It! Soda Studies Cite Stronger Link to Obesity (nlm.nih.gov)