The Scatter of Medical Research and What to do About it.

Paul Glasziou, GP and professor in Evidence Based Medicine, co-authored a new article in the BMJ [1]. Similar to another paper [2] I discussed before [3] this paper deals with the difficulty for clinicians of staying up-to-date with the literature. But where the previous paper [2,3] highlighted the mere increase in number of research articles over time, the current paper looks at the scatter of randomized clinical trials (RCTs) and systematic reviews (SR’s) accross different journals cited in one year (2009) in PubMed.

Hofmann et al analyzed 7 specialties and 9 sub-specialties, that are considered the leading contributions to the burden of disease in high income countries.

They followed a relative straightforward method for identifying the publications. Each search string consisted of a MeSH term (controlled  term) to identify the selected disease or disorders, a publication type [pt] to identify the type of study, and the year of publication. For example, the search strategy for randomized trials in cardiology was: “heart diseases”[MeSH] AND randomized controlled trial[pt] AND 2009[dp]. (when searching “heart diseases” as a MeSH, narrower terms are also searched.) Meta-analysis[pt] was used to identify systematic reviews.

Using this approach Hofmann et al found 14 343 RCTs and 3214 SR’s published in 2009 in the field of the selected (sub)specialties. There was a clear scatter across journals, but this scatter varied considerably among specialties:

“Otolaryngology had the least scatter (363 trials across 167 journals) and neurology the most (2770 trials across 896 journals). In only three subspecialties (lung cancer, chronic obstructive pulmonary disease, hearing loss) were 10 or fewer journals needed to locate 50% of trials. The scatter was less for systematic reviews: hearing loss had the least scatter (10 reviews across nine journals) and cancer the most (670 reviews across 279 journals). For some specialties and subspecialties the papers were concentrated in specialty journals; whereas for others, few of the top 10 journals were a specialty journal for that area.
Generally, little overlap occurred between the top 10 journals publishing trials and those publishing systematic reviews. The number of journals required to find all trials or reviews was highly correlated (r=0.97) with the number of papers for each specialty/ subspecialty.”

Previous work already suggested that this scatter of research has a long tail. Half of the publications is in a minority of papers, whereas the remaining articles are scattered among many journals (see Fig below).

Click to enlarge en see legends at BMJ 2012;344:e3223 [CC]

The good news is that SRs are less scattered and that general journals appear more often in the top 10 journals publishing SRs. Indeed for 6 of the 7 specialties and 4 of the 9 subspecialties, the Cochrane Database of Systematic Reviews had published the highest number of systematic reviews, publishing between 6% and 18% of all the systematic reviews published in each area in 2009. The bad news is that even keeping up to date with SRs seems a huge, if not impossible, challenge.

In other words, it is not sufficient for clinicians to rely on personal subscriptions to a few journals in their specialty (which is common practice). Hoffmann et al suggest several solutions to help clinicians cope with the increasing volume and scatter of research publications.

• a central library of systematic reviews (but apparently the Cochrane Library fails to fulfill such a role according to the authors, because many reviews are out of date and are perceived as less clinically relevant)
• registry of planned and completed systematic reviews, such as prospero. (this makes it easier to locate SRs and reduces bias)
• Synthesis of Evidence and synopses, like the ACP-Jounal Club which summarizes the best evidence in internal medicine
• Specialised databases that collate and critically appraise randomized trials and systematic reviews, like www.pedro.org.au for physical therapy. In my personal experience, however, this database is often out of date and not comprehensive
• Journal scanning services like EvidenceUpdates from mcmaster.ca), which scans over 120 journals, filters articles on the basis of quality, has practising clinicians rate them for relevance and newsworthiness, and makes them available as email alerts and in a searchable database. I use this service too, but besides that not all specialties are covered, the rating of evidence may not always be objective (see previous post [4])
• The use of social media tools to alert clinicians to important new research.

Most of these solutions are (long) existing solutions that do not or only partly help to solve the information overload.

I was surprised that the authors didn’t propose the use of personalized alerts. PubMed’s My NCBI feature allows to create automatic email alerts on a topic and to subscribe to electronic tables of contents (which could include ACP journal Club). Suppose that a physician browses 10 journals roughly covering 25% of the trials. He/she does not need to read all the other journals from cover to cover to avoid missing one potentially relevant trial. Instead it is far more efficient to perform a topic search to filter relevant studies from journals that seldom publish trials on the topic of interest. One could even use the search of Hoffmann et al to achieve this.* Although in reality, most clinical researchers will have narrower fields of interest than all studies about endocrinology and neurology.

At our library we are working at creating deduplicated, easy to read, alerts that collate table of contents of certain journals with topic (and author) searches in PubMed, EMBASE and other databases. There are existing tools that do the same.

Another way to reduce the individual work (reading) load is to organize journals clubs or even better organize regular CATs (critical appraised topics). In the Netherlands, CATS are a compulsory item for residents. A few doctors do the work for many. Usually they choose topics that are clinically relevant (or for which the evidence is unclear).

The authors shortly mention that their search strategy might have missed  missed some eligible papers and included some that are not truly RCTs or SRs, because they relied on PubMed’s publication type to retrieve RCTs and SRs. For systematic reviews this may be a greater problem than recognized, for the authors have used meta-analyses[pt] to identify systematic reviews. Unfortunately PubMed has no publication type for systematic reviews, but it may be clear that there are many more systematic reviews that meta-analyses. Possibly systematical reviews might even have a different scatter pattern than meta-analyses (i.e. the latter might be preferentially included in core journals).

Furthermore not all meta-analyses and systematic reviews are reviews of RCTs (thus it is not completely fair to compare MAs with RCTs only). On the other hand it is a (not discussed) omission of this study, that only interventions are considered. Nowadays physicians have many other questions than those related to therapy, like questions about prognosis, harm and diagnosis.

I did a little imperfect search just to see whether use of other search terms than meta-analyses[pt] would have any influence on the outcome. I search for (1) meta-analyses [pt] and (2) systematic review [tiab] (title and abstract) of papers about endocrine diseases. Then I subtracted 1 from 2 (to analyse the systematic reviews not indexed as meta-analysis[pt])

Thus:

(ENDOCRINE DISEASES[MESH] AND SYSTEMATIC REVIEW[TIAB] AND 2009[DP]) NOT META-ANALYSIS[PT]

I analyzed the top 10/11 journals publishing these study types.

This little experiment suggests that:

1. the precise scatter might differ per search: apparently the systematic review[tiab] search yielded different top 10/11 journals (for this sample) than the meta-analysis[pt] search. (partially because Cochrane systematic reviews apparently don’t mention systematic reviews in title and abstract?).
2. the authors underestimate the numbers of Systematic Reviews: simply searching for systematic review[tiab] already found appr. 50% additional systematic reviews compared to meta-analysis[pt] alone
3. As expected (by me at last), many of the SR’s en MA’s were NOT dealing with interventions, i.e. see the first 5 hits (out of 108 and 236 respectively).
4. Together these findings indicate that the true information overload is far greater than shown by Hoffmann et al (not all systematic reviews are found, of all available search designs only RCTs are searched).
5. On the other hand this indirectly shows that SRs are a better way to keep up-to-date than suggested: SRs  also summarize non-interventional research (the ratio SRs of RCTs: individual RCTs is much lower than suggested)
6. It also means that the role of the Cochrane Systematic reviews to aggregate RCTs is underestimated by the published graphs (the MA[pt] section is diluted with non-RCT- systematic reviews, thus the proportion of the Cochrane SRs in the interventional MAs becomes larger)

Well anyway, these imperfections do not contradict the main point of this paper: that trials are scattered across hundreds of general and specialty journals and that “systematic reviews” (or meta-analyses really) do reduce the extent of scatter, but are still widely scattered and mostly in different journals to those of randomized trials.

Indeed, personal subscriptions to journals seem insufficient for keeping up to date.
Besides supplementing subscription by  methods such as journal scanning services, I would recommend the use of personalized alerts from PubMed and several prefiltered sources including an EBM search machine like TRIP (www.tripdatabase.com/).

*but I would broaden it to find all aggregate evidence, including ACP, Clinical Evidence, syntheses and synopses, not only meta-analyses.

**I do appreciate that one of the co-authors is a medical librarian: Sarah Thorning.

References

1. Hoffmann, Tammy, Erueti, Chrissy, Thorning, Sarah, & Glasziou, Paul (2012). The scatter of research: cross sectional comparison of randomised trials and systematic reviews across specialties BMJ, 344 : 10.1136/bmj.e3223
2. Bastian, H., Glasziou, P., & Chalmers, I. (2010). Seventy-Five Trials and Eleven Systematic Reviews a Day: How Will We Ever Keep Up? PLoS Medicine, 7 (9) DOI: 10.1371/journal.pmed.1000326
3. How will we ever keep up with 75 trials and 11 systematic reviews a day (laikaspoetnik.wordpress.com)
4. Experience versus Evidence [1]. Opioid Therapy for Rheumatoid Arthritis Pain. (laikaspoetnik.wordpress.com)

Related articles

• Can Guidelines Harm Patients? (laikaspoetnik.wordpress.com)
• Plausibility bias? You say that as though that were a bad thing! (sciencebasedmedicine.org)
• “Plausibility bias”? Try “reality bias” when it comes to clinical trials. [Respectful Insolence] (scienceblogs.com)
• Connecting Researchers to Unpublished Research (semanticweb.com)
• Systematic review / synthesis of qualitative evidence – issues (phtwitjc.wordpress.com)

Advertisement

PubMed’s Higher Sensitivity than OVID MEDLINE… & other Published Clichés.

Is it just me, or are biomedical papers about searching for a systematic review often of low quality or just too damn obvious? I’m seldom excited about papers dealing with optimal search strategies or peculiarities of PubMed, even though it is my specialty.
It is my impression, that many of the lower quality and/or less relevant papers are written by clinicians/researchers instead of information specialists (or at least no medical librarian as the first author).

I can’t help thinking that many of those authors just happen to see an odd feature in PubMed or encounter an unexpected  phenomenon in the process of searching for a systematic review.
They think: “Hey, that’s interesting” or “that’s odd. Lets write a paper about it.” An easy way to boost our scientific output!
What they don’t realize is that the published findings are often common knowledge to the experienced MEDLINE searchers.

Lets give two recent examples of what I think are redundant papers.

The first example is a letter under the heading “Clinical Observation” in Annals of Internal Medicine, entitled:

“Limitations of the MEDLINE Database in Constructing Meta-analyses”.[1]

As the authors rightly state “a thorough literature search is of utmost importance in constructing a meta-analysis. Since the PubMed interface from the National Library of Medicine is a cornerstone of many meta-analysis,  the authors (two MD’s) focused on the freely available PubMed” (with MEDLINE as its largest part).

The objective was:

“To assess the accuracy of MEDLINE’s “human” and “clinical trial” search limits, which are used by authors to focus literature searches on relevant articles.” (emphasis mine)

O.k…. Stop! I know enough. This paper should have be titled: “Limitation of Limits in MEDLINE”.

Limits are NOT DONE, when searching for a systematic review. For the simple reason that most limits (except language and dates) are MESH-terms.
It takes a while before the indexers have assigned a MESH to the papers and not all papers are correctly (or consistently) indexed. Thus, by using limits you will automatically miss recent, not yet, or not correctly indexed papers. Whereas it is your goal (or it should be) to find as many relevant papers as possible for your systematic review. And wouldn’t it be sad if you missed that one important RCT that was published just the other day?

On the other hand, one doesn’t want to drown in irrelevant papers. How can one reduce “noise” while minimizing the risk of loosing relevant papers?

1. Use both MESH and textwords to “limit” you search, i.e. also search “trial” as textword, i.e. in title and abstract: trial[tiab]
2. Use more synonyms and truncation (random*[tiab] OR  placebo[tiab])
3. Don’t actively limit but use double negation. Thus to get rid of animal studies, don’t limit to humans (this is the same as combining with MeSH [mh]) but safely exclude animals as follows: NOT animals[mh] NOT humans[mh] (= exclude papers indexed with “animals” except when these papers are also indexed with “humans”).
4. Use existing Methodological Filters (ready-made search strategies) designed to help focusing on study types. These filters are based on one or more of the above-mentioned principles (see earlier posts here and here).
   Simple Methodological Filters can be found at the PubMed Clinical Queries. For instance the narrow filter for Therapy not only searches for the Publication Type “Randomized controlled trial” (a limit), but also for randomized, controlled ànd  trial  as textwords.
   Usually broader (more sensitive) filters are used for systematic reviews. The Cochrane handbook proposes to use the following filter maximizing precision and sensitivity to identify randomized trials in PubMed (see http://www.cochrane-handbook.org/):
   (randomized controlled trial [pt] OR controlled clinical trial [pt] OR randomized [tiab] OR placebo [tiab] OR clinical trials as topic [mesh: noexp] OR randomly [tiab] OR trial [ti]) NOT (animals [mh] NOT humans [mh]).
   When few hits are obtained, one can either use a broader filter or no filter at all.

In other words, it is a beginner’s mistake to use limits when searching for a systematic review.
Besides that the authors publish what should be common knowledge (even our medical students learn it) they make many other (little) mistakes, their precise search is difficult to reproduce and far from complete. This is already addressed by Dutch colleagues in a comment [2].

The second paper is:

PubMed had a higher sensitivity than Ovid-MEDLINE in the search for systematic reviews [3], by Katchamart et al.

Again this paper focuses on the usefulness of PubMed to identify RCT’s for a systematic review, but it concentrates on the differences between PubMed and OVID in this respect. The paper starts with  explaining that PubMed:

provides access to bibliographic information in addition to MEDLINE, such as in-process citations (..), some OLDMEDLINE citations (….) citations that precede the date that a journal was selected for MEDLINE indexing, and some additional life science journals that submit full texts to PubMed Central and receive a qualitative review by NLM.

Given these “facts”, am I exaggerating when I am saying that the authors are pushing at an open door when their main conclusion is that PubMed retrieved more citations overall than Ovid-MEDLINE? The one (!) relevant article missed in OVID was a 2005 study published in a Japanese journal that MEDLINE started indexing in 2007. It was therefore in PubMed, but not in OVID MEDLINE.

An important aspect to keep in mind when searching OVID/MEDLINE ( I have earlier discussed here and here). But worth a paper?

Recently, after finishing an exhaustive search in OVID/MEDLINE, we noticed that we missed a RCT in PubMed, that was not yet available in OVID/MEDLINE.  I just added one sentence to the search methods:

Additionally, PubMed was searched for randomized controlled trials ahead of print, not yet included in OVID MEDLINE. 

Of course, I could have devoted a separate article to this finding. But it is so self-evident, that I don’t think it would be worth it.

The authors have expressed their findings in sensitivity (85% for Ovid-MEDLINE vs. 90% for PubMed, 5% is that ONE paper missing), precision and  number to read (comparable for OVID-MEDLINE and PubMed).

If I might venture another opinion: it looks like editors of medical and epidemiology journals quickly fall for “diagnostic parameters” on a topic that they don’t understand very well: library science.

The sensitivity/precision data found have little general value, because:

• it concerns a single search on a single topic
• there are few relevant papers (17- 18)
• useful features of OVID MEDLINE that are not available in PubMed are not used. I.e. Adjacency searching could enhance the retrieval of relevant papers in OVID MEDLINE (adjacency=words searched within a specified maximal distance of each other)
• the searches are not comparable, nor are the search field commands.

The latter is very important, if one doesn’t wish to compare apples and oranges.

Lets take a look at the first part of the search (which is in itself well structured and covers many synonyms).

This part of the search deals with the P: patients with rheumatoid arthritis (RA). The authors first search for relevant MeSH (set 1-5) and then for a few textwords. The MeSH are fine. The authors have chosen to use Arthritis, rheumatoid and a few narrower terms (MeSH-tree shown at the right). The authors have taken care to use the MeSH:noexp command in PubMed to prevent the automatic explosion of narrower terms in PubMed (although this is superfluous for MesH terms having no narrow terms, like Caplan syndrome etc.).

But the fields chosen for the free text search (sets 6-9) are not comparable at all.

In OVID the mp. field is used, whereas all fields or even no fields are used in PubMed.

I am not even fond of the uncontrolled use of .mp (I rather search in title and abstract, remember we already have the proper MESH-terms), but all fields is even broader than .mp.

In general a .mp. search looks in the Title, Original Title, Abstract, Subject Heading, Name of Substance, and Registry Word fields. All fields would be .af in OVID not .mp.

Searching for rheumatism in OVID using the .mp field yields 7879 hits against 31390 hits when one searches in the .af field.

Thus 4 times as much. Extra fields searched are for instance the journal and the address field. One finds all articles in the journal Arthritis & Rheumatism for instance [line 6], or papers co-authored by someone of the dept. of rheumatoid surgery [line 9]

Worse, in PubMed the “all fields” command doesn’t prevent the automatic mapping.

In PubMed, Rheumatism[All Fields] is translated as follows:

“rheumatic diseases”[MeSH Terms] OR (“rheumatic”[All Fields] AND “diseases”[All Fields]) OR “rheumatic diseases”[All Fields] OR “rheumatism”[All Fields]

Oops, Rheumatism[All Fields] is searched as the (exploded!) MeSH rheumatic diseases. Thus rheumatic diseases (not included in the MeSH-search) plus all its narrower terms! This makes the entire first part of the PubMed search obsolete (where the authors searched for non-exploded specific terms). It explains the large difference in hits with rheumatism between PubMed and OVID/MEDLINE: 11910 vs 6945.

Not only do the authors use this .mp and [all fields] command instead of the preferred [tiab] field, they also apply this broader field to the existing (optimized) Cochrane filter, that uses [tiab]. Finally they use limits!

Well anyway, I hope that I made my point that useful comparison between strategies can only be made if optimal strategies and comparable  strategies are used. Sensitivity doesn’t mean anything here.

Coming back to my original point. I do think that some conclusions of these papers are “good to know”. As a matter of fact it should be basic knowledge for those planning an exhaustive search for a systematic review. We do not need bad studies to show this.

Perhaps an expert paper (or a series) on this topic, understandable for clinicians, would be of more value.

Or the recognition that such search papers should be designed and written by librarians with ample experience in searching for systematic reviews.

NOTE:
* = truncation=search for different word endings; [tiab] = title and abstract; [ti]=title; mh=mesh; pt=publication type

Photo credit

The image is taken from the Dragonfly-blog; here the Flickr-image Brain Vocab Sketch by labguest was adapted by adding the Pubmed logo.

References

1. Winchester DE, & Bavry AA (2010). Limitations of the MEDLINE database in constructing meta-analyses. Annals of internal medicine, 153 (5), 347-8 PMID: 20820050
2. Leclercq E, Kramer B, & Schats W (2011). Limitations of the MEDLINE database in constructing meta-analyses. Annals of internal medicine, 154 (5) PMID: 21357916
3. Katchamart W, Faulkner A, Feldman B, Tomlinson G, & Bombardier C (2011). PubMed had a higher sensitivity than Ovid-MEDLINE in the search for systematic reviews. Journal of clinical epidemiology, 64 (7), 805-7 PMID: 20926257
4. Search OVID EMBASE and Get MEDLINE for Free…. without knowing it (laikaspoetnik.wordpress.com 2010/10/19/)
5. 10 + 1 PubMed Tips for Residents (and their Instructors) (laikaspoetnik.wordpress.com 2009/06/30)
6. Adding Methodological filters to myncbi (laikaspoetnik.wordpress.com 2009/11/26/)
7. Search filters 1. An Introduction (laikaspoetnik.wordpress.com 2009/01/22/)

Related articles

• What’s the Difference Between MEDLINE and PubMed? Fact Sheet (nlm.nih.gov)
• Back Door Method to Getting Articles in PubMed: Is Indexing so Important? (kraftylibrarian.com)
• PubMed Clinical Queries Page Updated (jflahiff.wordpress.com)
• Updated Online Tutorials for NCBI resources including an NCBI Overview and PubMed and the Gene Expression Omnibus tutorials (prweb.com)

RIP Statistician Paul Meier. Proponent not Father of the RCT.

This headline in Boing Boing caught my eye today:  RIP Paul Meier, father of the randomized trial

Not surprisingly, I knew that Paul Meier (with Kaplan) introduced the Kaplan-Meier estimator (1958), a very important tool for measuring how many patients survive a medical treatment. But I didn’t know he was “father of the randomized trial”….

But is he really?: “Father of the randomized trial and “probably best known for the introduction of randomized trials into the evaluation of medical treatments”, as Boing Boing states?

Boing Boing’s very short article is based on the New York Times article: Paul Meier, Statistician Who Revolutionized Medical Trials, Dies at 87. According to the NY Times “Dr. Meier was one of the first and most vocal proponents of what is called “randomization.” 

Randomization, the NY-Times explains, is:

Under the protocol, researchers randomly assign one group of patients to receive an experimental treatment and another to receive the standard treatment. In that way, the researchers try to avoid unintentionally skewing the results by choosing, for example, the healthier or younger patients to receive the new treatment.

(for a more detailed explanation see my previous posts The best study designs…. for dummies and #NotSoFunny #16 – Ridiculing RCTs & EBM)

Meier was a very successful proponent, that is for sure. According to Sir Richard Peto, (Dr. Meier) “perhaps more than any other U.S. statistician, was the one who influenced U.S. drug regulatory agencies, and hence clinical researchers throughout the U.S. and other countries, to insist on the central importance of randomized evidence.”

But an advocate need not be a father, for advocates are seldom the inventors/creators. A proponent is more of a nurse, a mentor or a … foster-parent.

Is Meier the true father/inventor of the RCT? And if not, who is?

Googling “Father of the randomized trial” won’t help, because all 1.610  hits point to Dr. Meier…. thanks to Boing Boing careless copying.

What I read so far doesn’t point at one single creator. And the RCT wasn’t just suddenly there. It started with comparison of treatments under controlled conditions. Back in 1753, the British naval surgeon James Lind published his famous account of 12 scurvy patients, “their cases as similar as I could get them” noting that “the most sudden and visible good effects were perceived from the uses of the oranges and lemons and that citrus fruit cured scurvy [3]. The French physician Pierre Louis and Harvard anatomist Oliver Wendell Holmes (19th century) were also fierce proponents of supporting conclusions about the effectiveness of treatments with statistics, not subjective impressions.[4]

But what was the first real RCT?

Perhaps the first real RCT was The Nuremberg salt test (1835) [6]. This was possibly not only the first RCT, but also the first scientific demonstration of the lack of effect of a homeopathic dilution. More than 50 visitors of a local tavern participated in the experiment. Half of them received a vial  filled with distilled snow water, the other half a vial with ordinary salt in a homeopathic C30-dilution of distilled snow water. None of the participants knew whether he got the “actual medicine or not” (blinding). The numbered vials were coded and the code was broken after the experiment (allocation concealment).

The first publications of RCT’s were in the field of psychology and agriculture. As a matter of fact one other famous statistician, Ronald A. Fisher  (of the Fisher’s exact test) seems to play a more important role in the genesis and popularization of RCT’s than Meier, albeit in agricultural research [5,7]. The book “The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century” describes how Fisher devised a randomized trial at the spot to test the contention of a lady that she could taste the difference between tea into which milk had been poured and tea that had been poured into milk (almost according to homeopathic principles) [7]

According to Wikipedia [5] the published (medical) RCT appeared in the 1948 paper entitled “Streptomycin treatment of pulmonary tuberculosis”. One of the authors, Austin Bradford Hill, is (also) credited as having conceived the modern RCT.

Thus the road to the modern RCT is long, starting with the notions that experiments should be done under controlled conditions and that it doesn’t make sense to base treatment on intuition. Later, experiments were designed in which treatments were compared to placebo (or other treatments) in a randomized and blinded fashion, with concealment of allocation.

Paul Meier was not the inventor of the RCT, but a successful vocal proponent of the RCT. That in itself is commendable enough.

And although the Boing Boing article was incorrect, and many people googling for “father of the RCT” will find the wrong answer from now on, it did raise my interest in the history of the RCT and the role of statisticians in the development of science and clinical trials.
I plan to read a few of the articles and books mentioned below. Like the relatively lighthearted “The Lady Tasting Tea” [7]. You can envision a book review once I have finished reading it.

Note added 15-05 13.45 pm:

Today a more accurate article appeared in the Boston Globe (“Paul Meier; revolutionized medical studies using math”), which does justice to the important role of Dr Meier in the espousal of randomization as an essential element in clinical trials. For that is what he did.

Quote:

Dr. Meier published a scathing paper in the journal Science, “Safety Testing of Poliomyelitis Vaccine,’’ in which he described deficiencies in the production of vaccines by several companies. His paper was seen as a forthright indictment of federal authorities, pharmaceutical manufacturers, and the National Foundation for Infantile Paralysis, which funded the research for a polio vaccine.

A good blog post (by @mitchelzoler)can be found at Elsevier’s Global Medical News (EGMN): “Cruising the Kaplan-Meier Curve”

Related articles

1. RIP Paul Meier, father of the randomized trial (boingboing.net)
2. Paul Meier, Statistician Who Revolutionized Medical Trials, Dies at 87 (nytimes.com)
3. M L Meldrum A brief history of the randomized controlled trial. From oranges and lemons to the gold standard. Hematology/ Oncology Clinics of North America (2000) Volume: 14, Issue: 4, Pages: 745-760, vii PubMed: 10949771  or see http://www.mendeley.com
4. Fye WB. The power of clinical trials and guidelines,and the challenge of conflicts of interest. J Am Coll Cardiol. 2003 Apr 16;41(8):1237-42. PubMed PMID: 12706915. Full text
5. http://en.wikipedia.org/wiki/Randomized_controlled_trial
6. Stolberg M (2006). Inventing the randomized double-blind trial: The Nuremberg salt test of 1835. JLL Bulletin: Commentaries on the history of treatment evaluation (www.jameslindlibrary.org).
7. The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century Peter Cummings, MD, MPH, Jama 2001;286(10):1238-1239. doi:10.1001/jama.286.10.1238  Book Review.
   Book by David Salsburg, 340 pp, with illus, $23.95, ISBN 0-7167-41006-7, New York, NY, WH Freeman, 2001.
8. Kaptchuk TJ. Intentional ignorance: a history of blind assessment and placebo controls in medicine. Bull Hist Med. 1998 Fall;72(3):389-433. PubMed PMID: 9780448. abstract
9. The best study design for dummies/ (https://laikaspoetnik.wordpress.com: 2008/08/25/)
10. #Notsofunny: Ridiculing RCT’s and EBM (https://laikaspoetnik.wordpress.com: 2010/02/01/)
11. RIP Paul Meier : Research Randomization Advocate (mystrongmedicine.com)
12. If randomized clinical trials don’t show that your woo works, try anthropology! (scienceblogs.com)
13. The revenge of “microfascism”: PoMo strikes medicine again (scienceblogs.com)

#NotSoFunny #16 – Ridiculing RCTs & EBM

I remember it well. As a young researcher I presented my findings in one of my first talks, at the end of which the chair killed my work with a remark, that made the whole room of scientists laugh, but was really beside the point. My supervisor, a truly original and very wise scientist, suppressed his anger. Afterwards, he said: “it is very easy ridiculing something that isn’t a mainstream thought. It’s the argument that counts. We will prove that we are right.” …And we did.

This was not my only encounter with scientists who try to win the debate by making fun of a theory, a finding or …people. But it is not only the witty scientist who is to *blame*, it is also the uncritical audience that just swallows it.

I have similar feelings with some journal articles or blog posts that try to ridicule EBM – or any other theory or approach. Funny, perhaps, but often misunderstood and misused by “the audience”.

Take for instance the well known spoof article in the BMJ:

“Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials”

It is one of those Christmas spoof articles in the BMJ, meant to inject some medical humor into the normally serious scientific literature. The spoof parachute article pretends to be a Systematic Review of RCT’s  investigating if parachutes can prevent death and major trauma. Of course, no such trial has been done or will be done: dropping people at random with and without a parachute to proof that you better jump out of a plane with a parachute.

I found the article only mildly amusing. It is so unrealistic, that it becomes absurd. Not that I don’t enjoy absurdities at times, but  absurdities should not assume a live of their own.  In this way it doesn’t evoke a true discussion, but only worsens the prejudice some people already have.

People keep referring to this 2003 article. Last Friday, Dr. Val (with whom I mostly agree) devoted a Friday Funny post to it at Get Better Health: “The Friday Funny: Why Evidence-Based Medicine Is Not The Whole Story”.* In 2008 the paper was also discussed by Not Totally Rad [3]. That EBM is not the whole story seems pretty obvious to me. It was never meant to be…

But lets get specific. Which assumptions about RCT’s and SR’s are wrong, twisted or put out of context? Please read the excellent comments below the article. These often put the finger on the spot.

1. EBM is cookbook medicine.
Many define EBM as “make clinical decisions based on a synthesis of the best available evidence about a treatment.” (i.e. [3]). However, EBM is not cookbook medicine.

The accepted definition of EBM  is “the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients” [4]. Sacket already emphasized back in 1996:

Good doctors use both individual clinical expertise and the best available external evidence, and neither alone is enough. Without clinical expertise, practice risks becoming tyrannised by evidence, for even excellent external evidence may be inapplicable to or inappropriate for an individual patient. Without current best evidence, practice risks becoming rapidly out of date, to the detriment of patients.

2. RCT’s are required for evidence.

Although a well performed RCT provides the “best” evidence, RCT’s are often not appropriate or indicated. That is especially true for domains other than therapy. In case of prognostic questions the most appropriate study design is usually an inception cohort. A RCT for instance can’t tell whether female age is a prognostic factor for clinical pregnancy rates following IVF: there is no way to randomize for “age”, or for “BMI”. 😉

The same is true for etiologic or harm questions. In theory, the “best” answer is obtained by RCT. However RCT’s are often unethical or unnecessary. RCT’s are out of the question to address whether substance X causes cancer. Observational studies will do. Sometimes cases provide sufficient evidence. If a woman gets hepatic veno-occlusive disease after drinking loads of a herbal tea the finding of  similar cases in the literature may be sufficient to conclude that the herbal tea probably caused the disease.

Diagnostic accuracy studies also require another study design (cross-sectional study, or cohort).

But even in the case of  interventions, we can settle for less than a RCT. Evidence is not present or not, but exists on a hierarchy. RCT’s (if well performed) are the most robust, but if not available we have to rely on “lower” evidence.

BMJ Clinical Evidence even made a list of clinical questions unlikely to be answered by RCT’s. In this case Clinical Evidence searches and includes the best appropriate form of evidence.

1. where there are good reasons to think the intervention is not likely to be beneficial or is likely to be harmful;
2. where the outcome is very rare (e.g. a 1/10000 fatal adverse reaction);
3. where the condition is very rare;
4. where very long follow up is required (e.g. does drinking milk in adolescence prevent fractures in old age?);
5. where the evidence of benefit from observational studies is overwhelming (e.g. oxygen for acute asthma attacks);
6. when applying the evidence to real clinical situations (external validity);
7. where current practice is very resistant to change and/or patients would not be willing to take the control or active treatment;
8. where the unit of randomisation would have to be too large (e.g. a nationwide public health campaign); and
9. where the condition is acute and requires immediate treatment.
   Of these, only the first case is categorical. For the rest the cut off point when an RCT is not appropriate is not precisely defined.

Informed health decisions should be based on good science rather than EBM (alone).

Dr Val [2]: “EBM has been an over-reliance on “methodolatry” – resulting in conclusions made without consideration of prior probability, laws of physics, or plain common sense. (….) Which is why Steve Novella and the Science Based Medicine team have proposed that our quest for reliable information (upon which to make informed health decisions) should be based on good science rather than EBM alone.

Methodolatry is the profane worship of the randomized clinical trial as the only valid method of investigation. This is disproved in the previous sections.

The name “Science Based Medicine” suggests that it is opposed to “Evidence Based Medicine”. At their blog David Gorski explains: “We at SBM believe that medicine based on science is the best medicine and tirelessly promote science-based medicine through discussion of the role of science and medicine.”

While this may apply to a certain extent to quack or homeopathy (the focus of SBM) there are many examples of the opposite: that science or common sense led to interventions that were ineffective or even damaging, including:

• protective effects of beta-carotene on lung cancer (opposite)
• protective effects of vitamin E on cardiovascular disease (opposite) or certain cancers (neutral)
• protective effects of the prone sleeping on Sudden Infant Death Syndrome (opposite)
• Protective effects of corticosteroids in severe head trauma (opposite)
  etcetera.

As a matter of fact many side-effects are not foreseen and few in vitro or animal experiments have led to successful new treatments.

At the end it is most relevant to the patient that “it works” (and the benefits outweigh the harms).

Furthermore EBM is not -or should not be- without consideration of prior probability, laws of physics, or plain common sense. To me SBM and EBM are not mutually exclusive.

Why the example is bullshit unfair and unrealistic

I’ll leave it to the following comments (and yes the choice is biased) [1]

Nibu A George,Scientist :

First of all generalizing such reports of some selected cases and making it a universal truth is unhealthy and challenging the entire scientific community. Secondly, the comparing the parachute scenario with a pure medical situation is unacceptable since the parachute jump is rather a physical situation and it become a medical situation only if the jump caused any physical harm to the person involved.

Richard A. Davidson, MD,MPH:

This weak attempt at humor unfortunately reinforces one of the major negative stereotypes about EBM….that RCT’s are required for evidence, and that observational studies are worthless. If only 10% of the therapies that are paraded in front of us by journals were as effective as parachutes, we would have much less need for EBM. The efficacy of most of our current therapies are only mildly successful. In fact, many therapies can provide only a 25% or less therapeutic improvement. If parachutes were that effective, nobody would use them.
While it’s easy enough to just chalk this one up to the cliche of the cantankerous British clinician, it shows a tremendous lack of insight about what EBM is and does. Even worse, it’s just not funny.

Aviel Roy-Shapira, Senior Staff Surgeon

Smith and Pell succeeded in amusing me, but I think their spoof reflects a common misconception about evidence based medicine. All too many practitioners equate EBM with randomized controlled trials, and metaanalyses.
EBM is about what is accepted as evidence, not about how the evidence is obtained. For example, an RCT which shows that a given drug lowers blood pressure in patients with mild hypertension, however well designed and executed, is not acceptable as a basis for treatment decisions. One has to show that the drug actually lowers the incidence of strokes and heart attacks.
RCT’s are needed only when the outcome is not obvious. If most people who fall from airplanes without a parachute die, this is good enough. There is plenty of evidence for that.
EBM is about using outcome data for making therapeutic decisions. That data can come from RCTs but also from observation

Lee A. Green, Associate Professor

EBM is not RCTs. That’s probably worth repeating several times, because so often both EBM’s detractors and some of its advocates just don’t get it. Evidence is not binary, present or not, but exists on a heirarchy (Guyatt & Rennie, 2001). (….)
The methods and rigor of EBM are nothing more or less than ways of correcting for our imperfect perceptions of our experiences. We prefer, cognitively, to perceive causal connections. We even perceive such connections where they do not exist, and we do so reliably and reproducibly under well-known sets of circumstances. RCTs aren’t holy writ, they’re simply a tool for filtering out our natural human biases in judgment and causal attribution. Whether it’s necessary to use that tool depends upon the likelihood of such bias occurring.

Scott D Ramsey, Associate Professor

Parachutes may be a no-brainer, but this article is brainless.

Unfortunately, there are few if any parallels to parachutes in health care. The danger with this type of article is that it can lead to labeling certain medical technologies as “parachutes” when in fact they are not. I’ve already seen this exact analogy used for a recent medical technology (lung volume reduction surgery for severe emphysema). In uncontrolled studies, it quite literally looked like everyone who didn’t die got better. When a high quality randomized controlled trial was done, the treatment turned out to have significant morbidity and mortality and a much more modest benefit than was originally hypothesized.

Timothy R. Church, Professor

On one level, this is a funny article. I chuckled when I first read it. On reflection, however, I thought “Well, maybe not,” because a lot of people have died based on physicians’ arrogance about their ability to judge the efficacy of a treatment based on theory and uncontrolled observation.

Several high profile medical procedures that were “obviously” effective have been shown by randomized trials to be (oops) killing people when compared to placebo. For starters to a long list of such failed therapies, look at antiarrhythmics for post-MI arrhythmias, prophylaxis for T. gondii in HIV infection, and endarterectomy for carotid stenosis; all were proven to be harmful rather than helpful in randomized trials, and in the face of widespread opposition to even testing them against no treatment. In theory they “had to work.” But didn’t.

But what the heck, let’s play along. Suppose we had never seen a parachute before. Someone proposes one and we agree it’s a good idea, but how to test it out? Human trials sound good. But what’s the question? It is not, as the author would have you believe, whether to jump out of the plane without a parachute or with one, but rather stay in the plane or jump with a parachute. No one was voluntarily jumping out of planes prior to the invention of the parachute, so it wasn’t to prevent a health threat, but rather to facilitate a rapid exit from a nonviable plane.

Another weakness in this straw-man argument is that the physics of the parachute are clear and experimentally verifiable without involving humans, but I don’t think the authors would ever suggest that human physiology and pathology in the face of medication, radiation, or surgical intervention is ever quite as clear and predictable, or that non-human experience (whether observational or experimental) would ever suffice.

The author offers as an alternative to evidence-based methods the “common sense” method, which is really the “trust me, I’m a doctor” method. That’s not worked out so well in many high profile cases (see above, plus note the recent finding that expensive, profitable angioplasty and coronary artery by-pass grafts are no better than simple medical treatment of arteriosclerosis). And these are just the ones for which careful scientists have been able to do randomized trials. Most of our accepted therapies never have been subjected to such scrutiny, but it is breathtaking how frequently such scrutiny reveals problems.

Thanks, but I’ll stick with scientifically proven remedies.

parachute experiments without humans

* on the same day as I posted Friday Foolery #15: The Man who pioneered the RCT. What a coincidence.

** Don’t forget to read the comments to the article. They are often excellent.

Photo Credits

• http://www.Army.mil http://www.flickr.com/photos/soldiersmediacenter/ / CC BY 2.0
• EBM triad: http://knol.google.com/k/evidence-based-medicine-ebm#
• Royal Army Parachute Dog http://www.flickr.com/photos/wurzeltod/ / CC BY-NC 2.0
• Somewhere Over the Rainbow http://www.flickr.com/photos/daybeezho/ / CC BY-NC-SA 2.0

References

1. Smith, G. (2003). Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials BMJ, 327 (7429), 1459-1461 DOI: 10.1136/bmj.327.7429.1459
2. The Friday Funny: Why Evidence-Based Medicine Is Not The Whole Story”. (getbetterhealth.com) [2010.01.29]
3. Call for randomized clinical trials of Parachutes (nottotallyrad.blogspot.com) [08-2008]
4. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, & Richardson WS (1996). Evidence based medicine: what it is and what it isn’t. BMJ (Clinical research ed.), 312 (7023), 71-2 PMID: 8555924

Related articles by Zemanta

• Friday Foolery #15: The Man who pioneered the RCT (laikaspoetnik.wordpress.com)
• Friday Foolery #11. Is Friday the 13th bad for your Health? (laikaspoetnik.wordpress.com)
• Randomized trial versus observational study challenge, II: meta-comments (scienceblogs.com)
• Randomized trial versus observational study challenge, VII: randomization, second part (scienceblogs.com)
• Evidence based medicine – misinterpreted and not sufficient (medrants.com)
• NOT ONE RCT on Swine Flu or H1N1?! – Outrageous! (laikaspoetnik.wordpress.com)
• Look before we LEAP again (junkfoodscience.blogspot.com)

are very well edged off

Friday Foolery #15: The Man who pioneered the RCT

This BMJ video certainly belongs on a blog like this, focussing on EBM. This video shows “John Crofton who pioneered the randomised controlled trial in a 1948 BMJ paper which looked at the antibiotic streptomycin to treat TB. Now in his 90s, Dr Crofton talks to Colin Blakemore about the importance of randomisation and blinding, and how it has helped to make medicine more evidence based.”

First seen on the Clinical Cases and Images: CasesBlog of Ves Dimov.

NOT ONE RCT on Swine Flu or H1N1?! – Outrageous!

Last week doctorblogs (Annabel Bentley) tweeted: “Outrageous- there isn’t ONE randomised trial on swine flu or #H1N1“

Annabel referred to an article at Trust the Evidence, the excellent blog of the Centre for Evidence-Based Medicine (CEBM) in Oxford, UK.

In the article “Is swine flu the most over-published and over-hyped disease ever?” Carl Heneghan first showed the results of a quick PubMed search using the terms ‘swine flu’ and ‘H1N1’: this yielded 4,475 articles on the subject, with approximately one third (1,437 articles) published in the last 7 months (search: November 27th). Of these 107, largely news articles, were published in the BMJ, followed by the Lancet and NEJM at 35 each.

Top News stories on H1N1 generated appr. 2000 to 4000 news articles each (in Google). Items included outbreak of a new form of ‘swine flu’ which prompted the United States and the World Health Organization to declare a public health emergency (April), Southern Hemisphere being mostly spared in the swine flu epidemic (May), Tamiflu, i.e. the effects of Tamiflu in children in the BMJ (co-authored by Carl) in August and the availability of the vaccine H1N1 vaccine clinics to offer seasonal flu shots in November.

According to Heneghan this must be the most over-hyped disease ever, and he wonders: “are there any other infections out there?”

Finally he ends with: “Do you know what the killer fact is in all of this? There isn’t one randomized trial out there on swine flu or H1N1 – outrageous.”

My first thoughts were: “is H1N1 really so over-published compared to other (infectious) diseases?”, “Is it really surprising that there are no RCTs yet? The H1N1-pandemics just started a few months ago!” and even “are RCT’s really the study designs we urgently need right now?”

Now the severity of the H1N1 flu seems less than feared, it is easy to be wise. Isn’t is logic that there are a lot of “exploratory studies” first: characterization of the virus, establishing the spread of H1N1 around the world, establishing mortality and morbidity, and patterns of vulnerability among the population? It is also understandable that a lot of news articles are published, in the BMJ or in online newspapers. We want to be informed. In the Netherlands we now have a small outbreak of Q-fever, partly because the official approach was slow and underestimated the public health implications of Q-fever. So the public was really underinformed. That is worse than being “overexposed”.

News often spreads like wildfire, that is no news. When I google “US Preventive Services Task Force” (who issued the controversial US breast cancer screening guidelines last month) 2,364 hits still pop up in Google News (over the last month). All papers and other news sources echo the news. 2,000 hits are easily reached.

4,475 PubMed articles on ‘swine flu’ and ‘H1N1’ isn’t really that much. When I quickly search PubMed for the rather “new” disease Q-fever I get 3,752 hits, a search for HPV (Alphapapillomavirus OR papilloma infections OR HPV OR human papilloma virus) gives 19,543 hits (1,330 over the last 9 months), and a quick search for (aids) AND “last 9 months”[edat] yields 4,073 hits!

The number of hits alone doesn’t mean much, certainly not if news, editorials and comments are included. But lets go to the second comment, that there is “not ONE RCT on H1N1.”

Again, is it reasonable to expect ONE RCT published and included in PubMed over a 9 month period? Any serious study takes time from concept to initiation, patient-enrollment, sufficient follow-up, collection of data, writing and submitting the article, peer review, publication, inclusion in PubMed and assignment of MeSH-terms (including the publication type “Randomized Controlled Trial”).

Furthermore RCTs are not always the most feasible or appropriate study designs for answering certain questions. For instance for questions related to harm, etiology, epidemiology, spreading of virus, characteristics, diagnosis and prognosis. RCTs may be most suitable to evaluate the efficacy of treatment or prevention interventions. Thus in case of H1N1 the efficacy of vaccines and of neuraminidase inhibitors to prevent or treat H1N1 flu. However, it may not always be ethical to do so (see below).

I’ve repeated the search, and using prefab “My NCBI filters” for RCTs discussed before I get the following results:

Using the Randomized Controlled Trials limits in PubMed I do get 7 hits, and using broader filters, like the Therapy/Narrow Filter under  Clinical Queries I even find 2 more RCTs that have not yet been indexed by PubMed. With the Cochrane Highly sensitive Filter even more hits are obtained, most of which are “noise”, inherent to the use of a broad filter.

The found RCTs are safety/immunogenicity/stability studies of subunit or split vaccines to H1N1, H3N2, and B influenza strains. This means they are not restricted to H1N1, but this is true for the entire set of H1N1 publications. 40 of the 1443 hits are even animal studies. Thus the total number of articles dealing with H1N1 only -and in humans- is far less than 1443.
By the way, one of the 15 H1N1-hits in PubMed obtained with the SR-filter (see Fig) is a meta-analysis of RCTs in the BMJ, co-authored by Heneghan. It is not about H1N1, but contains the sentence: “Their (neuraminidase inhibitors) effects on the incidence of serious complications, and on the current A/H1N1 influenza strain remain to be determined.”

More important, if studies have been undertaken in this field they are probably not yet published. Thus, the place to look is a clinical trials register, like Clinical trials.gov (http://clinicaltrials.gov/), The International Clinical Registry Platform Search Portal at the WHO (www.who.int/trialsearch) , national or pharmaceutical industry trials registers.

A search for H1N1 OR swine flu in Clinical trials.gov, that offers the best searching functions, yields 132 studies, of which 116 were first recieved this year.

Again, most trials concern the safety and efficacy of H1N1 vaccines and include the testing of vaccines on subgroups, like pregnant women, children with asthma and people with AIDS. 30 trials are phase III.
Narrowing the search to H1N1 OR swine flu | neuraminidase inhibitors OR oseltamivir OR zanamivir (treatment filled in in the filed “Interventions”) yields 8 studies. One of the studies is a phase III trial.

This yield doesn’t seem bad per se. However, numbers of trials don’t mean a lot and a more pertinent issue is, whether the most important and urgent questions are investigated.

Three issues are important with respect to interventions:

1. Are H1N1 vaccines safe and immunogenic? in subpopulations?
2. Do H1N1 vaccines lower morbidity and mortality due to the H1N1 flu?
3. Are neuraminidase inhibitors effective in preventing or treating H1N1 flu?

Question [1] will be answered by current trials.
Older Cochrane Reviews on the seasonal influenza flu (and updates) cast doubt on the efficacy of [2] vaccines (see the [poor*] Atlantic news article) ànd [2] neuraminidase inhibitors in children (Cochrane 2007 and BMJ 2009) ànd adults  (Cochrane 2006, update 2008 and BMJ 2009) against symptoms or complications of the seasonal flu. The possibility has even been raised that seasonal flu shots are linked to swine flu risk.

However, the current H1N1 isn’t a seasonal flu. It is a sudden, new pandemic that requires different actions. Overall H1N1 isn’t as deadly as the regular influenza strains, but it hits certain people harder: very young kids, people with asthma and pregnant women. About the latter group, Amy Tuteur (obstetrician-gynecologist blogging at The Skeptical OB) wrote a guest post at Kevin MD:

(…) the H1N1 influenza has had an unexpectedly devastating impact among pregnant women. According to the CDC, there have been approximately 700 reported cases of H1N1 in pregnant women since April.** Of these, 100 women have required admission to an intensive care unit and 28 have died. In other words, 1 out of every 25 pregnant women who contracted H1N1 died of it. By any standard, that is an appalling death rate. (……)
To put it in perspective, the chance of a pregnant woman dying from H1N1 is greater than the chance of a heart patient dying during triple bypass surgery. That is not a trivial risk.

The H1N1 flu has taken an extraordinary toll among pregnant women. A new vaccine is now available. Because of the nature of the emergency, there has not been time to do any long term studies of the vaccine. Yet pregnant women will need to make a decision as soon as possible on whether to be vaccinated. (Emphasis mine)

…. Given the dramatic threat and the fact that we know of no unusual complications of vaccination, the decision seems clear. Every pregnant woman should get vaccinated as soon as possible.

Thus the anticipated risks must be balanced against the anticipated benefits, Amy urges pregnant women to get vaccinated, even though no one can be sure about side effects ànd about the true efficacy of the vaccine.

For scientific purposes it would be best to perform a double randomized trial with half of a series of pregnant women receiving the vaccine, and the other half a placebo. This would provide the most rigid evidence for the true efficacy and safety of the vaccine.

However it would not be ethical to do so. As “Orac” of Orac Knows explains so well  in his post “Vaccination for H1N1 “swine” flu: Do The Atlantic, Shannon Brownlee, and Jeanne Lenzer matter?” RCTs are only acceptable from an ethical standpoint if we truly do not know whether one treatment is superior to another or a treatment is better than a placebo. There is sufficient reason to believe that vaccination for H1N1 will be more efficacious than “doing nothing”. Leaving a control group unvaccinated will certainly mean that a substantial percentage of pregnant women is going to die. To study the efficacy of the H1N1 among pregnant women observational studies (like cohort studies) are also suitable and more appropriate.

Among the studies found in ClinicalTrials.gov there are a few H1N1 Vaccine Clinical Studies in Pregnant Women, including RCTs. But these RCT’s never compare vaccinated women with a non-vaccinated women. All pregnant women are vaccinated, but the conditions vary.

In one Danish study the arms (study groups) are as follows:

Thus two doses of H1N1 with adjuvant are compared with a higher dose H1N1 without adjuvant. As a control non-pregnant women are vaccinated with the adjuvant H1N1.*** The RCT is performed within a prospective, birth-cohort study recruiting 800 pregnant mothers between Q1- 2009 and Q4-2010. As a natural control women pregnant in the H1N1 season (Q4) will be compared with women outside the season. Please note that the completion date of this study will be 2012, thus we will have to wait a number of years before the study describing the results will be found in PubMed….

To give an impression of the idea behind the study, here is the summary of that trial in the register (not because it is particularly outstanding, but to highlight the underlying thoughts):

“Pregnant women are at particular risk during the imminent H1N1v influenza pandemic. The new H1N1v virus requires urgent political and medical decisions on vaccination strategies in order to minimize severe disease and death from this pandemic. However, there is a lack of evidence to build such decisions upon. A vaccine will be provided in the fourth quarter of 2009, but there is little knowledge on the immunogenicity. Particularly its clinical effectiveness and duration of immunity in pregnant women and their newborn infants is unknown. Therefore, it will be important to study the optimal vaccination regimens with respect to dosing and use of adjuvant to decide future health policies on vaccination of pregnant women. We have a unique possibility to study these aspects of H1N1v infection in pregnant women in our ongoing unselected, prospective, birth-cohort study recruiting 800 pregnant mothers between Q1- 2009 and Q4-2010. Pregnant women from East-Denmark are being enrolled during the 2nd trimester and their infant will undergo a close clinical follow-up. The H1N1v pandemic is expected to reach Denmark Q4-2009. The timing of this enrollment and the imminent pandemic allows for an “experiment of nature” whereby the first half of the mothers completes pregnancy before the H1N1v pandemic. The other half of this cohort will be pregnant while H1N1v is prevalent in the community and will require H1N1v vaccination.The aim of this randomized, controlled, trial is to compare and evaluate the dose-related immune protection conferred by vaccine and adjuvant (Novartis vaccine Focetria) in pregnant women and non-pregnant women. In addition the protocol will assess the passive immunity conferred to the newborn from these vaccine regimes. The study will provide evidence-based guidance for health policies on vaccination for the population of pregnant women during future H1N1v pandemics.”

Although with regard to H1N1-vaccination, appropriate studies are being done, it is feasible that certain measures might not be appropriate on basis of what we know. For instance, pretreating people in the non-risk groups (healthy young adults) with neuraminidase-inhibitors, because they are “indispensable employees”. Perhaps Heneghan, who as you remember is a co-author of the BMJ paper on neuraminidase -inhibitors in children with the seasonal flu, was thinking of this when writing his post.
If Heneghan would have directed his arrows at certain interventions in certain circumstances in certain people he might have had a good point, but now his arrows don’t hit any target. Revere from Effect Measure and Orac from Orac Knows might well have diagnosed him as someone who suffers from “methodolatry,” which is, as Revere puts it, the “profane worship of the randomized clinical trial as the only valid method of investigation.”

Notes

* But see the excellent post of Orac who trashes the Atlantic paper in Flu vaccination: Do The Atlantic, Shannon Brownlee, and Jeanne Lenzer matter? (scienceblogs.com). He also critiques the attitude of the Cochrane author Jefferson, who has a different voice in the media compared to the Cochrane Reviews he co-authors. Here he is far more neutral.

** There is no direct link to the data in the post. I’m not sure whether all pregnant women in the US are routinely tested for H1N1. (if not the percentage of H1N1 deaths among H1N1 infected pregnant women might be overestimated)

***In the US, vaccins given to pregnant women are without adjuvant.

Related articles by Zemanta

• Seasonal flu vaccine programs being questioned (ctv.ca)
• WHO: H1N1 is now world’s dominant flu virus (cnn.com)
• Quick Take: Novartis Vaccine Chief Andrin Oswald on Swine Flu and Vaccine Business (xconomy.com)
• FDA To America: Please, Don’t Be Idiots (boingboing.net)
• WHO says Tamiflu still works against H1N1, despite cases (ctv.ca)
• Swine Flu May Have Hit One Peak; More To Come (nlm.nih.gov)
• Allergists Urge People with Asthma Get Swine and Seasonal Flu Shots Following CDC Vaccine Advisory Committee Recommendations (prweb.com)
• Public Health, News, Epidemiology: H1N1 News and a Feed from CDC (creakysites.wordpress.com)
• Studies link seasonal flu shots to swine flu risk (thestar.com)
• Britain: Swine flu less deadly than first thought (seattletimes.nwsource.com)
• Flu vaccination: Do The Atlantic, Shannon Brownlee, and Jeanne Lenzer matter? (scienceblogs.com)

45,982

#CECEM Bridging the Gap between Evidence Based Practice and Practice Based Evidence

A very interesting presentation at the CECEM was given by the organizer of this continental Cochrane meeting, Rob de Bie. De Bie is Professor of Physiotherapy Research and director of Education of the Faculty of Health within the dept. of Epidemiology of the Maastricht University. He is both a certified physiotherapist and an epidemiologist. Luckily he kept the epidemiologic theory to a minimum. In fact he is a very engaging speaker who keeps your attention to the end.

Guidelines

While guidelines were already present in the Middle Ages in the form of formalized treatment of daily practice, more recently clinical guidelines have emerged. These are systematically developed statements which assists clinicians and patients in making decisions about appropriate treatement for specific conditions.

Currently, there are 3 kinds of guidelines, each with its own shortcomings.

• Consensus based. Consensus may be largely influenced by group dynamics
  Consensus = non-sensus and Consensus guidelines are guidelies.
• Expert based. Might be even worse than consensus. It can have all kind of biases, like expert and opinion bias or external financing.
• Evidence based. Guideline recommendations are based on best available evidence, deals with specific interventions for specific populations and are based on a systematic approach.

The quality of Evidence Based Guidelines depends on whether the evidence is good enough, transparent, credible, available, applied and not ‘muddled’ by health care insurers.
It is good to realize that some trials are never done, for instance because of ethical considerations. It is also true that only part of what you read (in the conclusions) has actually be done and some trials are republished several times, each time with a better outcome…

Systematic reviews and qualitatively good trials that don’t give answers.

Next Rob showed us the results of a study ( Jadad and McQuay in J. Clin. Epidemiol. ,1996) with efficacy as stated in the review plotted on the X-axis and the Quality score on the Y-axis. Surprisingly meta-analysis of high quality were less likely to produce positive results. Similar results were also obtained by Suttorp et al in 2006. (see Figure below)

Photo made by Chris Mavergames

There may be several reasons why good trials not always give good answers. Well known reasons are  the lack of randomization or blinding. However Rob focused on a less obvious reason. Despite its high level of evidence, a Randomized Controlled Trial (RCT) may not always be suitable to provide good answers applicable to all patients, because RCT’s often fail to reflect the true clinical practice. Often, the inclusion of patients in RCT’s is selective: middle-aged men with exclusion of co-morbidity. Whereas co-morbidity occurs in > 20% of the people of 60 years and older and in >40% of the people of 80 years and older (André Knottnerus in his speech).

Usefulness of a Nested Trial Cohort Study coupled to an EHR to study interventions.

Next, Rob showed that a nested Trial cohort study can be useful to study the effectiveness of  interventions. He used this in conjunction with an EHR (electronic health record), which could be accessed by practitioner and patient.

One of the diseases studied in this way, was Intermittent Claudication. Most commonly Intermittent Claudication is a manifestation of  peripheral arterial disease in the legs, causing pain and cramps in the legs while walking (hence the name). The mortality is high: the 5 year mortality rates are in between those of colorectal cancer and Non-Hodgkin Lymphoma. This is related to the underlying atherosclerosis.

There are several risk factors, some of which cannot be modified, like hereditary factors, age and gender. Other factors, like smoking, diet, physical inactivity and obesity can be tackled. These factors are interrelated.

Rob showed that, whereas there may be an overall null effect of exercise in the whole population, the effect may differ per subgroup.

• Patients with mild disease and no co-morbidity may directly benefit from exercise-therapy (blue area).
• Exercise has no effect on smokers, probably because smoking is the main causative factor.
• People with unstable diabetes first show an improvement, which stabilized after a few weeks due to hypo- or hyperglycaemia induced by the exercise,
• A similar effect is seen in COPD patients, the exercise becoming less effective because the patients become short of breath.

It is important to first regulate diabetes or COPD before continuing the exercise therapy. By individually optimizing the intervention(s) a far greater overall effect is achieved: 191% improval in the maximal (pain-free) walking distance compared to for instance <35% according to a Cochrane Systematic Review (2007).

Another striking effect: exercise therapy affects some of the prognostic factors: whereas there is no effect on BMI (this stays an important risk factor), age and diabetes become less important risk factors.

Because guidelines are quickly outdated, the findings are directly implemented in the existing guidelines.

Another astonishing fact: the physiotherapists pay for the system, not the patient nor the government.

More information can be found on https://www.cebp.nl/. Although the presentation is not (yet?) available on the net, I found a comparable presentation here.

** (2009-06-15) Good news: the program and all presentations can now be viewed at: https://www.cebp.nl/?NODE=239

Some Sugars Worse than Others? The Bittersweet Fructose/Glucose Debate.

Excessive consumption of sugar has been associated with increased incidences of type 2 diabetes, formerly called adult-onset diabetes, obesity and tooth decay.

There are many sugars around. Natural sugars and refined sugars. The refined table sugar and sugar cubes would be called “sucrose” by a chemist. Sucrose consists of two simple sugars (monosaccharides): 1 fructose and 1 glucose molecule (5).

Glucose is a sugar that occurs in the blood. Because of its name, fructose (Latin= fructus, fruit) is often regarded as more “natural” and therefore as a healthier alternative to glucose. However, unlike glucose, that can be metabolized anywhere in the body, fructose has to be metabolized by the liver. Here, fructose is easily converted to fat.

There is an intensive debate whether glucose or fructose is the real culprit for overweight and related health problems. This discussion is  relevant, because of the shift towards use of (cheaper) high fructose corn syrup from sucrose (especially in the US).

Last week a journal article was published in the Journal of Clinical Investigation, written by Stanhope et al (1) that was widely covered in the media. Headlines were for instance “Fructose slechter dan glucose” (NRC, Dutch, 8), “Fructose is the bad sugar, not glucose” (Indo-Asian News Service, i.e. 9) “Fructose-Sweetened Beverages Linked to Heart Risks” (NY-times, 10).

Is this study a breakthrough? What has been done?

This study was a double-blinded parallel arm study that assessed the relative effects of fructose- versus glucose – sweetened beverages in 32 matched, obese individuals, 40 to 72 years old (see 1).

The study consisted of 3 phases:

1. The first 2 weeks the volunteers lived in a clinical research center, consuming an energy- balanced high complex carbohydrate diet. This phase established baseline measurements for the study.
   
2. An 8-week outpatient intervention period during which subjects consumed either fructose- or glucose-sweetened beverages providing 25% of daily energy requirements along with their usual ad libitum diet. The aim was to imitate the ‘normal situation’, where sugar-sweetened beverages are typically consumed as part a normal energy-rich diet.
3. A 2-week inpatient intervention period during which subjects consumed fructose- or glucose-sweetened beverages providing 25% of daily energy requirements with an energy-balanced diet.

Results

Both study groups put on the same amount of weight, but people drinking fructose showed an increase in intra-abdominal fat, an increased hepatic de-novo (new) synthesis of lipids, higher triglyceride, LDL and oxidized LDL (“bad fats”), and higher fasting plasma glucose and insulin levels, but lowered insulin sensitivity. All these parameters are associated with a higher risk for diabetes and cardiovascular disease.

Positive Aspects of the study

• Intervention directly comparing fructose and glucose
• Human study
• Randomized Controlled Trial
• Many variables measured, related to diabetes and cardiovascular disease.

Critique:

• The first thing that came to my mind was: is it ethical to expose obese man and woman (or any healthy volunteer) to 10 weeks of a very unhealthy diet: extra glucose or fructose beverages making up 25% of the calorie intake?
• Because the subjects were obese, the results may not directly apply to lean persons.
• Minor point: It is a rather difficult to read paper, with a plethora of data. I wonder why SEM are given instead of SD and why the statistical significance is only determined versus baseline.
• Only surrogate markers were tested.
• Most important: the doses of sugars used are excessive, not reflecting a real-life diet.
• Nor can results with pure fructose be directly translated to health effects of high-fructose corn syrup, which is not pure fructose, but still contains 45% glucose.
• In addition the abstract and introduction suggests that it is the first human intervention study, which it isn’t.

Quite coincidentally the Journal of Nutrition published a supplement about “the State of the Science on Dietary Sweeteners Containing Fructose” [2-4]. In his paper Geoffrey Livesey [2] stresses the pitfalls of studies on Fructose, not only of animal and epidemiological studies, but also of intervention studies using excessive high fructose (excessive is > 400 kcal/day = >20% of energy intake), that may bear little relevance to the normal situation.

Many hypotheses of disease risk and prevention depend on inferences about the metabolic effects of fructose; however, there is inadequate attention to dose dependency. Fructose is proving to have bidirectional effects. At moderate or high doses, an effect on any one marker may be absent or even the opposite of that observed at very high or excessive doses; examples include fasting plasma triglyceride, insulin sensitivity (..) Among markers, changes can be beneficial for some (..) but adverse for others (e.g., plasma triglycerides at very high or excessive fructose intake). Evidence on body weight indicates no effect of moderate to high fructose intakes, but information is scarce for high or excessive intakes. The overall balance of such beneficial and adverse effects of fructose is difficult to assess but has important implications for the strength and direction of hypotheses about public health, the relevance of some animal studies, and the interpretation of both interventional and epidemiological studies.

References:

1. Stanhope, K., Schwarz, J., Keim, N., Griffen, S., Bremer, A., Graham, J., Hatcher, B., Cox, C., Dyachenko, A., Zhang, W., McGahan, J., Seibert, A., Krauss, R., Chiu, S., Schaefer, E., Ai, M., Otokozawa, S., Nakajima, K., Nakano, T., Beysen, C., Hellerstein, M., Berglund, L., & Havel, P. (2009). Consuming fructose-sweetened, not glucose-sweetened, beverages increases visceral adiposity and lipids and decreases insulin sensitivity in overweight/obese humans Journal of Clinical Investigation DOI: 10.1172/JCI37385
2. Livesey, G. (2009). Fructose Ingestion: Dose-Dependent Responses in Health Research Journal of Nutrition DOI: 10.3945/jn.108.097949
3. White, J. (2009). Misconceptions about High-Fructose Corn Syrup: Is It Uniquely Responsible for Obesity, Reactive Dicarbonyl Compounds, and Advanced Glycation Endproducts? Journal of Nutrition DOI: 10.3945/jn.108.097998
4. Jones, J. (2009). Dietary Sweeteners Containing Fructose: Overview of a Workshop on the State of the Science Journal of Nutrition DOI: 10.3945/jn.108.097972
5. Wikipedia: http://en.wikipedia.org/wiki/Sugar
   
6. Essentially Healthy Food: Sugar, a bittersweet story part 2
7. http://www.askmen.com/sports/foodcourt_250/257_health-benefits-of-sugar.html
8. NRC, April 21, 2009. http://www.nrc.nl/wetenschap/article2219138.ece/Fructose_slechter_dan_glucose
9. The Idian http://www.thaindian.com/newsportal/sci-tech/fructose-is-the-bad-sugar-not-glucose_100184408.html
10. NY Times,  April 2, 2009: Fructose-Sweetened Beverages Linked to Heart Risks

Photo Credits

• Sugarcube: http://www.flickr.com/photos/uwehermann/132244825/ (CC)
• Glucose/Frucose structure: http://www.elmhurst.edu/~chm/vchembook/543fructose.html
• Softdrinks: http://www.flickr.com/photos/tuis_imaging/3198244845/ (CC)
  

The Best Study Design… For Dummies.

When I had those tired looks again, my mother in law recommended coenzyme Q, which research had proven to have wondrous effects on tiredness. Indeed many sites and magazines advocate this natural energy producing nutrient which mobilizes your mitochondria for cellular energy! Another time she asked me if I thought komkommerslank (cucumber pills for slimming) would work to loose some extra weight. She took my NO for granted.

It is often difficult to explain people that not all research is equally good, and that outcomes are not always equally “significant” (both statistically and clinically). It is even more difficult to understand “levels of evidence” and why we should even care. Pharmaceutical Industries (especially the supplement-selling ones) take advantage of this ignorance and are very successful in selling their stories and pills.

If properly conducted, the Randomized Controlled Trial (RCT) is the best study-design to examine the clinical efficacy of health interventions. An RCT is an experimental study where individuals who are similar at the beginning are randomly allocated to two or more treatment groups and the outcomes of the groups are compared after sufficient follow-up time. However an RCT may not always be feasible, because it may not be ethical or desirable to randomize people or to expose them to certain interventions.

Observational studies provide weaker empirical evidence, because the allocation of factors is not under control of the investigator, but “just happen” or are choosen (e.g. smoking). Of the observational studies, cohort studies provide stronger evidence than case control studies, because in cohort studies factors are measured before the outcome, whereas in case controls studies factors are measured after the outcome.

Most people find such a description of study types and levels of evidence too theoretical and not appealing.

Last year I was challenged to tell about how doctors search medical information (central theme = Google) for and here it comes…. “the Society of History and ICT”.

To explain the audience why it is important for clinicians to find ‘the best evidence’ and how methodological filters can be used to sift through the overwhelming amount of information in for instance PubMed, I had to introduce RCT’s and the levels of evidence. To explain it to them I used an example that stroke me when I first read about it.

I showed them the following slide :

And clarified: Beta-carotene is a vitamine in carrots and many other vegetables, but you can also buy it in pure form as pills. There is reason to believe that beta-carotene might help to prevent lung cancer in cigarette smokers. How do you think you can find out whether beta-carotene will have this effect?

• Suppose you have two neighbors, both heavy smokers of the same age, both males. The neighbor who doesn’t eat much vegetables gets lung cancer, but the neighbor who eats a lot of vegetables and is fond of carrots doesn’t. Do you think this provides good evidence that beta-carotene prevents lung cancer?
  There is a laughter in the room, so they don’t believe in n=1 experiments/case reports. (still how many people don’t think smoking does not necessarily do any harm because “their chainsmoking father reached his nineties in good health”).
  I show them the following slide with the lowest box only.
• O.k. What about this study? I’ve a group of lung cancer patients, who smoke(d) heavily. I ask them to fill in a questionnaire about their eating habits in the past and take a blood sample, and I do the same with a simlar group of smokers without cancer (controls). Analysis shows that smokers developing lung cancer eat much less beta-carotene containing vegetables and have lower bloodlevels of beta-carotene than the smokers not developing cancer. Does this mean that beta-carotene is preventing lung cancer?
  Humming in the audience, till one man says: “perhaps some people don’t remember exactly what they eat” and then several people object “that it is just an association” and “you do not yet know whether beta-carotene really causes this”. Right! I show the box patient-control studies.
• Than consider this study design. I follow a large cohort of ‘healthy’ heavy smokers and look at their eating habits (including use of supplements) and take regular blood samples. After a long follow-up some heavy smokers develop lung cancer whereas others don’t. Now it turns out that the group that did not develop lung cancer had significantly more beta-carotene in their blood and eat larger amount of beta-carotene containing food. What do you think about that then?
  Now the room is a bit quiet, there is some hesitation. Then someone says: “well it is more convincing” and finally the chair says: “but it may still not be the carrots, but something else in their food or they may just have other healthy living habits (including eating carrots). Cohort-study appears on the slide (What a perfect audience!)
• O.k. you’re not convinced that these study designs give conclusive evidence. How could we then establish that beta-carotene lowers the risk of lung cancer in heavy smokers? Suppose you really wanted to know, how do you set up such a study?
  Grinning. Someone says “by giving half of the smokers beta-carotene and the other half nothing”. “Or a placebo”, someone else says. Right! Randomized Controlled Trial is on top of the slide. And there is not much room left for another box, so we are there. I only add that the best way to do it is to do it double blinded.

Than I reveal that all this research has really been done. There have been numerous observational studies (case-control as well cohorts studies) showing a consistent negative correlation between the intake of beta-carotene and the development of lung cancer in heavy smokers. The same has been shown for vitamin E.

“Knowing that”, I asked the public: “Would you as a heavy smoker participate in a trial where you are randomly assigned to one of the following groups: 1. beta-carotene, 2. vitamin E, 3. both or 4. neither vitamin (placebo)?”

The recruitment fails. Some people say they don’t believe in supplements, others say that it would be far more effective if smokers quit smoking (laughter). Just 2 individuals said they would at least consider it. But they thought there was a snag in it and they were right. Such studies have been done, and did not give the expected positive results.
In the first large RCT (appr. 30,000 male smokers!), the ATBC Cancer Prevention Study, beta-carotene rather increased the incidence of lung cancer with 18 percent and overall mortality with 8 percent (although harmful effects faded after men stopped taking the pills). Similar results were obtained in the CARET-study, but not in a 3rd RCT, the Physician’s Health Trial, the only difference being that the latter trial was performed both with smokers ànd non-smokers.
It is now generally thought that cigarette smoke causes beta-carotene to breakdown in detrimental products, a process that can be halted by other anti-oxidants (normally present in food). Whether vitamins act positively (anti-oxidant) or negatively (pro-oxidant) depends very much on the dose and the situation and on whether there is a shortage of such supplements or not.

I found that this way of explaining study designs to well-educated layman was very effective and fun!
The take-home message is that no matter how reproducible the observational studies seem to indicate a certain effect, better evidence is obtained by randomized control trials. It also shows that scientists should be very prudent to translate observational findings directly in a particular lifestyle advice.

On the other hand, I wonder whether all hypotheses have to be tested in a costly RCT (the costs for the ATCB trial were $46 million). Shouldn’t there be very very solid grounds to start a prevention study with dietary supplements in healthy individuals ? Aren’t their any dangers? Personally I think we should be very restrictive about these chemopreventive studies. Till now most chemopreventive studies have not met the high expectations, anyway.
And what about coenzyme-Q and komkommerslank? Besides that I do not expect the evidence to be convincing, tiredness can obviously be best combated by rest and I already eat enough cucumbers…. 😉
To be continued…

SOURCES:
Clinical Studies and designs: several paper books; online e.g. GlossClinStudy on a vetenarian site
The ATCB study: The Alpha-Tocopherol, Beta Carotene Cancer Prevention Study Group. The effect of vitamin E and beta carotene on the incidence of lung cancer and other cancers in male smokers. N Engl J Med 1994;330:1029-35. See free full text here
Overview of ATBC and CARET study 2 overwiews at www.cancer.gov, one about the ATBCfollowup and one about the CARET-trial
Overview of other RCT’s with surprising outcomes: on wikipedia

—————————-

Toen ik er weer eens donkere kringen onder mijn ogen had, raadde mijn schoonmoeder me coenzym Q aan, waarvan ze net gelezen had dat het een wetenschappelijk bewezen effectieve vermindering van vermoeidheid geeft. Inderdaad bevelen veel webpagina’s en huis-aan-huis bladen deze “natuurlijke en energie-mobiliserende voedingstof die je mitochondrien aanzet tot cellulaire energie” aan. Een andere keer vroeg ze me of ik dacht dat je wat pondjes zou kunnen kwijtraken door komkommerslank. Ze nam met een stellig NEE genoegen.

Het is vaak heel moeilijk duidelijk te maken dat niet alle research even goed is en niet alle uitkomsten even significant (zowel statistisch als klinisch). Nog moeilijker is het om de “bewijsniveau’s (levels of evidence)” uit de doeken te doen of om uit te leggen waarom je uberhaupt zoiets zou willen weten. De pharmacie en zeker de supplement verkopende bedrijfjes varen wel bij de goedgelovigheid van de gemiddelde consument. Hun verhalen en pillen vinden gretig aftrek.

Een gerandomiseerde gecontroleerde trial (RCT) is, indien goed uitgevoerd, het ‘beste’ studietype om aan te tonen of een behandeling werkzaam of nuttig is. Een RCT is een experimentele studie waarbij de deelnemers, die bij start van de studie vergelijkbaar zijn, door het lot worden toegewezen aan een (bepaalde) interventie- of controlegroep en waarbij na een bepaalde tijd de uitkomsten in beide groepen worden vergeleken. Een RCT is echter niet altijd haalbaar of wenselijk, bijvoorbeeld omdat het niet ethisch is om mensen te randomiseren of om hen bloot te stellen aan bepaalde interventies.

Observationele studies zijn zwakker in bewijskracht, omdat de toewijzing van de factoren niet in handen liggen van de onderzoeker, maar van het lot (natuur, werkomstandigheden, eigen keuzes). Van de observationele studies hebben cohort studies een hogere bewijskracht dan case control studies, omdat de te onderzoeken factoren bij cohort studies worden bepaald vòòrdat de uitkomst bekend is, terwijl dit bij case control studies juist gebeurt nadat de uitkomst bekend is.

De meeste mensen vinden zo’n beschrijving van studietypes the theoretisch en nietszeggend. Het spreekt ze niet aan, omdat ze zich er niets bij voor kunnen stellen.

Afgelopen jaar werd ik uitgedaagd om te vertellen “hoe artsen medische informatie zoeken” (het centrale thema was Google en informatie) voor …en nou komt-ie.. “De Vereniging voor Geschiedenis en Informatica”.

Om het publiek uit te leggen waarom het belangrijk is dat klinici de beste evidence vinden en hoe zij met behulp van methodologische filters het kaf van het koren kunnen scheiden, moest ik hen eerst uitleggen wat RCT’s en bewijsniveau’s zijn. Ik legde hen dit uit aan de hand van een onderwerp dat mij na aan het hart ligt.

Ik toonde hen een dia met daarop de vraag: Voorkomt bètacaroteen longkanker?
En ik legde hen uit dat bètacaroteen een vitamine is die veel in worteltjes en bepaalde andere groente voorkomt, maar die je ook als pillen kan slikken. Er zijn aanwijzingen dat bètacaroteen longkanker bij rokers zou helpen voorkomen. Hoe denkt u dat u dit het best kunt aantonen?

• Stel u heeft 2 buren, beiden kettingroker, man, en even oud. De buurman die weinig groente eet krijgt longkanker, maar de buurman die heel veel groente eet en gek is op worteltjes krijgt het niet. Kunnen we nu concluderen, dat bètacaroteen longkanker bij rokers voorkomt?
  Er wordt hartelijk gelachen, dus men gelooft hier niet in n=1 experimenten/case reports. (maar je moet ze maar de kost geven die denken dat roken geen kwaad kan omdat hun shaggies verslindende vader een taaie kerel was die nog tot op hoge leeftijd volop van het leven genoot).
  Ik toon hen een nieuwe slide met daarop het woord “casus”.
• O.k. Wat vind u hiervan? Er is een groep longkankerpatienten, die veel gerookt heeft. Ik vraag of de patienten een vragenlijst willen invullen en neem wat bloed af. Datzelfde doe ik met een vergelijkbare groep rokers die géén kanker ontwikkeld heeft. Ik analyseer alles en wat blijkt? In de groep met longkanker zitten vooral rokers met een minder caroteenrijk voedingspatroon en minder caroteen in hun bloed. Betekent dit dat bètacaroteen longkanker voorkomt?
  Geroezemoes. Iemand zegt: “misschien kunnen sommige mensen zich helemaal niet zo goed herinneren wat ze vroeger precies hebben gegeten”. Anderen werpen tegen “dat het alleen maar een associatie is” en “dat je niet zeker weet of het echt alleen de betacaroteen is die dit effect geeft.” Heel goed! Op het scherm verschijnt “patient-controle onderzoek”.
• Dan doe ik het anders. Ik volg een grote groep ‘gezonde’ zware rokers, ik volg hun eetgewoonten (inclusief het gebruik van supplementen) nauwgezet en neem regelmatig bloed af. Na lange tijd krijgen sommige rokers longkanker, maar andere niet. Nou blijkt wéér dat de groep die geen longkanker kreeg meer caroteenrijk voedsel at en meer betacaroteen in zijn bloed had. Wat denkt u hiervan?
  Het is een beetje stiller, je voelt de aarzeling, tot iemand zegt. “Nou, het overtuigt wel iets meer”. Een ander werpt tegen: “maar wie zegt dat het de worteltjes zijn, misschien is het wel iets anders in het eten of misschien is de levensstijl sowieso gezonder. Cohort-studie verschijnt op het scherm. Wat een perfect publiek, daar kan menig geneeskunde student nog een puntje aan zuigen.
• Ik begrijp dat u niet geheel overtuigd bent van de bewijskracht van deze studies. Maar hoe zou u dan vaststellen dat bètacaroteen de kans op longkanker verlaagt bij rokers? Stel dat u het echt zou willen weten, hoe zet u de studie dan op?
  Gegrinnik. Iemand zegt: “Geef de helft van de rokers beta-caroteen en de andere helft niets”. “Of een placebo”. zegt een ander. Prima! Er verschijnt “Randomized Controlled Trial” boven aan de slide. Er is geen ruimte meer over, dus we zijn er. Ik vertel alleen nog iets over het belang van dubbelblind onderzoek.

Dan onthul ik dat dergelijke onderzoeken ook werkelijk gedaan zijn. Uit tal van observationele (case-control en cohort) studies is gebleken dat er een omgekeerde relatie bestaat tussen inname van bètacaroteen en ontwikkelen van longkanker bij rokers. Hetzelfde geldt voor vitamine E.

“Nu u dat weet”, vraag ik het publiek “Zou u dan als kettingroker deelnemen aan een studie waar u random toegewezen wordt aan een van de volgende behandelingen:
1. bètacaroteen, 2. vitamine E, 3. beiden of 4. geen van beiden (placebo)?”

De inclusie faalt. Sommigen geloven niet in supplementen, anderen stellen dat het beter zou zijn als rokers stopten met roken (instemmend gelach). Twee mensen zeggen dat ze het in overweging zouden nemen. Maar men vermoed (gezien het voorafgaande) een addertje onder het gras en ze hebben gelijk. Dergelijke studies zijn gedaan en hebben niet het gewenste resultaat opgeleverd.
In de eerste grote RCT (ca. 30.000 mannelijke rokers!), de ATBC Cancer Prevention Study, verhoogde bètacaroteen juist het aantal longkankergevallen met 18% en de algehele sterfte met 8% (hoewel het effect langzaam uitdooft als mensen met de pillen stoppen). Vergelijkbare resultaten werden verkregen in de CARET-studie, maar niet in een 3e RCT, de Physician’s Health Trial, die ook niet-rokers had geincludeerd.
Aangenomen wordt nu dat sigaretterook bètacaroteen afbreekt tot gevaarlijke afvalproducten, een proces dat gekeerd kan worden door andere oxidanten (normaal aanwezig in het voedsel). Of vitaminen een positieve (anti-oxidant) of negatieve (pro-oxidant) werking hebben hangt erg af van de dosis, de vorm, en de situatie: vitamines in fysiologische concentraties zijn met name nuttig bij een tekort eraan.

Deze manier van studietypes uitleggen aan goed-ontwikkelde leken vond ik erg effectief en leuk om te doen bovendien.

De boodschap is dat hoezeer observationele studies ook op een bepaald effect wijzen, beter evidence verkregen kan worden met een RCT (hoewel dit niet altijd kan en mag). Het laat ook zien dat je als wetenschapper (en dokter) heel voorzichtig moet zijn met het direct vertalen van ‘waarnemingen’ naar adviezen richting een bepaalde therapie of levensstijl.

Aan de andere kant vraag ik me wel af, of alle hypothesen getest moeten worden in een overigens zeer kostbare RCT (kosten van de ATCB trial bedroegen $46 miljoen). Zou er niet een heel stevige basis moeten zijn vòòr je een preventieve studie doet met supplementen bij gezonde individuen? Zijn er geen risico’s ? Zelf denk ik dat we heel terughoudend moeten zijn met chemopreventie studies. Tot op heden hebben ze trouwens niet aan de hoge verachtingen voldaan.
Wat coenzym-Q en komkommerslank betreft? Behalve dat ik er gewoon niet in geloof (ik bedoel dat er evidence bestaat dat ze werken), denk ik dat vermoeidheid het best bestreden kan worden met rust en ik eet zat komkommers. 😉 Volgende keer meer…..