The Scatter of Medical Research and What to do About it.

18 05 2012

ResearchBlogging.orgPaul Glasziou, GP and professor in Evidence Based Medicine, co-authored a new article in the BMJ [1]. Similar to another paper [2] I discussed before [3] this paper deals with the difficulty for clinicians of staying up-to-date with the literature. But where the previous paper [2,3] highlighted the mere increase in number of research articles over time, the current paper looks at the scatter of randomized clinical trials (RCTs) and systematic reviews (SR’s) accross different journals cited in one year (2009) in PubMed.

Hofmann et al analyzed 7 specialties and 9 sub-specialties, that are considered the leading contributions to the burden of disease in high income countries.

They followed a relative straightforward method for identifying the publications. Each search string consisted of a MeSH term (controlled  term) to identify the selected disease or disorders, a publication type [pt] to identify the type of study, and the year of publication. For example, the search strategy for randomized trials in cardiology was: “heart diseases”[MeSH] AND randomized controlled trial[pt] AND 2009[dp]. (when searching “heart diseases” as a MeSH, narrower terms are also searched.) Meta-analysis[pt] was used to identify systematic reviews.

Using this approach Hofmann et al found 14 343 RCTs and 3214 SR’s published in 2009 in the field of the selected (sub)specialties. There was a clear scatter across journals, but this scatter varied considerably among specialties:

“Otolaryngology had the least scatter (363 trials across 167 journals) and neurology the most (2770 trials across 896 journals). In only three subspecialties (lung cancer, chronic obstructive pulmonary disease, hearing loss) were 10 or fewer journals needed to locate 50% of trials. The scatter was less for systematic reviews: hearing loss had the least scatter (10 reviews across nine journals) and cancer the most (670 reviews across 279 journals). For some specialties and subspecialties the papers were concentrated in specialty journals; whereas for others, few of the top 10 journals were a specialty journal for that area.
Generally, little overlap occurred between the top 10 journals publishing trials and those publishing systematic reviews. The number of journals required to find all trials or reviews was highly correlated (r=0.97) with the number of papers for each specialty/ subspecialty.”

Previous work already suggested that this scatter of research has a long tail. Half of the publications is in a minority of papers, whereas the remaining articles are scattered among many journals (see Fig below).

Click to enlarge en see legends at BMJ 2012;344:e3223 [CC]

The good news is that SRs are less scattered and that general journals appear more often in the top 10 journals publishing SRs. Indeed for 6 of the 7 specialties and 4 of the 9 subspecialties, the Cochrane Database of Systematic Reviews had published the highest number of systematic reviews, publishing between 6% and 18% of all the systematic reviews published in each area in 2009. The bad news is that even keeping up to date with SRs seems a huge, if not impossible, challenge.

In other words, it is not sufficient for clinicians to rely on personal subscriptions to a few journals in their specialty (which is common practice). Hoffmann et al suggest several solutions to help clinicians cope with the increasing volume and scatter of research publications.

  • a central library of systematic reviews (but apparently the Cochrane Library fails to fulfill such a role according to the authors, because many reviews are out of date and are perceived as less clinically relevant)
  • registry of planned and completed systematic reviews, such as prospero. (this makes it easier to locate SRs and reduces bias)
  • Synthesis of Evidence and synopses, like the ACP-Jounal Club which summarizes the best evidence in internal medicine
  • Specialised databases that collate and critically appraise randomized trials and systematic reviews, like www.pedro.org.au for physical therapy. In my personal experience, however, this database is often out of date and not comprehensive
  • Journal scanning services like EvidenceUpdates from mcmaster.ca), which scans over 120 journals, filters articles on the basis of quality, has practising clinicians rate them for relevance and newsworthiness, and makes them available as email alerts and in a searchable database. I use this service too, but besides that not all specialties are covered, the rating of evidence may not always be objective (see previous post [4])
  • The use of social media tools to alert clinicians to important new research.

Most of these solutions are (long) existing solutions that do not or only partly help to solve the information overload.

I was surprised that the authors didn’t propose the use of personalized alerts. PubMed’s My NCBI feature allows to create automatic email alerts on a topic and to subscribe to electronic tables of contents (which could include ACP journal Club). Suppose that a physician browses 10 journals roughly covering 25% of the trials. He/she does not need to read all the other journals from cover to cover to avoid missing one potentially relevant trial. Instead it is far more efficient to perform a topic search to filter relevant studies from journals that seldom publish trials on the topic of interest. One could even use the search of Hoffmann et al to achieve this.* Although in reality, most clinical researchers will have narrower fields of interest than all studies about endocrinology and neurology.

At our library we are working at creating deduplicated, easy to read, alerts that collate table of contents of certain journals with topic (and author) searches in PubMed, EMBASE and other databases. There are existing tools that do the same.

Another way to reduce the individual work (reading) load is to organize journals clubs or even better organize regular CATs (critical appraised topics). In the Netherlands, CATS are a compulsory item for residents. A few doctors do the work for many. Usually they choose topics that are clinically relevant (or for which the evidence is unclear).

The authors shortly mention that their search strategy might have missed  missed some eligible papers and included some that are not truly RCTs or SRs, because they relied on PubMed’s publication type to retrieve RCTs and SRs. For systematic reviews this may be a greater problem than recognized, for the authors have used meta-analyses[pt] to identify systematic reviews. Unfortunately PubMed has no publication type for systematic reviews, but it may be clear that there are many more systematic reviews that meta-analyses. Possibly systematical reviews might even have a different scatter pattern than meta-analyses (i.e. the latter might be preferentially included in core journals).

Furthermore not all meta-analyses and systematic reviews are reviews of RCTs (thus it is not completely fair to compare MAs with RCTs only). On the other hand it is a (not discussed) omission of this study, that only interventions are considered. Nowadays physicians have many other questions than those related to therapy, like questions about prognosis, harm and diagnosis.

I did a little imperfect search just to see whether use of other search terms than meta-analyses[pt] would have any influence on the outcome. I search for (1) meta-analyses [pt] and (2) systematic review [tiab] (title and abstract) of papers about endocrine diseases. Then I subtracted 1 from 2 (to analyse the systematic reviews not indexed as meta-analysis[pt])

Thus:

(ENDOCRINE DISEASES[MESH] AND SYSTEMATIC REVIEW[TIAB] AND 2009[DP]) NOT META-ANALYSIS[PT]

I analyzed the top 10/11 journals publishing these study types.

This little experiment suggests that:

  1. the precise scatter might differ per search: apparently the systematic review[tiab] search yielded different top 10/11 journals (for this sample) than the meta-analysis[pt] search. (partially because Cochrane systematic reviews apparently don’t mention systematic reviews in title and abstract?).
  2. the authors underestimate the numbers of Systematic Reviews: simply searching for systematic review[tiab] already found appr. 50% additional systematic reviews compared to meta-analysis[pt] alone
  3. As expected (by me at last), many of the SR’s en MA’s were NOT dealing with interventions, i.e. see the first 5 hits (out of 108 and 236 respectively).
  4. Together these findings indicate that the true information overload is far greater than shown by Hoffmann et al (not all systematic reviews are found, of all available search designs only RCTs are searched).
  5. On the other hand this indirectly shows that SRs are a better way to keep up-to-date than suggested: SRs  also summarize non-interventional research (the ratio SRs of RCTs: individual RCTs is much lower than suggested)
  6. It also means that the role of the Cochrane Systematic reviews to aggregate RCTs is underestimated by the published graphs (the MA[pt] section is diluted with non-RCT- systematic reviews, thus the proportion of the Cochrane SRs in the interventional MAs becomes larger)

Well anyway, these imperfections do not contradict the main point of this paper: that trials are scattered across hundreds of general and specialty journals and that “systematic reviews” (or meta-analyses really) do reduce the extent of scatter, but are still widely scattered and mostly in different journals to those of randomized trials.

Indeed, personal subscriptions to journals seem insufficient for keeping up to date.
Besides supplementing subscription by  methods such as journal scanning services, I would recommend the use of personalized alerts from PubMed and several prefiltered sources including an EBM search machine like TRIP (www.tripdatabase.com/).

*but I would broaden it to find all aggregate evidence, including ACP, Clinical Evidence, syntheses and synopses, not only meta-analyses.

**I do appreciate that one of the co-authors is a medical librarian: Sarah Thorning.

References

  1. Hoffmann, Tammy, Erueti, Chrissy, Thorning, Sarah, & Glasziou, Paul (2012). The scatter of research: cross sectional comparison of randomised trials and systematic reviews across specialties BMJ, 344 : 10.1136/bmj.e3223
  2. Bastian, H., Glasziou, P., & Chalmers, I. (2010). Seventy-Five Trials and Eleven Systematic Reviews a Day: How Will We Ever Keep Up? PLoS Medicine, 7 (9) DOI: 10.1371/journal.pmed.1000326
  3. How will we ever keep up with 75 trials and 11 systematic reviews a day (laikaspoetnik.wordpress.com)
  4. Experience versus Evidence [1]. Opioid Therapy for Rheumatoid Arthritis Pain. (laikaspoetnik.wordpress.com)




Evidence Based Point of Care Summaries [2] More Uptodate with Dynamed.

18 10 2011

ResearchBlogging.orgThis post is part of a short series about Evidence Based Point of Care Summaries or POCs. In this series I will review 3 recent papers that objectively compare a selection of POCs.

In the previous post I reviewed a paper from Rita Banzi and colleagues from the Italian Cochrane Centre [1]. They analyzed 18 POCs with respect to their “volume”, content development and editorial policy. There were large differences among POCs, especially with regard to evidence-based methodology scores, but no product appeared the best according to the criteria used.

In this post I will review another paper by Banzi et al, published in the BMJ a few weeks ago [2].

This article examined the speed with which EBP-point of care summaries were updated using a prospective cohort design.

First the authors selected all the systematic reviews signaled by the American College of Physicians (ACP) Journal Club and Evidence-Based Medicine Primary Care and Internal Medicine from April to December 2009. In the same period the authors selected all the Cochrane systematic reviews labelled as “conclusion changed” in the Cochrane Library. In total 128 systematic reviews were retrieved, 68 from the literature surveillance journals (53%) and 60 (47%) from the Cochrane Library. Two months after the collection started (June 2009) the authors did a monthly screen for a year to look for potential citation of the identified 128 systematic reviews in the POCs.

Only those 5 POCs were studied that were ranked in the top quarter for at least 2 (out of 3) desirable dimensions, namely: Clinical Evidence, Dynamed, EBM Guidelines, UpToDate and eMedicine. Surprisingly eMedicine was among the selected POCs, having a rating of “1” on a scale of 1 to 15 for EBM methodology. One would think that Evidence-based-ness is a fundamental prerequisite  for EBM-POCs…..?!

Results were represented as a (rather odd, but clear) “survival analysis” ( “death” = a citation in a summary).

Fig.1 : Updating curves for relevant evidence by POCs (from [2])

I will be brief about the results.

Dynamed clearly beated all the other products  in its updating speed.

Expressed in figures, the updating speed of Dynamed was 78% and 97% greater than those of EBM Guidelines and Clinical Evidence, respectively. Dynamed had a median citation rate of around two months and EBM Guidelines around 10 months, quite close to the limit of the follow-up, but the citation rate of the other three point of care summaries (UpToDate, eMedicine, Clinical Evidence) were so slow that they exceeded the follow-up period and the authors could not compute the median.

Dynamed outperformed the other POC’s in updating of systematic reviews independent of the route. EBM Guidelines and UpToDate had similar overall updating rates, but Cochrane systematic reviews were more likely to be cited by EBM Guidelines than by UpToDate (odds ratio 0.02, P<0.001). Perhaps not surprising, as EBM Guidelines has a formal agreement with the Cochrane Collaboration to use Cochrane contents and label its summaries as “Cochrane inside.” On the other hand, UpToDate was faster than EBM Guidelines in updating systematic reviews signaled by literature surveillance journals.

Dynamed‘s higher updating ability was not due to a difference in identifying important new evidence, but to the speed with which this new information was incorporated in their summaries. Possibly the central updating of Dynamed by the editorial team might account for the more prompt inclusion of evidence.

As the authors rightly point out, slowness in updating could mean that new relevant information is ignored and could thus affect the validity of point of care information services”.

A slower updating rate may be considered more important for POCs that “promise” to “continuously update their evidence summaries” (EBM-Guidelines) or to “perform a continuous comprehensive review and to revise chapters whenever important new information is published, not according to any specific time schedule” (UpToDate). (see table with description of updating mechanisms )

In contrast, Emedicine doesn’t provide any detailed information on updating policy, another reason that it doesn’t belong to this list of best POCs.
Clinical Evidence, however, clearly states, We aim to update Clinical Evidence reviews annually. In addition to this cycle, details of clinically important studies are added to the relevant reviews throughout the year using the BMJ Updates service.” But BMJ Updates is not considered in the current analysis. Furthermore, patience is rewarded with excellent and complete summaries of evidence (in my opinion).

Indeed a major limitation of the current (and the previous) study by Banzi et al [1,2] is that they have looked at quantitative aspects and items that are relatively “easy to score”, like “volume” and “editorial quality”, not at the real quality of the evidence (previous post).

Although the findings were new to me, others have recently published similar results (studies were performed in the same time-span):

Shurtz and Foster [3] of the Texas A&M University Medical Sciences Library (MSL) also sought to establish a rubric for evaluating evidence-based medicine (EBM) point-of-care tools in a health sciences library.

They, too, looked at editorial quality and speed of updating plus reviewing content, search options, quality control, and grading.

Their main conclusion is that “differences between EBM tools’ options, content coverage, and usability were minimal, but that the products’ methods for locating and grading evidence varied widely in transparency and process”.

Thus this is in line with what Banzi et al reported in their first paper. They also share Banzi’s conclusion about differences in speed of updating

“DynaMed had the most up-to-date summaries (updated on average within 19 days), while First Consult had the least up to date (updated on average within 449 days). Six tools claimed to update summaries within 6 months or less. For the 10 topics searched, however, only DynaMed met this claim.”

Table 3 from Shurtz and Foster [3] 

Ketchum et al [4] also conclude that DynaMed the largest proportion of current (2007-2009) references (170/1131, 15%). In addition they found that Dynamed had the largest total number of references (1131/2330, 48.5%).

Yes, and you might have guessed it. The paper of Andrea Ketchum is the 3rd paper I’m going to review.

I also recommend to read the paper of the librarians Shurtz and Foster [3], which I found along the way. It has too much overlap with the Banzi papers to devote a separate post to it. Still it provides better background information then the Banzi papers, it focuses on POCs that claim to be EBM and doesn’t try to weigh one element over another. 

References

  1. Banzi, R., Liberati, A., Moschetti, I., Tagliabue, L., & Moja, L. (2010). A Review of Online Evidence-based Practice Point-of-Care Information Summary Providers Journal of Medical Internet Research, 12 (3) DOI: 10.2196/jmir.1288
  2. Banzi, R., Cinquini, M., Liberati, A., Moschetti, I., Pecoraro, V., Tagliabue, L., & Moja, L. (2011). Speed of updating online evidence based point of care summaries: prospective cohort analysis BMJ, 343 (sep22 2) DOI: 10.1136/bmj.d5856
  3. Shurtz, S., & Foster, M. (2011). Developing and using a rubric for evaluating evidence-based medicine point-of-care tools Journal of the Medical Library Association : JMLA, 99 (3), 247-254 DOI: 10.3163/1536-5050.99.3.012
  4. Ketchum, A., Saleh, A., & Jeong, K. (2011). Type of Evidence Behind Point-of-Care Clinical Information Products: A Bibliometric Analysis Journal of Medical Internet Research, 13 (1) DOI: 10.2196/jmir.1539
  5. Evidence Based Point of Care Summaries [1] No “Best” Among the Bests? (laikaspoetnik.wordpress.com)
  6. How will we ever keep up with 75 Trials and 11 Systematic Reviews a Day? (laikaspoetnik.wordpress.com
  7. UpToDate or Dynamed? (Shamsha Damani at laikaspoetnik.wordpress.com)
  8. How Evidence Based is UpToDate really? (laikaspoetnik.wordpress.com)

Related articles (automatically generated)





Evidence Based Point of Care Summaries [1] No “Best” Among the Bests?

13 10 2011

ResearchBlogging.orgFor many of today’s busy practicing clinicians, keeping up with the enormous and ever growing amount of medical information, poses substantial challenges [6]. Its impractical to do a PubMed search to answer each clinical question and then synthesize and appraise the evidence. Simply, because busy health care providers have limited time and many questions per day.

As repeatedly mentioned on this blog ([6-7]), it is far more efficient to try to find aggregate (or pre-filtered or pre-appraised) evidence first.

Haynes ‘‘5S’’ levels of evidence (adapted by [1])

There are several forms of aggregate evidence, often represented as the higher layers of an evidence pyramid (because they aggregate individual studies, represented by the lowest layer). There are confusingly many pyramids, however [8] with different kinds of hierarchies and based on different principles.

According to the “5S” paradigm[9] (now evolved to 6S -[10]) the peak of the pyramid are the ideal but not yet realized computer decision support systems, that link the individual patient characteristics to the current best evidence. According to the 5S model the next best source are Evidence Based Textbooks.
(Note: EBM and textbooks almost seem a contradiction in terms to me, personally I would not put many of the POCs somewhere at the top. Also see my post: How Evidence Based is UpToDate really?)

Whatever their exact place in the EBM-pyramid, these POCs are helpful to many clinicians. There are many different POCs (see The HLWIKI Canada for a comprehensive overview [11]) with a wide range of costs, varying from free with ads (e-Medicine) to very expensive site licenses (UpToDate). Because of the costs, hospital libraries have to choose among them.

Choices are often based on user preferences and satisfaction and balanced against costs, scope of coverage etc. Choices are often subjective and people tend to stick to the databases they know.

Initial literature about POCs concentrated on user preferences and satisfaction. A New Zealand study [3] among 84 GPs showed no significant difference in preference for, or usage levels of DynaMed, MD Consult (including FirstConsult) and UpToDate. The proportion of questions adequately answered by POCs differed per study (see introduction of [4] for an overview) varying from 20% to 70%.
McKibbon and Fridsma ([5] cited in [4]) found that the information resources chosen by primary care physicians were seldom helpful in providing the correct answers, leading them to conclude that:

“…the evidence base of the resources must be strong and current…We need to evaluate them well to determine how best to harness the resources to support good clinical decision making.”

Recent studies have tried to objectively compare online point-of-care summaries with respect to their breadth, content development, editorial policy, the speed of updating and the type of evidence cited. I will discuss 3 of these recent papers, but will review each paper separately. (My posts tend to be pretty long and in-depth. So in an effort to keep them readable I try to cut down where possible.)

Two of the three papers are published by Rita Banzi and colleagues from the Italian Cochrane Centre.

In the first paper, reviewed here, Banzi et al [1] first identified English Web-based POCs using Medline, Google, librarian association websites, and information conference proceedings from January to December 2008. In order to be eligible, a product had to be an online-delivered summary that is regularly updated, claims to provide evidence-based information and is to be used at the bedside.

They found 30 eligible POCs, of which the following 18 databases met the criteria: 5-Minute Clinical Consult, ACP-Pier, BestBETs, CKS (NHS), Clinical Evidence, DynaMed, eMedicine,  eTG complete, EBM Guidelines, First Consult, GP Notebook, Harrison’s Practice, Health Gate, Map Of Medicine, Micromedex, Pepid, UpToDate, ZynxEvidence.

They assessed and ranked these 18 point-of-care products according to: (1) coverage (volume) of medical conditions, (2) editorial quality, and (3) evidence-based methodology. (For operational definitions see appendix 1)

From a quantitive perspective DynaMed, eMedicine, and First Consult were the most comprehensive (88%) and eTG complete the least (45%).

The best editorial quality of EBP was delivered by Clinical Evidence (15), UpToDate (15), eMedicine (13), Dynamed (11) and eTG complete (10). (Scores are shown in brackets)

Finally, BestBETs, Clinical Evidence, EBM Guidelines and UpToDate obtained the maximal score (15 points each) for best evidence-based methodology, followed by DynaMed and Map Of Medicine (12 points each).
As expected eMedicine, eTG complete, First Consult, GP Notebook and Harrison’s Practice had a very low EBM score (1 point each). Personally I would not have even considered these online sources as “evidence based”.

The calculations seem very “exact”, but assumptions upon which these figures are based are open to question in my view. Furthermore all items have the same weight. Isn’t the evidence-based methodology far more important than “comprehensiveness” and editorial quality?

Certainly because “volume” is “just” estimated by analyzing to which extent 4 random chapters of the ICD-10 classification are covered by the POCs. Some sources, like Clinical Evidence and BestBets (scoring low for this item) don’t aim to be comprehensive but only “answer” a limited number of questions: they are not textbooks.

Editorial quality is determined by scoring of the specific indicators of transparency: authorship, peer reviewing procedure, updating, disclosure of authors’ conflicts of interest, and commercial support of content development.

For the EB methodology, Banzi et al scored the following indicators:

  1. Is a systematic literature search or surveillance the basis of content development?
  2. Is the critical appraisal method fully described?
  3. Are systematic reviews preferred over other types of publication?
  4. Is there a system for grading the quality of evidence?
  5. When expert opinion is included is it easily recognizable over studies’ data and results ?

The  score for each of these indicators is 3 for “yes”, 1 for “unclear”, and 0 for “no” ( if judged “not adequate” or “not reported.”)

This leaves little room for qualitative differences and mainly relies upon adequate reporting. As discussed earlier in a post where I questioned the evidence-based-ness of UpToDate, there is a difference between tailored searches and checking a limited list of sources (indicator 1.). It also matters whether the search is mentioned or not (transparency), whether it is qualitatively ok and whether it is extensive or not. For lists, it matters how many sources are “surveyed”. It also matters whether one or both methods are used… These important differences are not reflected by the scores.

Furthermore some points may be more important than others. Personally I find step 1 the most important. For what good is appraising and grading if it isn’t applied to the most relevant evidence? It is “easy” to do a grading or to copy it from other sources (yes, I wouldn’t be surprised if some POCs are doing this).

On the other hand, a zero for one indicator can have too much weight on the score.

Dynamed got 12 instead of the maximum 15 points, because their editorial policy page didn’t explicitly describe their absolute prioritization of systematic reviews although they really adhere to that in practice (see comment by editor-in-chief  Brian Alper [2]). Had Dynamed received the deserved 15 points for this indicator, they would have had the highest score overall.

The authors further conclude that none of the dimensions turned out to be significantly associated with the other dimensions. For example, BestBETs scored among the worst on volume (comprehensiveness), with an intermediate score for editorial quality, and the highest score for evidence-based methodology.  Overall, DynaMed, EBM Guidelines, and UpToDate scored in the top quartile for 2 out of 3 variables and in the 2nd quartile for the 3rd of these variables. (but as explained above Dynamed really scored in the top quartile for all 3 variables)

On basis of their findings Banzi et al conclude that only a few POCs satisfied the criteria, with none excelling in all.

The finding that Pepid, eMedicine, eTG complete, First Consult, GP Notebook, Harrison’s Practice and 5-Minute Clinical Consult only obtained 1 or 2 of the maximum 15 points for EBM methodology confirms my “intuitive grasp” that these sources really don’t deserve the label “evidence based”. Perhaps we should make a more strict distinction between “point of care” databases as a point where patients and practitioners interact, particularly referring to the context of the provider-patient dyad (definition by Banzi et al) and truly evidence based summaries. Only few of the tested databases would fit the latter definition. 

In summary, Banzi et al reviewed 18 Online Evidence-based Practice Point-of-Care Information Summary Providers. They comprehensively evaluated and summarized these resources with respect to coverage (volume) of medical conditions, editorial quality, and (3) evidence-based methodology. 

Limitations of the study, also according to the authors, were the lack of a clear definition of these products, arbitrariness of the scoring system and emphasis on the quality of reporting. Furthermore the study didn’t really assess the products qualitatively (i.e. with respect to performance). Nor did it take into account that products might have a different aim. Clinical Evidence only summarizes evidence on the effectiveness of treatments of a limited number of diseases, for instance. Therefore it scores bad on volume while excelling on the other items. 

Nevertheless it is helpful that POCs are objectively compared and it may help as starting point for decisions about acquisition.

References (not in chronological order)

  1. Banzi, R., Liberati, A., Moschetti, I., Tagliabue, L., & Moja, L. (2010). A Review of Online Evidence-based Practice Point-of-Care Information Summary Providers Journal of Medical Internet Research, 12 (3) DOI: 10.2196/jmir.1288
  2. Alper, B. (2010). Review of Online Evidence-based Practice Point-of-Care Information Summary Providers: Response by the Publisher of DynaMed Journal of Medical Internet Research, 12 (3) DOI: 10.2196/jmir.1622
  3. Goodyear-Smith F, Kerse N, Warren J, & Arroll B (2008). Evaluation of e-textbooks. DynaMed, MD Consult and UpToDate. Australian family physician, 37 (10), 878-82 PMID: 19002313
  4. Ketchum, A., Saleh, A., & Jeong, K. (2011). Type of Evidence Behind Point-of-Care Clinical Information Products: A Bibliometric Analysis Journal of Medical Internet Research, 13 (1) DOI: 10.2196/jmir.1539
  5. McKibbon, K., & Fridsma, D. (2006). Effectiveness of Clinician-selected Electronic Information Resources for Answering Primary Care Physicians’ Information Needs Journal of the American Medical Informatics Association, 13 (6), 653-659 DOI: 10.1197/jamia.M2087
  6. How will we ever keep up with 75 Trials and 11 Systematic Reviews a Day? (laikaspoetnik.wordpress.com)
  7. 10 + 1 PubMed Tips for Residents (and their Instructors) (laikaspoetnik.wordpress.com)
  8. Time to weed the (EBM-)pyramids?! (laikaspoetnik.wordpress.com)
  9. Haynes RB. Of studies, syntheses, synopses, summaries, and systems: the “5S” evolution of information services for evidence-based healthcare decisions. Evid Based Med 2006 Dec;11(6):162-164. [PubMed]
  10. DiCenso A, Bayley L, Haynes RB. ACP Journal Club. Editorial: Accessing preappraised evidence: fine-tuning the 5S model into a 6S model. Ann Intern Med. 2009 Sep 15;151(6):JC3-2, JC3-3. PubMed PMID: 19755349 [free full text].
  11. How Evidence Based is UpToDate really? (laikaspoetnik.wordpress.com)
  12. Point_of_care_decision-making_tools_-_Overview (hlwiki.slais.ubc.ca)
  13. UpToDate or Dynamed? (Shamsha Damani at laikaspoetnik.wordpress.com)

Related articles (automatically generated)





The #TwitJC Twitter Journal Club, a New Initiative on Twitter. Some Initial Thoughts.

10 06 2011

There is a new initiative on Twitter: The Twitter Journal Club. It is initiated by Fi Douglas (@fidouglas) a medical student at Cambridge,  and Natalie Silvey (@silv24)  a junior doctor in the West Midlands.

Fi and Natalie have set up a blog for this event: http://twitjc.wordpress.com/

A Twitter Journal Club operates in the same way as any other journal club, except that the forum is Twitter.

The organizers choose a paper, which they announce at their website (you can make suggestions here or via a tweet). Ideally, people should read the entire paper before the Twitter session. A short summary with key points (i.e. see here) is posted on the website.

The first topic was:  Early Goal-Directed Therapy in the Treatment of Severe Sepsis and Septic Shock [PDF]

It started last Sunday 8 pm (Dutch time) and took almost 2 hours to complete.

@twitjournalclub (the twitter account of the organizers) started with a short introduction. People introduced themselves as they entered the discussion. Each tweet in the discussion was tagged with #TwitJC (a so called hashtag), otherwise it won’t get picked up by people following the hashtag. (Tweetchat automatically provides the hashtag you type in).

Although it was the first session, many people (perhaps almost 100?!) joined the Journal Club, both actively and more passively. That is a terrific achievement. Afterwards it got a very positive Twitter “press”. If you know to engage people like @nothern_doctor, @doctorblogs, @amcunningham and @drgrumble and people like @bengoldacre, @cebmblog and @david_colquhoun find it a terrific concept, then you know that it is a great idea that meets a need. As such, enough reason to continue.

There were also not purely positive voices. @DrVes sees it as a great effort, but added that “we need to go beyond this 1950s model rather than adapt it to social media.” Apparently this tweet was not well received, but I think he made a very sensible statement.

We can (and should) asks ourselves if Twitter is the right medium for such an event.

@DrVes has experience with Twitter Journal Clubs. He participated in the first medical journal club on Twitter at the Allergy and Immunology program of Creighton University back in 2008 and presented a poster at an allergy meeting in 2009.

BUT, as far as I can tell, that Twitter Journal Club was both much more small-scale (7 fellows?) and different in design. It seems that Tweets summarized what was being said at a real journal club teaching session. Ves Dimov:

“The updates were followed in real time by the Allergy and Immunology fellows at the Louisiana State University (Shreveport) and some interested residents at Cleveland Clinic, along with the 309 subscribers of my Twitter account named AllergyNotes“.

So that is the same as tweeting during a conference or a lecture to inform others about the most interesting facts/statements. It is one-way-tweeting (overall there were just 24 updates with links).

I think the present  Twitter Journal Club was more like a medical Twitter chat (also the words of Ves).

Is chatting on Twitter effective?

Well that depends on what one wants to achieve.

Apparently for all people participating, it was fun to do and educative.

I joined too late to tell, thus I awaited the transcript. But boy, who wants to read 31 pages of “chaotic tweets”? Because that is what a Twitter chat is if many people join.  All tweets are ordered chronologically. Good for the archive, but if the intention is to make the transcribed chat available to people who couldn’t attend, it needs deleting, cutting, pasting and sorting. But that is a lot of work if done manually.

I tried it for part of the transcript. Compare the original transcript here with this Google Doc.

The “remix of tweets” also illustrates that people have their own “mini-chats”, and “off-topic” (but often very relevant) questions.

In addition, the audience is very mixed. Some people seem to have little experience with critical appraisal or concepts like “intention to treat” (ITT) and would perhaps benefit from supplementary information beforehand (i.e. documents at the TwitJC website). Others are experienced doctors with a lot of clinical expertise, who always put theoretical things in perspective. Very valuable, but often they are far ahead in the discussion.

The name of the event is Twitter  Journal Club. Journal Club is a somewhat ambiguous term. According to Wikipedia “A journal club is a group of individuals who meet regularly to critically evaluate recent articles in scientific literature”. It can deal with any piece which looks interesting to share, including hypotheses and preclinical papers about mechanisms of actions.

Thus, to me Journal club is not per definition EBM (Evidence Based Medicine).

Other initiatives are a critical appraisal of a study and a CAT,  a critical appraisal of a topic (sometimes wrongly called PICO, PICO is only part of it).

The structure of the present journal club was more that of a critical appraisal. It followed the normal checklist for an RCT: What is being studied? Is the paper valid (appropriately allocated, blinded etc ), what are the results (NNT etc) and are the results valid outside of the context of the paper?

Imo, official critical appraisal of the paper costs a lot of time and is not the most interesting. Looking at my edited transcript you see that half of the people are answering the question and they all say the same: “Clearly focused question” is answer to first question (but even in the edited transcript this takes 3 pages), “clear interventions (helpful flowcharts) is the answer to the second question.

Half of the people have their own questions. Very legitimate and good questions, but not in line with the questions of @twitjournalclub. Talking about the NNT and about whether the results are really revolutionary, is VERY relevant, but should be left till the end.

A twitter chat with appr. 100 people needs a tight structure.

However, I wonder whether this  approach of critical appraisal is the most interesting. Even more so, because this part didn’t evoke much discussion.

Plus it has already been done!!

I searched the TRIP database and with the title of the paper, to find critical appraisals or synopses of the paper. I found 3 synopses, 2 of which follow more or less the structure of this journal club here, here (and this older one). They answer all the questions about validity.

Wouldn’t it have better with this older key paper (2001) to just use the existing critical appraisals as background information and discuss the implications? Or discuss new supporting or contradictory findings?

The very limited search in TRIP (title of paper only) showed some new interesting papers on the topic (external validation, cost effectiveness, implementation, antibiotics) and I am sure there are many more.

A CAT may also be more interesting than a synopsis, because “other pieces of evidence” are also taken into consideration and one discusses a topic not one single paper. But perhaps this is too difficult to do, because one has to do a thorough search as well and has too much to discuss. Alternatively one could choose a recent systematic review, which summarizes the existing RCT’s.

Anyway, I think the journal club could improve by not following the entire checklist (boring! done!), but use this as a background. Furthermore I think there should be 3-5 questions that are very relevant to discuss. Like in the #HSCMEU discussions, people could pose those questions beforehand. In this way it is easier to adhere to the structure.

As to the medium Twitter for this journal club. I am not fond of  long Twitter chats, because it tends to be chaotic, there is a lot of reiteration, people tend to tweet not to “listen” and there is a constriction of 140 characters. Personally I would prefer a webinar, where people discuss the topic and you can pose questions via Twitter or otherwise.
Other alternatives wouldn’t work for me either. A Facebook journal club (described by of Neil Mehta) looks more static (commenting to a short summary of a paper), and Skyping is difficult with more than 10 people and not easy to transcribe.

But as said there is a lot of enthusiasm for this Twitter Journal Club. Even outside the medical world. This “convincing effort” inspired others to start a Astronomy Twitter Journal Club.

Perhaps a little modification of goals and structure could make it even more interesting. I will try to attend the next event, which is about Geoffrey Rose’s ‘Prevention Paradox’ paper, officially titled ”Strategy of prevention: lessons from cardiovascular disease”, available here.

Notes added:

[1] A summary of the first Twitter journal club is just posted. This is really valuable and takes away the disadvantages of reading an entire transcript (but one misses a lot of interesting aspects too)!

[2] This is the immediate response of one of the organizers at Twitter. I’m very pleased to notice that they will put more emphasis on implications of the Journal. That would take away much of my critic.

(Read tweets from bottom to top).

References

  1. Welcome (twitjc.wordpress.com)
  2. An important topic for the first Twitter Journal Club (twitjc.wordpress.com)
  3. Rivers E, Nguyen B, Havstad S, Ressler J, Muzzin A, Knoblich B, Peterson E, Tomlanovich M; Early Goal-Directed Therapy Collaborative Group. Early goal-directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001 Nov 8;345(19):1368-77. PubMed PMID: 11794169. (PDF).
  4. The First Journal Club on Twitter – Then and Now (casesblog.blogspot.com)
  5. Allergy and Immunologyclub on Twitter (allergynotes.blogspot.com)
  6. The Utility of a Real-time Microblogging Service for Journal Club in Allergy and Immunology. Dimov, V.; Randhawa, S.; Auron, M.; Casale, T. American College of Allergy, Asthma & Immunology (ACAAI) 2009 Annual Meeting. Ann Allergy Asthma Immunol., Vol 103:5, Suppl. 3, A126, Nov 2009.
  7. https://docs.google.com/document/pub?id=1qzk1WzjNO5fbWd0PAax6cIDdUGGg1sDn86FPT1li-sQ (short remix of the transcript)
  8. Model for a Journal Club using Google Reader and Facebook OR if the prophet does not go to the Mountain…. bring the journal club to FB! (blogedutech.blogspot.com)
  9. Astronomy Twitter Journal Club/ (sarahaskew.net)
  10. A summary of week one: Rivers et al (twitjc.wordpress.com)




#CECEM Bridging the Gap between Evidence Based Practice and Practice Based Evidence

15 06 2009

cochrane-symbol A very interesting presentation at the CECEM was given by the organizer of this continental Cochrane meeting, Rob de Bie. De Bie is Professor of Physiotherapy Research and director of Education of the Faculty of Health within the dept. of Epidemiology of the Maastricht University. He is both a certified physiotherapist and an epidemiologist. Luckily he kept the epidemiologic theory to a minimum. In fact he is a very engaging speaker who keeps your attention to the end.

Guidelines

While guidelines were already present in the Middle Ages in the form of formalized treatment of daily practice, more recently clinical guidelines have emerged. These are systematically developed statements which assists clinicians and patients in making decisions about appropriate treatement for specific conditions.

Currently, there are 3 kinds of guidelines, each with its own shortcomings.

  • Consensus based. Consensus may be largely influenced by group dynamics
    Consensus = non-sensus and Consensus guidelines are guidelies.
  • Expert based. Might be even worse than consensus. It can have all kind of biases, like expert and opinion bias or external financing.
  • Evidence based. Guideline recommendations are based on best available evidence, deals with specific interventions for specific populations and are based on a systematic approach.

The quality of Evidence Based Guidelines depends on whether the evidence is good enough, transparent, credible, available, applied and not ‘muddled’ by health care insurers.
It is good to realize that some trials are never done, for instance because of ethical considerations. It is also true that only part of what you read (in the conclusions) has actually be done and some trials are republished several times, each time with a better outcome…

Systematic reviews and qualitatively good trials that don’t give answers.

Next Rob showed us the results of a study ( Jadad and McQuay in J. Clin. Epidemiol. ,1996) with efficacy as stated in the review plotted on the X-axis and the Quality score on the Y-axis. Surprisingly meta-analysis of high quality were less likely to produce positive results. Similar results were also obtained by Suttorp et al in 2006. (see Figure below)

12066264  rob de bie CECEM

Photo made by Chris Mavergames

There may be several reasons why good trials not always give good answers. Well known reasons are  the lack of randomization or blinding. However Rob focused on a less obvious reason. Despite its high level of evidence, a Randomized Controlled Trial (RCT) may not always be suitable to provide good answers applicable to all patients, because RCT’s often fail to reflect the true clinical practice. Often, the inclusion of patients in RCT’s is selective: middle-aged men with exclusion of co-morbidity. Whereas co-morbidity occurs in > 20% of the people of 60 years and older and in >40% of the people of 80 years and older (André Knottnerus in his speech).

Usefulness of a Nested Trial Cohort Study coupled to an EHR to study interventions.

Next, Rob showed that a nested Trial cohort study can be useful to study the effectiveness of  interventions. He used this in conjunction with an EHR (electronic health record), which could be accessed by practitioner and patient.

One of the diseases studied in this way, was Intermittent Claudication. Most commonly Intermittent Claudication is a manifestation of  peripheral arterial disease in the legs, causing pain and cramps in the legs while walking (hence the name). The mortality is high: the 5 year mortality rates are in between those of colorectal cancer and Non-Hodgkin Lymphoma. This is related to the underlying atherosclerosis.

There are several risk factors, some of which cannot be modified, like hereditary factors, age and gender. Other factors, like smoking, diet, physical inactivity and obesity can be tackled. These factors are interrelated.

Rob showed that, whereas there may be an overall null effect of exercise in the whole population, the effect may differ per subgroup.

15-6-2009 3-06-19 CI 1

  • Patients with mild disease and no co-morbidity may directly benefit from exercise-therapy (blue area).
  • Exercise has no effect on smokers, probably because smoking is the main causative factor.
  • People with unstable diabetes first show an improvement, which stabilized after a few weeks due to hypo- or hyperglycaemia induced by the exercise,
  • A similar effect is seen in COPD patients, the exercise becoming less effective because the patients become short of breath.

It is important to first regulate diabetes or COPD before continuing the exercise therapy. By individually optimizing the intervention(s) a far greater overall effect is achieved: 191% improval in the maximal (pain-free) walking distance compared to for instance <35% according to a Cochrane Systematic Review (2007).

Another striking effect: exercise therapy affects some of the prognostic factors: whereas there is no effect on BMI (this stays an important risk factor), age and diabetes become less important risk factors.

15-6-2009 3-35-10 shift in prognostic factors

Because guidelines are quickly outdated, the findings are directly implemented in the existing guidelines.

Another astonishing fact: the physiotherapists pay for the system, not the patient nor the government.

More information can be found on https://www.cebp.nl/. Although the presentation is not (yet?) available on the net, I found a comparable presentation here.

** (2009-06-15) Good news: the program and all presentations can now be viewed at: https://www.cebp.nl/?NODE=239





How to make EBM easy to swallow: BMJ PICO

8 02 2009

Guest author: Shamsha Damani (@shamsha)

As a medical librarian, I try to instill the importance of Evidence Based Medicine (EBM) to all my users. They agree that EBM is important, and yet, still resort to shortcuts (like using Google, asking colleagues, etc). And you know what, I don’t blame them. Given the amount of medical literature published today, it is very difficult to keep up with it all. There are some very bad and poorly designed studies published, which makes it difficult to identify good ones. And once you’ve identified a good article to read, evaluating and critiquing it is another daunting task. I keep wondering if this has to be so difficult. Shouldn’t there be stricter standards for publications? Shouldn’t publishers care about the quality of research that is associated with their name? I know that some journals like ACP Journal Club critique articles but they don’t cover nearly enough topics.

As I pondered these thoughts, something very interesting happened that gives me hope. BMJ recently announced that they will be publishing two summaries for each research article published. One is called BMJ PICO, is prepared by the authors, and breaks down the article into the different EBM elements. The other is called Short Cuts, which is written by BMJ itself. This is where I hope BMJ will shine, provide an unbiased view of the article, and set itself apart from other journals by doing some extra work. Imagine reading a brief synopsis of a research article, not written by the author, which will tell you whether the study was any good and if the results were valid. What a time saver! I hope that BMJ continues this practice and that other journals follow suit. Right now BMJ is still testing the waters and trying to figure out which format would be most appealing to readers. Personally I think it would have been better to have the BMJ reviewers write the PICO format, and do a bit more thorough critiquing. The reviewers already critique the article before it gets accepted; it only makes sense that the results of such a thorough critique be published as well. An unbiased view would make it easier for readers to trust (or not!) the results and proceed accordingly.

I still believe that EBM skills are very important and should be learned.
However, busy health care providers will find value in such pre-packaged articles and will use the evidence more if it has been critiqued already. And isn’t that the point of EBM: to make more use of the evidence?

Shamsha Damani, Clinical Librarian





The Web 2.0-EBM Medicine split. [1] Introduction into a short series.

4 01 2009

Since the three years I’m working as a medical information specialist, I’ve embraced the concept of evidence based medicine or EBM. As a searcher I spend hours if not days to find as much relevant evidence as possible on a particular subject, which others select, appraise and synthesize to a systematic review or an evidence based guideline. I’m convinced that it is important to find the best evidence for any given intervention, diagnosis, prognostic or causal factor.

Why? Because history has shown that despite their expertise and best intentions, doctors don’t always know or feel what’s best for their patients.

An example. For many years corticosteroids had been used to lower intracranial pressure after serious head injury, because steroids reduce the inflammation that causes the brain to swell. However, in the 1990’s, meta-analyses and evidence-based guidelines called the effectiveness of steroids into question. Because of the lack of sufficiently large trials, a large RCT (CRASH) was started. Contrary to all expectations, there was actually an excess of 159 deaths in the steroid group. The overall absolute risk of death in the corticosteroid group was shown to be increased with 2%. This means that the administration of corticosteroids had caused more than 10,000 deaths before the 1990’s.[1,2,3]

Another example. The first Cochrane Systematic Review, shows the results of a systematic review of RCTs of a short, inexpensive course of a corticosteroid given to women about to give birth too early. The diagram below, which is nowadays well known as the logo of the Cochrane Collaboration, clearly shows that antenatal corticosteroids reduce the odds of the babies dying from the complications of immaturity by 30 to 50 per cent (diamond left under). Strikingly, the first of these RCTs showing a positive effect of corticosteroids, was already reported in 1972. By 1991, seven more trials had been reported, and the picture had become still stronger. Because no systematic review of these trials had been published until 1989, most obstetricians had not realized that the treatment was so effective. As a result, 10.000s of premature babies have probably suffered and died unnecessarily. This is just one of many examples of the human costs resulting from failure to perform systematic, up-to-date reviews of RCTs of health care.[4,5]

The Cochrane logo explained

Less than I year ago I entered the web 2.0-, and (indirectly) medicine 2.0 world, via a library 2.0 course. I loved the tools and I appreciated the approach. Web 2.0 is ‘all about sharing‘ or as Dean Giustini says it: ‘all about people. It is very fast and simple. It is easy to keep abreast of new information and to meet new interesting people with good ideas and a lot of knowledge.

An example. Bertalan Mesko in a comment on his blog ScienceRoll:

I know exactly that most of these web 2.0 tools have been around for quite a long time. Most of these things are not new and regarding the software, there aren’t any differences in most of the cases. But!
These tools and services will help us how to change medicine. In my opinion, the most essential problem of medicine nowadays is the sharing of information. Some months ago, I wrote about a blogger who fights Pompe disease, a rare genetic disorder and he told me about the diagnostic delay. I try to help physicians how they can find information easier and faster. For example: I gave tips how to search for genetic diseases.

Other examples are good functioning and dedicated patient web 2.0 sites, like PatientsLikeMe.

In the medical literature, blogs and slideshare, differences between medicine 2.0 and 1.0 are already described in detail (for instance see the excellent review of Dean Giustini in the BMJ), as well as the differences between medicine 1.0 and EBM (e.g. see the review of David Sackett et al in BMJ).

However, the longer I’m involved in web 2.0, the more I feel it conflicts with my job as EBM-librarian. The approach is so much different, other tools are used and other views shared. More and more I find ideas and opinions expressed on blogs that do EBM no justice and that seem to arise out of ignorance and/or prejudice. On the other hand EBM and traditional medicine often are not aware of web 2.0 sources or mistrust them. In science, blogs and wiki’s seldom count, because they express personal views, echo pre-existing data and are superficial.

split-1231

I’m feeling like I’m in a split, with one leg in EBM and the other in web 2.0. In my view each has got his merits, and these approaches should not oppose each other but should mingle. EBM getting a lower threshold and becoming more digestible and practical, and medicine 2.0 becoming less superficial and more underpinned.

It is my goal to take an upright position, standing on both legs, integrating EBM, medicine 2.0 (as well as medicine 1.0).

As a first step I will discuss some discrepancies between the two views as I encounter it in blogs, in the form of a mini-series: “The Web 2.0-EBM Medicine split”.

Before I do so I will give a short list of what I consider characteristic for each type of medicine, EBM-, Web 1.0 (usual)- and Web 2.0- medicine. Not based on any evidence, only on experience and intuition. I’ve just written down what came to my mind. I would be very interested in your thoughts on this.

EBM – medicine

  • centered round the best evidence
  • methodology-dependent
  • objective, transparent
  • thorough
  • difficult (to make, but for many also to find and also to understand)
  • time-consuming
  • published in peer reviewed papers (except for guidelines)
  • searching: PubMed and other bibliographic databases (to produce) and guideline databases, TRIP, and PubMed (Clinical Queries) or specific sources, i.e. specialist guidelines (to find).
  • Mostly Web 1.0 (with some web 2.0 tools, like podcasts, RSS and e-learning)

Web 1.0 – traditional medicine*

  • centered round clinical knowledge, expertise and intuition
  • opinion-based
  • authority based, i.e.strong beliefs in opinion leaders, expert opinion or ‘authority opinion’ (i.e. head of departments, professor) and own authority versus patient.
  • subjective
  • fast
  • act! (motto)
  • searching: browsing ( a specific list, site or Journals), quick search, mostly via Google**, in pharmacopeia, or protocols and UpToDate seldom in Pubmed (dependent on discipline)
  • Web 1.0: mail, patient-records, quick search via Google and Pubmed

Web 2.0 medicine

  • people-centered and patient-centered (although mostly not in individual blogs of doctors)
  • heavily based on technology (easy to use and free internet software)
  • social-based: based on sharing knowledge and expertise
  • (in theory) personalized
  • subjective, nondirected.
  • often:superficial
  • fast
  • generally not peer reviewed, i.e. published on blogs and wiki’s
  • searching: mostly via free internet sources and search engines, e.g. wikipedia, emedicine, respectively Google**, health metasearch engines, like Mednar and Health Sciences Online. PubMed mainly via third-party-tools like GoPubMed, HubMed and PubReminer. (e.g. see recent listings of top bedside health search engines on Sandnsurf’s blog ‘Life in the Fast Lane’
  • heavily dependent on web 2.0 tools both for ‘publishing’, ‘finding information’ and ‘communication’

*very general. of course dependent on discipline.
** this is not merely my impression, e.g. see: this blogpost on the “Clinical Cases and Images blog” of Ves Dimov, referring to four separate interviews of Dean Giustini with Physician bloggers.

Other references

[1] Final results of MRC CRASH, a randomised placebo-controlled trial of intravenous corticosteroid in adults with head injury-outcomes at 6 months. Edwards P et al. Lancet. 2005 Jun 4-10;365(9475):1957-9.
[2] A CRASH landing in severe head injury. Sauerland S, Maegele M. Lancet. 2004 Oct 9-15;364(9442):1291-2. Comment on: Lancet. 2004 Oct 9-15;364(9442):1321-8.
[3] Corticosteroids for acute traumatic brain injury.Alderson P, Roberts IG. Cochrane Database of Systematic Reviews 2005, Issue 1. Art. No.: CD000196.
[4] http://www.cochrane.org/logo/
[5] Antenatal corticosteroids for accelerating fetal lung maturation for women at risk of preterm birth.Roberts D, Dalziel SR.Cochrane Database of Systematic Reviews 2006, Issue 3. Art. No.: CD004454
[6] How Web 2.0 is changing medicine. Giustini D. BMJ. 2006 Dec 23;333(7582):1283-4.
[7] Evidence based medicine: what it is and what it isn’t. Sackett DL et al. BMJ. 1996 Jan 13;312(7023):71-2.









Follow

Get every new post delivered to your Inbox.

Join 610 other followers