How Evidence Based is UpToDate really?

KevinMD or Kevin Pho is one of the top physician bloggers. He writes many posts per day, often provocatively commenting on breaking medical news or other blogposts.

A few weeks ago Kevin wrote a post on comparative effectiveness research [5] (tweet below), which is “(funded) research to evaluate and compare clinical outcomes, effectiveness, risk, and benefits of two or more medical treatments or services that address a particular medical condition” (definition from DB Medical Rants).

Kevin stated that doctors certainly need an authoritative, unbiased, source to base their decisions on and that that kind of information is already there in the form of UpToDate®. According to Kevin:

For those who don’t know, UpToDate is a peer-reviewed, evidence-based, medical encyclopedia [1] available via DVD or online that’s revised every 3 months. It does not carry advertisements, and is funded entirely via paid subscriptions [3]. I am a big proponent, and like many other doctors, could not practice medicine effectively without it by my side.” [2]

Kevin Pho also refers to a recent study showing that hospitals who used UpToDate scored better on patient safety and complication measures, as well as length of stay, when compared to institutions who did not use the resource.[4]

Kevin’s post actually summarized a post of yet another well known blogger, Val Jones MD (dr Val) of Better Health. In her blog post Dr Val wonders whether we should incentivize hospitals and providers to use UpToDate more regularly. Incentives can range from pay for performance bonuses to malpractice immunity for physicians who adhere to UpToDate’s, evidence-based, unbiased, clinical recommendations. According to Dr. Val, this might be an effective and easy way to target the problem of inconsistent practice styles on a national level, since many physicians know and respect UpToDate.[5]

The tweet of KevinMD elicited many responses on Twitter. To read most of the discussion on twitter follow this link.

Below I shall discuss the points addressed in the blogpost of KevinMD and DrVal. When relevant I will show/discuss the tweets as well.***

[1] Is UpToDate Evidence based?
The main discussion point on twitter was to which extent UpToDate is evidence based. As you can see below (the oldest tweet is at the bottom, the newest at the top) the opinions differ as to the level of UpToDate’s “evidence-basedness”. It varies from the one extreme of UpToDate doing systematic reviews and being entirely evidence based (drval) to ‘a slant of EBM*’ (@kevinmd) and UpToDate being an online book with narrative reviews.

UpToDate used to be entirely an online book with (excellent) narrative reviews written by experts in the field. From 2006 onwards UpToDate began grading recommendations for treatment and screening using a modification of the GRADE system. Nowadays UpToDate calls its database an evidence based, peer reviewed information resource. According UpToDate the evidence is compiled from:

• Hand-searching of over 400 peer-reviewed journals
• Electronic searching of databases including MEDLINE, The Cochrane Database, Clinical Evidence, and ACP Journal Club
• Guidelines that adhere to principles of evidence evaluation
• Published information regarding clinical trials such as reports from the FDA and NIH
• Proceedings of major national meetings
• The clinical experience and observations of our authors, editors, and peer reviewers

Although it is an impressive list of EBM-sources, this does not mean that UpToDate itself is evidence based. A selection of journals to be ‘handsearched’ will undoubtedly lead to positive publication bias (most positive results will reach the major journals). The electronic searches -if done- are not displayed and therefore the quality of any search performed cannot be checked. It is also unclear on which basis articles are in- or excluded. And although UpToDate may summarize evidence from Systematic Reviews, including Cochrane Systematic Reviews it does not perform Systematic Reviews itself. At the most it gives a synthesis of the evidence, which is (still) gathered in a rather nontransparent way. Thus the definition of @kevinmd comes closest: “it gives an evidence based slant”. After all, Evidence-based medicine is a set of procedures, pre-appraised resources and information tools to assist practitioners to apply evidence from research in the care of individual patients” (McKibbon, K.A., see defintions at  the scharr webpage). Merely summarizing and /or referring to evidence is not enough to be evidence based.
It is also not clear what peer reviewed implies, i.e.can articles (chapters) be rejected by peer reviewers?

As a consequence the chapters differ in quality. Regularly I don’t find the available evidence in UpToDate. That is also true for students and docs preparing a Critically Appraised Topic (CAT). In my experience, UpToDate is hardly ever useful for finding recent evidence on a not too common question. @Allergynotes tweeted a specific example on chronic urticaria and H. pylori, where the available evidence could not be found in UpToDate.
In an older post (2007)*** @Allergynotes (Ves Dimov) commented on an interesting post by Dr. RW: “Are you UpToDate dependent?” by citing an old proverb: “beware the man of a single book (homo unius libri), which describes people with limited knowledge. The current version of the Internet has billions of scientific journal pages and the answer to your questions must be somewhere out there.” Ves:

“I don’t think anybody should be dependent on a single source. If one cannot practice medicine without UpToDate, may be one should not practice at all.”

Likewise, an anonymous commenter on Kevin’ posts stated:

“Don’t overlook the fact that there is a lot of good research outside of UpToDate. This is a great source, but if it’s your only source you’re closing off a tremendous amount of the literature. The articles are also written by people, and are subject to the biases of individuals.”

In another comment Dr. Matthew Mintz of the excellent blog with the same name puts forward that many of the authors have substantial ties to the pharmaceutical industry, meaning that UptoDate (although not financed) is not completely unbiased.

[2] Usefulness
@Allergynotes rightly states that usability/perceived usefulness my be more important to physicians (than real usefulness) and that we should look at what make UpToDate so useful rather than just say “it’s not EBM”. In one of his posts Ves Dimov (@allergynotes) refers to a (Dutch) paper showing that answers to questions posed during daily patient care are more likely to be answered by UpToDate than PubMed.** At my hospital some doctors (especially intern med docs) consider UpToDate as their Bible. It is without doubt that UpToDate is a very useful source both for clinicians, patients and even librarians. It is ideal for background questions (How can disease X be treated, what is the differential diagnosis?, what might be the cause of this disease?), to look up things and as a starting point. And it has a broad coverage. However the point here was not whether UpToDate is a useful source for clinicians – but whether it is a sufficiently unbiased evidence based source to incentive docs to follow its recommendations and its recommendations alone. Or as Shamsha says it: “I don’t like putting all my eggs in UpToDate’s basket.

[3] Disadvantages/Alternatives
As highlighted by the twitter discussions (read from down up), the major disadvantages of UpToDate are its high pricing, its ridigity, monopolistic tendencies and strict denial of remote access. I don’t know if you have seen the recent post of David Rothman on a very unpolite, aggressive vendor trying to push a trial. Most of David readers guess the vendor was from UpToDate (2nd: MD consult). Is it reasonable to positively discriminate in favor of UpToDate, while not everyone may be able to afford this costly database or may prefer another source? Incentives will only enhance UpToDate’s monopolistic position.
The most ideal situation would be an open source UTD, as suggested by @nursedan. Allergynotes thinks that this should be possible. A role for Web 2.0 in EBM?

It should be noted that (besides the databases mentioned in the tweets) there are also other freely available evidence based sources, like

• National Guideline Clearinghouse (NGC), a public resource for evidence-based clinical practice guidelines
• National Library of Guidelines, a collection of guidelines for the NHS.  It is based on the guidelines produced by NICE and other national agencies. NICE guidelines of very high quality, based on a systematic review of the evidence and extensive consultation with clinicians and patients and, where relevant, industry. Dr. Matthew Mintz also stresses the usefulness of NICE-guidelines in his comment to Kevin’s post
• TRIP database, that grazes various evidence based resources, including the ones above.

[4] Hospitals using UptoDate score better?
Kevin and Dr Val also refer to a study in International Journal of Medical Informatics showing hospitals that used UptoDate scored better than hospitals that didn’t (even in a dose-response way). This study is shown prominently at the UpToDate’s site.

Now let’s just “score” the Evidence.

First one can wonder how representative this article is. A quick and dirty Google search gives many hits on the very same subject not (directly) linking to UpToDate. For instance, a paper published in the January issue of Ann Intern Med tells us the results from a large-scale study of more than 40 hospitals and 160,000 patients showing that when health information technologies replace paper forms and handwritten notes, both hospitals and patients benefit strongly (fewer complications, lower mortality rates, and lower costs). Etcetera. One would like to know how the evidence in the “UpToDate paper” would relate to other studies or even better one would like to see a head to head comparison of UpToDate with any other (specific) evidence based source.

The Impact Factor of INT J MED INFORM is 1.579. This says nothing about its value, but such a paper wouldn’t likely appear in UpToDate’s handsearch list.

More important 2 of the 4 authors are from UpToDate. This is an important bias.

Furthermore the study is a retrospective and observational study, comparing hospitals with online access to UpToDate with other acute care hospitals. According to the GRADE system this would automatically yield a Grade C: low-quality evidence from observational evidence. Most important, as admitted by the authors, the study could not fully account for additional features at the included hospitals that may also have been associated with better health outcomes. It is easy to imagine, for instance that a hospital being able to subscribe to UpToDate has a medical staff that was already predisposed to delivering higher quality care or might have a greater budget ;). And although the average, severity-adjusted lenght of stay was significantly shorter in UpToDate® hospitals than in other hospitals with a P value of less than 0.0001, the mean difference was only 0.167 days with a not very impressive 95% confidence interval of 0.081–0.252 days.

[5] Incentives?
Based on the above arguments I don’t think it would be reasonable, effective or fair to incentive those hospitals or doctors that consult (and can afford) UpToDate or to indirectly punish those that don’t (because they don’t have the money or they have a good alternative).

Furthermore such a positive discrimination would not solve the problem of lack of head to head comparison, what was what it was all about. Dr Mintz explains this very clearly in his comment to Kevin.

“… the authors of UptoDate are providing their own summary of already published data, which most is funded by industry. This is similarly true of other so-called unbiased sources.(..)
The problem goes even deeper than the potential bias of industry funded research, which has been consistently shown to be favorable to the sponsor. The fact that most research, and virtually all therapeutic research is funded by the industry allows the industry to dictate what scientific knowledge is available, and by default clinical practice.(…)
There are hundreds of important studies that are never done because the industry only takes a “safe” bet.
We need comparative effectiveness not just to see whether the more expensive treatment is worth the cost, but we also need it to answer scientifically important questions that the industry will unlikely fund.”

*EBM = evidence based medicine

** It should be noted though that an other interface of PubMed is used in this hospital, to allow recording of the queries.The study participants were doctors in internal medicine.

*** I’ve added this sentence because people thought I merely summarized the tweets. In addition I added some new references.

References:

• Kevin MD: (2009/03): Do doctors already have a source of comparative effectiveness research?
• Dr. Val Jones from Better Health (2009.02.25) Attention Health Policy Makers: How To Win Docs And Influence Patients.
• R. centor from DB’s Medical Rant: Comparative Effectiveness research (2009/02/23)
  
• UpToDate.com (with info about the database)
• Bonis PA, Pickens GT, Rind DM, Foster DA Association of a clinical knowledge support system with improved patient safety, reduced complications and
  shorter length of stay among Medicare beneficiaries in acute care hospitals in the United States. Int J Med Inform. 2008 Nov;77(11):745-53. (abstract)
• Tweets about UpToDate by several people (shamsha, kevinmd, drval, allergynotes, digicmb, laikas) note that result is not static, several tweets added since this post.
  
• Amarasingham R, Plantinga L, Diener-West M, Gaskin DJ, Powe NR. Clinical information technologies and inpatient outcomes: a multiple hospital study. Arch Intern Med. 2009 Jan 26; 169(2): 108-14
  
• Clinical Cases and Images – Blog. Study: UpToDate More Likely than PubMed to Answer Patient Care Questions (2008/10/)
• Arjen Hoogendam, Anton FH Stalenhoef, Pieter F de Vries Robbé, A John PM Overbeke, D Answers to Questions Posed During Daily Patient Care Are More Likely to Be Answered by UpToDate Than PubMed. J Med Internet Res 2008;10(4):e29) – doi:10.2196/jmir.1012
• Clinical Cases and Images – Blog. Are You Dependent on UpToDate for Your Clinical Practice? (2007/07)
• Notes from Dr. W. Are you UpToDate dependent? (2007/07)
• David Rothman on Davidrothman.net. Venting about a lousy vendor sales rep (2009/03/24)

You may also want to read:

• Walter Jessen on Highlighthealth. US encourages evidence based medicine in economic stimulus (2009/02/17) (2009
• Guest Post of Shamsha Damani: Evidence based medicine the facebook of medicine. (2009/04/04)
• The web-2.0 EBM medicine split (1) introduction into a short series (2009/01/04/) The above post could have been the 2nd in this series,