How Evidence Based is UpToDate really?

5 04 2009

logo-uptodate-2KevinMD or Kevin Pho is one of the top physician bloggers. He writes many posts per day, often provocatively commenting on breaking medical news or other blogposts.

A few weeks ago Kevin wrote a post on comparative effectiveness research [5] (tweet below), which is “(funded) research to evaluate and compare clinical outcomes, effectiveness, risk, and benefits of two or more medical treatments or services that address a particular medical condition” (definition from DB Medical Rants).

utd1-uptodate-kevin

Kevin stated that doctors certainly need an authoritative, unbiased, source to base their decisions on and that that kind of information is already there in the form of UpToDate®. According to Kevin:

For those who don’t know, UpToDate is a peer-reviewed, evidence-based, medical encyclopedia [1] available via DVD or online that’s revised every 3 months. It does not carry advertisements, and is funded entirely via paid subscriptions [3]. I am a big proponent, and like many other doctors, could not practice medicine effectively without it by my side.[2]

Kevin Pho also refers to a recent study showing that hospitals who used UpToDate scored better on patient safety and complication measures, as well as length of stay, when compared to institutions who did not use the resource.[4]

Kevin’s post actually summarized a post of yet another well known blogger, Val Jones MD (dr Val) of Better Health. In her blog post Dr Val wonders whether we should incentivize hospitals and providers to use UpToDate more regularly. Incentives can range from pay for performance bonuses to malpractice immunity for physicians who adhere to UpToDate’s, evidence-based, unbiased, clinical recommendations. According to Dr. Val, this might be an effective and easy way to target the problem of inconsistent practice styles on a national level, since many physicians know and respect UpToDate.[5]

The tweet of KevinMD elicited many responses on Twitter. To read most of the discussion on twitter follow this link.

Below I shall discuss the points addressed in the blogpost of KevinMD and DrVal. When relevant I will show/discuss the tweets as well.***

[1] Is UpToDate Evidence based?
The main discussion point on twitter was to which extent UpToDate is evidence based. As you can see below (the oldest tweet is at the bottom, the newest at the top) the opinions differ as to the level of UpToDate’s “evidence-basedness”. It varies from the one extreme of UpToDate doing systematic reviews and being entirely evidence based (drval) to ‘a slant of EBM*’ (@kevinmd) and UpToDate being an online book with narrative reviews.

utd2-tot2-uptodate

UpToDate used to be entirely an online book with (excellent) narrative reviews written by experts in the field. From 2006 onwards UpToDate began grading recommendations for treatment and screening using a modification of the GRADE system. Nowadays UpToDate calls its database an evidence based, peer reviewed information resource. According UpToDate the evidence is compiled from:

  • Hand-searching of over 400 peer-reviewed journals
  • Electronic searching of databases including MEDLINE, The Cochrane Database, Clinical Evidence, and ACP Journal Club
  • Guidelines that adhere to principles of evidence evaluation
  • Published information regarding clinical trials such as reports from the FDA and NIH
  • Proceedings of major national meetings
  • The clinical experience and observations of our authors, editors, and peer reviewers

Although it is an impressive list of EBM-sources, this does not mean that UpToDate itself is evidence based. A selection of journals to be ‘handsearched’ will undoubtedly lead to positive publication bias (most positive results will reach the major journals). The electronic searches -if done- are not displayed and therefore the quality of any search performed cannot be checked. It is also unclear on which basis articles are in- or excluded. And although UpToDate may summarize evidence from Systematic Reviews, including Cochrane Systematic Reviews it does not perform Systematic Reviews itself. At the most it gives a synthesis of the evidence, which is (still) gathered in a rather nontransparent way. Thus the definition of @kevinmd comes closest: “it gives an evidence based slant”. After all, Evidence-based medicine is a set of procedures, pre-appraised resources and information tools to assist practitioners to apply evidence from research in the care of individual patients” (McKibbon, K.A., see defintions at  the scharr webpage). Merely summarizing and /or referring to evidence is not enough to be evidence based.
It is also not clear what peer reviewed implies, i.e.can articles (chapters) be rejected by peer reviewers?

As a consequence the chapters differ in quality. Regularly I don’t find the available evidence in UpToDate. That is also true for students and docs preparing a Critically Appraised Topic (CAT). In my experience, UpToDate is hardly ever useful for finding recent evidence on a not too common question. @Allergynotes tweeted a specific example on chronic urticaria and H. pylori, where the available evidence could not be found in UpToDate.
In an older post (2007)*** @Allergynotes (Ves Dimov) commented on an interesting post by Dr. RW: “Are you UpToDate dependent?” by citing an old proverb: “beware the man of a single book (homo unius libri), which describes people with limited knowledge. The current version of the Internet has billions of scientific journal pages and the answer to your questions must be somewhere out there.” Ves:

“I don’t think anybody should be dependent on a single source. If one cannot practice medicine without UpToDate, may be one should not practice at all.”

Likewise, an anonymous commenter on Kevin’ posts stated:

“Don’t overlook the fact that there is a lot of good research outside of UpToDate. This is a great source, but if it’s your only source you’re closing off a tremendous amount of the literature. The articles are also written by people, and are subject to the biases of individuals.”

In another comment Dr. Matthew Mintz of the excellent blog with the same name puts forward that many of the authors have substantial ties to the pharmaceutical industry, meaning that UptoDate (although not financed) is not completely unbiased.

utd1-uptodate-allergynotes-laikas-evidence-not-always-found-3[2] Usefulness
@Allergynotes rightly states that usability/perceived usefulness my be more important to physicians (than real usefulness) and that we should look at what make UpToDate so useful rather than just say “it’s not EBM”. In one of his posts Ves Dimov (@allergynotes) refers to a (Dutch) paper showing that answers to questions posed during daily patient care are more likely to be answered by UpToDate than PubMed.** At my hospital some doctors (especially intern med docs) consider UpToDate as their Bible. It is without doubt that UpToDate is a very useful source both for clinicians, patients and even librarians. It is ideal for background questions (How can disease X be treated, what is the differential diagnosis?, what might be the cause of this disease?), to look up things and as a starting point. And it has a broad coverage. However the point here was not whether UpToDate is a useful source for clinicians – but whether it is a sufficiently unbiased evidence based source to incentive docs to follow its recommendations and its recommendations alone. Or as Shamsha says it: “I don’t like putting all my eggs in UpToDate’s basket.

utd-ebm-eggs-shamsha[3] Disadvantages/Alternatives
As highlighted by the twitter discussions (read from down up), the major disadvantages of UpToDate are its high pricing, its ridigity, monopolistic tendencies and strict denial of remote access. I don’t know if you have seen the recent post of David Rothman on a very unpolite, aggressive vendor trying to push a trial. Most of David readers guess the vendor was from UpToDate (2nd: MD consult). Is it reasonable to positively discriminate in favor of UpToDate, while not everyone may be able to afford this costly database or may prefer another source? Incentives will only enhance UpToDate’s monopolistic position.
The most ideal situation would be an open source UTD, as suggested by @nursedan. Allergynotes thinks that this should be possible. A role for Web 2.0 in EBM?

It should be noted that (besides the databases mentioned in the tweets) there are also other freely available evidence based sources, like

utd1-uptodate-price-disadv

[4] Hospitals using UptoDate score better?
Kevin and Dr Val also refer to a study in International Journal of Medical Informatics showing hospitals that used UptoDate scored better than hospitals that didn’t (even in a dose-response way). This study is shown prominently at the UpToDate’s site.

Now let’s just “score” the Evidence.

First one can wonder how representative this article is. A quick and dirty Google search gives many hits on the very same subject not (directly) linking to UpToDate. For instance, a paper published in the January issue of Ann Intern Med tells us the results from a large-scale study of more than 40 hospitals and 160,000 patients showing that when health information technologies replace paper forms and handwritten notes, both hospitals and patients benefit strongly (fewer complications, lower mortality rates, and lower costs). Etcetera. One would like to know how the evidence in the “UpToDate paper” would relate to other studies or even better one would like to see a head to head comparison of UpToDate with any other (specific) evidence based source.

The Impact Factor of INT J MED INFORM is 1.579. This says nothing about its value, but such a paper wouldn’t likely appear in UpToDate’s handsearch list.

More important 2 of the 4 authors are from UpToDate. This is an important bias.

Furthermore the study is a retrospective and observational study, comparing hospitals with online access to UpToDate with other acute care hospitals. According to the GRADE system this would automatically yield a Grade C: low-quality evidence from observational evidence. Most important, as admitted by the authors, the study could not fully account for additional features at the included hospitals that may also have been associated with better health outcomes. It is easy to imagine, for instance that a hospital being able to subscribe to UpToDate has a medical staff that was already predisposed to delivering higher quality care or might have a greater budget ;). And although the average, severity-adjusted lenght of stay was significantly shorter in UpToDate® hospitals than in other hospitals with a P value of less than 0.0001, the mean difference was only 0.167 days with a not very impressive 95% confidence interval of 0.081–0.252 days.

[5] Incentives?
Based on the above arguments I don’t think it would be reasonable, effective or fair to incentive those hospitals or doctors that consult (and can afford) UpToDate or to indirectly punish those that don’t (because they don’t have the money or they have a good alternative).

Furthermore such a positive discrimination would not solve the problem of lack of head to head comparison, what was what it was all about. Dr Mintz explains this very clearly in his comment to Kevin.

“… the authors of UptoDate are providing their own summary of already published data, which most is funded by industry. This is similarly true of other so-called unbiased sources.(..)
The problem goes even deeper than the potential bias of industry funded research, which has been consistently shown to be favorable to the sponsor. The fact that most research, and virtually all therapeutic research is funded by the industry allows the industry to dictate what scientific knowledge is available, and by default clinical practice.(…)

There are hundreds of important studies that are never done because the industry only takes a “safe” bet.
We need comparative effectiveness not just to see whether the more expensive treatment is worth the cost, but we also need it to answer scientifically important questions that the industry will unlikely fund.”

*EBM = evidence based medicine
** It should be noted though that an other interface of PubMed is used in this hospital, to allow recording of the queries.The study participants were doctors in internal medicine.
*** I’ve added this sentence because people thought I merely summarized the tweets. In addition I added some new references.

References:

You may also want to read:

About these ads

Actions

Information

26 responses

5 04 2009
Robin

What a wonderful post on a topic that seems so touchy to so many in the medical community. I know there are several topics close to my heart that are not “up-to-date” on UpToDate when compared to current research.

I somehow missed a lot of this conversation on Twitter due to work overload, but am so glad to see that it took place. Excellent use of twitter, of blogs, and of sharing of resources, thoughts, and ideas. Thank you so much for pulling it all together and for adding to it.

I haven’t digested all this, yet, but wanted to take time to tell you how much I appreciate it.

5 04 2009
Clinical Cases and Images - Blog

You didn’t like my “beware the man of a single book” blog post for inclusion in “relevant blog posts”?
:)

http://casesblog.blogspot.com/2007/07/are-you-uptodate-dependent-for-your.html

5 04 2009
precordialthump

Up To Date is really just a textbook in my view – one that is “up to date” – providing predominantly background rather than foreground knowledge.

Still have to search Cochrane, clinical evidence.org, guidelines, or get “down and dirty” with Pubmed (etc) to get the real evidence.

Great post!

5 04 2009
Weekly News Round-Up « Women’s Health News

[...] From libraryland: Laika is poking the UpToDate beehive with a stick. [...]

5 04 2009
Luca DF

An extremely interesting debate. Your notes are really elegant and to the point: the very question is “Could/Should doctors base their clinical decision on secondary publications?” The primary medical literature is suffering an overall crisis of credibility (see the Jefferson’s article on vaccines’ papers in BMJ – Feb 2009): but (sadly) the crisis involves also the e-b pre-appraised tools (such as UpToDate or POEMS, etc,)…

6 04 2009
smnewsletters

Please see <a href=www.searchmedica.com, an authoritative (physician-vetted), free clinical search engine containing a separate article category “Evidence-based Medicine” limited to randomized trials, meta-analyses, systematic reviews. SearchMedica’s database of journal sources includes PubMed pre-filtered to exclude molecular and animal studies, as well as direct searches of hundreds of major medical journals. It is also the only clinical search engine to include secondary and tertiary publications.

This is the resource for those who can’t afford UpToDate etc.

6 04 2009
Rose

Evidence based or not, it’s incredibly user friendly. Therefore all the hand wringing in the world is not going to get people to stop using it in favor of less biased more up-to-date sources that have poor interfaces rendering them useless despite the content. Better to focus on getting good content coupled with the winning interface that people will actually *USE*. Easier to build a better interface than to try to change individual’s habits, they will always want to use the source that is the easiest.

6 04 2009
Lori Graham

What I do not understand is why is UpToDate SO MUCH MORE expensive than other databases of it kind? I am talking for comparable services like system-wide and remote access, it was SIX THOUSAND MORE than their competition for our small institution two years ago. Plus the competition is able to get the updates sooner and theirs is equally evidence-based. That makes a difference in this time.

7 04 2009
Hvor evidensbasert er UpToDate? « Bibliotektjenesten SI Blog

[...] Se hele diskusjonen gjengitt her [...]

7 04 2009
danil

i am not that convinced that uptodate is evidence based as often you cannot find any info as to why they are making that recommendation and they dont refernce the evidnece or rct’s or literature behind. plus I just dont have the time to spend hours browsing through pages and pages of information when I just need a quick answer to a clinical question that I am facing on the ward. its just like a big text book which might be ok for learning but for use at the point of care, ? i dont think so. i just got some info from the guys at the bmj about their new US product, Point of Care: http://group.bmj.com/products/evidence-centre/bmj-point-of-care-1

I am going to test this out for the US, but I noticed they have a product for outside of the US which I guess is more international and certinaly totally evidence based: http://bestpractice.bmj.com, some of this product aparantrly contains clincial evidence which is THE evidence base, so I am interested to see how the new offering works. thanks

7 04 2009
Sarah

I have my suspicions. UpToDate gives astronomical institutional pricing, when compared to individual subscriptions. This is counter to what other publishers are doing. I suspect that UpToDate is using their subscription information for nefarious purposes. If someone at hospital X (but you don’t know who) is searching for information about asthma, that is a lot less valuable information than Dr. Smith of 123 Johnson Street, (202) 555-1234 searched for information about asthma. And hey, guy from Merck with new asthma drug, aren’t you going to see Dr. Smith this week?

7 04 2009
7 04 2009
laikaspoetnik

First, thank you all for your comments. I must say I’m a bit overwhelmed by the attention this post got.

@robin, thank u very much for your compliments. I appreciate it that you share your opinion directly (can imagine you didn’t digest it directly). And thanks for another example of “missing evidence”.

@allergynotes: I’m glad you took care of it – It is a relevant reference I forgot. It is now included, both as a reference and in the text.

@precordialthump. I share your view completely. Glad that many doctors are very information literate – and sensible.

Womans health news (@rachel-w?) Beehive? stick? Not on purpose. But apparently I touched a nerve (or a beehive) here.

@ Luca DF It is even more complex. Indeed it is not sufficient to rely on secondary literature alone. There is definitely a need for head to head comparisons and other research that will close gaps in evidence (that matter). On the other hand I do believe that secondary evidence is helpful in presenting (and appraising) the available evidence. There are many examples in the past (CRASH-trial, corticosteroid treatment in preterm birth), that lack of synthesis (and thus unawareness) of the available evidence indirectly caused many deaths. See my post here.
One can also wonder (as I did) whether UpToDate can be considered truly Evidence Based.

@searchmedica I will have a detailed look at your search engine later. Hope Leman has also written a quite positive post on searchmedica. Although interesting, I do not expect the search engine to be a replacement for a good quality textbook (as I see UpToDate) nor for the missing head to head comparisons.

@rose. This is the same argument referred to by @allergynotes and is discussed under [2]. I don’t discourage the use of UpToDate. In fact I find it a very useful, jet costly, source. People shouldn’t think in either..(EBM) or… (textbook). Each serves different purposes. However, it is the “evidencebasedness” of UpToDate that is discussed. Because UpToDate states that this is what it is and others see it as a sufficient reason to incentive people not using it. Everybody (hospital, library, doctor) may buy what it wants (and can pay for), but it should not be imposed.

@lori. I don’t understand it either. Please whisper the name of the competitor.

@danil. You make some important statements.
“i am not that convinced that uptodate is evidence based as often you cannot find any info as to why they are making that recommendation and they dont reference the evidence or rct’s or literature behind.” Exactly!
“plus I just dont have the time to spend hours browsing through pages and pages of information when I just need a quick answer to a clinical question that I am facing on the ward. its just like a big text book” Opinions differ (see @rose, kevinmd), but I understand what you mean.
“but I noticed they (BMJ) have a product for outside of the US which I guess is more international and certainly totally evidence based: http://bestpractice.bmj.com, some of this product apparently contains clinical evidence which is THE evidence base.”
Indeed BMJ Clinical Evidence is a very good evidence based product and bestpractice is an extension of it. However it only deals with interventions of the most (200) common diseases. It isn’t real cheap either. But I loved it very much. I said “loved” because the library has decided to cancel the subscription as off April. Can’t use it in my class tomorrow where it gives the most direct answer to 2 out of 3 questions: “is spironolactone effective in reducing hirsutism in PCOS?” and “what is most effective in inducing pregnancy in subfertile females with low grade endometriosis: surgery or medicinal treatment?”.

@sarah While it is true that UpToDate gives astronomical institutional pricing, when compared to individual subscriptions, I don’t think they are playing FBI. Probably they are *only* misusing their monopolistic position. It is “money” or “no deal”. Do you want to disappoint your patrons? The client is King. We buy them what they want (to a certain extent).

8 04 2009
Medlib’s Round, April 2009 « First Person Narrative

[...] Meanwhile, round founder, Jacqueline, plunges in more depth into one EBM tool and asks, ‘How evidence based is UpToDate really?‘ -  a reminder that however neat and convenient the product may be, we always need to keep [...]

8 04 2009
Anne Marie Cunningham

Hello!

I can’t access UptoDate and had not heard of it before reading about it on twitter and blogs. But last week I got a letter or invitation through the post. So it is coming to the UK.
Do you think it is more or less evidence based that GPNotelbook?
I’m not sure if you can access it but here is a tiny paste of the page on managing Type 2 DM http://tinypaste.com/f150741
Anne Marie

8 04 2009
danil

i am pretty sure Clinical Evidence covers closer to 500 conditions but these really are main condition categories within which there are multiple diagnoses, so it really is quite large. I am starting a trial of BMJ Best Practice which they say covers 10.000 diagnosis, so thats massive and I look forward to see how it all works, obviously you wont find evidence for each diagnosis so would be good to see how the folkes at the bmj cover recommendations when there is no evidence or weak evidence

9 04 2009
Christian Sinclair

Great post. Stopped using UTD after residency and have not missed it. Keeping on top of the literature with RSS feeds, delicious, and focused searches has been great. Thanks for the critical analysis of UTD (And thanks for MedLib Carnival, which is how I found this post!)

16 04 2009
UpToDate, and Twitter « MHSLA Blog

[...] UpToDate, and Twitter Posted on April 15, 2009 by Sandy Swanson Laika’s MedLibLog posts an interesting article summarizing a blog-and-twitter discussion of  UpToDate in How Evidence Based Is UpToDate really? [...]

16 04 2009
MDoctor Information

yep i have been using rss to…its just easier to me

28 05 2009
joel topf

I found an interesting paragraph in UpToDate which is about as anti-EBM as you can get. See: http://www.pbfluids.com/2009/05/uptodate-evidence-based-medicine-or-not.html

26 06 2009
Kris

What if utd was in the IT budget due to being point of care and integrated to workflow?Would there be such emotion? They do allow remote. Much of the info above was not factual, again it was emotional.

7 02 2010
another md

it appears taht this post has much anti-UTD sentiments. It claims UTD is not evidence based, yet this post attempts to simulate with “unbiased” references to twitter conversations to make its points.

I think the bottom line is this — if you don’t like it, don’t use it! Not everyone are specialists who requiers latest news on research developments. There are a large population of practionitionesr that just want some uptodate background info on certain clinical questions and don’t need to boggle through pubmed.

7 02 2010
laikaspoetnik

Dear “another MD”

IMHO, UpToDate is an excellent background source and some chapters give a real good -and up-to-date overview. Please (re)read point 2 which discusses the usefulness of UpToDate. UpToDate is also very expensive, so choosing UpToDate may mean giving up another good source (or more than one).

The main reason why I wrote this post is that the claim that UpToDate is “authoritative and unbiased”, and the suggestion that hospitals and providers should be incentivized to use UpToDate more regularly, seem absurd to me. I also wasn’t particularly impressed by the article that claims that hospitals do better when using UpToDate.

I use both arguments and other people’s views and sentiments (on Twitter or blogposts) to illustrate these points.
Besides “hard evidence” (or the lack thereof), people’s sentiments and experiences are also important.

If your experience is only positive, well that is fine with me.

Thanks for responding.

20 03 2010
An Evidence Pyramid that Facilitates the Finding of Evidence « Laika’s MedLibLog

[...] my colleagues and I decide whether a source is evidence based (we don’t include UpToDate for instance) and where it  belongs. Each clinical librarian (we all serve different departments) [...]

13 10 2011
Evidence Based Point of Care Summaries [1] No “Best” Among the Bests? « Laika's MedLibLog

[...] According to the “5S” paradigm[9] (now evolved to 6S -[10]) the peak of the pyramid are the ideal but not yet realized computer decision support systems, that link the individual patient characteristics to the current best evidence. According to the 5S model the next best source are Evidence Based Textbooks. (Note: EBM and textbooks almost seem a contradiction in terms to me, personally I would not put many of the POCs somewhere at the top. Also see my post: How Evidence Based is UpToDate really?) [...]

18 10 2011
Evidence Based Point of Care Summaries [2] More Uptodate with Dynamed. « Laika's MedLibLog

[...] How Evidence Based is UpToDate really? (laikaspoetnik.wordpress.com) [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




Follow

Get every new post delivered to your Inbox.

Join 607 other followers

%d bloggers like this: