The Trouble with Wikipedia as a Source for Medical Information

14 09 2009

This post was chosen as an Editor's Selection for

Do you ever use Wikipedia? I do and so do many other people. It is for free, easy to use, and covers many subjects.

But do you ever use Wikipedia to look up scientific or medical information? Probably everyone does so once in a while. Dave Munger (Researchblogging) concluded a discussion on Twitter as follows:

Logo of the English Wikipedia
Image via Wikipedia

“Wikipedia’s information quality is better than any encyclopedia, online or off. And, yes, it’s also easy to use”.

Wikipedia is an admirable initiative. It is a large online collaborative, multilingual encyclopedia written by contributors around the world.
But the key question is whether you can rely on Wikipedia as the sole source for medical, scientific or even popular information.

Well, you simply can’t and here are a few examples/findings to substantiate this point.



When you search  for diabetes in Google (EN), Wikipedia’s entry about diabetes ranks second, below the American Diabetes Association Home Page. A recent study published in the J Am Med Inform Assoc [1] confirms what you would expect: that the English Wikipedia is a prominent source of online health information. Wikipedia ranked among the first ten results in more than 70% of search engines and health-keywords tested, and outranked other sources in case of rare disease-related keywords. Wikipedia’s articles were viewed more frequently than the corresponding MedlinePlus Topic pages. This corroborates another study that can be downloaded from the internet here [10]. This study by Envision Solutions, LLC, licensed under the Creative Commons License, concluded that the exposure of Internet user’s to health-related user-generated media (UGM) is significant, Wikipedia being the most reference resource on Google and Yahoo.

The following (also from, from 2007 [10]) illustrates the impact of this finding:

According to the Pew Internet & American Life Project*, 10 million US adults search online for information on health each day [1]. Most (66%) begin their research on a search engine like Yahoo or Google. In addition, Americans are saying that the information they find on the Internet is having an impact. According to Pew, “53% of health seekers report that their most recent health information session [influenced] how they take care of themselves or care for someone else.” In addition, 56% say the information they find online has boosted their confidence in their healthcare decision-making abilities.

And according to an update from the Pew Internet & American Life Project (2009) [11]:

In 2000, 46% of American adults had access to the internet, 5% of U.S. households had broadband connections, and 25% of American adults looked online for health information. Now, 74% of American adults go online, 57% of American households have broadband connections, and 61% of adults look online for health information.

Thus a lot of people look online for health care questions and are more inclined to use highly ranked sources.
This is not unique for Health topics but is a general phenomenon, i.e. see this mini-study performed by a curious individual: 96.6% of Wikipedia Pages Rank in Google’s Top 10 [12]. The extreme high traffic to Wikipedia due to search referrals has  even been been denounced by SEO-people (see here) [13]: if you type “holiday” Wikipedia provides little value when ranking in the top 10: everybody knows what a holiday is 😉

Medical students use it too.

A nightmare for most educators in the curriculum is that students rely on UGM or Web 2.0 sites as a source  of medical information. Just walk along medical students as they work behind their computers and take a quick glance at the pages they are consulting. These webpages often belong to the above category.

AnneMarie Cunningham, GP and Clinical Lecturer in the UK, did a little informal “survey” on the subject. She asked 31 first year medical students about their early clinical attachments in primary and secondary care and summerized the results on her blog Wishful Thinking in Medical Education [14]. By far and away Wikipedia was the most common choice to look up unfamiliar clinical topics.


‘Many students said I know I shouldn’t but….’ and then qualified that they used Wikipedia first because it was easy to understand, they felt it was reasonably reliable, and accessible. One student used it to search directly from her phone when on placement..

50% of the doctors use it!

But these are only medical students. Practicing doctors won’t use Wikipedia to solve their clinical questions, because they know where to find reliable medical information.


The New Scientist cites a report [15] of US healthcare consultancy Manhattan Research (April 2009), stating that that 50 percent of the doctors turn to Wikipedia for medical information.

A recent qualitative study published in Int J Med Inform [2] examined the “Web 2.0” use by 35 junior physicians in the UK. Diaries and interviews encompassing 177 days of internet use or 444 search incidents, analyzed via thematic analysis. Although concepts are loosely defined (Web 2.0, internet and UMG are not properly defined, i.e. Google is seen as a web 2.0 tool (!) [see Annemarie’s critical review [16] the results clearly show that 89% of these young physicians use at least one “Web 2.0 tool” (including Google!) in their medical practice, with 80% (28/35) reporting the use of wikis. The visit of wiki’s is largely accounted for by visits to Wikipedia: this was the second most commonly visited site, used in 26% (115/44) of cases and by 70% (25/35) of all physicians. Notably, only one respondent made regular contribution to a medical wiki site.

The main motivation for using the Internet for information seeking was the accessibility and ease of use over other tools (like textbooks), the uptodateness, the broad coverage and the extras such as interactive immages. On the other hand most clinicians realized that there was a limitation in the quality or usefulness of information found. It is reassuring that most doctors used UGM like Wikipedia for background or open questions, to fulfill the need for more in depth knowledge on a subject, or to find information for patients, not for immediate solving of clinical questions.

The Int J Med Inform article has been widely covered by blogs: i.e. see Wishful Thinking in Medical Education [16], Dr Shock, MD, PhD [17], Life in the Fast Lane [18], Clinical Cases and Images Blog [19] and Scienceroll [20].

Apparently some doctors also heavily rely on Wikipedia that they refer to Wikipedia articles in publications (see the Int. J Cardiol. PubMed [3] abstract below)!!

8-9-2009 14-03-15 Int J cardiol wikipedia references 2


Whether the common use of Wikipedia by e-patient, medical students and doctors is disadvantageous depends on the quality and the trustworthiness of the Wikipedia articles, and that is in its turn dependent on who writes the articles.

Basically, the strength of Wikipedia is it weakness: anyone can write anything on any subject, and anyone can edit it, anonymously.

Negative aspects include its coverage (choice of subjects but also the depth of coverage), the “overlinking”, the sometimes frustating interactions between authors and editors, regularly leading to (often polite) “revision wars“, but above all the lack of ‘expert’ authors or peer review. This may result in incomplete, wrong or distorted information.

Positive aspects are its accessibility, currency, availability in many languages, and the collective “authorship” (which is an admirable concept).

The following humorist video shows how the wisdom of the crowds can lead to chaos, incorrect and variable information.

SCOPE AND ACCURACY (What has been covered, how deep and how good) :

Too much, too little, too ….

With respect to its coverage one study in the Journal of Computer-Mediated Communication (2008) [4] concludes:

Differences in the interests and attention of Wikipedia’s editors mean that some areas, in the traditional sciences, for example, are better covered than others. (…)
Overall, we found that the degree to which Wikipedia is lacking depends heavily on one’s perspective. Even in the least covered areas, because of its sheer size, Wikipedia does well, but since a collection that is meant to represent general knowledge is likely to be judged by the areas in which it is weakest, it is important to identify these areas and determine why they are not more fully elaborated. It cannot be a coincidence that two areas that are particularly lacking on Wikipedia—law and medicine—are also the purview of licensed experts.

It is not unexpected though that Wikipedia’s topical coverage is driven by the interests of its users.

Sometimes data are added to Wikipedia, that are in itself correct, but controversial. Recently, Wikipedia published the 10 inkblots (Scienceroll, [21]) of the Rorschach test, along with common responses for each. This had led to complaints by Psychologists , who argue that the site is jeopardizing one of the oldest continuously used psychological assessment tests (NY Times [22]).

The actual coverage of medical subjects may vary greatly. In one study [5], abstract-format, 2007) Wikipedia entries were screened for the most commonly performed inpatient surgical procedures in the U.S. Of the 39 procedures, 35 were indexed on Wikipedia. 85.7% of these articles were deemed appropriate for patients. All 35 articles presented accurate content, although only 62.9% (n=22) were free of critical omissions. Risks of the procedures were significantly underreported. There was a correlation between an entry’s quality and how often it was edited.

Wikipedia may even be less suitable for drug information questions, questions that one-third of all Internet health-seekers search for. A study in Annals of Pharmacotherapy [6] comparing the scope, completeness, and accuracy of drug information in Wikipedia to a free, online, traditionally edited database (Medscape Drug Reference [MDR]) showed that  Wikipedia answered significantly fewer drug information questions (40.0%) compared with MDR (82.5%; p < 0.001) and that Wikipedia answers were less complete. Although no factual errors were found, errors of omission were higher in Wikipedia (n = 48) than in MDR (n = 14). The authors did notice a marked improvement in Wikipedia over time. The authors conclude:

This study suggests that Wikipedia may be a useful point of engagement for consumers looking for drug information, but that it should be supplementary to, rather than the sole source of, drug information. This is due, in part, to our findings that Wikipedia has a more narrow scope, is less complete, and has more errors of omission versus the comparator database. Consumers relying on incomplete entries for drug information risk being ill-informed with respect to important safety features such as adverse drug events, contraindications, drug interactions, and use in pregnancy.
These errors of omission may prove to be a substantial and largely hidden danger associated with exclusive use of
user-edited drug information sources.

Alternatively, user-edited sites may serve as an effective means of disseminating drug information and are promising as a means of more actively involving consumers in their own care. However, health professionals should not use user-edited sites as authoritative sources in their clinical practice, nor should they recommend them to patients without knowing the limitations and providing sufficient additional information and counsel…

Not Evidence Based

German researches found [7], not surprisingly, that Wikipedia (as well as two major German statutory health insurances):

“…failed to meet relevant criteria, and key information such as the presentation of probabilities of success on patient-relevant outcomes, probabilities of unwanted effects, and unbiased risk communication was missing. On average items related to the objectives of interventions, the natural course of disease and treatment options were only rated as “partially fulfilled”. (..)  In addition, the Wikipedia information tended to achieve lower comprehensibility. In conclusion(..) Wikipedia (..) does not meet important criteria of evidence-based patient and consumer information though…”

Wrong, misleading, inaccurate

All above studies point at the incompleteness of Wikipedia. Even more serious is the fact that some of the Wikipedia addings are wrong or misleading. Sometimes on purpose. The 15 biggest wikipedia blunders [23] include the death announcements of Ted Kennedy (when he was still alive),  Robert Byrd and others. Almost hilarious are the real time Wikipedia revisions after the presumed death of Kennedy and the death of Ken Lay (suicide, murde, heart attack? [24).

In the field of medicine, several drug companies have been caught altering Wikipedia entries. The first drug company messing with Wikipedia was AstraZeneca. References claiming that Seroquel allegedly made teenagers “more likely to think about harming or killing themselves” were deleted by a user of a computer registered to the drug company [25], according to Times [26]. Employees of Abbott Laboratories have also been altering entries to Wikipedia to “eliminate information questioning the safety of its top-selling drugs.”(See WSJ-blog [27] , [28], and recently Kevin MD[29])

These are “straightforward” examples of fraudulent material. But sometimes the Wikipedia articles are more subtly colored by positive or negative bias.

Take for instance the English entry on Evidence Based Medicine (in fact the reason why I started this post). Totally open-minded I checked the entry, which was automatically generated in one of my posts by Zemanta. First I was surprised by the definition of EBM:

Evidence-based medicine (EBM) aims to apply the best available evidence gained from the scientific method to medical decision making. It seeks to assess the quality of evidence of the risks and benefits of treatments (including lack of treatment).

instead of the usually cited Sacket-definition (this is only cited at the end of the paper):

“the practice of evidence based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research”

In short, the whole article lacks cohesion: the definitions of EBM are not correct, there is too much emphasis on not directly relevant information (4 ways to grade the evidence and 3 statistical measures), the limitations are overemphasized (cf. chapter 7 with 6 in the Figure below) and put out of perspective.

Apparently this has also been noted by Wikipedia, because there is a notice on the Evidence Based Medicine Page saying:

This article has been nominated to be checked for its neutrality. Discussion of this nomination can be found on the talk page. (May 2009)

9-9-2009 9-55-04 wikipedia EBM start smal

Much to my surprise the article had been written by Mr-Natural-Health, who’s account seems not to be in use since 2004  and who is currently active as User:John Gohde. Mr Natural Health is a member of WikiProject Alternative medicine.

Now why in earth would some advocate of CAM write the Wikipedia EBM-entry? I can think of 4 (not mutually exclusive) reasons:

  1. When you’re an EBM-nonbeliever or opponent this is THE chance to misinform readers about EBM (to the advantage of CAM).
  2. The author was invited to write this entry.
  3. No EBM-specialist or epidemiologist is willing to write the entry, or to write for Wikipedia in general (perhaps because they find Wikipedia lacks trustworthiness?)
  4. EBM specialists/epidemiologists are not “allowed”/hindered to make major amendments to the text, let alone rewrite it.

According to Mr Naturopath point 2 is THE reason he wrote this article. Now the next question is “exactly by whom was he invited?” But the TALK-page reveals that Mr Naturapath makes it a tough job for other, better qualified writers, to edit the page (point 4). To see how difficult it is for someone to re-edit a page, please see the TALK-page. In fact, one look at this page discourages me from ever trying to make some amendments to any Wikpedia text.


Changes to Wikipedia’s organization

Wikipedia has long grasped that its Achilles heel is the free editability (see for instance this interview with Wikipedia’s founder [30]). Therefore, “WikiProjects” was initiated to help coordinate and organize the writing and editing of articles on a certain topic, as well as “Citizendium” which is an English-language wiki-based free encyclopedia project aimed to improve the Wikipedia model by providing a “reliable” encyclopedia. “It hopes to achieve this by requiring all contributors to use their real names, by strictly moderating the project for unprofessional behavior, by providing what it calls “gentle expert oversight” of everyday contributors, and also through its “approved articles,” which have undergone a form of peer-review by credentialed topic experts and are closed to real-time editing.”

Starting this fall Wikipedia will launch an optional feature called “WikiTrust” will color code every word of the encyclopedia based on the reliability of its author and the length of time it has persisted on the page: Text from questionable sources starts out with a bright orange background, while text from trusted authors gets a lighter shade.

9-9-2009 15-25-36 wikipedia wikiproject medicine

The Wikipedia EBM article is within the scope of these two projects, and this is good news. However, Wikipedia still clings to the idea that: “Everyone is welcome to join in this endeavor (regardless of medical qualifications!).” In my opinion, it would be better if Wikipedia gave precedence to experts instead of hobbyists/ people from another field, because the former can be expected to know what they are talking about. It is quite off-putting for experts to contribute. See this shout-out:

Who are these so-called experts who will qualify material? From what I’ve seen so far, being an academic expert in a particular field hardly protects one from edit wars–Julie and 172 are two primary examples of this. Meanwhile, the only qualification I have seen so far is that they have a B.A. Gimme a friggin’ break! (and before I get accused of academic elitism, I make it known that I dropped out of college and spend an inordinate amount of time at work correcting the BS from the BAs, MAs, and PhDs).

While anyone can still edit entries, the site is testing pages that require changes to be approved by an experienced Wikipedia editor before they show up, the so called Flagged protection and patrolled revisions. (see Wikimedia) This proposal is only for articles that are currently under normal mechanisms of protection (i.e. the Obama-article cannot be edited by a newcomer).

Although this seems logic, it is questionable whether “experienced” editors are per definition better qualified than newcomers. A recent interesting analysis of the Augmented Social Cognition group [31], (cited in the Guardian [32]) shows a slowdown in growth of Wikipedia activity, with the activity slightly declining in all classes of editors except for the highest-frequency class of editors (1000+ edits). Here is an increase in their monthly edits.

In addition the study shows growing resistance from the Wikipedia community to new content. The total percentage of reverted edits increased steadily over the years, but more interestingly, low-frequency or occasional editors experienced a visibly greater resistance compared to high-frequency editors . Together this points at a growing resistance from the Wikipedia community to new content, especially when the edits come from occasional editors.

This is more or less in line with an earlier finding [9] showing that Wikipedia members feel more comfortable expressing themselves on the net than off-line and scored lower on agreeableness and openness compared to non-Wikipedians, a finding that was interpreted as consistent with the possibility that contributing to Wikipedia serves mainly egocentric motives.

Image representing Medpedia as depicted in Cru...
Image via CrunchBase

Encouraging students, doctors and scientists (provisional)

One way of improving content, is to encourage experts to write. To achieve that the information on Wikipedia is of the highest quality and up-to-date, the NIH is encouraging its scientists and science writers to edit and even initiate Wikipedia articles in their fields [36]. It joined with the Wikimedia Foundation, to host  a training session on the tools and rules of wiki culture, at NIH headquarters in Bethesda.

A less noncommital approach is the demand to “Publish in Wikipedia or perish”, as described in Nature News [9]. Anyone submitting to a section of the journal RNA Biology will, in the future, be required to also submit a Wikipedia page that summarizes the work. The journal will then peer review the page before publishing it in Wikipedia.” The project is described in detail here [10] and the wiki can be viewed here

Wiki’s for experts.

One possible solution is that scientist and medica experts contribute to wiki’s other than the Wikipedia. One such wiki is the wiki-surgery [5]. PubDrugRxWiki , WikiProteins [11] and Gene Wiki [12] are other examples. In general, scientists are more inclined to contribute to these specialists wiki’s, that have oversight and formal contributions by fellow practitioners (this is also true for the RNA-wiki)

A medical Wikipedia

Yet another solution is a medical wikipedia, such as Ganfyd or Medpedia . Ganfyd is written by medical professionals. To qualify to edit or contribute to the main content of Medpedia approved editors must have an M.D., D.O., or Ph.D. in a biomedical field. Others, however, may contribute by writing in suggestions for changes to the site using the “Make a suggestion” link at the top of each page. Suggestions are reviewed by approved editors. Whether these medical wikipedias will succeed will depend on the input of experts and their popularity: to what extent will they be consulted by people with health questions?

I would like to end with a quote from Berci during twitterview (link in Wikipedia):

@Berci : @diariomedico And as Wikipedians say, Wikipedia is the best source to start with in your research, but should never be the last one. #DM1 9 months ago


ResearchBlogging.orgScientific Articles

  1. Laurent, M., & Vickers, T. (2009). Seeking Health Information Online: Does Wikipedia Matter? Journal of the American Medical Informatics Association, 16 (4), 471-479 DOI: 10.1197/jamia.M3059
  2. Hughes, B., Joshi, I., Lemonde, H., & Wareham, J. (2009). Junior physician’s use of Web 2.0 for information seeking and medical education: A qualitative study International Journal of Medical Informatics, 78 (10), 645-655 DOI: 10.1016/j.ijmedinf.2009.04.008
  3. Lee, C., Teo, C., & Low, A. (2009). Fulminant dengue myocarditis masquerading as acute myocardial infarction International Journal of Cardiology, 136 (3) DOI: 10.1016/j.ijcard.2008.05.023
  4. Halavais, A., & Lackaff, D. (2008). An Analysis of Topical Coverage of Wikipedia Journal of Computer-Mediated Communication, 13 (2), 429-440 DOI: 10.1111/j.1083-6101.2008.00403.x
  5. Devgan, L., Powe, N., Blakey, B., & Makary, M. (2007). Wiki-Surgery? Internal validity of Wikipedia as a medical and surgical reference Journal of the American College of Surgeons, 205 (3) DOI: 10.1016/j.jamcollsurg.2007.06.190
  6. Clauson, K., Polen, H., Boulos, M., & Dzenowagis, J. (2008). Scope, Completeness, and Accuracy of Drug Information in Wikipedia Annals of Pharmacotherapy, 42 (12), 1814-1821 DOI: 10.1345/aph.1L474 (free full text)
  7. Mühlhauser I, & Oser F (2008). [Does WIKIPEDIA provide evidence-based health care information? A content analysis] Zeitschrift fur Evidenz, Fortbildung und Qualitat im Gesundheitswesen, 102 (7), 441-8 PMID: 19209572
  8. Amichai–Hamburger, Y., Lamdan, N., Madiel, R., & Hayat, T. (2008). Personality Characteristics of Wikipedia Members CyberPsychology & Behavior, 11 (6), 679-681 DOI: 10.1089/cpb.2007.0225
  9. Butler, D. (2008). Publish in Wikipedia or perish Nature DOI: 10.1038/news.2008.1312
  10. Daub, J., Gardner, P., Tate, J., Ramskold, D., Manske, M., Scott, W., Weinberg, Z., Griffiths-Jones, S., & Bateman, A. (2008). The RNA WikiProject: Community annotation of RNA families RNA, 14 (12), 2462-2464 DOI: 10.1261/rna.1200508
  11. Mons, B., Ashburner, M., Chichester, C., van Mulligen, E., Weeber, M., den Dunnen, J., van Ommen, G., Musen, M., Cockerill, M., Hermjakob, H., Mons, A., Packer, A., Pacheco, R., Lewis, S., Berkeley, A., Melton, W., Barris, N., Wales, J., Meijssen, G., Moeller, E., Roes, P., Borner, K., & Bairoch, A. (2008). Calling on a million minds for community annotation in WikiProteins Genome Biology, 9 (5) DOI: 10.1186/gb-2008-9-5-r89
  12. Huss, J., Orozco, C., Goodale, J., Wu, C., Batalov, S., Vickers, T., Valafar, F., & Su, A. (2008). A Gene Wiki for Community Annotation of Gene Function PLoS Biology, 6 (7) DOI: 10.1371/journal.pbio.0060175
    Other Publications, blogposts
    (numbers in text need to be adapted)

  13. Envision Solutions, LLC. Diving Deeper Into Online Health Search – Examining Why People Trust Internet Content & The Impact Of User-Generated Media (2007) Accessed August 2009 (CC)
  14. New data available of the the Pew Internet & American Life Project are available here)
  22. Wikipedia used by 70% of junior physicians, dominates search results for health queries (
  25. (Rorschach)
Reblog this post [with Zemanta]

Auteursrecht nekt de digitale bibliotheek

5 05 2008

Via bakkel dot com een bespreking gelezen: “Auteursrecht maakt meer kapot dan je lief is” van Arjen Kamphuis naar aanleiding van een interessant artikel in het NRC van Martin Bossenbroek en Hans Jansen van de KB ( Auteurswet nekt de digitale bibliotheek).

Het auteursrecht eist dat tot tientallen jaren later nog toestemming wordt gevraagd voor hergebruik. Dit dient ter bescherming van auteurs, maar werkt in deze tijd van digitalisering averechts. Alleen al het achterhalen van (alle) auteurs is een schier onmogelijke klus als het om een verweesd werk (auteurs onbekend of onvindbaar) of krant/tijdschrift gaat (freelance journalisten, fotografen, tekenaars, schrijvers, advertentieontwerpers en niet te vergeten ingezondenbrievenschrijvers).

Uiteindelijk biedt het auteursrecht ook voor de auteur geen bescherming maar louter uitzicht op vergetelheid, aldus Bossenbroek en Jansen. Dat kan toch niet de bedoeling zijn.

Overigens heeft volgens een van de commentaren op het blog van Arjen Kamphuis (#7) kleine webloggers ook te lijden onder auteursrechten. Het opsporen van auteursrechtschendingen wordt door sommige auteurs gezien als het nieuwe brood op de plank voor de auteur om eens wat ‘extra te verdienen op artikelen’.

Ik moet zeggen dat ik er zelf ternauwernood op let. Qua tekst refereer ik netjes mijn bronnen (indien inhoudelijk relevant), maar plaatjes pluk ik her en der van het net. Een tijdje geleden sloeg de schrik mij om het hart toen Google zoekacties op “Laika Spoetnik” bepaalde zoekresultaten steevast niet toonden, omdat hier het auteursrecht geschonden zou zijn. Het zal je in alle onschuld (lees naïviteit) maar overkomen…

Het volgende slot heb ik zeer toepasselijk overgenomen van Arjen Kamphuis:

Dit werk valt, net als al mijn publieke werk, onder een
Creative Commons licentie. Want het grootste risico dat een auteur vandaag de dag kan lopen is niet dat je werk ‘gestolen’ wordt maar dat je werk niet gelezen wordt. Share & enjoy!