PubMed versus Google Scholar for Retrieving Evidence

8 06 2010

ResearchBlogging.orgA while ago a resident in dermatology told me she got many hits out of PubMed, but zero results out of TRIP. It appeared she had used the same search for both databases: alopecea areata and diphenciprone (a drug with a lot of synonyms). Searching TRIP for alopecea (in the title) only, we found a Cochrane Review and a relevant NICE guideline.

Usually, each search engine has is its own search and index features. When comparing databases one should compare “optimal” searches and keep in mind for what purpose the search engines are designed. TRIP is most suited to search aggregate evidence, whereas PubMed is most suited to search individual biomedical articles.

Michael Anders and Dennis Evans ignore this “rule of the thumb” in their recent paper “Comparison of PubMed and Google Scholar Literature Searches”. And this is not the only shortcoming of the paper.

The authors performed searches on 3 different topics to compare PubMed and Google Scholar search results. Their main aim was to see which database was the most useful to find clinical evidence in respiratory care.

Well quick guess: PubMed wins…

The 3 respiratory care topics were selected from a list of systematic reviews on the Website of the Cochrane Collaboration and represented in-patient care, out-patient care, and pediatrics.

The references in the three chosen Cochrane Systematic Reviews served as a “reference” (or “golden”) standard. However, abstracts, conference proceedings, and responses to letters were excluded.

So far so good. But note that the outcome of the study only allows us to draw conclusions about interventional questions, that seek to find controlled clinical trials. Other principles may apply to other domains (diagnosis, etiology/harm, prognosis ) or to other types of studies. And it certainly doesn’t apply to non-EBM-topics.

The authors designed ONE search for each topic, by taking 2 common clinical terms from the title of each Cochrane review connected by the Boolean operator “AND” (see Table, ” ” are not used). No synonyms were used and the translation of searches in PubMed wasn’t checked (luckily the mapping was rather good).

“Mmmmm…”

Topic

Search Terms

Noninvasive positive-pressure ventilation for cardiogenic pulmonary edema “noninvasive positive-pressure ventilation” AND “pulmonary edema”
Self-management education and regular practitioner review for adults with asthma “asthma” AND “education”
Ribavirin for respiratory syncytial virus “ribavirin” AND “respiratory syncytial virus”

In PubMed they applied the narrow methodological filter, or Clinical Query, for the domain therapy.
This prefab search strategy (randomized controlled trial[Publication Type] OR (randomized[Title/Abstract] AND controlled[Title/Abstract] AND trial[Title/Abstract]), developed by Haynes, is suitable to quickly detect the available evidence (provided one is looking for RCT’s and doesn’t do an exhaustive search). (see previous posts 2, 3, 4)

Google Scholar, as we all probably know, does not have such methodological filters, but the authors “limited” their search by using the Advanced option and enter the 2 search terms in the “Find articles….with all of the words” space (so this is a boolean “AND“) and they limited it the search to the subject area “Medicine, Pharmacology, and Veterinary Science”.

They did a separate search for publications that were available at their library, which has limited value for others, subscriptions being different for each library.

Next they determined the sensitivity (the number of relevant records retrieved as a proportion of the total number of records in the gold standard) and the precision or positive predictive value, the  fraction of returned positives that are true positives (explained in 3).

Let me guess: sensitivity might be equal or somewhat higher, and precision is undoubtedly much lower in Google Scholar. This is because (in) Google Scholar:

  • you can often search full text instead of just in the abstract, title and (added) keywords/MeSH
  • the results are inflated by finding one and the same references cited in many different papers (that might not directly deal with the subject).
  • you can’t  limit on methodology, study type or “evidence”
  • there is no automatic mapping and explosion (which may provide a way to find more synonyms and thus more relevant studies)
  • has a broader coverage (grey literature, books, more topics)
  • lags behind PubMed in receiving updates from MEDLINE

Results: PubMed and Google Scholar had pretty much the same recall, but for ribavirin and RSV the recall was higher in PubMed, PubMed finding 100%  (12/12) of the included trials, and Google Scholar 58% (7/12)

No discussion as to the why. Since Google Scholar should find the words in titles and abstracts of PubMed I repeated the search in PubMed but only in the title, abstract field, so I searched ribavirin[tiab] AND respiratory syncytial virus[tiab]* and limited it with the narrow therapy filter: I found 26 papers instead of 32. These titles were missing when I only searched title and abstract (between brackets: [relevant MeSH (reason why paper was found), absence of abstract (thus only title and MeSH) and letter], bold: why terms in title abstract are not found)

  1. Evaluation by survival analysis on effect of traditional Chinese medicine in treating children with respiratory syncytial viral pneumonia of phlegm-heat blocking Fei syndrome.
    [MesH:
    Respiratory Syncytial Virus Infections/]
  2. Ribavarin in ventilated respiratory syncytial virus bronchiolitis: a randomized, placebo-controlled trial.
    [MeSH:
    Respiratory Syncytial Virus Infections/[NO ABSTRACT, LETTER]
  3. Study of interobserver reliability in clinical assessment of RSV lower respiratory illness.
    [MeSH:Respiratory Syncytial Virus Infections*]
  4. Ribavirin for severe RSV infection. N Engl J Med.
    [MeSH: Respiratory Syncytial Viruses
    [NO ABSTRACT, LETTER]
  5. Stutman HR, Rub B, Janaim HK. New data on clinical efficacy of ribavirin.
    MeSH: Respiratory Syncytial Viruses
    [NO ABSTRACT]
  6. Clinical studies with ribavirin.
    MeSH: Respiratory Syncytial Viruses
    [NO ABSTRACT]

Three of the papers had the additional MeSH respiratory syncytial virus and the three others respiratory syncytial virus infections. Although not all papers (2 comments/letters) may be relevant, it illustrates why PubMed may yield results, that are not retrieved by Google Scholar (if one doesn’t use synonyms)

In Contrast to Google Scholar, PubMed translates the search ribavirin AND respiratory syncytial virus so that the MeSH-terms “ribavirin”, “respiratory syncytial viruses”[MeSH Terms] and (indirectly) respiratory syncytial virus infection”[MeSH] are also found.

Thus in Google Scholar articles with terms like RSV and respiratory syncytial viral pneumonia (or lack of specifications, like clinical efficacy) could have been missed with the above-mentioned search.

The other result of the study (the result section comprises 3 sentences) is that “For each individual search, PubMed had better precision”.

The Precision was 59/467 (13%) in PubMed and 57/80,730 (0.07%)  in Google Scholar (p<0.001)!!
(note: they had to add author names in the Google Scholar search to find the papers in the haystack 😉

Héhéhé, how surprising. Well why would it be that no clinician or librarian would ever think of using Google Scholar as the primary, let alone the only, source to search for medical evidence?
It should also ring a bell, that [QUOTE**]:
In the Cochrane reviews the researchers retrieved information from multiple databases, including MEDLINE, the Cochrane Airways Group trial register (derived from MEDLINE)***, CENTRAL, EMBASE, CINAHL, DARE, NHSEED, the Acute Respiratory Infections Group’s specialized register, and LILACS… ”
Note
Google Scholar isn’t mentioned as a source! Google Scholar is only recommendable to search for work citing (already found) relevant articles (this is called forward searching), if one hasn’t access to Web of Science or SCOPUS. Thus only to catch the last fish.

Perhaps the paper could have been more interesting if the authors had looked at any ADDED VALUE of Google Scholar, when exhaustively searching for evidence. Then it would have been crucial to look for grey literature too, (instead of excluding it), because this could be a possible strong point for Google Scholar. Furthermore one could have researched if forward searching yielded extra papers.

The specificity of PubMed is attributed to the used therapy-narrow filter, but the vastly lower specificity of Google Scholar is also due to the searching in the full text, including the reference lists.

For instance, searching for ribavirin AND respiratory syncytial virus in PubMed yields 523 hits. This can be reduced to 32 hits when applying the narrow therapy filter. This means a reduction by a factor of 16.
Yet a similar search in Google Scholar yield
4,080 hits. Thus without the filter there is still an almost 8 times higher yield from Google Scholar than from PubMed.

That evokes another  research idea: what would have happened if randomized (OR randomised) would have been added to the Google Scholar search? Would this have increased the specificity? In case of the above search it lowers the yield with a factor 2, and the first hits look very relevant.

It is really funny but the authors bring down their own conclusion that “These results are important because efficient retrieval of the best available scientific evidence can inform respiratory care protocols, recommendations for clinical decisions in individual patients, and education, while minimizing information overload.” by saying elsewhere that “It is unlikely that users consider more than the first few hundred search results, so RTs who conduct literature searches with Google Scholar on these topics will be much less likely to find references cited in Cochrane reviews.”

Indeed no one would take it into ones head to try to find the relevant papers out of those 4,080 hits retrieved. So what is this study worth from a practical point of view?

Well anyway, as you can ask for the sake of asking you can research for the sake of researching. Despite being an EBM-addict I prefer a good subjective overview on this topic over a weak scientific, quasi-evidence based, research paper.

Does this mean Google Scholar is useless? Does it mean that all those PhD’s hooked on Google Scholar are wrong?

No, Google Scholar serves certain purposes.

Just like the example of PubMed and TRIP, you need to know what is in it for you and how to use it.

I used Google Scholar when I was a researcher:

  • to quickly find a known reference
  • to find citing papers
  • to get an idea of how much articles have been cited/ find the most relevant papers in a quick and dirty way (i.e. by browsing)
  • for quick and dirty searches by putting words string between brackets.
  • to search full text. I used quite extensive searches to find out what methods were used (for instance methods AND (synonym1 or syn2 or syn3)). An interesting possibility is to do a second search for only the last few words (in a string). This will often reveal the next words in the sentence. Often you can repeat this trick, reading a piece of the paper without need for access.

If you want to know more about the pros and cons of Google Scholar I recommend the recent overview by the expert librarian Dean Giustini: “Sure Google Scholar is ideal for some things” [7]”. He also compiled a “Google scholar bibliography” with ~115 articles as of May 2010.

Speaking of librarians, why was the study performed by PhD RRT (RN)’s and wasn’t the university librarian involved?****

* this is a search string and more strict than respiratory AND syncytial AND virus
**
abbreviations used instead of full (database) names
*** this is wrong, a register contains references to controlled clinical trials from EMBASE, CINAHL and all kind of  databases in addition to MEDLINE.
****other then to read the manuscript afterwards.

References

  1. Anders ME, & Evans DP (2010). Comparison of PubMed and Google Scholar Literature Searches. Respiratory care, 55 (5), 578-83 PMID: 20420728
  2. This Blog: https://laikaspoetnik.wordpress.com/2009/11/26/adding-methodological-filters-to-myncbi/
  3. This Blog: https://laikaspoetnik.wordpress.com/2009/01/22/search-filters-1-an-introduction/
  4. This Blog: https://laikaspoetnik.wordpress.com/2009/06/30/10-1-pubmed-tips-for-residents-and-their-instructors/
  5. NeuroDojo (2010/05) Pubmed vs Google Scholar? [also gives a nice overview of pros and cons]
  6. GenomeWeb (2010/05/10) Content versus interface at the heart of Pubmed versus Scholar?/ [response to 5]
  7. The Search principle Blog (2010/05) Sure Google Scholar is ideal for some things.
Advertisements




The Trouble with Wikipedia as a Source for Medical Information

14 09 2009

This post was chosen as an Editor's Selection for ResearchBlogging.org

Do you ever use Wikipedia? I do and so do many other people. It is for free, easy to use, and covers many subjects.

But do you ever use Wikipedia to look up scientific or medical information? Probably everyone does so once in a while. Dave Munger (Researchblogging) concluded a discussion on Twitter as follows:

Logo of the English Wikipedia
Image via Wikipedia

“Wikipedia’s information quality is better than any encyclopedia, online or off. And, yes, it’s also easy to use”.

Wikipedia is an admirable initiative. It is a large online collaborative, multilingual encyclopedia written by contributors around the world.
But the key question is whether you can rely on Wikipedia as the sole source for medical, scientific or even popular information.

Well, you simply can’t and here are a few examples/findings to substantiate this point.

RANKING AND USE

E-patients

When you search  for diabetes in Google (EN), Wikipedia’s entry about diabetes ranks second, below the American Diabetes Association Home Page. A recent study published in the J Am Med Inform Assoc [1] confirms what you would expect: that the English Wikipedia is a prominent source of online health information. Wikipedia ranked among the first ten results in more than 70% of search engines and health-keywords tested, and outranked other sources in case of rare disease-related keywords. Wikipedia’s articles were viewed more frequently than the corresponding MedlinePlus Topic pages. This corroborates another study that can be downloaded from the internet here [10]. This study by Envision Solutions, LLC, licensed under the Creative Commons License, concluded that the exposure of Internet user’s to health-related user-generated media (UGM) is significant, Wikipedia being the most reference resource on Google and Yahoo.

The following (also from envisionsolutionsnow.com, from 2007 [10]) illustrates the impact of this finding:

According to the Pew Internet & American Life Project*, 10 million US adults search online for information on health each day [1]. Most (66%) begin their research on a search engine like Yahoo or Google. In addition, Americans are saying that the information they find on the Internet is having an impact. According to Pew, “53% of health seekers report that their most recent health information session [influenced] how they take care of themselves or care for someone else.” In addition, 56% say the information they find online has boosted their confidence in their healthcare decision-making abilities.

And according to an update from the Pew Internet & American Life Project (2009) [11]:

In 2000, 46% of American adults had access to the internet, 5% of U.S. households had broadband connections, and 25% of American adults looked online for health information. Now, 74% of American adults go online, 57% of American households have broadband connections, and 61% of adults look online for health information.

Thus a lot of people look online for health care questions and are more inclined to use highly ranked sources.
This is not unique for Health topics but is a general phenomenon, i.e. see this mini-study performed by a curious individual: 96.6% of Wikipedia Pages Rank in Google’s Top 10 [12]. The extreme high traffic to Wikipedia due to search referrals has  even been been denounced by SEO-people (see here) [13]: if you type “holiday” Wikipedia provides little value when ranking in the top 10: everybody knows what a holiday is 😉

Medical students use it too.

A nightmare for most educators in the curriculum is that students rely on UGM or Web 2.0 sites as a source  of medical information. Just walk along medical students as they work behind their computers and take a quick glance at the pages they are consulting. These webpages often belong to the above category.

AnneMarie Cunningham, GP and Clinical Lecturer in the UK, did a little informal “survey” on the subject. She asked 31 first year medical students about their early clinical attachments in primary and secondary care and summerized the results on her blog Wishful Thinking in Medical Education [14]. By far and away Wikipedia was the most common choice to look up unfamiliar clinical topics.

AnneMarie:

‘Many students said I know I shouldn’t but….’ and then qualified that they used Wikipedia first because it was easy to understand, they felt it was reasonably reliable, and accessible. One student used it to search directly from her phone when on placement..

50% of the doctors use it!

But these are only medical students. Practicing doctors won’t use Wikipedia to solve their clinical questions, because they know where to find reliable medical information.

Wrong!

The New Scientist cites a report [15] of US healthcare consultancy Manhattan Research (April 2009), stating that that 50 percent of the doctors turn to Wikipedia for medical information.

A recent qualitative study published in Int J Med Inform [2] examined the “Web 2.0” use by 35 junior physicians in the UK. Diaries and interviews encompassing 177 days of internet use or 444 search incidents, analyzed via thematic analysis. Although concepts are loosely defined (Web 2.0, internet and UMG are not properly defined, i.e. Google is seen as a web 2.0 tool (!) [see Annemarie’s critical review [16] the results clearly show that 89% of these young physicians use at least one “Web 2.0 tool” (including Google!) in their medical practice, with 80% (28/35) reporting the use of wikis. The visit of wiki’s is largely accounted for by visits to Wikipedia: this was the second most commonly visited site, used in 26% (115/44) of cases and by 70% (25/35) of all physicians. Notably, only one respondent made regular contribution to a medical wiki site.

The main motivation for using the Internet for information seeking was the accessibility and ease of use over other tools (like textbooks), the uptodateness, the broad coverage and the extras such as interactive immages. On the other hand most clinicians realized that there was a limitation in the quality or usefulness of information found. It is reassuring that most doctors used UGM like Wikipedia for background or open questions, to fulfill the need for more in depth knowledge on a subject, or to find information for patients, not for immediate solving of clinical questions.

The Int J Med Inform article has been widely covered by blogs: i.e. see Wishful Thinking in Medical Education [16], Dr Shock, MD, PhD [17], Life in the Fast Lane [18], Clinical Cases and Images Blog [19] and Scienceroll [20].

Apparently some doctors also heavily rely on Wikipedia that they refer to Wikipedia articles in publications (see the Int. J Cardiol. PubMed [3] abstract below)!!

8-9-2009 14-03-15 Int J cardiol wikipedia references 2

WHY WIKIPEDIA IS NOT (YET) A TRUSTWORTHY AND HIGH QUALITY HEALTH SITE

Whether the common use of Wikipedia by e-patient, medical students and doctors is disadvantageous depends on the quality and the trustworthiness of the Wikipedia articles, and that is in its turn dependent on who writes the articles.

Basically, the strength of Wikipedia is it weakness: anyone can write anything on any subject, and anyone can edit it, anonymously.

Negative aspects include its coverage (choice of subjects but also the depth of coverage), the “overlinking”, the sometimes frustating interactions between authors and editors, regularly leading to (often polite) “revision wars“, but above all the lack of ‘expert’ authors or peer review. This may result in incomplete, wrong or distorted information.

Positive aspects are its accessibility, currency, availability in many languages, and the collective “authorship” (which is an admirable concept).

The following humorist video shows how the wisdom of the crowds can lead to chaos, incorrect and variable information.

SCOPE AND ACCURACY (What has been covered, how deep and how good) :

Too much, too little, too ….

With respect to its coverage one study in the Journal of Computer-Mediated Communication (2008) [4] concludes:

Differences in the interests and attention of Wikipedia’s editors mean that some areas, in the traditional sciences, for example, are better covered than others. (…)
Overall, we found that the degree to which Wikipedia is lacking depends heavily on one’s perspective. Even in the least covered areas, because of its sheer size, Wikipedia does well, but since a collection that is meant to represent general knowledge is likely to be judged by the areas in which it is weakest, it is important to identify these areas and determine why they are not more fully elaborated. It cannot be a coincidence that two areas that are particularly lacking on Wikipedia—law and medicine—are also the purview of licensed experts.

It is not unexpected though that Wikipedia’s topical coverage is driven by the interests of its users.

Sometimes data are added to Wikipedia, that are in itself correct, but controversial. Recently, Wikipedia published the 10 inkblots (Scienceroll, [21]) of the Rorschach test, along with common responses for each. This had led to complaints by Psychologists , who argue that the site is jeopardizing one of the oldest continuously used psychological assessment tests (NY Times [22]).

The actual coverage of medical subjects may vary greatly. In one study [5], abstract-format, 2007) Wikipedia entries were screened for the most commonly performed inpatient surgical procedures in the U.S. Of the 39 procedures, 35 were indexed on Wikipedia. 85.7% of these articles were deemed appropriate for patients. All 35 articles presented accurate content, although only 62.9% (n=22) were free of critical omissions. Risks of the procedures were significantly underreported. There was a correlation between an entry’s quality and how often it was edited.

Wikipedia may even be less suitable for drug information questions, questions that one-third of all Internet health-seekers search for. A study in Annals of Pharmacotherapy [6] comparing the scope, completeness, and accuracy of drug information in Wikipedia to a free, online, traditionally edited database (Medscape Drug Reference [MDR]) showed that  Wikipedia answered significantly fewer drug information questions (40.0%) compared with MDR (82.5%; p < 0.001) and that Wikipedia answers were less complete. Although no factual errors were found, errors of omission were higher in Wikipedia (n = 48) than in MDR (n = 14). The authors did notice a marked improvement in Wikipedia over time. The authors conclude:

This study suggests that Wikipedia may be a useful point of engagement for consumers looking for drug information, but that it should be supplementary to, rather than the sole source of, drug information. This is due, in part, to our findings that Wikipedia has a more narrow scope, is less complete, and has more errors of omission versus the comparator database. Consumers relying on incomplete entries for drug information risk being ill-informed with respect to important safety features such as adverse drug events, contraindications, drug interactions, and use in pregnancy.
These errors of omission may prove to be a substantial and largely hidden danger associated with exclusive use of
user-edited drug information sources.

Alternatively, user-edited sites may serve as an effective means of disseminating drug information and are promising as a means of more actively involving consumers in their own care. However, health professionals should not use user-edited sites as authoritative sources in their clinical practice, nor should they recommend them to patients without knowing the limitations and providing sufficient additional information and counsel…

Not Evidence Based

German researches found [7], not surprisingly, that Wikipedia (as well as two major German statutory health insurances):

“…failed to meet relevant criteria, and key information such as the presentation of probabilities of success on patient-relevant outcomes, probabilities of unwanted effects, and unbiased risk communication was missing. On average items related to the objectives of interventions, the natural course of disease and treatment options were only rated as “partially fulfilled”. (..)  In addition, the Wikipedia information tended to achieve lower comprehensibility. In conclusion(..) Wikipedia (..) does not meet important criteria of evidence-based patient and consumer information though…”

Wrong, misleading, inaccurate

All above studies point at the incompleteness of Wikipedia. Even more serious is the fact that some of the Wikipedia addings are wrong or misleading. Sometimes on purpose. The 15 biggest wikipedia blunders [23] include the death announcements of Ted Kennedy (when he was still alive),  Robert Byrd and others. Almost hilarious are the real time Wikipedia revisions after the presumed death of Kennedy and the death of Ken Lay (suicide, murde, heart attack? [24).

In the field of medicine, several drug companies have been caught altering Wikipedia entries. The first drug company messing with Wikipedia was AstraZeneca. References claiming that Seroquel allegedly made teenagers “more likely to think about harming or killing themselves” were deleted by a user of a computer registered to the drug company [25], according to Times [26]. Employees of Abbott Laboratories have also been altering entries to Wikipedia to “eliminate information questioning the safety of its top-selling drugs.”(See WSJ-blog [27] , brandweeknrx.com [28], and recently Kevin MD[29])

These are “straightforward” examples of fraudulent material. But sometimes the Wikipedia articles are more subtly colored by positive or negative bias.

Take for instance the English entry on Evidence Based Medicine (in fact the reason why I started this post). Totally open-minded I checked the entry, which was automatically generated in one of my posts by Zemanta. First I was surprised by the definition of EBM:

Evidence-based medicine (EBM) aims to apply the best available evidence gained from the scientific method to medical decision making. It seeks to assess the quality of evidence of the risks and benefits of treatments (including lack of treatment).

instead of the usually cited Sacket-definition (this is only cited at the end of the paper):

“the practice of evidence based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research”

In short, the whole article lacks cohesion: the definitions of EBM are not correct, there is too much emphasis on not directly relevant information (4 ways to grade the evidence and 3 statistical measures), the limitations are overemphasized (cf. chapter 7 with 6 in the Figure below) and put out of perspective.

Apparently this has also been noted by Wikipedia, because there is a notice on the Evidence Based Medicine Page saying:

This article has been nominated to be checked for its neutrality. Discussion of this nomination can be found on the talk page. (May 2009)

9-9-2009 9-55-04 wikipedia EBM start smal

Much to my surprise the article had been written by Mr-Natural-Health, who’s account seems not to be in use since 2004  and who is currently active as User:John Gohde. Mr Natural Health is a member of WikiProject Alternative medicine.

Now why in earth would some advocate of CAM write the Wikipedia EBM-entry? I can think of 4 (not mutually exclusive) reasons:

  1. When you’re an EBM-nonbeliever or opponent this is THE chance to misinform readers about EBM (to the advantage of CAM).
  2. The author was invited to write this entry.
  3. No EBM-specialist or epidemiologist is willing to write the entry, or to write for Wikipedia in general (perhaps because they find Wikipedia lacks trustworthiness?)
  4. EBM specialists/epidemiologists are not “allowed”/hindered to make major amendments to the text, let alone rewrite it.

According to Mr Naturopath point 2 is THE reason he wrote this article. Now the next question is “exactly by whom was he invited?” But the TALK-page reveals that Mr Naturapath makes it a tough job for other, better qualified writers, to edit the page (point 4). To see how difficult it is for someone to re-edit a page, please see the TALK-page. In fact, one look at this page discourages me from ever trying to make some amendments to any Wikpedia text.

SOLUTIONS?

Changes to Wikipedia’s organization

Wikipedia has long grasped that its Achilles heel is the free editability (see for instance this interview with Wikipedia’s founder [30]). Therefore, “WikiProjects” was initiated to help coordinate and organize the writing and editing of articles on a certain topic, as well as “Citizendium” which is an English-language wiki-based free encyclopedia project aimed to improve the Wikipedia model by providing a “reliable” encyclopedia. “It hopes to achieve this by requiring all contributors to use their real names, by strictly moderating the project for unprofessional behavior, by providing what it calls “gentle expert oversight” of everyday contributors, and also through its “approved articles,” which have undergone a form of peer-review by credentialed topic experts and are closed to real-time editing.”

Starting this fall Wikipedia will launch an optional feature called “WikiTrust” will color code every word of the encyclopedia based on the reliability of its author and the length of time it has persisted on the page: Text from questionable sources starts out with a bright orange background, while text from trusted authors gets a lighter shade.

9-9-2009 15-25-36 wikipedia wikiproject medicine

The Wikipedia EBM article is within the scope of these two projects, and this is good news. However, Wikipedia still clings to the idea that: “Everyone is welcome to join in this endeavor (regardless of medical qualifications!).” In my opinion, it would be better if Wikipedia gave precedence to experts instead of hobbyists/ people from another field, because the former can be expected to know what they are talking about. It is quite off-putting for experts to contribute. See this shout-out:

Who are these so-called experts who will qualify material? From what I’ve seen so far, being an academic expert in a particular field hardly protects one from edit wars–Julie and 172 are two primary examples of this. Meanwhile, the only qualification I have seen so far is that they have a B.A. Gimme a friggin’ break! (and before I get accused of academic elitism, I make it known that I dropped out of college and spend an inordinate amount of time at work correcting the BS from the BAs, MAs, and PhDs).

While anyone can still edit entries, the site is testing pages that require changes to be approved by an experienced Wikipedia editor before they show up, the so called Flagged protection and patrolled revisions. (see Wikimedia) This proposal is only for articles that are currently under normal mechanisms of protection (i.e. the Obama-article cannot be edited by a newcomer).

Although this seems logic, it is questionable whether “experienced” editors are per definition better qualified than newcomers. A recent interesting analysis of the Augmented Social Cognition group [31], (cited in the Guardian [32]) shows a slowdown in growth of Wikipedia activity, with the activity slightly declining in all classes of editors except for the highest-frequency class of editors (1000+ edits). Here is an increase in their monthly edits.

In addition the study shows growing resistance from the Wikipedia community to new content. The total percentage of reverted edits increased steadily over the years, but more interestingly, low-frequency or occasional editors experienced a visibly greater resistance compared to high-frequency editors . Together this points at a growing resistance from the Wikipedia community to new content, especially when the edits come from occasional editors.

This is more or less in line with an earlier finding [9] showing that Wikipedia members feel more comfortable expressing themselves on the net than off-line and scored lower on agreeableness and openness compared to non-Wikipedians, a finding that was interpreted as consistent with the possibility that contributing to Wikipedia serves mainly egocentric motives.

Image representing Medpedia as depicted in Cru...
Image via CrunchBase

Encouraging students, doctors and scientists (provisional)

One way of improving content, is to encourage experts to write. To achieve that the information on Wikipedia is of the highest quality and up-to-date, the NIH is encouraging its scientists and science writers to edit and even initiate Wikipedia articles in their fields [36]. It joined with the Wikimedia Foundation, to host  a training session on the tools and rules of wiki culture, at NIH headquarters in Bethesda.

A less noncommital approach is the demand to “Publish in Wikipedia or perish”, as described in Nature News [9]. Anyone submitting to a section of the journal RNA Biology will, in the future, be required to also submit a Wikipedia page that summarizes the work. The journal will then peer review the page before publishing it in Wikipedia.” The project is described in detail here [10] and the wiki can be viewed here

Wiki’s for experts.

One possible solution is that scientist and medica experts contribute to wiki’s other than the Wikipedia. One such wiki is the wiki-surgery [5]. PubDrugRxWiki , WikiProteins [11] and Gene Wiki [12] are other examples. In general, scientists are more inclined to contribute to these specialists wiki’s, that have oversight and formal contributions by fellow practitioners (this is also true for the RNA-wiki)

A medical Wikipedia

Yet another solution is a medical wikipedia, such as Ganfyd or Medpedia . Ganfyd is written by medical professionals. To qualify to edit or contribute to the main content of Medpedia approved editors must have an M.D., D.O., or Ph.D. in a biomedical field. Others, however, may contribute by writing in suggestions for changes to the site using the “Make a suggestion” link at the top of each page. Suggestions are reviewed by approved editors. Whether these medical wikipedias will succeed will depend on the input of experts and their popularity: to what extent will they be consulted by people with health questions?

I would like to end with a quote from Berci during twitterview (link in Wikipedia):

@Berci : @diariomedico And as Wikipedians say, Wikipedia is the best source to start with in your research, but should never be the last one. #DM1 9 months ago

REFERENCES

ResearchBlogging.orgScientific Articles

  1. Laurent, M., & Vickers, T. (2009). Seeking Health Information Online: Does Wikipedia Matter? Journal of the American Medical Informatics Association, 16 (4), 471-479 DOI: 10.1197/jamia.M3059
  2. Hughes, B., Joshi, I., Lemonde, H., & Wareham, J. (2009). Junior physician’s use of Web 2.0 for information seeking and medical education: A qualitative study International Journal of Medical Informatics, 78 (10), 645-655 DOI: 10.1016/j.ijmedinf.2009.04.008
  3. Lee, C., Teo, C., & Low, A. (2009). Fulminant dengue myocarditis masquerading as acute myocardial infarction International Journal of Cardiology, 136 (3) DOI: 10.1016/j.ijcard.2008.05.023
  4. Halavais, A., & Lackaff, D. (2008). An Analysis of Topical Coverage of Wikipedia Journal of Computer-Mediated Communication, 13 (2), 429-440 DOI: 10.1111/j.1083-6101.2008.00403.x
  5. Devgan, L., Powe, N., Blakey, B., & Makary, M. (2007). Wiki-Surgery? Internal validity of Wikipedia as a medical and surgical reference Journal of the American College of Surgeons, 205 (3) DOI: 10.1016/j.jamcollsurg.2007.06.190
  6. Clauson, K., Polen, H., Boulos, M., & Dzenowagis, J. (2008). Scope, Completeness, and Accuracy of Drug Information in Wikipedia Annals of Pharmacotherapy, 42 (12), 1814-1821 DOI: 10.1345/aph.1L474 (free full text)
  7. Mühlhauser I, & Oser F (2008). [Does WIKIPEDIA provide evidence-based health care information? A content analysis] Zeitschrift fur Evidenz, Fortbildung und Qualitat im Gesundheitswesen, 102 (7), 441-8 PMID: 19209572
  8. Amichai–Hamburger, Y., Lamdan, N., Madiel, R., & Hayat, T. (2008). Personality Characteristics of Wikipedia Members CyberPsychology & Behavior, 11 (6), 679-681 DOI: 10.1089/cpb.2007.0225
  9. Butler, D. (2008). Publish in Wikipedia or perish Nature DOI: 10.1038/news.2008.1312
  10. Daub, J., Gardner, P., Tate, J., Ramskold, D., Manske, M., Scott, W., Weinberg, Z., Griffiths-Jones, S., & Bateman, A. (2008). The RNA WikiProject: Community annotation of RNA families RNA, 14 (12), 2462-2464 DOI: 10.1261/rna.1200508
  11. Mons, B., Ashburner, M., Chichester, C., van Mulligen, E., Weeber, M., den Dunnen, J., van Ommen, G., Musen, M., Cockerill, M., Hermjakob, H., Mons, A., Packer, A., Pacheco, R., Lewis, S., Berkeley, A., Melton, W., Barris, N., Wales, J., Meijssen, G., Moeller, E., Roes, P., Borner, K., & Bairoch, A. (2008). Calling on a million minds for community annotation in WikiProteins Genome Biology, 9 (5) DOI: 10.1186/gb-2008-9-5-r89
  12. Huss, J., Orozco, C., Goodale, J., Wu, C., Batalov, S., Vickers, T., Valafar, F., & Su, A. (2008). A Gene Wiki for Community Annotation of Gene Function PLoS Biology, 6 (7) DOI: 10.1371/journal.pbio.0060175
    Other Publications, blogposts
    (numbers in text need to be adapted)

  13. Envision Solutions, LLC. Diving Deeper Into Online Health Search – Examining Why People Trust Internet Content & The Impact Of User-Generated Media (2007) http://www.envisionsolutionsnow.com/pdf/Studies/Online_Health_Search.pdf Accessed August 2009 (CC)
  14. New data available of the the Pew Internet & American Life Project are available here)
  15. http://www.thegooglecache.com/white-hat-seo/966-of-wikipedia-pages-rank-in-googles-top-10/
  16. http://www.seoptimise.com/blog/2008/05/why-wikipedias-google-rankings-are-a-joke.html
  17. http://wishfulthinkinginmedicaleducation.blogspot.com/2009/06/where-do-first-year-medical-students.html
  18. http://www.newscientist.com/article/mg20327185.500-should-you-trust-health-advice-from-the-web.html?page=1
  19. http://wishfulthinkinginmedicaleducation.blogspot.com/2009/07/where-do-junior-doctors-look-things-up.html
  20. http://www.shockmd.com/2009/07/06/how-and-why-junior-physicians-use-web-20/
  21. http://sandnsurf.medbrains.net/2009/07/how-and-why-junior-docs-use-web-20/
  22. Wikipedia used by 70% of junior physicians, dominates search results for health queries (casesblog.blogspot.com)
  23. http://scienceroll.com/2009/07/06/junior-physicians-and-web-2-0-call-for-action/
  24. http://scienceroll.com/2009/08/03/rorschach-test-scandal-on-wikipedia-poll/
  25. http://www.nytimes.com/2009/07/29/technology/internet/29inkblot.html (Rorschach)
  26. http://www.pcworld.com/article/170874/the_15_biggest_wikipedia_blunders.html
  27. http://www.futureofthebook.org/blog/archives/2006/07/reuters_notices_wikipedia_revi.html
  28. http://en.wikipedia.org/w/index.php?diff=prev&oldid=144007397
  29. http://business.timesonline.co.uk/tol/business/industry_sectors/media/article2264150.ece
  30. http://blogs.wsj.com/health/2007/08/30/abbott-labs-in-house-wikipedia-editor/
  31. http://www.brandweeknrx.com/2007/08/abbott-caught-a.html
  32. http://www.kevinmd.com/blog/2009/08/op-ed-wikipedia-isnt-really-the-patients-friend.html
  33. http://www.businessweek.com/technology/content/dec2005/tc20051214_441708.htm?campaign_id=topStories_ssi_5
  34. http://asc-parc.blogspot.com/2009/08/part-2-more-details-of-changing-editor.html
  35. http://www.guardian.co.uk/technology/2009/aug/12/wikipedia-deletionist-inclusionist
  36. http://www.washingtonpost.com/wp-dyn/content/article/2009/07/27/AR2009072701912.html
Reblog this post [with Zemanta]