The Scatter of Medical Research and What to do About it.

18 05 2012

ResearchBlogging.orgPaul Glasziou, GP and professor in Evidence Based Medicine, co-authored a new article in the BMJ [1]. Similar to another paper [2] I discussed before [3] this paper deals with the difficulty for clinicians of staying up-to-date with the literature. But where the previous paper [2,3] highlighted the mere increase in number of research articles over time, the current paper looks at the scatter of randomized clinical trials (RCTs) and systematic reviews (SR’s) accross different journals cited in one year (2009) in PubMed.

Hofmann et al analyzed 7 specialties and 9 sub-specialties, that are considered the leading contributions to the burden of disease in high income countries.

They followed a relative straightforward method for identifying the publications. Each search string consisted of a MeSH term (controlled  term) to identify the selected disease or disorders, a publication type [pt] to identify the type of study, and the year of publication. For example, the search strategy for randomized trials in cardiology was: “heart diseases”[MeSH] AND randomized controlled trial[pt] AND 2009[dp]. (when searching “heart diseases” as a MeSH, narrower terms are also searched.) Meta-analysis[pt] was used to identify systematic reviews.

Using this approach Hofmann et al found 14 343 RCTs and 3214 SR’s published in 2009 in the field of the selected (sub)specialties. There was a clear scatter across journals, but this scatter varied considerably among specialties:

“Otolaryngology had the least scatter (363 trials across 167 journals) and neurology the most (2770 trials across 896 journals). In only three subspecialties (lung cancer, chronic obstructive pulmonary disease, hearing loss) were 10 or fewer journals needed to locate 50% of trials. The scatter was less for systematic reviews: hearing loss had the least scatter (10 reviews across nine journals) and cancer the most (670 reviews across 279 journals). For some specialties and subspecialties the papers were concentrated in specialty journals; whereas for others, few of the top 10 journals were a specialty journal for that area.
Generally, little overlap occurred between the top 10 journals publishing trials and those publishing systematic reviews. The number of journals required to find all trials or reviews was highly correlated (r=0.97) with the number of papers for each specialty/ subspecialty.”

Previous work already suggested that this scatter of research has a long tail. Half of the publications is in a minority of papers, whereas the remaining articles are scattered among many journals (see Fig below).

Click to enlarge en see legends at BMJ 2012;344:e3223 [CC]

The good news is that SRs are less scattered and that general journals appear more often in the top 10 journals publishing SRs. Indeed for 6 of the 7 specialties and 4 of the 9 subspecialties, the Cochrane Database of Systematic Reviews had published the highest number of systematic reviews, publishing between 6% and 18% of all the systematic reviews published in each area in 2009. The bad news is that even keeping up to date with SRs seems a huge, if not impossible, challenge.

In other words, it is not sufficient for clinicians to rely on personal subscriptions to a few journals in their specialty (which is common practice). Hoffmann et al suggest several solutions to help clinicians cope with the increasing volume and scatter of research publications.

  • a central library of systematic reviews (but apparently the Cochrane Library fails to fulfill such a role according to the authors, because many reviews are out of date and are perceived as less clinically relevant)
  • registry of planned and completed systematic reviews, such as prospero. (this makes it easier to locate SRs and reduces bias)
  • Synthesis of Evidence and synopses, like the ACP-Jounal Club which summarizes the best evidence in internal medicine
  • Specialised databases that collate and critically appraise randomized trials and systematic reviews, like www.pedro.org.au for physical therapy. In my personal experience, however, this database is often out of date and not comprehensive
  • Journal scanning services like EvidenceUpdates from mcmaster.ca), which scans over 120 journals, filters articles on the basis of quality, has practising clinicians rate them for relevance and newsworthiness, and makes them available as email alerts and in a searchable database. I use this service too, but besides that not all specialties are covered, the rating of evidence may not always be objective (see previous post [4])
  • The use of social media tools to alert clinicians to important new research.

Most of these solutions are (long) existing solutions that do not or only partly help to solve the information overload.

I was surprised that the authors didn’t propose the use of personalized alerts. PubMed’s My NCBI feature allows to create automatic email alerts on a topic and to subscribe to electronic tables of contents (which could include ACP journal Club). Suppose that a physician browses 10 journals roughly covering 25% of the trials. He/she does not need to read all the other journals from cover to cover to avoid missing one potentially relevant trial. Instead it is far more efficient to perform a topic search to filter relevant studies from journals that seldom publish trials on the topic of interest. One could even use the search of Hoffmann et al to achieve this.* Although in reality, most clinical researchers will have narrower fields of interest than all studies about endocrinology and neurology.

At our library we are working at creating deduplicated, easy to read, alerts that collate table of contents of certain journals with topic (and author) searches in PubMed, EMBASE and other databases. There are existing tools that do the same.

Another way to reduce the individual work (reading) load is to organize journals clubs or even better organize regular CATs (critical appraised topics). In the Netherlands, CATS are a compulsory item for residents. A few doctors do the work for many. Usually they choose topics that are clinically relevant (or for which the evidence is unclear).

The authors shortly mention that their search strategy might have missed  missed some eligible papers and included some that are not truly RCTs or SRs, because they relied on PubMed’s publication type to retrieve RCTs and SRs. For systematic reviews this may be a greater problem than recognized, for the authors have used meta-analyses[pt] to identify systematic reviews. Unfortunately PubMed has no publication type for systematic reviews, but it may be clear that there are many more systematic reviews that meta-analyses. Possibly systematical reviews might even have a different scatter pattern than meta-analyses (i.e. the latter might be preferentially included in core journals).

Furthermore not all meta-analyses and systematic reviews are reviews of RCTs (thus it is not completely fair to compare MAs with RCTs only). On the other hand it is a (not discussed) omission of this study, that only interventions are considered. Nowadays physicians have many other questions than those related to therapy, like questions about prognosis, harm and diagnosis.

I did a little imperfect search just to see whether use of other search terms than meta-analyses[pt] would have any influence on the outcome. I search for (1) meta-analyses [pt] and (2) systematic review [tiab] (title and abstract) of papers about endocrine diseases. Then I subtracted 1 from 2 (to analyse the systematic reviews not indexed as meta-analysis[pt])

Thus:

(ENDOCRINE DISEASES[MESH] AND SYSTEMATIC REVIEW[TIAB] AND 2009[DP]) NOT META-ANALYSIS[PT]

I analyzed the top 10/11 journals publishing these study types.

This little experiment suggests that:

  1. the precise scatter might differ per search: apparently the systematic review[tiab] search yielded different top 10/11 journals (for this sample) than the meta-analysis[pt] search. (partially because Cochrane systematic reviews apparently don’t mention systematic reviews in title and abstract?).
  2. the authors underestimate the numbers of Systematic Reviews: simply searching for systematic review[tiab] already found appr. 50% additional systematic reviews compared to meta-analysis[pt] alone
  3. As expected (by me at last), many of the SR’s en MA’s were NOT dealing with interventions, i.e. see the first 5 hits (out of 108 and 236 respectively).
  4. Together these findings indicate that the true information overload is far greater than shown by Hoffmann et al (not all systematic reviews are found, of all available search designs only RCTs are searched).
  5. On the other hand this indirectly shows that SRs are a better way to keep up-to-date than suggested: SRs  also summarize non-interventional research (the ratio SRs of RCTs: individual RCTs is much lower than suggested)
  6. It also means that the role of the Cochrane Systematic reviews to aggregate RCTs is underestimated by the published graphs (the MA[pt] section is diluted with non-RCT- systematic reviews, thus the proportion of the Cochrane SRs in the interventional MAs becomes larger)

Well anyway, these imperfections do not contradict the main point of this paper: that trials are scattered across hundreds of general and specialty journals and that “systematic reviews” (or meta-analyses really) do reduce the extent of scatter, but are still widely scattered and mostly in different journals to those of randomized trials.

Indeed, personal subscriptions to journals seem insufficient for keeping up to date.
Besides supplementing subscription by  methods such as journal scanning services, I would recommend the use of personalized alerts from PubMed and several prefiltered sources including an EBM search machine like TRIP (www.tripdatabase.com/).

*but I would broaden it to find all aggregate evidence, including ACP, Clinical Evidence, syntheses and synopses, not only meta-analyses.

**I do appreciate that one of the co-authors is a medical librarian: Sarah Thorning.

References

  1. Hoffmann, Tammy, Erueti, Chrissy, Thorning, Sarah, & Glasziou, Paul (2012). The scatter of research: cross sectional comparison of randomised trials and systematic reviews across specialties BMJ, 344 : 10.1136/bmj.e3223
  2. Bastian, H., Glasziou, P., & Chalmers, I. (2010). Seventy-Five Trials and Eleven Systematic Reviews a Day: How Will We Ever Keep Up? PLoS Medicine, 7 (9) DOI: 10.1371/journal.pmed.1000326
  3. How will we ever keep up with 75 trials and 11 systematic reviews a day (laikaspoetnik.wordpress.com)
  4. Experience versus Evidence [1]. Opioid Therapy for Rheumatoid Arthritis Pain. (laikaspoetnik.wordpress.com)
Advertisements




“Pharmacological Action” in PubMed has no True Equivalent in OVID MEDLINE

11 01 2012

Searching for EMBASE Subject Headings (the EMBASE index terms) for drugs is relatively straight forward in EMBASE.

When you want to search for aromatase inhibitors you first search for the Subject Heading mapping to aromatase inhibitors (aromatase inhibitor). Next you explode aromatase inhibitor/ if you are interested in all its narrower terms. If not, you search both for the general term aromatase inhibitor and those specific narrower terms you want to include.
Exploding aromatase inhibitor (exp aromatase inhibitor/) yields 15938 results. That is approximately twice what you get by searching aromatase inhibitor/ alone (not exploded). This yields 7434 hits.

It is different in MEDLINE. If you search for aromatase inhibitors in the MeSH database you get two suggestions.

The first index term “Aromatase Inhibitors” is a Mesh. It has no narrower terms.
Drug-Mesh are generally not arranged by working mechanism, but by chemical structure/type of compound. That is often confusing. Spironolactone for instance belongs to the MeSH Lactones (and Pregnenes) not to the MeSH Aldosterone Antagonists or Androgen Antagonist. Most Clinicians want to search for a group of compounds with the same mechanism of action, not the same biochemical family

The second term “Aromatase Inhibitors” [Pharmacological Action]  however does stand for the working mechanism. It does have narrower terms, including 2 MeSH terms (highlighted) and various substance names, also called Supplementary Concepts. 

For complete results you have to search for both MeSH and Pharmacological action: “Aromatase Inhibitors”[Mesh] yields 3930 records, whereas (“Aromatase Inhibitors”[Mesh]) OR “Aromatase Inhibitors” [Pharmacological Action] yields 6045. That is a lot more.

I usually don’t search PubMed, but OVID MEDLINE.

I know that Pharmacological Action-subheadings are important, so I tried to find the equivalent in OVID .

I found the MeSH Aromatase Inhibitors, but -unlike PubMed- OVID showed only two narrower Drug Terms (called Non-MeSH here versus MeSH in PubMed).

I found that odd.

I reasoned “Pharmacological action” might perhaps be combined with the MESH in OVID MEDLINE. This was later confirmed by Melissa Rethlefsen (see Twitter discussion below)

In Ovid MEDLINE I got 3937 hits with Aromatase Inhibitors/ and 5219 with exp Aromatase Inhibitors/ (thus including aminogluthemide or Fadrozole)

At this point I checked PubMed (shown above). Here I found  that “Aromatase Inhibitors”[Mesh] OR “Aromatase Inhibitors” [Pharmacological Action] yielded 6045 hits in PubMed, against 5219 in OVID MEDLINE for exp Aromatase Inhibitors/

The specific aromatase inhibitors Aminogluthemide/and Fadrozole/ [set 60] accounted fully for the difference  between exploded [set 59] and non-exploded Aromatase Inhibitors[set 58].

But what explained the gap of approximately 800 records between “Aromatase Inhibitors”[Mesh] OR “Aromatase Inhibitors”[Pharmacological Action]* in PubMed and exp aromatase inhibitors/ in OVID MEDLINE?

Could it be the substance names, mentioned under “Aromatase Inhibitors”[Pharmacological Action], I wondered?

Thus I added all the individual substance names in OVID MEDLINE (code= .rn.). See search set 61 below.

Indeed these accounted fully for the difference (set 62= 59 or 61 : the total number of hits in PubMed is similar)

It obviously is a mistake of OVID MEDLINE and I will inform them.

For the meanwhile, take care to add the individual substance names when you search for drug terms that have a pharmacological action-equivalent in PubMed. The substance names are not automatically searched when exploding the MeSH-term in OVID MEDLINE.

——–

For more info on Pharmacological action, see: http://www.nlm.nih.gov/bsd/disted/mesh/paterms.html

Twitter Discussion between me and Melissa Rethlefsen about the discrepancy between PubMed and OVID MEDLINE (again showing how helpful Twitter can be for immediate discussions and exchange of thoughts)

[read from bottom to top]





Search OVID EMBASE and Get MEDLINE for Free…. without knowing it

19 10 2010

I have the impression that OVIDSP listens more to librarians than the NLM, who considers the end users of databases like PubMed more important, mainly because there are more of them. On the other hand NLM communicates PubMed’s changes better (NLM Technical Bulletin) and has easier to find tutorials & FAQs, namely at the PubMed homepage.

I gather that the new changes to the OVIDSP interface are the reason why two older OVID posts are the recent number 2 and 3 hits on my blog. My guess is that people are looking for some specific information on OVID’s interface changes that they can’t easily access otherwise.

But this post won’t address the technical changes. I will write about this later.

I just want to mention a few changes to the OVIDSP databases MEDLINE and EMBASE, some of them temporary, that could have been easily missed.

[1] First, somewhere in August, OVID MEDLINE contained only indexed PubMed articles. I know that OVID MEDLINE misses some papers PubMed already has -namely the “as supplied by publisher” subset-, but this time the difference was dramatic: “in data review” and “in process” papers weren’t found as well. I almost panicked, because if I missed that much in OVID MEDLINE, I would have to search PubMed as well, and adapt the search strategy…. and, since I already lost hours because of OVID’s extreme slowness at that time, I wasn’t looking forward to this.

According to an OVID-representative this change was not new, but was already there since (many) months. Had I been blind? I checked the printed search results of a search I performed in June. It was clear that the newer update found less records, meaning that some records were missed in the current (August) update. Furthermore the old Reference Manager database contained non-indexed records. So no problems then.

But to make a long story short. Don’t worry: this change disappeared as quickly as it came.
I would have doubted my own eyes, if my colleague hadn’t seen it too.

If you have done a MEDLINE OVID search in the second half of August you might like to check the results.

[2] Simultaneously there was another change. A change that is still there.

Did you know that OVID EMBASE contains MEDLINE records as well? I knew that you could search EMBASE.com for MEDLINE and EMBASE records using the “highly praised EMTREE“, but not that OVID EMBASE recently added these records too.

They are automatic found by the text-word searches and by the EMTREE already includes all of MeSH.

Should I be happy that I get these records for free?

No, I am not.

I always start with a MEDLINE search, which is optimized for MEDLINE (with regard to the MeSH).

Since indexing by  EMTREE is deep, I usually have (much) more noise (irrelevant hits) in EMBASE.

I do not want to have an extra number of MEDLINE-records in an uncontrolled way.

I can imagine though, that it would be worthwhile in case of a quick search in EMBASE alone: that could save time.
In my case, doing extensive searches for systematic reviews I want to be in control. I also want to show the number of articles from MEDLINE and the number of extra hits from EMBASE.

(Later I realized that a figure shown by the OVID representative wasn’t fair: they showed the hits obtained when searching EMBASE, MEDLINE and other databases in Venn diagrams: MEDLINE offered little extra beyond EMBASE, which is self-evident, considering that EMBASE includes almost all MEDLINE records.- But I only learned this later.)

It is no problem if you want to include these MEDLINE records, but it is easy to exclude them.

You can limit for MEDLINE or EMBASE records.

Suppose your last search set is 26.

Click Limits > Additional Limits > EMBASE (or MEDLINE)

Alternatively type: limit 26 to embase (resp limit 26 to medline) Added together they make 100%

If only they would have told us….


3. EMBASE OVID now also adds conference abstracts.

A good thing if you do an exhaustive search and want to include unpublished material as well (50% of the conference abstracts don’t get published).

You can still exclude them if you like  (see publication types to the right)

Here is what is written at EMBASE.com

Embase now contains almost 800 conferences and more than 260,000 conference abstracts, primarily from journals and journal supplements published in 2009 and 2010. Currently, conference abstracts are being added to Embase at the rate of 1,000 records per working day, each indexed with Emtree.
Conference information is not available from PubMed, and is significantly greater than BIOSIS conference coverage. (…)

4. And did you know that OVID has eliminated StopWords from MEDLINE and EMBASE? Since  a few years you can now search for words or phrases like is there hope.tw. Which is a very good thing, because it broadens the possibility to search for certain word strings. However, it isn’t generally known.

OVID changed it after complaints by many, including me and a few Cochrane colleagues. I thought I had written a post on it before, but I apparently I haven’t ;).

Credits

Thanks to Joost Daams who always has the latest news on OVID.

Related Articles





Problems with Disappearing Set Numbers in PubMed’s Clinical Queries

18 10 2010

In some upcoming posts I will address various problems related to the changing interfaces of bibliographic databases.

We, librarians and end users, are overwhelmed by a flood of so-called upgrades, which often fail to bring the improvements that were promised….. or which go hand-in-hand with temporary glitches.

Christina of Christina’s Lis Rant even made rundown of the new interfaces of last summer. Although she didn’t include OVID MEDLINE/EMBASE, the Cochrane Library and Reference manager in her list, the total number of changed interfaces reached 22 !

As a matter of fact, the Cochrane Library was suffering some outages yesterday, to repair some bugs. So I will postpone my coverage of the Cochrane bugs a little.

And OVID send out a notice last week: This week Ovid will be deploying a software release of the OvidSPplatform that will add new functionality and address improvements to some existing functionality.”

In this post I will confine myself to the PubMed Clinical Queries. According to Christina PubMed changes “were a bit ago”, but PubMed continuously tweaks  its interface, often without paying much attention to its effects.

Back in July, I already covered that the redesign of the PubMed Clinical Queries was no improvement for people who wanted to do more than a quick and dirty search.

It was no longer possible to enter a set number in the Clinical Queries search bar. Thus it wasn’t possible to set up a search in PubMed first and to then enter the final set number in the Clinical Queries. This bug was repaired promptly.

From then on, the set number could be entered again in the clinical queries.

However, one bug was replaced by another: next, search numbers were disappearing from the search history.

I will use the example I used before: I want to know if spironolactone reduces hirsutism in women with PCOS, and if it works better than cyproterone acetate.

Since little is published about this topic,  I only search for  hirsutism and spironolactone. These terms  map correctly with  MeSH terms. In the MeSH database I also see (under “see also”) that spironolactone belongs to the aldosterone antagonists, so I broaden spironolactone (#2) with “Aldosterone antagonists”[pharmacological Action] using “OR” (set #7). My last set (#8) consists of #1 (hirsutism) AND #7 (#2 OR #6)

Next I go to the Clinical Queries in the Advanced Search and enter #8. (now possible again).

I change the Therapy Filter from “broad”  to “narrow”, because the broad filter gives too much noise.

In the clinical queries you see only the first five results.

Apparently even the clinical queries are now designed to just take a quick look at the most recent results, but of course, that is NOT what we are trying to achieve when we search for (the best) evidence.

To see all results for the narrow therapy filter I have to go back to the Clinical Queries again and click on see all (27) [5]

A bit of a long way about. But it gets longer…


The 27 hits, that result from combining the Narrow therapy filter with my search #8 appears. This is set #9.
Note it is a lower number than set #11 (search + systematic review filter).

Meanwhile set #9 has disappeared from my history.

This is a nuisance if I want to use this set further or if I want to give an overview of my search, i.e. for a presentation.

There are several tricks by which this flaw can be overcome. But they are all cumbersome.

1. Just add set number (#11 in this case, which is the last search (#8) + 3 more) to the search history (you have to remember the search set number though).

This is the set number remembered by the system. As you see in the history, you “miss” certain sets. #3 to #5 are for instance are searches you performed in the MeSH-database, which show up in the History of the MeSH database, but not in PubMed’s history.

The Clinical query set number is still there, but it doesn’t show either. Apparently the 3 clinical query-subsets yield a separate set number, whether the search is truly performed or not. In this case  #11 for (#8) AND systematic[sb], #9 for (#8) AND (Therapy/Narrow[filter]). And #10 for (#8) AND the medical genetics filter.

In this way you have all results in your history. It isn’t immediately clear, however, what these sets represent.

2. Use the commands rather than going to the clinical queries.

Thus type in the search bar: #8 AND systematic[sb]

And then: #8 AND (Therapy/Narrow[filter])

It is easiest to keep all filters in Word/Notepad and copy/paste each time you need the filter

3. Add clinical queries as filters to your personal NCBI account so that the filters show up each time you do a search in PubMed. This post describes how to do it.

Anyway these remain just tricks to try to make something right that is wrong.

Furthermore it makes it more difficult to explain the usefulness of the clinical queries to doctors and medical students. Explaining option 3 takes too long in a short course, option 1 seems illogical and 2 is hard to remember.

Thus we want to keep the set numbers in the history, at least.

A while ago Dieuwke Brand notified the NLM of this problem.

Only recently she received an answer saying that:

we are aware of the continuing problem.  The problem remains on our programmers’ list of items to investigate.  Unfortunately, because this problem appears to be limited to very few users, it has been listed as a low priority.

Only after a second Dutch medical librarian confirmed the problem to the NLM, saying it not only affects one or two librarians, but all the students we teach (~1000-2000 students/university/yearly), they realized that it was a more widespread problem than Dieuwke Brand’s personal problem. Now the problem has a higher priority.

Where is the time that a problem was taken for what it was? As another librarian sighed: Apparently something is only a problem if many people complain about it.

Now I know this (I regarded Dieuwke as a delegate of all Dutch Clinical Librarians), I realize that I have to “complain” myself, each time I and/or my colleagues encounter a problem.

Related Articles





Medical Information Matters 2.8 is up!

15 10 2010

The new edition of Medical Information Matters (formerly Medlibs round) is up at Danielhooker.com.

The main theme is “Programs in libraries or medical education”.
Besides two posts from this blog (A Filter for Finding Animal Studies in PubMed” and more on the topic: An Educator by Chance) the following topics are included: a new MeSH (inclusion under mild librarian pressure), PubMed in your pocket, embedding Google Gadgets in normal webpages and experiences with introducing social bookmarking to medical students.
If you find this description to cryptic (and I bet you do), then I invite you to read the entire post here. I found it a very pleasant read.

Since we are already midway October, I would like to invite you to start submitting here (blog carnival submission form).

Our next host is Dean Giustini of the The Search Principle blog. The deadline is in about 3 weeks ( November 6th).

Related Articles





A Filter for Finding “All Studies on Animal Experimentation in PubMed”

29 09 2010

ResearchBlogging.orgFor  an introduction to search filters you can first read this post.

Most people searching PubMed try to get rid of publications about animals. But basic scientists and lab animal technicians just want to find those animal studies.

PubMed has built-in filters for that: the limits. There is a limit  for “humans” and a limit for “animals”. But that is not good enough to find each and every article about humans, respectively animals. The limits are MeSH, Medical Subject Headings or index-terms and these are per definition not added to new articles, that haven’t been indexed yet. To name the main disadvantage…
Thus to find all papers one should at least search for other relevant MeSH and textwords (words in title and abstract) too.

A recent paper published in Laboratory Animals describes a filter for finding “all studies on animal experimentation in PubMed“, to facilitate “writing a systematic review (SR) of animal research” .

As the authors rightly emphasize, SR’s are no common practice in the field of animal research. Knowing what already has been done can prevent unnecessary duplication of animal experiments and thus unnecessary animal use. The authors have interesting ideas like registration of animal studies (similar to clinical trials registers).

In this article they describe the design of an animal filter for PubMed. The authors describe their filter as follows:

“By using our effective search filter in PubMed, all available literature concerning a specific topic can be found and read, which will help in making better evidencebased decisions and result in optimal experimental conditions for both science and animal welfare.”

Is this conclusion justified?

Design of the filter

Their filter is subjectively derived: the terms are “logically” chosen.

[1] The first part of the animal filter consists of only MeSH-terms.

You can’t use animals[mh] (mh=Mesh) as a search term, because MeSH are automatically exploded in PubMed. This means that narrower terms (lower in the tree) are also searched. If “Animals” were allowed to explode, the search would include the MeSH, “Humans”, which is at the end of one tree (primates etc, see Fig. below)

Therefore the MeSH-parts of their search consists of:

  1. animals [mh:noexp]: only articles are found that are indexed with “animals”, but not its narrower terms. Notably, this is identical to the PubMed Limit: animals).
  2. Exploded Animal-specific MeSH-terms not having humans as a narrow term, i.e. “fishes”[MeSH Terms].
  3. and non-exploded MeSH in those cases that humans occurred in the same branch. Like “primates”[MeSH Terms:noexp]
  4. In addition two other MeSH are used: “animal experimentation”[MeSH Terms] and “models, animal”[MeSH Terms]

[2] The second part of the search filter consist of terms in the title and abstract (command: [tiab]).

The terms are taken from relevant MeSH, two reports about animal experimentation in the Netherlands and in Europe, and the experience of the authors, who are experts in the field.

The authors use this string for non-indexed records (command: NOT medline[sb]). Thus this part is only meant to find records that haven’t (yet) been indexed, but in which (specific) animals are mentioned by the author in title or text. Synonyms and spelling variants have been taken into account.

Apparently the authors have chosen NOT to search for text words in indexed records only. Presumably it gives too much noise, to search for animals mentioned in non-indexed articles. However, the authors do not discuss why this was necessary.

This search string is extremely long. Partly because truncation isn’t used with the longer words: i.e. nematod*[tiab] instead of nematoda[Tiab] OR nematode[Tiab] OR nematoda[Tiab] OR nematode[Tiab] OR nematodes[Tiab]. Partly because they aim for completeness. However the usefulness of the terms as such hasn’t been verified (see below).

Search strategies can be freely accessed here.

Validation

The filter is mainly validated against the PubMed Limit “Animals”.

The authors assume that the PubMed Limits are “the most easily available and most obvious method”. However I know few librarians or authors of systematic reviews who would solely apply this so called ‘regular method’. In the past I have used exactly the same MeSH-terms (1) and the main text words (2) as included in their filter.

Considering that the filter includes the PubMed limit “Animals” [1.1] it does not come as a surprise that the sensitivity of the filter exceeds that of the PubMed limit Animals…

Still, the sensitivity (106%) is not really dramatic: 6% more records are found, the PubMed Limit “animals” is set as 100%.

Apparently records are very well indexed with the MeSH “animals”. Few true animal records are missed, because “animals” is a check tag. A check tag is a MeSH that is looked for routinely by indexers in every journal article. It is added to the record even if it isn’t the main (or major) point of an article.

Is an increased sensitivity of appr. 6% sufficient to conclude that this filter “performs much better than the current alternative in PubMed”?

No. It is not only important that MORE is found but to what degree the extra hits are relevant. Surprisingly, the authors ONLY determined SENSITIVITY, not specificity or precision.

There are many irrelevant hits, partly caused by the inclusion of animal population groups[mesh], which has some narrower terms that often not used for experimentation, i.e. endangered species.

Thus even after omission of animal population groups[mesh], the filter still gives hits like:

These are evidently NOT laboratory animal experiments and mainly caused by the inclusion invertebrates  like plankton.

Most other MeSH are not extremely useful either. Even terms as animal experimentation[mh] and models, animal[mh] are seldom assigned to experimental studies lacking animals as a MeSH.

According to the authors, the MeSH “Animals” will not retrieve studies solely indexed with the MeSH term Mice. However, the first records missed with mice[mesh] NOT animals[mh:noexp] are from 1965, when they apparently didn’t use “animals” as a check tag in addition to specific ‘animal’ MeSH.

Thus presumably the MeSH-filter can be much shorter and need only contain animal MeSH (rats[mh], mice[mh] etc) when publications older than 1965 are also required.

The types of vertebrate animals used in lab re...

Image via Wikipedia

Their text word string (2) is also extremely long.  Apart from the lack of truncation, most animal terms are not relevant for most searches. 2/3 of the experiments are done with rodents (see Fig). The other animals are often used for specific experiments (zebra-fish, Drosophila) or in another context, not related to animal experiments, such as:

swine flu, avian flu, milk production by cows, or allergy by milk-products or mites, stings by insects and bites by dogs and of course fish, birds, cattle and poultry as food, fetal calf serum in culture medium, but also vaccination with “mouse products” in humans. Thus most of the terms produce noise for most topics. An example below (found by birds[mesh] 🙂

On the other hand strains of mice and rats are missing from the search string: i.e. balb/c, wistar.

Extremely long search strings (1 page) are also annoying to use. However, the main issue is whether the extra noise matters. Because the filter is meant to find all experimental animal studies.

As Carlijn Hooijmans notices correctly, the filters are never used on their own, only in combination with topic search terms.

Hooijmans et al have therefore “validated” their filter with two searches. “Validated” between quotation marks because they have only compared the number of hits, thus the increase in sensitivity.

Their first topic is the use of probiotics in experimental pancreatitis (see appendix).

Their filter (combined with the topic search) retrieved 37 items against 33 items with the so called “regular method”: an increase in sensitivity of 21,1%.

After updating the search I got  38 vs 35 hits. Two of the 3 extra hits obtained with the broad filter are relevant and are missed with the PubMed limit for animals, because the records haven’t been indexed. They could also have been found with the text words pig*[tiab] or dog*[tiab]. Thus the filter is ok for this purpose, but unnecessary long. The MeSH-part of the filter had NO added value compared to animals[mh:noexp].

Since there are only 148 hits without the use of any filters, researchers could also use screen all hits. Alternatively there is a trick to safely exclude human studies:

NOT (humans[mh] NOT animals[mh:noexp])

With this double negation you exclude PubMed records that are indexed with humans[mh], as long as these records aren’t indexed with animals[mh:noexp] too. It is far “safer” than limiting for “animals”[mesh:noexp] only. We use a similar approach to ” exclude”  animals when we search for human studies.

This extremely simple filter yields 48 hits, finding all hits found with the large animal filter (plus 10 irrelevant hits).

Such a simple filter can easily be used for searches with relatively few hits, but gives too many irrelevant hits in case of  a high yield.

The second topic is food restriction. 9280 Records were obtained with the Limit: “Animals”, whereas this strategy combined with the complete filter retrieved 9650 items. The sensitivity in this search strategy was therefore 104%. 4% extra hits were obtained.

The MeSH-search added little to the search. Only 21 extra hits. The relevant hits were (again) only from before 1965.

The text-word part of the search finds relevant new articles, although there are quite some  irrelevant findings too, i.e. dieting and obtaining proteins from chicken.

4% isn’t a lot extra, but the aim of the researchers is too find all there is.

However, it is the question whether researchers want to find every single experiment or observation done in the animal kingdom. If I were to plan an experiment on whether food restriction lowers the risk for prostate cancer in a transgenic mice, need I know what the effects are of food restriction on Drosophila, nematodes, salmon or even chicken on whatever outcome? Would I like to screen 10,000 hits?

Probably most researchers would like separate filters for rodents and other laboratory animals (primates, dogs) and for work on Drosophila or fish. In some fields there might also be a need to filter clinical trials and reviews out.

Furthermore, it is not only important to have a good filter but also a good search.

The topic searches in the current paper are not ideal: they contain overlapping terms (food restriction is also found by food and restriction) and misses important MeSH (Food deprivation, fasting and the broader term of caloric restriction “energy intake” are assigned more often to records about food deprivation than caloric restriction).

Their search:

(“food restriction”[tiab] OR (“food”[tiab] AND “restriction”[tiab]) OR “feed restriction”[tiab] OR (“feed”[tiab] AND “restriction”[tiab]) OR “restricted feeding”[tiab] OR (“feeding”[tiab] AND “restricted”[tiab]) OR “energy restriction”[tiab] OR (“energy”[tiab] AND “restriction”[tiab]) OR “dietary restriction”[tiab] OR (dietary”[tiab] AND “restriction”[tiab]) OR “caloric restriction”[MeSH Terms] OR (“caloric”[tiab] AND “restriction”[tiab]) OR “caloric restriction”[tiab])
might for instance be changed to:

Energy Intake[mh] OR Food deprivation[mh] OR Fasting[mh] OR food restrict*[tiab] OR feed restrict*[tiab] OR restricted feed*[tiab] OR energy restrict*[tiab] OR dietary restrict*[tiab] OR  caloric restrict*[tiab] OR calorie restrict*[tiab] OR diet restrict*[tiab]

You do not expect such incomplete strategies from people who repeatedly stress that: most scientists do not know how to use PubMed effectively” and that “many researchers do not use ‘Medical Subject Headings’ (MeSH terms), even though they work with PubMed every day”…..

Combining this modified search with their animal filter yields 21920 hits instead of 10335 as found with their “food deprivation” search and their animal filter. A sensitivity of 212%!!! Now we are talking! 😉 (And yes there are many new relevant hits found)

Summary

The paper describes the performance of a subjective search filter to find all experimental studies performed with laboratory animals. The authors have merely “validated”  this filter against the Pubmed Limits: animals. In addition, they only determined sensitivity:  on average 7% more hits were obtained with the new animal filter than with the PubMed limit alone.

The authors have not determined the specificity or precision of the filter, not even for the 2 topics where they have applied the filter. A quick look at the results shows that the MeSH-terms other than the PubMed limit “animals” contributed little to the enhanced sensitivity. The text word part of the filter yields more relevant hits. Still -depending on the topic- there are many irrelevant records found, because  it is difficult to separate animals as food, allergens etc from laboratory animals used in experiments and the filter is developed to find every single animal in the animal kingdom, including poultry, fish, nematodes, flies, endangered species and plankton. Another (hardly to avoid) “contamination” comes from in vitro experiments with animal cells, animal products used in clinical trials and narrative reviews.

In practice, only parts of the search filter seem useful for most systematic reviews, and especially if these reviews are not meant to give an overview of all findings in the universe, but are needed to check if a similar experiment hasn’t already be done. It seems impractical if researchers have to make a systematic review, checking, summarizing and appraising  10,000 records each time they start a new experiment.

Perhaps I’m somewhat too critical, but the cheering and triumphant tone of the paper in combination with a too simple design and without proper testing of the filter asked for a critical response.

Credits

Thanks to Gerben ter Riet for alerting me to the paper. He also gave the tip that the paper can be obtained here for free.

References

  1. Hooijmans CR, Tillema A, Leenaars M, & Ritskes-Hoitinga M (2010). Enhancing search efficiency by means of a search filter for finding all studies on animal experimentation in PubMed. Laboratory animals, 44 (3), 170-5 PMID: 20551243

———————-





MEDLIB’s ROUND 1.6

27 09 2009

shht-librarian-costume1Welcome to the sixth edition of MedLib’s Round, a blog carnival of “excellent blog posts in the field of medical librarianship”.

First I have to apologize for the postponement in publication. There were so few submissions (5, including one on this blog), that I needed more time to find some material myself. Time that I didn’t have at that moment.

After a flying start with many volunteering hosts and submissions the enthusiasm for the Medlib’s Round seems to have faded somewhat. There are far less submissions. Luckily there is a core of  enthusiastic people regularly submitting to the Medlib’s Round and I’m very grateful for that. However, there are many more bloggers out there, who also write very useful MedLib stuff. Why aren’t they contributing? Are they not aware of the round, do they lack time, don’t they like blog carnivals? Should the rounds be better promoted or differently organized? I know that postponement does the round no good, but it is a bit the chicken-and-egg problem. Anyway, I would like to hear your thoughts on this.

But lets start…..

Social Media

A previous host and regular contributor to the round, Nikki Dettmar (@eagledagw) of the Eagle Dawg Blog makes a good point in  “Social Media & Emergency Preparedness: Can Your Family Text?”: “Does your family know to text when there is an emergency? Traditional phone lines may be down and traditional methods of communication may not be working.” Learn about an upcoming drill conducted by a national safety foundation and the Federal Emergency Management Agency (FEMA) over the next few months to use texting and social media channels for emergency communication. And don’t forget to instruct your mother. By the way, the use of Twitter is included in the advise.

Another regular contributor to the Medlib’s round is Ves Dimov (@DrVes). Dr. Dimov is an Allergy and Immunology Fellow at Creighton University and the author of the Clinical Cases and Images – Blog. Blogging for several years and with more than 7000 RSS readers we can trust him for some good advice on blogging In What makes a blogger go on in a field where so many others stop, fail and disappear?” Dr Ves shortly gives 4 reasons and several tips from his own experience.

Google Health

Alisha

Speaking about blogging, it is only a half year ago that Alisha Miles (@alisha764) started with her blog Alisha 764 saying: “I am no longer a mushroom, I am now a tree.” Which refers to @sandnsurf‘s post: Is Twitter the essential blogging nutrient and his comment on my blog: “the most important thing is that you are actually a tree in this ecosystem, you are out there experimenting, thinking and trying to drive the revolution further…Most of my colleagues are still mushrooms….
Alisha, who is a contributor to this round from the start, has definitely developed into a full blossoming tree, a top librarian blogger and tweeter,  She is featured, for instance, in Novoseek’s top 10 medical librarian list (as all current librarian submitters with a public blog).
Her submitted post is a classical post already. It is quite long (hear, hear who is saying) but offers good information. In “Google Health® Information: Surprising Facts” she describes the pros and cons of Google Health®, concluding:

“It is a good product; however, it should be used with caution. Remember Google Health® is not bound by HIPPA, resources should always be double or triple checked, the Google® Health Drug Interaction program is missing some key interactions, and the Google Health® Topics are missing the reference section, reviewer information, and date stamp.

Again, I applaud Google® for its efforts and for including links to MedlinePlus® as a trusted resource. As with any information source, even MedlinePlus®, all information should be checked against at least 1 other source.”

With regard to MedlinePlus and Google, Rachel Walden wrote a post: “Where is MedlinePlus in Google Drug Search Results?” where she notices that Google searches for drug information no longer seem to return results from MedlinePlus and FDA pages.

PubMed, MeSH and the like

Rachel

Rachel Walden (@rachel_w on Twitter) is the woman behind the successful blog Women’s Health News and writer for Our bodies ourselves. She not only knows a lot about women’s health and medical information, but she is always ready to reach a helping hand or join a discussion on Twitter, which is actually a quality of all MEDLIB round contributors.  In “Improving the Findability of Evidence & Literature on DoulasRachel describes  the lack of a specific MeSH for “Doula” in PubMed. A doula is an assistant who provides various forms of non-medical and non-midwifery support (physical and emotional) in the childbirth process. MeSH (or Medical Subject Headings) are controlled terms in MEDLINE, or as explained by Rachel:

MeSH are “right” terms to use to conduct a literature search in PubMed, it can really help to start with the MeSH term database, because you know those are the official subject terms being assigned to the articles. MeSH is a hierarchy, and it can help you focus a search, or expand it when needed, by moving up and down the list of subject words. It’s a nice tool to have, when it works.

As highlighted by Rachel, this gap in the MeSH makes searching less efficient and less precise: for instance, nursing and midwivery are too broad terms. But instead of whining, Rachel decided to do something about it. Via this form she send the National Library of Medicine a request to add the “doula” concept to the MeSH terms. I would recommend others to do the same when terms they search for are not (appropriately) covered by the MESH.

Librarian Mark Rabnett agrees hartfully with Rachel as he has encountered exactly the problems and yes, “there is no question that this is a satisfactorily distinct and widely accepted term, and its entry into the MeSH pantheon is long overdue.”
On his blog Gossypobima Mark had earlier posted the “Top 5 results to improve PubMedfrom the brainstorming suggestions during the Canadian Health Libraries Association conference. These include “Adding adjacency and real string searching” (YES!) and “Improval of the MeSH database”. His group found “The MeSH database stiff and laboured , and the visual display of the thesaurus and subheadings not intuitive, the ‘Add To’ feature for inserting MeSH terms to a search box kludgy, and the searching for MeSH headings difficult and unpredictable. [..] So he concludes with: “We need a MeSH mashup.”

Wouldn’t that be wonderful indeed? Rather than the current “enhancements”, why not introduce some web 2.0 tools in PubMed? As Patricia Anderson tweeted a long time ago:

“It would be so cool to do a # search, then display word cloud of top major MESH terms in results.”

Yes I would like a visual MeSH, but even better, one that would show up in the sidebar and that you would be able to “walk up and down (and sideways) and with “drag and drop to your search possibilities”. That would be cool. My imagination runs away with me when I think of it.

Grey Literature

cappadocia1_bigger shamshaNot having a public blog @shamsha has contributed to this round by writing a guest post on this blog. This interesting post is about grey literature: what is grey literature, why do you need it and why not have guidelines for searching grey literature? She gives many tips and a wealth of references, including links to her own delicious page and a wonderful resource from the Canadian Agency for Drugs and Technologies in Health.

**********************************************************************************

This concludes the official part of this MEDLIB’s September round.

The next round is hosted by Alisha Miles on her blog Alisha 764.
Officially the deadline is next Saturday
. (But it may be postponed a little. If so I will post the new deadline here)
Anyway, Alisha is looking forward to your posts. So send them in as soon as possible HERE at the Blog Carnival form.
(registration required; see the medlibs-archive for more information.
)

And some good news about the round: We already have hosts for November and December, namely Walter Jessen of Highlight Health and Valentin Vivier of at the Novoseek Blog.

Would you like to host the Medlibs round in 2010? It is never to early! Please dm me at twitter, comment on this post or write an email to laika.spoetnik@gmail.com.

**********************************************************************************

Bonus

Here are some other posts I also found worth while to read.
(I didn’t include too recent ones, so they can be included in the next round)

Dr Mike Cadogan (@Sandnsurf) writes  frequently about medical information on his blog Life in the fast Lane (his blog has moved to http://lifeinthefastlane.com, so check out old links that you may have). One of the co-authors of the blog,  Chris Nickson (precordialthump) gives emergency physicians advise how to deal withinformation-overload”. Needless to say the tips are useful to all people dealing with medical information-overload.

Dr Shock also writes a lot about medical information and web 2.0 tools. Here a video he posted about iPhone and iPod Touch as a Medical Tool.

Another good source for info about i-phones, palms can be found on http://palmdoc.net/. Medical librarians frequently writing about this subject include the Krafty Librarian and David Rothman.

I don’t have a palm or sophisticated phone, nor does our library supports its use, so I choose some other posts from these excellent bloggers.

From the KraftyLibrarian Michelle: Rapid Research about Rapid Research Notes , a new resource developed by the National Center for Biotechnology Information (NCBI) to quickly disseminate the research results to the public in an open access archive. Michelle wonders why only PLOS-articles are included and not other quality information from for instance EBSCO and Cochrane.

From palmdoc : Evernote as your peripheral brain (Evernote is a note taking application)

Rapid Research Notes is also covered by Alison of Dragonfly, a previous host of the round. She also mentions the fact that Medlineplus is now on Twitter.

David Rothman ‘s paternity leave seems over since he posts several interesting posts per week on his blog Davidrothman.net. Typically he shortly refers to a new tool or a post he encountered, like:

Dean Giustini of the The Search Principle blog published part one of a Top Fifty Twitter Users List in Medicine and has written a post on Using Twitter to manage information.

Patricia Anderson of Emerging Technologies Librarian is been very active lately with posts on social media, like “Conversation and Context in Social Media (Cautionary Tales)“, with four scenarios, including the Clinical Reader fiasco. And as always she has a lot of tips on web 2.0 tools. There is for instance a post on Listening Tools to track what your community is saying about you or to you and about Social Media Metrics

Another techy librarian working at the National University of Singapore is Aaron Tay. Aaron Tay (@aarontay) is not working in the field of medicine, but his web 2.0 tips are useful for anyone, and his blog Musings about Librarianship is certainly a must for libraries that want to use web  2.0 tools to the benefit of their users. Personally, I found the tips onViewing research alerts – full text within Google reader very useful.

Phil Bradley highlights Google Fast Flip and Bing’s Visual Search.

Alan from The health Informaticist discusses in “NHS Evidence boo vs guidelinesfinder hurrah” that a simple search for backpain in NHS Evidence yielded 1320 hits (!) of which only a handful are useful guidelines, whereas the good old Guidelines Finder (now a ’specialist collection’), yields 47 mostly useful and relevant hits. He ends this discussion with a  request to NICE: please keep the specialist collections. And I agree.

On EBM and Clinical Support Librarians@UCHC this month an overview of current news, advisories and practical information about Pandemic Flu (H1N1) .

Related articles by Zemanta

his imagination had run away
Reblog this post [with Zemanta]