A Filter for Finding “All Studies on Animal Experimentation in PubMed”

29 09 2010

ResearchBlogging.orgFor  an introduction to search filters you can first read this post.

Most people searching PubMed try to get rid of publications about animals. But basic scientists and lab animal technicians just want to find those animal studies.

PubMed has built-in filters for that: the limits. There is a limit  for “humans” and a limit for “animals”. But that is not good enough to find each and every article about humans, respectively animals. The limits are MeSH, Medical Subject Headings or index-terms and these are per definition not added to new articles, that haven’t been indexed yet. To name the main disadvantage…
Thus to find all papers one should at least search for other relevant MeSH and textwords (words in title and abstract) too.

A recent paper published in Laboratory Animals describes a filter for finding “all studies on animal experimentation in PubMed“, to facilitate “writing a systematic review (SR) of animal research” .

As the authors rightly emphasize, SR’s are no common practice in the field of animal research. Knowing what already has been done can prevent unnecessary duplication of animal experiments and thus unnecessary animal use. The authors have interesting ideas like registration of animal studies (similar to clinical trials registers).

In this article they describe the design of an animal filter for PubMed. The authors describe their filter as follows:

“By using our effective search filter in PubMed, all available literature concerning a specific topic can be found and read, which will help in making better evidencebased decisions and result in optimal experimental conditions for both science and animal welfare.”

Is this conclusion justified?

Design of the filter

Their filter is subjectively derived: the terms are “logically” chosen.

[1] The first part of the animal filter consists of only MeSH-terms.

You can’t use animals[mh] (mh=Mesh) as a search term, because MeSH are automatically exploded in PubMed. This means that narrower terms (lower in the tree) are also searched. If “Animals” were allowed to explode, the search would include the MeSH, “Humans”, which is at the end of one tree (primates etc, see Fig. below)

Therefore the MeSH-parts of their search consists of:

  1. animals [mh:noexp]: only articles are found that are indexed with “animals”, but not its narrower terms. Notably, this is identical to the PubMed Limit: animals).
  2. Exploded Animal-specific MeSH-terms not having humans as a narrow term, i.e. “fishes”[MeSH Terms].
  3. and non-exploded MeSH in those cases that humans occurred in the same branch. Like “primates”[MeSH Terms:noexp]
  4. In addition two other MeSH are used: “animal experimentation”[MeSH Terms] and “models, animal”[MeSH Terms]

[2] The second part of the search filter consist of terms in the title and abstract (command: [tiab]).

The terms are taken from relevant MeSH, two reports about animal experimentation in the Netherlands and in Europe, and the experience of the authors, who are experts in the field.

The authors use this string for non-indexed records (command: NOT medline[sb]). Thus this part is only meant to find records that haven’t (yet) been indexed, but in which (specific) animals are mentioned by the author in title or text. Synonyms and spelling variants have been taken into account.

Apparently the authors have chosen NOT to search for text words in indexed records only. Presumably it gives too much noise, to search for animals mentioned in non-indexed articles. However, the authors do not discuss why this was necessary.

This search string is extremely long. Partly because truncation isn’t used with the longer words: i.e. nematod*[tiab] instead of nematoda[Tiab] OR nematode[Tiab] OR nematoda[Tiab] OR nematode[Tiab] OR nematodes[Tiab]. Partly because they aim for completeness. However the usefulness of the terms as such hasn’t been verified (see below).

Search strategies can be freely accessed here.

Validation

The filter is mainly validated against the PubMed Limit “Animals”.

The authors assume that the PubMed Limits are “the most easily available and most obvious method”. However I know few librarians or authors of systematic reviews who would solely apply this so called ‘regular method’. In the past I have used exactly the same MeSH-terms (1) and the main text words (2) as included in their filter.

Considering that the filter includes the PubMed limit “Animals” [1.1] it does not come as a surprise that the sensitivity of the filter exceeds that of the PubMed limit Animals…

Still, the sensitivity (106%) is not really dramatic: 6% more records are found, the PubMed Limit “animals” is set as 100%.

Apparently records are very well indexed with the MeSH “animals”. Few true animal records are missed, because “animals” is a check tag. A check tag is a MeSH that is looked for routinely by indexers in every journal article. It is added to the record even if it isn’t the main (or major) point of an article.

Is an increased sensitivity of appr. 6% sufficient to conclude that this filter “performs much better than the current alternative in PubMed”?

No. It is not only important that MORE is found but to what degree the extra hits are relevant. Surprisingly, the authors ONLY determined SENSITIVITY, not specificity or precision.

There are many irrelevant hits, partly caused by the inclusion of animal population groups[mesh], which has some narrower terms that often not used for experimentation, i.e. endangered species.

Thus even after omission of animal population groups[mesh], the filter still gives hits like:

These are evidently NOT laboratory animal experiments and mainly caused by the inclusion invertebrates  like plankton.

Most other MeSH are not extremely useful either. Even terms as animal experimentation[mh] and models, animal[mh] are seldom assigned to experimental studies lacking animals as a MeSH.

According to the authors, the MeSH “Animals” will not retrieve studies solely indexed with the MeSH term Mice. However, the first records missed with mice[mesh] NOT animals[mh:noexp] are from 1965, when they apparently didn’t use “animals” as a check tag in addition to specific ‘animal’ MeSH.

Thus presumably the MeSH-filter can be much shorter and need only contain animal MeSH (rats[mh], mice[mh] etc) when publications older than 1965 are also required.

The types of vertebrate animals used in lab re...

Image via Wikipedia

Their text word string (2) is also extremely long.  Apart from the lack of truncation, most animal terms are not relevant for most searches. 2/3 of the experiments are done with rodents (see Fig). The other animals are often used for specific experiments (zebra-fish, Drosophila) or in another context, not related to animal experiments, such as:

swine flu, avian flu, milk production by cows, or allergy by milk-products or mites, stings by insects and bites by dogs and of course fish, birds, cattle and poultry as food, fetal calf serum in culture medium, but also vaccination with “mouse products” in humans. Thus most of the terms produce noise for most topics. An example below (found by birds[mesh] :-)

On the other hand strains of mice and rats are missing from the search string: i.e. balb/c, wistar.

Extremely long search strings (1 page) are also annoying to use. However, the main issue is whether the extra noise matters. Because the filter is meant to find all experimental animal studies.

As Carlijn Hooijmans notices correctly, the filters are never used on their own, only in combination with topic search terms.

Hooijmans et al have therefore “validated” their filter with two searches. “Validated” between quotation marks because they have only compared the number of hits, thus the increase in sensitivity.

Their first topic is the use of probiotics in experimental pancreatitis (see appendix).

Their filter (combined with the topic search) retrieved 37 items against 33 items with the so called “regular method”: an increase in sensitivity of 21,1%.

After updating the search I got  38 vs 35 hits. Two of the 3 extra hits obtained with the broad filter are relevant and are missed with the PubMed limit for animals, because the records haven’t been indexed. They could also have been found with the text words pig*[tiab] or dog*[tiab]. Thus the filter is ok for this purpose, but unnecessary long. The MeSH-part of the filter had NO added value compared to animals[mh:noexp].

Since there are only 148 hits without the use of any filters, researchers could also use screen all hits. Alternatively there is a trick to safely exclude human studies:

NOT (humans[mh] NOT animals[mh:noexp])

With this double negation you exclude PubMed records that are indexed with humans[mh], as long as these records aren’t indexed with animals[mh:noexp] too. It is far “safer” than limiting for “animals”[mesh:noexp] only. We use a similar approach to ” exclude”  animals when we search for human studies.

This extremely simple filter yields 48 hits, finding all hits found with the large animal filter (plus 10 irrelevant hits).

Such a simple filter can easily be used for searches with relatively few hits, but gives too many irrelevant hits in case of  a high yield.

The second topic is food restriction. 9280 Records were obtained with the Limit: “Animals”, whereas this strategy combined with the complete filter retrieved 9650 items. The sensitivity in this search strategy was therefore 104%. 4% extra hits were obtained.

The MeSH-search added little to the search. Only 21 extra hits. The relevant hits were (again) only from before 1965.

The text-word part of the search finds relevant new articles, although there are quite some  irrelevant findings too, i.e. dieting and obtaining proteins from chicken.

4% isn’t a lot extra, but the aim of the researchers is too find all there is.

However, it is the question whether researchers want to find every single experiment or observation done in the animal kingdom. If I were to plan an experiment on whether food restriction lowers the risk for prostate cancer in a transgenic mice, need I know what the effects are of food restriction on Drosophila, nematodes, salmon or even chicken on whatever outcome? Would I like to screen 10,000 hits?

Probably most researchers would like separate filters for rodents and other laboratory animals (primates, dogs) and for work on Drosophila or fish. In some fields there might also be a need to filter clinical trials and reviews out.

Furthermore, it is not only important to have a good filter but also a good search.

The topic searches in the current paper are not ideal: they contain overlapping terms (food restriction is also found by food and restriction) and misses important MeSH (Food deprivation, fasting and the broader term of caloric restriction “energy intake” are assigned more often to records about food deprivation than caloric restriction).

Their search:

(“food restriction”[tiab] OR (“food”[tiab] AND “restriction”[tiab]) OR “feed restriction”[tiab] OR (“feed”[tiab] AND “restriction”[tiab]) OR “restricted feeding”[tiab] OR (“feeding”[tiab] AND “restricted”[tiab]) OR “energy restriction”[tiab] OR (“energy”[tiab] AND “restriction”[tiab]) OR “dietary restriction”[tiab] OR (dietary”[tiab] AND “restriction”[tiab]) OR “caloric restriction”[MeSH Terms] OR (“caloric”[tiab] AND “restriction”[tiab]) OR “caloric restriction”[tiab])
might for instance be changed to:

Energy Intake[mh] OR Food deprivation[mh] OR Fasting[mh] OR food restrict*[tiab] OR feed restrict*[tiab] OR restricted feed*[tiab] OR energy restrict*[tiab] OR dietary restrict*[tiab] OR  caloric restrict*[tiab] OR calorie restrict*[tiab] OR diet restrict*[tiab]

You do not expect such incomplete strategies from people who repeatedly stress that: most scientists do not know how to use PubMed effectively” and that “many researchers do not use ‘Medical Subject Headings’ (MeSH terms), even though they work with PubMed every day”…..

Combining this modified search with their animal filter yields 21920 hits instead of 10335 as found with their “food deprivation” search and their animal filter. A sensitivity of 212%!!! Now we are talking! ;) (And yes there are many new relevant hits found)

Summary

The paper describes the performance of a subjective search filter to find all experimental studies performed with laboratory animals. The authors have merely “validated”  this filter against the Pubmed Limits: animals. In addition, they only determined sensitivity:  on average 7% more hits were obtained with the new animal filter than with the PubMed limit alone.

The authors have not determined the specificity or precision of the filter, not even for the 2 topics where they have applied the filter. A quick look at the results shows that the MeSH-terms other than the PubMed limit “animals” contributed little to the enhanced sensitivity. The text word part of the filter yields more relevant hits. Still -depending on the topic- there are many irrelevant records found, because  it is difficult to separate animals as food, allergens etc from laboratory animals used in experiments and the filter is developed to find every single animal in the animal kingdom, including poultry, fish, nematodes, flies, endangered species and plankton. Another (hardly to avoid) “contamination” comes from in vitro experiments with animal cells, animal products used in clinical trials and narrative reviews.

In practice, only parts of the search filter seem useful for most systematic reviews, and especially if these reviews are not meant to give an overview of all findings in the universe, but are needed to check if a similar experiment hasn’t already be done. It seems impractical if researchers have to make a systematic review, checking, summarizing and appraising  10,000 records each time they start a new experiment.

Perhaps I’m somewhat too critical, but the cheering and triumphant tone of the paper in combination with a too simple design and without proper testing of the filter asked for a critical response.

Credits

Thanks to Gerben ter Riet for alerting me to the paper. He also gave the tip that the paper can be obtained here for free.

References

  1. Hooijmans CR, Tillema A, Leenaars M, & Ritskes-Hoitinga M (2010). Enhancing search efficiency by means of a search filter for finding all studies on animal experimentation in PubMed. Laboratory animals, 44 (3), 170-5 PMID: 20551243

———————-

About these ads

Actions

Information

6 responses

6 10 2010
How will we ever keep up with 75 Trials and 11 Systematic Reviews a Day? « Laika's MedLibLog

[...] A Filter for Finding “All Studies on Animal Experimentation in PubMed” (laikaspoetnik.wordpress.com) [...]

15 10 2010
Medical Information Matters 2.8 is up! « Laika's MedLibLog

[...] is “Programs in libraries or medical education”. Besides two posts from this blog (A Filter for Finding Animal Studies in PubMed” and more on the topic: An Educator by Chance) the following topics are included: a new MeSH [...]

18 10 2010
Problems with Disappearing Set Numbers in PubMed’s Clinical Queries « Laika's MedLibLog

[...] A Filter for Finding “All Studies on Animal Experimentation in PubMed” (laikaspoetnik.wordpress.com) [...]

19 10 2010
Search OVID EMBASE and Get MEDLINE for Free…. without knowing it « Laika's MedLibLog

[...] A Filter for Finding “All Studies on Animal Experimentation in PubMed” (laikaspoetnik.wordpress.com) [...]

8 01 2011
Aandacht voor diergeneeskunde / dierexperimenteel onderzoek « UBU Medisch Team

[...] als co-auteur. Een uitgebreid commentaar op het artikel (met kanttekeningen) is te lezen op het blog van Jacqueline Limpens [...]

15 03 2012
C. Hooijmans, R de Vries, A Tillema

Ms Jacqueline Limpens posted interesting comments on her blog Laika’s MedLibLog regarding the search filter we developed for finding animal studies in PubMed (Hooijmans, 2010). We would like to respond to those comments.

In our article, we state that collecting and analysing all available literature before starting an animal experiment is of the utmost importance. It is a means of reducing unnecessary duplication of experiments and unnecessary animal use and of improving the safety of translating animal research into clinical benefits (since different species might react differently). It is an elaborate venture to obtain a complete overview of studies on a certain topic in laboratory animal science. Among factors that influence the current lack of systematic reviews (SR) in animal research are the intricacy of search options in bibliographic databases like PubMed and scientists’ unawareness of those options or their inability to use several options efficiently. To increase awareness and to facilitate more effective searching for animal studies in PubMed we created a search filter.

One of the comments that Ms Limpens’ raises is that the filter is too long (i.e. longer than necessary) and that not all MeSH terms are extremely useful. She notes that the first records missed with Mice[mesh] NOT Animals[mh:noexp] are from 1965, when they apparently didn’t use “animals” as a check tag in addition to specific ‘animal’ MeSH.
We agree with Ms Limpens that indeed most of the records missed when using the searchstring Mice[mesh] NOT Animals[mh:noexp] are from 1965 since in the last few years many mistakes have been retrospectively corrected by PubMed. Nevertheless the goal of this filter is to find all papers concerning experimental animal studies and therefore in our view publications older than 1965 should not be disregarded.
When we performed our first searches for all relevant papers concerning omega-3 fatty acid supplementation in experimental Alzheimer’s disease, some very relevant papers were not found. One of the missing papers was one of my own, which at that time (although it was about transgenic mice, and this was even a title word) did not receive the MesH term Animals. Another example of a paper clearly concerning animals, but which did not receive the MesH term Animals is the article by Molinari et al. (2002) on pigs. (The effect of testosterone on regional blood flow in prepubertal anaesthetized pigs; Molinari et al.)
Even though these may be rare omissions we decided to include the separate MeSH terms because of the considerable advantage of missing fewer relevant articles.
It is true that using the animal search filter will inevitably also result in many irrelevant hits especially because the search results will include studies on animal cells, animal products and names of species of strains used in diseases, e.g.swine flu. We welcome suggestions for improving the balance between precision and sensitivity of the search filter.
Last but not least; the length of the filter does not detract from the ease of use. We agree with Limpens that indeed it would have been a problem in case you had to type the entire filter into the search box every time you want to use it. We created a supplement, however, from which you can directly copy and paste the filter into the search box.

Other issues mentioned by Ms Limpens regarding the filter’s length are the lack of truncation and the fact that most animal terms do not seem relevant for most searches.
Our decision to leave out truncation, and describe all the relevant synonyms ourselves was motivated by the aim to prevent retrieval of even more irrelevant hits. Although many search terms included in the search filter may seem irrelevant in most searches, they are relevant in some searches. It is important to include those “more unusual animal models” in your search as well, because we ultimately want to improve the safe translation of the results from animal studies to the clinical situation. The filter was developed to be used by everyone searching for animal studies. In addition, when a specific species is not used in a specific research area, and is thus superfluous, this term will not yield many extra hits, and is therefore quite harmless.

Ms Limpens’also suggests to use the search string NOT (Humans[mh] NOT Animals[mh:noexp]) instead of the animal filter to safely exclude human studies.
Limpens’ approach is not in line with our aim to find all animal studies and exclude human studies in one go. Limpens’ approach also includes in vitro studies and other irrelevant topics which will lead to many irrelevant hits, and is likely to be even less specific compared to our search filter

One of the other comments Ms Limpens raises is that we did not validate the search filter in the strict sense of the word.
We agree with her that we indeed did not validate our search filter in the strict sense of the word and we explained the reasons extensively in the discussion (page 174 2nd column), but in short:
It would be practically unfeasible to formally validate the entire filter via the regular process (for arguments see discussion section of the paper) and was beyond the scope of our paper as well. Nevertheless we used a more informal way of validating the filter by comparing it with the only alternative currently available, the PubMed Limit: Animals, and calculate sensitivity relative
to this alternative.

Ms Limpens also states that only parts of the search filter seem useful for most systematic reviews, and especially if these reviews are not meant to give an overview of all findings in the universe, but are needed to check if a similar experiment hasn’t already be done. She deems it impractical for researchers each time they start a new experiment to have to make a systematic review, checking, summarizing and appraising 10,000 records.
We agree with Ms Limpens here. Our comment, however, is that the search filter is meant to help researchers to detect all animal studies in their specific field of research. In addition, from our own experience we can relate that when conducting and writing a SR in laboratory animal science using the search filter we end up with roughly 1000 abstracts. This is not more than is common in SR of humane studies. In addition, it is important to stress that the search filter for animal studies is only one component of the total search strategy (for more information; Leenaars et al., Lab Anim. 2012 Jan;46(1):24-31).

To conclude, we do not advocate performing a SR before starting a new experiment, but we do stress the importance of searching systematically for all original papers before the start of an experiment. In this way, unnecessary duplication of experiments may be prevented and experiments may be designed that really add to the existing knowledge. Our filter is an important tool in this context.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




Follow

Get every new post delivered to your Inbox.

Join 610 other followers

%d bloggers like this: