For an introduction to search filters you can first read this post.
Most people searching PubMed try to get rid of publications about animals. But basic scientists and lab animal technicians just want to find those animal studies.
PubMed has built-in filters for that: the limits. There is a limit for “humans” and a limit for “animals”. But that is not good enough to find each and every article about humans, respectively animals. The limits are MeSH, Medical Subject Headings or index-terms and these are per definition not added to new articles, that haven’t been indexed yet. To name the main disadvantage…
Thus to find all papers one should at least search for other relevant MeSH and textwords (words in title and abstract) too.
A recent paper published in Laboratory Animals describes a filter for finding “all studies on animal experimentation in PubMed“, to facilitate “writing a systematic review (SR) of animal research” .
As the authors rightly emphasize, SR’s are no common practice in the field of animal research. Knowing what already has been done can prevent unnecessary duplication of animal experiments and thus unnecessary animal use. The authors have interesting ideas like registration of animal studies (similar to clinical trials registers).
In this article they describe the design of an animal filter for PubMed. The authors describe their filter as follows:
“By using our effective search filter in PubMed, all available literature concerning a specific topic can be found and read, which will help in making better evidencebased decisions and result in optimal experimental conditions for both science and animal welfare.”
Is this conclusion justified?
Design of the filter
Their filter is subjectively derived: the terms are “logically” chosen.
[1] The first part of the animal filter consists of only MeSH-terms.
You can’t use animals[mh] (mh=Mesh) as a search term, because MeSH are automatically exploded in PubMed. This means that narrower terms (lower in the tree) are also searched. If “Animals” were allowed to explode, the search would include the MeSH, “Humans”, which is at the end of one tree (primates etc, see Fig. below)
Therefore the MeSH-parts of their search consists of:
- animals [mh:noexp]: only articles are found that are indexed with “animals”, but not its narrower terms. Notably, this is identical to the PubMed Limit: animals).
- Exploded Animal-specific MeSH-terms not having humans as a narrow term, i.e. “fishes”[MeSH Terms].
- and non-exploded MeSH in those cases that humans occurred in the same branch. Like “primates”[MeSH Terms:noexp]
- In addition two other MeSH are used: “animal experimentation”[MeSH Terms] and “models, animal”[MeSH Terms]
[2] The second part of the search filter consist of terms in the title and abstract (command: [tiab]).
The terms are taken from relevant MeSH, two reports about animal experimentation in the Netherlands and in Europe, and the experience of the authors, who are experts in the field.
The authors use this string for non-indexed records (command: NOT medline[sb]). Thus this part is only meant to find records that haven’t (yet) been indexed, but in which (specific) animals are mentioned by the author in title or text. Synonyms and spelling variants have been taken into account.
Apparently the authors have chosen NOT to search for text words in indexed records only. Presumably it gives too much noise, to search for animals mentioned in non-indexed articles. However, the authors do not discuss why this was necessary.
This search string is extremely long. Partly because truncation isn’t used with the longer words: i.e. nematod*[tiab] instead of nematoda[Tiab] OR nematode[Tiab] OR nematoda[Tiab] OR nematode[Tiab] OR nematodes[Tiab]. Partly because they aim for completeness. However the usefulness of the terms as such hasn’t been verified (see below).
Search strategies can be freely accessed here.
Validation
The filter is mainly validated against the PubMed Limit “Animals”.
The authors assume that the PubMed Limits are “the most easily available and most obvious method”. However I know few librarians or authors of systematic reviews who would solely apply this so called ‘regular method’. In the past I have used exactly the same MeSH-terms (1) and the main text words (2) as included in their filter.
Considering that the filter includes the PubMed limit “Animals” [1.1] it does not come as a surprise that the sensitivity of the filter exceeds that of the PubMed limit Animals…
Still, the sensitivity (106%) is not really dramatic: 6% more records are found, the PubMed Limit “animals” is set as 100%.
Apparently records are very well indexed with the MeSH “animals”. Few true animal records are missed, because “animals” is a check tag. A check tag is a MeSH that is looked for routinely by indexers in every journal article. It is added to the record even if it isn’t the main (or major) point of an article.
Is an increased sensitivity of appr. 6% sufficient to conclude that this filter “performs much better than the current alternative in PubMed”?
No. It is not only important that MORE is found but to what degree the extra hits are relevant. Surprisingly, the authors ONLY determined SENSITIVITY, not specificity or precision.
There are many irrelevant hits, partly caused by the inclusion of animal population groups[mesh], which has some narrower terms that often not used for experimentation, i.e. endangered species.
Thus even after omission of animal population groups[mesh], the filter still gives hits like:
These are evidently NOT laboratory animal experiments and mainly caused by the inclusion invertebrates like plankton.
Most other MeSH are not extremely useful either. Even terms as animal experimentation[mh] and models, animal[mh] are seldom assigned to experimental studies lacking animals as a MeSH.
According to the authors, the MeSH “Animals” will not retrieve studies solely indexed with the MeSH term Mice. However, the first records missed with mice[mesh] NOT animals[mh:noexp] are from 1965, when they apparently didn’t use “animals” as a check tag in addition to specific ‘animal’ MeSH.
Thus presumably the MeSH-filter can be much shorter and need only contain animal MeSH (rats[mh], mice[mh] etc) when publications older than 1965 are also required.
Image via Wikipedia
Their text word string (2) is also extremely long. Apart from the lack of truncation, most animal terms are not relevant for most searches. 2/3 of the experiments are done with rodents (see Fig). The other animals are often used for specific experiments (zebra-fish, Drosophila) or in another context, not related to animal experiments, such as:
swine flu, avian flu, milk production by cows, or allergy by milk-products or mites, stings by insects and bites by dogs and of course fish, birds, cattle and poultry as food, fetal calf serum in culture medium, but also vaccination with “mouse products” in humans. Thus most of the terms produce noise for most topics. An example below (found by birds[mesh] 
On the other hand strains of mice and rats are missing from the search string: i.e. balb/c, wistar.
Extremely long search strings (1 page) are also annoying to use. However, the main issue is whether the extra noise matters. Because the filter is meant to find all experimental animal studies.
As Carlijn Hooijmans notices correctly, the filters are never used on their own, only in combination with topic search terms.
Hooijmans et al have therefore “validated” their filter with two searches. “Validated” between quotation marks because they have only compared the number of hits, thus the increase in sensitivity.
Their first topic is the use of probiotics in experimental pancreatitis (see appendix).
Their filter (combined with the topic search) retrieved 37 items against 33 items with the so called “regular method”: an increase in sensitivity of 21,1%.
After updating the search I got 38 vs 35 hits. Two of the 3 extra hits obtained with the broad filter are relevant and are missed with the PubMed limit for animals, because the records haven’t been indexed. They could also have been found with the text words pig*[tiab] or dog*[tiab]. Thus the filter is ok for this purpose, but unnecessary long. The MeSH-part of the filter had NO added value compared to animals[mh:noexp].
Since there are only 148 hits without the use of any filters, researchers could also use screen all hits. Alternatively there is a trick to safely exclude human studies:
NOT (humans[mh] NOT animals[mh:noexp])
With this double negation you exclude PubMed records that are indexed with humans[mh], as long as these records aren’t indexed with animals[mh:noexp] too. It is far “safer” than limiting for “animals”[mesh:noexp] only. We use a similar approach to ” exclude” animals when we search for human studies.
This extremely simple filter yields 48 hits, finding all hits found with the large animal filter (plus 10 irrelevant hits).
Such a simple filter can easily be used for searches with relatively few hits, but gives too many irrelevant hits in case of a high yield.
The second topic is food restriction. 9280 Records were obtained with the Limit: “Animals”, whereas this strategy combined with the complete filter retrieved 9650 items. The sensitivity in this search strategy was therefore 104%. 4% extra hits were obtained.
The MeSH-search added little to the search. Only 21 extra hits. The relevant hits were (again) only from before 1965.
The text-word part of the search finds relevant new articles, although there are quite some irrelevant findings too, i.e. dieting and obtaining proteins from chicken.
4% isn’t a lot extra, but the aim of the researchers is too find all there is.
However, it is the question whether researchers want to find every single experiment or observation done in the animal kingdom. If I were to plan an experiment on whether food restriction lowers the risk for prostate cancer in a transgenic mice, need I know what the effects are of food restriction on Drosophila, nematodes, salmon or even chicken on whatever outcome? Would I like to screen 10,000 hits?
Probably most researchers would like separate filters for rodents and other laboratory animals (primates, dogs) and for work on Drosophila or fish. In some fields there might also be a need to filter clinical trials and reviews out.
Furthermore, it is not only important to have a good filter but also a good search.
The topic searches in the current paper are not ideal: they contain overlapping terms (food restriction is also found by food and restriction) and misses important MeSH (Food deprivation, fasting and the broader term of caloric restriction “energy intake” are assigned more often to records about food deprivation than caloric restriction).
Their search:
(“food restriction”[tiab] OR (“food”[tiab] AND “restriction”[tiab]) OR “feed restriction”[tiab] OR (“feed”[tiab] AND ”restriction”[tiab]) OR “restricted feeding”[tiab] OR (“feeding”[tiab] AND “restricted”[tiab]) OR “energy restriction”[tiab] OR (“energy”[tiab] AND “restriction”[tiab]) OR ”dietary restriction”[tiab] OR (“dietary”[tiab] AND “restriction”[tiab]) OR “caloric restriction”[MeSH Terms] OR (“caloric”[tiab] AND “restriction”[tiab]) OR “caloric restriction”[tiab])
might for instance be changed to:
Energy Intake[mh] OR Food deprivation[mh] OR Fasting[mh] OR food restrict*[tiab] OR feed restrict*[tiab] OR restricted feed*[tiab] OR energy restrict*[tiab] OR dietary restrict*[tiab] OR caloric restrict*[tiab] OR calorie restrict*[tiab] OR diet restrict*[tiab]
You do not expect such incomplete strategies from people who repeatedly stress that: “most scientists do not know how to use PubMed effectively” and that “many researchers do not use ‘Medical Subject Headings’ (MeSH terms), even though they work with PubMed every day”…..
Combining this modified search with their animal filter yields 21920 hits instead of 10335 as found with their “food deprivation” search and their animal filter. A sensitivity of 212%!!! Now we are talking! 
(And yes there are many new relevant hits found)
Summary
The paper describes the performance of a subjective search filter to find all experimental studies performed with laboratory animals. The authors have merely “validated” this filter against the Pubmed Limits: animals. In addition, they only determined sensitivity: on average 7% more hits were obtained with the new animal filter than with the PubMed limit alone.
The authors have not determined the specificity or precision of the filter, not even for the 2 topics where they have applied the filter. A quick look at the results shows that the MeSH-terms other than the PubMed limit “animals” contributed little to the enhanced sensitivity. The text word part of the filter yields more relevant hits. Still -depending on the topic- there are many irrelevant records found, because it is difficult to separate animals as food, allergens etc from laboratory animals used in experiments and the filter is developed to find every single animal in the animal kingdom, including poultry, fish, nematodes, flies, endangered species and plankton. Another (hardly to avoid) “contamination” comes from in vitro experiments with animal cells, animal products used in clinical trials and narrative reviews.
In practice, only parts of the search filter seem useful for most systematic reviews, and especially if these reviews are not meant to give an overview of all findings in the universe, but are needed to check if a similar experiment hasn’t already be done. It seems impractical if researchers have to make a systematic review, checking, summarizing and appraising 10,000 records each time they start a new experiment.
Perhaps I’m somewhat too critical, but the cheering and triumphant tone of the paper in combination with a too simple design and without proper testing of the filter asked for a critical response.
Credits
Thanks to Gerben ter Riet for alerting me to the paper. He also gave the tip that the paper can be obtained here for free.
References
- Hooijmans CR, Tillema A, Leenaars M, & Ritskes-Hoitinga M (2010). Enhancing search efficiency by means of a search filter for finding all studies on animal experimentation in PubMed. Laboratory animals, 44 (3), 170-5 PMID: 20551243
———————-
Recent Comments