Grey Literature: Time to make it systematic

6 09 2009

Guest author: Shamsha Damani (@shamsha)

Grey literature is a term I first encountered in library school; I remember dubbing it “the-wild-goose-chase search” because it is time consuming, totally un-systematic, and a huge pain altogether. Things haven’t changed much in the grey literature arena, as I found out last week, when my boss asked me to help with the grey literature part of a systematic review.

Let me back up a bit and offer the official definition for grey literature by the experts of the Grey Literature International Steering Committee: “Information produced on all levels of government, academics, business and industry in electronic and print formats not controlled by commercial publishing i.e. where publishing is not the primary activity of the producing body.” Grey literature can include things such as policy documents, government reports, academic papers, theses, dissertations, bibliographies, conference abstracts/proceedings/papers, newsletters, PowerPoint presentations, standards/best practice documents, technical specifications, working papers and more! (Benzies et al 2006). So what is so time consuming about all this? There is no one magic database that will search all these at once. Translation: you have to search a gazillion places separately, which means you have to learn how to search each of these gazillion websites/databases separately. Now if doing searches for systematic reviews is your bread-and-butter, then you are probably scoffing already. But for a newbie like me, I was drowning big time.

After spending what seemed like an eternity to finish my search, I went back to the literature to see why inclusion of grey literature was so important. I know that grey literature adds to the evidence base and results in a comprehensive search, but it is often not peer-reviewed, and the quality of some of the documents is often questionable. So what I dug up was a bit surprising. The first was a Cochrane Review from 2007 titled “Grey literature in meta-analyses of randomized trials of health care interventions (review).” The authors concluded that not including grey literature in meta-analyses produced inflated results when looking at treatment effects. So the reason for inclusion of grey literature made sense: to reduce publication bias. Another paper published in the Bulletin of the World Health Organization concluded that grey literature tends to be more current, provides global coverage, and may have an impact on cost-effectiveness of various treatment strategies. This definitely got my attention because of the new buzzword in Washington: Comparative Effectiveness Research (CER). A lot of the grey literature is comprised of policy documents so it definitely has a big role to play in systematic reviews as well. However, the authors also pointed out that there is no systematic way to search the grey literature and undertaking such a search can be very expensive and time consuming. This validated my frustrations, but gave no solutions.

When I was struggling to get through my search, I was delighted to find a wonderful resource from the Canadian Agency for Drugs and Technologies in Health. They have created a document called “Grey Matters: A Practical Search Tool for Evidence-Based Medicine”, which is a 34-page checklist of many of the popular websites for searching grey literature, including a built-in documentation system. It was still tedious work because I had to search a ton of places, many resulting in no hits. But at least I had a start and a transparent way of documenting my work.

However, I’m still at a loss for why there are no official guidelines for librarians to search for grey literature. There are clear guidelines for authors of grey literature. Benzies and colleagues give compelling reasons for inclusion of grey literature in a systematic review, complete with a checklist for authors! Why not have guidelines for searching too? I know that every search would require different tools; but I think that a master list can be created, sort of like a must-search-these-first type of a list. It surely would help a newbie like me. I know that many libraries have such lists but they tend to be 10 pages long, with bibliographies for bibliographies! Based on my experience, I would start with the following resources the next time I encounter a grey literature search:

  1. National Guideline Clearinghouse
  2. Centre for Reviews and Dissemination
  3. Agency for Healthcare Research and Quality (AHRQ)
  4. Health Technology Assessment International (HTAI)
  5. Turning Research Into Practice (TRIP)

Some databases like Mednar, Deep Dyve, RePORTer, OAIster, and Google Scholar also deserve a mention but I have not had much luck with them. This is obviously not meant to be an exhaustive list. For that, I present my delicious page:, which is also ever-growing.

Finally, a request for the experts out there: if you have any tips on how to make this process less painful, please share it here. The newbies of the world will appreciate it.

Shamsha Damani

Clinical Librarian

Reblog this post [with Zemanta]



7 responses

6 09 2009
Twitter Trackbacks for Grey Literature: Time to make it systematic « Laika’s MedLibLog [] on

[…] Grey Literature: Time to make it systematic « Laika’s MedLibLog – view page – cached #RSS 2.0 Laika's MedLibLog » Grey Literature: Time to make it systematic Comments Feed Laika's MedLibLog Hallo iedereen! Friday Foolery Showing the invisible — From the page […]

6 09 2009
Twitted by DougSalz

[…] This post was Twitted by DougSalz […]

8 09 2009
Twitted by MDAndersonLib

[…] This post was Twitted by MDAndersonLib […]

9 09 2009
Dennis D. McDonald

In my opinion, solely focusing on lists and databases for solving the “grey literature problem” as it exists in health care and medical research would be short-sighted. As much as we want and need to create searchable databases and indexing and search guidelines, a more productive — and possibly more realistic — focus would be on establishing relationships with the producers and users of what we refer to as “grey literature.”

By this I mean that any database development or indexing effort should also incorporate social networking and collaboration technologies that support such functionality as expertise profiling, recommendations, ratings, and personal experience and commentary. In addition, we need to address the question of physical access and availability, and that may mean collecting documents and creating repositories that ensure access long after the originator has moved on. Many of the organizations that produce “grey literature” do so as only a secondary function and don’t devote the time and effort to access and distribution that people familiar with library and publishing processes take for granted. Solving indexing and search without physical access would therefore be an incomplete solution.

Dennis D. McDonald, Ph.D.
Twitter: @ddmcd

28 09 2009
MEDLIB’s ROUND 1.6 « Laika’s MedLibLog

[…] has contributed to this round by writing a guest post on this blog. This interesting post is about grey literature: what is grey literature, why do you need it and why not have guidelines for searching grey […]

8 08 2010
Collaborating and Delivering Literature Search Results to Clinical Teams Using Web 2.0 Tools « Laika's MedLibLog

[…] bell: she is a prominent twitterer and has written quest posts at this blog on several occasions (here, here, here and […]

9 03 2011
Multi-Author Medical Blogs – At the End it is all about Credibility « Laika's MedLibLog

[…] medicine, library-related topics), but had a fresh new look at certain topics (see for instance Grey literature time to make it systematic and  Uptodate versus Dynamed. The post were written by Shamsha, but I reviewed them before […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: