FUTON Bias. Or Why Limiting to Free Full Text Might not Always be a Good Idea.

8 09 2011

ResearchBlogging.orgA few weeks ago I was discussing possible relevant papers for the Twitter Journal Club  (Hashtag #TwitJC), a succesful initiative on Twitter, that I have discussed previously here and here [7,8].

I proposed an article, that appeared behind a paywall. Annemarie Cunningham (@amcunningham) immediately ran the idea down, stressing that open-access (OA) is a pre-requisite for the TwitJC journal club.

One of the TwitJC organizers, Fi Douglas (@fidouglas on Twitter), argued that using paid-for journals would defeat the objective that  #TwitJC is open to everyone. I can imagine that fee-based articles could set a too high threshold for many doctors. In addition, I sympathize with promoting OA.

However, I disagree with Annemarie that an OA (or rather free) paper is a prerequisite if you really want to talk about what might impact on practice. On the contrary, limiting to free full text (FFT) papers in PubMed might lead to bias: picking “low hanging fruit of convenience” might mean that the paper isn’t representative and/or doesn’t reflect the current best evidence.

But is there evidence for my theory that selecting FFT papers might lead to bias?

Lets first look at the extent of the problem. Which percentage of papers do we miss by limiting for free-access papers?

survey in PLOS by Björk et al [1] found that one in five peer reviewed research papers published in 2008 were freely available on the internet. Overall 8,5% of the articles published in 2008 (and 13,9 % in Medicine) were freely available at the publishers’ sites (gold OA).  For an additional 11,9% free manuscript versions could be found via the green route:  i.e. copies in repositories and web sites (7,8% in Medicine).
As a commenter rightly stated, the lag time is also important, as we would like to have immediate access to recently published research, yet some publishers (37%) impose an access-embargo of 6-12 months or more. (these papers were largely missed as the 2008 OA status was assessed late 2009).

PLOS 2009

The strength of the paper is that it measures  OA prevalence on an article basis, not on calculating the share of journals which are OA: an OA journal generally contains a lower number of articles.
The authors randomly sampled from 1.2 million articles using the advanced search facility of Scopus. They measured what share of OA copies the average researcher would find using Google.

Another paper published in  J Med Libr Assoc (2009) [2], using similar methods as the PLOS survey examined the state of open access (OA) specifically in the biomedical field. Because of its broad coverage and popularity in the biomedical field, PubMed was chosen to collect their target sample of 4,667 articles. Matsubayashi et al used four different databases and search engines to identify full text copies. The authors reported an OA percentage of 26,3 for peer reviewed articles (70% of all articles), which is comparable to the results of Björk et al. More than 70% of the OA articles were provided through journal websites. The percentages of green OA articles from the websites of authors or in institutional repositories was quite low (5.9% and 4.8%, respectively).

In their discussion of the findings of Matsubayashi et al, Björk et al. [1] quickly assessed the OA status in PubMed by using the new “link to Free Full Text” search facility. First they searched for all “journal articles” published in 2005 and then repeated this with the further restrictions of “link to FFT”. The PubMed OA percentages obtained this way were 23,1 for 2005 and 23,3 for 2008.

This proportion of biomedical OA papers is gradually increasing. A chart in Nature’s News Blog [9] shows that the proportion of papers indexed on the PubMed repository each year has increased from 23% in 2005 to above 28% in 2009.
(Methods are not shown, though. The 2008 data are higher than those of Björk et al, who noticed little difference with 2005. The Data for this chart, however, are from David Lipman, NCBI director and driving force behind the digital OA archive PubMed Central).
Again, because of the embargo periods, not all literature is immediately available at the time that it is published.

In summary, we would miss about 70% of biomedical papers by limiting for FFT papers. However, we would miss an even larger proportion of papers if we limit ourselves to recently published ones.

Of course, the key question is whether ignoring relevant studies not available in full text really matters.

Reinhard Wentz of the Imperial College Library and Information Service already argued in a visionary 2002 Lancet letter[3] that the availability of full-text articles on the internet might have created a new form of bias: FUTON bias (Full Text On the Net bias).

Wentz reasoned that FUTON bias will not affect researchers who are used to comprehensive searches of published medical studies, but that it will affect staff and students with limited experience in doing searches and that it might have the same effect in daily clinical practice as publication bias or language bias when doing systematic reviews of published studies.

Wentz also hypothesized that FUTON bias (together with no abstract available (NAA) bias) will affect the visibility and the impact factor of OA journals. He makes a reasonable cause that the NAA-bias will affect publications on new, peripheral, and under-discussion subjects more than established topics covered in substantive reports.

The study of Murali et al [4] published in Mayo Proceedings 2004 confirms that the availability of journals on MEDLINE as FUTON or NAA affects their impact factor.

Of the 324 journals screened by Murali et al. 38.3% were FUTON, 19.1%  NAA and 42.6% had abstracts only. The mean impact factor was 3.24 (±0.32), 1.64 (±0.30), and 0.14 (±0.45), respectively! The authors confirmed this finding by showing a difference in impact factors for journals available in both the pre and the post-Internet era (n=159).

Murali et al informally questioned many physicians and residents at multiple national and international meetings in 2003. These doctors uniformly admitted relying on FUTON articles on the Web to answer a sizable proportion of their questions. A study by Carney et al (2004) [5] showed  that 98% of the US primary care physicians used the Internet as a resource for clinical information at least once a week and mostly used FUTON articles to aid decisions about patient care or patient education and medical student or resident instruction.

Murali et al therefore conclude that failure to consider FUTON bias may not only affect a journal’s impact factor, but could also limit consideration of medical literature by ignoring relevant for-fee articles and thereby influence medical education akin to publication or language bias.

This proposed effect of the FFT limit on citation retrieval for clinical questions, was examined in a  more recent study (2008), published in J Med Libr Assoc [6].

Across all 4 questions based on a research agenda for physical therapy, the FFT limit reduced the number of citations to 11.1% of the total number of citations retrieved without the FFT limit in PubMed.

Even more important, high-quality evidence such as systematic reviews and randomized controlled trials were missed when the FFT limit was used.

For example, when searching without the FFT limit, 10 systematic reviews of RCTs were retrieved against one when the FFT limit was used. Likewise when searching without the FFT limit, 28 RCTs were retrieved and only one was retrieved when the FFT limit was used.

The proportion of missed studies (appr. 90%) is higher than in the studies mentioned above. Possibly this is because real searches have been tested and that only relevant clinical studies  have been considered.

The authors rightly conclude that consistently missing high-quality evidence when searching clinical questions is problematic because it undermines the process of Evicence Based Practice. Krieger et al finally conclude:

“Librarians can educate health care consumers, scientists, and clinicians about the effects that the FFT limit may have on their information retrieval and the ways it ultimately may affect their health care and clinical decision making.”

It is the hope of this librarian that she did a little education in this respect and clarified the point that limiting to free full text might not always be a good idea. Especially if the aim is to critically appraise a topic, to educate or to discuss current best medical practice.

References

  1. Björk, B., Welling, P., Laakso, M., Majlender, P., Hedlund, T., & Guðnason, G. (2010). Open Access to the Scientific Journal Literature: Situation 2009 PLoS ONE, 5 (6) DOI: 10.1371/journal.pone.0011273
  2. Matsubayashi, M., Kurata, K., Sakai, Y., Morioka, T., Kato, S., Mine, S., & Ueda, S. (2009). Status of open access in the biomedical field in 2005 Journal of the Medical Library Association : JMLA, 97 (1), 4-11 DOI: 10.3163/1536-5050.97.1.002
  3. WENTZ, R. (2002). Visibility of research: FUTON bias The Lancet, 360 (9341), 1256-1256 DOI: 10.1016/S0140-6736(02)11264-5
  4. Murali NS, Murali HR, Auethavekiat P, Erwin PJ, Mandrekar JN, Manek NJ, & Ghosh AK (2004). Impact of FUTON and NAA bias on visibility of research. Mayo Clinic proceedings. Mayo Clinic, 79 (8), 1001-6 PMID: 15301326
  5. Carney PA, Poor DA, Schifferdecker KE, Gephart DS, Brooks WB, & Nierenberg DW (2004). Computer use among community-based primary care physician preceptors. Academic medicine : journal of the Association of American Medical Colleges, 79 (6), 580-90 PMID: 15165980
  6. Krieger, M., Richter, R., & Austin, T. (2008). An exploratory analysis of PubMed’s free full-text limit on citation retrieval for clinical questions Journal of the Medical Library Association : JMLA, 96 (4), 351-355 DOI: 10.3163/1536-5050.96.4.010
  7. The #TwitJC Twitter Journal Club, a new Initiative on Twitter. Some Initial Thoughts. (laikaspoetnik.wordpress.com)
  8. The Second #TwitJC Twitter Journal Club (laikaspoetnik.wordpress.com)
  9. How many research papers are freely available? (blogs.nature.com)
Advertisements




The Second #TwitJC Twitter Journal Club

14 06 2011

In the previous post I wrote about  a new initiative on Twitter, the Twitter Journal Club (hashtag #TwitJC). Here, I shared some constructive criticism. The Twitter Journal Club is clearly an original and admirable initiative, that gained a lot of interest. But there is some room for improvement.

I raised two issues: 1. discussions with 100 people are not easy to follow on Twitter, and 2. walking through a checklist for critical appraisals is not the most interesting to do (particularly because it had already been done).

But as one of the organizers explained, the first session was just meant for promoting #twitjc. Instead of the expected 6 people, 100 tweople showed up.

In the second session, last Sunday evening, the organizers followed a different structure.

Thus, I thought it would only be fair, to share my experiences with the second session as well. This time I managed to follow it from start to finish.

Don’t worry. Discussing the journal club won’t be a regular item. I will leave the organization up to the organizers. The sessions might inspire me, though, to write a blog post on the topic now and then. But that may only work synergistic. (at least for me, because it forces me to rethink it all)

This time the discussion was about Rose’s Prevention Paradox (PDF), a 30 year old paper that is still relevant. The paper is more of an opinion piece, therefore the discussion focused on the implications of the Prevention Paradox theory. It was really helpful that Fi wrote an introduction to the paper, and a Points of Discussion beforehand. There were 5 questions (and many sub-questions).

I still found it very hard to follow it all at Twitter, as illustrated by the following tweet:

  • laikas I think I lost track. Which question are we? #twitjc Sun Jun 12 20:07:03
  • laikas @MsPhelps ik werd wel helemaal duizelig van al die tweets. Er zijn toch wel veel mensen die steeds een andere vraag stellen voor de 1e is beantwoord -9:47 PM Jun 12th, 2011 (about instant nausea when seeing tweets rolling by and people already posing a new question before the first one is answered)

I followed the tweets at http://tweetchat.com/room/twitjc. Imagine tweets rolling by and you try to pick up those tweets you want to respond to (either bc they are very relevant, or because you disagree). By the time you have finished your tweet, already 20 -possibly very interesting tweets- passed by, including the next question by the organizers (unfortunately they didn’t use the official @twitjournalclub account for this).

Well, I suppose I am not very good at this. Partly because I’m Dutch (thus it takes longer to compose my tweets), partly because I’m not a fast thinker. I’m better at thorough analyses, at my blog for instance.

But this is Twitter.  To speak with Johan Cruyff, a legendary soccer-player from Holland, “Every disadvantage has its advantage”.

Twitter may not favor organized discussions, but on the other hand it is very engaging, thought-provoking and easy accessible. Where else do you meet 100 experts/doctors willing  to exchange thoughts about an interesting medical topic?

The tweets below are in line with/reflect my opinion on this second Twitter Journal Club (RT means retweeting/repeating the tweet):

  • laikas RT @themattmak@fidouglas @silv24 Congratulations again on a great #twitjc. Definitely more controversial and debate generating than last week’s! -9:18 PM Jun 12th, 2011
  • laikas @silv24 well i think it went well (it is probably me, I’m 2 slow). This paper is broad, evokes much discussion & many examples can B given -9:45 PM Jun 12th, 2011
  • DrDLittle Less structure to #twitJC last night but much wider debate 7:41 AM Jun 13th, 2011
  • amitns @DrDLittle It’s obviously a very complex topic, more structure would have stifled the debate. A lot of food for thought.#twitJC -7:45 AM Jun 13th, 2011

Again, the Twitter Journal Club gained a lot of interest. Scientist and teachers consider to borrow the concept. Astronomers are already preparing their first meeting on Thursday… And Nature seems to be on top of it as well, as it will interview the organizers of the medical and the astronomy journal club for an interview.

Emergency Physician Tom Young with experience in critically appraisal just summarized it nicely: (still hot from the press):

The two meetings of the journal club so far have not focussed in on this particular system; the first used a standard appraisal tool for randomised controlled trials, the second was more laissez-faire in its approach. This particular journal club is finding its feet in a new setting (that of Twitter) and will find its strongest format through trial and error. indeed, to try to manage such a phenomenon might be likened to ‘herding cats’ that often used description of trying to manage doctors, and I think, we would all agree would be highly inadvisable. Indeed, one of its strengths is that participants, or followers, will take from it what they wish, and this will be something, rather than nothing, whatever paper is discussed, even if it is only contact with another Tweeter, with similar or divergent views. 

Indeed, what I gained from these two meetings is that I met various nice and interesting people (including the organizers, @fidouglas and @silv24). Furthermore, I enjoyed the discussions, and picked up some ideas and examples that I would otherwise wouldn’t know about. The last online meeting sparked my interest in the prevention paradox. Before the meeting, I only read the paper at a glance. After the session I decided to read it again, and in more detail. As a matter of fact I feel inspired to write a blog post about this theory. Originally I planned to write a summary here, but probably the post is getting too long. Thus I will await the summary by the organizers and see if I have time to discuss it as well.

Related articles