FUTON Bias. Or Why Limiting to Free Full Text Might not Always be a Good Idea.

8 09 2011

ResearchBlogging.orgA few weeks ago I was discussing possible relevant papers for the Twitter Journal Club  (Hashtag #TwitJC), a succesful initiative on Twitter, that I have discussed previously here and here [7,8].

I proposed an article, that appeared behind a paywall. Annemarie Cunningham (@amcunningham) immediately ran the idea down, stressing that open-access (OA) is a pre-requisite for the TwitJC journal club.

One of the TwitJC organizers, Fi Douglas (@fidouglas on Twitter), argued that using paid-for journals would defeat the objective that  #TwitJC is open to everyone. I can imagine that fee-based articles could set a too high threshold for many doctors. In addition, I sympathize with promoting OA.

However, I disagree with Annemarie that an OA (or rather free) paper is a prerequisite if you really want to talk about what might impact on practice. On the contrary, limiting to free full text (FFT) papers in PubMed might lead to bias: picking “low hanging fruit of convenience” might mean that the paper isn’t representative and/or doesn’t reflect the current best evidence.

But is there evidence for my theory that selecting FFT papers might lead to bias?

Lets first look at the extent of the problem. Which percentage of papers do we miss by limiting for free-access papers?

survey in PLOS by Björk et al [1] found that one in five peer reviewed research papers published in 2008 were freely available on the internet. Overall 8,5% of the articles published in 2008 (and 13,9 % in Medicine) were freely available at the publishers’ sites (gold OA).  For an additional 11,9% free manuscript versions could be found via the green route:  i.e. copies in repositories and web sites (7,8% in Medicine).
As a commenter rightly stated, the lag time is also important, as we would like to have immediate access to recently published research, yet some publishers (37%) impose an access-embargo of 6-12 months or more. (these papers were largely missed as the 2008 OA status was assessed late 2009).

PLOS 2009

The strength of the paper is that it measures  OA prevalence on an article basis, not on calculating the share of journals which are OA: an OA journal generally contains a lower number of articles.
The authors randomly sampled from 1.2 million articles using the advanced search facility of Scopus. They measured what share of OA copies the average researcher would find using Google.

Another paper published in  J Med Libr Assoc (2009) [2], using similar methods as the PLOS survey examined the state of open access (OA) specifically in the biomedical field. Because of its broad coverage and popularity in the biomedical field, PubMed was chosen to collect their target sample of 4,667 articles. Matsubayashi et al used four different databases and search engines to identify full text copies. The authors reported an OA percentage of 26,3 for peer reviewed articles (70% of all articles), which is comparable to the results of Björk et al. More than 70% of the OA articles were provided through journal websites. The percentages of green OA articles from the websites of authors or in institutional repositories was quite low (5.9% and 4.8%, respectively).

In their discussion of the findings of Matsubayashi et al, Björk et al. [1] quickly assessed the OA status in PubMed by using the new “link to Free Full Text” search facility. First they searched for all “journal articles” published in 2005 and then repeated this with the further restrictions of “link to FFT”. The PubMed OA percentages obtained this way were 23,1 for 2005 and 23,3 for 2008.

This proportion of biomedical OA papers is gradually increasing. A chart in Nature’s News Blog [9] shows that the proportion of papers indexed on the PubMed repository each year has increased from 23% in 2005 to above 28% in 2009.
(Methods are not shown, though. The 2008 data are higher than those of Björk et al, who noticed little difference with 2005. The Data for this chart, however, are from David Lipman, NCBI director and driving force behind the digital OA archive PubMed Central).
Again, because of the embargo periods, not all literature is immediately available at the time that it is published.

In summary, we would miss about 70% of biomedical papers by limiting for FFT papers. However, we would miss an even larger proportion of papers if we limit ourselves to recently published ones.

Of course, the key question is whether ignoring relevant studies not available in full text really matters.

Reinhard Wentz of the Imperial College Library and Information Service already argued in a visionary 2002 Lancet letter[3] that the availability of full-text articles on the internet might have created a new form of bias: FUTON bias (Full Text On the Net bias).

Wentz reasoned that FUTON bias will not affect researchers who are used to comprehensive searches of published medical studies, but that it will affect staff and students with limited experience in doing searches and that it might have the same effect in daily clinical practice as publication bias or language bias when doing systematic reviews of published studies.

Wentz also hypothesized that FUTON bias (together with no abstract available (NAA) bias) will affect the visibility and the impact factor of OA journals. He makes a reasonable cause that the NAA-bias will affect publications on new, peripheral, and under-discussion subjects more than established topics covered in substantive reports.

The study of Murali et al [4] published in Mayo Proceedings 2004 confirms that the availability of journals on MEDLINE as FUTON or NAA affects their impact factor.

Of the 324 journals screened by Murali et al. 38.3% were FUTON, 19.1%  NAA and 42.6% had abstracts only. The mean impact factor was 3.24 (±0.32), 1.64 (±0.30), and 0.14 (±0.45), respectively! The authors confirmed this finding by showing a difference in impact factors for journals available in both the pre and the post-Internet era (n=159).

Murali et al informally questioned many physicians and residents at multiple national and international meetings in 2003. These doctors uniformly admitted relying on FUTON articles on the Web to answer a sizable proportion of their questions. A study by Carney et al (2004) [5] showed  that 98% of the US primary care physicians used the Internet as a resource for clinical information at least once a week and mostly used FUTON articles to aid decisions about patient care or patient education and medical student or resident instruction.

Murali et al therefore conclude that failure to consider FUTON bias may not only affect a journal’s impact factor, but could also limit consideration of medical literature by ignoring relevant for-fee articles and thereby influence medical education akin to publication or language bias.

This proposed effect of the FFT limit on citation retrieval for clinical questions, was examined in a  more recent study (2008), published in J Med Libr Assoc [6].

Across all 4 questions based on a research agenda for physical therapy, the FFT limit reduced the number of citations to 11.1% of the total number of citations retrieved without the FFT limit in PubMed.

Even more important, high-quality evidence such as systematic reviews and randomized controlled trials were missed when the FFT limit was used.

For example, when searching without the FFT limit, 10 systematic reviews of RCTs were retrieved against one when the FFT limit was used. Likewise when searching without the FFT limit, 28 RCTs were retrieved and only one was retrieved when the FFT limit was used.

The proportion of missed studies (appr. 90%) is higher than in the studies mentioned above. Possibly this is because real searches have been tested and that only relevant clinical studies  have been considered.

The authors rightly conclude that consistently missing high-quality evidence when searching clinical questions is problematic because it undermines the process of Evicence Based Practice. Krieger et al finally conclude:

“Librarians can educate health care consumers, scientists, and clinicians about the effects that the FFT limit may have on their information retrieval and the ways it ultimately may affect their health care and clinical decision making.”

It is the hope of this librarian that she did a little education in this respect and clarified the point that limiting to free full text might not always be a good idea. Especially if the aim is to critically appraise a topic, to educate or to discuss current best medical practice.

References

  1. Björk, B., Welling, P., Laakso, M., Majlender, P., Hedlund, T., & Guðnason, G. (2010). Open Access to the Scientific Journal Literature: Situation 2009 PLoS ONE, 5 (6) DOI: 10.1371/journal.pone.0011273
  2. Matsubayashi, M., Kurata, K., Sakai, Y., Morioka, T., Kato, S., Mine, S., & Ueda, S. (2009). Status of open access in the biomedical field in 2005 Journal of the Medical Library Association : JMLA, 97 (1), 4-11 DOI: 10.3163/1536-5050.97.1.002
  3. WENTZ, R. (2002). Visibility of research: FUTON bias The Lancet, 360 (9341), 1256-1256 DOI: 10.1016/S0140-6736(02)11264-5
  4. Murali NS, Murali HR, Auethavekiat P, Erwin PJ, Mandrekar JN, Manek NJ, & Ghosh AK (2004). Impact of FUTON and NAA bias on visibility of research. Mayo Clinic proceedings. Mayo Clinic, 79 (8), 1001-6 PMID: 15301326
  5. Carney PA, Poor DA, Schifferdecker KE, Gephart DS, Brooks WB, & Nierenberg DW (2004). Computer use among community-based primary care physician preceptors. Academic medicine : journal of the Association of American Medical Colleges, 79 (6), 580-90 PMID: 15165980
  6. Krieger, M., Richter, R., & Austin, T. (2008). An exploratory analysis of PubMed’s free full-text limit on citation retrieval for clinical questions Journal of the Medical Library Association : JMLA, 96 (4), 351-355 DOI: 10.3163/1536-5050.96.4.010
  7. The #TwitJC Twitter Journal Club, a new Initiative on Twitter. Some Initial Thoughts. (laikaspoetnik.wordpress.com)
  8. The Second #TwitJC Twitter Journal Club (laikaspoetnik.wordpress.com)
  9. How many research papers are freely available? (blogs.nature.com)




The University Library (UBA) goes Mobile.

4 04 2010
UBA mobielOur Medical Library at the AMC hospital is one of main (autonomous) libraries of the UBA, the University Library of the University of Amsterdam.

The UBA developed the Spoetnik (library 23 things-like) course -inspiring the start of this blog-, has a library-coach with chat function, a library blog (UBA-e), and is now on Twitter as @bibliotheekuva.
Plus, as I just learned, a small team of the UBA recently launched a mobile version of the library website.

I like their approach. This team consisting of Driek Heesakkers (project leader), Lukas Koster, Gre Ootjers, Roxana Popistasu en Alice Doek, realized this “perpetual beta version” in no more than 7 weeks (from first meeting till launch at April 1st). There aim was not to strive for perfection, but to develop a version first and to learn from their mistakes and the feedback from the users. Thus highly interactive.

Another excellent principle was that they designed ONE mobile app for all smart phones.

This is what UBA mobile offers right now:

  • The library catalog (searching; reserve items; renew loans)
  • Opening hours and addresses of library locations
  • Locations (on a map)
  • Contact phone numbers
  • Questions, feedback
  • News via @bibliotheekuva-tweets

The most important feature, full access to the digital library (with link to all subscriptions) is not yet realized.

I hope our medical library will follow this shining example. Many medical students and doctors use smart-phones and I’m sure a digital version of our medical library website would surely be appreciated by our clients.

Mobile is the future. What do you think?

Below a short and clear presentation by Lukas Koster at UGUL (UGame ULearn) 2010.

The web address of the mobile site is: http://cf.uba.uva.nl/mobiel.

Short notice about UBA mobile at the news section of the UBA.

Janneke Staaks (librarian for: Psychology, Cultural Anthropology and Pedagogical and Educational Sciences) has dealt more in depth with this subject. See this post at her (Dutch) blog FMG Library.





Auteursrecht nekt de digitale bibliotheek

5 05 2008

Via bakkel dot com een bespreking gelezen: “Auteursrecht maakt meer kapot dan je lief is” van Arjen Kamphuis naar aanleiding van een interessant artikel in het NRC van Martin Bossenbroek en Hans Jansen van de KB ( Auteurswet nekt de digitale bibliotheek).

Het auteursrecht eist dat tot tientallen jaren later nog toestemming wordt gevraagd voor hergebruik. Dit dient ter bescherming van auteurs, maar werkt in deze tijd van digitalisering averechts. Alleen al het achterhalen van (alle) auteurs is een schier onmogelijke klus als het om een verweesd werk (auteurs onbekend of onvindbaar) of krant/tijdschrift gaat (freelance journalisten, fotografen, tekenaars, schrijvers, advertentieontwerpers en niet te vergeten ingezondenbrievenschrijvers).

Uiteindelijk biedt het auteursrecht ook voor de auteur geen bescherming maar louter uitzicht op vergetelheid, aldus Bossenbroek en Jansen. Dat kan toch niet de bedoeling zijn.

Overigens heeft volgens een van de commentaren op het blog van Arjen Kamphuis (#7) kleine webloggers ook te lijden onder auteursrechten. Het opsporen van auteursrechtschendingen wordt door sommige auteurs gezien als het nieuwe brood op de plank voor de auteur om eens wat ‘extra te verdienen op artikelen’.

Ik moet zeggen dat ik er zelf ternauwernood op let. Qua tekst refereer ik netjes mijn bronnen (indien inhoudelijk relevant), maar plaatjes pluk ik her en der van het net. Een tijdje geleden sloeg de schrik mij om het hart toen Google zoekacties op “Laika Spoetnik” bepaalde zoekresultaten steevast niet toonden, omdat hier het auteursrecht geschonden zou zijn. Het zal je in alle onschuld (lees naïviteit) maar overkomen…

Het volgende slot heb ik zeer toepasselijk overgenomen van Arjen Kamphuis:


Dit werk valt, net als al mijn publieke werk, onder een
Creative Commons licentie. Want het grootste risico dat een auteur vandaag de dag kan lopen is niet dat je werk ‘gestolen’ wordt maar dat je werk niet gelezen wordt. Share & enjoy!