To Retract or Not to Retract… That’s the Question

7 06 2011

In the previous post I discussed [1] that editors of Science asked for the retraction of a paper linking XMRV retrovirus to ME/CFS.

The decision of the editors was based on the failure of at least 10 other studies to confirm these findings and on growing support that the results were caused by contamination. When the authors refused to retract their paper, Science issued an Expression of Concern [2].

In my opinion retraction is premature. Science should at least await the results of two multi-center studies, that were designed to confirm or disprove the results. These studies will continue anyway… The budget is already allocated.

Furthermore, I can’t suppress the idea that Science asked for a retraction to exonerate themselves for the bad peer review (the paper had serious flaws) and their eagerness to swiftly publish the possibly groundbreaking study.

And what about the other studies linking the XMRV to ME/CFS or other diseases: will these also be retracted?
And what happens in the improbable case that the multi-center studies confirm the 2009 paper? Would Science republish the retracted paper?

Thus in my opinion, it is up to other scientists to confirm or disprove findings published. Remember that falsifiability was Karl Popper’s basic scientific principle. My conclusion was that “fraud is a reason to retract a paper and doubt is not”. 

This is my opinion, but is this opinion shared by others?

When should editors retract a paper? Is fraud the only reason? When should editors issue a letter of concern? Are there guidelines?

Let first say that even editors don’t agree. Schekman, the editor-in chief of PNAS, has no direct plans to retract another paper reporting XMRV-like viruses in CFS [3].

Schekman considers it “an unusual situation to retract a paper even if the original findings in a paper don’t hold up: it’s part of the scientific process for different groups to publish findings, for other groups to try to replicate them, and for researchers to debate conflicting results.”

Back at the Virology Blog [4] there was also a vivid discussion about the matter. Prof. Vincent Ranciello gave the following answer in response to a question of a reader:

I don’t have any hard numbers on how often journals ask scientists to retract a paper, only my sense that it is very rare. Author retractions are more frequent, but I’m only aware of a handful of those in a year. I can recall a few other cases in which the authors were asked to retract a paper, but in those cases scientific fraud was involved. That’s not the case here. I don’t believe there is a standard policy that enumerates how such decisions are made; if they exist they are not public.

However, there is a Guideline for editors, the Guidance from the Committee on Publication Ethics (COPE) (PDF) [5]

Ivanoranski, of the great blog Retraction Watch, linked to it when we discussed reasons for retraction.

With regard to retraction the COPE-guidelines state that journal editors should consider retracting a publication if:

  1. they have clear evidence that the findings are unreliable, either as a result of misconduct (e.g. data fabrication) or honest error (e.g. miscalculation or experimental error)
  2. the findings have previously been published elsewhere without proper crossreferencing, permission or justification (i.e. cases of redundant publication)
  3. it constitutes plagiarism
  4. it reports unethical research

According to the same guidelines journal editors should consider issuing an expression of concern if:

  1. they receive inconclusive evidence of research or publication misconduct by the authors 
  2. there is evidence that the findings are unreliable but the authors’ institution will not investigate the case 
  3. they believe that an investigation into alleged misconduct related to the publication either has not been, or would not be, fair and impartial or conclusive 
  4. an investigation is underway but a judgement will not be available for a considerable time

Thus in the case of the Science XMRV/CSF paper an expression of concern certainly applies (all 4 points) and one might even consider a retraction, because the results seem unreliable (point 1). But it is not 100%  established that the findings are false. There is only serious doubt……

The guidelines seem to leave room for separate decisions. To retract a paper in case of plain fraud is not under discussion. But when is an error sufficiently established ànd important to warrant retraction?

Apparently retractions are on the rise. Although still rare (0.02% of all publications by the late 2000s) there has been a tenfold increase in retractions compared to the early 1980s (see review at Scholarly Kitchen [6] about two papers: [7] and [8]). However it is unclear whether increasing rates of retraction reflect more fraudulent or erroneous papers or a better diligence. The  first paper [7] also highlights that, out of fear of litigation, editors are generally hesitant to retract an article without the author’s permission.

At the blog Nerd Alert they give a nice overview [9] (based on Retraction Watch, but then summarized in one post ;) ) . They clarify that papers are retracted for “less dastardly reasons then those cases that hit the national headlines and involve purposeful falsification of data”, such as the fraudulent papers of Andrew Wakefield (autism caused by vaccination). Besides the mistaken publication of the same paper twice, data over-interpretation, plagiarism and the like, the reason can also be more trivial: ordering the wrong mice or using an incorrectly labeled bottle.

Still, scientist don’t unanimously agree that such errors should lead to retraction.

Drug Monkey blogs about his discussion [10] with @ivanoransky over a recent post at Retraction Watch, which asks whether a failure to replicate a result justifies a retraction [11]“. Ivanoransky presents a case, where a researcher (B) couldn’t reproduce the findings of another lab (A) and demonstrated mutations in the published protein sequence that excluded the mechanism proposed in A’s paper. This wasn’t retracted, possibly because B didn’t follow the published experimental protocols of A in all details. (reminds me of the XMRV controversy). 

Drugmonkey says (quote):  (cross-posted at Scientopia here – hmmpf isn’t that an example of redundant publication?)

“I don’t give a fig what any journals might wish to enact as a policy to overcompensate for their failures of the past.
In my view, a correction suffices” (provided that search engines like Google and PubMed make clear that the paper was in fact corrected).

Drug Monkey has a point there. A clear watermark should suffice.

However, we should note that most papers are retracted by authors, not the editors/journals, and that the majority of “retracted papers” remain available. Just 13.2% are deleted from the journal’s website. And 31% are not clearly labelled as such.

Summary of how the naïve reader is alerted to paper retraction (from Table 2 in [7], see: Scholarly Kitchen [6])

  • Watermark on PDF (41.1%)
  • Journal website (33.4%)
  • Not noted anywhere (31.8%)
  • Note appended to PDF (17.3%)
  • PDF deleted from website (13.2%)

My conclusion?

Of course fraudulent papers should be retracted. Also papers with obvious errors that invalidate the conclusions.

However, we should be extremely hesitant to retract papers that can’t be reproduced, if there is no undisputed evidence of error.

Otherwise we should retract almost all published papers at one point or another. Because if Professor Ioannides is right (and he probably is) “Much of what medical researchers conclude in their studies is misleading, exaggerated, or flat-out wrong”. ( see previous post [12],  “Lies, Damned Lies, and Medical Science” [13])  and Ioannides’ crushing article “Why most published research findings are false [14]”)

All retracted papers (and papers with major deficiencies and shortcomings) should be clearly labeled as such (as Drugmonkey proposed, not only at the PDF and at the Journal website, but also by search engines and biomedical databases).

Or lets hope, with Biochembelle [15], that the future of scientific publishing will make retractions for technical issues obsolete (whether in the form of nano-publications [16] or otherwise):

One day the scientific community will trade the static print-type approach of publishing for a dynamic, adaptive model of communication. Imagine a manuscript as a living document, one perhaps where all raw data would be available, others could post their attempts to reproduce data, authors could integrate corrections or addenda….

NOTE: Retraction Watch (@ivanoransky) and @laikas have voted in @drugmonkeyblog‘s poll about what a retracted paper means [here]. Have you?

References

  1. Science Asks to Retract the XMRV-CFS Paper, it Should Never Have Accepted in the First Place. (laikaspoetnik.wordpress.com 2011-06-02)
  2. Alberts B. Editorial Expression of Concern. Science. 2011-05-31.
  3. Given Doubt Cast on CFS-XMRV Link, What About Related Research? (blogs.wsj.com)
  4. XMRV is a recombinant virus from mice  (Virology Blog : 2011/05/31)
  5. Retractions: Guidance from the Committee on Publication Ethics (COPE) Elizabeth Wager, Virginia Barbour, Steven Yentis, Sabine Kleinert on behalf of COPE Council:
    http://www.publicationethics.org/files/u661/Retractions_COPE_gline_final_3_Sept_09__2_.pdf
  6. Retract This Paper! Trends in Retractions Don’t Reveal Clear Causes for Retractions (scholarlykitchen.sspnet.org)
  7. Wager E, Williams P. Why and how do journals retract articles? An analysis of Medline retractions 1988-2008. J Med Ethics. 2011 Apr 12. [Epub ahead of print] 
  8. Steen RG. Retractions in the scientific literature: is the incidence of research fraud increasing? J Med Ethics. 2011 Apr;37(4):249-53. Epub 2010 Dec 24.
  9. Don’t touch that blot. (nerd-alert.net/blog/weeklies/ : 2011/02/25)
  10. What_does_a_retracted_paper_mean? (scienceblogs.com/drugmonkey: 2011/06/03)
  11. So when is a retraction warranted? The long and winding road to publishing a failure to replicate (retractionwatch.wordpress.com : 2011/06/03/)
  12. Much Ado About ADHD-Research: Is there a Misrepresentation of ADHD in Scientific Journals? (laikaspoetnik.wordpress.com 2011-06-02)
  13. “Lies, Damned Lies, and Medical Science” (theatlantic.com :2010/11/)
  14. Ioannidis, J. (2005). Why Most Published Research Findings Are False. PLoS Medicine, 2 (8) DOI: 10.1371/journal.pmed.0020124
  15. Retractions: What are they good for? (biochembelle.wordpress.com : 2011/06/04/)
  16. Will Nano-Publications & Triplets Replace The Classic Journal Articles? (laikaspoetnik.wordpress.com 2011-06-02)

NEW* (Added 2011-06-08):

 





Science Asks to Retract the XMRV-CFS Paper, it Should Never Have Accepted in the First Place.

2 06 2011

Wow! Breaking!

As reported in WSJ earlier this week [1], editors of the journal Science asked Mikovits and her co-authors to voluntary retract their 2009 Science paper [2].

In this paper Mikovits and colleagues of the Whittemore Peterson Institute (WPI) and the Cleveland Clinic, reported the presence of xenotropic murine leukemia virus–related virus (XMRV) in peripheral blood mononuclear cells (PBMC) of patients with chronic fatigue syndrome (CFS). They used the very contamination-prone nested PCR to detect XMRV. This 2 round PCR enables detection of a rare target sequence by producing an unimaginable huge number of copies of that sequence.
XMRV was first demonstrated in cell lines and tissue samples of prostate cancer patients.

All the original authors, except for one [3], refused to retract the paper [4]. This prompted Science editor-in-chief Bruce Alberts to  issue an Expression of Concern [5], which was published two days earlier than planned because of the early release of the news in WSJ, mentioned above [1]. (see Retraction Watch [6]).

The expression of concern also follows the publication of two papers in the same journal.

In the first Science paper [7] Knox et al. found no Murine-Like Gammaretroviruses in any of the 61 CFS Patients previously identified as XMRV-positive, using the same PCR and culturing techniques as used by Lombardi et al. This paper made ERV (who consistently critiqued the Lombardi paper from the startlaugh-out-loud [8], because Knox also showed that human sera neutralize the virus in the blood,indicating it can hardly infect human cells in vivo. Knox also showed the WPIs sequences to be similar to the XMRV plasmid VP62, known to often contaminate laboratory agents.*

Contamination as the most likely reason for the positive WPI-results is also the message of the second Science paper. Here, Paprotka et al. [9]  show that XMRV was not present in the original prostate tumor that gave rise to the XMRV-positive 22Rv1 cell line, but originated -as a laboratory artifact- by recombination of two viruses during passaging the cell line in nude mice. For a further explanation see the Virology Blog [10].

Now Science editors have expressed their concern, the tweets, blogposts and health news articles are preponderantly negative about the XMRV findings in CFS/ME, where they earlier were positive or neutral. Tweets like “Mouse virus #XMRV doesn’t cause chronic fatigue #CFS http://t.co/Bekz9RG (Reuters) or “Origins of XMRV deciphered, undermining claims for a role in human disease: Delineation of the origin of… http://bit.ly/klDFuu #cancer” (National Cancer Institute) are unprecedented.

Thus the appeal by Science to retract the paper is justified?

Well yes and no.

The timing is rather odd:

  • Why does Science only express concern after publication of these two latest Science papers? There are almost a dozen other studies that failed to reproduce the WPI-findings. Moreover, 4 earlier papers in Retrovirology already indicated that disease-associated XMRV sequences are consistent with laboratory contamination. (see an overview of all published articles at A Photon in the Darkness [11])
  • There are still (neutral) scientist who believe that genuine human infections with XMRV still exist at a relatively low prevalence. (van der Kijl et al: xmrv is not a mousy virus [12])
  • And why doesn’t Science await the results from the official confirmation studies meant to finally settle whether XMRV exist in our blood supply and/or CFS (by the Blood Working Group and the NIH sponsored study by Lipkin et al.)
  • Why (and this is the most important question) did Science ever decide to publish the piece in the first place, as the study had several flaws.
I do believe that new research that turns existing paradigms upside down deserves a chance. Also a chance to get disproved. Yes such papers might be published in prominent scientific journals like Science, provided they are technically and methodologically sound at the very least. The Lombardi paper wasn’t.

Here I repeat my concerns expressed in earlier posts [13 and 14]. (please read these posts first, if you are unfamiliar with PCR).

Shortcomings in PCR-technique and study design**:

  • No positive control and no demonstration of the sensitivity of the PCR-assay. Usually a known concentration or a serial dilution of a (weakly) positive sample is taken as control. This allows to determine sensitivity of the assay.
  • Aspecific bands in negative samples (indicating suboptimal conditions)
  • Just one vial without added DNA per experiment as a negative control. (Negative controls are needed to exclude contamination).
  • CFS-Positive and negative samples are on separate gels (this increases bias, because conditions and chance of contamination are not the same for all samples, it also raises the question whether the samples were processed differently)
  • Furthermore only results obtained at the Cleveland Clinic are shown. (were similar results not obtained at the WPI? see below)
Contamination not excluded as a possible explanation
  • No variation in the XMRV-sequences detected (expected if the findings are real)
  • Although the PCR is near the detection limit, only single round products are shown. These are much stronger then expected even after two rounds. This is very confusing, because WPI later exclaimed that preculturing PBMC plus nested PCR (2 rounds) were absolutely required to get a positive result. But the Legend of Fig. 1 in the original Science paper clearly says PCR after one round. Strong (homogenous) bands after one round of PCR are highly suggestive of contamination.
  • No effort to exclude contamination of samples with mouse DNA (see below)
  • No determination of the viral DNA integration sites.

Mikovits also stressed that she never used the XMRV-positive cell lines in 2009. But what about the Cleveland Clinic, nota bene the institute that co-discovered XMRV and that had produced the strongly positive PCR-products (…after a single PCR-round…)?

On the other hand, the authors had other proof of the presence of retrovirus: detection of (low levels of) antibodies to XMRV in patient sera, and transmissibility of XMRV. On request they later applied the mouse mitochondrial assay to successfully exclude the presence of mouse DNA in their samples. (but this doesn’t exclude all forms of contamination, and certainly not at Cleveland Clinic)

These shortcomings alone should have been sufficient for the reviewers, had they seen it and /or deemed it of sufficient importance, to halt publication and to ask for additional studies**.

I was once in a similar situation. I found a rare cancer-specific chromosomal translocation in normal cells, but I couldn’t exclude PCR- contamination. The reviewers asked me to exclude contamination by sequencing the breakpoints, which only succeeded after two years of extra work. In retrospect I’m thankful to the reviewers for preventing me from publishing a possible faulty paper which could have ruined my career (yeah, because contamination is a real problem in PCR). And my paper improved tremendously by the additional experiments.

Yes it is peer review that failed here, Science. You should have asked for extra confirmatory tests and a better design in the first place. That would have spared a lot of anguish, and if the findings had been reproducible, more convincing and better data.

There were a couple of incidents after the study was published, that made me further doubt the robustness of WPI’s scientific data and even (after a while) I began to doubt whether WPI, and Judy Mikovits in particular, is adhering to good scientific (and ethical) practices.

  • WPI suddenly disclosed (Feb 18 2010) that culturing PBMC’s is necessary to obtain a positive PCR signal.  As a matter of fact they maintain this in their recent protest letter to Science. They refer to the original Science paper, but this paper doesn’t mention the need for culturing at all!! 
  • WPI suggests their researchers had detected XMRV in patient samples from both Dr. Kerr’s and Dr. van Kuppeveld’s ‘XMRV-negative’ CFS-cohorts. Thus in patient samples obtained without a culture-enrichment step…..  There can only be one truth:  main criticism on negative studies was that improper CFS-criteria were used. Thus either this CFS-population is wrongly defined and DOESN’t contain XMRV (with any method), OR it fulfills the criteria of CFS and the XMRV can be detected applying the proper technique. It is so confusing!..
  • Although Mikovits first reported that they found no to little virus variation, they later exclaimed to find a lot of variation.
  • WPI employees behave unprofessional towards colleague-scientists who failed to reproduce their findings.
Other questionable practices 
  • Mikovits also claims that people with autism harbor XMRV. One wonders which disease ISN’t associated with XMRV….
  • Despite the uncertainties about XMRV in CFS-patients, let alone the total LACK of demonstration of a CAUSAL RELATIONSHIP, Mikovits advocates the use of *not harmless* anti-retrovirals by CFS-patients.
  • At this stage of controversy, the WPI-XMRV test is sold as “a reliable diagnostic tool“ by a firm (VIP Dx) with strong ties to WPI. Mikovits even tells patients in a mail: “First of all the current diagnostic testing will define with essentially 100% accuracy! XMRV infected patients”. WTF!? 
  • This test is not endorsed in Belgium, and even Medicare only reimbursed 15% of the PCR-test.
  • The ties of WPI to RedLabs & VIP Dx are not clearly disclosed in the Science Paper. There is only a small Note (added in proof!)  that Lombardi is operations manager of VIP Dx, “in negotiations with the WPI to offer a diagnostic test for XMRV”.
Please see this earlier post [13] for broader coverage. Or read the post [16] of Keith Grimaldi, scientific director of Eurogene, and expert in personal genomics, who I asked to comment on the “diagnostic” tests. In his post he very clearly describes “what is exactly wrong about selling an unregulated clinical test  to a very vulnerable and exploitable group based on 1 paper on a small isolated sample”.

It is really surprising this wasn’t picked up by the media, by the government or by the scientific community. Will the new findings have any consequences for the XMRV-diagnostic tests? I fear WPI will get away with it for the time being. I agree with Lipkin, who coordinates the NIH-sponsored multi-center CFS-XMRV study that calls to retract the paper are premature at this point . Furthermore, -as addressed by WSJ [17]- if the Science paper is retracted, because XMRV findings are called into question, what about the papers also reporting a  link of XMRV-(like) viruses and CFS or prostate cancer?

WSJ reports, that Schekman, the editor-in chief of PNAS, has no direct plans to retract the paper of Alter et al reporting XMRV-like viruses in CFS [discussed in 18]. Schekman considers it “an unusual situation to retract a paper even if the original findings in a paper don’t hold up: it’s part of the scientific process for different groups to publish findings, for other groups to try to replicate them, and for researchers to debate conflicting results.”

I agree, this is a normal procedure, once the paper is accepted and published. Fraud is a reason to retract a paper, doubt is not.

Notes

*samples, NOT patients, as I saw a patient erroneous interpretation: “if it is contamination in te lab how can I have it as a patient?” (tweet is now deleted). No, according to the contamination -theory” XMRV-contamination is not IN you, but in the processed samples or in the reaction mixtures used.

** The reviewers did ask additional evidence, but not with respect to the PCR-experiments, which are most prone to contamination and false results.

  1. Chronic-Fatigue Paper Is Questioned (online.wsj.com)
  2. Lombardi VC, Ruscetti FW, Das Gupta J, Pfost MA, Hagen KS, Peterson DL, Ruscetti SK, Bagni RK, Petrow-Sadowski C, Gold B, Dean M, Silverman RH, & Mikovits JA (2009). Detection of an infectious retrovirus, XMRV, in blood cells of patients with chronic fatigue syndrome. Science (New York, N.Y.), 326 (5952), 585-9 PMID: 19815723
  3. WPI Says No to Retraction / Levy Study Dashes Hopes /NCI Shuts the Door on XMR (phoenixrising.me)
  4. http://wpinstitute.org/news/docs/FinalreplytoScienceWPI.pdf
  5. Alberts B. Editorial Expression of Concern. Science. 2011 May 31.
  6. Science asks authors to retract XMRV-chronic fatigue paper; when they refuse, issue Expression of Concern. 2011/05/31/ (retractionwatch.wordpress.com)
  7. K. Knox, Carrigan D, Simmons G, Teque F, Zhou Y, Hackett Jr J, Qiu X, Luk K, Schochetman G, Knox A, Kogelnik AM & Levy JA. No Evidence of Murine-Like Gammaretroviruses in CFS Patients Previously Identified as XMRV-Infected. Science. 2011 May 31. (10.1126/science.1204963).
  8. XMRV and chronic fatigue syndrome: So long, and thanks for all the lulz, Part I [erv] (scienceblogs.com)
  9. Paprotka T, Delviks-Frankenberry KA, Cingoz O, Martinez A, Kung H-J, Tepper CG, Hu W-S , Fivash MJ, Coffin JM, & Pathak VK. Recombinant origin of the retrovirus XMRV. Science. 2011 May 31. (10.1126/science.1205292).
  10. XMRV is a recombinant virus from mice  (Virology Blog : 2011/05/31)
  11. Science asks XMRV authors to retract paper (photoninthedarkness.com : 2011/05/31)
  12. van der Kuyl AC, Berkhout B. XMRV: Not a Mousy Virus. J Formos Med Assoc. 2011 May;110(5):273-4. PDF
  13. Finally a Viral Cause of Chronic Fatigue Syndrome? Or Not? – How Results Can Vary and Depend on Multiple Factor (laikaspoetnik.wordpress.com: 2010/02/15/)
  14. Three Studies Now Refute the Presence of XMRV in Chronic Fatigue Syndrome (CFS) (laikaspoetnik.wordpress.com 2010/04/27)
  15. WPI Announces New, Refined XMRV Culture Test – Available Now Through VIP Dx in Reno (prohealth.com 2010/01/15)
  16. The murky side of physician prescribed LDTs (eurogene.blogspot.com : 2010/09/06)
  17. Given Doubt Cast on CFS-XMRV Link, What About Related Research? (blogs.wsj.com)
  18. Does the NHI/FDA Paper Confirm XMRV in CFS? Well, Ditch the MR and Scratch the X… and… you’ve got MLV. (laikaspoetnik.wordpress.com : 2010/08/30/)

Related articles





Three Studies Now Refute the Presence of XMRV in Chronic Fatigue Syndrome (CFS)

27 04 2010

ResearchBlogging.org.“Removing the doubt is part of the cure” (RedLabs)

Two months ago I wrote about two contradictory studies on the presence of the novel XMRV retrovirus in blood of patients with Chronic Fatigue Syndrome (CFS).

The first study, published in autumn last year by investigators of the Whittemore Peterson Institute (WPI) in the USA [1], claimed to find XMRV virus in peripheral blood mononuclear cells (PBMC) of patients with CFS. They used PCR and several other techniques.

A second study, performed in the UK [2] failed to show any XMRV-virus in peripheral blood of CFS patients.

Now there are two other negative studies, one from the UK [3] and one from the Netherlands [4].

Does this mean that XMRV is NOT present in CFS patients?

No, different results may still be due do to different experimental conditions and patient characteristics.

The discrepancies between the studies discussed in the previous post remain, but there are new insights, that I would like to share.*

1. Conflict of Interest, bias

Most CFS patients seem “to go for” WPI, because WPI, established by the family of a chronic fatigue patient, has a commitment to CFS. CFS patients feel that many psychiatrists, including authors of the negative papers [2-4] dismiss CFS as something “between the ears”.  This explains the negative attitude against these “psych-healers” on ME-forums (i.e. the Belgium forum MECVS.net and http://www.forums.aboutmecfs.org/). MECVS even has a section “faulty/wrong” papers, i.e. about the “failure” of psychiatrists to demonstrate  XMRV!

Since a viral (biological) cause would not fit in the philosophy of these psychiatrists, they might just not do their best to find the virus. Or even worse…

Dr. Mikovits, co-author of the first paper [1] and Director of Research at WPI, even responded to the first UK study as follows (ERV and Prohealth):

“You can’t claim to replicate a study if you don’t do a single thing that we did in our study,” …
“They skewed their experimental design in order to not find XMRV in the blood.” (emphasis mine)

Mikovits also suggested that insurance companies in the UK are behind attempts to sully their findings (ERV).

These kind of personal attacks are “not done” in Science. And certainly not via this route.

Furthermore, WPI has its own bias.

For one thing WPI is dependent on CFS and other neuro-immune patients for its existence.

WPI has generated numerous press releases and doesn’t seem to use the normal scientific channels. Mikovits presented a 1 hr Q&A session about XMRV and CFS (in a stage where nothing has been proven yet). She will also present data about XMRV at an autism meeting. There is a lot of PR going on.

Furthermore there is an intimate link  between WPI and VIP Dx, both housed in Reno. Vip DX is licensed by WPI to provide the XMRV-test. Vipdx.com links to the same site as redlabsusa.com, for Vip Dx is the new name of the former RedLabs.

Interestingly Lombardi (the first author of the paper) co-founded Redlabs USA Inc. and  served as the Director of Operations at Redlabs, Harvey Whittemore owns 100% of VIP Dx, and was the company President until this year and  Mikovits is the Vice President of VIP Dx. (ME-forum). They didn’t disclose this in the Science paper.

TEST_RQN_Feb_2010

Vip/Dx offers a plethora of tests, and is the only RedLab -branch that performs the WPI-PCR test, now replaced by the “sensitive” culture test (see below). At this stage of controversy, the test is sold as “a reliable diagnostic tool“(according to prohealth). Surely their motto “Removing the doubt is part of the cure” appeals to patients. But how can doubt be removed if the association of XMRV with CFS has not been confirmed, the diagnostic tests offered have yet not been truly validated (see below), as long as a causal relationship between XMRV and CFS has not been proven and/or when XMRV does not seem that specific for CFS: it has also been found in people with prostate cancer, autism,  atypical multiple sclerosis, fibromyalgia, lymphoma)(WSJ).

Meanwhile CFS/ME websites are abuzz with queries about how to obtain tests -also in Europe- …and antiretroviral drugs. Sites like Prohealth seem to advocate for WPI. There is even a commercial XMRV site (who runs it is unclear)

Project leader Mikovits, and the WPI as a whole, seem to have many contacts with CSF patients, also by mail. In one such mail she says (emphasis and [exclamations] mine):

“First of all the current diagnostic testing will define with essentially 100% accuracy! XMRV infected patients”. [Bligh me!]….
We are testing the hypothesis that XMRV is to CFS as HIV is to AIDS. There are many people with HIV who don’t have AIDS (because they are getting treatment). But by definition if you have ME you must have XMRV. [doh?]
[....] There is so much that we don’t know about the virus. Recall that the first isolation of HIV was from a single AIDS patient published in late 1982 and it was not until 2 years later that it was associated with AIDS with the kind of evidence that we put into that first paper. Only a few short years later there were effective therapies. [...]. Please don’t hesitate to email me directly if you or anyone in the group has questions/concerns. To be clear..I do think even if you tested negative now that you are likely still infected with XMRV or its closest cousin..

Kind regards, Judy

These tests costs patients money, because even Medicare will only reimburse 15% of the PCR-test till now. VIP Dx does donate anything above costs to XMRV research, but isn’t this an indirect way to support the WPI-research? Why do patients have to pay for tests that have not proven to be diagnostic? The test is only in the experimental phase.

I ask you: would such an attitude be tolerated from a regular pharmaceutical company?

Patients

Another discrepancy between the WPI and the other studies is that only the WPI use the Fukuda and Canadian criteria to diagnose CFS patients. The Canadian  criteria are much more rigid than those used in the European studies. This could explain why WPI has more positives than the other studies, but it can’t fully explain that WPI shows 96% positives (their recent claim) against 0% in the other studies. For at least some of the European patients should fulfill the more rigid criteria.

Regional Differences

Patients of the positive and negative studies also differ with respect to the region they come from (US and Europe). Indeed, XMRV has previously been detected in prostate cancer cells from American patients, but not from German and Irish patients.

However, the latter two reasons may not be crucial if the statement in the open letter* from Annette Whittemore, director of the WPI, to Dr McClure**, the virologist of the second paper [2], is true:

We would also like to report that WPI researchers have previously detected XMRV in patient samples from both Dr. Kerr’s and Dr. van Kuppeveld’s cohorts prior to the completion of their own studies, as they requested. We have email communication that confirms both doctors were aware of these findings before publishing their negative papers.(……)
One might begin to suspect that the discrepancy between our findings of XMRV in our patient population and patients outside of the United States, from several separate laboratories, are in part due to technical aspects of the testing procedures.

Assuming that this is true we will now concentrate on the differences in the PCR -procedures and results.

PCR

All publications have used PCR to test the presence of XMRV in blood: XMRV is present in such low amounts that you can’t detect the RNA without amplifying it first.

PCR allows the detection of a single or few copies of target DNA/RNA per milligram DNA input, theoretically 1 target DNA copy in 105 to 106 cells. (RNA is first reverse transcribed to DNA). If the target is not frequent, the amplified DNA is only visible after Southern blotting (a radioactive probe “with a perfect fit to” the amplified sequence) or after a second PCR round (so called nested PCR). In this second round a set of primers is used internal to the first set of primers. So a weak signal is converted in a strong and visible one.

All groups have applied nested PCR. The last two studies have also used a sensitive real time PCR, which is more of a quantitative assay and less prone to contamination.

Twenty years ago, I had similar experiences as the WPI. I saw very vague PCR bands that had all characteristics of a tumor-specific sequence in  normal individuals, which was contrary to prevailing beliefs and hard to prove. This had all to do with a target frequency near to the detection limit and with the high chance of contamination with positive controls. I had to enrich tonsils and purified B cells to get a signal and sequence the found PCR products to prove we had no contamination. Data were soon confirmed by others. By the way our finding of a tumor specific sequence in normal individuals didn’t mean that everyone develops lymphoma (oh analogy)

Now if you want to proof you’re right when you discovered something new you better do it good.

Whether a PCR assay at or near the detection limit of PCR is successful depends on:

  • the sensitivity of the PCR
    • Every scientific paper should show the detection limit of the PCR: what can the PCR detect? Is 1 virus particle enough or need there be 100 copies of the virus before it is detected? Preferably the positive control should be diluted in negative cells. This is called spiking. Testing a positive control diluted in water doesn’t reflect the true sensitivity. It is much easier for primers to find one single small piece of target DNA in water than to find that piece of DNA swimming in a pool of DNA from 105 cells. 
  • the specificity of the PCR.
    • You can get aspecific bands if the primers recognize other than the intended sequences. Suppose you have one target sequence competing with a lot of similar sequences, then even a less perfect match in the normal genome has every chance to get amplified. Therefore you should have a negative control of cells not containing the virus (i.e. placental DNA), not only water. This resembles the PCR conditions of your test samples.
  • Contamination
    • this should be prevented by rigorous spatial separation of  sample preparation, PCR reaction assembly, PCR execution, and post-PCR analysis. There should be many negative controls. Control samples should be processed the same way as the experimental samples and should preferably be handled blinded.
  • The quality and properties of your sample.
    • If XMRV is mainly present in PBMC, separation of PBMC by Ficoll separation (from other cells and serum) could make the difference between a positive and a negative signal. Furthermore,  whole blood and other body fluids often contain inhibitors, that may lead to a much lower sensitivity. Purification steps are recommended and presence of inhibitors should be checked by spiking and amplification of control sequences.

Below the results per article. I have also made an overview of the results in a Google spreadsheet.

WPI
The PCR conditions are badly reported in the WPI paper, published in Science[1]. As a matter of fact I wonder how it ever came trough the review.

  • Unlike XMRV-positive prostate cancer cells, XMRV infection status did not not correlate with the RNASEL genotype.
  • The sensitivity of the PCR is not shown (nor discussed).
  • No positive control is mentioned. The negative controls were just vials without added DNA.
  • Although the PCR is near the detection limit, only first round products are shown (without confirmation of the identity of the product). The positive bands are really strong, whereas you expect them to be weak (near the detection limit after two rounds). This is suggestive of contamination.
  • PBMC have been used as a source and that is fine, but one of WPI’s open letters/news items (Feb 18), in response to the first UK study, says the following:
    • point 7. Perhaps the most important issue to focus on is the low level of XMRV in the blood. XMRV is present in such a small percentage of white blood cells that it is highly unlikely that either UK study’s PCR method could detect it using the methods described. Careful reading of the Science paper shows that increasing the amount of the virus by growing the white blood cells is usually required rather than using white blood cells directly purified from the body. When using PCR alone, the Science authors found that four samples needed to be taken at different times from the same patient in order for XMRV to be detected by PCR in freshly isolated white blood cells.(emphasis mine)
  • But carefully reading the methods,  mentioned in the “supporting material” I only read:
    • The PBMC (approximately 2 x 107 cells) were centrifuged at 500x g for 7 min and either stored as unactivated cells in 90% FBS and 10% DMSO at -80 ºC for further culture and analysis or resuspended in TRIzol (…) and stored at -80 ºC for DNA and RNA extraction and analysis. (emphasis mine)

    Either …. or. Seems clear to me that the PBMC were not cultured for PCR, at least not in the experiments described in the science paper.

    How can one accuse other scientists of not “duplicating” the results if the methods are so poorly described and the authors don’t adhere to it themselves??

  • Strikingly only those PCR-reactions are shown, performed by the Cleveland Clinic (using one round), not the actual PCR-data performed by WPI. That is really odd.
  • It is also not clear whether the results obtained by the various tests were consistent.
    Suzanne D. Vernon, PhD, Scientific Director of the CFIDS Association of America (charitable organization dedicated to CFS) has digged deeper into the topic. This is what she wrote [9]:
    Of the 101 CFS subjects reported in the paper, results for the various assays are shown for only 32 CFS subjects. Of the 32 CFS subjects whose results for any of the tests are displayed, 12 CFS subjects were positive for XMRV on more than one assay. The other 20 CFS subjects were documented as positive by just one testing method. Using information from a public presentation at the federal CFS Advisory Committee, four of the 12 CFS subjects (WPI 1118, 1150, 1199 and 1125) included in the Science paper were also reported to have cancer – either lymphoma, mantle cell lymphoma or myelodysplasia. The presentation reported that 17 WPI repository CFS subjects with cancer had tested positive for XMRV. So how well are these CFS cases characterized, really?

The Erlwein study was published within 3 months after the first article. It is simpler in design and was reviewed in less then 3 days. They used whole blood instead of PBMC and performed nested PCR using another set of primers. This doesn’t matter a lot, if the PCR is sensitive. However, the sensitivity of the assay is not shown and the PCR bands of the positive control look very weak, even after the second round (think they mad a mistake in the legend as well: lane 9 is not a positive control but a base pair ladder, I presume). It also looked like they used a “molecular plasmid control in water”, but in the comments on the PLoS ONE paper, one of the authors states that the positive control WAS spiked into patient DNA.(Qetzel commenting to Pipeline Corante) Using this PCR none of the 186 CSF samples was positive.

Groom and van Kuppeveld studies
The two other studies use an excellent PCR approach[3,4]. Both used PBMC, van Kuppeveld used older cryoperserved PBMC. They first tried the primers of Lombardi using a similar nested PCR, but since the sensitivity was low they changed to a real time PCR with other optimized primers. They determined the sensitivity of the PCR by serially diluting a plasmid into PBMC DNA from a healthy donor. The limit of sensitivity equates to 16 and 10 XMRV-gene copies in the UK and the Dutch study respectively. They have appropriate negative controls and controls for the integrity of the material (GAPDH, spiking normal control cDNAs in negative DNA to exclude sample mediated PCR inhibition[1], phocine distemper virus[2]), therefore also excluding that cryopreserved PBMC were not suitable for amplification.

The results look excellent, but none of the PCR-samples were positive using these sensitive techniques. A limitation of the Dutch study is the that numbers of patients and controls were small (32 CSF, 43 controls)

Summary and Conclusion

In a recent publication in Science, Lombardi and co-authors from the WPI reported the detection of XMRV-related, a novel retrovirus that was first identified in prostate cancer samples.

Their main finding, presence of XMRV in peripheral blood cells could not be replicated by 3 other studies, even under sensitive PCR conditions.

The original Science study has severe flaws, discussed above. For one thing WPI doesn’t seem to adhere to the PCR to test XMRV any longer.

It is still possible that XMRV is present in amounts at or near the detection limit. But it is equally possible that the finding is an artifact (the paper being so inaccurate and incomplete). And even if XMRV was reproducible present in CFS patients, causality is still not proven and it is way too far to offer patients “diagnostic tests” and retroviral treatment.

Perhaps the most worrisome part of it all is the non-scientific attitude of WPI-employees towards colleague-scientists, their continuous communication via press releases. And the way they try to directly reach patients, who -i can’t blame them-, are fed up with people not taking them serious and who are longing for a better diagnosis and most of all a better treatment. But this is not the way.

Credits

*Many thanks to Tate (CSF-patient) for alerting me to the last Dutch publication, Q&A’s of WPI and the findings of Mrs Vernon.
- Ficoll blood separation. Photo [CC] http://www.flickr.com/photos/42299655@N00/3013136882/
- Nested PCR: ivpresearch.org

References

  1. Lombardi VC, Ruscetti FW, Das Gupta J, Pfost MA, Hagen KS, Peterson DL, Ruscetti SK, Bagni RK, Petrow-Sadowski C, Gold B, Dean M, Silverman RH, & Mikovits JA (2009). Detection of an infectious retrovirus, XMRV, in blood cells of patients with chronic fatigue syndrome. Science (New York, N.Y.), 326 (5952), 585-9 PMID: 19815723
  2. Erlwein, O., Kaye, S., McClure, M., Weber, J., Wills, G., Collier, D., Wessely, S., & Cleare, A. (2010). Failure to Detect the Novel Retrovirus XMRV in Chronic Fatigue Syndrome PLoS ONE, 5 (1) DOI: 10.1371/journal.pone.0008519
  3. Groom, H., Boucherit, V., Makinson, K., Randal, E., Baptista, S., Hagan, S., Gow, J., Mattes, F., Breuer, J., Kerr, J., Stoye, J., & Bishop, K. (2010). Absence of xenotropic murine leukaemia virus-related virus in UK patients with chronic fatigue syndrome Retrovirology, 7 (1) DOI: 10.1186/1742-4690-7-10
  4. van Kuppeveld, F., Jong, A., Lanke, K., Verhaegh, G., Melchers, W., Swanink, C., Bleijenberg, G., Netea, M., Galama, J., & van der Meer, J. (2010). Prevalence of xenotropic murine leukaemia virus-related virus in patients with chronic fatigue syndrome in the Netherlands: retrospective analysis of samples from an established cohort BMJ, 340 (feb25 1) DOI: 10.1136/bmj.c1018
  5. McClure, M., & Wessely, S. (2010). Chronic fatigue syndrome and human retrovirus XMRV BMJ, 340 (feb25 1) DOI: 10.1136/bmj.c1099
  6. http://scienceblogs.com/erv/2010/01/xmrv_and_chronic_fatigue_syndr_5.php
  7. http://scienceblogs.com/erv/2010/01/xmrv_and_chronic_fatigue_syndr_6.php
  8. http://scienceblogs.com/erv/2010/03/xmrv_and_chronic_fatigue_syndr_11.php
  9. http://www.cfids.org/xmrv/022510study.asp
Sensitivity of PCR screening for XMRV in PBMC DNA. VP62 plasmid was serially diluted 1:10 into PBMC DNA from a healthy donor and tested by Taqman PCR with env 6173 primers and probe. The final amount of VP62 DNA in the reaction was A, 2.3 × 10-2 ng, B, 2.3 × 10-3 ng, C, 2.3 × 10-4 ng, D, 2.3 × 10-5 ng, E, 2.3 × 10-6 ng, F, 2.3 × 10-7 ng or G, 2.3 × 10-8 ng. The limit of sensitivity was 2.3 × 10-7 ng (shown by trace F) which equates to 16 molecules of VP62 XMRV clone.




Finally a Viral Cause of Chronic Fatigue Syndrome? Or Not? – How Results Can Vary and Depend on Multiple Factors

15 02 2010

Last week @F1000 (on Twitter) alerted me to an interesting discussion at F1000 on  a paper in Science, that linked Chronic fatigue syndrome (CFS) to a newly discovered human virus XRMV [1]ResearchBlogging.org.

This finding was recently disputed by another study in PLOS [2], that couldn’t reproduce the results.  This was highlighted in an excellent post by neuroskeptic “Chronic Fatigue Syndrome in “not caused by single virus” shock!

Here is my take on the discrepancy.

Chronic fatigue syndrome (CFS) is a debilitating disorder with unknown etiology. CFS causes extreme fatigue of the kind that does not go  away after a rest. Symptoms of CFS include fatigue for 6 months or more and experiencing other problems such as muscle pain, memory problems, headaches, pain in multiple joints and  sleep problems. Since other illnesses can cause similar symptoms, CFS is hard to diagnose. (source: Medline Plus).

No one knows what causes CFS, but a viral cause has often been suspected, at least in part of the CFS patients. Because the course of the disease often resembles a post-viral fatigue, CFS has also been referred to as post-viral fatigue syndrome (PVFS).

The article of Lombardi [1], published in October 2009 in Science, was a real breakthrough. The study showed that two thirds of patients with CFS were infected with a novel gamma retrovirus, xenotropic murine leukaemia virus-related virus (XMRV). XMVR was previously linked to prostate cancer.

Lombardi et al  isolated DNA from white blood cells (Peripheral Blood Mononuclear Cells or PBMCs) and assayed the samples for XMRV gag sequences by nested polymerase chain reaction (PCR).

The PCR is a technique that allows the detection of a single or few copies of target DNA by amplifying it across several orders of magnitude, generating thousands to millions of copies of a particular DNA. Nested PCR amplifies the resultant amplicon several orders of magnitude further. In the first round external primers are used (short DNA-sequences that fit the outer end of the piece of DNA to be amplified) and an internal set of primers is used for the second round. Nested PCR is often used if the target DNA is not abundantly present and to avoid the comtamination with products that are amplified as a spin-off due to the amplification of artifacts (sites to which the primers bind as well)

[I used a similar approach 15-20 years ago to identify a lymphoma-characteristic translocation in tonsils and purified B cells of (otherwise) healthy individuals. By direct sequencing I could prove that each sequence was unique in its breakpoint sequence, thereby excluding that the PCR-products arose by contamination of an amplified positive control. All tumor cells had the translocation against one in 100,000 or 1,000,000 normal cells. To be able to detect the oncogene in B cells, B cells had to be purified by FACS. Otherwise the detection limit could not be reached]

Lombardi could detect XMRV gag DNA in 68 of 101 patients (67%) as compared to 8 of 218 (3.7%) healthy controls. Detection of gag as well as env XMRV was confirmed in 7 of 11 CFS samples at the Cleveland Clinic (remarkably these are only shown in Fig 1A of the paper, thus not the original PCR-results).
In contrast, XMRV gag sequences were detected in 8 of 218 (3.7%) PBMC DNA specimens from healthy individuals. Of the 11 healthy control DNA samples analyzed by PCR, only one sample was positive for gag and none for env. The XMRV gag and env sequences were more than 99% similar to those previously reported for prostate tumor–associated strains of XMRV. The authors see this as proof against contamination of samples with prostate cancer associated XMRV-DNA.

Not only PCR experiments were done. Using intracellular flow cytometry and Western blot assays XMRV proteins were found to be expressed in PBMCs from CFS patients. CFS patiens had anti-XMRV antibodies and cell culture experiments revealed that patient-derived XMRV was infectious. These findings are consistent with but do not prove that XMRV may be a contributing factor in the pathogenesis of CFS. XMRV might just be an innocent bystander. However, unlike XMRV-positive prostate cancer cells, XMRV infection status did not not correlate with the RNASEL genotype.

The Erlwein study was published within 3 months after the first article. It is much simpler in design. DNA was extracted from whole blood (not purified white blood cells) and subjected to a nested PCR using another set of primers. The positive control was an end-point dilution of the plasmid. Water served as a negative control. None of the 186 CSF samples was positive.

The question then is: which study is true? (although it should be stressed that the Science paper just shows a link between the virus and CFS, not a causal relationship)

Regional Differences

Both findings could be “real” if there was a regional difference in occurrence of the virus. Indeed XMRV has previously been detected in prostate cancer cells from American patients, but not from German and Irish patients.

Conflict of Interest

Lombardi’s clinic [1] offers $650 diagnostic test to detect XMRV, so it is of real advantage to the authors of the first paper that the CSF-samples are positive for the virus. On the other hand Prof. Simon Wessely of the second paper has built his career on the hypothesis that CFS is a form of psychoneurosis, that should be treated with cognitive behavior therapy. The presence of a viral (biological) cause would not fit in.

Shortcomings of the Lombardi-article [1]

Both studies have used nested PCR to detect XMRV. Because of the enormous amplification potential, PCR can easily lead to contamination (with the positive control) and thus false positive results. Indeed it is very easy to get contamination from an undiluted positive into a weakly positive or negative sample.

Charles Chiu who belongs to the group detecting XMRV in a specific kind of hereditary prostate cancer, puts it like this [5]:

In their Dissenting Opinion of this article, Moore and Shuda raise valid concerns regarding the potential for PCR contamination in this study. Some concerns include 1) the criteria for defining CFS/ME in the patients and in controls were not explicitly defined, 2) nested PCR was used and neither in a blinded nor randomized fashion, 3) the remarkable lack of diversity in the six fully sequenced XMRV genomes (<6 nucleotide average difference across genome) — with Fig. S1 even showing that for one fully sequenced isolate two of the single nucleotide differences were “N’s” — clearly the result of a sequencing error, 4) failure to use Southern blotting to confirm PCR results, and 5) primary nested PCR screening done in one lab as opposed to independent screening from start to finish in two different laboratories. Concerns have also been brought up with respect to the antigen testing

Shortcomings of the Erlwein-article [2]

Many people have objected that the population of CSF patients is not the same in both studies. Sure it is difficult enough to diagnose CSF (which is only done by exclusion), but according to many commenters of the PLOS study there was a clear bias towards more depressed patients. Therefore, a biological agent is less likely the cause of the disease in these patients. In contrast the US patients had all kinds of physical constraints and immunological problems.

The review process was also far less stringent: 3 days versus several months.

The PLOS study might have suffered from the opposite of contamination: failure to amplify the rare CSF-DNA. This is not improbable. The Erlwein group did not purify the blood cells, used other primers, amplified another sequences and did not test DNA of normal individuals. The positive control was diluted in water not in human DNA. The negative control was water.

Omitting cell purification can lead to a lower relative amount of the XMRV-DNA or to inhibition (often seen this with unpurified samples). Furthermore the gel results seem of poor quality (see Fig 2). The second round of the positive PCR sample results in an overloaded lane with too many aspecific bands (lane 9), whereas the first round leads to a very vague low molecular band (lane 10). True that the CSF-samples also run two rounds, but why aren’t the aspecific bands seen here? It would have been better to use a tenfold titration of the positive control in human DNA (this might be a more real imitation of the CSF samples: (possibly) a rare piece of XMRV DNA mixed with genomic DNA) and to use normal DNA as control, not water.Another point is that the normal XMRV-incidence of 1-3,7% in healthy controls is not reached in the PLOS study, although this could be a matter of chance (1 out of 100).

Further Studies

Anyway, we can philosophize, but the answer must await further studies. There are several ongoing efforts.

References

  1. Lombardi VC, Ruscetti FW, Das Gupta J, Pfost MA, Hagen KS, Peterson DL, Ruscetti SK, Bagni RK, Petrow-Sadowski C, Gold B, Dean M, Silverman RH, & Mikovits JA (2009). Detection of an infectious retrovirus, XMRV, in blood cells of patients with chronic fatigue syndrome. Science (New York, N.Y.), 326 (5952), 585-9 PMID: 19815723
  2. Erlwein, O., Kaye, S., McClure, M., Weber, J., Wills, G., Collier, D., Wessely, S., & Cleare, A. (2010). Failure to Detect the Novel Retrovirus XMRV in Chronic Fatigue Syndrome PLoS ONE, 5 (1) DOI: 10.1371/journal.pone.0008519
  3. http://f1000biology.com/article/yxfr5q9qnc967kn/id/1166366/evaluation/sections
  4. http://neuroskeptic.blogspot.com/2010/01/chronic-fatigue-syndrome-in-not-caused.html
  5. Charles Chiu: Faculty of 1000 Biology, 19 Jan 2010 http://f1000biology.com/article/id/1166366/evaluation

Photo Credits

Nested PCR ivpresearch.org







Follow

Get every new post delivered to your Inbox.

Join 610 other followers