When to retract?

Today we are retracting a study that seemed to show that the outcome of pregnancy in diabetic women in northeast England was worse than that of diabetic women in Norway.^1-3 The authors have realised that they made a fundamental mistake.² Data collected in Norway were meant to exclude codes for gestational diabetes but didn't.² The conclusions cannot be allowed to stand, and a subsequent analysis shows no significant difference in outcome between women in the two countries known to have diabetes before pregnancy.² Studies are most commonly retracted because of fraud, but there is no question of misconduct in this case. It was a simple mistake, and as soon as the authors realised it they asked us to retract the study.² But when should editors retract studies?

Retraction is topical following two recent high profile cases. Nature Medicine has just retracted a German study that described how three patients with metastatic kidney cancer responded to a vaccine produced by fusing their tumour cells with immune cells.⁴,^{5 w1} An ombudsman's committee from Göttingen University conducted an inquiry. It did not find scientific misconduct but did find negligence in the documentation of the trial and preparation of the manuscript. Some of the 17 authors needed convincing that the study should be retracted.

The second retraction concerns a study in Science that showed that injection of 3,4-methylene-dioxymethamphetamine (ecstasy) into primates at doses close to those used recreationally caused severe damage to dopaminergic neurons.⁶,^{7 w2} The research was widely reported, including in the BMJ,^w3 with the suggestion that the widespread use of ecstasy might cause an epidemic of Parkinson's disease. The authors asked for the study to be retracted once they realised that nine of the 10 animals had been given the wrong drug. The bottle was mislabelled.⁶

Some critics have accused the authors of “rushing their results into print” because of discussions in the US Congress on legislation to curb the use of ecstasy.^w2 Colin Blakemore, a professor of physiology from Oxford and chief executive designate of the Medical Research Council, said that it was “hard to see how that process [peer review] operated properly in the case of this article” and that “respect for science” had been degraded.^{w2 w4} Some of the press have talked about Science being humiliated by having to retract the article.

All this is over the top. Peer review works on trust. If authors write that they injected a particular drug or had 200 patients in their trial, then they are believed. Nobody asks to see the drug and analyse it or to see the records of the 200 patients. Peer review will sometimes pick up fraud, but it isn't designed to do so—and mostly doesn't.⁸ Peer review is held to be almost sacred within science, and yet accumulating evidence shows it to be highly subjective, something of a lottery, and prone to bias and abuse.^{w5 w6} Several studies—including some undertaken by the BMJ—show that reviewers are poor at detecting major errors that are apparent in the text.^w7 w8 They cannot be expected to detect errors that are not apparent from the text.

All journals publish studies that turn out eventually to be nonsense. All studies have some flaws, and critics often call for studies to be retracted. There were calls for us to retract our recent study on passive smoking.^{w9 w10} Subjects who felt abused by the Bristol Cancer Help Centre study, which wrongly said that women who went to the centre were likely to die sooner than “controls,”⁹ went as far as the Charity Commission to try to get the study retracted. Many would like to see a retraction of the Lancet paper that linked the MMR (measles, mumps, and rubella) vaccine with autism.¹⁰ Indeed, almost every week people call for retraction of papers they don't like.

Many use the word retraction loosely. They probably mean that they want the editors to make clear that there are great anxieties over the studies. To editors retraction is a formal process that means that the study will be marked as retracted in Medline, their own index, and other indices—with the implication that it should be ignored. In fact retracted studies continue to be cited, usually without any indication that they have been retracted.^w11 In a paper world the retracted studies could not be entirely destroyed (although in a recent case—here retraction was inappropriate—the publisher urged subscribers to tear out the pages^w12), but in an electronic world they could be deleted. Editors and publishers mostly don't do so—because it destroys the scholarly record. For the same reason we don't go back and correct the electronic pages: rather, we leave the error and make clear in the original that there is a correction.

Studies are most commonly retracted because they are fraudulent. Since 1994, when bmj.com began, the BMJ has retracted three studies and one letter, all because of fraud. As far as the editorial team can remember we have only once before retracted a study because of a major error.¹¹,¹² Studies should certainly be retracted when discovered to be fraudulent—because even if the distortion of the data seems small there are immediately questions about the integrity of all the data: the trust that underpins peer review is destroyed. Questions are also immediately raised about previous studies by the authors. But even retractions for misconduct are not straightforward. Some misconduct is minor. Should a study be retracted because authors fail to declare competing interests? Probably not, but it's a form of misconduct—and there are many other minor forms of misconduct. When does the conduct become sufficiently major to retract the article?

Still more difficult is the decision over when to retract a flawed study. Retraction is surely right for the Nature Medicine, Science, and BMJ papers, but it wouldn't seem to be right for the Lancet's MMR study. It's widely regarded as a very weak study, but “what you see is what you get.” The weaknesses are there for all to see, while the defects in the three retracted studies are not apparent from what has been published. The severe defects in the Bristol Cancer Help Centre study are largely apparent—and so retraction is probably not appropriate, even though the conclusions go well beyond the strength of the methods and results.

Retraction should thus be reserved for studies that involve scientific misconduct and severe errors that are not discernible from the text. But when retraction is indicated editors should not hesitate. There is no need for shame. Retractions are like corrections. Excellent publications—like the New York Times—are full of corrections, whereas inferior publications—like British tabloid newspapers—have few. Everybody makes mistakes but not everybody admits them.