The effectiveness of the practice of correction and republication in the biomedical literature

Abstract

Objective:

This research measures the effectiveness of the practice of correction and republication of invalidated articles in the biomedical literature by analyzing the rate of citation of the flawed and corrected versions of scholarly articles over time. If the practice of correction and republication is effective, then the incidence of citation of flawed versions should diminish over time and increased incidence of citation of the republication should be observed.

Methods:

This is a bibliometric study using citation analysis and statistical analysis of pairs of flawed and corrected articles in MEDLINE and Web of Science.

Results:

The difference between citation levels of flawed originals and corrected republications does not approach statistical significance until eight to twelve years post-republication. Results showed substantial variability among bibliographic sources in their provision of authoritative bibliographic information.

Conclusions:

Correction and republication is a marginally effective biblioremediative practice. The data suggest that inappropriate citation behavior may be partly attributable to author ignorance.

Highlights.

• The citation of flawed articles occurs at a rate nearly equal to that of corrected versions.

• The practice of correction and republication is only marginally effective and does not prevent the continued citation of flawed articles post-correction, with the analysis finding only a slight reduction in the citation of flawed articles after publication of the corrected version.

• Neither MEDLINE nor Web of Science consistently alert users when dealing with corrected and republished literature.

Implications.

• The practice of correction and republication would be more effective if prominent sources of bibliographic information were more consistent in providing users with information about the status of corrected and republished articles and the existence of post-publication modifications to the literature.

• It is incumbent upon the scientific community to raise the profile of post-publication changes to the literature to prevent the wasteful and potentially tragic consequences of scientists and medical professionals applying flawed information. Failure to do so will surely result in a reduction of public trust in the reliability of the scientific literature and its users.

INTRODUCTION

Occasionally, authors cite literature that contains factual errors. Examples of cited flawed literature include a published erratum in the Journal of Pediatrics in which a dosage was misprinted as milligrams rather than grams; an article in which a chemotherapy dosage was mistranscribed from “every day for twenty-eight days” to “every twenty-eight days,” and a highly cited article in which the publisher printed the wrong figures in a paper about techniques to return blood to the heart (reperfusion) immediately following myocardial infarction [1, 2]. Approximately 10% of biomedical articles indexed in MEDLINE that are marked as having been changed post-publication are indexed as review articles, and the impact of errors in this influential subset alone could be significant. The potential impact of errors in the literature can be great if inaccurate information forms the basis of subsequent research or patient treatment. The evaluation of the effectiveness of corrections to the scientific literature is significant because applying flawed biomedical research findings can have substantial negative health, scientific, and economic consequences.

When flaws are recognized in the literature, the publisher is alerted and a notification of the error and a correction are published. Of particular interest to this study is evaluating the effectiveness of the publication type that the National Library of Medicine (NLM) designates as the “corrected and republished article.” Since 1987, NLM's policy regarding these documents has been that whenever a labeled, citable correction to a document is published, NLM annotates the bibliographic record of the original article with a reference to the notice of erratum [3]. NLM also identifies a variety of articles that for various reasons (including error on the part of the authors or publishers, article duplication, and scholarly malfeasance) have been formally withdrawn from the official scientific record. These are designated with “retracted publication,” while those with published corrections receive the annotation “published erratum.” Traditionally, it has been incumbent on the reader to search all subsequent issues to learn of any post-publication modification to an article, either retracted or republished. While preserving the historical integrity of the scientific archive, this post-hoc approach has the significant drawback of being an inefficient way of alerting the public to the fact that a given article has been invalidated. Because researchers cannot readily distinguish between flawed and reliable published documents, making users aware of the status of a document is fundamental to reducing citation of retracted or republished articles, as well as any errata [4].

Unlike other biblioremediative mechanisms such as the publication of retractions or errata, the practice of correction and republication has resulted in a body of nearly identical pairs of articles in which one member of the pair contains erroneous information. An article in MEDLINE that has been corrected and republished by its producers (authors or publishers) is referenced by the newer version, and the original document's bibliographic entry is changed to indicate that it is now not considered reliable and is linked in the index to the newer, more correct and authoritative version. In the current study, the uncorrected, flawed version of a published scholarly article is referred to as the “original,” which is withdrawn by its authors or publisher in lieu of the “republication” or “republished version.”

The retraction was the first type of literature requiring remediation that NLM identified. Evaluation of the effectiveness of biblioremediative techniques for scientific literature, particularly in the study of the effectiveness of retractions, has generally used citation counts. Pfeifer and Snodgrass questioned whether “current methods to disseminate knowledge of retracted publications were adequate to prevent their future use” and applied citation analysis to a cohort of articles containing fradulent results. The authors concluded that citation of previously invalidated works is “abundant and ubiquitous,” observing a 35% diminution of citation among retracted articles compared to non-retracted articles and concluding that “invalid work is not being effectively purged from the literature.” They also noted the responsibilities that publishers and indexers play in alerting the public to changes in the state of the veracity of the literature [5]. In the same issue of JAMA, Garfield and Welljams-dorof published findings based on their citation analysis of the research impact of fraud on the scientific literature: Approximately 30% of citations to this cohort were explicitly negative, that is, “references disagreed with findings/methods” [6]. They concluded that the scientific literature is purging itself of elements that have been identified as being flawed. Garfield's interpretation is at odds with that of Pfeifer and Snodgrass in that Pfeifer and Snodgrass interpreted a 70% tacit or explicit positive citation rate as evidence that inappropriate citation is rampant.

Wright performed an analysis of retracted literature that included examining citations to 53 retracted articles, as a way of determining whether retraction reduces subsequent citation frequency and determining whether post-retraction citations were positive or negative, and concluded that continued affirmative citation indicated that some retractions were going unheeded and that invalid information might be perpetuated through subsequent citation [7]. In 1999, Budd et al. established through context analysis that the overwhelming majority (93%) of citations were indeed positive, concluding that retraction, even when noted in the MEDLINE database, did not ensure reader awareness and a subsequent diminution in referencing of retracted works [8].

Whitely concludes that users do not and probably cannot identify fraudulent articles if the articles have made it through the peer-review process and into print. However, when alerted to the status of such articles, the scientific community responds by reducing citation of those articles [4]. Whitely further concludes that news articles and reviews of the author's work were more effective at alerting the scientific community than other print notices in scholarly publications or in MEDLINE. Whitley's study focuses on literature whose retraction has been highly publicized, and the practice of focusing on the work of an author whose disclosure of misconduct has been particularly public has been followed by other researchers [6, 9]. Though all of these studies focused on highly publicized bodies of work (which theoretically ought to be shunned by authors and therefore should show a lower incidence of citation than less notorious work), all noted significant post-retraction positive citation.

The aforementioned studies have evaluated the continued use of retracted literature by examining the degree of citation and the nature of citation through citation context analysis. The results of these studies are similar: post-retraction citation is usually positive and continues to a significant extent after the article has been retracted. Articles are retracted because of fraud or error, making this kind of post-publication modification particularly high profile. Corrections and republications have not commanded the same level of attention as retractions, but their impact may be no less significant. Nearly all of the previous studies examine retracted articles; very few studies of other corrective mechanisms of the literature exist. Poworoznek's [10] and Sievert et al.'s [11] studies concerning correction and errata are 2 rare exceptions. Poworoznek observed considerable variability among publishers' linking errata to original papers by examining the rate of linkage between corrigenda in online versions of high-impact physical science journals and finding that only 26 of 43 journals reviewed (60%) include at least 1 link connecting errata with original papers. Porowoznek's study is relevant to the current research project, because it evaluates the frequency with which a bibliographic entry for 1 member of a document pair directs the user to the complementary member.

METHODS

Bibliographic information for republished articles was obtained by searching MEDLINE via PubMed for the document type “Corrected and Republished Article,” with searching limited to the clinically relevant “Core Clinical Journals.” This option selects high-profile journals that are known to have a lower incidence of inappropriate, explicit citation of retracted articles than non-core clinical journal articles [8]. The search was further limited to articles published in English between “1990/01/01” and “2000/01/01,” ensuring the currency of the literature while allowing for permeation of the correction into public awareness. After retrieving this cohort, the bibliographic information was extracted from the Web of Science database so that citation data for each version could be obtained. The relative incidence of citation of corrected and republished versions of article pairs was compared at four-, eight-, and twelve-year intervals. These intervals were selected because the half-life of an article in the core clinical journal subset is approximately eight years [12]. Thus, measurements at four, eight, and twelve years should provide insight into the probable most-active period of citation for an article. Citing articles included in the sample were published not less than one year post-republication. This one-year publication lag was intended to account for potential legitimate citations that might have been in press at the time that the republication was released, thus carrying a post-republication citation date while legitimately citing the original as valid.

To determine the degree to which prominent sources of biomedical bibliographic information alert users to the status of corrected and republished articles, the main bibliographic entry for each article version in MEDLINE and Web of Science was examined for information directing the searcher to the definitive, republished member of the pair. It is important to note that this part of the study examined the main entry for each article; only information included on those pages was considered. In some cases, title searches yielded multiple entries. It cannot be assumed that all searchers meticulously scrutinize each entry when pursuing a particular document.

The raw data were collected and tabulated, and outliers were removed. Outliers fell into three categories:

• Outlier type 1: articles cited much more frequently than the norm. The average lifetime citation rate of articles pairs overall was 24 citations over 12 years, with a standard deviation of 14.4. Four members of the original sample had been individually cited more than 70 times (more than 3 standard deviations from the norm). Eliminating these outliers in accordance with the empirical rule guaranteed the normal distribution of the sample.

• Outlier type 2: articles pairs that received no citations over their lifetime. Three articles fell into this group.

• Outlier type 3: articles whose republication's volume and page number were indistinguishable from that of the original, making it impossible to determine which version a citing author intended to reference. In these cases, the Web of Science Cited Reference Search attributes all citations to the original and none to the republication. These entries do not measure citations to republished versions of articles and thus were eliminated. Four examples of this type of article pair were found in the sample.

After removing the outliers, the final sample contained 48 article pairs that had received a total of 1,064 citations, with a total citation rate of 496 (mean 10.3, standard deviation 13.5) for the original articles and a total citation rate of 568 (mean 11.8, standard deviation 13.6) for the republished versions. The data collected allowed the null hypothesis to be tested using a t-test to compare the difference in average citation rates of republished and original versions of articles. The t-test makes 3 assumptions (that the samples are paired, that the sample is a random selection, and that the sample size is greater than thirty), all of which are fulfilled by the set of 48 otherwise unrelated document pairs selected from the MEDLINE database.

RESULTS

Citation of republications is slightly more common than citation of original versions: 53% of post-republication citations to an article pair are to the republished version, 47% to the original. The bar graph (Figure 1) shows that republications seem to be preferentially cited over originals after 4 years post-republication and the relative citation levels of versions at 4-year intervals indicate that republications are preferentially cited by authors after 4 years post-republication. Data gathered indicate that post-correction, flawed articles continue to be cited at a rate nearly equal to that of the republished versions.

Figure 1.

Incidence of post-republication citation at four-, eight-, and twelve-year intervals

There was no statistically significant difference between the average incidence of citation of original and republished versions of articles when the article versions were compared in a single group (t = 1.67, df = 47, P = 0.218) or between the versions at 4-year intervals. At the 8–12 year interval, the difference began to approach statistical significance (t = 1.60, df = 47, P = 0.059). As document pairs aged, the republished versions of the articles were cited with more frequency than originals; however, the effect was not strong.

Bibliographic information sources showed considerable disparity in their presentation of information regarding the status of the articles in the dataset. MEDLINE provided information about the corrected version in main bibliographic entries for 33 of the 48 examined articles (69%) with invalidated original articles, while Web of Science provided that information for only 2 of the 48 articles (4%). Further, while every MEDLINE entry for a republished article indicated that it was a republication, only 28 of 48 (58%) of Web of Science entries for the same articles contained the same information.

DISCUSSION/CONCLUSION

The central question of this study compares the incidence of citation of each version of the article to determine if the practice of correction and republication is an effective mechanism for replacing a flawed document with a corrected one. The data support a number of conclusions regarding the effectiveness of correction and republication. Analysis indicates that republications may replace originals as authoritative over time, but the process is slow: Citation of invalidated literature occurs at a rate nearly the same as that of valid literature for at least the first eight years after publication. Thus, flawed literature is being cited (and thus presumably used) by authors with great frequency. Though the practice of correction and republication is marginally effective, it does not prevent the inappropriate citation of documents that have been identified as containing errors. The results suggest that authors have difficulty distinguishing between corrected and republished article versions.

Considering the prominence of electronic databases as tools for literature retrieval, if republications are identified by these resources, then the continued use of flawed literature is likely not attributable to lack of information presented to the user. Results indicate that the major repositories of bibliographic information for biomedical research literature do not consistently provide users with the best information possible. The scholarly community would surely benefit if measures to alert users about corrected literature were improved.

Responsibility for continued use of flawed literature in derivative works is less important than the fact that the phenomenon exists at all. Authors and information providers alike have a role to play in preventing the use of flawed biomedical literature in research. Authors have a responsibility to ensure that they are using the most up-to-date resources possible and to ensure that source material is valid. The information provider has a significant role in providing the user with such material and in alerting the user when a literary artifact is in some way anomalous. Greater exertion on the part of both parties would help reduce this phenomenon and improve the quality of modern science. It is apparent that a tool that helps both authors and indexers more effectively manage such literature would be beneficial. The need for such a tool suggests an obvious avenue for continued research. An information management tool that compares the bibliographies of works to an index of subsequently corrected literature would be a contribution from the information science community that would benefit the entire community of science.

Ultimately, future efforts should be directed at the problem of making post-publication changes to the literature more effective and at diminishing the erroneous use of invalidated literature. In an era of instant delivery of information from many sources, it is incumbent on the providers of information to provide the best possible information to users at the cutting edge of their research fields. Failure to do so may not only have negative human health impacts, it also diminishes the reputability of the scientific record, which must be maintained at the highest levels to ensure the robustness of modern research in all fields of study. The derivative nature of science assumes and relies utterly on the accuracy of the scientific record. The assumption that each article can be individually validated allows researchers to be confident that the accumulated literature for any given topic is reliable. If the assumption of the validity of the scientific record can be challenged, science is undermined in a fundamental way.