The Virtues of Correct Citation

Scientific statements have to be supported by evidence. The usual way to do this in academic articles is by citing the literature—that is, by giving references. A reference citation is supposed to provide accurate underpinning for a statement and to represent the current state of research, or, in the case of a maverick opinion, to ensure it is recognizable as such. Reference to a published source allows the reader to probe more deeply into the subject and puts him or her in a position to judge the underlying evidence. Using references puts the author under a duty to formulate them precisely so as to reproduce the source publication accurately.

The convention of reference citation is a constituent element of scientific texts; it distinguishes them from journalism and other forms of writing and gives their content a particular authority. The majority of readers will have more confidence in a referenced statement than in one without a reference, even without checking it. The choice and placing of references, however, are complex processes prone to error. The following mistakes can appear:

• The source does not support the statement, or does so only partially (“quotation errors”).

• The reference citation is placed in such a way that it is not clear which statement it relates to.

• The bibliographic data (authors, title, volume number, page numbers, or journal title) are incomplete or wrong (“citation errors”). This can make it difficult or impossible to find the source.

Quotation errors are particularly problematic because it is difficult to spot them: Often the reader can only judge the accuracy of a reference if he or she has relevant specialist knowledge or takes the trouble to research the sources cited. Another form of referencing error, which will not be treated here, is plagiarism: In this case the source of information is knowingly suppressed and the statement is presented as the plagiarizing author’s own intellectual achievement (1).

Quotation errors

In a systematic review, Wager and Middleton evaluated 66 studies, in which a total of 3836 references from 74 biomedical journals were examined (2). In addition to references in general medical journals such as The Lancet and the British Medical Journal, articles in journals of surgery, pediatrics, and other specialties were also included. In this Cochrane Review the median overall rate of major and minor quotation errors was 20%, with a range from 0% to 50% (2). The systematic review included publications up to and including the year 2007.

A literature search carried out by ourselves on Medline was restricted to publications dating from 2008 and later, in order to ensure that no publication was counted twice. These publications are summarized in the Table. From 2008 on, five publications were identified, which analyzed a total of 1521 references in 16 journals. Overall, 299 errors were identified, giving an error rate of 19.7%, which confirms the earlier values. These publications gave no clear indication of whether the error rates correlated with, for example, the impact factor of the journal under investigation or with any particular medical discipline. Interestingly, the error rate appears to have remained steady from 1977, when the first studies were published, up until 2011, as our own analysis of the data showed.

Table. Studies in which errors of content were found in the references.

Study	References checked (n)	Incorrect referencing*1 (%)	Journals examined (n)
Wager, Middleton 2008, Cochrane Review (2)	3836	20, range: 0 to 50	74 (various specialties)
Reddy et al. 2008 (3)	258	7.8, range: 3.5 to 10.3	Surgery (4)
Al-Benna et al. 2009 (6)	117	13.7, 95% CI: 8.6–19.5	Burns (2)
Singh, Chaudhary 2009 (7)	46	43.5, 95% CI: 29.2–57.8	Dermatology (1)
Davis et al. 2010 (8)	200	38, 95% CI: 30.1–47	Orthopedics (4)
Awrey et al. 2011 (5)	900	18.5, range: 10.5 to 22.0	Surgery (5)
Authors‘ own pilot study *2	50	Major errors: 18, 95% CI: 9.2–30.5; minor errors: 14, 95% CI: 6.3–25.7	Medicine, general (Deutsches Ärzteblatt)

The study by Wager and Middleton is a systematic review (Cochrane) analyzing publications up until the end of 2007. In order not to include any publication twice, the studies listed below Wager and Middleton include only articles published from 2008 onwards.

^*1 Reference does not support statement made in the article, or only partially supports it.

^{* 2} The pilot study did not analyze references to books.

Agreement between the two evaluators (authors) was satisfactory (weighted Cohen‘s ? = 0.583)

Deutsches Ärzteblatt

How common reference quotation errors are has been investigated several times in the past few years, although so far not in any German-language medical journal. For this reason, we carried out a small pilot study of references in Deutsches Ärzteblatt.

To get an idea of the extent to which references are used correctly in the scientific section of Deutsches Ärzteblatt, 50 randomly selected reference citations in articles published during 2010 were evaluated. The agreement between statement and reference was determined independently by S. M. and C. B. on the basis of the abstract or the full text of the reference. Each citation was scored as either correct or incorrect; “incorrect” citations were divided into minor errors and substantial errors. A reference source in which, for example, there were slight discrepancies as to the numbers quoted, but which nevertheless supported the statement, was classed as showing a minor error. Where the two authors scored reference citations differently, they discussed them and reached a joint evaluation.

Substantial quotation errors were found in 9 of the 50 statements (18%). The incidence of minor errors was 14%. Although the small size of the sample allows only a rough estimation of the error rate (95% confidence interval [CI] of substantial quotation errors: 9.2% to 30.5%), this result agrees well with the rates identified in the literature.

However, it is difficult to compare these data with the literature because the definition of errors is not homogeneous. Even Wager and Middleton had to leave some publications out of their Cochrane Review for this reason (2). Moreover, the ratio of major to minor errors is variable: The proportion of errors that are major is reported at up to 80% of the overall error rate (3). Against this background, the error rates found in Deutsches Ärzteblatt seem to be comparable to those found in other titles (Figure).

Figure.

Quotation errors and citation errors

Sources: (2) for literature up until 2008, thereafter our own literature search and, for Deutsches Ärzteblatt, the pilot study.

^*1 substantial errors

Citation errors

In their Cochrane Review, Wager and Middleton analyzed 66 publications that included checking of bibliographic data. Out of the more than 27 000 references in more than 100 journals, a median of 38% contained errors (2). Depending on the journal, error rates ranged between 4% and 67%. The Cochrane Review authors found indications that technical editing is associated with a lower error rate.

In our own Medline search of publications from 2008 onwards, we identified six further articles in which bibliographic data were checked. Out of the total of 1928 citations in 18 journals, on average 16.4% contained errors (range: 7.5% to 59.6%) (3– 8). Major errors, which made it impossible to identify the original source, were rare. Overall it became clear that the rate of bibliographic errors (citation errors) is markedly lower in publications since 2008 than in the preceding years. This is in contrast to the correctness of reference quotations, where the error rate seems to be remaining steady over time. The reduced citation error rate could possibly be ascribed to increased use of electronic reference software.

Deutsches Ärzteblatt

Since there are evidently no publications that have investigated the accuracy of bibliographic data in German-language journals, we analyzed 50 randomly chosen references in 50 articles in Deutsches Ärzteblatt published in 2010. Different references were chosen than in the first part of the study. Four of the references contained errors (8%: 95% CI: 2.6% to 18.2%). All of these, however, were minor errors such as misspellings of an author’s name. None of them affected a reader’s ability to locate the article. This error rate is markedly below the values found in the published literature (Figure) and may be due to the extensive editorial process that manuscripts undergo at Deutsches Ärzteblatt.

Conclusion

References in scientific publications often contain errors. Published data and those from our own small study indicate an error rate of around 20%. This would tend rather to be a conservative estimate, because the studies only checked whether the statement agreed with the source. It would be more interesting—but require disproportionately more work—to test whether the reference chosen is scientifically correct and representative of the current state of research, or whether the choice is rather influenced by the author’s own hypotheses and interests.

If 20% of reference quotations are incorrect, this means that the impact factor has a correspondingly high error rate. The impact factor, it will be remembered, measures how often articles in a journal are cited during a certain period of time and is used to evaluate journals and authors (9). In calculating the impact factor, it is assumed that there are no errors in the quotation of references. If the inaccuracy was systematic in nature—that is, if it were to affect all articles equally—the absolute impact factor values would be wrong, but at least the relationship between journals would remain the same. It is, however, not very likely that such a systematic distortion is the case. It is therefore possible that the quotation error rate for popular publications is disproportionately high, simply because many scientists know about them. This could lead to a distortion of the impact factor in favor of much-cited articles.

The comparatively high proportion of incorrect reference citations might, however, be more than just an indication of carelessness on the part of the authors. It could also mean that the requirement to prove every statement is in many cases asking too much of scientists. After all, the authors of scientific articles are primarily focused on presenting their own results or—in the case of a review article—opinion on a clinical problem. In this context, one or two gaps in documentation from the literature can exist without invalidating the main statements of the article. However, it is unclear whether quotation errors relate to peripheral or core aspects.

To obtain greater transparency about the choice of sources, it would be helpful to describe—not just in systematic reviews, but, ideally, also in original articles and in narrative review articles—how the authors selected the references they have used. Are there inclusion and exclusion criteria for the publications cited? Which databases were searched, and for what period? Are the citations based only on original articles, or also on review articles? Giving information on these points would allow the critical reader to check and understand the selection criteria by which the references employed were chosen, and thus allow him or her better to judge the authorial balance in the use of references. This could motivate authors to search more intensively and to cite further, previously unknown but relevant publications.