How accurate are the references in Emergency Medical Journal?

Abstract

Objective

To determine the accuracy of references in Emergency Medicine Journal during 2003.

Materials and methods

All references cited in Emergency Medicine Journal during 2003 were examined carefully, and the accuracy of the citations was checked against reliable electronic and manual resources. References were categorised as correct or incorrect. The errors were classified as minor if the integrity of the reference was not greatly compromised and major if the error severely detracted from the quality of the reference.

Results

Overall, errors were found in 19% of all citations checked (n = 2561), and in 8% the errors were major and markedly detracted from the quality of the reference.

Conclusions

Citation errors reflect badly on authors and the publishing journal and may reflect underlying flaws in other areas of the research published. It is hoped that identification of this problem will lead to attempts to improve the accuracy of reference citation in the emergency medicine literature and to an improvement in the credibility of research in our specialty.

The reference list at the end of a scientific article serves a number of important and useful functions. Accuracy of referencing reflects on the credibility of the author, the journal, and the research itself. Furthermore, it gives credit to the original researchers and allows readers to easily retrieve cited articles should they want to read more widely about a particular subject.

Inaccuracy of reference citations in the published medical literature is an ongoing problem that has been identified in a variety of specialist medical journals.¹,²,³,⁴,⁵,⁶ Goldberg et al in 1993 published the only previous study addressing this issue in the emergency medicine literature.⁷ They found an error rate of 27.5% in a random sample of citations from three major American emergency medicine journals.

The Emergency Medical Journal (EMJ) is the major emergency medical journal in the UK. The aim of the present study was to assess the accuracy of the references cited in EMJ during 2003.

Materials and methods

We carefully scrutinised the six issues of EMJ from 2003 and initially included all cited references in the study. References comprising journal articles were initially checked electronically via the Medline (1951 – present), EMBASE (1974 – present), and CINAHL (1982 – present) databases. Journal article citations that disagreed with the indexed reference were then checked manually where possible, and otherwise in a variety of full text online and other electronic sources. For each journal article we examined six bibliographic elements¹,³: author(s), title, journal, year, volume, and page numbers. If multiple errors occurred in a single element, only one error was noted. Books and book chapters were checked manually or electronically via a wide variety of resources. Other references, such as government documents, policy statements, and websites, were all checked for authenticity either manually or electronically.

Box 1: Examples of poor quality referencing (major errors)

• Citation so bad that finding the reference very difficult or impossible

• Reference not cited in the text

• First author markedly wrong or omitted from the reference

• Wrong journal

• Errors in four (or more) of the six bibliographic elements

• Book chapters referenced without page numbers or chapter authors

We then excluded references for which we were unable to find a reputable source to check the authenticity of the reference.

For every incorrect reference, a judgment was made whether the error in the citation markedly detracted from the integrity of the reference (box 1). We had two criteria for considering an error as a major error. Our first concern was that it should be easy to find the information to which the reference was alluding. Searching through wrong journals, volumes, and pages is tedious, as is looking through books for the relevant nugget of information when page numbers are not given. However, if a book or document was referenced in a general sense then we did not view the omission of page numbers as an error. Secondly, we regarded as major those errors that did not give accurate credit to the original authors (especially the first author) by either omitting them from the citation or badly misspelling their names so that it would be difficult to credit them for their original work. Minor misspellings of author names that did not compromise the integrity of the reference were not counted as errors.

Results

We scrutinised 2724 references from 254 articles in the 2003 issues of EMJ. In 163 cases (6%) we were unable to access an authentic source with which to compare the reference. These citations were thus excluded from the study, leaving 2561 references to be checked.

Overall, we found errors in 491 citations (19%). In 212 cases (8%) the errors were major and markedly detracted from the quality of the reference (see box 1). The breakdown of these references can be seen in table 1.

Table 1 Breakdown of the references checked.

	Checked	Incorrect	Poor quality
Journal articles	2269	404	151
Books and chapters	102	65	58
Other references	190	22	3
Total	2561	491	212

When checking the incorrect journal articles (n = 404), four articles were so badly incorrect that they could not be found despite extensive searching. One reference was not cited in the text at all. The remaining 399 journal article references were split based on the six bibliographic elements, and 83 articles were found to contain errors in two or more elements (table 2).

Table 2 Errors in journal article citations.

Error	No.
Unable to find despite resource	4
Reference not cited in text	1
Error in 1 element	315
Error in 2 elements	66
Error in 3 elements	11
Error in 4 elements	5
Error in 5 elements	1
Error in 6 elements	1
Total	404

Discussion

More than a decade on from the previous study,⁷ the problem of citation errors in the emergency medical literature remains. The ethics of high quality research methods have been eloquently documented elsewhere and clearly state that accurate referencing is part of good research practice.⁸,⁹ The 19% error rate in reference citations found in the 2003 issues of EMJ does give cause for concern that authors do not take care with this part of their research. Some of the errors are so bad (8%) one might wonder if authors may not even be obtaining, let alone reading, the references they are citing. Bibliographic errors can often be traced back over several publications in which they were previously incorrectly cited—the error then becoming repetitious. This may be evidence of authors having drawn incorrect conclusions from another source without even examining the original context of the citation.⁸ Despite this, the findings of this study compare favourably with previous research undertaken in a number of different specialty journals in which citation error rates of 3–60% were found.¹,²,³,⁴,⁵,⁶,⁷,¹⁰

Methods to improve the accuracy of reference citation include: submission of photocopies of first and last page of all references cited in the submitted article²,⁸,¹⁰; the direct downloading of references from Medline to avoid errors instead of copying them manually from the original article¹,⁵,¹¹; and reviewer checking of a sample of references from submitted articles.¹,³,⁸ Whatever method of verification is used, the aim should be to minimise the reference error rate in medical literature. Errors in citation reflect poorly on the author, the validity of the article, the peer review process, and the reputation of the journal.

Although by far the biggest study of its kind, our study does have some limitations. Of the references initially selected we could check 94%, but in 6% it was not possible to access a reliable source with which to compare the index citation. Independent reviewers were not used as a gold standard and this may have led to biases in the review process. As the study included only one journal, our results may not be applicable to all emergency medical journals. We would aim to widen the scope of this work with a future study to include a stratified random sample of references from multiple journals, in which the qualitative impact of citation errors could also be assessed.

Conclusions

There is a sizeable rate of citation errors (19%) in the EMJ. It is hoped that identification of this problem will lead to attempts to improve the accuracy of reference citation in the emergency medicine literature, and to an improvement in the credibility of research in our specialty.