Accuracy of references in the ophthalmic literature

INTRODUCTION

The practice of citing previously published work to support current investigations is one of the foundations of medical literature. It places an article in a context, builds on prior work, establishes credibility, and leads readers to additional resources on the topic. According to the Uniform Requirements for Manuscripts Submitted to Biomedical Journals, “Readers should … be provided with direct references to original research sources wherever possible” [1]. Peer review and editing guarantee the accuracy and relevancy of the scientific information presented in a paper, and the same measures should be applied to the references (also referred to in this paper as “citations”). Unfortunately, as demonstrated in reference accuracy studies, including studies done in major medical publications, verification of reference accuracy is not taking place in many journals.

The authors of this article both work in ophthalmic libraries and were specifically interested in the accuracy of this literature and how it would compare to other medical fields. The objectives of this study were to assess the accuracy of bibliographic citations in a number of specialty journals in the ophthalmic literature, to compare the results with earlier studies involving a variety of medical specialties and publishers, and to discuss roles for librarians in assuring a higher accuracy rate for published citations. The authors were also interested in how different publishers dealt with reference accuracy and how this affected the error rates. The assumption was that the more thorough the checking process, the smaller the error rate among references in the published articles.

Attention has been paid to reference accuracy in the biomedical literature in a variety of reported studies, most of which stick to a single medical specialty. One other previous study by Buchan, Norris, and Kuper looked specifically at ophthalmic literature. The error rate in that study ranged from 0 to 35% with overall error rate of 16% [2]. A small sample consisting of 20 references each from 10 journals was examined. References to non-English materials, books, reports, and electronic materials were excluded. The study found a low frequency of citation errors among those titles with extra accuracy checks. Two journal titles had no errors: the Journal of Cataract & Refractive Surgery had a librarian check references and Survey of Ophthalmology had an automated reference-checker program. The journal, Ophthalmology, that employed a reference checker had a 10% error rate. Although the Buchan et al. study dealt with some ophthalmic journals, it did not include optometry journals, non-journal literature, or publisher information. It also involved fewer references; 200 total references to the current study's 800 [2].

Wager and Middleton [3] recently completed a systematic review of technical editing in biomedical journals, involving a large range of medical specialties. Their review included a reference accuracy component, including more than 15,000 references. They showed a median citation error rate of 39%, with a range of 4% to 67%.

Table 1 lists the error rates found in a number of single specialty studies. The majority of these other studies involved references to journal literature only; other materials such as websites and books were excluded.

Table 1.

Overview of biomedical reference accuracy studies

METHODS

Eight peer-reviewed ophthalmic journal titles, covering a broad range of specialty topics, distributed among three major publishers, were selected for this study. Six of the titles were ophthalmology journals, and two were optometry journals. Adding optometry journals broadened the scope of the study by including the vision science literature from a related but separate field. Two ophthalmology titles, Acta Ophthalmologica Scandinavica and Progress in Retinal Eye Research, and two optometry titles, Optometry and Vision Science and Ophthalmic and Physiological Optics, were unique to this study. Journals were selected not only for their ophthalmic content but to study different publishers' methods of reference checking. The three major biomedical publishers represented were Elsevier, Lippincott Williams & Wilkins, and Blackwell Publishing.

A sample of 100 references per journal published in the year 2004 were reviewed for citation accuracy for a total of 800 references. Every third article listed in the table of contents from each journal title was chosen for the study. Every fifth reference from each of these articles was picked for inclusion in accuracy verification until 100 references were selected from the title. This process helped ensure that more than 1 reference from an individual article could be included and that they would not be consecutive. They would also be selected from more than 1 month of each title.

As in other studies, references to journal literature were included, but this study also included books, reports, electronic formats, and so on. Accuracy was measured by comparing the reference with the original article or book using the authors' own libraries and institutional libraries, online access to portable document format (PDF) files, and interlibrary loan. PubMed was used only as a last resort for a very small number of references as PubMed itself contains errors. Errors were classified, analyzed, and compared with previous studies.

Finally, an editor from each journal was consulted directly to find out who was responsible for checking accepted article references.

RESULTS

As shown in Table 2 (online), this study recorded eight varieties of mistakes in citations. These included an author error (a typographical error or a missing initial), an incorrect or missing author name, an article title typographical error, an incorrect article title or missing article title words, an error in the journal title, an error in the date, an error in the volume, and/or an error in the page number. Any reference might have more than one kind of error.

These eight kinds of errors were also divided into major or minor categories. An error was considered minor if the article could still be easily located despite the erroneous information. A typographical error or a missing initial in an author's name, a wrong or missing author name, and/or a typographical error in an article title were labeled minor mistakes.

An error was considered major if it would impede the article from being easily found. A major error was defined as wrong or missing words in the article title and/or a mistake in the journal title, date, volume, or page number. A journal could have a higher percentage of major than minor errors but still have fewer total errors when compared to other journals.

In terms of total errors (Table 2, online), the journal error rate, in order from the title that contained the least amount of errors to the most, was: Journal of Cataract and Refractive Surgery (4%), Optometry and Vision Science (9%), Progress in Retinal Research (9%), Cornea (17%), Ophthalmic and Physiological Optics (19%), Clinical and Experimental Ophthalmology (21%), Acta Ophthalmologica Scandinavica (23%), and Journal of Glaucoma (25%). In terms of major errors only, Progress in Retinal Eye Research would switch places with Optometry and Vision Science to become the journal with the second lowest error rate, and Ophthalmic and Physiological Optics would move from fifth highest to the seventh highest, with only 1 journal having more major errors. The results of this study, with 6 ophthalmology journals and 2 optometry journals, showed a total error rate that ranged from 4.0% in the Journal of Cataract and Refractive Surgery to a high of 25.0% in the Journal of Glaucoma. The mean error rate was 15.9%, and the median error rate was 18.0%.

The other study of ophthalmic journal errors by Buchan et al. showed an error rate that ranged from 0 in the Journal of Cataract and Refractive Surgery and Survey of Ophthalmology to an error rate of 35% in Ophthalmic Epidemiology, with a mean error rate of 16% [2]. These results were similar to this study's results, even though different journals were sampled, different types of literature were included, and a different reference selection method was used.

Table 3 shows the error rate from the eight journals in this study by publisher. Elsevier, which was the only publisher to use librarians to check references, had the lowest averaged error rate, followed by Lippincott Williams & Wilkins and then Blackwell.

Table 3.

Reference errors by publisher

DISCUSSION

Accurate references are important in scientific discourse for a variety of reasons. Credit given to an author in citation databases is reduced if the name is incorrect. Correct references support the author's familiarity with the history and background of a topic. Invalid references create questions as to the validity of the article itself. Sloppy references reduce the credibility of the authors, editorial board, and publisher. Errors impede retrieval of information based on those references. Readers should be able to easily identify and retrieve cited articles.

One would expect a direct association between prepublication reference checking and lower citation error rates. The results of this study support this expectation. The Journal of Cataract and Refractive Surgery, which used a librarian to check the references, had the lowest error rate.

The highest error rate appeared in journals that leave it up to the authors to submit correct references with no additional checking. The Journal of Glaucoma, which had the highest error rate, had several instances of missing data in the references. It seems that missing data should have been caught early on in the editing process.

The amount of error in the citations is not positively associated with the number of total articles printed, as the journal with the lowest percentage of errors had the highest number of articles. The journal with the highest percentage of errors had the second lowest number of articles.

CONCLUSIONS

Based on two studies, reference accuracy in the ophthalmic literature is lower than the average shown for other medical specialties, while falling well in the range established in the systematic review cited above [3].

In spite of the obvious importance of reference accuracy, articles with erroneous references are still too common. All references should be checked for accuracy; leaving reference accuracy up to the author can result in a high incidence of error. This study showed that those journals that had reference checking methods in place had a lower reference error rate than those who left it up to the author. The journal that relied on a librarian to check references had the lowest error rate in the study.

Aside from references that can be verified in PubMed, authors commonly cite books, chapters, older and non-peer-reviewed materials, reports, electronic formats, and so on. References of this type cannot be handled by automated reference-checkers and can be difficult to track down and verify. Librarians or others who are very familiar with medical literature and citation standards should be involved in this phase of the editorial process.

LIMITATIONS OF THE STUDY

This study involved 800 references from 8 journals in the ophthalmic literature in the year 2004. Conclusions cannot be drawn regarding titles that were not included in the study or beyond the timeframe examined. References to be checked were not selected randomly according to conventional methods.