Abstract
Objective To determine the rate and type of errors in biomedical informatics journal article references.
Methods References in articles from the first 2004 issues of five biomedical informatics journals, Journal of the American Medical Informatics Association, Journal of Biomedical Informatics, International Journal of Medical Informatics, Methods of Information in Medicine, and Artificial Intelligence in Medicine were compared with MEDLINE for journal, authors, title, year, volume, and page number accuracy. If discrepancies were identified, the reference was compared with the original publication. Two reviewers independently evaluated each reference.
Results The five journal issues contained 37 articles. Among the 656 eligible references, 225 (34.3%) included at least one error. Among the 225 references, 311 errors were identified. One or more errors were found in the bibliography of 31 (84%) of the 37 articles. The reference error rates by journal ranged from 22.1% to 40.7%. Most errors (39.0%) occurred in the author element, followed by the journal (31.2%), title (17.7%), page (7.4%), year (3.5%), and volume (1.3%) information.
Conclusion The study identified a considerable error rate in the references of five biomedical informatics journals. Authors are responsible for the accuracy of references and should more carefully check them, possibly using informatics-based assistance.
Introduction
Authors of publications in scientific journals are expected to provide the readership with references to related and previously published work. Most medical journals follow the “Uniform Requirements for Manuscripts Submitted to Biomedical Journals,” which follow the Vancouver style for formatting references.1 The requirements specify that authors are expected to check the accuracy of references against the original article. Accurate references allow interested readers to easily locate additional publications that are relevant to the subject of a specific journal article. Unless an error occurs in a critical element of the reference, such as journal, title, year of publication, volume, or first page number, most errors would not hinder their retrieval. However, if references are cited incorrectly, their location may become time-consuming, frustrating, or even impossible. Thus, the accuracy of references is critical.
Previous investigators have reported on reference accuracy in various general and specialist biomedical journals and found error rates ranging from 3% to 60%.2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26 High error rates (4.1%–40.3%) were even found among leading biomedical journals that check references prior to publication.23 Most errors affect the names of authors, followed by errors in the title element of a reference.5,6,20,21,22,23,25 The reference accuracy in biomedical informatics journals, the aim of this study, has not been previously examined.
Materials and Methods
The references in articles of the first standard, nonspecial 2004 issues of five biomedical informatics journals were selected: Journal of the American Medical Informatics Association (J Am Med Inform Assoc), Journal of Biomedical Informatics (J Biomed Inform), International Journal of Medical Informatics (Int J Med Inform), Methods of Information in Medicine (Methods Inf Med), and Artificial Intelligence in Medicine (Artif Intell Med).
Adapting the methodology of previous studies,9,21,22,23,25 we included all references to published journal articles that were indexed in MEDLINE,27 such as research articles, reviews, editorials, and letters to the editor. Each reference listed in an article's bibliography was checked against its corresponding MEDLINE citation. If a reference did not exactly match the MEDLINE citation, it was compared with the actual full-text original publication that was referenced. We excluded references to books, proceedings (even if indexed in MEDLINE), abstracts, technical reports, dissertations, theses, Web sites, public media, manuals, or other types of reports, for which the original publication may not be easily accessible for verifying the accuracy of the reference.
We defined a reference to a publication as incorrect if one or more errors were identified in any of the following six elements: journal, authors, title, year, volume, and page numbers. We defined a journal error as the full journal name or abbreviation differing from the official MEDLINE version, which follows the style used in the List of Journals Indexed in Index Medicus.28 We further classified inaccurate author information by errors involving the author's last name, the author's first and middle initials, and whether the error involved the first or subsequent authors. We also noted missing, additional, or an incorrect order of authors. The “Uniform Requirements for Manuscripts Submitted to Biomedical Journals” recommends an abbreviated list of authors using et al. after the sixth author.1,29 Some biomedical journals, however, use et al. after the third author. In this study, a missing author error was noted if et al. was used after the first or second author. For references using et al. after the third author, we verified author information for listed authors. An error in the title element was present if the title included misspelled, changed, omitted, or additional text. Page numbers were checked for the accuracy of the first and last page referenced. In addition, we noted punctuation errors in any part of the reference and the listing of duplicate references within the bibliography of an article.
Each included reference was independently checked by two reviewers. Disagreements among the two reviewers were resolved by consensus involving the three participating reviewers. We used descriptive statistics for the overall error rate and the rate in each error category. Punctuation errors and duplicate references were not included in the calculation of the overall error rates.
Results
The five journal issues included 37 publications with 1,356 references, of which 656 (48.4%) met the inclusion criteria. We identified 225 (34.3%) references that contained a total of 311 errors in the journal, authors, title, year, volume, or page number element of a reference. A single error occurred in 170 (75.6%) and multiple errors in 55 (24.4%) references (two errors in 51 and three errors in four references). Six (16%) of the 37 articles had no errors as defined in the study. ▶ summarizes the type and number of errors. We found 50 punctuation errors in any part of the 656 study references and one duplicate reference; neither error type was included in the overall error rate.
Table 1.
Reference Errors in Five Biomedical Informatics Journals
| J Am Med Inform Assoc. | J Biomed Inform. | Int J Med Inform. | Methods Inf Med. | Artif Intell Med. | |
|---|---|---|---|---|---|
| Total articles | 10 | 6 | 7 | 10 | 4 |
| Total references | 141 | 130 | 223 | 131 | 75 |
| Included references | 181 | 175 | 145 | 109 | 46 |
| References with any errors | 40 | 70 | 59 | 38 | 18 |
| Reference error rate (%) | 22.1 | 40.0 | 40.7 | 34.9 | 39 |
| Total errors | 56 | 105 | 73 | 52 | 25 |
| Journal title errors | 11 | 19 | 43 | 16 | 8 |
| Authors errors | 27 | 49 | 19 | 21 | 5 |
| Title errors | 12 | 22 | 7 | 8 | 6 |
| Year errors | 2 | 4 | 2 | 2 | 1 |
| Volume errors | 0 | 2 | 1 | 1 | 0 |
| Page number errors | 4 | 9 | 1 | 4 | 5 |
The author element was the most frequent error category and contained 121 (39.0%) errors in 99 references. An error in the first author's name was found 39 times (eight errors in the last name and 31 in the first or middle initial). An error in the list of coauthors was found 52 times (16 errors in the last name and 36 in the first or middle initial). A missing or additional author or an incorrect order of authors was found 30 times. Inaccurate journal titles accounted for 97 (31.2%) errors. An incorrect journal abbreviation was listed 93 times, and an incorrect journal name appeared four times. One reference listed Vis Graphics Image Proces instead of Comput Med Imaging Graph, one listed Int J Antimicrob Chemother instead of Int J Antimicrob Agents, one listed a reference to a submitted manuscript, and one mixed the authors and the title of a book with the journal, year, and volume of a journal issue. The title element included 55 (17.7%) errors, the year element 11 (3.5%), and the volume element four (1.3%). In 18 references, an inaccurate page listing accounted for 23 (7.4%) errors (four in the first page, nine in the last page, and five in both pages).
Discussion
In this study of five biomedical informatics journals, we found error rates between 22.1% and 40.7%. These error rates are within the range of reference accuracy rates found in other biomedical journals.2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26 We were, however, surprised by the high error rate, particularly because authors publishing in biomedical informatics journals are expected to be more familiar with and to more frequently use computerized tools that support the retrieval, management, and preparation of references and bibliographies. Authors publishing in the field of biomedical informatics should realize the importance of correct references. However, a previous study that examined the error rate in the library literature, in which lower error rates could also be expected due to author familiarity with bibliographies and references, did not identify a lower overall reference error rate between biomedical and library publications.5
Our study identified a large number of errors in the abbreviation of journal names. For readers who are unfamiliar with the biomedical informatics field, it is more difficult to identify the correct journal. For example, at the time of this study in summer 2004, a MEDLINE query for “Int J Med Inform [jour]” (International Journal of Medical Informatics) returned no results. At that time, a look up in MEDLINE's journal database returned the applicable MEDLINE abbreviation Int J Med Inf. The official abbreviation for “informatics” is “inform,” which was corrected in MEDLINE's electronic journal database after the completion of this study.
Errors in the author's name may only demonstrate a lack of attention to detail. However, the names of authors are not only critical for retrieving articles but are also relevant when author citations are used to measure research productivity.30 For example, the Science Citation Index® from Thomson ISI allows users to examine the citation frequency of an author's publications.31 Misspellings and variations in an author's name may prevent the program from retrieving all relevant citations and influence bibliometric measures and information retrieval approaches.
Many journals, including four considered in this study, publish papers in electronic format and provide readers with a direct link to the abstract or the original publication of the cited references. If errors exist in references, successful linking does not occur in online publications.22,23 Furthermore, unsuccessful linking increases the visibility of errors to the readers.
Our study examined only one issue of each journal, and it is possible that other issues contain fewer errors in references. In addition, our study did not examine MEDLINE's reference error rate, which has been reported to be 1.5% in a study analyzing publications from 1976 to 1998.32 We would have missed errors if authors incorporated references from MEDLINE that were incorrectly indexed. Such an error is present in reference 20 of the present article, in which MEDLINE lists the third author's name as “Kee WD,” whereas the original article lists the correct name of the author as “Ngan Kee WD” (same author as reference 26). This study included only references to journal articles that were indexed in MEDLINE; error rates for excluded references may differ from our results and need to be examined in a separate study.
Proposed strategies to minimize error rates from an author's perspective include the use of reference management software and careful verification with the original publications. Possible editorial office strategies include manual or automated in-house reference checking, asking reviewers to check references, requesting the submission of the first page of a cited reference at the time of manuscript submission, or limiting the number of references.3,14,20,21,22,23,25
Conclusion
Many references in biomedical informatics journals include errors. Inaccurate references are frustrating and may reflect poorly on the biomedical informatics field. Authors should review and validate the accuracy of references included in submitted manuscripts through verification with the original article.
References
- 1.International Committee of Medical Journal Editors. Uniform requirements for manuscripts submitted to biomedical journals: writing and editing for publication [home page on the Internet]. [updated 2003 Nov 9; cited 2004 Oct 22]. Available from: http://www.icmje.org
- 2.Arrowsmith JE. The accuracy of references in journal articles. Anaesthesia. 1996;51:517–8. [DOI] [PubMed] [Google Scholar]
- 3.Asano M, Mikawa K, Nishina K, Maekawa N, Obara H. Improvement of the accuracy of references in the Canadian Journal of Anaesthesia. Can J Anaesth. 1995;42:370–2. [DOI] [PubMed] [Google Scholar]
- 4.Asano M, Mikawa K, Nishina K, Maekawa N, Obara H. The accuracy of references in Anaesthesia. Anaesthesia. 1995;50:1080–2. [DOI] [PubMed] [Google Scholar]
- 5.Benning SP, Speer SC. Incorrect citations: a comparison of library literature with medical literature. Bull Med Libr Assoc. 1993;81:56–8. [PMC free article] [PubMed] [Google Scholar]
- 6.Celayir AC, Sander S, Celayir S. Accuracy of references in the pediatric surgery journals. J Pediatr Surg. 2003;38:653–4. [DOI] [PubMed] [Google Scholar]
- 7.de Lacey G, Record C, Wade J. How accurate are quotations and references in medical journals? BMJ. 1985;291:884–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Eichorn P, Yankauer A. Do authors check their references? A survey of accuracy of references in three public health journals. Am J Public Health. 1987;77:1011–2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Evans JT, Nadjari HI, Burchell SA. Quotational and reference accuracy in surgical journals. A continuing peer review problem. JAMA. 1990;263:1353–4. [PubMed] [Google Scholar]
- 10.Fenton JE, Brazier H, De Souza A, Hughes JP, McShane DP. The accuracy of citation and quotation in otolaryngology/head and neck surgery journals. Clin Otolaryngol. 2000;25:40–4. [DOI] [PubMed] [Google Scholar]
- 11.George PM, Robbins K. Reference accuracy in the dermatologic literature. J Am Acad Dermatol. 1994;31:61–4. [DOI] [PubMed] [Google Scholar]
- 12.Goldberg R, Newton E, Cameron J, et al. Reference accuracy in the emergency medicine literature. Ann Emerg Med. 1993;22:1450–4. [DOI] [PubMed] [Google Scholar]
- 13.Hansen ME, McIntire DD. Reference citations in radiology: accuracy and appropriateness of use in two major journals. AJR Am J Roentgenol. 1994;163:719–23. [DOI] [PubMed] [Google Scholar]
- 14.Jackson K, Porrino JA Jr, Tan V, Daluiski A. Reference accuracy in the Journal of Hand Surgery. J Hand Surg [Am]. 2003;28:377–80. [DOI] [PubMed] [Google Scholar]
- 15.McLellan MF, Case LD, Barnett MC. Trust, but verify. The accuracy of references in four anesthesia journals. Anesthesiology. 1992;77:185–8. [PubMed] [Google Scholar]
- 16.Nishina K, Asano M, Mikawa K, Maekawa N, Obara H. Accuracy of references in Anesthesia & Analgesia does not improve. Anesth Analg. 1995;80:641–2. [DOI] [PubMed] [Google Scholar]
- 17.Nishina K, Mikawa K, Asano M, Maekawa N, Obara H. Reference accuracy in Critical Care Medicine. Crit Care Med. 1995;23:1610–1. [DOI] [PubMed] [Google Scholar]
- 18.O'Connor AE. A review of the accuracy of references in the Journal Emergency of Medicine. Emerg Med (Fremantle). 2002;14:139–41. [DOI] [PubMed] [Google Scholar]
- 19.Orlin W, Pehling J, Pogrel MA. Do authors check their references? A survey of 500 references from the Journal of Oral and Maxillofacial Surgery. J Oral Maxillofac Surg. 1996;54:200–2. [DOI] [PubMed] [Google Scholar]
- 20.Roach VJ, Lau TK, Ngan Kee WD. The quality of citations in major international obstetrics and gynecology journals. Am J Obstet Gynecol. 1997;177:973–5. [DOI] [PubMed] [Google Scholar]
- 21.Siebers R. The accuracy of references of three allergy journals. J Allergy Clin Immunol. 2000;105:837–8. [DOI] [PubMed] [Google Scholar]
- 22.Siebers R. How accurate are references in clinical chemistry? Clin Chem. 2001;47:606–7. [PubMed] [Google Scholar]
- 23.Siebers R, Holt S. Accuracy of references in five leading medical journals. Lancet. 2000;356:1445. [DOI] [PubMed] [Google Scholar]
- 24.Sutherland AG, Craig N, Maffulli N, Brooksbank A, Moir JS. Accuracy of references in the orthopaedic literature. J Bone Joint Surg Br. 2000;82:9–10. [DOI] [PubMed] [Google Scholar]
- 25.Vargas-Origel A, Gomez-Martinez G, Vargas-Nieto MA. The accuracy of references in paediatric journals. Arch Dis Child. 2001;85:497–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Ngan Kee WD, Roach VJ, Lau TK. How accurate are references in the Australian and New Zealand Journal of Surgery? Aust N Z J Surg. 1997;67:417–9. [DOI] [PubMed] [Google Scholar]
- 27.MEDLINE [database on the Internet]. Bethesda (MD): National Library of Medicine. [cited 2004 Oct 22]. Available from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi.
- 28.MEDLINE Journals Database [database on the Internet]. Bethesda (MD): National Library of Medicine [cited 2004 Oct 22]. Available from: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=journals
- 29.International Committee of Medical Journal Editors. Uniform requirements for manuscripts submitted to biomedical journals: sample references [updated 2003 Nov 9; cited 2004 Oct 22]. Available from: http://www.nlm.nih.gov/bsd/uniform_requirements.html
- 30.Hecht F, Hecht BK, Sandberg AA. The journal “impact factor”: a misnamed, misleading, misused measure. Cancer Genet Cytogenet. 1998;104:77–81. [DOI] [PubMed] [Google Scholar]
- 31.Thomson ISI [home page on the Internet]. Thomson Corporation, Stamford, CT. [cited 2004 Oct 22]. Available from: http://www.isinet.com.
- 32.De S, Jones T, Brazier H, Jones AS, Fenton JE. The accuracy of MEDLINE and journal contents pages for papers published in Clinical Otolaryngology. Clin Otolaryngol. 2001;26:39–42. [DOI] [PubMed] [Google Scholar]