Skip to main content
NCBI home page
JBJS Open Access logoLink to JBJS Open Access
. 2021 Aug 25;6(3):e21.00019. doi: 10.2106/JBJS.OA.21.00019

Quotation Errors in High-Impact-Factor Orthopaedic and Sports Medicine Journals

Aaron Gazendam 1,a, Daniel Cohen 1, Samuel Morgan 2, Seper Ekhtiari 1, Michelle Ghert 1
PMCID: PMC8386904  PMID: 34476322

Abstract

Background:

Inappropriate referencing of the existing literature has the potential to propagate false information. Quotation errors are defined as citations in which the referenced article fails to substantiate the authors’ claims. The aim of this study was to determine the rate of quotation errors in high-impact general orthopaedic and sports medicine journals and to determine whether there are article or journal-related factors that are related to the rate of inaccuracies.

Methods:

A total of 250 citations from the 5 orthopaedic and sports medicine journals with the highest impact factors in 2019 (per Journal Citation Reports) were chosen using a random sequence generator. Reviewers rated the chosen citations by comparing the claims made by the authors with the data and conclusions of the referenced source to determine whether quotation errors were present. Logistic regression was utilized to assess for article- and journal-related factors related to the rate of quotation errors.

Results:

The overall quotation error rate was 13.6%. A total of 2.8% of the claims were completely unsubstantiated. The number of quotation errors did not significantly differ between the included journals. Single citations were significantly more likely than string citations to result in citations that could not be fully substantiated (χ2 = 4.57; odds ratio = 2.22; 95% confidence interval = 1.06 to 4.66; p = 0.03). No relationship was found between the rate of quotation errors and the total number of citations in the article, study type, or the graded level of evidence of the article.

Conclusions:

Quotation errors in high-impact factor orthopaedic and sports medicine journals are common. This is particularly important given the higher likelihood that studies in these journals are cited elsewhere, thus propagating the inaccuracies. Efforts from both authors and journals are needed to reduce quotation errors in the orthopaedic literature.

In scientific communication, appropriate citations support claims made by authors and help provide the reader with a context of the subject matter in relation to the body of literature as a whole. Inappropriate referencing of the existing literature has the potential to propagate false information1. In general, there are 2 types of inaccuracies. The first is referred to as a “citation error,” which includes bibliographic errors of the citation style, missing authorships, or missing citations2. The second is a “quotation error,” defined as a citation in which the referenced article fails to substantiate the authors’ claims2. Quotation errors are more serious than citation errors, as they may mislead readers and have the potential to affect patient care1,3-5.

Several studies have documented the rate of quotation errors in various areas of the medical literature2,6-8. Recently (2020), Smith and Cumberledge demonstrated a 25% quotation-error rate in high-impact general science journals2. Within orthopaedics, quotation errors have been quantified in specific areas of interest, including pediatric surgery, foot and ankle surgery, scaphoid pathology, and spinal surgery6,7,9,10. However, the quotation-error rate in high-impact-factor general orthopaedic and sports medicine journals has not been investigated, to our knowledge. This is of critical importance because papers in high-impact journals are likely to be cited extensively, thus substantially propagating any potential quotation errors. Additionally, the majority of the literature on quotation errors has not addressed the relationship between article-related and journal-related factors and the accuracy of quotations6.

The aims of the present study were to determine the rate of quotation errors in high-impact-factor general orthopaedic and sports medicine journals and to determine whether there are article- or journal-related factors that are related to the rate of inaccuracies. We hypothesized that there would be differences in quotation errors based on impact factor and that single citations would have significantly fewer quotation errors than string citations (i.e., multiple citations used to support the same claim).

Materials and Methods

Journal and Citation Selection

The 5 orthopaedic and sports medicine journals with the highest impact factors in 2019 per the Web of Science, Journal Citation Reports were utilized: British Journal of Sports Medicine, The American Journal of Sports Medicine, The Journal of Bone & Joint Surgery, Sports Medicine, and Arthroscopy: The Journal of Arthroscopic and Related Surgery11. All articles published in 2019 were extracted into a collaborative spreadsheet12. A random sequence generator was utilized to select 50 articles from each journal, for a total of 250 articles13. All articles published in each of the 5 journals were included in the random sequence generator. Articles were excluded if (1) the manuscript was without references, or (2) the manuscript was a personal communication, letter to the editor, or conference proceeding. Excluded manuscripts were replaced by others using the random sequence generator. From each article, a random citation was then chosen using the same random sequence generator13. The first citation of the reference in the article was located. For consistency, the first use of the citation was always employed, regardless of whether it was a single citation or part of a string citation.

Quotation Accuracy Assessment

Two authors (D.C. and S.M.) rated the chosen citations by comparing the claims made by the authors with the data and conclusions of the referenced source. The first 20 citations chosen were analyzed in a duplicate, blinded fashion to resolve any discrepancies. Interobserver agreement for the assessments of quotation rating was calculated using the Cohen kappa (κ) statistic; κ values were interpreted according to McHugh, and the agreement was found to be almost perfect (κ = 0.96; 95% confidence interval [CI] = 0.88 to 1.00)14. Following this, the citations were analyzed independently by the 2 reviewers. Quotation inaccuracies were categorized according to the scoring system proposed by Smith and Cumberledge, as described below2:

Quotation Errors

  1. Fully substantiated: the claim made by the article was entirely substantiated by the reference cited. If the citation was part of a string citation, it was scored as fully substantiated if the quotation substantiated at least part of the claim as long as the references in the rest of the citation string were able to substantiate the claim.

  2. Partially substantiated: the claim made by the article contained a minor error. This category was differentiated from the Unsubstantiated category by the following: does the error invalidate the purpose of the citation? If not, partially substantiated was utilized. (For example, if an article falsely states that posttraumatic arthritis occurs in 10% of patients with ankle fractures instead of 14%, it would be considered partially substantiated.)

  3. Unsubstantiated: the claim made by the article was not substantiated at all by the citation used. This occurred because the citation was either contradictory to, unrelated to, or failed to back up the claims.

  4. Impossible to substantiate: the claim was logically impossible to substantiate using a reference. This was due to the article citing a reference for actions undertaken as part of the study. An outside reference, written prior to the current study, would not be able to substantiate actions of the current study.

Data Extraction

To determine whether any journal- or article-related factors were associated with rates of quotation error, the following journal and article characteristics were extracted: journal impact factor, single or string citation, and the total number of citations in the article. The study type was recorded and categorized as clinical, biomechanical, methodological, basic science, systematic review or meta-analysis, or unstructured review, narrative, or commentary. A level of evidence (I through V) was assigned to each included article on the basis of the classification system adopted by the American Academy of Orthopaedic Surgeons (AAOS)15.

Statistical Analysis

Descriptive statistics were utilized to describe the overall quotation errors and characteristics of the included studies. Logistic regression was used to identify any variables, or set of variables, that significantly predicted a citation that could not be fully substantiated (the dependent variable). The independent variables were (1) string or single citation, (2) journal, (3) level of evidence, (4) type of study, and (5) total number of citations in the paper. Each independent variable was entered into a univariate logistic-regression model to assess for a relationship with the dependent variable, and all variables with a p value of <0.25 were then included in the multivariable logistic-regression model16. Given the small number of variables and unclear relative importance of each, it was determined a priori that any variables meeting this threshold would be entered into the multivariable model. G*Power 3.1 software (Heinrich-Heine-Universität Düsseldorf) was used to perform post-hoc power analysis. Based on a post-hoc power analysis using a beta value of 0.80 and alpha value of 0.05, 215 observations (citation ratings) would be sufficient to detect an odds ratio of 1.5.

Source of Funding

No external funding was received in the preparation of this manuscript.

Results

Article Characteristics

A total of 250 articles were reviewed, 50 from each of the 5 included journals. The median number of references cited in each article was 36 (range, 2 to 258). Of the citations evaluated, 114 (46%) were single citations and 136 (54%) were part of a string of citations. The 2-year impact factor of the included journals ranged from 11.65 (British Journal of Sports Medicine) to 4.4 (Arthroscopy: The Journal of Arthroscopic and Related Surgery)11. Of the 250 articles, 19 (8%) were Level I, 24 (10%) were Level II, 68 (27%) were Level III, 78 (31%) were Level IV, and 61 (24%) were Level V. There were 94 clinical studies (38%), 63 systematic reviews and meta-analyses (25%), 62 unstructured reviews, narratives, or commentaries (25%), 17 biomechanical studies (7%), 8 basic science studies (3%), and 6 methodological studies (2%).

Quotation Errors

A total of 250 references were evaluated and rated on the basis of their accuracy. Of the 250 references, 34 (13.6%) had quotation errors (Table I). With respect to the type of quotation error, 24 (71%) of the 34 were partially substantiated claims, 7 (21%) were unsubstantiated claims, and 3 (9%) were impossible-to-substantiate claims.

TABLE I.

Quotation Errors by Journal

British Journal of Sports Medicine The American Journal of Sports Medicine Sports Medicine The Journal of Bone & Joint Surgery Arthroscopy Total No. (%)
Fully substantiated (no.) 43 41 44 45 43 216 (86.4%)
Partially substantiated (no.) 4 7 3 5 5 24 (9.6%)
Unsubstantiated (no.) 3 1 3 0 0 7 (2.8%)
Impossible to substantiate (no.) 0 1 0 0 2 3 (1.2%)
Total no. 50 50 50 50 50 250
Open in a new tab

Journal and Article Factors

The number of quotation errors did not significantly differ between the included journals. The only independent variable that had a p value of <0.25 was whether the citation was a single citation or part of a string of citations (Table II); this significantly predicted the likelihood that a citation could not be fully substantiated. Single citations were significantly more likely to result in citations that could not be fully substantiated (χ2 = 4.57; odds ratio = 2.22; 95% CI = 1.06 to 4.66; p = 0.03). Among single citations, 92 (81.4%) of 113 were fully substantiated, compared with 124 (90.5%) of 137 of string citations. No relationship was found between the rate of quotation errors and the total number of citations in the article, study type, or the graded level of evidence of the article.

TABLE II.

Quotation Errors by Citation Type

Single Citation String Citation
Fully substantiated 92 124
Partially substantiated 16 8
Unsubstantiated 4 3
Impossible to substantiate 1 2
Open in a new tab

Discussion

In this study, we quantified the rate and type of quotation errors present in orthopaedic and sports medicine journals with a high impact factor. Although accurate referencing in a scientific article appears to be trivial and straightforward on the surface, quotation errors are prevalent even among top journals. The most notable finding of the current study is that quotation errors were present in 13.6% of the 250 analyzed citations in high-impact-factor orthopaedic and sports medicine journals. Moreover, 2.8% of the claims were completely unsubstantiated by their corresponding citations. This is particularly important given the higher likelihood that papers in these highly cited journals will be further referenced, thus propagating the inaccuracies.

Findings in Relation to Previous Literature

These findings are consistent with the published literature within other orthopaedic surgery subspecialties and other areas of medicine6-8,10. Luo et al. demonstrated a quotation error of 20% in the foot and ankle literature7. Montenegro et al. recently demonstrated similar findings in the spine literature, with a quotation-error rate of 18.6%10. These findings have been demonstrated in other surgical specialties and subspecialties, with a quotation error rate of 7.8% to 18.6%10,17-19. Across disciplines, the quotation error varies considerably, with a systematic review in 2015 demonstrating a range from 6.7% to 83% among 28 included studies8.

In the current study, string citations were significantly less likely to contain quotation errors compared with single citations. The majority of previous studies examining quotation errors did not examine the relationship between string citations and the substantiation of claims. However, a recent study examining high-impact medical journals found, similar to the present study, a lower rate of quotation errors among string citations compared with single citations2. This finding is logical given that citations in a string often substantiate the same claim with overlapping and corroborative information20. This may also be due, in part, to methodology: we did not require a citation that was part of a string to fully substantiate the claim, but only to partially substantiate the claim, as long as the references in the rest of the citation string were able to substantiate the claim.

Implications

Quotation errors have the potential to mislead clinicians and affect patient care1,4. For example, a classic paper by Yablon et al. demonstrated the importance of the lateral malleolus in the management of ankle fracture fixation21. The authors famously demonstrated that the “displacement of the talus faithfully followed that of the lateral malleolus.”21 However, this research work was carried out on exclusively bimalleolar ankle fractures, and the findings have often been misquoted in support of surgical fixation of isolated lateral malleolar fractures1,22. Misquotation of this classic orthopaedic literature has led to a surgical dogma associated with this injury pattern that is not supported by the current literature1. Similarly, from the sports medicine literature, recent guidelines recommend that exercise testing should last between 8 and 12 minutes to attain a reliable VO2MAX (maximal oxygen uptake)23. However, these recommendations are based on a single study in 1983 of 5 healthy males24. The results of the original study have been extrapolated and applied to a variety of patient populations25. This example highlights the concerns regarding quotation errors, even when the claims made are partially substantiated.

There is a growing body of literature that has suggested potential causes for the high rates of quotation errors seen across the medical literature8. Given that quotation errors often occur because the claim cannot be supported by the cited reference, it suggests that the original reference was not read by the authors6. Statistical modeling also suggests that the majority of citations are copied from other article reference lists and not the original text26. Similarly, it is common practice for authors to attribute findings to a review and not to the original research27. It is important to note that the perpetuation of misquotations of the literature is rarely due to purposeful deception and more commonly because the authors did not read the original text. It has been stated that citation of the original text is preferred over abstracts or reviews of the original research to reduce the risk of error28. The expansion of open-access journals and full-text articles online will allow authors to access original articles and may reduce the incidence of misquotations. Greater diligence and precision when referencing will reduce the risk of quotation errors.

A number of authors have suggested strategies that journals can undertake to minimize the presence of quotation errors in their articles8. Journals that provide technical editing of the manuscript between acceptance and publication have been shown to improve readability and reduce both citation and quotation errors29. Buijze et al. demonstrated that journals with dedicated technical editing services had significantly lower rates of quotation errors when compared with the other journals in their review6. Other authors have suggested that journals explicitly state that quotation accuracy is expected and to ask for confirmation of quotation accuracy prior to submission30,31. However, it is our opinion that the burden rests principally with the authors to ensure that quotation errors are minimized. Requiring the authors to state, at the time of manuscript submission, that the article has been reviewed for quotation errors may alert authors to the seriousness of the issue and reduce errors.

Strengths and Limitations

A strength of this study is the novelty of its findings; this is, to our knowledge, the first study to document the rate of quotation errors in high-impact journals in orthopaedics and sports medicine. The major limitation of this study was its relatively small sample size when compared with the total number of citations that were utilized in the 5 included journals. Given the low rate of unsubstantiated or impossible-to-substantiate claims, the study was not powered to detect differences between the journals in those types of claims. However, the use of a random sequence generator reduced the risk of bias, and the current sample size is in line with previous literature on the topic2,10,18,19.

Conclusions

We found that quotation errors were common (13.6% of 250 citations) in high-impact orthopaedic and sports medicine journals, with single citations having a higher rate of errors. Given the higher likelihood that studies in these journals are cited elsewhere, this indicates a risk of the propagation of inaccuracies. Authors should be responsible for reviewing each citation for quotation accuracy prior to manuscript submission. Both author and journal vigilance are required to reduce quotation errors and thus improve the accuracy of the body of literature as a whole, and in particular in high-impact journals that are likely to be further cited.

Footnotes

Investigation performed at McMaster University, Hamilton, Ontario, Canada

Disclosure: The Disclosure of Potential Conflicts of Interest forms are provided with the online version of the article (http://links.lww.com/JBJSOA/A311).

References

  • 1.Smith G. The isolated lateral malleolar fracture: where are we and how did we get here? Surgeon. 2013February;11(1):6-9. Epub 2012 Mar 28. [DOI] [PubMed] [Google Scholar]
  • 2.Smith N, Cumberledge A. Quotation errors in general science journals. Proc Royal Soc A Math Phys Sci. 2020;476:20200538. [Google Scholar]
  • 3.Rivkin A. Manuscript Referencing Errors and Their Impact on Shaping Current Evidence. Am J Pharm Educ. 2020July;84(7):ajpe7846. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Porrino JA, Jr, Tan V, Daluiski A. Misquotation of a commonly referenced hand surgery study. J Hand Surg Am. 2008January;33(1):2-7. [DOI] [PubMed] [Google Scholar]
  • 5.Leung PTM, Macdonald EM, Stanbrook MB, Dhalla IA, Juurlink DNAA. A.1980 Letter on the Risk of Opioid Addiction. N Engl J Med. 2017June1;376(22):2194-5. [DOI] [PubMed] [Google Scholar]
  • 6.Buijze GA, Weening AA, Poolman RW, Bhandari M, Ring D. Predictors of the accuracy of quotation of references in peer-reviewed orthopaedic literature in relation to publications on the scaphoid. J Bone Joint Surg Br. 2012February;94(2):276-80. [DOI] [PubMed] [Google Scholar]
  • 7.Luo M, Li CC, Molina D, 4th, Andersen CR, Panchbhavi VK. Accuracy of citation and quotation in foot and ankle surgery journals. Foot Ankle Int. 2013July;34(7):949-55. Epub 2013 Feb 4. [DOI] [PubMed] [Google Scholar]
  • 8.Jergas H, Baethge C. Quotation accuracy in medical journal articles-a systematic review and meta-analysis. PeerJ. 2015October27;3:e1364. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Davids JR, Weigl DM, Edmonds JP, Blackhurst DW. Reference accuracy in peer-reviewed pediatric orthopaedic literature. J Bone Joint Surg Am. 2010May;92(5):1155-61. [DOI] [PubMed] [Google Scholar]
  • 10.Montenegro TS, Hines K, Partyka PP, Harrop J. Reference accuracy in spine surgery. J Neurosurg Spine. 2020September25;1:1-5. Epub 2020 Sep 25. [DOI] [PubMed] [Google Scholar]
  • 11.2019 Journal Citation Reports. Web of Science Group. 2019. Accessed 2020 Dec 15. https://clarivate.com/webofsciencegroup/wp-content/uploads/sites/2/dlm_uploads/2019/08/JCR_Full_Journal_list140619.pdf [Google Scholar]
  • 12.Google Sheets. Accessed 2020 Dec 16. https://docs.google.com/spreadsheets/u/0/
  • 13.Random Sequence Generator. Random.org. Accessed 2020 Dec 17. https://www.random.org/sequences/
  • 14.McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22(3):276-82. [PMC free article] [PubMed] [Google Scholar]
  • 15.Wright J. Levels of Evidence and Grades of Recommendations: An Evaluation of Literature. 2005. Accessed 2021 Jun 15. https://www.researchgate.net/publication/238721645_Levels_of_Evidence_and_Grades_of_Recommendations_An_Evaluation_of_Literature
  • 16.Bursac Z, Gauss CH, Williams DK, Hosmer DW. Purposeful selection of variables in logistic regression. Source Code Biol Med. 2008December16;3:17. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Reddy MS, Srinivas S, Sabanayagam N, Balasubramanian SP. Accuracy of references in general surgical journals—an old problem revisited. Surgeon. 2008April;6(2):71-5. [DOI] [PubMed] [Google Scholar]
  • 18.Armstrong MF, Conduff JH, 3rd, Fenton JE, Coelho DH. Reference Errors in Otolaryngology-Head and Neck Surgery Literature. Otolaryngol Head Neck Surg. 2018August;159(2):249-53. Epub 2018 Apr 24. [DOI] [PubMed] [Google Scholar]
  • 19.Al-Benna S, Rajgarhia P, Ahmed S, Sheikh Z. Accuracy of references in burns journals. Burns. 2009August;35(5):677-80. Epub 2009 Mar 20. [DOI] [PubMed] [Google Scholar]
  • 20.Moravcsik MJ, Murugesan P. Some Results on the Function and Quality of Citations. Soc Stud Sci. 1975;5(1):86-92. [Google Scholar]
  • 21.Yablon IG, Heller FG, Shouse L. The key role of the lateral malleolus in displaced fractures of the ankle. J Bone Joint Surg Am. 1977March;59(2):169-73. [PubMed] [Google Scholar]
  • 22.Vander Griend R, Michelson JD, Bone LB. Instructional Course Lectures, The American Academy of Orthopaedic Surgeons - Fractures of the Ankle and the Distal Part of the Tibia. J Bone Joint Surg Am. 1996;78(11):1772-83. [PubMed] [Google Scholar]
  • 23.American Thoracic SocietyAmerican College of Chest Physicians. ATS/ACCP statement on cardiopulmonary exercise testing. Am J Respir Crit Care Med. 2003;167(2):211-77. [DOI] [PubMed] [Google Scholar]
  • 24.Buchfuhrer MJ, Hansen JE, Robinson TE, Sue DY, Wasserman K, Whipp BJ. Optimizing the exercise protocol for cardiopulmonary assessment. J Appl Physiol Respir Environ Exerc Physiol. 1983November;55(5):1558-64. [DOI] [PubMed] [Google Scholar]
  • 25.Midgley AW, Bentley DJ, Luttikholt H, McNaughton LR, Millet GP. Challenging a dogma of exercise physiology: does an incremental exercise test for valid VO 2 max determination really need to last between 8 and 12 minutes? Sports Med. 2008;38(6):441-7. [DOI] [PubMed] [Google Scholar]
  • 26.Simkin MV, Roychowdhury VP. Stochastic modeling of citation slips. Scientometrics. 2005;62(3):367-85. [Google Scholar]
  • 27.Gavras H. Inappropriate attribution: the “Lazy Author Syndrome”. Am J Hypertens. 2002September;15(9):831-831. [DOI] [PubMed] [Google Scholar]
  • 28.ICMJE. Recommendations. 2019. Accessed 2020 Dec 16. http://www.icmje.org/recommendations/
  • 29.Wager E, Middleton P. Effects of technical editing in biomedical journals: a systematic review. JAMA. 2002June5;287(21):2821-4. [DOI] [PubMed] [Google Scholar]
  • 30.Wright M, Armstrong JS. The Ombudsman: Verification of Citations: Fawlty Towers of Knowledge? Interfaces. 2008;38:125-39. [Google Scholar]
  • 31.Goldberg R, Newton E, Cameron J, Jacobson R, Chan L, Bukata WR, Rakab A. Reference accuracy in the emergency medicine literature. Ann Emerg Med. 1993September;22(9):1450-4. [DOI] [PubMed] [Google Scholar]

Articles from JBJS Open Access are provided here courtesy of Wolters Kluwer Health

RESOURCES