Abstract
Background
With increased legislative efforts to utilize evidence-based medicine as a guide for clinical practice, orthopaedists feel increasing pressure to publish research in higher-quality journals that reach a larger audience. Impact factor (IF) is used to quantify and rank journal apparent quality, and is the most standardized method for journal appraisal. In this study, we assessed the trends for IF among orthopaedic journals and compared these trends to those of medicine and general surgery journals.
Methods
Journal IFs from Journal Citation Reports (JCR) between the years 2010 to 2016 were obtained and analyzed for trends. Only journals that were considered primarily orthopaedic journals were included. The top 10 journals by IF in both internal medicine and surgery were also included for comparison. Each journal was analyzed by IF, and trends across time were noted. The differences in mean IF between orthopaedic specialty groups were analyzed using an independent samples t-test.
Results
The mean IF of orthopaedic increased from 1.4 (range, 0.0–3.9) in 2010 to 1.9 (range, 0.5–5.7) in 2016. In 2016, the percentage of English journals increased to 87.3% (n=48), while the percentage of journals published in the United States was 47.3% (n=26). There was a significant difference between the IF of journals published in English and those published in other languages (P=0.004). The mean IF of both general and specialized orthopaedic journals increased from 2010 to 2016, but the difference was nonsignificant. The mean IF of the top 10 journals in both surgery and internal medicine also increased from 2010 to 2016, but the increase was also nonsignificant.
Conclusions
Overall, the mean IF for peer-reviewed orthopaedic journals has increased in the past years, as has the number of journals. English journals from the United States continue to have the largest impact when compared to non-English journals and journals from outside the United States. Future studies should aim to better qualify journal impact, while limiting confounders such as self-citation.
Keywords: Orthopaedics, research, impact factor (IF)
Introduction
Increasing legislative efforts to improve quality of care has led to a rejuvenated emphasis on utilizing evidence-based medicine as a guide for clinical practice (1,2). These efforts have spurred an increased demand for the academic community to publish high-quality studies. To meet this demand, there has been a significant rise in the number of scientific medical journals. According to Journal Citation Report (JCR)—Thomson Reuters, the number of “clinical medicine journals” published between the years 1999 and 2010 grew from 1,291 to 1,986 (3). However, despite the increase in the total number of scientific journals, investigators still seek to publish in journals that garner the largest audiences and potential impact. Particularly for orthopaedics, in which new research findings often dictate healthcare policy and reimbursement, the pressure to publish in journals with the largest impact is of paramount importance.
In 1951, Eugene Garfield developed an algorithm, designated “impact factor” (IF), to quantify and rank journal quality (4). IF is calculated by measuring the frequency with which articles from a journal are cited over the total number of published articles from the same journal in a particular year (Figure 1). This algorithm was devised with the intent of controlling for journals with a larger number of publications. Despite some objection against the use IF as a means to measure journal quality, IF remains the most standardized method for journal appraisal and is commonly used by medical librarians in selecting journals for their institutions (5-9).
Figure 1.
Impact factor equation.
Moverley et al. (10) investigated IF trends among orthopaedic journals between the years 2000 and 2010 and reported an increase in the mean IF for orthopaedic journals. We aimed to update their study by assessing IF trends in orthopaedic journals between the years 2010 to 2016. More specifically, we assessed trends for IF trends among orthopaedic journals and aimed to compare these trends to that of medicine and general surgery journals.
Methods
Journal IFs were obtained directly from JCR for the years 2010–2016 and analyzed for trends. Two authors (NS Mohamed and CU Gwam) screened and categorized each journal into either general, or subspecialty journals, based on whether the journal covered broad topics in orthopaedics or contained focused deeper knowledge (Table 1). Only journals that were considered primarily orthopaedic journals were included, as with previous studies (11,12). We also included the top 10 journals by IF in both internal medicine and surgery for comparison (Table 2).
Table 1. Journals in 2016 primarily focusing on orthopaedics, subdivided into general and specialist journal (accepted JCR abbreviations were used).
| Title | Language | Published |
|---|---|---|
| General orthopaedic journals (n=27) | ||
| Acta CHIR ORTHOP TR | Czech | Czech Republic |
| Acta Orthop | English | UK |
| Acta Orthop Belg | English | Belgium |
| Acta Orthop Traumato | Turkish | Turkey |
| Acta Ortop Bras | English | Brazil |
| Am J Sport Med | English | USA |
| Arch Orthop Traum Su | English | Germany |
| Bmc Musculoskel Dis | English | UK |
| Bone Joint J | English | UK |
| Bone Joint Res | English | UK |
| Clin Orthop Relat R | English | USA |
| Eklem Hast Cerrahisi | Turkish | Turkey |
| Indian J Orthop | English | India |
| Int Orthop | English | USA |
| J Am Acad Orthop Sur | English | USA |
| J Bone Joint Surg Am | English | USA |
| J Orthop Res | English | USA |
| J Orthop Sci | English | USA |
| J Orthop Surg Res | English | UK |
| J Orthop Surg-Hong K | English | China |
| J Orthop Trauma | English | USA |
| Oper Orthop Traumato | German | Germany |
| Orthop Clin N Am | English | USA |
| Orthop Surg | English | China |
| Orthop Traumatol-Sur | English | France |
| Orthopade | German | Germany |
| Orthopedics | English | USA |
| Specialist orthopaedic journals (n=28) | ||
| Arthroscopy | English | USA |
| Cartilage | English | USA |
| Chir Main | French | France |
| Connect Tissue Res | English | UK |
| Eur Spine J | English | USA |
| Foot Ankle Clin | English | USA |
| Foot Ankle Int | English | USA |
| Foot Ankle Surg | English | UK |
| Hand Clin | English | USA |
| Hip Int | English | Italy |
| Int J Shoulder Surg | English | India |
| J Arthroplasty | English | USA |
| J Foot Ankle Res | English | UK |
| J Foot Ankle Surg | English | USA |
| J Hand Surg-Am | English | USA |
| J Hand Surg-Eur Vol | English | UK |
| J Knee Surg | English | USA |
| J Pediatr Orthop B | English | USA |
| J Pediatr Orthoped | English | USA |
| J Plast Surg Hand Su | English | Sweden |
| J Shoulder Elb Surg | English | USA |
| Knee | English | Netherlands |
| Knee Surg Sport Tr A | English | Germany |
| Osteoarthr Cartilage | English | UK |
| Skeletal Radiol | English | USA |
| Spine | English | USA |
| Spine J | English | USA |
| Z Orthop Unfallchir | German | Germany |
JCR, Journal Citation Report.
Table 2. Journals focusing on orthopaedics added and removed between 2010 and 2016 (accepted JCR abbreviations were used).
| Title | Language | Published |
|---|---|---|
| Orthopaedic journals added (n=14) | ||
| Acta Ortop Bras | English | Brazil |
| Acta Chir Orthop Tr | Czech | Czech Republic |
| J Orthop Surg-Hong K | English | China |
| J Plast Surg Hand Su | English | Sweden |
| Foot Ankle Clin | English | USA |
| Int J Shoulder Surg | English | India |
| Orthop Surg | English | China |
| Foot Ankle Surg | English | UK |
| J Foot Ankle Res | English | UK |
| J Orthop Surg Res | English | UK |
| J Knee Surg | English | USA |
| Cartilage | English | USA |
| Bone Joint Res | English | UK |
| Bone Joint J | English | UK |
| Orthopaedic journals removed (n=7) | ||
| Minerva Ortop Trauma | Italian | Italy |
| Med Chir Pied | French | France |
| Eur J Orthop Surg Tr | English | France |
| Osteologie | German | Switzerland |
| Rev Bras Fisioter | Portuguese | Brazil |
| Rev Chir Orthop | Multi-language | France |
| J Bone Joint Surg Br | English | UK |
JCR, Journal Citation Report.
Each journal was analyzed by IF, and trends across time were evaluated. The differences in mean IF between orthopaedic specialty groups, geographical regions, and language of publication were analyzed using an independent samples t-test. The trends in mean IF for internal medicine and surgery were analyzed using a one-way Analysis of Variance (ANOVA). Significance was set at a value of P<0.05. All analysis was conducted using SPSS version 24 (IBM corporation, Armonk, New York, USA).
Results
In 2010, the number of orthopaedic journals with IF totaled 48, and increased to 55 in 2016 (Table 1). There were 27 general orthopaedic journals published, and 28 specialist journals published. The mean IF increased from 1.4 (range, 0.0–3.9) in 2010 to 1.9 (range, 0.5–5.7) in 2016. In 2010, English was the most common language of publication (n=37, 77.1%), followed by German (n=4, 8.3%). The majority of journals were published in the United States (n=23, 47.9%). In 2016, the percentage of English journals increased to 87.3% (n=48), while the percentage of journals published in the United States remained about the same (n=26, 47.3%). A significant difference found between the IF of journals published in English and those published in other languages (P=0.004). IFs for 2016 ranged from 5.7 for the American Journal of Sports Medicine to 0.5 for the Acta Ortopedica Brasileira (Figures 2,3).
Figure 2.
2016 impact factors of the journals listed in the JCR deemed to be primarily orthopaedic journals (as per Table 1), listed by accepted JCR abbreviation. JCR, Journal Citation Report.
Figure 3.
2010 impact factors of the journals listed in the JCR deemed to be primarily orthopaedic journals (as per Table 1), listed by accepted JCR abbreviation. JCR, Journal Citation Report.
The mean IF of both general and specialized orthopaedic journals increased from 2010 to 2016 (Figure 4). There was no significant difference found between the mean IF of these groups in any of the years included (2010: P=0.563; 2011: P=0.945; 2012: P=0.807; 2013: P=0.689; 2014: P=0.807; 2015: P=0.812; 2016: P=0.889).
Figure 4.
Change in mean impact factor of orthopaedic journals between 2010 and 2016. Orthopaedic journals (as per Table 1), listed by accepted JCR abbreviation. JCR, Journal Citation Report.
The majority of journals published in both 2010 and 2016 experienced an increase in IF (range, 0.04–1.9), while 5 journals had a decrease in IF: Spine (−0.011); Spine Journal (−0.062); BMC Musculoskeletal Disorders (−0.148); Connective Tissue Research (−0.261); and Journal of Orthopaedic Research (−0.284) (Figure 5).
Figure 5.
Change in impact factor from 2010 to 2016 for orthopaedic journals.
The mean IF of the top 10 journals in both surgery and internal medicine increased from 2010 to 2016 (Figure 6). However, the increase within each specialty was found to be nonsignificant [internal medicine (IM): P=0.202; surgery: P=0.573] (Figure 7). The surgical journal with the highest IF, Annals of Surgery, experienced an increase in IF similar to the top orthopaedic journals (1.506). The internal medicine journals with the highest IF had major increases in IF from 2010 to 2016: Journal of the American Medical Association (JAMA) (14.394); Lancet (14.198), and New England Journal of Medicine (18.92) (Figure 8).
Figure 6.
Change in mean impact factor of orthopaedic, internal medicine, and surgery journals from 2010 to 2016.
Figure 7.
Change in mean impact factor of internal medicine and surgery journals from 2010 to 2016 (IM: P=0.202; surgery: P=0.573). IM, internal medicine.
Figure 8.
Change in impact factor from 2010 to 2016 for internal medicine and surgery journals.
The field of orthopaedics has added at least 14 new journals since 2010, but concomitantly, approximately 7 journals ceased publication and no longer had any publication data (Table 3).
Table 3. Top 10 journals in 2010, primarily focusing on internal medicine and surgery (accepted JCR abbreviations were used).
| Title | Language | Published |
|---|---|---|
| Internal medicine journals (n=10) | ||
| Ann Intern Med | English | USA |
| Annu Rev Med | English | USA |
| Arch Intern Med | English | USA |
| Brit Med J | English | UK |
| Can Med Assoc J | English | Canada |
| Cochrane Db Syst Rev | English | UK |
| Jama-J Am Med Assoc | English | USA |
| Lancet | English | UK |
| New Engl J Med | English | USA |
| Plos Med | English | USA |
| Surgery journals (n=10) | ||
| Am J Surg Pathol | English | USA |
| Am J Transplant | English | Denmark |
| Ann Surg | English | USA |
| Ann Surg Oncol | English | USA |
| Arch Surg-Chicago | English | USA |
| Brit J Surg | English | UK |
| Endoscopy | English | Germany |
| J Am Coll Surgeons | English | USA |
| J Neurol Neurosur Ps | English | UK |
| J Vasc Surg | English | USA |
JCR, Journal Citation Report.
Discussion
Legislative efforts to improve healthcare quality have led to an increased demand for high quality research as a means of directing clinical practice (1,2). For orthopaedists, the importance of publishing the latest research in journals that garner large audiences is crucial, as both private and government insurance institutions look to improve care value by utilizing data from high impact journals (13,14). Currently, the most widespread method for assessing a journal’s impact on the scientific community is through Eugene Garfield’s “IF” algorithm. Therefore, this study was a continuation of Moverley et al.’s 2013 study (10), which assessed the IF trends in orthopaedics and provided comparisons to trends in major internal medicine and general surgery journals. Our findings revealed a mean increase in IF among orthopaedic journals, similar to the increase demonstrated among internal medicine and general surgery journals. Additionally, our study revealed an increase in the total number of orthopaedic journals published between 2010 and 2016, with a significant portion of journals published in English and in the United States.
This study was not without limitations. By utilizing an amalgamated database created by the Journal of Citation reports website (3), our results are subject to the same documentation and data collection errors that may have persisted on the website. Furthermore, this study is retrospective and does not denote causality, nor does it offer conclusive rationale to explain the phenomenon. At best, we are able to offer a hypothesis based on evidence in order to explain our findings. Despite these limitations, our study is a useful addition to the current literature and demonstrates an increase in orthopaedic journal IF between 2010 and 2016.
Similar to our study, Moverley et al. (10) reported an increase in the number of orthopaedic journals (+17) and mean IF between the years 2000 and 2010. However, this increase may be the result of an increased number of inter-citations and self-citations between orthopaedic journals. Hakkalamani et al. (11) evaluated seven general orthopaedic journals {Journal of Bone and Joint Surgery [JBJS (Am)], the British volume of the Journal of Bone and Joint Surgery [JBJS (Br)], Clinical Orthopaedics and Related Research (CORR), Acta Orthopedica Scandinavica (Acta Orthop Scand), International Orthopaedics (Int Orthop), Orthopaedic Clinics of North America (OCNA) and the Archives of Orthopaedics and Trauma Surgery (Arch Orthop Trauma Surg)} and reported a self-citation rate greater than 20%. Additionally, the authors reported a strong correlation between journal self-citation and reported IF (r=0.613; P<0.001). This may indicate that, while the amount of orthopaedic literature may be increasing, this literature may not be permeating non-orthopaedic audiences.
Despite the demonstrated increase in mean IF, its use as a measure of journal quality has been highly contested. Kodumuri et al. (15) evaluated publications from the top thirteen IF ranked journals in trauma and orthopaedics between the years 2007 and 2008, and assessed the number of citations at 2 and 5 years after publication. The authors revealed lower rates of mean citation among surgical papers at 2 years when compared to basic science papers (2.18 vs. 4.68, respectively). However, by 5 years post-publication, the citation rates were similar (26.57 for surgery, 30.35 for basic science/medical). The authors concluded that it may be necessary to adjust IF to measure citation rate at 5 years post-publication in order to be a more appropriate indicator of journal impact. The limitations housed by IF were further exemplified by a 2008 publication in Acta Crystallographica in which the authors explicitly asked readers to cite their publication. This resulted in over 10,000 citations, thus inflating the journal’s IF from 2.051 (in 2008) to 49.926 (in 2009) and 54.333 (in 2010) (16,17).
Despite these objections, in the absence of a better system, IF is still largely used as a surrogate marker for excellence among universities and research funding institutes (15). Unfortunately, this places orthopaedic literature at a marked disadvantage when compared to medicine and surgery journals that also report on outcomes following orthopaedic intervention. Delanois et al. (13) evaluated all publications from JAMA, NEJM, and Lancet between January 1976 and December 2016 and found that 56% of orthopaedic publications reported on complications. Furthermore, the authors reported potential policy impact from nearly half (46%) of the publications that reported conclusively about the efficacy and/or advisability of orthopedic intervention. This effect may be augmented due to rise in IF among major IM journals, as demonstrated by our study.
Conclusions
Despite concerns, a journal’s IF remains the most commonly used measure for evaluation of journal quality and impact. Between the years 2010 and 2016, there has been a significant increase in the number of peer-reviewed orthopaedic journals, and in mean orthopaedic journal IF. Furthermore, English journals from the United States continue to have the largest impact when compared to non-English journals and journals from outside the United States. However, this increased IF does not seem to translate to increased non-orthopaedic audiences, as health-care policy makers, and private institutions still utilize higher impact internal medicine journals to shape health-care. Future studies should aim to better qualify journal impact, while limiting confounders such as self-citation.
Acknowledgements
None.
Footnotes
Conflicts of Interest: The authors have no conflicts of interest to declare.
References
- 1.Bourdeau M, Winter R, Marshall R. Opportunities for improving legislative public health policy in Rhode Island through evidence-based education. R I Med J (2013) 2013;96:29-31. [PubMed] [Google Scholar]
- 2.Harolds JA, Merrill JK. Evidence-based medicine and health care reform legislation. Clin Nucl Med 2010;35:65. 10.1097/RLU.0b013e3181cf3963 [DOI] [PubMed] [Google Scholar]
- 3.InCites https://jcr-incites-thomsonreuters-com.ccmain.ohionet.org/JCRJournalHomeAction.action?SID=A2-Rl8GAi7nxxxxqBkOjWm65NXocNM4ov16Xv-18x2duSF8x2B93x2BHM0ILyGpmx2Fh9wgx3Dx3DoovcjO4gKcRPt9SoKFhwTwx3Dx3D-9vvmzcndpRgQCGPd1c2qPQx3Dx3D-wx2BJQh9GKVmtdJw3700KssQx3Dx3D
- 4.Garfield E. Citation indexes for science; a new dimension in documentation through association of ideas. Science 1955;122:108-11. 10.1126/science.122.3159.108 [DOI] [PubMed] [Google Scholar]
- 5.Isohanni M. Peer review--still the well-functioning quality control and enhancer in scientific research. Acta Psychiatr Scand 2005;112:165-6. 10.1111/j.1600-0447.2005.00601.x [DOI] [PubMed] [Google Scholar]
- 6.Baethge C. Impact factor--a useful tool, but not for all purposes. Dtsch Arztebl Int 2012;109:267-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Ironside PM. Rethinking the meaning and significance of journal impact factors. J Gerontol Nurs 2007;33:3-4. [DOI] [PubMed] [Google Scholar]
- 8.Seglen PO. Citation rates and journal impact factors are not suitable for evaluation of research. Acta Orthop Scand 1998;69:224-9. 10.3109/17453679809000920 [DOI] [PubMed] [Google Scholar]
- 9.Smith R. Beware the tyranny of impact factors. J Bone Joint Surg Br 2008;90:125-6. 10.1302/0301-620X.90B2.20258 [DOI] [PubMed] [Google Scholar]
- 10.Moverley R, Rankin KS, McNamara I, et al. Impact factors of orthopaedic journals between 2000 and 2010: trends and comparisons with other surgical specialties. Int Orthop 2013;37:561-7. 10.1007/s00264-012-1769-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Hakkalamani S, Rawal A, Hennessy MS, et al. The impact factor of seven orthopaedic journals: factors influencing it. J Bone Joint Surg Br 2006;88:159-62. 10.1302/0301-620X.88B2.16983 [DOI] [PubMed] [Google Scholar]
- 12.Siebelt M, Siebelt T, Pilot P, et al. Citation analysis of orthopaedic literature; 18 major orthopaedic journals compared for Impact Factor and SCImago. BMC Musculoskelet Disord 2010;11:4. 10.1186/1471-2474-11-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Delanois RE, Gwam CU, Piuzzi NS, et al. Hip and Knee Arthroplasty Orthopedic Literature in Medical Journals—Is It Negatively Biased? J Arthroplasty 2018;33:615-9. 10.1016/j.arth.2017.09.017 [DOI] [PubMed] [Google Scholar]
- 14.De Oliveira GS, Jr, Chang R, Kendall MC, et al. Publication bias in the anesthesiology literature. Anesth Analg 2012;114:1042-8. 10.1213/ANE.0b013e3182468fc6 [DOI] [PubMed] [Google Scholar]
- 15.Kodumuri P, Ollivere B, Holley J, et al. The impact factor of a journal is a poor measure of the clinical relevance of its papers. Bone Joint J 2014;96-B:414-9. 10.1302/0301-620X.96B3.32279 [DOI] [PubMed] [Google Scholar]
- 16.Van Noorden R. Scientists join journal editors to fight impact-factor abuse: News blog. Available online: http://blogs.nature.com/news/2013/05/scientists-join-journal-editors-to-fight-impact-factor-abuse.html
- 17.Sheldrick GM. A short history of SHELX. Acta Crystallogr A 2008;64:112-22. 10.1107/S0108767307043930 [DOI] [PubMed] [Google Scholar]