The Journal Impact Factor of Orthopaedic Journals Does not Predict Individual Paper Citation Rate

Abstract

Background:

The journal impact factor (JIF) is thought to reflect the average number of citations an article will receive and therefore can influence study impact and clinical decision making. However, analysis of citation rates across multiple scientific and research domains has shown that most articles will not reach this expected number of citations. This phenomenon is known as citation skew and it has not previously been examined in the orthopaedic literature. The objective of this study was to determine the extent to which citation skew exists within orthopaedic journals and thus to determine whether the JIF in the orthopaedic literature reflects individual study citation rates.

Methods:

We used data from the Thomson Reuters (now Clarivate Analytics) Web of Science to determine the 2015 JIF and citation distribution for all orthopaedic journals listed in the database. We calculated the percentage of articles with fewer citations than the JIF for each journal. Finally, we analyzed the citation distribution within groups of orthopaedic subspecialty publications.

Results:

We identified a total of 74 orthopaedic journals and 29,296 publications for the years 2013 and 2014. Across all orthopaedic journals, 85% of published articles are cited fewer times than the JIF would indicate. The median number of citations of all articles was zero for all journals (interquartile range = 0-0) except for seven journals, for which the median number of citations per article was 1.

Conclusion:

Citation skew is prevalent across the orthopaedic literature. Most published work is not cited in the first 2 years following publication, and the JIFs are the result of a few highly cited articles. The assessment of an individual orthopaedic study's quality should not be determined by the JIF but rather by direct evaluation of the methodology, relevance, and appropriateness of the study's conclusions.

Orthopaedic surgeons and evidence users may be familiar with the journal impact factor (JIF). The JIF was originally conceived in 1972 to help bibliometricians decide which journals to purchase for their institutional libraries¹ and was neither designed nor intended to be applied to the assessment of individual researchers or manuscripts.² The calculation of the JIF of any given year is straightforward: it is the number of citations accrued over that given year for articles published in that journal during the preceding 2 years divided by the number of articles published in that journal during those 2 years. For example, many clinicians would expect that an article published in a journal with a JIF of 20 will accrue, on average, 20 citations per year over the first 2 years after publication. However, analysis of citation rates across multiple scientific and research domains has shown that most articles will not reach this expected number of citations.³ This phenomenon is known as citation skew and to our knowledge has not previously been examined in the orthopaedic literature.

The shortcomings of JIFs are well described.^4,5,6,7,8,9 These include the small number of highly cited articles skewing the impact factor of a journal; the risk for manipulation by self citation in reviews, editorials and letters; the lack of transparency in its calculation; and the lack of publication of a median citation number.¹⁰ Furthermore, it has been determined recently that the JIF serves as an inappropriate indicator of research quality⁶ and does not predict an individual author's future citations.¹¹

Nature Publishing Group, known for having some of the highest JIFs in academic publishing, is now among the publishing groups openly criticizing the JIF and calling for the use of a new metric.¹² International granting agencies have not only taken notice but have also taken action. Groups such as the Research Councils of the United Kingdom (RCUK),¹³ Canadian Institutes of Health Research (CIHR),¹⁴ and the Australian Research Council (ARC)¹⁵ have abandoned consideration of the JIF in grant applications. In 2015, the San Francisco Declaration on Research Assessment (DORA)² outlined the pitfalls of the JIF in evaluating individual articles and researchers; the declaration was signed by an international group of journal publishers and granting agencies.¹⁶

Recent evidence suggests that orthopaedic surgeons do consider JIF to be important when making clinical decisions.^17,18 Therefore, the potential for citation skew could undermine this process. If citation skew does indeed exist in the orthopaedic literature, then these clinical decisions could be considered to be based on injudicious information. The objective of this study was to determine the extent to which citation skew exists in orthopaedic research.

Methods

Using the Institute for Scientific Information (ISI) listings available online by Thomson Reuters (now Clarivate Analytics), we retrieved all orthopaedic journals under the automated search term orthopaedics on August 21, 2016. Using a previously described technique,³ we searched each journal individually on Thomson Reuters' Web of Science for all their publications for the years 2013 and 2014. The website allows for the creation of a citation report on each publication, and we analyzed these data using a Microsoft Excel file. We calculated frequency data, medians, and interquartile ranges (IQRs) of the number of citations for each journal in 2015 for 2013 and 2014 publications. We report discrete variables as counts or proportions, and normally distributed continuous variables as mean with SDs.

We created a citation distribution for each journal by plotting the frequency of the number of citations accrued by each article. We then compiled the citation distribution data of all orthopaedic journals into one. We selected the journals with the 15 highest impact factors for separate analysis. We then identified the percentage of publications within each journal that accrued fewer citations than the journal's JIF. Finally, we divided the number of publications having zero citations by the total number of publications to determine the percentage of publications with no citations.

We grouped all 74 journals into their respective subspecialties by examining the website of each journal for the description of their academic and clinical scope. Any uncertainty as to which subspecialty a journal belongs to was resolved by a consensus with the senior author. We then calculated the mean and SD of the IF for each subspecialty grouping.

Results

Seventy-four journals in orthopaedics (Table 1, http://links.lww.com/JG9/A1) were included in the analysis. These journals comprise 29,296 publications for the years 2013 and 2014 and 52,214 citations in 2015. The journals are presented along with their 2-year IF for 2015 in Table 1 (http://links.lww.com/JG9/A1). While most journals included are orthopaedic surgical journals, several physiotherapy journals (Physical Therapy, Gait Posture, and Journal of Physiotherapy), one nursing journal (Orthopaedic Nursing), and one plastic surgery hand journal (Journal of Plastic Hand Surgery) are listed and included in the analyses.

The IFs of the orthopaedic journals range from 5.163 (Journal of Bone and Joint Surgery of America) to 0.357 (Isokinetic Exercise Science), with a median IF of 1.429 (IQR = 0.878–2.387). The cumulative citation distribution for all orthopaedic publications for 2013 and 2014 is presented in Figure 1. A total of 18,910 of 29,296 (64.5%) orthopaedic publications for 2013 and 2014 were not cited in 2015.

Figure 1.

Combined citation distribution for all 29,296 articles included in the calculation of the 2015 journal impact factor for all 74 orthopaedic journals. Frequency is plotted against the number of citations per publication. Inset is for zero to 10 times cited. Overall, 18,910/29,296 (64.5%) of 2013 and 2014 publications were not cited in 2015. IQR = interquartile range.

The citation distributions of the journals with 15 highest JIFs are presented in Figure 2, and the remainder is provided in Appendix 1 (http://links.lww.com/JG9/A1). The distribution of citation data in all orthopaedic journals demonstrates a skew, with a preponderance of publications having zero citations. The maximum number of citations for an individual article ranged from 125 (Osteoarthritis and Cartilage) to 2 (Zeitschrift fur Orthopadie und Unfallchirurgie).

Figure 2.

Individual citation distribution for the top 15 orthopaedic journals, plotting the number of citations accrued per publication by frequency. JIF = journal impact factor.

Across all orthopaedic journals, 85% of published articles were cited less often than the JIF would indicate. The proportion of articles with citations below the JIF for the top 15 orthopaedic journals is 87% and ranges from 97% (Osteoarthritic Cartilage) to 77% (Journal of American Academy of Orthopaedic Surgeons), as shown in Table 2 (http://links.lww.com/JG9/A1). The median number of citations was zero (IQR = 0–0) for all journals except for seven (Knee Surgery Sports Traumatology Arthroscopy, Journal of Orthopaedic Research, Gait Posture, Bone & Joint Research, Bone & Joint Journal, and Arthroscopy), where the median number of citations was 1.

The journal with the lowest number of zero-citation publications is the American Journal of Sports Medicine at 47%, in comparison with 92% of publications receiving zero citations in Sportverletzung-Sportschaden.

Subspecialty analyses are illustrated in Figure 3. The subspecialties with the highest median JIFs are sports medicine (median = 2.36, Q₁ = 1.21, Q₃ = 3.9) and spine (median = 2.36, Q₁ = 2.16, Q₃ = 2.60), and the lowest median JIFs are arthroplasty (median = 0.89, Q₁ = 0.845, Q₃ = 2.41) and pediatrics (median = 0.89, Q₁ = 0.45, Q₃ = 1.33).

Figure 3.

Median impact factor plotted of subspecialty journals in orthopaedics. General orthopaedics includes trauma and basic science journals. Other includes hand journals, those not fitting other categories, and non-English journals.

Discussion

Our findings strongly indicate that citation skew holds true in orthopaedic research and is prevalent across all orthopaedic journals. Most research articles published in orthopaedic journals (85%) fall short of the JIF, and the median number of citations per publication per year is zero for all but seven orthopaedic journals, for which the median citation rate is one. Therefore, the JIF of orthopaedic journals does not reflect the citation rate of individual articles. Not only does the JIF overestimate most individual publication citation rates, but the JIF also fails to properly acknowledge the articles that outperform the expected citation rate.

Our results are consistent with the citation skews that have been reported in the literature across other scientific domains. Lariviere et al³ found that 75% of articles published in both Nature and Science were cited less than each journal's respective JIF, whereas 67% of articles published in the biology journals EMBO and PLOS Biology were cited less than their JIFs.

Some journals, such as 8 of the 11 American Society for Microbiology journals, have consciously decided to stop advertising their JIFs online,¹² and alternatives to the JIF have been recently described. Alternative metrics, or altmetrics, measure the number of online mentions that an academic article receives, including mentions in social media and online news outlets.^19,20 These altmetrics have been shown to predict future citation counts and accrue much more rapidly than do citations.²¹ As we seek to improve the understanding of knowledge translation and exchange in a time of rapidly emerging information channels, the importance of monitoring and measuring the dispersion of orthopaedic research in all forms should not be underestimated.²²

Another alternative to the JIF would be to publish journal citation distributions, or a metric that draws attention to the spread and variation of citations, as a countermeasure to the tendency to rely wholly on JIFs in the assessment of research impact.³ Efforts have also been made to normalize the differing citation rates across scientific disciplines,²³ and the Relative Citation Rate (RCR) is one such example. The RCR seeks to standardize citation rates by dividing the citation rate derived from articles in the same field and benchmarked to a peer comparison group.²⁴ As opposed to JIF, the RCR was shown to closely reflect the 15 most cited NIH-funded articles in 2014, and the NIH has now adopted the RCR as a tool in their granting decision-making process.²⁵

Audience factor (AF) is another alternative to JIF that aims to control for the main determinant of IF variability: the propensity to cite across fields and article types.²⁶ It has been noted that AF removes variance across fields while retaining variance of citation rates within a field.²⁶ The AF would adjust for the lower average citation rates in disciplines such as mathematics and the higher citation rates in certain health science and surgical domains. The potential difference in citation rates and skewness across disciplines is highlighted by a 2015 article that found a very strong correlation between JIF and median citation rates across five nonsurgical subject domains (physics, genetics, marine biology, multidisciplinary science, and information science).¹⁰ This contrasts with our findings that all but seven of the orthopaedic journals had a median citation of zero, and that therefore, citation skew may be more prevalent in orthopaedic journals.

One strength of this study is the fact that the entire corpus of the orthopaedic literature has been included in the analysis. In addition, the JIF used is the most recent available, and this study is the first to our knowledge to analyze the presence of citation skew in the orthopaedic literature. However, we acknowledge some limitations. One may be that the results and conclusions drawn in this article rely on data sourced from ISI and Web of Science. Furthermore, although we know with certainty the proportion of published articles cited above and below each journal's JIF, we do not know all the characteristics of the articles driving the IFs. Certain types of research, such as review papers, are known to generally garner more citations across all scientific fields.²⁷ Bhandari et al²⁸ showed that research with superior methodological safeguards against bias, such as randomized controlled trials and meta-analyses, was cited more than observational studies and case reports in one orthopaedic journal. However, the breakdown of the most cited articles across all orthopaedic journals remains unknown. Determining which individual article was the most cited for each journal may be of interest but was beyond the scope of this article. Furthermore, citations may accrue because of both positive and negative attention,²⁹ as many retracted articles have accrued hundreds of citations.³⁰

Despite these limitations, our study underscores the importance of evaluating each individual article for quality, as opposed to simply assigning quality based solely on the JIF. The JIF was designed to measure journal popularity, not individual article quality. Furthermore, citation rate is not a proven indicator of research quality. In fact, poorly designed or controversial articles have been shown to accrue a large number of citations.³¹

Conclusion

Citation skew is prevalent across the body of the orthopaedic literature. Most published work is not cited in the first 2 years following publication, and the reported JIFs are the result of a few highly cited articles. Extrapolating an orthopaedic journal's historical impact to an individual article is a poor predictor of that article's quality and future citations. Orthopaedic surgeons should be aware that more journals and granting agencies are moving away from the JIF and toward other metrics that assess the actual impact of the individual studies. The assessment of an individual study's quality and any bearing on clinical decision making should not be determined by the JIF but rather by direct evaluation of the methodology, relevance, and appropriateness of the study's conclusions.