Subspecialty Variation in Academic Citations of Orthopedic Surgery Publications

Abstract

Background

The objective of this study was to assess the variation in academic citation and social media mentions across orthopedic subspecialties in general orthopedic surgery journals.

Methods

An internet-based study was performed of 666 articles from four orthopedic journals published from January 2018 to February 2019. Each publication was categorized by its subspecialty within orthopedics: arthroplasty, hand and upper extremity, foot and ankle, orthopedic oncology, pediatric orthopedics, shoulder, spine, sports medicine, orthopedic trauma, basic science, and miscellaneous. For each publication, academic citations were quantified using Google Scholar and Web of Science, and social media mentions were quantified using Twitter. Comparisons of continuous data among various subspecialties were performed using analysis of variance.

Results

The average number of citations per publication was 7.4 ± 9.0 on Google Scholar, 4.5 ± 5.3 on Web of Science, and 2.8 ± 8.7 on Twitter. The number of academic citations differed significantly by subspecialty on Google Scholar (p < 0.001) and Web of Science (p < 0.001). There was no difference in social media mentions on Twitter by subspecialty (p = 0.8). The most highly cited subspecialties, adjusted for number of publications, were arthroplasty, orthopedic oncology, and sports medicine, while the least highly cited subspecialties were hand and upper extremity, pediatric orthopedics, and orthopedic basic science.

Conclusions

There is significant baseline variation in the citation of orthopedic publications among various subspecialties. Our findings argue against the use of a uniform threshold to gauge scholarly success in orthopedic surgery. The variation in citation of orthopedic publications across subspecialties support the use of subspecialty-specific benchmarks to gauge academic productivity.

Research productivity is a central consideration in academic promotion.¹⁾ One metric of research productivity is the h-index, defined as the number of publications (h) that have attained at least h number of citations.²⁾ By design, the h-index measures both the quantity and quality of an author's scholarly works, though it is based exclusively on academic citations. In orthopedic surgery, the h-index has been correlated with academic rank¹,³,⁴⁾ and funding.⁵⁾ Some have advocated for the use of discrete h-index thresholds for consideration of promotion.³⁾

While the h-index has merit, it is difficult to simplify an author's scholarly success into a single statistic. Whether the citation of orthopedic surgery publications differs across subspecialties is not well-described. Significant baseline subspecialty variation in the citation of orthopedic literature would argue against a single, uniform benchmark for an orthopedic surgery department.

The primary objective of this study was to assess the variation across subspecialties of academic citations of recent orthopedic surgery publications in high-impact general interest orthopedic surgery journals. The secondary objective of this study was to assess the variation across subspecialties of social media mentions in these publications. Our null hypothesis was that there is no difference in academic citations or social media mentions across subspecialties.

METHODS

An Internet-based bibliometric study was performed without human subjects, and thus, institutional review board approval was deferred. This study comprised all original scientific research articles from four high-impact general interest orthopedic surgery journals, published from January 2018 to February 2019. The journals were the Bone & Joint Journal (BJJ), Clinical Orthopedics and Related Research (CORR), Journal of Bone & Joint Surgery (JBJS), and Journal of the American Academy of Orthopedic Surgeons (JAAOS). Editorials, commentaries, review articles, technique articles, short reports, case reports, conference proceedings, and errata were excluded. A final sample of 666 original full-length scientific research articles, including 214 articles from BJJ, 110 articles from CORR, 264 articles from JBJS, and 78 articles from JAAOS, were included for analysis.

Each publication was categorized by its subspecialty within orthopedic surgery: arthroplasty, hand and upper extremity, foot and ankle, orthopedic oncology, pediatric orthopedics, shoulder, spine, sports medicine, orthopedic trauma, basic science, and miscellaneous. When the topic of an article was pertinent to more than one subspecialty, it was categorized into the most relevant subspecialty by the judgment of the authors. All articles pertaining to the shoulder were categorized as shoulder. Articles pertaining to the non-shoulder upper extremity, including traumatic injuries, were categorized as hand and upper extremity. Articles pertaining to the foot and ankle, including traumatic injuries, were categorized as foot and ankle. Articles on joint preservation were categorized as sports medicine, whereas articles on joint replacement were categorized as arthroplasty. The number of conventional academic citations and social media mentions for each publication was assessed. Academic citations were quantified using Google Scholar and Web of Science, both internet-based indices of research and metadata across multiple databases and academic disciplines. Social media mentions were assessed using Twitter, a social networking platform with over 300 million active monthly users.⁶⁾ For each article, the total number of mentions on Twitter was determined in a standardized way by searching the Twitter platform with each full article title and counting the resultant social media posts. All data were collected from April to June 2020.

The number of citations adjusted per 1,000 surgeons in each subspecialty was calculated based on the number of surgeons by practice specialty, using the American Academy of Orthopedic Surgeons (AAOS) member directory (https://www7.aaos.org/member/directory): 3,224 for arthroplasty, 1,974 for hand and upper extremity, 1,104 for foot and ankle, 228 for orthopedic oncology, 949 for pediatric orthopedics, 705 for shoulder, 1,979 for spine, 4,003 for sports medicine, and 983 for orthopedic trauma (Table 1). The community size of basic science orthopedic researchers is not available using the AAOS member directory, and thus, not included.

Table 1. Number of Surgeons by Practice Specialty, Using the American Academy of Orthopedic Surgeons Member Directory.

Variable	Number
Arthroplasty	3,224
Foot & ankle	1,104
Hand & upper extremity	1,974
Oncology	228
Pediatrics	949
Shoulder	705
Spine	1,979
Sports	4,003
Trauma	983

Descriptive statistics were calculated, and results were given as mean ± standard deviation. One article was not indexed in Google Scholar, and 1 article was not indexed in Web of Science, and comparative statistics were performed on complete datasets only. Comparisons of continuous data among various subspecialties were performed using analysis of variance. Comparisons of paired data between indices were performed using paired t-test. The standard significance criteria of α = 0.05 was employed.

RESULTS

The numbers of academic citations and social media posts were determined for 666 orthopedic surgery publications. The average number of citations per publication was 7.4 ± 9.0 on Google Scholar and 4.5 ± 5.3 on Web of Science. Publications received significantly fewer citations on Web of Science compared with Google Scholar (p < 0.001). The average number of social media mentions per publication was 2.8 ± 8.7 on Twitter.

The most frequently published orthopedic subspecialty was arthroplasty, followed by orthopedic trauma, shoulder, and pediatric orthopedics. Sports medicine and orthopedic basic science comprised the least frequently published orthopedic subspecialties (Fig. 1, Table 2).

Fig. 1. The total number of orthopedic surgery publications by subspecialty.

Table 2. Total Number of Orthopedic Surgery Publications by Subspecialty.

Variable	Number
Arthroplasty	190
Foot & ankle	39
Hand & upper extremity	43
Oncology	40
Pediatrics	62
Shoulder	64
Spine	49
Sports	34
Trauma	83
Basic science	27
Miscellaneous	35

The number of academic citations of orthopedic surgery publications differed significantly by subspecialty on Google Scholar (p < 0.001) and Web of Science (p < 0.001). There was no difference in social media mentions on Twitter by subspecialty (p = 0.8) (Table 3). The most highly cited subspecialties, adjusted for number of publications, were arthroplasty, orthopedic oncology, and sports medicine, while the least highly cited subspecialties were hand and upper extremity, pediatric orthopedics, and orthopedic basic science (Fig. 2). The most highly cited subspecialty, adjusted for the number of publications and surgeons per subspecialty, was orthopedic oncology, while the least cited subspecialties were spine, hand and upper extremity, and sports medicine (Fig. 3).

Table 3. Citations of Orthopedic Surgery Publications by Subspecialty.

Variable	Google Scholar	Web of Science	Twitter
Arthroplasty	10.6 ± 13.4	6.5 ± 7.5	2.8 ± 5.8
Foot & ankle	6.1 ± 6.5	3.6 ± 4.2	1.5 ± 4.4
Hand & upper extremity	5.6 ± 4.3	2.7 ± 2.6	2.3 ± 4.1
Oncology	7.9 ± 7.1	5.3 ± 4.8	1.8 ± 2.6
Pediatrics	4.8 ± 4.0	3.0 ± 2.7	2.1 ± 3.4
Shoulder	6.3 ± 5.3	3.7 ± 3.5	3.7 ± 7.9
Spine	6.3 ± 5.6	4.0 ± 3.6	4.7 ± 24.4
Sports	7.3 ± 6.2	4.4 ± 4.4	3.2 ± 7.4
Trauma	6.2 ± 5.0	3.8 ± 3.4	2.5 ± 6.5
Basic science	3.7 ± 3.1	2.6 ± 2.1	2.4 ± 4.3
Miscellaneous	6.2 ± 12.2	3.5 ± 6.2	4.3 ± 9.5

Values are presented as mean ± standard deviation.

Fig. 2. The average number of Google Scholar, Web of Science, and Twitter citations per publication by subspecialty.

Fig. 3. The average number of Google Scholar, Web of Science, and Twitter citations per publication by subspecialty, adjusted for the number of surgeons in each subspecialty.

DISCUSSION

Academic productivity and scholarly impact are important considerations in academic promotion, but can be difficult to measure. While some departments consider a candidate's raw number of publications, this metric assesses only the quantity, not the influence, of a body of scholarly work.⁵⁾ In 2005, Hirsch²⁾ proposed the h-index as a metric of scholarly impact, calculated by the number of publications (h) that have attained at least h number of citations. For instance, an author who has published 10 articles, each with at least 10 citations, would have an h-index of 10. Since its introduction, the h-index has gained popularity across medical specialties.⁷⁾ Within orthopedic surgery, the h-index has been correlated with academic faculty rank¹,³,⁴⁾ and research funding.⁵⁾ Some authors have identified discrete thresholds in h-index between junior and senior orthopedic faculty and proposed the use of these thresholds in considerations for promotion in orthopedic surgery departments.³⁾ In this study, we have demonstrated significant variation among subspecialties in academic citation of orthopedic surgery publications.

The reason for the substantial subspecialty variation in academic citation of orthopedic literature is likely multifactorial. Citation is in part contingent upon readership, and one important factor may be the difference in the size of the readership among the various subspecialties, both within and outside of the field of orthopedic surgery. Within orthopedic surgery, the target audience of publications across subspecialties is likely to differ in size. This is seen in the striking difference in active membership of national subspecialty societies, which ranges from approximately 4,366 members in the American Association of Hip and Knee Surgeons⁸⁾ to 653 members in the Pediatric Orthopedic Society of North America⁹⁾ and to 193 members in the Musculoskeletal Tumor Society.¹⁰⁾ Moreover, the scholarly works of a particular subspecialty may induce interest and readership from fields outside of orthopedic surgery. For example, an article in orthopedic oncology may be of interest to medical oncologists and radiation oncologists, which may greatly expand the interest in and academic impact and citations of a particular article. The differences in academic citation of orthopedic publications across subspecialties argue against the use of a single, discrete benchmark for scholarly success within the field of orthopedic surgery.

Interestingly, we have found that social media mentions of orthopedic surgery publications do not differ by subspecialty. There have been recent proponents of the use of alternative metrics of scholarly impact, so called altmetrics, based on social media mentions.¹¹,¹²,¹³⁾ Altmetrics have a number of advantages over conventional citations as a metric of scholarly impact. First, while conventional citations often take years to accrue, altmetrics may provide a more rapid assessment of the impact of a publication. Second, conventional citations do not account for the proportion of readership who do not publish or cite themselves, yet may be influenced by its content, and therefore, altmetrics may represent a scholarly work's overall societal impact. Third, conventional citations may underestimate an article's impact if subsequent authors cite review articles or secondary sources. Finally, conventional citations are subject to the Matthew effect, which describes the tendency for more renowned authors to receive a disproportionate share of citations compared with less well-known authors.¹⁾ Due to the merits of altmetrics, complementary algorithms for scholarly productivity that weigh both conventional citations and social media mentions have been proposed.¹¹⁾ Interestingly, we have found no significant difference in the social media mentions of orthopedic surgery research on Twitter across subspecialties. As the use of social media to promote and disseminate orthopedic surgery research matures, future studies are needed to verify whether altmetrics can overcome subspecialty variation in orthopedic surgery research citations.

There are several limitations to this study. First, the study included only publications between January 2018 and February 2019. We chose this study period to assess the recent orthopedic literature and allow at least one year since publication for citations to accrue. Second, four high-impact general interest orthopedic surgery journals were studied. While we believe these journals provide an accurate reflection of the recent orthopedic literature across subspecialties, they are not exhaustive of the orthopedic research landscape. General interest orthopedic surgery journals may publish a certain proportion of articles from each subspecialty with differing acceptance rates. Future studies may benefit from analysis of subspecialty-specific journals. Third, citation rates can be influenced by a number of factors extrinsic to the scientific merit of the scholarly work. Examples include the length and punctuation of the article title.¹⁴⁾ Fourth, our data do not provide insight into the underlying reasons for the subspecialty variation of academic citations in orthopedic surgery publications. Fifth, adjustments for the number of surgeons within a given subspecialty were made using AAOS membership information, but this may not be fully representative of the number of practicing surgeons and the journal readership. Lastly, Twitter was chosen as our metric of social media research impact because it is the most commonly used online platform for sharing orthopedic research⁶⁾; however, it is one of many social media platforms on which research is shared.

We have demonstrated that there is significant and substantial baseline variation in the citation of orthopedic publications among various subspecialties. Social media mentions of orthopedic publications did not differ among subspecialties. Future studies may focus on the factors driving the subspecialty variation in academic citation of orthopedic publications, but a likely factor is the scope of readership. Our findings argue against the use of a single, uniform threshold to gauge scholarly success in orthopedic surgery and in favor of subspecialty-specific benchmarks.