Evaluation of the National Institutes of Health–supported relative citation ratio among American orthopedic spine surgery faculty: A new bibliometric measure of scientific influence

Highlights

• •Analyze new NIH-Supported bibliometric tool, the relative citation ratio (RCR).
• •Evaluate research productivity among fellowship trained Orthopedic Spine Surgeons.
• •Conduct demographic analysis of research productivity among fellowship trained Orthopedic Spine Surgeons.

Abstract

Background

Publication metrics have been traditionally used to compare research productivity amongst academic faculty. However, traditional bibliometrics lack field-normalization and are often biased towards time-dependent publication factors. The National Institutes of Health (NIH) has developed a new, field-normalized, article-level metric, known as the “relative citation ratio” (RCR), that can be used to make accurate self, departmental, and cross-specialty comparisons of research productivity. This study evaluates the use of the RCR amongst academic orthopedic spine surgery faculty and analyzes physician factors associated with RCR values.

Methods

A retrospective data analysis was performed using the iCite database for all fellowship trained orthopedic spine surgery (OSS) faculty associated with Accreditation Council for Graduate Medical Education (ACGME)-accredited orthopedic surgery residency program. Mean RCR, weighted RCR, and total publication count were compared by sex, career duration, academic rank, and presence of additional degrees. A value of 1.0 is the NIH-funded field-normalized standard. Student t-tests were used for two-group analyses whereas the analysis of variance tests (ANOVA) was used for between-group comparisons of three or more subgroups. Statistical significance was achieved at P < 0.05.

Results

A total of 502 academic OSS faculty members from 159 institutions were included in the analysis. Overall, OSS faculty were highly productive, with a median RCR of 1.62 (IQR 1.38-2.32) and a median weighted RCR of 68.98 (IQR 21.06-212.70). Advancing academic rank was associated with weighted RCR, career longevity was associated with mean RCR score, and male sex was associated with having increased mean and weighted RCR scores. All subgroups analyzed had an RCR value above 1.0.

Conclusions

Academic orthopedic spine surgery faculty produce impactful research as evidenced by the high median RCR relative to the standard value set by the NIH of 1.0. Our data can be used to evaluate research productivity in the orthopedic spine community.

Introduction

Research activity amongst academic orthopedic faculty plays a significant role in decisions regarding their hiring, promotion, tenure, and funding support [1], [2], [3]. Historically, the academic output of an individual or department has been measured by publication count, publication type, or the amount of funding received through research grants [4]. The Hirsch index (H-index) has been the most commonly utilized citation-based bibliometric ⁴used to quantify research productivity and scientific impact within the field of orthopedic surgery [3,[5], [6], [7]],. However, the H-index has been scrutinized for its simplicity and lack of field normalization, which limits accurate cross-specialty comparisons of research productivity [8]. This limitation was alluded to by Hirsch in 2005: “Scientists working in non-mainstream areas will not achieve the same very high h values as the top echelon of those working in highly topical areas” [9]. Moreover, because the H-index is influenced by total publication count, its value is limited by time-dependent publication factors such as career longevity [ 10,11].

In 2015 the National Institutes of Health (NIH) developed a new time- and field- normalized article-level metric known the “relative citation ratio” (RCR), which capitalizes on several shortcomings of traditional bibliometric indices [3]. The RCR is calculated as the total number of citations per year of a specific publication divided by the average number of citations received per year by NIH-funded papers within the same field [3]. Dynamic field normalization is a unique feature of the RCR that allows for more accurate comparisons of author impact across interdisciplinary fields [3,12]. Author-level derivatives of the RCR such as the mean RCR and the weighted RCR are calculated by taking the average or sum of all article-level RCR scores for an individual author, respectively. As a statistical average, the mean RCR eliminates the bias imposed by time-related factors (i.e., career longevity) and serves as a measure of research impact [3]. Conversely, much like the H-index, the weighted RCR is influenced by the quantity of total publications and serves as a measure of overall research productivity.

Herein, we evaluate the use of the RCR amongst academic orthopedic spine surgery faculty associated with ACGME-accredited orthopedic residency programs across the United States. The purpose of this study was to provide benchmark data for RCR scores within the field of orthopedic spine surgery and to analyze physician factors associated with these values. We hypothesize that the mean RCR score for academic orthopedic spine surgery faculty members would fall above the NIH standard of 1.0. The information in this study may be used for individual self and departmental evaluations of research impact within the field of orthopedic spine surgery.

Methods

Departmental and faculty inclusion criteria

All fellowship-trained orthopedic spine surgeons employed as faculty at ACGME-accredited orthopedic surgery residency programs were included in our analysis. Individual departmental websites for each accredited residency program were accessed on October 3, 2021 (https://apps.acgme.org/ads/Public/Programs/Search) to identify all fellowship trained orthopedic spine surgeons. Faculty sex, degrees (PhD or no), academic ranks and residency start years were obtained using physician profiles on departmental websites or via publicly available outlets. Academic rank included assistant, associate, and full professors. Clinical instructors, staff physicians, private practice surgeons, or other faculty not otherwise specified were listed as ‘Other’. Residency start years were obtained to categorize faculty into the following groups: ≤ 1980, 1981–1990, 1991–2000, >2000.

Bibliometric analysis

The RCR for an individual publication is described as the total number of citations per year for that publication divided by the average field-specific citations per year received for all NIH- funded publications in the same field. Thus, a ratio of 1.0 represents the field-normalized, NIH-funded standard. Author-level RCR scores (mean and weighted RCR) are calculated from the aggregate article-level RCR scores for all publications produced by an individual author. The mean RCR is simply the statistical average of all RCR scores for publications produced by an individual author. The weighted RCR is the sum of all RCR scores for publications produced by an individual author and is therefore influenced by total publication count.

Orthopedic spine surgery faculty members were individually indexed using the NIH iCite database website (https://icite.od.nih.gov/). Non-original research articles (i.e., editorials, reviews, and meeting abstracts) as defined by the iCite database were excluded. The iCite database currently contains PubMed listed articles from 1980 to present. The number of total publications, mean RCR score and weighted RCR score were collected for each author on November 14, 2021.

Statistical analysis

The mean and weighted RCR were collected from the iCite search output for each orthopedic spine surgery faculty member and compared by sex, degree, academic rank, and career longevity as defined by residency start date. Student t-tests were used for two-group analyses whereas the analysis of variance tests (ANOVA) were used for between-group comparisons of three or more subgroups. Statistical significance was achieved at P < 0.05. The data herein is presented as the median and interquartile range (in addition to the mean and standard deviation) to account for outliers of the mean and weighted RCR scores.

Results

A total of 502 fellowship-trained academic orthopedic spine surgeon faculty members were included in this study (Table 1). The majority of OSS faculty identified were male (n = 484; 96.4%), and approximately 2.6% of all faculty had a PhD (n = 13). Overall, RCR scores were high but widely variable with a median RCR of 1.62 (IQR 1.38-2.32) (Table 2) and median weighted RCR of 68.98 (IQR 21.06-212.70) (Table 3). The median number of total publications produced per OSS faculty member was 46 (IQR 14-106) (Table 4). An overview of the mean RCR data and mean weighted RCR data for all OSS faculty members are depicted as boxplots in Fig. 1 and Fig 2, respectively.

Table 1.

Demographics for Orthopedic Spine Surgery Faculty members.

Characteristic	No.	%
Sex
Female	18	3.59
Male	484	96.41
PhD Degree
No	489	97.41
Yes	13	2.59
Academic Ranking
Assistant professor*	185	36.85
Associate professor	93	18.53
Professor	107	21.31
Other⁎⁎	118	23.51
Residency Start Year
≤1980	24	4.78
1981-1990	85	16.93
1991-2000	124	24.70
2001-2010	149	29.68
>2010	120	23.90

^⁎Assistant professor includes clinical assistant professor, instructor, and lecturer.

^⁎⁎“Other” indicates clinical instructors, staff physicians, private practice surgeons, or other faculty not otherwise specified.

Table 2.

Mean RCR by sex, PhD acquisition, academic ranking, and residency start year.

Characteristic	No.	Mean	SD	Median	25th Percentile	75th Percentile	p value
Sex
Female	18	1.33	0.68	1.47	1.16	1.57	0.019
Male	484	1.64	1.10	1.51	0.98	2.04
PhD Degree
No	489	1.63	1.10	1.51	0.96	2.04	0.008
Yes	13	1.32	0.46	1.49	1.19	1.55
Academic Ranking
Assistant professor*	185	1.65	1.09	1.51	0.96	2.06	0.089
Associate professor	93	1.39	0.84	1.38	0.87	1.73
Professor	107	1.89	0.97	1.72	1.33	2.24
Other⁎⁎	118	1.47	1.25	1.17	0.72	1.87
Residency Start Year
≤1980	24	1.39	0.86	1.19	0.88	1.74	0.030
1981-1990	85	1.77	1.07	1.65	1.13	2.22
1991-2000	124	1.77	1.32	1.67	1.13	2.13
2001-2010	149	1.52	0.99	1.39	0.92	1.89
>2010	120	1.51	0.94	1.48	0.85	2.01

^⁎Assistant professor includes clinical assistant professor, instructor, and lecturer.

^⁎⁎“Other” indicates clinical instructors, staff physicians, private practice surgeons, or other faculty not otherwise specified.

Statistical Significance determined by p-value<0.05.

Table 3.

Weighted RCR by sex, PhD acquisition, academic ranking, and residency start year.

Characteristic	No.	Mean	SD	Median	25th Percentile	75th Percentile	p value
Sex
Female	18	42.51	76.40	12.56	5.42	27.05	0.040
Male	484	80.41	177.61	16.84	4.68	66.88
PhD Degree
No	489	75.30	165.88	15.24	4.71	61.65	0.204
Yes	13	220.05	372.38	71.25	3.46	126.70
Academic Ranking
Assistant professor*	185	52.39	136.35	12.13	4.68	38.77	<.0001
Associate professor	93	68.91	133.36	17.01	5.88	71.25
Professor	107	181.51	253.07	91.31	28.23	207.60
Other⁎⁎	118	36.30	133.94	5.78	2.04	23.34
Residency Start Year
≤1980	24	43.77	99.95	9.52	4.22	25.68	0.261
1981-1990	85	81.91	146.68	22.81	6.23	67.04
1991-2000	124	103.07	225.70	20.44	3.72	81.23
2001-2010	149	74.73	166.81	16.91	4.94	71.28
>2010	120	64.79	154.24	11.48	4.55	43.98

^⁎Assistant professor includes clinical assistant professor, instructor, and lecturer.

^⁎⁎“Other” indicates clinical instructors, staff physicians, private practice surgeons, or other faculty not otherwise specified.

Statistical Significance determined by p-value<0.05.

Table 4.

Total number of publications by sex, PhD acquisition, academic ranking, and residency start year.

Characteristic	No.	Mean	SD	Median	25th Percentile	75th Percentile	p value
Sex
Female	18	28.17	43.93	7.50	6.00	24.25	.122
Male	484	46.17	90.05	15.00	4.00	44.25
PhD Degree
No	489	42.88	80.89	14.00	5.00	42.00	.134
Yes	13	145	229.12	53.00	3.00	92.00
Academic Ranking
Assistant professor*	185	31.02	76.57	11.00	4.00	23.00	<.001
Associate professor	93	47.25	84.61	15.00	6.00	46.00
Professor	107	96.86	118.22	54.00	22.00	130.00
Other⁎⁎	118	20.79	54.96	5.00	2.00	15.00
Residency Start Year
≤1980	24	29.00	63.00	8.50	3.75	20.75	.540
1981-1990	85	41.26	62.16	16.00	5.00	47.00
1991-2000	124	56.56	108.26	15.00	4.00	50.25
2001-2010	149	44.03	86.24	15.00	6.00	46.00
>2010	120	42.28	90.39	10.50	4.00	27.25

^⁎Assistant professor includes clinical assistant professor, instructor, and lecturer.

^⁎⁎“Other” indicates clinical instructors, staff physicians, private practice surgeons, or other faculty not otherwise specified.

Statistical Significance determined by p-value<0.05.

Fig. 1.

Overview of mean RCR data for fellowship-trained orthopedic spine surgeons employed as faculty at ACGME-accredited orthopedic surgery residency programs. The center line shows the median; box limits indicate the 25th and 75th percentiles as determined by R software; whiskers extend 1.5 times the interquartile range from the 25th and 75th percentiles, outliers are represented by dots. n = 502 sample points.

Fig. 2.

Overview of weighted RCR data for all fellowship-trained orthopedic spine surgeons employed as faculty at ACGME-accredited orthopedic surgery residency programs. The center line shows the median; box limits indicate the 25th and 75th percentiles as determined by R software; whiskers extend 1.5 times the interquartile range from the 25th and 75th percentiles, outliers are represented by dots. n = 502 sample points.

Academic rank

Assistant professor, which includes clinical assistant professor, instructor, and lecturer, was the most common academic rank with 185 members (36.9%), with professors composing (21.3%), and associate professors composing (18.5%). The remaining 23.5% of members were categorized as “other” which includes clinical instructors, staff physicians, private practice surgeons, or faculty not otherwise specified.

There was a positive correlation of weighted RCR (P<.001) with academic rank in our sample. Full professors were the most productive subgroup in our study with a median RCR of 1.72 (IQR1.33-2.24) and median weighted RCR of 91.31 (IQR 28.23-207.60). There was no significant association between mean RCR and academic rank (p = .0885).

Sex

There was a statistically significant difference between the median female and male RCR score (1.47, IQR 1.16-1.57 vs 1.51, IQR .98-2.04 (p = .019)) and weighted RCR score (12.56, IQR 5.42-27.05 vs 16.84, IQR 4.68-66.88 (p = .04)). However, there was no statistically significant difference between the median female and male total publication count (7.5, IQR 6.0- 24.25 vs 15.0, IQR 4.0-44.25 (p = .122)).

PhD Degree

PhD faculty had a median RCR of 1.49 (IQR 1.19-1.55) and non-PhD faculty had a median RCR of median RCR of 1.51 (IQR .096-2.04), however this finding was not significant (p = .929). There was also no statistically significant difference between the median PhD and non-PhD weighted RCR scores (71.25, IQR 3.46-126.7 vs 15.24, IQR 4.71-61.65 (p = .204)) or median total publication count (53, IQR 3.0-92.0 vs 14, IQR 5.0-42.0 (p = 0.134)).

Career Longevity

Longer career duration, as defined by residency start date, had a significant impact on median RCR scores (p = .03). Those with residency start dates between 1981-1990 and 1991-2000 had the highest median RCR scores of 1.65 (IQR 1.13-2.22) and 1.67 (IQR 1.13-2.13), respectively, whereas lower RCR scores were found in those with more recent residency start dates between 2001-2010 (1.39, IQR 0.92-1.89) and >2010 (1.48, IQR 0.85-2.01). No significant association between median weighted RCR (p = 0.2613) or total publications (p = 0.54) with career longevity was found.

Discussion

Academic communities are in need of a standardized bibliometric index capable of drawing accurate comparisons of research impact within a field and across all scientific disciplines[6]. As a time- and field-normalized article-level metric, the NIH-supported RCR is thought to satisfy the needs for such standardization. While several academic disciplines have evaluated the use of the RCR since its development in 2015 [3,6,10,13,14], a correlative study is yet to be performed within the field of orthopedic surgery. This study expanded upon the utilization of the RCR to academic orthopedic spine surgeons. Our benchmark data may be used for individual and departmental evaluation within the field of academic spine surgery and can be used to make cross-specialty comparisons.

Previous studies have shown PubMed-listed publications and NIH-funded publications included in the iCite database had a median RCR of 0.37 (range, 10^th percentile to 90^th percentile, 0 to 2.24) and 1.00 (range, 10^th percentile to 90^th percentile, 0.38 to 3.81), respectively[14]. In our study, the median RCR was 1.62 (IQR 1.38-2.32), which falls within the 80^th to 90^th percentile of RCR scores for all publications within the iCite database, and within the 60^th to 70^th percentile of all NIH-funded publications (iCite). This suggests that publications of academic orthopedic spine surgeons are highly influential when compared to PubMed-listed and NIH-funded publications.

Several recent studies have evaluated the use of the RCR across a variety of academic disciplines[3,6,10,14]. Within these studies, a median RCR of 1.32 (0.87-1.94) was reported amongst 1299 radiation oncologists across 75 institutions [14]; a median RCR of 1.37 (IQR 0.93-1.97) was reported amongst 1687 academic neurosurgery faculty members across 125 institutions [3]; and a median RCR of 1.38 (IQR 0.94-1.95) was reported amongst 358 academic neurosurgery spine surgeons across 125 institutions. These findings suggest that research and productivity of orthopedic spine surgeon faculty may be higher when compared to that of other specialties.

Within our study, career longevity was the principal factor associated with higher research impact. Those who started residency between the years 1981-2000 demonstrated significantly higher median RCR scores than those who started after that time period (p = .03). While these findings suggest that more experienced OSS are likely to generate more impactful research publications, the lowest median RCR was observed in the group with the longest career duration (residency start date <1980). However, this group also had the lowest publication count and weighted RCR amongst those of other career durations. We believe these findings represent an outlier to the trends observed, given the fact that the iCite database does not include data on research published prior to 1980.

In addition, although one would expect longer career duration to be associated with greater publication counts and higher overall weighted RCR scores, we did not find a significant relationship (p =  .54 and p = .216) and was also seen in the study of spine neurosurgeons by Grogan et al. which additionally did not show a relationship between career duration and mean or weighted RCR (p = 0.397 and p = .735, respectively) [6]. This is in contrast to the findings reported for academic radiation oncologists [14], neurosurgeons [3], and spine trained neurosurgeons [6]. One possible explanation for this could be the increase in international competition for publishing in U.S Spine journals [15]. Park et al. studied the publication characteristics in a leading spine journal between 2005 and 2015. Within that study, the percentage of publications from international authors increased from 17.8% in 2005 to 69.1% in 2015 (P<0.001). Furthermore, the percentage of studies with orthopedic authors decreased from 67% to 44.9%, with a corresponding increase in the percentage of studies with neurosurgeon authors and studies with a collaboration of authors from both specialties. Given the difficulty in achieving spine-related publications, as well as the recent push toward increasing the quality of orthopedic spine studies [15], the association of median RCR [i.e., study quality] and career longevity amongst academic OSS faculty members becomes more apparent.

Within our study, a significant association between advancing academic rank and weighted RCR was found (p<0.001). However, no significant association was found between advancing academic rank and median RCR (p = 0.089). This suggests that while those of higher academic rank may have greater overall research productivity (i.e., higher weighted RCR scores), the quality of publications is similar regardless of academic position (i.e., no difference in mean RCR). These findings are in contrast to academic neurosurgery spine surgeons, who's median and weighted RCR scores have been shown to increase with advancing academic rank(^3,6as has been similarly seen with h-index analysis [16]. These findings represent a specialty-specific distinction between research impact amongst those of different academic positions within the fields of neurosurgery spine and orthopedic spine surgery. Perhaps further study into these differences should occur, especially since decisions regarding faculty promotion and tenure often incorporate comparisons of research productivity[3,6].

With regard to sex-specific analysis, we found that female OSS faculty members had lower median RCR and weighted RCR scores (p = .019 and p = .040, respectively) when compared to their male counterparts. Similarly, Grogan et al. reported higher median RCR scores amongst male academic neurosurgery spine surgeons. However, it is important to note that the majority of OSS faculty members in our study were male (96.4%), which limits the accuracy of our findings. Underrepresentation of female faculty were similarly reported amongst academic neurosurgery spine surgeons (6.42%) [6], academic neurosurgeons (9%)[3], and radiation oncologists (31%) [14]. Upon further sex-specific analysis, Grogan et al. found that the observed differences in research impact and productivity amongst male and female neurosurgery spine surgeons was likely due to the underrepresentation of women in positions of higher rank. While overall analysis revealed males to have a higher median RCR score, this difference was not seen when these metrics were compared by academic rank. For example, the majority (56.5%) of females within that study were academic professors, compared with only 40.3% of males[6].

When mean and weighted RCR scores were compared between male and female assistant professors, no significant difference in either metric was found. Interestingly, the authors commented that all other ranks had too few female physicians for robust analysis[6]. Similarly based on the limited number of female academic neurosurgery spine surgeons available to be included in this analysis it is difficult to draw any true conclusions from this gender analysis. Perhaps with increasing opportunities for career advancement amongst female spine surgeons[3,6,17], the true measure of sex-specific research impact will become more clear.

Similarly, we found that only a fraction of academic OSS faculty members had a PhD (2.6%). While this percentage was roughly 5-fold greater amongst academic neurosurgery spine surgeons (13.65%) [6], the presence of a PhD had no significant association with the median RCR or weighted RCR in either study. These findings imply that spine surgeons with and without a PhD seem to generate comparable levels of research output of similar quality. This may be due to the increasingly competitive nature of the field, which often requires considerable research productivity in order to obtain a fellowship position [6]. Similarly to the limited number of females represented in this analysis, the lack of robust PhD representation in our data set limits any true conclusions to be drawn from analysis based on having a PhD.

This study has several limitations related to our sample population. First, although our sample size was large (n = 502), only fellowship-trained orthopedic spine surgeons employed at ACGME-accredited institutions were included. Thus, our analysis fails to incorporate general orthopedists who perform spine surgery, international spine surgeons not affiliated with US residency programs, and those employed within a private practice setting. Therefore, out data may not be generalizable to the entirety of the orthopedic spine surgeons. However, because the evaluation of the RCR is intended for the purpose of academic advancement and departmental evaluations, inclusion of academic-affiliated surgeons is thought to provide the most accurate specialty-specific benchmark data. Furthermore, in order to facilitate the most accurate cross-specialty comparisons, it is prudent to establish inclusion criteria in accordance with previous studies that have evaluated use of the RCR within other medical disciplines [3,6,10,13,14]. Second, our analysis relied on information provided on department websites. This represents another possible limitation as information on these website may not have been accurate or up to date at the time period of our data collection. Additionally, the low number of female orthopedic spine surgeons and those with PhD degrees may have confounded our findings.

Other limitations stem from the RCR and iCite database. One of the main benefits of the RCR also serves as a limitation. The field-normalization of the RCR allows for most accurate intra-specialty comparisons, however it is limited in comparison between specialties. Due to the difference in overall productivity among specialties, with certain specialties such as oncology and vascular having much higher gross publication numbers than specialties such as neurosurgery[1], caution must be taken with cross-specialty comparisons. Furthermore, the iCite website does not differentiate amongst researchers with the same name. Potential errors were limited by searching middle initials and reviewing individual publication titles. Additionally, the iCite website only includes PubMed articles published between 1980 and present, which may limit the accuracy of RCR and publication amongst researchers with publications prior to 1980.

Our study did not provide a comparison of RCR analyses to h-index analyses using the same data set. Previous studies have utilized comparisons between RCR and h-index analyses in order to draw the conclusion that the RCR is an effective measure of research yield and resolve many of the deficiencies present in the h-index. Due to the relative newness of the RCR metric it has yet to be established whether this metric is better than previously used metrics like the RCR. This represents a current limitation of our study, and an area for future study [10].

Conclusions

The NIH-supported RCR and its derivatives serve as new, more accurate metric of academic research impact that address many of the shortcomings associated with traditional bibliometric indices. Overall, fellowship-trained academic orthopedic spine surgeons are highly productive and produce highly impactful research, as evidenced by their weighted and median RCR scores. This data can be used as a standard for the continued evaluation of research influence within the orthopedic spine community.

• •Permission to reproduce copyrighted materials or signed patient consent forms.
• •The authors received no financial, technical or corporate support for the research, authorship, and/or publication of this article.
• •Utilized publically available data, no IRB review required.

Conflict of Interest

None.