The Relative Citation Ratio: A Modern Approach to Assessing Academic Productivity within Plastic Surgery

Background:

The accurate assessment of physician academic productivity is paramount and is frequently included in decisions for promotion and tenure. Current metrics such as h-index have been criticized for being biased toward older researchers and misleading. The relative citation ratio (RCR) is a newer metric that has been demonstrated within other surgical subspecialties to be a superior means of measuring academic productivity. We sought to demonstrate that RCR is a valid means of assessing academic productivity among plastic surgeons, and to determine demographic factors that are associated with higher RCR values.

Methods:

All Accreditation Council for Graduate Medical Education-accredited plastic and reconstructive surgery residency programs and faculty throughout the United States were compiled from the American Council of Academic Plastic Surgeons website. Demographic information was obtained for each surgeon via the program’s website, and RCR data were obtained utilizing iCite, a bibliometrics tool provided by the National Institutes of Health. Surgeons were excluded if any demographic or RCR data were unavailable.

Results:

A total of 785 academic plastic surgeons were included in this analysis. Surgeons who belonged to departments with more than six members had a higher median RCR (1.23). Increasing academic rank (assistant: 12.27, associate: 24.16, professor: 47.58), chief/chairperson status (47.58), male gender (25.59) and integrated model of residency training program (24.04) were all associated with higher median weighted RCR.

Conclusions:

RCR is a valid metric for assessing plastic surgeon academic productivity. Further research is warranted in assessing disparities among different demographics within academic plastic surgery.

Takeaways

Question: Is relative citation ratio (RCR) a valid means of assessing academic productivity among plastic surgeons and are there demographic factors among plastic surgeons associated with higher RCR values?

Findings: Demographic information and mean (mRCR) and weighted RCR (wRCR) values for 785 academic plastic surgeons were obtained. Statistical analyses assessing differences in mRCR and wRCR were performed. Increasing academic rank, chief/chairperson status, male gender, and integrated residency training model were associated with higher wRCR.

Meaning: RCR is a valid metric for assessing academic productivity among plastic surgeons.

INTRODUCTION

Research productivity is one of the most significant factors in evaluating the achievements of academic faculty and tracking their career progression.^1–3 Bibliometric data have long been used to assess the quantity and quality of an academic’s research production. Previous literature has shown certain metrics have served as a valid metric for evaluating academic productivity in plastic surgery.^1,4 Utilizing bibliometric data allow for objective assessments of performance rather than relying on subjective measures, such as letters of recommendation or reputation within a field. Having objective data is crucial for decisions related to funding, promotion, and tenure. Many comprehensive measurements, such as the Hirsch index (h-index), I-10 index, g-index, e-index, and m-quotient, were developed to overcome the limitations of simple quantitative metrics.^2,5 However, all of these metrics are still fraught with limitations. For instance, the h-index, which is the current standard metric, accounts for the quantity of a research work and attempts to control for the quality by factoring in the impact of those works. Criticisms of the h-index include inability to compare scientists from different fields, inherent benefit to older researchers, and misleading depiction of productivity and impact.^4,6

The relative citation ratio (RCR) is a modernized, effective method of measuring productivity that addresses the disadvantages of other popular indices. The National Institutes of Health (NIH) developed the RCR as a validated tool in 2015 to combat the limitations of the h-index.⁵ RCR generates an article-level score derived from dividing the total number of citations per year of that publication (article citation rate) by the average annual number of citations for NIH-funded publications in the same field (expected citation rate), as determined by the article’s co-citation network.⁴ Advantages of RCR over h-index include the ability to make inter-field comparisons and reduction of temporal discrepancies between researchers at varying career stages.⁷ The mean RCR is an average of the author’s article-level RCR scores and provides a more fair representation of impact without allowing gross number of publications to disadvantage younger researchers.⁴ The weighted RCR is the sum of all article-level RCR scores for each of an author’s publications.

To our knowledge, no research has evaluated the RCR with respect to the field of plastic surgery. This study had two aims: (1) analyze mean and weighted RCR scores for a cohort of 785 academic plastic surgeons located in the United States, and (2) investigate the impact of factors such as gender, academic rank, and size of plastic surgery faculty on their RCR scores.

METHODS

Inclusion Criteria

A list of all Accreditation Council for Graduate Medical Education-accredited plastic and reconstructive surgery (PRS) residency programs throughout the United States was compiled from the American Council of Academic Plastic Surgeons website. Each program’s website was used to identify currently practicing academic PRS surgeons for inclusion in this study’s analysis. Departmental websites and the American Council of Academic Plastic Surgeons website were also utilized to collect data for use as variables in our analysis, including program geographic region, faculty size of program (<6 members or greater), model of residency training program (Independent versus integrated), and whether the program was considered its own department or as a division within the department of surgery. Data were also collected on each faculty member, including name, gender, whether they were fellowship trained, type of fellowship(s) (microsurgery, hand, craniofacial, aesthetic, or other) completed, academic rank (assistant professor, associate professor, or professor), chief or chairperson status, and program director status.

Bibliometric Data Collection and Exclusion Criteria

As aforementioned, the RCR is calculated by standardizing the number of citations an article receives to the NIH benchmark for that field. Thus, an RCR of 1.0 is defined as the median for all NIH-funded publications within the author’s specialty. The mean and weighted RCR is the calculated average and sum, respectively, of all RCRs for each author’s collection of publications as a whole. Therefore, the mean RCR is effective for comparisons of research impact, while the weighted RCR is a practical measure to assess productivity. Each of these metrics are indexed in the NIH iCite website for publications included in PubMed from 1980 to 2021. PRS faculty members from each institution were individually queried in the NIH iCite system in September and October 2021. Total number of publications, mean RCR, and weighted RCR were acquired per faculty member. However, faculty members were excluded from the study if their name was common to the degree that it made identifying their particular publications from another person with the same name impossible. Additionally, PRS residency programs were excluded from analysis if they did not include academic rank on their faculty pages.

Ethical Considerations

This study was exempt from institutional review board approval by Rutgers New Jersey Medical School as it did not involve human subject research.

Statistical Analysis

Mean and median RCR and weighted RCR were calculated for all academic PRS surgeons. Data were compared by academic rank, chief or chairperson, program director status, gender, model of residency training program, status of plastic surgery section (department versus division), and faculty size of program (<6 members or greater). Significance (P < 0.05) was assessed via the Mann-Whitney U test for two groups and the Kruskal-Wallis test for more than two groups. All statistical analyses were performed using SPSS 27.0 (IBM, Armonk, N.Y.) and Microsoft Excel.

RESULTS

A total of 785 academic plastic surgeons were included in the data analysis. Demographics of participants are shown in Table 1. Tables 2 and 3 provide mean and weighted RCR values respectively‚ both with detailed breakdowns by gender, academic rank, chief/chairperson categorization, program director status, model of residency training program, status of plastic surgery section, and size of plastic surgery faculty.

Table 1.

Demographic Information

	No.	Percentage
Academic rank
Assistant	345	43.9
Associate	211	26.9
Professor	229	29.2
Chief or Chairperson
Yes	121	15.4
No	664	84.6
Chief/Chairperson subcategorized
Chief	93	76.9
Chairperson	28	23.1
Program Director
Yes	85	10.8
No	700	89.2
Gender
Women	186	23.7
Men	599	76.3
Model of residency training program
Integrated	659	83.9
Independent	126	16.1
Status of plastic surgery section
Department	232	29.6
Division	553	70.4
Size of plastic surgery faculty
6 or more members	707	90.1
Fewer than 6 members	78	9.9

Table 2.

Mean RCR of Academic Plastic Surgeons Categorized by Subgroup

	Number	Mean	SD	Median	Interquartile Range	P
Academic rank
Assistant	345	1.3362	1.95759	1.14	0.74––1.65	0.052
Associate	211	1.4346	1.13202	1.25	0.95–1.65
Professor	229	1.3031	0.62474	1.21	0.87–1.65
Chief or Chairperson
Yes	121	1.3548	0.80821	1.23	0.88–1.58	0.688
No	664	1.3527	1.55349	1.2	0.81–1.65
Chief/Chairperson subcategorized
Chief	93	1.3863	0.87063	1.25	0.86–1.65	0.531
Chairperson	28	1.25	0.55277	1.12	0.89–1.48
Program Director
Yes	85	1.3368	0.75584	1.25	0.81–1.65	0.731
No	700	1.355	1.52725	1.205	0.84–1.65
Gender
Women	186	1.2917	0.76522	1.13	0.74–1.74	0.728
Men	599	1.372	1.61989	1.22	0.86–1.62
Model of Residency Training Program
Integrated	659	1.3476	1.4836	1.21	0.86–1.64	0.685
Independent	126	1.381	1.35652	1.155	0.76–1.72
status of plastic surgery section
Department	232	1.2651	0.72309	1.15	0.82–1.58	0.332
Division	553	1.3899	1.67846	1.22	0.84–1.66
Size of plastic surgery faculty
6 or more members	707	1.33	0.84801	1.23	0.85–1.65	0.021
Fewer than 6 members	78	1.5618	3.89299	1.07	0.69–1.50

Bolded values are significant at P < 0.05.

Table 3.

Weighted RCR of Academic Plastic Surgeons Categorized by Subgroup

	No.	Mean	SD	Median	Interquartile Range	P
Academic rank
Assistant	345	27.615	40.3642	12.27	3.84–35.20	<0.001
Associate	211	51.8784	69.39453	24.16	11.57–67.58
Professor	229	97.5998	153.8206	47.58	17.85–115.38
Chief or Chairperson status
Yes	121	89.6926	125.7242	47.58	16.76–112.70	<0.001
No	664	48.1492	91.65134	19.57	6.91–55.12
Chief/Chairperson Subcategorized
Chief	93	88.7133	130.6664	47.4	16.08–112.70	0.547
Chairperson	28	92.945	109.81848	48.765	19.96–137.78
Program Director
Yes	85	47.318	57.0663	23.64	9.41–56.77	0.555
No	700	55.4312	102.64086	21.735	7.67–64.52
Gender
Women	186	36.0772	67.24433	15.285	4.75–35.12	<0.001
Men	599	60.2897	106.02951	25.59	9.87–71.86
Model of residency training program
Integrated	659	56.2765	102.92977	24.04	8.48–66.03	0.021
Traditional	126	45.5368	72.61967	17.455	5.55–47.17
Status of plastic surgery section
Department	232	47.6076	62.69092	23.385	10.49–60.50	0.492
Division	553	57.4663	110.32608	21.24	7.17–65.11
Size of plastic surgery faculty
6 or more members	707	57.7684	102.88746	24.04	8.23–67.58	0.065
Fewer than 6 members	78	25.4055	35.20553	13.59	6.54–28.95

Bolded values are significant at P < 0.05.

Academic Rank

Among assistant, associate, and full professor academic positions, assistant professor was the majority with 345 surgeons (43.9%), followed by full professor with 229 surgeons (29.2%) and associate professor with 211 surgeons (26.9%). Although median RCR values did not show significant difference across academic positions, median weighted RCR showed positive correlation with successive rank with values of 47.58, 24.16, and 12.27 (P < 0.001) for professor, associate professor, and assistant professor, respectively (Fig. 1).

Fig. 1.

Median weighted RCR stratified by academic rank.

Gender

The majority of academic plastic surgeons in our study were men (n = 599 | 76.3%). Male plastic surgeons had a mean RCR score of 1.22, whereas female plastic surgeons had a mean RCR score of 1.13. Although there was no significant difference in mean RCR (P = 0.728), there was significant difference in weighted RCR (P < 0.001). The median weighted RCR for male plastic surgeons was 25.59, whereas the value for female plastic surgeons was 15.29 (Fig. 2).

Fig. 2.

Median weighted RCR stratified by sex.

Chief or Chairperson Status

Plastic surgeons with a chairperson or chief status had significantly different median weighted RCR of 47.58 compared with those who were not chairperson or chief‚ with RCR of 19.57 (P < 0.001) (Fig. 3). No difference was found when comparing between chairperson and chief positions. Program director position also showed no significant difference with a median weighted RCR of 23.64.

Fig. 3.

Median Weighted RCR stratified by chief/chairperson status.

Plastic Surgery Program Structure

Plastic surgery programs with faculty size greater than or equal to six had a median RCR value of 1.23 compared with those with fewer than six with median RCR value of 1.07 (P = 0.021). This was the only category with significant difference when looking at unweighted RCR values. However, its weighted RCR did not show any significant difference. Looking at the model of plastic surgery residency training program, the integrated program had a significantly higher median weighted RCR of 24.04 compared with the independent program with 17.455 (P = 0.021) (Fig. 4). Status of plastic surgery section did not show any significant difference.

Fig. 4.

Median Weighted RCR stratified by model of residency training program.

DISCUSSION

Understanding the academic productivity of a physician is a paramount component in evaluating one’s contributions to academia. However, provided there are various metrics for analysis and each varies in its components, determining which computational metric is most representative of a physician’s true academic impact has proven to be difficult. The RCR is a relatively new metric and there has been no research demonstrating its utility in evaluating the academic productivity of plastic surgeons.

Upon comparison of subgroups of academic plastic surgeons via mean RCR, only size of plastic surgery faculty demonstrated a significant difference, with surgeons working in departments/divisions with more than six members having higher mean RCRs. Our postulation for this difference is that faculty at larger programs have access to more funding and resources, allowing them to publish more meaningful and higher quality research. This finding is consistent with previous research on productivity among academic plastic surgeons.³ The positive correlation across academia between productivity and department size has been suggested by numerous studies for many years now. Some researchers have attributed this to a more enriched environment as a result of the greater number of faculty. Although the impact of size on academic productivity when controlling for other variables such as research support and faculty rating is not as clear.^6,8-11

Other studies assessing the academic productivity of neurosurgeons found significant differences among similarly categorized subgroups using mean RCR.¹² The comparison of mean RCRs among subgroups only found one significant difference, whereas weighted RCR demonstrated differences between academic rank, chief/chairperson status, gender, and model of training program. Weighted RCR is the sum of all article-level RCR scores over a surgeon’s career.⁴ As other authors have suggested, weighted RCR is a better metric than mean RCR when assessing the overall academic productivity of plastic surgeons‚ and the aforementioned discrepancy between weighted and mean RCR tables in our research supports this.

Our research demonstrated increasing median weighted RCR values as academic rank increases. This is a reasonable trend, as similar research assessing academic productivity in other surgical subspecialties found similar results. This research among both ophthalmologists and neurosurgeons found a positive correlation between academic rank and RCR values.^7,13,14 Previous research assessing the relationship between academic rank and productivity among plastic surgeons using separate measures found similar results. Although academic rank is a product of successes in various areas, authors suggested utilizing these numeric productivity metrics to determine a surgeon’s impact.² The academic status of chief or chairperson was also associated with an increased median weighted RCR when compared with nonchiefs or chairpersons. Again, this follows the aforementioned trend of increasing productivity with increased academic status. However, there was no statistically significant difference between chief status and chairperson status.

It is evident that academic productivity plays a significant role in allowing academic surgeons to move up in the ranks. The data demonstrate an increase in weighted RCR as rank increases. This further validates the argument that RCR is a proper metric for assessing productivity, as it adequately captures the trends established by other metrics, without many of the drawbacks. Utilizing RCR as a metric for measurement has already been demonstrated to be successful in other surgical subspecialties and should be in strong consideration when assessing an academic plastic surgeon.

We identified a significant difference in weighted RCR between the genders. Male plastic surgeons had a much higher median weighted RCR when compared with female surgeons. This trend has been demonstrated across a variety of subspecialties. A study assessing academic productivity among head and neck surgeons found that the mean RCR of male surgeons was nearly double that of female surgeons.¹⁵ In research among academic radiation oncologists, it was shown that there was no statistically significant difference in mean RCR between the genders, but that male physicians had a higher weighted RCR than female physicians.¹⁶ Previous research utilizing h-index also demonstrated a difference between male and female surgeon academic productivity.³ Unfortunately, this seems to be a trend across various medical and surgical subspecialties. Historically, plastic surgery has been a male dominated field. As these data take into account decades of research, it is reasonable to see a gap in productivity. However, as plastic and reconstructive surgery as a field works to close this gap, it is likely that the gap in academic productivity will close as well.¹⁷

Additionally, our findings demonstrate a significant difference in mean weighted RCR between integrated and independent programs. This is consistent with research of academic plastic surgeons utilizing other bibliometric indices.³ This may be suggestive of a trend in program research culture, with integrated programs placing greater value on research than independent, as a whole. Furthermore, research comparing applicants to other integrated surgical subspecialty programs as compared with traditional found that integrated applicants had a greater number of peer reviewed publications as well as increased interest in remaining in academia.^18,19 Further research is warranted in understanding this difference in productivity that exists between surgeons in integrated and independent programs.

Our findings demonstrate that RCR is a valid and important tool in assessing academic productivity among plastic surgeons. Many of the relationships between different demographic variables and academic productivity found in this study utilizing RCR are also seen using other, older measures such as h-index and g-index.^1,3 In addition to being a valid measurement of academic productivity, RCR can be easily translated across different medical specialties as well as scientific fields. Furthermore, older measurements, such as h-index, have been criticized for being poorly translated across fields as well as being biased against younger female researchers.^20,21 This study demonstrates the great utility of the RCR and provides insight into differences in academic productivity among academic plastic surgeons.

There a few notable limitations to this study. Some plastic surgeons with common names were unable to be included in this study due to the nature of the iCite database. Furthermore, other surgeons were excluded from the analysis if their program websites lacked information such as academic rank and chief/chairperson status. Although not numerous, these missing pieces of data could have influence on study findings. Researchers who have changed their name may have some work left unaccounted for when utilizing the iCite system to collect data, as the data collection is driven primarily by first, middle and last name. Lastly, the accuracy of departmental websites provides a limiting factor for the study, as some webpages may be out of date.

CONCLUSIONS

RCR is a new and effective means of measuring academic productivity. It yields accurate, up-to-date numerical data with fewer of the drawbacks seen in older measures such as h-index. Among academic plastic surgeons, increasing rank, chief or chairperson status, male gender, and integrated residency training model were associated with greater academic productivity. RCR should continue to be utilized in the future to accurately assess plastic surgeon academic productivity. Further research regarding disparities in academic productivity among other demographic variables is warranted.

ACKNOWLEDGEMENTS

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.