High Research Productivity During Orthopaedic Surgery Residency May Be Predicted by Number of Publications as a Medical Student

Abstract

Introduction:

Orthopaedic applicants have increased the average number of publications on their residency application to compete with the growing competitiveness of the field. The purpose of this study was to assess whether research productivity before orthopaedic residency and caliber of one's institution is correlated with academic productivity during residency.

Methods:

Scopus was used to extract publication metrics. Quantity and quality (how often the publications were cited) were analyzed at 2 different time periods: before and during residency. All subjects in the study had graduated an ACGME-accredited orthopaedic surgery residency in 2021. Military residents, international medical graduates, and residents not listed on their department's website were excluded. Residents were categorized as both high (≥2 publications) or low (<2 publications) publishers according to their pre-residency publications. They were also categorized based on their program's Jones et al. research productivity ranking.

Results:

For the 758 residents, the median number of publications was 0 (Interquartile Range [IQR]: 0-2) and 3 (IQR: 1-6) before and during residency, respectively. High publishing medical students had more publications during training than low publishers (6 [IQR: 3-14] and 2 [IQR: 1-4], p < 0.001). Residents at higher ranked programs also had more publications (4 [IQR: 2-9] and 2 [IQR: 0-4], p < 0.001). High publishing students now training at lower ranked institutions had more publications during residency than low publishers who trained at more productive institutions (4 [IQR: 1-9] and 3 [IQR: 1-6], p < 0.001).

Conclusion:

Having 2 or more publications before residency is correlated with an increased number of publications during residency. While attending a higher academically productive program is associated with increased resident publications, a high publishing medical student would be expected to have more publications during residency than a low publishing student, regardless of program rank. Notably, most matched applicants continue to have zero publications before matriculation.

Orthopaedic research is becoming an integral part of one's residency application, with 60% to 71% of program directors viewing it as a crucial aspect influencing their decision of who to invite for an interview^1-3. Using National Resident Matching Program (NRMP) data, DePasse et al. revealed that the average number of publications and presentations among orthopaedic applicants doubled from 2007 to 2014; however, there were no data on the median⁴. Likewise, orthopaedic surgery programs that are higher tier in terms of research usually match applicants with greater academic productivity⁵. This “higher tier” status was derived from a study by Jones et al. which ranked residency-associated orthopaedic surgery departments based on the research productivity of the faculty within the program. Research productivity was determined by H-index, which counts the total number of publications along with the number of times those publications have been cited. This ranking was based off data collected from as recent as 2015 and has been validated against other metrics and published surrogates⁶.

Medical students may be involved in research for a variety of reasons beyond personal ambition. Some medical schools strongly encourage completion of a research year before graduation and many offer various funding opportunities for such experiences⁷. As orthopaedic surgery remains a competitive specialty, dedicated research years in the field have become popular and have shown to increase one's chances of matching into the field⁸. One orthopaedic residency program saw a doubling of applicants with a research fellowship gap year from 2014 to 2020⁹. Research fellowships are 1 to 2-year programs that permit medical students or recent graduates to conduct research, present at conferences, and build connections. As of 2020, there were at least 30 orthopaedic research fellowship programs in the United States¹⁰.

In addition, there has also been an increase in the number of orthopaedic residents pursuing subspecialty training in fellowships. In a 2012 survey, 87% of the orthopaedic trainees planned to pursue a fellowship. In 2021, that number had risen to 90%^8,9. This has led many fellowship programs to place heavier emphasis on research productivity in residency as a measure for their selection process. This is supported by the increase in publications among trainees applying for sports fellowship⁷. In one study, Carr et al. sought to determine whether publications during residency was an indicator of continued academic achievement after graduation. They found that residents who decide to pursue a fellowship or career in academia have a greater research quantity and quality during training¹¹. Because of this trend toward fellowship, the pressure of academic productivity has increased on both orthopaedic applicants and residents. Interestingly though, only 14% of orthopaedic surgeons as of 2018 were employed by academic institutions, and only 42% of current residents felt they would go on to perform research as an attending^12,13.

While various strategies have successfully increased resident research, there has been little investigation into the effect that prior research experience has on research productivity during residency^14-16. The purpose of this study was to assess whether research productivity before orthopaedic residency and research caliber of one's residency institution are correlated with academic productivity during residency. With the United States Medical Licensing Examination Step 1 becoming pass/fail, we will likely see an increase in publications among applicants, and these data will provide program directors with information to assess how the research characteristics of an applicant will best align with the goals of their program.

Methods

Database

A list of 2021 graduates from US ACGME-accredited orthopaedic surgery residency programs was obtained by visiting individual program websites. Residents of military programs, international medical graduates (IMGs), and residents not listed on their department's website were excluded. No information regarding additional degrees or dedicated research years was obtained. Institutional review board approval was not sought for this study because all data were extracted from publicly available records. Using the Scopus database (Elsevier, Amsterdam, Netherlands), authors were identified using medical school and residency affiliations. The unique Scopus ID of each author was used in an Application Programming Interface (API) query of the Scopus database. Our algorithm is based on Pybliometrics¹⁷. If an author did not have a Scopus ID, a manual count of publications and H-index was conducted if the publication time points could accurately be confirmed. The query was performed on August 25, 2022.

Quantity and Quality Metrics

All publications, regardless of medical topic and type, were included and used as an author's quantity metric. H-index was the metric used to assess the quality of an author's publications. A greater H-index indicates higher productivity and quality of an author. It is important to note that the H-index is subject to increase over time, and we extracted each author's H-index as of the day the query was run.

Data were extracted at different time points in an author's career, the first time point being before December 31, 2016, which was used to determine research productivity before residency. A grace period until December 31 was given to authors to account for research conducted before residency that was not published until after the start of their training. This method has been used in previous studies¹¹. The second time frame was January 1, 2017, to December 31, 2021, which was used to represent an author's productivity during residency, giving the same ending grace period for projects submitted during residency. The end point was determined to avoid having publications from either an author's post-residency career or fellowship.

Productivity ranking of a program was determined using the Jones et al. rankings published in 2018, which ranked programs 1 through 157 in terms of academic productivity of the department⁶. If a program was not listed, they were given a ranking of 158. There were no ACGME-approved osteopathic residency programs included in the ranking.

Statistics

Residents were divided into 2 groups: low (less than 2 publications before residency) or high (2 or greater publications) publishers. The value 2 was chosen as the benchmark because it was the closest whole number to the average number of publications for the class entering residency. A second grouping of each resident was made based on if they attended a top or bottom 50% program according to the Jones et al. rankings. Using these groupings, continuous variables were then compared using a nonparametric Mann-Whitney U test. Median values were used to determine significance and represented publication counts, H-index values, and program rank to account for outliers. H-index was not used to categorize the residents because the H-index today is not reflective of the H-index at the time they started residency. Significance was set at 0.05.

Results

Seven hundred eighty-eight orthopaedic surgery residents in the graduating class of 2021 met inclusion criteria. Thirty residents were excluded for being IMGs, military residency graduates, or those not found on the program's website. The remaining 758 residents encompassed 138 different orthopaedic residency programs. A Scopus ID was available for 514 authors which were used in the API query. Thirty-two programs had resident information available but were not listed on the Jones et al. productivity ranking and thus were given a ranking of 158.

Table I shows the mean and median publication counts and H-index of published studies before and during residency. The median pre-residency publication count of the 2021 graduating class of orthopaedic surgery residents was 0 (Interquartile Range [IQR]: 0-2). The range of pre-residency publications was 0 to 61. During residency, the median was 3 (IQR: 1-6) publications. The range of publications during residency was 0 to 92. There were 411 residents (54.2%) who had no publications before residency and 129 residents (17%) who had none during residency (Figs. 1 and 2).

Fig. 1.

Bar graph illustrating the frequency of residents having each number of publications before starting residency.

Fig. 2.

Bar graph illustrating the frequency of residents having each number of publications during residency.

TABLE I.

Quantity and Quality of Publications^*

N = 758 Residents	Before Residency		During Residency
	Mean (SD), [95% CI], Median (IQR)	Range	Mean (SD), [95% CI], Median (IQR)	Range
Publications per resident	1.9 (4.5), [1.6-2.2], 0 (0-2)	0-61	5.8 (9.9), [5.1-6.5], 3 (1-6)	0-92
H-Index of all publications per resident	1.4 (2.8), [1.2-1.6], 0 (0-2)	0-32	2.3 (2.8), [2.0-2.4], 1 (1-3)	0-21
Residents with zero publications (n = 758)	N = 411 (54.2%)		N = 129 (17.0%)

^*Analysis of all residents graduating from ACGME-accredited orthopaedic surgery residency programs in 2021. The H-index was as of 2022.

Five hundred thirty-six residents (71%) had less than 2 publications before residency (low publishers), and 222 residents (29%) had 2 or more publications (high publishers). During residency, low publishers had a median of 2 (IQR: 1-4) publications while high publishers had 6 (IQR: 3-14). Publication count (p < 0.001) and H-index (p < 0.001) were both significantly different, with high publishers outperforming in both. When categorized by their program's research ranking, residents who attended a program ranked in the top 50% had a median of 4 (IQR: 2-9) publications while those who attended a program in the bottom 50% had a median of 2 (IQR: 0-4) (p < 0.001) (Table II). Median program rank between these 2 groups was significantly different, with low publishers being from programs with a median rank of 82 (IQR: 43.3-135) and high publishers from programs with a more competitive median rank of 36 (IQR: 13-78.3) (p < 0.001) (Table III).

TABLE II.

Comparison of All Residents^*

	Publications During Residency			H-index of Residency Publications
	N	Mean (SD) [95% CI] Median (IQR)	p	N	Mean (SD) [95% CI] Median (IQR)	p
Low publishers	536	3.5 (5.2) [3.0-3.9] 2 (1-4)	<0.001†	535	1.5 (1.6) [1.3-1.6] 1 (0-2)	<0.001†
High publishers	222	11.4 (14.9) [9.5-13.4] 6 (3-14)		218	4.2 (3.8) [3.7-4.7] 3 (2-6)
Residents at a bottom 50% program	334	3.1 (5.1) [2.5-3.6] 2 (0-4)	<0.001†	332	1.3 (1.7) [1.2-1.5] 1 (0-2)	<0.001†
Resident at a top 50% program	424	8 (11.9) [6.8-9.1] 4 (2-9)		421	3.0 (3.2) [2.7-3.3] 2 (1-4)

^*A lower program rank indicated a higher academically productive program. A top 50% program is indicative of a more productive program. Low publishers and high publishers refer to <2 or ≥2 publications before residency, respectively. “N” represents the number of residents.

^†Indicates statistical significance.

TABLE III.

Comparison of All Residents^*

	Program Rank
	N	Mean (SD) [95% CI] Median (IQR)	p
Low publishers	536	86.9 (51.1) [82.6-91.3] 82 (43.3-135)	<0 0.001†
High publishers	222	50.2 (44.6) [44.3-56.1] 36 (13-78.3)

^*A lower program rank indicated a higher academically productive program. The most academically productive program would be ranked #1. Low publishers and high publishers refer to <2 or ≥2 publications before residency, respectively. The program rank was determined by Jones et al. 2016. “N” represents the number of residents.

^†Indicates statistical significance.

For residents entering training with less than 2 publications, the median number of publications during residency differed significantly between attendees of top and bottom 50% programs (3 [IQR: 1-6] vs. 1 [IQR: 0-3], respectively, p < 0.001). The median number of publications during residency for high publishers who attended a top 50% program was 7 (IQR: 4-15) vs. a median of 4 (IQR: 1-9) for those who attended a bottom 50% program (p < 0.001). Furthermore, high publishing medical students now training at lower ranked institutions had more publications during residency than low publishers who trained at more productive institutions (4 [IQR: 1-9] and 3 [IQR: 1-6], p < 0.001) (Table IV). A higher median H-index was also seen in the subgroup who attended a top 50% program vs. a bottom 50% program (1 [IQR: 1-3] vs. 1 [IQR: 0-2], respectively, p < 0.001) (Table V).

TABLE IV.

Publications During Residency Based on Publication Benchmark and Program Ranking^*

	Bottom 50% Ranked Program		Top 50% Ranked Program		p
	N	Mean (SD) [95% CI] Median (IQR)	N	Mean (SD) [95% CI] Median (IQR)
Low publishers	279	2.4 (3.4) [1.9-2.8]1 (0-3)	257	4.7 (6.4) [3.9-5.5] 3 (1-6)	<0.001†
High publishers	55	6.8 (9.2) [4.3-9.3] 4 (1-9)	167	13 (16.1) [10.5-15.4] 7 (4-15)	<0.001†
p-value		<0.001*		<0.001*

^*A lower program rank indicated a higher academically productive program. A top 50% program is indicative of a more productive program. Low publishers and high publishers refer to <2 or ≥2 publications before residency, respectively. “N” represents the number of residents.

^†Indicates statistical significance.

TABLE V.

H-index During Residency Based on Publication Benchmark and Program Ranking^*

	Bottom 50% Ranked Program		Top 50% Ranked Program		p
	N	Mean (SD) [95% CI] Median (IQR)	N	Mean (SD) [95% CI] Median (IQR)
Low publishers	278	1.0 (1.1) [0.9-1.2] 1 (0-2)	257	1.9 (2) [1.7-2.2] 1 (1-3)	<0.001†
High publishers	54	2.9 (2.8) [2.1-3.6] 2 (1-5)	164	4.6 (4) [4.0-5.2] 3 (2-6)	<0.001†
p		<0.001†		<0.001†

^*A lower program rank indicated a higher academically productive program. A top 50% program is indicative of a more productive program. Low publishers and high publishers refer to <2 or ≥2 publications before residency, respectively. “N” represents the number of residents.

^†Indicates statistical significance.

Discussion

Orthopaedic surgery has been increasingly competitive over recent years resulting in applicants searching for ways to strengthen their applications. One avenue is through research, which many programs value because it aligns with their own goals. Because of this, we wanted to explore how research productivity during medical school relates to productivity during residency. Our study found a positive correlation between research productivity as a medical student and research productivity as an orthopaedic surgery resident. Furthermore, another positive correlation was shown between higher academically productive programs and a resident's research output. We also discovered that higher publishing medical students typically attended higher academically productive institutions compared with low publishers.

Reports by the NRMP show that since 2014, the mean research experiences in a matched orthopaedic applicant increased from 3.7 to 6.6, although these average values may be subject to outliers^18-22. When analyzing median values, we found that the number of publications for matched applicants is zero. This mean-median discrepancy suggests that while the gross number of publications coming from applicants increases, the median applicant still has no publications before residency. However, one possibility is that many more students are conducting research that does not result in publication. In addition, the Electronic Residency Application Service (ERAS) allows applicants to list research that is not currently published; however, less than two-thirds of these manuscripts listed as in-submission will eventually lead to publication²³. Unfortunately, for program directors, current ways to predict future research productivity of an applicant can be ambiguous. Our results in Table II show that high publishing students had more publications during residency compared with low publishers (p < 0.001). Likewise, these publications were ultimately of higher quality, judged by the H-index. This trend shows that a higher publishing student will likely bring more quality along with the quantity of publications to a program.

While our results show that prior research publications correlate with an applicant's productivity during residency, we also aimed to determine how the research caliber of an individual's program affects their research productivity during residency. Regardless of previous publication count, Table II presents that residents who attended a top 50% program had more publications during residency compared with a resident at a program in the bottom 50% (p < 0.001). Likewise, high publishing medical students and those who attended a higher ranking orthopaedic surgery residency program had a significantly higher quality of research, as measured by the H-index. An important consideration when looking at the H-index of students and residents is the overall effect of the faculty contributions to the publications. It is certainly likely that academic orthopaedic surgeons at highly ranked programs enhance the quality and publicity of the projects they work on, skewing the results of this study, which specifically aim to represent the merit of the students and residents themselves.

Many strategies have been shown to increase research during residency. Implementation of research meetings, hiring a research coordinator, restructuring the curriculum, and protected research days have all shown to increase productivity during training^14-16,24. In terms of restructuring the curriculum, this includes incorporating research milestones for each training year, mentoring by National Institutes of Health–funded scientists, and protected time to engage in required research and prepare scholarly peer-reviewed publications¹⁶. Although these listed strategies have the potential to boost research productivity of a program, it is up to the residents to efficiently use these resources. The results in Table IV analyze how different institutions enhance research productivity. A low publisher who attended a top orthopaedic residency program had more than double the number of publications of a low publisher who attended a bottom 50% program (p < 0.001). The same trend was seen for high-publishing residents (p < 0.001). When comparing low vs. high publishers at a bottom 50% program, the high publishers had more publications than the low publishers (p < 0.001). Similarly, at top 50% programs, high publishers had more publications during training (<0.001). The median number of publications during residency for residents who were high publishers during medical school is higher regardless of program tier compared with residents who were low publishers during medical school.

The same can be said for the quality of research, as presented in Table V. High publishers also tend to have a higher H-index during their training than low publishers, regardless of the institution they attend. This likely stems from the ability of these residents to use their developed skills to contribute to the growing body of high-quality literature within orthopaedics.

This study does not come without limitations. Our method of data extraction by using an API query was limited by a failure to identify which field of medicine the publication pertained to, nor the study design, thus limiting our ability to make conclusions regarding the effect these factors have on predicting academic success. Examining only a single year of orthopaedic residency graduates limited the study's ability to analyze trends in publications over the years among successful orthopaedic surgery applicants. In addition, there were more than twice as many students with fewer than 2 publications before residency than students with 2 or more, which likely presents selection bias due to potential confounding variables between groups. Furthermore, this study did not consider factors such as co-authorship, primary authorship, and multiple student or resident authors listed on publications. We do, however, believe that because we used a threshold of ≥2 publications before residency rather than reporting the actual publication numbers of each student, the likely padded numbers resulting from poly co-authorship were controlled for because it is likely that primary authorship required more time and effort than co-authorship.

Other limitations include demographic data not being collected on the residents as well as not being able to determine whether a medical student's publications came from a research year position. In addition, because more programs have become accredited since the Jones et al. ranking and these programs were ranked arbitrarily at 158 in our study, we inevitably introduced bias to our results. However, we decided to include these programs at this rank because it could be inferred that these newly accredited programs likely have a lower research productivity. Finally, only using one database in our study to retrieve publication metrics limits the study's ability to ensure all publications by individual authors were included in the analysis.

Conclusion

Having 2 or more publications before residency is correlated with an increased number of publications during residency. While attending a higher academically productive program is associated with increased resident publications, a high publishing medical student would be expected to have more publications during residency than a low publishing student, regardless of program rank. Notably, most matched applicants continue to have zero publications before matriculation.