Which Application Factors Are Associated With Outstanding Performance in Orthopaedic Surgery Residency?

Abstract

Background

Identifying ideal candidates for orthopaedic surgery residency is difficult. Data available for applicant selection are evolving; preclinical grades and the Alpha Omega Alpha (AOA) honors society are being phased out at some medical schools. Similarly, three-digit United States Medical Licensing Examination (USMLE) Step 1 scores have been eliminated. There is renewed interest in improving resident selection to provide a diverse, comprehensive educational opportunity that produces orthopaedic surgeons who are prepared for practice.

Questions/purposes

We sought to identify whether (1) academic achievements, (2) letters of recommendation, (3) research activity, and (4) miscellaneous factors available on Electronic Residency Application Service (ERAS) applications were associated with outstanding residency performance.

Methods

Ten faculty members (22% of all full-time faculty) with extensive educational involvement for at least 7 years, whose expertise covered all subspecialty departments at an urban, academic orthopaedic surgery residency program, were given an anonymous survey on the performance of the four most recent classes of residency graduates (24 residents). This survey was developed due to the lack of a validated residency outcomes tool or objective metrics for residency performance. The evaluated criteria were decided upon after discussion by a relatively large group of academic orthopaedic surgeons considering the factors most important for graduating orthopaedic residents. The faculty were selected based on their long-term knowledge of the residency, along with their diversity of specialty and backgrounds; there were no nonresponders. Faculty graded each resident on a scale from 1 to 10 (higher is better) on six criteria: surgical technical skills, research productivity, clinical knowledge, professionalism, personality, and fellowship match. The mean of the faculty ratings made by all faculty for all six criteria was calculated, producing the overall residency performance score. Factors available on each resident’s ERAS application were then correlated with their overall residency performance score. Categorical ERAS factors, including AOA status, five or more honors in core clerkships, at least three exceptional letters of recommendation, collegiate athletics participation, expertise with a musical instrument, and research (6-year) track residents, were correlated with overall residency performance score via point biserial analysis. Continuous ERAS factors including USMLE Step 1 and Step 2 scores, number of publications before residency, number of research years before residency, medical school ranking, and number of volunteer experiences were correlated with overall residency performance score via Pearson correlation. USMLE Step 1 three-digit scores were evaluated despite their recent elimination because of their historic importance as a screening tool for residency interviews and for comparison to USMLE Step 2, which retains a three-digit score. Application factors with a p < 0.2 on univariate analysis (five or more honors in core clerkships, at least three exceptional letters of recommendation, research track residents) were included in a stepwise linear regression model with “overall residency performance score” as the outcome variable. All p values < 0.05 were considered significant.

Results

The mean overall residency performance score was 7.9 ± 1.2. Applicants with at least five honors grades in core clerkships had overall residency performance scores 1.2 points greater than those of their peers (95% confidence interval (CI) 0.3 to 2.0; p = 0.01, Cohen ƒ² = 0.2, representing a small effect size). ERAS applications including at least three exceptional letters of recommendation were associated with a 0.9-point increase in residency performance (95% CI 0.02 to 1.7; p = 0.046, Cohen ƒ² = 0.1, representing a small effect size). Participation in the residency research (6-year) track was associated with a 1-point improvement in residency performance (95% CI 0.1 to 1.9; p = 0.03, Cohen ƒ² = 0.2, again, representing a small effect size). Together, these three factors accounted for 53% of the variance in overall residency performance score observed in this study.

Conclusion

Past clinical excellence, measured by core clerkship grades and exceptional letters of recommendation, is associated with slightly improved overall orthopaedic residency performance scores. Applicants meeting both criteria who also complete a research track residency may perform substantially better in residency than their counterparts, as these three factors accounted for half of all the variance observed in the current study. Although minimum requirements are necessary, traditionally used screening factors (such as USMLE scores, AOA status, medical school rank, and number of publications) may be of less utility in identifying successful future residents than previously thought.

Level of Evidence

Level III, therapeutic study.

Introduction

Selecting the best applicants for orthopaedic residency is one of the most important decisions orthopaedic departments make each year, yet it often remains a confusing and contentious process [6-8, 11]. Reliably identifying which students are most likely to succeed during residency remains difficult [1, 23]. Ideally, each residency program’s admissions committee would have empirical data about which preresidency selection factors predict outcomes important to their program, such as clinical acumen, surgical technical performance, research productivity, or accreditation examination scores [1]. Programs would then be able to develop evidenced-based models by which to make better-informed decisions when narrowing down a large applicant pool for a limited number of residency positions.

The number of orthopaedic residency applications per residency position continues to increase, leading to an increasingly heavy application burden for residency program administration each year [35]. Unfortunately, the data available to programs for differentiation of those applications have decreased in recent years. Because of its positive impact on medical student wellbeing, pass/fail preclinical grading is being implemented at many medical schools across the United States [22, 32]. Similarly, fewer medical schools are selecting students for Alpha Omega Alpha (AOA) honors society membership [2, 21]. In January 2022, three-digit United States Medical Licensing Examination (USMLE) Step 1 scores, traditionally a major screening factor for orthopaedic residency interviews, were discontinued in favor of pass/fail grading [14, 24, 35]. A consequence of these changes is increased difficulty for residency programs to make decisions based on applications lacking these data [24].

Prior studies have attempted to determine the attributes available on an Electronic Residency Application Service (ERAS) application that are associated with clinical success during orthopaedic residency [4, 7, 8, 11, 31, 36]. Although these studies were important, most lacked a multivariate analysis, increasing the potential for confounding [4, 7, 31, 36]. Additionally, these studies have disagreed with one another in important ways, resulting in the absence of any consensus on the definition of success or the factors associated with excellent performance as an orthopaedic surgical resident [4, 7, 8, 11, 31, 36].

We therefore evaluated faculty members’ perceptions of four classes of residents at an urban, academic orthopaedic surgery residency program to determine whether (1) academic achievements (USMLE scores and clerkship grades), (2) letters of recommendation, (3) research activity, and (4) miscellaneous factors available on ERAS applications are associated with outstanding residency performance.

Materials and Methods

ERAS Application Review

We reviewed the ERAS residency applications from our four most recent classes of residency program graduates (2018 to 2021). Although we knew that this small sample size (n = 24) would impose some analytic limitations, we felt that ensuring good memory of all aspects of resident performance by each faculty member was important, and for that reason we chose this time span and study size. We evaluated ERAS factors associated with academic performance including USMLE Step 1 and Step 2 scores, number of honors grades in core clerkships, AOA membership, and US News and World Report medical school research ranking. Despite their recent elimination, we evaluated USMLE Step 1 three-digit scores because they: (1) allow programs to better understand the value that Step 1 provided as the data available during the residency application process changes and (2) they permit comparison with USMLE Step 2, an exam in which three-digit scores are still available and thus still may be used for residency application screening. Similarly, research activity, sports participation, and other extracurriculars were reviewed by one author (RJF) for each resident (Table 1). Orthopaedic in Training Examination scores throughout residency were also collected. Each resident’s medical school reputation was assessed via US News and World Report medical school research rankings, with schools ranked 1 to 20 earning a score of 1, 21 to 40 earning a score of 2, 41 to 80 earning a score of 3, and more than 80 earning a score of 4. To assess research productivity during residency, we conducted PubMed searches using each resident’s last name and first initial. Articles published during the residency years plus fellowship year (to account for delays in manuscript submission for publication) that were completed at one of our teaching institutions were tallied.

Table 1.

Electronic Residency Application Service (ERAS) application and residency data (n = 24)

	Parameter	Median (IQR) or % (n)
Before residency	Medical school ranka	26 (19-50)
	USMLE Step 1 score	248 (239-255)
	USMLE Step 2 score	252 (248-270)
	Number of honors in core clerkships	4 (3-5)
	Number of volunteer experiences	7 (5-9)
	Number of exceptional LORs	2 (1-3)
	Number of publications	2 (1-5)
	First-author publications	0 (0, 1)
	AOA membershipb	48 (10)
	≥ 1 research year before residency	33 (8)
	Men	88 (21)
	First-time applicant	100 (24)
	Research grant	25 (6)
	College athlete	17 (4)
	Plays musical instrument	17 (4)
	Time off to work in nonmedical field	21 (5)
During residency	Overall performance (1-10)c	8.1 (7.2-8.7)
	Number of publications	6.5 (3-15)
	PGY-5 OITE	42 (27, 65)
	Number of first-author publications	4 (1-7)
	Research (6-year) track	33 (8)

^aUS News and World Report research ranking.

^bAOA chapter not available at medical school of three residents.

^cRange of overall residency performance score 4.9 to 9.5; LORs = letters of recommendation; PGY = postgraduate year; OITE = Orthopaedic in Training Examination.

Guidelines for differentiating exceptional letters of recommendation from strong letters (Table 2) were developed by a senior faculty member (RWL) with experience reviewing more than 1500 residency applications, which would have included 4500 to 6000 letters of recommendation. These guidelines were universally agreed upon by three other senior faculty members at our institution, who had more than 44 years of combined teaching experience. Inclusion of just one of these exceptional comments, or their equivalents, would designate a letter as exceptional. However, it was rare for a letter to only have one exceptional comment and unexceptional verbiage throughout the rest of the letter. According to these guidelines, each applicant’s letters of recommendation were categorically sorted into “exceptional” or “strong” by two authors together (RJF, RWL). Our residency requires three letters of recommendation but will review a fourth recommendation letter if provided by the applicant.

Table 2.

Key terminology differentiating exceptional LORs, as defined by faculty members with experience evaluating more than 1500 applications

Parameter	Comments defining an exceptional LOR	Positive comments not considered exceptional
Recruitment	“We will be recruiting (student) to become a member of our department”	“I would interview and rank (student) highly. We will”
	“(Student) will be at the top of our rank list and we hope to be able to match (student) here”	“(Student) is going to be very competitive for a position in our program”
	“(Student) will be ranked highly and recruited aggressively in the match here”	“(Student) would be a ‘boon’ to any program”
	“I view (student) as a can’t miss orthopaedic resident. I personally hope we are able to match (student) to our program”	“Any program (student) joins will be a fortunate one”
	“(Student) should only go to the very best in the country … I give my highest recommendation … You will be very pleased if you are able to lure (student) to your institution”	“I am sure (student) will be interviewed at the top programs across the county and will certainly be interviewed here”
	“I will do my best to convince (student) to stay and train at our residency”	“I would strongly encourage you to interview (student). You will not be disappointed”
	“A ‘must take’ applicant”	“A ‘must interview’ candidate”
Performance	“Top 10% of students” (or better)	“Top 15% of students” (or worse)
	“Our top student this year”	“Among our very best students this year”
	“Favorite medical student according to our chief residents”	“Top 3 to 5 students for us (this year)”
	“One of two ‘franchise players’ from our class this year”	“I consider (student) to be a top candidate here”
	“This is our strongest endorsement”	“I give my highest recommendation”
	“One of the very best students with whom I’ve had an opportunity to work (in my career)”	“(Student) is greater than the sum of the parts of (his/her) application”
Career prospects	“One of the future leaders of our specialty”	“(Student) is a superb candidate for residency training”
	“There is no doubt (student) will go on to become a leader in our field”	“I am certain (student) will become an outstanding orthopaedic resident and ultimately an outstanding surgeon”
Specific examples of performance	“That was perhaps the most I’ve ever let a medical student perform in my 25 years of practice!”	N/A
	“The only student in my 20-year career that I could not stump in my anatomy interrogation”

Faculty Assessment of Graduated Residents

A six-item online survey was developed to assess key aspects of resident performance on a 1- to 10-point scale: surgical skills, clinical knowledge, professionalism, fellowship match, research productivity, and personality. These six criteria were defined after extensive discussion with senior faculty regarding the necessary competencies of a graduating resident. This methodology for resident assessment was selected out of necessity; no validated residency performance outcome tool exists, and resident evaluations are inherently subjective. Surgical skills, clinical knowledge, and professionalism were included as these are three traits expected of any orthopaedic surgeon, regardless of practice setting. Fellowship match was included as most residents complete a fellowship. Residency programs hope to send their residents to the best and most prestigious fellowship training programs and routinely publish their fellowship match results [5, 28]. Although research is not a primary focus for many residents, research productivity was included as a criterion because research activity is a major variable considered by most orthopaedic training programs in their evaluation of applicants [12, 20, 33]. Both residency and fellowship programs hope that their applicants will be productive researchers while training and thus typically recruit applicants with extensive research backgrounds more heavily [5, 28, 39]. Finally, personality was included as a parameter because a sizeable body of literature from over a dozen centers exists with the goal of identifying personality traits such as grit, self-control, and conscientiousness [3, 15, 18, 19, 34].

Each of the six factors was subjectively evaluated. As an example, the item assessing clinical knowledge was written as follows: “Please grade [resident]’s clinical knowledge throughout residency.” Each of the other five items assessed a different criterion in an identical manner. For each item, a score of 1 was defined as “one of the worst-performing residents in program history” and a score of 10 was defined as “one of the best-performing residents in program history” for the evaluated criterion. Fellowship match was assessed considering both program reputation and resident fit, with higher scores being assigned to fellowship matches subjectively considered to be more prestigious or better meeting a particular graduating resident’s needs. For all items, an “unable to assess” option was available if the faculty member did not feel equipped to answer. This survey was copied 24 times, once for each resident who was evaluated, and distributed to 10 faculty members representing all subspecialty departments and each having educational involvement for at least 7 years at our urban, academic orthopaedic surgery residency program. Based on our residency rotation schedules, it would not be possible for any of these 10 faculty to have had limited exposure to any of the evaluated residents. The mean value assigned for each of the six survey items produced the residency performance score for each resident from each faculty member. We identified the mean of the resulting 10 residency performance scores for each resident to determine the overall residency performance score, the main outcome variable for this study. To minimize bias, participating faculty members reviewed resident ERAS applications only after they completed the resident performance surveys.

We considered asking faculty to stack-rank residents rather than assign scores of 1 to 10 for each criterion to each resident. A major advantage of stack-ranking would be avoidance of any potential for “clumping,” or similar grades being assigned to multiple residents. The considerable drawback of stack-ranking would be an inability for our long-tenured faculty evaluators to compare these four residency classes to the many that came before. Fortunately, our methodology of resident evaluation did not result in clumping, but rather produced a relatively linear progression in residency performance with a de facto 1 to 24 resident performance ranking (Fig. 1). Enough score variation was obtained to allow for identification of application factors that predicted residents who would be nearer the top of the performance distribution.

Fig. 1.

This graph shows the study’s distribution of overall residency performance scores. It is important to note that our methodology of resident evaluation did not result in clumping but rather produced a relatively linear progression in residency performance with a de facto 1 to 24 resident performance ranking. Although our mean scores ranged from 5.1 (average) to 9.6 (excellent), enough variation in scores was obtained to allow for identification of application factors that were associated with residents who were nearer the top of the performance distribution.

Primary and Secondary Study Outcomes

Our primary study goal was to identify which academic achievements available on ERAS applications were associated with overall residency performance. To achieve this, we performed a univariate analysis of the following ERAS factors against overall residency performance score: USMLE Step 1 and Step 2 scores, AOA status, number of honors in core clerkships, and US News and World Report medical school research ranking. We could not study whether a graduate passed or failed the American Board of Orthopaedic Surgery exam, as all assessed residents passed on their first attempt. Of these factors, only “number of honors in core clerkships” was eligible for inclusion for further analysis based on a cutoff of p < 0.2.

Our secondary study goals were to identify whether letters of recommendation, research activity, and extracurriculars available on ERAS applications are associated with residency performance. To achieve these goals, we performed a univariate analysis of the following factors against overall residency performance score: number of exceptional letters of recommendation, number of total publications and first author publications prior to residency, number of research years prior to residency, research (6-year) track residents, participation in collegiate sports, expertise with a musical instrument, years spent working in a nonmedical field, and number of volunteer experiences. Of these factors, only “number of exceptional letters of recommendation” and “research (6-year) track resident” were eligible for inclusion for further analysis based on a cutoff of p < 0.2.

Ethical Approval

We obtained ethical review board approval for this study.

Statistical Analysis

Application factors meeting our univariate cutoff of p < 0.2 were included in a stepwise multivariate linear regression model to identify factors independently associated with improved overall residency performance. We assessed clerkship grades and letters of recommendation in a continuous and categorical manner (for example, number of honors grades versus honors in medicine and surgery versus honors in five or more core clerkships). In these scenarios, we used the factor with the smallest p value in the multivariate model. Because of the retrospective nature of our study, the number of years since graduation was included as a factor in the stepwise regression model to mitigate potential recency bias from faculty reviewers. We used Cronbach alpha testing to assess interrater reliability for the 10 faculty members on each of the six residency performance criteria and the overall residency performance score. Significance was defined as p < 0.05. The statistical analysis was performed using IBM SPSS, version 27 (IBM Corp). Study data were managed using REDCap electronic data capture tools.

Results

The mean overall residency performance score was 7.9 ± 1.2 (Fig. 1). Our final multivariate model contained three factors: five or more honors in core clerkships, at least three exceptional letters of recommendation, and research (6-year) track resident. This model accounts for 53% of the variance in overall residency performance score observed in this study.

Academic Achievements

USMLE Step 1 and Step 2 scores had no correlation with residency performance (Fig. 2). In contrast, applicants who earned at least five honors grades in core clerkships had a 1.2-point improvement in the overall residency performance score (p = 0.01) (Table 3). Thirty-three percent (8 of 24) of residents met this criterion. The Cohen ƒ² was 0.2 for this application factor, indicating a small effect size.

Fig. 2.

A-B These graphs show the correlation between overall residency performance and (A) USMLE Step 1 and (B) USMLE Step 2 scores. No correlation was found between overall residency performance and score on either standardized test. Scores on the x-axis were omitted to avoid identifying individual residents, but all USMLE scores were above 230 on both examinations.

Table 3.

Factors independently associated with overall residency performance

Factor	Improvement in overall performance (1-10 scale)	95% CI	p value
≥ 5 honors in core clerkships	1.2	0.3-2.0	0.01
Research (6-year) track	1.0	0.1-1.9	0.03
≥ 3 exceptional letters of recommendation	0.9	0.02-1.7	0.046

Parameters excluded by stepwise regression: years since graduation. Model accounts for 53% of the variance in overall residency performance.

Letters of Recommendation

After controlling for variables like core clerkship grades and residency track, we found that residents with three or more exceptional letters of recommendation had slightly higher overall residency performance scores than those who did not (∆0.9 points; p = 0.046) (Table 3). Thirty-eight percent (9 of 24) of residents met this criterion. The Cohen ƒ² was 0.1 for this ERAS application factor, indicating a small effect size.

Research Activity

Applicants who participated in the residency research (6-year) track had a 1-point improvement in overall residency performance score (p = 0.03) (Table 3). Twenty-nine percent (7 of 24) of residents completed the research 6-year track. The Cohen ƒ² was 0.2 for this factor, indicating a small effect size in its association with overall residency performance.

Miscellaneous Factors

Medical school reputation, participation in collegiate sports, expertise with a musical instrument, years spent working in a nonmedical field, and number of volunteer experiences did not correlate with performance during residency.

Discussion

There are many more qualified residency applicants than available training spots, and the important factors for identifying the applicants most likely to be successful during residency remain unclear [1, 4, 7, 8, 11, 23, 31, 35]. This difficulty is not because of variability in what programs are looking for. The high interrater reliability observed between faculty in the current study indicate homogeneity in our institution’s definition of an outstanding resident, a finding consistent with studies from other institutions [7, 8]. Our data indicate that three readily available ERAS application factors—at least five honors in core clerkships, three or more exceptional letters of recommendation, and research (6-year) track residents—are associated with improved performance as an orthopaedic surgeon in training (Table 3). Although their effect sizes are individually small, applicants meeting multiple criteria appear to perform notably better than their counterparts; taken together, those three factors accounted for over half of the variance in performance that we observed.

Limitations

A major criticism of the current study is its unvalidated, subjective endpoint of overall residency performance. It should be noted that, in the absence of universally agreed upon parameters that denote a successful orthopaedic surgery residency graduate, it is difficult to refine a methodology that all parties will agree with. Our study endpoint was defined considering these constraints.

At first glance, including a subjective “personality” evaluation as a component of overall residency performance may seem particularly problematic. For example, evaluators might give higher personality scores to residents who share their race, gender, interests, and experiences. Although the aforementioned traits are irrelevant to orthopaedic residency performance [29, 30], characteristics such as grit, self-control, and conscientiousness are valued enough that a sizeable evidence base from more than a dozen centers has been published with the goal of identifying these personality traits in potential orthopaedic residents [3, 15, 18, 19, 34]. Our faculty evaluators included those of different races, genders, and ages. However, we still found consistency in desired resident personality traits among these evaluators, evidenced by an interrater reliability of 0.91. This reliability was slightly higher than that of surgical skills or clinical knowledge.

Similarly, our inclusion of “research productivity” as a component of resident performance could be seen as problematic. Many residents have little desire to publish, and most graduates do not do so [20]. Despite this, research activity remains one of the major variables considered by most orthopaedic training programs, both community and academic, in their evaluation of applicants [12, 33]. Applicants with extensive research backgrounds are more heavily recruited with the hope and expectation that they will remain productive researchers in residency [39]. Research output directly affects faculty member promotion at academic institutions and potentiates their involvement in national orthopaedic societies. Moreover, research productivity is a main differentiator for resident applicants when applying to fellowship programs [5, 28]. The pressure to publish research affects enough applicants, residents, and faculty that its exclusion as a criterion for residency performance would weaken our study.

Additionally, the small number of residents evaluated (n = 24) in this study is a common limitation in other studies because of the small number of orthopaedic residents training nationally and the extended duration of training [7, 8, 31]. However, our three factors identified in the multivariate analysis indicate that our study was adequately powered to achieve our goal of identifying ERAS application factors influencing residency performance. Our time range was chosen so all faculty graders had extensive experience working with each of the residents. Because this was a single-center study, our results may not be fully applicable to programs dissimilar from ours, which is a relatively large, academic orthopaedic residency program covering two urban Level I trauma centers.

Finally, only residency performance was evaluated in the current study rather than fellowship performance or quality of future orthopaedic practice. It was not possible to accurately determine performance of the studied residents after residency due to the variability in faculty evaluators across institutions. Similarly, since all residents passed their board exams on their first try, we were unable to evaluate board passage as a study endpoint.

Academic Achievements

USMLE Step 1 scores historically were a critical resident selection criterion, with most programs having a minimum USMLE Step 1 score as a prerequisite for an interview [35]. Although good performance on USMLE Step 1 is associated with American Board of Orthopaedic Surgery passage rates in some studies [1, 9, 31, 35, 37], it has never been correlated with residency clinical performance, a finding consistent with our study. Notably, all our residents had USMLE Step 1 scores substantially above the mean. As such, there may be a minimal score needed for excellent performance that could not be detected with our dataset. With USMLE Step 1 recently becoming pass/fail, it is reasonable to expect that USMLE Step 2 may substitute as the imperfect screening tool used to reduce application burden. Although the use of USMLE Step 2 as a screening tool may increase until another means of application reduction is reached, our results caution against using high standardized scores as a proxy for successful future residents.

Our work supports prior studies that found strong correlations between numbers of honors grades from third-year medical school clerkships and subsequent residency performance [7, 8, 31]. Although different medical schools have variable grading methods, obtaining a grade of honors typically requires rapid integration into a medical team, appropriate social interactions, and noteworthy improvement in clinical knowledge over the course of 1 to 2 months. In our study, obtaining at least five honors in core clerkships was associated with a 1.2-point increase in residency performance. Although a 1.2-point change on a 1 to 10 scale is small, it may have clinical relevance. This is because our range of overall residency performance scores in the current study was only 4.5 points (range 5.1 to 9.6). Similarly, other factors that would be expected to be associated with residency performance (such as, USMLE scores, research productivity, AOA status) did not have even the small effect observed here. Finally, this factor is one of three identified in the current study that, together, are responsible for 53% of the observed variation in overall residency performance. This ERAS application factor is easily identified and relatively objective. It seems to be a reasonable tool for identifying successful future orthopaedic residents.

Letters of Recommendation

After assessment of confounding variables such as AOA status, USMLE Step 1 and Step 2 scores, clinical grades, research before residency, extracurriculars, collegiate athletics participation, and years since graduation from our orthopaedic residency, we found that applicants with three or more exceptional letters of recommendation had slightly higher (1 point out of 10) residency performance scores. Although programs still rely heavily on letters of recommendation and personal phone calls recommending candidates for residency [1, 7], to our knowledge, no other study has found an association between exceptional letters of recommendation and residency performance. Notably, narrative orthopaedic letters of recommendation were shown to do a poor job of communicating the letter writer’s intended strength of recommendation [10]. Similarly problematic, the standardized orthopaedic letter of recommendation suffers from massive “grade inflation,” as more than one-third of applicants are noted to be in the top 10% of the applicant pool [27]. This a finding replicated in other specialties [13, 16, 17, 38]. We consider letters of recommendation to be an imperfect tool of applicant evaluation. However, our results suggest key phrase identification across several letters of recommendation may be effective in identifying applicants who have consensus approval from all faculty members they worked with. Our reliance on the information gleaned from multiple letters in aggregate may explain the difference in our results and those of prior studies. As datapoints for applicant evaluation are reduced, appropriately using the information available in letters of recommendation will be critical for successful orthopaedic applicant selection.

Research Activity

During the past 10 years, the focus on research in the orthopaedic application process has drastically increased. In 2011, matched US senior medical students had an mean of 4.5 research items (abstracts, presentations, and publications) on their application [25]. This number more than tripled in 2020, with matched US senior medical students having a mean of 14.3 research items on their ERAS applications [26]. Increasing numbers of medical students are also completing research years to improve their chances of a successful match [39]. In our study, the number of publications before residency had no association with residency performance, a finding supported by other studies [7, 36]. Our results suggest that the level of commitment to research prior to residency is not reliably associated with the quality of residency performance.

Miscellaneous Factors

ERAS application factors such as participation in an undergraduate varsity sport, playing a musical instrument, medical school reputation, work experience, and volunteer experience are often discussed during the assessment of orthopaedic applicants, with unclear implications [11]. In the current study, none of these factors was associated with orthopaedic residency performance. Studies on their association with residency performance have been mixed; although Dirschl et al. [8] found that participation in activities involving fine motor skills (such as playing a musical instrument) correlated with intraoperative psychomotor skills, Spitzer et al. [36] found no association. Similarly, activities involving gross motor skills, such as athletics and carpentry, are not associated with intraoperative performance [8]. Overall, our results and others indicate that these nonorthopaedic components of the ERAS application have limited utility in identifying candidates who will become successful orthopaedic residents.

Conclusion

Past clinical excellence, measured by core clerkship grades and exceptional letters of recommendation, are each associated with a small improvement in overall orthopaedic residency performance scores. When these factors are combined with completing a research (6-year) track residency, they are associated with a substantial improvement in residency performance, accounting for half of the variance observed in the current study. In comparison, traditionally used screening factors such as USMLE scores, AOA membership, medical school reputation, and preresidency research activity may have less utility in identifying successful future orthopaedic surgery residents.