Characteristics Associated with Successful Residency Match in General Surgery

Abstract

Objective:

To evaluate characteristics of matched and unmatched general surgery residency (GSR) applicants.

Background:

Given the recent change of the United States Medical Licensing Exam Step 1 grading to pass/fail, understanding the factors that influence GSR match success is integral to identifying potential interventions to improve match rates for diverse medical students.

Methods:

Retrospective review of GSR National Residency Matching Program (NRMP) applicant and Accreditation Council for Graduate Medical Education (ACGME) active resident data between 2011 and 2022. Data included application characteristics for United States (“US”) and “independent” applicants, factors cited by program directors in the interview and ranking process, paths pursued if applicants went unmatched, and racial/ethnic representation.

Results:

A total of 9149 US and 3985 independent applicants applied to GSR between 2011 and 2021. Matched versus unmatched applicants had higher step 1 scores (US: 236 vs 218, P = 0.005; independent: 237 vs 228, P = 0.001), higher step 2 scores (US: 248 vs 232, P = 0.006; independent: 245 vs 234, P < 0.001), more likely to belong to alpha omega alpha (US: 17.1% vs 1.6%, P = 0.002) or to attend a top 40 National Institutes of Health-funded school (US: 31.0% vs 19.4%, P = 0.002) compared to unmatched applicants. Program directors heavily factored step 1 and step 2 scores, letters of recommendation, interactions with faculty and trainees, and interpersonal skills when interviewing and ranking applicants. The proportion of active general surgery residents versus applicants was lower for Asians (12.3% vs 20.9%, P < 0.001), Black/African American (5.0% vs 8.8%, P < 0.001), Hispanic/Latino (5.0% vs 9.4%, P = 0.001), and underrepresented in medicine students (10.3% vs 19.1%, P < 0.001).

Conclusions:

In the pass/fail step 1 era, factors including step 2 score and other subjective metrics may be more heavily weighted in the GSR match process.

INTRODUCTION

The Association of American Medical Colleges (AAMC) has identified promoting a racially and ethnically diverse physician workforce as a key approach to alleviating health inequities in the United States (US).¹ This is in part due to the greater likelihood of minoritized physicians treating minority patients, practicing in underserved areas, and the benefits both culturally sensitive and racially concordant care have on patient-physician relationships and health outcomes.^2,3 Nonetheless, minority surgeons are direly underrepresented; 2% of surgeons are Black, 5% are Hispanic/Latino, and 0.1% are American Indian/Alaska Native compared to 13.6%, 18.9%, and 1.3% of the US population, respectively.^4,5

As the US becomes increasingly racially and ethnically diverse, promoting a diverse surgical workforce is essential to providing culturally competent care. General surgery residencies (GSR) have historically emphasized academic characteristics such as United States Medical Licensing Exam (USMLE) scores, clerkship grades, and alpha omega alpha (AOA) membership as surrogates for future residency success, although these vary based on race/ethnicity, socioeconomic status, and training program.^6–10 In 2020, 90% of residency programs used USMLE step 1 scores as an initial screening tool for applicants.^11–14 When using the mean step 1 score in 2014 as a theoretical cutoff, 74.8% of Hispanic applicants and 84.3% of Black applicants would have been filtered out of the application pool.¹⁴ Thus, the overt reliance on academic metrics is known to lead to bias and limit diversity in residency programs, underscoring the need for a more comprehensive, inclusive approach to evaluating applicants’ competency.

The change in reporting of USMLE step 1 scores hopes to prompt residency programs to adopt a more holistic review of applicants to evaluate students’ clinical aptitude and suitability. A 2021 survey of GSR program directors identified step 2 score, personal contacts (mentors or academic acquaintances), medical school prestige, and sub-internship performance as metrics that would be more heavily weighted following the scoring change.⁷ Congruent with previous work, USMLE step 2 score, honors in clerkships, AOA membership, and number of publications are significant predictors of a successful GSR match.^6,9 Despite 70.9% of program directors disagreeing with the pass/fail scoring change, more than 40% agreed that a holistic review would decrease socioeconomic disparities and promote diversity and inclusion in their programs.⁷

A more comprehensive understanding of factors influencing successfully matching into a GSR is necessary to identify potential interventions to improve diversity, equity, and inclusivity in the selection process. We aimed to evaluate academic and demographic characteristics associated with matching into GSR and assess what paths unmatched applicants pursue.

METHODS

Data Source

This is a retrospective review of general surgery (GS) applicants and residents from the 2011–2012 to 2021–2022 academic years. Data were obtained from the National Residency Matching Program (NRMP) Applicant and Program Director Surveys, the NRMP Charting Outcomes from the Match reports for GSR, and the Accreditation Council for Graduate Medical Education (ACGME) Data Resource Books over the study period. This study was deemed exempt from institutional review board approval.

The NRMP Applicant Survey and Charting Outcomes in the Match reports were accessed for data on matched and unmatched applicants who selected a categorical GSR as their “preferred specialty” in the match. Data from each reporting year were recorded and aggregated for the analysis. Characteristics obtained from the charting outcomes in the match reports included research experience, publications, work or volunteer experience, AOA membership, top 40 National Institutes of Health-funded schools, PhD or graduate degree, and USMLE step 1 and 2 scores. Number of applications submitted, interview offers, interviews attended, programs ranked, and paths pursued by unmatched applicants were obtained from the NRMP Applicant Survey. The likelihood of an unmatched applicant pursuing various paths was reported on a 5-point Likert scale, where 5 corresponded to “extremely likely” and 1 to “not at all likely”. Likelihood scores for each path were recorded and averaged over the study period.

NRMP Program Director Surveys were assessed for top factors cited for interviewing and ranking an applicant. For the 2012–2013 academic year, factors were reported on a 5-point Likert scale and excluded from the analysis. Percentages of program directors citing each factor were recorded and averaged over the study period.

Self-reported race/ethnicity data of GS applicants and residents were obtained from the ACGME data resource book over the study period. The underrepresented in medicine (URM) category is based on AAMC designation and includes students who self-identify as American Indian, Black/African American, Hispanic/Latino, or Native Hawaiian/Pacific Islander. Applicants from Doctor of Medicine and Doctor of Osteopathic Medicine medical schools, including US and international medical graduates, were included in the analysis.

Based on reporting from the NRMP, “US applicants” included seniors from US allopathic medical schools. “Independent” applicants included allopathic medical school graduates, US citizen and non-US citizen students, graduates of international, osteopathic, and Canadian medical schools, and graduates of Fifth Pathway programs. Starting in 2020, application and interview reporting metrics for Independent applicants were sub-grouped into “DO” or “Other” for ease of comparing data before and after COVID-19 prompted the change to the virtual interview format. The NRMP applicant data clearly differentiates US versus independent applicants, however, data on race/ethnicity is only available as an aggregate for all applicants and all active residents and thus is reported as such.

Data Analysis

Data analysis was conducted using R version 4.2.2 with a statistically significant P value < 0.05. Continuous and categorical variables were summarized as median (interquartile range) and n (percentage), respectively. Matched and unmatched US and independent applicant characteristics were compared using the Wilcoxon rank sum test. The AAMC and NRMP do not publicly report the race/ethnicity of matched and unmatched applicants, thus, differences in race/ethnicity proportions were assessed using the Wilcoxon rank sum test using data from GS applicants and residents.^15,16 Trends in race/ethnicity proportions of GS applicants and residents were assessed using the Mann-Kendall trend test.

RESULTS

Academic Characteristics

During the study period, a total of 9149 US and 3985 independent applicants applied to GSR positions with a 90.1% and 44.9% match rate, respectively. For both US and independent applicants, matched applicants received more interview offers (US: 18.0 interview offers vs 8.0, P = 0.005; independent: 8.0 vs 3.0, P = 0.009), attended more interviews (US: 14.0 interviews attended vs 8.0, P = 0.017; independent: 7.0 vs 3.0, P = 0.009), and ranked more programs (US: 14.0 programs ranked vs 8.0, P = 0.020; independent: 6.2 vs 2.7, P < 0.001) as compared to unmatched applicants (Table 1). Matched applicants had significantly higher step 1 (US: 236 vs 218, P = 0.005; independent: 237 vs 228, P = 0.001) and step 2 (US: 248 vs 232, P = 0.006; independent: 245 vs 234, P < 0.001) scores than unmatched applicants. There was no difference in match status when evaluating the number of applications, research experience, publications, work or volunteer experience, PhD or graduate degree. Matched US applicants were more likely to belong to AOA (17.1% vs 1.6%, P = 0.002) or attend a top 40 National Institutes of Health-funded school (31.0% vs 19.4%, P = 0.002).

TABLE 1.

Characteristics of Matched and Unmatched General Surgery applicants from 2011 to 2021

Characteristic	US Applicants (N = 6212)†			Independent Applicants (N = 3947)†,‡
	Matched (N = 5201)	Unmatched (N = 1016)	P Value	Matched (N = 1426)	Unmatched (N = 2521)	P Value
Applications§	48.0 (40.0, 54.0)	60.0 (52.0, 60.0)	0.177	77.0 (75.0, 84.0)	80.0 (70.0, 100.0)	0.675
Interview offers§	18.0 (17.0, 18.5)	8.0 (7.5, 12.5)	*0.005	8.0 (4.0, 9.0)	3.0 (3.0, 3.0)	*0.009
Interviews attended§	14.0 (13.0, 15.0)	8.0 (7.0, 10.0)	*0.017	7.0 (4.0, 8.0)	3.0 (3.0, 3.0)	*0.009
Programs ranked§	14.0 (12.5, 15.0)	8.0 (7, 10.5)	*0.020	6.2 (4.3, 7.8)	2.7 (2.4, 3.6)	*<0.001
Research experience	3.4 (3.1, 3.9)	2.9 (2.6, 3.3)	0.260	2.8 (2.2, 3.2)	2.7 (2.1, 2.9)	0.368
Publications∥	5.5 (4.5, 6.9)	4.0 (2.9, 4.8)	0.172	5.0 (4.1, 10.5)	5.1 (3.6, 8.6)	0.581
Work experience	3.3 (3.1, 3.5)	3.4 (3.2, 3.6)	0.810	4.3 (3.6, 4.8)	4.2 (3.9, 4.8)	0.765
Volunteer experience	6.9 (6.7, 8.0)	6.6 (6.2, 7.5)	0.240	4.9 (4.3, 5.8)	4.4 (4.0, 5.1)	0.312
AOA, %	17.1 (2.5)	1.6 (0.5)	*0.002	—	—	—
Top 40 NIH-funded school, %	31.0 (1.8)	19.4 (3.7)	*0.002	—	—	—
PhD degree	1.9 (0.3)	2.6 (0.9)	0.229	1.3 (1.8)	1.1 (1.4)	0.777
Graduate degree	18.6 (4.2)	20.3 (3.6)	0.423	23.1 (5.0)	27.0 (4.3)	0.094
USMLE score
Step 1	236 (233, 237)	218 (214, 219)	*0.005	237 (232, 239)	228 (223, 232)	*0.001
Step 2	248 (246, 249)	232 (227, 234)	*0.006	245 (243, 246)	234 (232, 236)	*<0.001

^*Denotes statistical significance with P < 0.05.

^†N applicants who responded to the NRMP Applicant Survey.

^‡Independent includes allopathic medical school graduates, US citizen and non-US citizen students and graduates of international medical schools, students and graduates of osteopathy, students and graduates of Canadian medical schools, and graduates of Fifth Pathway programs.

^§Excludes Independent applicants from 2020–2021 to 2021–2022 given difference in reporting format.

^∥Includes abstracts/presentations/publications.

Application Changes During COVID-19 Pandemic

The change to virtual interviews in 2020 resulted in an overall increase in applications submitted for all applicants. Matched 2011–2019 US applicants submitted 34–51 applications compared to a median of 56 applications in 2020 and 62 in 2021. From 2011 to 2019, matched independent applicants submitted 49–90 applications compared to a median of 84 applications in 2020 and 84–90 in 2021, consequently resulting in an increased number of interviews completed after 2020. From 2011 to 2019, matched US applicants completed 12–14 interviews compared to 16 in both 2020 and 2021. Matched independent applicants completed 4–9 interviews from 2011 to 2019 compared to 4–10 interviews in 2020 and 5–13 in 2021.

Race/Ethnicity Characteristics

Assessing 2011–021 trends, there was a significant increase in the number of applicants who identified as White (39.2%–48.3%, tau = 0.71, P = 0.003) or Hispanic/Latino (8.6%–11.5%, tau = 0.75, P < 0.001). There was no difference in the proportion of applicants identifying as Black and a decrease in those identifying as Asian (23.1%–20.9%, tau = −0.53, P = 0.029) or other/unknown (21.1%–8.8%, tau = −0.89, P = 0.003). There was no difference in the race/ethnicity proportions of active GSRs over the same period. Comparing proportions of GS applicants to residents, there was a significant drop in representation among those who identified as Asian (20.9% of applicants vs 12.3% of active residents, P < 0.001), Black/African American (8.8% of applicants vs 5.0% of active residents, P < 0.001), or Hispanic/Latino (9.4% of applicants vs 5.0% of active residents, P = 0.001) (Table 2 and Figure 1). Combining Black/African American, Hispanic/Latino, American Indian, and Native Hawaiian/Pacific Islander to form the URM category, there was a similar drop in representation of active GSRs compared with applicants (19.1% of applicants vs 10.3% of active residents, P < 0.001). Comparatively, there was a significant increase in the other/unknown group (11.0% of applicants vs 30.1% of active residents, P = 0.001) and no difference among White representation (48.3% of applicants vs 46.9% of active residents, P = 0.478).

TABLE 2.

Race/ethnicity proportions of general surgery applicants and active general surgery residents, 2011–2021

Race/Ethnicity	GS Applicants	Active GS Residents	P Value
Asian	20.9 (20.2–24.0)	12.3 (12.0–13.4)	*<0.001
URM	19.1 (18.5–19.5)	10.3 (9.6–11.4)	*<0.001
Black/African American	8.8 (7.9–9.5)	5.0 (4.5–5.1)	*<0.001
Hispanic/Latino	9.4 (8.5–10.3)	5.0 (4.9–5.8)	*0.001
White	48.3 (42.6–50.1)	46.9 (46.1–50.6)	0.478
Other/unknown†	11.0 (10.2–12.9)	30.1 (24.4–26.9)	*0.001

^*Denotes statistical significance with P < 0.05; URM includes American Indian, Black/AA, Hispanic/Latino, Native Hawaiian/Pacific Islander.

^†Other/unknown includes other, unknown, non-US citizen/nonpermanent resident, and multiple race/ethnicity (starting in 2021).

FIGURE 1.

Race/ethnicity proportions of applicants and matriculants to categorical general surgery from 2011 to 2021.

Program Director Perspective

The distribution of factors cited by program directors for offering an interview and ranking an applicant can be visualized in Figure 2. Top cited measures for offering interviews include step 1 score (92.7%), letters of recommendation (LOR) (90.2%), and step 2 score (83.0%). Interactions with faculty during the interview/visit (92.5%), interpersonal skills (91.9%), and interactions with house staff during the interview/visit (86.8%) were the most cited factors for ranking an interviewee.

FIGURE 2.

Top factors cited by program directors for (A) offering an applicant an interview and (B) ranking an applicant.

Paths Pursued by the Unmatched

During the study period, 9.9% of US and 55.0% of independent applicants did not match into a GSR position. When assessed on a 5-point Likert scale, unmatched applicants from 2013 to 2019 were most likely to participate in the Supplemental Offer and Acceptance Program (SOAP) for a GSR position (4.51/5), SOAP for a preliminary position (4.01/5), or pursue research (3.03/5) and re-apply the next cycle (4.01/5).

DISCUSSION

In a retrospective review of GSR applicants and residents, higher USMLE scores, being granted more interviews, and ranking more programs were associated with successful match outcomes for both US and independent applicants. Program directors factored step scores, LORs, and interactions/interpersonal skills when interviewing and ranking an applicant. Concerningly, there was a decrease in the proportion of active non-White GSRs compared with applicants during the 10-year study period. Excluding USMLE scores, metrics used to rank students are highly subjective and susceptible to bias; thus, programs will need to be intentionally proactive to address biases in the application and interview process.

By comprehensively assessing all components of an individual’s application, a holistic approach to evaluating GSR applications is integral to diversifying the surgical workforce. Our study found that USMLE scores were significantly higher for matched than unmatched applicants. This is congruent with prior literature, as many programs still use USMLE step 1 and step 2 scores as initial screening thresholds, although there is limited data to suggest a correlation between scores and GSR success.^9,10,17,18 With the change of step 1 to pass/fail, results from a survey of program directors suggest that Step 2 may subsequently be more heavily factored.⁷ However, when programs rely solely on score thresholds to screen applicants, they may have unintended consequences on interviewing and ranking a diverse pool of applicants as scores can be influenced by environmental and nonacademic factors, including race/ethnicity, socioeconomic status, and medical school.^12,19–22 Other important metrics identified in this study for interviewing an applicant include LORs, personal statements, and clerkship grades. However, each of these are known to be highly subject to bias.^23–28 In LORs, standout adjectives (eg, “excellent”, “exceptional”) are more likely to be used when describing White and male applicants while grindstone adjectives (eg, “dedicated”, “hardworking”) are more likely to be used when describing URM and female applicants.^29,30 Similarly, prior work has demonstrated a lack of inter-rater reliability when evaluating the personal statement—a single statement could receive contradictory comments from separate reviewers.³¹ Finally, clerkship grades vary widely depending on medical school, region of the country, and race/ethnicity with the percentage of students achieving honors in their surgery clerkship ranging from 6.5% to 78% across the country, favoring students from the top 20 medical schools.^24,32,33 Prior work has also suggested that non-White race and self-identified disadvantaged backgrounds are associated with lower grades in all clerkships.^12,33 Thus, programs may be inadvertently limiting the diversity of their residency class through nonstandardized evaluation of LORs, personal statements, and overemphasizing clerkship grades. This is supported by our finding of a discrepancy in URM representation among active GS residents.

Students who receive and attend more interviews are more likely to successfully match. Furthermore, interactions with faculty and house staff during the interview process were top-factored measures when ranking an applicant. However, perceptions of interpersonal interactions are subjective and heavily influenced by perceived differences in culture and background that may negatively affect candidates of differing age, gender, sex, race/ethnicity, sexual identity, religion, geographic region, family background, and physical ability.³⁴ During the selection process, an applicant’s interpersonal skills are often referred to as an applicant’s “fit” with the program. However, “fit” is highly susceptible to implicit bias, particularly against applicants who may not share the same sex, ethnicity, religion, socioeconomic status, or sexual orientation as the individual conducting the evaluation.^35,36 Prior familiarity with applicants is correlated with application review and interview scores, suggesting more interpersonal bias and underscoring the importance of standardizing the interview process to minimize potential bias.³⁷ It is unclear where in the application process URM students may be lost, but biases in the interview and ranking process warrant further evaluation when considering our finding of a discrepancy in active URM GS residents.

While the shift away from scored USMLE examinations is intended to promote a more holistic review of GS applicants, actualizing this vision will come with challenges. Several initiatives show promise to achieving this goal.^38,39 The switch to a virtual format eliminated the financial burden associated with in-person interviews, likely contributing to the increase in the number of applications submitted and interviews attended observed in this study. Continuation of this may promote equity in the ability for individuals to attend interviews regardless of socioeconomic status.^40,41 Additional proposed interventions to facilitate holistic review include appointing diverse interview committees, deemphasizing USMLE scores and overall grades, emphasizing important program characteristics, and standardizing interview scoring.⁴² The challenge is each program receives thousands of applications for an average of 5 categorical spots every cycle, with some programs receiving over 150 applications per position.⁴³ Artificial intelligence may augment the holistic review of GSR applications, by leveraging large language models that can provide an initial summative evaluation of the range of data provided by applicants.⁴⁴ Furthermore, the implementation of program and regional preference signaling into the match process may be used to show applicants’ strong interest in particular programs. Recommendations from the 2023 Association of Program Director in Surgery to standardize this information include implementing interview release dates to prevent interview “hoarding,”⁴⁵ providing applicants with additional information before accepting an interview so they can focus on programs they are genuinely interested in, thus allowing interviews to be more evenly distributed across all applicants. The AAMC should also consider limiting the number of applications submitted and interviews attended to minimize the burden of holistically reviewing all applicants.^11,46

All data obtained from the NRMP and ACGME surveys are self-reported and limited to those who responded to the surveys, limiting the generalizability of the results. Current AAMC demographic data is limited to the number of students applying to each specialty and the total number of active residents in a specific specialty. Thus, our results are limited to a comparison of race/ethnicity proportions of applicants to active residents, resembling prior work.¹⁵ Based on the available data, URM students who applied to a GSR during the study period had a significantly lower proportion of active enrollment in a GSR program. Although this does not directly correspond to a lower proportion of URM students matching into a GS program, it is imperative to assess what happens to students lost along the pipeline. There was also an increase in the proportion of active GSR self-identifying as “other” race/ethnicity. While it is possible that this may explain some of the drop in minority representation, it is unlikely that people who self-identified as minorities would change to selecting “other.” Further, the drop in representation was consistent across all non-White racial groups and may remain true. Our findings also show that unmatched applicants are likely to participate in SOAP for a GSR, a preliminary position, and re-apply the next cycle or pursue research and re-apply the next cycle. However, there are no formal programs to mentor and develop students who go unmatched. Programs designed to help URM students navigate reapplying are important for supporting these students. Finally, this study only spans 2 years of data from the COVID-19 pandemic, and as such may not reflect current trends in the match. The shift toward virtual interviews will continue to have ramifications for years to come.

CONCLUSIONS

As residency programs transition to evaluating applications via pass/fail, subjective metrics (eg, LOR) may be more heavily weighted. Interactions with faculty, staff, and residents were important for ranking interviewees but may be subject to inherent bias; this is a cause for concern to diversify the surgical workforce. Structural barriers affecting match status warrant further evaluation. Future studies may focus on what characteristics predict success when applicants matriculate as GSRs.