Did the COVID-19 Pandemic Impact the Caliber of Trainees Taken During Match 2021? A Survey of General Surgery Residency Program Directors

Abstract

Background

The cancellation of clinical rotations (CRs) and implementation of virtual interviews (VIs) profoundly affected the residency selection process leading up to the 2021 NRMP Match. The authors investigated how these changes influenced the caliber of applicants taken by general surgery (GS) residency programs from the perspectives of program directors (PDs).

Methods

A 14 question, web-based electronic survey was emailed to PDs of ACGME-accredited GS residency programs. Questions sought program characteristics and PDs’ perspectives regarding potential differences in subjective characteristics and clinical skills demonstrated by their 2021 Match class relative to previous resident classes.

Results

A total of 75 PDs (27.2%) responded to our survey. Most respondents observed no changes in residents’ fit with their program (72.0%), communication skills (68.0%), responsiveness to clinical instruction and feedback (73.3%), work ethic (73.3%), and rotation evaluations (68.0%). Only 21.3% of PDs believed that VIs negatively impacted their ability to accurately assess applicant intangibles. Conversely, 56.0% of PDs reported that the cancellation of CRs in 2020 negatively affected residents’ clinical competency at the start of residency. At 1-year following the 2021 NRMP Match, 30.7% of PDs reported that the clinical skills exhibited by their 2021 Match class were poorer than previous resident classes.

Discussion

Our findings suggest that VIs limited selection committees’ ability to accurately assess applicant’s subjective characteristics to a lesser degree than previously described in the literature. Canceled CRs adversely affected the 2021 Match Class’s clinical skills at the start of residency and at 1 year following the 2021 NRMP Match.

Key Takeaways

• • The majority of program directors reported no changes in their 2021 Match class’s subjective characteristics relative to previous resident classes.

• • Virtual interviews limited selection committees’ ability to accurately assess applicant’s subjective characteristics to a lesser degree than previously described in the literature.

• • The cancellation of clinical rotations in 2020 negatively affected the clinical skills exhibited by the 2021 Match class at the start of their training and at 1-year following the 2021 NRMP Match.

Introduction

The residency selection process is a high-stakes endeavor with lasting implications for general surgery (GS) residency programs and applicants alike. This long-standing practice, however, was greatly affected by the COVID-19 pandemic. The most significant change to the residency selection process during the 2020-2021 application cycle was the introduction of virtual interviews (VIs).¹ While VIs reduced interview-related costs for both applicants and residency programs, many program directors (PDs) raised concerns regarding their ability to measure applicant intangibles during their interview.^2-7 GS residency programs were also tasked with choosing from an applicant pool that demonstrated significant heterogeneity in their clinical exposure to surgery resulting from the cancellation of clinical rotations (CRs) in 2020.⁸ Despite these changes, 1564 out of 1569 (99.7%) of categorical GS training positions were filled at the conclusion of the 2021 NRMP Match.⁹

The 2021 NRMP Match was a success for GS residency programs from the standpoint of filling training positions; however, the aforementioned changes to the 2020-2021 application cycle raised several questions regarding the quality of applicants who secured these positions. To date, no studies have measured the outcomes of the 2021 NRMP Match in terms of the clinical skills and subjective characteristics (such as personality and work ethic) of the 2021 Match class. Moreover, the absence of this data has limited the analysis of the true impact that the cancellation of CRs had on medical students’ ability to develop clinical skills prior to beginning residency, and the utility of VIs for evaluating applicant intangibles during residency interviews. In this study, we seek to elucidate the impact that the cancellation of CRs and implementation of VIs had on the caliber of residents taken in the 2021 NRMP Match from the perspectives of PDs of American Council for Graduate Medical Education (ACGME)-accredited GS residency programs.

Methods

Data Source

A web-based, electronic survey was designed using REDCap© (Vanderbilt University, Nashville, TN) and piloted prior to its distribution. The survey pilot entailed multiple cycles of sending our survey to academic surgeons across multiple subspecialties, obtaining feedback to ensure questions were being answered as intended, and modifying the survey to address areas that may confuse the respondent. Surveys were sent via email to PDs of ACGME-accredited GS residency programs during the data collection period (3/10/2022 - 5/4/2022). PDs of GS residency programs with an accreditation status other than those listed under the “continued” category (such as the “initial accreditation,” “withhold and probation,” or “withdrawal” categories) were excluded from our study. Additionally, PDs with email addresses that could not be identified and those with email addresses that blocked our recruitment email were excluded. Participation in our cross-sectional study was voluntary, and participants were able to withdraw from the study at any time. Responses were collected in a deidentified fashion and were analyzed at the conclusion of the collection period. Incomplete responses were not included in our analysis. This study was approved by the Institutional Review Board at Baylor College of Medicine.

Measurements

Our survey included 14 questions that were presented in multiple choice, true/false, and Likert scale formats (Supplemental file 1). Questions included in the survey sought program characteristics, the number of training positions offered during the 2021 NRMP Match, and the number of home students taken in the 2021 Match relative to prior application cycles (only programs that offered > 1 training position and are affiliated with a medical school). Program geographic location was classified using The Census Regions and Divisions of the United States Classification System.¹⁰ Several questions on our survey were designed to collect PDs’ perspectives regarding potential differences in the subjective characteristics and clinical skills demonstrated by their 2021 Match class (pandemic resident cohort) and previous resident classes (pre-pandemic resident cohort). Specific to the pandemic resident cohort, PDs were asked to provide their perspectives regarding the relationship between the 2021 Match class’s clinical competency at the start of residency and canceled CRs. Additionally, PDs were asked about their outlooks on the utility of VIs for evaluating applicant intangibles based on their experiences with their 2021 Match class. Lastly, participants were provided an optional free-response text box to add comments that they believed would be beneficial to our study.

Results

A total of 75 PDs of ACGME-accredited GS residency programs responded to our survey (27.2% response rate). Forty-seven (62.7%) PDs represented an academic residency program with the majority of programs residing in the South (34.7%), Northwest (28.0%), and Midwest (26.7%) geographic regions (Table 1). GS residency programs represented in our sample offered a median of 6 [Interquartile Range (IQR), 4-8] categorical training positions in the 2021 NRMP Match. Forty-four (58.7%) GS residency programs in our sample were affiliated with a medical school and offered more than 1 training position during the 2020-2021 application cycle; with 79.5% taking approximately the same number of home medical students in the 2021 NRMP Match.

Table 1.

Program Characteristics.

Characteristic	n (%/IQR)
Type of program
Academic	47 (62.7)
Community-based	28 (37.3)
Program location
Midwest	20 (26.7)
Northeast	21 (28.0)
South	26 (34.7)
West	8 (10.7)
Median number of positions offered	6 (4-8)
Number of home students taken during 2020-2021 cycle when compared to previous cyclesa (n = 44)
More	5 (11.4)
Less	4 (9.1)
Approximately equal	35 (79.5)

Abbreviations: IQR, interquartile range.

^aOnly academic programs offering >1 training position and are located at an institution with a medical school.

When comparing subjective characteristics of individuals belonging to the pre-pandemic and pandemic resident cohorts, most respondents observed no differences in residents’ fit with their program (72.0%), communication skills (68.0%), responsiveness to clinical instruction and feedback (73.3%), initiative and work ethic (73.3%), and end-of-rotation evaluations (68.0%) (Figure 1). The number of residents who required disciplinary action or left their program (both voluntarily and involuntarily) were largely similar between both resident cohorts with only 8.0% and 5.3% of PDs observing increases in each respective outcome in their 2021 Match class (Table 2). Only 21.3% of PDs believed that VIs negatively impacted their selection committee’s ability to accurately assess applicant intangibles during the residency interview (Figure 2).

Figure 1.

Clinical and subjective characteristics exhibited by residents belonging to the 2021 match class when compared to previous resident classes.

Table 2.

Program Directors’ Observations Regarding Resident Conduct and Attrition.

In Comparison to Previous Resident Classes, Residents Matched during the 2020-2021 Application Cycle...	True (%)	False (%)
Received more complaints requiring disciplinary action.	6 (8.0)	69 (92.0)
More residents have been dismissed or have left your residency program.	4 (5.3)	71 (94.7)

Figure 2.

Program directors’ responses regarding the utility of virtual interviews for evaluating applicant subjective characteristics (left) and the impact of canceled clinical rotations on the 2021 match Class’s clinical competency at the start of residency (right).

Fifty-six percent of PDs reported that the cancellation of CRs (both home and away) in 2020 negatively affected their 2021 Match class’s clinical competency at the start of residency (Figure 2). When comparing the clinical and technical skills exhibited by individuals belonging to the pre-pandemic and pandemic resident cohorts 1 year after the 2021 NRMP Match, 30.7% of PDs reported that their 2021 Match class exhibited poorer clinical skills relative to previous resident classes (Figure 1). A list of all comments provided by responding PDs can be found in Supplemental file 2.

Discussion

In this cross-sectional study of 75 PDs representing ACGME-accredited GS residency programs, we observed that most PDs believed that their 2021 Match class exhibited similar subjective characteristics but worsened clinical skills when compared to previous resident classes. Additionally, most PDs reported that VIs did not limit the ability of their program to evaluate applicant intangibles during the interview and that the cancellation of CRs in 2020 negatively affected the 2021 Match class’s clinical competency at the start of their training.

Collectively, GS residency programs primarily employ quantifiable metrics, such as research output and USMLE score reports, when selecting candidates to interview.¹¹ After applicants are selected for the interview, subjective metrics, such as personality and communication skills, serve as the primary variables used by selection committees when drafting their rank list for the NRMP Match.¹¹ With the cancellation of away rotations during the 2020-2021 academic calendar year, GS residency programs were limited to evaluating applicants’ subjective metrics exclusively during VIs and virtual pre-interview socials. At the conclusion of the 2020-2021 application cycle, the majority of PDs of surgical residency programs expressed concerns regarding their ability to measure applicant intangibles when hosting VIs.^2-7 Specific to GS residency programs, we had previously found that PDs believed that VIs limited their program’s ability elucidate applicants’ fit with their residency program (75%), personality and communication skills (69%), and commitment to GS (57%) when compared to in-person interviews.² When analyzing these parameters 1 year after the 2021 NRMP Match, our findings suggest that VIs are a viable alternative to in-person interviews for both accurately evaluating applicant intangibles and successfully matching applicants with desirable subjective characteristics. This is highlighted in one PD’s comment:

“… we used the same criteria (when evaluating applicants), and we have always intensely involved our current residents regarding fit into our program/system. Our best process to do this was a "meet and greet" the night before hosted by the residents without any faculty. (Residents) could still discern who fit and who did not. I do not think there is any reason to return to in-person interviews given the results we have seen…”

Of note, recent recommendations issued by the Association of American Medical Colleges suggest that VIs will likely remain the norm for residency interviews moving forward. Taking this into account, the authors surmise that refinement of the VI format over time, along with the removal of away rotation restrictions, will result in more programs being able to accurately gauge applicants’ intangibles in future application cycles.

Leading up to the 2020-2021 residency application cycle, approximately 24% of clinical-level medical students in the United States believed that the cancellation of CRs negatively affected their ability to choose a specialty prior to submitting their residency application.¹² Many residency programs were concerned about how this decreased clinical exposure would affect resident attrition rates for their 2021 Match class. Interestingly, only 5.3% of PDs observed an increase in the number of candidates who voluntarily withdrew or were dismissed from their residency program. At face value, our findings suggest that the 2021 Match class’s exposure to surgery during medical school, while limited, allowed them to make an informed decision when choosing their specialty. More likely, however, is that our findings were skewed given the high baseline attrition rate of GS residents under normal circumstances. GS residency programs have resident attrition rates up to 18% annually, and responding PDs may have failed to identify the additional resident attrition resulting from pandemic-related decreases in exposure to surgery during medical school.^13,14 This rationale is supported by the anecdotal experiences of several GS PDs:

“I have become aware of more openings at the PGY-2 level. I am concerned that students did not get a good CR (in 2020) and thus did not know what they were getting into. The other thing that may be at play here is that they arrived at the program, (found that) it was not what they thought it would be, and decided to leave.”

“It is my opinion that applicants did not get adequate exposure surgery in many institutions due to COVID-19 restrictions/limitations. The results of this appears to be some applicants choosing the wrong profession.”

Presently, discussion of the COVID-19 pandemic’s impact on attrition rates of GS residents is limited to conjecture and represent a potential area for future research efforts.

At the time of CRs being canceled in March 2020, only 51%-55% of clinical-level medical students in the United States had completed a full-length, in-person surgery rotation at their institution.^12,15 Instead, medical students were limited to experiencing surgery through virtual didactics and learning modules. Much of the literature investigating the utility of the aforementioned learning tools strongly suggests that they lack the educational value normally provided by in-person CRs.^12,15,16 Hernandez et al. investigated the educational value of virtual surgical curriculum when compared to in-person surgery rotations by performing a cross-sectional study of medical students who participated in both a virtual surgery rotation and an abbreviated, in-person CR at a later date.¹⁵ The authors found that the in-person surgery rotation provided more educational value than the virtual rotation in terms of students learning how to manage patients in the perioperative setting, developing technical skills, and acquiring an understanding of operating room etiquette.¹⁵ More than half of PDs in our sample reported that the cancellation of CRs in 2020 negatively affected the clinical competency of their 2021 Match class at the start of their training. Our findings reinforce the notion that virtual alternatives to in-person CRs used during the COVID-19 pandemic were of limited educational value to medical students who intended to pursue surgical training. This was echoed in several comments provided by PDs:

“Several residents from cities that had long-term COVID-19 shutdowns came in severely behind in clinical and technical skills.”

“…Some of the new interns have more limited clinical skills which appeared to be due to less CRs during medical school…”

Of note, many of the virtual learning tools used following the cancellation of CRs were hastily created by medical schools, as such, proactive development and refinement of virtual surgical curriculum would likely provide a better educational experience should similar circumstances arise in the future.

While many PDs in our sample reported that the 2021 Match class arrived at their residency program with worsened clinical skills, there was significant heterogeneity in PDs’ outlooks regarding their clinical skills relative to previous resident classes 1 year following the 2021 NRMP Match. One possibility is that our findings reflect pandemic-related disparities in surgical training observed across institutions during the 2021-2022 academic calendar year. At the height of the pandemic, the quality of surgical training at healthcare facilities across the United States decreased greatly owing to profound decreases in surgical volume and the implementation of restrictive perioperative safety protocols.^17-20 While burdensome safety protocols have since been lifted, surgical volume has recovered unevenly across institutions based on their ability to respond to widespread shortages of surgical staff. It is also possible that our findings are the result of residency programs’ differing approaches to “catching up” residents during their first year of residency. Presently, there are no published articles in the literature detailing how GS residency programs addressed the clinical shortcomings exhibited by their 2021 Match class. While this was outside of the scope of our study, one PD in our sample stated that their residency program extended their intern bootcamp; however, owing to the anonymous nature of our survey, we could not follow-up with them regarding their bootcamp curriculum or its observed educational value.

Our study has several limitations. First and foremost, our study is susceptible to recall bias given that our survey did not objectively measure the clinical competency and subjective characteristics of individuals belonging to the pre-pandemic and pandemic resident cohorts. Second, our findings are susceptible to non-respondent bias owing to the cross-sectional design of our study. We do not believe this skewed our data given that our sample was composed of PDs belonging to programs that were proportionally represented across various characteristics. Third, while our study inquired about the number of home medical students that were taken in the 2021 NRMP Match, it did not inquire about the number of outside medical students that were taken in the 2021 NRMP Match who had previously pursued a research year or CR with the respective program. Lastly, the data published in this study reflects the perspectives of PDs 1 year following the NRMP Match, as such, future studies are necessary to account for changes in their outlooks as their 2021 Match class progresses through their training.

Conclusion

The COVID-19 pandemic has greatly influenced the results of the 2021 National Resident Matching Program Match for many ACGME-accredited general surgery residency programs. Most notably, the cancellation of CRs at the height of the pandemic negatively impacted the clinical competency of interns upon arriving to residency; however, deficits in residents’ clinical skills appeared to improve throughout their first year of training to a variable extent. Conversely, the majority of GS residency programs were able to match applicants with subjective characteristics comparable to previous resident classes. VIs appear to limit selection committees’ ability to accurately assess an applicant’s subjective characteristics to a lesser degree than previously described in the literature. While our study presents outcomes data for residents who were taken in the 2021 NRMP Match, future studies are needed to evaluate the professional trajectory of the 2021 Match class as they progress through their training.