Impact of the COVID‐19 Pandemic on the 2021 Otolaryngology Residency Match: Analysis of the Texas STAR Database

Abstract

Objectives/Hypothesis

To estimate the impact of the coronavirus disease 2019 (COVID‐19) pandemic on the 2021 otolaryngology match with regard to geographic clustering, interview distribution, applicant‐reported costs, and matched applicant characteristics.

Study Design

Retrospective cohort study.

Methods

Survey data from applicants to otolaryngology residency programs were obtained from the Texas Seeking Transparency in Applications to Residency database. Applicant differences between the 2021 match year and prior match years (2018, 2019, and 2020) were analyzed using two‐sided t‐tests, Chi‐square tests, and Fisher's exact tests.

Results

A total of 442 otolaryngology residency applicants responded to the survey, including 329 from the match years 2018 to 2020 and 113 from match year 2021. In 2021, 30.7% of responding applicants reported matching at a program where they had a geographic connection, compared to 40.0% in prior years (P = .139). Matched applicants in 2021 reported attending less interviews than applicants in prior years (mean 12.2 vs. 13.3, P = .040), and 26.1% of responding applicants reported matching at a program where they sent a preference signal. Applicants in the 2021 match reported significantly lower total costs than applicants in prior years (mean difference −$5,496, 95% confidence interval −$6,234 to −$4,759; P < .001). Compared to prior match years, matched applicants in 2021 had no meaningful differences in characteristics such as United States Medical Licensing Exam board scores, clerkship grades, honors society memberships, research output, volunteer experiences, or leadership experiences.

Conclusion

Based on this sample, there was no evidence of significant interview hoarding or increased geographic clustering in the 2021 otolaryngology match, and the COVID‐19 pandemic did not appear to result in significantly different matched applicant characteristics.

Level of Evidence

4 Laryngoscope, 132:1177–1183, 2022

INTRODUCTION

The coronavirus disease 2019 (COVID‐19) pandemic brought several unprecedented changes to the transition from medical school to residency, including delays in standardized testing, cancellation of in‐person away electives, virtual interviews, and shortened clerkships. ¹ There was significant uncertainty among both otolaryngology applicants and residency programs about how these changes would affect the residency selection process. ² , ³ Specifically, otolaryngology applicants and program directors were concerned about their ability to gather enough information to make informed decisions about their rank lists. ² , ³

Stakeholders in otolaryngology found creative ways to mitigate some of the disruptions caused by the COVID‐19 pandemic; these included advocating for holistic review of applications, ¹ , ⁴ the advent of virtual away rotations or subinternships, ⁵ and increased flexibility in number and type of letters of recommendation. ⁶ In addition to these adaptations, 2020 to 2021 was the first application cycle to implement preference signaling, a mechanism by which applicants formally indicate their highest‐interest program choices. ⁷

Given all of the changes in the otolaryngology residency application process during the 2020 to 2021 cycle, we sought to evaluate whether there were significant differences in matched applicant characteristics in 2021 compared to prior years. The results of this study could help inform future decisions about the otolaryngology residency selection process as it relates to virtual interviews, virtual subinternships, and preference signaling. We hypothesized that during the 2021 match year, otolaryngology applicants would attend more interviews and be more likely to match at a program with geographic ties compared to prior years.

MATERIALS AND METHODS

This study was approved by the Institutional Review Board at the University of North Carolina at Chapel Hill.

Sample Selection

Data were obtained from the Texas Seeking Transparency in Applications to Residency (STAR) database, which contains self‐reported information from residency applicants at U.S. medical schools during the 2018, 2019, 2020, and 2021 match years. ⁸ The Texas STAR survey was distributed by the dean of student affairs at participating medical schools, and it was available for students to complete between match day and April 10th of each application cycle. Applicants were included in this study if they applied to otolaryngology residency and completed the Texas STAR survey between 2018 and 2021. The overall response rate for all specialties was 46% in 2018, 41% in 2019, 46% in 2020, and 40% in 2021. The response rate for each year, calculated as the number of respondents at each medical school over the total number of graduating students at each medical school receiving the survey, was provided to study authors by the creators of the Texas STAR database. The preferred specialty makeup of nonrespondents was not known, so we were unable to calculate a response rate for otolaryngology applicants specifically.

Texas STAR Survey

The Texas STAR survey asked applicants to report information as it would have appeared on their residency applications. Data collected included United States Medical Licensing Exam (USMLE) board scores (reported within a five‐point range), clerkship honors, honors society memberships, second degrees, research years, research experiences, research output, volunteer experiences, and leadership experiences. Applicants were also asked to report the number of interviews attended, whether they successfully matched, if they had a geographic connection to the program at which they matched, and costs related to the application cycle. For the 2021 match year survey, otolaryngology applicants were also asked to report where they chose to send preference signals and whether they matched at a program where they sent a signal. Data on applicant demographics (age, sex, and race) and medical school were not collected in effort to protect confidentiality.

Statistical Analysis

Descriptive statistics were used to compare applicant characteristics in the 2021 match year compared to prior years. USMLE Step 1 and Step 2 Clinical Knowledge (CK) scores were centered for the analysis (e.g., a reported score of 220–224 was centered at 222). Bivariate testing methods included two‐sided t‐tests, Chi‐square tests, Fisher's exact test, and Wilcox rank‐sum test. A secondary analysis was performed for significant variables in the bivariate analysis to determine if the effect was unique to the 2021 match year or related to a broader trend. Correlations with preference signal yield (# of interviews at signaled programs/total # of signals sent) were calculated using the Pearson correlation coefficient. The significance criterion was set at P < .05 for all testing. Stata 16.0 (College Station, Texas) was used for all analyses.

RESULTS

A total of 442 otolaryngology applicants responding to the Texas STAR survey were included in this study. The breakdown of survey response by year was 81 in 2018, 105 in 2019, 143 in 2020, and 113 in 2021. In this sample, there were 386 applicants who matched (87.3%) and 56 who did not match (12.7%).

Geographic Connections and Preference Signaling

Applicants were asked to report if they had a geographic connection to the program at which they matched. There was no significant difference in percentage of applicants with a geographic connection to their matched program in the 2021 match year compared to prior years (30.7% vs. 40.0%; P = .139) (Fig. 1).

Fig 1.

Geographic connections of matched applicants.

In the 2021 match year, a total of 88 applicants reported data on preference signaling (not reported by 12.9% of respondents). On average, applicants sent 29.0% (standard deviation [SD] 31.6%) of signals to programs where they had a geographic connection, 25.7% (SD 27.3%) of signals to programs ranked in the top 20 for reputation on Doximity, and 15.4% (SD 20.0%) of signals to programs ranked in the top 10 for reputation on Doximity (Fig. 2).

Fig 2.

Where applicants sent signals.

The mean signal yield (# of interviews at signaled programs/total # of signals sent) was 0.61 (SD 0.25). There was no significant correlation between signal yield and percentage of signals an applicant sent to programs where they had a geographic connection (Pearson's R = 0.164; P = .127), percentage of signals sent to Doximity top 20 programs (Pearson's R = 0.024; P = .821), or percentage of signals sent to Doximity top 10 programs (Pearson's R = 0.015; P = .891). Among the 88 applicants reporting signaling data, 23 applicants (26.1%) matched at a program where they sent a signal.

Interview Distribution

Matched applicants from 2021 reported attending an average of 12.2 (SD 5.4) interviews, compared to an average of 13.3 (SD 4.3) interviews reported by matched applicants in prior match years (mean difference [MD] −1.10, 95% confidence interval [CI] −2.15 to −0.05; P = .040) (Table I). The number of interviews attended by matched applicants followed a relatively normal distribution in the 2021 match year as well as prior years, with no significant skew in either direction (Table I and Fig. 3). There was no significant difference in the number of interviews attended by unmatched applicants in 2021 compared to prior years (mean [SD], 9.4 [6.2] vs. 9.4 [6.1]; MD 0.05, 95% CI −3.93 to 4.04; P = .979) (Table I).

TABLE I.

Distribution of Interviews Attended by Applicants.

	2018–2020	2021	P‐Value
Matched (mean, standard deviation [SD])	13.3 (4.3)	12.2 (5.4)	.040
Skewness	−0.298	−0.020	n/a
Kurtosis	0.315	−0.410	n/a
Maximum reported	26	27	n/a
Unmatched (mean, SD)	9.4 (6.1)	9.4 (6.2)	.979
Skewness	0.693	0.278	n/a
Kurtosis	−0.168	−0.777	n/a
Maximum reported	24	20	n/a

Fig 3.

Histogram of interviews attended by matched applicants in 2021 versus 2018 to 2020. [Color figure can be viewed in the online issue, which is available at www.laryngoscope.com.]

Costs Related to Residency Application

Data on costs related to the residency application cycle were reported by 349 applicants, including 237 applicants from match years 2018 to 2020 and 112 from the 2021 match year. Applicants in the 2021 match year reported significantly lower total costs compared to applicants in prior years (mean [SD], $2,013 [$1,063] vs. $7,509 [$3,898]; MD −$5,496, 95% CI −$6,234 to −$4,759; P < .001).

When broken down by categories, applicants from the 2021 match year reported spending an average of $1,651 (SD $774) on application fees, $64 (SD $48) on virtual interviews, and $667 (SD $653) on other costs. Applicants from prior years reported spending an average of $1,612 (SD $707) on application fees (P = .642 vs. 2021), $3,536 (SD $2,520) on interviews (P < .001 vs. 2021), and $2,437 (SD $1,866) on other costs (P < .001 vs. 2021).

Applicant Characteristics

Differences in characteristics between matched applicants in 2021 compared to prior years were summarized (Table II). Matched applicants in 2021 reported a significantly higher number of abstracts, posters, or presentations (mean [SD], 8.0 [3.3] vs. 7.2 [3.5]; P = .040) and peer‐reviewed publications (mean [SD], 5.1 [3.3] vs. 4.3 [3.3]; P = .027) compared to matched applicants in 2018 to 2020. There were no significant differences in mean number of honored clerkships, honors in Otolaryngology clerkship, Alpha Omega Alpha or Gold Humanism Honor Society membership, USMLE Step 1 or Step 2 CK score, second degrees or research years, number of research experiences, number of volunteer experiences, or number of leadership experiences (Table II).

TABLE II.

Characteristics of Matched Applicants in 2021 Versus 2018–2020.

	2018–2020 (n = 285)	2021 (n = 101)	P‐Value
# of honored clerkships (mean, SD)	4.6 (2.3)	4.2 (2.3)	.101
Honors in otolaryngology clerkship	230 (96.6%)	78 (97.5%)	.520
AOA	125 (46.6%)	40 (44.0%)	.657
GHHS	50 (18.4%)	16 (17.4%)	.831
Step 1 score (mean, SD)	249.2 (11.3)	247.9 (11.6)	.328
Step 2 CK score (mean, SD)	256.6 (10.0)	256.5 (9.6)	.912
Second degree	56 (19.7%)	13 (12.9%)	.127
Research year	40 (14.0%)	19 (18.8%)	.252
Number of research experiences (mean, SD)	6.2 (2.8)	6.5 (2.4)	.351
Number of abstracts, posters, or presentations (mean, SD)	7.2 (3.5)	8.0 (3.3)	.040
Number of peer‐reviewed publications (mean, SD)	4.3 (3.3)	5.1 (3.3)	.027
Volunteer experiences (mean, SD)	7.3 (2.9)	7.5 (2.6)	.480
Leadership positions (mean, SD)	4.8 (2.8)	5.2 (2.8)	.275

AOA = alpha omega alpha; CK, clinical knowledge; GHHS = Gold Humanism Honor Society; SD = standard deviation.

Differences in characteristics between unmatched applicants in 2021 compared to prior years were also summarized (Table III). Unmatched applicants in 2021 reported a significantly higher number of peer‐reviewed publications compared to unmatched applicants in 2018 to 2020 (mean [SD], 6.4 [4.0] vs. 3.2 [3.1]; P = .008). There were no significant differences in any of the other applicant characteristics (Table III).

TABLE III.

Characteristics of Unmatched Applicants in 2021 Versus 2018–2020.

	2018–2020 (n = 44)	2021 (n = 12)	P‐Value
# of honored clerkships (mean, SD)	3.8 (2.4)	3.5 (2.6)	.670
Honors in otolaryngology clerkship	33 (89.2%)	8 (100.0%)	1.000
AOA	15 (35.7%)	7 (58.3%)	.194
GHHS	5 (11.4%)	1 (9.1%)	1.000
Step 1 score (mean, SD)	245.5 (12.6)	247.9 (12.0)	.574
Step 2 CK score (mean, SD)	252.9 (12.4)	253.8 (15.2)	.836
Second degree	10 (22.7%)	2 (16.7%)	1.000
Research year	9 (20.5%)	2 (16.7%)	1.000
Number of research experiences (mean, SD)	5.3 (2.6)	7.0 (4.2)	.083
Number of abstracts, posters, or presentations (mean, SD)	6.5 (4.0)	7.7 (4.5)	.381
Number of peer‐reviewed publications (mean, SD)	3.2 (3.1)	6.2 (4.0)	.008
Volunteer experiences (mean, SD)	7.3 (3.1)	7.2 (3.6)	.881
Leadership positions (mean, SD)	4.3 (2.7)	4.8 (3.3)	.664

AOA = alpha omega alpha; CK, clinical knowledge; GHHS = Gold Humanism Honor Society; SD = standard deviation.

Secondary analysis revealed that the significant findings in the bivariate analysis for matched applicants were driven by the incremental increases between 2018 and 2020 (Table IV). For unmatched applicants, there did appear to be a significant increase in peer‐reviewed publications unique to 2021 (P = .029).

TABLE IV.

Secondary Analysis for Significant Variables in Bivariate Analysis.

	2018	2019	2020	2021	P‐Value 2018 vs. 2019	P‐Value 2019 vs. 2020	P‐Value 2021 vs. 2020
Matched applicants
Number of abstracts, posters, or presentations	6.2 (3.3)	6.8 (3.5)	8.1 (3.4)	8.0 (3.3)	.285	.005	.832
Number of peer‐reviewed publications	2.9 (2.8)	4.2 (3.3)	5.2 (3.4)	5.1 (3.3)	.009	.041	.934
Unmatched applicants
Number of peer‐reviewed publications*	6.0 (2.7)	2.4 (2.5)	3.0 (3.3)	6.2 (4.0)	.002	.824	.029

^* Wilcox rank sum test used to estimate P‐values for unmatched applicants; n = 6 for 2018, n = 19 for 2019, n = 16 for 2020, n = 12 for 2021 for unmatched applicants.

DISCUSSION

In this study, we used the Texas STAR database to estimate differences between otolaryngology applicants in the 2021 match year compared to prior years given the disruptions caused by the COVID‐19 pandemic. Our findings suggest that traditional metrics such as board scores, honor society membership, clerkship grades, volunteering, and leadership were not significantly different for matched applicants in 2021. Matched applicants in 2021 had significantly more abstracts/posters/presentations and peer‐reviewed publications than prior years, although the secondary analysis suggested that this finding is part of a trend over time rather than specific to 2021. In addition, while unmatched applicants appeared to have significantly more peer‐reviewed publications in 2021, the validity of this finding is limited given that only 12 unmatched applicants responded to the survey that year. In contrast to our hypothesis, matched applicants in 2021 did not attend significantly more interviews than applicants in prior years and were not more likely to match at a program where they had a geographic connection. In its first year of implementation, preference signaling resulted in a yield of 0.61 interviews per signal and approximately a quarter of applicants matching at a program where they sent a signal. Finally, applicants in the 2021 match year reported significantly lower total costs related to the residency application cycle than prior years.

One of the major changes precipitated by the COVID‐19 pandemic was a shift to virtual residency interviews, recommended by the Association of American Medical Colleges in order to comply with public health and safety measures. ⁹ A major concern about virtual residency interviews was that without the monetary and time constraints typically associated with in‐person interviews, some applicants would be able to attend significantly more interviews than previous years, which could negatively affect the rest of applicants. ¹⁰ , ¹¹ , ¹² , ¹³ This possibility was especially concerning for stakeholders in otolaryngology ¹⁴ , ¹⁵ where there was already evidence of interview inequality before the pandemic, with one study showing that approximately a quarter of all applicants account for 50% of interview positions offered. ¹⁶ Interestingly, our study did not find an exacerbation of this phenomenon among otolaryngology applicants in the 2021 match year. In fact, in this sample, matched applicants in 2021 reported attending statistically significantly fewer interviews than applicants in prior years (P = .040). While this finding should be validated by additional, larger studies, these initial data are reassuring if virtual residency interviews are to be continued in the future.

It has also been hypothesized that lack of in‐person away rotations would result in more applicants matching in familiar geographic territory. ¹⁷ In this sample, however, only 30.7% of responding otolaryngology applicants reported a geographic connection to the program at which they matched in 2021, compared to 40.0% of responding applicants in prior years (P = .139). It is possible that preference signaling could have helped applicants obtain interviews at more diverse geographic regions in the 2021 match year. In addition, without travel expenses for distant interviews, applicants may have been more likely to keep interviews outside of their regions.

The lack of meaningful differences in matched applicant characteristics in 2021 compared to prior years after the secondary analysis is also reassuring. Without in‐person away rotations, there was concern among otolaryngology applicants that programs would not be able to gather sufficient information to make an informed decision about their candidacy. ² In the same survey, 36.1% of otolaryngology applicants believed that there would be more emphasis placed on board scores and research. ² It is possible that an increased focus on holistic application review and the advent of virtual rotations could have helped close the gap left by absence of traditional away rotations. Year‐to‐year differences in otolaryngology applicant characteristics may also be blunted by a ceiling effect, in which the average applicant metrics are already so high given the competitiveness of the specialty that there is little room for improvement.

Overall, our findings suggest that the virtual adaptations precipitated by the COVID‐19 pandemic may not have objectively impacted the 2021 otolaryngology match outcomes as expected. Some of these adaptations, such as virtual informational webinars and virtual away rotations may be worth considering for future application cycles post‐pandemic, potentially as a supplement to in‐person activities. In a 2021 National Resident Matching Program (NRMP) survey of 1,033 program directors, 60% reported that they “intended to rely on the virtual environment for at least some portion of the recruitment cycle in the future.” ¹⁸ In addition, some authors argue for hybrid model of virtual interviews with “optional, non‐evaluative open house days for revisit and second look opportunities.” ¹⁹ Despite our reassuring findings, there are certain intangible elements of the in‐person experience that applicants and programs may be reluctant to give up. ¹⁸ Our goal was to provide some early evidence to help inform future decisions regarding the continuation of virtual interviews, but we believe more qualitative and quantitative studies are warranted to support decisions beyond the 2021 to 2022 application cycle. More research will be needed to determine the most effective, preferable, and equitable changes to the residency selection process going forward.

Our study is one of the first to evaluate applicant‐reported data on preference signaling in its inaugural year. The finding that approximately one‐fourth of applicants matched at a program where they sent a preference signal is perhaps lower than expected given that applicants received 5 signals, and the 2021 NRMP report found that among all specialties, 72.3% of U.S. MD seniors matched at one of the top three programs on their rank lists. ²⁰ However, based on our results, about 39% ²¹ of signals did not result in an interview. Additionally, some applicants may have strategically signaled programs not in their true top five in attempt to provide a safety net. It is worth noting that while 88 out of the 113 Texas STAR respondents in 2021 reported data for preference signaling, this only represents 15.8% of the total 558 otolaryngology applicants who participated in preference signaling in 2021 according to the Otolaryngology Program Directors Organization (OPDO). Our findings suggest that preference signaling resulted in a yield of 0.61 interviews per signal, which is very similar to the 58% reported by OPDO. ²¹ However, our finding that 29% of applicants sent signals to programs with geographic ties underestimates the 50% reported by OPDO. ²¹ Given our limited sample size, any inconsistent findings on preference signaling should be deferred to the results published by OPDO using the full dataset.

The OPDO has made several recommendations for 2021 to 2022 application cycle to help meet the ongoing and ever‐changing challenges that the COVID‐19 pandemic poses on residency recruitment and selection. ²² Specifically, the OPDO recommends that applicants with a home program are limited to one away rotation, and applicants without a home program are limited to two. With regard to interview format, the OPDO recommends that individual programs choose the interview format that best meets the needs of the applicants and programs while complying with institutional and travel guidelines. Finally, the OPDO recommends that applicants receive four preference signals to use during the 2021 to 2022 cycle given the ability to do at least one away rotation. Our findings suggest that these changes will not have a significant impact on interview distribution, geographic clustering, or matched applicant characteristics in the 2022 otolaryngology match. However, with the return of in‐person away rotations and potentially some in‐person interviews, we do expect to see an increase in applicant‐reported costs for the 2021 to 2022 cycle.

One of the primary weaknesses of this study is the limited potential generalizability to all otolaryngology applicants during the study time period. Although the response rate among eligible participants who received the Texas STAR survey was over 40% for all specialties, the number of otolaryngology applicants from 2018 to 2021 responding to the survey only accounts for about 20.7% of all otolaryngology applicants during this time period (442 applicants in this sample out of 2,131 total applicants reported by the Electronic Residency Application Service ²³ ). This apparent discrepancy is partially attributable to lack of participation of some medical schools in the Texas STAR survey as well as the exclusion of international medical graduates from the Texas STAR study. In terms of geographic representation, 43.6% of respondents in this sample were from medical schools in the South (AL, AK, FL, GA, KY, LA, MS, NC, SC, OK, TN, TX, VA, WV), 24.8% from the Northeast (CT, DC, ME, MA, NH, NJ, MD, NY, PA, RI, VT), 22.7% from the Midwest (IL, IN, IA, KS, MI, MN, MO, NE, ND, OH, SD, WI), and 8.8% from the West (AZ, CA, CO, HI, NV, NM, OR, UT, WA). In comparison, data from a single‐institution otolaryngology program in the West during a similar time period reported the following distribution of otolaryngology applicants: 36.5% from the South, 24.1% from the Northeast, 24.8% from the Midwest, and 14.6% from the West. ²⁴ We suspect that the relatively high percentage of respondents from the South in our study may be a bias related to the affiliation of Texas STAR with UT Southwestern Medical Center, potentially creating increased visibility and awareness among medical schools in proximity.

There are some additional limitations related to the Texas STAR database. The Texas STAR database relies on voluntary response to surveys and may be subject to both recall bias and selection bias. The percentage of matched applicants in our sample (87.3%) overestimates the match rate for otolaryngology based on NRMP data, suggesting that matched applicants were more likely to respond to the survey. As a result, our analysis for unmatched applicants (n = 56) had limited statistical power and should be interpreted with caution. Finally, the Texas STAR database lacked information on applicant race, gender, and socioeconomic status, which has been shown to impact applicant's access to opportunities and match success in surgical fields. ²⁵ , ²⁶ , ²⁷

Despite the potential for selection bias and limited generalizability, the Texas STAR survey has some unique strengths that distinguish it from other databases such as the NRMP. To our knowledge, it is the only national residency selection survey that collects data on applicant‐reported connections with matched programs such as geographic ties and away rotations. In addition, it collects data on costs associated with the residency application process, which are an important consideration given the ongoing deliberations about continuing virtual interviews and virtual away rotations after the COVID‐19 pandemic. Finally, it provides individual‐level data allowing for statistical modeling and significance testing, which is not readily available in the aggregate data reported publicly by NRMP. As such, this is the first study to objectively examine the impact of the COVID‐19 pandemic on otolaryngology match, including interview distribution, geographic ties to programs, and total costs. These findings can be used to help inform future research and decisions regarding the otolaryngology residency selection process.

CONCLUSION

The COVID‐19 did not appear to result in significant differences in matched applicant characteristics for otolaryngology. There was no evidence of interview hoarding or increased chances of matching in familiar geographic regions, and applicants in the 2021 match year reported significantly lower costs. Some of the virtual adaptations precipitated by the pandemic may be worth considering for future application cycles.