Not the Last Word: Predicting Chaos in the Residency Match

Let me start with a bold prediction: The orthopaedic surgery residency match of 2021 will have a record number of unfilled positions. For the past decade, there have been, on average, fewer than two unfilled positions annually. This year, I predict, there will be at least 10.

Perhaps you might think this is not such a bold prediction after all. We are in the midst of a once-in-a-century pandemic. Some processes might break down. Predicting chaos in chaotic times, you may say, is trite.

Nonetheless, my prediction might still be worth examining because it’s based not on generalized COVID-19-chaos but on a specific factor: namely, the newly-employed method of virtual, online interviews. And if virtual interviews will be used in the future (even when there is no pandemic demanding them), their untoward effects will remain.

The problem is this: Virtual interviews are too easy to attend. Applicants will not be limited by time or budget, and thus, applicants will not limit themselves to attending only about 15 interviews at most (as I have seen them do in the past). This freedom will allow the best students to monopolize interview slots.

Of course, programs do not share a uniform definition of “best student.” However, if there are, say, 295 orthopaedic applicants who are members of the Alpha Omega Alpha (AOA) honor society (as there were last year), and if those students accepted 20 interview invitations each, then nearly every interview slot available nationally will be taken up by this cohort alone. The resulting interview pool will be too small to fill every available position.

There can be reasonable disagreement about how large the interview pool should be. There can be no disagreement that it requires at least 849 available applicants to fill 849 available positions (the number offered in the 2020 orthopaedic surgery residency match [23]).

What’s the problem? According to the Electronic Residency Application Service (ERAS), last year nearly twice that number, 1577, applied [5].

Although 1577 seems like plenty, it might not be. The problem is hinted at by the phrase “available applicants.” In fact, by the time the match rolled around, there were only 1117 applicants available: 460 of those who filed an application did not ultimately participate in the orthopaedic surgery match [22]. That is, nearly one-third of the original pool dropped out between the date when applications were due and the time a few months later when rank lists were submitted.

Some of these applicants may have had a change of heart, and one or two may have gotten sick or decided to postpone graduation for other reasons. In my experience, however, the chief reason for dropping out is a failure to secure interview invitations.

Normally, when we think of those rejected, we consider those who did not match. The size of that group, however, is considerably smaller than the number who were rejected implicitly when they failed to secure interview invitations. A majority of the rejections last year, 63%, were in this “rejected implicitly” group.

To the extent that virtual interviews make it easier for some applicants to accept invitations without limit, many other applicants will not receive interview invitations and in turn will be forced to drop out. Using last year’s numbers as an example, the mathematical Pigeon Hole Principle [4] states that if there were an additional 200 such implicit rejections, then at least 30 spots will go unfilled.

What to do?

I see three possible solutions. The first is that interviews could be offered through a central clearing house that monitors the size of the applicant pool. Such a system perhaps might even limit the number of interview invitations a candidate is permitted to accept.

Another option is to conduct asynchronous video interviews. In this process, applicants submit brief videos responding to prompts. This will allow programs to conduct many more interviews (and thereby consider many more candidates). Indeed, programs might as well invite all applicants to participate. Programs can elect not to watch every video response submitted, but they need not inform the candidates of that decision. In this fashion, no applicant is implicitly rejected and none will have reason to despair and drop out.

A third option is to limit the number of applications per applicant. I like this for many reasons [8], but foremost because if applications per applicant were capped at 15 programs, our friend the Pigeon Hole Principle guarantees that no applicant can scoop up 16 or more interview slots.

Of course, there is no central authority willing and able to impose such a limit explicitly. The one organization that is able (ERAS) is likely not willing. (ERAS collects a large fee for every additional program designated, though its marginal cost for additional applications is zero.) Still, de facto limits can be imposed without ERAS’s cooperation. As I previously described [8], a trusted system in which applicants could designate “core programs of interest” could simulate a market in which real limits were present. Such a system would ensure that the applicant pool was sufficiently large. And to those who say “it can’t be done,” I note with a mixture of satisfaction and disappointment that it has been done. (I’m disappointed because a rival specialty, otolaryngology, beat us to it [28]).

Woody Allen correctly noted that “showing up is 80% of life.” Showing up is valuable because usually showing up has an opportunity cost. When showing up is free, many of its advantages dissipate.

Showing up to a virtual interview costs almost nothing and therefore signals little. That loss of signal creates chaos, independent of any pandemic-induced chaos. If we use virtual interviews, we will need an explicit signaling system more than ever.

John C. Hagedorn II MD

Department of Orthopaedic Surgery and Rehabilitation, University of Texas Medical Branch

Mass implementation of virtual interviews has been a shock to the 2021 orthopaedic residency match, especially for orthopaedic residency programs where virtual interviews have never been used. Dr. Bernstein makes several good points as to why virtual interviews may lead to an unprecedented number of unmatched positions in the 2021 orthopaedic residency match. While I agree that virtual interviews may contribute to an increase in unmatched residents in 2021, the virtual interview may be only one part of a larger problem resulting in unmatched positions.

In a traditional match cycle, the interviews receive a great deal of attention in orthopaedic surgery as they are pivotal in narrowing down the applicant pool to a final rank list; however, we should not forget that interviews are only one tool used to select applicants. Applicants must first demonstrate capability and interest to even receive interview offers. Applicants’ capabilities and interest in orthopaedic surgery are demonstrated through academic performance and clinical rotations, which are compiled with other life experience into an application via the ERAS. Unfortunately, many universities had to suspend research and other academic opportunities during the COVID-19 pandemic. This left many medical student research projects incomplete and other opportunities (helping with high school sports physicals, for example) cancelled, which could have helped bolster a candidate’s application. Furthermore, current 4^th-year medical students had many of their core clinical rotations cancelled, moved to virtual experiences, or fulfilled other ways during the pandemic, and this was extended into orthopaedic home and away rotations. Clinical rotations, whether home or away, are used by medical students not only to confirm specialty choice but also to network and get to know practicing orthopaedic surgeons. Programs also use these rotations to attract candidates to their program and as an opportunity to evaluate a candidate firsthand. The impact of this lack of clinical rotations during this match cycle cannot be understated, especially given that 60% of orthopaedic applicants match at their home program or at a place where they completed an away rotation [26].

The lack of academic and clinical experiences during medical school because of the pandemic will probably cause little harm to the top applicants, as these applicants often already have stellar applications and will be offered interviews. However, the middle group of applicants, who may have a good application but some minor deficiencies (like low United States Medical Licensing Examination [USMLE] scores), will be negatively affected. These applicants count on their academic and clinical experiences, which were largely cancelled, to improve their applications and show programs why their minor deficiencies should be overlooked. I fear that interviews previously offered to this middle group of applicants will disproportionately go to the top applicants, giving top applicants a still-greater opportunity to hoard interview spots. In response to Dr. Bernstein, I would suggest that any increase in the number of unfilled spots in orthopaedic surgery is likely multifactorial and cannot solely be blamed on virtual interviews.

Helen K. Morgan MD

University of Michigan Medical School

Albert Einstein reportedly said that if he had an hour to solve a problem, he would spend 55 minutes thinking about the problem, and the remaining 5 minutes thinking of a solution. Clear identification of the problem is crucially important to the current state of residency application processes.

In his latest column, Dr. Bernstein warns of the potentially dire consequences of virtual interviews and predicts more unfilled orthopaedic surgery residency positions this year. His concern regarding the skewed distribution of interview offers is appropriate. However, this is not a unique problem related to virtual interviews. An examination of 2016 National Resident Matching Program data reported that 12% of orthopaedic surgery applicants were already accounting for 31% of total interviews [19]. In the midst of these growing worries, the implementation of virtual interviews has amplified concerns of a coming “match crisis” [2].

But the data must inform our concerns. The December 2020 release of interview data from Thalamus (a GME management platform) showed few differences in interview offer distributions this year compared with previous cycles [34], suggesting that disaster may be averted for now. Still, the root problems need to be urgently defined and addressed.

One pressing problem is that an increasingly inefficient selection process favors a subset of applicants based on flawed metrics. The average number of applications received by orthopaedic surgery residency programs increased from 457.5 in 2013 to 553.9 in 2020 [5]. This makes it increasingly challenging for programs to review applications holistically, resulting in the reliance on metrics such as the USMLE Step 1 score, AOA status, and clerkship grades [25]. Racial and gender differences in USMLE scoring [31], clerkship grades, and evaluation comments [12, 20, 30], as well as racial biases in AOA selection [10, 38], now are being recognized. These emerging data are especially relevant given how gender, racial, and ethnic disparities have persisted in orthopedics despite valiant recruitment efforts [29].

In order to address these issues, key stakeholder groups need to thoughtfully create learner-centered solutions. Dr. Bernstein accurately notes that the ERAS receives a fee for every additional application filed, so this is a propitious time for specialty societies to provide much-needed leadership. Application and interview caps [21] will need to be one of many parts of the solution. Incremental measures such as standardized video interviews [3] and secondary application essays have faltered [33] because the onus fell too squarely on the applicants with limited added benefit for residency programs. The Otolaryngology Program Directors Association opted to centrally organize a preference-signaling system this year [32], in which applicants could indicate “tokens” of interest to up to five programs at the time of application submission. More-equitable distribution of interview offers is one of the goals of their initiative, although this alone will not address the rising numbers of applications per applicant. In order to fully address current inefficiencies, disruptive measures such as the proposed Early Result Acceptance Program (ERAP) need to be considered. In this model, applicants would apply to a small number of programs prior to the match, and programs could fill a small percent of their positions through ERAP [17]. As this year’s unprecedented application cycle completes this month, we have a much-needed opportunity to pause and to identify clearly the problems that we need to address for future cycles. Only then will we identify the solutions that will truly improve the training of our future physician workforce.

C. W. David Chang MD

Department of Otolaryngology—Head and Neck Surgery, University of Missouri

The application process that drives the medical residency market has its advantages, disadvantages, and unforeseen repercussions. One repercussion is the prolific rise in applications submitted per applicant. In the ERAS 2011 season, US graduate orthopedic surgery applicants applied to an average of 56 programs [7]. Ten years later in the ERAS 2021 season, this number is 77 [6]. There are many reasons for this, but the change principally is driven by supply and demand, as well as by the “all-in” approach taken by many orthopaedic surgery applicants. By that I mean that the number of applicants for whom orthopaedic surgery was their only choice (921) was larger than the number of available positions (849) in the 2021 orthopaedic match [24]. In otolaryngology, applicants are likewise driven to apply prolifically to mitigate risk [14]. With the deluge of applications, applicants are less able to distinguish themselves from one another. Programs therefore must invest a considerable amount of time and opportunity cost in the applicant selection process.

The impetus behind instituting the preference-signaling process for otolaryngology this year was to help applicants to credibly show interest in specific programs. In a nonpandemic year, applicants can mitigate match risk by “auditioning”: traveling for away rotations to cultivate faculty advocates. Such tactics advantage financially secure applicants and those from schools of higher reputations. While the pandemic environment may have eliminated some of these financial pressures, the social determinants have become more influential. Preference signaling provides a means in which applicants can signal interest in programs credibly and equitably.

In the otolaryngology preference-signaling process, each applicant is given five equally weighted signals to send to five different programs [15, 27]. The system was created with the intent of maximizing the applicants’ preferences. However, applicants must send their signals strategically, relying on a calculus that accounts for the attractiveness of their candidacy, professional interests, and programs’ interests. If the system is built with too many signals, not only is the value of each signal diluted, but the absence of a signal would more strongly become a signal of applicant disinterest. Programs that are inundated with signals may find less utility. We plan on investigating this further upon completion of match 2021.

A complementary solution to this problem would be to advocate for an application cap. It will require multiple specialties to work together to advocate for this change. Signaling might serve as a proxy for an application cap by reducing the incentive for residents to send so many applications. However, if applicants are allowed too few signals, they will continue to mitigate risk with large numbers of applications. Preliminarily for the 2021 season, it has not changed the number of applications per otolaryngology applicant. However, a cultural change like this may take a few seasons to bear fruit. With the right balance, the hope is that applicants would rely less on over-applying with the aid of signaling.

Margaret (Meg) Wolff MD, MHPE

University of Michigan Medical School

Dr. Bernstein voiced the concerns that many medical educators have been quietly repeating for months—namely, that the 2021 match cycle may yield an unprecedented number of unfilled positions and unmatched applicants.

As Dr. Bernstein outlined, the transition to virtual interviews removed the financial and logistical barriers of in-person interviews that naturally restricted the number of interviews an applicant could complete. Without these constraints, educators have noted anecdotally that top-tier candidates have scheduled more interviews than in prior years, leaving average students with fewer interviews [36, 37]. This maldistribution of interviews with programs vying for the same top-tier candidates could result in a mismatch of open positions and number of applicants. Educators from general surgery [36], obstetrics and gynecology [21], and the Association of American Medical Colleges [37] have expressed concern about this potential crisis for the 2021 match.

Despite these concerns, virtual interviews offer several advantages. The traditional interview process imposes substantial financial and educational costs [1, 9, 11]. The annual cost to a program to fill a single postgraduate position in internal medicine has been estimated at USD 9899, with an average annual total of USD 148,000 per program [11]. Applicants typically spend between USD 2500 to more than USD 10,000 (in travel, lodging, and application fees) for primary care and surgical specialties, respectively [1, 11, 13]. In 2015, the average cost to apply to orthopaedic surgery was USD 5415, with some students spending upwards of USD 25,000 [13]. Virtual interviews, however, are free for the applicant after the initial applications. Eliminating this financial barrier may enable students from socioeconomically disadvantaged backgrounds to consider specialties that previously had been cost-prohibitive.

The traditional interview process is also time-intensive and results in lost productivity for faculty and educational time for trainees [16, 35]. Virtual interviews still require considerable faculty time, though it may be possible to shift some portion outside of normal clinical hours. The time saved for medical students is noteworthy. Without travel, applicants have more flexibility with interview scheduling, increasing time available for educational endeavors. The virtual format also allows more flexibility for how information is delivered [39]. Advance delivery of program information through webinars or online content allows interview time to be spent on an applicant’s strengths and characteristics.

This pandemic has caused disruption at every level, making it unlikely that recruitment will completely return to the pre-pandemic state. The recruitment system had challenges prior to the pandemic—applications per applicant were skyrocketing, and programs were challenged to discern which applicants were truly interested. The pandemic has simply exacerbated these preexisting conditions [8, 18, 21]. We should use this opportunity to evaluate the residency-recruitment process and the role virtual interviewing may play going forward. Lessons learned from studying this recruitment cycle will inform recruitment for years to come. As we anxiously await March 15 to see if we will need to participate in the Supplemental Offer and Acceptance Program, I will offer some hope. Recent data from a residency interview reservation system demonstrate that for the approximately 60 orthopaedic surgery residency programs utilizing this system, there is less than 15% of overlap between any two participating programs in the applicants invited to interview [34]. In other words, programs are not inviting the same group of applicants. Although this only represents a fraction of the orthopaedic surgery programs participating in the 2021 match, these data offer hope for another successful match.