Does “Hoarding” Impact Interview Distribution Among Orthopaedic Surgery Residency Applicants? A Modeling Study Based on a Large Database

Abstract

Background

“Interview hoarding” is commonly used to refer to the concentration of interview offers among a small number of high-performing residency applicants. Theoretically, if the same applicants interview at every program, fewer rank lists will be submitted than open residency positions, leading to a “match crisis” with unfilled positions after the match. There are no published studies we are aware of that describe the observed distribution of residency interview offers among orthopaedic surgery applicants or the potential impact of “hoarding” on that distribution.

Questions/purposes

We examined the distribution of interview invitations extended to orthopaedic surgery residency applicants in the 2020 to 2021 and 2021 to 2022 application cycles. The change in the shape of the interview invitation distribution was the primary outcome for two central questions: (1) Does the interview offer distribution curve among orthopaedic surgery applicants change meaningfully from baseline with implementation of an interview cap (Model 1)? (2) What is the impact on the distribution of invitations with a reduction in the number of applicants in the field (Model 2)?

Methods

This was a retrospective study of orthopaedic surgery interview invitations extended to applicants via the Thalamus interview management program during the 2020 to 2021 and 2021 to 2022 residency application cycles. The Thalamus database was chosen because it contains data on interview invitations for orthopaedic surgery residency positions and has the largest market share in orthopaedics compared with similar databases. Thalamus data represent 1565 applicants and 53 residency programs (90% and 25% of the national total, respectively) in 2021 to 2022 and 993 applicants and 46 programs (77% and 23%, respectively) in 2020 to 2021. It has been shown to contain a representative sample of orthopaedic residency programs. An interview cap (Model 1) was simulated by removing excess interviews held by applicants above the 75th and 95th percentiles, which were chosen to represent a formal cap and an informal cap, respectively. A reduction in the size of the applicant pool was similarly modeled by randomly removing 5% and 25% of applicants, chosen to simulate informal and formal application requirements, respectively. In both models, the excess interviews were redistributed among the remaining applicants.

Results

Applicants received a mean of 1.8 ± 2.2 Thalamus interview invitations in 2020 to 2021 and 1.7 ± 2.4 invitations in 2021 to 2022, with no change to the overall distribution curve. A total of 39% (606 of 1565) of applicants received no Thalamus interview invitations in 2021 to 2022, 75% (1176 of 1565) received two or fewer, and < 1% (14 of 1565) of applicants received 10 or more invitations. Redistributing excess interviews held by the top 5% of applicants resulted in 2% (61 of 2651) of interviews being redistributed (Model 1). Removing 5% of the total applicant pool resulted in a redistribution of 3% (87 of 2651) of the interview invitations (Model 2).

Conclusion

Orthopaedic surgery interview data demonstrated an expected uneven distribution of interview invitations, with a small proportion of highly competitive applicants receiving a higher number of interview offers as well as a large group of applicants receiving no interview invitations in Thalamus. Concerns that “hoarding” would lead to a crisis resulting in many unmatched residency positions seemed unfounded, given the excess of applicants relative to positions and the minimal change in the distribution of interviews in the cap model.

Clinical Relevance

Medical students applying to orthopaedic residency should seek individual advising to improve their individual odds of matching, while understanding that interview hoarding does not seem to alter the distribution of interviews. Program directors and medical students’ advisors should be cognizant that a small proportion of applicants are broadly interviewed and may benefit from steps taken to ensure applicants have genuine interest in the program.

Introduction

Orthopaedic surgery is one of the most competitive residency specialties, as evidenced by a > 30% rate of unmatched applicants in the 2021 to 2022 season [18]. The number of orthopaedic surgery applicants continues to outpace the number of available residency positions. During the 2021 to 2022 application cycle, 1737 candidates applied via the Electronic Residency Application Service for 875 available orthopaedic surgery positions, with 1470 of those applicants submitting rank-order lists to the National Resident Matching Program [2, 18]. These ratios were stable across the two prior application cycles [16-18].

Some studies have been published specific to orthopaedic surgery residency application and interview data [4, 9, 15, 19, 20]. However, these studies are either based on survey data or do not include information on interview distribution. As such, the main sources of information available to applicants regarding interview invitations are unverified online forums such as Google spreadsheets, Reddit, and Discord, as well as word-of-mouth advising from anecdotal experience [19]. One frequently voiced concern in online forums is maldistribution of interviews, colloquially labeled “interview hoarding,” wherein a disproportionately high number of interviews are offered to a small percentage of the most competitive applicants. This term appears regularly in studies of graduate medical education processes and public forums, although it lacks a consensus definition [5, 6, 21-23]. This is perceived to limit the number of interviews available to other applicants, and application or interview caps have been proposed in order to address this perceived problem. Despite a lack of supporting data, the maldistribution was predicted to be so extreme in the 2020 to 2021 application cycle that it would lead to a “match crisis” with many unfilled residency positions broadly across medical and surgical specialties [4, 21, 22]. Not only has this predicted crisis failed to materialize, but also, to date, no data substantiating this phenomenon have been published for any medical subspecialty.

We therefore used a large, generalizable database to analyze the distribution of interviews across candidates in two typical match cycles, and model whether two proposed solutions to the “problem” of interview hoarding would have the desired effect. Specifically, we asked, (1) Does the interview offer distribution curve among orthopaedic surgery applicants change meaningfully from baseline with implementation of an interview cap (Model 1)? (2) What is the impact on the distribution of invitations with a reduction in the number of applicants in the field (Model 2)?

Patients and Methods

Study Design and Setting

This was a retrospective analysis of data from two interview seasons drawn from a large, longitudinally maintained database of graduate medical education information. Thalamus (SJ MedConnect, Inc dba ThalamusGME: https://thalamusgme.com/) is a commercially available cloud-based graduate medical education interview management platform and scheduling software used by residency programs that is hosted on the Microsoft Azure/SQL server. The Thalamus database is distinct because it contains longitidinal information on applicants and residency programs, including Electronic Residency Application Service application data, interview offer releases, and scheduling and completion of interviews. Similarly structured longitudinal data on interview offers, acceptances, and completion are not available from any other database that contains a similar proportion of residency programs. This study was designed and reported using the STROBE guidelines.

We used data from Thalamus to perform a retrospective study, examining demographic data and interview invitation distributions from the 2020 to 2021 and 2021 to 2022 orthopaedic surgery residency application cycles. All analyses were performed after completion of the 2022 National Resident Matching Program match. Only aggregate, deidentified data were shared for analysis by Thalamus.

Study Population

In the 2021 to 2022 residency application cycle, 53 orthopaedic surgery residency programs (25% of the national total) used Thalamus for interview management, representing a total of 1565 applicants (90% of the national total). This is an increase from the 2020 to 2021 application cycle, for which Thalamus captured data from 993 applicants and 46 different orthopaedic programs (77% and 23% of the national total, respectively) [10]. This increase was because more orthopaedic surgery residency programs were using Thalamus for scheduling interviews. Orthopaedic surgery programs in Thalamus have been shown to geographically span the United States and include programs from the entire spectrum of the Doximity orthopaedic surgery rankings [8, 10, 11].

Aggregate Program-level Data and Individual Data

Interview and demographic data were collected or calculated for each program, including the total number of interview invitations extended, total number of interview positions offered, number of completed interviews, and total number of applicants. National data on orthopaedic surgery application and match cycles are publicly available from the National Resident Matching Program and Association of American Medical Colleges [2, 18]. Individual applicants were then assessed for the number of interview invitations received.

Primary and Secondary Study Outcomes

Our primary goal was to model changes in the distribution of orthopaedic surgery residency interview invitations by instituting an interview cap and by reducing the number of applicants. To model an interview cap, we removed a proportion of interviews from high-performing applicants and redistributed them to the field of applicants. To model a smaller applicant pool, we removed a proportion of applicants and redistributed any interview offers to the remaining applicants. Additionally, we compared the two application cycles’ interview distributions visually and by comparing the mean and SD for the number of interviews per applicant.

Ethical Approval

This study was reviewed by the institutional review board at Oregon Health & Science University (STUDY00022821) and was determined to fall outside the regulatory definitition of human subject study because all data were deidentified before analysis and were aggregated in their presentation. As such, no written consent or additional review was required by applicants or residency programs.

Statistical Analysis and Modeling Interview Distribution

All analyses were performed using R Statistical Software (v4.2.3) [12]. Interview distribution was evaluated with a graphical plot of interview invitations and number of applicants. Models were then applied to these data for the two central questions above. In Model 1, excess invitations held by high-performing applicants were redistributed uniformly among the remaining applicants. Two separate cutoffs were used: a 95th percentile cutoff (redistribution of excess interviews from the top 5% of applicants) to represent the informal cap resulting from logistical conflict (multiple programs with the same interview date preclude an applicant’s ability to attend all interviews) and a 75th percentile cutoff (redistribution of excess interviews from the top 25% of applicants) to represent a formal interview cap or functional interview cap such as preference signaling, for example, limiting the number of interview invitations per applicant. In this population, the 25% cutoff corresponded to just above the number of interviews associated with a 90% probability of a successful match, as predicted by the Association of American Medical Colleges, if the number of interview offers per applicant in Thalamus is extrapolated to programs not included in Thalamus [3]. In Model 2, applicants were removed from the pool with random distribution. Two separate cutoffs were again used: a 5% reduction in applicants as well as a 25% reduction in applicants to simulate formal application requirements that would more substantially limit the application pool. Because these models are the function of a single, historical variable, we felt there was little value and substantial complexity in treating these cutoffs as a function to be analyzed.

The number of interview invitations and the number of completed interviews were then calculated by applicant and by program. In both models, interview invitations were redistributed using an uninformed Bayesian prior, which assumes a uniform density function for the redistribution of offers. In other words, we assumed that all applicants were equally competitive, while it is more likely that an applicant with a higher number of interview invitations will receive an additional interview and that one with a low number of invitations will not receive an additional interview. This is a conservative approach that tends to overestimate the impact of reallocation. The distribution shape and mean number of interview invitations were recalculated in each model. Interview invitations were analyzed to best reflect the distribution of program interest in applicants instead of completed interviews, meaning cancellations were excluded. We ran the same analysis using interview offers and completed interviews for a sensitivity analysis, and there was no meaningful difference. Where appropriate, p values were calculated using a two-tailed t-test.

Results

What Was the Interview Distribution Among Applicants?

We found a right-skewed distribution of interview offers among applicants, with a large number of applicants with few or no interview offers and a small number of applicants with a high number of interview offers. The mean number of interview invitations received by applicants was 1.7 ± 2.4 in 2021 to 2022 and 1.8 ± 2.2 in 2020 to 2021 (Fig. 1). There was no difference between cycles in the mean number of interviews (p = 0.20). When visualized by overlaying histograms, the distribution of interviews among applicants was grossly unchanged between the two seasons, and neither season appeared to be bimodally distributed. For the 2021 to 2022 application cycle, 39% (606 of 1565) of applicants received no interview invitations, and 75% (1176 of 1565) of applicants received two or fewer invitations.

Fig. 1.

This graph shows the interview invitation distribution among applicants, 2020 to 2021 and 2021 to 2022 application cycles. The dashed lines represent the mean number of interview invitations per applicant in 2020 to 2021 (1.8 invitations) and in 2021 to 2022 (1.7 invitations). A color image accompanies the online version of this article.

Interview Caps and Resulting Interview Invitation Distribution

When we modeled the impact of an interview cap (Model 1), with redistribution of excess interviews held by the top 5% of applicants, the maximum number of interview invitations per applicant was eight, down from an observed maximum of 13 (Table 1). Two percent (61 of 2651) of all interview invitations were redistributed, with no visible changes in the shape of the curve (Fig. 2A). When excess interviews from the top 25% of applicants were redistributed, the invitation maximum decreased to four, and 12% (318 of 2651) of interview invitations were redistributed (Fig. 2B). When we modeled the removal of 5% and 25% of the candidate pool (Model 2), mimicking implementation of an interview cap, 87 and 551 of 2651 invitations were redistributed and 3% and 21% of all interview invitations were reallocated, respectively (Fig. 2C-D). A sensitivity analysis was completed, including only programs from which data were available from both application cycles; this did not result in any meaningful change in the curve shapes nor the proportions of redistributed interview invitations.

Table 1.

Number and proportion of interview invitations redistributed under model 1and model 2

	Model 1			Model 2
Percentile	n interviews	% interviews	Interview cap	% removed	n interviews	% interviews
97.5	37	1%	9	2.5	42	2%
95	61	2%	8	5	87	3%
90	154	6%	6	10	186	7%
85	228	9%	5	15	292	11%
80	318	12%	4	20	414	16%
75	318	12%	4	25	551	21%

Model 1 (functional interview cap): redistributing excess interview invitations above a set percentile by n interviews and % interviews redistributed. Model 2 (reduction in applicants): removing a proportion of applicants from the field and redistributing interview invitations by n interviews and % interviews.

Fig. 2.

These graphs show the distribution of interview invitations before and after redistribution of interview invitations in the 2021 to 2022 application cycle using (A-B) Model 1 and (C-D) Model 2: (A) redistribution of interview invitations above the 95th percentile of invitations per applicant, (B) redistribution of interview invitations above the 75th percentile of invitations per applicant, (C) redistribution after removing 5% of applicants from the pool, and (D) redistribution after removing 25% of applicants from the pool. A color image accompanies the online version of this article.

Discussion

Orthopaedic surgery is one of the most competitive subspecialties in terms of residency selection, with approximately two applicants per residency position [18]. “Interview hoarding,” the concentration of interview invitations among a small proportion of high-performing applicants, is commonly mentioned as a concern in a competitive field, but to our knowledge, no studies have described the distribution of interviews among orthopaedic surgery applicants nor attempted to quantify the impact of any “hoarding” on the overall distribution of interviews. We therefore used a large, generalizable database to examine the distribution of interview offers among orthopaedic surgery applicants as well as model the impact of interview caps and a reduction in the applicant pool size on the overall distribution of interview invitations. Interviews were distributed unevenly, with 75% of applicants receiving two or fewer invitations. A modest reduction in the size of the applicant pool (Model 2) caused a slightly higher rate of interview redistribution than did the creation of interview caps (Model 1), although neither intervention meaningfully changed the shape of the curve (Fig. 2). Medical students, their advisors, and other stakeholders in the orthopaedic surgery residency application process should look beyond “interview hoarding” when considering targets for reform.

Limitations

One concern is whether the Thalamus database contains a broadly representative sample, given that it includes fewer than half of orthopaedic surgery residency programs. However, 90% of the applicants for the 2021 to 2022 cycle are captured in Thalamus, and prior analysis has highlighted the geographic spread and range of Doximity rankings of orthopaedic surgery programs in the database [10]. Therefore, given the representative nature of the cross-section of residency programs included in the Thalamus database, it is unlikely that including the remaining residency programs would drastically alter the overall shape of the interview distribution curve. Importantly, national databases such as the National Resident Matching Program do not contain all interview invitation data relevant to the primary outcome of interest, making Thalamus an appropriate source of data. It would be ideal to have all applicant data and all interview invitation and completion data, and to be able to link these directly to match data at an individual level, but such data are not available. Should such combined data become available for analysis in the future through collaboration between organizations, it could narrow the CI of these results and facilitate an assessment of impacts on ultimate applicant outcomes. Although interview invitations are an important metric in the application process, they do not directly lead to understanding match outcomes.

Another limitation is the relative simplification of using the uninformed Bayesian before redistribution of interview invitations in the model. More complex models of the impact of interview redistribution or capping are possible [13, 14]. However, these models are based on the impact of interview distribution on an individual applicant instead of on the overall applicant pool. The concern about a “match crisis” because of interview hoarding is one that impacts the entire applicant field, not individual applicants. Our simulations based on the distribution of interview invitations over the applicant field do not suggest interview clustering among high-performing applicants as the prime target for reform. Orthopaedic surgery, among other specialties, has recently instituted preference signaling to address overapplication and perhaps more closely align programs and applicants who have mutual interest [15]. Although data are currently lacking, this mechanism may be promising, with 30 signals per applicant acting as a proxy for an application limit. Medical students’ advisors and orthopaedic surgery program leadership must also be engaged with and honest in advising medical students, given that many applicants ultimately receive few interviews and are at risk of not matching.

Finally, year-over-year analysis of the orthopaedic application process has been challenging because of the many co-occurring structural changes to medical education. This includes changes in the interview process because of the coronavirus-19 pandemic and associated novel procedural alterations each year, such as transitioning to virtual interviews, adding applicant preference signaling, and instituting the universal interview offer day, all of which make direct comparison difficult [10].

Discussion of Key Findings

A major finding of this study was the quantification of the overall right-skewed distribution of interview invitations for orthopaedic surgery applicants, in which many applicants had few interviews and few applicants received many interview invitations (Fig. 1). The stability of the distribution between cycles, especially given the rapid changes to interview format and application requirements as well as the record-high mean number of applications submitted (up to 90 per applicant) in recent years is notable [2]. This problem may have been exacerbated by the coronavirus-19 pandemic and widespread virtual interviews. Although this distribution is somewhat expected in a competitive process, quantifying it is helpful for applicants and advisors during the application cycle. For example, applicants might use this distribution to contextualize the number of interview offers they receive to make decisions about how to move forward in the application cycle, such as declining interviews with less-preferred programs if they have an above-average number of invitations or moving forward with dual applying in two specialties if they have fewer invitations. These are not theoretical decisions; a high percentage (26% in 2021 to 2022) of orthopaedic surgery applicants submit rank lists in multiple specialties to increase their chances of matching, and there is substantial attrition between the number of applications submitted via the Electronic Residency Application Service and the number of rank-order lists submitted to the National Resident Matching Program [2, 18]. This distribution might serve as quantitative reference point and a useful adjunct to survey-based studies, anecdotal experiences, and crowd-sourced data with questionable validity [19].

An important secondary aim of this study was to model the impact of interview caps, which are a proposed solution to mitigate “interview hoarding.” In Model 1, there was no observable change in the shape of the distribution curve when interviews above the 95th percentile were redistributed (Fig. 2A). This scenario most closely mimics the policies enacted by the Council of Orthopaedic Residency Directors for the 2022 to 2023 cycle, because there is no formal cap but rather a suggested, but not enforced, maximum number of applications and preference signaling where applicants “signaled” their interest to 30 programs [1, 7]. The scenario in which interviews above the 75th percentile are redistributed (more on par with a stricter formal cap) did demonstrate a greater change on the left side of the curve, with interview offers extended to more than 150 applicants who previously had no interviews (Fig. 2B). Redistribution of the top 25% of interview invitations was the most effective way to reduce the SD of interviews among applicants among the four modeled scenarios. However, implementing this model would require a strict cap that would do far more than correct for any “interview hoarding” present and would fundamentally change the overall distribution of interviews among applicants, while still leaving a large proportion of applicants with zero interviews. Therefore, a restrictive limit in the absence of a reduction in the size of the applicant pool may still result in an undesirable distribution of interview offers for both applicants and residency programs.

The second model, a reduction in the size of the applicant pool, demonstrated no change in the shape of the curve with 5% of the population removed. There was a larger mismatch between the original distribution and redistribution when 25% of the population was removed (Fig. 2D). However, this mismatch was almost entirely among those who received zero or one interview invitation; there was little change in the remaining categories. Removing 25% of the population was the most effective at reducing the number of applicants with zero interviews and was twofold the rate of reduction with Model 1. Ultimately, the fact that there are more applicants than available residency positions in orthopaedic surgery is what results in the lower overall match percentage. Although denying an applicant’s ability to apply to orthopaedic surgery may not be desired or possible, changes in the application process such as preference signals may mimic these results to at least help programs and applicants with high mutual interest to identify each other. Policies that reduce the number of programs that applicants apply to would also reduce costs to the applicants in the application and interview portions of the process.

We hope the data gathered will serve as an objective, quantitative reference that can help guide policy and advising for residency application in orthopaedic surgery and be used to track interview distributions over time. In 2020, orthopaedic applicants with a rank-order list submitted to 12 or more programs had an approximately 90% probability of a successful match [3]. Although 90% is a measure that applicants, programs, and medical school advisors consistently use to benchmark candidate competitiveness and likelihood of success in the match, the challenge here is that even if all applicants had 12 interviews in the past two cycles, approximately 50% would still fail to match in orthopaedic surgery because the number of positions available was far lower than the number of applicants. That is, the sheer imbalance between positions and applicants means it is impossible to redistribute interviews so all applicants receive enough interview invitations to have a more than 90% chance of matching. This is further illustrated by a recent model using otolaryngology residency application data [14]. Given that both the number of applicants in orthopaedics and the number applications per applicant are currently excessive and increasing, decreasing the size of the applicant pool and making the distribution of interview invitations more uniform may be needed to improve the probability that any given applicant obtains enough interviews to have a reasonable chance of matching.

Conclusion

We found that the distribution of orthopaedic surgery interview invitations included an exceedingly small group of high-performing applicants and a large portion of applicants with few or no interviews. The models required 25% of applicants to be removed or interviews redistributed to meaningfully decrease the number of applicants with zero interviews and alter the shape of the interview distribution curve to redistribute interviews. Neither a modest interview cap nor a small decrease in the number of applicants independently changed the distribution of interviews among applicants. Rather than implicating “interview hoarding,” for which we found no evidence, medical students and those who advise them should recognize that many applicants get few interviews; this is primarily a result of the mismatch between the high number of applicants and the relative paucity of training positions. Changes to the application process, such as preference signaling, may decrease applications to individual programs and help connect medical students and programs who have mutual interest. Publishing the number of preference signals each program receives each year, or similar objective data, could be useful for applicants and programs. With preference signaling in its second year for orthopaedic surgery in 2023 to 2024, we recommend that available data sources be used to study how preference signaling changes distributions of applications and interviews and ideally the impact of these factors on match outcomes.