I Dream of Gini: Quantifying Inequality in Otolaryngology Residency Interviews

Abstract

Objectives:

Otolaryngology applicants routinely decry conflicting interview dates as it limits the number of interviews that one can attend, despite being offered an interview. Conversely, applicants also perceive that a large number of interviews are offered to a minority of applicants. We sought to verify and quantify the inequality in distribution of interviews attended.

Study Design:

Retrospective analysis of the National Resident Matching Program (NRMP) 2016 Charting Outcomes in the Match and Electronic Residency Application Service (ERAS) historic specialty data.

Methods:

The Gini coefficient, a commonly used indicator of economic inequality, was calculated using data from the 2016 Charting Outcomes in the Match to estimate the distribution of interviews attended. This data was compared to nine other specialties, comprising a wide range of competitiveness and specialty size.

Results:

26% (110/416) of otolaryngology applicants accounted for half (1721/3426) of all possible interview positions. The Gini coefficient ranged from 0.43 to 0.84 across 10 specialties, with a higher coefficient indicating higher inequality. The Gini coefficient among otolaryngology applicants was 0.43, indicating lower inequality than most other specialties. When including only applicants who interviewed, the Gini coefficient was 0.23.

Conclusions:

There is an unequal distribution of interview invitations, which likely reflects the reality of asymmetry in applicant competitiveness. Otolaryngology demonstrates the greatest equality in distribution, which may stem from a greater burden of hoarding. The specialty’s perceived competitiveness mitigates factors such as cost and time, essentially encouraging more people to take as many interviews as they can

INTRODUCTION:

The Otolaryngology – Head & Neck Surgery (OHNS) match is recognized as a highly competitive match, having among the highest average USMLE^® (United States Medical Licensing Examination) Step 1 scores, higher proportion of Alpha Omega Alpha members, and a lower match rate than most other specialties.¹ Indeed, the mean USMLE Step 1 scores of matched OHNS applicants have been consistently 0.5 to 1 standard deviation above the national mean.²

Given the competitive nature of the OHNS match, there are two sentiments that are frequently expressed on the OHNS interview trail. On the one hand, applicants denounce the restriction of conflicting interview dates as it limits the number of interviews that the applicant can attend. On the other hand, they also decry the perceived hoarding of interviews by a minority of highly desirable applicants. Message boards such as otomatch.com are often fraught with posts from frustrated applicants requesting that individuals to give up interviews in the name of fairness and collegiality.

As with other scarce resources, the distribution is not equal. In this study, we sought to verify and quantify the inequality in distribution of interview invitations using the Gini coefficient. The Gini coefficient, originally developed in 1912 by Italian statistician and sociologist Corrado Gini, is a popular measure of inequality and ranges from 0 (perfect equality) to 1 (perfect inequality).³ It is typically used to represent the income or wealth distribution of a nation’s residents. We hypothesized that a small group of applicants accounted for a disproportionately large number of interview positions. We also sought to compare the results in OHNS with the Gini coefficient and interview distribution patterns of other specialties, with the purpose of exploring potential mitigating or exacerbating qualities in perpetuating inequality.

METHODS:

Database Selection:

The 2016 National Residency Match Program (NRMP) Charting Outcomes in the Match data was used to determine the total contiguous ranks for U.S. allopathic graduates, U.S. osteopathic graduates, and International Medical Graduates (IMG’s).^4–6 The year 2016 was selected as it was the most recent year where the complete Charting Outcomes in the Match data was available. The total number of applicants to a given specialty was determined by aggregated historical specialty specific data provided by the Electronic Residency Application Service (ERAS).⁷ ERAS updates its data monthly during the match season and the February data was used as it most accurately reflects the application burden within an application cycle.⁸ In contrast, the end of match data, which is formally published by ERAS, includes extraneous applicants such as those applying through the Supplemental Offer and Acceptance Program (SOAP).⁹

Calculating Proportional Representation of Applicants:

Within the Charting Outcomes in Match, the “number of contiguous ranks” figure was examined for each specialty to determine the number of applicants vs. the total number of contiguous ranks ranging from 1 to “16 or more.”⁴ It was assumed that the number of ranked positions corresponded to total number of attended interview positions. According to the NRMP Applicant Survey from 2017, the majority of applicants (89%) responded that they ranked all programs at which they interviewed.¹⁰ These “16 or more” individuals were defined as the “Interview Rich” Applicants (IRA’s).

The proportion of IRA’s in the overall pool of applicants in a given specialty was calculated by dividing the # of IRA’s, provided by the number of contiguous ranks, from the total number of applicants, provided by ERAS. The total number of interview positions was calculated by summing all of the contiguous ranks in a given specialty.

An index to normalize and compare the distribution of interviews across specialties – the “interview distribution index” (IDI)-was calculated by dividing the proportion of interviews taken by IRA’s by the proportion of applicants comprised of IRA’s. Hence, an IRI of “1” would be denoted to be completely fair, and increasing values above that, less fair.

Calculating the Gini Coefficient:

The Gini index was calculated by using an online Gini coefficient.¹¹ The Gini coefficients were calculated in 2 major populations: those including all applicants (including those that received zero interviews) and those that ranked at least one program in a given specialty. In order to remove any potential bias related to international and osteopathic school status, the Gini coefficients were also separately calculated for U.S. allopathic medical students in each of these groups.

Data for the Gini calculator was provided by the NRMP number of continuous ranks. For the purposes of calculating the Gini coefficient, the IRA’s were assumed to have attended 16 interviews as further granularity was unavailable. The number of applicants who attended zero interviews was calculated by subtracting the total number of individuals who ranked programs (U.S. allopathic + U.S. osteopathic + IMG) based on NRMP data, from the total ERAS applicants.^4–6,9 The number of U.S. allopathic applicants who attended zero interviews was calculated by subtracting the number of U.S. allopathic applicants who ranked programs through NRMP, from the total number of ERAS U.S. allopathic applicants. Comparisons of the Gini coefficient to countries around the world was based on the Central Intelligence Agency (CIA) World Factbook.¹²

RESULTS:

Proportionate Representation:

IRA representation among all applicants ranged from 3–25% across specialties and accounted for 12–55% of the total interviews [Figure 1]. In order to eliminate any applicants who simply “threw their hat into the ring” and to look at only applicants who had a chance of matching, the data was re-examined to include only applicants who ranked a given specialty at least once. In this group, IRA’s comprised a higher proportion of the applicant population, ranging from 6–40% [Figure 2].

Figure 1. Proportion of IRA’s and IRA interviews across all applicants:

Interview Rich Applicants (IRA’s) account for 19% of all OHNS applicants but account for 38% of the interviews.

Figure 2. Proportion of IRA’s and IRA interviews across only those applicants who interviewed:

Interview Rich Applicants (IRA’s) account for 26% of OHNS applicants who received an interview but comprise 38% of interview slots.

Specific to OHNS, according to ERAS, there were 416 total individuals who submitted an application, including U.S. allopathic, IMG’s, and osteopathic students.^1,9 IRA’s comprised 19.5% (81/416) of applicants and alone attended an estimated 37.8% (1296/3426) of interviews.^1,5,6,9,10 Only neurosurgery had a greater proportion of IRA’s making up the total applicant pool. Amongst only applicants who attended at least 1 interview, IRA’s comprised 26.2% (81/309) of applicants. [Figure 2].

Plastic surgery was the most comparable specialty in IRA representation, with IRA’s comprising 14% (42/296) of the total applicant pool but attending 39% (672/1718) of the total interviews [Figure 1]. Among applicants who attended at least 1 interview, IRA’s comprised 25% (42/168) of applicants. Neurosurgery IRA’s took the greatest proportion of interviews compared to all specialties, comprising 24.5% (117/477) of applicants but accounting for 54.6% (1872/3428) of total interviews. Looking at only applicants who attended at least one interview, IRA’s comprised 39.9% of the neurosurgery applicant pool.

The “interview distribution index” (IDI) ranged from 2.0 to 8.6 [Table 1]. Family medicine demonstrated the highest IDI, suggesting the greatest gap between IRA’s and the rest of the applicant pool with respect to their share of the total interviews. Across all applicants, Family Medicine IRA’s had 2.6% (316/12,207) of applicants comprising 22.2% (5056/22753) of interviews. OHNS had the lowest IDI of 2.0, suggesting the smallest gap between IRA’s and the rest of the applicants.

Table 1. Interview Distribution Index:

Normalized representation of IRA’s across specialties. The IDI was calculated by dividing the proportion of interviews taken by IRA’s by the proportion of applicants comprised of IRA’s. An IRI of “1” would be denoted to be completely fair, and increasing values above that, less fair. Otolaryngology was among the most fair in interview distribution.

Specialty	Interview Distribution Index (IDI)
Otolaryngology	2.0
Neurosurgery	2.2
Orthopedic Surgery	2.5
Plastic Surgery	2.8
Diagnostic Radiology	2.8
Dermatology	3.6
General Surgery	4.5
Internal Medicine (categorical)	4.6
Pathology	5.8
Family Medicine	8.6

Finally, in order to standardize the comparison of the proportion beyond just IRA’s, the number of applicants to comprise 50% of total interview slots was calculated [Figure 3]. This was performed for both the total number of ERAS applicants and only those applicants who interviewed. In a hypothetically fair world, 50% of the applicants would take up 50% of interview slots.

Figure 3. Percentage of applicants (relative to the total number of applicants and those who interviewed) needed to interview to comprise 50% of interview spots:

26% of all OHNS applicants comprised 50% of total interviews. 36% of applicants who actually interviewed comprised 50% of total interviews.

The order of the specialties largely mirrors that of the representation of IRA’s. Indeed, in neurosurgery, IRA’s alone captured greater than half of the total interviews. OHNS likewise had more equality compared to other specialties, with 36% of applicants who interviewed comprising 50% of the interview spots. When examining all applicants who submitted an application to OHNS, about 26% (110/416) of applicants comprised half of the total interviews. Family medicine again demonstrated greater inequality with just 23% of applicants who interviewed and 6.6% of total applicants, taking up half of the interviews.

Gini coefficient:

The calculated range of Gini coefficients for all specialties across all applicants was 0.428 (OHNS) to 0.843 (Family Medicine) [Table 2]. Across all applicants, OHNS had the greatest equality (closer value to zero) in the distribution of interviews among all applicants, whereas family medicine had the greatest inequality (closer value to one). According to the CIA World Factbook, the OHNS GINI coefficient of 0.428 is comparable to the distribution of family income in Argentina (0.427).¹²

Table 2. Gini Coefficients across specialties:

The Gini coefficient ranges from 0 (perfect equality) to 1 (perfect inequality). Interview distributions were “more fair” when including only applicants who interviewed and also when comparing only U.S. Allopathic students.

	All Applicants		Only Applicants who Interviewed
Specialty	Total	U.S. Allopathic	Total	U.S. Allopathic
Otolaryngology	0.428	0.372	0.23	0.204
Orthopedic Surgery	0.519	0.46	0.27	0.242
Neurosurgery	0.528	0.393	0.232	0.159
Plastic Surgery	0.593	0.526	0.284	0.263
Diagnostic Radiology	0.599	0.545	0.3	0.215
Dermatology	0.6	0.545	0.336	0.311
Internal Medicine	0.718	0.576	0.403	0.204
General Surgery	0.742	0.635	0.369	0.201
Pathology	0.762	0.615	0.403	0.231
Family Medicine	0.843	0.706	0.432	0.251

When accounting for only those applicants who ranked at least 1 OHNS spot, the Gini coefficient was 0.23, similar to the distribution of family income of the Faroe Islands (0.227). Removing IMG’s and osteopathic medical school students from the Gini coefficient calculation lowered the Gini coefficient to 0.372 and 0.204 in the total applicant group and the applicants who interviewed group, respectively [Table 2]. The Gini coefficient of OHNS (0.428) was most similar to orthopedic surgery (0.519) when looking at all applicants. The Gini coefficient was most similar to neurosurgery (0.23 vs. 0.232) when only including applicants who interviewed.

As in other specialties, the Gini coefficient of OHNS decreased (−0.06) when excluding osteopathic and international medical graduates, but this was more pronounced in specialties such as internal medicine, pathology, and family medicine where international medical graduates and osteopathic students make up a larger portion of the interview and applicant pools. The specialties with the largest drop in the Gini coefficient when examining all applicants vs. only U.S. allopathic applicants were: Internal Medicine (−0.14), Family Medicine (−0.137), and Pathology (−0.147).

DISCUSSION:

In this study, we sought to verify and quantify the inequality in distribution of interview invitations with the hypothesis that a small subset of applicants receive a disproportionately large amount of residency interviews. This was achieved by examining the proportion of interviews captured by IRA’s relative to their representation in the overall applicant cohort. This was also examined by calculating the Gini index. These calculations were performed across several other specialties to act as benchmarks.

Confirmation and Quantification of Inequality in Residency Interviews:

Our results verify the phenomenon of a smaller subset of applicants taking a larger portion of interviews. Indeed, across all OHNS applicants, IRA’s comprised 19% of the applicant pool but took 38% of interviews. In this same cohort, 26% of applicants took 50% of the interviews. When including only applicants who actually interviewed, the gap between the “haves” and the “have-nots” decreased, but was still notable. This finding of IRA’s capturing a disproportionate amount of interviews was true across all specialties, including those perceived as less competitive. Indeed, contrary to what might ostensibly be expected, this effect was most apparent in family medicine, where 3% of applicants took nearly a quarter of interviews and where the IDI and Gini coefficients were greatest.

However, upon further consideration, that less competitive specialties may show greater inequality makes sense. In competitive specialties such as OHNS, there may be a perceived need to apply to a large number of residency programs. For example, Kaplan, et al. noted that medical students believed that applying to a large number of residency programs enhanced their chances for matching.¹³ In contrast, in less competitive specialties, there may be a perception that a large number of interviews are not necessary, thus lowering the number of interviews that even highly qualified applicants attend

Certainly, this perception is somewhat grounded reality as the NRMP match data also demonstrates an advantage in hoarding interviews in otolaryngology compared to family medicine. According to the NRMP Charting Outcomes in the Match 2016, the number of ranked programs required to reach a 95% probability of matching was 7 programs in family medicine compared to 13 programs in Otolaryngology, with minimal incremental improvement.¹ Taking into consideration allopathic, osteopathic, and international applicants who matched, the NRMP Charting Outcomes in the Match data shows that Family Medicine applicants ranked 8.68 programs compared to 12.46 in OHNS.^4–6 Furthermore, the Association of American Medical Colleges (AAMC) recently calculated the point of diminishing returns in the number of programs to apply to for a variety of specialties. In family medicine, the number of programs to apply to was 19, whereas for OHNS it was 39–44.¹⁴ ERAS data shows that the average U.S. allopathic OHNS applicant applied to 49 programs in 2016, whereas the average family medicine applicant applied to 27.5 programs.⁹ Hence, it is likely that in competitive specialties, the top applicants would feel more compelled to attend more interviews and IRA’s would thus comprise a larger portion of the overall candidate pool. This is further compounded by the greater proportion of “qualified” applicants in more competitive specialties due to self-selection of less competitive applicants away from those specialties. This would presumably lead to a greater proportion of applicants who are offered the opportunity to attend more interviews. Because the Gini coefficient quantifies the distribution of attended interviews but not the context of the distribution, having a large burden of IRA’s and the resulting hoarding of interviews would actually increase the Gini coefficient (reflecting less equality) in those specialties.

Disadvantage For IMG and Osteopathic Students:

A major attribute driving the higher Gini coefficients of specialties such as pathology, internal medicine, and family medicine is the large number of applicants who received no invites. This was particularly prevalent among specialties with a large number of Osteopathic and IM applicants.

Specialties with a large number of D.O. and IMG applicants saw the largest drops in their Gini coefficients when excluding them from the calculation and examining only U.S. allopathic applicants. This seems to confirm what others have previously perceived and demonstrated, that there is a disadvantage in the match of being a D.O. or IMG applicant.^15,16

Mitigating Characteristic & Solutions:

Despite this recognized phenomenon of asymmetric distribution of resources, there are several mitigating factors that may decrease inequality such as self-selection, resource limitations, and conflicting interview dates. First, there is likely self-selection in competitive specialties where those less likely to obtain interviews would be intimidated from applying in the first place, thus filtering out candidates. Over the past several years,OHNS has experienced decreasing applicant numbers, potentially related to its perception as a difficult field in which to match.¹⁷ Furthermore, applying to residency is expensive. Cabrera-Muffly, et al. reported that the average OHNS applicant spent $5,400 on application fees and interview travel expenses.¹⁸ Chang, et al. estimated that the aggregate OHNS application cost was approximately $560,395 in 2015.⁸ As a result, although candidates may wish to attend more interviews, they may be financially limited in doing so. Furthermore, applicants know that in the end, they can only match to a single program and therefore may be less inclined to hoard residency interviews compared to other scarce resources. However, these traditional mitigating factors may be diminished in the context of a perception of increased competition. If there is no point at which an applicant feels “safe,” the incremental costs of applying may not be a barrier.

In addition to financial limitations, there may be temporospatial limitations in that an applicant cannot be at two places at the same time. This may play a larger factor in smaller fields such as OHNS where interview dates often conflict from program to program. OHNS applicants often denounce the restriction of conflicting interview dates as it limits the number of interviews that the applicant can attend. This is supported by the NRMP applicant survey which showed that in 2017, matched applicants received 19 interview offers, but were only able to attend and rank 14.¹⁰ Furthermore, according to Cabrera-Muffly, et al., 74% of applicants reported declining an interview offer due to scheduling conflicts.¹⁸ In comparison, fields such as Diagnostic Radiology tend to offer a wider selection of interview dates and thus conflicts may play a role in mitigating hoarding.

The authors do not wish to place any ethical or moral judgment on the unequal distribution of interviews as there is a real asymmetry in applicant competitiveness, with certain applicants simply more qualified than others. One may argue that certain applicants worked hard to receive those interviews and deserve to attend them. However, there are some objective problems with poor interview distribution. As applicants can ultimately only match at one program, if programs are disproportionately interviewing the same applicants, then it increases the scenario of programs going unmatched. This problem became a reality in 2017 when 14 slots went unfilled for 10 programs and in the most recent match when 12 spots went unfilled, despite having more total applicants than residency positions.^7,19 Furthermore, there is a financial cost to programs. Chang, et al. noted that having a larger number of residency applications taxes programs. He calculated that an application cap of 30 programs per applicant would save over $400,000 in application costs and 19.3 hours per OHNS program.⁸ Likewise, Weissbart, et al. calculated that a cap of 20 programs per applicant would save $4.7 million yearly in application costs aggregately across 13 specialties.²⁰

Recognizing the asymmetric distribution of interviews, some solutions have been proposed. Since the introduction of the program specific application requirement for OHNS, the number of applications per applicant has decreased.⁹ Wong, recognizing the phenomenon of hoarding interviews among a pool of highly desirable applicants, proposed an early decision plan to match these applicants early and eliminate these applicants from the general interview pool.²¹ Understanding that traditional mitigating factors are limited by the perceived competitiveness in otolaryngology, OHNS could benefit from outreach efforts to correct perceptions of competitiveness in order to encourage more interested students to apply and to reassure qualified applicants to limit hoarding. For example, knowing that the probability of matching in OHNS does not substantially increase after ranking 13 programs may encourage IRA’s to feel safe matching without attending more interviews.

Limitations:

There are a number of limitations to this study. First, we were limited to using the NRMP and ERAS databases and were therefore constrained to the data provided. For example, there was no contextual breakdown in this data, with no demographic data between the “haves” and “have-nots.” Furthermore, one of the biggest limitations was the method in which the contiguous ranks was published. We have already discussed the limitation of aggregating 16 or more interviews into one group. Furthermore, NRMP lists only contiguous ranks for a given specialty, so applicants who apply to more than one specialty and list a different specialty will reflect an artificially low numbers of interviews attended and thus lead to a higher Gini coefficient. This may be particularly evident in specialties where it is likely for applicants to cross-apply. The number of contiguous ranks also reflects interviews attended and not offered and thus does not fully expose the behind-the-scenes decision making that drives an applicant to actually attend or cancel an interview.

Furthermore, the analysis was only performed for 2016 and therefore only represents a cross-section of a dynamic phenomenon where the distribution of interviews and proportion of IRAs may vary from year to year. These differences would most likely be related to the natural swings in the number of applicants. Therefore, calculated inequalities in OHNS may have been less obvious in 2016 when 570 applicants applied according to ERAS, compared to 2013 when 774 applicants applied.⁹ Especially in the last 2 application cycles where the number of applicants to OHNS has precipitously decreased and positions have gone unmatched, there will be value in revisiting this analysis in the future.²²

CONCLUSIONS:

Through calculating proportional representation and the Gini coefficient, our results verify the phenomenon of a smaller subset of applicants taking a larger portion of interviews and also confirm the disadvantage of being an osteopathic and IMG applicant across all specialties. In 2016, OHNS is more equal in distribution compared to other specialties analyzed. Otolaryngology demonstrates the greatest equality in distribution, which may stem from a greater burden of hoarding. The specialty’s perceived competitiveness mitigates traditional factors, essentially encouraging more people to take as many interviews as they can.

Self-selection, resource limitations, and conflicting interview dates likely act as mitigating factors from hoarding. The Gini coefficient may be a useful tool in quantifying interview distribution and will be important to perform this analysis in subsequent years.