Abstract
Background
There has been an increase in the number of applications medical students have submitted for the National Residency Matching Program (NRMP). These additional applications are associated with significant costs and may contribute to match inefficiency.
Objective
We explored if match rates improved in years when an increased number of applications were submitted.
Methods
We analyzed yearly published data from the NRMP and the Electronic Residency Application Service for 13 specialties. A generalized linear model was used to assess the relationship between the annual match rate and the mean number of applications submitted per applicant, while controlling for the number of positions available and the number of applicants in the given year.
Results
Over the last 13 years there has been an increase in the mean number of applications submitted per applicant (P < .001). For the 13 assessed medical specialties, there was no statistically significant relationship between the mean number of applications per applicant per year submitted to the NRMP, and the annual match rate (odds ratios near 1.00 and nonsignificant, P values > .05).
Conclusions
There was no improvement in the match rate in years when medical students submitted an increased number of applications. Therefore, it would appear that the applicants do not benefit from the larger number of applications submitted. Further study is required to assess the cost and benefit of these additional applications.
What was known and gap
Medical students are submitting a larger number of applications. The effectiveness and impact on the match have not been studied.
What is new
A study of aggregated data suggests applicants do not benefit when all submit a large number of applications.
Limitations
Aggregate data preclude analysis if certain types of applicants benefit from current practices; data apply to US senior applicants only.
Bottom line
The current high number of applications submitted has no utility for applicants and adds financial and opportunity costs for applicants and programs.
Introduction
Matching into residency has become increasingly competitive.1–5 As a result, medical student applicants are submitting an increasing number of applications to the National Residency Matching Program (NRMP). These additional applications can have a deleterious effect on the match in 3 ways: (1) they create an added cost for applicants, (2) they result in an additional time commitment for program directors who need to review these applications, and (3) they cause congestion in the application review process; therefore, program directors may overlook preferred applicants as a result. There is no study investigating whether these additional applications are associated with a benefit to match participants.
Game theory is the economic study of decision making. The prisoner's dilemma game illustrates the counterintuitive phenomenon of all members of a group being worse off when each member of the group acts in his or her self-interest (figure 1).6,7 Applying the prisoner's dilemma model to the residency match in aggregate, we hypothesize that applicants may be better off if they could agree to submit a finite number of applications rather than trying to out-apply each other. We have previously published on the possibility of a prisoner's dilemma existing in the urology match, suggesting that the urology match could do without such a large number of applications, and that participants would be served by a reasonably set application limit.8
FIGURE 1.
The Prisoner's Dilemma Payoff Matrix
In the prisoner's dilemma model, 2 criminal partners, Criminal A and Criminal B, are arrested and separated into individual holding cells for interrogation. There were no witnesses at the scene of the crime and, therefore, if both criminals remain silent and refuse to testify against one another they will only be convicted of a small charge, which carries a 3-year prison sentence. During interrogation, Criminal A is offered a plea deal and told that if he testifies against his partner (Criminal B) he will only be sentenced to 1 year in prison while his partner (Criminal B) will be sentenced to 15 years in prison. Criminal B is offered the same deal to testify against Criminal A. In the event that both criminals testify against one another they will each be sentenced to 10 years in prison. Individually, it appears to each criminal that testifying is always the best strategy. From Criminal A's perspective, if Criminal B stays silent, Criminal A is better off to testify instead of staying silent (will receive 1 year in prison instead of 3) and if Criminal B testifies, Criminal A is better off to testify instead of remaining silent (will receive 10 years in prison instead of 15). Criminal B has the same perspective. Given that it is always in the individual criminal's perceived best interest to testify, the result of this “economic game” is that both criminals will testify against one another and they will each wind up serving a 10-year prison sentence. They would have both been better off if they could have worked together and agreed to not testify against one another (3-year prison sentence each instead of 10). The prisoner's dilemma model thus demonstrates the counterintuitive phenomenon of all members of a group being individually worse off, when each member of the group acts in his or her self-interest. The reader is referred to the cited website7 for a further discussion and explanation of the prisoner's dilemma and game theory.
In this study, we investigated whether other medical specialty matching markets and applicants are benefited by a higher number of applications submitted. We also investigated if there is a related improvement in the match rate in years when larger numbers of applications were submitted.
Methods
Data were obtained from the NRMP9 and Electronic Residency Application Service (ERAS)10 websites. On the NRMP website, a robust amount of match data is published, beginning with 1984. For each specialty, the NRMP reports for each year contain the number of programs, the number of positions offered, the number of unfilled programs, the number of US senior applicants, and the number of US senior matches. Data from the ERAS website contain tables citing the yearly mean number of applications submitted in each specialty, and the number of applicants submitting those applications.11
The match rate was calculated with the following formula: Match Rate = No. of US Seniors Matched/No. of US Senior Applicants. All data necessary for match rate calculations were obtained from the NRMP website in the annually released “Results and Data” packets for years 2008 through 2013.12–17 From each packet, the following information was extracted for 13 specialties: (1) positions offered, (2) number of US senior applicants, and (3) number of US senior matches. Analysis was limited to 2008 and later, because starting with 2008 the data show applicants who ranked multiple specialties.
For specialties offering both postgraduate year (PGY)-1 and PGY-2 positions (anesthesiology, dermatology, emergency medicine, psychiatry, radiology-diagnostic), the values from PGY-1 and PGY-2 positions were combined for final analyses, with the assumption that overall trends in match rate should optimally be reflected in the sum, rather than analyzed separately. In addition, trending for 6 years mitigated year-to-year variability.
To test if the annual average number of applications was related to the match rate, a generalized linear model with a logit link was used. The response was the yearly number of US seniors matched over the number of US senior applicants, and the predictor was the mean number of applications per student during that year (derived from ERAS data).11 In addition, the number of positions offered, and the number of applicants, were included in the model as covariates. The effect of the mean number of applications was measured by an odds ratio (OR), and the α level for statistical significance was set at .05.
To estimate the cost savings effect of a residency application limit, a model was created using these parameters: the number of US applicants for the 2013 match, the mean number of submitted applications per specialty for the 2013 match (US applicants), the cited ERAS application costs,18 and a hypothetical application limit of 20 applications per applicant (table 1). Cost savings of a match with an application limit of 20 for the 13 studied specialties was calculated by the following formula: Cost savings = Cost of Match (No. of Applicants 2013, No. of Applications 20) – Cost of Match (No. of Applicants 2013, No. of Applications 20).
TABLE 1.
Cost and Time for Applying and Reviewing Applications
To estimate the time saving effect of a residency application limit for program directors, we created a model using these parameters: the number of US applicants for the 2013 match, the mean number of applications per specialty for the 2013 match (US seniors), a hypothetical application limit of 20 applications per applicant, and an 8-minute application review time per application (table 1). As there were no data available that determined the mean amount of time program directors spend reviewing an application, we assumed an application review time of 8 minutes, on the basis of how long it would take a program director to read through a student's application. Time savings of a match with an application limit of 20 for the 13 studied specialties was calculated by the following formula: Time Savings = Time Saved (No. of Applicants 2013, No. of Applications 20) – Time Saved (No. of Applicants 2013, No. of Applications 20).
Results
The mean number of applications submitted by each applicant has significantly increased over the last 13 years (P < .001; figure 2). The specialty with the highest rate of increase in mean applications submitted was orthopedic surgery (1.85 applications per year), and the specialty with the lowest rate of increase in mean applications submitted per year was pediatrics (0.44 applications per year).
FIGURE 2.
Mean Number of Applications Submitted During the Last 13 Years According to Specialty11,19
table 2 shows the results from the generalized liner model by specialty and for all specialties combined. For every specialty, the effect of the mean number of applications was small (odds ratios near 1.00) and nonsignificant (P values > .05). The number of positions offered significantly increased the odds of a match for dermatology (P = .001) and emergency medicine (P = .05), while the number of applicants significantly decreased the odds of a match for dermatology (P = .02), emergency medicine (P < .001), otolaryngology (P < .001), and diagnostic radiology (P = .001). When all specialties were combined in a single analysis, with specialty included as a stratified variable, there still was no effect for the mean number of applications (OR = 1.00; 95% CI 0.99–1.01; P = .61). The number of positions offered increased the odds (P < .001), and the number of applicants decreased the odds (P < .001).
TABLE 2.
Relationship Between Match Rate With Mean Number of Submitted Applications by Specialtya
If a hypothetical application limit of 20 was created for the 2013 match (for the 13 specialties studied), each applicant on average would save $174 from a total application cost of $355. With 27 027 applicants, the total savings would be $4,696,979. The amount of time saved per program director, with a hypothetical application limit of 20, would be 11 hours and 19 minutes. For the 2936 program directors, in the specialties analyzed, the time savings would be 33 250 hours.
Discussion
Applicants individually have an incentive to submit a large number of applications. However, when every applicant submits a large number of applications, these applications in aggregate may “cancel” each other out, leaving participants with added costs yet without an improvement in the match rate.
The reason why medical students are not benefitted by submitting an unlimited number of applications can be explained by game theory and the prisoner's dilemma model. Consider 2 highly qualified students, A and B, applying for a residency position in orthopedics. If medical student A and medical student B submit 20 applications, they have an equal chance of matching. If medical student A submits 40 applications while medical student B submits 20 applications, medical student A will have an increased chance of matching over medical student B (and vice versa). If medical student A and medical student B each submit 40 applications, they have the same chance of matching, yet they incur the added cost of the 20 additional applications. Using this model, it would appear that these medical students would be served by an application limit.
The idea of limiting the number of residency applications is not new. We surmise that the tiered ERAS application pricing system was used to discourage the overzealous submission of applications. ERAS charges $92 for the initial 10 applications, $9 each for applications 11 to 20, $15 each for applications 21 to 30, and $25 each for applications 31 or more.18 However, it is unlikely that most medical students will let a $25 application fee stand in the way. Thus, the current pricing system seems to be a poor deterrent. Although increasing the ERAS prices for submitting additional applications would deter applicants from overapplying, wealthy applicants would be able to out-apply poorer applicants, and individual resources would become a factor in match success. A more effective and equitable method for limiting the high number of applications would be to impose an application limit.
The match system algorithm20 was designed by recent Nobel Prize winners Roth and Shapley to facilitate the creation of matches that optimize the outcome for both applicants and programs. In this context, an evaluation of the current application system is important. For medical students, these additional applications add significant costs to the application process.21 For program directors who already have limited time,22 the added applications add a significant time commitment. Most important, the high number of applications may cause congestion and cause program directors to overlook preferred applicants as a result of the large number of applications. Anecdotally, this phenomenon is best illustrated when we hear of top-notch applicants not receiving interview offers from less competitive programs, by program directors noting that, given the large number of applications, they are forced to interview applicants they perceive are interested in their program as opposed to the best applicants.
Our study has limitations, including the use of only aggregate data. We thus do not know if certain applicants (ie, American Osteopathic Association, higher US Medical Licensing Examination scores) are benefitted by additional applications, nor do we know which applicants (ie, poor, strong, average) are submitting a higher number of applications. Our data pertain only to US allopathic graduates and should not be generalized to other groups. Additionally, data are not available that can be used to assess the effect that additional applications have on rank lists.
The current system is not efficient, and in certain specialties (otolaryngology and orthopedic surgery), applicants are on average applying to nearly half of the available residency programs. This suggests an ineffective utilization of the match. Having said that, our study does not offer sufficient information to suggest an application limit to the match, nor do we know the level at which such a limit should be set. It may, however, serve as the groundwork for further research to assess the utility of an application limit.
Conclusion
Despite an increase in the number of applications submitted, there is no associated increase in the match rate, suggesting that applicants may be done a disservice by a system where they can submit an unlimited number of applications. Research is needed to assess if these additional applications have led to an improvement in applicants' rank list outcomes, and to assess how these additional applications have affected applicants of varying quality.
Footnotes
Steven J. Weissbart, MD, is Chief Resident, Department of Urology, Icahn School of Medicine at Mount Sinai; Soo Jeong Kim, BA, is a Medical Student, Icahn School of Medicine at Mount Sinai; Richard S. Feinn, PhD, is Assistant Professor of Medical Sciences, Department of Medical Science, Frank H. Netter, MD, School of Medicine, Quinnipiac University; and Jeffrey A. Stock, MD, is Associate Clinical Professor of Urology, Department of Urology, Icahn School of Medicine at Mount Sinai.
Funding: The authors report no external funding source for this study.
Conflict of interest: The authors declare they have no competing interests.
References
- 1.Stain SC, Hiatt JR, Ata A, Ashley SW, Roggin KK, Potts JR, et al. Characteristics of highly ranked applicants to general surgery residency programs. JAMA Surg. 2013;148(5):413–417. doi: 10.1001/jamasurg.2013.180. [DOI] [PubMed] [Google Scholar]
- 2.Aneja S, Wilson LD, Haffty BG, Yu JB. National Residency Matching Program (NRMP) results for radiation oncology: 2011 update. Int J Radiat Oncol Biol Phys. 2012;83(3):771–772. doi: 10.1016/j.ijrobp.2012.01.073. [DOI] [PubMed] [Google Scholar]
- 3.Harper JG, Given KS, Pettitt B, Losken A. The independent plastic surgery match: an in-depth analysis of the applicants and process. Ann Plast Surg. 2011;66(5):568–571. doi: 10.1097/SAP.0b013e31820b3cea. [DOI] [PubMed] [Google Scholar]
- 4.Stratman EJ, Ness RM. Factors associated with successful matching to dermatology residency programs by reapplicants and other applicants who previously graduated from medical school. Arch Dermatol. 2011;147(2):196–202. doi: 10.1001/archdermatol.2010.303. [DOI] [PubMed] [Google Scholar]
- 5.de Oliveira GS, Jr, Akikwala T, Kendall MC, Fitzgerald PC, Sullivan JT, Zell C, et al. Factors affecting admission to anesthesiology residency in the United States: choosing the future of our specialty. Anesthesiology. 2012;117(2):243–251. doi: 10.1097/ALN.0b013e31825fb04b. [DOI] [PubMed] [Google Scholar]
- 6.Jeffrey R. The Logic of Decision. New York, NY: Gordon and Breach; 1965. [Google Scholar]
- 7.Kuhn S. The Stanford Encyclopedia of Philosophy. 2014. Prisoner's dilemma. http://plato.stanford.edu/archives/fall2014/entries/prisoner-dilemma/. Accessed November 1, 2014. [Google Scholar]
- 8.Weissbart SJ, Hall SJ, Fultz BR, Stock JA. The urology match as a prisoner's dilemma: a game theory perspective. Urology. 2013;82(4):791–798. doi: 10.1016/j.urology.2013.04.061. [DOI] [PubMed] [Google Scholar]
- 9.National Resident Matching Program. Match data 2014. http://www.nrmp.org/match-data/. Accessed October 25, 2014. [Google Scholar]
- 10.Association of American Medical Colleges. Electronic Residency Application Service (ERAS) data. https://www.aamc.org/data/facts/erasmdphd/. Accessed October 25, 2014. [DOI] [PubMed] [Google Scholar]
- 11.Association of American Medical Colleges. Table 41: residency applicants of US medical schools by specialty, 2009–2014. https://www.aamc.org/download/321564/data/factstable41.pdf. Accessed January 7, 2015. [Google Scholar]
- 12.National Resident Matching Program. Results and Data: 2008 Main Residency Match. Washington, DC: National Resident Matching Program; 2008. [Google Scholar]
- 13.National Resident Matching Program. Results and Data: 2009 Main Residency Match. Washington, DC: National Resident Matching Program; 2009. [Google Scholar]
- 14.National Resident Matching Program. Results and Data: 2010 Main Residency Match. Washington, DC: National Resident Matching Program; 2010. [Google Scholar]
- 15.National Resident Matching Program. Results and Data: 2011 Main Residency Match. Washington, DC: National Resident Matching Program; 2011. [Google Scholar]
- 16.National Resident Matching Program. Results and Data: 2012 Main Residency Match. Washington, DC: National Resident Matching Program; 2012. [Google Scholar]
- 17.National Resident Matching Program. Results and Data: 2013 Main Residency Match. Washington, DC: National Resident Matching Program; 2013. [Google Scholar]
- 18.Association of American Medical Colleges. The cost of applying for medical residency. 2013. https://www.aamc.org/download/94416/data/applyingformedicalresidency.pdf. Accessed October 23, 2014. [Google Scholar]
- 19.Association of American Medical Colleges. 2008. Table 34: residency applicants of US medical school graduates by specialty, 2001–2008. https://www.aamc.org. Accessed January 7, 2015. [Google Scholar]
- 20.National Resident Matching Program. How the matching algorithm works. http://www.nrmp.org/match-process/match-algorithm/. Accessed October 23, 2014. [Google Scholar]
- 21.Steinbrook R. Medical student debt—is there a limit. N Engl J Med. 2008;359(25):2629–2632. doi: 10.1056/NEJMp0808520. [DOI] [PubMed] [Google Scholar]
- 22.Malik MU, Diaz Voss Varela DA, Stewart CM, Laeeq K, Yenokyan G, Francis HW, et al. Barriers to implementing the ACGME Outcome Project: a systematic review of program director surveys. J Grad Med Educ. 2012;4(4):425–433. doi: 10.4300/JGME-D-11-00222.1. [DOI] [PMC free article] [PubMed] [Google Scholar]