Abstract
Background
Over the past decade, the number of residency applications has increased substantially, causing many residency programs to change their recruitment practices.
Objective
We determined how internal medicine (IM) residency programs have responded to increased applications by program type (community-based, community-based/university-affiliated, and university-based) and characteristics (percentage of international medical graduates, program size, and program director [PD] tenure).
Methods
The Association of Program Directors in Internal Medicine conducted a national survey of 363 IM PDs in 2017. Five questions assessed IM program responses to the increased number of residency applications in 3 areas: changes in recruitment strategies, impact on ability to perform holistic review, and interest in 5 potential solutions. We performed a subgroup analysis to measure differences by program type and characteristics.
Results
The response rate was 64% (233 of 363). There were no differences by program type or characteristics for experiencing an increase in the number of applicants, altering recruitment practices, or conducting holistic reviews. There were moderate differences in alterations of recruitment practices by program characteristics and moderate differences in interest in proposed solutions by program type. Community-based programs had the greatest interest in a program-specific statement (59%, P = .032) and the lowest percentage in a national database of matched applicants (44%, P = .034).
Conclusions
IM residency programs are experiencing an increasing number of applications and are accommodating by adjusting recruitment practices in a variety of ways. A majority of IM PDs supported 4 of the 5 solutions, although the level of interest differed by program type.
Introduction
The average number of applications received by residency programs across disciplines has increased 71% since 2007.1 The overall 2019 postgraduate year 1 Match rate is 80%, the highest since 1993.2 Applicants are submitting more applications than needed to secure a first-year residency position.3 Simultaneously, many specialties report increased competition to fill positions, with the average number of ranked applicants increasing from 2015 to 2019.2 The reasons for the increase in applications are complex and driven by applicants, medical school advisors, and residency programs.4
This increase is changing how residency programs screen applicants.4 In a national survey of internal medicine (IM) program directors (PDs), we found two-thirds of PDs adjusted their recruitment practices, with most inviting more applicants, raising the standards for whom to invite, and adding additional interview days to accommodate more applicants.5,6 When presented with potential solutions to address the increase in applications, the majority of PDs were most interested in limiting the number of applications each candidate is allowed to submit and asking them to indicate which programs were of high interest to them.5,6 However, residency programs differ from each other in multiple ways; there may be unique program attributes that influence how a program reacts to this increase.
In a follow-up to this study, we examined the associations between IM programs' changes to recruitment strategies, ability to perform holistic review, and interest in 5 potential solutions by program type (community-based, community-based/university-affiliated, and university-based) and program characteristics (percentage of international medical graduates [IMGs], size, and PD tenure).
Methods
The Association of Program Directors in Internal Medicine (APDIM), a charter organization of the Alliance for Academic Internal Medicine (AAIM), conducts a national survey of IM PDs on important issues facing the IM educational community. Five questions were selected for inclusion in the 2017 APDIM Spring Survey through a blinded, peer-review process to assess the effect of the increased number of residency applications, including recruitment adjustments and impact on holistic review (provided as online supplemental material). We assessed interest in 5 potential solutions that are based on a review of the literature, perceived cost, and ease of implementation.4,7–10 The survey was pilot-tested by the APDIM Survey Committee, members of the AAIM Resident to Fellow Interface Committee, and the AAIM Medical Student to Resident Interface Committee. All pilot testers were charged with reviewing the survey questions for construct validity, clarity of the questions, exhaustiveness of the response options, logic and flow of the survey, and cognitive load. Critical comments were resolved by the authors. In March 2017, the survey was deployed to 373 AAIM member programs accredited by the Accreditation Council for Graduate Medical Education (ACGME), representing 89% of 418 IM programs accredited prior to July 1, 2016. Nonrespondents received up to 4 e-mail reminders until survey closure in May 2017.
Prior to blinding program identity for analysis, responses were merged with publicly available third-party data, accessed in 2018. Data from the ACGME website were used to obtain PD tenure (years) and total number of filled positions (program size).11 Program type and 3-year mean percentage of IMGs (2015–2016) were obtained from the American Medical Association Fellowship and Residency Electronic Interactive Database Access system.12
The study protocol was granted exempt status by the Mayo Clinic Institutional Review Board.
We used Pearson's chi-square test to report P values for group-based comparisons involving 3 types of program characteristics. Welch's t test was used to compare respondents to nonrespondents and to respondents who selected a response option to those who did not. For programs that reported experiencing an increase in application volume and adjusted recruitment practices, Bartlett's test for equal variances was used to test for differences in self-reported likelihood of conducting holistic reviews and interest in potential solutions. Holistic review variables were collapsed for reporting and analysis: more likely (“more likely” and “much more likely”) and less likely (“less likely” and “much less likely”). “No opinion” responses were excluded from subsequent analysis. An alpha level of .05 was used to determine statistical significance. One author (C.M.W.) conducted data analysis with Stata SE 14.2 (StataCorp LLC, College Station, TX).
Results
After excluding 10 military and unknown program types, the overall response rate was 64% (233 of 363). There were no differences in respondents and nonrespondents for program type, program size, or PD tenure (Table 1). The mean IMG percentage was higher among nonrespondents (55%) than it was with respondents (44%, P < .001). Ninety-two percent (214 of 233) of respondents indicated that they had experienced an increase in the number of applications due to increased application volume. Forty-four percent (57 of 130) of respondents who conducted holistic review were “much less likely” or “less likely” to do so with no differences based on assessed program characteristics or types (Table 2). Eight programs reported not conducting holistic reviews regardless of application volume.
Table 1.
Comparison of Survey Respondents to Nonrespondents Based on Program Characteristic and Type (N = 363)
| Program Type | Total, n (%) | Respondents (n = 233), n (%) | Nonrespondents (n = 130), n (%) | P Valuea |
| Community-based | 42 (12) | 30 (13) | 12 (9) | .06 |
| Community-based, university-affiliated | 190 (52) | 111 (48) | 79 (61) | |
| University-based | 131 (36) | 92 (39) | 39 (30) | |
| Program Characteristic | Mean (SD) | Mean (95% CI) | Mean (95% CI) | P Valueb |
| IMG (2015–2016), % | 47.7 (35.1) | 43.7 (39.1–48.3) | 54.9 (48.8–61.0) | < .001c |
| Filled positions (ACGME) | 62.9 (39.1) | 70.1 (64.8–75.7) | 63.2 (56.9–69.4) | .14 |
| PD tenure, y | 6.5 (6.5) | 6.4 (5.5–7.2) | 6.7 (5.5–7.8) | .66 |
Abbreviations: CI, confidence interval; IMG, international medical graduate; ACGME, Accreditation Council for Graduate Medical Education; PD, program director.
Pearson's chi-square test used for categorical variables.
Welch's t test used for continuous variables.
P < .05.
Table 2.
Program Self-Reports: Increased Application Volume, Adjustments to Recruitment Practice, and Impact on Holistic Review
| Program Type | Experienced Increase in Applicationsa | Adjusted Recruitment Practicesb | Impact on Conducting Holistic Reviewc | |||||||
| Yes, n (%) | No, n (%) | P Valued | Yes, n (%) | No, n (%) | P Valued | More Likely,e n (%) | No Impact, n (%) | Less Likely,e n (%) | P Valued | |
| Community-based | 27 (90) | 3 (10) | .81 | 20 (74) | 7 (26) | .54 | 4 (20) | 7 (35) | 9 (45) | .96 |
| Community-based, university-affiliated | 101 (92) | 9 (8) | 64 (63) | 37 (37) | 8 (14) | 24 (41) | 26 (45) | |||
| University-based | 86 (93) | 6 (7) | 54 (63) | 32 (37) | 8 (15) | 22 (42) | 22 (42) | |||
| Total | 214 (92) | 18 (8) | 138 (64) | 76 (36) | 20 (15) | 53 (41) | 57 (44) | |||
| Program Characteristics | Mean (SD) | Mean (SD) | P Valuef | Mean (SD) | Mean (SD) | P Valuef | Mean (SD) | Mean (SD) | Mean (SD) | P Valueg |
| IMG (2015–2016), % | 43.9 (35.1) | 45.0 (37.8) | .86 | 39.7 (34.2) | 49.9 (35.8) | .045 | 44.2 (40.1 ) | 36.4 (34.1) | 39.1 (32.2) | .50 |
| Filled positions (ACGME) | 65.2 (40.6) | 66 (43.1) | .94 | 61.8 (39) | 71.4 (42.8) | .11 | 59 (45.2) | 63.8 (42.3) | 58.0 (34.3) | .19 |
| PD tenure, y | 6.7 (6.6) | 6.2 (6.9) | .90 | 6.7 (6.8) | 5.9 (6.3) | .38 | 7.2 (7.3) | 7.7 (7.7) | 5.3 (5.6) | .07 |
Abbreviations: IMG, international medical graduate; ACGME, Accreditation Council for Graduate Medical Education; PD, program director.
N = 232.
N = 214.
N = 130.
Pearson's chi-square test used for categorical variables.
Holistic review variables were collapsed for reporting and analysis: more likely (“more likely” and “much more likely”) and less likely (“less likely” and “much less likely”).
Welch's t test used for continuous variables.
Bartlett's test for equal variances.
Among the 64% (138 of 214) of respondents that altered recruitment, some recruitment strategies differed by program characteristics (Table provided as online supplemental material). Programs that indicated they invited more applicants had a lower mean IMG percentage (35% versus 48%, P < .004). Respondents from smaller programs indicated they had (1) hosted more applicants per day, and (2) conducted more telephone interviews than larger programs (mean, 54 versus 67, P = .031, and 26 versus 62, P < .001, respectively). The PDs who had been in their role for fewer years reported conducting more online and telephone interviews (mean, 2.5 versus 6.9, P < .001, and 1.4 versus 6.9, P < .001, respectively).
Table 3 shows the percentage of respondents reporting interest in the 5 proposed solutions. Interest in the creation of a national database of matched applicant data per program and a requirement for a program-specific personal statement differed based on program type. Community-based programs had the greatest interest in program-specific personal statements (59%, 17 of 29, P = .032) and the lowest percentage in an interest in a national database (44%, 12 of 27, P = .034). Interest in the proposed solutions did not differ based on program characteristics.
Table 3.
“Somewhat” and “Very Interested” Responses to Potential Solutions to Increasing Application Volume
| Program Typea,b | Limit Number of Applications | Indicate High Interest in Subset of Programs | Create National Database of Matched Applicant Data | National Invitation System | Program-Specific Personal Statement | |||||
| n (%) | P Value | n (%) | P Value | n (%) | P Value | n (%) | P Value | n (%) | P Value | |
| Community-based | 23/29 (79) | .66 | 21/29 (72) | .33 | 12/27 (44) | .034c | 15/27 (56) | .98 | 17/29 (59) | .032c |
| Community-based, university-affiliated | 70/98 (71) | 71/100 (71) | 55/103 (53) | 49/98 (50) | 37/93 (40) | |||||
| University-based | 70/88 (80) | 59/83 (71) | 54/87 (62) | 45/85 (53) | 27/87 (31) | |||||
| Total | 163/215 (76) | 151/212 (71) | 121/217 (56) | 109/210 (52) | 81/209 (39) | |||||
| Program Characteristicsb,d | Mean (SD) | P Value | Mean (SD) | P Value | Mean (SD) | P Value | Mean (SD) | P Value | Mean (SD) | P Value |
| IMG (2015–2016), % | 49.0 (39.4) | .40 | 43.4 (37.3) | .76 | 40.5 (34.1) | .75 | 41.5 (35.0) | .98 | 39.0 (36.2) | .34 |
| Filled positions (ACGME) | 61.8 (35.2) | .24 | 64.3 (44.9) | .47 | 59.6 (37.4) | .07 | 63.4 (41.0) | .97 | 72.9 (42.5) | .29 |
| PD tenure, y | 7.3 (5.9) | .30 | 7.3 (7.5) | .25 | 6.8 (6.5) | .25 | 7.0 (7.1) | .16 | 6.3 (6.4) | .05 |
Abbreviations: IMG, international medical graduate; ACGME, Accreditation Council for Graduate Medical Education; PD, program director.
Pearson's chi-square test used for categorical variables.
Analysis assessed differences based on 3 outcomes (“not at all interested,” “somewhat interested,” and “very interested”).
P < .05.
Bartlett's test for equal variances.
Discussion
This survey of a majority of US IM residency PDs found that nearly all are experiencing increased applications and are altering recruitment practices by, for example, increasing numbers of interviews, using web-based or telephone interviews, and raising standards. We determined that program responses differed by program type and program characteristics. Some programs are decreasing their use of holistic review. The majority of programs are interested in approaches to reduce the number of applications that require review, with differences by program type.
Programs with fewer IMGs more often adjusted recruitment in favor of the resource-heavy option of hosting more applicants. The US medical graduates are largely responsible for the increases in applications,13 thus potentially burdening programs with fewer IMGs. Smaller programs adjusted by hosting more applicants per interview day and conducting more telephone interviews.
Community-based programs reported the most interest in program-specific personal statements, which we speculate may highlight their desire to identify highly motivated applicants. Conversely, community-based programs were less interested in the creation of a national database of matched applicant data. We theorize some programs are hoping to attract applicants with greater academic achievements, but applicants may be less likely to apply if their achievements were above or below the typical matched student for that program. Despite variability in preferred solutions by program type, PDs shared a desire for a solution. One-fit solutions that encompass all specialties will pose an even greater challenge; numerous combined solutions may be required.
Other specialties are grappling with the increase in applications and seeking solutions. Otolaryngology implemented program-specific personal statements.14 Years ago, emergency medicine developed a standardized letter of evaluation, now the most important factor when selecting applicants for interviews.15 Others are addressing this from the medical student perspective, using a data-driven approach to counsel students to apply to appropriate programs.16,17 The Association of American Medical Colleges launched the Residency Explorer to allow medical students access to a national database of residency programs' matched applicants.18
Our study has limitations, including moderate overrepresentation of respondents from programs with fewer IMGs, possibly introducing nonresponse bias and confounding study conclusions by understating or overstating statistical significance. This survey included prepopulated, multiple-choice questions; the results may have been different had we included free-text responses. Other possible solutions have been noted in the literature10,19 but were not represented among the prepopulated responses; therefore, the study was not inclusive of all possible potential solutions that PDs could have reported. Although our survey response rate was over 60%, our survey population was not inclusive of all IM programs, meaning the results cannot be generalized to the entire population of IM residency programs.
The relationship between smaller programs and increased interview resources, such as direct expenses, education and service disruptions, and other indirect costs, needs to be examined further.13,20 The effects of a national database and the Residency Explorer resource on programs, particularly community-based programs, requires further study as well.18
Conclusions
This survey found that most IM residency programs are altering recruitment practices, and some programs are less likely to perform a holistic review. Although there is widespread support for potential solutions, IM programs express preferences for solutions that vary by program type.
Supplementary Material
References
- 1.Association of American Medical Colleges. ERAS national statistics: historical specialty specific data, residencies specialty summary, ERAS year 2011 and ERAS year 2018.
- 2.National Resident Matching Program. Results and data: 2019 main residency match. 2019 https://mk0nrmp3oyqui6wqfm.kinstacdn.com/wp-content/uploads/2019/04/NRMP-Results-and-Data-2019_04112019_final.pdf Accessed September 19.
- 3.Angus SV, Williams CM, Kwan B, Vu TR, Harris L, Muntz M, et al. Drivers of application inflation: a national survey of internal medicine residents. Am J Med. 2018;131(4):447–452. doi: 10.1016/j.amjmed.2018.01.002. [DOI] [PubMed] [Google Scholar]
- 4.Pereira AG, Chelminski PR, Chheda SG, Angus SV, Becker J, Chudgar SM, et al. Medical Student to Resident Interface Committee Workgroup on the Interview Season. Application inflation for internal medicine applicants in the match: drivers, consequences, and potential solutions. Am J Med. 2016;129(8):885–891. doi: 10.1016/j.amjmed.2016.04.001. [DOI] [PubMed] [Google Scholar]
- 5.Angus SV, Williams CM, Stewart E, Sweet ML, Kisielewski M, Willett LL. The current state of internal medicine residency recruitment: an analysis of program directors' perceptions and screening practices. Acad Med. In press. [DOI] [PubMed]
- 6.Alliance for Academic Internal Medicine. Response and solutions to application inflation: a national survey of internal medicine program directors. 2019 https://www.im.org/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=2d649ced-e4c9-1190-0656-7914f23a99e5&forceDialog=0 Accessed September 19.
- 7.Weissbart SJ, Kim SJ, Feinn RS, Stock JA. Relationship between the number of residency applications and the yearly match rate: time to start thinking about an application limit? J Grad Med Educ. 2015;7(1):81–85. doi: 10.4300/JGME-D-14-00270.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Puscas L, Esclamado R. Use of a secondary essay in the residency application process. JAMA Otolaryngol Head Neck Surg. 2015;141(7):591–592. doi: 10.1001/jamaoto.2015.0677. [DOI] [PubMed] [Google Scholar]
- 9.Gliatto P, Karani R. Viewpoint from 2 undergraduate medical education deans the residency application process: working well, needs fixing, or broken beyond repair? J Grad Med Educ. 2016;8(3):307–310. doi: 10.4300/JGME-D-16-00230.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Katsufrakis PJ, Uhler TA, Jones LD. The residency application process: pursuing improved outcomes through better understanding of the issues. Acad Med. 2016;91(11):1483–1487. doi: 10.1097/ACM.0000000000001411. [DOI] [PubMed] [Google Scholar]
- 11.Accreditation Council for Graduate Medical Education. Data collection systems. 2019 http://www.acgme.org/Data-Collection-Systems/Overview Accessed September 19.
- 12.American Medical Association. Match: FREIDA. 2019 https://www.ama-assn.org/residents-students/match/freida Accessed September 19.
- 13.Berger JS, Cioletti A. Viewpoint from 2 graduate medical education deans: application overload in the residency match process. J Grad Med Educ. 2016;8(3):317–321. doi: 10.4300/JGME-D-16-00239.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Stepan KO, Kaul VF, Raquib AR, Kozin ED, Sethi RK, Malkin BD, et al. An evaluation of the program—specific paragraph in the otolaryngology residency application. Laryngoscope. 2018;128(11):2508–2513. doi: 10.1002/lary.27553. [DOI] [PubMed] [Google Scholar]
- 15.Garmel GM, Grover CA, Quinn A, Lotfipour S, Langdorf MI, Cheaito MA, et al. Letters of recommendation. J Emerg Med. 2019] doi: 10.1016/j.jemermed.2019.04.020. [published online ahead of print July 30. [DOI] [PubMed]
- 16.Frayha N, Raczek J, Lo J, Martinez J, Parker D. An electronic interview tracking tool to guide medical students through the match: improvements in advising and match outcomes. Acad Med. 2019;94(3):348–352. doi: 10.1097/ACM.0000000000002522. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Liao NL, Mahan JD, Scherzer R. The pediatric match frenzy: an overview and an approach for mentoring medical students. Acad Pediatr. 2019] doi: 10.1016/j.acap.2019.06.007. [published online ahead of print June 12. [DOI] [PubMed]
- 18.Association of American Medical Colleges. Residency Explorer. 2019 https://www.residencyexplorer.org/Account/Login?ReturnUrl=%2F Accessed September 19.
- 19.Hueston WJ. A proposal to address the increasing number of residency applications [comment] Acad Med. 2017;92(7):896–897. doi: 10.1097/ACM.0000000000001740. [DOI] [PubMed] [Google Scholar]
- 20.Brummond A, Sefcik S, Halvorsen AJ, Chaudhry S, Arora V, Adams M, et al. Resident recruitment costs: a national survey of internal medicine program directors. Am J Med. 2013;126(7):646–653. doi: 10.1016/j.amjmed.2013.03.018. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.