Use of Filters for Residency Application Review: Results From the Internal Medicine In-Training Examination Program Director Survey

Abstract

Background

The increase in applications to residency programs, known as “application inflation,” creates challenges for program directors (PDs). Prior studies have shown that internal medicine (IM) PDs utilize criteria, such as United States Medical Licensing Examination (USMLE) Step examination performance, when reviewing applications. However, little is known about how early these filters are utilized in the application review cycle.

Objective

This study sought to assess the frequency and types of filters utilized by IM PDs during initial residency application screening and prior to more in-depth application review.

Methods

A web-based request for the 2016 Internal Medicine In-Training Examination (IM-ITE) PD Survey was sent to IM PDs. Responses from this survey were analyzed, excluding non-US programs.

Results

With a 50% response rate (214 of 424), IM PDs responded that the most commonly used data points to filter applicants prior to in-depth application review were the USMLE Step 2 Clinical Knowledge score (32%, 67 of 208), USMLE Step 1 score (24%, 50 of 208), and medical school attended (12%, 25 of 208). Over half of US IM PD respondents (55%, 114 of 208) indicated that they list qualifying interview criteria on their program website, and 31% of respondents (50 of 160) indicated that more than half of their applicant pool does not meet the program's specified interview criteria.

Conclusions

Results from the 2016 IM-ITE PD Survey indicate many IM PDs use filters for initial application screening, and that these filters, when available to applicants, do not affect many applicants' decisions to apply.

Introduction

The rising number of applications submitted in the Electronic Residency Application Service continues to outpace the more gradual growth of internal medicine (IM) applicants, contributing to the phenomenon known as “application inflation.”¹ As the largest specialty, IM experiences the highest volume of applications, averaging over 3000 applicants per categorical program.²

The downstream impact of application inflation creates challenges for IM program directors (PDs), who face real-world resource constraints in managing increased application volumes. One strategy for managing application inflation is the utilization of screening filters. Although programs' use of filters predates application inflation,^3,4 screening practices are not well understood due to a lack of data about temporal use of filters. While past studies have reported that over 89% of IM PDs since 2014 used both United States Medical Licensing Examination (USMLE) Step 1 and Step 2 as selection factors for interviews^5,6 and over 70% have a Step 1 target score,⁷ no previous studies of IM residency programs have identified which filters are commonly used for initial application screening. We sought to understand application filtering practices during the initial review period, prior to in-depth application review.

Methods

A workgroup of the Alliance for Academic Internal Medicine Medical Student to Resident Interface Committee provided questions on the 2016 Internal Medicine In-Training Examination (IM-ITE) PD Survey (provided as online supplemental material). Administered by the American College of Physicians, the web-based survey request was sent to 474 IM PDs whose programs participated in the 2016 IM-ITE. This survey was selected due to survey timing and the IM PD target population. Subsequent analysis excluded non–US program responses (n = 50). The survey period was October 2016 through April 2017. Seven e-mail reminders were sent to nonrespondents.

The survey asked IM PDs to choose the most common data point used to screen applicants prior to an in-depth application review. In addition, IM PDs indicated whether interview criteria appeared on their program websites and estimated the percentage of program applicants that did not meet these criteria.

This survey is exempt from human subjects review by the University of Connecticut School of Medicine Institutional Review Board.

Results

A total of 214 of 424 US IM PDs completed the survey for an overall response rate of 50%. Six respondents indicated that they did not want their responses included in research shared in scholarly presentations and/or publications and were excluded from analysis. When respondents were asked about the single most common data point used to filter applicants prior to beginning in-depth review of individual applicants, the 3 most common responses were the USMLE Step 2 Clinical Knowledge score (32%, 67 of 208), the USMLE Step 1 score (24%, 50 of 208), and medical school attended (12%, 25 of 208; Table 1). Fifteen percent of respondents (31 of 208) reported that they do not use filters or criteria to sort applicants prior to reviewing all applications.

Table 1.

Single Most Commonly Used Filter for Initial Application Review^a

Criterion	US IM PD Respondents Using Criterion to Filter (n = 208)
USMLE Step 2 CK score	32
USMLE Step 1 score	24
Medical school	12
USMLE Step 2 CS failure, first attempt	9
Failing grade on medicine clerkship	5
Class standing on MSPE	2
Failing grade in other clerkships	1
Lack of AOA membership	0
We do not use filters	15

Abbreviations: IM, internal medicine; PD, program director; USMLE, United States Medical Licensing Examination; CK, clinical knowledge; CS, clinical skills; MSPE, Medical Student Performance Evaluation; AOA, Alpha Omega Alpha.

^aSurvey question: “If you use criteria/filters to sort out applicants in your pool prior to beginning in-depth review of individual applicants, which of the following data points is the single most common data point used to sort/filter applicants?”

The majority of US IM PDs who completed the survey (55%, 114 of 208) indicated that they list qualifying criteria on their program websites to help applicants determine whether they qualify for an interview. Despite this, approximately one-third (31%, 50 of 160) of IM PD respondents estimated that more than half of their applicants do not meet this specified interview criteria (Table 2).

Table 2.

Applicants Not Meeting Disclosed IM PD Criteria for Interview Invitation^a

% of Applicants Not Meeting Criteria	US IM PD Respondents (n = 160), No. (%)
< 25	50 (31)
25–50	60 (38)
51–75	33 (21)
> 75	17 (11)

Abbreviations: IM, internal medicine; PD, program director.

^an = 48 of 208 (23%) of US IM PDs responded they do not disclose criteria on webpage.

Discussion

Results from the IM-ITE PD Survey suggest that many IM PDs use filters during initial application screening prior to in-depth review. Of the choices provided, the most commonly utilized filters were USMLE Step examination scores. Survey respondents reported using Step 2 Clinical Knowledge scores more often than Step 1 scores and that applicants did not appear to be deterred from applying to programs that post qualifying criteria for interviews.

These findings align with prior reports³ noting the importance of USMLE scores as selection criteria for residency. The use of filters to manage application inflation has raised concerns about the lack of correlation of USMLE scores with future resident performance.^8,9 A recent observational study¹⁰ did demonstrate a weak association linking higher Step 2 Clinical Knowledge scores to lower risk of future disciplinary action by a medical board. However, using high-stakes test scores for purposes other than those for which they are designed may result in undesired consequences.¹¹ These include influencing medical students to place even more time and effort on achieving the highest possible USMLE scores rather than focusing on clinical care, team, and communication skills. Furthermore, use of test scores for initial review may introduce racial and socioeconomic biases into the selection process.^12,13 Yet our findings demonstrate that a large number of IM PDs are using test scores as initial filters.

Finally, 15% (31 of 208) of IM PD respondents noted that they do not use any filters, which was higher than we anticipated. We assume this group holistically reviews their applications before making a decision regarding interview invitations. Without knowing the individual program characteristics of respondents, it is difficult to ascertain how this subgroup is able to handle their application burden. A follow-up survey of this group may help to understand how they approach application review in the era of application inflation.

This study is limited in that our initial filter options were not based on an open-ended survey of IM PDs; therefore, we could not include all the filters that programs currently use or elicit additional details to responses, including specific information programs provide as qualifying criteria on their websites. With a 50% response rate and an inability to compare respondents to nonrespondents, we do not know if the responses are representative of all IM programs. In addition, the survey lacks response testing; thus respondents may have interpreted questions differently than we intended. Finally, our study analysis is limited to responses from US IM PDs only, and we are unable to determine whether our findings could be generalizable to the wider graduate medical education community.

As the current national debate regarding changing some or all of the USMLE score reports to pass/fail continues, it is important to determine through additional quantitative and qualitative studies the strategies IM PDs and other PDs plan to use in their place and whether these strategies will enhance or deter more holistic application review in the future.

Conclusions

Our results suggest many IM PDs use filters for initial residency application screening prior to more in-depth review. These filters, when available to applicants, do not affect many applicants' decisions to apply to residency programs.