What Can We Learn From Resident Selection Interviews?

In this issue of the Journal of Graduate Medical Education, Stephenson-Famy and colleagues¹ report the findings of their literature review on the use of interviews in resident selection. Through demonstrating the generalized use of interviews across specialties and program types, the authors highlight issues common to the selection of residents across all specialties. Medical educators should view this type of review as an opportunity to consider both the results of the individual studies and what the aggregate conclusions might mean for the practice of resident selection.

As has been recognized through the creation of faculty development programs,² Masters in Health Professions Education,³ and expectations by our colleagues,⁴ medical education contains a body of theory and expertise that requires focused effort to master skills beyond what is commonly taught in medical school. We have moved past the idea that being a good physician automatically makes you a good teacher,⁵ but as a community we must continue to ask hard questions of ourselves and the larger medical profession if we are going to succeed in an area of specialized knowledge and practice. The article by Stephenson-Famy et al¹ helped me to crystalize a number of ideas that have been persistent in my thinking about the resident interview process and the future of medical education. In this commentary, I propose the following talking points derived from their article for consideration.

Graduate medical education (GME) is costly, and successful resident selection is very important. The US government invests more than $9 billion to support the direct and indirect costs of GME.⁶ As referenced by the authors, costs for selection and recruitment to a residency also are high, with 1 study reporting a median annual resident recruitment cost of about $148,000 for internal medicine residency programs and a median cost of $9,900 per resident slot per year.⁷ There is a reason why residency programs and program directors are willing to spend almost $10,000 per resident in recruitment: residents who fail out of programs or are disruptive to others are even more costly.⁸ In addition to a lack of return on the individual portion of the $9 billion total spent on a failed resident's training, there are additional costs to the resident, the program, and program leadership. These additional costs include opportunity costs to others who might have otherwise trained in the residency program; opportunity costs due to failed training of the resident who faces dismissal, program transfer, or a new career path; time and financial costs related to possible litigation; and reputation and goodwill costs to the program, its faculty, and the other residents when dismissal or frequent remediation becomes public knowledge.⁸ Given the large sums of money involved in resident selection and training entrusted to graduate medical educators, we must earnestly try to use these resources responsibly and in an evidence-directed fashion.

The resident interview process may not be as useful as is generally believed and appears to have not been adequately studied to date. As described by the authors, the current resident interview process is generally seen as very important in making rank list decisions for both program directors and applicants. Stephenson-Famy et al¹ reported a number of consistent attitudes toward the resident interview. Program directors use the interview process to assess applicants for noncognitive skills and “red flags,”⁹ whereas applicants value the opportunity to interact with faculty and residents and assess the degree of programmatic and social fit for themselves.¹⁰

Despite the expressed beliefs by both program directors and residents that the resident interview provides critical information about the selection process, this perception is not supported by the available literature. Instead, the literature shows a lack of rigorous study of the issue, and the limited available evidence does not demonstrate that residency selection interviews provide the benefits expressed by program directors or resident applicants. The majority of the studies included in the literature review report data from single institution studies. The resulting underpowered studies and the reduced ability to generalize based on the findings make each of the studies less robust. However, the type of aggregation the authors conducted is useful for determining the impact of the selection interview, including whether programs and institutions obtain a return on the considerable expenditures associated with the resident interview visit.

In addition, the methodology described in a large number of the studies was lacking in specificity regarding interview methods, bias mitigation, or psychometric evidence of the interview instruments. With regard to the benefits of the interview process, a number of studies showed changes in rank list ascribed to the interview process, although 1 study¹¹ did not demonstrate a clear additional value of the interview beyond the use of preexisting academic metrics. This is perhaps not surprising, as a minimal demonstrable benefit of the interview has been described in medical school selection.¹² A more concerning possibility is that the interview may actually make the process less fair, as similar interviews in other professional disciplines have been accused of primarily replicating social and class stratification instead of the stated goal of fair selection.¹³ Finally, in evaluating the predictive ability of the interview by correlating it with residency training outcomes, Stephenson-Famy et al¹ showed, at best, mixed results for the 34 studies they reviewed. In my opinion, this is concerning because the most important benefit of the interview process should be to improve the specificity and sensitivity of the resident selection process with regard to learner outcomes and success.

Given the above concerns regarding the traditional interview process, it is not surprising that alternatives have been explored. The work my colleagues and I have previously completed using the multiple mini-interview (MMI) is 1 of the alternative approaches discussed in the review.¹⁴ There is some evidence that MMI performance correlates with the objective structured clinical examination in the first third of the intern year, making the MMI approach a possible useful adjunct in resident selection.¹⁵ A different novel interview process (CanMEDS) correlated in a negative fashion with academic metrics, but is thought to provide additional data for resident selection.¹⁶ Unfortunately, our continued work with the MMI was unable to show an increased predictive value of the MMI beyond the standard selection process for end of year 1 performance in emergency medicine residents, which confirms similar concerns as those for the traditional approach in predicting success in residency.¹⁷ Compounding the efficacy issue of the MMI (and likely other novel approaches) are problems in the varied reactions of prospective subjects to alternative interview approaches, which range from a positive¹⁸ to a mixed response.¹⁴ Finally, while the MMI uses institutional resources (such as faculty time) differently than the traditional selection interview process, the overall costs are comparable.¹⁹

The issues reported in the literature review on resident selection interviews are not insurmountable. We can all work together to improve the resident selection process and make the medical education research on this topic more rigorous. As suggested by Stephenson-Famy and colleagues,¹ there already are concrete ways, such as the adoption of standard interview practices, to ensure that residency selection interviews are more likely to provide reliable information to decision makers. Similarly, the research on the predictive value of the resident interview can be improved through more explicitly described methodology and correlation with meaningful outcomes. Medical education professionals can make a significant contribution through a renewed focus on principles such as these.