Use of the Interview in Resident Candidate Selection: A Review of the Literature

Abstract

Background

Although the resident candidate interview is costly and time-consuming for both applicants and programs, it is considered critically important for resident selection. Noncognitive attributes, including communication skills and professionalism, can be assessed by the personal interview.

Objective

We conducted a review of the literature on the residency interview to identify the interview characteristics used for resident selection and to ascertain to what extent the interview yields information that predicts future performance.

Methods

We searched PubMed and Scopus using the following search terms: residency, internship, interview, selection, and performance. We extracted information on characteristics of the interview process, including type of interview format, measures taken to minimize bias by interviewers, and testing of other clinical/surgical skills.

Results

We identified 104 studies that pertained to the resident selection interview, with highly varied interview formats and assessment tools. A positive correlation was demonstrated between a medical school academic record and the interview, especially for unblinded interview formats. A total of 34 studies attempted to correlate interview score with performance in residency, with mixed results. We also identified a number of studies that included personality testing, clinical skills testing, or surgical skills testing.

Conclusions

Our review identified a wide variety of approaches to the selection interview and a range of factors that have been studied to assess its effectiveness. More research needs to be done not only to address and ascertain appropriate interview formats that predict positive performance in residency, but also to determine interview factors that can predict both residents' “success” and program attrition.

Editor's Note: The online version of this article contains a list of the data extracted for each article.

Introduction

The year 2014 marked a record high of 26 678 postgraduate year (PGY)-1 positions offered in the United States. To achieve a Match rate of 96%, 17 374 US medical school seniors ranked a median of 11.5 programs, and often ranked more than 1 specialty.¹ Residency programs screen large numbers of applications for a limited number of interview slots, and the residency selection process creates a significant expense for both applicants and programs. A recent survey of plastic surgery applicants found that interview costs ranged from $2,500 to $10,000,² excluding the academic “cost” of lost days of medical education. The cost to residency programs is multifactorial, with the mean cost of recruiting 1 PGY-1 position estimated at $9,899.³

The screening, interview, and ranking processes are critical, as applicant selection has enduring consequences for the programs. Ideally, an applicant should be a good “fit” for the program, with a high likelihood of success and a low likelihood of problems. Poor performance in residency may require remediation, and resident attrition has been reported to be as high as 22% to 27% in general surgery programs,^4–6 which can create significant workflow issues, reduce morale, and have a negative impact on future recruitment.

Academic qualifications of candidates are accessible via the electronic residency application service (ERAS) and include United States Medical Licensing Examination (USMLE) scores, clerkship and preclinical grades, and class rank. Although readily available, there is wide variation in grading, class rank, and academic honors among medical schools. Studies of the use of academic data in predicting future performance have produced mixed results. A recent meta-analysis of factors showed that examination-based selection strategies (eg, USMLE Step 1) had a strong positive association with in-service training examinations, whereas medical school grades had a less robust association with subjective outcomes such as resident performance evaluations.⁷ USMLE Part 1 may even have a negative correlation with future clinical performance and professionalism.⁸ Attributes such as leadership and professionalism are not assessable through standardized testing, but may be evident in a review of the curriculum vitae, personal statement, dean's letter, and letters of recommendation. A combination of cognitive and noncognitive factors was shown in 1979 by Keck et al⁹ to be the most predictive of postgraduate clinical performance compared with any individual variable, and it is unlikely that this paradigm has changed 36 years later.

The interview process allows for assessment of noncognitive factors, such as interpersonal and communication skills, maturity, interest in the field, dependability, and honesty.^10,11 Program directors reported that the interview is the most important factor in determining resident selection.^11–20 The interview may also identify negative applicant characteristics such as anxiety or aggression.²¹ Candidates value the interview process highly,^22,23 and cite the quality of interactions with the program director²⁴ and, most important, with the residents^25–27 during the interview as important in their decision making. Applicants value the interaction with residents during an interview to assess resident morale,²⁸ esprit de corps,²⁹ and program “red flags” as perceived by applicants.³⁰ Ultimately, program directors and residents have reported that their “gut feeling” is important in creating their respective rank lists.^29,31

The resident selection interview also has been criticized for its “dubious value,”³² due to the lack of a standardized approach and low interrater reliability.²⁷ During the interview, there is a strong potential for a “halo effect,” in which interviewers' prior knowledge about an applicant's academic record (grades, test scores) affect the outcome.³³ Additionally, the interview can be a venue for unethical questioning regarding applicants' marital status, reproductive plans, and health, especially by faculty without adequate training.^34–36

Given the high cost of the resident interview and its importance in resident selection, this review attempts to identify data-driven strategies to optimize resident interview processes. The objectives were (1) to identify interview characteristics utilized by residency programs to evaluate candidates for selection; (2) to establish a relationship between the interview score and applicant characteristics and rank-order position in blinded and unblinded interview formats; and (3) to examine associations between applicant interviews and trainee/physician performance.

Methods

Identifying and reviewing articles that met inclusion criteria involved 2 phases. During the first phase, we conducted a pilot search of articles to develop a data abstraction tool. This initial search yielded 107 articles in PubMed, using medical subject heading (MeSH) terms such as internship and residency, interviews as topic/methods, and personnel selection covering the period between 1966 and 2013. The tool was developed by a research team member (S.K.), with 1 revision after the pilot phase. A kappa value of 0.80 (for accuracy of data abstraction) was attained across 3 research team members (S.K., S.O., A.M.) who reviewed 5 articles. Subsequently, the team reviewed all 30 articles and completed the data abstraction tool with the addition of 2 added researchers (B.H., S.C.). A list of the data extracted for each article is provided as online supplemental material.

The second review phase involved an expanded search of articles with consultation from a health sciences librarian. A search for English-language articles from 1966 through October 2014 in PubMed used the following terms: (1) internship and residency (MeSH terms), (2) interview, and (3) selection or performance. The search strategy was repeated in Scopus, and we also reviewed references of identified articles. Figure 1 illustrates the article selection process. A full-text review of 264 articles was performed, which included the initial 30 studies. We excluded review articles, surveys of program directors or applicants, letters to the editor, and commentaries. Forty-seven additional articles focused on other aspects of resident selection and did not include data on the residency interview. A total of 104 articles contained program-level data on the interview processes for resident selection or as a predictor of future performance. The lead author (A.S-F.) completed the review of all 104 articles using the review process established during phase 1. Data analysis was performed using Stata version 13 (StataCorp LP) to generate descriptive statistics.

FIGURE 1.

Selection Process for Included Studies

For the type of interview structure, we defined traditional or unstructured interviewing as the use of academic criteria and curricula vitae to generate questions that the interviewer determined to be relevant to the applicants' credentials. In contrast, structured interviews used predefined questions established by the selection committee prior to the interview and posed to all applicants. Behavioral interviews (also referred to as accomplishment interviews) are a type of structured interview in which applicants are asked to describe a past experience to provide an example of when they demonstrated characteristics desirable to the residency program. Multiple mini interviews include a timed circuit of stations to assess skills, including interpersonal communication, professionalism, and ethics.

Results

Interview Characteristics

Of the 104 articles, 91 (88%) were from the United States or Canada; the remainder were from the United Kingdom (n = 8), Australia, Ireland, Thailand, and the United Arab Emirates. The specialties studied in the reviewed articles are shown in figure 2. Table 1 reports the key results of studies. Studies were predominantly single institution studies. Less than half (46%, 48 of 104) of the articles reported the interview format. Of the studies that provided data on the interview format, 54% (26 of 48) described using web- or video-based skills testing or multiple mini formats, 31% (15 of 48) reported a structured (including behavioral) interview, and 15% (7 of 48) used a traditional or unstructured format.

FIGURE 2.

Distribution of Specialties

TABLE 1.

Characteristics of Interviews (n = 104)

We found that most studies lacked substantial information regarding the scope and the format of the interview, the steps taken to reduce possible interview biases, and the psychometric evidence associated with the interview instrument, such as reliability and validity. Only one-third of studies tracked resident performance over time. The type and number of interviewers were not consistently reported. For studies that included this information, in addition to physician faculty, interviewers comprised of residents (n = 20), PhD faculty (n = 2), psychologists (n = 3), or other health professionals (n = 4).

Correlational Studies

Studies frequently reported on the relationship of the interview score with academic criteria (USMLE, transcript, class rank, Alpha Omega Alpha Honor Medical Society, research publications), final rank, or match results. The cognitive and noncognitive criteria most commonly cited in studies as components of the final rank are shown in table 2. Many studies demonstrated a positive correlation among interview score, academic data, and final rank.^32,37–43 One study found that the interview could so closely correlate with academic variables that the R² values for final rank and preinterview academic rank were unchanged whether the interview score was added to the model or not.⁴⁴ Three studies showed that the interview significantly changed the rank of some applicants, moving them both higher and lower (> 10 positions) than their preinterview rank.^37,38,45 In a study using an interview format based on the CanMEDS competency-based framework (Communicator, Collaborator, Manager, Professional, Health Advocate, Medical Expert, Scholar), poor correlation was found between the interview and academic record, suggesting the interview was measuring factors other than academic performance.⁴⁶

TABLE 2.

Selection Criteria Reported by 55 Studies (53%)

Blinding Interviewers

Studies that evaluated the effect of blinding of interviewers to reduce bias consistently showed that an open or unblinded file resulted in higher interviewer ratings.^{33,42,47–49} Unblinded interviews correlated more closely to USMLE Step 1 scores,^33,42,48 whereas 2 studies where interviewers were blinded to the applicant's academic record found a negative correlation with USMLE scores.^33,42 One study found that 30% of the variance in interview scores was due to interviewer bias and concluded that unblinded interview scores were influenced significantly by other variables (USMLE scores, transcript, class rank, Alpha Omega Alpha), minimizing the weight of the interview in resident selection.⁴⁷

Performance

Thirty-four studies attempted to evaluate whether the interview predicted performance. Outcomes studied included (1) clinical evaluations during residency by program directors, faculty, and/or chief residents; (2) global evaluation or ranking of residents; (3) in-training examination results; (4) attrition; or (5) problems with professionalism in residency or subsequently in their careers. The duration of follow-up ranged from 7 to 12 months for those outcomes that were assessed during the course of internship,^50–53 and from 36 to 60 months for outcomes in residency. Results from these studies varied. Of the 34 studies (table 3), 17 showed that the interview did not predict subsequent clinical performance (weak positive, negative, or no correlation) in internship or residency, particularly with a traditional or unstructured interview format.^{41,50,53–67} Eleven studies showed a positive correlation between interview and subsequent performance.^{8,51,52,68–75} Performance metrics with a positive correlation included clinical evaluations, in-training examinations, licensing board examinations, and a composite score or rank of resident performance. Attrition from residency was not consistently predicted by the interview process in the 6 studies that assessed this variable.^{4–6,43,55,56} In a case control study of psychiatry residents, minor or major problems during residency may have only been associated with negative comments in the dean's letter and were not picked up in the interview.⁷⁶ A longitudinal study from 1965 to 1999 showed that neither the residency interview nor the performance evaluations in a psychiatry program predicted long-term professionalism issues, resulting in referral to an impaired physician program.⁷⁷

TABLE 3.

Prediction of Performance

The behavioral or accomplishment interview was more predictive of subsequent residency performance evaluation than the traditional interview or other academic variables.⁵⁴ Although 1 study did not find a correlation between the overall behavioral interview score and resident performance, a subscore for suitability/trainability was negatively associated with attrition.⁵⁵ In another study, components of the accomplishment interview correlated with specific resident behaviors such as confidence with interpersonal skills (r = 0.38) or recognition of personal limits (r = −0.46).⁷⁵

Clinical and Surgical Skills

The 7 studies that included an assessment of surgical skills as part of the interview or selection process showed mixed results.^78–84 A soap carving task as part of an otolaryngology residency interview was not predictive of cognitive knowledge, visuospatial ability, manual dexterity, decision making, or overall resident performance.⁸² The use of surgical skills testing at an otolaryngology residency interview did not correlate with USMLE Step 1 scores,⁷⁸ although in a larger cohort, it was predictive of faculty performance evaluation.⁷⁹

Assessment of clinical skills in the form of an objective structured clinical examination or a multiple mini interview (MMI) to assess competencies desired by the program has been described for individual programs and regional assessment centers for selection of residents. Many of the studies examining validity evidence for the MMIs or multiple station interviews were identified in this review (MMI^70,85–91 and competency center^71,92–98). Several studies showed a positive correlation between multiple station interviews and future clinical examination testing.^70,71,95 A study of the use of the MMI for emergency medicine interns at 3 institutions found that the results did not correlate with clerkship grades, medical school quality, or USMLE scores, but did correlate with emergency medicine clerkship grades. This suggests that the MMI may measure abilities that contribute to success in the specialty.⁹¹

Personality Testing

Eight studies explored the use of personality^99–103 or emotional intelligence^104–106 testing as part of resident selection. Use of the Myers-Briggs personality inventory for interviewers and applicants demonstrated that clinician faculty ranked candidates more favorably when they shared certain personality styles.¹⁰³ Bohm et al¹⁰⁶ evaluated a moral reasoning assessment tool in orthopedic resident applicants and found no association between moral reasoning and resident rank or USMLE scores.

Discussion

Our comprehensive review shows the complexity of the interview process in resident selection. The scope of this review, which included a broad range of medical and surgical specialties, provides an overview of the diverse characteristics of the interview in historical and contemporary resident selection. It suggests that the interview process varies greatly among programs and specialties, with regard to the assessment of interview format, logistics, characteristics of interviewers, and noncognitive skills.

Although 34 studies attempted to establish a relationship between the interview and future performance, the results are mixed as to whether the interview itself, or the interview combined with other data used in the application, predicted future “success” or “problems” for residency applicants. It was not possible to determine whether a specific interview format is superior for predicting performance.

It would be informative if selection strategies could predict subsequent resident performance, but quality data are limited and mixed. This is partly due to the lack of useful performance outcomes and limited power in small studies. Thus, longitudinal and multi-institution studies of performance during and after residency with clinically meaningful performance measures are needed to fully assess the predictability of various interview techniques or academic records. With improved specialty-specific trainee assessment tools, program directors may have a common language to assess overall performance and design interview processes that could predict more long-term physician success.

Gordon and Lincoln²⁷ criticized the traditional interview for poor inter-interviewer reliability, content that varies from applicant to applicant, interviewer bias from stereotypes or idiosyncratic rapport, and the fact that questions may be anticipated or rehearsed by the applicant, resulting in a skewed view of an applicant's values and motivation. The box shows a series of recommendations to improve the reliability and value of the resident interview as a selection tool.

box Attributes of the Resident Interview That Improve Reliability

• 1Explicit written description of the desired traits in an applicant/resident
• 2Standardized questions to every applicant
• 3Provision of behavior-specific anchors for rating scales for interviewers and using a scoring rubric to improve interrater and intrarater scoring
• 4Use of multiple observers rather than a single interviewer
• 5Training of interviewers in the format and scoring and including unethical and “illegal” question rules
• 6Blinding of the interviewer to cognitive application data to minimize bias¹⁰⁷

In the future, if the regional interview fair¹⁰⁸ or web-based conference interview^109,110 has additional evidence of validity, this may diminish the expense of interviewing to both programs and applicants. The major drawback will be the lack of interactions with resident and staff that an onsite interview provides.

This review has 2 limitations. The first is the heterogeneity of the source articles, which do not consistently report many of the variables of interest to the authors, limiting the ability to make interpretation of findings across studies. Additionally, a single author reviewed all included studies, which could have introduced bias in the data abstraction and conclusions raised in this study.

Conclusion

The interview in resident selection often is used to assess noncognitive factors and can provide information to assess the mutual fit of the applicant and the program. Ideally, the interview contributes to the final rank list by systematically assessing specialty-specific personal qualities, skills, and competencies. However, the selection interview does not predict clinical performance, problems with professionalism, or resident attrition. The predictive value of the interview may be limited due to “halo effects,” as interviewers base their assessment of candidates on academic criteria, especially in unblinded formats. Although there is insufficient evidence to recommend an optimal interview format to predict future performance, use of the unstructured, unblinded interview should be replaced with a more rigorous interview strategy.