Using structured interviews to reduce bias in emergency medicine residency recruitment: Worth a second look

INTRODUCTION

The in‐person interview is a critical component of resident recruitment and selection. For programs, it is one of the most important factors in consideration when forming a rank list.¹ Applicants and programs commit significant time and money to the interview process.², ³ In the authors’ collective experience, most emergency medicine (EM) residency interviews are unstructured. They typically consist of informal conversations on a variety of topics (e.g., hobbies, activities, and experiences shared by interviewer and applicant). This format has several drawbacks and may facilitate age, race, and gender bias.⁴ As a result, EM interviews do not consistently predict an applicant’s future job performance⁵ and potentially opens the door for interviewers to ask discriminatory and illegal questions.⁶

Alternatively, structured interviews aim to control multiple variables by using the same interviewers, job‐relevant questions, and anchored grading scales for every applicant.⁷ This approach has been shown to improve inter‐rater reliability, validity, and predictive value for future job performance.⁸ The structured interview can reduce race, age, and gender bias and increase workforce diversity.⁹, ¹⁰, ¹¹, ¹² It has become standard practice for recruitment in numerous industries ranging from leadership positions and police officers to customer service.¹³ Despite these potential benefits, EM residencies have been slow to implement structured interviews due to concern over applicants’ perceptions of them to be intimidating and difficult, leaving candidates feeling inadequate and more likely to rank a program lower.⁹, ¹⁴, ¹⁵, ¹⁶

In light of the limited use of structured multiple mini‐interviews (MMIs) and halted implementation of standardized video interviews (SVIs) in EM,¹⁵, ¹⁷, ¹⁸ we seek to share our perspective, backed by relevant hiring literature, to call for renewed consideration of structured interviews in EM. Specifically, this article will highlight the potential limitations and hazards of unstructured interviews, describe the essential components of structured interviews, and explore the experience with structured interviews thus far in EM. Finally, we propose a hybrid approach whereby some aspects of the structured interview are implemented to reduce bias, while maintaining a welcoming atmosphere with the ultimate goal of fostering diversity and inclusion in EM.

LIMITATIONS OF UNSTRUCTURED INTERVIEWS

Unstructured interviews provide ample opportunity for the introduction of bias into the recruitment process. A large meta‐analysis found that unstructured interviews result in Black and Hispanic applicants scoring approximately one‐fourth of a standard deviation lower on interviews than White applicants. The more structure that was applied to interviews, the more this gap between races closed.⁴ Structuring interviews consistently reduces similarity bias or the tendency to select applicants that are more alike to the interviewer. While the data within EM are lacking, multiple studies outside of medicine found that using rigorously designed structured interviews significantly lowers, and in some cases even eliminates, race and gender bias.⁹, ¹⁰, ¹¹, ¹²

Current interview practices also provide openings to ask illegal or unethical questions. One survey study reported that 47.6% of surveyed EM applicants were asked questions about their marital status, 22.6% about current children, 11.5% about plans for child‐rearing, 7.9% about ethnicity, 5.3% about gender, 5.7% about religion, 7.8% about age, and 0.5% about sexual orientation. Free‐text responses were equally alarming, with participants reporting questions that constitute sexual harassment and ageism.¹⁹ The authors note that such incidences likely occurred during the informal “get to know you” lines of questioning.¹⁹

Additionally, a meta‐analysis of predictive factors of physician performance shows that a traditional residency interview adds little value and may even negatively correlate with future performance.²⁰ These results have been replicated in EM, where the interview score has no correlation with clinical performance as judged at graduation.⁵

WHAT CONSTITUTES A STRUCTURED INTERVIEW?

Campion and Palmer⁸ conducted one of the first large‐scale meta‐analyses of structured interviews and proposed a framework that is now widely used. He suggested 15 elements of structure, of which six are the most widely used and validated, including use of anchored rating scales, training for interviewers, and standardized and job‐specific questions (Figure 1). Each of these structured elements provides incremental benefits,²¹ allowing programs to tailor this framework to their own needs.

Figure 1.

Fifteen points of structure in the interview process, adapted from Campion and Palmer.⁸ PBQs = past behavior questions; SQs = situational questions.

Campion and Palmer also proposed four types of structured interview questions (Figure 2), of which past behavior questions (PBQs) and situational questions (SQs) are the most commonly used and studied. Both PBQs and SQs exhibit impressive inter‐rater reliability (>75%)²² and have similarly high validity for predicting future performance in multiple fields.²¹

Figure 2.

Four types of structured questions, based on Campion and Palmer⁸ with examples. PBQs = past behavior questions; SQs = situational questions.

THE EXPERIENCE WITH STRUCTURED INTERVIEWS IN EM

Published experiences using structured interviews in EM are limited. One example previously employed in EM, among other specialties, is the MMI.¹⁷, ²³, ²⁴, ²⁵, ²⁶, ²⁷ The MMI consists of multiple brief stations with standardized scenarios, questions, or challenges that are scored on a standardized scale. Tasks may include role‐playing or testing fine motor skills.²³ Some evidence suggests that the MMI may reduce gender bias.¹⁴ The MMI is better correlated with first‐year performance when compared to rank list position, particularly with patient care and procedural skills.¹⁷ However, this correlation was lost after the first year of residency.²⁸ One major shortfall of structured interviews such as the MMI is that some applicants perceive the process to be intimidating and more difficult than traditional interviews,¹⁴, ¹⁵ which can result in candidates ranking a program lower.¹⁶ Indeed, the friendliness of a program was the most important factor when applicants formed a rank list.²⁹ If there is any concern that structured interviews might feel intimidating or unfriendly, programs may be hesitant to implement them. While the evidence supporting the implementation of structured interviews is significant, there are still reasons for organizations to continue to use unstructured interviews. For instance, a more informal atmosphere may help interviewers better portray themselves as “attentive, warm, thoughtful, and socially perceptive, and likable” in addition to allowing the interviewer the flexibility to better sell the job to the applicant.³⁰ This may be especially relevant in the match process where applicants interview at and consider many other programs for future employment. After all, the EM interview serves not only in applicant selection, but as a recruitment tool as well.

Another notable example of structured interviews is the recent trial of SVIs within residency applications. The SVI was designed to improve validity and reduce bias during application review using many of the principles of Campion and Palmer, including standardized PBQs and SQs relevant to EM.¹⁸ However, as an asynchronous, graded and generic element of the initial application, few applicants or programs believed the SVI to be of added value on top of the wealth of information in existing applications (e.g., grades, personal statements, standardized test results, and recommendation letters). In fact, SVI scores showed weak correlation with other faculty assessments of communication and professionalism, including traditional interviews.³¹, ³² The SVI pilot ended after the 2020 application cycle. In contrast, adding structure to existing synchronous interviews allows for follow‐up questioning that emphasizes in‐depth discussion rather than practiced responses.

In the age of COVID‐19, EM interviews have shifted from the in‐person to the virtual online format. Video interviews may complicate applicant assessment as the format uses a less rich communication channel. For example, the online video format may make it more difficult to interpret complex and subjective information. As a result, virtual interviews may actually benefit even more from adding structured elements through standardizing, simplifying, and making discussions more concrete with PBQs or SQs.²¹

DISCUSSION

Despite the identified limitations, the concept of structured interviews is worthy of further consideration by the EM community. Equitable hiring practices are necessary to narrow the gap between the number of EM physicians who self‐identify as underrepresented minorities (URM) and the number of URM patients.³³ A study published in 2013 reported that even though URMs represent 30% of U.S. population, only 9% of emergency physicians and 14% of EM residents self‐identify as URMs.³³ Years later, gender and racial disparities still persist in EM today.³⁴ URM providers have been shown to improve health care for minority, Medicaid, and uninsured patients.³⁵ As more programs recognize the importance of a diverse workforce and make efforts to recruit URM physicians,³⁶, ³⁷ a more rigorous evaluation of the interview process, as well as optimal implementation of “best practices” to reduce bias is warranted.

Based on the extensive literature supporting the use of structured interviews in other industries, taking into account the limited suboptimal experience within EM, we recommend the implementation of a hybrid interview model whereby programs can selectively add structure to their current interview process in a manner that best meets the needs of their applicants, interviewers, and program. This approach allows programs to avoid the drawbacks of strictly adhering to a rigid structure like the MMI and SVI while potentially benefiting from more a structured approach. We suggest that programs consider implementing the six elements that are most widely used and validated (Figure 3).⁸, ²¹ At least some questions should be standardized, using problem‐based or situational formats and focusing on job‐relevant qualities. Detailed rating scales that can be anchored to specific examples or benchmarks for job‐appropriate responses or can simply range from “excellent” to “poor.”²¹ Interviewers will benefit from additional training on the use of standardized questions, the anchored rating scales, identification of implicit bias, and illegal or unethical questions to avoid.

Figure 3.

Sample adaptation for EM residency interviews based on the six most validated and commonly used elements from Campion and Palmer.⁸ PBQs = past behavior questions; SQs = situational questions.

The potential pitfall of our hybrid interview proposal relates to validation. Structured interviews are psychological tests for which the development and validation require a rigorous scientific approach. While we base our recommendation on the use of previously validated elements of the structured interview of Campion et al.,⁷ their incomplete adaptation to a new industry, in a novel environment with a truncated interview format, will undoubtedly affect their psychometric properties. However, prior studies on similarity bias in interviews that did not incorporate all elements of structure still showed reduction or elimination of racial and gender bias despite varying levels of reported structure.¹¹ Nevertheless, meticulous internal and external validation efforts examining future performance and diversity outcomes in EM are necessary to ensure their value to serve intended purpose. Such work is long overdue and would allow for the development of validated structured interview questions that can be included in otherwise unstructured EM residency interviews.

CONCLUSION

There exists significant literature supporting the utilization of structured interviews to better predict future performance and reduce bias. However, emergency medicine has been slow to adopt this practice, in part due to concerns about the potentially negative impact on applicant perceptions of the program. We believe that as the importance of workforce diversity becomes increasingly recognized, the benefits of a structured approach outweigh the risks. To strike the optimal balance between structuring interviews and creating an enjoyable interview experience, we recommend the use of a hybrid interview approach in which programs selectively add structure to previously unstructured interviews, based on program‐specific needs and goals. Further research is needed to rigorously validate added structured elements based on diversity and future performance outcomes. The ultimate goal of such efforts is to adopt a transparent and fair recruitment process that will build the most competent and inclusive workforce in EM.

CONFLICTS OF INTEREST

The authors have no relevant financial information or potential conflicts to disclose.

AUTHOR CONTRIBUTIONS

RH and SK made substantial contributions to the conception of the work; RH performed the literature search and drafted the initial manuscript; SK performed additional literature search and contributed to editing the manuscript; AS provided close mentorship throughout the entire process, made extensive edits to the manuscript, and prepared it for publication; and all authors gave final approval of the version to be published and is in agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Hughes RH, Kleinschmidt S, Sheng AY. Using structured interviews to reduce bias in emergency medicine residency recruitment: Worth a second look. AEM Education and Training. 2021;5(Suppl. 1):S130–S134. 10.1002/aet2.10562