AAIM Recommendations to Promote Equity and Inclusion in the Internal Medicine Residency Interview Process

Perspectives Viewpoints.

• •Programs should be aware of potential systemic biases impacting certain groups, including underrepresented in medicine populations, sexual and gender minorities, first-generation or low-income applicants, international medical graduates, and individuals with disabilities.
• •Programs should mitigate potential biases that arise in the interview process and implement strategies to promote equity and inclusion.
• •Programs should structure their interviews and scoring systems to reflect the merits and qualities they seek in an applicant.

Alt-text: Unlabelled box

Introduction

The coronavirus disease 2019 (COVID-19) pandemic upended residency recruitment by transforming interviews into an all-virtual process, and it heightened awareness of inequities impacting specific applicant populations.^{1 , 2} Overall, program directors were satisfied with the process and outcomes of the all-virtual interview process and planned to keep virtual elements in future years.3, 4, 5 Both programs and applicants found advantages in an all-virtual recruitment process, including cost savings, improved efficiency, flexibility of interviews, faculty recruitment, opportunities to interview more candidates, program website innovations, decreased interruptions in education from travel, and the ability to attend more interviews.3, 4, 5 However, virtual interviews also potentially introduced or amplified implicit and structural biases.^{6 , 7} Furthermore, applicants expressed challenge with virtually assessing a program's culture.^{3 , 4} They found it difficult to assess whether programs equitably provided opportunities to learners and whether programs were inclusive of diverse learner groups.³

A demographically representative physician workforce improves both health care access for underserved populations⁸ as well as medical research and innovations for all populations.⁹ Despite the efforts of the Liaison Committee on Medical Education,¹⁰ medical school matriculants from underrepresented in medicine (URiM) backgrounds continue to remain underrepresented compared with the US population, suggesting that additional efforts are needed to realize improvements in URiM representation.⁹ Similar data demonstrating a gap in representation, or lack of data, applies to other specific groups, such as individuals with disabilities and sexual and gender minorities (SGM).

Gonzaga et al¹¹ recommends five key steps to increase diversity in medical education to meet the needs of the population: setting diversity as a priority, seeking out candidates, implementing inclusive recruitment practices, investing in learner success, and building the pipeline. This guide focuses specifically on implementing inclusive recruitment practices by mitigating bias in the interview process at the individual and systemic levels.

Background

The Alliance for Academic Internal Medicine (AAIM), a national organization representing internal medicine leaders in undergraduate medical education (UME) and graduate medical education (GME), created a task force in November 2021 to develop interview standards for the internal medicine residency interview process with a lens specific to diversity, equity, and inclusion (DEI) for applicants and programs.

This guide provides strategies to reduce individual and systemic biases in the interview process. The task force conducted an iterative process to identify areas in need of guidance, including reviewing the literature, consulting with DEI experts, identifying evidence-based strategies and resources, and engaging with UME and GME stakeholders. Alliance governance councils and DEI expert representatives also provided input. Although this article focuses on residency interviews, the principles and strategies are, for the most part, transferable to the fellowship interview process.

The implementation of these recommendations necessitates seeking out the support, participation, and expertise of faculty, residents, and leadership with knowledge and skills related to DEI. Many of these individuals are likely to be from underrepresented populations, thus putting them at risk of being impacted by the “minority tax.”¹² The minority tax is the concept that individuals from underrepresented groups take on additional responsibilities to support an organization's DEI efforts without compensation for their efforts.¹² Programs should be intentional in mitigating the effect of the minority tax on these individuals, specifically by providing resources, funding, and recognition in the promotion process for their contributions.¹³

Demographic group identifiers are based on terms that are accepted within their communities, including use of the terms URiM, SGM, first-generation or low-income applicants, international medical graduates (IMGs), and individuals with disabilities.

Reducing Bias in the Interview Process

Impact of Individual and Systemic Bias

Bias permeates the interview process at both the individual and systemic levels and negatively impacts both applicants and residency programs. Programs may unintentionally eliminate applicants who are qualified and who could also contribute important perspectives and values. Society is also impacted; physicians from diverse backgrounds often represent their community in discussions around clinical care, health policy, research, and innovations, thereby elevating the needs of the community.^{9 , 11} Awareness of individual and systemic biases is critical for programs to ensure equity in the interview process. Table 1 lists recommendations to address individual and systemic biases.

Table 1.

Recommendations to Mitigate Effects of Individual and Systemic Bias

Recommendations to Reduce Individual Bias • Provide implicit bias education and training for faculty and staff who interact with applicants on interview days to cover ○ How to identify one's implicit biases in the interviewing process, including how virtual interviews may introduce and magnify other biases6 and ○ Strategies to mitigate one's implicit bias in the interview process.6 •Allow sufficient protected time in an interview as bias is stronger when under time pressure.14 , 15 •Aim to schedule breaks in between interviews (eg, 10 min) for both applicants and interviewers.16

Recommendations for Programs to Reduce Systemic Bias and Promote Equity and Inclusion •Create residency selection committees with diverse faculty, such as representation from URiM and SGM. Invite DEI leadership, diversity champions, or diversity committee members; this may help mitigate how implicit bias influences the selection process.11 •Create a standardized process of reviewing evaluations for biased narrative language (eg, by a trained program coordinator) and a procedure for dealing with a biased evaluation. For example, after recognizing a biased evaluation, the coordinator can remove biased language prior to review by the selection committee and then provide feedback to the interviewer. •During the interview day, emphasize support for applicants from these demographic groups in general, without identifying or singling out individuals. This can be done during general program and diversity presentations. •Display commitment to inclusiveness (eg, pronoun pins, Pride flags, posters for DEI). Include pronouns when displaying names during virtual events. •Host specific diversity group forums so current residents of that demographic group can interact with applicants who self-identify in similar groups. Provide protected time in schedules for residents to attend these forums. •Ask for volunteer faculty or residents with various identities to be point contacts for whom applicants can contact for questions during the interview season. •Develop a backup plan in case of connectivity issues with videoconferencing (eg, telephone interview or another videoconferencing platform). •After positions have been filled, seek anonymous feedback regarding equity and inclusion from applicants who did and did not match into the program. •Emphasize continuous quality improvement in the interview process and reflect on how the interview process can be improved after the interview season.

Recommendations for Medical Schools to Reduce Systemic Bias and Promote Equity and Inclusion •Set expectations and prepare students for the interview process and recruitment activities. Topics include how to prepare for interviews, postinterview communication with programs, and questions to ask during an interview. Applicants with privilege may have better access to these resources, so explicitly including these expectations in the curricula can promote equity. •Offer standardized virtual background templates for students who prefer to use a template. •Provide spaces for interviews with stable internet access and a high-quality camera. •Provide mock interviews and workshops on best practices with virtual and in-person interviews.

DEI = diversity, equity, and inclusion; SGM = sexual and gender minorities; URiM = underrepresented in medicine.

Implicit biases are the attitudes, stereotypes, and feelings people harbor unconsciously.¹⁷ They impact our interactions and decisions and are more insidious than overt biases because they are, by definition, outside of our awareness.¹⁷ Implicit bias permeates all levels of medical education advancement, including clinical evaluations, where group differences in clerkship narrative language and grades exist even after adjusting for academic metrics.^{18 , 19}

Within the residency interview process, implicit bias can present as rater bias, which includes affinity and video background biases. Affinity bias is seen when the rater's scores favor applicants with similar backgrounds, interests, and appearances, which include vocal tone, race, gender, age, shared schools, and geography.^{6 ,} 20, 21, 22, 23, 24 Virtual interviews also introduce new biases inherent to the video platform; for example, fatigue with video conferencing may lead to reliance on implicit bias and hardware/software instability (internet disruptions, communication delays) can lead to a negative perception of an applicant. Video background bias may occur when an interviewer judges an applicant based on the applicant's physical setting, which could include household tidiness, items, other people, or background noise.^{6 , 20}

Systemic Bias in Specific Demographic Groups

Multiple groups face systemic bias and barriers in the residency interview process; we are not able to address all marginalized groups within the scope of this article and have limited discussion to five groups. Intersectionality, or the interconnected nature of all of one's identities, plays a significant role in these groups, which can compound barriers.²⁵ Table 2 lists recommendations to address systemic bias affecting these groups.

Table 2.

Recommendations to Promote Equity and Inclusion for Specific Populations During the Interview Process

Recommendations Specific to URiM Applicants •Ensure a diverse group of residents and faculty who can participate in general interview day activities. This requires protecting time in their schedules to participate in these recruitment activities. •Hold applicant sessions with upper-level URiM residents and faculty to create a sense of connection and ability to ask questions in a “low-stakes” situation. This can also help applicants envision themselves thriving in the program.26 •For programs without strong diversity within the residency or faculty, be transparent in discussing a thoughtful plan on how to create positive change toward diversity in their department.26

Recommendations Specific to SGM Applicants •Know the appropriate contact, such as the human resources office or central GME office, for positive SGM benefits and policies, such as adoption benefits, health care benefits, and family planning. •Use an OUTlist to allow applicants to see “out” faculty (eg, https://lgbt.ucsf.edu/outlist-directory).

Recommendation Specific to Applicants with Low-Income •Waive application fees (eg, for secondary applications) for applicants who have undue financial burdens.

Recommendations Specific to Applicants with Disabilities •In a standardized manner (eg, in the body of the interview invitation), provide contact information for the appropriate office/designee that could discuss the accommodations confidentially. This office/individual should be informed of legal requirements and clinical accommodations. •Include individuals with disabilities in diversity definitions or mission statements on website. For example, “As part of our commitment to diversity, [name of program] seeks to attract and educate trainees who will make the population of health care professionals representative of the national population. We are, therefore, committed to the full and meaningful inclusion of qualified trainees with disabilities.”27

Recommendations Specific to IMG Applicants •Establish a point person in the program leadership to have a deeper understanding of the educational systems of international medical schools and the medical education received, including evaluations and testing in these systems and how they compare to US medical schools. This individual should receive protected time and resources for this role. •Program directors and staff should seek professional development to stay current on visas and requirements of the immigration process. •Institutions with Immigration Offices can serve as resources for IMG applicants, and this information can be included in a standardized manner (eg, in the body of the interview invitation). •Provide examples of particularly successful IMG faculty or program graduates to interviewers and individuals involved in the recruitment process. •Have readily available the medical school institution of interviewing faculty so that IMG applicants can identify other IMG faculty, similar to an OUTlist for SGM applicants. •Collaborate with IMG groups (eg, American Medical Association-IMG and American Colleges of Physicians-IMG) to keep updated on the unique circumstances of applicants from different countries. •For virtual interviews, the time zone discrepancies can be a barrier, particularly for IMG applicants; provide flexibility in interview times to accommodate different time zones. •Waive application fees (eg, for secondary applications) for applicants who have undue financial burdens.

GME = graduate medical education; IMG = international medical graduate; SGM = sexual and gender minorities; URiM = underrepresented in medicine.

Underrepresented in Medicine Populations

Black/African American, Latinx/Hispanic, and American Indian/Alaskan Native applicants have historically faced systemic racism in medical education, leading to these groups being classified as URiM.²⁸ The negative impact of implicit bias on learner advancement through the medical education continuum has been well-documented. Even after adjusting for confounding educational metrics, disparities continue to persist in clerkship grades, medical student performance evaluation narratives^{18 , 29 , 30} and Alpha Omega Alpha membership,³¹ and favor non-URiM students. Small differences in clinical assessment ratings between URiM and non-URiM can lead to larger differences in grades and awards received (amplification cascade), which may potentially limit career opportunities for learners who are URiM.¹⁹ Despite the flaws of these metrics, residencies continue to weigh them significantly in resident selection.

Sexual and Gender Minorities

SGM is an all-encompassing term that includes but is not limited to lesbian, gay, bisexual, transgender, queer, intersex, asexual/agender (LGBTQIA+).³² Approximately 8% of people living in the United States identify as SGM.³² Sexual orientation (SO) and gender identity (GI) data are not routinely collected in the workplace due to historical and current laws that allow employees to be terminated for their identity, depending on residential jurisdiction. Because SO/GI data is not routinely collected on the physician workforce, it is not possible to assess if this population is underrepresented. Additionally, “being out” is an individual choice and is not an all or none phenomenon, meaning being “out” may or may not include work. Therefore, the SGM population is likely underestimated. Research has shown that discrimination is prevalent in the general workforce for SGM.^{33 , 34}

Prospective LGBTQIA+ residents face several barriers in the interview process. First, they cannot easily identify SGM faculty or residents. Second, Electronic Residency Application Services does not collect SO/GI data. Programs cannot track their recruitment efforts of SGM residents. Finally, it is unclear whether implicit bias workshops hosted at institutions includes bias against LGBTQIA+ applicants.

First-Generation Applicants and Applicants with Low Income

The Association of American Medical Colleges (AAMC) estimates that first-generation applicants, defined as “candidates whose parents have not earned an associate degree or higher,” make up 12.4% of 2021-2022 matriculants to allopathic medical schools.³⁵ The US Department of Human Health Services defines candidates as low-income if their family earns less than 200% of the agency's cutoff for poverty.³⁶ Medical school costs have increased exponentially³⁷ and less than 6% of medical students come from the bottom quintile, whereas nearly half come from the top quartile.³⁸ First-generation applicants are a heterogenous group, with variability in racial and ethnic backgrounds and immigration status.³⁹ Considerable overlap exists among groups of applicants who are first-generation, low-income, and URiM. These applicants may share barriers related to finances and social support.^{25 , 37}

Individuals with Disabilities

Approximately 4.5% of medical students across allopathic and osteopathic medical schools reported having a disability,²⁷ which includes physical, sensory, learning, psychological, and chronic health conditions. The true number is likely underreported given students may not feel safe disclosing their disabilities.^{27 , 40} Learners with disabilities form a heterogeneous group with varying degrees of impairment.⁴⁰ During the interview process, applicants with disabilities may face stress and logistical challenges associated with in-person travel,⁴¹ with these challenges being mitigated with virtual interviews. In addition, applicants with disabilities may have concerns about disclosing their disabilities during interviews, leading them to hesitate to ask about accommodations that could be important in their decision-making.⁴²

International Medical Graduates

IMGs are a heterogeneous group representing different races and ethnicities, with the majority of licensed IMGs graduating from medical schools in India, the Caribbean Islands, Pakistan, the Philippines, and Mexico.⁴³ IMGs disproportionately practice in underserved rural and urban areas, providing important health care services in these regions.⁴⁴ In 2020, approximately 25% of applicants who matched into internal medicine residencies were IMGs born outside the United States;^{45 , 46} among different specialties, more than 4,000 IMGs were offered visas to facilitate residency training.^{46 , 47} IMGs face several barriers in the interviewing process. First, IMGs incur excessive costs in the residency process because of the high number of programs they must apply for to match.^{48 , 49} To be considered competitive, they often must have an application that includes additional research fellowships and observerships that may be unfunded.⁵⁰ Another barrier is that programs lack understanding of issues related to visas.⁵⁰ Finally, departmental pressure, institutional priority, and reputational concerns have been considered factors impacting the recruitment of IMGs to university programs, with the reputational concern raising the possibility of bias.⁴⁵

Interview Content and Evaluation

This section focuses on three structural strategies to reduce the influence of individual biases during and after the interview: defining the concept of “fit,” structuring the content of interviews, and standardizing the evaluation of interviews. Table 3 includes specific recommendations to reduce bias in the interview and evaluation process.

Table 3.

Recommendations to Reduce Bias During the Interview and Evaluation Process

Pre-defining Merit and Compatibility Between Applicants and Programs •Prior to the interview season, define the program's desired qualities in an applicant in the context of the program's brand identity (eg, goals, mission, and learning environment) and recruitment goals.51 , 52 •Develop these criteria of merit using a group of stakeholders who are familiar with the residency program's mission and with residency education.53 •Discount gestalt impressions if not supported by evidence.52

Structure and Content of Interviews •Design interviews to have structure, such as using behavioral questions, or multiple mini-interviews. •For each candidate, consider having at least one partially or completely blinded interview (eg, withholding academic performance or medical school).

Evaluation of Interviews •Develop a scoring rubric for the interviews with descriptive anchors based on specific behaviors related to competencies or attributes valued by a residency program. •Train interviewers on the use of scoring rubrics to ensure the reliability of evaluations.53

Programming to Promote Inclusivity •Signal commitment to equity and inclusion through the website and social media,7 include photos that represent diversity and inclusion in a program's overview. •Include diverse faculty and residents among those interviewing applicants and those facilitating general “meet-and-greet” events. •Demonstrate inclusion through DEI-focused recruitment events, including using smaller breakout rooms during virtual sessions, which may help improve candidates’ comfort to ask questions. •Ensure that faculty and resident efforts aimed toward enhancing equity and inclusion are appropriately valued, funded, and counted toward academic promotion.13

DEI = diversity, equity, and inclusion.

Predefining Merit and Compatibility Between Applicants and Programs

The Association of American Medical Colleges defines applicant-program fit as the compatibility between an applicant's personality, attitudes, work, and learning style or preferences and the program's goals and culture.⁵¹ Applicant-program fit is one of the most important factors that residency programs consider when evaluating applicants. However, residency programs rarely predefine what they mean by fit, an ambiguous term that is often based on gestalt and which may be subject to implicit bias. It can imply similarity to other residents in the program, which could represent a threat to diversity by eliminating qualified applicants.⁵²

Programs are encouraged to define the characteristics they feel will allow the applicant to thrive in their program. These characteristics can be divided into values fit and culture fit. Values fit is when an applicant shares the same mission or values as the program. Culture fit can be seen as the mindset of recruiting applicants similar to their current residents. However, programs are encouraged to transition from the mindset of culture fit, which may be seen as preserving comfort and familiarity, to culture add, the concept of seeking applicants with diverse ideas and experiences.^{54 , 55}

Structure and Content of Interviews

Unstructured interviews can be influenced by an interviewer's implicit bias. Structured interviews with standardized elements may reduce bias and improve the reliability and validity of these interviews.^{51 , 56} In this section, we discuss the use of structured interviews as a strategy to mitigate bias.

Behaviorally Based Interviews

Interviews with behavioral questions are common in the business setting and are increasingly being used in residency interviews to assess an applicant's skills needed as an entering intern. The use of structured questions can reduce potential racial and gender bias in the interview process by using a set of standardized and open-ended questions that address an applicant's thought process, action, and outcomes of a certain situation, with a focus on past behaviors and situations.^{57 , 58}

Multiple Mini-Interviews

The multiple mini-interview (MMI) is one type of structured interview technique with multiple stations with different interviewers and standardized questions.⁵⁶ MMI may mitigate bias by reducing the variability of questions and limiting the impact of both the interview context and interviewer biases (eg, gender, race, or leniency/stringency tendency).^{56 ,} 59, 60, 61 MMIs include scenarios that may provide insight into an applicant's noncognitive domains, such as interpersonal and communication skills.⁵⁶ Studies have shown improved reliability as well as perceived fairness, effectiveness, and acceptability,60, 61, 62, 63 although some applicants felt it difficult to establish a connection with interviewers during MMIs.⁶⁴

Blinded Interviews

Applicant files may trigger interviewer implicit bias based on information such as name, gender, home institution, and academic performance. Partial or complete blinding refers to the withholding of information to the interviewer and may help mitigate bias. Partially blinding the interviewer to academic data enables the interviewer to focus on the nonacademic attributes, such as communication skills,⁶⁵ or the characteristics the program defines in their definition of program-applicant compatibility. Blinded interviews also help in eliminating the “halo” effect in which an interviewer's preconceptions based on academic record influence the interview evaluation.⁶⁶ Studies with residency applicants who were interviewed under blinded conditions have shown a higher interviewer rating in likelihood to succeed than candidates interviewed under unblinded conditions.^{67 , 68}

Standardizing the Evaluation of Interviews

The use of rating scales for the evaluation of an applicant can enhance the reliability, validity, and fairness of interview scores.^{51 , 53} Each standardized question can be associated with a rating scale that uses narrative anchors. Faculty development on the use of these scales is critical.

Programming to Promote Inclusivity

The interview day experience is critical for applicants to assess program culture and compatibility. Applicants may encounter difficulty assessing program culture through virtual interviews. Intentionally demonstrating inclusivity is therefore particularly important. For example, URiM applicants reported that URiM-focused virtual events provided the opportunity for current URiM residents to impart important knowledge about a program and its culture.⁴¹ The virtual nature of such sessions also promoted equity in allowing more URiM applicants to attend these events, thus enabling more applicants to have the opportunity to gauge the inclusivity and culture of a program.

The Appendix, available online includes resources on equitable and inclusive interview strategies.

Conclusion

Diversity is an essential component of a productive and innovative workforce. Although the virtual interview process has benefits to both programs and applicants, it also has challenges, including the potential introduction of or magnification of biases held by individuals and those embedded in the current system. Educators in UME and GME can become change agents in promoting, sustaining, and increasing DEI through intentional, equitable, and inclusive interview strategies, including faculty development and standardization of the interview process.

Appendix

Resources to Guide Programs in Implementation of an Equitable and Inclusive Interview Process

Examples of General Interview Guidance for Program Directors and Administrators •AAMC's Tips for Program Directors. https://www.aamc.org/media/44676/download •AAMC's Virtual Interviews: Tips for Interviewers. https://www.aamc.org/media/44811/download •AAIM guidance for virtual residency interviewing season. https://www.im.org/resources/ume-gme-program-resources/guidance-virtual-interviewing

Examples of Resources That Apply an Equity Lens to the In-Person and Virtual Interviewing Process •AAIM Recruitment Health Disparities Collaborative's toolkit for a holistic application and interview process, with brief (<6 minutes) video links to diversity, equity, and inclusion (DEI) concepts (bias and heuristics, schemas, attitudes and stereotypes, real world consequences, implicit association test [IAT], countermeasures) https://higherlogicdownload.s3.amazonaws.com/IM/fecab58a-0e31-416b-8e56-46fc9eda5c37/UploadedImages/Documents/resources/Recruitment_Subgroup_Health_Disparities_Collaborative.pdf •Explanation of bias associated with virtual interviews and strategies to mitigate these bias. https://journals.lww.com/academicmedicine/Fulltext/2021/08000/Avoiding_the_Virtual_Pitfall__Identifying_and.38.aspx •Reflections from Underrepresented in Medicine applicants on elements that promoted and detracted from inclusivity during the virtual interview process. https://doi.org/10.4300/JGME-D-21-00674.1

Examples of Faculty and Staff Development Resources on Implicit Bias •Implicit Association Test: Commonly used to raise awareness of one's own personal bias with the idea that this may help reduce the impact of the bias. Individuals can choose different tests to assess one's automatic preferences (eg, race, gender, disabilities, sexuality, transgender) https://implicit.harvard.edu/implicit/takeatest.html •AAMC's webinar on “Addressing Implicit Bias in Virtual Interviews”: 15-minute webinar on implicit versus explicit bias, and strategies to mitigate biases when conducting and rating interviews. https://vimeo.com/443088643

Examples of Behavioral Interview Questions •Develop behavioral interview questions that link to skills or “core values” that a residency feels are essential for an entering intern. Questions should be standardized across applicant interviews. • Potential domains include interpersonal skills, conflict resolution, communication, motivation, compassion, leadership, teamwork, reflection, resiliency, learning from previous mistakes, and diversity, among others. Examples of characteristics and behaviorally based questions: - Resilience: “Our residents often address difficult or challenging situations. Please give an example of a time when you faced a challenge that tested your coping skills.”53 - Internal motivation: “Our residents often go above and beyond. Please describe a time when you went the extra mile when it would have been just as acceptable not to, and why.”53 - Conflict resolution: “Tell me about a time you had to deal with conflict within a team.”58

AAIM = Alliance for Academic Internal Medicine; AAMC = Association of American Medical Colleges.