Bias in recruitment: A focus on virtual interviews and holistic review to advance diversity

INTRODUCTION

In 2019, nearly 45,000 medical students applied for the residency match.¹ Interview invitations are commonly influenced by a variety of factors such as academic performance metrics, personal statement, and research interests.² On the 2020 survey of program directors, the application components that ranked highest were quantifiers of academic achievement; among those were the United States Medical Licensing Examination (USMLE) Step 1 and 2 scores, clerkships grades, and class ranking. Other factors included were the narratives accounting for student performance via letters of recommendation and the Dean's letter.² After interviews are complete, each candidate is ranked against their peers. Residency programs heavily base ranking of candidates on performance during the interview day. All three of these review stages—selection for interview, performance during interview, and overall ranking compared to peers—leave room for bias.

The residency application and interview processes are subject to implicit and explicit bias.³, ⁴ Furthermore, in 2020 national organizations such as The Coalition on Physician Accountability, which includes the Association of American Medical Colleges (AAMC), 10 national emergency medicine organizations, and local hospital institutions, recommended virtual interviews in response to the COVID‐19 pandemic.⁵, ⁶ However, this new interview format may potentiate bias and disproportionately affect students who historically have been underrepresented in medicine (UiM).⁷, ⁸, ⁹ It is essential to seek a foundational understanding of the complexity and impact of bias against UiM students and how the upcoming virtual interview season format creates additional avenues for bias. Consequently, it is critical to discuss the importance of holistic review during this application cycle.

Just as bias affects diversity and inclusion in medicine, the pandemic has a compounded effect on the landscape of medical education with recruitment in particular. Across the country, routine clinical opportunities for students are now limited as an infection control measure and to preserve personal protection equipment (PPE). Further, traditional subinternships, a critical component of an application where medical students often travel to different institutions, are also restricted. These rotations allow medical students to better understand the practice variation of a field and an institution's culture, obtain a letter of evaluation, and gain access to potential mentors and collaborators.⁷ For UiM applicants, this opportunity offers them access to advocates in that program. Historically Black colleges and universities (HBCUs), who traditionally have higher percentages of UiM students, are credited for training the majority of Black physicians in the United States.¹⁰ HBCU medical schools, however, have limited affiliated residency programs especially given hospital closures compared to large academic institutions, thereby limiting students’ access to early advising specific to the specialty.¹¹, ¹² Students from HBCU programs often use rotations to showcase their clinical strengths, gain exposure to the field, and build connections. The inability to travel for these rotations, therefore, affects every candidate's application, although much more so for students from HBCU programs with limited internal residency programs. In addition to the loss of away rotation opportunities, students also experience difficulty scheduling USMLE test dates due to travel restrictions and site closures.¹³ Because of these changes to the application experience, it is imperative to develop and implement a true holistic review that looks beyond inherently biased metrics (scores, evaluations, AOA, grades) and instead focuses on life experiences, social identities, and extracurriculars of the candidates to ensure equity, especially for UiM candidates.

INTERVIEW BIAS

Biases can be formed and magnified within seconds of meeting someone.¹⁴ Implicit biases place vulnerable groups at risk for premature and unfair judgment.¹⁵ Also, while a person's name has never been directly linked to economic success, data reveal that a person's given name can affect the likelihood of an interview offer and success.¹⁶ Another source of bias is how candidates convey their thoughts through language. A study from the University of Chicago has explored the possibility of linguistic and speech bias during interviews and found that the ability to understand someone's speech as a heavily desired factor—therefore, accents could lower the desirability of a candidate. Subconsciously we make judgments on others based on accents, pitch, tone, and inflection points.¹⁷, ¹⁸ In addition, it has been shown that interviewers typically look for applicants who remind them of themselves, the concept of homophily.¹⁹ Bias can be gendered as well. We know that women are questioned and interrupted more often during the interview process and face more follow‐up questions than men, who were given space and time to comment on their potential in the business.²⁰ In the field of medicine, where diversity metrics continue to lag behind current estimates of the racial and ethnic composition of the U.S. population, these are essential details that must be proactively addressed.

VIRTUAL INTERVIEW BIAS

Vulnerable groups such as women and those applicants who identify as UiM deserve special attention during virtual interviews. Compared to in‐person interviews (where unstructured, impromptu conversations are natural), the transition to virtual interviews limits downtime interactions and, therefore, the familiarity between the applicants and programs.²¹ Although safer and cost‐effective, virtual interviews in response to needs and restrictions due to the pandemic introduce different additional biases.⁸, ²² When considering the intersectionality of various identities and demographics including age, ethnicity, race, gender identity and sexual orientation, abilities, and religion, these unanticipated biases may infringe on critical efforts to increase diversity, equity, and inclusion within medicine.

Bias in video interviews within medicine has been documented. In the 2017 to 2018 application cycle, standardized video interview (SVI) was created by the AAMC and was piloted as an opportunity for an applicant to “showcase the intangibles about themselves” that were not highlighted in their application. The SVI aimed to offer a more holistic evaluation of an application by providing more information regarding communication, professional, and interprofessional skills.²³, ²⁴ However, the data received from the SVI were mixed, leaving programs unable to assess how to use this additional data point, and the SVI format was later abandoned.²⁴, ²⁵, ²⁶ One study revealed no correlation between USMLE scores and performance on the SVI.²⁷ Another study, while favoring women over men and US‐MD applicants in SVI, also showed standardized evaluation ratings having bias against Black men.²⁸ Further an additional study found that emergency medicine interviewees were concerned about bias toward their physical appearance, ethnicity, and video production quality.²⁹ Despite the AAMC having personnel trained to recognize and avoid bias during the SVI scoring, standardizing interviews across all programs and specialties for academic recruitment cycle is challenging.³⁰ Furthermore, students with unreliable internet or are from low socioeconomic status might be judged unfairly based on connectivity issues or background decor. All of these, when left unaddressed will likely migrate into the 2020 to 2021 application cycle due to heavy reliance on the virtual format across all programs. Table 1 lists potential interventions that can be incorporated to remove bias during the interview process.

TABLE 1.

Approach and impact of potential interventions to limit bias

Intervention	Approach	Impact
Virtual background	Create an expectation that all applicants use a solid color for their background (a bare wall without decor or a solid color image). Offer several plain, standardized, virtual background images to suit interviewees preferences.	Standardize the screen so that applicants are not judged or graded by their surroundings.
Pronunciation of names	Invite students to submit audio recordings of their names via an online free software which will be shared with interviewers.	Removes potential discomfort that appears when unable to pronounce a name. A person's name is a significant connection to one's identity.
Offer a technology check session	Have an administrator host sessions for students unfamiliar with the technology, especially if the program is hosting interviews on less frequently used software platforms.	Allows students to gain familiarity with software and a chance to explore compatibility and fix technological mishaps.
Implicit bias training for interviewers	Encourage the use of widely available open access educational videos and incorporation of self‐assessment.	While not sufficient to create change alone, training starts the conversation and introduces the language and the concept of bias.
Standardized questions	Develop prospective core questions for interviewers to create consistency in areas asked and set agreed‐upon criteria that align with departmental goals. Focus questions on skills.	Avoids stereotypical questions and also offers the ability to compare among candidates. Avoids personality questions and instead aligns with the needs of the program.
Multiple mini‐interviews	Applicants get interviews from a series of multiple reviewers.	Applicants gain a better understanding of the institution by talking with different faculty members and residents. Allows programs to get to know the applicant from an array of reviewers.
Mock interviews	For students interviewing from schools without an associated residency program, consider offering mock virtual interviews with faculty or alumni.	While mock interviews should have no bearing on the applicants’ final evaluation, it provides an opportunity for applicants without home institution support.
Diverse Interviewers	Increase UiM faculty and resident presence during the interview day. Compensate UiM faculty and residents for the time given above minimum requirements or expectations to be part of recruitment.	Applicants can meet faculty and residents with shared experiences of being UiM. Avoids minority tax and also sets the model for inclusion efforts as being part of the culture rather than on a voluntary basis.
Anonymous post‐interview feedback	Reaching out to UiM applicants after the interview cycle for ongoing quality review and improvement.	Applicants have the opportunity to report negative and positive experiences. The program receives feedback to continue to reiterate and reinforce positive experiences.

Abbreviation: UiM, underrepresented in medicine.

HOLISTIC REVIEW

The holistic review of applications humanizes each applicant beyond quantitative metrics and narrative evaluations. Studies reveal that holistic application reviews create a diverse pool of applicants in part due to a shift in emphasis from USMLE step scores alone to letters of recommendations, life experiences, and extracurriculars.³¹, ³² Unfortunately, as of 2020, program directors rated “other life experience,” “volunteer/extracurricular activities,” or “fluency in the language spoken by [the] patient population” with lower importance as it pertains to rank list placement.² However, all three of these factors help create a picture of personhood—the distance traveled and lived journey—and are therefore critical in performing a holistic review that considers and values the entire application without unfairly placing weight on factors that have historically limited diversity (e.g., step scores, evaluations, AOA).

CHALLENGES IN HOLISTIC REVIEW

Holistic review is not without its challenges. Bias is present at multiple junctures of the application, recruitment, and interview cycle. Even when institutions have attempted to reduce bias through blinding of an applicant's name and implementing automated interview processes, the effects of biases persist.³³ In one study, when an employer blinded the applicants’ names in file reviews, they found other ways of discriminating candidates such as looking at the applicant's affiliations and participation in the Black student organization.³³ In response, studies have shown that Black and Asian applicants have attempted to avoid job discrimination through the “whitening” of their applications, which yielded an increase in interview offers. This was achieved by deliberately concealing aspects of the application, including race‐related extracurricular activities, such as being president of a diversity and inclusion organization.³³ In a meta‐analysis performed regarding employer–applicant callbacks, Black applicants were less likely to receive a callback than White applicants, a staggering difference that has persisted over the 25 years evaluated in the study.³⁴

BENEFITS OF HOLISTIC REVIEW

While one's application is vulnerable to bias at several junctures, performing a holistic review offers a big picture about each candidate and creates connections in each aspect of the application entry that can help overcome stereotypes.³⁵ Programs that have applied the use of holistic review have increased interviews for UiM and the matriculation of these candidates.³⁶ The AAMC has also shown that implementing a holistic review increases the interview and matriculation of first‐generation college, Black/African American, and Hispanic students in medical school.³⁵

CHALLENGES IN PERFORMING HOLISTIC REVIEW

The changes in the 2020 to 2021 application cycle also create a new challenge for performing a holistic review of candidates given an anticipated higher volume of applications with less time to review them thoroughly. Performing holistic reviews will be time‐ and resource‐intensive. While restrictions on travel will lessen the financial and time commitments for applicants, there is concern that applicants will be financially available to apply to more programs, which will ultimately shift supply and demand for interview positions.³⁷ Medical schools have seen a 16% increase in their applicant pool this year.³⁸ Residency programs anticipate a similar increase in trend.³⁹ If programs receive more applications to review over a shorter time allotment, especially since ERAS opens later this cycle on October 21, 2020, performing a thorough holistic review of each applicant will be challenging given that the deadline to the submit rank list is unchanged. Despite these high stakes decisions that are susceptible to bias and variations in practice, programs will be expected to do more in less time. To combat these challenges, implementing quantitative, preweighted scoring pre‐ and postinterview can help eliminate potential bias, and advocating for transparency can help reduce potential bias. In one study, they placed a cap on the number of applications evaluated per reviewer, selected interviewers to eliminate bias, standardized interview questions, performed multiple mini‐interviews, and weighted z‐scores to create heat maps for ratings to evaluate for outliers.⁴⁰

LETTERS OF REFERENCES

Letters of recommendation have also been shown to have a linguistic bias against women and racial minorities applying to residency. One study revealed that agentic words, descriptors related to leadership and work ethic, were less likely to be used when describing Black and Latinx applicants when compared to White and Asian applicants. Female applicants were also more likely than male applicants to have both agentic and communal terms used in their recommendations, with communal terms being used to describe stereotypes associated with being a woman such as kind and sympathetic.⁴¹ A way to overcome bias is to standardize letters of recommendations and evaluations. In emergency medicine, after the implementation of the standardized letters of evaluation (SLOE), a study found no “pervasive differences” in linguistic categories in terms of gender bias.⁴² Of note, the specialty also moved from using letters of recommendations to SLOE to develop an objective focus on assessment. This example highlights the importance of standardizing the format for letters of references to limit bias and as a result provide a structured format and minimize biasing language in evaluations.

CONCLUSION

Advancing diversity in medicine continues to be a critical challenge and heavily affects women and underrepresented in medicine applicants. Virtual interviews will likely pose additional bias against women and underrepresented in medicine applicants. At the same time, there are time and resource limitations that interfere with the implementation of formal, standardized holistic reviews. These reviews would minimize bias when creating an environment of inclusion during the current recruitment season. Moving forward, it is necessary to prioritize the unique challenges in holistic reviews and virtual interviews in context of diversity, equity and inclusion to advance representation in medicine.

CONFLICT OF INTEREST

The authors have no potential conflicts to disclose.

AUTHOR CONTRIBUTIONS

All authors contributed to the writing, drafting and editing of this manuscript concept and design, acquisition of the data, analysis and interpretation of the data, and critical revision of the manuscript for important intellectual content.

Otugo O, Alvarez A, Brown I, Landry A. Bias in recruitment: A focus on virtual interviews and holistic review to advance diversity. AEM Educ Train. 2021;5(Suppl. 1):S135–S139. 10.1002/aet2.10661