Virtual Residency Interviews: Applicant Perceptions Regarding Virtual Interview Effectiveness, Advantages, and Barriers

Abstract

Background

Studies of the virtual interview format are needed to inform medical residency program leaders as they plan for future virtual interview seasons.

Objective

In the current study, completed in 2021, we sought to assess applicant perspectives of virtual interview effectiveness, advantages, and barriers, including factors that might impact equity and inclusion.

Methods

Interviewees applying to 7 residency programs and 2 clinical psychology programs at an academic medical center in the Pacific Northwest completed a post-interview survey.

Results

A total of 565 of 1429 interviewees (40%) completed the survey. A vast majority (83%–96%) agreed virtual interviews were effective in each measured domain, except for learning institutional culture (352 of 565, 62%). Many also found information regarding social/living environments inadequate. Participants selected advantages to virtual interviews more frequently than disadvantages. Commonly selected advantages included cost savings, time efficiency, reduced burden of travel, and reduced carbon footprint. Disadvantages included time zone differences, access to an appropriate interview setting, and reliable access to internet. The majority of interviewees (84%, 456 of 542) desired to keep a component of virtual interviews in the future. There were no significant disparities in results based on gender, rural/suburban/urban location, race, or underrepresented minority status.

Conclusions

Virtual interviews were perceived as effective, more advantageous than burdensome, and widely acceptable, with no disparities in these findings by included demographic characteristics.

Introduction

Due to the COVID-19 pandemic, US graduate medical education (GME) programs conducted virtual interviews in 2020–2021. Applicant perceptions regarding the virtual interview format are needed to inform program leaders as they plan for subsequent interview cycles.

Most studies of GME program virtual interviews were published prior to the technological advances and widespread use of virtual conference platforms that occurred during the pandemic.^1-11 These studies were limited to single program assessments, and none to date included an assessment of potential impacts on diversity, equity, or inclusion.¹²

In this study, we assessed applicant perceptions of virtual interviews in a multispecialty approach at one institution during the 2020–2021 virtual residency interview cycle. To begin to explore the impacts of virtual interviewing on equity and inclusion, we assessed key advantages, barriers, and perceived differences in experience based on gender, race, and location.

Methods

This observational study surveyed interviewees in family medicine, obstetrics and gynecology, psychiatry, emergency medicine, pediatrics, general surgery, and anesthesiology residency programs, as well as in clinical psychology internship and PhD programs, following their virtual interviews at one urban academic medical center in the Pacific Northwest in 2020–2021.

The survey was created by a team of clinician educators, including faculty, program directors, a statistician, residents, and medical students (Qualtrics.com; provided as online supplementary data). The instrument assesses effectiveness, advantages, barriers, and future preferences regarding virtual interviews. We included items that may amplify or alleviate structural inequalities in the setting of virtual interviews, such as pregnancy, breastfeeding, dependent care, disability, and mental health conditions. We also included self-reported urban/rural/suburban interview location, because rurally based applicants may not always have reliable access to internet but may particularly benefit from reduced travel in the setting of virtual interviews. The author group relied on the literature and expert discussion to select items that would optimize content validity. To increase response process validity, we piloted the survey with 10 people, including faculty, program directors, residents, and medical students, and revised for clarity, brevity, and readability.

Descriptive statistics of responses to items related to effectiveness, advantages and barriers, and future interview format preferences were analyzed. Subgroup analyses were performed on each item above by gender, race, underrepresented in medicine (UiM; excludes White or Asian) and urban/suburban/rural status. A narrow age distribution precluded analysis by age. Participants reporting mixed race with at least one UiM selection were included in our aggregated UiM category. Participant responses were excluded from subgroup analyses when requisite demographic data were not provided or a demographic category contained fewer than 5 responses.

All statistical tests were performed using SAS 9.4 software (SAS Institute Inc, Cary, NC). We conducted Wilcoxon–Mann–Whitney and Kruskal–Wallis tests for ordinal variables. Chi-square and Fisher's exact tests were utilized for analysis of nominal variables. To account for multiple comparisons in our subgroup analyses, we calculated Benjamini–Hochberg critical values using a false discovery rate of 0.05 and compared each test's P value with its corresponding Benjamini–Hochberg critical value to determine significance.

This study was exempted by the Oregon Health & Science University Institutional Review Board.

Results

Demographics

A total of 565 out of 1429 interviewees completed the survey, with a response rate of 40%. Demographics are reported in the Table. Our sample of residency applicants was representative of the Electronic Residency Application Service (ERAS) 2019–2020 applicant population in terms of Asian race and Latinx race.¹³ More respondents were female (63.9% vs 54.4%) and White (55.6 vs 42.4%) and fewer were Black (6.3% vs 9.4%) or categorized as UiM (17.9% vs 19.2%) than the ERAS population.

Table.

Demographic Characteristics of Interviewees and Comparison With ERAS Data

Demographic	Interviewees		ERAS Applicantsa
	N (%)	Disaggregated Mixed Race Categoryb N (%)	N (%)	Weighted Averagec
Gender
Female	361 (63.9)		20 913 (49.6)	54.4
Male	181 (32.0)		21 196 (50.3)	45.5
Non-binary	3 (0.5)
Transgender	1 (0.2)
Prefer not to answer	4 (0.7)
Unknown	15 (2.7)		20 (0.0)	0.0
UiMd
Yes	108 (19.1)	112 (17.9)	8072 (19.2)	19.2
No	437 (77.3)	492 (78.8)	27 067 (64.2)
Unknown	20 (3.5)	20 (3.2)	3377 (8.0)
Race
White	306 (54.2)	347 (55.6)	17 365 (41.2)	42.4
Asian	106 (18.8)	133 (21.3)	9702 (23.0])	22.3
Latinxe	32 (5.7)	53 (8.5)	3694 (8.8)	8.9
Blackf	29 (5.1)	39 (6.3)	3976 (9.4)	9.4
American Indian or Alaska Native	6 (1.1)	12 (1.9)	302 (0.7)	0.7
Native Hawaiian or Pacific Islander	0 (0.0)	7 (1.1)	100 (0.2)	0.2
Multiracial	55 (9.7)
Other	11 (2.0)	13 (2.1)	1684 (4.0)	3.8
Unknown	20 (3.5)	20 (3.2)	1693 (4.0)	3.8
Location
Rural	35 (6.2)
Suburban	185 (32.7)
Urban	331 (58.6)
Unknown	14 (2.5)
GME program
Family medicine	148 (26.2)
Pediatrics	137 (24.3)
OB/GYN	78 (13.8)
Anesthesia	59 (10.4)
Emergency medicine	59 (10.4)
Psychiatry	40 (7.1)
General surgery	34 (6.0)
Clinical psychology	10 (1.8)

Abbreviations: ERAS, Electronic Residency Application System; UiM, underrepresented in medicine; GME, graduate medical education; OB/GYN, obstetrics and gynecology.

^aIncludes only applicants to the same residency specialties as included in our study (excluding clinical psychology) in the 2019–2020 interview cycle.

^bIncluded for comparison to ERAS data. Participants who reported multiple races in the ERAS data are counted separately in each race category, rather than in a mixed-race category.

^cWeighted by residency specialty representativeness in our sample.

^dUnderrepresented in medicine excludes Asian and White.

^eLatinx includes ERAS categories of Latino, Hispanic, or Spanish origin.

^fBlack includes ERAS category of Black or African American.

Effectiveness

A large majority of respondents agreed or strongly agreed that virtual interviews were effective in establishing comfort (469 of 565, 83%) and connection (500 of 565, 89%), demonstrating applicant strengths (488 of 564, 87%), and answering their questions (539 of 564, 96%). A smaller majority of interviewees felt virtual interviews were effective for clinical tours (337 of 564, 60%), learning about the program culture (352 of 565, 62%), institutional social environment (384 of 565, 68%), and city (345 of 564, 61%). Forty-seven percent (265 of 563) of interviewees agreed or strongly agreed they would need to still visit the city before selecting a program. Additionally, when comparing median Likert scores, fewer Black (2.0 [95% CI 2-3] vs 3.0 [95% CI 1-5] to 4.0 [95% CI 2-5] for other races; P=.0009), female (3.0 [95% CI 3-4] vs 4.0 male [95% CI 3-4]; P=.0055), urban, and rural (3.0 [95% CI 3-4] and 3.0 [95% CI 2-3] vs 4.0 [95% CI 3-4] suburban; P=.0025) interviewees agreed that they would need to visit compared with their subgroup counterparts (online supplementary data).

Advantages and Barriers

Survey participants reported more advantages than barriers of virtual interviews (Figure). The most common perceived advantages were cost savings (539 of 555, 97%), time efficiency (505 of 555, 91%), less burdensome scheduling (469 of 555, 85%), and reduced carbon footprint (397 of 555, 72%). The most frequently reported barriers were time zone differences (233 of 555, 42%), access to an appropriate interview setting (148 of 555, 27%), and reliable access to internet (129 of 555, 23%).

Figure.

Perceived Advantages Versus Barriers of the Virtual Interview Season

Women reported more often than men that virtual interviews were less burdensome to their schedule (89.2% [95% CI 86.0-92.4] vs 75.7% [95% CI 69.4-81.9], P≤.0001). No other significant subgroup differences were found.

Eighty-four percent of interviewees (456 of 542) desired some incorporation of the virtual format into future interview cycles, and there were no significant differences in subgroup analysis (online supplementary data).

Discussion

Most interviewees perceived virtual interviews to be effective and preferable as a component of future interview seasons. Applicants perceived virtual interviews to be less effective for communicating program culture and providing adequate socio-environmental information. We did not identify negative impacts on equity in our subgroup analyses of gender, race, UiM status, or rural/suburban/urban location.

The results of our multispecialty study support prior findings demonstrating the effectiveness and acceptability of virtual interviews.^1-12 Others have reported similar advantages of cost savings, time efficiency, and reduced scheduling burden. Carbon footprint reduction was among the most frequently selected advantages in our study, reflecting a similar portion of applicants concerned about the environmental impact of travel as reported by Fung et al (74.4%).¹⁴ Technological issues and suboptimal tours/social information are often cited as main barriers in other studies. Our study echoes these issues and adds time zone differences as a major barrier. To our knowledge, our study is the first to assess gaps in equity of virtual interviews by gender, UiM status, race, or rural/urban/suburban location.^12,15

There were several limitations to our study. Validity may be affected by lack of a traditional interview comparator group and limited validity evidence for the survey. Generalizability is limited by the single collection site, but we did collect a relatively large multispecialty national sample of applicants. Despite a relatively low response rate (40%), our population was representative of the greater ERAS population in many ways. However, our overrepresentation of those identifying as female and White and underrepresentation of those identifying as Black may have limited our study's ability to identify equity gaps, particularly for Black applicants. We did not include objective criteria for urban/rural/suburban which may have introduced heterogeneity in these groups.

Future studies that explore factors affecting diversity, equity, and inclusion, such as the experience of applicants with different abilities or who are from international medical schools would be helpful with purposive or representative samples of the general and underrepresented applicant populations. As US medical students cite geographic location and “goodness of fit” as leading factors in program selection, ongoing attention should be paid to improving methods for conveying the institutional and regional environment.¹⁶

Conclusions

Virtual interviews were perceived as effective, more advantageous than burdensome, and widely acceptable to our respondents, with no significant differences by respondent race, gender, or location.