Impact of Virtual Recruitment on Costs, Time Spent, and Applicant Perspectives Within a Family Medicine Residency Program

Abstract

Background

Virtual recruitment for all residency programs was endorsed by the Accreditation Council for Graduate Medical Education (ACGME) for the 2021 and 2022 recruitment seasons. This study assesses the impact of virtual recruitment on cost and outcome in a family medicine residency program.

Methods

We assessed program recruitment costs and interview-day time with applicants in one program for the 2019 to 2022 recruitment seasons, and we sent an anonymous survey to interviewed applicants (n=98) for the 2022 match year. In-person interviews were conducted in 2019 and 2020. Virtual interviews were conducted in 2021 and 2022.

Results

Program recruitment costs decreased from over $70,000 annually for in-person interview seasons to between $10,000 and $20,000 annually for the virtual interview years. Applicant time with the program on interview days decreased from 515 minutes when held in-person, to 345 minutes when virtual. Applicants expressed that they were generally satisfied with the virtual interview format though their preference for the virtual format was only slightly greater than for in-person interviews (38.6% and 35.1%, respectively); 26.3% of the responding applicants had no preference for either format. During virtual interview years, applicants interviewed at an average of 16.6 programs with 80% indicating that virtual interviews allowed for consideration of more programs.

Conclusion

The virtual interview format was associated with decreased interview-day costs for programs and interviewees, and decreased time on interview days for both groups. It allowed applicants to consider more programs.

Background

Prior to 2020, few residency programs used virtual recruitment to fill incoming classes,^1,2 despite recruitment being an historically important, time intensive, and costly effort.³ During the COVID-19 pandemic, the Accreditation Council for Graduate Medical Education (ACGME) advised all programs to conduct interviews virtually for the 2021 recruitment season.⁴ Virtual recruitment was then strongly encouraged for the 2022 season. Subsequently, the ACGME called for research on the virtual interview process.⁵ Studies examining virtual recruitment within family medicine residency programs remains scarce.

This study aimed to analyze the impact of virtual recruitment within one university-based program in three areas: financial cost to the program, time spent by applicants with the program, and applicant satisfaction with the virtual process. Additionally, given this program’s focus on increasing equity for applicants from diverse backgrounds,⁶ this study analyzed the impact of virtual recruitment on applicants who self-identify as being underrepresented in medicine (URiM).

Methods

We assessed recruitment data at the University of Utah Family Medicine Residency program for the 2019 through 2022 recruitment seasons. Expenses incurred during calendar years 2018 and 2019 (covering 2019 and 2020 seasons), were reviewed and compared to the expenses incurred during calendar years 2020 and 2021 (covering 2021 and 2022 seasons). We compared staff, resident, and faculty time on interview day for the same years.

For the 2022 season, an anonymous survey was sent to interviewed applicants via SurveyMonkey (Momentive), after applicants certified their rank lists but before match day. The survey consisted of multiple choice, Likert scale, and free response questions. Questions examined the program’s ability to represent itself virtually and applicant satisfaction with the virtual interview format (see Appendix). For comparing data between URiM and non-URiM interviewees, we used Pearson χ², continuous, and t tests. We set statistical significance at .05. All statistical tests were two-sided. This study was included under an umbrella exemption from review, provided by the University of Utah Institutional Review Board.

Results

Finances and Time

Program Expenditures. Recruitment expenses for in-person seasons of 2019 and 2020 were $70,355 and $70,604, respectively. Recruitment expenses for virtual recruiting years of 2021 and 2022 were $21,949 and $9,932, respectively. Most expenses for in-person interviews were related to lodging and meals for interviewees. Additional expenses for both in-person and virtual interviews were for travel for program representatives to out-of-state recruitment events, and meals for program representatives (Table 1).

Table 1.

Recruitment Expenses through years 2018–2021

Expense	2018	2019	2020	2021
Advertising	$722	$1,493	$180	$0
Business meals/entertainment	$25,381	$26,336	$6,618	$9,483
Printing and photocopying	$2,038	$956	$269	$408
Travel: domestic, in-state	$5,022	$4,350	$2,963	$41
Travel: domestic, out-of-state	$24,478	$20,294	$2,947	$0
Applicant lodging	$12,715	$17,175	$8,972	$0
Grand Total	$70,355	$70,604	$21,949	$9,932

^*2018 and 2019 reflect in-person recruitment years while 2020 and 2021 reflect virtual recruitment years.

Time Spent by Applicants With the Program. Applicants spent 515 minutes directly engaging the program on in-person interview days and 345 minutes on virtual days (Table 2). The reduced time was due to the elimination of in-person facility tours, lunch, and travel time between sites and shorter welcome session and interview times. Virtual interview-day interactions were conducted via the Zoom platform. In 2019 and 2020 there were 9 interview days, compared to 11 days in 2021 and 2022.

Table 2.

Time Spent in Minutes by Each Applicant in Contact With the Program on Interview Day

	2019	2020	2021	2022
Welcome and introduction	70	70	60	60
Tour	90	90	0	0
Lunch	50	50	0	0
Travel with resident guide	20	20	0	0
Interviews	150	150	150	140
Wrap-up	15	15	15	25
Evening dinner/social	120	120	120	120
Total Time (in minutes)	515	515	345	345

Survey Results

Out of 60 responders (response rate 67%), 70% (n=42) were female, and 20% were URiM (Table 3). Regarding participation, 3.5% stated they would not have interviewed if interviews were held in person, 80.4% indicated that virtual interviews allowed them to interview with more programs, and 82.8% indicated they would have needed to fly to interview with the program. In addition, 45.2% indicated that finances would have been a barrier to interviewing in person. Respondents were split on preferred format, with 38.6% preferring virtual, 35.1% in-person, and 26.3% no preference. However, URiM candidates had a higher preference for the virtual interview format (54.5%) compared to non URiM candidates (34.8%; Table 4).

Table 3.

Participant Sociodemographics (n=60)

Variable		All participants	Female	URiM
Age (years)
	22–25	13.3% (8)	16.7% (7)	25.0% (3)
	26–30	71.7% (43)	66.7% (28)	41.7% (5)
	>30	10.0% (6)	9.5% (4)	33.3% (4)
	Not disclosed	5.0% (3)	7.1% (3)	25.0% (3)
Gender
	Male	28.3% (17)	----	16.7% (2)
	Female	70.0% (42)	----	75.0% (9)
	Not disclosed	1.7% (1)	----	8.3% (1)
URiM
	Yes	20.0% (12)	32.2% (9)	----
	No	80.0% (48)	68.8% (33)	----
	Not disclosed	0.0%	0.0%	----

Abbreviation: URiM, underrepresented in medicine

Table 4.

Virtual vs In-Person Interview Process

		All Participants1	Non-URiM1	URiM1	P Value2
Would you have chosen to interview with the UU FM program if it had only been in person? (n=58)
	Yes	70.0% (40)	70.2% (33)	63.6%(7)	.520
	No	3.5% (2)	2.1% (1)	9.1% (1)
	Maybe	27.6% (16)	27.7% (13)	27.3% (3)
Would finances have been a barrier to interview with the UU FM Program? (n=42)
	Yes	45.2% (19)	40% (14)	71.4% (5)	.309
	No	23.8% (10)	25.7% (9)	14.3% (1)
	Maybe	31.0% (13)	34.4% (12)	14.3% (1)
Would you have needed to fly to Salt Lake City if interviews had been in person? (n=58)
	Yes	82.8% (48)	83.0% (39)	81.8 (9)	.479
	No	13.8% (8)	14.9% (7)	9.1% (1)
	Maybe	3.4% (2)	2.1% (1)	9.1% (1)
Did virtual interviews allow you to consider more programs than you would have if it had been in person? (n=56)
	Yes	80.4% (45)	82.2% (37)	72.7 (8)	.437
	No	3.6% (2)	4.4% (2)	0.0 (0)
	Maybe	16.1% (9)	13.3% (6)	27.3 (3)
Which interview format do you prefer? (n=57)
	Virtual	38.6% (22)	34.8% (16)	54.5% (6)	.132
	In person	35.1% (20)	41.3% (19)	9.1% (1)
	No preference	26.3% (15)	23.9% (11)	36.4% (4)

Abbreviations: URiM, underrepresented in medicine; UU, University of Utah.

¹Mean (SD);

²Pearson χ²

Respondents interviewed at, on average, 16.6 programs virtually and 0.4 programs in-person. URiM respondents interviewed virtually at four more programs on average than their non-URiM counterparts (19.9 vs 15.9; P=.052). Non-URiM respondents on average attended 0.5 in-person interviews, whereas URiM respondents attended no interviews in person (0.5 vs 0; P=.033). This suggests a statistically significant difference between non-URiM and URiM in-person interviews (Table 5).

Table 5.

Evaluation and Frequency of Virtual Interviews

	All participants1	Non-URiM1	UriM1	P value4
How well did the virtual interview help you understand the UU FM program? (n=57)2	4.0 (0.8)	4.0 (0.7)	4.0 (1.0)	.435
How well did the virtual interview allow you to distinguish differences between programs? (n=56)2	3.5 (0.9)	3.4 (0.9)	3.8 (1.0)	.100
How satisfied were you with the virtual interview process? (n=56)2	4.0 (0.6)	4.0 (0.6)	4.2 (0.6)	.153
How helpful was the introductory presentation and slideshow? (n=52)2	4.3 (0.8)	4.3 (0.8)	4.5 (0.5)	.306
How was the afternoon interview process and time spent in the chat room? (n=52)2	4.2 (0.8)	4.2 (0.9)	4.3 (0.8)	.362
During this recruitment season, how many interviews did you attend virtually? (n=54)3	16.6 (7.1)	15.9 (6.9)	19.9 (7.4)	.052
During this recruitment season, how many interviews did you attend in person? (n=54)3	0.4 (0.8)	0.5 (0.9)	0.0	.033

Abbreviations: URiM, underrepresented in medicine; UU, University of Utah; FM, family medicine.

¹Mean (SD);

²Scale: 1–5 (5=best);

³Continuous;

⁴t test.

Conclusion

The financial benefits of virtual recruitment were substantial; transitioning from in-person to virtual format saved the program about $50,000 annually. Cost savings due to reduced air travel were noted by 82.8% of respondents. Nearly half of respondents indicated that costs would have been a barrier interviewing with this program. The average cost of a roundtrip flight within the United States during quarter 3 of 2021 was $314.39.⁷ The cost of flying to 16.6 in-person interviews could have totaled $5,218.87 per applicant in flights alone. Prior studies suggest applicants are particularly interested in virtual interviews when distance is a variable.¹ The reduced costs to applicants of virtual interviews may allow them to interview with more programs.

However, virtual recruitment is associated with the loss of some interview day activities, including physical hospital, clinics, and community tours, and in-person meals. Eliminating the tours and lunch resulted in lost opportunities for applicants to gain insight into the program by seeing program and community facilities and sharing casual conversations.

In the 2022 recruitment season, this program did not fill during the initial Match. As a result, the program participated in the Supplemental Offer and Acceptance Program through the National Resident Matching Program, which required added time commitments from staff, faculty, and residents. It is unclear if the failure to fill is linked to virtual recruiting. Further study comparing program personnel engagement in in-person and virtual recruitment, and related success in the match, will be beneficial.

Overall, survey respondents were satisfied with the virtual process, and reported sufficient understanding of the program, but also reported less ability to distinguish between programs. The virtual format may allow applicants to consider more programs. Whether the increased number of interviews is beneficial for applicants remains to be determined; if individual applicants interview at extremely high numbers of programs, there are fewer interview opportunities for other applicants. Additionally, the difference between URiM and non-URIM in-person attendance should be further studied.

A limitation of this project is that data were collected from a single program. Additionally, the URiM applicant sample size was small. Strengths include the longitudinal nature of objective outcome measures. While the COVID-19 pandemic created challenges, it provided opportunities to expand understanding of virtual recruitment options for family medicine residencies.