Do Lower Costs for Applicants Come at the Expense of Program Perception? A Cross-Sectional Survey Study of Virtual Residency Interviews

Abstract

Background

Travel costs and application fees make in-person residency interviews expensive, compounding existing financial burdens on medical students. We hypothesized virtual interviews (VI) would be associated with decreased costs for applicants compared to in-person interviews (IPI) but at the expense of gathering information with which to assess the program.

Objective

To survey senior medical students and postgraduate year (PGY)-1 residents regarding their financial burden and program perception during virtual versus in-person interviews.

Methods

The authors conducted a single center, multispecialty study comparing costs of IPI vs VI from 2020-2021. Fourth-year medical students and PGY-1 residents completed one-time surveys regarding interview costs and program perception. The authors compared responses between IPI and VI groups. Potential debt accrual was calculated for 3- and 7-year residencies.

Results

Two hundred fifty-two (of 884, 29%) surveys were completed comprising 75 of 169 (44%) IPI and 177 of 715 (25%) VI respondents. The VI group had significantly lower interview costs compared to the IPI group (median $1,000 [$469-$2,050 IQR] $784-$1,216 99% CI vs $3,200 [$1,700-$5,500 IQR] $2,404-$3,996 99% CI, P<.001). The VI group scored lower for feeling the interview process was an accurate representation of the residency program (3.3 [0.5] vs 4.1 [0.7], P<.001). Assuming interview costs were completely loan-funded, the IPI group will have accumulated potential total loan amounts $2,334 higher than the VI group at 2% interest and $2,620 at 6% interest. These differences were magnified for a 7-year residency.

Conclusions

Virtual interviews save applicants thousands of dollars at the expense of their perception of the residency program.

Introduction

The residency application, interviewing, and rank list process has always been a psychologically and financially stressful undertaking for applicants.^1,2 While the COVID-19 pandemic introduced challenges to residency recruitment by requiring virtual interviews, an unintentional benefit was the eliminated travel costs associated with in-person interviews, lightening the financial burden on students. Traditionally, the interview process has been costly, with expenses including, but not limited to, airfare, fuel, and overnight accommodations,^3-6 ranging from $4,500 to over $10,000 per recruitment season.^7,8 This cost is especially burdensome, since 73% of medical students graduate with debt, with median borrowing of ∼$200,000 in 2019.³ Since most applicants must finance medical school with loans, additional costs are magnified by interest accrual during residency. Students are acutely aware of these overall costs of medical training, which negatively impact well-being and academic performance, and even drive students to pursue higher paying specialties.⁹

While it is unclear if virtual interviews allow for the same degree of interaction between applicants and residency programs as in-person interviews, virtual interviews are likely to persist.^10,11 As such, the Undergraduate Medical Education to Graduate Medical Education Review Committee encourages the “ongoing study of the impact and benefits of virtual interviewing as a permanent means of interviewing for residency.”¹² Given the uncertainty of the future of the residency interview process, applicants and programs alike must be kept abreast of the impact of the virtual interview process.

The objective of our study was to survey senior medical students and postgraduate year (PGY)-1 residents regarding the financial burden of virtual versus in-person interviews. We hypothesized the virtual interview (VI) group would appreciate significant financial cost savings, at the expense of gaining deeper, more intimate insight into the programs at which they interviewed as compared to those in the in-person interview (IPI) group.

Methods

Settings and Participants

UPMC is an integrated health care provider and insurance system home to a multitude of graduate medical education programs, including 43 residency programs with primary teaching sites at 11 separate Pennsylvania hospitals geographically dispersed in urban and rural areas. Teaching sites include tertiary care referral hospitals, a free-standing pediatric hospital, a free-standing psychiatric hospital, a women and infants hospital, and several community-based hospitals. UPMC is affiliated with the University of Pittsburgh School of Medicine (UPSOM).

Objectives

To survey senior medical students and postgraduate year 1 residents regarding their financial burden and program perception during virtual versus in-person interviews.

Findings

The virtual interview group had significantly lower interview costs compared to the in-person interview group, but the virtual interviewers scored lower for feeling the interview process was an accurate representation of the residency.

Limitations

This was a single center study and also retrospective, which may have introduced recall bias.

Bottom Line

Virtual interviews save applicants thousands of dollars at the expense of their perception of the residency program.

We conducted a single center, multispecialty survey study of UPMC PGY-1 physicians and fourth-year medical students applying to residency during the 2020-2021 recruiting cycle. The IPI group (cohort A) comprising PGY-1 UPMC residents was surveyed in the spring of 2021. The IPI group was compared to 2 groups who went through the VI process. The VI group included UPSOM students (cohort B) who matched in spring 2021 and external medical school students (cohort C) who matched at UPMC at the same time. Figure 1 depicts the sampling timeline for each of the groups. While we surveyed the financial impact of interviews in all groups, we surveyed only the interview perceptions of the program in cohorts A and C, given cohort B comprised UPSOM students who matched elsewhere in the country and were outside of our institution for their 2021 PGY-1.

Figure 1.

Timing of Surveys Among the Various Study Cohorts

Abbreviations: MS4, medical student year 4; PGY, postgraduate year; MS3, medical student year 3; UPSOM, University of Pittsburgh School of Medicine.

Note: The survey was sent to cohorts A and B in spring 2020 and to cohort C in spring 2021. The financial analysis comprised responses from all 3 cohorts. The perceptions of the interview's representation of the program were compared between cohorts A and C.

Intervention

Our study group agreed that a survey was the appropriate tool for determining interview costs and applicants' perceptions of their residency programs. We developed our survey instrument to determine direct interview costs and other factors that may have indirectly affected costs. We then created a 13-item instrument aimed at assessing residents' accuracy of their perceptions of their program from their interview experience. All survey items were derived by content experts, such as a Designated Institutional Official (G.M.B.), an Associate Dean for Student Affairs (A.M.G.), Co-Chairs, UPMC Medical Education Professional Development Subcommittee (J.B.M., M.F.H.), a Senior Vice Chair of Education (D.G.M.), and Residency Program Directors and Recruitment Committee Chair (P.S.A., E.A.U., L.S.). All items were reviewed, and any leading, double-barreled, or negatively worded items were removed or edited. All items received consensus agreement for final inclusion.

A voluntary, anonymous survey hosted by REDCap (Vanderbilt University) was sent to participants' institutional emails (see online supplementary data). Medical students received a series of 3 emails from the Office of Medical Education at UPSOM between April and May 2021. Residents received a series of 3 emails drafted by the Graduate Medical Education Office and sent by their program directors between April and May 2021 for cohort A and between July and August 2021 for cohort C. These were one-time surveys and not a longitudinal series.

Outcomes Measured

All participants were surveyed about the type of PGY-1 program applied to, whether they participated in couples matching, and baseline student loan debt. The financial survey assessed the number of applications and interviews conducted, geographic limitations, and costs associated with the application and interview process. Cohorts A and C were surveyed about how well their perceptions about the programs during the interview matched reality once residency had commenced (provided as online supplementary data).

The long-term financial impact of IPI and VI upon residency graduation was calculated using the annual compound interest formula.¹³ Several assumptions were made for the calculation. The median interview costs for IPI and VI were used as the principal amount assuming that these costs would have been covered through additional loans, through existing funds from prior loans, or from funds that could have otherwise been used to pay off existing loans. We used the expense amount instead of additional loan amount, as it better encompasses the opportunity cost associated with interviewing, and because many medical schools account for increased interview expenses in their financial aid package in the final year of medical school. The time variable was set to 3 and 7 years reflecting the common minimal and maximal duration of residency: 3 years (internal medicine, family medicine, pediatrics, emergency medicine) to 7 years (surgical specialties). Interest rates were assumed to be either 2% (common rate after refinancing) or 6% (average federal loan interest rate).³ The debt was modeled assuming no principal payments were made during residency.

Analysis of Outcomes

Data are presented as counts with percentages, means (standard deviation), or medians (interquartile range). Continuous data were assessed for normality using the Shapiro-Wilk test and assessing histograms. No continuous data were transformed. Our primary objective was to assess the financial impact of VI vs IPI on applicants. Thus, we utilized Wilcoxon rank-sum testing to compare the median costs between the VI vs IPI groups. Wilcoxon rank-sum testing and Pearson chi-square testing (Fisher exact testing for expected values <5) were utilized to assess other variables associated with the financial impact of VI vs IPI. Given we assessed 11 variables, we chose to utilize Bonferroni correction of the P value for multiple comparisons and accepted P=.05/11=.0045 to define statistical significance. Next, we aimed to assess perceptions of the residency programs based on the interview between the IPI and VI groups. This assessment comprised a 13-item instrument (1=strongly disagree, 2=disagree, 3=neutral, 4=agree, 5=strongly agree). Data were compared using independent Student's t tests. Because we assessed 13 items, we chose to utilize Bonferroni correction of the P value for multiple comparisons and accepted P=.05/13=.0038 to define statistical significance.

Given the heterogeneity of our data regarding the diversity of trainees in the study and the various medical specialties for which they interviewed, we chose to utilize 99% confidence intervals (median confidence intervals derived as described by McGill et al).¹⁴ We utilized Cohen's d to determine effect sizes for continuous data and Cohen's h for proportional effect sizes. Effect size of 0.2 was considered small, 0.5 medium, 0.8 large, and 1.3 very large.¹⁵ Statistical analysis was completed using Stata/SE 16.1 (StataCorp LLC). This study was granted exempt status by the University of Pittsburgh Institutional Review Board.

Results

Participants

There were 252 of 884 (29%) responses to the surveys comprising 73 of 150 (49%) UPSOM students and 169 of 734 (23%) PGY-1 residents. We received 75 of 169 (44%) responses from cohort A (IPI), 73 of 150 (49%) from cohort B (VI), and 94 of 169 (56%) from cohort C (VI) (Figure 1). Ten in the VI group did not respond to level of training. There were no major baseline differences between the IPI and VI groups other than small-medium effect sizes for more in the VI group applying to surgical preliminary years and having no student debt (Table 1).

Table 1.

Baseline Variable Comparison Between Those in the In-Person vs Virtual Interview Groups

Variable	In-Person Interviews (N=75), n (%)	Virtual Interviews (N=177), n (%)	Cohen's h
PGY-1 position applied to
Categorical	74 (99)	172 (97)	0.108
Transitional	8 (11)	16 (9)	0.057
Preliminary-medicine	6 (8)	18 (10)	0.077
Preliminary-surgery	0 (0)	8 (5)	0.428
Couples Match	9 (12)	15 (9)	0.115
Current student loan debt
None	9 (12)	43 (24)	0.32
$0-50K	6 (8)	13 (7)	0.026
$51-100K	2 (3)	8 (5)	0.102
$101-150K	6 (8)	7 (4)	0.174
$151-250K	20 (27)	39 (22)	0.112
>$250K	25 (34)	58 (33)	0.017
Prefer not to answer	6 (8)	8 (5)	0.145

Abbreviation: PGY, postgraduate year.

Note: Cohen's h calculated for proportion effect sizes.

Financial Impact

The VI group had significantly lower interview costs compared to the IPI group (median $1,000 [$469-$2,050 IQR] $784-$1,216 99% CI vs $3,200 [$1,700-$5,500 IQR] $2,404-$3,996 99% CI, P<.001). There were no significant differences between the number of applications submitted, specialties applied to, invitations received, interviews accepted, or geographical interview limitations (Table 2). A greater proportion in the VI group did not require additional loans for interviewing, while a greater proportion of the IPI group obtained $2,501 to $5,000 in additional loans for interviewing (Table 2). The odds of taking out an additional loan(s) for the application and interview process for the VI group were 0.33 times that of the IPI group (0.13-0.81 99% CI, P=.001).

Table 2.

Comparison of the Number of Interviews Applied for, Invited to, and Accepted, as Well as the Costs Associated With Interviewing Between the In-Person vs Virtual Interview Groups

Variables	In-Person Interviews (N=75)		Virtual Interviews (N=177)		Cohen's d/h	P Value
	Value	99% CI	Value	99% CI
Total interview costs (US $)	3200 (1700-5500)	2404-3996	1000 (469-2050)	784-1216	0.88	<.001
Number of applications	38 (28-68)	30-46	45 (29-80)	38-52	0.24	.19
Number of specialties applied to	1 (1-1)	1-1	1 (1-1)	1-1	0.159	.08
Number of interview invitations	16 (12-23)	14-18	15 (11-21)	14-16	0.168	.26
Number of accepted interviews	13 (11-15)	12-14	14 (10-17)	13-15	0.073	.45
Interview geographic limitations, n (%)
No limitations	39 (52)		95 (54)		0.04	.40
US region	31 (41)		61 (35)		0.136
1 state + surrounding state(s)	5 (7)		20 (11)		0.165
Application fees (US $)	1000 (500-1700)	749-1252	1000 (429-2000)	786-1214	0.112	.69
Travel/flight costs (US $)	1500 (800-2500)	1144-1856	0 (0-0)	0-0	2.236	<.001
Hotel/lodging costs (US $)	1000 (500-2000)	686-1314	0 (0-0)	0-0	2.217	<.001
Additional costs (US $)	300 (150-500)	227-373	100 (50-200)	80-121	0.485	.06
Additional loans for interviews, n (%)
None	53 (72)		154 (89)		0.432	.004
$1-2500	4 (5)		10 (6)		0.013
$2501-5000	8 (11)		3 (2)		0.408
$5001-7500	3 (4)		2 (1)		0.188
$7501-10000	1 (1)		1 (< 1)		0.082
>$10000	1 (1)		1 (< 1)		0.082
Prefer not to answer	4 (5)		3 (2)		0.208

Note: Cohen's d calculated for continuous data effect sizes. Cohen's h calculated for proportion effect sizes. Bonferroni adjustment to the P value: .05/11=.0045.

Debt Projections

Extrapolating from our observed median total interview expense amount ($3,200 for IPI, $1,000 for VI), we calculated the projected debt assuming costs were 100% financed through loans as described in the methods. After a 3-year residency, IPI applicants would have accumulated $2,334 and $2,620 more debt than the VI group at 2% and 6% interest rates, respectively (Figure 2). This difference is magnified in applicants after a 7-year residency, for which IPI applicants would have accumulated $2,527 and $3,307 more debt than the VI group at 2% and 6% interest rates, respectively. Utilizing a 10-year loan payoff projection, after a 3-year residency and at 6% interest, the IPI group will have spent an additional $4,692 (10-year payoff of $6,825 for the IPI group and $2,133 for the VI group).

Figure 2.

Median Principal Amount of Additional Loans Acquired for Residency Application and Interview Process as Well as Accrued Interest Throughout Either 3 or 7 Years of Postgraduate Training

Abbreviations: IPI, in-person interviews; VI, virtual interviews; i, interest rate.

Note: Solid black bars represent the 3-year amounts. The 7-year amounts are the sum of the black and grey bars (ie, total bar amount). Dashed line indicates median principal amount.

Interview Representation of Program

All PGY-1 trainees (both IPI and VI participants) were asked to complete a supplemental 13-item Likert response survey. Of the 169 PGY-1 trainees, 61 of 75 (89%) in the IPI group and 50 of 94 (53%) in the VI group responded. Cronbach's alpha for our instrument was 0.77. Trainees in PGY-1 who participated in virtual interviews responded with significantly lower ratings for being able to appreciate resident camaraderie and culture, gaining an understanding of living in the city, getting a feel for daily life, and developing rapport with faculty and residents during the interview process (Table 3). The VI group felt the interview process was less reflective of how the program turned out to be as compared to the IPI group (Table 3). Assessment of satisfaction with residency choice was among the highest-rated items, and there was no significant difference between the IPI and VI PGY-1 trainees (Table 3).

Table 3.

Survey Assessing Applicants' Perceptions of Residency Programs Compared Between In-Person and Virtual Interview Formats

Statement	In-Person Interviews (N=61)		Virtual Interviews (N=50)		Cohen's d	P Value
	Mean (SD)	99% CI	Mean (SD)	99% CI
1. I was able to get a good impression of the residency camaraderie and culture during my interview process.	4.4 (0.7)	4.2-4.7	3.3 (1)	2.9-3.6	1.413	<.001
2. I was able to gain an understanding of what living in the city the residency program I was interviewing at would be like.	4.2 (0.8)	3.9-4.5	2.6 (1)	2.3-3	1.786	<.001
3. I got a feel for what daily life would be like at the programs I interviewed.	3.9 (0.8)	3.7-4.2	3.1 (0.9)	2.8-3.5	0.933	<.001
4. I felt that I gained a good understanding of additional opportunities at the program for residents (research, advocacy, moonlighting, etc).	4.2 (0.8)	4-4.5	4 (0.8)	3.7-4.3	0.25	.19
5. The pre-interview dinner/social was instrumental in making my rank list.	3.9 (1.1)	3.5-4.3	3.2 (1.1)	2.8-3.6	0.676	.001
6. The social media presence of the residency affected my ability to learn about the program.	2.6 (1.3)	2.1-3	3 (1.2)	2.6-3.5	0.373	.05
7. Educational and informational content of their website was crucial in learning about the program.	4 (1)	3.6-4.3	4.5 (0.8)	4.2-4.8	0.561	.004
8. I feel that I was able to make a positive impression with faculty during my interview.	4.5 (0.6)	4.3-4.7	4.2 (0.8)	3.9-4.5	0.467	.016
9. I feel that I was able to develop rapport with faculty and residents during my interview experience.	4.4 (0.7)	4.2-4.6	3.8 (0.9)	3.4-4.2	0.746	<.001
10. The interview process greatly influenced my final rank list.	4.5 (0.7)	4.3-4.8	4.1 (1)	3.7-4.5	0.471	.015
11. I feel that an open house separate from my interviews would have helped me make my rank list.	3 (1.1)	2.6-3.3	3.6 (1)	3.2-4	0.628	.001
12. I feel the interview process was a good representation of what the residency program actually turned out to be after arriving.	4.1 (0.7)	3.9-4.4	3.3 (0.5)	3.1-3.5	1.291	<.001
13. I am satisfied with my choice of residency.	4.7 (0.6)	4.5-4.9	4.5 (0.9)	4.1-4.8	0.255	.19

Note: Cohen's d calculated for continuous data effect sizes. Bonferroni adjustment to the P value of .05/13=.0038.

Discussion

When compared to the in-person interview system, our study showed that the virtual interview process was associated with significantly lower expenditures for residency applicants. We observed the cost of virtual interviews was less than one-third that of in-person interviews for an applicant over a recruitment cycle. This difference becomes amplified when accounting for accrued interest over the course of their residency. Theoretically, a resident at the conclusion of a 7-year program who did not refinance their loans would save an estimated $3,307 with virtual interviews compared to in-person interviews. However, these financial savings appear to come at the expense of gaining an impression of resident camaraderie, understanding what living in the city might be like, getting a feel for daily life in the program, and developing rapport with the interviewing faculty. There was significantly less agreement that the interview represented what the residency turned out to be like in the VI group, which may be due to these aforementioned factors.

Previous studies have investigated the cost of IPI for residents and observed similar findings. The range of average IPI costs was found to be $3,000 to >$10,000, with surgical subspecialties incurring higher costs.^5,6,8,16,17 However, no previous study has been conducted to project the long-term cost differential between the 2 interview formats across multiple specialties in a large academic institution. When interest accrual is considered—whether it be from new loans, money from prior loans, or money that could have been used to pay down existing loans—the financial impact is further augmented. Additionally, our study found that VI applicants are more able to accommodate multiple interviews in one day due to lack of travel limitation between cities, thus providing additional unrealized cost savings. Similar observations were made in other studies, and VI was associated with fewer missed clinical days secondary to lack of travel requirements and interview consolidation.¹⁸ Benefits have been observed for residency programs as well.¹⁹ A survey of surgery program directors found an average saving of $600 per applicant with VI compared to IPI.¹¹

Ultimately IPI cost more upfront and in the long run, but are the differences in program perception significant enough to warrant that cost? Virtual recruitment is likely to stay, whether as purely virtual or as a part of a hybrid format.^11,12 Therefore, it will be critical to evaluate the benefits and disadvantages as compared to traditional in-person recruiting. Given our results suggest efforts at exemplifying resident camaraderie, the program city, daily life, and building rapport with faculty may help to provide a more accurate picture of the program, future studies should be aimed at determining which factors provide applicants the most authentic representation of the residency program. In addition, examining whether a hybrid-type format provides both applicants and programs a more robust perception of one another is warranted.

Limitations

First, this study was conducted at a single, large, urban academic institution. The applicants who are drawn to this system may not be the same demographic as applicants more interested in community-based programs and who may have different residency recruitment strategies. While our UPSOM medical student response rate was 49%, we acknowledge the low response rate of 23% from PGY-1 residents. The low PGY-1 response rate is likely multifactorial and may include survey fatigue and our reliance on individual program directors to distribute to their residents. Also, this study is retrospective and may be subject to recall bias. While we tried to limit this by encouraging respondents to refer to credit card statements, receipts, etc. for costs, the accuracy of the cost estimates cannot be guaranteed. Additionally, our survey did not assess applicants by specialties. While this makes the study more generalizable to the average applicant, it also leads to the possibly that a few applicants with highly competitive specialties could have skewed the results. Finally, the 13 items in our program perception survey were developed by the study authors, and respondents may not have interpreted these items as intended.

Conclusions

This single center, multispecialty survey study suggests virtual interview formats save applicants ∼$2,000 in initial costs and potentially another ∼$2,000 from accrued interest if interview costs are financed with loans. However, while we observed these financial savings, responses from the VI group suggest impairment in perceiving resident camaraderie, understanding the city, getting a sense of daily resident life, and feeling that the interview was not representative of what the program actually turned out to be.

Funding Statement

Funding: The authors report no external funding source for this study.