Abstract
Background:
For the 2019–2020 interview season, the anesthesia residency program at Augusta University offered candidates a choice between in-person (IP) and video conference (VC) interviews to accommodate a greater number of qualified candidates.
Methods:
The same applicant selection criteria were used for both interview types. However, we modified the informal interactions with residents, campus tours, and interview formats for VC interviews. We sought to compare the 2 methods by analyzing the respective costs, benefits, and match results.
Results:
Of 159 candidates interviewed, we ranked 127 and matched with 12. The IP (n = 135) and VC (n = 24) groups were similar in gender distribution but not by the type of medical school, with more international medical graduates interviewing by VC than IP. There was no statistically significant difference between the 2 interview types for being ranked (81% of IP, 71% of VC) or matched (6% of IP, 17% of VC). US Medical Licensing Examination Step 1 and Step 2 scores and type of medical school did not affect the likelihood of being ranked or matched. Program costs per candidate were higher for the IP group ($431 for IP, $294 for VC).
Conclusion:
Our single-center study indicates that the interview type did not affect the likelihood of a candidate being ranked by or matched to our program. Further, VC interviews were more cost-effective and time-effective than IP interviews. Our findings suggest that VC interviews are a viable alternative and should be an option for residency interviews.
Keywords: Interview, in-person, virtual, web-based, video conference
Introduction
The residency interview process allows programs and candidates to assess one another. It is key to evaluating the fit of a candidate for a specific residency program, ensuring that resident candidates rank appropriate programs.1 The interview season traditionally requires candidates to travel to multiple locations for face-to-face interactions with faculty, staff, and current residents, thus generating costs in time and money for both applicants and programs.2,3 Given the need to support the nation’s public health efforts during the COVID-19 pandemic, the Association of American Medical Colleges issued a statement mandating virtual-only interviews for the 2020–2021 interview season. The shift to a virtual experience required both interviewees and programs to adapt and get creative. There is a paucity of comparative data between in-person (IP) and virtual interviews, and our 2019–2020 data will inform programs as they pivot to a mixed option in the future.
Residency programs use cognitive data such as test scores and medical school grades to screen applicants. On interview day, programs evaluate candidates’ noncognitive and interpersonal skills and determine their compatibility with the program. Candidates find one-on-one interviews with program faculty, interactions with current residents, and tours of the program’s facilities to be valuable in determining if they are a good fit for the program.4 They use the interview itself to form subjective and objective opinions about program quality and compatibility. Such interactions enable residency programs and candidates to assess one another and prepare rank order lists that ultimately determine the match outcome.3
In the past, our residency program granted interviews by videoconferencing (VC) on a case-by-case basis (eg, if military obligations prevented a candidate from traveling). For the 2019–2020 interview season, we integrated VC into the interview process to expand opportunities to meet eligible applicants. To determine if VC interviews were an acceptable alternative to IP interviews, we sought to determine the likelihood of ranking and matching to the program based on interview type. We also compared the costs related to the 2 interview methods.
Methods
Applicant Selection
Applicants were screened using standardized, holistic criteria including the Medical Student Performance Evaluation, US Medical Licensing Examination (USMLE) Step 1 and Step 2 scores, medical school transcripts, and letters of recommendation. We also considered anesthesiology rotations, scholarly activity, and whether the applicant had local ties to the region. Candidates selected for interview were asked to choose from among the available IP and VC interview days and a statement advised applicants that IP and VC interviews would be considered equivalent in determining the program rank-order list. VC applicants were allowed to visit the institution and tour with a resident after the interview process if they wished to do so. Neither IP nor VC applicants were allowed to interact with the ranking committee members if they chose to visit for a postinterview second look (or first look in the case of VC applicants).
Interview Process
IP interview candidates had dinner with 4 residents, 1 from each residency class, the evening before their scheduled interview, providing an opportunity to meet with future colleagues in an informal setting. The next morning, the chief residents delivered a presentation about the program and took the applicants on a tour of the adult and children’s hospitals and of the simulation and ultrasound labs. A meet-and-greet with the program director and chair followed, permitting the applicants to ask additional questions. The interview itself was a series of 3 one-on-one interviews with 2 faculty and 1 senior resident. Four faculty and 2 senior residents were assigned each day for IP interviews and each interviewer saw 6 applicants per day. Candidates answered preselected questions for 15 minutes and were allowed to ask questions about the program for 5 minutes.
We conducted VC interviews similarly. VC candidates could either participate in a virtual meet-and-greet with residents or contact current residents by phone before the scheduled interview. Links to 2 videos, a narrated presentation about the program and a virtual hospital tour, were emailed to applicants before their interviews, along with links to various hospital resources (Figure 1). Candidates connected to the interview session with video-capable and audio-capable devices using the virtual meeting software Cisco WebEx (Cisco Systems, Milpitas, CA). Each of the 5 candidates interviewed with 2 faculty and 1 resident as a group rather than individually. Junior faculty were paired with senior faculty to observe and learn from their senior colleagues. Interviewers asked preselected questions for 15 minutes, with 5 minutes reserved for questions from the applicant. This format proved to be more time-efficient and prevented candidates from having to answer the same questions repeatedly.
Figure 1.
Comparison of in-person and virtual interview sessions.
Candidate Evaluations
Interviewers evaluated each applicant independently immediately after each interview, whether IP or VC, then met as a group at the end of the day to debrief, discuss applicants, and create a preliminary rank list of applicants who interviewed that day (Figure 1).
Time and Financial Costs
We estimated the program costs of IP and VC interviews by multiplying the average hourly wage for residents, faculty, and residency coordinators who participated in the interview process by the number of hours spent in the interview process and preinterview dinner. The per-person-hour cost was then calculated by dividing the total cost by the number of applicants in each group. We also calculated the per-applicant costs of Sunday dinners and interview day lunches for IP interviews.
Distance Travelled
We used Google (Google LLC, Mountain View, CA) to estimate the distance between our institution and the state or country from which an applicant was located, rounded to hundred miles. Candidates were divided into 5 categories: 0 to 500, 501 to 1000, 1001 to 1500, 1501 to 2000, and more than 2000 miles away from our city.
Interviewer and Matched Applicant Satisfaction Survey
We sought to gain insight into how the applicants viewed the interview process by querying our current postgraduate year one residents, who interviewed during the 2019–2020 cycle, about their thoughts in a confidential survey using SurveyMonkey (San Mateo, CA). We also confidentially surveyed our faculty and residents who interviewed resident candidates for the 2019–2020 season. Response options were on a 5-point Likert scale, and all questions provided the option for feedback in free-text comments boxes. The survey questions were a modification of a previously validated electronic questionnaire by an epidemiologist and 2 experts on virtual learning used in a cross-sectional study designed for medical students and residents to gauge their perception of virtual interviews for residency applications in the United States.5
Data Analysis
The χ2 test of independence and post-hoc Bonferroni correction were used to determine whether VC candidates had an equal chance of being ranked and matched to the program as those who had a more interactive IP interview using Microsoft Excel (Microsoft Corp, Redmond, WA). Additionally, we assessed the choice of IP and VC interviews and rank and match outcomes, according to applicant Step 1 and Step 2 scores, by performing independent-samples t test and logistic regression in RStudio version 1.3.1093 (Boston, MA) to search for significance at the P = .05 level. Furthermore, we analyzed the choice of IP and VC interviews, and rank and match outcomes to our program, according to whether the applicant would be a graduate of an osteopathic (Commission on Osteopathic College Accreditation), allopathic (Liaison Committee on Medical Education), or international medical school. We then analyzed the time and monetary costs per candidate for IP and VC interviews.
Results
For the 2019–2020 interview season, there were 1047 anesthesia residency candidates. Of the 465 applicants who passed our program’s screening, 159 were interviewed (135 [85%] by IP, 24 [15%] by VC), 125 were ranked, and 12 matched. The gender breakdown of IP and VC applicants, average Step 1 and Step 2 scores, and school type can be found in Table 1.
Table 1.
Gender Breakdown, Average Step 1 and Step 2 Scores, and School Type of IP and VC Applicants
| Characteristic | In Person, n = 135 | Virtual, n = 24 | P |
|---|---|---|---|
| Male gender, n (%) | 99 (73) | 16 (67) | >.05 |
| School type, n (%) | |||
| Allopathic medical school (LCME) | 63 (47) | 6 (25) | |
| Osteopathic medical school (DO) | 41 (30) | 1 (4) | |
| International medical school (IMG) | 31 (23) | 17 (71) | |
| Step 1 scores, mean ± SD | 231 ± 11 | 225 ± 11 | >.05 |
| Step 2 scores, mean ± SD | 239 ± 16 | 237 ± 11 | >.05 |
| Ranked, n (%) | 110 (81) | 17 (71) | >.05; |
| Matched, n (%) | 8 (6) | 4 (17) | χ2 = 3.40 |
Abbreviations: DO, doctor of osteopathy; IMG, international medical graduate; IP, in-person; LCME, Liaison Committee on Medical Education; VC, videoconference.
Rank and Match Outcomes According to Interview Format Preference
We ranked 110 (81%) IP and 17 (71%) VC candidates and subsequently matched with 8 (6%) IP and 4 (17%) VC candidates. (Table 1). Comparing the proportion of candidates who interviewed by IP and VC to their corresponding rank and match outcomes, there were no significant difference in the distribution of interview format within the subsequent rank (χ2 (df = 1, n = 159) = 0.17, P > .05) or match (χ2 (df = 1, n = 159) = 2.71, P > .05) outcomes. Furthermore, applicants’ choice of conducting interviews by IP or VC format did not impact their likelihood of being ranked or matched (χ2 (df = 1, n = 139) = 3.40, P > .05).
Interview Format Preference, Rank, and Match Outcomes According to Step 1 and 2 Scores
Analysis by independent-samples t test and logistic regression demonstrated that Step 1 and 2 scores did not impact applicants’ choice of IP or VC interviews. Step 1 and 2 scores were similar for IP [(231 ± 11) (239 ± 16)] and VC [(225 ± 11) (237 ± 11)] interviewees [(t(26) = 1.48, P > .05)] [(t(30) = 0.743, P > .05)] and were comparable between IP ranked [(μ = 230) (μ = 240)] and unranked [(μ = 233) (μ = 237)] candidates, and between VC ranked [(μ = 229) (μ = 239)] and unranked [(μ = 218) (μ = 234)] candidates. Additionally, Step 1 and 2 scores were similar between IP candidates who matched [(μ = 225) (μ = 241)], and VC candidates who matched [(μ = 234) (μ = 238)]. Moreover, Step 1 and 2 scores were similar in ranked IP and VC applicants and in the matched IP and VC applicants (Table 2a). Logistic regression confirmed that Step 1 and 2 scores were not a significant predictor for ranking or matching in the context of interview format (Table 2b).
Table 2a.
T-tests Comparing Ranking and Matching by USMLE Step Scores
| Mean | Variance | t Test | |||||
|---|---|---|---|---|---|---|---|
| Not Ranked IP/VC vs Ranked IP/VC | Ranked IP/VC vs Matched IP/VC | Overall IP vs Overall VC | Ranked IP vs VC | Matched IP vs VC | |||
| Step 1 Scores | |||||||
| IP | |||||||
| Overall | 230.91 | 123.80 | P 2-tail = .15 df = 26 t Stat = 1.48 t Crit 2-tail = 2.06 | ||||
| Not ranked | 233.42 | 145.76 | P 2-tail = .30 df = 27 t Stat = 1.06 t Crit 2-tail = 2.05 | ||||
| Ranked | 230.43 | 119.41 | P 2-tail = .27 df = 8 t Stat = 1.17 t Crit 2-tail = 2.30 | P 2-tail = .76 df = 17 t Stat = −0.31 t Crit 2-tail = 2.11 | |||
| Matched | 224.75 | 178.79 | P 2-tail = .17 df = 9 t Stat = 1.48 t Crit 2-tail = 2.26 | ||||
| VC | |||||||
| Overall | 225.82 | 248.24 | |||||
| Not ranked | 218.14 | 227.48 | P 2-tail = .13 df = 12 t Stat= 1.609 t Crit 2-tail = 2.18 | ||||
| Ranked | 229.18 | 233.50 | P 2-tail = .41 df = 8 t Stat = −0.88 t Crit 2-tail = 2.30 | ||||
| Matched | 234.25 | 74.92 | |||||
| Step 2 Scores | |||||||
| IP | |||||||
| Overall | 239.21 | 114.21 | P 2-tail = .46 df= 30 t Stat = 0.74 t Crit 2-tail = 2.04 | ||||
| Not ranked | 237.33 | 144.53 | P 2-tail = .43 df = 26 t Stat = −0.80 t Crit 2-tail = 2.06 | ||||
| Ranked | 239.57 | 108.84 | P 2-tail = .64 df = 10 t Stat = −0.48 t Crit 2-tail = 2.23 | P 2-tail = .82 df = 19 t Stat = −0.23 t Crit 2-tail = 2.09 | |||
| Matched | 240.75 | 40.50 | P 2-tail = .44 df = 8 t Stat = −0.81 t Crit 2-tail = 2.31 | ||||
| VC | |||||||
| Overall | 237.35 | 123.60 | |||||
| Not ranked | 233.86 | 123.14 | P 2-tail = .34 df = 12 t Stat = −0.99 t Crit 2-tail = 2.18 | ||||
| Ranked | 238.88 | 123.85 | P 2-tail = .82 df = 11 t Stat = 0.23 t Crit 2-tail = 2.20 | ||||
| Matched | 238.00 | 26.00 | |||||
Abbreviations: t Crit, t critical value; IP, in-person; Stat, t statistic; USMLE, US Medical Licensing Examination; VC, video conference.
Table 2b.
Logistic Regression Analysis for choosing IP or VC interviews, Ranking or Matching by USMLE Step Scores
| Predictors | Coefficient Estimate | Standard Error | z Score | P Value | 95% CI | ||
|---|---|---|---|---|---|---|---|
| Lower | Upper | ||||||
| Step 1 Scores | Choosing IP or VC Interviews | −0.0377 | 0.0218 | −1.732 | >.05 | −0.0808 | 0.00542 |
| Rankingab | −0.0126 | 0.0218 | −0.579 | >.05 | −0.0571 | 0.0288 | |
| Matchingb | −0.0288 | 0.0284 | −1.013 | >.05 | −0.0838 | 0.0292 | |
| Step 2 Scores | Choosing IP or VC Interviews | 0.00645 | 0.0254 | 0.253 | >.05 | −0.0438 | 0.0568 |
| Rankinga | 0.0347 | 0.0241 | 1.436 | >.05 | −0.0119 | 0.0834 | |
| Matching | 0.0268 | 0.0336 | 0.796 | >.05 | −0.0393 | 0.0939 | |
Abbreviations: CI, confidence interval; IP, in-person; USMLE, US Medical Licensing Examination; VC, video conference.
Logistic regression for ranking: Step 1: β = −0.0126, P > .05, CI [−0.0571,0.0288]; Step 2: β = 0.0347, P > .05, CI [−0.0119,0.0834].
Logistic regression for matching: Step1: β = −0.0288, P > .05, CI [−0.0838,0.0292]; Step 2: β = 0.0268, P > .05, CI [−0.0393,0.0939].
Interview Format Preference, Rank, and Match Outcomes, According to Applicant Medical School
The distribution of applicants who preferred to conduct interviews by IP and VC demonstrated significant differences in the type of medical school from which they would graduate (χ2 (df = 2, n = 159) = 22.97, P < .001). Post-hoc analysis using Bonferroni correction indicated that a significantly higher proportion of international medical graduate candidates chose to interview by VC compared to US Doctor of Medicine (χ2 (df = 1, n = 117) = 12.80, P < .05) or US Doctor of Osteopathy candidates (χ2 (df = 1, n = 90) = 15.28, P < .05). Simultaneously, there was no difference in the proportion of US Doctor of Medicine and US Doctor of Osteopathy candidates who chose to participate in IP or VC interviews. However, regardless of medical school type, candidates’ interview format preference did not impact whether they would rank (χ2 (df = 2, n = 125) = 0.68, P > .05) or match (χ2 (df = 2, n = 12) = 0.12, P > .05) at our program (Table 3).
Table 3.
Rank and Match Outcomes to Our Program, According to Whether the Applicant Would Be a Graduate of an Osteopathic, Allopathic, or International Medical School
| Interview Format | |||
|---|---|---|---|
| In Person | Virtual | ||
| Percent of Ranked Applicants by School Background | χ2 = 0.68; P > .05 | ||
| US-DO | 87.80 | 0 | |
| IMG | 74.20 | 76.50 | |
| US-MD | 80.90 | 66.70 | |
| Percent of Matched Applicants by School Background | χ2 = 0.12; P > .05 | ||
| US-DO | 2.8 | 0 | |
| IMG | 17.4 | 30.8 | |
| US-MD | 5.9 | 0 | |
Abbreviations: IMG, international medical graduate; LCME, Liaison Committee on Medical Education; US-DO, US Doctor of Osteopathy; US-MD, US Doctor of Medicine.
Time and Financial Costs
Personnel costs amounted to $311 per applicant in the IP group and $294 for the VC group. Meals for IP group interviews amounted to $120 per applicant, making the overall per-applicant costs $431 for IP and $294 for VC interviewees. Thus, IP interviews cost $137 more per applicant. The Cisco Webex VC platform cost was not included because it was a negligible amount provided by the institution. Time spent per IP or VC candidate (total hours divided by the total number of applicants in each group) was also higher for the IP group (4.0 hours for IP, 3.0 hours for VC; Table 4).
Table 4.
Time and Monetary Costs per Candidate for In-person and Virtual Interviews a
| In Person (n = 135), 12 Interview Days | Virtual (n = 24), 6 Interview Days | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| N | h/d | Total Hours | Total Cost | Cost/Applicant | N | h/d | Total Hours | Total Cost | Cost/Applicant | |
| Personnel (per-hour wage) | ||||||||||
| Faculty ($171) | 4 | 4 | 192 | $32,832 | $243 | 2 | 3 | 36 | $6,156 | $257 |
| Residents/Interview ($29) | 2 | 6 | 144 | $4,176 | $31 | 1 | 3 | 18 | $522 | $22 |
| Residents/Dinner ($29)b | 4 | 2 | 96 | $2,784 | $21 | 0 | 0 | 0 | 0 | 0 |
| Coordinators ($20) | 2 | 4.5 | 108 | $2,160 | $16 | 1 | 3 | 18 | $360 | $15 |
| Meals | ||||||||||
| Sunday dinner | $9,356 | $69 | $0 | $0 | ||||||
| Monday lunch | $6,949 | $51 | $0 | $0 | ||||||
| Total costs | $58,257 | $431 | $7,038 | $294 | ||||||
In-person applicants interviewed with 2 faculty and 1 resident individually, while 2 faculty and 1 resident, as a group, interviewed each virtual applicant.
For the 12 in-person interviews, 4 residents went to dinner with applicants the night before. Four faculty, 2 residents, and 2 coordinators were involved on the day of the interview. For the 6 virtual interviews, 2 faculty, 1 resident, and 1 coordinator were involved on the interview day.
Traveling Distance
In determining whether distance affected IP or VC interview choice, we found that most (75%) of those choosing IP interviews were located within 1500 miles of our institution. Of those, 66% were located within 500 miles of us. On the other hand, 63% of those choosing VC interviews were located more than 1500 miles from us. The mean distance was 1241 miles (range, 0–7900 miles) for the IP group and 3329 miles (range, 100–7900 miles) for the VC group. No data were removed from the data set.
Interviewer and Matched Applicant Satisfaction Survey
Of 30 interviewers, 21 (70%) responded to a survey about the interview process. The responses were evenly split among those who participated in IP interviews only (7), VC interviews only (6), or both (8). Among respondents, 57% preferred the IP method, and 62% said that it was very important or extremely important to meet applicants in person. Fifteen (71%) were satisfied or very satisfied with the VC interview process, and 17 (81%), with the IP interview process. While 43% felt the interview format did not play a role in the match outcome, 71% said that having an IP interview would impact their personal connection with the applicant. A majority (81%) said that both IP and virtual options should be offered to candidates.
Of our 12 postgraduate year one residents, 11 responded to an internal survey of their IP or VC interview preference. Nearly all (91%) reported a preference for an IP interview, and most said it was extremely important or very important to visit a residency program’s location (64%) and to meet the residents and faculty in person (82%). About a third stated that finances played a role in their choice of interview type. When it came to VC interviews, only 27% stated that they were somewhat concerned or very concerned about technical difficulties. For IP interviews, none of the respondents reported being concerned about the time spent away from school. The 4 residents who opted for a VC interview were either somewhat satisfied or very satisfied with the virtual format. Most believed that the type of interview could affect personal connection as well as match results. Nevertheless, nearly all said that programs should offer both IP and VC interview options to future applicants.
Discussion
In our single-center study comparing rank and match outcomes of IP and VC anesthesiology residency interviews, there was no statistically significant difference between the 2 interview formats for being ranked (81% of IP, 71% of VC) or matched (6% of IP, 17% of VC), and USMLE Step 1 and Step 2 scores and type of medical school did not affect the likelihood of being ranked or matched. Similarly, Vadi et al also reported that virtual interviews do not appear to negatively impact an applicant’s overall perception or actual chance to match to an anesthesiology residency program compared to IP interviews.3 Pasadhika et al6 also reported that the percentage of face-to-face versus VC interview applicants who were ranked in the top 25 in an ophthalmology residency were not statistically significant.6 Program directors at American medical residency programs have rated Step 1 and 2 scores, clerkship grades, class ranking, and failed examinations as important factors when assessing the potential for academic success among applicants. Kremer et al reported that Step 1 scores significantly predicted interview offers for most specialties, except for dermatology.7 De Oliveira et al reported that the USMLE Step 2 score was associated with a successful admission to an anesthesiology residency.8 In our study, we demonstrated that Step 1 and Step 2 scores, regarded as 2 of the strongest metrics in determining residency matching outcomes, did not influence applicants’ choice of interview format and whether applicants were ranked or matched with our program (Table 2a). We were able to holistically review each application and make decisions about ranking order regardless of interview format and examination scores for both interview groups. It has been suggested that residency application reviews should be comprehensively conducted while placing less emphasis on examination scores, and our study supports the notion that a review of residency applications is possible without undue focus on Step scores.
The interview process can be burdensome for candidates in the face of travel and hotel costs and time lost from educational pursuits and clinical responsibilities. From the residency program’s perspective, there are opportunity costs related to time spent away from clinical duties, research, or other administrative tasks.2,9,10 VC platforms can improve cost efficiencies, potentially alleviating the financial burden on applicants and training programs.11–13 Pourmand et al noted that web-based interviews saved applicants about $566 and decreased direct salary costs for programs by $482 compared to IP interviews. Moreover, purchasing and installing the technology necessary for VC interviews amounted to only $130, saving the program $586 for each applicant who opted for a VC interview.14 Edje et al found that the direct and indirect savings for their program amounted to $5864, with a time savings of 7 interview days.2 Further, web-based interviews lead to increased clinical and educational productivity among residency programs.14 These figures could be higher for applicants applying to certain specialties such as plastic surgery.12 Another possible financial benefit for training centers is that VC interviews can be held during evening or weekend hours to avoid the potential loss of clinical revenue, with the trade-off of possible loss of nonclinical time for the individual faculty members and residents.6
Overall, these findings support VC as a cost-effective alternative to IP interviews (Table 4). Any savings gained by incorporating the virtual format could be invested in developing a series of more polished, professionally produced virtual tours. In addition, the use of virtual reality technology could leave a lasting impression of the program in the minds of applicants.15
We demonstrated in this single-institution study that those who interviewed by VC were not disadvantaged by the interview format in terms of our impression of the candidates or their image of our program, a notion mirrored by Vadi et al.3 Two previous studies found the standardized VC interview score format to yield total scores that are reliable and comparable across applicants. It also provided information beyond what is currently available from academic metrics to assess professionalism as well as interpersonal and communication skills.16,17
The dinner, campus tour, and lunch are opportunities for informal interactions between the candidate and our residents and faculty during the IP interview process, yielding valuable information about applicants’ interpersonal and communication skills and determining whether there is a good fit between the applicant and the program. Fit is an essential factor in the residency selection process, according to program directors and applicants alike.1,18,19 For the VC candidates, we asked the interviewers to focus on interview questions selected (see Online Supplemental Material) to evaluate fit for the program, in the absence of informal interactions that would otherwise facilitate this interpersonal analysis.
Applicants who choose web-based interviews may cite scheduling, travel, or financial considerations as the reason.3 In our study, a higher proportion of international medical graduate candidates conducted interviews by VC, perhaps to limit the cost and time associated with extensive travel. Those choosing virtual interviews tended to live farther away from us compared to those choosing IP interviews. Visa considerations could also make the virtual option attractive to international medical graduate candidates as it may be difficult to obtain a temporary visa to travel to the United States for interviews. Our study results suggest that applicants can take advantage of the cost and time benefits associated with interviews conducted virtually without having to weigh them against the perceived disadvantages to rank or match outcomes. Potential disadvantages to foregoing the IP interview experience have been reported. A meta-analysis suggested that lower interviewer ratings and applicant reactions could occur in technology-assisted interviews.20 Other concerns include technical issues, being less effective in presenting themselves, and the inability to evaluate the hospital and city in which the residency program is located.4,9,14,20 Nearly all of our postgraduate year one residents who responded to a confidential survey reported a preference for an IP interview, visiting a residency program, and meeting the residents and faculty. Residency programs and their governing bodies are working to address these critical concerns through social media outreach to promote the unique qualities of residents and staff, department and hospital accomplishments, and city attractions.21 Another approach, including a virtual tour as part of the interview process to walk prospective applicants through their facilities, was reported to be as good as or better than IP tours.15 Alternatively, VC interviews could be used to supplement IP interviews.13,17 Our study attempted to model the IP experience by offering a virtual meet-and-greet with residents and providing links to videos highlights various aspects of our program, institution, and city.
While applicants’ satisfaction was not addressed, we sought to gain insight into our matched applicants’ and interviewers’ views about their interview experiences at our institution and elsewhere. Our findings were consistent with what has been reported by others. For both resident and interviewer groups surveyed, the consensus was that IP interviews are preferred for the opportunity to interact with the faculty, residents, and candidates.5 Overall, both groups believed that programs should offer both IP and VC interviews.5,22 Residents surveyed also indicated that a visit to the area and institution had an impact on their rank order. A majority of the residents reported that personal costs influenced their choice of interview type, confirming that among interview-related stressors, the financial impact is an important one.
This study highlights the effectiveness and utility of virtual interviews even before the pandemic necessitated them. Interactive VC interviews, allowing candidates and interviewers to see one another in real-time, tend to be more time-efficient and cost-efficient and closely replicate the traditional IP interview. Drawbacks include not being able to see the campus on the day of the interview, reduced interactions with residents and faculty, and more difficulty gaining detailed knowledge about the city, program, and institution.23
Our experience with virtual interviews was positive, confirming the findings of other studies. If conducted properly, training programs will match with the same desirable applicants they are likely to attract with IP interviews. Most specialty and student organizations and the Coalition of Physician Accountability Work Group urged programs to commit to online interviews and virtual visits for all applicants, including local students, rather than IP interviews for the 2020–2021 interview cycle.
In addition to cost savings, virtual interviews offer residency programs an opportunity to increase diversity of geographic and cultural representation. By reducing the cost and travel burdens for applicants, virtual interviews make it possible for programs to interview and potentially accept applicants from more regions of the United States and other countries. This could broaden the cultural diversity in our programs, which would in turn benefit our culturally diverse patients. The medical education community should take specific steps to create a robust digital environment and tools to yield the best experiences for programs and applicants. Residency programs can ensure that candidates and programs will be well-positioned to make essential ranking decisions in future residency match cycles through enhanced website content, creative media solutions, and implementation of best practices for interviewing.4
Limitations of the study include its single-center nonrandomized design and relatively small sample size. While randomization may limit bias and balance confounding factors, it may not be appropriate in residency interviews as candidates assigned to virtual interviews may be concerned that they may be disadvantaged in the high-stakes residency match. The interview experiences should have been standardized as much as possible to improve interview equity. The 2 interview experiences were different, above and beyond the IP vs virtual nature of the interview. The IP interview was the standard with synchronous interactions with applicants. The VC interactions were devised to be as similar to IP as possible to improve interview equity; however, only the virtual interview component was synchronous. All other components were asynchronous. In addition, VC candidates interviewed with 2 faculty and 1 resident as a group rather than individually. Time spent per candidate was also higher for the IP group because of the asymmetric study design and not a natural result of IP vs VC. Thus, caution should be used in interpreting the results.
In addition, while we chose to use the state or country of the applicant’s medical school as the starting point for travel to our institution, an applicant may not actually have been at that location during the interview season. For example, an applicant may have been living with a relative in California during the interview season despite being a senior medical student at an international school. Another limitation is a lack of formal feedback from applicants. For this reason, we instead surveyed our interns and interviewers. Nearly all said that both IP and virtual options should be offered to candidates, and about half felt the interview format did not play a role in matching residents to our program.
Our study results indicate that residency candidates who interviewed by VC were not disadvantaged compared to those who were interviewed by IP. The results, taken together with the time and financial cost savings, strongly suggest that virtual interviews should be an option for the interview season from this point forward, even after pandemic-related travel restrictions have been lifted.
Acknowledgments
The authors thank Carol Dickerson, research manager, for helping with the cost analysis and editing of the manuscript.
Online Supplemental Material
Interview Questions for the 2019–2020 Residency Interview Season
Questions Assessing Fit for the Program
What makes this program appealing to you?
What do you hope to gain from our residency program?
Why should we want you to come to our program?
What will you bring to our program?
We have many suitable applicants. Why should we choose you?
What do you feel you could add to our program?
General Questions
Who is your role model? Why?
Tell me something about you that is not on your CV.
Why are you so sure anesthesiology is right for you?
What is one characteristic you would change about yourself?
What is the last non-academic book you read?
Describe the most challenging decision you have ever had to make. How did you go about it?
Describe the worst or most disappointing clinical experience you’ve had so far.
What will you do if you don’t match in anesthesiology?
What do you do to cope with stress?
Footnotes
Funding: None
Conflicts of interest: None
References
- 1.Association of American Medical Colleges Best practices for conducting residency program interviews; 2016. https://www.aamc.org/media/44746/download Accessed July 19, 2021.
- 2.Edje L, Miller C, Kiefer J, Oram D. Using Skype as an alternative for residency selection interviews. J Grad Med Educ. 2013;5(3):503–5. doi: 10.4300/JGME-D-12-00152.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Vadi MG, Malkin MR, Lenart J, et al. Comparison of web-based and face-to-face interviews for application to an anesthesiology training program: a pilot study. Int J Med Educ. 2016;7:102–8. doi: 10.5116/ijme.56e5.491a. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Joshi A, Bloom DA, Spencer A, Gaetke-Udager K, Cohan RH. Video interviewing: a review and recommendations for implementation in the era of COVID-19 and beyond. Acad Radiol. 2020;27(9):1316–22. doi: 10.1016/j.acra.2020.05.020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Seifi A, Mirahmadizadeh A, Eslami V. Perception of medical students and residents about virtual interviews for residency applications in the United States. PLoS One. 2020;15(8):e0238239. doi: 10.1371/journal.pone.0238239. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Pasadhika S, Altenbernd T, Ober RR, Harvey EM, Miller JM. Residency interview video conferencing. Ophthalmology. 2012;119(2):426–426.e5. doi: 10.1016/j.ophtha.2011.09.032. [DOI] [PubMed] [Google Scholar]
- 7.Kremer TR, Kremer MJ, Kremer KP, Mihalic A. Predictors of getting a residency interview: Differences by medical specialty. Med Educ. 2021;5(2):198–212. doi: 10.1111/medu.14303. [DOI] [PubMed] [Google Scholar]
- 8.de Oliveira GS, Akikwala T, Kendall MC, et al. Factors affecting admission to anesthesiology residency in the United States: choosing the future of our specialty. Anesthesiology. 2012;117(2):243–51. doi: 10.1097/ALN.0b013e31825fb04b. [DOI] [PubMed] [Google Scholar]
- 9.Shah SK, Arora S, Skipper B, et al. Randomized evaluation of a web based interview process for urology resident selection. J Urol. 2012;187(4):1380–4. doi: 10.1016/j.juro.2011.11.108. [DOI] [PubMed] [Google Scholar]
- 10.Nilsen K, Callaway P, Phillips JP, Walling A. How much do family medicine residency programs spend on resident recruitment? A CERA study. Fam Med. 2019;51(5):405–12. doi: 10.22454/FamMed.2019.663971. [DOI] [PubMed] [Google Scholar]
- 11.Chandler NM, Litz CN, Chang HL, Danielson PD. Efficacy of videoconference interviews in the pediatric surgery match. J Surg Educ. 2019;76(2):420–6. doi: 10.1016/j.jsurg.2018.08.010. [DOI] [PubMed] [Google Scholar]
- 12.Melendez MM, Dobryansky M, Alizadeh K. Live online video interviews dramatically improve the plastic surgery residency application process. Plast Reconstr Surg. 2012;130(1):240e–1e. doi: 10.1097/PRS.0b013e3182550411. [DOI] [PubMed] [Google Scholar]
- 13.Breitkopf D, Hopkins MR, Camp CL, et al. One way video interviewing as a method to augment the residency application [5C] Obstet Gynecol. 2018;131:31s. [Google Scholar]
- 14.Pourmand A, Lee H, Fair M, Maloney K, Caggiula A. Feasibility and usability of tele-interview for medical residency interview. West J Emerg Med. 2018;19(1):80–6. doi: 10.5811/westjem.2017.11.35167. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Zertuche JP, Connors J, Scheinman A, Kothari N, Wong K. Using virtual reality as a replacement for hospital tours during residency interviews. Med Educ Online. 2020;25(1):1777066. doi: 10.1080/10872981.2020.1777066. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Bird SB, Hern HG, Blomkalns A, et al. Innovation in residency selection: the AAMC standardized video interview. Acad Med. 2019;94(10):1489–97. doi: 10.1097/ACM.0000000000002705. [DOI] [PubMed] [Google Scholar]
- 17.Gallahue FE, Hiller KM, Bird SB, et al. The AAMC standardized video interview: reactions and use by residency programs during the 2018 application cycle. Acad Med. 2019;94(10):1506–12. doi: 10.1097/ACM.0000000000002714. [DOI] [PubMed] [Google Scholar]
- 18.Blindspot: hidden biases of good people. Choice Rev Online. 2014;51(10):51–5867. [Google Scholar]
- 19.Fuchs JW, Youmans QR. Mitigating bias in the era of virtual residency and fellowship interviews. J Grad Med Educ. 2020;12(6):674–7. doi: 10.4300/JGME-D-20-00443.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Blacksmith N, Willford JC, Behrend TS. Technology in the employment interview: a meta-analysis and future research agenda. Personnel Assess Decis. 2016;2(1):12–20. [Google Scholar]
- 21.Nguyen JK, Shah N, Heitkamp DE, Gupta Y. COVID-19 and the radiology match: a residency program’s survival guide to the virtual interview season. Acad Radiol. 2020;27(9):1294–7. doi: 10.1016/j.acra.2020.06.023. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Davis MG, Haas MRC, Gottlieb M, et al. Zooming in versus flying out: virtual residency interviews in the era of COVID-19. AEM Educ Train. 2020;4(4):443–6. doi: 10.1002/aet2.10486. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Healy WL, Bedair H. Videoconference interviews for an adult reconstruction fellowship: lessons earned. J Bone Joint Surg Am. 2017;99(21):e114. doi: 10.2106/JBJS.17.00322. [DOI] [PubMed] [Google Scholar]
- 24.Green M, Jones P, Thomas JX., Jr. PhD selection criteria for residency: results of a national program directors survey. Acad Med. 2009;84(3):362–7. doi: 10.1097/ACM.0b013e3181970c6b. [DOI] [PubMed] [Google Scholar]
- 25.Chou DW, Pletcher SD, Bruss D, et al. Otolaryngology residency interviews in a socially distanced world: strategies to recruit and assess applicants. Otolaryngol Head Neck Surg. 2021;164(5):903–8. doi: 10.1177/0194599820957961. [DOI] [PubMed] [Google Scholar]