Virtual Interviews During COVID-19: Perspectives of Female Pelvic Medicine and Reconstructive Surgery Program Directors

Supplemental digital content is available in the text.

Abstract

Objective

The aim of the study was to evaluate female pelvic medicine and reconstructive surgery (FPMRS) fellowship program directors’ opinions regarding the effectiveness of virtual interviews for selecting fellows and their future interview mode preferences.

Methods

This was a cross-sectional online survey of all FPMRS program directors in the United States conducted from April 29, 2020, to May 30, 2020. At the time of this study, there were 73 program directors and 69 obstetrics and gynecology and urology–accredited FPMRS programs nationwide. The primary outcome was to subjectively assess the effectiveness of virtual interviews as compared with in-person interviews for evaluating applicants.

Results

Fifty seven (82.6%) of the program directors completed the survey. A total of 80.7% (46/57) of the respondents had participated in interviews for the active match cycle. Of the programs that participated in the interview process, almost all conducted interviews using virtual platforms (97.8%, 45/46). Program directors who conducted interviews virtually found them effective in evaluating applicants (88.9%, 40/45) and were satisfied with the virtual interview process (86.7%, 39/45). A total of 31.1% of respondents (14/45) preferred a virtual platform to an in-person setting for future interviews, and 60% (27/45) reported that they will likely perform future interviews virtually.

Conclusions

Although the pandemic resulted in a sudden reformatting of FPMRS fellowship interviews, most program directors nationally were satisfied with the process and found virtual interviews effective for assessing applicants. More than 50% of FPMRS program directors are likely to consider the virtual format for future interviews.

During the spring of 2020 as the coronavirus disease 2019 (COVID-19) pandemic spread in the United States, female pelvic medicine and reconstructive surgery (FPMRS) was one of the few medical training programs scheduled to conduct interviews because of an early match process. As travel and social distancing restrictions imposed by the COVID-19 pandemic made face-to-face interviews less feasible, the FPMRS program director community recognized the need to quickly adapt. Most programs registered for the FPMRS 2021 match cycle transitioned from in-person to virtually conducted interviews. This transition was reinforced by the May 2020 statement from the Association of American Medical Colleges “strongly encouraging” interviews for medical school, residency, and faculty, to be conducted using virtual platforms.¹

Interviews have traditionally been a way for applicants and programs to assess each other. Applicants can showcase their personality and learn about the relationship between the faculty and trainees to assess whether the environment is a good fit for them. Programs highlight their unique attributes and facilities. Although the current literature shows mixed results with regard to the ability of interviews to predict trainees’ attrition, problems with professionalism, or clinical performance, this tradition is an essential part of the application process.²

The shift in the interview format and process, caused by COVID-19, raises several questions with regard to the suitability of the virtual format for programs to assess individual applicants and make rank list decisions. The current literature assessing and comparing virtual with in-person interviews has consisted of small groups and single-institution evaluations of the process.^3–9 In addition, these studies were all conducted in nonpandemic times. With the current Association of American Medical Colleges recommendation and the persistence of the COVID-19 crisis, the experience of the FPMRS subspecialty may be useful to other fields as they consider changing the format of their interviews. The primary objective of this study was to evaluate the FPMRS program directors’ perceptions on the subjective effectiveness of virtual interviews compared with in-person interviews for the assessment and evaluation of fellowship candidates. The secondary objectives were to assess the satisfaction of program directors with virtual interviews for evaluating applicants, their preference of in-person or virtual format for future interviews, as well as their likelihood of using virtual interviews in future match cycles.

MATERIALS AND METHODS

Exempt status was obtained from the institutional review board. A novel, voluntary, and anonymous survey questionnaire (see supplemental material, http://links.lww.com/FPMRS/A189) was distributed to program directors for all Accreditation Council for Graduate Medical Education–accredited FPMRS fellowships in the United States 5 times from April 29, 2020, to May 30, 2020. Study data were collected using the Mount Sinai Hospital–secure Research Electronic Data Capture (REDCap) tool.¹⁰ REDCap is a secure, web-based application designed to support data capture for research studies, providing the following: (1) an intuitive interface for validated data entry, (2) audit trails for tracking data manipulation and export procedures, (3) automated export procedures for seamless data downloads to common statistical packages, and (4) procedures for importing data from external sources.

There were a total of 69 FPMRS fellowship programs with 73 program directors (3 programs had 2 individuals serving as program directors) at the time of our survey. The list of gynecology-based FPMRS programs was obtained from the American Urogynecology Society website, and the list of urology-based FPMRS programs from the Society of Urodynamics and Female Urology website. This list was cross-referenced with the available list of programs on the Accreditation Council for Graduate Medical Education website. The survey was distributed via e-mail to 72 program directors (68 programs), after excluding the Mount Sinai FPMRS fellowship. Program directors were asked to complete the survey after completing at least one of their fellowship interviews for the 2021 application year (Fig. 1). Sixty-nine program directors remained eligible to participate in the survey after 3 program directors were excluded: 2 program directors contacted the study team to inform them that they would not participate in the study because their interview days were after study conclusion, and 1 program director informed the study team via e-mail that their program had not participated in this interview season (Fig. 1).

FIGURE 1.

Flow chart. PDs, program directors.

A 17-item questionnaire that followed the CHERRIES (Checklist for Reporting Results of Internet E-Surveys) guidelines was created by the Mount Sinai Hospital FPMRS Division.¹¹ The survey assessed program directors’ perspectives on use of virtual interviews for evaluating applicants. The survey questions queried program directors about participation of their FPMRS fellowship program in the 2020 interview season; those who had participated in interviews further identified the format of their interviews as virtual or not. Responses were evaluated using a 5-point Likert scale by asking about effectiveness (“very ineffective, ineffective, unsure, effective, very effective”) and satisfaction (“very dissatisfied, dissatisfied, unsure, satisfied, very satisfied”) with use of virtual interviews to assess applicants as compared with in-person interviews. The programs’ degree of workflow disruption as a result of the COVID-19 pandemic was evaluated by asking about the degree of interruption to patient care: all operating room cases except emergency cases were canceled (significant), nonurgent cases and office visits were canceled (moderate), normal operating room case flow and some nonurgent office visits were canceled (mild), and normal operating room and office schedule (not at all). Demographic information was also collected, including age, years acting as program director, and location of the program (U.S. census region).

We hypothesized that although transitioning to virtual interviews was sudden because of the pandemic, program directors would find virtual interviews effective to assess and evaluate applicants for choosing their future fellow compared with in-person interviews.

Descriptive statistics were used for analysis of the cohort’s characteristics. Univariate analyses (Fisher exact test for categorical variables, Wilcoxon rank sum test for continuous variables) were performed to evaluate the association between the program directors’ perceptions of the virtual interview process and their stated likeliness to continue to perform virtual interviews in the future using R Version 3.6.3.

RESULTS

A total of 57 surveys were completed yielding an 82.6% response rate (57/69). A total of 29.6% of respondents serve as a program director in an FPMRS fellowship in the Northeast, 24.1% in the Midwest, 31.5% in the South, and 14.8% in the West. This distribution corresponds to the U.S. distribution of FPMRS program directors: 24% Northeast, 26% Midwest, 34% South, and 15% West. Most responders were in the age group 41–50 years (50.9%, 29/57) and have been serving in the role of program director for 5 or more years (52.6%, 30/57; Table 1). Every program director noted that the pandemic either moderately (17.0%, 9/53) or significantly (83.0%, 44/53) disrupted their workflow (Table 1). Of the respondents, 11 had not participated in the current interview cycle and 1 had not used a virtual setting; therefore, these 12 respondents were not included in the final analysis (Fig. 1).

TABLE 1.

Demographics

	Program Directors Participating in Virtual Interviews, n (%)	Program Directors Not Participating in Virtual Interviews, n (%)	Program Directors That Did Not Participate in the Match for 2021 Cycle, n (%)	P
Virtual interview participation	45 (78.9)	1 (1.8)	11 (19.3)	<0.001
Program director age, y		—		0.09
30-40	4 (8.9)	—	0 (0.0)
41-50	26 (57.8)	—	3 (27.3)
51-60	15 (33.3)	1 (100.0)	4 (36.4)
≥61	0 (0.0)		1 (9.1)
Years as program director				1
0-1	2 (4.4)	—	0 (0.0)
1-5	18 (40.0)	—	4 (36.4)
≥5	25 (55.6)	1 (100.0)	4 (36.4)
Program location (U.S. region)				0.36
Northeast	14 (31.1)	—	2 (18.2)
Midwest	9 (20.0)	1 (100.0)	3 (27.3)
South	14 (31.1)	—	3 (27.3)
West	8 (17.8)	—	0 (0.0)
Workflow disruption due to COVID-19
No disruption	—	—	0 (0.0)
Mild disruption	—	—	0 (0.0)
Moderate disruption	7 (15.6)	—	2 (18.2)
Significant disruption	38 (84.4)	1 (100.0)	5 (45.5)
Virtual session format				—
Single applicant per session	4 (8.9)	—	—
Multiple applicant per session	41 (91.1)	—	—
Virtual interview format		—	—
1 applicant per interviewer	26 (57.8)	—	—
Panel of applicants per interviewer	1 (2.2)	—	—
1 applicant per panel of interviewers	13 (28.9)	—	—
Panel of applicants per panel of interviewers	3 (6.7)	—	—
Combination of above formats	5 (11.1)	—	—

One third (15/45) of program directors modified their interview format significantly to adapt to the virtual setting, whereas the other two thirds (30/45) did not. Most programs held 2 interview sessions (53.5%, 23/43), and most respondents (91.1%, 41/45) had interviewed multiple applicants per interview session. Most program directors (57.8%, 26/45) indicated that their interview format was 1 applicant and interviewer at a time, and 28.9% (13/45) had 1 applicant per panel of at least 2 interviewers. One program conducted all of their interviews as panels of applicants per interviewer. A total of 11.1% (5/45) of program directors indicated that their program used a combination of the previously mentioned formats for conducting their virtual interviews (Table 1).

Compared with in-person interviews, 88.9% (40/45) of program directors found the virtual interview format effective for assessing and evaluating applicants (Fig. 2). Most (86.7%, 39/45) were satisfied with the virtual interview process when compared with in-person interviews (Fig. 2). Ninety-five percent (38/40) of program directors who found the virtual interview process effective and 20% (1/5) who found it ineffective were satisfied with the process for evaluating applicants (P = 0.0005). Although there was a high satisfaction rate, only 31.1% (14/45) of program directors preferred virtual interviews to in-person interviews. Although only one third of program directors preferred virtual interviews, 60% noted that they are likely to use virtual interviews in the future, and the remaining 40% (28.9% [13/45] neutral, 11.1% [5/45] unlikely, and 0% [0/45] extremely unlikely) felt that they are unlikely to use a virtual format in the future (Fig. 3). Program directors who found organizing virtual interviews easy were more likely to consider conducting future interviews virtually than those who found it difficult (76.9% [20/26] vs 36.8% [7/19], P = 0.013). The majority of the 33.3% (15/45) of program directors who had significantly modified their interview session format to accommodate the virtual setting were more likely to prefer in-person to virtual interviews (46.7% [7/15] vs 26.7% [4/15], P = 0.57).

FIGURE 2.

A, Percentage of program directors that found virtual interviews effective versus ineffective. B, Percentage of program directors that were satisfied versus dissatisfied with virtual interviews. C, Percentage of program directors that found organizing virtual interviews easy versus difficult.

FIGURE 3.

Program directors' preference and likelihood to conduct future fellowship interviews using virtual platforms.

DISCUSSION

The COVID-19 pandemic coincided with the start of the interview season for FPMRS programs nationally. As a result, most programs conducting interviews transitioned from in-person to a virtual format. In this nationwide survey of FPMRS program directors, the majority found virtual interviews to be effective (88.9%) in evaluating applicants for choosing a future fellow. There was also a high satisfaction rate (86.7%) with the virtual interview process. Although most program directors were satisfied with and found the process effective, only 31.1% indicated that they preferred conducting interviews virtually rather than in-person. Despite most program directors preferring in-person interviews, 60% responded that they would be likely to use virtual interviews in the future.

Before the pandemic, use of web-based and video conferencing programs for interviews in graduate medical education was uncommon. Although a limited number of studies have assessed virtual interviews, most were single-institution studies that focused on the applicant’s perspective on the process. Our finding that FPMRS program directors thought that virtual interviews were effective for evaluating applicants supports prior studies’ findings that rank list order is not affected when comparing virtual interviews with in-person interviews.^12,13 Furthermore, factors that are associated with a successful match in obstetrics and gynecology fellowships, such as multiple research publications and letters of recommendations from leaders in the field, are not influenced by the virtual format.¹⁴

The current data suggest that candidates have found virtual interviews to be acceptable, satisfactory, and, as expected, cost-effective.^6,7,12,15,16 Our data focused on the program directors’ perspective and show that regardless of years in the role of program director, age, and region of the country, most found virtual interviews to be satisfactory, a finding that is supported by previous data that have also found virtual interviews satisfactory by faculty.¹²

Although most program directors were satisfied with a virtual format for fellowship interviews and found it effective, only 31.1% prefer it to in-person interviews and 60% were likely to use virtual interviews in the future. A recent single-institution study also found that virtual interviews conducted during the COVID-19 pandemic were seamless but video interviews were only preferred by 33.3% of faculty participants.¹⁷ For future interview seasons, some participants in our study commented that they could envision virtual interviews being used as a screening tool followed by in-person interviews, as an adjunct to in-person interviews, or as an alternative option for candidates who cannot attend in-person interviews. These comments support findings by other studies that virtual interviews could serve as a preinterview screening process by interviewers and a great introductory tool to the program for candidates.^12,18

According to our data, the opinions of FPMRS program directors were varied about their personal preferences for virtual interviews in the future (they were fairly equally divided among the responses yes, no, and neutral regarding preferring virtual interviews). The lack of overwhelming enthusiasm for virtual interviews may be related to the limited time for programs to transition their interview processes and familiarize themselves with the technology as they were adapting to while dealing with the COVID-19 surge both on a personal and professional level. It seems that program directors acknowledge the utility of the format, likely considering its benefits to applicants and its possible inevitability in the future.

With advances in technology and the ease of virtual interviews confirmed by early match fields like FPMRS, the pros and cons of virtual interviews will need to be sorted out to determine the utility of virtual interviews in the nonpandemic future. A deeper look at the details of this process, such as types of virtual platforms, applicants’ perspectives, cost-analysis for both programs and applicants, rank list order, and match data, will be required to clarify the pros and cons of this process.

Limitations of this study include the use of a nonvalidated questionnaire and the lack of applicant perspective. However, given the study timeline and applicant rank list submission and finalization deadline, it was decided against assessing applicants’ perspectives on the process during the same period as program directors. Response bias, given that 20% of FPMRS program directors did not participate in the study, is a concern; however, the relatively high response rate is reassuring. Eleven programs did not participate in the match. The reasons why were not assessed, and are thus unknown. This may present a selection bias if these programs did not participate because of the virtual format. The data representing a single specialty are also a source of selection bias, and thus, the conclusion may not be applicable to other fields in medicine. Another limitation may be the small number of applicants interviewed for fellowship positions, as compared with residency programs; thus, these data may not be applicable to programs that interview a larger number of applicants in a single interview session.

Strengths of this study include its comprehensiveness as a survey of most program directors from an entire subspecialty, the national distribution, and its high response rate. The short 2-minute format of the survey allowed participation during a busy and uncertain time. In addition, the anonymous format and compliance with the CHERRIES checklist for survey studies made it secure to complete and helped ensure useful results.

The FPMRS program director community was at the forefront of adapting the virtual format for fellowship interviews as the pandemic modified social interactions. Based on our survey results, FPMRS program directors who conducted virtual interviews during the COVID-19 pandemic found the process highly effective and were greatly satisfied with the use of virtual formats for assessing applicants. Many are likely to consider the use of virtual platforms in the future. Further investigation regarding the role of virtual interviews for graduate medical education as a substitute, screening, or in adjunct to in-person interviews is needed.

Appendix A

The survey instrument.