Is the changing landscape of fellowship recruitment during COVID-19 here to stay?

Abstract

Background

The 2020 Pediatric Surgery (PS) fellowship selection process was heavily impacted by the COVID-19 pandemic. A review of lessons learned can help determine best practices for the future. The purpose of the study was to analyze the virtual interview experience and assess opportunities to improve the post-pandemic fellowship recruitment process.

Study Design

Using a 28-question survey of Program Directors (PDs) of PS fellowships as well as a 44-question survey of applicants to PS fellowships in the US and Canada, we gathered information on the recruitment process during the COVID-19 pandemic (2020). Dichotomous, multiple choice and open-ended questions about the changes in process, platforms used, format, comparison to on-site interviews and overall satisfaction were used for objective and subjective feedback.

Results

A 95% participation rate was recorded for the PD survey. 24 out of 55 programs (44%) changed their on-site interviews to virtual format due to the pandemic. Most PDs described their overall impression of virtual interviews as satisfactory (66%, 16/24) and did not have an impact on the applicant's success in the match (35/54; 65%). About 50% of PDs preferred to have on-site interviews with virtual screening in the future. While the participation rate from applicants was much less (26 of 70), responses confirmed our survey results. Majority preferred on-site interviews (17/26), 6 of which preferred virtual screening followed by on-site interviews.

Conclusion

Components of virtual screening and interviews were found to have benefits financially and from both time and stress perspectives, and thus might survive past the pandemic.

Levels of Evidence Level IV

.

1. Introduction

The SARS-CoV-2 (COVID-19) pandemic forced many changes throughout the healthcare system, many of which were felt to be long overdue. Telemedicine and tele-interviews had been previously underutilized in the field of medicine in the US. While many forays into creating this change had been previously attempted, there was no universal acceptance. The public health threat of COVID-19 ultimately forced a move to virtual interactions, and the surgical subspecialty training programs were required to rapidly adapt their recruitment process to these new conditions.

According to the National Resident Matching Program (NRMP), there were a total of 8 active fellowship matches during Spring 2020, six of which were surgical subspecialties. Pediatric Surgery (PS)1 was one of these surgical fellowships whose interview season overlapped with the pandemic, social distancing guidelines, and travel restrictions. Thus, the fellowship interview process was quickly modified to a virtual format. Applicants and Program Directors (PD)2 pivoted to modify schedule and interviewing tools. Platforms such as Zoom (Zoom Video Communications, San Jose, Ca), FaceTime (Apple, Cupertino, CA), WebEx (Cisco WebEx, Milpitas, CA) and Skype (Skype, Palo Alto, CA) became crucial to the interviewing process. On the one hand, applicants had to best portray themselves through new technologies and formats and find a quiet yet professional setup for interviews at home, while program directors and coordinators had to create a virtual experience that simulated an onsite interview giving applicants a realistic feel of the program. Molina et al. recently outlined their experience in the Complex General Surgical Oncology Fellowship at Dana-Farber/Partners describing how they simulated an on-site experience using various tools such as breakout rooms on Zoom [1].

While this switch from on-site to virtual interviews was acutely required during the pandemic, this was not the first time PS fellowships had explored changing their interview process. Prior to COVID-19, the work restrictions on general surgical residents, American Board of Surgery rules about absences from training, and the financial hardships faced by candidates led PS educational leaders to consider changes to the application process to benefit all stakeholders (the candidates, the general surgery programs and the PS fellowship programs). Previously the concept of central or virtual interviews was explored [2]. However, many felt the on-site experience was invaluable for both candidates and programs alike in determining the rank list and best fit for match. To address these concerns, the Applicant Committee was created within the Association of Pediatric Surgery Training Program Directors (APSTPD) to explore different ways to improve the interview process. Recent recommendations from this committee has included limiting the total number of interviews offered thereby decreasing the interview burden for applicants, utilizing a centralized electronic scheduling platform, and centralizing program information into one location on the American Pediatric Surgery Association (APSA) website. In 2018, Johns Hopkins All Children's Hospital in Florida screened candidates via videoconference to determine who to invite for on-site interviews. According to Chandler et al., both faculty and applicants agreed that the video interviews were a reasonable option and allowed them to accurately represent themselves. Despite these positive responses, a single, “high-stakes” on-site interview remained the mainstay for the majority of the PS fellowships in the US and Canada [2].

Following these COVID-19 pandemic interview process changes, the APSTPD wished to survey the applicants and the PDs from the 2020 PS fellowship interview process on the challenges and overall experience of the virtual interview process to not only provide insight into the actual process itself (logistics, advantages, disadvantages), but also determine how we can best utilize this process for the future, particularly given the uncertainty surrounding the pandemic's course and lingering restrictions.

2. Study design

An IRB exemption was obtained from National Institutes of Health (NIH) research compliance office to anonymously survey the PS PDs via the Applicant Committee of the APSTPD as well as the applicants to PS fellowship programs during the 2020 match. Web-based survey comprising 63 questions on the 2020 PS fellowship process was sent to the PDs of the 58 programs in the US and Canada. In this survey, 28 questions were based on the 2020 COVID-19 related PS fellowship recruitment process alone (Appendix 1). There were no financial incentives offered for participation. The survey remained open for a month with two rounds of reminders.

Data were collected through the web-based SurveyMonkey (SurveyMonkey Inc. Palo Alto, Ca; www.surveymonkey.com). Each institutional representative, PD or Associate PD, was instructed to complete the survey with only one survey for each institution. The PD or Associate PD were felt to be representative of the faculty at their institution. Applicants were also instructed to complete only one survey. Each survey respondent remained anonymous (however, one study administrator was not blinded to confirm no duplicate program responses were recorded). Objective questions on the virtual interview process were asked in dichotomous (Yes/No) fashion. Questions on the format of the on-site versus virtual interview set up were multiple choice questions. Most questions were followed by an open comment section in order to obtain more granular details and descriptions to the specific questions. Questions about the virtual interview experience and satisfaction were asked in multiple choice format. Both PDs and applicants were asked to describe the benefits of the virtual interview season and its effects on the ranking and match process. A poll was conducted at the end of the survey regarding how they would incorporate virtual interviews in the future.

The data were analyzed for means and ranges to the dichotomous nature of the questions using the SAS software 9.4. Categorical responses were tabulated, and percentages for the PD data were calculated using either 55 or 24 as the denominator. Total of 55 programs responded. Sample size of 24 was used for the questions related to virtual interviews (questions 26–53) as 24 programs utilized the virtual format in the 2020 fellowship match. For each proportion, 95% confidence limits were calculated using the Clopper-Pearson (exact) method. For continuous question, non-parametric summary statistics were calculated (quartiles, and 10th/90th percentiles). Statistical analyzes were not performed for the applicant data since the response rate was so poor. Rather, the results are presented more holistically.

3. Results

Fifty-five PDs of the 58 programs completed the survey for a 95% participation rate. Twenty-four of the 55 programs reported a need to change their interview format to virtual in the midst of the COVID-19 pandemic. Of these 24 programs, most (20/24; 83%) performed only one day of virtual interviews (1–10 range) as most interviews were underway prior to the onset of the pandemic. A median of 10 applicants were interviewed virtually per program (Lower quartile= 8, upper quartile= 11.5).

The format of the virtual interviews was one-on-one for 79.2% programs (19/24; 95% CI = 57.2, 92.9) followed by one applicant to 2–3 faculty for 38% programs (9/24; 95% CI = 18.8, 59.4). Similar to on-site interviews, virtual interview format was mostly free form conversations for 88% programs (21/24; 95% CI = 67.6, 97.3) and each program used the same supplemental tools such as the APSA website, ERAS, Thalamus, link to program, etc. to provide program information to the applicants. However, a few programs (5/24, 20.8%) did provide a video tour and electronic packet with program information. Zoom was the preferred platform for 18/24 programs (75%, 95% CI = 53.3, 90.2) (Fig. 1 ). The length of the interview day was an average of 3–6 h which was the same as on-site interviews for 15/24 programs. During the virtual interview, applicants interacted with current fellows in separate one-on-one sessions for 75% programs (18/24). When asked about the applicant interaction with the advanced practice providers, a majority of programs (20/24) chose none for the answer (83.3%, 95% Ci = 62.6, 95.3).

Fig. 1.

Different platforms and their usage in pediatric surgery programs during COVID-19

based on the 24 PD responses, zoom was the most preferred platform (18/24 or 75.0%, 95% CI 53.3%, 90.2%).

A majority of PDs (93%) felt that the lack of a recruitment event the night before the interview day affected the applicant's experience. Despite these challenging circumstances, nearly 70% PDs were satisfied with the overall experience of using the virtual format (16/24; 66%, Fig. 2 ). The PDs surveyed noted that the virtual format did not change their perceptions of the applicants, and, thus, did not interfere with the applicant's position on the rank list (24/28; 86%) and overall success in the match (35/54; 65%). In open-ended responses regarding the benefits of virtual interviews, the PDs noted the advantage of time and cost savings for the applicants. Some of the limitations noted most frequently were “Did not get to know them as well” and “Did not get to see them interact with others.” The virtual format limited the ability to assess “chemistry,” “personal rapport,” “demeanor,” “personality,” “feel,” and “body language.” These comments all pointed to the intangible connection that forms in-person and is difficult to replicate in virtual interactions. In the future, 48% PDs preferred a hybrid approach of on-site interview with virtual screening interviews (Fig. 3 ). In the open responses, most programs agreed that virtual interviews could provide a screening tool and further narrow the number of on-site interview applicants thus refining the interview process for both the programs and the applicants.

Fig. 2.

Overall impression of virtual interview format

Nearly half PDs (11/24 or 45.8%; 95% CI: 25.5%, 67.2%) were satisfied with the virtual interview format. Two-thirds (11+5 = 16 or 66.7%) were satisfied or very satisfied. Only 16.7% (4/24) were dissatisfied or very dissatisfied.

Fig. 3.

PD Poll on future use of virtual interview format

Most respondents chose ‘On-site w/ virtual’ (28/55 or 50.9%; 95% CI: 37.1%, 64.7%) with ‘On-site inter.’ as a close second (22/55 or 40.0%; 95% CI: 27.0%, 54.1%). Only 3.64% (2/55; 95% CI: 0.44%, 12.5%) chose ‘Virtual’ alone.

Twenty-six of 70 applicants in the 2020 PS match completed the survey for a 37% participation rate. Given this low response rate, statistics were not performed on this data. However, the applicant class of 2020 provided additional data from their own internal survey which was consistent with our data. Thus, overall impressions and trends will be reported here. Applicants continue to apply to a large number of PS fellowship programs and schedule many fellowship interviews in order to optimize or maximize their chances of matching in this competitive fellowship. This causes significant personal financial hardship as well as stress upon their co-residents, programs, and families. For those who participated in virtual interviews, applicants reported decreased interaction with current pediatric surgery fellows as well as a decreased ability to get a good “feel” of the program, obtain the same information about the programs, and determine if the program was a “right fit”. The Zoom platform was also most preferred (16 of the 26). Some benefits cited for on-site interviews were the ability to fully immerse oneself at the institution without other clinical duties; the opportunity to observe interactions between faculty, staff, and the fellows; the chance to ask questions of the fellows and staff; the benefit of attending a social event the night before thus providing another chance to observe interactions and understand the culture of the program; the ease of in-person communication for people with English as a second language; and the avoidance of having to coordinate a quiet place to conduct a virtual interview. Benefits cited for the virtual interviews include less time away from family and clinical duties, lower cost, and opportunity to show some personality if interviewed at home (showing decorations, etc.)

Applicants also noted that due to technical difficulties, their actual interview time was often less compared to on-site interviews. Many also noted that with time, as people became more accustomed to the virtual interview process, the technical barriers were less.

Few thought that virtual interviews could substitute for on-site interviews, and the majority (17 of 26) preferred either on-site interviews (11/17) or virtual screening followed by limited on-site interviews (6/17).

4. Discussion

Virtual interviews have been a long-discussed topic in the field of medicine. Surgical fellowships have an arduous match process that is time consuming, financially straining, and stressful for surgical residents. Due to the COVID-19 pandemic, virtual interviews became a reality overnight for many programs. Our survey captures the objective and subjective feedback on the virtual interview process as compared to the traditional on-site interview process from both the program directors’ and applicants’ perspectives, highlighting some of the benefits and limitations of virtual interviews in surgical fellowships.

One of the foremost advantages of virtual interviews is the budget friendliness of the process. Tseng et al. quoted nearly $6000 in travel costs that surgical fellowship candidates incur via a comprehensive review of literature from various fellowships (plastics, neurosurgery, ophthalmology, orthopedics, otolaryngology, etc.) [3], [4], [5], [6]. This is not a small number given that the residents’ average annual salary is estimated to be a mean of $64,255 for a post-graduate fourth year (PGY-4) [7]. As surgical specialties become more competitive, general surgery residents feel compelled to extend their geographic boundaries and increase the number of attended interviews in order to increase the possibility of successfully matching. Pediatric surgery, in particular, given its competitiveness, often leads applicants to attend as many interviews as possible across the country. This gives rise to more interviews as well as proportional increase in expenses. Previous reports of the number of pediatric surgery interviews attended correlating with a successful match have led some residents to misinterpret this to mean that attending more interviews increases their chances of matching [10]. However, the number of interviews attended is more likely a surrogate for the quality of the candidate. Similarly, time is a valuable resource that is heavily scrutinized in general surgical residency. Most residents do not have the built-in time for numerous interview and travel days for the PGY-4 or PGY-5 residents requiring them to use vacation time to interview. In addition, the ABS has set time-in-training requirements in order to sit for the Board Examination. A candidate who spends more than 30 days per year away from training puts their career and the general surgery program's standing at risk. This causes increased amount of stress on the applicant as well as their programs. Conversion to virtual interviews allows for substantial financial savings as well as decrease in time burden for the recruitment season. Another survey by Vining et al., of the applicants and faculty in the field of complex general surgical oncology, highlighted that the decreased costs and time savings caused less stress for the applicants [8]. Our survey of the PS PDs and applicants also received similar comments in the open-ended sections regarding time and resource savings from the administrative and program perspective as well as the significant decrease in time away from clinical duties for applicants as well as significant cost savings.

One of the major highlights of this survey was to assess and report the overall satisfaction of the virtual interviews from both the PD and applicant perspectives. Similar to the videoconferencing trial done in 2018 by Johns Hopkins All Children's to screen patients for an on-site interview invitation, the majority of PDs were satisfied with the virtual interviews. A popular opinion based on the open-ended responses was using virtual interviews as a potential screening tool followed by limited on-site interviews [2,4]. Concerns about missing the opportunity to see the hospital, city and people in-person remained high among the applicants. Both applicants and PDs alike did not feel that you could assess “the fit” as well virtually. This has been echoed in the previous survey studies as well [2,8,9].

One of the major limitations of this survey is the retrospective nature and inherent recall bias as well as the short term follow-up. The COVID-19 crisis disrupted the traditional interview season for PS and left virtual interviews as the only alternative for both the applicants and the PDs. With such little time to prepare for virtual interviews, programs may not have been able to provide the same virtual experience as in the coming years with more preparation time. The survey itself is not comprehensive in gathering the granular details of the interview day simulated over the videoconference. We did not discuss the details of how the concerns of seeing the hospital and facilities in-person, or meeting the faculty and fellows in-person, could have impacted the rank list for both the parties.

A repeat, comprehensive survey of both the applicants and PDs in future virtual interview seasons such as this year are planned to further delineate the aspects of the virtual interviews that can be improved for future implementation. Additionally, follow-up surveys of both PDs and fellows after these matched applicants have started their fellowship will enable better determination of the ultimate impact of virtual vs. in-person interviews. In other words, did the programs and candidates enjoy a good match?

5. Conclusion

Virtual interviews in the recruitment process of surgical fellowships has resurfaced as a viable alternative to the classic in-person interviews due to the COVID-19 pandemic crisis. While virtual interviews may not be able to completely replace the information obtained from the in-person interview experience, we may be able to learn from the experiences of this year to improve the virtual process in order to better replace the “intangibles” of the on-site process. However, programs and applicants are realizing the time and financial advantages of virtual technology that may prove beneficial past the pandemic crisis. As the classic saying goes, “Never let a good crisis go to waste.” Thus, components of the virtual interview especially as a screening tool might be in the future of the pediatric surgery fellowship interview process, a change that has been years in the making.