Abstract
Background:
The COVID-19 pandemic in 2020 led to multiple changes in graduate medical education programs across the country, including the switch to virtual interviews for all residency applicants instead of on-site visits. The rapid transition to virtual interviews introduced challenges, including limited opportunities to formally and informally interact with residents and faculty, observe the clinical and educational environments, and explore the local culture and community. As a result, programs were advised to heavily invest in and create comprehensive digital resources including but not limited to video tours and multimedia resources describing programmatic details.
Methods:
In preparation for the virtual interview season of 2020–2021, digital recruitment materials were created for the University of Nebraska Medical Center’s Anesthesiology residency applicants to provide the information that they would traditionally receive during an in-person interview experience. The objectives of the study were (1) to assess which digital materials residency applicants accessed most frequently during the interview season, and (2) to determine if the digital materials were helpful for the residency applicant in best determining program fit as part of the interview process. A post-interview survey and user analytics were analyzed.
Results:
With a survey response rate of 58% (n = 87 of 150) and a Web-based email-open rate of 98% (n =147 of 150), the data revealed that the favored digital materials were the “What Residents Say” video and the Residency Applicant Handbook. These were also the most helpful for the residency applicant in best determining program fit.
Conclusion:
This study shows that resources that allowed students to better assess their “fit” in the program were highly accessed and valued, as were detailed descriptions of the clinical and educational aspects of the training program found in the resident handbook.
Keywords: Residency recruitment, residency match, anesthesiology digital media
Introduction
The COVID-19 pandemic in 2020 led to multiple changes in graduate medical education (GME) programs across the country, including the switch to virtual interviews for all residency applicants instead of on-site visits. The rapid transition to virtual interviews introduced challenges including limited opportunities to formally and informally interact with residents and faculty, observe the clinical and educational environments, and explore the local culture and community.1 As a result, programs were advised to heavily invest in and create comprehensive digital resources including but not limited to video tours and multimedia resources describing programmatic details. Programs also explored new ways to leverage social media to provide exposure to the faculty and residency culture.1,2 These recommendations presented incredible demands on time and financial resources, pushing residency programs to focus on the most desired recruiting content.
The top 5 considerations for allopathic medical students applying to all residency programs include desired geographic location, perceived goodness of fit, reputation of program, academic medical center program, and quality of residents in program.3 In addition, residency applicants have previously reported the importance of program Web sites in deciding where to apply and interview as well as formulate their rank list; however, content analyses showed that perhaps only 46% to 60% of desired information is present.4,5 Training programs across the United States scrambled to create and distribute digital materials focused on high-yield information for the prospective residents. We developed a system to measure the utilization and impact of the newly created digital content for our program. The objectives of our study were (1) to assess which digital materials residency applicants accessed most frequently during the interview season, and (2) to determine if the digital materials were helpful for the residency applicant in best determining program fit as part of the interview process.
Methods
An institutional review board (IRB) protocol was submitted to University of Nebraska Medical Center (UNMC) IRB describing the proposed study designed to assess information regarding the use of digital materials for residency applicant recruitment during the virtual interview season, which was determined to be exempt from human subjects research (IRB 313-21-EX).
Setting and Participants
After a holistic review of residency applications, 150 applicants were invited to interview in consideration for anesthesiology residency class of 2024 (10 positions available) at UNMC. Virtual interviews were conducted via a teleconference video platform (Zoom, San Jose, CA) on 10 different dates with 15 applicants per session, starting in November 2020 through January 2021.
Digital Recruiting Materials Via Web-Based Email Marketing Software
In the week preceding their interview, applicants received an email using a Web-based email marketing software (Constant Contact, Loveland, CO). The software allows the department to build contact databases; create interactive, media-rich emails; and track data on how recipients interact with the email and its contents. The email (Figure 1) was crafted by the department’s communication specialist and reviewed by the education team. The team sought to re-create, as best as possible, the information typically delivered during the traditional in-person campus visit. The email included information about the interview day (ie, applicant’s interview date, schedule, and instructions to join the teleconference meeting on the interview day), and links to the following digital recruiting materials:
Figure 1.
Pre-interview day e-mail via Web-based email marketing software.
What Our Residents Say Video. https://youtu.be/ZR7xQlSVx2Q. This 5-minute, 45-second video features 4 current residents describing our program in their own words. Each outlined their perceived strengths of the program, which included clinical variety and acuity, the quality and number of mentors, the variety of teaching and learning methods, the friendship among peers, the family-friendly culture, and the livability of Omaha. Video footage that coincides with narration depicts the residents working in a variety of hospital environments and spending time with family. The video was created by the department’s communication specialist.
2020 Anesthesiology Graduation Video. https://youtu.be/ubheQ8TzlW0. The 2020 Anesthesiology Graduation video is the department’s socially distant celebration of our graduating residents during the pandemic. It was included to illustrate the strong collegial bonds formed among mentors and mentees. At nearly 20 minutes long, the video was not intended to be viewed in its entirety. The video was created by the department’s communication specialist.
GME Programs Video. https://youtu.be/hqKfKtwkKCA. The university-wide GME video is 7 minutes long and is narrated by the university chancellor, the associate dean of GME, the dean of the College of Medicine, and current trainees. Video footage and narration depict university strengths, opportunity, and culture. The video was created by the university’s public relations department, with much of the footage captured by an external, professional videographer. Costs related to this video creation are unknown to the authors.
Campus From Above Video. https://youtu.be/mhcys_l6fAQ. The Campus From Above video, also created by the university’s public relations department, features drone footage of our main campus near midtown Omaha. The 4-minute video was included with intentions of orienting our off-campus applicants to UNMC’s resources. Costs related to this video creation are unknown to the authors.
Residency Handbook. https://www.unmc.edu/anesthesia/residency/Testimonials-Tours.html. In previous application cycles, each applicant to UNMC’s Department of Anesthesiology residency program received a spiral bound handbook that showcased specific program details, including rotations, call schedules, faculty information, and learning resources. The handbook layout was updated in 2020 by the department’s communication specialist to directly address the anticipated questions applicants would have about a program and to be more visually appealing in a digital PDF format.
Residency Website. https://www.unmc.edu/anesthesia/. All residency program Web pages within the department Website were updated to mimic the organization of the residency handbook but with visual, more digestible content for the Web experience. Information was divided into the following pages: Education Team, Clinical Rotations, Didactics, Testimonials and Tours, and Salary/Benefits/Call. The Web site is updated and maintained by the department’s communication specialist and the director of education development.
PDFs. Links to 3 PDFs featuring pertinent information about our department and university, including a highlight of gender equality within the department called “We Strive” http://webmedia.unmc.edu/anesthesia/learningportal/resources/WomenSucceedHere.pdf, the department’s annual report https://www.unmc.edu/anesthesia/about/annual-reports.html, and UNMC “Points of Pride” acknowledging the university’s achievements http://webmedia.unmc.edu/anesthesia/learningportal/resources/UNMC-Points-of-Pride-2019-20.pdf. The gender equality PDF and the annual report PDF were created by the department’s communication specialist, and the Points of Pride PDF was created by the university’s public relations department.
Social Media. Quick links to the department’s social media resources (Facebook, Menlo Park, CA, https://www.facebook.com/UNMCanesthesiology; Twitter, San Francisco, CA, https://twitter.com/UNMCanesthesia; Instagram, Menlo Park, CA, https://www.instagram.com/unmcanesthesiology/) were included. These sites are maintained by the department’s communication specialist.
The authors were able to access user analytics related to the digital materials from YouTube and the Web-based email marketing software. The process used for accessing YouTube data included the following: (1) Log in to YouTube, (2) Select “Your Data in YouTube” from Menu, (3) Click on Analytics button next to video, and (4) Select Engagement data. The data were collected from the day the first interview email was sent on October 27, 2020, to the day Match Rank Lists were due, March 3, 2021. In addition to user analytics data, the authors also were able to collect and analyze data from the Residency Interview Day survey (Appendix 1). Applicants who completed virtual interviews with our program were invited to complete a survey of the virtual interview process. This survey was originally intended for internal programmatic review; however, the educational team determined that the initial analysis of the outcomes was worth pursuing as scholarly work. The survey was designed by the educational team and included 11 questions. Question styles were mixed, including ranking, yes/no, and open-ended response. Before distribution of the survey, the content validity of the instrument was established by subjecting it to review by 2 of the authors, who are educational research experts (F.H., A.D.) and members of the campus Interprofessional Academy of Educators. There were no survey edits noted after the review. The survey was then distributed on February 8, 2021, with a reminder email sent on February 25, 2021. The survey closed on March 3, 2021. The survey was anonymous.
Results
User Analytics
We examined user analytics data provided by the Web-based email marketing software and by YouTube. The email software reported that 98% (n = 147) of applicants opened the pre-interview day email. Of those openers, 100% (n = 147) clicked on at least 1 link within the email. In total, the email had 594 recruiting-media-related clicks from applicants during this period, which is an average of 4 clicks per applicant. The most popular links were the What Residents Say video with 76.7% (n = 115) of applicants clicking a total of 116 times and the Residency Applicant Handbook with 64.0% (n = 96) of applicants clicking a total of 103 times (Table 1). The data from the Web-based email marketing software was not able to provide information about time spent on materials; however, the authors were able to look at additional data from YouTube for those items stored there (ie, videos).
Table 1.
User Analytics Data From the Web-based E-mail Marketing Software
| Recruiting Media Item | Total Recruiting Media Clicks | Average Clicks Per Email | Percent of Applicants Clicked | |
|---|---|---|---|---|
| 1 | What Residents Say video | 116 | 11.6 | 76.7 |
| 2 | Residency Applicant Handbook | 103 | 10.3 | 64.0 |
| 3 | 2020 Anesthesiology Graduation video | 85 | 8.5 | 54.7 |
| 4 | GME Programs video | 77 | 7.7 | 50.0 |
| 5 | UNMC “Points of Pride” PDF | 49 | 4.9 | 32.7 |
| 6 | Annual Report PDFsa | 43 | 4.3 | 28.7 |
| 7 | Gender Equity “We Strive” PDFa | 35 | 3.5 | 23.3 |
| 8 | Department Web site link | 21 | 2.1 | 14.0 |
| 9 | Instagram page | 5 | 0.5 | 3 |
| 10 | Facebook page | 0 | 0 | 0 |
| 11 | Twitter page | 0 | 0 | 0 |
GME, graduate medical education; UNMC, University of Nebraska Medical Center.
Departmental reports.
Video Analytics
The YouTube links were considered as unlisted within the department’s YouTube account, and shared only on the “Residency Testimonials” page on the department’s Website and the embedded video link within the applicant email. This allowed the authors to retain the accuracy of the data in accounting for only those applicants who were interested in our program. The What Residents Say video had 518 total views. In addition, of the views during this period, 41.7% resulted in the viewer watching the video in its entirety. On average, people stopped watching after 3 minutes, 9 seconds. The 2020 Anesthesiology Graduation video had 216 views on YouTube. Of the 216 views during this period, 13.6% resulted in the viewer watching the video in its entirety. On average, people stopped watching after 3 minutes, 58 seconds. Viewership of both videos showed steady decline over time with no drastic drops in viewing.
Post-Interview Day Survey
The response rate for the post-interview day survey was 58% (n = 87 of 150).
Ninety-eight percent (n = 85) of applicants who responded to the survey indicated that they had explored our Website (www.unmc.edu/anesthesia) before interviewing with our program. Pages on the Website that were most helpful included Clinical Rotations, Salary/Benefits, Didactics, Testimonials and Tours, Program Structure, and the Educational Team. Ninety-four percent (n = 82) of applicants said the media resources provided positively affected their view of the program, whereas only 6% (n = 5) said the media resources did not affect their view of the program either way. The media resources that were most helpful for applicants to familiarize with our program were the resident video (n = 71, 84%), the resident handbook (n = 63, 74%), and the graduation video (n = 41, 48%).
Spearman’s rho correlation coefficient was used to assess the relationship between the ordinal variables of number of clicks a media element received by applicants (user analytics data) and the perceived helpfulness of that same item (survey data). The post hoc analysis revealed weak relationships for all of the media elements: What Residents Say video (rs = −0.045, P = .582), the Residency Applicant Handbook (rs = −0.009, P = .913), the 2020 Anesthesiology Graduation video (rs = 0.078, P = .344), the GME Programs Video (rs = 0.218, P = .007), and the Departmental Reports (rs = 0.210, P = .010). In addition, the What Residents Say video, the Residency Applicant Handbook, and the 2020 Anesthesiology Graduation video showed no statistically significant correlation between variables. However, the GME Programs video and the Departmental Reports showed statistically significant associations. The overall percentage of number of clicks a media element received by applicants (user analytics data) and the perceived helpfulness of that same item (survey data) can be seen in Figure 2.
Figure 2.
Percentage of applicant clicks compared with percentage of applicant perceived usefulness for media resources.
Discussion
The COVID-19 pandemic created major changes in how programs recruited and interacted with potential candidates for residency and fellowship positions. Throughout the process, the opportunity to grow digital resources and outreach increased the likelihood that programs and candidates would build connections. This descriptive analysis of different tools and their role in recruitment of residency applicants in a completely digital setting found that department-specific resources (What Residents Say video, the resident handbook, the graduation video, and the GME video) were beneficial to applicants in making decisions on where they want to train. Although widespread vaccination efforts are under way, it will be necessary to continue remote recruitment and interviewing as the pandemic continues to cause reduced travel and social distancing protocols. As we move forward in this new environment in which remote recruitment and interviews are an essential part of a successful match season, understanding the resources candidates access and find valuable when analyzing programs is an excellent first step to optimizing the recruitment process for individual programs.
Data from the Web-based email marketing software (Constant Contact, Loveland, CO) tracking revealed that among our electronic resources, the What Residents Say video, the resident handbook, the graduation video, and the GME video received the highest levels of traffic. The ability to recreate the sense of connection with current residents in the absence of being able to physically visit a city and program is challenging.6,7 Our program sought to help candidates achieve some insight into their future resident colleagues as they discussed important aspects and overall satisfaction with the program in the video. Not only was this resource highly accessed, but 83.5% reported that the resource was valuable. We sought to expand this experience on the interview day via dedicated Zoom meetings with both our chief residents and 2 additional representatives from the residency classes.
The second most-commonly accessed electronic resource was our resident handbook. This was unsurprising to the interview team given that it provided a comprehensive overview of the program and logistics. Candidates found the description of the education and clinical aspects to be valuable as they decided if the program fit their future goals and needs. Evaluation of post-interview data revealed that the handbook was the most accessed resource between the last interview date and the date candidates submitted their rank list. This seems to highlight that at this late stage in the process, candidates are interested in the details of the programs they are highly considering. Given that this is the “last look” many of the candidates are taking, it is essential to ensure any handbook information is up to date, informative, and highlights the strengths of the program.
YouTube analytics were used to determine the average length of time viewers spent watching the 2 program videos. The What Residents Say video was 5 minutes, 45 seconds in length and the average individual watched for 3 minutes, 9 seconds. The graduation video was 18 minutes, 1 second in length and the average viewer stopped watching after 3 minutes, 58 seconds. This provides insight into the optimal duration of video resources. Based on these data, programs should consider further research regarding the duration of video resources.8,9
The lack of connection to social media sites through the original email invitation is an interesting phenomenon. In fact, links to Twitter and Facebook were never accessed and candidates clicked on the Instagram link only 5 times. This observation may be related to generational differences in what today’s learners find valuable in terms of social media resources. Several potential reasons for these findings exist, including that the students may not use these social media platforms as much as the education team predicted. An alternative hypothesis is that when searching for information regarding a professional position, they are looking for more official information than might be found on social media. Finally, because the Web-based email marketing software program only tracks access of resources through the original email, candidates may have accessed these sites through other modalities, such as by searching for the department directly from the social media platforms.
Feasibility
The relevant feasibility factors for this project include the following: technical, personnel, scheduling, and costs. The technical resources were determined by the program’s educational leadership team and identified as tools that our department already owned, licensed, or subscribed to or were provided (supported) by the university (camera equipment, video editing software, video storage, and Web-based email marketing software). The personnel tasked with creating digital recruitment resources were also members of the department (communications specialist and director of education development). The scheduling consisted of time spent creating, editing, and publishing, with all digital recruitment resources completed within 6 months. Specific costs are difficult to determine within this study, but consideration should be made to the accessibility and availability of the previous factors.
Limitations
There are several limitations to the present report. First, the Web-based email marketing software used cannot commit to a 100% accurate tracking rate because of variables such as practices in the email marketing industry, firewalls within educational institutions, and variations in email software. In addition, when candidates directly accessed resources through the UNMC Department of Anesthesiology Web site or through another modality (such as through a Web browser search engine), these visits would also not appear in the pre-interview email marketing software data. This limitation will lead to an underestimation of the frequency with which residency resources were accessed by applicants. The analytics in YouTube also may have caused an increase in number of views. Although the videos were unlisted and shared only on the residency recruitment Web page and within the applicant email, external viewers of the Web site (ie, medical students who were not applicants to our program, faculty, families) may have increased the visitor data count and the viewing times. Another factor that has potential to influence which resources were accessed includes the order of resources on the email and use of differential fonts/colors. The 2 most frequently accessed links were those at the top of the email and they were emphasized using larger font and an alternative color. These resources were purposefully highlighted in the interview invitation email, as the education team hypothesized that they would be the most valuable to candidates. Also important to the limitations is that the survey was intended to provide internal programmatic feedback on the virtual interview process and was distributed only to the candidates who interviewed with our program. This does not represent the general population of residency applicants, nor do all the survey questions pertain to this study. In addition, because of the anonymous responses on the post-interview survey, the authors were not able to review analytics for those who completed the survey alone. Those who did not complete the survey may have responses different from those who did. It is also important to note that the survey was distributed before the rank list due date, and although this was intentional in order to receive timely feedback related to the interview process, this may have also introduced a positive bias in survey results by those applicants who wanted to train with our program.
Conclusions
Video-based resident testimonials and a digital handbook of the residency program were the most reviewed materials of resident applicants to a single midwestern residency training program in anesthesiology. Survey results confirmed that residents largely viewed the materials provided favorably in allowing them to assess their “fit” within the program, and the average length of time spent viewing videos was just more than 3 minutes, regardless of overall length of video. Future studies should be conducted to explore video-based materials and social media preferences of applicants, in the context of virtual recruitment, to guide future efforts.
Appendix 1.
Post-Interview Day Survey
continued
continued
Funding Statement
Financial support: No institutional or corporate funding was used to conduct this activity.
References
- 1.Ashrafzadeh S, Nambudiri V. Fostering certainty in an uncertain era of virtual residency interviews. J Grad Med Educ . 2020;12(5):561–5. doi: 10.4300/JGME-D-20-00503.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Deitte L, Mian A, Esfahani S, Hu J. Going virtual: redesigning the interview experience. J Am Coll Radiol . 2021;18(2):337–9. doi: 10.1016/j.jacr.2020.06.021. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.American Medical Association. Murphy B. Which factors do applicants weigh most when picking residency programs. https://www.ama-assn.org/residents-students/residency/which-factors-do-applicants-weigh-most-when-picking-residency-programs Accessed February 18, 2021.
- 4.Chu LF, Young CA, Zamora AK, et al. Self-reported information needs of anesthesia residency applicants and analysis of applicate related web sites resources at 131 United States training programs. Anesth Analg . 2011;112(2):430–9. doi: 10.1213/ANE.0b013e3182027a94. [DOI] [PubMed] [Google Scholar]
- 5.Joshi A, Bloom D, Spencer A, Gaetke-Udager K, Cohan R. Video interviewing: a review and recommendations for implementation in the era of COVID-19 and beyond. Acad Radiol . 2020;27(9):1316–22. doi: 10.1016/j.acra.2020.05.020. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Schlitzkus LL, Schenarts PJ, Schenarts KD. It was the night before the interview: perceptions of resident applicants about the pre-interview reception. J Surg Educ . 2013;70(6):750–7. doi: 10.1016/j.jsurg.2013.05.008. [DOI] [PubMed] [Google Scholar]
- 7.DeSantis M, Marco CA. Emergency medicine residency selection: factors influencing candidate decisions. Acad Emerg Med . 2005;12:559–61. doi: 10.1197/j.aem.2005.01.006. [DOI] [PubMed] [Google Scholar]
- 8.Bordes S, Walker D, Modica L, Buckland J, Sobering A. Towards the optimal use of video recordings to support the flipped classroom in medical school basic sciences education. Med Educ Online . 2021;26 doi: 10.1080/10872981.2020.1841406. 1841406. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Abeysekera L, Dawson P. Motivation and cognitive load in the flipped classroom: definition, rationale and a call for research. High Educ Res Dev . 2015;34:1–14. [Google Scholar]