Abstract
Objective
During the COVID‐19 pandemic, away rotations were suspended for safety purposes. This led to the development of online interactive learning modules for students, now known as virtual sub‐internships (VSIs). To date, VSIs within otolaryngology‐head and neck surgery (OHNS) have been limited in their description and design.
Study design
Cross‐sectional survey.
Setting
ACGME‐accredited OHNS residency program.
Methods
Our curriculum for an OHNS VSI is presented. Based on the model used by our OHNS residency program, the VSI consisted of a 2‐week block of activities and interactive small‐group discussions. A post‐VSI feedback survey was distributed to participants; results are reported.
Results
Six 2‐week VSI sessions were administered. Twenty‐one individuals participated from all US regions, median 4 individuals per session. Fifteen participants (71.4%) completed the feedback survey. Of survey respondents, 20% lacked a home OHNS residency program, and no respondents' home‐programs featured a VSI. All respondents were satisfied with the schedule and organization of the VSI, and 73.3% (11 of 15) felt it was of appropriate duration. All respondents reported a high degree of familiarity with the program, with a greater (86.7%) or equal (13.3%) level of interest in applying to the residency program as a result of participating in the VSI.
Conclusion
This VSI curriculum offers a well‐received virtual learning experience for medical students applying to OHNS residency. It provides an opportunity for programs to expand their appeal to potential applicants who may otherwise be restricted in their ability to travel.
Keywords: away rotation, education, ENT, learning, medical school, OHNS, otolaryngology, residency, sub‐I, subinternship, sub‐internship, surgery, teaching, virtual sub‐internship, VSI
1. INTRODUCTION
Visiting sub‐internships, also known as externships or away rotations, are an important aspect of the fourth‐year curriculum for medical students, particularly those applying into a surgical residency. Within competitive surgical fields such as otolaryngology‐head and neck surgery (OHNS), the away rotation serves as an opportunity for education and audition, allowing programs and applicants to demonstrate their strengths and assess compatibility. For students from institutions without an OHNS residency program, away rotations are a crucial time to gain experience, demonstrate competence, and establish connections in the field. 1 Prior to COVID‐19, participation in away rotations was nearly universal for successful applicants to OHNS, with a significant majority completing 2 months of rotations away from their home program. 2 , 3 , 4 In surgical specialties, completion of an in‐person sub‐internship has been shown to increase the odds of matching at the host program. 5 , 6 , 7
In response to COVID‐19, away rotations were suspended during the 2020 to 2021 academic year. 8 Recommendations for “novel teleconferencing experiences” consisting of “virtual didactic programs and grand rounds” were suggested in lieu of in‐person sub‐internship rotations. 8 , 9 , 10 As program websites and social media accounts have become a key means of connection with prospective applicants, programs across multiple specialties developed various forms of virtual sub‐internships (VSIs). 11 , 12 , 13 , 14 , 15 , 16 Dean et al (2020) first described the implementation of a plastic surgery VSI built around the established Accreditation Council for Graduate Medical Education (ACGME) core competencies for in‐person rotations. 13 In other surgical subspecialties, similar curricula have been well‐received and highly valued by the majority of participating students. 15 , 16 , 17 In OHNS, only single‐day, townhall, or webinar format VSIs have been described in the literature to date. 18 , 19 For the 2021 to 2022 application year, the Coalition for Physician Accountability and the Association of American Medical Colleges (AAMC) announced that in‐person away rotations are to be limited to one per student, with specific exceptions for students without home‐programs. 20 In this new paradigm, OHNS programs will continue to face the problems of accurately assessing candidate appropriateness‐of‐fit and developing program interest through inherently limited virtual interactions. Therefore, implementing a comprehensive and interactive VSI in OHNS is critical, and may soon become a practice used by residency programs throughout the United States.
2. METHODS
Our VSI consisted of a 2‐week curriculum comprised of scheduled meetings and activities with our institution's OHNS faculty and residents. Core objectives addressed by the curriculum were based upon ACGME objectives and a surgical committee statement, outlined in Issa et al (2015). 6 , 21 (curriculum objectives are reported in Table 1).
TABLE 1.
Virtual sub‐internship curriculum objectives and educational modalities
| Objective | Educational modality |
|---|---|
| Didactic learning | 1. Weekly journal club |
| 2. Case reviews with faculty and residents | |
| 3. Virtual grand rounds | |
| 4. Subspecialty‐specific lectures | |
| 5. Virtual conferences held by professors | |
| 6. Exposure to broad spectrum of subspecialties, including head and neck surgical oncology, facial plastic and reconstructive surgery, rhinology, laryngology, sleep surgery, salivary surgery, and otology | |
| Opportunity for mentorship | Weekly sessions with faculty and residents |
| Professional development | 1. Interview preparation session |
| 2. Student final presentation | |
| Assessing mutual program‐applicant “personality fit” | Virtual happy hour with residents |
2.1. Curriculum
The 2‐week curriculum consisted of a rotating schedule of events arranged throughout the day. Sessions included journal club, faculty‐led discussions of various OHNS topics, resident‐led study sessions, operating room sessions, Zoom interview preparation, grand rounds, student presentations, and a social hour. Topics for faculty‐led discussion included otology, rhinology, facial nerve reanimation, facial plastic and reconstructive surgery, head and neck surgery, and laryngology. Sessions were held via Zoom from faculty offices, the resident workroom, and various operating rooms. The resident study session is an informal discussion of surgical cases with an emphasis on board review and clinical pearls. The operating room sessions consisted of tours of the operating rooms along with faculty‐ and resident‐led discussions of surgical planning, relevant anatomy, and diagram‐based teaching in a question‐and‐answer format. No identifying personal health information or images of patients in surgery were shared during the VSI. The Zoom interview preparation session was led by faculty and consisted of a common interview question review, a detailed account of the structure of the Zoom interview day, tips and tricks to maximize presentability, and a workshop on how to objectively decide and prepare a rank list. At the start of the VSI, participants were assigned to a resident mentor and were encouraged to schedule additional time with individual faculty throughout the course (a complete course schedule is outlined in Table 2).
TABLE 2.
Virtual sub‐internship course schedule
| Activity | Topic | Duration (min) | Frequency | |
|---|---|---|---|---|
| Monday | Journal club | Head and neck oncology | 30 | Weekly |
| Faculty‐led discussions | Facial trauma and reconstruction | 30 to 60 | Weekly | |
| Tuesday | Faculty‐led discussion | Facial plastics and reconstructive surgery | 30 to 60 | Weekly |
| Operating room session | Various subspecialties | 60 to 90 | Weekly | |
| Wednesday | Faculty‐led discussion | Rhinology; otology | 30 to 60 | Weekly |
| Resident office hours | Miscellaneous topics | 60 | Weekly | |
| Thursday | Faculty‐led discussion | Laryngology; salivary surgery; facial nerve injury case review; head and neck reconstruction | 90 to 150 | Weekly |
| Operating room session | Various subspecialties | 60 to 90 | Weekly | |
| Friday | Resident study session | Miscellaneous topics | 60 | Week 1 |
| Zoom interview preparation | Professional growth/education | 30 | Week 1 | |
| Grand rounds | Various subspecialties | 180 | Week 2 | |
| Student presentations | Various subspecialties | 10 min/presentation | Week 2 | |
| Resident social hour | Program atmosphere | 60 | Week 2 |
2.2. Feedback survey
A brief feedback survey was generated using Qualtrics software. (Qualtrics, Provo, UT). The survey consisted of 14 multiple choice and free‐response questions. Upon conclusion of the 2‐week VSI, all participants were emailed a survey link and informed that completing the survey was optional, and that results from the survey could be used in a study analysis. Completing the survey indicated their consent. Responses were kept anonymous and it was emphasized several times throughout the survey that participation and content of responses would not be shared with residency selection committee members to influence the chance of interviewing or matching. Totals and frequencies were calculated. Inclusion criteria were defined as a fourth‐year US medical student who participated in the entire 2 weeks of the VSI. This study was deemed exempt from review by our institution's Internal Review Board.
3. RESULTS
Across the study period, 21 individuals participated in the OHNS VSI, of whom 33.3% (7) identified as female. The VSI involved a total of six 2‐week courses beginning in September 2020, with a median of four participants per 2‐week course. Upon conclusion of the course, 16 (76.2%) participants completed the exit survey. One respondent was unable to complete the 2‐week curriculum due to personal reasons and was excluded from the final analysis. The 15 participants were distributed across United States census‐designated regions evenly: South (33.3%), Midwest (33.3%), Northeast (20.0%), and West (13.3%). All applicants from the West region were from California‐based programs. Among participants, 53.3% first learned of the VSI through the affiliated residency website, whereas 33.3% found it via the Otomatch.com message board or through social media posts by our institution's OHNS‐affiliated accounts (Instagram, Facebook, and Twitter). (Table 3).
TABLE 3.
VSI survey demographics and program characteristics
| Category | No. (%) |
|---|---|
| Total participants | 21 |
| Female | 7 (33.3) |
| Total VSI sessions | 6 |
| Median class size [range] | 4 [2‐4] |
| Duration (weeks) | 2 |
| VSI feedback survey (n = 15) | |
| US region | |
| South | 5 (33.3) |
| Midwest | 5 (33.3) |
| Northeast | 3 (20.0) |
| West | 2 (13.3) |
| OHNS residency offered at home institution | |
| Yes | 12 (80.0) |
| No | 3 (20.0) |
| OHNS VSI offered at home institution | |
| Yes | 0 |
| No | 15 |
| Previously completed OHNS sub‐internship | |
| Yes | 13 (86.7) |
| No | 2 (13.3) |
| Number of additional intended VSIs | |
| 0 | 5 (33.3) |
| 1 to 2 | 10 (66.6) |
| ≥3 | 0 |
Abbreviations: OHNS, otolaryngology‐head and neck surgery; VSI, virtual sub‐internship.
Twelve of 15 (80%) survey respondents were from programs that featured an OHNS residency, and 73.3% (13 of 15) had already completed at least one in‐person sub‐internship in OHNS. None of the participants had home‐programs which featured a VSI. Of the participants who completed the OHNS VSI, 66.6% (10 of 15) reported their intent to complete 1 to 2 additional VSI's at other institutions, whereas the remaining 33.3% did not intend to do other VSI's.
Regarding the length of the OHNS VSI curriculum, 11 of 15 participants (73.3%) felt the 2‐week timeline was of appropriate duration, whereas 2 (13.3%) wanted a longer, and 2 (13.3%) wanted a more condensed curriculum. All participants reported being at least “satisfied” with the schedule and organization of the VSI, with 11 of 15 (73.3%) reportedly being “highly‐satisfied.” (Table 4).
TABLE 4.
Responses from the 2020 to 2021 OHNS VSI feedback survey
| Survey | No. responses (%) (n = 15) |
|---|---|
| 1. Where did you first hear about this OHNS VSI? | |
| OtoMatch.com message board | 3 (20.0) |
| Otolaryngology residency website | 8 (53.3) |
| Social media (Instagram, Facebook, etc.) | 2 (13.3) |
| Word of mouth | 1 (6.7) |
| Other | 1 (6.7) |
| 2. Regarding the 2‐week length of the VSI | |
| It could be shorter | 2 (13.3) |
| It was “just right” | 11 (73.3) |
| I wish it was longer | 2 13.3) |
| Other | 0 |
| 3. Satisfaction with VSI schedule and organization | |
| Highly satisfied | 11 (73.3) |
| Satisfied | 4 (26.7) |
| Unsatisfied | 0 |
| 4. New level of interest/excitement about the affiliated OHNS residency program | |
| Greater than before VSI | 13 (86.7) |
| Same (baseline) | 2 (13.3) |
| Less | 0 |
Abbreviations: OHNS, otolaryngology‐head and neck surgery; VSI, virtual sub‐internship.
Having completed the OHNS VSI, all participants reported continued interest in applying and matching to the affiliated otolaryngology residency program, with 86.7% (13 of 15) reporting a greater level of interest in the program than before participating in the VSI. Following completion of the VSI, participants reported a median rating of “Extremely Comfortable” (5 out of 5) with topics related to the residency program, including living in the region, interacting with residents and faculty, and exploring research options within the field (Figure 1).
FIGURE 1.
Average self‐reported familiarity with each topic is represented by bar height (0 = not at all familiar or comfortable, 5 = extremely familiar or comfortable). VSI, virtual sub‐internship; OHNS, otolaryngology‐head and neck surgery; ENT, ear/nose/throat. 
, median familiarity rating
4. DISCUSSION
The OHNS VSI format described can be easily implemented by programs to offer a representative virtual learning experience. This curriculum served to capture the attention of potential applicants from around the country, with the effect of maintaining or increasing their interest in the affiliated residency program. After completing the VSI, despite having limited to no in‐person experience, participants reported a high level of comfort with topics related to residency at the affiliated institution. Though there can be no substitute for an in‐person rotation, a VSI provided a high degree of program familiarity, suggesting that this format can be a surrogate experience in lieu of an in‐person rotation.
The format used for the VSI was well‐received based on the survey feedback responses. For programs interested in implementing their own VSI, the development of a 2‐week schedule is a reasonable length of time based upon our model. Programs should encourage participation by faculty from as many subspecialties as possible, set side adequate time for each session, and feature a mix of group activities as well as one‐on‐one sessions with faculty and residents. Of note, because the VSI was classified as an informal course by the affiliated medical school, grades, and course credit were not assigned to participants.
In the era of COVID‐19, many opportunities for travel and in‐person education were suspended for safety purposes. 8 , 20 In light of these changes, the VSI offers an opportunity for programs to expand their appeal to potential applicants who may otherwise be restricted in their travel. As travel restrictions are lifted, the success of the VSI should not be discounted in the post‐COVID‐19 era. Based on this data, the VSI format can stimulate applicant interest in the field and in hosting institutions. In addition, it has been suggested that VSIs may enhance the diversity of a program's applicant pool. 22 Importantly, these benefits are possible without the added cost burden to applicants for travel and housing, and without the legal liability typically assumed by institutions for visiting students. 23 , 24 , 25 Much like working from home and the associated increased utilization of tele‐medicine, the VSI may become a permanent experience alongside in‐person away‐rotations for residency programs well into the future.
There are myriad ways to improve upon this OHNS VSI model. Feedback received from VSI participants voiced a desire for greater opportunities to demonstrate their unique value as future residents. Dean et al outline a series of hands‐on workshops in addition to virtual meetings, which aim to challenge students' motor skills and demonstrate their competency for surgical mechanics. 13 Programs looking to develop the VSI model may consider adding hands‐on experiences and graded assignments.
Limitations of the VSI survey include a low number of participants, which may affect the generalizability of the findings. Furthermore, it was not possible to remove self‐selection bias where only the most interested and impressed candidates may have replied to the survey. The timing of the survey shortly after completion of the VSI may also introduce a recency bias, allowing for the conflation of participants' excitement over the prospect of matching to residency. Despite numerous reassurances of anonymity, given the highly competitive residency match environment, participants may have felt compelled to rate aspects of the VSI more highly. In addition, reporting familiarity with certain topics is a relative metric, and may only reflect one's perception of familiarity more than his or her true knowledge. Lastly, lacking a pre‐VSI questionnaire limits our ability to determine the true effect of the VSI on building familiarity and positive feelings about the program. These limitations highlight the need for future efforts and a further collaborative, multi‐site study on this evolving topic in medical education.
5. CONCLUSION
A successful and easy‐to‐implement VSI curriculum for OHNS programs is described. As during the COVID‐19 pandemic, VSI opportunities can foster interest in programs for those applicants who are otherwise unable to travel. VSIs are convenient, effective, and may become more prevalent in years to come.
CONFLICT OF INTEREST
The authors declare no conflicts of interest.
AUTHOR CONTRIBUTIONS
Jeffrey D. Bernstein: study design, data collection, data analysis, manuscript writing, manuscript editing, and preparation of figures. Laurel L. Ball: manuscript writing, manuscript editing, preparation of figures. Zachary B. Nardone: manuscript writing, manuscript editing, preparation of figures. Deborah Watson: study design, manuscript editing.
ACKNOWLEDGEMENTS
The authors would like to recognize and give thanks to Norissa Gastelum for her many hours of work towards ensuring success in the administration of the virtual sub‐internship.
Bernstein JD, Ball LL, Nardone ZB, Watson D. A virtual sub‐internship for otolaryngology‐head and neck surgery. Laryngoscope Investigative Otolaryngology. 2021;6(5):952‐957. doi: 10.1002/lio2.637
This paper was presented at Scientific Conference: UC San Diego Department of Surgery Spring Research Symposium 2021.
BIBLIOGRAPHY
- 1. Wang JC, Pillutla P, Tello N, et al. Competitiveness of otolaryngology residency applicants without a home program. Ann Otol Rhinol Laryngol. 2020;129(5):462‐468. 10.1177/0003489419892016 [DOI] [PubMed] [Google Scholar]
- 2. Kasle DA, Torabi SJ, Izreig S, Rahmati RW, Manes RP. COVID‐19's impact on the 2020‐2021 resident match: a survey of otolaryngology program directors. Ann Otol Rhinol Laryngol. 2020;0003489420967045:666‐673. 10.1177/0003489420967045 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Academy of American Medical Colleges (AAMC) . Away rotations of U.S Medical School graduates by intended specialty. 2019 AAMC Medical School Graduation Questionnaire (GQ); 2019.
- 4. AAMC . Away rotations of U.S. Medical School Graduates by Intended specialty. 2020 AAMC Medical School Graduation Questionnaire (GQ); 2020.
- 5. Jacobson RA, Daly SC, Schmidt JL, et al. The impact of visiting student electives on surgical match outcomes. J Surg Res. 2015;196(2):209‐215. 10.1016/j.jss.2015.03.031 [DOI] [PubMed] [Google Scholar]
- 6. Issa N, Ladd AP, Lidor AO, Sippel RS, Goldin SB. Surgical subinternships: bridging the chiasm between medical school and residency. Am J Surg. 2015;209(1):8‐14. 10.1016/j.amjsurg.2014.10.006 [DOI] [PubMed] [Google Scholar]
- 7. National Resident Matching Program . Results of the 2020 NRMP program director survey. 2020.
- 8. Society of University Otolaryngologists SUO/AADO/OPDO statement regarding away rotations. 2020. https://cdn.ymaws.com/suo-aado.org/resource/resmgr/covid-19/suo-aado-opdo_away_rotation_.pdf. Accessed May 17, 2020.
- 9. Dedeilia A, Sotiropoulos MG, Hanrahan JG, Janga D, Dedeilias P, Sideris M. Medical and surgical education challenges and innovations in the COVID‐19 era: a systematic review. In Vivo. 2020;34(3 Suppl):1603‐1611. 10.21873/invivo.11950 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Chou DW, Pletcher SD, Bruss D, et al. Otolaryngology residency interviews in a socially distanced world: strategies to recruit and assess applicants. Otolaryngol Head Neck Surg. 2020;164:903–908. 10.1177/0194599820957961 [DOI] [PubMed] [Google Scholar]
- 11. Gaeta TJ, Birkhahn RH, Lamont D, Banga N, Bove JJ. Aspects of residency programs' web sites important to student applicants. Acad Emerg Med. 2005;12(1):89‐92. 10.1197/j.aem.2004.08.047 [DOI] [PubMed] [Google Scholar]
- 12. DeAtkine AB, Grayson JW, Singh NP, Nocera AP, Rais‐Bahrami S, Greene BJ. #ENT: otolaryngology residency programs create social media platforms to connect with applicants during COVID‐19 pandemic. Ear Nose Throat J. 2020;1‐5. 10.1177/0145561320983205 [DOI] [PubMed] [Google Scholar]
- 13. Dean RA, Reghunathan M, Hauch A, Reid CM, Gosman AA, Lance SH. Establishing a virtual curriculum for surgical subinternships. Plast Reconstr Surg. 2020;146(4):525e‐527e. 10.1097/prs.0000000000007267 [DOI] [PubMed] [Google Scholar]
- 14. Jiang J, Key P, Deibert CM. Improving the residency program virtual open house experience: survey of urology applicants. Urology. 2020;146:1‐3. 10.1016/j.urology.2020.08.077 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Williams C, Familusi OO, Ziemba J, et al. Adapting to the educational challenges of a pandemic: development of a novel virtual urology subinternship during the time of COVID‐19. Urology. 2021;148:70‐76. 10.1016/j.urology.2020.08.071 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Margolin EJ, Gordon RJ, Anderson CB, Badalato GM. Reimagining the away rotation: a 4‐week virtual subinternship in urology. J Surg Educ. 2021. 10.1016/j.jsurg.2021.01.008 [DOI] [PubMed] [Google Scholar]
- 17. Mikhail D, Margolin EJ, Sfakianos J, et al. Changing the status quo: developing a virtual sub‐internship in the era of COVID‐19. J Surg Educ. 2021. 10.1016/j.jsurg.2021.03.007 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Farlow JL, Marchiano EJ, Fischer IP, Moyer JS, Thorne MC, Bohm LA. Addressing the impact of COVID‐19 on the residency application process through a virtual subinternship. Otolaryngol Head Neck Surg. 2020;163(5):926‐928. 10.1177/0194599820934775 [DOI] [PubMed] [Google Scholar]
- 19. Ruthberg JS, Quereshy HA, Ahmadmehrabi S, et al. A multimodal multi‐institutional solution to remote medical student education for otolaryngology during COVID‐19. Otolaryngol–Head Neck Surg. 2020;163(4):707‐709. 10.1177/0194599820933599 [DOI] [PubMed] [Google Scholar]
- 20. Coalition for Physician Accountability Updated recommendations on away rotations for medical education institutions of LCME‐accredited, U.S. osteopathic, and non‐U.S. medical school applicants. 2021. https://www.aamc.org/media/52326/download. Accessed April 14, 2020.
- 21. Accredation Council for Graduate Medical Education (ACGME) Policies and procedures. 2020. http://www.acgme.org/portals/0/pdfs/ab_acgmepoliciesprocedures.pdf. Accessed May 11, 2020.
- 22. Asaad M, Glassman GE, Allam O. Virtual rotations during COVID‐19: an opportunity for enhancing diversity. J Surg Res. 2021;260:516‐519. 10.1016/j.jss.2020.11.071 [DOI] [PubMed] [Google Scholar]
- 23. Chen AF, Secrist ES, Scannell BP, Patt JC. Matching in orthopaedic surgery. J Am Acad Orthop Surg. 2020;28(4):135‐144. 10.5435/jaaos-d-19-00313 [DOI] [PubMed] [Google Scholar]
- 24. Winterton M, Ahn J, Bernstein J. The prevalence and cost of medical student visiting rotations. BMC Med Educ. 2016;16(1):291. 10.1186/s12909-016-0805-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Griffith M, DeMasi SC, McGrath AJ, Love JN, Moll J, Santen SA. Time to reevaluate the away rotation: improving return on investment for students and schools. Acad Med. 2019;94(4):496‐500. 10.1097/acm.0000000000002505 [DOI] [PubMed] [Google Scholar]