Abstract
Otolaryngology surgical education continues to evolve where trainees increasingly use videos to learn technical skills. Trainees commonly use YouTube, but no study to date has evaluated the educational quality (EQ) of otologic surgical videos on YouTube. We aim to assess the EQ of cholesteatoma surgical videos. Cholesteatoma surgical videos were queried using YouTube search terms, assessed using LAParoscopic surgery Video Educational GuidelineS (LAP-VEGaS), a validated assessment tool for publication, and categorized into low (0-6), medium (7-12), and high (13-18) EQ groups. In total, 74 videos were identified (mean LAP-VEGaS score = 9.6 ± 4.0) and 44.6% had medium EQ. Videos commonly lacked graphic aids to highlight anatomy (71.6%) and postprocedural outcomes (68.9%). LAP-VEGaS scores were greater in videos originating from US surgeons compared to non-US surgeons (12.4 ± 3.4 vs 8.0 ± 3.5; P < .001). Our study highlights that otolaryngology trainees may experience difficulty finding high-EQ cholesteatoma surgery videos on YouTube. Areas for improved EQ content are discussed.
Level of evidence: IV
Keywords: cholesteatoma, surgical education, otolaryngology education, internet videos, surgery, otology, otolaryngology trainee
Surgical education in otolaryngology has rapidly progressed over the past decades. Trainees have traditionally used written resources to learn technical skills but are faced with work-hour restrictions.1 To augment the surgical learning experience, videos are now increasingly used, driven by technological advances and ease of accessing online information.2 YouTube, the world’s largest video-sharing platform, offers trainees access to a large number of surgical videos.3-5
The ability for trainees to easily identify high-quality educational content is unclear given YouTube’s proprietary search algorithm,5 which likely promotes video exposure based upon noneducational factors. While prior studies in the general surgery literature have assessed YouTube videos for educational quality (EQ),2,6-8 none have focused on otologic surgical videos. This study aims to explore the EQ of cholesteatoma surgical YouTube videos, a procedure that otolaryngology trainees may encounter early in training and may find challenging due to altered anatomy from chronic disease.
Methods
Search Strategy
This study was deemed exempt from review by the Massachusetts Eye and Ear Institutional Review Board because it involves publicly available videos. To simulate an Internet search by a trainee in “real-world” settings (eg, limited time), a YouTube search according to “relevance” with a cache-cleared browser on January 1, 2021, was performed using “cholesteatoma surgery,” “cholesteatoma procedure,” “cholesteatoma removal,” and “cholesteatoma endoscopic surgery.” The first 50 results from each keyword query were evaluated. Videos were included if cholesteatoma operative footage and English commentary were present. Videos that were non-English were excluded.
Video Evaluation
Video characteristics (eg, view count), presence of auditory commentary and subtitles, and the surgeon’s country of origin were recorded. Each video was graded using the LAParoscopic surgery Video Educational GuidelineS (LAP-VEGaS), a validated surgical video assessment tool for presentation and publication9 previously used to assess pyeloplasties,8 appendectomies,7 and neck dissections.10 Nine items are assessed and scored from 0 (not present) to 2 (extensively presented)9 and used to categorize videos into low (0-6), medium (7-12), and high (13-18) EQ groups.
The LAP-VEGaS scoring was performed by 2 reviewers on a subset of 38 videos.11 The intraclass correlation coefficient was excellent with a value of 0.92 (95% CI, 0.84-0.96; P < .0001), and 1 reviewer then scored the remaining videos.
Statistical Analysis
Categorical variables were summarized by frequency and percentage. Continuous variables were summarized by mean and standard deviation. The Mann-Whitney U test was performed to compare LAP-VEGaS scores between US and non-US-based surgeons. Spearman correlation was performed to evaluate the association between LAP-VEGaS scoring and video characteristics. Statistical analyses were performed using SPSS Statistics. P values <.05 were considered statistically significant.
Results
Video Characteristics
The search query identified 74 videos that met inclusion criteria. The mean view count, video age, and length were 16,141, 4.6 ± 3.2 years, and 14′33′′± 21′9′′, respectively (Table 1). Auditory commentary was present in 60.8% of videos, and 63% did not originate from the United States.
Table 1.
Cholesteatoma Surgical Video Characteristics.
| Characteristic | Value |
|---|---|
| View count, mean (SD) | 16,141 (48,751) |
| Video age, mean (SD), y | 4.6 (3.2) |
| Video length, mean (SD), min:s | 14:33 (21:09) |
| Likes, mean (SD) | 77 (284) |
| Dislikes, mean (SD) | 5 (15) |
| Auditory commentary, No. (%) | |
| Present | 45 (60.8) |
| Not present | 29 (39.2) |
| Subtitles, No. (%) | |
| Present | 49 (66.2) |
| Not present | 25 (33.8) |
| Country of origin, No. (%)a | |
| United States | 27 (36.5) |
| Not the United States | 46 (63.0) |
Less than 100% due to missing data.
Educational Assessment
The mean LAP-VEGaS score was 9.6 ± 4.0, with 19 (25.7%), 33 (44.6%), and 22 (29.7%) videos being categorized into low, medium, and high EQ, respectively. Overall, videos most commonly lacked graphic aids to highlight pertinent anatomy (71.6%), information on postprocedural outcomes (68.9%), and surgical positioning (48.6%). Most videos had clear image quality (91.9%), thoroughly displayed surgical procedure steps (62.2%), and had English commentary (58.1%). LAP-VEGaS scores were significantly greater in videos originating from US surgeons compared to non-US surgeons (12.4 ± 3.4 vs 8.0 ± 3.5; P < .001) (Table 2). LAP-VEGaS scores were positively correlated with higher video like count (r = 0.43; P < .001) and longer video length (r = 0.25; P = .033).
Table 2.
Educational Assessment of Cholesteatoma Surgical Videos.
| Characteristic | Value | P value |
|---|---|---|
| LAP-VEGaS score, mean (SD) | ||
| Overall | 9.6 (4.0) | |
| US surgeon | 12.4 (3.4) | <.001 |
| Non-US surgeon | 8.0 (3.5) | |
| Correlation between video metric and LAP-VEGaS score | ||
| Video count | r = 0.43 | <.001 |
| Video length | r = 0.25 | .033 |
Abbreviation: LAP-VEGaS, LAParoscopic surgery Video Educational GuidelineS.
Discussion
Our study is the first to assess the EQ of otologic surgical videos from YouTube. In our simulated search as an otolaryngology trainee, most cholesteatoma surgery videos had average EQ. Videos often had high image quality and thoroughly presented surgical steps, with videos from US authors having higher EQ.
Developing useful videos for trainees is a challenging process as it involves balancing time spent producing videos12 while maximizing EQ. Our study identifies common areas for improvement, such as using graphic aids and discussion of postsurgical outcomes. Given the regular use of cameras in otologic procedures (endoscopy, microscopy),13 numerous opportunities exist to record educational content. In diseases with chronic destructive processes, such as cholesteatoma,14 distinguishing anatomical structures may be challenging. Therefore, embedding videos with arrows or creating representative diagrams may benefit the viewer.15
The findings from this study and others10,16 highlight the difficulty of trainees to reliably identify high-EQ otolaryngology surgical content on YouTube. Higher LAP-VEGaS scores were found to be positively correlated with greater like counts, which may be a helpful indicator of high-EQ videos. Nonetheless, this platform does not require peer review prior to publishing and may lead to decreased EQ. Currently, few surgical video resources are peer reviewed or created by academic professional societies such as the American Academy of Otolaryngology–Head and Neck Surgery and Headmirror.17 While these atlases use YouTube for video sharing, our search rarely found these videos, highlighting that trainees may need to independently search for these. Emerging journal formats allowing users to submit surgical videos for peer review18 may help maintain academic rigor by following outlined criteria.
This study has several limitations. First, authors may not have been aware of LAP-VEGaS criteria when creating their videos. Nonetheless, the high number of videos with average EQ suggests the need for otolaryngologists to publish more content, potentially in a peer-reviewed format. Second, the LAP-VEGaS criteria were originally created to assess laparoscopic surgery, where elements (eg, port placement) may be less relevant to otologic surgery. However, most elements have strong overlap with otologic surgery in assessing EQ in intraoperative and perioperative settings. Next, LAP-VEGaS criteria, which overall are a suitable fit for cholesteatoma surgery by providing several opportunities to create annotations and commentary, consider author and institution names and intraoperative steps as having the same educational value, which may be less beneficial when learning technical skills. A future study may revise the scoring system to award additional points for technical skills to promote thorough explanations. Finally, the decision to review the first 50 videos of keyword searches may have prevented analysis of high-EQ videos appearing in later search results. Nonetheless, we intended that our search strategy would reflect those of the majority of otolaryngology trainees with time constraints.
Conclusion
This is the first study to evaluate the EQ of otologic surgical videos on YouTube. Otolaryngology trainees may experience difficulty finding high-EQ content on YouTube for cholesteatoma surgery.
Author Contributions
Matthew J. Wu, conceptualization, data analysis, manuscript preparation; Renata M. Knoll, data analysis, manuscript preparation; Karim Bouhadjer, data analysis, manuscript preparation; Aaron K. Remenschneider, data analysis, manuscript preparation; Elliott D. Kozin, conceptualization, data analysis, manuscript preparation.
Disclosures
Competing interests: None.
Sponsorships: None.
Funding source: None.
Footnotes
This article was presented as a poster presentation at the 125th AAO-HNSF 2021 Annual Meeting & OTO Experience; October 5, 2021; Los Angeles, California.
References
- 1. Poulose BK, Ray WA, Arbogast PG, et al. Resident work hour limits and patient safety. Ann Surg. 2005;241(6):847-860. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Ahmet A, Gamze K, Rustem M, Sezen KA. Is video-based education an effective method in surgical education? A systematic review. J Surg Educ. 2018;75(5):1150-1158. [DOI] [PubMed] [Google Scholar]
- 3. Rapp AK, Healy MG, Charlton ME, Keith JN, Rosenbaum ME, Kapadia MR. YouTube is the most frequently used educational video source for surgical preparation. J Surg Educ. 2016;73(6):1072-1076. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Jabbour J, Bakeman A, Robey T, Jabbour N. Self-directed learning in otolaryngology residents’ preparation for surgical cases. Ann Otol Rhinol Laryngol. 2017;126(4):296-303. [DOI] [PubMed] [Google Scholar]
- 5. Covington P, Adams J, Sargin E. Deep neural networks for YouTube recommendations. In: Proceedings of the 10th ACM Conference on Recommender Systems, Boston, MA, USA, September, 2016. Association for Computing Machinery; 2016:191–198. [Google Scholar]
- 6. Rodriguez HA, Young MT, Jackson HT, Oelschlager BK, Wright AS. Viewer discretion advised: is YouTube a friend or foe in surgical education? Surg Endosc. 2018;32(4):1724-1728. [DOI] [PubMed] [Google Scholar]
- 7. de’Angelis N, Gavriilidis P, Martinez-Perez A, et al. Educational value of surgical videos on YouTube: quality assessment of laparoscopic appendectomy videos by senior surgeons vs. novice trainees. World J Emerg Surg. 2019;14:22. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Haslam RE, Seideman CA. Educational value of YouTube surgical videos of pediatric robot-assisted laparoscopic pyeloplasty: a qualitative assessment. J Endourol. 2020;34(11):1129-1133. [DOI] [PubMed] [Google Scholar]
- 9. Celentano V, Smart N, Cahill RA, et al. Development and validation of a recommended checklist for assessment of surgical videos quality: the LAParoscopic surgery Video Educational GuidelineS (LAP-VEGaS) video assessment tool. Surg Endosc. 2019;35(3):1362-1369. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Luu NN, Yver CM, Douglas JE, Tasche KK, Thakkar PG, Rajasekaran K. Assessment of YouTube as an educational tool in teaching key indicator cases in otolaryngology during the COVID-19 pandemic and beyond: neck dissection. J Surg Educ. 2021;78(1):214-231. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420-428. [DOI] [PubMed] [Google Scholar]
- 12. Poon C, Stevens SM, Golub JS, Pensak ML, Samy RN. Pilot study evaluating the impact of otology surgery videos on otolaryngology resident education. Otol Neurotol. 2017;38(3):423-428. [DOI] [PubMed] [Google Scholar]
- 13. Kozin ED, Gulati S, Kaplan AB, et al. Systematic review of outcomes following observational and operative endoscopic middle ear surgery. Laryngoscope. 2015;125(5):1205-1214. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Hamed MA, Nakata S, Sayed RH, et al. Pathogenesis and bone resorption in acquired cholesteatoma: current knowledge and future prospectives. Clin Exp Otorhinolaryngol. 2016;9(4):298-308. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Ward TM, Fer DM, Ban Y, Rosman G, Meireles OR, Hashimoto DA. Challenges in surgical video annotation. Comput Assist Surg (Abingdon). 2021;26(1):58-68. [DOI] [PubMed] [Google Scholar]
- 16. Shires CB, Wilson CD, Sebelik M. Thyroid surgery YouTube videos: estimating quality by surgeon characteristics and view rate. Gland Surg. 2019;8(3):207-211. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Goates AJ, Chweya CM, Choby G, Carlson ML. An open-access, comprehensive otolaryngology-head and neck surgery video atlas for resident education. Am J Otolaryngol. 2020;41(6):102628. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Kozin ED, Lustig LR. Inaugural otology and neurotology “video report.” Otol Neurotol. 2020;41(4):429-430. [DOI] [PubMed] [Google Scholar]