Evaluating YouTube videos on sialendoscopy as an educational resource for patients

Abstract

Objectives

Patients are increasingly relying on YouTube for health information. We objectively evaluated the quality and comprehensiveness of sialendoscopy YouTube videos available to patients. We further investigated the relationship between video content and video popularity.

Methods

We identified 150 videos using the search term “sialendoscopy.” Videos were excluded if they were lectures for medical professionals, operating room (OR) recordings, unrelated, non‐English, or non‐audio. Video quality and comprehensiveness were evaluated using modified DISCERN criterion (range: 5–25) and novel sialendoscopy criterion (NSC, range: 0–7), respectively. Secondary outcomes included standard video metrics and Video Power Index to measure popularity. Videos were classified binarily by uploader type as from an academic medical center or from other sources.

Results

Twenty‐two (14.7%) of 150 videos were included for review, with 7 (31.8%) uploaded from academic medical institutions. One hundred‐nine (72.7%) videos were excluded as lectures for medical professionals or OR recordings. Overall mean modified DISCERN (13.45 ± 3.42) and NSC (3.05 ± 0.96) scores were low; however, videos uploaded by academic medical institutions were significantly more comprehensive (NSC mean difference = 0.98, 95% CI: 0.16–1.80, p = .02). There were no significant correlations between video popularity and objective measures of quality or comprehensiveness.

Conclusions

This study highlights the paucity and low quality of sialendoscopy videos for patients. More popular videos are not higher quality, and most videos are targeted more toward physicians rather than patients. As YouTube becomes increasingly used by patients, there is opportunity for otolaryngologists to produce more informative videos for patients while implementing targeted strategies to increase viewership.

Level of Evidence

NA.

Patients are increasingly watching YouTube videos as a means of educating themselves on their potential treatment options. YouTube videos available to patients on sialendoscopy are overall sparse and of low quality. Furthermore, more popular videos are not of higher quality.

1. INTRODUCTION

Patients are increasingly using online resources for health information. Recent data suggest that about 80% of Internet users look for health information online, and of these users, 56% look specifically for information on medical treatments or procedures. ¹ Although numerous websites host video content, YouTube is by far the largest and most‐visited repository of videos worldwide. ² The most recent 2020 Health Information National Trends Survey estimates that 39.7% of Americans watched health‐related videos on YouTube, making YouTube the most‐accessed video streaming website specifically for health information. ² , ³ As of August 2021, YouTube's mobile application is the most‐used smartphone application in the United States, and its user base is projected to continue growing as consumers increasingly access information from mobile devices. ⁴

Sialendoscopy is a diagnostic and therapeutic tool used to manage sialolithiasis and sialadenitis refractory to first‐line, conservative treatment. It is a minimally invasive technique that has been shown to preserve function and spare excision of salivary glands in non‐neoplastic obstructive pathologies. ⁵ , ⁶ Sialendoscopy is highly effective in relieving salivary obstruction, with reported success rates approaching 91% and is associated with minimal complications. ⁷ , ⁸ Since its introduction in the 1990s, usage of sialendoscopy has increased, and it is now performed in a variety of surgical settings. ⁹ , ¹⁰ , ¹¹ , ¹²

Despite the growth in patient use of YouTube and clinical use of sialendoscopy, the quality of the information found in sialendoscopy YouTube videos has not been investigated. Given that YouTube videos are unregulated and often contain incomplete or inaccurate health information, there is growing concern over the quality of information that patients access on the website. ¹³ , ¹⁴ , ¹⁵ Thus, this study seeks to objectively evaluate the quality and comprehensives of YouTube videos as a source of health information for patients on sialendoscopy. We hypothesize that the information within YouTube videos available for patient on sialendoscopy is of low quality and comprehensiveness.

2. MATERIALS AND METHODS

This research study received exemption from the Institutional Review Board at Thomas Jefferson University. Videos were searched using the Google Chrome web browser. To minimize user search result bias, browsing history, cookies, and cache were cleared before searching, and all videos were accessed via Google Chrome's private browsing feature. On March 26, 2021, “www.youtube.com” was searched using default settings for the term “sialendoscopy.”

The first 150 videos were identified. A search of 150 videos is likely far beyond what a typical user would do, allowing for analysis of the entire breadth of sialendoscopy video content and proper powering of the study. ¹⁶ , ¹⁷ Videos were excluded if they were (1) lecture format videos intended for medical professionals, (2) non‐English videos, (3) duplicates, (4) surgical and/or operating room (OR) technique‐specific videos, (5) irrelevant videos (defined as videos with titles that do not mention sialendoscopy and contain less than 50% of video time spent discussing sialendoscopy or topics directly related to sialendoscopy), or (6) videos without both audio and subtitles.

Videos were classified based on their uploader as either from an academic medical institution (“academic”) or from any other uploader source (“other”). Other uploaders included private medical institutions, news and media agencies, and individual users. This video stratification is consistent with previous literature ¹⁸ , ¹⁹ , ²⁰ and allowed for proper powering of the study.

Primary outcome measures were scores from two video scoring criteria: a novel sialendoscopy criteria (NSC) score and a modified DISCERN score. Each video was independently scored by two authors. The authors created the NSC to evaluate the comprehensiveness of sialendoscopy information within a video. It was designed with input from other novel scoring criteria in the literature, and it represents the breadth of information an otolaryngologist would likely present to patients undergoing sialendoscopy. ²¹ , ²² , ²³ Scoring (range: 0–7) is based on the discussion of information pertaining to surgical indications, alternatives to surgery, complications, potential need for additional intraoperative interventions, and postoperative prognosis (Table 1).

TABLE 1.

Novel sialendoscopy criteria scoring system

Score range: 0–7 points

Surgical indications (1 pt)

Non‐surgical alternatives (1 pt)

Procedure‐specific details (1 pt)

Complications (2 pts) Intraprocedural (1 pt) Postprocedural (1 pt)

Potential need for additional intraoperative intervention (1 pt)

Postoperative prognosis (1 pt)

Note: Novel sialendoscopy criteria (NSC) used to assess for comprehensiveness of information in videos. Each item is scored 0 or 1 and combined for a total score ranging from 0 to 7. Videos received a “1” if the corresponding criterion was adequately discussed within the video body.

The DISCERN criteria are a validated scoring system developed to help health consumers judge information about treatment choices (i.e., sialendoscopy), and it has been previously used in the literature to evaluate the overall quality and reliability of health information in YouTube videos. ²¹ , ²² , ²⁴ , ²⁵ , ²⁶ The original DISCERN instrument is a 15‐item survey broken up into two sections: reliability (items 1–8) and specific details about treatment choices (items 9–15). ²⁴ The modified DISCERN implements and combines the reliability items (1–8) into a 5‐item score, each graded on a scale of 1–5, yielding a total score of 5–25 (Table 2). ²¹ Items 4 and 5 from the original DISCERN instrument are excluded, as they are specific to written forms of patient education. Items 9–15 are excluded from the modified DISCERN because they (1) are addressed more specifically in the context of sialendoscopy within the NSC, or (2) assume multiple treatment modalities are being compared, whereas our investigation evaluates a single treatment. The DISCERN bias score was the third criteria in the modified DISCERN score and was used to assess the presence of bias in videos (Table 2).

TABLE 2.

Modified DISCERN criteria scoring system ^a

Question	Scoring: Range 5–25
1. Are the aims clear and achieved?	1–5
2. Are reliable sources of information used (i.e., publication cited, speaker is otolaryngologist)?	1–5
3. Is the information balanced and unbiased?	1–5
4. Are additional sources of information listed for patient reference?	1–5
5. Are areas of uncertainty mentioned?	1–5

Note: Modified DISCERN criteria used to evaluate the quality of information in videos. Each item is scored on a scale of 1 to 5 and combined for a total score ranging from 5 to 25.

^a Adapted from Wu et al. ¹⁹

Secondary outcome measures included: (1) audiovisual (AV) quality score, (2) reported video metrics, and (3) calculated video metrics. The AV quality score has previously been used to evaluate YouTube videos and ranges from zero to three, with higher scores indicating clearer, more professional editing (Table 3). ¹⁶ Reported video metrics included the number of views, comments, likes, dislikes, days since upload, and duration. Calculated video metrics included like ratio (likes/[likes + dislikes]), view ratio (views/days since upload), and video power index (VPI). VPI is a previously published measure of video popularity calculated as: VPI = ([view ratio * like ratio]/100). ¹⁹ , ²¹ , ²² , ²⁷ , ²⁸ Data used for basic and calculated video metrics were collected at the date of the initial YouTube search (March 26, 2021).

TABLE 3.

Audiovisual quality scoring system ^a

Score Range: 0–3 points

3—Excellent: clear, professional editing

2—Average: non‐professional editing

1—Poor: blurry, out of focus, unintelligible

0—Unable to view

Note: Assessment of video audiovisual quality.

^a Adapted from Singh et al. ²⁰

All statistical analyses were performed using IBM SPSS Statistics Version 27. Statistical significance was set at p < .05. Intraclass Correlation Coefficient (ICC) was calculated based on NSC and modified DISCERN scores from both reviewers, with ICC > 0.90 defined as “excellent” reliability. ²⁹ Basic descriptive statistics and Pearson Correlation Coefficient were calculated for all primary outcomes (using the mean of both reviewers' scores) and secondary outcomes, regardless of uploader type. Primary and secondary outcomes were compared between uploader types (academic and other) using an independent t‐test supplemented by Levene's test and Cohen's D measurement.

3. RESULTS

Of the 150 videos identified from the initial YouTube search, 22 videos (14.7%) met inclusion criteria. The 128 excluded videos included the following: 8 lecture‐format videos intended for a live audience, 9 non‐English videos, 101 surgical or OR technique‐specific videos, 7 irrelevant videos, and 3 videos without both audio and subtitles. Of the 22 included videos, 7 (31.8%) were uploaded from an academic medical institution, and 15 (68.2%) were uploaded from other sources. Other uploader sources included 10 videos from private medical institutions, 4 videos from independent users, and 1 video from a news/media organization (Table 4).

TABLE 4.

Videos stratified by uploader source

Uploader source	Number of videos (% of total videos)
Academic Medical Institution	7 (31.8%)
Other Uploader Sources	15 (68.2%)
Private medical institution	10 (45.5%)
Independent uploader	4 (18.2%)
News/media organization	1 (4.5%)

Note: Videos included in the study stratified by uploader source. Other uploaders included all videos that were not from academic medical institutions and could be categorized into three categories: private medical institutions, independent uploaders, and news/media organizations.

The scorers' video evaluations using the NSC are displayed in Figure 1. Across all videos, the mean NSC score was 3.05 (±0.96), the modified DISCERN score was 13.45 (±3.42), and the DISCERN bias score was 3.36 (±0.79) (Table 2). ICCs between the two independent reviewers were excellent for both NSC (ICC = 0.997) and modified DISCERN (ICC = 0.949). ²⁹ There was a significant positive correlation between total NSC and modified DISCERN scores (r = .833, p < .01). Academic videos scored significantly higher on the NSC score than other videos (mean difference = 0.98, 95% CI: 0.16–1.80, p = .02). There was no significant difference between academic and other videos in the modified DISCERN or DISCERN bias scores (Table 5).

FIGURE 1.

Reviewer scoring for novel sialendoscopy criteria (NSC) by two authors. A score of “1” was given if a reviewer believed that the topic was clearly and adequately discussed, and a score of “0” was given if a reviewer believed that a topic was not clearly and adequately discussed during the video. Colors are used to represent agreement (green or red) or disagreement (yellow) between the two reviewers.

TABLE 5.

Primary outcome results by video uploader type

Outcome	Overall	Academic	Other	p‐value
Novel Sialendoscopy Criteria Score	3.05 (±0.96)	3.71 (±0.81)	2.73 (±0.88)	.02*
Modified DISCERN Score	13.45 (±3.42)	15.14 (±2.01)	12.67 (±3.70)	.12
DISCERN Bias Score	3.36 (±0.79)	3.64 (±0.48)	3.23 (±0.88)	.27

Note: Assessments of video comprehensiveness and quality stratified by uploader type. Data are represented as means (±SD). Reported p‐value represents difference between academic and other videos. Asterisk (*) indicates significance (p < .05).

Abbreviation: SD, standard deviation.

AV quality score, reported video metrics, and calculated video metrics for all videos are displayed (Table 6). Academic videos had significantly higher AV quality scores than videos from other uploaders (mean difference = 0.73, 95% CI: 0.35–1.11, p = .01). Videos that were from other sources had significantly longer duration than those from academic medical institutions (276.3 ± 180 s vs. 156.7 ± 82 s, p < .05). There were no other significant differences in secondary outcomes between the two groups.

TABLE 6.

Secondary outcome results by video uploader type

Metric	Overall	Academic	Other	p‐value
AV quality score (0–3)	2.43 (±0.64)	2.93 (±0.19)	2.20 (±0.65)	.009*
Views	18,208 (±30,023.13)	17,933 (±39,339.00)	18,337 (±26,213)	.977
Comments	23.59 (±52.94)	11.50 (±23.00)	27.31 (±59.50)	.617
Duration (seconds)	238.23 (±163.49)	156.71 (±81.70)	276.27 (±179.90)	.045*
Likes	67.89 (±135.87)	54.33 (±122.89)	74.15 (±145.82)	.777
Dislikes	8.79 (±14.49)	8.33 (±19.45)	9.00 (±12.54)	.929
Like ratio	88.15 (±17.45)	0.79 (±0.39)	0.59 (±0.44)	.434
View ratio	10.60 (±17.38)	7.47 (±12.28)	12.06 (±19.53)	.577
VPI	12.46 (±19.12)	7.42 (±12.47)	15.25 (±22.18)	.485

Note: Reported and calculated video metrics stratified by video uploader type. Data are represented as means (±SD). Reported p‐value represents difference between academic and other videos. Asterisk (*) indicates significance (p < .05).

Abbreviations: AV, audiovisual; SD, standard deviation; VPI, video power index.

Across all videos, AV score was positively correlated with NSC score (R = .429, p < .05) but not modified DISCERN score. Neither NSC nor modified DISCERN scores significantly correlated with any other secondary characteristics. Within secondary characteristics, AV score was significantly positively correlated with like ratio (R = .749, p < .01) and negatively correlated with days since upload (R = −.53, p = .01).

4. DISCUSSION

This is the first study to evaluate the quality and comprehensiveness of sialendoscopy information within YouTube videos for patients. Using the search term “sialendoscopy,” only 22 of the first 150 videos (14.7%) met inclusion criteria, highlighting the paucity of suitable sialendoscopy video content for patients. Users typically only click on videos found within the first search engine result page (SERP), with less than 1% of all user SERP clicks occurring beyond the first page and the likelihood of interaction significantly diminishing as search position increases. ¹⁶ , ¹⁷ , ²⁰ Even in the first 25 search results, only 8 (32%) videos were suitable for patients. Thus, YouTube may not be an appropriate resource for patient's to reliably find sialendoscopy videos suitable to their education level. Physicians may consider directing patients toward other resources, as most users are likely unwilling or unable to reliably search for an appropriate video within the large quantities of unsuitable videos on YouTube.

Given the paucity of videos for patients, it is important to consider who else may be using YouTube for information on sialendoscopy. One hundred‐nine (72.7%) of the initial 150 videos were excluded for being surgical technique videos/OR recordings or lectures for medical professionals that contained technical language or medical jargon. Of these excluded videos, 15 (60%) were in the top 25 search results. This suggests that physicians may be using YouTube to share sialendoscopy content with other physicians or medical personnel. Physicians frequently use online videos to study medical concepts: one study reported that 90% of polled physicians used online videos and, of these physicians, 86% specifically used YouTube. ³⁰ Future research may seek to investigate the quality and utility of physician‐oriented YouTube educational videos.

The overall low NSC and modified DISCERN scores indicate poor video comprehensiveness and quality, respectively. These findings are consistent with our initial hypothesis and reflect the findings of other surgery literature investigating the health information within YouTube videos. ¹⁴ , ¹⁵ , ¹⁹ , ²¹ , ²² Within the overall low scores, academic videos scored significantly higher on NSC than videos uploaded from other sources, suggesting that academic uploaders provide patients with more comprehensive information. While otolaryngologists may encourage patients to seek videos uploaded from academic sources to access the most comprehensive information, YouTube overall appears to be a poor source of quality health information on sialendoscopy.

The videos included in this study were poor at discussing complications. None (0%) of the videos described intra‐procedural complications, and only three videos (13.7%) discussed any complications; these low reporting rates are consistent with our initial hypothesis and literature in other fields. ²² , ²³ This lack of discussion may be due to the low complication rate of sialendoscopy procedures overall or because not all included videos were intended for patient education. ⁷ , ³¹ Physicians may choose to increase discussion of these topics in the future if their goals are to create comprehensive videos on sialendoscopy.

The above findings highlight that YouTube may not be an ideal resource for physicians to recommend to their patients. Physicians seeking to use YouTube for sialendoscopy patient education may consider preselecting videos at their own discretion and providing their patients with weblinks. This would eliminate the need for patients to siphon through the large quantity of videos intended for medical professionals and encourage targeted selection of higher quality videos. Physicians seeking to create their own high‐quality videos may use the NSC as a reference for pertinent inclusion points.

Viewers of online videos often perceive more popular videos as more credible sources of information. ³² We found no correlations between the two scoring criteria (NSC and modified DISCERN) and popularity, indicating that more popular videos are not necessarily of higher quality. Moreover, although videos uploaded from academic sources were on average more comprehensive than from other sources, there was no difference in popularity between uploader sources. Taken together, this suggests that a highly informative video in and of itself will not ensure viewership, and it highlights an opportunity for academic medical institution uploaders to use strategies to increase video viewership, as even the best videos will not impact patient education if they cannot be viewed.

Growing bodies of literature studying online user behavior have outlined strategies to increase the likelihood of video viewership and can be implemented by otolaryngologists seeking to create content that is both highly informative and accessible. When a YouTube search is queried, the most popular videos appear at the top of YouTube's SERP. ²⁰ Strategies to maximize a video's SERP presence include (1) using thoughtful “keyword” and title selection, (2) sharing the video on corresponding social media platforms, and (3) keeping the video within the 3–5 min range. ²⁰ Keywords and titles should be in colloquial language and kept consistent throughout the video title, the text description, and within corresponding social media posts. The uploader may identify other popular sialendoscopy YouTube videos and choose identical keywords or similar title verbiage to mirror these videos' SERP successes. ²⁰ , ³³ Promoting the video on corresponding social media pages (i.e., Twitter, Instagram) will further establish video validity and direct users of multiple media networks back to YouTube. ²⁰ , ³² Reducing video length augments engagement and increases the likelihood that the video will be watched to completion, a metric than has been shown to improve SERP. ²⁰ , ³⁴ While academic organizations can seek to implement these strategies themselves, various internet services exist to guide creators toward successful content creation and promotion, including the YouTube Creator Academy and Google Trends for YouTube. ²⁰ , ³⁵

Videos uploaded from academic medical institutions had significantly higher AV quality scores. While AV quality score did positively correlate with like ratio, it did not correlate with any other metrics of user engagement, such as number of views or VPI. While this suggests that academic medical institutions can produce high AV quality videos, the patients cannot appreciate such quality if it is not viewed. These findings reiterate the importance of physicians engaging in strategies to maximize their position on YouTube's SERP.

This study has potential limitations. Our small sample size may fail to represent the true extent of videos available to patients on sialendoscopy. However, as discussed previously, such a sample size is a finding itself that highlights the scarcity of videos available to patients on the topic. Furthermore, although we excluded videos that were not suitable for patients and found no difference in the DISCERN bias score between uploaders, we did not categorize videos by their intended purpose. Thus, it is possible that some of the remaining videos were not intended directly for patient education, potentially creating bias in certain discussion points based on the uploader's intentions. Future investigations should seek to evaluate video bias as it relates to the video's intended purpose (e.g., educational, testimonial, or marketing). Additionally, while this study provides valuable insight into the quality and comprehensiveness of sialendoscopy YouTube videos, it does not evaluate videos uploaded outside of YouTube and it does not reveal the extent to which these videos may truly educate patients. Finally, although the modified DISCERN has been extensively used to evaluate online patient education, no validation studies specific to the modified instrument have been published. Further studies may seek to validate the modified DISCERN score against other previously validated reliability grading systems.

5. CONCLUSION

Our study highlights the overall paucity and low quality of videos for patients on sialendoscopy. More popular videos are not necessarily higher quality or more comprehensive, and videos uploaded on YouTube are targeted more toward physicians rather than patients. As YouTube becomes an increasingly used source of health information, there is opportunity for otolaryngologists to produce more informative videos for patients while implementing targeted strategies to increase viewership.

FUNDING INFORMATION

This research did not receive any specific grant from funding agencies in the public, commercial, or not‐for‐profit sectors.

CONFLICT OF INTEREST

The authors have no funding, financial relationships, conflicts of interest, or competing interests to disclose.

Campbell DJ, Xiao K, Mastrolonardo E, Cognetti DM. Evaluating YouTube videos on sialendoscopy as an educational resource for patients. Laryngoscope Investigative Otolaryngology. 2023;8(1):82‐88. doi: 10.1002/lio2.991