Abstract
Extended Summary
Introduction:
Parents who make decisions about hypospadias repair for their child may seek information from online platforms such as YouTube.
Objective:
The purpose of this study was to evaluate the health literacy demand of hypospadias videos on YouTube using the Patient Education Materials Assessment Tool for Audiovisual Materials (PEMAT-A/V).
Study Design:
We performed a YouTube search using the term “hypospadias,” limiting results to the first 100 videos. We excluded videos that were < 1 minute or > 20 minutes and videos that were not in English or did not include subtitles. Two evaluators independently examined videos and determined PEMAT-A/V scores for understandability and actionability (i.e. ability to identify actions the viewer can take). Videos with scores >70% are understandable or actionable. The inter-rater reliability (kappa) and intra-class correlation coefficient (ICC) of PEMAT scores were calculated. Bivariate and multivariable linear regression models assessed the association of video characteristics with respective scores.
Results:
Of the 100 videos that were identified on YouTube, 47(47%) were excluded leaving 53 for analysis: 14 were >20 minutes, 14 were <1 minute, 9 had no audio or subtitles, 7 were not in English, 1 was a duplicate, 1 was unrelated to hypospadias and 1 was deleted at the time of data analysis. Three (5.6%) were understandable (mean score 54.5%, standard deviation (SD) 14.9) and eight (15.1%) were actionable (mean score 21.8%, SD 16.6) (Extended Summary Figure). Kappa values ranged from 0.4 to 1. The ICC’s were 0.55 and 0.33 for understandability and actionability respectively. In the bivariate analysis, mean understandability scores were significantly higher for English language videos (p=0.04), videos with animation (p=0.002) and those produced by industry (p=0.02). In the multivariable analysis, mean understandability scores were significantly higher for “expert testimonial” or “other” video types after adjusting for graphics type and overall tone (p=0.04). Mean understandability scores were also significantly higher for videos with animation after adjusting for video type and overall tone (p=0.01). Mean actionability scores were significantly higher for videos with a negative tone (p=0.01).
Discussion:
The vast majority of hypospadias-related YouTube content is not appropriate for users with low health literacy although certain types of videos, such those with animation and expert testimonials, scored higher on understandability than other types.
Conclusion:
Due to the lack of sufficient online informational content regarding hypospadias we plan to engage parents of sons with hypospadias in the development of high quality patient educational materials about hypospadias.
Keywords: Decision making, pediatrics, hypospadias, qualitative research
Introduction
Hypospadias is the most common congenital anomaly of the male genitalia with an incidence of 1:250. [1] Urologists typically offer surgical repair in infancy with over 400 different types of surgical repairs described in the literature. [2] The quality of information available online about hypospadias is variable in terms of accuracy and reliability.[3, 4] Routh et al evaluated 100 websites on five common and uncommon pediatric urology topics. .[4] The authors concluded that, compared with uncommon topics, websites devoted to common topics such as hypospadias were more likely to contain accurate information about disease treatments. On average, a minimum of a 12th-grade reading level was required to understand the content.[4] In their evaluation of 46 hypospadias websites, Cisu et al noted that most were of “adequate quality” and those from institutions had significantly more accurate information compared to commercial websites.[3] They also noted that the average reading level of the websites was 11th-12th grade which is far beyond the reading level of most adults.[3]
In addition to traditional websites, video-sharing sites such as YouTube may serve as educational tools to assist in healthcare decision-making for conditions such as hypospadias. YouTube has gained popularity as an online source for medical information and as a social networking platform for sharing health information among viewers.[5] Healthcare providers and government agencies have recently expressed concerns, however, about the reliability and quality of the information on this platform given the anecdotal nature of much of the information and the limited guidelines regulating its content.[6] To our knowledge, \no prior studies have evaluated information about hypospadias on YouTube. The objective of this study was to assess the quality and reliability of videos about hypospadias on YouTube using a validated instrument for patient educational materials.
Materials and Methods
We queried YouTube in April 2018 using the search term “hypospadias” and limited our search to the first 100 videos. Studies of user behavior on Internet search engines indicate that over 90% of users click on results within the first three pages of search results.[7] Therefore, we limited our search to the first 100 videos in order to mimic the search strategy of a hypothetical parent of a child with hypospadias. We excluded videos that were not about hypospadias as well as those that were less than 1 minute or greater than 20 minutes long. We selected a 20-minute cut-off based on a prior study of participants in a video-based lecture series demonstrating 50% audience retention at 14 minutes.[8] We also excluded videos that were not in English or did not include subtitles due to limited availability of translation services for our study. Two evaluators independently reviewed video characteristics including language, video type (i.e. surgical procedure, parent testimonial, expert testimonial, or educational), intended audience, overall tone, presence of advertising, country of origin and “thumbs up/thumbs down” ratings. The “thumbs up and thumbs down ratings” are indicators of the popularity of a particular YouTube video amongst viewers. We did not record the number of views a video had at the time of the evaluation. The two evaluators who reviewed the YouTube videos were graduate students in the School of Informatics at Indiana University-Purdue University Indianapolis. In order to learn how to use the PEMAT tool, the evaluators read the PEMAT user guide.[9] Two of the authors (KC and JP), conducted a formal training session with the evaluators during which we reviewed the PEMAT user guide, assessed five sample videos as a group and discussed our ratings.
We used a validated instrument called The Patient Education Materials Assessment Tool (PEMAT) to assess the quality of the videos. [10]. The tool has moderate to excellent overall inter-rater reliability.[11] There are two versions of the PEMAT, one for print materials (PEMAT-P) and one for audiovisual materials (PEMAT-A/V). PEMAT-A/V deems patient education materials as “understandable” when consumers of diverse backgrounds and varying levels of health literacy can process and explain key messages. The tool deems videos “actionable” when consumers can identify what they can do based on the information presented.[10] The understandability domain of PEMAT-A/V consists of 13 binary questions (agree/disagree) pertaining to content, word choice and style, organization, layout and design, and use of visual aids. The user calculates the score for understandability as a percentage of total possible points (total points/total possible points x 100) with scores ranging from 0–100%. The actionability domain consists of four criteria: the video must address the viewer directly, clearly identify at least one action they can take, instruct them about how to take the action, and provide a tangible tool to facilitate the action. A video with a PEMAT score of 70% or below is poorly understandable or actionable.[10]
Statistical analysis
PEMAT scores of the two raters were averaged, and the inter-rater reliability of the PEMAT-A/V tool was calculated. For item scores, frequencies of combinations of ratings, percent of perfect agreement and kappa were used to measure the strength of agreement between raters. The strength of agreement was classified using Landis and Koch’s suggested benchmarks for kappa statistic with values of 0.21–0.40 indicating fair agreement and values of 0.41–0.60 suggesting moderate agreement.[12] For continuous measures, (i.e. the total scores for understandability and actionability respectively) summary statistics and the intra class correlation coefficient (ICC) were used to assess agreement between raters. The ICC was calculated as the variation between videos divided by the total variation. We calculated the within video variation (1-ICC), which is a measure of how much the raters varied in their scores. “We also calculated “best and worst case scenarios” based on understandability and actionability. We defined a “best case scenario” for understandability as the percentage of videos that either reviewer thought were understandable and the “best case scenario” for actionability as the percentage of videos that either reviewer thought were actionable. We defined a “worst case scenario” for understandability as the percentage of videos that both reviewers thought were not understandable and the “worst case scenario” for acitionability as the percentage of videos that both reviewers thought were not actionable.”
Two separate analyses were conducted for PEMAT scores, one with scores as binary (>70%) and one with scores as continuous. Videos were classified as understandable and acceptable respectively if the mean scores on each subscale were greater than 0.70. Video characteristics were summarized by understandability and acceptability.
Bivariate analyses:
Bivariate linear regression models were used to assess the association of video characteristics with understandability and actionability scores respectively.
Multivariable analysis:
Characteristics that were significant at the 0.25 level on bivariate analysis were eligible for inclusion in a multivariable linear regression model. Backwards variable selection was used until all remaining effects were significant at the 0.1 level. Residuals plots were examined to assess model assumptions of normality and homogeneity of error terms. Statistics were computed using the SAS/STAT ® Software version 9.4 (copyright 2002–2012 by SAS Institute, Cary NC).
Results
Of the 100 videos that were identified on YouTube 47(47%) were excluded leaving 53 for analysis: 14 were >20 minutes, 14 were <1 minute, 9 had no audio or subtitles, 7 were not in English, 1 was a duplicate, 1 was unrelated to hypospadias and 1 was deleted at the time of data analysis.
Inter-rater Reliability of PEMAT Scores
Table 3 shows the kappa scores of the seventeen PEMAT items, in the moderate to strong range, illustrating the strength of agreement between the two raters about each of the item scores (Table 3). The ICC for understandability was 0.55. This means that 45% of total variation was from differences in rater scores of the same video. The ICC for actionability was 0.33, implying that 67% of total variation was from difference in rater scores of the same video. The “best case scenarios” were as follows: 7 (13.2%) of videos had an understandability score greater than 70% from either reviewer and 16 (30.2%) videos had an actionability score greater than 70% from either reviewer. The “worst case scenarios” were as follows: 49 (87.0%) of videos had an understandability score of 70% or less from both reviewers and 42 (65.6%) of videos had an actionability score of 70% or less from both reviewers.
Table 3:
Inter-rater reliability of PEMAT scores: item agreement
| Item | Rater 1 | Rater 2 | n (%) | Perfect Agreement | Kappa (Standard Error) | 95% Confidence Interval |
|---|---|---|---|---|---|---|
| 1 | 0 | 0 | 9(17) | 47(88.7%) | 0.73(0.10) | (0.52,0.93) |
| 0 | 1 | 1(1.9) | ||||
| 1 | 0 | 5(9.4) | ||||
| 1 | 1 | 38(71.7) | ||||
| 3 | 0 | 0 | 43(81.1) | 53(100.0%) | 1.00(0.00) | (1.00,1.00) |
| 1 | 1 | 10(18.9) | ||||
| 4 | 0 | 0 | 24(45.3) | 47(88.7%) | 0.80(0.08) | (0.65,0.95) |
| 0 | 1 | 6(11.3) | ||||
| 1 | 1 | 23(43.4) | ||||
| 5 | 0 | 0 | 53(100) | 53(100.0%) | 1.00(0.00) | (1.00,1.00) |
| 8 | 0 | 0 | 30(56.6) | 43(81.1%) | 0.64(0.10) | (0.44,0.84) |
| 0 | 1 | 8(15.1) | ||||
| 1 | 0 | 2(3.8) | ||||
| 1 | 1 | 13(24.5) | ||||
| 9 | 0 | 0 | 23(43.4) | 40(75.5%) | 0.57(0.10) | (0.37,0.77) |
| 0 | 1 | 11(20.8) | ||||
| 1 | 0 | 2(3.8) | ||||
| 1 | 1 | 17(32.1) | ||||
| 10 | 0 | 0 | 6(11.3) | 36(67.9%) | 0.36(0.11) | (0.14,0.57) |
| 0 | 1 | 16(30.2) | ||||
| 1 | 0 | 1(1.9) | ||||
| 1 | 1 | 30(56.6) | ||||
| 11 | 0 | 0 | 41(77.4) | 45(84.9%) | 0.57(0.14) | (0.29,0.84) |
| 0 | 1 | 3(5.7) | ||||
| 1 | 0 | 5(9.4) | ||||
| 1 | 1 | 4(7.5) | ||||
| 12 | NA | NA | 53(100) | 53(100.0%) | 1.00(0.00) | (1.00,1.00) |
| 13 | 0 | 0 | 1(1.9) | 39(73.6%) | 0.48(0.11) | (0.26,0.71) |
| 1 | 0 | 6(11.3) | ||||
| 1 | 1 | 30(56.6) | ||||
| 1 | NA | 2(3.8) | ||||
| NA | 1 | 6(11.3) | ||||
| NA | NA | 8(15.1) | ||||
| 14 | 0 | 0 | 4(7.5) | 42(79.2%) | 0.65(0.09) | (0.47,0.83) |
| 1 | 0 | 6(11.3) | ||||
| 1 | 1 | 28(52.8) | ||||
| 1 | NA | 3(5.7) | ||||
| NA | 0 | 1(1.9) | ||||
| NA | 1 | 1(1.9) | ||||
| NA | NA | 10(18.9) | ||||
| 18 | 0 | 0 | 5(9.4) | 44(83.0%) | 0.75(0.07) | (0.60,0.89) |
| 0 | 1 | 4(7.5) | ||||
| 1 | 0 | 3(5.7) | ||||
| 1 | 1 | 17(32.1) | ||||
| NA | 0 | 1(1.9) | ||||
| NA | 1 | 1(1.9) | ||||
| NA | NA | 22(41.5) | ||||
| 19 | NA | 1 | 1(1.9) | 52(98.1%) | 0.79(0.20) | (0.40,1.00) |
| NA | NA | 52(98.1) | ||||
| 20 | 0 | 0 | 23(43.4) | 35(66.0%) | 0.39(0.12) | (0.15,0.63) |
| 0 | 1 | 6(11.3) | ||||
| 1 | 0 | 12(22.6) | ||||
| 1 | 1 | 12(22.6) | ||||
| 21 | 0 | 0 | 15(28.3) | 32(60.4%) | 0.35(0.10) | (0.15,0.55) |
| 0 | 1 | 2(3.8) | ||||
| 1 | 0 | 19(35.8) | ||||
| 1 | 1 | 17(32.1) | ||||
| 22 | 0 | 0 | 27(50.9) | 40(75.5%) | 0.55(0.11) | (0.35,0.76) |
| 0 | 1 | 11(20.8) | ||||
| 1 | 0 | 2(3.8) | ||||
| 1 | 1 | 13(24.5) | ||||
| 25 | NA | NA | 53(100) | 53(100.0%) | 1.00(0.00) | (1.00,1.00) |
Descriptive statistics
Characteristics of the 53 included videos are summarized by understandability and acceptability in Tables 1 and 2 respectively. Only three videos (5.6%) were “understandable” with a PEMAT-A/V understandability score >70%: mean score 54.5%, standard deviation (SD) 14.9 (Extended Summary Figure). Only eight videos (15.1%) were actionable with a PEMAT-A/V actionability score >70%: mean score 21.8%, SD 16.6 (Extended Summary Figure).
Table 1:
PEMAT understandability scores summarized by video characteristics
| Video Characteristic | Overall | Understandable | Not Understandable | |
|---|---|---|---|---|
| n (%) | n (%) | n (%) | ||
| Video Length (minutes) | Mean ± SD | 4.4 ± 2.9 | 4.7 ± 2.3 | 4.4 ± 2.9 |
| Median (Min, Max) | 4.0 (1.0, 13.0) | 6.0 (2.0, 6.0) | 4.0 (1.0, 13.0) | |
| English Language | No | 3 (5.7) | 0 (0.0) | 3 (6.0) |
| Yes | 50 (94.3) | 3 (100.0) | 47 (94.0) | |
| Has Subtitles | No | 36 (67.9) | 2 (66.7) | 34 (68.0) |
| Yes | 17 (32.1) | 1 (33.3) | 16 (32.0) | |
| Has Audio | No | 13 (24.5) | 0 (0.0) | 13 (26.0) |
| Yes | 40 (75.5) | 3 (100.0) | 37 (74.0) | |
| Country of Origin | US | 15 (28.3) | 2 (66.7) | 13 (26.0) |
| Outside of US | 27 (50.9) | 1 (33.3) | 26 (52.0) | |
| Unknown | 11 (20.8) | 0 (0.0) | 11 (22.0) | |
| Type of Publisher | Hospital/Practice | 19 (35.8) | 0 (0.0) | 19 (38.0) |
| Industry | 6 (11.3) | 1 (33.3) | 5 (10.0) | |
| Non-Profit | 2 (3.8) | 0 (0.0) | 2 (4.0) | |
| Individual | 17 (32.1) | 0 (0.0) | 17 (34.0) | |
| Other/Unknown | 9 (17.0) | 2 (66.7) | 7 (14.0) | |
| Number of “Thumbs Up” Ratings | Mean ± SD | 45.4 ± 98.9 | 213.3 ± 307.5 | 35.3 ± 68.5 |
| Median (Min, Max) | 15.0 (0.0, 566.0) | 73.0 (1.0, 566.0) | 15.0 (0.0, 320.0) | |
| Number of “Thumbs Down” Ratings | Mean ± SD | 9.6 ± 24.4 | 15.3 ± 13.3 | 9.3 ± 24.9 |
| Median (Min, Max) | 1.0 (0.0, 149.0) | 23.0 (0.0, 23.0) | 1.0 (0.0, 149.0) | |
| Percent of “Thumbs Up” Ratings | Mean ± SD | 77.9 ± 28.8 | 90.7 ± 12.9 | 77.1 ± 29.4 |
| Median (Min, Max) | 86.8 (0.0, 100.0) | 96.1 (76.0, 100.0) | 86.0 (0.0, 100.0) | |
| Video Type | Surgical Procedure | 16 (30.2) | 0 (0.0) | 16 (32.0) |
| Parent Testimonial | 9 (17.0) | 0 (0.0) | 9 (18.0) | |
| Expert Testimonial | 13 (24.5) | 1 (33.3) | 12 (24.0) | |
| Educational | 8 (15.1) | 1 (33.3) | 7 (14.0) | |
| Other | 7 (13.2) | 1 (33.3) | 6 (12.0) | |
| Graphics Type | Live Action | 31 (58.5) | 1 (33.3) | 30 (60.0) |
| Animation | 3 (5.7) | 1 (33.3) | 2 (4.0) | |
| Photographs | 4 (7.5) | 1 (33.3) | 3 (6.0) | |
| Illustrations/Graphics | 8 (15.1) | 0 (0.0) | 8 (16.0) | |
| Other/Unknown | 7 (13.2) | 0 (0.0) | 7 (14.0) | |
| Audience Type | Parents | 22 (41.5) | 0 (0.0) | 22 (44.0) |
| Patients | 1 (1.9) | 0 (0.0) | 1 (2.0) | |
| Physicians/Nurses | 9 (17.0) | 1 (33.3) | 8 (16.0) | |
| Students | 2 (3.8) | 1 (33.3) | 1 (2.0) | |
| Other/Unknown | 19 (35.8) | 1 (33.3) | 18 (36.0) | |
| Overall Tone | Positive | 20 (37.7) | 0 (0.0) | 20 (40.0) |
| Negative | 4 (7.5) | 1 (33.3) | 3 (6.0) | |
| Neutral | 29 (54.7) | 2 (66.7) | 27 (54.0) | |
| Advertising | No | 39 (73.6) | 2 (66.7) | 37 (74.0) |
| Yes | 14 (26.4) | 1 (33.3) | 13 (26.0) | |
Table 2:
PEMAT actionability scores summarized by video characteristics
| Video Characteristic | Overall n=53 | Actionable (n=8) | Not Actionable (n=45) | |
|---|---|---|---|---|
| n (%) | n (%) | n (%) | ||
| Video Length (minutes) | Mean ± SD | 4.4 ± 2.9 | 4.9 ± 3.9 | 4.4 ± 2.7 |
| Median (Min, Max) | 4.0 (1.0, 13.0) | 4.0 (1.0, 13.0) | 4.0 (1.0, 12.0) | |
| English Language | No | 3 (5.7) | 0 (0.0) | 3 (6.7) |
| Yes | 50 (94.3) | 8 (100.0) | 42 (93.3) | |
| Has Subtitles | No | 36 (67.9) | 5 (62.5) | 31 (68.9) |
| Yes | 17 (32.1) | 3 (37.5) | 14 (31.1) | |
| Has Audio | No | 13 (24.5) | 1 (12.5) | 12 (26.7) |
| Yes | 40 (75.5) | 7 (87.5) | 33 (73.3) | |
| Country of Origin | US | 15 (28.3) | 3 (37.5) | 12 (26.7) |
| Outside of US | 27 (50.9) | 5 (62.5) | 22 (48.9) | |
| Unknown | 11 (20.8) | 0 (0.0) | 11 (24.4) | |
| Type of Publisher | Hospital/Practice | 19 (35.8) | 2 (25.0) | 17 (37.8) |
| Industry | 6 (11.3) | 3 (37.5) | 3 (6.7) | |
| Non-Profit | 2 (3.8) | 0 (0.0) | 2 (4.4) | |
| Individual | 17 (32.1) | 1 (12.5) | 16 (35.6) | |
| Other/Unknown | 9 (17.0) | 2 (25.0) | 7 (15.6) | |
| Number of Thumbs Up Ratings |
Mean ± SD | 45.4 ± 98.9 | 19.4 ± 29.2 | 50.0 ± 106.2 |
| Number of Thumbs Up Ratings |
Median (Min, Max) | 15.0 (0.0, 566.0) | 3.5 (0.0, 73.0) | 18.0 (0.0, 566.0) |
| Number of Thumbs Down Ratings |
Mean ± SD | 9.6 ± 24.4 | 5.4 ± 7.9 | 10.4 ± 26.2 |
| Number of Thumbs Down Ratings |
Median (Min, Max) | 1.0 (0.0, 149.0) | 2.5 (0.0, 23.0) | 1.0 (0.0, 149.0) |
| Percent of Thumbs Up Ratings |
Mean ± SD | 77.9 ± 28.8 | 71.7 ± 35.0 | 79.0 ± 27.9 |
| Median (Min, Max) | 86.8 (0.0, 100.0) | 81.4 (0.0, 100.0) |
87.1 (0.0, 100.0) | |
| Video Type | Surgical Procedure | 16 (30.2) | 1 (12.5) | 15 (33.3) |
| Parent Testimonial | 9 (17.0) | 2 (25.0) | 7 (15.6) | |
| Expert Testimonial | 13 (24.5) | 2 (25.0) | 11 (24.4) | |
| Educational | 8 (15.1) | 1 (12.5) | 7 (15.6) | |
| Other | 7 (13.2) | 2 (25.0) | 5 (11.1) | |
| Graphics Type | Live Action | 31 (58.5) | 5 (62.5) | 26 (57.8) |
| Animation | 3 (5.7) | 0 (0.0) | 3 (6.7) | |
| Photographs | 4 (7.5) | 2 (25.0) | 2 (4.4) | |
| Illustrations/Graphics | 8 (15.1) | 1 (12.5) | 7 (15.6) | |
| Other/Unknown | 7 (13.2) | 0 (0.0) | 7 (15.6) | |
| Audience Type | Parents | 22 (41.5) | 5 (62.5) | 17 (37.8) |
| Patients | 1 (1.9) | 0 (0.0) | 1 (2.2) | |
| Physicians/Nurses | 9 (17.0) | 1 (12.5) | 8 (17.8) | |
| Students | 2 (3.8) | 0 (0.0) | 2 (4.4) | |
| Other/Unknown | 19 (35.8) | 2 (25.0) | 17 (37.8) | |
| Overall Tone | Positive | 20 (37.7) | 1 (12.5) | 19 (42.2) |
| Negative | 4 (7.5) | 2 (25.0) | 2 (4.4) | |
| Neutral | 29 (54.7) | 5 (62.5) | 24 (53.3) | |
| Advertising | No | 39 (73.6) | 3 (37.5) | 36 (80.0) |
| Yes | 14 (26.4) | 5 (62.5) | 9 (20.0) | |
Extended Summary Figure:
Distribution of Actionability and Understandability Scores
Bivariate analyses
In a bivariate linear regression model examining the association between video characteristics and continuous PEMAT understandability scores, we noted that mean understandability was significantly higher for English language videos (versus non-English; p=0.04), videos with animation (versus live action; p=0.002) and those produced by industry versus hospitals or practices (p=0.02). In another bivariate linear regression model examining the association between video characteristics and continuous PEMAT actionability scores, we noted that mean actionability was significantly higher for videos with a negative tone (p=0.01).
Multivariable analyses
We examined the association between video characteristics and continuous PEMAT understandability scores adjusting for the effects of other variables in a multivariable model (Table 4). This model shows that, adjusted for graphics type and overall tone, mean understandability scores are significantly higher for “expert testimonial” or “other “video types than for surgical procedure videos. This model also shows that, adjusted for video type and overall tone, mean understandability scores are significantly higher for videos with animation than live action videos. Adjusted for video type and graphics type, overall tone has a marginally significant association with understanding scores. Videos with a positive overall tone have on average lower understandability scores than do videos with a neutral tone.
Table 4:
Multivariable linear regression model examining the association between video characteristics and understandability score
| Effect | Estimate (Std Error) | Overall Test | p-value | |
|---|---|---|---|---|
| Intercept | 41.48(4.57) | . | <.0001 | |
| Video Type | Educational | 0.13(6.84) | 0.04 | 0.9854 |
| Expert Testimonial | 15.18(5.63) | 0.01 | ||
| Other | 16.56(7.80) | 0.04 | ||
| Parent Testimonial | 12.02(6.14) | 0.06 | ||
| Surgical Procedure | 0.00 | . | ||
| Graphics Type | Animation | 33.43(9.97) | 0.01 | 0.002 |
| Illustrations/Graphics | −2.37(7.16) | 0.7 | ||
| Other/Unknown | –1.81(6.34) | 0.8 | ||
| Photographs | –3.81(8.37) | 0.7 | ||
| Live Action | 0.00 | . | ||
| Overall Tone | Negative | 6.66(8.93) | 0.06 | 0.5 |
| Positive | –9.67(5.07) | 0.06 | ||
| Neutral | 0.00 | . | ||
Discussion
We found that the majority of hypospadias-related video content on YouTube is neither understandable nor actionable based on evaluation with a validated rating tool called PEMAT. We also noted that videos with animation and expert testimonials were easier to understand than other types of videos. Prior authors have expressed similar concerns about the quality of information available online regarding common pediatric urology conditions.[3, 4, 13] Cisu et al noted that only 23.9% (1¼6) of the websites they evaluated had a Health on the Net (HoN)-certified seal, indicating quality as determined by the Health on the Net Foundation.[3] There is even less regulatory oversight on YouTube given the lack of any certification process for the quality of content. Cisu et al also classified websites equivalently to our study using the following categories: institutional/ reference, commercial, non-profit/charitable and patient support groups/blogs. Fast et al used a validated rating tool called DISCERN to evaluate the reliability and quality of information on 60 websites addressing circumcision, vesicoureteral reflux and posterior urethral valves.[13] The average DISCERN score ranged from 40 to 60 (out of a maximum of 80) for each of the three topics respectively. [3] The two areas that received the lowest scores were education on quality of life issues and risks for treatment. [3] These studies highlight the poor quality of information that is available to parents who are making decisions about treatment choices for common pediatric urology conditions.
Unlike prior studies in pediatric urology, we limited our evaluation to YouTube video content about hypospadias since it is likely resource for this generation of parents of young children.[14] Social media has increased the potential for communication of health information and in the past decade with an estimated 3.02 billion users projected for 2021.[15, 16] A recent study suggested that patients are using social media as a health information platform for four primary reasons: 1) to obtain information about a newly diagnosed disease, 2) to communicate and receive advice from healthcare providers, 3) to gain knowledge about other patient’s experiences and 4) to receive social support.[17] Although there has been a steady increase in the number of internet users there are disparities in internet usage based on sociodemographic status. [18] Prestin et al found that young adults with higher education, non-Hispanic White race/ethnicity were more likely to use the Internet based on data from the Health Information National Trends Survey (HINTS), a nationally representative survey of internet access of US adults.[18] Huo et al used the HINTS to determine an individual’s use of social media for the purposes of sharing health information or exchanging medical information with a healthcare professional.[15] They noted that young adults were more likely than older adults to use social media as a platform for health communication but they found no racial or ethnic disparities in internet usage.[15] One study showed that 72% of patients access the internet to obtain information about healthcare conditions online.[19] Current evidence suggests that patients seeking urological care rely on social media for information on adult and pediatric urology conditions.[20, 21] Sood et al found that 58.3% YouTube videos provide useful information on kidney stones and 47.2% of these videos had total viewership.[20] A similar study by Nason et al reported that 56.7% of parents whose children were diagnosed with hydroceles accessed the internet for more information on this condition.[21] The increasing patient awareness of social media as a health information resource warrants adequate and comprehensive online education materials.[22]
Healthcare providers and government agencies have expressed concern about quality and veracity of the information available on this platform in recent years.[23–25] The minimal guidelines regulating YouTube content raise questions about the trustworthiness of this source and the risk of disseminating misleading information. A recent systematic review identified three major concerns when consumers use information from YouTube to make health care decisions: 1) YouTube is a medium for promoting unscientific therapies; 2) YouTube contains information contradicting reference standards/guidelines; and (3) YouTube has the potential to change the beliefs of patients about controversial topics such as vaccinations. [6]
To our knowledge, this is the first study to evaluate pediatric urology-focused content on YouTube. One of the strengths of this study is the use of a validated instrument that provides a comprehensive, quantitative assessment of a subset of online patient education materials about a common pediatric urology condition. We chose the PEMAT for this study because it addresses the key components of health literacy, defined as the capacity to “obtain, process and understand basic health information and services needed to make health decisions.”[26] Low health literacy is strongly associated with poorer use of healthcare and subsequent health outcomes, leading to higher utilization of emergency departments and inpatient beds.[27–29] The U.S. Department of Health and Human Services’ National Action Plan to Promote Health Literacy advocates for health information that is “accurate, accessible and actionable.”[26]
The primary limitation of our study is the relatively low ICCs for content and actionability, measuring the consistency of the quantitative measurements of the two evaluators. This may be due to the subjective nature of some of the questions and individual differences in the evaluators (i.e. bias). With additional training, we might have been able to reduce or eliminate these differences but time and resource limitations did not allow for re-evaluation of the data. Nonetheless, several previous studies using a limited number of evaluators have reported similar results regarding inter-rater reliability.[11, 30] In addition the evaluators pursuing graduate-level degrees may have had a higher level of competency and understanding in the sciences than the average YouTube viewer. Their strong science background may have increased the likelihood that videos with a high reading level were also rated highly on the understandability scale.
Finally, we did not evaluate the scientific accuracy of the hypospadias-related content on YouTube because we felt that this type of evaluation was beyond the scope of this study and we did not have an objective, validated measure to conduct such an evaluation. Our goal was to analyze the YouTube experience from the parents’ perspective, focusing on the health literacy demands of the materials.
In conclusion, our study is the first objective assessment of the hypospadias-related content on YouTube using a validated instrument. We found that the majority of hypospadias-related video content on YouTube not appropriate for users with low health literacy although certain types of videos such those with animation and expert testimonials were easier to understand than others. In light of these findings, we would advocate for more regulatory oversight of content on social media sites such as YouTube that are popular resources for health information. This presents an opportunity for a non-profit foundation such as Health on the Net (https://www.hon.ch/en/) to extend its reach beyond traditional websites into social media platforms in order to provide a certification of quality. In future studies, we plan to engage parents of sons with hypospadias in both the evaluation of existing online content and the development of high quality, reliable educational materials about hypospadias.
Acknowledgements
The authors wish to acknowledge the following individuals who participated in data collection for this project as part of a graduate-level class project in the School of Informatics at Indiana University-Purdue University Indianapolis: Mounika Guntu, Krishna Sai Vasireddy, Melisa Cinbat, Jagdish Rayankula and Quenin Taylor Gill
Funding source
K23 Career Development Award (1K23DK111987–01) from the National Institutes of Health-National Institute of Diabetes and Digestive and Kidney Diseases.
Footnotes
Conflict of interest
The authors have no conflicts of interest to disclose.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
References
- [1].Springer A, van den Heijkant M, Baumann S. Worldwide prevalence of hypospadias. J Pediatr Urol. 2016;12:152 e1–7. [DOI] [PubMed] [Google Scholar]
- [2].Lambert SM, Snyder HM, 3rd, Canning DA. The history of hypospadias and hypospadias repairs. Urology. 2011;77:1277–83. [DOI] [PubMed] [Google Scholar]
- [3].Cisu TI, Mingin GC, Baskin LS. An evaluation of the readability, quality, and accuracy of online health information regarding the treatment of hypospadias. Journal of Pediatric Urology. 2018. [DOI] [PubMed] [Google Scholar]
- [4].Routh JC, Gong EM, Nelson CP. Pediatric urology and the internet--does an uncommon topic decrease content quality? J Urol. 2009;182:1569–74. [DOI] [PubMed] [Google Scholar]
- [5].Singh AG, Singh S, Singh PP. YouTube for information on rheumatoid arthritis--a wakeup call? J Rheumatol. 2012;39:899–903. [DOI] [PubMed] [Google Scholar]
- [6].Madathil KC, Rivera-Rodriguez AJ, Greenstein JS, Gramopadhye AK. Healthcare information on YouTube: A systematic review. Health Informatics J. 2015;21:173–94. [DOI] [PubMed] [Google Scholar]
- [7].iProspect Search Engine User Behavior Study. 2006.
- [8].Lau KHV, Farooque P, Leydon G, Schwartz ML, Sadler RM, Moeller JJ. Using learning analytics to evaluate a video-based lecture series. Med Teach. 2018;40:91–8. [DOI] [PubMed] [Google Scholar]
- [9].Shoemaker SJ WM, Brach C. Patient Education Materials Assessment Tool for Audiovisual Materials (PEMAT-A/V). Rockville, MD 2013. [Google Scholar]
- [10].Shoemaker SJ, Wolf MS, Brach C. Development of the Patient Education Materials Assessment Tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information. Patient Educ Couns. 2014;96:395–403. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [11].Vishnevetsky J, Walters CB, Tan KS. Interrater reliability of the Patient Education Materials Assessment Tool (PEMAT). Patient Educ Couns. 2018;101:490–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [12].Landis JR KG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–74. [PubMed] [Google Scholar]
- [13].Fast AM, Deibert CM, Hruby GW, Glassberg KI. Evaluating the quality of Internet health resources in pediatric urology. J Pediatr Urol. 2013;9:151–6. [DOI] [PubMed] [Google Scholar]
- [14].YouTube useage reach in the United States 2018, by age group. Statista.
- [15].Huo J, Desai R, Hong YR, Turner K, Mainous AG, 3rd, Bian J. Use of Social Media in Health Communication: Findings From the Health Information National Trends Survey 2013, 2014, and 2017. Cancer Control. 2019;26:1073274819841442. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [16].Clement J Most famous social network sites worldwide as of July 2019, ranked by number of active users (in millions). Statista; 2019. [Google Scholar]
- [17].Antheunis ML, Tates K, Nieboer TE. Patients’ and health professionals’ use of social media in health care: motives, barriers and expectations. Patient Educ Couns. 2013;92:426–31. [DOI] [PubMed] [Google Scholar]
- [18].Prestin A, Vieux SN, Chou WY. Is Online Health Activity Alive and Well or Flatlining? Findings From 10 Years of the Health Information National Trends Survey. J Health Commun. 2015;20:790–8. [DOI] [PubMed] [Google Scholar]
- [19].Fox SD M. Health online—2013. Washington, DC: Pew Research Center; 2013. [Google Scholar]
- [20].Sood A, Sarangi S, Pandey A, Murugiah K. YouTube as a source of information on kidney stone disease. Urology. 2011;77:558–62. [DOI] [PubMed] [Google Scholar]
- [21].Nason GJ, Tareen F, Quinn F. Hydrocele on the web: an evaluation of Internet-based information. Scand J Urol. 2013;47:152–7. [DOI] [PubMed] [Google Scholar]
- [22].O’Kelly F, Nason GJ, Manecksha RP, Cascio S, Quinn FJ, Leonard M, et al. The effect of social media (#SoMe) on journal impact factor and parental awareness in paediatric urology. J Pediatr Urol. 2017;13:513 e1- e7. [DOI] [PubMed] [Google Scholar]
- [23].Lewis SP, Heath NL, Sornberger MJ, Arbuthnott AE. Helpful or harmful? An examination of viewers’ responses to nonsuicidal self-injury videos on YouTube. J Adolesc Health. 2012;51:380–5. [DOI] [PubMed] [Google Scholar]
- [24].Briones R, Nan X, Madden K, Waks L. When vaccines go viral: an analysis of HPV vaccine coverage on YouTube. Health Commun. 2012;27:478–85. [DOI] [PubMed] [Google Scholar]
- [25].Keelan J, Pavri-Garcia V, Tomlinson G, Wilson K. YouTube as a source of information on immunization: a content analysis. JAMA. 2007;298:2482–4. [DOI] [PubMed] [Google Scholar]
- [26].National action plan to improve health literacy. Washington, DC 2010.
- [27].Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med. 2011;155:97–107. [DOI] [PubMed] [Google Scholar]
- [28].Dewalt DA, Berkman ND, Sheridan S, Lohr KN, Pignone MP. Literacy and health outcomes: a systematic review of the literature. J Gen Intern Med. 2004;19:1228–39. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [29].Keller DL, Wright J, Pace HA. Impact of health literacy on health outcomes in ambulatory care patients: a systematic review. Ann Pharmacother. 2008;42:1272–81. [DOI] [PubMed] [Google Scholar]
- [30].Harris VC, Links AR, Hong P, Walsh J, Schoo DP, Tunkel DE, et al. Consulting Dr. Google: Quality of Online Resources About Tympanostomy Tube Placement. Laryngoscope. 2018;128:496–501. [DOI] [PMC free article] [PubMed] [Google Scholar]