Abstract
Introduction
Many patients seek information online including on social media.
Aim
To assess the quality of information regarding erectile dysfunction (ED) in YouTube videos.
Methods
We searched “erectile dysfunction” on YouTube in October 2019 and evaluated the first 100 videos in English sorted by relevance.
Main Outcome Measure
We recorded the user engagement, video producer, intended audience, and content. Videos containing medical information were evaluated using the Patient Education Materials Assessment Tool (PEMAT) and the DISCERN quality criteria for consumer health information. The PEMAT evaluates the understandability and actionability of materials as a percentage. The DISCERN assesses the quality of information by a scale from 1 (serious or extensive shortcomings) to 5 (minimal shortcomings).
Results
The median number of total views was 22,450 (range 591–20,255,133) and the median number of views/month was 654 (range 9–723,398). 42 percent of the videos were posted by professional medical institutions, and 21% were posted by individual medical professionals. Most videos were aimed at the general public or patients suffering from ED. The median PEMAT understandability and actionability scores were both 100% (range 50–100% and 33–100%, respectively). The median DISCERN score was 2 (range 1-5) with 80.4% receiving a score of ≤3. Overall, 28% of the videos contained direct misinformation. DISCERN scores were higher in videos produced by medical institutions (P = .0104), not selling specific products (P = .007) and not promoting alternative medicine (P = .0002). The number of subscribers was an independent predictor of views/month (P < .0001).
Conclusion
Patients may be exposed to videos of poor quality when searching for information about ED on YouTube. The medical community needs to adapt a strategy to improve the quality of online medical information.
Fode M, Nolsøe AB, Jacobsen FM, et al. Quality of Information in YouTube Videos on Erectile Dysfunction. J Sex Med 2020;8:408–413.
Key Words: Communication, Erectile dysfunction, Information, Internet, Misinformation, Online, Social media, YouTube
Introduction
Online information and social media are playing an increasingly important role in health care, and many patients turn to these resources for information.1 With free and readily available videos and more than 30 billion daily users, YouTube represents one of the most used websites for dissemination of knowledge.2 However, only limited research on the quality of health care–related information in YouTube videos has been performed. This is concerning as anyone with an internet connection and a recording device can post videos on the website, which in turn can be viewed by patients anywhere. The problem is underlined by previous research showing that even on health-care provider websites, the quality of information is not always adequate and may sometimes be directly misleading.3
Within urology, the condition of main public interest on YouTube is prostate cancer4 and in a well-designed study, using validated assessment tools, Loeb et al recently found that the general quality of information in videos about prostate cancer is questionable and that many videos contain biases.5 Andrological urology is another topic of high public interest, but the available information in this field has not been adequately examined.4 The purpose of our study is to formally assess the quality of information regarding erectile dysfunction (ED) in videos on YouTube. We hypothesize that the information is widely accessed by users but that it is likely to be of low quality.
Materials and methods
We searched the term “erectile dysfunction” on YouTube in October 2019 (not logged into any personal account) without filters and sorted by relevance in the search function. We evaluated the first 100 videos in English and noted if they contained medical information, defined as material about biological mechanisms, and if they contained information about specific treatments. This approach was chosen as this reveals the videos that patients are most likely to be exposed to when searching the website. We recorded the user engagement by total views, views/month, and likes and dislikes to the videos. We also documented if the comment sections were enabled and if they included advertising, peer-to-peer medical advice, and/or signs of social support. The type of institution or individual who had posted the video was registered along with the intended audience, and it was recorded if the videos aimed at selling a specific product. For videos addressing the lay public, the target audience of the videos was defined as “patients suffering from ED” if the video made any reference to a problem in the viewer and as “general public” if the information was kept more neutral with no such reference. All videos containing medical information underwent a systematic evaluation using the Patient Education Materials Assessment Tool for Audiovisual Materials (PEMAT) and the validated DISCERN quality criteria for consumer health information. The PEMAT is a systematic method developed to select printable and audiovisual patient education materials, which are easier to understand and easier to act on.6,7 In this study, we used the version for audiovisual materials, which consists of 13 items measuring understandability and 4 items measuring actionability. For each question, the assessor selects either “Disagree,” “Agree,” or “Not Applicable.” “Agree” is chosen if a characteristic occurs in about 80–100% of the material. The score of all items is added together (“Agree” = 1 point and “Disagree” = 0 points), divided by the number of items on which the material was rated, and multiplied by 100 to give a percentage score for understandability and actionability, respectively. There is no set cutoff value for the scores. The DISCERN is a standardized index of quality of consumer health information on treatment choices, which can be used by anyone without the need for specialist knowledge.8 The tool uses 15 questions to judge the reliability of materials by assessing whether the sources of evidence are current, can be checked, or include apparent biases and if alternative options are mentioned. The publication is rated on a scale ranging from 1 (signifying “serious or extensive shortcomings”) to 5 (signifying “minimal shortcomings”) based on these questions. For the purposes of this study, we have rated videos by all relevant questions and given them an overall quality rating although not all videos were directly concerned with treatment choices.
To supplement the 2 instruments, we then evaluated if the videos favored new technology, recommended complementary/alternative medicine, or had an apparent commercial bias. The videos categorized as having an apparent commercial bias included specific products of which the presenter was deemed to have a personal financial interest. Videos selling a product were only classified as biased if they were unequivocally positive toward their product. In addition, we rated the extent of misinformation (defined as contradictions of established literature and/or guidelines) of the videos on a Likert score of 1-5 with the numbers corresponding to “none,” “low,” “moderate,” “high,” or “extreme.”
The videos were evaluated by 3 of the authors. 2 were medical doctors with a special interest in ED and andrology and one was a research fellow without prior specialized knowledge. This constellation was chosen to reduce bias in the PEMAT and DISCERN scores, while maintaining a high standard in the more subjective evaluation of content quality. Before evaluation, the assessors had familiarized themselves with the PEMAT user's guide and the DISCERN handbooks. Disagreements about video evaluations were settled by discussion and consensus.
We performed descriptive data analyses regarding the aforementioned parameters. In univariate analyses, Fisher's exact test was used to investigate if the type of producer or video content was associated with video quality as measured by the DISCERN score. Video characteristics including the video content, length, producer, PEMAT, and DISCERN scores were investigated using the Wilcoxon-Mann-Whitney test to identify potential predictors of video popularity measured as views per months and the number of likes per view. Factors which showed statistical significance on univariate analyses were tested using multivariate logistic regression and linear regression analyses to determine independent predictors. All tests were carried out using SAS Studio University Edition 3.8 (SAS Institute Inc, Cary, NC).
Results
The 100 videos on ED were posted between May 13th 2008 and August 6th 2019, and 92% of them contained medical information, whereas 67% contained specific information about treatments. The videos had been viewed an overall of 35,996,391 times and together they had 1,808,121 total monthly views. Information on video metrics and user engagement is listed in Table 1, and topics for videos with more than 1 million total views and more than 50,000 views per month are described in Table 2.
Table 1.
Video metrics and user engagement
| Median (range) | |
|---|---|
| Duration | 4.09 (0.38–74.51) minutes |
| The number of total views | 22,450 (591–20,255,133) |
| The number of views per month | 654 (9–723,398) |
| The number of likes | 115 (2–129,469) |
| The number of likes per view | 0.0049 (0.27–0.00039) |
| Dislikes | 12.5 (0–11,405) |
| The number of comments | 29 (0–2126) |
Table 2.
Topics and metrics for YouTube videos on erectile dysfunction with more than 1 million views and more than 50,000 views per month
| Topic | Date posted | Total views | Monthly views |
|---|---|---|---|
| Exercises for improving erections and performance in the bedroom (strength training and pelvic floor muscle exercises) | 18-06-2017 | 20,255,133 | 723,398 |
| Comedy video making fun of erectile dysfunction | 05-05-2019 | 3,573,383 | 714,677 |
| Information video on causes, symptoms, and treatment modalities for erectile dysfunction | 29-01-2013 | 2,493,900 | 50,896 |
| Advertisement video for an online men's health clinic selling medication online | 15-02-2018 | 1,002,990 | 50,150 |
97 of the videos had comments enabled, and 25 comment sections contained advertising, 4 contained peer-to-peer medical advice, and 7 contained signs of social support regarding ED.
42 percent of the videos were posted by professional medical institutions including hospitals and private clinics, and another 21% were posted by individual medical professionals. The remaining 37% of videos were posted by various YouTubers without a medical background. None of the videos were posted by patients or partners of patients. The majority of the videos were aimed at the general public (53%) or at patients suffering from ED (40%). Only 7% of the videos were produced for health-care professionals, and none of the videos were aimed specifically at partners to men who suffer from ED. 22 percent of the videos attempted to sell specific products or treatments for ED including low-intensity extracorporeal shockwave therapy, stem cell therapy, platelet-rich plasma injections, acupuncture, various herbal supplements, cannabidiol oil, and home delivery of phosphodiesterase type 5 inhibitors.
The median PEMAT understandability score for videos containing medical information was 100% (range 50–100%), and the median PEMAT actionability score was 100% (range 33–100%). The median DISCERN score was 2 (range 1-5). The distribution of scores is listed in Table 3. Table 4 contains information about new technology, complementary/alternative medicine, and commercial bias. Overall, 26/92 (28%) videos with medical information were deemed to contain direct misinformation. In 8 cases, the level of misinformation was low; in 11 cases, it was moderate; in 3 cases, it was high; and in 3 cases, it was deemed extreme.
Table 3.
DISCERN scores for videos containing medical information on ED (n = 92)
| Videos (n, %) | Total views | Views per month | Likes | Dislikes | |
|---|---|---|---|---|---|
| DISCERN 1 | 18 (19.6%) | 2,133,278 | 50,932 | 9,868 | 971 |
| DISCERN 2 | 35 (38%) | 24,413,118 | 899,653 | 159,476 | 14,752 |
| DISCERN 3 | 21 (22.8%) | 3,111,262 | 71,001 | 6,396 | 986 |
| DISCERN 4 | 16 (17.4%) | 1,239,940 | 38,134 | 6,370 | 812 |
| DISCERN 5 | 2 (2.2%) | 8,911 | 138 | 43 | 1 |
ED = erectile dysfunction.
Table 4.
Proportion of videos containing medical information on ED containing new technology, complementary/alternative medicine, and commercial bias, respectively (n = 92)
| Videos (n, %) | Total views | Views per month | Likes | Dislikes | |
|---|---|---|---|---|---|
| New technology | 13 (14.1) | 1,677,292 | 51,278 | 4,859 | 634 |
| Complementary/alternative medicine | 20 (21.7) | 2,663,818 | 113,839 | 18,579 | 1711 |
| Commercial bias | 23 (25) | 3,345,196 | 113,346 | 17,277 | 2319 |
ED = erectile dysfunction.
On univariate analyses, the type of video producer (P = .0002), if videos were selling specific products (P < .0001), and promotion of alternative medicine (P < .0001) were associated with the DISCERN score. The number of subscribers (P < .0001) and presence of medical advice (P = .0004) were associated with views per month. The number of subscribers (P = .0003) and the length of the videos (P = .0388) were associated with the number of likes per view. Multivariate logistic regression analysis showed that all parameters remained statistically significant predictors of video quality with higher DISCERN scores for videos produced by medical institutions (P = .0104), not selling specific products (P = .0070) and not promoting alternative medicine (P = .0002). Linear regression analyses showed that the number of subscribers was the only independent predictor of views per month with an increase of 0.0040 views (95% CI: 0.0021- 0.0059, P < .0001) per subscriber. There were no independent predictors of likes per view.
Discussion
The application of online resources containing information and support directly aimed at patients suffering from sexual dysfunction is still in the early phases, and there is limited knowledge about the available information.9 To our knowledge, our study represents the first systematic analysis with validated instruments of publicly available online information on ED. The results show that YouTube videos on the topic are primarily aimed at the general public and men suffering from ED and that the information is accessed by millions of people every month. At the same time, our data suggest a stigma surrounding ED as not a single video was posted by a patient with ED or the partner and as the second most viewed video was a comedy sketch making fun of the condition (Table 2). The stigma was also noted in the low degree of peer-to-peer interaction in the comment sections of the videos. The combination of high video popularity and apparent embarrassment could indicate that the field of sexual medicine is well suited for online information sources as these offer anonymity while gathering information.10 YouTube provides a free and easily accessible option for this. Moreover, social media in general may offer social peer-to-peer support of great potential value and even allows for communication between patients and health-care professionals. However, the lack of a barrier for posting on the website means that the information may be of varying value as illustrated in our study by more than 80% receiving a DISCERN score of 3 or less signifying low to moderate overall quality. This is concerning because people without a health-care education may have difficulty in distinguishing between good and bad information. The problem is exacerbated by our findings that low-quality videos were especially likely to advocate alternative medicine or to attempt selling a specific product. The latter is in accordance with previous studies on online medical information, which has shown that financial biases may in some cases influence the advice.11,12 However, the extent of advertisement is surprisingly high compared with that seen in prostate cancer–related videos, which only contained advertisements in 6% of the comment sections.5 In the light of this, it is of little comfort that our PEMAT scores show that videos are generally presented in a manner that is easy to understand and act upon. The problem seems to be universal as low quality of information in YouTube videos similar to our findings has recently been noted in a diverse range of other medical contexts including benign prostate hyperplasia,13 hydrocephalus,14 gastroesophageal reflux disease,15 HIV pre-exposure prophylaxis,16 and deep brain stimulation.17
Studies evaluating patient information videos professionally produced for specific medical conditions have shown positive effects with regard to a variety of issues including self-administration of treatments, anxiety reduction, and information recall.18,19,20 In line with this, we found that YouTube videos produced by medical institutions were of higher quality than those produced by individual professionals or nonprofessionals. A similar finding was recently made for videos concerning larynx cancer.21 Unfortunately, the quality of information in the videos did not predict popularity, and the highest quality videos only accounted for a small fraction of the views. The finding that video views were only associated with the existing number of subscribers to the YouTube channel from which the videos were posted shows that success on social media platforms need to be earned through a consistent effort over time. To our knowledge, the association has not been studied before and therefore not identified in previous studies on medical information on YouTube. However, it is crucial as even well-known and trusted medical institutions offering high-quality content cannot compete if this dynamic of online media is not observed. The implication for the medical community is that it is not enough to simply post good content. Rather, a joint effort over time will be needed to make an impact in the vast sea of online information. Within sexual medicine, such an effort may come from trusted associations such as the International Society for Sexual Medicine, the European Society for Sexual Medicine, and the European Association of Urology. These entities already offer resources of high quality to patients; however, the information is currently only available on the individual websites of the associations.22,23,24 To reach the broad public and to balance the current misinformation, it is likely necessary that the resources are offered on more well-known platforms of which YouTube is currently the most used one. As information of poor quality will not disappear from these platforms, a parallel effort to ensure that high-quality content is ranked more highly on search engines is also needed. This will require addressing the various manipulations that unscrupulous individuals and business will use to promote videos of lower scientific quality. To do so, it is likely needed to liaise with online marketing experts. The lack of independent predictors of likes per view in our study indicates that underlying mechanisms which we were unable to control for may play a role in this important metric. It may be speculated that such mechanisms could include soft measures as the attractiveness or likability of the presenter and the entertainment value of the videos. Focus-group studies with representatives for intended audiences during video development could be a way to explore and leverage such video characteristics.
The main strength of our study is the systematic evaluation of YouTube videos using a combination of validated questionnaires and expert opinion. The main limitation is the somewhat subjective evaluation of the videos inherent to this type of research including the risk of observer bias. We have attempted to limit this issue by using validated tools and engaging 3 assessors with different backgrounds including a person who was not an expert on andrology. In addition, we have little knowledge about the audience of the videos as this cannot be readily assessed on the website. It is natural to assume that younger men are the most prevalent viewers. However, previous research has shown that also middle-aged and older patients are likely to seek medical information online.25 Further studies are needed to elucidate this issue further. It would be of specific interest to engage YouTube users to evaluate how the video content is perceived and intercepted.
Conclusions
YouTube videos on ED are mainly produced for patients or the lay public, and there is no correlation between the video quality and popularity. This means that direct misinformation about the condition is being disseminated widely. To improve the situation, a concerted and lasting effort by the medical community is likely needed. This should be aimed at building YouTube channels with substantial amounts of followers to provide solid and relevant information on ED to a large audience.
Statement of authorship
Category 1
-
(a)Conception and Design
- Mikkel Fode; Alexander B. Nolsøe; Frederik M. Jacobsen; Giorgio Ivan Russo; Peter B. Østergren; Christian Fuglesang S. Jensen; Maarten Albersen; Paolo Capogrosso; Jens Sønksen
-
(b)Acquisition of Data
- Mikkel Fode; Alexander B. Nolsøe; Frederik M. Jacobsen
-
(c)Analysis and Interpretation of Data
- Mikkel Fode
Category 2
-
(a)Drafting the Article
- Mikkel Fode
-
(b)Revising It for Intellectual Content
- Alexander B. Nolsøe; Frederik M. Jacobsen; Giorgio Ivan Russo; Peter B. Østergren; Christian Fuglesang S. Jensen; Maarten Albersen; Paolo Capogrosso; Jens Sønksen
Category 3
-
(a)Final Approval of the Completed Article
- Mikkel Fode; Alexander B. Nolsøe; Frederik M. Jacobsen; Giorgio Ivan Russo; Peter B. Østergren; Christian Fuglesang S. Jensen; Maarten Albersen; Paolo Capogrosso; Jens Sønksen
Footnotes
Conflict of Interest: The authors report no conflicts of interest.
Funding: None.
References
- 1.Warren C. 30 Facts & Statistics on social media and healthcare [internet]. ReferralMD 2018. https://getreferralmd.com/2017/01/30-facts-statistics-on-social-media-and-healthcare/
- 2.The top 500 sites on the web. Alexa Internet, Inc. 2020. https://www.alexa.com/topsites Available at:
- 3.Dulaney C., Barrett O.C., Rais-Bahrami S. Quality of prostate cancer treatment information on cancer Center websites. Cureus. 2016;8:e580. doi: 10.7759/cureus.580. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Borgmann H., Salem J., Baunacke M. Mapping the landscape of urology: a new media-based cross-sectional analysis of public versus academic interest. Int J Urol. 2018;25:421–428. doi: 10.1111/iju.13527. [DOI] [PubMed] [Google Scholar]
- 5.Loeb S., Sengupta S., Butaney M. Dissemination of misinformative and biased information about prostate cancer on YouTube. Eur Urol. 2019;75:564–567. doi: 10.1016/j.eururo.2018.10.056. [DOI] [PubMed] [Google Scholar]
- 6.Shoemaker S.J., Wolf M.S.S.B.C. Development of the Patient Education Materials Assessment Tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information. Patient Educ Couns. 2014:395–403. doi: 10.1016/j.pec.2014.05.027. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Agency for Healthcare Research and Quality. The Patient Education Materials Assessment Tool (PEMAT) and user’s guide. Available at: https://www.ahrq.gov/ncepcr/tools/self-mgmt/pemat-av.html. Accessed March 16, 2020.
- 8.DISCERN Online Quality criteria for consumer health information. www.discern.org.uk/index.php Available at: Accessed March 16, 2020.
- 9.Minichiello V., Rahman S., Dune T. E-health: potential benefits and challenges in providing and accessing sexual health services. BMC Public Health. 2013;13:790. doi: 10.1186/1471-2458-13-790. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Leusink P.M., Aarts E. Treating erectile dysfunction through electronic consultation: a pilot study. J Sex Marital Ther. 2006;32:401–407. doi: 10.1080/00926230600835361. [DOI] [PubMed] [Google Scholar]
- 11.Read J., Mati E. Erectile dysfunction and the internet: drug company manipulation of public and professional opinion. J Sex Marital Ther. 2013;39:541–559. doi: 10.1080/0092623X.2012.736922. [DOI] [PubMed] [Google Scholar]
- 12.Garcia-Cruz E., Romero-Otero J., Fode M. The digital environment in men’s sexual disorders: a systematic review. Actas Urol Esp. 2017 doi: 10.1016/j.acuro.2017.05.008. [DOI] [PubMed] [Google Scholar]
- 13.Betschart P., Pratsinis M., Mullhaupt G. Information on surgical treatment of benign prostatic hyperplasia on YouTube is highly biased and misleading. BJU Int. 2020;125:595–601. doi: 10.1111/bju.14971. [DOI] [PubMed] [Google Scholar]
- 14.Szmuda T., Rosvall P., Hetzger T.V. YouTube as a source of patient information for hydrocephalus: a content-quality and Optimization analysis. World Neurosurg. 2020;138:e469–e477. doi: 10.1016/j.wneu.2020.02.149. [DOI] [PubMed] [Google Scholar]
- 15.Aydin M.F., Aydin M.A. Quality and reliability of information available on YouTube and Google pertaining gastroesophageal reflux disease. Int J Med Inform. 2020;137:104107. doi: 10.1016/j.ijmedinf.2020.104107. [DOI] [PubMed] [Google Scholar]
- 16.Whiteley L., Healy M.G., Lowery A. An evaluation and review of English language pre-exposure prophylaxis websites and YouTube videos. Int J STD AIDS. 2020;31:460–466. doi: 10.1177/0956462420905271. [DOI] [PubMed] [Google Scholar]
- 17.Tripathi S., ReFaey K., Stein R. The reliability of deep brain stimulation YouTube videos. J Clin Neurosci Off J Neurosurg Soc Australas. 2020;74:202–204. doi: 10.1016/j.jocn.2020.02.015. [DOI] [PubMed] [Google Scholar]
- 18.Ahlander B.-M., Engvall J., Maret E. Positive effect on patient experience of video information given prior to cardiovascular magnetic resonance imaging: a clinical trial. J Clin Nurs. 2018;27:1250–1261. doi: 10.1111/jocn.14172. [DOI] [PubMed] [Google Scholar]
- 19.Ter Laak S., Lucas I., Oude Nijeweeme J. An age-adjusted instruction video enhances the correct use of nasal corticosteroid sprays in children. Allergol Immunopathol (Madr) 2020 doi: 10.1016/j.aller.2019.11.006. [DOI] [PubMed] [Google Scholar]
- 20.Michie L., Cameron S.T., Glasier A. Giving information about the contraceptive implant using a DVD: is it acceptable and informative? A pilot randomised study. J Fam Plan Reprod Heal Care. 2016;42:194–200. doi: 10.1136/jfprhc-2015-101186. [DOI] [PubMed] [Google Scholar]
- 21.Enver N., Doruk C., Kara H. YouTube as an information source for larynx cancer: a systematic review of video content. Eur Arch Otorhinolaryngol. 2020;277:2061–2069. doi: 10.1007/s00405-020-05906-y. [DOI] [PubMed] [Google Scholar]
- 22.Patient information website for the International Society for sexual medicine n.d. https://www.issm.info/sexual-health-conditions Available at:
- 23.Patient information website for the European Society for sexual medicine n.d. https://www.essm.org/patients/ Available at:
- 24.Patient information website for the European association of urology n.d. https://patients.uroweb.org/ Available at:
- 25.Fode M., Frey A., Ostergren P.B. Patterns of knowledge Acquisition Among men Undergoing Radical Prostatectomy. Eur Urol. 2019;75:1040–1041. doi: 10.1016/j.eururo.2019.01.018. [DOI] [PubMed] [Google Scholar]