YouTube as a source of patient information for Coronavirus Disease (COVID‐19): A content‐quality and audience engagement analysis

Summary

YouTube is the second most popular website in the world and is increasingly being used as a platform for disseminating health information. Our aim was to evaluate the content‐quality and audience engagement of YouTube videos pertaining to the SARS (severe acute respiratory syndrome)‐CoV‐2 virus which causes the Coronavirus Disease 2019 (COVID‐19), during the early phase of the pandemic. We chose the first 30 videos for seven different search phrases: “2019 nCoV,” “SARS CoV‐2,” “COVID‐19 virus,” “coronavirus treatment,” “coronavirus explained,” “what is the coronavirus” and “coronavirus information.” Video contents were evaluated by two independent medical students with more than 5 years of experience using the DISCERN instrument. Qualitative data, quantitative data and upload source for each video was noted for a quality and audience engagement analysis. Out of the total 210 videos, 137 met our inclusion criteria and were evaluated. The mean DISCERN score was 31.33 out of 75 possible points, which indicates that the quality of YouTube videos on COVID‐19 is currently poor. There was excellent reliability between the two raters (intraclass correlation coefficient = 0.96). 55% of the videos discussed prevention, 49% discussed symptoms and 46% discussed the spread of the virus. Most of the videos were uploaded by news channels (50%) and education channels (40%). The quality of YouTube videos on SARS‐CoV‐2 and COVID‐19 is poor, however, we have listed the top‐quality videos in our article as they may be effective tools for patient education during the pandemic.

Abbreviations

COVID‐19

Coronavirus Disease 2019

MERS‐CoV

Middle East respiratory syndrome coronavirus

SARS‐CoV

severe acute respiratory syndrome

VPI

video power index

WHO

World Health Organization

1. INTRODUCTION

Nowadays, people often search for health information online. YouTube, an online video platform, is the second most popular website worldwide and is often used as a source of medical education. The reliability of YouTube videos has been evaluated for several other medical diseases. ¹ , ² , ³ However, no study has yet evaluated the quality and reliability of YouTube videos on SARS (severe acute respiratory syndrome)‐CoV‐2.

The outbreak of this novel coronavirus was reported in Wuhan, China on December 31, 2019. It infected and killed thousands of people worldwide in a matter of weeks. The health crisis spurred new research to be conducted on the epidemiology, transmission and clinical characteristics of the virus. ⁴ , ⁵ , ⁶ It is critical for the public to understand the basic features of the coronavirus so that they realize the threat of the viral illness on their own health, the health of society and follow proper health protocols (ie, wash heads frequently, maintain social distancing, seek medical care early, practice respiratory hygiene, avoid touching the face and practice food safety). By being informed, the public may prevent infection, reduce the spread of the virus and reduce the burden of the diseases on the healthcare facilities in a country. Therefore, it is critical to be aware of the information that the public is receiving on the coronavirus especially during a pandemic. This is especially true since previous health studies have shown that YouTube has spread misinformation. ⁷

On March 11, 2020, Coronavirus Disease 2019 (COVID‐19) was declared a pandemic by the World Health Organization (WHO). This study was conducted 1 day after this date to assess the reliability and quality of information the public was viewing on the novel coronavirus. Moreover, we aimed to establish what aspects of a video drove audience engagement (in terms of likes, comments and views) so that better content could be created in the future. Finally, we aimed to compile the most educational videos so that hospitals and health organizations could feature the most reliable information on their websites.

2. MATERIALS AND METHODS

2.1. Search strategy and data collection

Google Chrome browser was used in “incognito mode” when browsing YouTube. This was done so that no personal recommendations affected the search results. All searches were done with the YouTube default “relevance” sorting. We did not apply any time or date filters since most viewers search on YouTube without any of these filters and simply use the default search. In this way, we sought to replicate the same search pattern that users use most commonly. The first 30 videos for each of the following search terms were recorded: “2019 nCoV,” “SARS CoV‐2,” “COVID‐19 virus,” “coronavirus treatment,” “coronavirus explained,” “what is the coronavirus” and “coronavirus information.” Only the first 30 videos were chosen since 90% of YouTube users do not look past that number. ⁸ The videos were recorded into our database on March 12, 2020 (1 day after the WHO classified the COVID‐19 outbreak a pandemic).We understand that information about the coronavirus is rapidly changing. Therefore, we collected all the videos into our database and evaluated them in under 48 hours with the information that was available at that time. Therefore, the information about the Coronavirus and the videos on YouTube about the coronavirus were analyzed at almost the same time (within 48 hours). We did this to reduce any changes in scientific information from the point that we started our research to the point that we ended our analysis.

2.2. Inclusion and exclusion criteria

To choose relevant videos, we excluded videos that were (1) too long for the average viewer to watch (greater than 1 hour), (3) videos that were not in English, (3) duplicate videos and (4) videos that were not relevant to COVID‐19 (ie, that talked only about older coronaviruses).

2.3. Variables extracted

We classified the video upload source into one of the following categories: physician, hospital, educational channel, health organizations (eg, WHO) and news shows.

We determined if the video had any of the following qualitative video elements: if coronavirus is a zoonotic virus, common symptoms of the disease, which population is at risk of severe illness, a discussion of causes, the viral incubation period, methods of diagnosis, if a vaccine is available or in development, preventative measures, management, prognosis, animation, radiological images, diagrams, viral anatomy, methods of spread of the virus, a doctor speaker, a mention of Middle East Respiratory Syndrome Coronavirus (MERS‐CoV), mention of SARS (SARS‐CoV) and the difference between the novel coronavirus and previous coronaviruses (ie, SARS‐CoV and MERS‐CoV).These elements were selected as they were included in the frequently asked questions on Centers for Disease Control and Prevention webpage on COVID‐19. ⁹

The following descriptive statistics were recorded using “vidIQ Vision for YouTube,” a Google Chrome extension: views, duration (seconds), number of comments, number of likes, number of dislikes, average views per hour, referrers (number of external web pages that link to the video) and time since upload (days).

2.4. Scoring system

All videos were evaluated independently by two final year medical students with more than 5 years of experience using the DISCERN instrument. DISCERN instrument, shown on Table 1, is a validated 16‐part questionnaire to assess the quality of health information and has been used in several YouTube quality analysis studies. ¹⁰ , ¹¹

TABLE 1.

The 16 question DISCERN instrument

#	Question	Rating
1	Are the aims clear?	1	2	3	4	5
2	Does it achieve its aims?	1	2	3	4	5
3	Is it relevant?	1	2	3	4	5
4	Is it clear what sources of information were used to compile the publication (other than the author or producer)?	1	2	3	4	5
5	Is it clear when the information used or reported in the publication was produced?	1	2	3	4	5
6	Is it balanced and unbiased?	1	2	3	4	5
7	Does it provide details of additional sources of support and information?	1	2	3	4	5
8	Does it refer to areas of uncertainty?	1	2	3	4	5
9	Does it describe how each treatment works?	1	2	3	4	5
10	Does it describe the benefits of each treatment?	1	2	3	4	5
11	Does it describe the risks of each treatment?	1	2	3	4	5
12	Does it describe what would happen if no treatment is used?	1	2	3	4	5
13	Does it describe how the treatment choices affect overall quality of life?	1	2	3	4	5
14	Is it clear that there may be more than one possible treatment choice?	1	2	3	4	5
15	Does it provide support for shared decision making?	1	2	3	4	5
16	Based on the answers to all of these questions, rate the overall quality of the publication as a source of information about treatment choices	1	2	3	4	5

Before starting to score the videos, the raters only reviewed the official DISCERN scoring instructions as the DISCERN instrument does not require any pre‐calibration. ¹⁰ , ¹¹ Previous articles regarding the quality of information online using the DISCERN instrument also did not discuss how to rate the videos beforehand. ¹ , ³ In this way, the ratings were truly independent and are not influenced by external influence.

Each of the 16 questions may gain a score from 1 to 5 (where 1 is the minimum score and 5 is the maximum score). ¹⁰ The first 15 questions of the DISCERN score may be summed to determine the overall quality of a video as very poor (16‐26), poor (27‐38), fair (39‐50), good (51‐62) and excellent (63‐75). ¹²

2.5. Audience engagement

In our article, we used the term “audience engagement” to refer to the video power index (VPI), like ratio and average daily views. Previous publications have also used this term in reference to these metrics. ¹ , ³ From the descriptive statistics extracted, audience engagement metrics were calculated: VPI [(like*100/[like + dislike])*(views/day)/100], like ratio [(likes/likes + dislikes)*100] and average daily views [views/days since upload].

Videos were grouped according to their qualitative video content (eg, if a video included the prognosis of novel coronavirus) and were analyzed against the like ratio, average daily views, VPI, number of comments and the DISCERN score.

2.6. Statistical methods

Descriptive statistics for continuous variables covered mean, median, range and SD. To find differences between groups, the Kolmogorov‐Smirnov Test tested for normality, then the Mann‐Whitney U test found differences between categorical variables. The intraclass correlation coefficient was used to ascertain the inter‐rater agreement. A P‐value below .05 was deemed significant. Google Sheets (Google LLC, Mountain View, CA) and Past (Hammer and Harper, Øyvind Hammer, Natural History Museum, University of Oslo) were used for statistical analysis and illustrations.

3. RESULTS

3.1. Video contents

Out of the total 210 videos, 137 videos met our inclusion criteria and were evaluated. Figure 1 illustrates the qualitative video contents of the videos. Most videos (55%) mentioned prevention. However, less than half (49%) reviewed symptoms and talked about the spread of the virus (46%). Few videos mentioned the diagnostics involved (13%), the anatomy of the organs affected by the SARS‐CoV‐2 (12%) and radiological findings (5%) (Figure 1).

FIGURE 1.

Video contents of the videos reviewed

3.2. Video upload source

Figure 2 shows that most of the videos were uploaded by news channels (48.2%) and educational channels (38.7%). The rest were uploaded by health organizations (5.1%), hospitals (5.1%) and physicians (2.9%).

FIGURE 2.

Source of video uploads on COVID‐19

3.3. Video statistics

The following are the mean values for the descriptive statistics recorded: view count 469 514 (801‐8 546 863); comments 1317 (0‐19 190); likes 5939 (2‐171 000); dislikes 398 (0‐14 200); like ratio 90.42 (40‐100); duration 536.55 seconds (36‐2838); time since upload 22.79 days (1‐76); and VPI 41745.76 (9‐1 083 115.32).

3.4. Video quality evaluation

The mean DISCERN score for the first 15 questions between the two raters was 31.33 ± 9.92 (16‐64) indicating that the overall quality of the videos was poor ¹³ ; the first rater and the second rater had a DISCERN score of 31.35 ± 10 (16‐64) and 31.29 ± 9.8 (17‐60) respectively.

Figure 3 illustrates that videos uploaded by physicians, health organizations and educational channels had a higher average DISCERN score (above 30) than news shows and hospitals (below 30). However, videos uploaded by physicians had an overall fair quality (a score of 40.3).

FIGURE 3.

Average DISCERN scores by source of uploads

The mean score of question 16 of DISCERN, which requires a holistic judgment of the entire video was 1.23 ± 0.03 (1‐4); and was 1.25 (1–4) and 1.21 (1‐4) respectively for the individual raters.

The intraclass correlation coefficient for the absolute agreement was 0.96 for DISCERN between the two raters; this is regarded as an excellent reliability (Table 2). ¹⁴

TABLE 2.

Intraclass correlation coefficient for the DISCERN scores among the two raters

	Intraclass correlation a	95% Confidence interval
Single measures b	0.9598	0.9442‐0.9712
Average measures c	0.9795	0.9713‐0.9854

^aThe degree of absolute agreement among measurements.

^bEstimates the reliability of single ratings.

^cEstimates the reliability of averages of DISCERN ratings.

Figure 4 illustrates the mean for each of the 16 questions of DISCERN. Question 1 (are the aims clear?) has a low mean rating of 3.6, indicating that most videos failed to make their aims clear about the video content that they are providing. In addition, for Question 1, the DISCERN score does not match the mean for Question 2 (does it achieve its aims?), which shows a mean rating of 3.2. Thus, many videos claiming to provide information did not realize the aims they intended by their title. The average DISCERN score for Question 3 (is it relevant?) was 3.1 which indicates that about 40% of the videos were not relevant about educating the general population about the virus and the disease caused by it. There is a sharp decrease in the scores for Questions 9‐14 which are concerned about treatment. In this manner, many videos failed to mention anything about treatment, the quality of life after the treatment or the quality of life when a patient is not treated.

FIGURE 4.

The mean DISCERN score for each of the 16 DISCERN instrument criteria

In our study, news channels comprised the majority of videos uploaded on COVID‐19. However, their quality was the worst (mean DISCERN score of 27.9). Physician video uploaders had the best video quality (mean DISCERN score of 40.3) followed by health organizations (mean DISCERN score of 38.7) and education channels (mean DISCERN score of 37.6).

3.5. Video quality correlations

Videos that covered the following qualitative information all had a significantly higher DISCERN score than those that did not: zoonotic nature of the virus (P < .0001), clear information (P = .0044), causes (P < .0001), incubation period (P = 0054), diagnosis (P < .0004), vaccine possibility (P = .0005), management (P < .0001), prognosis (P < .0001), diagrams (P < .0001), anatomy of the viral structure (P < .0001), spread of virus (P = .0020), doctor speaking (P = .0070), MERS mentioned (P < .0001), SARS mentioned (P < .0001), the difference between SARS‐CoV‐2 and previous viral strains (P = .0001).

3.6. Audience engagement analysis

In our analysis we found that videos that discussed the zoonotic nature of the virus had a significantly lower average daily views (P = .0221) and VPI (P = .0173), yet, had a higher like ratio (P = .0045). Videos that mentioned the 2003 SARS outbreak also had lower average daily views (P = .0044) and a lower VPI (P = .0059).

A higher like ratio was linked to videos incorporating causes (P = .0211), management (P = .0002), diagrams (P < .0001) and anatomy of the structures involved (P = .0001).

3.7. Top quality videos

Table 3 shows that three of the five highest rated videos were from the channel “MedCram‐Medical Lectures Explained Clearly.” In addition, all the top five videos had one thing in common: the speaker was a doctor. The video titled “COVID‐19 (SARS‐CoV‐2) Epidemic with Dr. Forest Arnold” (YouTube ID: zFrghcp5pbY) contained almost all the qualitative features we analyzed (except the presence of animations). Two of the top five highest DISCERN scoring videos were over 30 minutes long. The top five videos contained high quality information as shown by their DISCERN scores in Table 3.

TABLE 3.

The best quality videos as a source of patient information for SARS‐CoV‐2 on YouTube

DISCERN	Title	Uploader	YouTube ID
66	Coronavirus Epidemic Update 34: US Cases Surge, Chloroquine & Zinc Treatment Combo, Italy Lockdown	MedCram‐Medical Lectures Explained Clearly	U7F1cnWup9M
66	How Coronavirus Kills: Acute Respiratory Distress Syndrome (ARDS) & Treatment	MedCram‐Medical Lectures Explained Clearly	okg7uq_HrhQ
63	2019 Novel Coronavirus (COVID‐19) Update	JAMA Network	wg5PjvbVioA
61.5	Coronavirus Epidemic Update 11: Antiviral Drugs, Treatment Trials for nCoV (Remdesivir, Chloroquine)	MedCram‐Medical Lectures Explained Clearly	pfGpdFNHoqQ
58.5	COVID‐19 (SARS‐CoV‐2) Epidemic with Dr. Forest Arnold	UofL Internal Medicine Lecture Series	zFrghcp5pbY

4. DISCUSSION

4.1. Quality analysis

We found that the overall quality and reliability of YouTube videos on SARS‐CoV‐2 and COVID‐19 is poor, which indicates that the information about this disease on YouTube does not provide a comprehensive review. We have included a table of the highest quality videos on YouTube as a reference so that people who may want to learn about COVID‐19 may use them as a relatively reliable source of knowledge. We suggest physicians and other medical professionals recommend these videos to the general population and feature them on their websites for a better understanding of COVID‐19. We also encourage YouTube content creators to use the DISCERN scoring system as a guide to making better quality and unbiased videos. ¹⁵ Our findings are novel since our study is the first to analyze the quality of health information concerning COVID‐19 on YouTube. Moreover, our results are relevant as our analysis was conducted just a few days after COVID‐19 was declared a pandemic.

As Figure 1 illustrates, more than half of the videos failed to provide rudimentary information such as symptoms, spread of the infection and prevention. Today the information on YouTube is neither regulated nor reviewed. Thus, anyone with an Internet connection may upload health information videos which may be accessed by millions worldwide. In the case of the COVID‐19 pandemic, misinformation spread on this platform has the potential to cause panic and worse health outcomes for patients.

Thus, we recommend that the public avoid news channels if they wish to obtain holistic knowledge about the SARS‐CoV‐2 virus and COVID‐19. Our results show that the videos uploaded by news channels had a relatively low average DISCERN score of 27.9. Videos uploaded by physicians had the highest DISCERN scores with a mean of 40.3 points (fair quality), which is better than any other result by a specific category of uploader. Thus, watching videos uploaded by physicians is better in gaining basic knowledge compared especially to news channels.

4.2. Audience engagement analysis

Videos containing the causes, management, diagrams and the anatomy of structures involved had a higher like ratio. Explanation of the anatomical structures which may be affected by the virus through diagrams may help viewers understand the pathogenesis of the disease more fluently and thus we suggest future content creators take this into consideration. If physicians and health organizations focus their videos more on the treatment options and possible pharmacotherapy (or the lack thereof), they may contribute to even higher quality videos and more engaging videos for the public.

Notably, including the zoonotic nature of the virus and mentioning the 2003 SARS outbreak significantly decreased the audience engagement (in terms of the average daily views and the VPI). We assert that historical information about the 2003 SARS outbreak may not seem relevant to the public. Moreover, amidst the emerging and rapidly evolving pandemic, explaining the zoonotic nature of the virus may not be as engaging as it does not directly pertain to practical information to help people prevent infection.

4.3. Context

The DISCERN criteria focus heavily on the treatment aspects of a disease ranging from the mechanism of action to its benefits, risks and the quality of life affected due to the treatment. It assesses what would happen if no treatment is used. Currently, there are no approved drugs for treating COVID‐19. Several randomized trials are underway to evaluate the efficacy of “remdesivir” for moderate or severe COVID‐19. ¹⁶ The use of a combined protease inhibitor “lopinavir‐ritonavir” has also been described in case studies. ¹⁷ , ¹⁸ , ¹⁹ Chloroquine ²⁰ and hydroxychloroquine ²¹ were also other interventions of interest but with limited or no clinical data.

Management of patients with suspected or diagnosed with COVID‐19 consists of ensuring appropriate infection control and supportive care, however, few videos mentioned this. Moreover, most videos failed to mention that there were drugs undergoing clinical trials against COVID‐19. This may be important, as when people know that there are ways to treat and manage COVID‐19, they may feel better prepared for the pandemic situation.

In our analysis, the news channels had a poor DISCERN score. These videos not only failed to mention the treatment and management strategies but often failed to provide almost any medical information that we took into our analysis, although their video title claimed to. We assert that the COVID‐19 crisis is not only a viral crisis but also a health information crisis. Recent studies have shown that wide dissemination of misinformation concerning the outbreak has created panic among the public. ²² , ²³ Larson et al has dubbed the biggest risk of COVID‐19 “viral misinformation.” ²² Thus, not only is there an urgent need to scale‐up public health measures, but also one to address the pandemic of social media panic and misinformation. ²⁴

4.4. Limitations

Medical student raters (as in our study) may be considered unreliable raters compared to physicians. However, the creators of DISCERN state that their instrument “was not dependent on specialist knowledge of a health condition or treatment.”. ¹⁰ Moreover, the intraclass correlation coefficient between the two raters was excellent. ¹⁴ Thus, we assert that the medical student raters, who each had 5 years of experience using the DISCERN instrument while in medical school, provided adequate results for this study.

4.5. Future directions

Since SARS‐CoV‐2 and COVID‐19 are such relevant topics in the emerging pandemic, we encourage researchers to conduct a follow‐up analysis in the future. We also encourage video content creators to use our audience engagement suggestions and the DISCERN instrument to assess their videos for educational content so that they may provide more robust information about the disease going forward.

4.6. Clinical implications

There are several simple yet effective steps that people may practice personally to prevent the spread of infection, such as washing heads regularly, practicing cough etiquette and avoiding public gatherings. However, these reminders need to be promoted and presented in health content on YouTube so that people adhere to these recommendations. Absence of this information may result in higher health‐care costs for a country, increased risk of viral spread and higher hospitalization and mortality rates. We hope that this quality analysis serves as a prompt call to action to physicians and health organizations to create more robust health content for viewers online.

5. CONCLUSION

The overall quality of YouTube videos on SARS‐CoV‐2 and COVID‐19 is poor. Thus, healthcare consumers obtaining health information on COVID‐19 on YouTube are obtaining an incomplete comprehension. This may lead to misunderstandings concerning treatment, prevention and the general characteristics of the virus. We recommend that people verify the facts they learn on YouTube with more reliable sources of information like peer‐reviewed research papers, healthcare professionals or online updates by the WHO or the Centers for Disease Prevention and Control. We recommend physicians to refer patients to the list of the highest quality videos that we provided if a patient wishes to use YouTube as a source of their medical information.

CONFLICT OF INTEREST

The authors have no competing interest.

Supporting information

Appendix S1. Supporting information

Szmuda T, Syed MT, Singh A, Ali S, Özdemir C, Słoniewski P. YouTube as a source of patient information for Coronavirus Disease (COVID‐19): A content‐quality and audience engagement analysis. Rev Med Virol. 2020;30:e2132. 10.1002/rmv.2132