Abstract
Coronavirus disease (COVID)-related misinformation is prevalent online, including on social media. The purpose of this study was to explore factors associated with user engagement with COVID-related misinformation on the social media platform, TikTok. A sample of TikTok videos associated with the hashtag #coronavirus was downloaded on September 20, 2020. Misinformation was evaluated on a scale (low, medium, and high) using a codebook developed by experts in infectious diseases. Multivariable modeling was used to evaluate factors associated with number of views and presence of user comments indicating intention to change behavior. One hundred and sixty-six TikTok videos were identified and reviewed. Moderate misinformation was present in 36 (22%) videos viewed a median of 6.8 million times (interquartile range [IQR] 3.6–16 million), and high-level misinformation was present in 11 (7%) videos viewed a median of 9.4 million times (IQR 5.1–18 million). After controlling for characteristics and content, videos containing moderate misinformation were less likely to generate a user response indicating intended behavior change. By contrast, videos containing high-level misinformation were less likely to be viewed but demonstrated a nonsignificant trend towards higher engagement among viewers. COVID-related misinformation is less frequently viewed on TikTok but more likely to engage viewers. Public health authorities can combat misinformation on social media by posting informative content of their own.
Keywords: TikTok, misinformation, COVID-19, public health
INTRODUCTION
Misinformation includes misleading or false information that contradicts scientific consensus.1 On social media, misinformation is shared relatively freely among users and spreads farther and faster than the truth.2–4 A single false post or story can dominate searches on a given topic.5 Even when users identify misinformation in real-time, corrections or rebuttals can register as user engagement that amplifies a story. Thus, misinformation can spread even as it is acknowledged to be wrong. When misinformation that threatens public health spreads across borders, it becomes an infodemic.6–9
Exposure to coronavirus disease (COVID)-related misinformation online can influence health-related beliefs, healthcare-seeking behaviors, and mental well-being.10–13 On social media, COVID-related misinformation has been documented on Twitter, Facebook, and TikTok.14–18 Compared to other platforms, TikTok has a user base that is expanding, particularly among younger, lower income adults living in cities.19 Thus, COVID-related content has the potential to reach an audience on TikTok that may not otherwise receive important public health messaging.20
Though COVID-related content has been described online and on social media, the response from users is not as well understood.21 The purpose of this study was to determine what types of videos on TikTok engage users, elicit user comments, and potentially lead to changes in behavior.
METHODS
We retrospectively reviewed a random sample of user-created videos on TikTok associated with the hashtag #coronavirus. Videos were eligible for inclusion if published in English on or before the date of the search (September 1, 2020). Conducting the search on a single day was used to capture a “snapshot” of misinformation on social media at a moment in time.14 The #coronavirus hashtag was selected because it generated relevant videos with the largest cumulative number of views (97.8 billion) and has been used by other investigators for similar purposes.16 Other search terms generated sets of videos with fewer cumulative views, including #covid19 (37.4 billion views), #covid (6.1 billion views), #covid_19 (1.8 billion), and #covid2019 (82.7 million views). Videos were excluded if unviewable due to privacy settings or determined based on preliminary review to be unrelated to COVID-19, related illness, or related public health measures. Preliminary review was used to identify videos that were unrelated to COVID-19 but used the #coronavirus hashtag to boost appearances in search results.
Videos and related metadata were downloaded using a third-party TikTok Scraper (https://github.com/drawrowfly/tiktok-scraper) that is freely available under an MIT license. This tool uses TikTok's Web API to download videos and related metadata.
Prior to review of video content, a codebook was developed by a hospital epidemiologist with expertise in infectious diseases (JB) and a clinician with expertise with social media research (RM). The codebook reflected public health guidance related to COVID-19 and served as a reference to identify misinformation (see Supplemental Material). Videos were viewed and annotated by RB, JF, DS, and RM between September 2020 and January 2021. The first 9 videos were reviewed and annotated by all reviewers, and independent ratings were compared for discrepancies and adjudicated to ensure consistency. Once interrater consistency was above 90%, subsequent videos were reviewed independently.
Prior to review of videos, evidence-based information related to each section of the codebook was reviewed by the research team. Presence of misinformation in each video was graded as low, moderate, or high based on the extent and degree of misinformation. For instance, a video implying that surgical masks were contaminated with COVID-19 because they originated in China was considered highly misinformative. A video implying that the COVID-19 pandemic was predicted on a television show was considered moderately misinformative.
Video quality was assessed using 2 validated scales: the Patient Education Materials Assessment Tool (PEMAT) and DISCERN.22,23 PEMAT is a systematic method available through the Agency for Healthcare Research and Quality to evaluate the understandability and actionability of patient education materials.23 PEMAT understandability and actionability scores were categorized as high (≥80) or low/moderate (<80). DISCERN was designed to help health consumers assess the quality of written information about treatment choices for a health problem.
Video comments were reviewed to identify requests for medical advice, provision of medical advice, reported intention to change behavior, or commercial advertising. Comments reflecting each category were considered present if at least one relevant comment was found. All comments potentially indicating intent to change behavior were reviewed by the research team as a group. Only comments that expressed a clear intention to change behavior (eg, “I will do…”) were included in this category. Comments that expressed mixed intention or uncertainty (eg, “Does anyone know if…?”) or agreement without related behavior (eg, “Your videos are so inspiring”) were not counted as reflecting intent to change behavior.
Multivariable regression with robust standard errors was used to evaluate the association between video characteristics and outcomes. Negative binomial regression was used for count outcomes, while logistic regression was used for binary outcomes. Covariates were included in multivariable models if associated with the outcome of interest on bivariate analysis with a P-value < .10 and not highly correlated with other covariates. Covariates related to characteristics of the video publisher were collected from their biography or username. Individuals who self-identified as a physician or doctor were categorized among other official sources. In one instance, a physician was a verified TikTok user. In other cases, these claims could not be verified. Coefficients from negative binomial regression are reported as the incidence rate ratio (IRR), while coefficients from logistic regression are reported as the odds ratio (OR). All coefficients from multivariable models are reported with 95% confidence intervals (95% CI).
RESULTS
Out of 1000 TikTok videos identified during the initial search, 166 (17%) met the inclusion criteria and were included in the study (Table 1). Videos were published between March 14, 2020 and August 22, 2020. Twenty-six videos (16%) were posted by an official source, including 7 videos from foundations or advocacy groups, 6 from governmental agencies, 3 from doctors, 3 from other healthcare providers, and 1 from a professional society. Seventy-six (46%) videos included humor, and 91 (55%) were set to music. Fifty-five (33%) videos were published by women, and 93 (56%) of publishers were under the age of 30. Only 2 (1.2%) videos were suspected of commercial bias. Videos were viewed a median of 8.4 million times (interquartile range [IQR] 4.9M–18M), and the median number of comments was 6871 (IQR 3348–13 000).
Table 1.
Video views by characteristic and content
| Video characteristics | Videos (%) | Median views (IQR) |
|---|---|---|
| n = 166 | ||
| Official publisher | ||
| Healthcare worker | 8 (5) | 14M (13–14M) |
| Foundation/government | 13 (4) | 28M (19–35M) |
| News media/outlet | 4 (2) | 7.2M (5.2–9.2M) |
| Any official sourcea | 26 (16) | 23M (10–35M) |
| Unofficial publisher | ||
| Patient | 5 (3) | 18M (14–21M) |
| Other | 136 (82) | 7.4M (4.4–1M) |
| Subject content | ||
| Risk factorsa | 22 (13) | 21M (13M–35M) |
| Symptomsa | 26 (16) | 16M (7.7M–37M) |
| Modes of transmissionb | 38 (23) | 12M (6.4M–24M) |
| Preventiona | 90 (54) | 11M (5.6M–23M) |
| Content related to prevention | ||
| Masks | 60 (36) | 9.5M (5.6M–22M) |
| Hand hygiene | 53 (32) | 8.4M (4.9M–27M) |
| Distancing | 45 (27) | 7.7M (4.7M–14M) |
| Quarantine/isolationa | 35 (21) | 5.0M (3.4M–11M) |
| Limiting exposures | 51 (30) | 9.5M (4.9M–23M) |
| Content related to masks | ||
| Demonstration of useb | 43 (26) | 10M (7.1M–24M) |
| Positive sentiment | 37 (22) | 8.1M (5–18M) |
| Neutral sentiment | 20 (12) | 10M (7.3–37M) |
| Negative sentiment | 3 (2) | 5.0M (4.7–21M) |
| Content related to testing | ||
| Testing discussedb | 13 (8) | 5.0M (3.7M–7.7M) |
| Positive sentiment | 9 (5) | 5.0M (3.7–7.7M) |
| Neutral sentiment | 7 (4) | 8.7M (5.1–20M) |
| Negative sentiment | 0 (0) | – |
| Misinformation | ||
| Absent | 119 (72) | 8.5M (5.3–19M) |
| Moderate | 36 (22) | 6.8M (3.6–16M) |
| High | 11 (7) | 9.4M (5.1–18M) |
| DISCERN rating | ||
| Lowb | 50 (30) | 9.9M (4.5–22M) |
| Moderate | 84 (51) | 9.9M (4.5M–22M) |
| High | 32 (19) | 8.2M (5.0M–15M) |
| PEMAT rating | ||
| High understandabilityb | 110 (66) | 9.4M (5.4–21M) |
| High actionability | 15 (9) | 26M (5.6M–35M) |
| Total | 166 | 8.4M (4.9M–18M) |
Abbreviation: PEMAT: Patient Education Materials Assessment Tool.
Significantly different than comparator group at an α < .001.
Significantly different than comparator group with P-value < .05.
Forty-seven (28%) videos contained misinformation, including 36 videos with a moderate level of misinformation and 11 videos with a high level of misinformation. Videos containing moderate misinformation were viewed a median of 6.8 million times (IQR 3.6–16 million), and videos containing high-level misinformation were viewed a median of 9.4 million times (IQR 5.1–18 million). Examples of high- and moderate-level misinformation are included in a joint display with comment content in Figure 1. Examples of misinformation included claims that recommendations for quarantine were the result of a conspiracy and that masks do not offer protection against infection.
Figure 1.
User response to TikTok videos based on level of misinformation.
Results from multivariable modeling to identify factors independently associated with how many times a video was viewed are listed in Table 2. Videos containing misinformation were viewed a median of 7.4M times (IQR 3.6M–18.1M). After controlling for video subject matter, videos published by an official source (IRR 1.49; 95% CI 1.09–2.05), such as a government agency, foundation, or advocacy group, were more likely to be viewed. View counts were significantly lower among videos suspected of commercial bias (IRR 0.23; 95% CI 0.15–0.35) and videos containing a high level of misinformation (IRR 0.59; 95% CI 0.39–0.90).
Table 2.
Factors associated with view counts and comments indicating intention to change behavior on TikTok videos with the #Coronavirus hashtag
| Factor | View counta | Behavior changeb |
|---|---|---|
| IRR (95% CI) | OR (95% CI) | |
| Predictor of interest | ||
| Moderate misinformation | 0.78 (0.57–1.05) | 0.25 (0.078–0.77) |
| High misinformation | 0.59 (0.39–0.90) | 6.14 (0.72–52.6) |
| Publisher characteristics | ||
| Publisher age < 30 | 0.57 (0.45–0.73) | 0.40 (0.14–1.11) |
| Official sourcec | 1.49 (1.09–2.05) | 0.35 (0.045–2.71) |
| Video content | ||
| Inclusion of humor | 1.20 (0.93–1.55) | 0.487 (0.18–1.28) |
| Content related to symptoms | 1.72 (1.25–2.37) | 3.18 (0.92–10.9) |
| Content related to testing | 0.96 (0.61–1.50) | 10.1 (2.25–45.5) |
| Content related to prevention | 1.66 (1.31–2.09) | 10.2 (3.46–29.8) |
| Content related to quarantine | 0.50 (0.39–0.64) | 0.14 (0.048–0.40) |
| Commercial bias | 0.23 (0.15–0.35) | -omitted- |
| Video quality | ||
| DISCERN rating of moderate-high | 0.67 (0.51–0.89) | 2.07 (0.66–6.46) |
| PEMAT understandability high | 1.18 (0.92–1.50) | 3.22 (1.24–8.34) |
| PEMAT actionability high | 0.97 (0.64–1.46) | 1.65 (0.14–19.4) |
Bold values represent factors that were significant at a threshold of α < .05.
View count was modeled using multivariable negative binomial regression.
Behavior change was evaluated based on user comments reporting intention to change behavior and was modeled using multivariable logistic regression.
Government, foundation, or advocacy group.
User comments indicating intention to change behavior were associated with 117 videos (70.5%), including 25 videos containing misinformation (15.1%). After controlling for video subject matter, user-reported intention to change behavior was associated with high PEMAT understandability (OR 3.22; 95% CI 1.24–8.34). Moderate misinformation (OR 0.25; 95% CI 0.078–0.77) was associated with decreased likelihood of user-reported intention to change behavior. High-level misinformation was associated with a nonsignificant trend towards increased likelihood of user engagement in terms of behavior change (OR 6.14; 95% CI 0.72–52.6).
DISCUSSION
In this retrospective review of early COVID-related content on the social media platform TikTok, 28% of videos contained at least moderate-level misinformation. The relationship between misinformation and user engagement was complex. Videos containing high-level misinformation were less likely to be viewed but more likely to engage viewers. On the other hand, videos containing moderate misinformation were viewed as frequently as videos without misinformation but were less engaging.
Previous research has identified that COVID misinformation on TikTok is highly viewed.24 Our study builds on this literature by examining user comments posted in response to misinformative videos. The experience of users on TikTok can include passive consumption of content or active interaction with other users.25 By differentiating videos based on the extent and degree of misinformation, we were able to explore further how misinformation provokes active engagement.
Misinformation spreads more easily than true information, and public health misinformation spreads more quickly when public health experts express uncertainty.3,26 In this study, though videos containing high-level misinformation were less likely to be viewed, misinformative content was nonetheless viewed millions of times. One in 4 misinformative videos elicited comments indicating intention to change behavior. Though misinformative videos were often humorous and frequently ridiculous, at least some users took them seriously. In the context of a coordinated public health response, those instances were troubling.
Behaviors spread on social media through social reinforcement.2,27,28 Because we were unable to measure user behavior directly, we represented behavior using proxy outcomes: view counts and user comments reporting intention to change behavior. Public health agencies should take note that videos posted by an official source were more likely to be viewed, and highly understandable videos were more likely to generate a user response indicating intended behavior change. Clear, understandable, and informative videos posted by trustworthy sources may be able to counteract the negative impact of misinformation on TikTok. To facilitate dissemination of informative content, authoritative sources may consider partnering with organizations that are better equipped or more experienced communicating with the public, such as mass media.4
This study had several limitations. First, the videos included in our analysis represent a sample of public English-language videos associated with the hashtag #coronavirus identified on the day we conducted the search. Our study is subject to potential bias that may have been introduced if the TikTok algorithm selectively displayed or suppressed specific videos. Second, our study was limited to videos in English despite COVID-19 related rumors, stigma, and conspiracy theories have been posted in 25 languages from 87 countries.29 Third, we did not examine the characteristics of users posting comments, which are known to influence susceptibility to health-related misinformation.12 Thus, we were unable to understand how viewer characteristics might interact with video characteristics to affect engagement. Finally, because we used human reviewers to identify misinformation, our work may be less scalable than research based on automated algorithms.30
In conclusion, the relationship between misinformation on TikTok and user engagement is complex. For a niche audience, misinformative content may be highly engaging. To combat misinformation, public health agencies should work to establish more of a presence on TikTok and other social media platforms.
Supplementary Material
Contributor Information
Jonathan D Baghdadi, Division of Genomic Epidemiology and Clinical Outcomes, Department of Epidemiology and Public Health, University of Maryland School of Medicine, Baltimore, Maryland, USA.
K C Coffey, Division of Genomic Epidemiology and Clinical Outcomes, Department of Epidemiology and Public Health, University of Maryland School of Medicine, Baltimore, Maryland, USA.
Rachael Belcher, Division of Urology, Department of Surgery, University of Maryland School of Medicine, Baltimore, Maryland, USA.
James Frisbie, Division of Urology, Department of Surgery, University of Maryland School of Medicine, Baltimore, Maryland, USA.
Naeemul Hassan, College of Information Studies, University of Maryland, College Park, Maryland, USA; Philip Merrill College of Journalism, University of Maryland, College Park, Maryland, USA.
Danielle Sim, Division of Urology, Department of Surgery, University of Maryland School of Medicine, Baltimore, Maryland, USA.
Rena D Malik, Division of Urology, Department of Surgery, University of Maryland School of Medicine, Baltimore, Maryland, USA.
FUNDING
Agency for Healthcare Research and Quality grant number 1K08HS028854-01 (to J.D.B.).
AUTHOR CONTRIBUTIONS
All authors reviewed and contributed to the final draft of this manuscript. RDM conceived of the study. NH, RB, JF, DS, and RDM performed data acquisition. JDB and RDM designed and conducted the analysis. KCC contributed interpretation of results. JDB and KCC wrote the initial draft.
SUPPLEMENTARY MATERIAL
Supplementary material is available at JAMIA Open online.
CONFLICT OF INTEREST STATEMENT
None declared.
DATA AVAILABILITY
The data underlying this article are available on Dryad (https://doi.org/10.5061/dryad.bvq83bkdp).
REFERENCES
- 1. Swire-Thompson B, Lazer D.. Public health and online misinformation: challenges and recommendations. Annu Rev Public Health 2020; 41 (1): 433–51. [DOI] [PubMed] [Google Scholar]
- 2. Desai AN, Ruidera D, Steinbrink JM, et al. Misinformation and disinformation: the potential disadvantages of social media in infectious disease and how to combat them. Clin Infect Dis 2022; 74 (Suppl_3): e34–39. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Vosoughi S, Roy D, Aral S.. The spread of true and false news online. Science 2018; 359 (6380): 1146–51. [DOI] [PubMed] [Google Scholar]
- 4. Tasnim S, Hossain MM, Mazumder H.. Impact of rumors and misinformation on COVID-19 in social media. J Prev Med Public Health 2020; 53 (3): 171–4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Waszak PM, Kasprzycka-Waszak W, Kubanek A.. The spread of medical fake news in social media – the pilot quantitative study. Health Policy Technol 2018; 7 (2): 115–8. [Google Scholar]
- 6. Goel A, Gupta L.. Social media in the times of COVID-19. J Clin Rheumatol 2020; 26 (6): 220–3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Ferrara E, Cresci S, Luceri L.. Misinformation, manipulation, and abuse on social media in the era of COVID-19. J Comput Soc Sci 2020; 3 (2): 271–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Zarocostas J. How to fight an infodemic. Lancet 2020; 395 (10225): 676. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Cinelli M, Quattrociocchi W, Galeazzi A, et al. The COVID-19 social media infodemic. Sci Rep 2020; 10 (1): 16598. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Liu M, Caputi TL, Dredze M, et al. Internet searches for unproven COVID-19 therapies in the United States. JAMA Intern Med 2020; 180 (8): 1116–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Lee JJ, Kang K-A, Wang MP, et al. Associations between COVID-19 misinformation exposure and belief with COVID-19 knowledge and preventive behaviors: cross-sectional online study. J Med Internet Res 2020; 22 (11): e22205. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Loomba S, de Figueiredo A, Piatek SJ, et al. Measuring the impact of COVID-19 vaccine misinformation on vaccination intent in the UK and USA. Nat Hum Behav 2021; 5 (3): 337–48. [DOI] [PubMed] [Google Scholar]
- 13. Verma G, Bhardwaj A, Aledavood T, et al. Examining the impact of sharing COVID-19 misinformation online on mental health. Sci Rep 2022; 12 (1): 8045. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Kouzy R, Abi Jaoude J, Kraitem A, et al. Coronavirus goes viral: quantifying the COVID-19 misinformation epidemic on Twitter. Cureus 2020; 12 (3): e7255. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Sheera Frenkel D. Surge of Virus Misinformation Stumps Facebook and Twitter. New York City, NY: The New York Times; 2020. [Google Scholar]
- 16. Southwick L, Guntuku SC, Klinger EV, et al. Characterizing COVID-19 content posted to TikTok: public sentiment and response during the first phase of the COVID-19 pandemic. J Adolesc Health 2021; 69 (2): 234–41. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Bastani P, Hakimzadeh SM, Bahrami MA.. Designing a conceptual framework for misinformation on social media: a qualitative study on COVID-19. BMC Res Notes 2021; 14 (1): 408. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Wang Y, Han R, Lehman TS, et al. Do Twitter users change their behavior after exposure to misinformation? An in-depth analysis. Soc Netw Anal Min 2022; 12 (1): 167. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Hellemans J, Willems K, Brengman M. The new adult on the block: daily active users of TikTok compared to Facebook, Twitter, and Instagram during the COVID-19 crisis in Belgium. In Digital Marketing & eCommerce Conference; 2021; Springer. [Google Scholar]
- 20. Ostrovsky AM, Chen JR.. TikTok and its role in COVID-19 information propagation. J Adolesc Health 2020; 67 (5): 730. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Yan M, Lin Y-R, Chung W-T. The Effect of Twitter User’s Commenting Behavior on the Propagation of COVID-19 Misinformation. In: Carnegie Mellon IDeaS Annual Conference; 2020.
- 22. Charnock D, Shepperd S, Needham G, et al. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 1999; 53 (2): 105–11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Shoemaker SJ, Wolf MS, Brach C.. Development of the Patient Education Materials Assessment Tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information. Patient Educ Couns 2014; 96 (3): 395–403. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Basch CH, Meleo-Erwin Z, Fera J, et al. A global pandemic in the time of viral memes: COVID-19 vaccine misinformation and disinformation on TikTok. Hum Vaccin Immunother 2021; 17 (8): 2373–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Montag C, Yang H, Elhai JD.. On the psychology of TikTok use: a first glimpse from empirical findings. Front Public Health 2021; 9: 641673. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Wang Y, McKee M, Torbica A, et al. Systematic literature review on the spread of health-related misinformation on social media. Soc Sci Med 2019; 240: 112552. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. Centola D. The spread of behavior in an online social network experiment. Science 2010; 329 (5996): 1194–7. [DOI] [PubMed] [Google Scholar]
- 28. Kaakinen M, Sirola A, Savolainen I, et al. Shared identity and shared information in social media: development and validation of the identity bubble reinforcement scale. Media Psychol 2020; 23 (1): 25–51. [Google Scholar]
- 29. Islam MS, Sarkar T, Khan SH, et al. COVID-19-related infodemic and its impact on public health: a global social media analysis. Am J Trop Med Hyg 2020; 103 (4): 1621–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Shang L, Kou Z, Zhang Y, et al. A multimodal misinformation detector for COVID-19 short videos on TikTok. In: 2021 IEEE International Conference on Big Data (Big Data). 2021.
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The data underlying this article are available on Dryad (https://doi.org/10.5061/dryad.bvq83bkdp).