A scoping review of digital health interventions for combating COVID-19 misinformation and disinformation

Abstract

Objective

We provide a scoping review of Digital Health Interventions (DHIs) that mitigate COVID-19 misinformation and disinformation seeding and spread.

Materials and Methods

We applied our search protocol to PubMed, PsychINFO, and Web of Science to screen 1666 articles. The 17 articles included in this paper are experimental and interventional studies that developed and tested public consumer-facing DHIs. We examined these DHIs to understand digital features, incorporation of theory, the role of healthcare professionals, end-user experience, and implementation issues.

Results

The majority of studies (n = 11) used social media in DHIs, but there was a lack of platform-agnostic generalizability. Only half of the studies (n = 9) specified a theory, framework, or model to guide DHIs. Nine studies involve healthcare professionals as design or implementation contributors. Only one DHI was evaluated for user perceptions and acceptance.

Discussion

The translation of advances in online social computing to interventions is sparse. The limited application of behavioral theory and cognitive models of reasoning has resulted in suboptimal targeting of psychosocial variables and individual factors that may drive resistance to misinformation. This affects large-scale implementation and community outreach efforts. DHIs optimized through community-engaged participatory methods that enable understanding of unique needs of vulnerable communities are urgently needed.

Conclusions

We recommend community engagement and theory-guided engineering of equitable DHIs. It is important to consider the problem of misinformation and disinformation through a multilevel lens that illuminates personal, clinical, cultural, and social pathways to mitigate the negative consequences of misinformation and disinformation on human health and wellness.

INTRODUCTION

Misinformation (and disinformation) is so pervasive that it has been listed by the World Economic Forum as a global risk for human wellness.^1,2 As engagement with online platforms for health information grows,^3–5 so has the spread of unfounded information that impedes the adoption of positive health behaviors that protect against disease.^6–11 With the current climate of distrust in scientific and medical institutions as a source of reliable information,^11–14 the dissemination of misinformation and disinformation presents a significant public health challenge. As has been apparent in the context of the SARS-CoV-2 (COVID-19) pandemic, inaccurate information, considered a “second pandemic of misinformation,” can lead to harmful health behaviors and impede the effectiveness of population health interventions and legitimate risk-related information.^4,8,15 In particular, the number of online platforms such as social media has been increasing, especially in low- and middle-income countries (LMICs).¹⁶ Social media also has become a predominant source of scientific and medical news in the United States as well as in European nations.^17,18 Although convenient, the search process for health information in social media makes it difficult for users to discern facts from opinions.¹⁹ Further, online platforms have allowed the rapid spread of questionable information posing a threat to individuals’ well-being^20,21 and creating unexpected challenges for public health agencies.¹⁵

To mitigate harm from health misinformation and disinformation during the COVID-19 pandemic, health agencies have called for a global movement to deliver evidence-based COVID-19 information.²² In response, several specialized Digital Health Interventions (DHIs) have been developed around the globe to address such misinformation and disinformation.²³ As defined by the World Health Organization (WHO), a DHI provides a discrete functionality of digital technology that is applied to achieve health objectives and is implemented within digital health applications and information systems, including communication channels such as text messages.²⁴ Several other definitions exist for DHIs, and in the context of this paper, DHIs are defined as interventions that utilize digital tools (eg, mobile apps, chatbots, online gaming, websites) to enable an end user to resist and rebut COVID-19 misinformation and disinformation.^25,26 To this end, we conduct a scoping review of DHIs to examine their development methods (specifically, the incorporation of theory and stakeholder engagement), digital technology utilization, end-user experience, and implementation barriers and facilitators.

BACKGROUND AND SIGNIFICANCE

Fox (1983) conducted the first work on misinformation, defining it as false information, while subsequent studies expanded its description to include uncertain, vague, and ambiguous information.^27,28 Disinformation is defined as information that “includes all forms of false, inaccurate, or misleading information designed, presented, and promoted to intentionally cause public harm or for profit.”^29,30 Notably, misinformation about critical health issues, which can lead to mistrust of the clinical and scientific community, has the potential to impose a threat to the value of medical advances such as vaccines.^1,31–36

Different strategies to address the circulation of misinformation, also known as a misinfodemic, have become urgently needed during the COVID-19 pandemic. A misinfodemic stems from an infodemic, which is defined as an overabundance of information, only some of which is accurate, that makes it difficult for people to find trustworthy sources and reliable guidance.³⁷ During the COVID-19 pandemic, the high virality of the large quantity of misleading information led to a misinfodemic, which WHO noted as a highly significant phenomenon that undermined public health responses to the pandemic around the globe.^38,39

Several online tools have been developed in the general domain, ranging from educational sources, bot-detection software to misinformation-detection algorithms, propagation models, and credibility assessments.^39–48 Some tools rely on statistical patterns and machine learning algorithms while others depend on nonautomatic methods such as crowdsourcing and human implementation or manual fact-checking.⁴⁹ A total of 90 tools have been identified by the RAND corporation and other organizations,⁴⁹ 98% of which are free of charge and 83%, fully operational. The main types of tools are presented in Table 1.

Table 1.

Tools for addressing misinformation and disinformation

Tool	Area(s) of focus	Implementation method	Method
Content verification	Content	Manual	Fact-checking through manual review
Standard implementation
Information literacy, education, and training
Disinformation tracking	Process	Automated	Machine learning
Whitelisting
Bot detection	Content and process	Manual and automated	Blockchain or crowdsourcing
Information and source credibility scoring

Building on the general domain tools/strategies shown in Table 1, several specialized DHIs that target COVID-19 misinformation and disinformation have been developed and evaluated.^23,50–54 Our understanding of these DHIs, however, is limited in terms of their: (1) methodological underpinnings, (2) facilitators and barriers concerning large-scale implementation efforts, and (3) end-user experience.

OBJECTIVE

The objective of this paper is to provide a scoping review and summary of research studies that develop and evaluate DHIs to mitigate COVID-19 misinformation and disinformation seeding and spread. We examine whether and how interventions: (1) utilize digital technologies; (2) incorporate cognitive frameworks and behavioral theory to guide intervention development; (3) leverage contributions from healthcare professionals (eg, clinicians, community health workers); (4) measure outcomes to establish an intervention’s efficacy; (5) capture end-user experiences; and (6) provide barriers and facilitators to the implementation of DHIs.

MATERIALS AND METHODS

Overview

This scoping review was conducted using the methodological framework for scoping reviews proposed by Arksey and O’Malley and the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA).^55,56 The 5 stages proposed by Arksey and O’Malley include: (1) identifying the research question(s); (2) identifying relevant studies; (3) selecting the study; (4) charting (or tabulating) the data; and (5) collating, summarizing, and reporting the results.

Literature search

After identifying our objectives, we conducted a comprehensive search on 3 electronic databases: PubMed, PsychINFO, and Web of Science. Numerous keywords and MeSH terms were used to build our queries (see the search strategy in the Supplementary File). All articles retrieved from the search are available in the public domain and were uploaded into Endnote 20 and consolidated into a single list.

Screening procedure

The complete list of articles was screened by 6 reviewers. Articles first were divided into 3 equal clusters, after which one of the 3 teams of 2 reviewers each reviewed a cluster. Within each team, the reviewers first independently screened the title and abstract using the eligibility criteria and then came together to compare the results of their independent screening and resolve any disagreements. Each team of 2 reviewers then screened the full texts and extracted all relevant information from the final list onto an Excel spreadsheet, after which the teams came together to combine their data into a comprehensive table that was reviewed by all to ensure inter-rater reliability.

Inclusion/exclusion criteria

Certain eligibility criteria were used to determine the final list of articles included in our review. We selected research studies focused on technology-based interventions or tools designed to mitigate COVID-19-related misinformation or disinformation. We included only those articles available in English that were published from 2019 to the present that described or evaluated such an intervention. We excluded all scoping reviews, meta-analyses, and systematic reviews and based our review on experimental studies as well as descriptive studies that mentioned a DHI.

RESULTS

Our search queries generated a total of 1666 articles. Of these, 626 articles were retrieved from PubMed, 97 from PsychInfo, 937 from Web of Science, and 6 through snowballing (Figure 1). After removing duplicates, the remaining 1159 titles/abstracts were initially screened, and the 73 articles that met our criteria were then charted in an Excel database and the full texts were reviewed by the entire team. During this phase, 54 articles were excluded because they described machine learning techniques for identifying false news, rumors, or misinformation that had been tested for functionality and acceptability but had not been incorporated into interventions for combating COVID-19 misinformation or disinformation. An additional 2 articles were excluded due to a lack of access to the full text. Of the 1159 articles screened and reviewed, 17 were included in the final analysis.^57–73 (see Supplementary Materials). A summary of the DHIs in these 17 studies and their methodological underpinnings is provided below. Figure 2 provides an overview of the DHIs that employed theoretical constructs and the data foundation to guide feature development.

Figure 1.

The PRIMSA diagram explains the identification and screening process applied to our literature review. Out of the original 1159 articles that were screened, 17 were included in the final analysis.

Figure 2.

This Venn diagram shows the classification of DHIs included in the final analysis in terms of data and theory. Only 6 of the 17 DHIs included in the final analysis were both data-driven and theory-based. DHIs: Digital Health Interventions.

Summary of digital features

Eleven studies utilized some aspects of social media in their interventions, such as platforms for data collection, group creation, or network modeling.^{57,60,61,63–67,70,71,73} These studies, however, are limited to 1 or 2 platforms, such as creating Facebook groups. Two studies created groups on a social media platform that allowed 2-way communication between members and moderators to share information,^57,63 but 2 other studies only pushed information (a 1-way channel) to the group.^60,61 One study organized town hall meetings about COVID-19 and then disseminated the recordings on Facebook, which makes it a hybrid, utilizing digital and nondigital channels.⁶³ Three studies made chatbots to help address misinformation and added support to include specific languages that are prevalent in their regions.^61,68,72 Three studies examined the best practices to debunk misinformation through messaging.^65,67,73 One study focused on how to stop rumors from spreading; however, it is limited to a single community on Sina Weibo and used a 2SI2R epidemic model to examine how to prevent misinformation spread.⁶⁶ Another 2 studies addressed the effects of misinformation-sharing intentions at the individual level.^58,69 One study investigated a market tool built to identify practitioners who spread misinformation and label them as noncompliant.⁶⁴ One study used a browser game and infographics to enable individuals to identify misinformation and mitigate its spread.⁵⁹

Incorporation of theory

Table 2 presents the transdisciplinary theoretical frameworks used in the studies. Of the 17 articles included in the final analysis, approximately half (52%; n = 9) specified and described a theory, framework, or model used in the development and implementation of the DHI.^{57–59,62,65–68,72} The theories, frameworks, and models that were noted came from a variety of disciplines, including behavioral science, psychology, sociology, communication, mathematics, epidemiology, computer science, and artificial intelligence. The most commonly used theories and models were psychosocial or computer science-based.

Table 2.

Summary of theoretical underpinnings

Author(s) (year)	Theory/framework/model	Behavioral outcomes measured
Albrecht et al. (2022)	Risk reduction model, interdisciplinary science communication, and a 2-way model of communication	No
Amin et al. (2021)	Visual selective attention theory	Yes
Basol et al. (2021)	Prebunking, inoculation theory	Yes
Ghaleb et al. (2022)	Machine learning, natural language processing	No
Königand Breves (2021)	Content-source integration model, theory of epistemic authority	Yes
Liu and Qi (2022)	I2S2R interactive infection model of multiple rumor engagers	No
Mourali and Drake (2022)	Extant theory	Yes
Pandey et al. (2022)	World Health Organization’s identify-simplify-amplify-quantify model	No
Siedlikowski et al. (2021)	Unsupervised machine learning Q&A model	No

Less than half of the studies (47%; n = 8) included in the final analysis measured and evaluated a behavioral construct. And, not all studies specified a theory, model, or framework for those constructs.^{58–60,64,65,67,70,73} The most common behavioral and psychosocial constructs targeted by the DHIs aimed at combating COVID-19 misinformation or disinformation were knowledge, attitudes, risk perceptions, and intentions. Intentions were measured in terms of inclination toward information dissemination (eg, intent to share social media posts) or adoption of preventive behaviors (eg, intent to wear a mask).^65,67 Many studies specified behavioral or psychosocial constructs when describing the development of the DHI, however, did not evaluate the effects of the DHI on users. The most common reason cited for the lack of evaluation was constraints due to the time-sensitive and evolving nature of the COVID-19 pandemic.

Role of healthcare professionals

More than half of the final studies (53%; n = 9) utilized the help of healthcare professionals in various roles to combat COVID-19-related misinformation and disinformation.^{57,61–65,69,71,72} The majority mentioned receiving help from an interdisciplinary team that consisted of medical doctors, nurses, epidemiologists, and public health practitioners to develop the content and features of DHIs. Medical professionals were involved in the development and use of chatbots to disseminate accurate COVID-19-related information to the public.^62,72 Their contribution was valuable in selecting relevant information about each topic, testing the accuracy of the content, providing feedback on major upgrades, and evaluating the efficiency and efficacy of the technology. In addition, healthcare professionals were directly involved in communicating COVID-19-related information to lay audiences via social media to thwart misinformation or disinformation spread.^57,61,63,71 They actively participated in online question-and-answer sessions,^57,61 contributed through town hall meetings,⁶³ managed newsletters and websites,⁵⁷ and corrected misinformation on social media.⁷¹

Summary of study outcomes

The design and study outcomes of the 17 articles are described in Supplementary Table S2. We report 12 descriptive studies, 1 observational study, and 4 experimental studies. Many investigations that evaluate online content were designed to target populations from local areas based on the language. Languages utilized outside of English included Spanish,⁵⁷ Arabic,⁶² Hindi,⁶⁸ French,⁷² Danish,⁶¹ and Chinese.^66,73 The variety of languages involved in these investigations has reflected a global effort to abrogate the spread of misinformation related to COVID-19. Sixteen studies considered the dissemination of educational information or the identification/rebuttal of misinformation. Five studies provided descriptive information regarding user preferences and/or psychosocial dynamics related to DHI design.^{65,66,68,72,73}

End-user experience

Only one study evaluated user awareness and satisfaction among a sample of 308 users who resided in Saudi Arabia. Ghaleb et al⁶² found that their chatbot’s accurate responses to inquiries led to a significant effect on user satisfaction (B = 0.799, P = 0.000) and that the chatbot positively and significantly increased users’ awareness of the DHI as a method to fight the infodemic (B = 0.567, P = 0.000). The remaining 16 studies did not provide any details about user experience and/or perceptions. Four studies were descriptive, summarizing the content and implementation of a DHI.^57,61,63,71 Two studies described the restrictions that affected the conduct of rigorous evaluations.^68,72 Siedlikowski et al⁷² stated that, due to the emergency caused by the evolving pandemic, they lacked the time and resources needed to evaluate their chatbot’s implementation and measure its acceptability and impact on users. Pandey et al⁶⁸ described how they were forced to remove their machine learning-based smartphone application (app) from the Google Play store in the middle of their study. The authors were ordered to revamp the app content so that it met new guidelines for COVID-19-related apps put in place by the clearinghouse, which severely cut the study timeframe. Finally, 5 social media-based studies lacked data on user perceptions due to simulated data/settings,^{65–67,70,74} and 3 due to retrospectively collected data.^64,66,73 As such, the studies included in this review are unable to provide insight into user experiences.

Examination of barriers to and facilitators of implementation

After assessing the limitations of the studies included in this review, 3 common themes were observed. First, 8 studies indicated bias due to limitations in the study sample and demonstrated that the findings are not necessarily representative of the greater population.^{58,63,65,67,69–71,73} For example, Pennycook et al⁷⁰ studied social media COVID-19 misinformation in the United States, and, therefore, their results could be quite different from those of other areas in the world. Second, 10 studies analyzed only one social platform, which potentially limits the generalizability of the findings.^{57,59,60,62,63,65,67,68,71,72} Finally, the most common shortcoming in these studies was the lack of methods that facilitate community engagement, which is essential to address the specialized needs of vulnerable groups, including minorities, older adults and seniors, and individuals with disabilities and other pre-existing conditions (such as Type 2 diabetes) that increase their risk for severe health consequences from COVID-19.

General strengths and facilitators also were assessed. We noted regular updates to the DHIs in real-time due to the pandemic, which required just-in-time informational updates. Given the need for social distancing, connectivity tools such as Zoom teleconferences and Facebook groups were optimized, as seen in Jayawardena et al and Furstrand et al, respectively, which proved to be effective in their implementation.^61,63 Overall, these studies demonstrated a user onboarding phase, which is associated with a steep learning curve highlighting the need for DHIs that are naturalistic and capable of leveraging our built-in cognitive awareness.

DISCUSSION

With the digitization of information dissemination and community outreach activities through online social media, we have a unique opportunity to capture, monitor, examine, and evaluate the components operationalized in trending misinformation (including semantics, syntax, framing, and behavioral constructs) that are permeating public health risk communications and diffusing in vulnerable communities globally. Several studies have conducted infoveillence of online social media interactions to monitor and examine the etiology of COVID-19 misinformation and disinformation.^75–77 The translation of data-based insight from large-scale observational studies into empirical technology evaluations, however, is limited.

Utilization of digital features and technologies

Evaluation of the literature on DHIs to combat COVID-19 misinformation reveals several common themes relevant to public health endeavors. It is clear that modern avenues of communication, including social media posts, websites, and text messages, are capable of impressive dissemination of information even on an international level.^{57,61,63,68,69,71} Attempts at effective public health outreach should concentrate on leveraging these channels of communication. DHIs that mitigate misinformation and disinformation generally fall into 3 categories: those that attempt to identify incorrect information,^59,70,72,78 promote accurate information,^{57,61–63,65,68,69,71} or counter misinformation directly.^{60,64,66,67,73,79}

A realistic intervention to combat misinformation will likely require the use of automation given the high volume of inaccurate information freely available to the public, a situation that is exacerbated by the willingness to spread information without confirmation of content accuracy.⁷⁰ Fortunately, early attempts to identify relevant information with automated methods have shown impressive accuracy. Pandey et al⁶⁸ report that this strategy can double the relevance of information in as little as 45 days. These early results show promise for further large-scale applications. Although identification of misinformation may help users to better evaluate information, Wang et al⁷³ and Kreps and Kriner⁷⁹ indicate that the more effective approach appears to involve direct rebuttal in the form of accurate information.

Role of theory

Most DHIs being developed are not informed by behavioral theory despite their goal to target users’ intentions to share information. For such interventions to be able to change human behavior, future research should aim to integrate evidence-based theories that seek to understand and influence human behavior centered around intention (or motivation), such as the integrated behavior model.⁸⁰ Such integration is key to increasing effectiveness and impact and can be accomplished by using an approach such as Intervention Mapping, a planning framework that provides a systematic process for developing behavior change interventions.⁸¹

Contributions of healthcare professionals and community engagement

The value of information dissemination directly from clinical experts will likely be the key to ongoing efforts to quell the spread of misinformation and disinformation related to COVID-19, as professional recommendations are considered more trustworthy by the public.⁶⁵ The United Nations has recommended the appointment of volunteers on social media as “digital first responders” to correct health misinformation.⁸² This need is currently being addressed in medical education and preliminary analysis suggests that social media training for medical students propagates accurate information on social media.⁷¹ This trend can be observed in the attempted monitoring of clinical providers’ social media accounts to ensure accurate postings.⁶⁴ Further, enabling clinicians to comprehend the terrain of COVID-19 misinformation and disinformation is essential to empower their patients with evidence-based clinical recommendations. Such clinical readiness is imperative for leveraging and implementing shared decision-making during patient encounters to promote COVID-19 testing, vaccines, and other infection prevention strategies. Some of the barriers encountered by health professionals actively involved in the development, implementation, and improvement of various technologies to fight COVID-19-related misinformation and disinformation include lack of time, lack of demonstrability of positive outcomes, avoidant behaviors, harassment and bullying, lack of social media training, and lack of organizational support.⁸³

Community engagement and participatory research methods are lacking in the development and evaluation phases of the majority of the DHIs considered in our review. Without insight into end-user acceptance, it is difficult to design and diffuse community outreach programs that leverage implementation science methodologies to counter health misinformation. In addition, a lack of community engagement leads to representative bias and suboptimal DHIs, which cannot meet the unique needs of vulnerable communities.²⁴

Implications for user engagement and experience

The studies included in our review indicate that the packaging, timing, and personalization of accurate information may be as important as the content. For example, the viewing and sharing of Twitter posts are more likely when content is in lower-case letters.⁶⁵ Alternative formatting, including false information indicators, are less effective,⁷⁹ whereas interventions implemented in the period of psychosocial resistance may augment the impact of the intervention.⁶⁶ Culturally competent DHIs are important to enhance user experience and ensure continued engagement. For example, a study of a Canadian application that provides educational information regarding COVID-19 found that responses to questions were more accurate in English than in French.⁷² Similarly, user engagement with a Hindi version of an Indian educational application was found to be higher than that in English.⁶⁸ This implies that the investment of resources for the contextual translation of educational interventions may be required for optimal user engagement. In addition, studies have found increased usage of educational bot software and increased knowledge gains following exposure to educational WhatsApp messages in females compared to males.^60,62 As such, personalized risk communications may be necessary for successful technology-based public health interventions related to COVID-19. None of the DHIs, however, employed methodologies that enabled examination of the needs of underserved populations who may have limited clinical access, or minority groups with tight-knit communities, which may be more vulnerable to online and offline misinformation.^84–87

Limitations

Our scoping review has certain limitations. We selected studies to review based on the experimental design. The evidence of how to support people in interacting with misinformation, spot misinformation, and stop its spread, however, also may be generated from questionnaires, behavioral data analysis, observational studies, or qualitative inquiry.^74,88–90 Although we used 3 well-established transdisciplinary repositories to collect articles, it may be possible that we omitted a few relevant publications not indexed in these databases. We reviewed articles published in peer-reviewed scholarly journals which means that potentially relevant evidence reported in formats more oriented toward practitioners and policymakers could have been overlooked. Future scoping reviews can present a more comprehensive view of the research area by exploring domain areas, search terms, and DHIs outside the domain of COVID-19. Although critical appraisal and meta-synthesis of articles are not required in scoping reviews, we performed an initial summarization. Future works, however, should consider such analyses so that they can be utilized to promote novel research questions, identify domain gaps, and improve quality in the literature.

CONCLUSION

This paper contributes a summary of a majority of the latest efforts to combat COVID-19 misinformation and disinformation around the globe. Due to the evolving nature of the pandemic researchers face many resource and time constraints. Nevertheless, they are able to develop and implement creative technological solutions for mitigating the impact of the misinfodemic. A lack of evidence-based and theory-informed interventions, however, prevents conclusions from being drawn regarding the effectiveness of such interventions. Future research efforts are needed to evaluate the impact of these DHIs on users in real-time situations. Institutions should take the initiative to provide social media training and domain summarization to healthcare professionals, to enable them to interact professionally through online platforms, as well as support their interactions with patients during clinical encounters. The internet has democratized information access, however, the general public within a country or throughout the globe may not have the same level of awareness, skills, or tools to mitigate their vulnerability to misinformation and disinformation exposure and its negative consequences on preventive behaviors. Although all individuals are exposed and susceptible to the adverse effects of COVID-19 misinformation and disinformation, populations that experience health disparities may face elevated harm, and equitable DHIs can play an important role in addressing this issue.

FUNDING

The work reported in this publication was supported by the National Library of Medicine of the National Institutes of Health under Award Numbers 1R01LM012974-01A1 and 3R01LM012974-02S1. This work was supported by the National Institute of General Medical Sciences of NIH (5T32GM008792). The content is the sole responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

AUTHOR CONTRIBUTIONS

All authors contributed equally to the study design, literature synthesis, text generation, and review of the manuscript. KC and SM led the development of the manuscript structure and theme identification for analysis.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

None declared.

DATA AVAILABILITY

The underlying data for this paper are available in the paper and in the online Supplementary Material.