Benefits and Pitfalls of Debunking Interventions to Counter mRNA Vaccination Misinformation During the COVID-19 Pandemic

Abstract

Misinformation about mRNA vaccination is a barrier in the global fight against the COVID-19 pandemic. Thus, authorities often rely on text-based refutations as a countermeasure. In two experiments (N = 2,444), text-based refutations effectively reduced the belief in misinformation and immunized participants against the impact of a misleading social media post. However, a follow-up (N = 817) questions the longevity of these debunking and prebunking effects. Moreover, the studies reveal potential pitfalls by showing a row of unintended effects of the refutations (lacking effect on intentions, backfire-effects among religious groups, and biased judgments when omitting information about vaccine side effects).

The development of effective vaccines was a milestone and one of the greatest challenges in the pandemic response to the coronavirus disease 2019 (COVID-19). Among the most promising and now widely approved vaccines are so-called messenger RNA (mRNA) vaccines. In contrast to conventional protein-based vaccines, mRNA vaccines contain genetic information for a single element of the virus rather than live attenuated or inactivated pathogens (Abbasi, 2020; Pardi et al., 2018). The approved COVID-19 mRNA vaccines have been shown to be effective in reducing the probability of a COVID-19 infection, reducing the transmission of the disease and, if infected, reducing the severity of the disease progression (Centers for Disease Control and Prevention [CDC], 2021). In addition, mRNA vaccines are considered beneficial over conventional vaccines because, among other things, these new types of vaccines allow rapid manufacturing, which is essential for a successful pandemic response (Jackson et al., 2020; Pardi et al., 2018). However, the mere availability of promising prevention measures does not guarantee public acceptance of such measures. In fact, in many countries it remains questionable whether the public acceptance of COVID-19 vaccines will be sufficient to achieve a coverage rate that will allow pandemic control (MacPherson, 2020). Thus, vaccine acceptance is widely discussed as another major challenge after the development of an effective COVID-19 vaccine itself (Habersaat et al., 2020; Lazarus et al., 2021).

One reason why vaccine acceptance is likely to remain too low despite overwhelming scientific evidence in favor of vaccination is the impact of misinformation on individuals’ judgments and health decision-making (Betsch, Schmid, et al., 2018). According to general health behavior theories (Ajzen, 1991) and domain-specific models of vaccine hesitancy (Betsch, Schmid, et al., 2018), misinformation can influence specific attitudinal beliefs about vaccination and thereby reduce an individual’s intention to get vaccinated. For example, if an individual encounters the widespread misinformation that mRNA vaccines alter the human genome (Löffler, 2021), then this can reduce the individual’s confidence in the safety of mRNA vaccines and ultimately reduce the intention to get vaccinated against COVID-19. The theoretically negative impact of such misinformation has been demonstrated in empirical evaluations during the course of the pandemic (Loomba et al., 2021).

However, individuals who are facing misinformation are not necessarily helpless. For example, individuals can evaluate the credibility of prominent dubious claims by consulting external sources (Farrell et al., 2019; Walter et al., 2020). Some external sources such as fact-checkers and health authorities even support individuals’ credibility evaluations by providing explicit refutations of prominent misinformation. For example, scientists have refuted the misinformation that mRNA vaccines alter the human genome because, among other reasons, the mRNA does not enter the nucleus where DNA is located (Hotez et al., 2021). Thus, the World Health Organization (WHO) presents this refutation as part of their COVID-19 MythBusters initiative on their webpages for the general public (WHO, 2021). Like WHO, many other fact-checkers rely on text-based approaches for the correction of misinformation. The ways in which text-based refutations are usually distributed vary from social media posts (Kim & Walker, 2020) and information leaflets at medical practices (Betsch, Rossmann, et al., 2018) to serious online games (Roozenbeek & van der Linden, 2019). Although there is some debate surrounding the unintended effects of refutations, challenging misinformation rather than ignoring it has been demonstrated repeatedly to be the superior strategy in mitigating its potential damage (Chan et al., 2017; Roozenbeek & van der Linden, 2019; Schmid & Betsch, 2019; Walter et al., 2021). Thus, providing text-based refutations of COVID-19 misinformation during the pandemic should support the public in evaluating the credibility of dubious claims. In fact, a recent evaluation of the MythBusters initiative revealed promising results for their effectiveness in social media (Vraga & Bode, 2021).

However, empirical evidence on the effectiveness of text-based refutations of vaccination misinformation during a pandemic is scarce and the extent to which corrective effects endure over time remains underexplored. Moreover, the introduction of COVID-19 mRNA vaccines may provide specific challenges for the design of text-based refutations of vaccination misinformation that are rarely addressed in previous research on vaccination misinformation. First, a text-based refutation about new mRNA vaccines may challenge the aversion to new technologies and scientific innovations that is assumed to be particularly prevalent among religious individuals (Allum et al., 2014; Brossard et al., 2009; Nisbet, 2005). Second, a text-based refutation about new mRNA vaccines based on scientific evidence may challenge the belief that truth can only be found through personal experience—a belief that is assumed to be particularly prevalent among spiritual individuals (Rutjens & van der Lee, 2020). Thus, both religiosity and spirituality may be barriers for the effectiveness of text-based refutations. Thus, the current studies aim to provide empirical evidence on (a) the effectiveness of text-based refutations of misinformation regarding COVID-19 mRNA vaccines during the COVID-19 pandemic and (b) examine the role of religiosity and spirituality in the success of refuting vaccination misinformation.

Designing Text-Based Refutations

Designing effective refutations does not mean simply to label misinformation as false. In fact, research suggests that refutations are most promising when they consist of specific components (Lewandowsky et al., 2012; van der Linden et al., 2017). One key component is to introduce a clear warning of the falsehood of the misinformation and to name the misinformation (Ecker et al., 2020; Lewandowsky et al., 2012, 2020). While repeating misinformation is inadvisable, mentioning the misinformation a single time can help belief-updating, that is the formation of a new attitude or knowledge base (Ecker et al., 2017). In addition, adding a warning that is directly connected to the misinformation makes it unlikely that the correction will be ignored or that the misinformation will be misinterpreted as a fact (Lewandowsky et al., 2012, 2020). Another key component is to provide a detailed causal explanation of why the misinformation is incorrect (Ecker et al., 2020; Lewandowsky et al., 2020). According to research on mental models, refutations that only state the falsehood of misinformation do not provide enough detail (Lewandowsky et al., 2012). As a consequence, individuals can experience difficulties in remembering which proposition was correct or incorrect (Chan et al., 2017). The two outlined components are key parts of recommended best-practice for effective refutations and should also be applicable to the context of misinformation around COVID-19 mRNA vaccines. Thus, the refutation hypothesis predicts that debunking texts that contain the relevant components will reduce individuals’ credibility judgments of prominent misinformation about COVID-19 mRNA vaccines compared to a control group. Based on studies on the long-term effectiveness of refutations (Maertens et al., 2021), we also expect that the refutation effect will be detectable after a 2-month delay.

Refutations as Prevention Measures

Refutations generally work via two distinct processes of belief updating. On one hand, refutations may serve as an intervention, given that individuals already believe in the misinformation; in this way, they serve as debunking (Chan et al., 2017; Lewandowsky et al., 2012). On the other hand, they may serve as prevention when individuals are entirely unfamiliar with the misinformation or when they encounter persuasive misinformation after receiving the refutation; in this way, they may serve as prebunking (Chan et al., 2017; Lewandowsky & van der Linden, 2021). For example, individuals reading the refutation of the MythBuster initiative on the WHO webpages may benefit from it because the refutation either corrects their false belief that mRNA vaccines alter the human genome (i.e., debunking), or it warns unknowing people that this misinformation is spreading and protects against future encounters with it (i.e., prebunking). While introducing a clear warning of the falsehood of the misinformation and providing a detailed causal explanation of why the misinformation is incorrect are components of effective debunking texts, these components may also serve as prebunking for reasons outlined in the following paragraph.

The design of effective prebunking interventions is usually based on two components that originate from inoculation theory (Lewandowsky & van der Linden, 2021; McGuire & Papageorgis, 1961). First, a weakened form of the misinformation is introduced, and individuals are warned that they may encounter this misinformation. This threat induction puts individuals in a state of arousal that makes them more sensitive to being a potential target for misinformation (Compton & Ivanov, 2012). Second, it provides a refutation of the misinformation, giving individuals strong counterarguments to use when they are exposed to misinformation at a later point in time. This process is called refutational preemption (Lewandowsky & van der Linden, 2021). Following this rationale, introducing a clear warning regarding the misinformation in a debunking text may serve as a threat induction, while providing a detailed causal explanation of why the misinformation is incorrect may serve as refutational preemption. Thus, the prebunking hypothesis predicts that texts that contain relevant components of effective debunking will mitigate the impact of subsequent encounters with prominent misinformation about COVID-19 mRNA vaccines on individuals’ credibility judgments compared to a control group. According to previous research, misinformation on vaccination also decreases individuals’ confidence in vaccination and individuals’ intentions to get vaccinated (Loomba et al., 2021; Schmid & Betsch, 2019). Thus, we also expect that the intervention texts reduce the negative impact of misinformation on confidence and intention measures.

Religiosity, Spirituality, and Unintended Effects

Despite the overall recommendation to challenge rather than ignore misinformation some researchers have raised concerns about potential backfire-effects when refuting misinformation. Backfire effects describe the counterintuitive finding that some individuals report higher (instead of lower) beliefs in misinformation after having received the refutation of the very same misinformation (Lewandowsky et al., 2020; Swire-Thompson et al., 2020). However, several studies have found no evidence for the presence of backfire-effects (Ecker et al., 2017, 2019; Wood & Porter, 2019). Thus, a recent review has concluded that backfire effects are not a robust empirical phenomenon and that it is unlikely that corrections cause such unintended effects on the group level (Swire-Thompson et al., 2020). While backfire effects seem to be a rare phenomenon, corrections may still prove ineffective under certain conditions. For example, refuting misinformation seems to be least effective when the misinformation aligns with the worldviews of the audience (van der Linden et al., 2017). This may also be an issue when refuting misinformation about mRNA vaccines (Dragojlovic & Einsiedel, 2013). In fact, increased religiosity has repeatedly been found to be associated with increased opposition to emerging technologies like stem-cell research, nanotechnology, and gene-technology (Allum et al., 2014; Brossard et al., 2009; Nisbet, 2005). The typical narratives are either that the new technology “interferes with God’s creation” or that the technology is deemed “unnatural” (Dragojlovic & Einsiedel, 2013). While many religious medical expert communities and leaders of world religions have explicitly expressed their support toward COVID-19 vaccination (British Islamic Medical Association, 2020; Card, 2020; Jewish Medical Doctors working in UK, 2020), some concerns with the emerging mRNA technologies may remain as objections to vaccination among religious individuals or single religious leaders. These objections, in turn, may foster motivated reasoning and lower the effectiveness of refutations of misinformation about mRNA vaccines. Thus, the religiosity as moderator hypothesis predicts that the effectiveness of debunking texts about mRNA vaccines decreases with increasing levels of individual religiosity.

In addition, some recent research suggests that spirituality plays a central role in explaining the differences in overall levels of support for vaccination (Rutjens et al., 2022). Spiritual individuals do not necessarily feel a sense of belonging to a particular traditional religion or church and tend to see their own inner selves as the gateway to truth (Rutjens & van der Lee, 2020). Therefore, spiritual individuals’ own experiences are often considered the primary method of discovery, and aggregate research data seem of little relevance to their own belief-updating (Rutjens et al., 2022). Measures of spirituality may thus cover a different spectrum of believers who oppose vaccination as part of a specific worldview rather than an objection to emerging human-made technologies. Thus, the spirituality as moderator hypothesis predicts that the effectiveness of debunking texts about mRNA vaccines decreases with increasing levels of spirituality.

Other unintended effects may occur when evaluating interventions to counter misinformation. For example, a debunking text may effectively reduce credibility judgments simply because it demands that participants distrust any subsequent information (Roozenbeek & van der Linden, 2019). Moreover, debunking texts may be designed in such a way that they talk people into vaccination, rather than informing them about the pros and cons of vaccination. Thus, the unintended effects research question explores the impact of debunking texts on credibility judgments of unrelated facts and misleading pro-vaccine statements.

Overview

Based on two online experiments (N = 2,444), this contribution provides insights into whether debunking texts can reduce individuals’ credibility judgments of prominent misinformation about COVID-19 mRNA vaccines (Experiment 1) and whether debunking texts can also serve as prebunking against the impact of subsequent misinformation (Experiment 2). Experiment 1 includes a 2-month follow-up measure of credibility to assess the long-term effects of debunking. Moreover, Experiment 2 includes the confidence in the safety of mRNA vaccines and the intention to get vaccinated against COVID-19 with mRNA-vaccines as additional outcome measures. Furthermore, we analyzed the role of religiosity (Experiment 1) and spirituality (Experiment 2) as potential moderators of the effectiveness of debunking texts and explored potential unintended effects of debunking texts.

Experiment 1

In this experiment, we tested the refutation hypotheses. We expected that debunking texts that contain relevant components of effective debunking reduce individuals’ credibility judgments of prominent misinformation about COVID-19 mRNA vaccines compared to a control group. We expected that this effect would occur immediately upon receipt of the debunking and after a delay of 2 months. Furthermore, we tested the religiosity as moderator hypothesis. That is, we expected that the effectiveness of debunking texts about COVID-19 mRNA vaccines decreases with increasing levels of religiosity. Finally, we explored the unintended effects research question. That is, we explored whether debunking texts also affect individuals’ credibility judgments of unrelated facts or misleading pro-vaccine statements about COVID-19 mRNA vaccines compared to a control group. Following best practices, we only label preregistered hypothesis as confirmatory hypothesis and all other analyses as exploratory research questions (Wagenmakers et al., 2012).

Methods

The hypotheses, measures, and analyses for this experiment were preregistered. The preregistration protocol, data, and script for all analyses can be accessed from the open-access repository OSF (Schmid & Betsch, 2022).

Design and Participants

Each participant was assigned randomly to one of two conditions, resulting from the 2 (debunking vs. control; between subjects) × 3 (measurement before vs. immediately after the debunking versus 2 months after the debunking; within subjects) mixed design. An a priori power analysis conducted with G*Power (Faul et al., 2009) revealed a minimum of n = 835 participants to detect an effect size of d = 0.25 (alpha = 0.05) with a power of 95% at the 2-month follow-up questionnaire. We expected a high drop-out rate of around 33% because the experiment was embedded in a larger survey; thus, we aimed to recruit at least 1,250 individuals for the onset of the experiment. The expected effect size was informed by the lower end of the confidence interval of the effectiveness of corrections from a previous meta-analysis (Walter et al., 2021). The external panel provider respondi.de invited and incentivized the participants. The provider used a quota sampling procedure to match the distribution of gender × age with that of the German population (Supplementary Table 1). A total of N = 1,387 individuals participated in the online experiment, and n = 821 individuals finished the 2-month follow-up. Seven individuals participated twice at the same measurement date. Data of the second participations were excluded from analyses and data at the 2-month follow-up was only analyzed if participants did not participate multiple times at the first measurement date. Thus, the following analyses use the final sample sizes of n = 1,382 (age Mean M = 45.40, Standard Deviation SD = 15.69; 49.7% female) and n = 817 (age M = 48.02, SD = 14.78; 50.1% female) for the follow-up. For the 2-month follow-up, we missed our planned sample size by n = 18 and reached a final power of 94.5% to detect a small effect size, which is still higher than for most psychological studies (Bakker et al., 2012). Dropping out was neither significantly correlated with subjective credibility judgments of misinformation at baseline (Pearson’s r = −.01, p = .834), nor with differences in credibility judgments between baseline and second measurement (r = −.01, p = .846), nor with experimental condition (r = −.00, p = .894). Thus, regarding the primary variables under investigation, we did not find evidence for any systematic dropout. In the follow-up sample, some participants were already vaccinated (n = 33) as the German vaccination campaign started on December 26, 2020. We did not exclude vaccinated individuals from analyses, but provide confirmatory analyses without vaccinated individuals as a robustness check.

Procedure

Participants indicated their self-reported religiosity and the perceived credibility of four fictitious news headlines. Depending on conditions, they then either received the debunking text or a control text (Figure 1). The credibility measures were again assessed immediately after the manipulation and at a 2-months follow-up.

Figure 1.

Excerpts of the experimental stimuli.

Note. Participants in Experiment 1 received facts about mRNA vaccines both in the control and the intervention conditions. In the debunking/prebunking conditions participants additionally received a warning regarding the misinformation and an explanation why the misinformation was wrong (Lewandowsky et al., 2012). Participants in the control condition of Experiment 2 received an unrelated text with no reference to vaccines. Full materials can be accessed in Supplementary Table 2.

Materials

The debunking text specifically addressed the myth that mRNA vaccines can alter human DNA. The text included (a) a warning of the falsehood of the misinformation and (b) provided a detailed explanation of why the misinformation was incorrect. Figure 1 provides an excerpt of the core components of the full debunking text. The content of the debunking text was designed in consultation with infectious disease specialists from the Bernhard Nocht Institute for Tropical Medicine, Hamburg, Germany. The full text is available in Supplementary Table 2.

The text in the control condition was identical regarding the information about the mRNA vaccines but missed the two central components of effective debunking: information that points to misinformation and an explanation of why the misinformation is wrong (Figure 1).

Measures

The relevant measures are listed below and ordered as they appeared in the questionnaire. Mean and standard deviations of primary measures stratified by condition are reported as Supplementary Table 3.

Religiosity

Self-reported religiosity was measured with a validated German version (Klumparendt & Drenckhan, 2014) of the Religious Commitment Scale (A. B. Cohen et al., 2006). The original scale measures religious commitment with six items (e.g., How religious are you?). The authors of the German version omitted the sixth item (How spiritual are you?) because this item measures spirituality rather than religiosity and, in line with our own thinking, spirituality is considered as a separate trait. Thus, we used the mean of the 5-item solution as an indicator of individuals’ religiosity. The commitment was measured on a verbally anchored 7-point scale ranging from 1 = not at all to 7 = very much. Reliability analysis in the final sample showed a sufficiently high reliability score for the religiosity scale, Cronbach’s alpha = .96.

Credibility judgments of (mis)information

Individuals judged the perceived credibility of a single false antivaccine news headline (mRNA vaccine alters the human genome) on a 7-point scale ranging from 1 = not at all credible to 7 = very credible. This item served as the primary outcome variable. The approach of judging news headlines as a primary measure was adapted from previous prebunking studies (Roozenbeek & van der Linden, 2019). We also measured the credibility of two factually correct statements (The next federal election in Germany is in 2021; The Summer Olympics are to be held in Tokyo) to test whether the debunking intervention merely demanded that participants judge any headline as not at all credible. Moreover, we measured the credibility of a misleading pro-vaccine headline (mRNA vaccine does not cause any side effects) to test whether the debunking intervention simply persuaded participants to falsely believe that side effects of vaccination do not exist. All credibility judgments were measured before and immediately and 2 months after individuals received the debunking or control text.

Additional variables

Participants also answered other pandemic-related questions reported elsewhere, as the experiment was embedded in a larger COVID-19 survey. The full list of variables of the survey can be accessed here: http://dx.doi.org/10.23668/psycharchives.4398.

Analysis

We used linear regression models to test the hypotheses and explore the research question. A significance level of α = .05 was employed for all analyses. The models were calculated with the PROCESS macro in R (Model 1: Hayes, 2013). Tables displaying the results of the models are presented in Supplementary Table 4 to 7. Predictor variables were mean-centered. Thus, the regression coefficients B in the Results section represent estimated unstandardized mean differences between the debunking group and the control group on outcome measures when controlling for all other predictors in the models. All models were controlled for baseline values of the outcome measures. In addition to the outlined regression models, we also preregistered separate linear models without religiosity as a predictor to test the refutation hypothesis in Experiment 1. However, these separate models were dropped due to scarcity in the Results section of this article because the full regression models include all effects of interest. We report the results of the separate linear models in Supplementary Table 8 for transparency. Results for the refutation hypotheses did not change when dropping religiosity as a predictor from the model. All continuous variables are reported as percentages of maximum possible scores of the original scales (POMP: P. Cohen et al., 1999). This linear transformation simplifies the interpretation of model parameters because all continuous scales range from 0 to 100. For example, a value of 50 on a POMP scale can be translated into 50% of the maximum possible score of the original scale independent of its original range.

Results

Test of Refutation Hypotheses

As shown in Figure 2A, participants who received the debunking text rated the false antivaccine news headline that mRNA vaccines alter the human genome as less credible compared to the control condition, b = −7.06, [−9.81, −4.31], t(1377) = −5.03, p < .001. This is in line with the refutation hypothesis. However, we did not find a significant difference in credibility judgments between conditions at the 2-months follow-up, b = −1.01, [−4.48, 2.47], t(810) = −0.57, p = .569. Explorative analyses revealed that this absence of a long-term effect did not result from a decay in the effectiveness of the debunking. In fact, as can be seen in Figure 2A, credibility judgments were slightly lower at the 2-months follow-up, compared to the immediate measurement, t(406) = −0.87, p = .386, Cohen’s d = .04. This effect occurred in both conditions alike, that is, the control group also rated the misinformation as less credible when asked 2 months later, t(409) = −5.74, p < .001, d = .28, which neutralized the overall long-term advantage of the debunking intervention. Thus, the debunking text effectively decreased the believe in the false antivaccine news headline, but this effect was only apparent in the short term. General long-term effects may be masked by learning effects in the control group.

Figure 2.

Effect of text-based debunking interventions on credibility judgments of misinformation in Experiment 1 (A) and effect of text-based prebunking on credibility judgments of misinformation, confidence in mRNA vaccines, and the intention to get vaccinated in Experiment 2 (B).

Note. Point estimates represent estimated means of primary outcomes, controlled for baseline values and model-specific moderators. Point estimates are reported in POMP values (percentage of maximum possible score). That is, scales range from 0 to 100, independent of the range of the original scale. Error bars are 95% confidence intervals.

Test of Religiosity as Moderator Hypothesis

As a next step, we tested the religiosity as moderator hypothesis, that is, whether effects of the debunking text varied by individuals’ religiosity (Figure 3A–B). As shown in Figure 3A, we did not find that the effectiveness of debunking on individuals’ credibility judgments was moderated by religiosity—at least in the short term, b = 0.01, [−0.09, 0.10], t(1377) = 0.10, p = .918. At the 2-month follow-up, however, we found a significant interaction effect between religiosity and debunking, b = 0.17, [0.05, 0.30], t(810) = 2.80, p = .005. People with higher religiosity demonstrated a backfire effect, such that after 2 months, highly religious people in the debunking condition judged misinformation as more credible than people in the control group (Figure 3B). Thus, while in the short run, debunking texts about mRNA vaccines seemed to work well, they may have triggered unintended effects in the long run when applied to highly religious groups.

Figure 3.

Credibility judgments of misinformation for intervention and control condition at different levels of religiosity for the immediate measurement (Experiment 1: A), the 2-month follow-up (Experiment 1: B), and at different levels of individuals’ spirituality (Experiment 2: C).

Note. Lines represent estimated regression lines and shades are 95% confidence intervals. Estimates are reported in POMP values (percentage of maximum possible score). That is, scales range from 0 to 100, independent of the range of the original scale.

The follow-up analyses were repeated without vaccinated participants (Supplementary Table 9). The patterns of results did not differ.

Exploration of Unintended Effects Research Question

To test whether the debunking intervention merely demanded participants to judge any headline as less credible, we asked individuals to also judge the credibility of factually correct statements before and after receiving the debunking. We did not find a significant decrease in credibility judgments for factually correct statements in the debunking condition, compared to the control condition (Supplementary Tables 5–6). In fact, participants in the debunking condition rated the factually correct statement about the Olympics as more credible compared to the control condition at the 2-month follow-up (Supplementary Table 6). Moreover, participants who received the debunking reported higher credibility judgments of the misleading news headline that the “mRNA vaccine does not cause any side effects” compared to the control group, b = 3.77, [1.28, 6.26], t(1377) = 2.97, p = .003. Thus, the debunking text may have created the false impression that because mRNA vaccines do not alter human DNA, they may have no side effects at all. However, we did not find the unintended effect of debunking at the 2-month follow-up, b = −0.77, [−4.49, 2.95], t(810) = −0.41, p = .684. An additional unsuccessful attempt to address the unintended effect with an additional fact box intervention at the 2-months follow-up is reported in Supplementary Results and Supplementary Table 10.

Additional Explorative Analyses

The religiosity as moderator hypothesis presumes that religiosity favors a worldview that is associated with lower support for vaccination. We indeed found that higher religiosity was associated with higher credibility judgments of misinformation at baseline (r = .106, p = <.001; Supplementary Figure 1A). This finding supports the notion that there is a positive association between religiosity and belief in misinformation about mRNA vaccination.

Experiment 2

In the second experiment, we tested the prebunking hypothesis. We expected that texts that contain relevant components of effective debunking will mitigate the impact of subsequent encounters with prominent misinformation about COVID-19 mRNA vaccines on individuals’ credibility judgments compared to a control group. Furthermore, we tested the spirituality as moderator hypothesis. That is, we expected that the effectiveness of the corrective text about COVID-19 mRNA vaccines decreases with increasing levels of spirituality. In line with Experiment 1, we explored the unintended effects research question. Moreover, Experiment 2 includes the confidence in the safety of mRNA vaccines and the intention to get vaccinated against COVID-19 with mRNA-vaccines as additional outcome measures. Again, preregistered hypothesis are labeled as confirmatory hypothesis and all other analyses as exploratory research questions.

Method

The hypotheses, measures, and analyses for this experiment were preregistered. The preregistration protocol, data, and script for all analyses can be accessed from the open-access repository OSF (Schmid & Betsch, 2022).

Design and Participants

Each participant was assigned randomly to one of two conditions, resulting from the 2 (prebunking vs. control; between subjects) × 2 (measurement before prebunking versus immediately after the misinformation; within subjects) mixed design. Again, G*Power (Faul et al., 2009) was used to estimate the required sample size. Compared to Experiment 1, we aimed to detect smaller effect sizes of d = .20 with a high statistical power of 90% and thus aimed to recruit 1,054 unvaccinated individuals (alpha = 0.05). Because we were interested in individuals’ intentions to get vaccinated, we recruited only unvaccinated individuals for Experiment 2. A total of 1,056 individuals participated in the online experiment. Two individuals participated twice. Data of the second participations were excluded from analyses, resulting in the final sample size of n = 1,054 (age M = 45.49, SD = 15.35; 50.3% female).

Procedure

Participants first reported their spirituality and then judged the credibility of the same news headlines as participants in Experiment 1. As additional dependent variables, they then indicated their confidence in the mRNA vaccine and their intention to get vaccinated against COVID-19 as the baseline measure. Depending on conditions, they then either received the prebunking text or a control text (Figure 1), followed by the misinformation in the form of a misleading social media post. The same intervention text from Experiment 1 served as prebunking with the expectation that having read the prebunking can prevent the negative effect of the misinformation. The credibility, confidence, and intention measures were then repeated immediately after receiving the misinformation. Finally, participants received an explicit debriefing that corrected the misinformation and provided official information about the safety of mRNA vaccines.

Materials

The prebunking text was identical to the debunking text of Experiment 1 (Figure 1), with one exception. As a precaution to avoid an increase in the false belief that mRNA vaccines do not cause any side effects, we added the disclaimer that “As with conventional vaccines, vaccine reactions and side effects can occur after mRNA vaccination. According to the Robert Koch-Institute, the most common reactions are pain at the injection site, fatigue and headaches.”

In this experiment, we also aimed to detect differences in individuals’ confidence in mRNA vaccines and their behavioral intention to get vaccinated. Following the rationale of general health behavior theories (Ajzen, 1991), the effect of the debunking text should be larger on credibility and confidence judgments and smaller on actual behavioral intentions. Thus, we replaced the conservative control condition from Experiment 1 with a less-challenging control condition to increase the power for detecting the weaker effects on the intention to get vaccinated (column 3 in Figure 1). The text of the new control condition was about technical information of updates for a Corona warning application for mobile phones and did not refer to mRNA vaccines whatsoever (Supplementary Table 2). As misinformation, all participants received an adaptation of a social media post that circulated in German social media groups at the time of the study and contained the claim that mRNA vaccines can alter the human genome. A translated version of the misinformation is provided as Supplementary Table 11.

Measures

The relevant measures are listed below and ordered as they appeared in the questionnaire. Mean and standard deviations of primary measures stratified by condition are reported in Supplementary Table 3.

Spirituality

Participants rated the following two items “To what extent do you consider yourself to be a spiritual person?” and “To what extent do others consider you to be a spiritual person?” on verbally anchored 7-point scales ranging from 1 = not at all to 7 = to a strong extent. The selected items were adapted from Rutjens and van der Lee (2020). Reliability analysis in the final sample showed a sufficiently high reliability score for the spirituality scale, Spearman Brown coefficient = .91.

Credibility of (mis)information

Individuals judged the perceived credibility of the same false and factually correct statements as in Experiment 1. The factually correct statement about Olympic games was dropped from Experiment 2 because, right before data collection, the media started to report about the possibility that the Olympic games may not take place at all.

Confidence in COVID-19 vaccination

Individuals judged their confidence in COVID-19 mRNA vaccines with a single item (I am completely confident that mRNA vaccines are safe) rated on a verbally anchored 7-point scale from 1 = strongly disagree to 7 = strongly agree. The selection of this item was informed by Betsch, Schmid, et al. (2018) and adapted to COVID-19.

Intention to get vaccinated

Individuals rated their intention to get vaccinated with an mRNA vaccine on a verbally anchored 7-point scale ranging from 1 = I would definitely not get vaccinated to 7 = I would definitely get vaccinated.

Additional variables

We also measured participants’ judgments of other, vector-based COVID-19 vaccines. This allowed us to explore whether the misinformation had a specific impact on the attitudes and intentions toward mRNA vaccines or whether there were spillover effects to vector-based vaccines, too. These additional analyses are reported in Supplementary Methods and Results.

Analysis

We used the same preregistered linear regression models to test the predicted main effects and interaction effects as described in Experiment 1. The models only differed in that religiosity was replaced by spirituality as the moderator variable. Moreover, additional models including confidence in vaccination and intention to get vaccinated as outcome measures were analyzed. Tables displaying the results of all models are presented in Supplementary Tables 3, 4, 6, and 12.

Results

Test of Prebunking Hypothesis

As can be seen in Figure 2B, the prebunking text mitigated the impact of the subsequent antivaccination social media post. The prebunking text effectively served as an “immunization” against misinformation, as credibility judgments in the prebunking condition were lower than in the control condition, b = −3.01, [−5.66, −0.36], t(1,049) = −2.23, p = .026. This benefit was, however, not observed for individuals’ intention to get vaccinated, b = 0.78, [−0.92, 2.48], t(1,049) = 0.90, p = .368. There was also no effect of the prebunking on confidence in mRNA vaccines, b = 1.74, [−0.37, 3.85], t(1049) = 1.62, p = .106.

Spirituality as Moderator Hypothesis

We did not find that the effectiveness of prebunking on individuals’ credibility judgments was moderated by spirituality, b = 0.04, [−0.05, 0.14], t(1,049) = 0.89, p = .375 (Figure 3C). Likewise, we did not find any statistically significant interaction effects of prebunking and spirituality on individuals’ intention or confidence in vaccination (Supplementary Table 12).

Exploration of Unintended Effects Research Question

Similar to Experiment 1, we did not find a significant decrease in credibility judgments for the factually correct statement in the prebunking condition as compared to the control condition (Supplementary Table 5). Unlike in Experiment 1, and potentially owing to the additional disclaimer that pointed to nonsevere side effects, we found no evidence that individuals who received the prebunking rated the misleading pro-vaccine statement as more credible (Supplementary Table 7).

Additional Explorative Analyses

The spirituality as moderator hypothesis presumes that spirituality favors worldviews that are associated with lower support for vaccination. We indeed found that higher spirituality was associated with higher credibility judgments of misinformation at the baseline measurement (r = .212, p ≤ .001), lower levels of confidence in vaccination (r = −.118, p ≤ .001) and a lower intention to get vaccinated (r = −.118, p ≤ .001; Supplementary Figure 1B). While spirituality may not moderate the effectiveness of prebunking, these findings support the notion that spirituality, just like religiosity (Experiment 1), is positively associated with the belief in misinformation about mRNA vaccination.

In some cases, prebunking effects may not be detected because the misinformation itself has no effect. Comparing premeasures and postmeasures in the control condition, however, revealed that in this experiment, the social media message indeed had an effect: it increased the perceived credibility of the false headline, t(515) = 2.07, p = .039, d = 0.09, and decreased the intention to get vaccinated with an mRNA vaccine, t(515) = −3.52, p < .001, d = 0.15. No such effect was found on individuals’ confidence in mRNA vaccines, t(515) = −0.04, p = .968, d = < 0.01. Thus, protective effects from the prebunking can be assumed, at least for judgments of the credibility of false headlines and the intention to get vaccinated with an mRNA vaccine.

Discussion

When new scientific innovations such as mRNA vaccines are introduced, they often provide a breeding ground for doubts, misunderstandings and misinformation. Results from the refutation hypothesis in Experiment 1 showed that text-based debunking attempts can effectively reduce the perceived credibility of misinformation about mRNA vaccines—at least in the short run. We did not observe a benefit of the text-based debunking on credibility judgments after a 2-month delay. However, the results provide no evidence that this absence of a benefit was due to a decay of the effect in the intervention group. Rather, the result of lower credibility judgments of the control group at the 2-months follow-up may account for this. This change in the control group’s perception may not be surprising, given the fact that several health authorities began to provide corrective information about the misinformation that mRNA alters human DNA in several media channels in Germany (e.g., Robert Koch-Institut [RKI], 2020)—corrective information that was available to both groups between the two measurements of Experiment 1. Thus, the results of Experiment 1 strengthen previous findings about the effectiveness of refutations (Chan et al., 2017; Vraga & Bode, 2021; Walter et al., 2020) by revealing their usefulness in the midst of a pandemic. In the light of the findings, we recommend to maintain and increase current active efforts by health authorities and fact-checkers to provide explicit corrections of misinformation (e.g., MythBusters; WHO, 2021), since they are an effective and economical measure of health communication during a pandemic.

The recommendation to use debunkings in times of a health crisis is also supported by theoretical considerations. According to research on the theory of motivated reasoning, people are motivated to be accurate, that is, they attend to relevant information more carefully when they have more incentive to invest effort in processing complex information (Bolsen & Druckman, 2015; Taber & Lodge, 2006). The possibility of preventing the risks of a potentially lethal COVID-19 infection may provide a good-enough incentive to invest effort into processing debunking texts about vaccination during a pandemic.

The results from the religiosity as moderator hypothesis suggest that accuracy may not be the only prevalent motivation when processing debunking about mRNA vaccination misinformation. In fact, we found evidence for a backfire effect of text-based debunking among religious individuals 2 months after debunking. According to research on the theory of motivated reasoning, people are motivated to arrive at a particular conclusion, that is, they attend more to information that is consistent with their prior beliefs when their worldviews are challenged by new information (Bolsen & Druckman, 2015, 2018). A debunking text about new mRNA vaccines may have challenged the aversion to new technologies and scientific innovations among religious individuals. As a response, they may have generated counterarguments against debunking, thus strengthening their initial worldviews.

But why does this effect occur only after 2 months and not immediately after reading the information? There is evidence that individuals who are motivated to arrive at a particular conclusion still do not openly lie to themselves (Taber & Lodge, 2006) but that a desired conclusion is drawn only if they can maintain the illusion of objectivity (Kunda, 1990). That is, they search for an apparently objective justification for their conclusion and do not realize that the process of building that justification is biased by focusing on specific memories that only confirm their conclusion (Kunda, 1990; Taber & Lodge, 2006). Maintaining this illusion may be more difficult immediately after being confronted with a disconfirming debunking text than after a delay of 2 months—when the debunking can be ignored and a memory search can be directed toward counterarguments that support the aversion against new technologies such as mRNA vaccination. In line with this, previous researchers have suggested that worldview backfire effects may, in fact, only occur after some delay (Wittenberg & Berinsky, 2020).

It is important to note that the sample in Experiment 1 was not very religious, and thus confirmatory replications, including highly religious groups, are needed to confirm the existence of a backfire effect among this target group. Moreover, replicating this finding is necessary before drawing conclusions for practitioners, since backfire effects after debunking are considered a rare phenomenon, and replications of previous backfire effects have failed (Haglin, 2017). Furthermore, using a single-item measure (as in this study) can further be unreliable (Swire-Thompson et al., 2020), which can also create false impressions of backfire effects (Swire-Thompson et al., 2022).

During the absence of further studies, giving religious leaders a voice when designing mRNA debunking texts may be useful (Valente & Pumpuang, 2007). Many leaders of world religions have explicitly expressed their support for vaccination against COVID-19 (British Islamic Medical Association, 2020; Card, 2020; Jewish Medical Doctors working in UK, 2020). Communicating this fact, along with debunking myths around mRNA vaccination, may reduce biased reasoning among religious groups.

Exploring the unintended effects research question also revealed potential pitfalls when designing refutations. Our findings from Experiment 1 suggest that debunking texts about COVID-19 vaccines can increase the false belief that COVID-19 vaccinations never cause any side effects. We assume this occurred because the debunking did not adequately address the potential risks of vaccination. Mentioning potential side effects in Experiment 2 prevented this unintended effect. Thus, to support informed decision-making and to build trust (Habersaat et al., 2020), we recommend reporting side effects in a transparent manner when designing debunking interventions about the risks of vaccination.

Results from the prebunking hypothesis in Experiment 2 revealed that materials designed for debunking can also serve as prebunking, and thereby protect individuals against subsequent misleading information, for example, from social media posts. These results are in line with previous findings about the effectiveness of corrections as prebunking interventions (Tay et al., 2022; Vraga et al., 2020), and further underline the usefulness of simple text-based debunkings during a pandemic. Moreover, the results are in line with research on the theory of motivated reasoning that suggests that prebunking can be particularly effective because it may address two motivated reasoning approaches (Bolsen & Druckman, 2015, 2018). First, individuals motivated to be accurate may use prebunking information as relevant information in their subsequent reasoning. Second, unknowing individuals that are not motivated to invest efforts in accuracy may use the prebunking to form a prior belief that is then defended against any subsequent (mis)information, regardless of its accuracy (Bolsen & Druckman, 2015).

Interestingly, we did not find evidence that prebunking protected against the impact of misleading social media posts on individuals’ intention to get vaccinated. Maybe this was a cost of simply turning the debunking into a prebunking without further changes to the intervention. Sophisticated prebunking attempts explicity warn individuals that they will be potential targets of disinformation (Lewandowsky & van der Linden, 2021). This warning is intended to trigger a feeling of threat and increase the receiver’s motivation to resist the maleficent attempt (van der Linden et al., 2020). The prebunking text used in this study instead warned individuals about the misinformation but did not explicitly raise the expectation that one will be a potential target of that misinformation in the near future. Moreover, typically the misleading argumentation is explicitly explained during the prebunking, which provides an additional explanation of why misinformation is false and allows individuals to generalize their knowledge to other topics (Ecker et al., 2022). The prebunking text used in this study focused only on the content and did not uncover the fallacies present in the social media post. Finally, the debunking text started with a technical summary of general facts (Figure 1), which was probably not particularly engaging, and may have made the intervention less effective due to a lack of engagement and interest.

We found no evidence for the spirituality as moderator hypothesis in Experiment 2. In light of the findings from Experiment 1, we assume that the moderating effects may only show after a delay. Future studies should investigate that assumption.

The studies have several limitations. Experiment 1 and Experiment 2 were conducted at different times during the pandemic. No mRNA vaccine had been approved yet by the European Medicines Agency at the start of Experiment 1, while vaccines were already being administered during the 2-months follow-up, and around 31 million Germans were already vaccinated at least once when Experiment 2 started (Supplementary Figure 2). At the time of Experiment 2, approximately 69.4 million of the total population in Germany (83.2 million) were over 18 years of age and thus eligible to be vaccinated. In relation to this group, 31 million vaccinated individuals correspond to a vaccination rate of about 44.7%. Thus, comparisons between the experiments need to be treated with caution because the results may have been influenced by time and context. However, assessing the effectiveness of debunking texts at different stages of the pandemic is also a unique strength of the current studies because it reveals the usefulness of simple, often-used and recommended text-based approaches at different stages of a pandemic. Another limitation is the rather highly educated sample pool. Debunking texts need to be understood in order to be effective. Thus, disseminating the chosen interventions in lower-educated or less-literate populations could lead to a decrease in the observed effectiveness. Other formats such as videos may be more suitable and should be tested in similar manners as prebunking and debunking interventions. Finally, both samples reported low levels of religiosity and spirituality. Thus, assumptions about how debunking affects the credibility judgments in highly religious or spiritual groups are limited and effects may be underestimated.

Conclusion

Text-based debunking containing detailed refutations can be an effective approach to counter misinformation about mRNA vaccination during a pandemic. In fact, detailed refutations can also serve as prebunking, that is, they can immunize individuals against subsequent misleading information, for example, as seen in social media posts. However, the conditions under which debunking and prebunking effects persist over time need to be further examined. Protective effects may be limited to beliefs rather than behavioral intentions. Despite the limitations of the present studies, we recommend actively providing text-based corrections of misinformation to the public because they can be an effective and economic measure of science communication during a pandemic. However, designers of debunking interventions should be aware of how such texts should be constructed (Ecker et al., 2020; Lewandowsky et al., 2012, 2020) and that the effect may be limited or backfire. As demonstrated, debunking that introduces a new technology, such as mRNA, may backfire among religious groups. To reduce the risk of backfire, we speculate that it can be helpful to highlight that leaders of world religions have explicitly expressed their support for the COVID-19 vaccination. Moreover, debunking texts that omit information about the side effects of vaccination may introduce new pro-vaccination myths that interfere with informed decision-making. Therefore, the risks of vaccination should be transparently reported to avoid overgeneralization and foster informed decision-making.

Author Biographies

Philipp Schmid is psychologist and postdoctoral researcher at the University of Erfurt, Germany. He studies the psychology of science denialism and health misinformation and aims to support people’s informed decision-making in health, for example, vaccination. He applies a persuasion psychology perspective to understand the impact of misinformation in health communication and to develop and evaluate promising interventions.

Cornelia Betsch is psychologist and professor of health communication at the University of Erfurt and leads the Health Communication working group at the Bernhard-Nocht-Institute for Tropical Medicine in Hamburg. She studies the psychological mechanisms of health behaviors and aims at increasing planetary health by evidence-based health communication. She applies a judgment and decision-making and strategic interaction perspective to understanding health behavior—especially with regard to fighting emerging infectious diseases, vaccination decision-making, prudent use of antibiotics, and climate friendly behavior.