Persuading US White evangelicals to vaccinate for COVID-19: Testing message effectiveness in fall 2020 and spring 2021

Abstract

The development of COVID-19 vaccines was an important breakthrough for ending the pandemic. However, people refusing to get vaccinated diminish the level of community protection afforded to others. In the United States, White evangelicals have proven to be a particularly difficult group to convince to get vaccinated. Here we investigate whether this group can be persuaded to get vaccinated. To do this, we leverage data from two survey experiments, one fielded prior to approval of COVID-19 vaccines (study 1) and one fielded after approval (study 2). In both experiments, respondents were randomly assigned to treatment messages to promote COVID-19 vaccination. In study 1, we find that a message that emphasizes community interest and reciprocity with an invocation of embarrassment for choosing not to vaccinate is the most effective at increasing uptake intentions, while values-consistent messaging appears to be ineffective. In contrast, in study 2 we observe that this message is no longer effective and that most messages produce little change in vaccine intent. This inconsistency may be explained by the characteristics of White evangelicals who remain unvaccinated vis à vis those who got vaccinated. These results demonstrate the importance of retesting messages over time, the apparent limitations of values-targeted messaging, and document the need to consider heterogeneity even within well-defined populations. This work also cautions against drawing broad conclusions from studies carried out at a single point in time during the COVID-19 pandemic.

Vaccination is the most effective way to mitigate the spread of COVID-19 and reduce fatalities. Yet some people are hesitant to get vaccinated. One group that has emerged as particularly difficult to convince to vaccinate is White evangelicals (1, 2), although this group has been shown to be persuadable in August 2020 to wear masks by values-consistent messaging (3). Important questions are whether White evangelicals can (still) be persuaded to get vaccinated against COVID-19 and are values-consistent messages a promising technique for this group?

To answer this, we draw on data from two survey experimental studies fielded on nationally representative samples. Study 1 (n = 855) was fielded in fall 2020 prior to COVID-19 vaccines being available in the United States and around the same time as the masking persuasion study targeting this group (3). Study 2, fielded in June 2021, was designed to explicitly sample this population and includes 2,419 still unvaccinated respondents. Study 2 allows us to test whether messaging that was effective prior to vaccines being available remains effective among still unvaccinated individuals.

We examine three outcomes measured using survey data: 1) willingness to vaccinate in the first/next 3 mo a vaccine is available, 2) willingness to advise a friend to vaccinate, and 3) negative judgments of someone who chose not to vaccinate along five traits: trustworthy, selfish, likeable, competent, and intelligent.

Results

In study 1, subjects were randomly assigned to seven conditions: a placebo message with content unrelated to COVID-19, a baseline message about vaccine efficacy, or five treatment messages that added specific content to that baseline.

The community interest and reciprocity (CIR) message emphasized the vaccination was a prosocial act that protects others and that others would likewise work to protect them. The CIR + embarrassment message added language that emphasized how embarrassed and ashamed one would feel if one did not vaccinate and infected someone else.

The remaining two messages highlighted values likely important to the respondent. The “not bravery” message reframed choosing to not be vaccinated as reckless rather than brave and emphasized that real bravery involves vaccinating to protect others. The “personal freedom” message argued vaccination would end government restrictions on freedom caused by COVID-19. A third values-based message was also tested: The “trust in science” message, highlighted that not vaccinating makes someone seem like they do not understand science.

Fig. 1, Left shows the effect of all messages relative to the placebo message. For all outcomes, the most effective message among this sample is the CIR + embarrassment message. It is associated with a large, ∼0.10 units increase in all outcomes, differences that are statistically significant (P < 0.05) compared to the placebo message, the baseline informational message, and the CIR message (the latter was the best performing message among the entire sample). These represent substantial increases from baseline scale scores in control: a 30% increase in intention to vaccinate, a 24% increase in intention to advise a friend, and a 38% increase in negative evaluations of a nonvaccinator. The CIR + embarrassment message also produced substantively large increases compared to the baseline vaccination message for all outcomes (vaccine intention: 0.10 units, P < 0.01, advise a friend: 0.07 units. P < 0.05, negative evaluation of nonvaccinator: 0.07 units, P < 0.05).

Fig. 1.

Covariate-adjusted treatment effects obtained using ordinary least squares (OLS) regression with robust Huber–White SEs. The placebo message serves as the baseline for both study 1 (Left) and study 2 (Right) and estimates are presented with 95% confidence intervals.

Notably, values-consistent messages, not bravery and personal freedom, are not robustly effective in the sample, despite how they mirrored rhetoric later espoused by President Trump and faith leaders.

Study 2 is a preregistered trial testing both whether messages effective among this population in fall 2020 were effective in spring 2021 and revised values-consistent messages.

Subjects were assigned to one of six conditions at equal rates, an untreated control, or one of five persuasive messages: 1) CIR, 2) CIR + embarrassment, and three novel messages: 3) Trump hero, 4) restore liberty, and 5) doctors endorse. The Trump hero message emphasized former President Donald Trump’s role in getting a vaccine developed and his taking the vaccine, a strategy shown to be effective among unvaccinated Republicans more broadly (4). The liberty message, which reworked the liberty message that was ineffective in fall 2020, emphasized that vaccination would get rid of government restrictions on individual freedom. Finally, the “doctors endorse” message highlighted that people trust their doctor and that doctors support vaccination.

Fig. 1, Right reports the analysis of study 2. In contrast with study 1, we find no evidence that the CIR + embarrassment message remains effective: The effects on one’s own intention to vaccinate, advising a friend, and evaluations of a nonvaccinator are all small and statistically insignificant. The CIR message (without embarrassment) was similarly ineffective.

Neither the Trump hero nor the restore liberty messages were effective in this setting. The Trump hero message is associated with a small increase in two outcomes. The doctors endorse message has small and statistically insignificant effects on one’s own intention to vaccinate and advising a friend to vaccinate (both ∼0.02, not significant).

Discussion

In the fall 2020 data, the CIR + embarrassment message was the most effective at increasing vaccination promoting intentions among White evangelicals, outperforming the CIR message that was most effective among the broader population. In spring 2021 this message was no longer effective for those who remained unvaccinated.

At neither point in time do we observe that values-based messaging or elite cues are successful at increasing vaccine uptake intentions among White evangelicals, which contrasts with prior work that has found that values-consistent messaging is effective at increasing positive attitudes toward masking (3). One potential explanation for this divergence is that the values-based messaging in previous work (3) also had a messenger (e.g., President Trump or an evangelical leader), making it difficult to separate out the effect of the value from the endorsement. Another is that elite discourse had already framed masking as an issue related to the values of freedom and protecting others in August 2020 (when data from ref. 3 were collected), while the conversation about vaccination in values terms had not matured in fall 2020 and White evangelicals who are more difficult to persuade remained unvaccinated in spring 2021. Finally, there is also the possibility of publication bias in which work that does not observe effects is not published. This leaves open the possibility that other tests of values-based messaging may have also failed, although more research is required to establish whether values-based messaging strategies are generically effective for this population in other domains.

A potential explanation for these changing results is selection—those who remained unvaccinated in June 2021 have resisted 6 mo of intensive attempted persuasion. Study 2 participants removed (pretreatment) for already being vaccinated were 8 points more likely to be age 65+, 5 points less likely to have less than a high school degree, 8 points less likely to be a Republican, and 12 points more likely to have voted for Biden in 2020. Comparing vaccine attitudes in the study 1 and study 2 samples, the fall 2020 sample was more likely to have gotten a seasonal flu vaccine in the preceding 5 y (average 2.6 versus 1.4) and scored much higher on general vaccine confidence (5). This scale (0 to 1) had a population mean of 0.66 in September 2020, while study 1 participants had a mean of 0.62. In the study 2 sample, by contrast, the mean is 0.46, fully 25% lower than in the study 1 sample.

An alternative explanation for the null finding from study 2 is that White evangelicals have become less persuadable over time. To gain partial leverage on this explanation, we use the observed sample proportions for age, gender, education, party identification, general vaccine confidence, and past flu vaccination from study 2 to construct sample weights for respondents in study 1 so that the reweighted sample approximates the study 2 sample. We then reestimate the models for study 1 using this reweighted sample to estimate how responsive individuals who remained unvaccinated in spring 2021 would have been to the treatment messages in fall 2020. This produces an estimate that the CIR + embarrassment message increases vaccine uptake intentions when compared to the pure control (0.09 units, P < 0.05) and the baseline vaccination message (0.11 units, P < 0.05). These effects are similar in magnitude to those observed in the original (unweighted) sample. For the advise a friend and judgment of a nonvaccinator outcomes the effect of the CIR + embarrassment message is approximately half as large as previously estimated and no longer statistically significant. This analysis requires the strong assumptions that observables, like vaccine confidence, did not change over time and that unobservable factors do not explain which individuals were persuaded by the treatment. This approach does, however, provide suggestive evidence both that those who remain unvaccinated were harder to persuade in fall 2020 and that the same White evangelicals have become more difficult to persuade to get vaccinated, although it is difficult to rule out concerns about selection on unobservables.

Our results have three broad implications for the study of vaccine uptake and persuasion more generally. First, they demonstrate that results obtained for a group, even a well-defined population, may not extrapolate to a future period, given differences in vaccine uptake and heterogeneous treatment effects. Second, they show the general importance of testing and retesting messages as the population needing to be persuaded changes. This also highlights that caution is necessary when generalizing results beyond a specific point in time during the COVID-19 pandemic. Of course, a limitation of this work is that we cannot assess the effectiveness of treatment on behavioral outcomes, and past work has documented divergent results between actual vaccine uptake and uptake intentions (6). Third, we show that values-consistent messaging is largely ineffective among White evangelicals, despite success with preventative behaviors.

Materials and Methods

Study 1 was fielded between September 9 and September 22, 2020 and study 2 was fielded between May 13 and May 31, 2021. Both samples only include self-identified White, non-Hispanic, evangelical Christians. The preregistration document and supplementary tables for Fig. 1 and the reweighted analysis in Discussion have been uploaded to Open Science Framework (7) and sample characteristics and extended methods can be found in SI Appendix.

Respondents provided consent prior to participation in either study. Both studies were fielded under an existing exemption granted by the Yale institutional review board as they pose minimal risk to respondents.

Data Availability

Anonymized survey data, preanalysis plan, and supplementary tables have been deposited in Harvard Dataverse, https://doi.org/10.7910/DVN/MA0OYO (8) and Open Science Framework, https://osf.io/b5rm9/?view_only=10a049d1aebe49a99529472619789fba (7).