Can financial incentives and other nudges increase COVID-19 vaccinations among the vaccine hesitant? A randomized trial

Abstract

Despite rapid initial uptake, COVID-19 vaccinations in the United States stalled within a few months of widespread rollout in 2021. In response, many state and local governments, employers and health systems used public health messaging, financial incentives and creative scheduling tools to increase vaccine uptake. Although these approaches drew on evidence from influenza and other vaccination efforts, they were largely untested in the context of SARS-CoV-2. In mid-2021, months after vaccines were widely available, we evaluated vaccination intentions and vaccine uptake using a randomized control trial. To do this, we recruited unvaccinated members of a Medicaid managed care plan in California (n = 2,701) and randomly assigned them to different public health messages, $10 or $50 financial incentives for vaccination, a simple vaccination appointment scheduler, or control. While messages increased vaccination intentions, none of the interventions increased vaccination rates. Estimates for financial incentives rule out even relatively small increases in vaccination rates. Small financial incentives and other behavioral nudges do not meaningfully increase COVID-19 vaccination rates amongst the vaccine hesitant.

1. Introduction

Despite ample supply of COVID-19 vaccines and strong evidence that vaccines protect against severe disease and death [1], many Americans remained unvaccinated more than nine months after FDA approval [2]. Although vaccine availability contributed to low initial uptake, supply constraints in the United States eased in early April 2021 and distribution sites became extensive and included more than 41,000 commercial pharmacy locations [3], [4], [5], [6]. Public vaccinations sites and pop-up locations offered additional access. Furthermore, vaccines were available free of charge in the US, irrespective of immigration or insurance status [7]. California, where our trial took place, offered free transportation to vaccination sites, home vaccinations and multi-lingual materials about COVID-19 vaccinations [8], [9].

An arguably more important contributor to the US’s stalled COVID-19 vaccination effort was vaccine hesitancy, the delay or outright refusal of vaccinations despite the availability of vaccination services [10], [11]. While estimates of vaccine hesitancy vary, a monthly tracking survey showed that as of February 2022, 10 months after all adults in the U.S. were eligible for vaccination, 19% of adults said that they would either “definitely not” get at least one dose of the COVID-19 vaccine or get it “only if required” [12]. An additional 4% continued to say they would “wait and see” [12], although this category had shrunk considerably over time.

The sources of vaccine hesitancy are varied and may include factors such as knowledge gaps about vaccine safety and effectiveness, concerns about side-effects, mistrust of the medical system, and perceptions of low disease risk [10]. In the case of COVID-19, concerns about the rapid development of the vaccines and mistrust of government are often cited as well [13].

In response to COVID-19 vaccine hesitancy, many state and local governments, employers and health systems have turned to financial incentives – gift cards, saving bonds and lotteries with grand prizes of $1 million or more – to increase vaccine uptake [14]. Public health messages, which have shown some promise in increasing COVID-19 vaccination intentions [15], [16], [17] and vaccination appointments among health care workers [18] are also widely used to combat vaccine hesitancy.

COVID-19 vaccination campaigns have also drawn on evidence from influenza immunizations. Financial incentives [19], default appointment scheduling [20], implementation prompts [21] and reminder messages [22], [23], [24], [25], [26] all increase influenza vaccinations. The unique aspects of COVID-19 vaccine hesitancy [27], however, suggest caution in applying this evidence to the pandemic. Yet, despite considerable time, interest, and money put towards COVID-19 vaccinations, experimental evidence on the effectiveness of financial incentives and other approaches to increasing vaccination, particularly among the vaccine hesitant, is relatively slim.

We aimed to fill the gap in our understanding of whether behavioral nudges, including small financial incentives, public health messaging, and an easy appointment scheduler, could increase COVID-19 vaccinations among the vaccine hesitant. We conducted a preregistered randomized control trial (NCT04867174) between May 24, 2021 and July 16, 2021 with unvaccinated adult members of Contra Costa Health Plan (CCHP), the public Medicaid managed care plan in Contra Costa County, California. Medicaid is public health insurance for people with low-income [28]. Contra Costa is a racially and ethnically diverse Bay Area county with over 1.1 million residents [29]. COVID-19 vaccine hesitancy was higher among individuals with low relative to high annual income and among those who identified as Black or Latino relative to White [30], and vaccination rates were significantly lower among Medicaid enrollees relative to the general population [30], [31]. At the beginning of our study, more than 30% of vaccine eligible individuals enrolled in the health plan were still unvaccinated despite living in a county with amongst the highest vaccination rates in the state [32]. Consequently, Contra Costa County partnered with us to gather evidence on what might work to get these vaccine hesitant individuals vaccinated. By design, our study focused on individuals who had delayed vaccination despite weeks of eligibility and ample supply of vaccine.

Unvaccinated adult health plan members were randomized prior to study invitation to control or one of three different public health video messages – a message from the California Department of Public Health focused on getting back to normal, a provider message emphasizing the safety and efficacy of the vaccines, and a provider message focused on the negative health consequences of remaining unvaccinated. In addition, we cross-randomized participants to $10 or $50 financial incentives for vaccination and a link to an easy vaccination scheduling system. For the two provider messages, we also randomly assigned respondents to a race and/or gender concordant provider. Experimental work has shown that in-person care from a race-concordant provider can increase uptake of influenza vaccination and other preventive health services among Black men [33]. In observational studies, gender concordance is positively correlated with cancer screenings [34], [35] but not influenza vaccinations [34]. We know of no experimental analysis of the impact of gender concordance on immunizations.

2. Background

Several quasi-experimental studies of state financial incentives for COVID-19 vaccinations come to conflicting findings. Studies of Ohio’s Vax-a-Million lottery, which offered weekly prizes of up to $1 million over the course of 5 weeks between the end of May and June 2021, have shown at best modest impacts on vaccination rates. A comparison of Ohio to the rest of the United States found no impact on vaccination rates [36] while comparisons using constructed “synthetic control” groups demonstrate increases on the order of 1 percentage point [37], [38], [39]. Studies simultaneously evaluating multiple state lottery programs have, with one exception [40], largely found null effects [41], [42], [43]. The inconsistent findings stem in part from heterogeneity in the incentives used and populations targeted across studies but also from sensitivity to the choice of control groups used for comparison purposes in these quasi-experimental analyses.

Only one randomized trial has studied the impact of financial incentives on COVID-19 vaccine uptake [44]. That work, which focused on vaccinations within 30 days after participants became eligible based on their age-group in Sweden, found that financial incentives of just $25 increased vaccination rates by over 4 percentage points. While the study provides convincing evidence for the role of financial incentives in increasing COVID-19 vaccinations in Sweden, the applicability of these findings to vaccine hesitant populations in the US may be limited. Swedish Public Health Agency data, for example, consistently show that about 90% of Swedes will certainly or probably accept the offer of a COVID-19 vaccine [45].

Two randomized trials have analyzed the impact of behaviorally-informed messages on COVID-19 vaccination rates. The first, which studied vaccinations in the UCLA Health System, found that reminder messages, specifically those that emphasize vaccine “ownership,” increased vaccinations [46]. As that study was conducted in February 2021, before vaccines were available to the general public and when the demand for vaccines outpaced supply, both nationwide [47] and in the study site of Los Angeles [48], it may also hold limited applicability for vaccine hesitant populations. The other study, compared “ownership” framed messages, in some conditions supplemented with other information about vaccines, relative to no messages and found no impact on vaccination rates [49]. This sample, which was made up of unvaccinated adults in Rhode Island in late May to June 2021, after more than a month of eligibility, was most similar to ours in focusing on the vaccine hesitant population.

Much of the remaining evidence on the impact of behavioral nudges on COVID-19 vaccination is survey based and focuses on intentions rather than actual vaccinations[15], [16], [17]. Evidence for a link between self-reported measures of vaccination intentions and actual vaccinations is relatively sparse. In Sweden, vaccination intentions and outcomes are closely linked [44]. In the US, vaccination intentions elicited from an online survey were not informative of actual vaccinations in a patient population [46], similar to the findings in this study.

3. Methods

The study flow diagrams (Fig. 1 and Appendix Fig. 1a-1d) show the factorial design and the allocation of participants to each intervention arm. Our protocol was approved by the Institutional Review Board (IRB) at the University of Southern California and Contra Costa Health Service’s Institutional Review Committee. The trial was registered on ClinicalTrials.gov, which specified the main primary and secondary outcomes. Our main analyses were prespecified and registered on the American Economic Association RCT Registry (AEARCTC-0007405). We conducted several robustness checks of our statistical models, which we were not prespecified.

Fig. 1.

Study Flow and Randomization. Notes: This CONSORT diagram describes the study flow, number of CCHP members invited, number of members who completed the survey, number excluded due to prior vaccination and number included in the analytic sample. For additional detail, see Appendix Figure 1.

3.1. Participants

Contra Costa Health Plan (CCHP) members aged 18 and over who were unvaccinated based on Contra Costa Health Services (CCHS) records and who had previously self-identified as White, Black, or Latino (n = 58,308) were invited via e-mail or SMS to participate in a survey about “COVID-19 in your community” (See Appendix for the invitation text). Treatment assignment was unknown to participants at the time of invitation. We excluded members who self-identified as Asian because the diversity of this group made studying provider race concordance difficult. We initially provided $5 gift cards for baseline survey completion as well as entry into a raffle for $250 gift cards.

A total of 2,825 individuals consented to and completed the survey; 124 were excluded due to prior vaccination, despite attesting to no prior COVID-19 vaccination in the pre-screening survey. The final analytic sample included 2,701 respondents (Fig. 1).

3.2. Interventions

Invited participants were randomized prior to invitation to the following arms:

• 1.Control Arm (25%).
• 2.Message Arm: emotion vs. safety vs. consequences of going unvaccinated (75%).

To assess whether video messages changed participants’ vaccination intentions, videos were played before a question about vaccination intentions. The first message, which we refer to as the “emotion” video, focused on getting back to normal was a publicly available video from the California Department of Public Health (CDPH message). The remaining two messages, which were developed by our team in consultation with outside medical and behavioral science experts, featured physicians from CCRMC. One of the messages highlighted the safety of the COVID-19 vaccines (“safety” message); the other, which was inspired by a study to counter childhood vaccination misinformation [50], focused on the negative health consequences of remaining unvaccinated (“consequences” message). Links to the CDPH “emotion” videos and the scripts of the provider “safety” and “health consequences” videos are available in the Appendix.

Both messages were recorded by a White female, White male, Black female, Black male, Latina female and Latino male physician. Latino physicians recorded the messages in both English and Spanish. These two messages were also randomized to be race/ethnicity and gender concordant/discordant with the respondent (Appendix Fig. 1). Each of these arms (control, emotion, safety, consequences) was interacted with a financial incentive of $10 (25%) or $50 (25%) and, separately with the highlighting of a convenient link to the county public vaccine appointment scheduling system (50%). The $10 or $50 financial incentives were paid for getting vaccinated within two weeks of survey completion. A two-week period was used given evidence that a shorter window can reduce procrastination in other health care contexts [51]. Those randomized to financial incentives also received a reminder about the incentive along with their e-gift card for survey completion. While vaccination status was verified using a link between the county health system and the state vaccination records system, we opted to pay incentives based on patient reports of the date and location of a vaccination or a photo of a vaccine card. We did not want to highlight access to the state’s vaccination system because monitoring and mistrust of the government may play an important role in vaccine hesitancy.

Participants randomly assigned to the county’s vaccination appointment scheduling system were given the link at the very end of the survey. The county scheduling system was designed to reduce scheduling frictions (e.g., it showed appointments on its landing page instead of requiring the user to first enter personal data.). Financial incentives were also presented after survey completion to eliminate the possibility that offering a financial incentive might have an effect on survey completion. A respondent randomized to both the appointment scheduler and a financial incentive would have seen both interventions on the same page.

3.3. Outcomes

Our pre-registered primary outcome was whether respondents received at least one dose of a COVID-19 vaccine within 30 days of survey completion. Our secondary outcome was respondent’s self-assessed probability (0–100) of getting vaccinated in the next 30 days at time of survey. We also pre-registered heterogeneity analyses of the primary outcomes in which we stratified respondents by age, gender, race/ethnicity, and 2020 presidential candidate support.

Vaccination data for all respondents were provided by CCHS, which receives near daily updates from the California Immunization Registry (CAIR), a system that captures all immunizations delivered in the state. Thus, we captured in-state immunizations irrespective of delivery site (e.g., CCHS clinic or retail pharmacy).

To reduce the chance of differential attrition from the survey, vaccination intention was asked after the video messages but prior to revealing financial incentives or the appointment scheduling link. As such, by design, we cannot estimate how intentions respond to treatments other than the video messages. Survey completion required answering the vaccination intention question so there was no missing data for this outcome.

3.4. Statistical analysis

Following our pre-registration, our primary analysis of the impact of our interventions on vaccination was estimated using the following linear probability model:

$${\mathit{Vaccinated}}_i=\alpha +{\beta }_1{\mathit{Financial}}_i+{\mathrm{M}}_i\mathrm{\theta }+{\beta }_3{\mathit{Link}}_i+\delta X_i+{\epsilon }_i$$ (1)

where ${\mathit{Vaccinated}}_i$ is an indicator (0/1) for whether a respondent received a COVID-19 vaccination within one month after completing the survey. ${\mathit{Financial}}_i$ is an indicator for whether the individual was randomized into the financial incentive arm, M is a vector of indicators for randomization into each of the three message types (CDPH, safety, or health consequences video) and ${\mathit{Link}}_i$ is an indicator for whether the individual was randomized to receive the highlighted link. The excluded group, the control condition, received no extra prompting to get vaccinated. To increase precision, we included $X_i$, a vector of predetermined characteristics including age and its square, race, gender, self-reported income, education, the language the respondent took the survey in (English/Spanish), whether the respondent was impaneled and indicators for calendar date. Our main hypotheses were that all of the interventions would increase vaccination rates, ${\beta }_1>0$, $\mathrm{\theta }>0$ and ${\beta }_3>0$.

To differentiate across the financial incentive amounts, we expanded on (1) to estimate:

$${Vaccinated}_i=\alpha +{\beta }_1{1}_i^{\mathrm{\$}10}+{\beta }_2{1}_i^{\mathrm{\$}50}+{\mathrm{M}}_i\mathrm{\theta }+{\beta }_3{\mathit{Link}}_i+\delta X_i+{\epsilon }_i$$ (2)

where $1_v^{\mathrm{\$}10}$ and $1_v^{\mathrm{\$}50}$ are indicators for being randomized into financial incentives of $10 or $50. Our hypothesis was that the magnitude of the effect is increasing in the incentive amount: ${0<\beta }_1<{\beta }_2$.

Our analysis of vaccinations intentions was based on a modification of equation (1):

$${\mathit{Intention}}_i=\alpha +{\beta }_1{\mathit{Financial}}_i+{\mathrm{M}}_i\mathrm{\theta }+{\beta }_3{\mathit{Link}}_i+\delta X_i+{\epsilon }_i$$ (3)

where ${\mathit{Intention}}_i$, a respondent’s self-assessed probability of getting vaccinated in the next 30 days, takes the place of ${Vaccinated}_i$. Note that since the financial incentive and scheduling link are presented to individuals after survey completion, they are included here only as indicators of treatment stratum and are not meant to generate causal estimates of their impact on vaccine intentions. Our main hypothesis was that messaging increases vaccine intentions, $\mathrm{\theta }>0$. We further hypothesized that the health consequences message would have the largest effect on intentions such that ${\mathrm{\theta }}_3>{\mathrm{\theta }}_1,\mathrm{a}\mathrm{n}\mathrm{d}{\mathrm{\theta }}_3>{\mathrm{\theta }}_2$, where 3 denotes the health consequences message, 2 denotes the safety message and 1 denotes the CDPH message.

We analyzed heterogeneity in the impacts of our interventions by respondent gender, race/ethnicity, age-group, and support for Trump or Biden during the 2020 presidential election. The 2020 election analysis was motivated by early indications that vaccination rates were lower in areas that had higher 2020 election vote shares for Trump; later data found a gap in vaccination rates based on presidential candidate support that widened over time.[52], [53].

To analyze whether race and gender concordance affected the impact of health messages, we re-estimated the models specified by equations 1 and 3, but include interactions between the relevant video messages and ${\mathit{RaceConcord}}_i$ and ${\mathit{GenderConcord}}_i$, which are indicator variables equal to 1 if the physician messenger and the recipient share the same race/ethnicity or gender, respectively.

We checked the robustness of our results to model choice - probit regression models of vaccine uptake (equations (1) and (2)) and censored regression models (tobit models) of vaccination intentions (3). Across all regression models, robust standard errors are used to control for heteroscedasticity. We report two-sided p-values.

4. Results

The trial sample was recruited from 58,308 adult CCHP members who were eligible for participation based on age and vaccination status. CCHP is a Medicaid plan. By virtue of eligibility for Medicaid, all respondents were low-income. As shown in column 1 of Appendix Table 1 , the majority of the sample we recruited from was female, reflecting the gender skew in Medicaid enrollment [54], about a quarter identified as Black and nearly 44% as Hispanic. About 1 in 5 listed Spanish as their primary language of communication and less than 10% were age 65 and over. Health plan members who completed the survey were more likely to be female and to be impaneled, meaning they receive primary care through the county integrated health system at Contra Costa Regional Medical Center (CCRMC). They were also less likely to have Spanish as their primary language of communication and less likely to be ages 40–64 and 65 and over relative to 18–39.

Table 1.

Treatment-control balance.

	Control mean	Treatment-Control Difference
Participant Characteristics		Video Message 1	Video Message 2	Video Message 3	Financial incentive	Scheduler Link
Age	36.9	−0.91	−0.56	−0.19	−0.70	−0.65
	(11.7)	(0.410)	(0.616)	(0.865)	(0.502)	(0.539)
Female (%)	72.8	−0.40	0.51	3.64	2.27	1.94
		(0.924)	(0.903)	(0.384)	(0.568)	(0.629)
English as preferred language (%)	94.1	−1.02	1.81	0.31	0.44	0.03
		(0.650)	(0.403)	(0.889)	(0.533)	(0.990)
Hispanic (%)	30.9	2.89	6.22	0.65	2.86	3.32
		(0.508)	(0.157)	(0.883)	(0.489)	(0.426)
Black (%)	28.7	2.75	−2.28	4.13	1.95	1.91
		(0.520)	(0.591)	(0.339)	(0.630)	(0.640)
Impaneled (%)	61.0	9.90	5.18	4.73	11.95	6.53
		(0.029)	(0.257)	(0.303)	(0.006)	(0.139)
At least high school graduate (%)	91.2	−0.28	0.53	0.38	0.45	0.35
		(0.916)	(0.843)	(0.886)	(0.859)	(0.891)
HH Income < $20,000 (%)	40.4	6.25	1.94	8.13	5.83	4.40
		(0.177)	(0.675)	(0.082)	(0.184)	(0.321)
Supported Biden (%)	55.4	0.50	−2.92	−1.04	−1.73	0.24
		(0.914)	(0.532)	(0.824)	(0.696)	(0.957)
Supported Trump (%)	14.7	−0.32	3.54	0.58	1.80	0.93
		(0.924)	(0.296)	(0.563)	(0.571)	(0.771)
F statistic	N/A	0.76	0.65	0.57	1.03	0.47
P-value	N/A	0.6717	0.7746	0.8399	0.4127	0.9108

Notes: We report the control mean (and standard deviation for continuous variables) and the treatment means (with p-values in parentheses) for the characteristics shown in the left-hand column. The final row reports the pooled F-statistic and p-value from testing treatment-control balance on the set of characteristics jointly. For variables that are percentages, the treatment–control difference is shown in percentage points. Due to a coding error during the early phase of the study, randomly assigned impaneled members at 1:1:1 instead of a 2:1:1 ratio to no financial incentives, a $10 incentive and a $50 incentive causing a disproportionate share of the impaneled receiving financial incentives.

Our analysis included 2,701 survey respondents, with an average age of 36.4 years (SD = 12.6). 74.3% were female. 35.9% self-identified as White, 30.5% as Black, and 33.6% as Latino (see Table 1). 67.4% of respondents were impaneled with the county health system, Contra Costa Regional Medical Center. The median self-reported household income was between $20,000 to $29,999 and 8.3% of respondents have less than a high school degree. 55.4% supported Biden in the 2020 presidential election and 15.7% supported Trump. As shown in Table 1, characteristics were well balanced across treatment arms; all joint tests of treatment-control differences are statistically indistinguishable from zero.

4.1. Vaccinations

In the control group, 8.82% of respondents were vaccinated, meaning they received at least one COVID-19 vaccine dose within 30 days of survey completion. Fig. 2 , which plots the main effects of the interventions shows that none of the treatments improved upon this rate. The estimates are small and close to zero. Only the CDPH and the safety videos are positively related to 30-day vaccination status. The upper 95% confidence intervals for financial incentives and the health consequences videos rule out increases in vaccination rates larger than 1.0 and 1.6 percentage points, respectively. The upper 95% confidence intervals for the CDPH and safety messages and the scheduling link are larger but still rule out increases larger than 3.0, 3.4 and 1.9 percentage points, respectively. We cannot reject that the impacts of the $10 and $50 incentives are the same. When fitted to a probit model, the effect sizes are similar in magnitude but with generally tighter confidence intervals (Appendix Table 2).

Fig. 2.

Impact of Main Treatments on 30-day Vaccination Rates. Notes: Regression-estimated change in COVID-19 vaccinations induced by each of our main behavioral interventions compared to control. Dashed whiskers depict 95% CIs.

Results of heterogeneity analyses, which stratified the regressions by age, gender, race/ethnicity, and 2020 presidential candidate support, also showed no meaningful effect of the treatments on vaccinations (Appendix Table 3). The one exception was financial incentives, which reduced vaccination rates in both older individuals (ages 40 and over) and those who indicated they supported Trump in the 2020 presidential election. For respondents ages 40 and over, 30-day vaccination rates declined by 4.5 percentage points (p = 0.045) and 4.7 percentage points (p = 0.041) in response to the $10 and $50 incentives, respectively. For respondents who indicated they supported Trump, the $50 incentive decreased vaccination rates by 4.2 percentage points (p = 0.047).

4.2. Effect of gender and race concordant physicians

We also analyzed the impact of race and gender concordance for the two physician videos on vaccinations (Appendix Table 4). Race concordance did not affect vaccinations for either of the physician-recorded messages (Panel A of Appendix Table 4). This was true for the sample as a whole as well as by key sub-groups of interest – young (under age 40) and old, male and female, Black, Hispanic and White and Trump versus Biden supporters. Although we do find a negative and marginally significant impact of race concordant messaging for Biden supporters viewing the safety video, we interpret this result with caution given the overall pattern of results and the likelihood that this one may be spurious.

In contrast, while gender concordant messaging had no clear impact for the safety message, a clearer pattern emerges for the message about the potential negative health consequences of remaining unvaccinated. In Panel B of Appendix Table 4, the main effect for each video message (safety or consequences) alone captures the impact of a message delivered by a gender discordant messenger relative to control and the interaction between the video type and an indicator for a gender concordant messenger captures any differential effect of the message delivered by a gender concordant relative to gender discordant messenger. Vaccination rates declined by 2.9 percentage points (p = 0.052) for those who saw the consequences video delivered by a gender discordant messenger relative to control. Relative to when the messenger is gender discordant, an identical message delivered by a gender concordant physician messenger increases vaccination rates by 3.8 percentage points (p = 0.019). On net, we cannot reject that the impact of the consequences message delivered by a gender concordant messenger is zero but the impact is large (33% relative to a mean vaccination rate of 8.8%) and negative when delivered by a discordant provider.

The effect of gender discordance was driven by individuals under age 40, men, and Latinx members. For respondents below age 40, a negatively framed message from a gender discordant physician decreased 30-day vaccination rates by 4.3 percentage points (p = 0.010). For men, the negatively-framed message delivered by a female physician decreased vaccination rates by 5.8 percentage points (p = 0.033). For Latinx, gender discordance decreased 30-day vaccination rates by 5.4 percentage points (p = 0.045). In exploratory analysis, the effect size for men below the age of 40 was over 10.0 percentage points (p = 0.012). In all case, these negative impacts were fully offset by having a gender concordant messenger such that the we cannot reject that the full effect of the video is null.

4.3. Vaccination intentions

While 8.82% of respondents in the control group were vaccinated within 30 days of the survey, 34.1% said they intended to get vaccinated over this period. Although our video messages did not impact actual vaccination rates, they did increase vaccination intentions (Fig. 3 ). Vaccination intentions increased by 5.5 percentage points (p = 0.002) in response to the safety videos and 8.6 percentage points (p < 0.001) in response to the health consequences videos (Appendix Table 5). These impacts are large, implying increases in intentions of 16–25%, compared to the 34.1% probability the control group respondents put on their intention to vaccinate in the 30 days post-survey. Results are similar when the data are fitted to a Tobit model (Appendix Table 5). The impact of the videos across demographic groups was typically positive although heterogenous in magnitude (Appendix Table 6).

Fig. 3.

Percentage Point Change in Vaccination Rates and Intentions due to Video Messages. Notes: Comparison of regression-estimated change in COVID-19 vaccinations (in blue) and vaccinations intentions (in red) induced by each of the three different public health video messages compared to control. Dashed whiskers depict 95% CIs.

As further shown in Fig. 3, the link between survey-elicited COVID-19 vaccination intentions and vaccinations was weak. Regression analysis of the relationship shows that a 10-percentage point increase in vaccination intentions is associated with only a 1.5 percentage point (p < 0.001) increase in actual vaccinations (Appendix Table 7).

In terms of race and gender concordance, neither factor seemed to affect subject’s vaccination intention (Appendix Table 8). Given that the negatively-framed message led to a large increase in overall vaccination intentions, the fact that gender discordant messages decreased actual vaccination rates (as discussed in the previous section) suggests that self-stated vaccination intentions may, in some cases, be not only a weak predictor of actual vaccinations, but a misleading one (Appendix Fig. 2).

5. Discussion

We studied common approaches to increasing vaccination rates. Although some public health messages increased vaccination intentions, none of the interventions meaningfully changed COVID-19 vaccination rates among the vaccine hesitant.

Consistent with concerns that financial incentives and other perks for COVID-19 vaccination might have perverse effects, our exploratory heterogeneity analysis provided suggestive evidence that financial incentives backfired [55], [56]. Financial incentives reduced vaccination rates among those 40 and over and those who supported Trump in the 2020 presidential election, two of the groups with the lowest vaccination intentions in our sample.

Our work further speaks to the potential role of public health messengers. In contrast to evidence that in-person visits with a race concordant provider increases influenza vaccinations [33] but consistent with recent evidence on video messages [57], we find no impact of provider-patient race concordance on vaccine uptake or vaccination intentions. Race concordance may matter less for one-way communication and, by extension, the impact found in in-person settings may reflect trust and communication that cannot be easily replicated. However, we do find suggestive evidence that negatively framed public health messages delivered by a gender discordant messenger reduce vaccinations for some groups, especially for men, Latinos, and adults under age 40.

Our work also sheds light on the relationship between COVID-19 vaccination intentions and actual vaccinations. Vaccination intentions have been widely studied during the pandemic to understand and counter vaccine hesitancy [15], [16], [17]. Unlike for influenza vaccination [19], [33], however, we found the link between an individual’s COVID-19 vaccination intention and actual vaccination status appears weak.

Our study has several limitations. First, because the survey was conducted online and recruitment was conducted by email or text, participants had to have internet access and either an email account or mobile phone capable of receiving text messages. They also had to opt in to a study about COVID-19. Thus, they may not be representative of the vaccine hesitant. Among those invited to our survey, 17.6% were vaccinated by August 31, 2021 but those who completed the survey and were in the control condition were 3.75 percentage points more likely (p-value 0.000) to be vaccinated compared to those who did not complete the survey. Second, given subjects were recruited from a single public Medicaid managed care plan and a single state, our findings may not generalize beyond our subject population.

Despite these limitations, our findings provide several lessons for more effective, evidence-based policy making on COVID-19 vaccinations. First, while vaccination intentions are an expedient way to study the impact of different policy options and can be illuminating in other contexts, such studies may have limited value for COVID-19 vaccinations. Our work demonstrates a weak link between self-stated COVID-19 vaccination intentions and subsequent vaccinations. In some cases, self-stated vaccination intentions are a misleading indicator of how actual vaccinations respond to messaging, providing false positive or false negative estimates of intervention effectiveness. Our findings suggest that studies that rely on self-stated COVID-19 vaccination intentions should, at a minimum, be interpreted with caution.

Second, many of the efforts that have proved popular in the past and in other contexts, including small financial incentives, are unlikely to convince a substantial share of the COVID-19 vaccine hesitant to get vaccinated. Reaching a goal of very high vaccination rates likely requires much stronger policy levers, such as employer rules or government mandates, which have shown considerable promise in Canada and several European countries [58].

Finally, to the extent that the publics’ response to future pandemics (or vaccinations more generally) resembles their response to the COVID-19 pandemic, our results suggest that the standard playbook for public health may be ineffective. High rates of vaccine hesitancy suggest a need to develop alternative approaches to help mitigate future pandemics.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following are the Supplementary data to this article: