Comparative effectiveness of mandates and financial policies targeting COVID-19 vaccine hesitancy: A randomized, controlled survey experiment

Abstract

Experts debate whether COVID-19 vaccine mandates or financial incentives will reduce, rather than increase, interest in vaccination. Among 3,698 unvaccinated U.S. residents, we conducted a randomized, controlled survey-embedded experiment to estimate the absolute and relative psychological effects of vaccine policies specifying: mandates by employers or airlines, bars, and restaurants; lotteries for $1 million, $200,000, or $100,000; guaranteed cash for $1000, $200, or $100; and $1,000 as either a tax credit or penalty. Vaccine intention —the study outcome— predicts uptake and provides insight into the psychological mechanism that is most proximal to behavior (i.e., vaccination). Compared to controls, those who learned about the $1,000 cash reward policy were 17.1 (±5.3)% more likely to want vaccination. Employer mandates are more promising than other mandate policies (8.6 [+/- 7.4]% vs. 1.4 [+/- 6.0]%). The full results suggest that neither mandates nor financial incentives are likely to have counterproductive psychological effects. These policies are not mutually exclusive and, if implemented well, they may increase vaccine uptake.

1. Introduction

The COVID-19 pandemic has prompted several types of vaccine policy proposals and intense debate about their absolute and relative effectiveness. In particular, vaccine mandates have been a subject of controversy, with some skeptics arguing that mandates are counterproductive because they will decrease the proportion of the population that wants to vaccinate [1], [2]. According to the theory called psychological reactance, people may resist a requirement they perceive as curtailing their freedom [3], [4]. Although there are few empirical tests relevant to COVID-19 vaccination mandates, the evidence does not appear to support this concern [5].

Some experts contend that financial incentives should be used to increase interest in COVID-19 vaccination [6], [7]. However, other experts have warned that, like with mandates, financial incentives are likely to reduce interest in vaccination [8], [9]. When financial incentives aiming to change various types of behavior (beyond vaccination, such as medication adherence and smoking cessation) have been tested, payments had inconsistent effects. In some cases, they may have failed to increase behavioral motivation [10], [11], [12].

There is a relatively small body of literature studying the effects of financial incentives specifically in the context of the COVID-19 pandemic and the results have not been consistent [13], [14]. For example, results from a quasi-experimental study in North Carolina suggest that a $25 guaranteed financial incentive slowed the decline in vaccination [13]. Analyses of Ohio’s lottery-based incentive program offering up to $1 million suggested this intervention was not particularly promising [14]. However, analyses of a dozen U.S. states using lottery incentives, found significant and substantial positive effects in ten of those states, including Ohio [15]. Conceivably, the small chance of winning a lottery could have different psychological effects on vaccine motivation those of a guaranteed cash payment, but systematic research on such differences is lacking.

Prior to the COVID-19 pandemic, reviews of interventions to increase uptake of older vaccines have found supportive evidence for various types of incentives [16], [17]. Few studies have been designed to compare the psychological effects (within the same study population) of financial incentive vaccination policies that vary by type and the monetary amounts. As an exception, see the survey-embedded experiment by Fishman et al. [18] and Robertson et al. [19]. None, to our knowledge, have randomized participants to directly compare the effects of different financial incentive amounts and types to various mandates.

Ideally, future vaccine policies are selected on the basis of their evidence and yet we lack studies designed to empirically compare the different types of policies that could be implemented. To build this evidence base, we conducted a randomized, controlled experiment (embedded within a survey) to estimate the absolute and relative effects of ten different policies, some of which have been recently implemented in the U.S. and elsewhere. The policies we compared include two types of mandates: employer mandates and mandates by airlines, bars and restaurants. In addition, the experiment includes eight types of financial policies, such as relatively large and small incentives and penalties. In the U.S., several types of financial incentives for COVID-19 vaccination have been implemented, including lotteries and guaranteed cash payments, and they have ranged dramatically in the size of the monetary reward [20].

Although incentives are, by definition, not mandatory, Attwell and Navin argue that “depending on their scale and setting they may be regarded as a form of coercion since to go without is to forego a benefit” [21]. They also recognize that any incentive (financial or otherwise) can function as a sanction (i.e., loss or punishment) for those who run afoul of the policy. We explore if such distinctions might matter to vaccination intentions by testing financial policies using the same monetary amount but framing it as a penalty on not being vaccinated or as reward for vaccination. We also test the same monetary amount but specify the delivery in cash or tax credits because these policies have been proposed and the distinctions may influence vaccine intention. Based on other research, we would expect the immediate cash rewards would be more motivating [17].

The present study was designed to examine the effects of potential vaccine policies on the psychological mechanism that influences vaccination behavior [17], [22], [23]. Intentions are the psychological mechanism that offers “insight into why people engage in health behaviors, including vaccination” [17]. Furthermore, according to a large body of literature, intentions are the most proximal and strongest determinant of future behavior [24], [25]. Vaccine uptake will ultimately depend not only on a policy’s psychological effect, but also on the degree to which logistical obstacles to vaccination are reduced [26], [27]. These logistics are beyond the scope of this experiment but, when logistical obstacles are reduced, vaccine intentions predict vaccine uptake [22], [23].

2. Methods

Pre-registration: The study design, analytical plans, including exclusions, and study outcomes were pre-registered at ClinicalTrials.gov (NCT04747327).

Study design: We conducted a randomized, controlled survey-embedded experiment. The experimental study design was also selected because it prioritizes internal validity, which is important since our goal is to compare the effects of different policies under controlled conditions. This study design allowed us to systematically manipulate the type of policy described to an individual and compare the psychological effects of such policies [28]. Participants were randomized to reduce the chance that observed effects are due to unmeasured factors. In addition, all study procedures were automated, which improves the control over how the experiment is conducted, allowing all procedures to be consistently standardized [29], [30]. Experiments without this automation risk several forms of bias [31], [32].

Subject recruitment and inclusion: Unvaccinated residents of the U.S. were recruited through an online platform (Prolific). Compared with traditional experiments, online experiments have made it feasible through “crowd-sourcing” to obtain a larger and more diverse sample [33], [34].

During recruitment, the study was described in vague terms as seeking to understand health-related preferences. The use of vague terms is recommended to reduce enrollment bias [29], [30]. Before participants joined the study, it was not apparent that the experiment was concerned with vaccination or COVID-19.

Prolific pre-screens all individuals who can participate in a survey panel. They allow participants to be screened for several, standard socio-demographic variables and relatively novel ones, such as COVID-19 vaccine status. Prolific creates panels for each study according to any pre-specified screening criteria. We enrolled a national convenience sample of 4,024 U.S. respondents who, when prescreened by Prolific, reported being unvaccinated. Since their vaccination status was measured by Prolific prior to our study, we also measured current vaccine status. After doing so, we excluded 636 (15.8%) who updated their status to being at least partially vaccinated.

Control and treatment conditions: In the control group, no vaccine policy was presented. In the treatment groups, the incentive policies included cash for $1000, $200, or $100; lotteries for $1 million, $200,000, or $100,000; $1,000 as either a tax credit for vaccination or a tax (penalty) on the unvaccinated; and mandates by employers or airlines, bars, and restaurants. Each experimental condition included a scenario that was described as shown below:

One of the below vaccine policies were randomized to appear in each experimental condition:

your employer requires that you get fully vaccinated against COVID-19.

airlines, and many restaurants and bars require proof of being fully vaccinated against COVID-19.

your state government starts a $1,000 tax penalty on those who are NOT fully vaccinated against COVID-19.

your state government offers a $1,000 tax credit for anyone who is fully vaccinated against COVID-19.

your state government offers a payment of $100 to anyone who is fully vaccinated against COVID-19.

your state government offers a payment of $200 to anyone who is fully vaccinated against COVID-19.

your state government offers a payment of $1,000 to anyone who is fully vaccinated against COVID-19.

your state government offers a weekly lottery with chances to win $100,000 for anyone fully vaccinated against COVID-19.

your state government offers a weekly lottery with chances to win $200,000 for anyone fully vaccinated against COVID-19.

your state government offers a weekly lottery with chances to win $1 million for anyone fully vaccinated against COVID-19.

Note: “Fully vaccinated” was defined as two Pfizer or Moderna shots or one Johnson & Johnson shot.

The cash payments and lottery payments are scaled in parallel at 1x, 2x, and 10x multipliers to evaluate diminishing marginal utility, and the $1,000 incentives were manipulated as cash, tax credit, or tax penalty to systematically evaluate such differences while keeping the amount fixed. The maximum lottery reward was listed at $1 million because that is the highest amount that has been proposed (or implemented) for such policies. The maximum amount of the cash policy was $1000 because debates about financial incentives for COVID-19 have focused on this amount more than any other [6], [35], [36], [37] and, while larger amounts may be effective, they are also considered excessive by some [35], [36], [37]. The specific mandates tested were selected because they were currently being debated and/or expected to be implemented soon in parts of the U.S.

Allocation ratios: Each participant was randomly assigned to a control condition or a treatment condition that discussed one of the ten vaccine policies. To allow power for planned contrasts, we used unequal allocation ratios, with twice as many assigned to the control condition and the $1000 cash condition compared to those assigned to each of the other conditions. Randomization was implemented automatically using Qualtrics software’s survey flow, setting it to “evenly present” each branch.

Before and After FDA Approval: The experiment was conducted from August 12 to September 2, 2021, and during this time the Food and Drug Administration (FDA) granted full approval to the Pfizer vaccine. Data were collected immediately before and after the FDA announcement, which was analyzed as a control variable. Throughout the experiment, in factorial design, we randomized inclusion (or omission) of a message asking respondents to consider their decision under the condition that the FDA had granted approval for the COVID vaccine. Prior to the FDA approval, this manipulation was a hypothetical vignette [28]; for example, the respondent was asked to imagine that they are offered $1,000 to vaccinate and that the vaccine has received full FDA approval. After the FDA approved the Pfizer vaccine, the FDA message was no longer hypothetical.

Vaccine intention outcomes: For the primary outcome, participants were asked, “Would you want to get vaccinated against COVID-19 in the next four weeks?” The response options included “yes,” “no” and “unsure.” As our dependent variable, we constructed a binary variable equal to one if the respondent selected “yes” and equal to zero otherwise.

As a secondary outcome, participants used a seven-point scale to report their likelihood of getting vaccinated soon: “How likely is it that you would get vaccinated in the next four weeks?” The response options ranged from “extremely likely” to “extremely unlikely.”

Both of these outcomes have been used in studies of vaccine hesitancy, which has been defined by the World Health Organization and others as “a delay in acceptance, or refusal of vaccines despite the availability of vaccine services” [38]. Therefore, this study sample includes those who are often characterized as “vaccine hesitant.” In addition, as discussed above, the item stems and response option sets are standardized and validated means of capturing the psychological mechanism that predicts voluntary, human behavior [24], [25] including vaccination [17], [22], [23].

Socio-demographic variables: Participants also reported their political affiliation, gender, ethnicity, age, education, and income. These variables were used as covariates and to describe the sample.

Data collection and cleaning: Qualtrics, which hosted the experiment, collected the data and provided measures of response quality that allowed us to eliminate those with duplicate IDs. We also eliminated those who finished in the shortest time, defined as the fastest 5%, which (based on standard deviations from the mean completion time) can capture those least likely to have read the items before selecting a response [39], [40]. Data were cleaned and analyzed (as described below) using the Stata 16 statistical software.

Analytic goals and procedures: Our analytic goal was to compare the effects of alternative policies by comparing the probability that respondents assigned to the treatment conditions intended to vaccinate relative to those assigned to the control group. Under our randomized, controlled experimental study design, where each respondent views one policy, a policy could increase or decrease motivation if the percent intending to vaccinate was higher or lower, respectively, under that policy condition compared to the control condition. Similarly, using the scaled study outcome, a policy could increase or decrease the strength of motivation to vaccinate compared to the control condition.

Primary outcome analytic procedures: We estimated a regression model–alternatively using ordinary least squares and logistic regression–with a binary-outcome dependent variable equal to one if the respondent selected “yes” when asked if they wanted to be vaccinated, and otherwise equal to zero. For explanatory variables, we include dummy variables for each of the ten treatment arms. These indicators are set equal to one if the respondent viewed the given treatment condition and are zero otherwise. We include as a control variable an indicator, which takes value one for respondents who viewed the FDA approval message.

We estimate the impacts of the employer mandate treatment only on respondents who viewed this condition and noted they were likely to have an employer during the time period in question. We excluded respondents who viewed the employer mandate condition but indicated they were unlikely to have an employer. We did not have any restrictions on the mandate for bars, restaurants and airlines.

FDA variable analytic procedures: For data collected after FDA approval, we included another indicator control variable (referred to as the “post-approval indicator”). Finally, we included a control variable, which is the interaction between the post-approval indicator and the FDA approval condition, to test whether the FDA-approval language may have had different effects once it was no longer hypothetical.

Subgroup analytic procedures: To explore correlations between respondent demographic characteristics and response to policy treatments, we compared mean outcomes of the binary vaccine intention variable across different demographic groups under each of the treatment conditions.

Secondary outcome analytic procedures: Finally, as a robustness check, we assessed the sensitivity of our empirical findings from the binary model, by estimating an ordered logit model, where the dependent variable is participants’ responses to the seven-point scale “likelihood of getting vaccinated,” scaled from 1 (extremely unlikely) to 7 (extremely likely). Explanatory variables in this model were consistent with those in the binary outcome specifications.

Sample size calculation: With at least 384 subjects assigned to each condition, we estimated that we would exclude < 20% and detect differences > 5%. Although we conducted subgroup analyses for certain demographic variables for robustness, we did not power this study to test hypotheses for these subgroups. We judged that the mandate policies were unlikely to be implemented in the United States among only a particular subgroup defined, for example, by race, age group, or gender.

3. Results

After data cleaning, the analytic sample included 3,698 individuals. As shown in Table 1 , the sample can be characterized as diverse financially and politically. The majority were under the age of 50 and female. About a third had some college education and no degree. The socio-demographic variables measured were well balanced between conditions (see Appendix Table 2).

Table 1.

Summary Statistics.

				Policy Intervention
			Control	$1000 Tax	$1000 Tax Credit	$1000 Cash	$1M Lottery	$200 Cash	$200 K Lottery	$100 Cash	$100 K Lottery	Air/Bar/Rest Mandate	Employer Mandate
Observations			421	298	307	587	296	283	288	300	288	293	195
	(after FDA approval)		(60)	(42)	(43)	(81)	(41)	(40)	(42)	(41)	(39)	(40)	(32)
Respondent Demographics (share of respondents within treatment condition)
	Political Affiliation
		GOP	0.37	0.32	0.34	0.36	0.37	0.31	0.34	0.38	0.33	0.34	0.38
		DEM	0.21	0.26	0.25	0.26	0.25	0.29*	0.22	0.24	0.27	0.22	0.29*
		IND	0.42	0.43	0.41	0.39	0.38	0.40	0.43	0.37	0.40	0.44	0.33*
	Gender
		Female	0.82	0.83	0.79	0.81	0.80	0.77	0.81	0.78	0.81	0.80	0.79
		Male	0.18	0.17	0.21	0.19	0.20	0.23	0.19	0.22	0.19	0.20	0.21
	Income Category
		$0-$50 k	0.52	0.54	0.53	0.55	0.45	0.53	0.49	0.51	0.56	0.51	0.42*
		$50 k-$100 k	0.32	0.33	0.33	0.34	0.38	0.27	0.33	0.35	0.32	0.34	0.41*
		$100 k-$150 k	0.11	0.09	0.08	0.07*	0.13	0.12	0.12	0.11	0.09	0.09	0.12
		>$150 k	0.05	0.03	0.06	0.05	0.04	0.08	0.06	0.04	0.03	0.05	0.05
	Ethnicity
		Hispanic	0.14	0.15	0.13	0.13	0.15	0.13	0.16	0.14	0.13	0.11	0.17
		White	0.88	0.85	0.83	0.86	0.84	0.82*	0.84	0.89	0.83	0.84	0.87
		Black	0.11	0.11	0.14	0.10	0.10	0.14	0.13	0.10	0.12	0.12	0.12
		Asian	0.02	0.01	0.02	0.03	0.02	0.05*	0.01	0.03	0.01	0.03	0.02
	Age
		18 to 29	0.69	0.65	0.67	0.70	0.73*	0.71	0.69	0.61*	0.64	0.67	0.74
		30 to 49	0.25	0.27	0.27	0.25	0.22	0.23	0.27	0.32*	0.30	0.27	0.24
		> 50	0.05	0.07	0.06	0.06	0.05	0.06	0.05	0.06	0.06	0.06	0.02*
	Education
		No Degree	0.02	0.03	0.02	0.01	0.00	0.02	0.03	0.01	0.03	0.03	0.02
		HS Grad	0.17	0.21	0.22*	0.25*	0.23*	0.23*	0.22*	0.28*	0.24*	0.24*	0.16
		Associates	0.13	0.11	0.14	0.10	0.11	0.13	0.11	0.09	0.15	0.15	0.12
		BS or Higher	0.29	0.31	0.30	0.30	0.30	0.24	0.32	0.34	0.25	0.23*	0.37
*Indicates significant difference from control group at 5% level.

Primary outcome: As described in further detail above, the primary study outcome measured whether participants would want to get vaccinated given a particular scenario and it is presented as the percent who reported “yes” (rather than “no” or “not sure”). In the control group, 20.7 (±3.9)% of respondents wanted to vaccinate.

Fig. 1 displays the estimated difference in the percent wanting to vaccinate for each treatment condition relative to the control group. In both the ordinary least squares and logistic regression models, seven of the ten vaccine policies generated positive and statistically significant effects. (For completeness, we estimated a linear probability model and a Logit model; the estimation strategy did not change the results.) The remaining three policies ($100 k lottery, $200 k lottery, and the airline/bars/restaurant mandate) elicited a small, positive effect statistically indistinguishable from zero.

Fig. 1.

Estimated Impact of Different Policies on Vaccine Intentions, Note: For each of the ten treatment conditions, this figure reports the estimated change in the decision to vaccinate (and 95% confidence intervals). The Ordinary Least Squares (OLS) and Logit regressions generated these estimates. The R-squared of OLS was 0.0259. The pseudo R-squared for the Logit model was 0.0212.

The employer mandate had a larger effect than the other mandates (8.6 [+/- 7.4]% vs. 1.4 [+/- 6.0]%). The $1,000 cash policy had the strongest effect; respondents who viewed this condition were 17.1 (±5.3)% more likely to want vaccination. The much smaller $100 and $200 cash payments also had positive and statistically significant effects (9.2 ± 6.3% and 9.3 ± 6.5%, respectively). However, the second most effective strategy was the $1,000 tax on the unvaccinated, which increased the proportion wanting vaccination by 13.8 (±6.5)%.

Comparing the magnitude of treatment effects between the $1000 cash versus $1000 tax and $1000 tax credit conditions, we find that the estimated treatment effect for the $1000 cash incentive is statistically significantly larger than the treatment effect for the $1000 tax credit in both the binary and ordinal models. Other comparisons between these treatment conditions are not statistically significantly different at the 5% level.

Secondary outcome: In an ordered-logit model using the secondary outcome (which used a scaled measure of “how likely” one was to get vaccinated soon), the above results were consistent with the results for the primary outcome. The results for the secondary outcomes are reported in the Appendix.

FDA effects: The coefficient on the FDA indicator control was positive and statistically significant (9.3 ± 3.2%), indicating that including the message about FDA approval increased the percentage of the sample that wanted to vaccinate. The estimated coefficient on the post-approval indicator was very small in magnitude and not statistically significant, indicating there were no significant differences among between responses when the FDA approval message was hypothetical or real. Similarly, the coefficient for the post-approval FDA interaction is small in magnitude and statistically insignificant, suggesting the FDA message generated a similar effect before and after the FDA announced full approval.

Subgroups: Fig. 2 compares responsiveness to mandates and vaccine incentive policies for different demographic groups. Comparing outcomes by political affiliation in panel 2.1 of the Figure, we find that, relative to Republicans and Independents, Democrats were statistically significantly more responsive to employer mandates and airline/bar/restaurant mandates. This result also holds for most of the incentive policies. Among incentive policies, unvaccinated Democrats appear particularly responsive to small ($100) cash payments, both compared to the responses of the Republicans and Independent groups and compared to Democratic group responses to other types of incentives.

Fig. 2.

Estimated Effects of Different Policies on Vaccine Intentions by Subgroups.

Referring to panels 2.2 and 2.4–2.6 of Fig. 2, we do not find statistically significant gender-, ethnicity-, education-, or age-based differences in responsiveness to different types of mandates. However, as shown in panel 2.3, respondents in lower income categories were more likely to respond to employer mandates and other mandates than those in higher income categories. We note that none of the respondents in the “>$150 k” income category indicated that they were likely to work for an employer. Thus, we do not estimate an effect for that category.

Finally, referring to panel 2.5 of Fig. 2, respondents with less than a high school education were more responsive to most incentive policies compared to individuals with higher levels of education. Small cash incentives were particularly effective among respondents without high school diplomas, compared to this group’s responses to other treatment conditions.

The subgroups defined by socio-demographics may be related to each other. Therefore, we tested associations among the socio-demographic variables measured. In the appendix, we report their correlations.

4. Discussion

This study was conducted in the U.S. about 9 months after COVID-19 vaccines had become available. Vaccination uptake had slowed, and a relatively low percentage of the control group reported that they wanted to vaccinate [41]. The U.S. had not reached herd immunity and the absolute and relative merits of these vaccine policies were still being widely debated.

Despite arguments (discussed above) that that employer mandates and financial incentives are likely to decrease the proportion of a population that wants to vaccinate, the current data suggest that this is unlikely. Compared to the base rate of the control group, neither policy decreased the proportion that would want to vaccinate, which is consistent with results from prior studies that examined COVID-19 vaccine mandates [5] or financial incentives [13], [14], [16], [17], [19]. Instead, employer mandates and financial incentives had positive psychological effects, increasing the proportion that wanted to vaccinate.

Employer mandates performed better than other mandates. The guaranteed cash performed better than lotteries for larger monetary amounts. The strongest effects were associated with guaranteed $1,000 cash rewards. Relatively small amounts, such as the $100 cash payment, had effects similar to the $1,000 tax credit, which suggests that the amount may be less important than the timeliness of the reward or the message framing when communicating about the incentives. Cash incentives of larger magnitude generally had greater effects on vaccine intention, but these policies exhibit diminishing marginal returns relative to increases in outlay size. For example, the $1,000 cash payment is ten times the size of the $100 cash payment, but the effect of the $1000 cash payment on vaccine intention is <2 times the size of the $100 cash payment.

Those who received a message about FDA approval of the vaccines were more likely to want to vaccinate compared to those who did not receive a message about FDA approval. This effect was observed before and after FDA approval was officially announced, suggesting that this regulatory body did inspire trust or otherwise positively influence some participants. It could be useful to make the FDA approval salient when communicating about the vaccine.

Experts have warned that employer mandates and financial incentives would be especially counterproductive among Republicans and those with lower socio-economic status—two groups that have been characterized as skeptical of such policies [8], [9]. However, we did not find evidence of a counterproductive effect among these subgroups. The positive effects of several policies are stronger among those who are less educated, minority race or ethnicity, male, and younger age. For example, the mandates tested had a stronger effect among those with lower financial status than those with higher financial status. In addition, the financial incentives often had stronger effects among those with lower socio-economic status, which makes them progressive policies. Importantly, the socio-demographic factors that are associated with these promising policy effects (Fig. 2) are factors that are also characteristics of U.S. populations with less vaccination coverage [41].

Our study does not address the practicality of implementing any of these policies. If the interventions are perceived as politically infeasible, it is worth considering the much greater costs produced by COVID-19 hospitalizations compared to a cash payment. In addition, the financial incentives are within the scope of financial stimulus payments already made by the U.S. Federal government, and if such payments were made conditional on vaccination, it could reduce the cost of an outlay that is otherwise made [6], [15]. However, large cash incentives are likely to be politically challenging, especially for state or local governments. As an alternative, tax credits and employer mandates may be less susceptible to political concerns (e.g., with economic inflation) and still effective. The $1,000 tax on those not being vaccinated was the most effective of the deterrent measures, and it would be revenue-positive since some people who still declined to vaccinate would pay the tax.

Our results are consistent with other research finding that financial incentives have increased vaccination rates for COVID-19 in other countries [42] and appear to increase uptake of other vaccines [43], [44], [45]. In addition, economic research suggests that even small effects of vaccine incentives in the U.S. can have a worthwhile cost-benefit ratio [15].

Limitations: This study provides insight into the psychological mechanism that can influence vaccination [17], [22], [23], [24], [25]. It was not designed to compare the effects of policies on vaccination coverage directly, and such effects will depend on how well the policy is implemented [22], [23], [24], [25]. Further research would be needed to identify the best ways to implement a particular policy. Policies can be set at national, state, and local levels and our sample is not weighted to represent particular jurisdictions considering these policies. The best implementation strategy may depend on the jurisdiction and other characteristics of the policy. For example, the $1000 incentive could be implemented on a national level but this amount would be impractical for a local jurisdiction with a more restrained budget. Airline mandates and tax credits can also be implemented by federal government. Employer mandates can be implemented by private actors.

There are other limitations of this study. Notably, the results may not be generalizable to other populations. The sample was limited to U.S. residents and the effects may differ in other countries. The study was also focused on testing effects for a particular vaccine and the effects may differ for other vaccines, such as those that protect against influenza, tetanus and other infections. In addition, we tested specific monetary amounts and other amounts may generate different results. Future research can examine the effects of additional policies not included here.

These limitations are important but they do not weaken the experiment’s internal validity. Our study tests a wide range of policies, including different strategies using the same incentive amount, so that the policies can be compared directly, under the same conditions. The random assignment strengthens our ability to consider causal effects. While field studies are ideal, randomization to the actual mandate and financial incentive policy was not feasible.

These results can contribute to the evidence-base when deciding between vaccine policies for COVID-19 and other immunization efforts. As of this writing, some large U.S. cities (e.g., Philadelphia) and the Veterans Administration are considering a cash incentive for COVID-19 vaccination [46]. Future research can evaluate the effectives of new policies and also test if effects may differ for COVID-19 booster vaccination.

Regardless of which vaccine policies may be implemented, there is a continuing need to ensure equity so that those who want to vaccinate are able to do so promptly and without disproportionate financial burden or logistical challenges [47], [48], [49]. Because lower-income populations tend to have inflexible work schedules, as well as fewer options for childcare and transportation, many who are interested in vaccination can still find it challenging to do so in a timely fashion [47], [48], [49]. Employers can promote equity by providing paid time off for vaccination, which is still rarely provided, or by hosting on-site, convenient vaccination opportunities [47], [48], [49]. Mobile vaccination opportunities provided by medical vans in neighborhoods could also help overcome these structural barriers [47], [48], [49].

In summary, this study found that neither mandates nor financial incentives had counterproductive effects. Moreover, some of the tested policies had promising effects, especially among subgroups that have lower rates of vaccine coverage. Guaranteed cash payments had larger effects than mandates, but these policies are not mutually exclusive. Instead, they may play a complementary role. Indeed, some parts of the U.S. have had both financial incentive and mandate policies [20], [50]. As the evidence base grows, the relative effectiveness of various public health policies may become clearer, helping to improve vaccination rates.

Availability of data and material: The dataset supporting the conclusions of this article is available from the Penn ALACRITY Data Sharing Committee by contacting research coordinator, Kelly Zentgraf at zentgraf@upenn.edu, 3535 Market Street, 3rd Floor, Philadelphia, PA 19,107 in https://hosting.med.upenn.edu/cmh/people/kelly-zentgraf/.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A.

Table 2. Ordered logit results.

Treatment	Coefficient	Standard Error
$1000 Tax	0.72	0.13	***
$1000 Tax Credit	0.21	0.12	*
$1000 Cash	0.65	0.11	***
$1M Lottery	0.05	0.13
$200 Cash	0.44	0.13	***
$200 K Lottery	0.16	0.12
$100 Cash	0.26	0.13	**
$100 K Lottery	0.17	0.12
Air/Bar/Rest. Mandate	−0.01	0.12
Emp Mandate	0.51	0.14	***
Observations	3,715
Pseudo R2	0.009
Standard errors are robust to heteroskedasticity of an unknown form.
***p < 0.01, **p < 0.05, *p < 0.1

Table 3. Correlations.

	Political Affiliation			Gender		Income Category				Ethnicity				Age			Education
	GOP	DEM	IND	Female	Male	$0-$50 k	$50 k-$100 k	$100 k-$150 k	>$150 k	Hispanic	White	Black	Asian	18 to 29	30 to 49	> 50	No Degree	HS Grad	Associates	BS or Higher
GOP
DEM	−0.42
IND	−0.60	−0.47
Female	0.00	0.09	−0.08
Male	0.00	−0.09	0.08	−1.00
$0-$50 k	−0.15	0.09	0.07	0.03	−0.03
$50 k-$100 k	0.07	−0.05	−0.02	−0.01	0.01	−0.73
$100 k-$150 k	0.08	−0.03	−0.05	−0.02	0.02	−0.35	−0.24
>$150 k	0.11	−0.06	−0.05	0.00	0.00	−0.23	−0.16	−0.08
Hispanic	−0.11	0.10	0.02	−0.02	0.02	0.03	0.01	−0.03	−0.04
White	0.23	−0.21	−0.04	0.03	−0.03	−0.11	0.05	0.06	0.06	−0.16
Black	−0.23	0.19	0.06	0.01	−0.01	0.09	−0.03	−0.07	−0.05	−0.05	−0.68
Asian	−0.02	0.03	−0.01	−0.03	0.03	0.00	0.00	0.01	0.00	−0.03	−0.16	−0.02
18 to 29	−0.03	0.05	−0.01	0.18	−0.18	0.04	−0.02	−0.05	0.01	0.13	−0.05	0.03	0.04
30 to 49	0.00	−0.03	0.02	−0.16	0.16	−0.05	0.03	0.05	−0.01	−0.09	0.03	−0.01	−0.03	−0.87
> 50	0.05	−0.04	−0.01	−0.05	0.05	0.02	−0.03	0.01	0.01	−0.08	0.05	−0.04	−0.02	−0.36	−0.15
No Degree	−0.05	0.03	0.02	−0.03	0.03	0.06	−0.05	−0.02	−0.01	−0.01	−0.02	0.02	−0.02	0.00	−0.01	0.01
HS Grad	−0.06	0.03	0.03	−0.05	0.05	0.10	−0.02	−0.10	−0.06	0.08	−0.05	0.03	0.02	0.09	−0.09	−0.01	−0.08
Associates	0.03	−0.02	0.00	0.04	−0.04	−0.01	0.03	−0.01	−0.03	−0.01	−0.01	0.00	0.02	−0.05	0.04	0.03	−0.05	−0.20
BS or Higher	0.12	−0.04	−0.08	−0.04	0.04	−0.17	0.07	0.11	0.08	−0.07	0.09	−0.08	−0.01	−0.13	0.15	−0.01	−0.09	−0.34	−0.24

Appendix B. Supplementary data

The following are the Supplementary data to this article: