Influence of a COVID-19 vaccine’s effectiveness and safety profile on vaccination acceptance

Significance

Acceptance of vaccines has been on the decline in recent years. Despite encouraging early results for coronavirus vaccine trials, achieving herd immunity requires substantial uptake. We presented scenarios varying vaccine efficacy, minor side effects, and severe reactions to a sample representative of the US population. Vaccine acceptance improved when the efficacy increased beyond 70%. Respondents were unaffected by the probability of minor side effects, such as a sore arm or fever lasting 24 h. The chances of accepting the vaccine were lower when the probability of serious adverse reactions was 1/100,000 in contrast to 1/million or 1/100 million. A replication showed that the results were largely unchanged following the public announcement that the vaccines were 95% effective.

Abstract

Although a safe and effective vaccine holds the greatest promise for resolving the COVID-19 pandemic, hesitancy to accept vaccines remains common. To explore vaccine acceptance decisions, we conducted a national survey of 1,000 people from all US states in August of 2020 and a replication in December of 2020. Using a 3 × 3 × 3 factorial experimental design, we estimated the impact of three factors: probability of 1) protection against COVID-19, 2) minor side effects, and 3) a serious adverse reactions. The outcome was respondents’ reported likelihood of receiving a vaccine for the coronavirus. Probability of vaccine efficacy (50%, 70%, or 90%) had the largest effect among the three factors. The probability of minor side effects (50%, 75%, 90%) including fever and sore arm, did not significantly influence likelihood of receiving the vaccine. The chances of a serious adverse reaction, such as temporary or permanent paralysis, had a small but significant effect. A serious adverse reaction rate of 1/100,000 was more likely to discourage vaccine use in comparison to rates of 1/million or 1/100 million. All interactions between the factors were nonsignificant. A replication following the announcement that vaccines were 95% effective showed small, but significant increases in the likelihood of taking a vaccine. The main effects and interactions in the model remained unchanged. Expected benefit was more influential in respondents’ decision making than expected side effects. The absence of interaction effects suggests that respondents consider the side effects and benefits independently.

The race to produce a safe and effective vaccine for the SARS-CoV-2 virus has yielded remarkable progress. Two vaccines have now been approved for emergency use in the United States. At least 10 other vaccines are under evaluation in other countries, and 57 candidates are reported to be in phase 1 or phase 2 clinical trials (1).

Even though safe and effective vaccines are now being administered, vaccination programs still face significant obstacles. One study suggested that 35.8% of adults refuse to take flu vaccines (2). Recent estimates from the Centers for Disease Control and Prevention (CDC) show that only about 63% of children under the age of 18 and 45% of adults received a flu vaccination during the 2018–2019 flu season. Our group recently completed a public opinion poll on vaccine hesitancy. Using a representative sample of the US population, we found that only 38% of the adult population reported being very likely to take a vaccine for the coronavirus, with another 29% being somewhat likely. About 21% of the US population reported they will not take the vaccine under any circumstance and 36% believed it was definitely or probably true that harmful effects of vaccines are not being disclosed to the public. Others have reported similar results. Using a May 2020 survey, Malik et al. (3) found 67% reported they would accept a COVID-19 vaccine (similar to our 38% very likely + 29% somewhat likely = 67%), but likely acceptance varied by demographic group, with males, older adults, Asians, and college graduates more prone to accept (3). In addition to the persistent determinants of vaccine hesitancy, and the concerns about the COVID-19 pandemic, 2020 was a highly charged presidential election year. Several studies indicated that candidate preference was highly correlated with likely vaccine acceptance (4).

There are legitimate reasons for being hesitant about vaccines. In an average year, individuals who take flu vaccines may only have a 50% probability of being protected. For example, in 2018–2019, the protection rate for adults was 47% (95% confidence interval [CI] = 34–57%) (5). If only 50% of the population gets vaccinated and the vaccine provides only 50% protection, only a quarter or the population will be protected (50% × 50% = 25%). However, the effectiveness of the first two coronavirus vaccines was about 95%—much higher than most flu vaccines. Thus, the announcement that COVID-19 vaccines were 95% effective was expected to significantly boost consumer confidence.

This study applies a conjoint measurement method to assess the influence of several factors on the decision to be vaccinated for coronavirus (6). In marketing research, conjoint analysis is widely used to assess consumer preferences and to predict consumer purchasing behavior. Conjoint analysis presents sets of options with various attributes (for example, high risk, low benefit vs. low risk, high benefit medicines) and asks respondents to choose a survey response that reflects their trade-offs among the attributes. When used to assess patient preferences, conjoint analysis presents the patient with combinations of health gains and risks that might be similar to those that the patient may encounter clinically (e.g., protection from infection, experience of minor side effects, experience of a serious adverse reaction). Because of limitations of alternative methods of assessing health-relevant decisions, use of conjoint analysis has expanded in studies of patient decision processes (7, 8). In this study, we used conjoint analysis to explore how variations in expected benefits and harms of a coronavirus vaccination affect the likelihood of respondents’ intent to be vaccinated. Following the completion of data collection in late August 2020, positive trial results for two new vaccines became dominant news stories and led to emergency use authorization and rapid deployment of the vaccines. To assess the effects of this important development, the August study was replicated in December of 2020.

Methods

Factors that affect the decision to be vaccinated have been summarized in a systematic review (9). In addition to demographic factors, vaccine confidence was identified as most influential variable. Three components of vaccine confidence are vaccine efficacy, probability of minor side effects, and probability of a serious adverse reaction. In this study, we systematically varied levels of these three attributes in a factorial experiment. We tested the main effects of each of the factors and the interactions between them.

Study Population.

Participants were members of the YouGov proprietary opt-in survey panel, which includes 1.8 million US residents. Participants were recruited using web advertising campaigns that target respondents based on the keywords they used for Google searches. Use of specific key words prompt an invitation from YouGov to be screened for membership on a panel. All recruited members go through a double opt-in procedure. After a first consent, respondents are required to confirm their consent again by responding to an email. Internal checks are used to confirm each participant is, indeed, new and that the provided address is valid.

The data were collected between August 20 and August 27 of 2020 and the replication data were collected between December 16 and December 22 of 2020. YouGov interviewed 1,196 (August) and 1,100 (December) respondents who were then matched down to a sample of 1,000 to a produce the final dataset for each time period. Respondents were from all 50 US states, the District of Columbia, and from all US territories.

Weighting.

The respondents were matched to a sampling frame on gender, age, race, and education. The frame was constructed by stratified sampling from the full 2018 American Community Survey 1-y sample with selection within strata by weighted sampling with replacements (using the person weights on the public use file). The matched cases were weighted to the sampling frame using logistic regression-based propensity scores. Variables in the propensity score model included age, gender, race/ethnicity, years of education, and region. The propensity scores were grouped into deciles in the frame and poststratified according to these deciles. The weights were then poststratified on 2016 Presidential vote choice, smoking status, and general health condition (benchmarks obtained from the 2017–2018 National Health and Nutrition Examination Survey adult sample), and a four-way stratification of gender, age (four categories), race (four categories), and education (four categories), to produce the final weight.

Validity of Sampling.

YouGov polling methods have been quite accurate when compared with publicly verifiable events, such as elections. For example, self-reported vote in the both the 2016 and 2020 presidential elections closely corresponded to the actual popular vote. See https://d25d2506sfb94s.cloudfront.net/cumulus_uploads/document/2uo7zs3zo8/Record_of_Accuracy_YG_w.pdf.

Table 1 shows the demographic distributions for the August and December study samples in comparison to the expected distribution in the US population.

Table 1.

Demographic summary of study population

	August sample (n = 1,000), %	December replication (n = 1,000), %	US* population, %
Sex
Male	48.7	48.5	49.2
Female	51.3	51.5	50.8
Race
White	63.2	63.2	60.4
Black	12.0	12.1	13.4
Hispanic	16.1	16.0	18.3
Asian	2.9	2.2	5.9
Native American	1.0	1.6	1.3
Mixed	1.6	2.6	2.7
Other	2.9	1.7
Age†
18–29	16.6	22.3	12.4
30–44	30.2	28.0	35.5
45–64	32.6	32.8	32.7
65+	20.6	18.9	19.4
Marital Status
Married	47.0	45.9	47.8
Separated	2.3	2.7	1.9
Divorced	8.5	10.3	10.8
Widowed	5.3	3.6	5.6
Never married	31.8	31.5	33.8
Domestic/civil partnership	5.1	6.0
Vote‡ in 2016 Presidential Election§
Hillary Clinton	48.7	47.2	48.2
Donald Trump	45.1	47.1	46.1
Other	6.2	5.8	5.7

^*https://en.wikipedia.org/wiki/2016_United_States_presidential_election.

^†Denominator adjusted to exclude 40.1% less than age 20.

^‡Denominator excludes 32.3% who reported they did not vote.

^§Ref. 10.

Conjoint Methodology.

A factorial conjoint experiment varied levels of three factors in a factorial design. The factors were vaccine benefits, minor side effects, and serious adverse reactions. The 3 × 3 × 3 factorial design requires judgments of 27 cases. The effectiveness factor varied level of vaccine effectiveness for preventing COVID-19: 50%, 70%, or 90% protection. The three levels were chosen to reflect the effectiveness of common vaccines, with particular emphasis on influenza vaccines. Although flu vaccine effectiveness tends toward 50% across years, effectiveness varies by flu subtype and vaccination history (11). The second factor is probability of minor reaction such as a sore arm, headache, or minor fever all lasting less than 1 d. We used levels of 50%, 75%, and 90% based on data from phase 2 trials of coronavirus vaccines. The third factor is probability of severe reaction (1 per 100,000, 1 per million, 1 per 100 million). The rationales for 1/100,000 and 1/million are based on high and low estimates of Guillain–Barré syndrome following 1976 swine flu vaccine (12). The 1 per 100 million assumes a severe reaction is possible, but extremely rare. Each of the 1,000 respondents was randomly assigned to one of nine groups. Each individual respondent completed three items. Their task was to read each case and rate how likely they would be to get vaccinated under the risk levels described in the scenario. SI Appendix, Fig. S1 shows the instructions to subjects and SI Appendix, Fig. S2 offers examples of two items. In addition to the conjoint exercise, the survey included a small experiment relevant to preference for being vaccinated. Half of the respondents were asked, “If President Trump assured the public that the coronavirus vaccine was safe and effective, how likely is it that you would be vaccinated?” For the other half, the name Dr. Anthony Fauci was substituted for President Trump.

Analysis.

The data were analyzed using factorial analysis of variance. Differences between individual means were evaluated using the least significant difference test. In the initial analysis, group assignment was considered a factor in the design. The effects of groups were nonsignificant as were all interactions involving group assignment. Therefore, the data reported here have group assignment removed. Calculations were completed using SPSS, version 26.

Institutional Review Board Review.

The protocol was reviewed by the Stanford University School of Medicine Committee on the Protection of Human Subjects (Institutional Review Board Protocol 56833) and approved as exempt. Each participant provided consent on three occasions. In addition to the two YouGov consents, participants were presented with a Stanford consent form, and all provided electronic consent.

Results

Each of the 27 cases was evaluated on a four-point scale of likeliness to take the vaccination where 1 indicated very likely; 2, likely; 3, unlikely; and 4, very unlikely. The 1,000 participants were randomly assigned to one of nine groups in order to keep the response burden low. We assume the differences between these nine groups could be attributable only to chance. To test that assumption, we used the analysis of variance to compare ratings of vaccine acceptance across the nine groups. As expected, the differences were nonsignificant (P = 0.991).

The marginal means for the variables in the factorial design are summarized in Table 2. As the probability of benefit from the vaccine increased, there was a linear increase in the reported likelihood of taking it (P < 0.0001). Individual mean comparisons found a significantly reduced likelihood of taking the vaccine if the benefit was identified as 50% protection in comparison to 70% or 90%. However, the difference between a 70% and a 90% protection rate was not statistically significant (P = 0.068).

Table 2.

Vaccine acceptance by benefit and risk

	N	Mean	SD	SE	95% confidence interval for mean
					Lower bound	Upper bound
Average chances of taking the vaccine by probability of benefit
50%	1,000	2.34	1.124	0.036	2.27	2.41
70%	1,000	2.19	1.143	0.036	2.12	2.26
90%	1,000	2.10	1.147	0.036	2.03	2.17
Total	3,000	2.21	1.142	0.021	2.17	2.25
Average chances of taking the vaccine by probability of minor side effect
50%	1,000	2.20	1.159	0.037	2.12	2.27
70%	1,000	2.21	1.108	0.035	2.14	2.28
90%	1,000	2.22	1.159	0.037	2.15	2.29
Total	3,000	2.21	1.142	0.021	2.17	2.25
Average chances of taking the vaccine by probability of a serious adverse reaction
1/100 thousand	1,005	2.29	1.121	0.035	2.22	2.36
1/million	991	2.18	1.135	0.036	2.11	2.25
1/100 million	1,004	2.16	1.166	0.037	2.08	2.23
Total	3,000	2.21	1.142	0.021	2.17	2.25

The probability of minor side effects (50%, 75%, 90%) had nonsignificant effects on rated likelihood of accepting the vaccine (P = 0.879). However, there were significant effects for serious adverse reactions (P = 0.016). Respondents reported significantly higher likelihood of taking the vaccine if the risk of a serious adverse reaction was 1/100 million or 1/million in comparison to 1 per 100,000 (P < 0.05). Differences between the first two categories were nonsignificant. All interactions between the three factors were nonsignificant. The full factorial analysis of variance is summarized in Table 3.

Table 3.

Tests of between-subjects effects

Dependent variable: Likely to take vaccine (four-point scale)
Source	Type III sum of squares	df	Mean square	F	Significance
Corrected model	47.079	26	1.811	1.393	0.089
Intercept	14,589.959	1	14,589.959	11,226.495	0.000
Level of protection	28.808	2	14.404	11.083	0.000
Minor side effects	0.336	2	0.168	0.129	0.879
A serious adverse reaction	10.775	2	5.388	4.146	0.016
Protection × Minor	0.505	4	0.126	0.097	0.983
Protection × Major	1.441	4	0.360	0.277	0.893
Minor × Major	1.452	4	0.363	0.279	0.891
Protection × Minor × Major	4.087	8	0.511	0.393	0.925
Error	3,863.713	2,973	1.300
Total	18,541.000	3,000
Corrected total	3,910.792	2,999

When asked how likely they would be to take a vaccine if endorsed by President Trump, 18% chose very likely. Swapping in a Fauci endorsement for a Trump endorsement more than doubled (to 38%) the percentage of people very likely to get vaccinated. Political attitude also had a stronger influence on the likelihood of getting vaccinated than statistics on a vaccine’s benefits and risks. Among those reporting they are very likely to take the vaccine, 64% reported favoring Joe Biden in the 2020 Presidential race, 27% favored Donald Trump, and 9% were undecided or favored other candidates.

December 2020 Replication.

SI Appendix, Table S1 shows the results of the December replication in comparison to the August evaluation. In December, respondents reported being slightly (t = 2.45, P = 0.014) more likely (mean [M] = 2.14, SD = 1.56) to take the vaccine than respondents in August (M = 2.25, SD = 1.16). The statistical significance for all main effects and interactions were unchanged in the replication sample. SI Appendix, Table S2 shows the complete analysis of variance model with time of data collection (August vs. December) as a factor in the design. All interactions with month of data collection were nonsignificant.

The percentage of respondents who would take the vaccine based on a Trump endorsement was comparable (21% in December vs. 18% in August). In December, 32% of the respondents reported being very likely to take the vaccine if it was endorsed by President Elect Biden. Even for vaccines that were described as 90% effective, more Biden voters reported they would be very likely to be inoculated in comparison to Trump voters (38.3 vs. 26.5%, P < 0.05).

Discussion

A safe and effective vaccine for the SARS-CoV-2 virus has high potential to mitigate a severely damaging COVID-19 pandemic. However, refusal to accept the vaccine may substantially diminish the population impact. A variety of studies document an increase in the refusal to accept vaccinations for other potentially epidemic illnesses (13, 14). In a national survey earlier in 2020, we documented that only about a third of the US population reported that they were very likely to accept a vaccination for the coronavirus and about one in five adults reported that they are very unlikely to take the vaccine under any circumstances. These findings suggest that, without a better understanding of the reasons for vaccine refusal, achieving herd immunity will be difficult.

Our findings are consistent with several other studies (3), although most used very different methodologies. After this study was completed and under review, Kreps et al. (4) reported a similar investigation that used conjoint analysis using quota-based sampling. They also found small but significant increases in vaccine acceptance with increases in efficacy and reduced acceptance with increases in a serious adverse reaction. Their study did not include minor side effects and their sample was slightly less representative of the demographics of the US population.

Our findings, based on two representative samples of the US population, suggest that the likelihood of reducing the chances of getting COVID-19 is the most important factor in accepting the vaccine. The probability of experiencing minor side effects, such as a fever or sore arm that last 1 d, had nonsignificant effects. This result is important because two phase 3 clinical trials showed minor side effects are very common. The chances of experiencing a serious adverse reaction, such as paralysis, do have an effect, but only if the chances are relatively high, such as 1/100,000. The respondents did not discriminate between 1/1 million and 1/100 million. To put this in context, anaphylaxis following the first dose of the Pfizer COVID vaccine occurred in about 1.1 per 100,000 doses administered (15). Combining data from the Pfizer and Moderna trials, Bell’s palsy occurred at a rate of about 3 per million among those receiving active vaccine. Although these all seem like small numbers, the differences between these three categories are important in public health policymaking. In a US population of 300 million, 1/100,000 would translate into about 3,000 people suffering severe health consequences from vaccination. At 1/million, there would be about 300 cases, and at 1/100 million there would be just three cases. Despite evidence that people have difficulty attending to small probabilities, it appears that subjects in this study were able to attend to these very small frequencies of severe side effects. These results suggest that many people may not be deterred by side effects that are possible, but highly improbable.

All interactions between the risks and benefits of the vaccine in the August study and the December replication were nonsignificant. This result suggests that subjects evaluated each of these sets of probabilities independently. Their judgments appeared not to be influenced by complex and unique combinations of the factors. For example, the effect of serious side effects is the same regardless of whether the vaccine is 50%, 70%, or 90% effective. This result is consistent with a significant body of research from the human information processing literature. Anderson (16) has shown that, when given complex combinations of information, people appear to attend independently to each factor when rendering their judgment.

Between the completion of the original study and the replication, significant public attention was devoted to coronavirus vaccines. In August of 2020, it was assumed the vaccines would be about 50% effective. Results from two trials released in November showed about 95% efficacy, and two vaccines were given emergency use authorization by the Food and Drug Administration by mid-December. The potential of vaccines to end the pandemic received daily news coverage and the initial rollout was greeted with great enthusiasm. Although there was a small increase in willingness to take a vaccine in the December replication, the impact of the minor side effects, serious adverse reactions, and probability of benefit remained largely unchanged.

Overall, the effects of expected benefit and a serious adverse reaction were statistically significant, but accounted for a relatively small portion of the variance. It is worth noting that other variables, including political ideology and preference for candidate in the 2020 presidential election, were more strongly associated with vaccine acceptance than was basic information about the expected benefits and side effects of the vaccinations. This finding is consistent with other reports (17). The mass behavioral impacts of social media communications may result in an amplification of beliefs held by an individual and a discounting of beliefs they do not hold (18). In order to improve acceptance of a vaccine, future studies should more systematically evaluate the psycho–political–social factors in concert with the traditional factors of risk and benefits.

Our study has a significant number of limitations. First, in order to simplify the task within a national survey, each respondent judged only one-ninth of the potential cases. We made the assumption that each subject was independent and a representative sample from the general population. Although it is difficult to verify this assumption, a test of average ratings across the nine groups revealed no differences or interactions associated with group assignment.

A second concern is that, as part of a larger survey, respondents might not have devoted sufficient attention to the task. However, the observation of significant effects in the expected direction, for two of the three dimensions, provides some reassurance that respondents were engaged and attending to the task. In addition, a replication produced almost identical results.

A third limitation concerns the representativeness of the sample. While the YouGov surveys are designed to be demographically representative of the US population, they do not use probability-based samples or sampling methods. Instead, they use opt-in participation and weighting methods to achieve demographic representativeness. We recognize that the demographic match between the weighted sample and the population does not assure generalizability. Furthermore, although our study successfully predicted the outcome of the 2020 presidential election, it overpredicted the popular vote difference between the two major candidates. Finally, initial surveys assessing public resistance to a newly discovered vaccine may overestimate actual resistance as social norms and physician advice evolve. For example, an early survey of Americans after the initial polio vaccine became available found substantial reluctance. However, actual vaccination rates soon rose considerably beyond predictions from early surveys (19).

In summary, vaccinations have their largest potential to end the CODID-19 pandemic if they are widely accepted and used. Regulators had planned for the likely scenario that a vaccine would provide protection for about 50% of those who receive it. If a vaccination becomes available that provides protection for a sustained period for about 50% of those who receive it and is taken by only half the population, only 25% of population would be protected. Under this scenario, those vaccinated remain at great peril if exposed to others who decline vaccination. On the other hand, coronavirus vaccines might be more effective than originally anticipated. Results from two phase 3 clinical trials indicate that at least two vaccines are 95% effective in preventing cases of COVID-19, and our results indicate that expected efficacy is the most important factor in the decision to accept a vaccine (19). Continued research to track changes in factors that influence vaccine acceptance will improve public health policymaking as clinical trial results, post-Food and Drug Administration approval studies, and public attitudes continue to evolve (https://en.wikipedia.org/wiki/2016_United_States_presidential_election; ref. 10).

Data Availability

All study data are included in the article and/or supporting information.