Digital public health interventions at scale: The impact of social media advertising on beliefs and outcomes related to COVID vaccines

Significance

This paper analyzes whether social media advertising can be a cost-effective tool to influence attitudes and beliefs about COVID-19 vaccines and ultimately vaccinations. We conduct a meta-analysis of more than 800 experiments conducted by 174 public health-related organizations. Each experiment randomly assigned advertisements to a subset of users and measured the outcome, attitudes, and beliefs, through surveys. Although each experiment individually has too few subjects to reliably detect small effects, by pooling the data, we obtain more precise estimates of the overall average effect. The estimated average cost per additional influenced person, $3.41, and the estimated cost per additional vaccination, $5.68, can be compared to the social benefit when evaluating the cost-effectiveness of public health campaigns.

Abstract

Public health organizations increasingly use social media advertising campaigns in pursuit of public health goals. In this paper, we evaluate the impact of about $40 million of social media advertisements that were run and experimentally tested on Facebook and Instagram, aimed at increasing COVID-19 vaccination rates in the first year of the vaccine roll-out. The 819 randomized experiments in our sample were run by 174 different public health organizations and collectively reached 2.1 billion individuals in 15 languages. We find that these campaigns are, on average, effective at influencing self-reported beliefs—shifting opinions close to 1% at baseline with a cost per influenced person of about $3.41. Combining this result with an estimate of the relationship between survey outcomes and vaccination rates derived from observational data yields an estimated cost per additional vaccination of about $5.68. There is further evidence that campaigns are especially effective at influencing users’ knowledge of how to get vaccines. Our results represent, to the best of our knowledge, the largest set of online public health interventions analyzed to date.

Throughout the rapidly evolving COVID-19 pandemic, policymakers and public health agencies needed to communicate with citizens about mitigation measures ranging from mask wearing and social distancing to vaccines. Advertising on social media emerged as a popular channel to quickly reach large numbers of people and has been used by public health organizations in nearly every country both to convey information and influence behavior. An understanding of the expected impact of these campaigns is important as such organizations continue to engage in interventions as the pandemic unfolds. Assessing these campaigns is further valuable as digital public health interventions become increasingly used to address broader health-related outcomes.

To speak to these questions, this paper aims to evaluate the impact of social media advertisements on a variety of COVID-19-related outcomes. Analyzing advertising campaigns run on Facebook and Instagram by 174 public health organizations around the world, we investigate three main questions. First, what effect did these social media advertising campaigns have? Second, how cost-effective were they? Third, which types of outcomes have the campaigns been most effective at influencing?

The campaigns in our sample were run between December 2020 and November 2021, reached users in nearly every country, and in aggregate consist of $39.4 million dollars of advertising spending. They include a wide range of public health organizations that span major multinational nonprofits, public health ministries, and local nongovernmental organizations. The identities of the individual advertisers are not included in this article to protect their confidentiality. Importantly, our dataset contains the near universe of relevant experiments that were run on Facebook and Instagram over this period.^* This feature of our sample allows us to draw conclusions that are not vulnerable to selection biases that commonly arise in meta-analyses, such as publication bias, whereby experiments with positive outcomes are more likely to be included in the sample. To our knowledge, the dataset we analyze is the largest set of online public health interventions studied to date.

The data have two key features relevant to our analysis. First, we have data from a large number of the campaigns that conducted experiments where exposure to the ads was randomized at the user level, allowing us to assess the causal effect of each campaign. This is especially important in the context of online advertisements, where selection bias is a significant obstacle in nonexperimental data (1–4). Randomized experiments have become more common in online advertising, and companies (including Meta) have developed standardized experimentation tools to facilitate testing. The campaigns we analyze all used these tools to conduct experiments.

Second, we are able to combine the experiments with user-level survey data for a subset of users. The surveys ask a variety of questions, namely, a user’s willingness to get a COVID-19 vaccine, belief in the importance of vaccination, belief in vaccine effectiveness, belief in vaccine safety, whether the advertiser is a trustworthy source of COVID-19 information, how knowledgeable the user feels about how to get the vaccines, and whether they think vaccines are socially acceptable (in this paper, we will refer to these questions shorthand as Willingness, Importance, Effectiveness, Safety, Trustworthy Source, Knowledgeable, and Social Norms, respectively). While not all advertisers asked all questions, the survey questions were largely standardized across campaigns, facilitating comparisons. As is common practice with such experiments, we classify responses into a binary outcome according to whether respondents were engaging in the public health behavior of interest (for instance, intending to get the vaccine) or had the relevant public health information (for instance, knowing where to get the vaccine). Looking across all studies, we can then see whether interventions had an impact on the binary outcome of interest (the wording of each question, possible responses, and how answers were classified are all listed in SI Appendix). This approach is similar in spirit to the approach taken by other papers that aggregate a set of experiments with distinct outcomes.^†

Overall, our combined findings suggest that these campaigns were effective at influencing peoples’ attitudes and beliefs about the vaccine. We find an average increase in the fraction of positive responses of 0.55% points (P = 2e-13) across all experiments, with a baseline 55.7% positive rate. While this point estimate is small on a per-person basis, the reach of the campaigns implies that even under conservative assumptions, around 11.6 million individuals were influenced by these campaigns alone, at a cost of about $3.41 per incremental person. Translating this estimate into a cost for incremental vaccinations requires additional assumptions and data; the survey outcome can be considered a “surrogate” for the outcome of interest, vaccination (8).^‡ In SI Appendix, we present the results of an analysis where we use data from the United States to estimate the correlation between county-level vaccination rates and county-level survey responses, finding a correlation of .6 (standard error .0174) in a sample of 2,710 counties with more than 20 survey responses. Using this estimate together with the result of our meta-analysis implies that the cost of an incremental vaccine is $5.68. These estimates suggest that campaigns may be an easily scalable intervention that can in aggregate shape the public health outcomes of a large number of citizens.^§ A limitation of this cross-validation approach is that we are ultimately interested in the relationship between marginal survey responses and marginal vaccination rates in response to the ads, whereas we observe a cross-sectional relationship between average survey responses and average vaccination rates at the county level. Our cost calculation approach is based on an assumption that a change in survey responses due to treatment would lead to a change in vaccination rates matching the observed cross-sectional correlation. Since we cannot link treatment to changes in vaccination status, we cannot directly assess whether a change in survey responses due to treatment would lead to a change in vaccination rates that matches the observed cross-sectional correlation. This requirement would fail if, for example, the effect of advertising was very short-lived or if there are barriers to vaccinations among individuals close to the margin of getting vaccinated that are not reflected in the average relationship between survey responses and vaccination decisions.

These results can be broadly compared to those from other initiatives aimed at influencing vaccination decisions.^¶ Barber and West (11) and Sehgal (12) estimate a $68 and $49 cost per incremental COVID-19 vaccination in Ohio from the Vax-a-Million lottery (though see refs. 13 and 14); in a separate study, Campos-Mercade (15) find a cost per incremental vaccination on the order of $400; and Krieger et al. (16) estimate costs of $88 to $380 for incremental flu vaccinations in seniors in the United States. Larsen et al. (17) find a much lower cost, specifically $1 per incremental COVID-19 vaccination from a location-randomized YouTube advertising experiment in select counties in the United States. The World Health Organization (18) has also explored methods of effective communication for influencing health outcomes, including through randomized experiments run on Meta. Since our data do not include information about actual vaccination decisions, it is hard to directly contrast our estimates to those. However, such estimates highlight how challenging it can be to influence health behaviors and the potential value of identifying low-cost, scalable interventions. When considering such interventions to influence vaccine uptake, there has been much advocacy for behaviorally informed promotions (19, 20). The campaigns we analyze broadly fit into this category.

Our findings also connect to the literature on health nudges, which similarly tends to focus on low-cost, scalable interventions. Much of this literature has focused on text-based interventions, which have shown potential across a number of domains, ranging from flu appointments to court appearances (21–23). Specifically, in the context of COVID-19, Dai et al. (24) sent participants in California text-based reminders to make vaccination salient and easy to remember. They find that reminders sent 1 d and 8 d after notification of vaccine eligibility increased vaccination rates by 3.57% points and 1.06% points, respectively. Banerjee et al. (25) analyze the effect of a video message randomly distributed via SMS to millions of individuals in West Bengal, India; they find substantial effects on both the treated individuals as well as nontreated community members on a broad range of COVID-related outcomes. One advantage of text-based interventions is that they may be more salient and thus have a larger effect relative to our effects. On the other hand, advertising campaigns do not require gathering phone numbers and may thus be more easily scaled to a large population.^#

Our results can be related to prior large-scale meta-analyses of online advertising. The literature has highlighted major challenges due to low statistical power (28). Meta-analysis is a natural way to address this challenge, but it requires access to data from the experiments of many advertisers and also creates challenges comparing effectiveness across heterogeneous advertising objectives. We are aware of only three other meta-analyses of digital advertisements that have a comparable scale to our study.^ǁ First, Johnson et al. (29) used internal data from Google’s display advertising platform to study the effect of digital advertisements on website visits for 432 digital advertising campaigns, finding effects of 8% of baseline website visits. Second, Goldfarb and Tucker (32) analyzed 2,892 experiments carried out by a brand research firm, each using a similar survey methodology and sample size to the experiments considered here, finding an effect of about 10% of the baseline on survey responses concerning intention to purchase. Third, Gordon et al. (33) analyze more than 600 advertising experiments on Facebook, comparing the estimated effects for different measures along what is referred to as a “funnel” or a customer’s journey to a final action of interest. They find effects of 28%, 19%, and 6% for measured outcomes that capture consumer behavior at the top, middle, and lower parts of the funnel. In our paper, we conduct a large-scale meta-analysis of online public health campaigns across multiple outcomes. Our estimated effects (about 1% over baseline) are substantially smaller than the effects found in these studies, suggesting that it is more difficult to change attitudes and beliefs about vaccination than it is to increase more standard advertising outcomes. Similar to ref. 33, we find smaller effects for outcome measures that are closer to the ultimate outcome of interest, vaccination.

Building on our main results, we next look across the different survey outcomes to see which ones are most impacted by the campaigns. We find significant effects on Knowledge, Safety, Social Norms, and Importance (all have P <  0.001), while no significant effects on Willingness, Effectiveness, or Trustworthy Source. There were individual campaigns that were able to significantly move these last three metrics, but we could not detect an overall average effect. Finally, we find evidence that the campaigns may have been particularly effective at shifting users’ knowledge around the vaccines. Knowledge has the largest treatment effect point estimate (1.23% points, P = 5e-7), and it is significantly higher than nearly all the other coefficients.^** This suggests that, on average, the digital advertisements in our sample may have been a particularly cost-effective channel for information dissemination or that information is easier to retain. In interpreting these comparisons, it is important to recall that different campaigns were designed for different purposes, and some campaigns may have conducted experiments on both primary and secondary outcomes so that smaller impacts might be expected for secondary outcomes. For example, a campaign focused on providing information might have evaluated its impact on Knowledge but also on Willingness. If Knowledge was less commonly included as a secondary outcome, it might be expected that the measured impact would be higher. In SI Appendix, we investigate this hypothesis by manually classifying advertisements as to which outcomes are their primary ones, and we find similar patterns (our manually labeled sample is smaller, but we see that the point estimate for Knowledge is again significantly larger than that for Importance, Safety, and Willingness; it is nonsignificantly different from that for Effectiveness and Social Norms).

Overall, our results suggest that social media advertising campaigns can be an important component of public health initiatives. Over the course of the past 2 y, health-oriented organizations have engaged in a wide range of tactics in an effort to shift attitudes and behaviors (see https://www.nga.org/center/publications/covid-19-vaccine-incentives/ for a list of different incentives offered in the United States alone, ranging from Girl Scout cookies to laps on a NASCAR track); key challenges with many of these include scalability, measurement, and generalizability. Digital advertising can help overcome these challenges. However, the small per-person impact highlights that these campaigns are best thought of as part of a broader set of strategies. To this end, our paper complements the growing literature on designing effective public health interventions for COVID-19.

The rest of our paper is organized as follows. The Data describes our sample, the outcome variables, and the approach we use to analyze the campaigns; the Results presents findings in greater detail; and the Conclusion summarizes. Additional analyses are provided in SI Appendix.

Data

Overview.

We analyze a set of 819 randomized experiments that were conducted between December, 2020, and November, 2021. The experiments in our sample are derived from 376 distinct advertising campaigns and 174 organizations. The advertising experiments and surveys were conducted by Meta and the public health agencies prior to our analysis. There are often multiple experiments associated within a single ad campaign, where each experiment corresponds to a specific survey outcome. For example, an advertiser may take one campaign and run three separate experiments that measure the impact of the campaign on Willingness, Importance, and Effectiveness. The average campaign in our data ran slightly more than two experiments; in other words, advertisers measured the impact of their campaigns on an average of about two outcome metrics each (the platform normally caps the number of questions per campaign at three).

The studies were all conducted using Meta’s infrastructure for conducting advertising effectiveness experiments across Facebook and Instagram; through the rest of this paper, we refer to this infrastructure as “the platform.” We focus specifically on experiments that measure the extent to which advertisements affect individuals’ attitude or beliefs as measured by survey questions.^††

We limit the set of experiments to those measuring outcomes in one of seven categories mentioned earlier (SI Appendix for details).^‡‡ Though not an exhaustive set of COVID-19 ad experiments, these seven categories were selected because they are the most prevalent across COVID-19 vaccine-related experiments. We use all studies that asked these questions and, following the platform’s policy (see https://www.facebook.com/business/help/2396060560411130), restrict to users aged 18 y and older. For our analysis, we used deidentified data for all campaigns.

The campaigns we study total $39.4 million in ad spend, with a reach of 2.1 billion unique users translated across 15 languages. The average campaign costs more than $100,000 and reached nearly 13 million people; these were substantive efforts, but importantly also not beyond the budget of what many public health organizations could conceivably spend on similar campaigns in the future. Table 1 provides summary statistics.

Table 1.

Summary statistics by outcome metric

Category	Effectiveness	Importance	Knowledge	Safety	Social norms	Trustworthy source	Willingness	Overall
# Experiments	64	237	94	218	107	59	40	819
# Unique campaigns	48	234	73	218	107	59	40	376
# Unique organizations	32	109	57	100	50	30	17	174
Earliest campaign start	2021-02-17	2020-12-08	2021-02-19	2020-12-15	2021-02-03	2021-02-24	2020-12-08	2020-12-08
Latest campaign end	2021-11-15	2021-11-12	2021-11-03	2021-11-15	2021-11-14	2021-10-21	2021-06-15	2021-11-15
Avg. # people reached	17,932,855 (3,216,101)	12,833,435 (1,353,134)	15,341,751 (2,957,774)	11,959,068 (1,746,021)	12,497,147 (2,396,430)	9,510,847 (1,971,257)	27,297,621 (6,773,069)	12,913,047 (1,225,276)
Avg. # survey resp per experiment	1,753.53 (151.09)	2,130.65 (90.27)	1,734.60 (128.96)	1,931.67 (90.81)	1,510.58 (120.24)	1,762.17 (148.21)	1,429.00 (155.72)	1,860.94 (45.75)
Avg. campaign cost	$136,807 ($41,543)	$65,487 ($6,376)	$122,796 ($36,000)	$132,491 ($26,694)	$63,556 ($13,409)	$104,177 ($24,353)	$392,678 ($142,627)	$105,183 ($16,872)
# Experiments rejecting no effect (0.1)	7	23	22	29	14	3	1	99
# Experiments rejecting no effect (0.05)	5	16	17	20	8	3	1	70
Implied false discovery rate (0.1)	0.914	1.000	0.427	0.752	0.764	1.000	1.000	0.827
Implied false discovery rate (.05)	0.640	0.741	0.276	0.545	0.669	0.983	1.000	0.585
# FDR survivor experiments (10% FDR)	2	14	5	5	1	2	0	27

Note: Standard errors in parentheses. # Experiments Rejecting No Effect (0.1) references the number of studies that are significant at the 0.1 level in a two-tailed t-test against the null of no treatment effect. Implied False Discovery Rate (0.1) estimates the false discovery rate if we accepted all experiments that were significant at the 0.1 level, as FDR(0.1)=(0.1 * n_e)/(n_rej), where 0.1 is the level of significance, n_e is the number of experiments, and n_rej is the number of experiments rejecting no effect at 0.1. # FDR Survivor Experiments is the number of experiments determined via the Benjamini–Hochberg algorithm to survive a false discovery rate of 10% (34).

Although the campaigns we study reached billions of users, as we can see in Table 1, we observe a much smaller number of survey responses. The platform provides the experimentation service to enable advertisers to estimate the incremental impact of their campaigns on survey-based outcomes; however, the number of responses per experiment is limited by the platform. The limits are presumably motivated by the fact that users may be willing to engage in only a small number of surveys, and the user experience may be negatively impacted by too many surveys. Thus, the platform caps the number of respondents per study to balance the tradeoff between statistical power and user experience (SI Appendix for more details on how these experiments are implemented).

In total, our dataset incorporates 1.5 million total responses across all experiments. Per experiment, the number of responses ranges from 300 to 4,507, split across test and control groups. This does not give us too much power for each individual experiment. If an experiment of average sample size (1861 in our sample) and average baseline positive response rate (55%) were to be analyzed using a difference in means between treatment and control, the minimum detectable effect size with 80% power and a 10% significance level would be about 0.06, close to 10% of the baseline. The average effects we find below are an order of magnitude lower than that (in response to an early draft of this manuscript, the platform started granting more exceptions to the normal response caps on these experiments in order to deliver better-powered results). Indeed, we see that only 99 out of the 819 experiments rejected the null at the 10% level, just slightly greater than the 10% of experiments that would be expected if there were no treatment effects.

Only 27 experiments on their own are included in a set of experiments determined to have a 10% false discovery rate using the Benjamini–Hochberg procedure (34). We also report the implied false discovery rate for the set of experiments that are individually significant at the 5% level, and we repeat this for the 10% level. This exercise is motivated by the idea that an organization might choose to further scale a campaign after seeing a statistically significant impact. We see that if organizations used the 5% threshold for scaling, for the Knowledge outcome, the false discovery rate would be about 1/4, while for the Importance outcome, the rate would be about 3/4. Although such a false discovery rate might not be problematic, as it is unlikely that the campaigns would be harmful and in aggregate these campaigns would be cost-effective, our findings as a whole suggest that we are not well powered to well identify individually effective campaigns. This motivates the approach we pursue in this paper of conducting a meta-analysis of hundreds of experiments together rather than seeking to identify individual campaigns with positive effects.

In SI Appendix, we provide graphs of the CDFs of P-values both overall and for each metric, where we can see that overall and particularly for the Knowledge outcome, the CDFs of P-values depart from the uniform distribution that would be expected if there were no effects.

Survey Questions.

We provide more detail on the questions, their possible responses, and their coding in SI Appendix. In Table 2 we provide text from our standardized survey questions as a reference.

Table 2.

Overview of survey questions

Question category	Wording
Importance	How important do you feel a vaccine is to prevent the spread of COVID-19?
Safety	How safe do you think a COVID-19 vaccine is for people like you?
Willingness	How likely are you to get vaccinated for COVID-19 when the vaccine is available to you?
Effectiveness	How effective do you think the COVID-19 vaccination is in preventing COVID-19?
Knowledge	Do you know where people in your local community can go to get a COVID-19 vaccine?
Social norms	When you think of most people whose opinion you value, how much would they approve of people getting a COVID-19 vaccine?
Trustworthy source	Do you agree or disagree that [advertiser name] is a trustworthy source of COVID-19 vaccine facts and information?

Validity of Survey-Based Outcomes.

We now turn to consider the validity of the survey-based outcome measures, and in particular, the extent to which they (or changes in them) do not capture changes in beliefs or knowledge. Privacy and legal constraints prevent advertisers from asking about or measuring some ultimate quantities of interest on platform (e.g., health or vaccination status), but the self-reported measures may still be meaningful.^§§ Here, we discuss two categories of potential concerns about these measures.

A first category of concerns relates to whether the survey outcomes as entered in the platform reflect beliefs and behaviors in the physical world. There are several considerations. Following established practice for social media brand campaigns, survey outcomes are a primary outcome that public health organizations have been using to evaluate their campaigns. Campaigns start, stop, and change based on the results of these experiments, dictating how entire ad budgets of COVID-19 interventions are spent. Hence, it is important to understand how these outcomes have responded to campaigns to date and to add to the understanding of public health organizations whose individual experiments to date have been underpowered to detect small effects.

Relatedly, a common goal of public health organizations is to simply shift attitudes and beliefs. Akin to traditional advertisers where campaigns may target different levels of the conversion funnel, many of these advertisers are aiming to move awareness or basic beliefs and may invest in complementary tactics to change behavior once beliefs have been influenced. To the extent that the implementation details and survey responses provide insight into awareness and beliefs, the campaign experiment outcomes are informative.

Finally, for social media campaigns in general, there is evidence that responses to platform surveys correlate reasonably well with behaviors of interest. Moehring et al. (36) find an R² of 0.83 in a regression of country-level vaccine uptake on self-reported vaccine status collected from a survey on Facebook. And Astley et al. (37) find a correlation between survey metrics on Facebook and off-platform COVID-19 cases (see refs. 38 and 39 for further discussions and caveats). Alekseev et al. (40) find a high degree of correlation between characteristics of businesses that Facebook users self-report to own and offline statistics from the US Census. Although the contexts and analyses from these studies are different, together, they suggest that there does appear to be informative signal in social media survey outcomes.

In SI Appendix, we explore the relationship between survey response positivity and county-level vaccine takeup and find that the two are strongly correlated. We use these findings to extrapolate an estimated cost of each additional vaccination.

A second category of concerns about survey results relates to whether the differences in survey outcomes between treated and control groups can be interpreted as the causal effect of the advertisements (1). One issue is that there can be systematic differences between the treatment and the control group due to the implementation of the randomized advertising experiment. Details are provided in SI Appendix, but in short, randomization of assignment to ads takes place just before an ad was intended to be shown to a user, so that whether a user is sent an ad is random within the experiment. However, after seeing an ad, whether a user in the treatment group is subsequently shown a survey depends on an additional factor that is not present for the control group. In particular, if after randomization into the treatment group, the platform intends to show the user an ad but the user scrolls past it or does not scroll to it at all, the user will not be sent a survey. This is done to only capture survey responses from users in the treatment group who actually saw the ad, but to the extent to which this behavior is correlated with the survey outcome, it could lead to confounding and thus biased estimates of treatment effects.

In addition, even if the set of users who were sent surveys was perfectly randomized within each experiment, there still is the potential for differential survey response between the treatment and the control group. This might occur if individuals influenced by the ads were more likely to respond to the survey. We address both of these issues in SI Appendix, where we show that along several observable dimensions, the treatment and control groups are similar. We further address these concerns by adjusting for several observable characteristics of individual respondents in our analysis.

A final concern is that the population answering the surveys differs from the overall target population along unobservables correlated with our outcome variable. For example, certain age groups may be more likely to answer the surveys. Poststratifying our results by age and gender (as we do) is an industry standard approach to address this concern; in addition, in SI Appendix, we also compare observables from a poststratified sample with those from the target population and find reasonable overlap.

Methods

We conduct a meta-analysis by first analyzing each experiment separately and then using inverse variance weighting to generate the average effect for each outcome and overall.

Meta-Analysis of Experiments.

We begin by analyzing each experiment separately using the following weighted linear model:

$$\begin{array}{c}\hfill {\text{response}}_i=({\tilde{X}}_i^{\prime }\beta +{\beta }_0)W_i+{\tilde{X}}_i^{\prime }\gamma +{\epsilon }_i,\end{array}$$ [1]

where response_i is an indicator for whether individual i gave a positive response, ${\tilde{X}}_i$ is a matrix of de-meaned controls that could potentially be related to outcomes (age bucket, gender, expected click-through rate, and expected conversion rate), and W_i is a dummy variable denoting whether i was in the treatment or control group. The expected click-through and conversion rates are platform-generated estimates age buckets are 18 to 24, 25 to 34, 35 to 44, 45 to 54, 55 to 64, and 65+. We de-mean and interact our covariates with the treatment indicator so that ${\widehat{\beta }}_0$ remains an unbiased and consistent estimate of the average treatment effect even in the presence of heterogeneous treatment effects by our covariates (41).

In our regression, each response is weighted with poststratification weights by age bucket and gender within the treatment and control group, such that both arms of each experiment are representative of the population reached by the relevant campaign. That is, we obtain the proportion of users in each age bracket and gender group reached by the campaign associated with a given experiment and divide it by the proportion of responses to obtain the weight for each response (to reduce variance, they are trimmed to an upper bound of 3 and a lower bound of 0.3 to decrease the influence of outlying observations. Rerunning without any trimming yields no material difference in the results). We poststratify by these two variables as age and gender are basic demographics that advertisers are both frequently interested in and where heterogeneous effects are often observed.^¶¶ In SI Appendix, we explore robustness to different weighting schema and find no material difference in the results.

This procedure is based on the one that Meta uses to analyze results from these survey-based experiments for advertisers. Repeating this approach for each of the experiments in our dataset, we are left with 819 estimates of average treatment effects and standard errors. The next step in the analysis is to combine these point estimates into estimated effects by outcome metric and to generate an overall, combined estimate, following standard meta-analytic methods (42, 43).

Specifically, to generate the average effect for each outcome and overall, we combine the respective experiments using inverse variance weighting. This approach estimates a single, homogeneous effect per category while minimizing variance. We present this approach due to its simplicity and the fact that there is no evidence of heterogeneity across all outcomes; in SI Appendix, we report several alternative specifications that allow for greater heterogeneity across experiments and outcomes and find very similar results.

Results

We now turn to our main results, which we break into two categories. First, we describe for each survey outcome, the average effect across all experiments that focused on that outcome. Second, we combine the treatment effect estimates with data about the number of unique people who received advertisements as well as the total cost of the campaigns to estimate how many people have been influenced and what the cost per influenced person is.

The results of our meta-analysis, including the average effect for each outcome and overall, are described in Table 3.

Table 3.

Meta-analysis of experiments by outcome

Category	Effectiveness	Importance	Knowledge	Safety	Social norms	Trustworthy source	Willingness	Overall
Treatment coefficient	0.0045 (0.0029)	0.0043*** (0.0012)	0.0123*** (0.0025)	0.0062*** (0.0016)	0.0081*** (0.0024)	0.0012 (0.0027)	0.0010 (0.0042)	0.0055*** (0.0008)
P-value	0.114	0.0004	5e-7	8e-5	0.0006	0.639	0.807	2e-13
Cost per influenced person	$2.43	$2.41	$0.77	$3.20	$1.00	$11.69	$17.14	$3.41
Baseline positive response rate	0.505	0.672	0.575	0.501	0.556	0.365	0.517	0.557
Treatment effect as % of baseline	0.89%	0.64%	2.14%	1.24%	1.46%	0.33%	0.19%	0.99%
Power calculations (approximate)
Minimum detectable effect	0.0071	0.0030	0.0061	0.0039	0.0059	0.0066	0.0104	0.0019
Power to detect given effect size	0.474	0.973	1.000	0.989	0.962	0.120	0.081	1.00
# experiments needed for 80% power	159	115	23	87	57	1,660	4,133	94
# survey resp. per exp for 80% power	4,349	1,036	426	770	801	49,567	147,635	214

Note: Standard errors in parentheses. For each column, we consider the set of associated experiments and calculate the inverse-variance weighted average treatment effect (row 1). The baseline % positive response is an unweighted mean across all the relevant experiments; calculating it using fixed or random effects models changes the numbers only slightly. The cost per influenced person for each subset is calculated using the spend and number of unique people reached across all campaigns in the relevant subset. Finally, we include power calculations based on the standard error of the treatment effects (abstracting away from heterogeneity across experiments). Power to detect a given effect size is calculated at the α = .1 level; for the last two rows, we want to convey how power could be improved by either increasing the number of experiments or the number of surveys per experiment. For those calculations, we ask how much of either we would need holding the other fixed to have 80% power to detect the given estimated treatment effect. SI Appendix for details.

Several comments are salient to the interpretation of these results. First, this dataset is very broad. Past efforts to understand what has and has not worked with shifting behaviors around COVID-19 have often by necessity studied a small number of treatments at modest scale or been one off ex post analyses. External validity with such studies is frequently a concern and potentially helps explain why studies to date have found conflicting results (e.g., refs. 12–15, 44 on effects of financial incentives on vaccination rates). In contrast, here, pooling hundreds of studies from a broadly representative population, we find a positive and statistically significant average main effect. The evidence may not be conclusive yet on what kinds of behavioral nudges work, but this is evidence that these digital advertising campaigns can help move the needle on COVID-related attitudes.

Second, consider results about the specific outcomes. We find that Importance, Knowledge, Safety, and Social Norms showed highly statistically significant effects. In contrast, we do not detect an effect for Effectiveness, Trustworthy Source, or Willingness. (Though Effectiveness is close to marginally significant in our main specification.)

We note that these last three metrics, particularly Effectiveness and Willingness, are arguably lower in the vaccine conversion funnel (that is, they are better proxies for a final desired action) than the first four metrics. A stylized fact from advertising is that such “lower-funnel” behaviors often see smaller effect sizes than more upper-level outcomes and are generally challenging to study as they may be influenced by a variety of unmeasured factors. While our finding thus accords with this intuition, a limitation of our study is that even with a very large sample size, we are not powered to generate more precise estimates of these averages (see the final rows of Table 3).

Third, we note that there is evidence that the estimated lift for Knowledge (1.23pp) is significantly higher than our other metrics. Our estimate for Knowledge is significantly greater than that for Effectiveness (P = 0.038), Importance (P = 0.003), Safety (P = 0.034), Trustworthy Source (P = 0.002), Willingness (P = 0.020), and our overall estimate (P = 0.008), including when dropping the Knowledge experiments (P = 0.006). It is nearly significantly greater than Social Norms in a one-sided t-test (P = 0.109). We note that Knowledge is a distinct outcome here in that the other outcomes relate more to persuasion, whereas Knowledge focuses simply on conveying information. These results suggest that social media campaigns may be particularly attractive for public health organizations interested in the latter.^## (See also our additional analysis in SI Appendix with manually labeled ads).

Finally, we note that in SI Appendix, we explore different specifications and find broadly similar results. In addition, the standard output provided to advertisers on Meta comes from a hierarchical Bayesian model; we chose a frequentist approach due to its simplicity, but in SI Appendix, we show robustness to using a similar Bayesian approach.

Number of Influenced People, Cost per Influenced Person.

Conditional on our results, how many people were influenced by these campaigns, and how cost-effective were they? As aforementioned, the survey data comes from only a subset of the overall users who saw the ads; to calculate the number of influenced people, we follow the common industry practice and scale the point estimate of the treatment effect from each experiment by the size of the overall population that saw the campaign. In our case, since we have data across many advertisers and some users were shown ads from multiple campaigns, to generate a (conservative) estimate of the number of influenced people per campaign, we can treat these collective campaigns as effectively one large one. Specifically, we combine the total spend, total unique reach, and our estimate of the average treatment effect to generate a back-of-the-envelope estimate of the number of people who were influenced by this combined effort.

Doing this calculation, we estimate that about 11.6 million people were influenced by these campaigns. To be clear, by “influenced” we mean shifted self-reported beliefs to a positive outcome; this does not capture people who, for example, moved along the intensive margin of these categories (in SI Appendix, we explore a less conservative way of estimating the number of influenced people and thus the cost per influenced person).

Conditional on estimates of how many people were influenced, how much does it cost to influence someone? For this, we divide the number of influenced people by the ad spend, again as is typical in the industry. From Table 3, we can see that the average cost per influenced person was $3.41.

To understand how cost-effectiveness translates to real-world public health outcomes, in SI Appendix, we explore the relationship between survey positivity rate and vaccine series completion rate at the county level in the United States. Across survey outcomes, we find that for each additional positive survey response, we would expect to see about a 0.6 increase in vaccination takeup in the CDC data. Applying this to our average cost per influenced person from Table 3, this implies an estimated cost per additional vaccine of about $5.68.

While we are hesitant to extrapolate substantially outside our sample, these magnitudes suggest that running even a few million dollars’ worth of additional campaigns could achieve relatively large shifts in the baseline fraction of outcome variables.

Conclusion

Over the course of the pandemic, public health agencies increasingly leveraged social media advertising to pursue public health goals. Our results show that public health interventions via digital advertising are an effective medium for changing important self-reported beliefs and attitudes around COVID-19. Combining with nonexperimental data on vaccination rates, our results suggest that these campaigns were a cost-effective approach to increasing rates as well. The cost-effectiveness and scale of these campaigns can make them appealing to a broad range of organizations around the world. The use of social media advertising more broadly has the potential to aid in the pursuit of other health policy goals, ranging from childhood vaccination to hand washing.

Supplementary Material

Appendix 01 (PDF)

Data, Materials, and Software Availability

We are not sharing individual level data as part of this work. However, on https://www.github.com/gsbDBI/CovidAdMeta/, we provide a simulated dataset and code to reproduce all our main results.