Quasi-experimental methods for pharmacoepidemiology: difference-in-differences and synthetic control methods with case studies for vaccine evaluation

Abstract

Difference-in-differences and synthetic control methods have become common study designs for evaluating the effects of changes in policies, including health policies. They also have potential for providing real-world effectiveness and safety evidence in pharmacoepidemiology. To effectively add to the toolkit of the field, however, designs—including both their benefits and drawbacks—must be well understood. Quasi-experimental designs provide an opportunity to estimate the average treatment effect on the treated without requiring the measurement of all possible confounding factors, and to assess population-level effects. This requires, however, other key assumptions, including the parallel trends or stable weighting assumptions, a lack of other concurrent events that could alter time trends, and an absence of contamination between exposed and unexposed units. The targeted estimands are also highly specific to the settings of the study, and combining across units or time periods can be challenging. Case studies are presented for 3 vaccine evaluation studies, showcasing some of these challenges and opportunities in a specific field of pharmacoepidemiology. These methods provide feasible and valuable sources of evidence in various pharmacoepidemiologic settings and can be improved through research to identify and weigh the advantages and disadvantages in those settings.

This article is part of a Special Collection on Pharmacoepidemiology.

Introduction

Determining the safety and efficacy of drugs and biologics is crucial, yet it faces many statistical, epidemiologic, regulatory, and logistical challenges. A related yet distinct challenge—determining the effects of health and social policies—has incorporated quasi-experimental study designs into its set of methods. Quasi-experimental methods can provide a convenient means for assessing the real-world effectiveness and safety of pharmacological products—especially those with population-level effects—in well-defined populations and specific circumstances that mitigate the risks of bias.

Quasi-experiments (in some cases also known as “natural experiments”) here refer to observational studies that identify a causal effect by taking advantage of circumstances that create variation in exposure status in ways that avoid the usual sources of confounding for that exposure and the outcome of interest.^1,2 This often comes through changes in the exposure that are not otherwise causally connected to the outcome, often referred to as exogeneity.^2,3

A suite of such approaches developed or formalized in the quantitative social sciences—including instrumental variables, regression discontinuity designs, interrupted time series, difference-in-differences, and synthetic control methods, among others—have become major empirical approaches in those fields^3,4 and have become popular in epidemiology as well.⁵ For example, during the coronavirus disease 2019 (COVID-19) pandemic, they have been used to assess the effectiveness of universal masking in schools,⁶ vaccine lotteries,⁷ and other interventions.^8,9 Along with this rise in the use of the methods have come articles describing the uses and limitations of these methods in health policy evaluation.^1,2,10-16

This article focuses on difference-in-differences (DiD) and the synthetic control method (SCM), quasi-experimental designs that are both controlled (ie, include exposed and unexposed units) and longitudinal (ie, exploit a change in exposure status over time to adjust for unmeasured confounders).^11,17

Despite their expanding use in health policy research, these methods have not been extensively studied for use in evaluating drugs and biologics. This article provides a brief overview of these methods (including recent developments from the econometrics literature), advice on settings where they are beneficial for pharmacoepidemiology (exemplified through 3 case studies of vaccine effectiveness evaluation), and discussion of the trade-offs and limitations they face. Finally, the article describes future research that can improve their use in pharmacoepidemiology.

Overview of methods

DiD and SCM analyses estimate the average treatment effect on the treated (ATT) by constructing the potential outcome for the exposed units in the (counterfactual) absence of the exposure using both cross-unit and within-unit comparisons. Below, I give a brief overview of these methods and their estimators, along with recent extensions. For a more comprehensive treatment, see the relevant econometrics^3,4 or public health review literature.^{1,10,14,15,17}

Difference-in-differences

Used (in function if not name) by John Snow in his investigation of the 1854 London cholera outbreak,^3,18,19 the DiD method grew in popularity after Card and Krueger’s 1994 study on minimum wage increases.²⁰ DiD methods rely on the parallel trends assumption: In the absence of exposure, the exposed and unexposed units would have equivalent trends over time in their expected outcomes. For the canonical “2-by-2” DiD analysis, with one preexposure and one postexposure period for the exposed unit and the same 2 time periods for the unexposed units, the ATT estimate is given by the difference between the pre–post difference in the exposed unit and the pre–post difference in the unexposed unit (see Figure 1A). For multiple units and time points, this is commonly extended into the following 2-way fixed effects (TWFE) regression model²¹:

where is the outcome of interest for unit i in time period t, are the unit-specific fixed effects, are the period-specific fixed effects, and is an indicator of whether unit i was exposed in period t. The ATT estimate is then given by . The fixed effects control for within-unit confounding over time and within-period confounding between exposure groups.

Figure 1.

Schematics showing simulated observed and counterfactual data for 4 quasi-experimental methods: A) difference-in-differences (DiD); B) DiD with staggered adoption; C) synthetic control method (SCM); D) augmented SCM. Vertical lines represent the beginning of exposure in exposed units/groups. Black dotted lines (A, B) and gray shaded regions (C, D) represent estimated effects. Note that different counterfactuals are possible with additional assumptions (B) and that augmented SCM is shown here when the exposed unit lies outside of the convex hull of the unexposed units (D).

Extensions of the DiD model

Several extensions to the DiD model aim to improve the validity of the parallel trends assumption. For example, covariates can be added when parallel trends holds only conditionally, or the so-called difference-in-difference-in-differences (or triple-differences) model can be used to remove nonparallel trends.⁴ Both of these have analogous approaches in the epidemiology literature: adjusting for measured covariates in cohort studies for the former, and using negative controls to adjust for residual bias for the latter.^22,23

The functional form of the parallel trends assumption can also be changed by transforming the outcome variable. For example, if the counterfactual outcomes are judged more likely to have equivalent multiplicative trends than linear trends, a logarithmic transformation can be used on the outcome. Importantly, only one functional form can truly have parallel trends, so the modeling and estimand choices determine the validity of the causal assumptions made.²⁴

Recently, literature has extensively investigated DiD in cases where there are units with different exposure-onset times (eg, “staggered treatment adoption”; see Figure 1B).²⁵ The TWFE model is subject to biased estimation of the ATT under this setting as the unit and time fixed effects do not properly capture treatment effect heterogeneity.^21,25 Various approaches have been proposed to ameliorate this problem in the DiD and related stepped-wedge trials literature^26-30; these include different estimators and explicit weighting of time- and unit-specific estimators.^3,26,27

Synthetic control method

The SCM, developed for comparative case studies, has also become a common method for policy evaluation.^12,31 The core of the design is modeling the counterfactual outcome of the exposed unit using a weighted average of the unexposed units (see Figure 1C). For a setting with 1 exposed unit and n unexposed units (with outcomes in period t denoted and , respectively), the estimator for the ATT in period t is:

where are nonnegative weights summing to 1. These weights are chosen by minimizing the preexposure difference between the outcomes of the not-yet-exposed unit and the weighted average of the outcomes of the unexposed units; minimizing covariate differences can also be incorporated.^3,31,32

The major assumption of SCM is that these weights are stable, in the sense that a weighting scheme that matches the preexposure outcome trends well will also give an unbiased estimate of the potential outcome after the exposure time point.³¹ A long preexposure time series on which to optimize the weights provides the best justification for this assumption under many data-generating processes.³¹ For multiple exposed units and/or multiple postexposure periods, the individual estimators can be combined to target the desired estimand.^29,33

Extensions of the synthetic control method

One reason for the growth of SCM in comparative case studies is its interpretability: All unexposed units have a nonnegative weight showing their contribution to the estimated counterfactual, allowing discussion of the reasonableness of the counterfactual. However, this also imposes restrictions on the model; SCM does not work when the outcomes of the exposed unit fall outside the convex hull (loosely, the range) of the unexposed units’ outcomes.³¹ Thinking of past outcomes as the “adjustment variables,” this requirement is roughly analogous to positivity requirements in cohort studies: Adjustment fails if there are combinations of covariates for which there are only exposed units.

Several extensions of SCM—including the generalized SCM,³⁴ augmented SCM (see Figure 1D),³⁵ and Bayesian structural time series model³⁶—trade off some interpretability by allowing outcome modeling and extrapolation outside of the convex hull. This allows the use of more control series, including those on different scales than the outcome itself.³⁷ This extrapolation, however, can lead to bias, especially when there are few pretreatment periods or unexposed units.³⁴

Quantifying uncertainty and negative controls

Pharmacoepidemiologic studies require not just point estimates but also quantification of uncertainty, often through P values, confidence intervals, or Bayesian alternatives.³⁸ For quasi-experiments, this can be more challenging, as they often involve only 1 or a limited number of exposed observations, and rarely form a sample of an identifiable study population. Because of this, model-based estimates of variability—even in cases when they can be computed, such as regression-based DiD and generalized SCM—may not have the desired interpretation or may require additional design-based assumptions.²⁷

Beyond statistical inference, it is desirable to quantify the causal or model-based uncertainty. Negative controls, sensitivity analyses, and robustness checks provide a means to assess the required assumptions to some extent. These include assessing the validity of the assumptions for the particular setting through models and graphical inspection of preexposure time trends, as well as assessing the model performance in settings with known effects.¹⁷

Repeated negative controls—often called placebo tests or dummy analyses—conducted for different time points (where no change in exposure occurs), different units (ie, only using untreated units), or different outcomes (where the exposure should have no effect) generate null distributions using cases that should identify effects of zero.^3,31 Comparing the observed estimate with these distributions allows hypothesis testing. Bayesian frameworks are also feasible but may be sensitive to the prior distributions used.³⁶ Properly reporting and interpreting these measures of uncertainty remains a challenge for these quasi-experimental methods, especially as they are applied to new fields.

Case studies: quasi-experimental vaccine evaluation

Case studies of the use of DiD and SCM in vaccine evaluation can illuminate the benefits and challenges of these designs. Vaccine evaluation studies demonstrate the use of large-scale routinely collected datasets in the context of exogenous variation to identify key public health effects as a supplement to randomized trial evidence or when it is not available. They also show potential pitfalls: the risk of concurrent events and misspecification leading to bias and the limited generalizability of targeted estimands.

Case 1: SCM analysis of meningococcal vaccines

Prunas et al analyzed the impact of early childhood meningococcal vaccination programs in Brazil and England in the 2010s (see Table 1).³⁹ Real-world estimates of vaccine effectiveness for hard-to-predict and highly variable diseases like invasive meningococcal disease are challenging, making quasi-experiments particularly useful. Using SCM on the logarithmic scale with Bayesian time-series modeling, the authors compared meningococcal disease incidence after the program roll-out with a synthetic control constructed from time series of other (nontargeted) diseases and the targeted disease in older age groups who were ineligible for vaccination.

Table 1.

Summary of meningococcal vaccine analyses in Prunas et al,^a 2022.

	Brazil analysis	England analysis
Research question	Did meningococcal vaccination programs reduce early childhood invasive meningococcal disease incidence?
Exposure	Within the target age group (<1 year and 1-4 years) of the MenC vaccination program	Within the target age group (18-51 weeks and 1 year old) of the MenB vaccination program
Outcome	Total MenC cases (monthly)	Total MenB cases (quarterly)
Setting	Brazil, 2007-2013	England, 2011-2019
Control series	Cases for various off-target infectious diseases in target age group; outcome of interest in nontargeted age groups
Method	SCM with Bayesian variable selection
Scale	Logarithmic
Identification assumption	Weighted average of control series is the expected outcome absent vaccination program
Results	Vaccine effectiveness:	Vaccine effectiveness:
	• 69% (95% CrI, 51-80) in < 1-year-old age group	• 75% (95% CrI, 69-80) in 18-51–week age group
	• 64% (95% CrI, 55-70) in 1-4–year-old age group	• 72% (95% CrI, 65-79) in 1-year-old age group
Negative controls/placebo tests	• Models tested on nontargeted age groups and compared with other methods
	• Sensitivity analyses excluding nontargeted age groups as control series

Abbreviations: CrI, credible interval; MenB, meningococcal serogroup B; MenC, meningococcal serogroup C; SCM, synthetic control method.

^aPrunas et al.³⁹

To demonstrate the value of the method, the authors conducted placebo tests using other quasi-experimental designs and found that the SCM better captured seasonality and nonlinear time trends; notably it avoids the need to specify a parametric time trend. It thus likely suffers from less bias in the desired analyses, although there could still be extrapolation bias due to the time series modeling.^32,36

Within-region controls (here, time series of nontargeted populations and outcomes) are particularly valuable for quasi-experimental evaluation of vaccines and drugs. By avoiding comparisons across geographic areas, they mitigate the risk of bias due to concurrent events or changes. However, they come with a potential increased risk of spillover or contamination, especially in infectious disease settings.⁴⁰ For the nontargeted age groups, bias could be caused by indirect protection of the vaccine (ie, a younger child receiving the vaccine is less likely to infect an older child in the household). For the nontargeted diseases, there is perhaps less risk of bias, but it could still occur due to off-target effects of vaccination or changes in health-seeking behavior and diagnosis in the wake of the vaccination policy. The authors used placebo tests and sensitivity analyses that dropped the older age groups as candidate control series, finding robust results that suggest these biases were minimal in this setting.³⁹

The results demonstrated effectiveness of the vaccination program against the targeted infections. Within each country of analysis, results on different early childhood age groups showed remarkable similarity, indicating that there may be some generalizability of these effects. However, as ATT estimands, these may still not be appropriate to transport to other countries with different infection risks. On the other hand, these estimates assess the overall effect of the program, rather than the individual-level direct effect, and so can provide an additional piece of evidence that is rarely achieved in preauthorization trials.⁴¹ This is useful for public health policy-makers as real-world evidence of both the vaccine itself and the vaccination program.⁴²

Case 2: DiD analysis of an off-target vaccine against COVID-19

Early in the COVID-19 pandemic, researchers sought to use existing drugs and vaccines to prevent infection, illness, and severe outcomes until COVID-19-specific drugs and vaccines could be developed and authorized. The bacille Calmette-Guérin (BCG) vaccine against tuberculosis was one such candidate because of its hypothesized off-target effects,⁴³ which were supported by cross-sectional analyses showing better COVID-19 outcomes in countries with high BCG vaccination coverage.⁴⁴ While randomized controlled trials would eventually occur in specific populations,⁴³ the emergency situation called for interim evidence to confirm or refute this hypothesis. A preprint by Matsuura et al in 2020 reexamined this hypothesis with a DiD analysis using existing country-level data (see Table 2).⁴⁴

Table 2.

Summary of BCG vaccine off-target effects vs. COVID-19 analysis in Matsuura et al,^a 2020.

Research question	Did prior BCG vaccination recommendation reduce COVID-19 incidence?
Exposure	Within age-group cohort covered by national BCG vaccination recommendation
Outcome	Confirmed COVID-19 cases per 1000
Setting	Various countries, unknown dates of analysis for outcome
Control series	Outcome of interest in age group cohorts not included in national BCG vaccination recommendations
Method	DiD TWFE model, with country and age-group fixed effects
Scale	Logarithmic
Identification assumption	Expected ratio of infection rates across age-groups and countries are equal absent BCG vaccination recommendation
Results	Do not support hypothesis of protective effect of BCG vaccine
Negative controls/placebo tests	None reported
Additional details	• Includes BCG strain as a covariate to address hypotheses generated from cross-sectional analysis of countries
	• Exposure onset is by age cohort, not calendar time

Abbreviations: BCG, bacille Calmette-Guérin; COVID-19, coronavirus disease 2019; DiD, difference-in-differences; TWFE, 2-way fixed effects.

^aMatsuura et al.⁴⁴

The authors identified countries that changed BCG vaccination recommendations at some point in the past and determined for various age groups in each country whether they were in a national vaccination cohort or not. They then conducted a TWFE analysis of the relationship between being in a BCG vaccination cohort and the log of confirmed COVID-19 cases for that age group and country, finding no protective effect of the vaccine.⁴⁴

In this study, incorporating within-country controls helped mitigate the bias of comparing across countries cross-sectionally. However, the use of age cohorts as the distinguishing feature risks contamination—as individuals of different ages interact and may provide indirect protection—and concurrent events, since many nonpharmaceutical interventions and recommendations in the COVID-19 pandemic were targeted to specific age groups. Identification for this design requires that the difference in log-cases between age cohorts would be constant (parallel multiplicative trends) across countries absent differential BCG vaccination policies. This would be threatened by differential age-targeted policies.

As in the previous study, the generalizability of the ATT is limited, since it is specific to the age cohorts and countries studied, as well as the time since BCG vaccination inherent in those age groups. However, the staggered adoption that occurs in this setting (different countries changed vaccination rules for different age cohorts) risks a larger problem of bias or incorrect interpretation of the ATT. The estimate given by the TWFE model is a weighted average of individual effects in each country-age cohort combination, with some given potentially negative weights.²¹ Careful specification of the estimand and the analysis method in quasi-experimental analyses, especially with staggered adoption, is thus crucial. In this case, individual 2-by-2 DiD analyses may be more useful, providing a range of estimated treatment effects in the different age group-country combinations.^21,45

The challenges of estimand interpretation and generalizability limit the internal and external validity of this approach. Nonetheless, it provided useful exploratory evidence that controlling to some extent for country-level factors could account for the correlation observed in a cross-sectional analysis. As this negative finding was borne out by a randomized controlled trial,⁴³ this study provided immediate useful interim evidence before that trial was concluded.

Case 3: DiD and SCM analyses of the indirect protection of a COVID-19 vaccine

After the authorization of COVID-19 vaccines to prevent symptomatic and severe illness in adults, questions remained about the effectiveness of the vaccines in providing indirect protection by preventing infection and transmission.⁴¹ Winner et al⁴⁶ sought to assess the indirect protection of children via adult vaccination by comparing infection rates among Austrian children in the district of Schwaz—which had high adult vaccination uptake due to a campaign during a localized outbreak—to other districts with much lower rates as of March 2021. They used both SCM analysis with other Austrian districts as controls and DiD analyses comparing Schwaz municipalities to neighboring municipalities outside the district (see Table 3).⁴⁶ All analyses reported strong indirect protection of children.⁴⁶

Table 3.

Summary of COVID-19 vaccine indirect protection analyses in Winner et al,^a 2022.

	SCM analysis	DiD analysis
Research question	Did a mass vaccination campaign in adults reduce COVID-19 incidence in children?
Exposure	Within ineligible age cohort (under 16 years old) in the district of Schwaz
Outcome	Cumulative daily SARS-CoV-2 infections per 100 000
Setting	Schwaz District of Austria, January to May 2021
Control series	Outcome of interest in other districts	Outcome of interest in bordering municipalities
Method	SCM using infection history, population size, geographic area, and number of municipalities as covariates	DiD, 2 models: • TWFE (municipality and week), with effect size varying by week • 2-by-2, averaging across weeks
Scale	Linear	• Linear (TWFE)
		• Logarithmic (2-by-2)
Identification assumption	Weighted average of control series is the expected outcome absent vaccination campaign	Additive (TWFE) or multiplicative (2-by-2) change in expected outcome pre- to postcampaign is the same across municipalities
Results	Vaccine effectiveness:	Vaccine effectiveness:
	675.3 avoided infections per 100 000 children in Schwaz (95% CI, 146.9-1232.6)	• Significant decrease after second dose in campaign (TWFE)
		• 64.5% (95% CI, 30.2-82.0) (2-by-2)
Negative controls/placebo tests	In-space placebo test	Preexposure trends
Additional details	All analyses reported results also for adults aged 16-50 (targeted by vaccination program and thus expected to exhibit a larger effect)

Abbreviations: CI, confidence interval; COVID-19, coronavirus disease 2019; DiD, difference-in-differences; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2; SCM, synthetic control method; TWFE, 2-way fixed effects.

^aWinner et al.⁴⁶

With a clear time point for the mass vaccination campaign, quasi-experimental methods are a natural design for this setting, and the neighboring municipalities and other districts in the country form a large pool of potential controls. Selecting unexposed units within the same country increases the plausibility of similar nonpharmaceutical interventions and other policies, supporting the assumptions of stable weights and parallel trends. The risk of cross-border contamination through indirect protection remains but would, if anything, yield conservative estimates. Negative controls included an in-space placebo test for SCM (ie, using the same method on nontargeted districts to create a null distribution) and assessment of preexposure parallel trends for DiD.⁴⁶

While policies spurred by a particular need create variation, there is risk in using them as quasi-experiments. In this case, a large outbreak in Schwaz spurred the vaccination campaign. This creates concurrent events and anticipation: The past outbreak and the vaccination both influence infection dynamics going forward and may violate the assumptions needed for these methods.⁸ The authors sought to use districts with similar prior infection trends to mitigate this bias. Parallel trends could also be threatened by the use of linear functional forms for a nonlinear outcome like infections, although the authors used both a linear TWFE model and a logarithmic 2-by-2 DiD model here.^8,17,47

Once again, the results are highly contingent on the campaign and the setting in which it was enacted. The differential outcomes result from both the vaccination itself and any changes in behavior that may have resulted from a mass vaccination campaign; this point is well-described in the quasi-experimental literature for policy evaluation.^1,19 For pharmacoepidemiologic purposes, then, this makes it difficult to ascribe the full effect to the pharmaceutical itself in such cases. In addition, indirect protection in infectious disease outbreaks is specific to the prior evolution of the outbreak and the population at risk, limiting the generalizability and interpretability of estimates.^23,40,48,49 Nonetheless, the finding of a large indirect effect could inform public health practice and motivate the study of this effect in other settings, even if precise estimates are not transportable.^41,42

Discussion

Quasi-experimental methods provide a promising set of designs for pharmacoepidemiology. The designs discussed here, among others, provide a class of observational study designs that can target useful public health-relevant estimands and provide meaningful real-world evidence in advance of or as a supplement to randomized controlled trials.^14,50,51 In the case of vaccines, for example, these designs can identify potential off-target effects of pharmaceutical products, encouraging further research into those effects,^44,51 and identify rare safety signals through the use of large, routinely collected datasets.⁵² In addition, these studies can provide important postmarketing evaluation of the effects of products on their target outcomes in actual use, including indirect effects and changing effectiveness over time as conditions change.^53,54

By avoiding the need to measure and model all confounders between exposure and outcome, quasi-experimental methods can achieve high internal validity with routine data sources.^3,4,32,52 This, however, requires other counterfactual assumptions, such as parallel trends or stable weights. Justifying these assumptions, especially the appropriate functional form, requires an understanding of the setting and likely effect.^8,17,24 As seen in the case studies, both linear and logarithmic scales are used in vaccine evaluation studies, leading to different assumptions and different estimands.

Moreover, spillover and contamination of control units by the exposure and anticipation or lagged effects of the intervention can cause bias, often towards the null.^8,27,32 For pharmacoepidemiology, this could occur through off-label prescribing or, as in the vaccine cases, indirect effects within networks and communities. Attributing the observed causal effect to the exposure also requires a lack of concurrent events in the study units, which may be threatened by changes in behavior that co-occur with pharmaceutical interventions, like behavior changes that follow vaccination.^5,8,9,17

As in all medical studies, understanding the estimand targeted by these methods is crucial to properly using the evidence.⁵⁵ When aggregate population data are used, these methods target aggregate or population-level effects, and thus cannot be transported to individuals.¹⁹ But this also provides useful real-world evidence, such as the overall effect measure in vaccine studies, as seen in the case studies.²³ This is rarely targeted by randomized trials and provides useful additional regulatory and public health evidence.^41,42,53

ATT estimands from quasi-experimental studies, however, are specific to the setting, exposure, outcomes, and exposed unit(s), as well as the contrast identified and other statistical analysis choices. Since the exposed units are rarely randomly selected (ie, the mass vaccination campaign in Schwaz occurred because of a prior outbreak⁴⁶), the ATT differs from the average treatment effect that is usually targeted by randomized trials and used in regulatory decisions.⁵⁶ It may be of specific interest in safety studies, as it represents the effect of removing the exposure from all exposed units (eg, compare Cowger et al⁶). In general, however, it may be less transportable to populations yet to be exposed or require further assumptions.^5,16

Compared with other observational study designs, quasi-experimental designs tend to trade away some external validity for internal validity, although these aspects should ideally be considered in concert.^3,57 Researchers need to be careful about generalizing results of these methods to other settings and populations, especially for outcomes that have marked spatiotemporal patterns (eg, infectious diseases) or for populations with specific health needs and vulnerabilities (eg, specific age groups or people with co-morbidities).¹⁹

Achieving the potential benefits of these designs will require further research and careful reporting and interpretation. The general methodology of these designs in the field of pharmacoepidemiology—as well as how to report results and place evidence in the context of other studies routinely conducted in the field—will be important for ensuring validity and appropriate interpretation.^9,11 Connecting the literature across various disciplines will improve uptake of the most appropriate methods as well. Moreover, research on the most appropriate designs for specific settings, considering the populations, interventions, and outcomes, is crucial.⁵⁰ This can include simulation studies in those settings to observe the relationship between estimates and true estimands.¹⁵

For example, quasi-experimental vaccine evaluation studies will need to determine appropriate lag times, functional forms, and time frames for the necessary DiD and SCM assumptions to hold.^14,52 This can build on the existing health policy and stepped-wedge trials literature.^8,47,48 Understanding and communicating the trade-offs of identifying population-level effects but potentially losing generalizability will be key to appropriately contextualizing this evidence.

Negative controls, placebo tests, and sensitivity analyses also have an important role to play. Assessing methods for sensitivity to the selection of control units and time periods can inform both internal and external validity. Quantitative placebo tests can also allow the reporting of causal uncertainty alongside statistical uncertainty.³ Investigators can improve generalizability by using multiple populations (as in the meningococcal vaccine case study³⁹), different methodological approaches or models (as in the COVID-19 vaccine case study⁴⁶), or multiple units and exposure points (as in the BCG vaccine case study⁴⁴). The latter, however, targets an estimand that averages potentially heterogeneous treatment effects, which may or may not have greater generalizability—depending on the setting—and can be statistically inefficient.^21,30,48 The proposed alternatives vary in their targeted estimands, required assumptions, and statistical properties.^25-27,45,58 Investigation of the assumptions and relative performance of these methods in specific pharmacoepidemiologic settings (see, eg, multiple references^8,9,17,47), alongside the appropriate reporting of these challenges, is needed.

Quasi-experimental methods can alleviate some of the challenges of pharmacoepidemiology, while introducing others. Understanding and using them provides another tool for regulators, physicians, and public health policy-makers to understand the benefits and risks of drugs and biologics.

Funding

None declared.

Conflict of interest

The author declares no conflicts of interest.

Disclaimer

The views expressed in this article are those of the author alone.

Data availability

None declared.