What to Do When Everything Happens at Once: Analytic Approaches to Estimate the Health Effects of Co-Occurring Social Policies

Abstract

Social policies have great potential to improve population health and reduce health disparities. Increasingly, those doing empirical research have sought to quantify the health effects of social policies by exploiting variation in the timing of policy changes across places. Multiple social policies are often adopted simultaneously or in close succession in the same locations, creating co-occurrence that must be handled analytically for valid inferences. Although this is a substantial methodological challenge for researchers aiming to isolate social policy effects, only in a limited number of studies have researchers systematically considered analytic solutions within a causal framework or assessed whether these solutions are being adopted. We designated 7 analytic solutions to policy co-occurrence, including efforts to disentangle individual policy effects and efforts to estimate the combined effects of co-occurring policies. We used an existing systematic review of social policies and health to evaluate how often policy co-occurrence is identified as a threat to validity and how often each analytic solution is applied in practice. Of the 55 studies, only in 17 (31%) did authors report checking for any co-occurring policies, although in 36 studies (67%), at least 1 approach was used that helps address policy co-occurrence. The most common approaches were adjusting for measures of co-occurring policies; defining the outcome on subpopulations likely to be affected by the policy of interest (but not other co-occurring policies); and selecting a less-correlated measure of policy exposure. As health research increasingly focuses on policy changes, we must systematically assess policy co-occurrence and apply analytic solutions to strengthen studies on the health effects of social policies.

Abbreviations

EITC

Earned Income Tax Credit

SNAP

Supplemental Nutrition Assistance Program

WIC

Special Supplemental Nutrition Program for Women, Infants, and Children

INTRODUCTION

Social policies are promising mechanisms to improve population health and reduce health disparities. Analyses of the health effects of social policies routinely leverage policy changes occurring in 1 or multiple different places at different times, with differences-in-differences or similar study designs (1). In epidemiology and related fields, empirical health research using these methods has proliferated rapidly and yielded important findings (2,3). However, the validity of this approach is threatened when multiple related policies are adopted simultaneously or in close succession in the same jurisdiction. Bundles of related policies with similar potential health effects are often changed together, creating co-occurrence that must be addressed analytically for valid inference. Specifically, analyses that do not account for co-occurring policies are likely to be confounded, and analyses that incorporate measures of co-occurring policies can encounter imprecise or unstable estimates and bias resulting from data sparsity (4, 5).

A rich literature exists on confounding and consequent data sparsity generally (4, 6–10); however, several aspects of the policy-co-occurrence problem make it important to consider separately from issues that arise with other exposures. By nature, policy making may create correlations among policy variables that are much stronger than those typically observed in nonpolicy studies (11–13). Governments may respond to the desires of their constituents by adopting multiple related policies simultaneously. For example, a state that moves to overhaul its social safety net is likely to change multiple related policies (e.g., income support and food insecurity benefit generosity) at the same time. The most promising analytic solutions also may be different. For example, if a set of policies is always adopted together, then estimating their combined effect is informative for real-world decision-making, whereas analyses of the combined effect of an exposure of interest and closely related confounders may be less useful. In addition, some data sparsity problems can be addressed by increasing sample sizes, but policy studies are typically based on a small, fixed set of jurisdictions. Meanwhile, stronger theories or substantive knowledge about how a specific social policy functions could guide causal analyses evaluating the health effects of mediating variables (14). For example, if it is understood that compulsory schooling laws affect all-cause mortality by increasing educational attainment for some individuals, then such laws could serve as an instrument for studying the effects of changing educational attainment on health (15). Thus, the policy-co-occurrence problem presents distinct challenges and possible analytic solutions beyond typical confounding.

In many policy domains, adopting groups of policies as a set is common (16). In these cases, researchers can implement a variety of study designs or statistical strategies to address potential bias or imprecision resulting from policy co-occurrence. Among these approaches, an overarching distinction is whether the approach aims to disentangle the effects of individual policies or conceptualizes the co-occurring policies as a group and evaluates their joint effects. For a given study, either approach may be policy relevant, depending on whether the goal is to deliver actionable evidence on the effects of a single policy or on the effects of a set of policies that would likely be adopted together.

To date, most approaches to handling social policy co-occurrence have been ad hoc. Authors of applied studies in fields including epidemiology, political science, and health economics have acknowledged the issue by critiquing existing studies in which co-occurring policies were not accounted for or by using specific analytic solutions (17–22). In methodological work in specific fields such as environmental epidemiology, statistics, substance use, political science, and economics, researchers have also discussed individual analytic solutions relevant to these domains (5, 6, 9, 23–27). The problem of multicollinearity is widely recognized in econometrics as a threat to causal inference (28–31), but to our knowledge, none has specifically addressed applications to research on the health effects of social policies. In this article, we aimed to address the need for a systematic assessment of the analytic solutions that are applicable to research on the health effects of social policies, how often these solutions are used in practice, and the tradeoffs to consider when selecting an approach.

This is the second article in a series on the social policy co-occurrence problem. In our other article in this issue (16), we demonstrated that co-occurring social policies are pervasive and that adequate adjustment for co-occurring policies is likely to substantially reduce the precision of estimated effects. Given this, delineating methods appropriate for this context is a high priority for the next generation of research on the health effects of social policies. In the present article, we describe analytic strategies researchers have adopted to address social policy co-occurrence with the goal of attaining valid inferences. We categorized these approaches on the basis of the type of causal question they answer (e.g., the effect of an individual policy on a population subgroup vs. the effect of a bundle of policies on the overall population). Using the sample of social policy evaluations we developed previously (16), we measured the proportion of studies in which authors assessed policy co-occurrence and the proportion applying each of the different analytic approaches to account for policy co-occurrence. We discuss the advantages and disadvantages of each approach and provide guidance on selecting among them.

METHODS

Identification of social policy studies

We used an existing sample of studies on the health effects of social policies to review common strategies for addressing policy co-occurrence. The details of this review are described in Part 1 of this article series. Briefly, we selected a multidisciplinary set of journals that publish health-related social policy research and are leading journals in their respective fields (American Journal of Public Health, American Journal of Epidemiology, Journal of the American Medical Association, New England Journal of Medicine, The Lancet, American Journal of Preventive Medicine, Social Science and Medicine, Health Affairs, Demography, and American Economic Review). We screened all 6,794 articles published in these journals in 2019 and included all original, empirical studies in which the authors estimated the causal effects of 1 or more social policies on health-related outcomes (n = 55). “Social policies” were defined as nonmedical, population-based, or targeted policies adopted at a community or higher level and hypothesized to affect health or health inequalities via changes in social or behavioral determinants. We defined “health-related outcomes” broadly to include morbidity, death, health conditions, and factors such as smoking, homelessness, and sales of unhealthy products. Given our focus on social interventions, we excluded studies that pertained to health care, health insurance, interventions delivered in the clinical setting, medications, or medical devices, including studies of the Affordable Care Act or Medicaid expansion.

Categorization of analytic approaches

Our framework (Table 1) applies to research questions about the health effects of 1 particular policy (the “index” policy) in a defined target population. We assumed investigators identified relevant co-occurring policies that might confound the index policy.

Table 1.

Types of Analytic Approaches to Address Policy Co-Occurrence and Corresponding Causal Research Questions

Analytic Approach	Corresponding Causal Research Question
Effect of individual policy of interest
1. Adjust for co-occurring policies	What is the effect of the policy of interest on the health outcome?
2. Restrict the study sample to the region of common support	What is the effect of the policy of interest on the health outcome in the restricted sample?
3. Define the outcome on subpopulations likely to be affected by the index policy but not other co-occurring policies	What is the effect of the policy of interest on the health outcome in the subpopulation?
4. Select a less-correlated measure of policy exposure	Example: How does a more generous version of the policy of interest affect the health outcome, compared with a less generous version of the policy interest?
5. Use formal Bayesian methods	What is the best estimate of the effect of the policy of interest on the health outcome, considering both prior knowledge on policy effects and the observed data on policies and outcomes?
Combined effects of multiple policies
6. Identify and evaluate the impacts of policy clusters	Example: What is the effect of adopting all policies in the cluster vs. no policies in the cluster on the health outcome?
7. Use an overall policy stringency or generosity score	What is the effect of differing levels of overall policy stringency or generosity on the health outcome?

A priori, we designated 7 categories of analytic approaches that researchers could adopt in the face of policy co-occurrence, on the basis of the causal question each method could answer (Table 1). We identified these approaches by reviewing multidisciplinary scientific literature on the study of co-occurring exposures, consulting with experts, and drawing on methods used in our own fields of research. We focused on methods that apply to study designs that leverage policy changes occurring in different places and different times, including aggregate or multilevel differences-in-differences and panel fixed effects. We defined “co-occurring” policies as policies whose adoption or implementation was correlated in space and time with an index policy (i.e., places and times with the primary policy of interest are also likely to have the “co-occurring” policy or policies) and that likely affected the health outcome under study. This co-occurrence could be at multiple jurisdictional levels (e.g., cities within states), although single-jurisdiction–level studies were the norm.

Approach 1: Adjust for co-occurring policies.

If co-occurrence of related policies with the index policy is not severe (see our other article in this issue), the researcher can adjust for measures of the other policies (e.g., by controlling for co-occurring policy measures in a regression). Under conventional assumptions, the resulting estimand corresponds to the effect of the index policy on the health outcome. This approach will often rely on some degree of model-based extrapolation, because not all possible combinations of policies actually occur. It is incumbent on the investigator to confirm that any extrapolation is well founded in theory or evidence.

As an example, Raifman et al. (32) estimated the effect of state same-sex marriage laws on adolescent suicide attempts using a differences-in-differences analysis while controlling for policies banning sexual orientation–based employment discrimination. Because co-occurrence between the 2 types of policies was only moderate, regression adjustment for the co-occurring was sufficient to isolate the index policy.

Policy co-occurrence will be severe if a co-occurring policy aligns perfectly or nearly perfectly in place and time with the policy of interest. After adjusting for co-occurring policies, there will be insufficient independent variation in the index policy left to study, giving extremely imprecise estimates. The only analytic solutions are based on modifying the research question. Approaches 2–7 involve such alternative research questions and corresponding analytic approaches to assessing the impact of the index policy.

Approach 2: Restrict the study sample to the region of common support.

The issue of policy co-occurrence can be conceptualized as a form of strong confounding of the index policy by the other co-occurring policies. This confounding and consequent data sparsity result in a lack of common support in the data, also known as a violation of the positivity assumption (4). Positivity violations occur when some confounder strata do not have variation in the exposure—for example, because the confounding policy and index policy are always adopted as a set. This situation can be resolved by restricting the analysis to the confounder strata for which there is variation in the index policy (i.e., the data region of “common support”). Similar to approach 1, this approach usually involves adjusting for co-occurring policies, but here, extrapolation is avoided by restricting the study sample. This approach changes the target population so the corresponding causal question refers to the effect of the index policy on the health outcome in the restricted sample, and results are only generalizable to the population represented by the restricted sample. In the extreme, if an index policy and co-occurring policy are always adopted as set, then there may be no region of common support, and alternative approaches must be considered (e.g., evaluating the combined impacts of a bundle of policies).

One way to implement this approach is to restrict the study sample to a subpopulation for whom exposure to the nonindex co-occurring policies does not vary. If policies that might confound the association of interest do not vary within a particular study population, then they cannot cause confounding. For example, many households are eligible for multiple social welfare programs, including the Supplemental Nutrition Assistance Program (SNAP) and the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC). This makes it difficult to disentangle program effects. Liu et al. (33) addressed this co-occurrence by studying outcomes for dual beneficiaries of SNAP and WIC versus WIC alone. Those without WIC could be considered “off support” and are excluded, so all study participants are WIC beneficiaries. This approach changes the target population, so the corresponding causal question refers to the effect of the index policy (e.g., SNAP) on the health outcome in the restricted sample (e.g., WIC beneficiaries), and results generalize to the population represented by the restricted sample.

When there are many confounder strata, an accessible way to assess positivity and identify the region of common support is using propensity scores. In the context of assessing policy impacts, the propensity score (34) is the probability of adopting the index policy, given the confounding policies. Units that are “on support” are those with propensity scores within the range of observed propensity scores both for units adopting the index policy and for units not adopting the index policy. A wide variety of matching and weighting methods involve using propensity scores to identify and restrict to the region of common support (8, 35). Numerous variations on this restriction have also been proposed, including restricting to units with propensity scores within a prespecified range (e.g., 0.1–0.9) or dynamic optimization procedures for selecting propensity score cutoffs (4, 6, 7).

An alternative approach to using propensity scores is to directly restrict the sample on the basis of the distribution of the co-occurring policies themselves. Several approaches have been proposed, including restricting to units inside the convex hull of the covariate space defined by the secondary policies (9, 36), restricting to a sufficiently data-dense, rectangular region of the covariate space defined by the co-occurring policies (37–39), or tree-based methods (e.g., see Chipman et al. (40)). These approaches are less common, but most can be readily implemented using existing software. In all cases, assessing the region of common support helps ensure that estimates are not relying on extrapolation to policy combinations that are never observed. The restricted study population should be well defined so that the investigator can transparently describe the places and times to which the results apply (37).

Chang et al. (41) applied this approach to study the impacts of prescription drug–monitoring programs and so-called pill-mill laws using a comparative interrupted time-series analysis. Rather than using all states in the analysis, some of which implemented other opioid policy changes in concert with prescription drug–monitoring programs or pill-mill laws, the authors restricted their analysis to Florida, which adopted prescription drug–monitoring program and pill-mill laws, and Georgia, which had a similar policy profile to Florida but did not implement the index policies during the study period. The authors determined that the combination of prescription drug–monitoring programs and pill-mill laws were associated with reductions in high-risk opioid prescribing for the Florida population.

Approach 3: Define the outcome on subpopulations likely to be affected by the index policy but not other co-occurring policies.

Identifying health effects that are specific to the index policy can be achieved by changing the outcome measure to one that is both closely aligned with the index policy and unlikely to be affected by other co-occurring policies. In particular, if the outcome is focused on a particular population subgroup (e.g., defined by age, gender, place, or time) that is likely to be most affected by the index policy and unlikely to be affected by other co-occurring policy changes, study results can provide pointers to the impacts of the individual policy.

For example, changes in state Earned Income Tax Credit (EITC) policies have often co-occurred with other changes in other social welfare policies such as SNAP (42). Rehkopf et al. (43) took advantage of the fact that EITC cash benefits are typically delivered in February, March, and April, whereas other benefits do not have the same seasonal dispersal pattern, to examine the association of EITC policies with health. They used a differences-in-differences approach to compared health outcomes that can change monthly (e.g., health behaviors, cardiovascular and metabolic biomarkers) for EITC-eligible versus noneligible individuals in months of income supplementation versus nonsupplementation. The authors thus were able to measure some potential short-term health impacts of EITC independent of other social welfare policies that do not have this seasonality.

This approach can be strengthened further by incorporating falsification tests or negative control analyses. Rehkopf et al. (43) strengthened their findings by confirming that treating non-EITC transfer months as “treated” months produced null findings and by confirming null associations for outcomes that do not change monthly.

Approach 4: Select a less-correlated measure of policy exposure.

Studies that use binary (0 or 1) characterizations of policy adoption are widespread in studies of the health impacts of social policies. However, more detailed characterizations of individual policies (e.g., the amount of funding allocated, benefit generosity, participation rate, or population reach of a program; the size of a tax; or the number of years a policy has been in place) can deliver policy measures that are less correlated with other related policies or opportunities to examine dose–response effects among jurisdictions adopting a policy. For example, adoption of more generous unemployment benefits, in terms of dollar amounts and durations for different types of households, tend to change in tandem with other worker protection and leave policies, but researchers have effectively disentangled effects of unemployment benefits by using continuous measures of maximum allowable unemployment benefit levels across states (44, 45). Similar approaches have been taken when studying the effects of alcohol taxes (46, 47), tobacco taxes (48), and EITC benefit generosity (49, 50).

One useful modification to this approach is to study factors that may specifically mediate the relationship between the index policy and the health outcomes. For example, Matthay et al. (51) generated evidence on the impacts of policies regulating gun shows by examining the impacts of gun-show events themselves on firearm-related injuries in differing policy environments. Similarly, the impacts of medical marijuana provisions allowing supply through dispensaries (as opposed to home cultivation) can be quantified by studying the direct effects of dispensaries on health (52, 53).

Studying mediators may also offer the opportunity to identify policy effects via the Front Door Criterion (14), a rarely used alternative to confounder-control or instrument-based methods (3). If all the pathways by which the index policy affects the outcome can be measured, and there are no unmeasured confounders of the index policy-mediator relationship or of mediator-outcome relationship (conditional on the index policy), then the effect of the index policy on the outcome can be identified without measuring the co-occurring policies. For example, Bellemare et al. (54) used the Front Door Criterion to estimate the effect of authorizing Uber and Lyft ride sharing with strangers on tipping drivers. They proposed that the only way in which sharing authorization affects tipping is whether a ride is actually shared (the mediator). This mediator is used as a tool to estimate the effect of sharing authorization while circumventing confounders of sharing authorization such as rider experience, mood, and social preferences. Although applications of the Front Door Criterion remain rare, a similar approach could be applied to social policy evaluations.

Approach 5: Use formal Bayesian methods.

Bayesian methods can be used to integrate information gleaned from other approaches to addressing policy co-occurrence. Bayesian methods can also be used as a way to treat estimation problems arising from policy co-occurrence without linking to other approaches we have mentioned. Several approaches to addressing policy co-occurrence depend on incorporating prior knowledge about the policies, determinants of the outcome, or hypothesized mechanisms of effect. For example, researchers may apply judgments about which policies affect the outcome or modify other policy processes. These insights can guide decisions about which policies need to be controlled and how. Bayesian methods offer a formal statistical method to incorporate prior knowledge about the plausible effects of the co-occurring and index policies, and to combine these with newly gathered empirical data.

When used alone, Bayesian methods can help address estimation issues and recover precision when highly co-occurring policies lead to convergence problems or imprecision. In particular, Bayesian approaches can stabilize estimates (i.e., address data sparsity-related problems of imprecision and sensitivity to different model specifications and influential data points) by constraining the effect sizes or interaction effects among policies and “shrinking” coefficients toward the specified prior distributions (21). This can be done without changing the set of adjustment variables, without restricting the study sample, and without changing the exposure or outcome measure. This approach is common in the environmental epidemiology literature as way to study multiple co-occurring exposures, such as air pollutants (5, 25).

Harper (55) used a Bayesian differences-in-differences approach to estimate the effects of adopting stronger enforcement of state seat belt laws on motor vehicle crash deaths. Because other road-safety policies may also affect motor vehicle crash deaths and change in tandem with seat belt enforcement policies, Harper also adjusted for laws controlling maximum speed limits, blood alcohol–concentration limits, graduated driver’s license programs, and annual state policy per capita as a proxy for traffic safety enforcement. Because these variables are correlated across states and years with each other and with seat belt enforcement policies, adjusting for them in a frequentist analysis reduces the precision of the estimated effect of interest—a major problem resulting from co-occurring policies. Harper enhanced precision by applying a Bayesian approach, incorporating existing evidence about the effects of seat belt laws into the statistical estimation of the effect of seat belt enforcement policies.

Approach 6: Identify and evaluate the impacts of policy clusters.

If a set of policies is typically adopted as a group, the effect of the combined set of policies may be the most pertinent parameter to estimate. By conceptualizing policy clusters as the exposure of interest, the investigator can preserve the original target population and outcome measure. For example, if 2 or more policies are highly co-occurring, it may be possible to estimate their combined impact (e.g., comparing health outcomes if both policies were adopted vs. if neither policy were adopted) (56). Policy clusters can be defined on the basis of substantive or policy-making considerations; this is useful if decision makers are considering adopting a set of policies. Alternatively, numerous data-driven clustering algorithms are applicable. Clusters or categories are defined on the basis of how frequently policies co-occur (i.e., the extent to which policies co-occur in the same place and time). Methods include hierarchical cluster analysis, latent class analysis, or principal components analysis (5, 57, 58). Clusters might also be defined on the basis of the strength of the relationship with the outcome (e.g., using supervised principal components analysis) (5, 25). No single algorithm is considered optimal for all settings (5, 25).

Among data-driven algorithms, it is common to distinguish between variable-centered methods that group similar policy variables (e.g., principal components analysis) and person-centered methods that group similar observations (e.g., latent class analysis). The underlying mechanics of variable-centered and person-centered approaches are distinct, but both approaches ultimately result in a small set of variables that summarize the policies to which each observation is exposed. This smaller set of variables is then used to assess health impacts. Erickson et al. (59) used latent class analysis to classify states on the basis of their position on 18 alcohol-control policies. The analysis categorized each state into 1 of 4 unordered groups, which the authors interpreted as follows: weak except serving policies; average; strong for underage use; and strong policies overall. State policy category was then associated with levels of past-month alcohol consumption.

Approach 7: Use an overall policy stringency or generosity score.

If the investigator is interested the effects of the overall policy environment on health, 1 method is to use a summary score of the stringency or generosity of a set of policies. In comparison with approach 6, this approach similarly involves reducing many policy variables to a few, but it typically focuses on creating an order along a predefined unidimensional scale, such as stringency. In contrast, policy clusters are usually unordered and defined on the basis of the covariation among the policy measures themselves without regard for underlying characterizations such as stringency, although such characterizations may be applied after the fact when interpreting or describing the clusters. This approach also differs from approach 4, in which one might characterize a single policy on a continuous scale to help disentangle the effects of that policy from other policies, because here we characterize a collection of policies with respect to their likely combined impact.

A simple way to apply this approach is to sum the number of policies in the set that apply in each place and time. Policies must be coded in the same direction so the presence of more policies indicates greater restrictiveness, or vice versa. This method is easy to operationalize, but it implies that all policies carry equal weight and are interchangeable in achieving health effects.

A more sophisticated application is to weight policies on the basis of existing evidence or expert opinion about the strength of the relationship with the outcome. This may be based on efficacy, restrictiveness, implementation, enforcement, enforceability, reach, or other metrics. Investigators have applied this approach in literature on firearm policy (60, 61), alcohol policy (19, 62–64), and marijuana policy (65, 66). Although there are infinite ways a set of policies can be ranked or weighted, using systematic methods can enhance rigor and replicability. For example, the Delphi technique is a structured communication approach to elicit consensus from a panel of experts and can be used to rank or score policies on the basis of stringency or effectiveness (67, 68). Assigned weights are typically outcome specific; for example, weighting state alcohol policies with different levels of effectiveness for binge drinking versus impaired driving and for adults versus youth (19, 62). Investigators can also explore different methods of weighting in sensitivity analyses (62).

Data extraction and analysis

For each social policy study, we reviewed the full text. Our main focus was on the primary analytic specification, which we understood to be the authors’ leading approach to estimate the effect of the social policy for the health outcome(s) of interest. We also reviewed any sensitivity analyses reported in the main text. We assessed 1) the overall analytic approach (e.g., differences-in-differences); 2) whether the authors reported checking for any co-occurring policies related to the health outcome of interest that might pose a threat to validity; 3) the authors’ determination of whether any co-occurring policies did, in fact, threaten validity (e.g., based on their analysis or prior literature); 4) whether there was any other indication that co-occurring policies exist for the study’s application (e.g., a co-occurring policy mentioned in the limitations); 5) if policy co-occurrence was identified as at threat, what analytic strategy the authors used to address it; and 6) any other aspects of the analytic strategy that may help address co-occurring policies, whether they were identified as a threat or not. We also documented whether study authors used any approaches to address policy co-occurrence not identified a priori. We then tabulated these characteristics.

RESULTS

We assessed 55 studies of social policies encompassing diverse topics, countries, and jurisdictional levels (69–124) (Web Table 1) (available at https://doi.org/10.1093/aje/mxab005). Studies included, for example, a comparative interrupted time-series evaluation of the impacts of lowering the blood-alcohol concentration limit for drivers on road-traffic accidents in Scotland (72) and a differences-in-differences analysis of the effects of state-paid family-leave policies on breastfeeding (97). The most common domains were poverty and social welfare policies such as the SNAP (n = 14 studies); food and beverage policies such as sugar-sweetened–beverage taxes (n = 6 studies); firearm restrictions (n = 5 studies); unemployment, sick leave, and pension benefit policies (n = 4 studies); tobacco control (n = 4 studies); alcohol control (n = 4 studies); and immigration (n = 4 studies).

Figure 1 presents a flowchart of the included studies, broken down by whether the authors evaluated policy co-occurrence and used techniques to address policy co-occurrence. Of the 55 studies, 4 involved methods for which assessing policy co-occurrence was not relevant: One involved a national policy with randomized rollout across village clusters, and for 3 others, the primary research question was about the overall policy environment, and authors used policy stringency scores. Of the remaining 51 studies, only for 17 did authors report checking for at least 1 co-occurring policy. Of these 17, at least 1 co-occurring policy was identified in 10 studies, and in 7, findings suggested that no co-occurring policies of concern were mentioned. For example, in a study of state texting-while-driving bans and traffic injuries, the authors acknowledged that administrative license suspension, speed limits, seat belt requirements, and graduated driver licensing laws had also changed over the study period and might have affected traffic injuries; therefore, the authors controlled for measures of these policies in their differences-in-differences analysis (111). In contrast, in a study of the effects of tuition-free primary education on access to family planning and health decision-making, researchers evaluated potentially co-occurring paid family-leave policies (85); they determined that these policies did not substantially co-occur with tuition-free primary education but acknowledged there may be other unmeasured co-occurring policies. Of the 34 studies that did not report checking for at least 1 co-occurring policy, 5 had some other indication that policy co-occurrence may be a threat.

Figure 1.

Flowchart of included social policy studies by evaluation of policy co-occurrence and use of techniques to address policy co-occurrence.

Table 2 summarizes the analytic methods used in each study, irrespective of whether the authors checked for co-occurring policies. Overall, in 36 of the 55 studies (65%), at least 1 approach was incorporated that addressed policy co-occurrence. Among studies in which at least 1 approach was used, the most common approaches were adjusting for co-occurring policies (n = 18 studies; 50%); defining the outcome on subpopulations likely to be affected by the index policy, but not other co-occurring policies (n = 14 studies; 39%); and selecting a less-correlated measure of policy exposure (n = 7 studies; 19%). Twelve (33%) used more than 1 approach. None used formal Bayesian methods. No co-occurring policies were reported for 2 studies but, without naming co-occurring policies as the motivation, nonetheless, at least 1 approach was applied that helps address co-occurring policies (Figure 1).

Table 2.

Number of Social Policy Studies Using Each Study Design and Analytic Approaches to Address Policy Co-Occurrence

	Study Design, No.
Approach to Address Policy Co-Occurrence	DID	Panel FE	CITS	Synth Control	Before-After	Regress	PSM	Rand Wedge	IV	Sim Model	Total No. of Studies Using Approach
Adjust for co-occurring policies	9	5	0	0	1	1	0	0	2	0	18
Restrict the study sample to the region of common support	0	0	0	1	0	1	0	0	0	0	2
Define the outcome on subpopulations likely to be affected by the index policy but not other co-occurring policies	8	0	0	1	1	3	1	0	0	0	14
Select a less-correlated measure of policy exposure	3	1	0	0	0	1	0	0	2	0	7
Use Bayesian methods	0	0	0	0	0	0	0	0	0	0	0
Identify and evaluate the impacts of policy clusters	3	0	0	0	0	0	0	0	1	0	4
Use an overall policy stringency or generosity scorea	1	0	0	0	0	2	0	0	0	0	3
No method used	5	2	1	1	7	1	0	1	0	0	20
Total studies using study designb	12	6	1	2	9	7	1	1	3	1	b

Abbreviations: CITS, comparative interrupted time series; DID, differences-in-differences; IV, instrumental variables; Panel FE, panel fixed effects; PSM, propensity score matching; Rand. wedge, randomized stepped wedge; Regress, other regression without place-specific controls; Sim, simulation; Synth, synthetic.

^a Studies in which an overall policy stringency or generosity score was used were those in which the primary research question was about the overall policy environment.

^b The totals reported in this column/row do not add up to the total number of studies included in this review because some studies used multiple study designs or multiple approaches to address policy co-occurrence.

Researchers used a range of designs, the most common being differences-in-differences (n = 12 studies), before-after (e.g., t tests or interrupted time series; n = 9 studies), and other regression approaches without place-specific controls (hereafter, “other regression approaches”; e.g., multilevel regression of an overall policy stringency score on an individual-level health outcome without fixed effects; n = 7 studies) (Table 2). Studies in which differences-in-differences, panel fixed effects, or other regression approaches were used were more likely to involve at least 1 approach to address policy co-occurrence than were studies in which before-after designs were used.

Of the 10 studies in which authors explicitly reported identifying 1 or more co-occurring policies, an attempt was made in 9 to address it (Figure 1). Of the 41 studies in which policy co-occurrence was not checked or identified, at least 1 technique that helps address policy co-occurrence was used nonetheless. For several studies, researchers used multiple approaches in the same analysis. For example, in a study of the effects of losing SNAP benefits, researchers controlled for participation in WIC and Temporary Assistance for Needy Families program (approach 1), excluded those with potential concurrent benefit changes in Supplemental Security Income (approach 2); and compared outcomes among SNAP participants who lost benefits to similar individuals with continuous benefits (approach 3) (94).

DISCUSSION

Co-occurring policies are common and can threaten the validity of studies of the health effects of social policies. In this review article, we have described 7 analytic approaches to address policy co-occurrence. The approaches are used either to seek to disentangle the effects of individual policies or to estimate the combined effects of clusters of policies. Using a sample of contemporary studies on the health effects of social policies, we found that potential policy co-occurrence is frequently unidentified and unaddressed: In only 33% of studies did authors report checking for policy co-occurrence as a potential threat to validity, and in only 65% was any approach incorporated that helps address policy co-occurrence, regardless of whether the authors checked for policy co-occurrence. Several studies that estimated the individual effect of the index policy used multiple, possibly complementary approaches to address co-occurrence in the same analysis; this may further enhance validity, although it is not guaranteed.

In future applied studies, systematically evaluating and reporting on policy co-occurrence would facilitate the evaluation of validity and interpretation of findings. In many of the studies we reviewed (35%), authors did not report any approach to address policy co-occurrence; they may not have considered it or may have checked for it but not reported their assessment, particularly if it was not found to be a concern. However, because failure to address policy co-occurrence (if it exists) poses a major threat to validity, readers need to understand if the authors believe that no policy co-occurrence exists or if they believe it has been addressed (and if so, through what analytic strategies). Stringent word limits imposed by many medical and public health journals preclude articles in which authors present full analytic exploration of issues, such as policy co-occurrence. An openness to incorporating such discussions, at least in appendices, would enhance the rigor and interpretability of social policy studies.

We assessed whether the studies in our sample checked for any co-occurring policies, but ideally, researchers would evaluate all policies and related social, economic, and political phenomena that co-occur with the index policy and that could affect the health outcome of interest. This is a formidable task, especially given that innumerable policies are continuously being passed at all levels of government, that databases measuring relevant policies often do not exist, or that the policies that may affect an outcome are not fully understood. Policy libraries such as the University of Kentucky Center for Poverty Research state welfare database (42), the Policy Surveillance Program at Temple University (http://lawatlas.org/), and the University of Iowa State Policy Innovation and Diffusion Database (125) are increasingly valuable resources, but they are burdensome to develop and maintain and require infrastructure support. Substantial, regular support for policy surveillance as well as scientific endeavors to link and harmonize large administrative data sets would support these efforts (126–128). Given that diverse policies across numerous disciplines are likely to affect health, interdisciplinary collaboration is also essential to these efforts.

Tradeoffs between different approaches

In this report, we described how approaches to address policy co-occurrence have been used practice; a logical next question is which approaches are best suited for different circumstances. The preferred approach to address policy co-occurrence should be driven by the target causal question (129). Investigators should select the approach that best answers their causal question while achieving sufficient accuracy and precision (e.g., based on likely sources of bias and evidence on the precision of different estimators). If an unbiased, precise estimate can be derived with simple adjustment for measures of the co-occurring policies (approach 1), this option will often be preferable, because it does not change the target causal question or study population. However, if the target causal question is deemed unanswerable because of severe policy co-occurrence, a different analytic approach, potentially corresponding with a modified causal question, is necessary (129). More severe policy co-occurrence may necessitate larger departures from the original causal question. The extent to which alternative approaches change the causal question depends on the application (e.g., whether restricting the study sample to the region of common support involves dropping many units or only a few); Web Figure 1 shows an approximate ordering.

Table 3 summarizes tradeoffs of different approaches to addressing policy co-occurrence. Overall, approaches that preserve estimates of the independent effect of the index policy may be particularly useful for decision makers comparing specific policy options. However, these approaches generally sacrifice some aspect of generalizability by restricting the analysis to certain populations, subgroups, outcomes, or periods for which policy effects can be estimated. Results, therefore, may serve as markers of policy impacts rather than as measures of overall impact. Estimating the combined effects of a group of co-occurring policies sacrifices estimates of the independent effects of the index policy but preserves generalizability to the original target population, outcomes, and period under study. The preferred approach depends both on what options are viable (i.e., unconfounded, sufficient precision) and which causal question is of greatest interest. For example, if certain combinations of policies are always adopted together, then their independent effects may be neither estimable nor of interest.

Table 3.

Advantages and Disadvantages of Alternative Approaches to Address Policy Co-Occurrence in Studies of the Health Effects of Social Policies

Approach	Advantages	Disadvantages
Approaches involving disentangling the effects of individual policies	Results are informative for decision makers interested in whether to adopt the index policy of interest.	Most approaches require sacrificing some aspect of generalizability by restricting the analysis to certain populations, subgroups, outcomes, or periods for which policy effects can be estimated.
1. Adjust for co-occurring policies	Does not require changing the original research question	Only works if policy co-occurrence is not severe (e.g., no perfectly aligned policies; sufficient statistical power and independent variation in index policy of interest after controlling for co-occurring policies)
2. Restrict the study sample to the region of common support	Must be able to identify the region of common support; propensity scores are most common but must be correctly estimated. Supported by a large body of literature on using propensity scores for analyzing policy effects. Helps ensure estimates do not rely on extrapolation to policy combinations that are never observed	Reduces sample size; can harm statistical power; restricts the population to whom the results generalize. If using propensity scores, they must be correctly estimated.
3. Define the outcome on subpopulations likely to be affected by the index policy but not other co-occurring policies	Can isolate individual policy effects in the face of severe policy co-occurrence. Encourages drilling down on the times, places, and people that are most affected or of greatest interest	Policy-specific outcomes must exist, be correctly identified (based on existing evidence or theory), and be relevant to the research question of interest. Can inhibit direct comparison of effect estimates from policy alternatives using uniform methods and measures of association. Assumes no spillover effects of the index policy on any comparison or control groups deemed “unaffected” by the index policy
4. Select a less-correlated measure of policy exposure	Can isolate individual policy effects in the face of severe policy co-occurrence. Encourages drilling down on the hypothesized mechanisms and policy aspects that are most affected or of greatest interest	Policy-specific exposures must exist, be correctly identified (based on existing evidence or theory), and be relevant to the research question of interest. Can inhibit direct comparison of effect estimates from policy alternatives using uniform methods and measures of association
5. Use formal Bayesian methods	Can solve estimation problems without sacrificing the ability to study individual policy effects in the original target population	Does not solve fundamental lack of support in the data. May still rely on extrapolation. Often computationally intensive. Methods and format of results are less familiar to some audiences.
Approaches involving estimating the combined effects of clusters of policies	Preserves generalizability of the original target population, outcomes, and period under study. May answer the most policy-relevant question if certain bundles of policies are always adopted together	Does not produce estimates of individual policy effects; cannot distinguish which policies in a cluster are driving health effects
6. Identify and evaluate the impacts of policy clusters	Can provide useful estimates of the combined impacts of realistic policy combinations	No consensus on optimal methods to identify policy clusters or optimal criteria for selecting a final set of clusters (particularly concerning if effect estimates are sensitive to the choice of clustering) (58). Results can be challenging to interpret when the summary policy measures are weighted combinations of policy variables, as in PCA or factor analysis, or if the clustering algorithm produces many distinct clusters that are difficult to define or interpret.
7. Use an overall policy stringency or generosity score	Summarizes the effect of the overall policy environment. May be the only viable option in the face of severe policy co-occurrence	Developing weighting schemes can be time consuming and subjective. Results can be sensitive to the choice of score, score weighting, or score components, unless using data-driven weighting schemes based on the strength of the relationship with the outcome. Implies that 2 policies are interchangeable in their effects if adopting one or the other results in the same numeric change in the score (possibly unrealistic)

Abbreviation: PCA, principal components analysis.

Among the individual approaches, key considerations include the circumstances in which the approach is feasible (e.g., controlling for co-occurring policies is not possible if policy co-occurrence is severe), the availability of evidence to support making analytic decisions (e.g., on how to use propensity scores, select weighting schemes for policy scores, or choose a clustering method), the extent to which the approach provides evidence that is relevant to the original causal question, ease of implementation, available data and measures, and interpretability of the results (see Table 3 for details). All the approaches discussed here can also be used to evaluate whether policy co-occurrence is a concern by comparing results of analyses that do not account for policy co-occurrence with results from analyses that do. Although none of these approaches will answer identical research questions, findings should generally align and comparison across methods can serve as a robustness check.

For all the approaches, we note 2 important limitations. First, none of the 7 approaches discussed here are guaranteed to resolve the analytic challenges presented by co-occurring policies. For example, 1 approach might isolate the effects of the primary policy of interest from some co-occurring policies but not others; another approach might help reduce problems of statistical power arising from policy co-occurrence for 1 outcome of interest but not another. Second, all the approaches rely on accurate measurement of all of the relevant policies. Missing or mismeasured policies may lead to bias. Careful attention to the structure and potential impact of measurement error, along with analytic tools such as quantitative bias analysis, can enhance validity (130).

Limitations

The 7 approaches we have presented are not an exhaustive list of all analytic solutions that could be applied to address policy co-occurrence and many suboptions exist. However, we did not encounter any other method that addresses policy co-occurrence in our sample of studies. In addition, this study is based on a systematically gathered set of exemplar studies of the health effects of social policies; a comprehensive review of all studies of the health effects of social policies would be valuable for future research (e.g., to characterize patterns of methods use across journals and disciplines, and to assess whether studies are trending toward more rigorous approaches over time). Finally, as with all studies, there may be some misclassification. In particular, if an analytic approach was applied but not identified as being for the purpose of addressing co-occurring policies (or an analogous problem under any other name), we may have missed it.

CONCLUSIONS

Policy co-occurrence plagues most research on the health effects of social policies. In combination with Part 1 of this articles series, which illustrated how to assess the pervasiveness and consequences of policy co-occurrence, we offer guidance in this review on how to address this challenge. Although randomization of policy rollouts can best estimate the causal effects of social policies, when not available, other methods, nonetheless, can indicate causality. These other methods demand careful selection of the research question and analytic approach and, guided by deep substantive knowledge and creativity, can help overcome policy co-occurrence and deliver stronger evidence on the health effects of social policies.