Making better use of natural experimental evaluation in population health

Natural experiments have long been used as opportunities to evaluate the health impacts of policies, programmes, and other interventions. Defined in the UK Medical Research Council’s guidance as events outside the control of researchers that divide populations into exposed and unexposed groups, natural experiments have greatly contributed to the evidence base for tobacco and air pollution control, suicide prevention, and other important areas of public health policy.¹

Although randomised controlled trials are often viewed as the best source of evidence because they have less risk of bias, reliance on them as the only source of credible evidence has begun to shift for several reasons. Firstly, policy makers are increasingly looking for evidence about “what works” to tackle pervasive and complex problems, including the social determinants of health,^{2 3} and these are hard to examine in randomised trials. In Scotland, for example, legislation to introduce a minimum retail price per unit of alcohol included a sunset clause, which means that the measure will lapse after six years unless evidence is produced that it works. This has resulted in multiple evaluations, including natural experimental studies using geographical or historical comparator groups.⁴ Similarly, the US National Institutes of Health has called for greater use of natural experimental methods to understand how to prevent obesity,⁵ and a consortium of European academies for their greater use to understand policies and interventions to reduce health inequalities.³

Secondly, a wider range of analytical methods developed within other disciplines, mostly by economists or other social or political scientists, are being increasingly applied to good effect. A good example is the use of synthetic control methods to evaluate the effect on mortality of the introduction of a pay-for-performance scheme for financing hospital care.⁶ There is also a greater availability of large administrative and other “big” data sources that link information on exposure to public policies with health and other outcomes.

Although natural experimental evaluations have an established foothold in population health research, particularly to support policy making, more work is needed to identify the best opportunities for natural experimental studies and to support their design, conduct, and synthesis to realise their full potential.

Diversifying the sources of evidence

The idea that there is a hierarchy of study designs, ranked according to susceptibility to bias, remains influential.^{7 8} A common shorthand for this view is that randomised controlled trials are the gold standard for evaluation, and that observational study designs are irredeemably weaker in all circumstances. An alternative view is that, while unbiased estimates of effectiveness are an important goal of evaluations, they are not the only goal, and may be unachievable in some circumstances.⁹ If research seeks to produce evidence that is useful for policy and other decision making, a wider range of study designs is needed, including those that work in situations when a planned experiment would not be feasible or ethical.

During the covid-19 pandemic, for example, randomised trials provided crucial evidence about the efficacy of vaccines in reducing risk of infection and severe disease, as well as the efficacy of treatments.^{10 11} But observational studies of the effects of interventions in practice have also made important contributions^{12 13}: evidence on longer term effectiveness of vaccines, has been established through cohort studies conducted in the context of large scale vaccination campaigns when it was unethical to withhold vaccines that had already been shown to be efficacious¹² and evidence on the impact of physical distancing interventions was generated employing interrupted time series analyses using routine data from 149 countries.¹³ Furthermore, evidence on the effectiveness of surveillance was obtained from ingenious natural experimental evaluations that exploited flaws in the implementation of the test and trace programme in England.^{14 15} The adverse effects of the UK Treasury’s “eat out to help out” scheme on infection rates were likewise identified by treating the scheme as a natural experiment.¹⁶

Despite growing acceptance of the value of natural experimental evalutions,^17–20 most discussion of their design focuses on quantitative methods.^{17 18 21} Other aspects of study design, conduct, and interpretation, such as how to identify a good opportunity for a natural experimental evaluation, how open science principles such as registering a study should be applied, or how to place effect estimates into a broader framework of whether, how, and in what circumstances the intervention achieves its effects, have been largely neglected.

Moving from classifications to opportunities

One key conceptual issue is how broadly or narrowly natural experiments should be defined.²² A related question is whether it is useful to distinguish sharply on methodological grounds between natural experimental evaluations and other observational studies that attempt to identify causal relationships using change or variation in exposure that is not associated with a specific event or process, such as the implementation of a new policy.

The UK MRC’s broad definition of a natural experiment contrasts with attempts to narrow the definition to include only studies that use one of a prescribed range of analytical methods,²³ or that satisfy some other criterion such as “as if randomisation” (a real world process leading to variation in exposure that approximates random allocation in a trial,²⁴ such as the use of lotteries to allocate military conscription²⁵ or school places.²⁶ However, broad study design labels are an inadequate proxy for study quality, which depends on the extent to which assumptions are tested, threats to validity evaluated, and robustness checks performed.²⁷ Lists of approved analytical methods can rapidly become dated as new methods are developed and existing ones refined. For example, synthetic control methods, which use a weighted composite of control areas rather than a geographical control area, have been widely applied to evaluate public health and healthcare interventions such as state level tobacco²⁸ and firearms control²⁹ policies in the US, but rarely feature in such lists.³⁰

Additionally, even though “as if randomisation” provides a strong basis for causal inference from a natural experiment, other than in clear-cut cases such as lotteries it is difficult to define precisely when the criterion is satisfied: few population health interventions are or could be implemented in this way. Rather than trying to sharply differentiate natural experimental evaluations from other observational studies on methodological grounds, it may be more useful to think about the sets of circumstances that are likely to generate useful opportunities for robust research using natural experimental evaluations.¹⁴

Recognising opportunities for natural experimental studies

Natural experimental evaluations are most commonly used in situations where there is a clear division in presence, level, time, place, or type of exposure between two or more otherwise similar subpopulations—for example, when a policy is implemented in one state within a federal jurisdiction but not in neighbouring states. Several other situations recur in the literature and provide useful pointers for the design of future studies, including policies with eligibility criteria that clearly define exposure, phased implementation of policy, the use of randomisation to determine entitlements or obligations, and flaws or shortcomings in policy delivery (table 1).

Table 1. Policy-related opportunities for natural experimental evaluations.

Type of opportunity	Examples
New policy with a clearly defined inception date or administrative jurisdiction	Gun control laws are implemented at state level within the US, so the effect of introducing or withdrawing a law in one state can be evaluated using other states as controls29
Policy that applies to some members of a population but not others	Social security policies often define eligibility for payment in terms of age or income, allowing comparisons between individuals with ages or incomes just above or just below the eligibility cut-off31
Phased implementation of a policy across a population	Universal Credit, a new system of social security benefits and tax credits in the UK, was rolled out area by area, allowing comparisons of recipients of benefits under the old system and the new system using data from surveys running throughout the rollout period32 33
Policies using randomisation as an assignment mechanism, generating otherwise similar exposed and unexposed individuals or groups	Election laws in two Indian states required a minimum number of council seats be reserved for women in a randomly selected proportion of villages, allowing comparisons of decision making in villages with larger or smaller numbers of women involved34
Flaws or shortcomings in policy implementation	Database errors led to people in some areas being randomly excluded from England’s test and trace programme for covid-19 until the error was noticed. This enabled trends in infection rates to be compared in areas with and without tracing in operation14

Whether such situations generate opportunities for good evaluations depends as much on availability and quality of data as on the nature of events or processes themselves.²² Many natural experimental evaluations are conducted retrospectively so good quality, routinely collected data from administrative systems, population surveys, or other sources is critical. Similarly, it is important to be able to accurately characterise the nature and timing and, where relevant, the intensity and implementation of the intervention being evaluated to correctly identify individuals or groups who were or were not exposed, or had varying levels of exposure. This often relies on access to good quality documentary evidence, as well as access to key informants who can remember and reliably describe the intervention, including how and when it was implemented.

Study registration policies

Another question that has been relatively neglected with respect to natural experimental evaluations is the registration of study protocols. For prospective studies such as randomised controlled trials, registration, especially if enforced by funders and stipulated by journal editors, is a powerful safeguard against some forms of manipulation, such as selective publication of favourable findings. For retrospective studies, where researchers may be familiar with a dataset before the study begins, transparency about how such prior knowledge has affected design choices is vital.

The protocols for natural experimental evaluations may have to be amended to accommodate changes as the evaluation progresses, such as developments in theoretical understanding of the nature of the intervention or a fuller appreciation of the characteristics of the data. For example, missing data may require modification of the analysis plan to use a different set of covariates or an alternative analytic method. Often protocols are published in a journal with no facility for updating. Natural experimental evaluation protocols may benefit from the flexibility now accepted for systematic review protocols, which can have amendments recorded.^{35 36}

Improved evidence base

Evidence from natural experimental evaluations can provide insight beyond estimates of effect size and contribute to understanding the importance of mechanisms or context of interventions within systems.^{37 38} Greater awareness and use of a range of methods to estimate the effects of interventions not under researchers’ control—and to understand how, where, and for whom those effects are realised—is essential for developing a robust and useful evidence base for policy. Indeed, natural experimental evaluations have already proved their value across a wide and disparate range of health and non-health policy areas, providing otherwise unobtainable evidence about the effects on population health of clean air legislation, suicide prevention, tobacco and gun control, trade agreements, non-pharmaceutical pandemic control measures, and many other kinds of interventions.

We believe they can contribute much further to these and other areas if the focus moves beyond justifying their use to optimising their execution. Making the most of evidence that can be obtained from natural experiments requires incorporating economic evaluations and modelling^{39 3} as well as qualitative methods that could provide vital information about possible causal mechanisms.^{40 41} Further guidance on how to identify opportunities for natural experimental evaluations, on how to design, conduct, report, and synthesise the evidence from such studies, and on what kinds of research infrastructure and governance processes are needed will help to realise this potential.

Key messages.

• Natural experimental evaluations can provide useful information to guide decision making about interventions

• Most discussion has focussed on what quantitative methods are suitable for natural experimental evaluations

Key definitions and concepts remain contested and there is a lack of consensus about the circumstances in which natural experimental evaluations can provide trustworthy and useful evidence for decision-making

• Guidance should help identify the circumstances that make for good natural experimental evaluation, and a range of applicable methods