Commentary: Natural and Unnatural Experiments in Epidemiology

In the past 1-2 decades, the use of natural experiments to produce more rigorous estimates of the effects of various risk factors on health has proliferated. While the term natural experiment is used to encompass different study designs and can involve any number of analytic techniques, these types of studies generally aim to take advantage of natural—or in many cases, unnatural—variation in an exposure that may otherwise be challenging to randomize. The goal is to overcome confounding more robustly than standard correlational analyses of observational data by identifying some event that resulted in an element of randomness (or quasi-randomness) in the exposure of interest. Quasi-random is another term that can be used in different ways, typically referring to an exposure that was not truly random, but rather arbitrary, resulting in near-random assignment conditional on some observed or (less ideally) unobserved factors. Among the first studies of this type was published by Hearst et al. in the New England Journal of Medicine (1986), showing that military service in the Vietnam War increased subsequent deaths from suicide and motor vehicle accidents. While a comparison of military service members to others may suffer from confounding due to unobserved differences that lead some individuals to enlist and others not to, this study leveraged the Vietnam draft lottery, which randomly conscripted some men into military service based on their dates of birth.¹ The use of such natural experiments in the epidemiology literature has dramatically increased since this seminal study; a PubMed search for “natural experiments” returns more than 150 studies annually in recent years, up from fewer than 30 annually in the early 2000s.

The use of natural experiments holds particular appeal for social epidemiologists, since social factors are often exceptionally difficult to randomize due to ethical reasons or feasibility. Estimating the effects of neighborhood-level and other place-based characteristics is notoriously thorny,² given the potential for selection of unhealthy individuals into less desirable neighborhoods and the rarity of opportunities to randomize individuals’ place of residence.³ A better understanding of the causal role of various neighborhood characteristics would help to guide interventions to improve the wellbeing of vulnerable populations by altering the composition and content of their environments.

In this issue, Shiba et al. share the results of one of several recent studies that take advantage of relocation of disaster victims in the wake of the 2011 Great East Japan Earthquake to examine the health effects of neighborhood factors.⁴ Drawing on cohort data on a sample of older adults, the authors leveraged variation in the proximity to local “food and recreation destinations” induced by relocation in the wake of the earthquake to evaluate the influence of these neighborhood characteristics on cardiometabolic outcomes. They conclude that relocation to a region with a high density of food and recreation facilities was associated with worsening of several indicators of cardiometabolic health including body mass index, waist circumference, and low- and high-density lipoprotein cholesterol. Since there were only 57 individuals in the study who were victims of the forced displacement and for whom health outcomes were available, Shiba et al. commendably couple this analysis with a more traditional fixed-effects analysis, in which they instead leverage within-person variation in the density of these neighborhood resources over time in a larger sample of 1630 individuals who were not displaced, with similar findings to the natural experiment.

The strength of any study that employs a natural experiment hinges upon the circumstances surrounding the natural experiment itself and the credibility of the (quasi-)randomization. In the case of Shiba et al., readers must be convinced that—among those who were forcibly displaced as a result of the earthquake—assignment to neighborhoods of higher or lower density of food and recreation destinations was as good as random. In this case, satisfying this criteria would mean that the government officials who assigned individuals into the trailer homes or subsidized apartments did not have information on unobserved or unmeasured characteristics of the study subjects (e.g., health status, frailty, family connections) that may have altered placement decisions and may have also influenced the health outcomes in question. Similarly, the study subjects themselves would have had to have had no input into their placement, or else the risk of self-selection would lead to the same problems as in a standard observational study. The burden of proof for supporting these assertions rests with the authors. In this case, a closer examination of official Japanese government records or interviews with affected individuals might clarify this point and strengthen the credibility of the natural experiment.

Another limitation of natural experiments more generally, compared with standard RCTs, is that researchers do not have control over the nature of the exposure in question, and instead rely on the natural or unnatural circumstances that altered the nature of the exposure. In many cases, however, the conditions that change as a result of the natural experiment are multifaceted and related, making it difficult to tease apart the salient feature of the exposure. In this case, the authors are interested in the effects of proximity to food and recreational facilities. Yet it is likely that these characteristics are correlated with, and confounded by, other neighborhood characteristics like housing quality, violent crime, and healthcare access that may be related to the outcomes of interest. Indeed, this same natural experiment in Japan has previously been used to examine the effects of neighborhood social capital.⁵ While the theoretical links between cardiometabolic outcomes and the food and physical environment strengthen the conceptual basis for the current study, the selection of any particular neighborhood characteristic as the exposure of interest is nevertheless somewhat indiscriminate, such that the results cannot be construed as definitively implicating the effects of the proximity to the food and recreational facilities themselves. Once again, this is a challenge for the broader neighborhood-health literature, as neighborhood resources often tend to cluster together, making it challenging to disentangle the effects of one specific feature. While this makes it difficult to advance theory on specific mechanisms through which neighborhoods may affect health, it is nevertheless reflective of the complexities of the real world in which individuals live.

Relatedly, the present study relies on the relocation of affected individuals as the source of variation in the exposure, rather than changes to neighborhood quality of stably housed individuals. This is similar to other previous natural experiments in the neighborhood-health literature.^6-8 The authors argue that this is perhaps more policy-relevant than altering the neighborhood environment. While the climate crisis is unfortunately likely to make such forced relocations more common due to natural disasters, displacement in this fashion is nevertheless perhaps more of an unnatural rather than a natural experiment. In this way, natural experiments share a drawback with RCTs, namely threats to generalizability beyond the specific circumstances of the (quasi-)randomization, and challenges to transportability of estimates from studies that rely on different study designs, populations, and analytic methods.⁹ It is true that, in the U.S. at least, major housing policies often rely on issuing vouchers to allow low-income individuals to relocate, e.g., through the Section 8 Housing Choice Voucher Program. At the same time, it may be more plausible and productive for governments and community groups to invest in specific community resources rather than relocating families and potentially disrupting existing social ties. There is limited work that tries to identify and exploit variation in single neighborhood features—most notably prior research on the effects of grocery stores on dietary habits and health^10,11—and this represents an opportunity for future research to examine neighborhood characteristics such as transportation, green spaces, and crime. Comparability of estimates from separate studies could also be enhanced by attempting to harmonize study procedures across analyses, or at the least carrying out sensitivity analyses to understand the source of discrepancies.⁹

Ultimately, all natural experiments are flawed in some way. Even studies exploiting the Vietnam draft lottery are imperfect, since the lottery was later demonstrated to be less than truly random.¹² Rather than dismissing the findings of natural experiments in which the circumstances of the randomization are less than ideal, building a rich and convincing evidence base about the effects of neighborhood characteristics—or other factors that are challenging to randomize—requires that we take a pluralistic approach to study designs.¹³ Shiba et al. employ this strategy by employing both a natural experiment and a standard fixed-effects analysis. While these two approaches are no longer targeting the same causal parameter, since the fixed-effects analysis only exploits variation in neighborhood factors within rather than across individuals, the two methods arrive at similar conclusions. By taking this approach, researchers can “triangulate” evidence from multiple studies using different study designs, data sources, analytic approaches, and contexts, to converge on a more holistic understanding of the causal effects of neighborhoods on health.