Understanding the Ethics of Natural Experiments in a Pandemic

Pandemics invite natural experiments: testing hypotheses by observing the effects of interventions without manipulating exposure to the intervention.¹ With novel infections like COVID-19 that spread rapidly and widely, knowledge gaps may be extensive. Quick action may be necessary and randomized trials impracticable if not downright impossible. But what can justify such interventions ethically, given that they are in some sense widescale experiments on unaware members of the public who have no opportunity to choose not to participate? And what ethical limits to them should there be?

John Snow’s iconic 1854 experiment with the Broad Street pump handle is often cited as an admirable example of public health experimentation. Everyone in public health has no doubt heard the story: Snow meticulously tracked cholera infections in the Soho area of London, England, to households and businesses drawing water from a particular well on Broad Street, removed the pump handle, and stopped the epidemic cold.

Yet perhaps not as well recognized is Snow’s prescient grasp on how to experiment ethically. Leaving aside that Snow’s experiment worked spectacularly, why has his experimental intervention so unequivocally garnered acclaim? Snow began with what he viewed as the best available science of the day: the germ theory of disease. Before he began the experiment, he meticulously collected data about occurrences of infection and water sources. The experiment removed a likely risk—accessible cholera-contaminated water—without creating new risks or risks for different people. For example, Snow did not divert the water from the well to a different location to see whether a new outbreak would occur. Snow did not have an economic conflict of interest such as a competing nearby well—although he did stand to achieve reputational fame from the experiment’s success. To achieve removal of the pump handle, Snow consulted the Board of Guardians of St. James Parish, the local parish.² Finally, Snow also collected data about the results of the experiment and did his best to make them public for the benefit of all.

These observations suggest many features of Snow’s experiment that are relevant to assessing it as ethical:

• It comported with the principle “do no harm.”

• It was based on Snow’s best assessment of the available evidence that it might be of benefit.

• It did not risk significant injustice; the intervention was not expected to treat some in a way that was significantly unjust compared with others or to further structural injustices of the day.

• Snow acted transparently, explaining to local leaders what he hoped could be done and engaging them in implementing the experiment.

• Snow was rigorous about collecting data about the experiment’s impact.

These features reflect standard principles of medical ethics: nonmaleficence, beneficence, and justice. Moreover, by collecting data rigorously, Snow did his very best to obtain knowledge for everyone in the community. Although he did not obtain the “informed consent” of each community member, he engaged with the community in a manner that was open and consultative.

Another noteworthy feature of Snow’s experiment is that it was planned. Snow used his best assessment of the science of the day and the likely benefits—or harms—of dismantling the pump handle to select and implement his intervention. When pandemics strike, and the need for action seems immediate, many interventions that are not planned as experiments may occur. If the effects of the intervention later appear worthy of study as natural experiments, some of the features that made Snow’s experiment ethical may not be present. The intervention may have already begun, possibly initially without careful assessment of the science, reflection on likely risks or benefits, or community consultation. But this lack of planning as an experiment does not mean that the experiment’s features were ethically irrelevant; rather, it may reinforce the importance of those features that can be applied. For example, it may be especially important to acquire information in a timely manner to assess the impact of the intervention and any need for retrofitting if the intervention is going wrong.

Transparency will be critical if the intervention is seeding mistrust. Just as clinical trials have data safety monitoring boards and stopping points if unanticipated risks or clear evidence of benefits or risks in one of the trial arms emerges, interventions that later are examined as natural experiments might be scrutinized for evidence of benefit or harm or of inequity or injustice. Although the intervention itself might have occurred without community consultation, even the slightest anecdotal evidence that the intervention might be risky or beneficial to some at the expense of others may feed rumors, misinformation, and mistrust. Public health depends on public trust in pandemic times, when people may be asked to behave in ways they find uncomfortable, constraining, or seriously deleterious to their own welfare; any sense that information is being hidden or manipulated may exacerbate suspicions that trust is unwarranted. Rigorous data collection and evaluation of unplanned interventions as natural experiments may be an ethical counterbalance to these sources of mistrust.

Many recently published examples of natural experiments exemplify some but perhaps not all of these ethical features. Several involve unplanned experiments, too. For example, studies of the impact of new transit lines consider health-related impacts for use in making later investment decisions in public transportation, in some cases with the notice and consent of community members from whom data are collected.³ In a different kind of example, phase IV postmarketing studies of the safety and efficacy of pharmaceuticals bear some resemblance to natural experiments, as they attempt to ascertain less frequent risks when a product goes into more widespread use. London et al. argue in response to the possibility that these studies might be thinly veiled marketing efforts that an “integrity framework” must be applied in which the driving values are promotion of health and assessment of interventions driven by evidence about safety, efficacy, and value, a framework that should include oversight rather than reliance on individual informed consent.⁴ There are many other examples of how natural experiments can be designed in a rigorous way to answer a question that may have widely beneficial results, without causing harm or putting some at unjustly differential risk.

In the midst of the COVID-19 pandemic, many interventions are under way that might be regarded as natural experiments. Some are planned as experiments, such as the concerts held in Leipzig, Germany, under varying conditions to attempt to ascertain how COVID-19 spreads at public events. As of this writing, the results of this experiment have been published in preliminary form, and they demonstrate that concerts conducted with social distancing, wearing masks, and especially good ventilation systems have a low risk of disease transmission.⁵ An experiment such as this one can be assessed under the ethical features exemplified in Snow’s experiment.

Many natural experiments during pandemics are not initially planned as experiments, however. Bars are opening, restaurants are serving indoors, and schools are bringing students back into classrooms with each other. Mask wearing is being required, those not wearing masks are cajoled to do so, or people are left to choose on their own, depending on the jurisdiction. These interventions may have widespread effects on people in the communities where they occur. Yet depending on how they are conducted and studied, they may not comport very well with some of the features of Snow’s work that made it so acceptable.

One set of concerns would be whether these interventions are being structured or evaluated after the fact based on the best currently available science. Rigorous data collection to assess outcomes is critical to this assessment. Many have the potential to cause real harm to some—those at greater risk for pandemic infection—albeit also with the hope of significant benefit to others, such as the businesses that can reopen. School openings, despite extensive community spread of COVID-19, are defended because of their educational importance to children and the economic importance to their parents of being able to work. Yet school openings may put teachers, staff, and some students or their families at significant risk. Without careful data collection about how these experiments are playing out, we will lack critical knowledge about disease spread in schools and its effects on students, their families, and school personnel. Moreover, ongoing community consultation may be especially important as more is learned about the impacts of the intervention on so many community members.

Now more than ever, our goal should be to encourage Snow-like models of ethical inquiry to learn from our natural experiments about how to address the evolving challenges of COVID-19 and plan sensibly for pandemics to come.