Rigorous Clinical Trial Design in Public Health Emergencies Is Essential

To obtain definitive information about the effects of treatments and vaccines, even in public health emergencies, randomized clinical trials are essential. Such trials are ethical and feasible with efforts to engage and collaborate with the affected communities.

Abstract

Randomized clinical trials are the most reliable approaches to evaluating the effects of new treatments and vaccines. During the 2014–2015 West African Ebola epidemic, many argued that such trials were neither ethical nor feasible in an environment of limited health infrastructure and severe disease with a high fatality rate. Consensus among the numerous organizations providing help to the affected areas was never achieved, resulting in fragmented collaboration, delayed study initiation, and ultimately failure to provide definitive evidence on the efficacy of treatments and vaccines. Randomized trials were in fact approved by local ethics boards and initiated, demonstrating that randomized trials, even in such difficult circumstances, are feasible. Improved planning and collaboration among research and humanitarian organizations, and affected communities, in the interepidemic periods are needed to ensure that questions regarding the efficacy of vaccines and treatments can be definitively answered during future public health emergencies.

(See the Brief Report by Rojek et al on pages 1454–7.)

The West African Ebola outbreak of 2014–2015 was unprecedented. As the first multicountry Ebola epidemic, it affected more individuals and caused more deaths than all previous Ebola outbreaks combined. Unfortunately, awareness of its scope was slow to develop, delaying the initiation of clinical trials. None of the completed therapeutic trials demonstrated efficacy (although the results of 1 study were suggestive); 1 of 4 vaccine trials produced results strongly suggestive of protective efficacy but with interpretive difficulties. To better plan for trials in a future outbreak—whether Ebola or another emerging infection—the US National Academies of Sciences, Engineering, and Medicine convened a committee to systematically review the studies conducted during the outbreak and to make recommendations for the future. The report, released in April 2017 [1], evaluated the study designs proposed/employed [2–9] and considered how to improve the quality of future research. Here we summarize the report’s conclusions about study designs.

Randomized controlled trials (RCTs) are generally recognized as the optimal way to evaluate new therapeutic and preventive interventions [10–14]. However, in situations involving serious diseases without satisfactory treatment, this approach has been questioned [15–19]. Although a handful of treatments have been shown to be effective in small, uncontrolled studies, such as platinum for treatment of testicular cancer [20], this is exceedingly rare. Most effective interventions provide modest to moderate improvements, which cannot be reliably identified in uncontrolled studies. The 2 primary attributes of RCTs—use of a concurrent control group and the random assignment of treatments—are critical to drawing valid conclusions about treatment effects.

CONCURRENT CONTROL GROUP

If individuals with a particular disease or condition had uniform outcomes, there would be no need for control groups. But this is rarely the case—even diseases with known poor prognosis typically have variable courses. In addition, emergence of new techniques allowing earlier diagnosis, or, as observed during the West Africa Ebola outbreak, the introduction of improved supportive care over time make the historical experience for evaluating new treatments problematic.

RANDOM TREATMENT ASSIGNMENT

Individuals who agree to participate in a clinical trial may differ from others with the same diagnosis in ways that affect prognosis. Even when adjusting for factors known to affect study outcomes, nonrandomized studies can be misleading due to differences in unmeasured or unknown factors. Countless examples of purported treatment effects emerging from observational data have been definitively refuted in subsequent RCTs [21–24].

CLINICAL TRIAL DESIGNS DURING THE 2014–2015 EBOLA OUTBREAK

At the outset of the Ebola epidemic, several experimental drugs and vaccines were in very early stages of development and none had yet been studied in humans. Unfortunately, consensus regarding priority interventions and trial designs was difficult to achieve. A particular area of debate was whether an RCT was ethical in the face of such a public health emergency [19, 25]. Some argued that randomization to a placebo control (in addition to all available supportive care) would be unethical given the expected high mortality, and presumed the affected communities would reject such a design. These concerns led several investigative teams to initiate uncontrolled trials of experimental treatments, hoping to observe a survival rate high enough to establish efficacy based on comparison with historical estimates [26–28]. Others suggested initial uncontrolled trials of investigational agents rapidly followed by RCTs for any agent found promising but not definitively effective [3]. This approach was applied to just 1 agent, which did not pass the first stage before the epidemic waned [29]. A “platform” design was also proposed, randomizing individuals among several different treatments and increasing the proportion randomized to treatments that appeared to be more effective as the trial progressed [5]; however, the epidemic was brought under control before it could be implemented. Only 1 RCT, comparing ZMapp to placebo with everyone receiving optimized supportive care, was initiated [30].

Ultimately the uncontrolled therapeutic trials did not demonstrate the extremely large effects required for credible conclusions, and the single RCT evaluating ZMapp did not enroll enough patients before the epidemic waned to definitively assess benefit. The observed mortality of 37% in the control group was substantially lower than the historical rates, demonstrating the difficulty in interpreting uncontrolled data. Had the mortality been as low as 37% in the single-arm trials, the products studied would have been viewed as extremely promising, as expected mortality was at least 50%.

In the case of vaccine trials, randomization was less controversial because the individuals involved were not currently ill. Several RCTs were initiated [31–34], but the only study able to evaluate efficacy used an innovative “ring” strategy in which clusters were defined around observed cases, and then randomized to immediate or delayed vaccination [31]. This approach defined clusters at elevated risk of infection so that despite the waning of the epidemic, enough cases were observed to permit meaningful efficacy analysis. A statistically significant estimate of 100% protection was obtained when individuals in the “immediate” clusters who actually received vaccine (approximately two-thirds of this randomized cohort) were compared to all individuals in the clusters randomized to “delayed” vaccination. However, this analysis violates the intention-to-treat (ITT) principle, which requires inclusion of all individuals randomized to both arms whether or not they received the experimental treatment [35]. The ITT results (included as an additional analysis in the final report) yielded a lower estimate of vaccine efficacy of 65%, which did not reach statistical significance. This is not a minor technicality—those randomized to be vaccinated but were not could be different from those who were vaccinated in ways that influenced the likelihood of infection [36]. Moreover, the trial was not masked to control potential biases in identifying cases.

These findings present major challenges for regulators, product manufacturers, and research organizations. Without definitive evidence of efficacy, will products be approved? Will manufacturers ramp up production of the promising products whether or not regulatory approval is granted, in anticipation of “compassionate use” in a future outbreak? Will trial organizers plan further studies using these products as controls instead of using placebo controls?

URGENCY DOES NOT OVERRIDE THE NEED FOR RELIABLE RESULTS

It is understandable that in a context with no known effective therapies, those treating the sick would want to use any accessible and potentially active treatment [37]. But conducting human research in a manner that does not conform to scientific standards and is thus unlikely to yield actionable findings is itself ethically questionable [38]. If drugs were approved based on promising early uncontrolled results, the outcome could be a plethora of new available treatments; individuals and physicians in desperate situations would have multiple drugs to choose from but no reliable information about their effects. No group has understood this dilemma better than the AIDS activists in the late 1980s, who very quickly became strong advocates for rigorous study designs to evaluate new treatments for human immunodeficiency virus infection [39].

Similar considerations apply to vaccines. Public health officials dealing with future outbreaks will face an inevitable and difficult trade-off between obtaining efficacy data as rapidly as possible, and obtaining the long-term observations needed to fully assess product safety and durability of protection.

SUMMARY: PROMOTING FURTHER CONVERSATION AND CONSENSUS

The scientific output from the clinical trials in West Africa has been characterized as “thin” [40]. It took too long for trials to be planned, vetted, and initiated, and as most of the trials were neither randomized nor adequately controlled, results could not in the end support conclusions about safety or efficacy. This experience should motivate investigators to plan for the inevitable future epidemics during the interepidemic periods and to drive consensus about trial design and conduct among the various research and humanitarian organizations and local communities before the next outbreak, whether of Ebola or another pathogen [41, 42]. Otherwise, we may well repeat the disappointing outcomes of the recent Ebola experience.

Notes

Acknowledgments. The authors acknowledge the invaluable assistance of Patricia Cuff, Michelle Mancher, Emily Busta, Julie Pavlin, and Michael Berrios in collecting and summarizing the information on Ebola treatment and vaccine studies for the National Academies Committee on Clinical Trials during the 2014–2015 Ebola epidemic.

Financial support. This work was supported by the National Academies of Science, Engineering, and Medicine, with funding from the Office of the Assistant Secretary for Preparedness and Response; the US Department of Health and Human Services; the National Institute of Allergy and Infectious Diseases; and the US Food and Drug Administration; the Division of Biostatistics, Epidemiology and Informatics in the University of Pennsylvania Perelman School of Medicine; the Medical Research Council (MRC_UU_12023/23); and the Boston University Schools of Medicine and Public Health.

Potential conflicts of interest. C. D. W. is an employee of Sanofi, Inc. A. G. B. has received grant funding from the UK Medical Research Council. J. D. L. has received grant funding from the Danish National Research Federation. K. M. E. has received grant funding from Novartis for unrelated research. S. S. E. served on a data monitoring committee for GeneOne and as an investigator on a study for which drugs were provided by AbbVie. All other authors report no potential conflicts. All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.