Obtaining Efficacy Data in an Environment of Variable and Uncertain Incidence: Ebola and Beyond

Many epidemic-prone infectious diseases present challenges that the current West African Ebola outbreak brings into sharp relief. Specifically, the urgency to evaluate vaccines, initially limited vaccine supplies, and large and unpredictable spatial and temporal fluctuations in incidence have presented huge logistical, ethical, and statistical challenges to trial design.

In the Ebola outbreak, long and intense discussion led to broad agreement on the need to evaluate the efficacy of Ebola vaccines through an individually randomized controlled trial (iRCT) (1), with cluster-randomized designs providing supplemental information. However, by the time an iRCT began in Liberia, the ability to estimate vaccine efficacy was threatened by the otherwise welcome declining incidence of Ebola virus infection (2). Other trials, planned to provide evidence on vaccines’ direct and indirect (herd immunity) effects, might not be able to include enough Ebola cases to provide statistically robust efficacy estimates (2, 3).

Similar challenges may arise when evaluating vaccines for diseases such as meningococcal meningitis, cholera, Middle East Respiratory Syndrome and other coronavirus infections, vector-borne viral diseases such as dengue and chikungunya, and novel influenza strains. Resource-poor populations continue to be at particularly high risk for such infections (4). We suggest three principles, all well-established in the clinical trials literature and applied to varying degrees in the Ebola vaccine trials, that will be of general use in designing vaccine trials during emergencies [Trials of therapies for infected persons arguably involve a different set of logistical and ethical challenges (5)]. Each principle is mainly responding to a challenge identified in the Ebola context: block randomization with matching is a response to heterogeneity of incidence; stepped rollout is a response to urgency; and adaptive design is a response to uncertainty.

Principle I: Block randomization within small centers, with analysis matched by center

For Ebola and other diseases, participants in different districts might experience a surge in cases and hence in infection risk at different times after randomization, depending on the local dynamics of the epidemic. Incidence is thus likely to be considerably more similar between intervention and control participants within a center than in the population as a whole.

To deal with very different incidence across sites, we suggest that randomization of participants to investigational vaccine or control should take place separately within each center (block randomization) and that vaccine efficacy estimates should be obtained for each center and combined statistically to obtain an overall efficacy estimate. A center would be a relatively small group of persons projected to have relatively homogeneous exposure to Ebola infection over the following months. In practice, centers could be composed of the frontline workers at a single Ebola treatment unit, burial teams in a single district, or geographic subgroups of the general population. Matching the analysis by center therefore compares individuals whose risks are more similar to one another and may thereby improve statistical efficiency (6).

Using block randomization of participants within small centers would also maintain balance (and limit loss of sample size) even if at the analysis phase, it becomes necessary to exclude centers in which data are expected to be unreliable—for example, due to expected failure of vaccine delivery or the cold chain or other overwhelming logistical challenges. Previously, such an approach was successfully used to evaluate approaches to national health insurance in Mexico and ensure that a trial design was “politically robust” (7, 8). If analysis provided evidence of interference by local politicians with the randomization scheme, matched pairs of districts could be removed.

Principle II: Stepped rollout

Individual centers may become ready to commence a trial at different times because of factors that include vaccine availability, timing of identification of centers and participants in areas of continuing incidence, establishment of a reliable cold chain for vaccine delivery, setting up of information systems, and contracting available trained personnel for vaccine administration and follow-up. A standard iRCT (see Fig. 1A) would await readiness of all centers before commencing. Instead, stepped rollout initiates the trial promptly in each center as soon as that center is ready (see Fig. 1B). Stepped rollout has been used for logistical (9) or political (7, 8) purposes. A key advantage of this approach is shortening the lag between starting a trial and accruing sufficient person-time (Fig. 1B). Shortening the lag may be particularly important when incidence in an area surges over several weeks due to intense transmission.

Fig. 1.

Strategies for vaccine trials in challenging conditions. (A) A classic iRCT. Randomization has occurred within the whole population, and the trial can only start when all of the centers are ready. The red line represents the maximum number of participants the trial could enroll, given increasing logistical capacity. The area of the box is the total person-time in the trial; it is the size of the trial measured in person-weeks of observation. (B) Illustration of stepped rollout (beginning the trial at each center when that center becomes ready to participate) and adaptive design [planning to add persons in new centers (green) and/or extend person-time of observation in existing centers (yellow), depending on incidence]. The estimated person-time required to do the trial (purple box) is the same as in (A), but because under stepped rollout the trial can start earlier, it can end earlier. Parameters are all illustrative, and (for simplicity) centers are shown as be-coming ready in a linear manner with time.

Principle III: Adaptive design

Vaccine trials require participants who are expected to be at high risk of infection weeks to months later (after an immune response has been generated). Thus, identification of trial sites relies on predictions of future incidence, which are exceedingly difficult. The third principle states that centers can be added adaptively, which counters the unpredictability of incidence by allowing flexibility in sample size. Adaptive designs can increase statistical power by continuing to observe participants already in the trial and/or by adding centers or participants according to prespecified rules (10). For example, in the Ebola outbreak, an early estimate of the sample size for the Liberia trial was about 27,000 persons followed for 3 to 4 months (11–13). However, because of spatiotemporal variations in incidence, the follow-up time was extended to 10 to 12 months before the trial began (14).

The design of adaptive trials requires the specification of rules by which decisions to add new centers, to continue follow-up in existing centers, or to end the trial for success or futility will be made at various stages (10). This approach also permits adaptive adjustments to shifting conditions in later centers (e.g., availability of vaccine or the need for additional trial arms), based on lessons learned in earlier centers (15).

Broader applications and a research agenda

Each design principle has been employed in other settings to resolve particular challenges of trial design. We believe this combination might find broad application in many such settings and contributes to a larger effort to define common principles and practices for such settings. For example, an Ebola vaccine trial in Guinea has just begun (16) with “ring vaccination”—vaccinating “rings” of contacts and geographic neighbors of confirmed cases, a strategy previously used for smallpox eradication. To evaluate vaccine effectiveness, rings will be randomized such that all individuals in certain rings will be offered experimental Ebola vaccine immediately upon identification of a case of infection, whereas individuals in other rings will be offered vaccine only after some delay. The differential incidence of disease between rings with immediate versus delayed vaccination will be a measure of vaccine effectiveness. Enrolling new rings as Ebola cases are detected is a form of stepped rollout that, by focusing the trial in areas of known transmission, is particularly well suited to circumstances of spotty and declining incidence. The design is adaptive as well, adding rings until there is sufficient statistical evidence to stop the trial, because at that point there is either enough evidence to declare the vaccine efficacious (success) or no reasonable likelihood of garnering such evidence by continuing the trial (futility). In addition, to improve comparability of the risk between vaccinated and not-yet-vaccinated people, immediate-vaccination rings can be matched for analysis with delayed rings that are similar in certain ways (e.g., geographically proximate rings).

Limited infrastructure, unpredictable variation in incidence, and a public health imperative to provide vaccine as quickly as possible are common in vaccine efficacy trials, even outside public health emergencies. Conduct of vaccine trials in outbreak settings and in populations most likely to benefit from vaccination is fraught with difficulties but reveals essential information about vaccine performance. Trial designs that are appropriately adapted to the most challenging settings are sorely needed, not only to improve the external validity of trial results but also to ensure that vaccine quickly reaches those most in need.

Outside the Ebola context, feasibility of particular designs will depend not only on the epidemiology of the relevant disease (time and spatial scale of transmission, ease of diagnosis, etc.) but also on the characteristics of the vaccines being tried (including number of doses required, timing of immunogenicity, and potential for post-exposure effectiveness). Ethical considerations of speeding the availability of possibly efficacious vaccines to large numbers of people will need to be balanced against the need for evaluation of vaccine efficacy so that resources can be concentrated on effective interventions. But discussions of ethical, logistical, and statistical considerations in trial design take time, and rapid implementation of studies is important for timely and reliable results before the epidemic wanes. The more such discussions can take place outside emergencies and establish general principles to inform vaccine trial designs in future outbreaks, the more effective the responses to such outbreaks will be.