A Deferred-Vaccination Design to Assess Durability of COVID-19 Vaccine Effect After the Placebo Group Is Vaccinated

Continued follow-up of participants in COVID-19 vaccine trials after vaccination of the placebo group permits determination of vaccine durability and allows the incorporation of a booster trial if waning vaccine efficacy is observed, while ensuring the ethical treatment of participants.

Abstract

Multiple candidate vaccines to prevent COVID-19 have entered large-scale phase 3 placebo-controlled randomized clinical trials, and several have demonstrated substantial short-term efficacy. At some point after demonstration of substantial efficacy, placebo recipients should be offered the efficacious vaccine from their trial, which will occur before longer-term efficacy and safety are known. The absence of a placebo group could compromise assessment of longer-term vaccine effects. However, by continuing follow-up after vaccination of the placebo group, this study shows that placebo-controlled vaccine efficacy can be mathematically derived by assuming that the benefit of vaccination over time has the same profile for the original vaccine recipients and the original placebo recipients after their vaccination. Although this derivation provides less precise estimates than would be obtained by a standard trial where the placebo group remains unvaccinated, this proposed approach allows estimation of longer-term effect, including durability of vaccine efficacy and whether the vaccine eventually becomes harmful for some. Deferred vaccination, if done open-label, may lead to riskier behavior in the unblinded original vaccine group, confounding estimates of long-term vaccine efficacy. Hence, deferred vaccination via blinded crossover, where the vaccine group receives placebo and vice versa, would be the preferred way to assess vaccine durability and potential delayed harm. Deferred vaccination allows placebo recipients timely access to the vaccine when it would no longer be proper to maintain them on placebo, yet still allows important insights about immunologic and clinical effectiveness over time.

Safe and durably effective SARS-CoV-2 vaccines hold the potential to dramatically alter the COVID-19 pandemic (1, 2). Several vaccine candidates are currently in phase 3 clinical testing, with key criteria for success including vaccine efficacy and safety. Early results from multiple trials (3–8) suggest high efficacy that far exceeds the U.S. Food and Drug Administration guidance threshold of 50% for symptomatic disease and severe disease. Yet critical questions remain, including the effects in such subgroups as elderly persons and ethnic minority populations and assessment of longer-term efficacy and safety, given theoretical concerns for harm, including vaccine-associated enhanced disease (a potential concern in such subgroups as elderly persons [9–11]), and waning protection (9, 12, 13). The latter concern must be considered in light of studies of seasonal coronaviruses (14–16) and natural infection by SARS-CoV-2 (17–20), which suggest that immunity may wane within 6 months to 2 years in some people.

Understanding whether durability of vaccine efficacy might be improved by revaccination is another critical question (13, 15, 16). Although long-term safety and durability of efficacy are best evaluated by continued blinded follow-up of the original study groups (21), at some point there is an ethical imperative for placebo recipients to be offered the vaccine. The timing of this offer is complex, with individual risk being weighed against the societal value of the continued placebo-controlled follow-up, society's perception of fairness, and the availability of the vaccine outside the trial (22).

We demonstrate the remarkable fact that vaccine efficacy versus placebo can still be estimated even after the placebo group has been completely immunized, provided that a trial continues with high levels of follow-up and compliance. We assert that the ideal way to maintain rigor is by implementing a blinded crossover design in which the original study groups receive the opposite product during the deferred immunization period (creating early versus deferred vaccination groups). Besides assessment of durability, continued follow-up allows for collecting measurements of postvaccination immune responses in the newly vaccinated recipients. These additional measurements can substantially increase the statistical power in immune correlate analyses that assess whether and how these immune responses correlate with risk for disease or with vaccine efficacy (23), 2 important goals in vaccine trials. In addition, continued follow-up allows a quick pivot to a randomized trial of a booster dose of the vaccine to assess whether waning efficacy can be reversed.

Recovery of Vaccine Efficacy Without a Placebo Group

Figure 1 shows a schematic for deferred immunization of the placebo group via blinded crossover. After randomization to the vaccine or the placebo group, participants are followed for COVID-19 case accrual and early efficacy is established. After some time (for example, once regulatory approval has been granted), the placebo group is offered vaccination so that all willing volunteers receive the efficacious vaccine. To keep the blind, the original vaccine group receives placebo and vice versa. Thus, the trial has changed into a blinded randomized crossover trial of immediate (original vaccine) versus deferred (original placebo) vaccination, so that 2 distinct interventions remain that can be contrasted. Intuitively, if the case accrual rate for the original vaccine group is higher than that for the recently vaccinated group, efficacy must be waning. Critically, post-crossover vaccine efficacy can be estimated by assuming the newly vaccinated persons in period 1 or period 2 receive the same benefit.

Figure 1. Schematic of a standard trial of vaccine versus placebo that pivots to a trial of immediate versus deferred vaccination using blinded crossover.

At some point after a positive primary efficacy signal, placebo group participants receive the vaccine and the vaccine group participants receive placebo. A balanced case split between study groups in period 2 supports maintenance of the period 1 vaccine efficacy. A key assumption is that vaccine efficacy for the newly vaccinated is the same whether at the start of period 1 or at the start of period 2. The tapering and fading blue wedge after vaccination indicates a potential waning of efficacy.

Figure 2 shows this concept for a vaccine with 80% efficacy in period 1. We deduce the case count for an inferred period 2 placebo group by requiring that the inferred count aligns with the vaccine efficacy observed in period 1 for the original vaccine recipients. We then use this inferred case count to deduce the period 2 vaccine efficacy. The approach generalizes beyond 2 equal periods and equal randomization, but the key aspects are easiest to develop in this simpler setting.

Figure 2. Schematic of how deferred placebo vaccination allows imputation of the case counts for an inferred placebo group.

The vaccine efficacy in period 2 for the newly vaccinated (deferred vaccine group) is assumed to be the same 80% that was observed in the newly vaccinated (immediate vaccine group) in period 1. This logic implies that a counterfactual placebo group of 35 persons would have about 10 cases, because this satisfies 100%[1–(2/35)/(10/35)] = 80%. Thus the vaccine efficacy for the original vaccine group in period 2 has waned 55%, calculated as 100% [1–(5/39)/(10/35)]. The lighter blue shade of the immediate vaccine group participants in period 2 indicates waning immunity.

Table 1 provides more detail. During period 1, vaccine efficacy is estimated as VE₁ = 100% (1 – 25/125) = 80%, the reduction in observed cases as a result of the vaccine. After vaccination of the placebo group, we evaluate 2 scenarios. In scenario 1, the number of cases is similar in period 2 with a relative risk (original vaccine/original placebo) in period 2 of RR₂ = 41/39, which suggests that little vaccine efficacy has been lost. Using the period 1 vaccine efficacy, we infer the number of placebo cases that could have been seen had there been a standard trial, as 39 × (125/25) = 195. We then calculate vaccine efficacy in period 2 as VE₂ = 100%(1 – 41/195) = 100%[1 – (25/125) × (41/39)] = 100%(1 – RR ₁ × RR ₂) = 79%. We obtain the same estimate of 79% even if the period 2 case counts are halved or tripled, and thus this procedure does not require a constant placebo case accrual rate. For scenario 2, there is clear evidence of loss of durability, as RR₂ = 53/9 = 5.89. Indeed, the vaccine efficacy becomes negative, suggesting potential harm in period 2. Such a scenario is extreme but illustrates that one can assess whether a vaccine eventually turns harmful.

Table 1. Calculation of Placebo-Controlled Vaccine Efficacy by Inferring a Placebo Group After Placebo Group Vaccination.

Assumptions

To obtain unbiased assessments about durability of efficacy, a standard trial requires 1) adequate and nondifferential retention after the original randomization such that the 2 randomized cohorts remain balanced at the start of period 2; 2) that there are no unmeasured host or viral modifiers of vaccine efficacy that have a different distribution across the follow-up periods; and 3) consistency in case assessment across periods. The only additional assumption required of the deferred vaccination design is that the newly vaccinated obtain the same benefit of vaccination in period 1 as in period 2. Of note, the deferred vaccination design is valid without making assumptions about whether and how background incidence changes from period 1 to period 2.

Because participants are removed from analysis for dropout or after acquiring disease, the two groups might not be similar at the start of period 2. One example would be differential censoring due to death or dropout. With an effective vaccine, the more vulnerable placebo group participants (for example, those with riskier behavior or frailer health) will acquire COVID-19 during period 1 at a greater rate than the matching vulnerable population of vaccinated participants, who were protected by vaccination (24–27). So period 2 can start with fewer vulnerable people in the original placebo group compared with the original vaccine group. Thus, the vaccine efficacy may appear to wane, but only because of a biased comparison. For large COVID-19 vaccine trials in which perhaps 1% to 5% of the participants develop disease, the proportion of participants excluded from period 2 and the resulting bias are unlikely to be a substantial concern (28). Nonetheless, any potential bias can be ameliorated by statistically adjusting for differential risk using measured characteristics that predict risk for disease. In addition, true waning vaccine efficacy is supported if the kinetics of antibody waning in individuals track with individual risk over time.

A developing issue that could lead to differences between the 2 periods is the emergence of variants of concern (VOCs) of SARS-CoV-2 (29–35) against which established vaccines may have reduced efficacy (36). Underlying this concern is the observation that differential vaccine efficacy by pathogen feature (such as genotype or amino acid sequence feature) has been seen, at least in some populations, for vaccines against malaria (37), HIV (38), and dengue (39). Provided that the VOCs are present in period 1, the methods in this article can be applied separately to different strains including original and VOCs where, for example, disease is redefined as disease caused by a specific VOC. More sophisticated sieve analysis methods that were developed for strain-specific efficacy of HIV can also be applied (40). If a completely new VOC emerges in period 2, one could check for waning vaccine efficacy for the new VOC by comparing case counts of the new VOC in the immediate versus deferred vaccination groups. If these were similar, it would suggest the vaccine efficacy for the VOC had not changed. However, the actual efficacy for the new VOC would be unknown and could be large or close to zero.

Estimation and Comparative Performance

A modified Poisson method to estimate overall vaccine efficacy can be generalized to provide period-specific estimates of relative risk and vaccine efficacy (41, 42). This person-year approach allows for differential follow-up within each period due to rolling enrollment or dropout. More than 2 periods with different durations can be assessed, and with sophisticated survival analysis methods, a period-free curve of placebo-controlled vaccine efficacy throughout follow-up can be derived (Supplement Figure 1) (28, 43, 44).

Table 2 evaluates the performance of a standard trial compared with a deferred vaccination trial in terms of power or the probability of detecting waning efficacy and the power of detecting harm in period 2, using the R package plaXdesign (45) (Supplement). We also report sample size ratios; for example, a sample size ratio of 2 means a deferred vaccination trial would require twice as many participants to achieve the power of a given standard trial. The first 4 rows correspond to a scenario with high initial efficacy that accrues about 200 COVID-19 cases among placebo recipients during period 1 and accrues 200 or 100 COVID-19 cases among placebo recipients during period 2 in a standard trial. Under deferred vaccination these period 2 placebo cases are inferred. We have good power to detect waning efficacy when vaccine efficacy changes from 90% to 75% with 200 cases in period 2 under either the deferred vaccination (0.96 power) or standard design (0.92 power). Power is lessened with 100 cases in period 2, with deferred vaccination having 0.72 power and standard design having 0.81 power.

Table 2. Statistical Performance of the Deferred Vaccination Design Compared With the Standard Design*.

The bottom 4 rows focus on a scenario where a subgroup may experience late harm from the vaccine. The subgroup accrues about 25 placebo group cases in period 1—thus, about one eighth of the total of 200 placebo cases. As before, we anticipate around 25 or 12 placebo group cases in period 2 under the standard trial. We consider subgroup vaccine efficacies in period 2 of –100% and –300% or a doubling and quadrupling of the case rate on vaccine compared with a placebo group, respectively. These estimates are meant to roughly parallel the vaccine-enhanced risk for hospitalized dengue disease and for severe dengue disease seen with the CYD-TDV dengue vaccine in baseline-seronegative participants (46). Power to detect waning efficacy is greater than 0.80 for all scenarios. The power to detect harm is substantially lower under deferred vaccination compared with the standard design, with poor power to detect harm with a period 2 vaccine efficacy of –100%, but powers of 0.71 and 0.84 to detect harm with a VE of –300% with 12 and 25 expected cases in period 2, respectively.

The approach can be generalized beyond 2 periods. For example, the vaccine efficacy estimate for 3 periods depends on all previous periods via the compounding formula 100% (1 – RR₁ × RR₂ × RR₃), where RR_k is the relative risk for the original study groups in period k. The compounding suggests a chain that is as strong as its weakest link. That is, if any period has an unreliable estimate of the relative risk, so will later periods. Conversely, if all periods have reliable estimates, later vaccine efficacy estimates will be reliable. This underscores the statistical benefit of maintaining the original placebo-controlled design for as long as possible to maximize the reliability of the first and thus subsequent estimates of vaccine efficacy. This behavior is illustrated in Supplement Figure 1.

Collateral Benefits of Placebo Crossover

Correlates of risk and correlates of protection analyses involve describing associations between the measured immune response shortly after the last dose of vaccine with subsequent risk for COVID-19 or with vaccine efficacy against COVID-19 (47–49). For example, neutralizing or binding antibodies have been identified as correlates of protection for many licensed vaccines (50, 51). If higher vaccine-induced levels of binding antibody to the spike protein or of SARS-CoV-2 neutralizing antibody responses are found to track with lower risk for disease (as might reasonably be hypothesized from, for example, retrospective analyses [52] and observational cohort studies [53] of natural infection), future vaccines that achieve high binding antibody or neutralizing antibody levels might be licensed on the basis of small-scale immunogenicity studies alone, thus precluding the need for large-scale efficacy trials. In fact, the U.S. Food and Drug Administration recently issued new guidance (54) that such an approach may be taken to support licensure of modified COVID-19 vaccines that have been tailored for enhanced efficacy against newly emerging VOCs. A proven or “reasonably likely” immune correlate of protection can also help inform assessments of durability and allow for immunobridging to populations not included in the trials either through traditional or provisional approval, such as through the U.S. Food and Drug Administration's accelerated approval mechanism (55, 56). After a strong positive vaccine efficacy result (for example, more than 80%), an immune correlates analysis (23) may be underpowered owing to relatively few breakthrough cases among the vaccine recipients. However, vaccination of the placebo group could effectively double the sample size for assessing immune correlates of risk and protection, a notable benefit of deferred vaccination.

If vaccine efficacy wanes substantially over the course of follow-up, a natural question will be whether revaccination can reverse the loss. Continued follow-up of the study participants provides a ready-to-go vehicle to experimentally evaluate revaccination by randomly assigning those who received vaccine to receive another course of vaccine or placebo. Although a boost trial can be quickly conducted in a standard trial that maintains follow-up, deferred vaccination allows for a doubling of those who are available for the boost trial relative to a trial that maintains a placebo control. A boost trial after deferred vaccination could proceed in 2 stages: The original vaccine group would be randomized first and the original placebo group randomized later, if needed (Supplement Figure 2). If the vaccine efficacy wanes from 80% to 40% and it is hoped that boosting will recover the vaccine efficacy to 80%, about 42 cases are required to achieve 90% power, substantially fewer than the roughly 150 cases required for a typical COVID-19 vaccine trial (3). The above discussion also applies to a heterologous boost or vaccination by a vaccine from the same platform but with an immunogen based on a newly emerged VOC. Another possibility is for the original vaccine group to be randomly assigned to receive a boost with the original vaccine versus a heterologous boost. The relative efficacy of the different boosting strategies could be directly compared. Furthermore, an inferred placebo-controlled vaccine efficacy for each boost could be derived by using the approach that we describe in this article, provided that the VOC was present in both periods.

Implementation

Once the vaccine is available, principles of ethics (including an evaluation of the societal benefit and individual risk profile of the trial), transparency, and fairness should govern the timing and implementation of the deferred vaccination (22). Ideally, the deferred vaccination would be part of the original design and part of the initial informed consent. Otherwise, a protocol amendment could be introduced and reconsent obtained, though this could be challenging to design and execute amidst the considerable activity after a positive efficacy signal.

Deferred vaccination should be initiated at a second immunization visit where blinded participants arrive and are immunized. The immunizations could be open-label (where participants are unblinded and only the placebo group is immunized) or blinded (where the participants who received vaccine now receive placebo and vice versa). This blinded crossover has several scientific advantages. Just as the pre-crossover period was blinded to address initial vaccine efficacy, so should the post-crossover period be blinded to address durability. Both questions are critical and deserve the same rigor. With open-label immunization, there is concern that the newly unblinded original vaccine recipients will forgo masks and physical distancing while the newly unblinded placebo recipients who are immunized will not. Another concern is that mild subjective symptoms might be differentially dismissed or elevated in the unblinded study groups. This is particularly important because efficacy is entirely predicated on volitional presentation with signs or symptoms of COVID-19. Such differential behavior could confound the assessment of waning vaccine efficacy.

With minimal assessment of safety and reactogenicity in newly vaccinated persons, operationally maintaining the blind could essentially only require the addition of dummy shots in the original vaccine recipients with some additional blood draws. The ongoing Novavax phase 3 SARS-CoV-2 vaccine trial, which started enrollment in late December 2020, has adopted a blinded crossover design (57).

Nonetheless, open label follow-up is logistically easier, which is especially important if deferred vaccination was not part of the original design. In addition, differential reactogenicity might allow study participants to guess the vaccine group to which they were assigned, thus undermining the benefits of maintaining the blind. Open-label follow-up provides important information, though at risk for some bias. To best analyze such trials, baseline variables predictive of risk and simple questions about actual risk behavior can be used to statistically ameliorate bias (28, 43). The Pfizer and Moderna phase 3 trials, for which deferred vaccination was considered around the time an efficacy signal was reached, are open-label.

An important goal for COVID-19 vaccine trials is to assess the effect of vaccine on asymptomatic infection, which can be assessed by periodic serologies and nasal or nasopharyngeal sampling. To maintain this goal, serologic and upper respiratory samples should be collected in all study participants while still blinded at the second immunization visit, with both study groups having the same schedules post crossover to ensure even-handed evaluation of this end point. Although more complex, our methods can be generalized to study vaccine efficacy against infrequently assessed end points, such as seroconversion.

Different approaches to the second immunization visit could be implemented, with a key requirement that both study groups are treated the same. If the vaccine is initially recommended for a subgroup, such as higher-risk individuals, their second immunization visit could be scheduled, with the blinded placebo-controlled trial continuing for the lower-risk subgroup. Second immunization visits could later occur for the lower-risk subgroup. Operationally, crossover could follow the order of the original enrollment but occur at an accelerated pace compared with enrollment, so that different cohorts would cross over on different days relative to the initial vaccination. Another possibility is to randomize the time of crossover. To reduce trial burden, late follow-up might be lessened provided that critical end points are still assessed as in the pre-crossover period.

In conclusion, the high efficacies reported for multiple vaccine candidates (3–8), although universally welcomed, add complexity and uncertainty to the environment surrounding access to the vaccine for trial participants who were assigned to receive placebo. Continued blinded follow-up in the original study groups is optimal to assess vaccine efficacy over time and is endorsed by the U.S. Food and Drug Administration in their guidance pertaining to COVID-19 vaccine development. Deferred vaccination allows placebo recipients timely access to the vaccine when it would no longer be proper to maintain participants on placebo. We demonstrate that critical information regarding durability of the vaccine effect can be obtained even after the placebo group participants receive the vaccine and argue that studies should maintain rigorous blinded follow-up after deferred vaccination via a blinded crossover design. Additional benefits of continued follow-up include a doubling of the number of participants who can contribute to an immune correlate analysis and the possibility of a quick pivot to a trial of boosting should the vaccine demonstrate waning efficacy.