Clinical Outcomes of a COVID-19 Vaccine: Implementation over Efficacy

Abstract

The global effort to develop a COVID-19 vaccine is likely to soon produce one or more authorized vaccines. We examine how different definitions and thresholds of vaccine efficacy, coupled with different levels of implementation effectiveness and background epidemic severity, translate into outcomes including cumulative infections, hospitalizations, and deaths. Using a mathematical simulation of vaccination in an at-risk population, we find that factors related to implementation will contribute more to the success of vaccination programs than the efficacy of a vaccine as determined in clinical trials. The benefits of a vaccine will decline substantially in the event of manufacturing or deployment delays, vaccine hesitancy, or greater epidemic severity. Our findings highlight the need for health officials to move swiftly to scale up production and distribution infrastructure, to promote widespread public confidence in COVID-19 vaccines, and to encourage continued adherence to other mitigation strategies, even after a vaccine becomes available.

INTRODUCTION

The public health, economic, and societal effects of COVID-19 in the United States and worldwide have been staggering. Non-pharmaceutical interventions such as physical distancing, mask-wearing, and limits on large gatherings have shown clear benefits in reducing the spread of SARS-CoV-2. However, widely variable adherence with those practices, inconsistent or unclear advice from government public health officials, and the politicization of many aspects of the pandemic response have limited their effectiveness in the United States and elsewhere. From the earliest stages of the pandemic, the development of safe and effective COVID-19 vaccines has therefore been widely considered an essential component of any durable strategy to control the virus, the disease, and its effects.

Since the publication of the SARS-CoV-2 viral sequence on January 10, 2020, an unprecedented global collaboration among governments, vaccine manufacturers, and academic researchers has been mounted to develop COVID-19 vaccines.¹ In the United States, this work has been supported through billions of dollars in public investment and new entities such as Operation Warp Speed and the Accelerating COVID-19 Therapeutics and Vaccines (ACTIV) public-private partnership.² Global coordination of vaccine research and development work ongoing in many nations has been provided by the Coalition for Epidemic Preparedness and Innovation (CEPI), Gavi, and the World Health Organization.³ According to CEPI, 321 COVID-19 vaccine candidates were in development worldwide as of September 2020.⁴ Of those, as of October 2020, over 40 have progressed to clinical testing in humans, 11 of which were in Phase 3 clinical trials — the large-scale population-based testing capable of producing the safety and efficacy evidence required for regulatory approval.⁵ As of mid-October 2020, four Phase 3 COVID-19 vaccine clinical trials are underway in the United States, with preliminary results likely to be made available in the coming months and more complete results thereafter.⁶

Vaccine efficacy is a particularly critical outcome that will be measured in these trials and subsequently evaluated by regulatory bodies such as the U.S. Food and Drug Administration (FDA) and its international counterparts. In a June 2020 guidance document to vaccine manufacturers, the FDA adopted a broad definition of vaccine efficacy, one that encompasses both transmission effects (i.e., the ability of the vaccine to prevent the spread of SARS-CoV-2 from an infected person to a susceptible person) and disease-modifying effects (i.e., the ability of the vaccine – among those vaccinated but who nonetheless become infected – to slow or prevent progression of illness, to speed recovery, to decrease utilization of critical-care resources, and/or to reduce mortality).⁷ While the guidance gave trial sponsors discretion to tailor endpoints to their specific populations and settings, it recommended both a transmission endpoint…

“Either laboratory-confirmed COVID-19 or laboratory-confirmed SARS-CoV-2 infection is an acceptable primary endpoint.…FDA recommends that either the primary endpoint or a secondary endpoint…be defined as virologically confirmed SARS-CoV-2 infection with one or more [COVID-19-defining symptoms]…”⁷

…and a disease-modification endpoint:

“As it is possible that a COVID-19 vaccine might be much more effective in preventing severe versus mild COVID-19, sponsors should consider powering efficacy trials for formal hypothesis testing on a severe COVID-19 endpoint. Regardless, severe COVID-19 should be evaluated as a secondary endpoint (with or without formal hypothesis testing) if not evaluated as a primary endpoint.^”7

The FDA also established a minimum efficacy threshold:

“To ensure that a widely deployed COVID-19 vaccine is effective, the primary efficacy endpoint point estimate for a placebo-controlled efficacy trial should be at least 50%…”⁷

These definitions and thresholds are highly consequential, yet the FDA guidance document provides no justifications for either. The choice of the 50% efficacy threshold most closely resembles the typical effectiveness of vaccines against influenza, a less transmissible, morbid, and lethal disease than COVID-19.^8,9 It is also a considerably lower efficacy standard than those of virtually all other approved and widely used vaccines.¹⁰ But in the context of a global pandemic with ruinous economic and public health consequences, the FDA’s 50% threshold raises the question: Might we settle for a vaccine with more modest effects and, if so, how modest? Would a vaccine that has a limited impact on transmission but significantly slows disease progression be acceptable? How might we compare such a vaccine to one that greatly lowers susceptibility to infection but has no impact on mortality?

A further complication is that biological efficacy is only one of many factors that will contribute to the real-world effectiveness of COVID-19 vaccination. How well a vaccine program “works” also depends on how quickly the vaccine can be manufactured, how efficiently it can be targeted and distributed to the settings in greatest need, how persuasive health messaging can be in promoting public acceptance, and how reliably the public can continue to adhere to the many complementary preventive strategies (e.g., masks, hand-washing, distancing) that may still be necessary to limit the spread and impact of the virus.

We sought to understand the interplay between these competing considerations. Specifically, we asked how vaccine-related changes in susceptibility to infection, progression of disease, and severity of illness might translate into population outcomes of interest, such as cumulative infections, hospitalizations, deaths, and intensive care unit (ICU) days averted. We explored how those downstream outcomes might vary in the face of alternative operational assumptions (e.g., the pace of scale-up and the degree of public acceptance) and changes in the epidemiological context at the time of vaccine introduction.

METHODS

Study Design

We used a simple mathematical model to estimate the population benefits of a vaccine against COVID-19. We considered vaccines with varying degrees of preventive benefit (transmission effect) and disease-modifying benefit (progression and mortality effect). We considered a range of different assumptions regarding the pace of manufacturing/distribution and public acceptance (coverage), two implementation parameters that are independent of vaccine clinical trial results. We also considered a range of different background epidemic severities, as measured by the reproduction number (R_t). Outcomes of interest – including total infections, deaths, and peak hospitalization/ICU volume – were reported on both an absolute basis and as a percent reduction from a “No Vaccination” scenario, using a 6-month planning horizon, a population size of 100,000, and an initial 100 (0.1%) exposed and 9,000 (9%) recovered cases.¹¹ The model was implemented as a spreadsheet and parameterized and validated using population-average data inputs (Table 1).

Table 1:

Input data and assumptions^*

		Base Value	Reference
Average time spent in state (days)
	Exposed	6	15
	Asymptomatic **	3	14, 15
	Mild-Moderate	6	15
	Severe	4	15, 18
	Critical	14	15, 18
Likelihood of state-to-state progression (%)
	Exposed to Asymptomatic	100	Assumption
	Asymptomatic to Mild/Moderate	60	15
	Mild-Moderate to Severe	30	15-18
	Severe to Critical	6	15-18
Mortality (likelihood of death from state, %)
	Asymptomatic	0	Assumption
	Mild-Moderate	1	19,20
	Severe	16	19-21
	Critical	25	19-22
Initial population***
	Uninfected	90,900	Assumption
	Exposed	100	Assumption
	Recovered	9,000	11
Population epidemiology
	Effective reproduction number, Rt	Base value 1.8; Range 1.5 – 2.1	Assumption
Vaccine assumptions****
	Efficacy lag (days)	Base value 30; Range 14 – 42	Assumption
	Pace (% of population that Can be vaccinated per day)	Base value 0.5; Range 0.1 – 2.0	12
	Coverage (maximum % ultimately vaccinated)	Base value 50; Range 25 – 75	13

^*Values presented in this table served as raw input to derive the rates and probabilities used by the model. See appendix for details on all derivations and calculations.

^**Among persons who progress to Mild/Moderate illness.

^***We assumed N = 100,000 and no persons in any vaccinated state or state of infection at time = 0.

^****Efficacy lag is defined as the delay between the time of vaccine administration and the time when effectiveness is first observed. Pace is defined as the fraction of the population that can be vaccinated on a given day. Coverage is defined as the maximum fraction of the population ultimately vaccinated.

The Institutional Review Boards of both the Massachusetts General Hospital (protocol number 2020P000967) and the Yale School of Medicine (protocol ID 2000029169) determined that this research did not require their review or approval because it used only aggregate, published data, and did not involve human participants.

Compartmental Model

The SEIR (Susceptible-Exposed-Infectious-Recovered) model is one of the simplest mathematical frameworks for portraying the trajectory of an infectious disease through an at-risk population. Briefly stated, the SEIR framework treats the process of viral transmission and disease progression as a sequence of transitions among a finite number of health states (or “compartments”) Transitions are governed by mathematical equations that capture both the transmission dynamics of the virus and what is known about the natural history of disease.

We adapted the classic SEIR framework in two important ways (Appendix Figure 1). First, we divided the “Infected” compartment into four distinct sub-compartments, to capture the increasing severity and resource use associated with more advanced COVID-19 disease: “Asymptomatic,” “Mild” (outpatient), “Severe” (hospitalized), and “Critical” (hospitalized in an ICU). Second, we introduced the possibility of vaccination by creating a parallel set of compartments to the ones described above. Individuals receiving vaccine moved from the “Susceptible Unvaccinated” state to the “Susceptible Vaccinated” state. From there, their progress to Exposure, Infection, Recovery, and Death was adjusted to reflect the transmission and disease-modifying benefits of the vaccine. This modeling device also permitted us to adjust the infectiousness of persons who received an imperfect vaccine but who nevertheless became infected (i.e., “breakthrough infections”).

Vaccine Efficacy

To capture the broad definition of “vaccine efficacy” in the FDA’s June 2020 guidance, we considered three different vaccine types (Table 2): First, a preventive vaccine that decreases susceptibility to infection in uninfected persons. Second, a disease-modifying vaccine that has no effect on susceptibility to infection but that improves the course of disease in infected persons, slowing progression, speeding recovery, reducing mortality, and decreasing infectiousness. Finally, a composite vaccine that combines the attributes of both the preventive and disease-modifying vaccines. We set the efficacy for each of these attributes at 50% in the base case and examined ranges of 25% to 75% in sensitivity analysis. (For the recovery rate increase, the base case value was 100% with a range of 75-150%.) To capture uncertainty about the lag time between when a vaccine is administered and when its effects take hold, we also considered a range of delays: 30 days in the base case (representing a two-dose vaccine with administration 30 days apart and partial efficacy after the first dose); 14 days (representing a fast-acting, single-dose vaccine); and 42 days (representing a two-dose vaccine with no efficacy after the first dose.)

Table 2:

Performance profiles of three illustrative vaccines

	Vaccine type
	Preventive	Disease- modifying	Composite
Decrease in susceptibility to infection (base value 50%; range 25% - 75%)	x		x
Decrease in infectiousness (base value 50%; range 25% - 75%)		x	x
Reduction in all disease progression rates (base value 50%; range 25% - 75%)		x	x
Reduction in all mortality rates (base value 50%; range 25% - 75%)		x	x
Increase in all recovery rates (base value 100%; range 75% - 150%)		x	x

Implementation Effectiveness

Recognizing that the challenges of vaccine development and vaccination programs do not end once an effective agent is identified, we also designed the model to respond to two implementation measures: pace and coverage. We defined pace as the percent of the population that could be vaccinated on a given day, a measure of manufacturing and logistical preparedness. As a point of departure, we assumed a base case estimate of 0.5% for the pace parameter, a value that roughly approximates the daily rate of influenza vaccination in the US during the peak period of vaccination efforts each fall.¹² We chose this value to reflect our assumptions that while a COVID-19 vaccine may need to be administered in two doses, the urgency of the pandemic may prompt sponsors to bring production and distribution to scale at twice the rate of the influenza vaccine. Given the uncertainty surrounding these assumptions, we considered a very broad range of alternative values for the pace parameter, ranging from 0.1% to 2%, in sensitivity analysis. We defined coverage as the percent of the population ultimately vaccinated, a measure of public acceptance and the success of public health efforts to make vaccines available to all who desire them. We used a base case value of 50% (range 25% to 75%) reflecting recent US polling data on vaccine acceptability.¹³

Epidemiology and Natural History

We defined three epidemic severity scenarios: a base case with a reproduction number (R_t) of 1.8; a best case (R_t = 1.5) representing strict adherence to social distancing and other preventive best practices; and a worst case (R_t = 2.1) reflecting the higher risks associated with winter weather and greater indoor activity. Assumptions regarding the development and natural history of COVID-19 (including incubation times, likelihood of symptoms, rates of progression, recovery, and fatality rates) were informed both by planning scenarios recommended by the Centers for Disease Control and Prevention and by the literature.^14-22

A comprehensive description of the model, its parameters, governing equations, and input data values is provided in the Appendix.

RESULTS

Base Case

In a population of 100,000 and at a baseline R_t of 1.8, the model projects 54,701 infections and 3,736 cumulative deaths over the course of 6 months without a vaccine. Introducing preventive, disease-modifying and composite vaccines would result in 37,903, 35,367 and 28,172 cumulative infections and 2,587, 1,803 and 1,642 cumulative deaths, respectively (Table 3). Across all values of R_t, a 50%-effective disease-modifying vaccine would have a greater impact on mortality and peak hospitalizations than a 50%-effective preventive vaccine. The impact of both vaccines on total infections would be similar in a high-severity epidemic (R_t = 2.1); but the disease-modifying vaccine would have a more pronounced impact on total infections in a lower-severity epidemic (R_t = 1.8 and 1.5). While the 50%-effective composite vaccine would outperform both the preventive and disease-modifying vaccines, its impact would be much less than the sum of the impacts of the other two vaccine types combined.

Table 3:

Output table for 3 vaccine types – with infections and deaths and 3 different R_t

	No Vaccine	Preventive			Disease-modifying			Composite
		Total	Unvaccinated	Vaccinated	Total	Unvaccinated	Vaccinated	Total	Unvaccinated	Vaccinated
Rt = 1.5
Total vaccinations	0	40,900			40,900			40,900
Total infections	34,940	14,048	11,176	2,872	11,424	8,208	3,216	8,365	7,115	1,250
Cumulative deaths	2,153	915	740	175	573	560	13	496	491	5
Peak hospitalizations	735	256			171			154
Rt = 1.8
Total vaccinations	0	40,561			40,900			40,900
Total infections	54,701	37,903	30,416	7,487	35,367	25,579	9,788	28,172	23,538	4,634
Cumulative deaths	3,736	2,587	2,087	500	1,803	1,760	43	1,642	1,622	20
Peak hospitalizations	1,604	1,005			726			662
Rt = 2.1
Total vaccinations	0	32,195			33,692			34,282
Total infections	63,896	52,904	44,263	8,641	52,939	40,876	12,063	46,253	39,572	6,681
Cumulative deaths	4,410	3,652	3,058	594	2,879	2,825	53	2,765	2,735	30
Peak hospitalizations	2,441	1,914			1,562			1,489

Sensitivity to Vaccine Efficacy

To understand the impact of potential outcomes of ongoing clinical trials, we considered vaccine efficacy variables set to 25%, 50%, and 75% while holding all program implementation parameters constant (Figure 1). We considered two implementation scenarios: first, a base case (left panel) with pace = 0.5% and coverage = 50%; and then a more aggressive implementation (right panel) with pace = 1% and coverage = 90%.

Figure 1. Total Infections: Sensitivity to Vaccine Type.

Bar graph representing the fraction of infections averted (vertical axis) under alternative vaccine types -- preventive, disease-modifying, and composite -- and at different background epidemic severities, represented by increasing R_ts. The different shades of each bar represent vaccine efficacies of 25% (darkest), 50% (middle) and 75% (lightest). Blue bars on the left represent an implementation scenario where the vaccination is scaled up in the population at 0.5% per day (pace) with a maximum of 50% coverage. Green bars on the right represent an implementation scenario where the vaccination pace is 1% per day with 90% coverage.

Greater vaccine efficacy always produced more favorable outcomes. In the case of preventive vaccines, the returns to increased efficacy were close to constant. For example, under base case implementation assumptions (Figure 1, left panel) and R_t = 1.8, the incremental contribution to infections and deaths averted from a preventive vaccine with efficacy 25%/50%/75% were 14%/17%/17% and 14%/17%/16%, respectively (see Appendix Figure 4 for results on deaths averted). By contrast, there were markedly diminishing marginal returns to increased efficacy using disease-modifying and composite vaccines; these vaccines attained much of their full potential effect on outcomes at efficacy level 25%. For example, under the aggressive implementation scenario (Figure 1, right panel) with R_t = 1.8, the incremental infections averted from a disease-modifying vaccine with efficacy 25%/50%/75% were 41%/22%/8%; incremental contribution to deaths averted were 62%/15%/4% (Appendix Figure 4).

Figure 1 also illustrates that potential benefits of even the most optimistically effective vaccine are greatly diminished if it is introduced into a more severe epidemic. For all three vaccine types, a 75%-effective vaccine implemented in a population where R_t = 2.1 averted fewer infections and deaths than a 25%-effective vaccine implemented under less severe pandemic conditions (R_t = 1.5). While the findings presented here all pertain to vaccines with an efficacy delay of 30 days, they persisted for efficacy delays ranging from 14 to 42 days (see Appendix Figures 2A and 2B).

Sensitivity to Implementation Effectiveness

To understand how imperfect implementation might affect the success of a vaccination program, we held all vaccine efficacy parameters constant at their base values and simultaneously varied the two program uptake parameters: pace and coverage (Figure 2). With R_t = 1.8, a disease-modifying vaccine (Figure 2, central panel) that attained even 90% coverage only averted 7% of infections at a pace of 0.1%; that same vaccine only averted 11% of infections at coverage 10%, even when it attained a pace of 2.0%. Bringing both coverage and pace up to their base case levels (50% and 0.5%) averted 35% of infections. (See black highlighted cells in the figure.) The pattern observed in the central panel of Figure 2 is one that persisted across all vaccine types, all values of R_t, all lag-time assumptions, and all outcome measures: sufficient pace and coverage are complements, not substitutes, and both are necessary elements of a successful vaccination intervention; success on one of these two implementation measures cannot fully make up for failure on the other.

Figure 2. Total Infections: Sensitivity to Implementation Factors.

Nine heat maps, each depicting an individual vaccine type (columns) at a given background epidemic severity (R_t, rows). Base case efficacies (50%) are used for the preventive, disease-modifying, and composite vaccines. Individual maps demonstrate the range of vaccination coverage (horizontal axis, 10-90%) and pace of scale up (vertical axis, 0.1%-2% per day). The color spectrum represents the proportion of infections averted: green averts the greatest number of infections; red the least.

Figure 2 again highlights that the impact of a vaccine dissipates dramatically as the severity of the epidemic (i.e., R_t) increases. For example, a preventive vaccine with 50% coverage and 1.0% pace averted 76%/48%/31% of infections when R_t = 1.5/1.8/2.1. (See white highlighted cells in the three left panels of the figure.) All other things being held equal, the proportional power of any vaccine to reduce infections, deaths, and peak hospitalization was greatest at lower values of R_t.

Figure 2 also illustrates the weak superiority of the composite vaccine (right-hand panels). While it achieved the greatest reduction in infections for any combination of pace and coverage, its impact was much less than the sum of the infections averted by the preventive and disease-modifying vaccines.

While shorter efficacy lag times invariably resulted in more favorable vaccine outcomes, the qualitative findings highlighted here for the 30-day efficacy lag were similar to those for vaccines with efficacy lags of 14 and 42 days (Appendix Figures 3A and 3B).

DISCUSSION

Our results demonstrate that the benefits of any COVID-19 vaccine — whether highly, moderately, or modestly efficacious by any trial-defined outcome — will depend at least as much on how swiftly and broadly it is implemented and the epidemiological environment into which it is introduced as it will on the vaccine’s physiological properties as shown through clinical trials. While these latter vaccine-specific characteristics are fixed, the medical, public health, and government communities can productively intervene with respect to the contextual considerations that would increase the benefits of a vaccine upon its introduction.

First, the effects of any COVID-19 vaccine are highly dependent on the effective reproductive number of the virus (R_t) at the time a vaccine is deployed. When R_t is comparatively low (1.5) — indicating that viral circulation is being controlled through other non-pharmaceutical measures — vaccines with low efficacy (25%) are capable of producing larger reductions in the fraction of infections and deaths than vaccines with much higher efficacy (75%) introduced at times when R_t is significantly higher (2.1). Furthermore, the additional benefit of a vaccine with 25% vs. 75% efficacy very much depends on the background R_t; in cases outbreak control (R_t ≤1.5), a vaccine with 25% efficacy might well have a substantive impact. Managing and reducing R_t requires a sustained commitment to the suite of public health tools known to reduce the spread of COVID-19, including the use of face coverings, limiting large gatherings, robust testing and contact tracing, and related measures. Attention to and investment in these activities remains imperative not simply until the arrival of a vaccine but throughout the likely prolonged period during which a vaccine is being deployed.

Second, our results show that the effectiveness of a COVID-19 vaccine will be shaped significantly by the success or failure of efforts to quickly and widely deliver it to the public. The pace of vaccination — how quickly the vaccine is introduced — will be determined by a combination of manufacturing capacity, the development of distribution systems and infrastructure, the creation of mass vaccination clinics in diverse settings and locations, and related logistical considerations. The vaccine benefit also depends on how many doses are required to reach its efficacy. A two-dose vaccine that takes 42 days — where maximum efficacy may be in winter months with a higher R_t — should be expected to have diminished impact when compared to a one-dose vaccine with only a 14-day delay to efficacy.

Vaccination coverage — the percentage of the population that ultimately receives a vaccine — is dependent on efforts that foster widespread public enthusiasm for vaccination and address sources of hesitancy for vaccines in general and COVID-19 vaccines, in particular.^23,24 It also requires efforts to ensure that vaccines are accessible to all communities, particularly underserved groups for which longstanding disparities in vaccination coverage have been observed. Included in this group are racial and ethnic minority groups among whom the effects of COVID-19 have been disproportionately felt. Delivering the vaccine to as many people as possible as quickly as possible can result in large reductions in infections and death, even at higher R_t. Conversely, a slow pace of vaccination or low vaccination coverage dramatically reduces the benefits of vaccines even with moderate or high efficacy.

Our results also suggest that the significant public optimism regarding the potential value of vaccines in reducing the burden of disease associated with COVID-19 is warranted, even if vaccines in development are shown to be only moderately efficacious. As would be expected, vaccine-associated benefits increase with greater levels of efficacy against infection, infectiousness, disease progression, and/or mortality. But even vaccines at or below the 50% efficacy threshold established in the June 2020 FDA guidance document could still make valuable contributions to COVID-19 prevention and response.

Our results should be placed in the context of several limitations. We assume a model of homogenous mixing and have not stratified vaccine deployment or coverage scenarios across different at-risk and vulnerable populations as suggested recently by the National Academies Framework for Equitable Allocation.²⁵ To some extent, sensitivity analyses on R_t might serve as surrogates for alternative communities of variable epidemic control. Next, published data used to populate the model are necessarily taken from early in the epidemic course. Over time, as clinical care and outcomes improve and as the pandemic reaches different demographic distributions,²⁶ our analysis likely merits adjustment using those forthcoming data. Finally, waning immunity after disease and vaccination remains an ongoing concern.²⁷ We intentionally examine a 6-month horizon and would caution extrapolation of our results beyond that time.

The efficacy of the COVID-19 vaccines currently being studied in phase 3 trials and soon to be reviewed by the FDA, while important, will be only one contributor to the overall effectiveness of the vaccination programs of which they may eventually be part. The ultimate success of COVID-19 vaccination efforts will depend on embracing a wide range of vaccine efficacy profiles at the time of authorization or approval, managing expectations regarding how a vaccine will contribute to public health responses in tandem with the continued use of non-pharmaceutical interventions, and investing substantially in efforts to rapidly deliver vaccines to as large a portion of the population as possible. Such a strategy would maximize the individual and population benefits of any authorized or approved COVID-19 vaccines and increase the likelihood that they approach the very high expectations placed upon them.