What defines an efficacious COVID-19 vaccine? A review of the challenges assessing the clinical efficacy of vaccines against SARS-CoV-2

Abstract

The novel coronavirus, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), has caused more than 1 million deaths in the first 6 months of the pandemic and huge economic and social upheaval internationally. An efficacious vaccine is essential to prevent further morbidity and mortality. Although some countries might deploy COVID-19 vaccines on the strength of safety and immunogenicity data alone, the goal of vaccine development is to gain direct evidence of vaccine efficacy in protecting humans against SARS-CoV-2 infection and COVID-19 so that manufacture of efficacious vaccines can be selectively upscaled. A candidate vaccine against SARS-CoV-2 might act against infection, disease, or transmission, and a vaccine capable of reducing any of these elements could contribute to disease control. However, the most important efficacy endpoint, protection against severe disease and death, is difficult to assess in phase 3 clinical trials. In this Review, we explore the challenges in assessing the efficacy of candidate SARS-CoV-2 vaccines, discuss the caveats needed to interpret reported efficacy endpoints, and provide insight into answering the seemingly simple question, “Does this COVID-19 vaccine work?”

Introduction

The novel coronavirus, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), has caused more than 1 million deaths in the first 6 months of the pandemic¹ and huge economic and social upheaval internationally.² An efficacious vaccine is considered essential to prevent further morbidity and mortality.³ To date, 44 candidate COVID-19 vaccines are in clinical development and 151 are in preclinical development, by use of a range of vaccine platforms.⁴ In this unprecedented pandemic, vaccine development is time-dependent, and considerable collaborative efforts are being expended to expedite preclinical and clinical assessment of candidate vaccines.⁵ The cost to manufacture and internationally deploy an efficacious COVID-19 vaccine will be vast, and the process will be at risk of politicisation.³ Although some countries might deploy COVID-19 vaccines on the strength of safety and immunogenicity data alone, the goal of vaccine development is to gain direct evidence of vaccine efficacy in protecting humans against SARS-CoV-2 infection and COVID-19.⁶

In their target product profile for COVID-19 vaccines, WHO suggested that a “clear demonstration of efficacy (on a population basis) ideally with ∼50% point estimate” should be a minimum criterion for any acceptable COVID-19 vaccine, and that efficacy can be assessed against “disease, severe disease, and/or shedding/transmission” endpoints.⁷ This definition is necessarily non-specific and reflects the complexities of assessing the clinical efficacy of candidate vaccines in the context of a novel pathogen. Indeed, a COVID-19 vaccine capable of reducing any of these elements might contribute to disease control where there are no efficacious prophylactic medications and few treatments.⁸ The US Food and Drug Administration (FDA) suggested that laboratory-confirmed COVID-19 or SARS-CoV-2 infection are appropriate primary endpoints for vaccine efficacy studies, with an endpoint estimate of at least 50% for placebo-controlled efficacy trials.⁶ However, protection against severe disease and death is difficult to assess in phase 3 clinical trials due to the unfeasibly large numbers of participants required. Instead, data to address this endpoint might be available only from large phase 4 trials or epidemiological studies done after widespread deployment of a vaccine. In this Review, we explore the challenges in assessing the efficacy of candidate SARS-CoV-2 vaccines and discuss the caveats needed to interpret reported efficacy endpoints.

Defining vaccine efficacy

Many different endpoints are used in vaccine research to define efficacy depending on the pathogen, consequences of infection, and transmission dynamics. Often, outcome data from randomised controlled trials (RCTs) are presented as a proportional reduction in disease between participants who were vaccinated and control participants to calculate the reduction that is attributable to the vaccine.⁹ Outcomes might include reduction in infection (ie, assessing sterilising immunity), severity of resultant clinical disease (ie, assessing disease-modifying immunity),⁹ or duration of infectivity.¹⁰ Such RCTs represent best-case scenarios of vaccine efficacy under idealised conditions in particular populations and provide key data necessary for vaccine licensure. However, vaccine efficacy does not always predict vaccine effectiveness—ie, the protection attributable to a vaccine administered non-randomly under field conditions.¹¹ For example, the effectiveness of rotavirus vaccines in children in low-income and middle-income settings was lower than the efficacy observed in children in high-income countries.¹² RCTs might not predict protection gained indirectly from herd protection (sometimes called herd immunity) following widespread vaccine deployment. Equally, RCTs done in a particular age group or geographical setting might not predict effectiveness if the vaccine is more widely deployed. It is possible that alternative vaccine platforms or the addition of adjuvants are required for adequate immunogenicity in older age groups, as for influenza vaccines.¹³ For this reason, prospective studies of vaccine effectiveness in real-world scenarios post licensure are routinely needed.

In the case of SARS-CoV-2, an efficacious vaccine might prevent infection, disease, or transmission (figure 1 ). The outcome of SARS-CoV-2 infection in individuals is heterogeneous and dependent on multiple variables, including age, sex, ethnicity, and comorbidities.¹⁴ On an individual level, the consequence of infection can range from paucisymptomatic states to hospital admission, requirement for respiratory support, and death.¹⁴ Transmission dynamics of SARS-CoV-2 are not yet fully understood but the ability of infected individuals to transmit infection when asymptomatic or in a presymptomatic period means that infection control strategies that focus solely on preventing transmission from symptomatic individuals will be insufficient alone to interrupt the transmission of SARS-CoV-2.¹⁵

Figure 1.

Potential endpoints of an efficacious COVID-19 vaccine

An efficacious COVID-19 vaccine could reduce the likelihood of infection of an individual, severity of disease in an individual, or degree of transmission within a population.

The effect of an efficacious vaccine on the course of the SARS-CoV-2 pandemic is complex and there are many potential scenarios after deployment. The ability of a vaccine to protect against severe disease and mortality is the most important efficacy endpoint, as hospital and critical-care admissions place the greatest burden on health-care systems. However, the beneficial effects of such a vaccine on a population can be observed only if the vaccine is efficacious in older adults (eg, approximately >60 years) and widespread distribution of the vaccine exists, including to people who are most susceptible to COVID-19. High coverage among these groups who are at high risk of severe COVID-19 would have the greatest effect against disease endpoints. Alternatively, vaccines that do not affect the clinical course, but reduce the transmissibility of SARS-CoV-2, could still be valuable interventions on a population level.

Study design of SARS-CoV-2 vaccine efficacy trials

Provided SARS-CoV-2 is circulating, comparison of clinical endpoints between vaccinated participants and an unvaccinated comparator group in an RCT is the most efficient study design to show vaccine efficacy.⁶ Although vaccine candidates can be assessed in isolation,⁴ WHO and the US FDA suggest that an adaptive trial design, evaluating multiple vaccine candidates in parallel against a single placebo group, could be an acceptable method to increase efficiency, provided that the trials are sufficiently powered.6, 16

Studies that rely on natural exposure to SARS-CoV-2 are vulnerable to multiple variables that influence whether a vaccinee is exposed to SARS-CoV-2 and then whether exposure leads to infection (figure 2 ). For example, older participants could be more likely to avoid social gatherings or use of public transport, reducing their likelihood of exposure to SARS-CoV-2. However, health-care workers might not only be more likely to be exposed to SARS-CoV-2 but also might receive higher infecting doses than would other participants in the study. Alternatively, by virtue of their recognised high-risk occupations, health-care workers might have better access to protective strategies, such as personal protective equipment, than other participants, reducing the likelihood of infection following exposure. These complex behavioural variables are difficult to control; therefore, it is important that participants are randomly assigned between vaccine and comparator groups, by use of a concealed method, to ensure reliable assessment of efficacy outcomes.

Figure 2.

Key variables for SARS-CoV-2 exposure, infection, and poor outcome

Efficacy studies should be adequately powered to meet efficacy endpoints, and multiple variables inform these calculations, including local transmission rates and participant characteristics. For example, the severity of COVID-19 and mortality rates vary according to age, sex, and ethnicity, with higher rates of hospital admissions, critical-care admissions, and death in older individuals, men, and individuals of Black, Asian, and minority ethnic groups.¹⁴ For example, if a phase 3 efficacy study enrolled participants aged only 20–29 years, the expected low mortality rate in this population would require an unfeasibly large sample size to adequately power the study to assess mortality as an endpoint, and the study would be reliant on a high rate of transmission to meet other efficacy endpoints (table 1 ). Selection of older participants with high rates of mortality, for example people older than 80 years, could reduce this number (table 1). However, given that older participants, especially people with comorbidities, are more likely to socially shield, they might be less likely to be exposed to SARS-CoV-2 and so a mortality efficacy endpoint might still not be met. Indeed, recruitment of older participants in vaccine trials has been historically challenging;20, 21 Cochrane reviews of influenza vaccine studies listed 52 RCTs in healthy adults with participants predominately aged 16–65 years, but only eight RCTs in adults older than 65 years,22, 23 despite the higher burden of disease in older adults. Given that mortality from SARS-CoV-2 disproportionately affects older adults, it is important that enrolment of older participants in COVID-19 vaccine trials is actively pursued via targeted engagement, minimising inconvenience to participants, and proactive sharing of study results.24, 25

Table 1.

Illustrative sample size calculations for a randomised controlled trial to assess efficacy of a SARS-CoV-2 vaccine candidate, calculated according to incidence of SARS-CoV-2 infection and age of participants

	Infection	Symptomatic infection	Hospital admission	Death
0·12 infections per 1000 people per day over 6 months*
20–29 years	1880	3154	183 930	619 130
>80 years	1880	3154	10 364	24 494
0·013 infections per 1000 people per day over 6 months†
20–29 years	17 876	29 816	1 722 106	5 796 166
>80 years	17 876	29 816	97 304	229 584

Data are n. Calculations assume no clustering and that participants are randomly assigned to either a SARS-CoV-2 vaccine or a comparator or placebo in a 1:1 ratio, with 80% power to detect 70% vaccine efficacy within 6 months of follow-up and with 5% significance, for various primary efficacy endpoints. Calculations assume that incidence is unchanged over the follow-up period, there is no difference in rates of infection on exposure according to age, and 60% of infected individuals become symptomatic.¹⁷ Hospital admission rates related to age and infection fatality ratio are taken from Verity and colleagues.¹⁸ Each scenario presumes participants aged only either 20–29 years or >80 years are enrolled in the vaccine efficacy trial. SARS-CoV-2=severe acute respiratory syndrome coronavirus 2.

^*Incidence of SARS-CoV-2 infection at peak of transmission in the UK (April, 2020; derived from UK Office of National Statistics data reporting 4793 RT-PCR positive cases per day, which are presumed to only include symptomatic cases).¹⁹

^†Incidence of SARS-CoV-2 infection post peak in the UK (July, 2020; derived from UK Office of National Statistics data reporting 512 RT-PCR positive cases per day, which are presumed to only include symptomatic cases).¹⁹

Dedicated trials will be needed to assess COVID-19 vaccines in individuals younger than 18 years as trial data in adults might not be predictive of vaccine safety and efficacy in this age group. However, since children are less affected by COVID-19 disease than are adults, substantial safety data should be collected from adults, and greater understanding should be acquired of the biology of paediatric multisystem inflammatory syndrome temporally associated with SARS-CoV-2 infection, before paediatric vaccine studies are initiated.²⁶

Incidence of SARS-CoV-2 infection varies hugely, with hotspots and surges occurring in an unpredictable way.¹ This unpredictability has a considerable effect on study sample sizes for efficacy endpoints (table 1), especially as infection rates are highly likely to change over the study follow-up period. For these reasons, assessment of vaccine efficacy against mortality is non-viable in current phase 3 clinical trials. However, as the pandemic spreads internationally, and other variables, such as poverty and decreased access to hospital care, contribute to high rates of severe disease in some populations, clinical trials in such settings could provide a measure of efficacy against severe disease with fewer participants than are suggested in table 1. Alternatively, pooling data from multiple trials that were not originally configured as a network of sites could mean that efficacy endpoints are met earlier, and conclusions about the efficacy of candidate vaccines are reached sooner.²⁷ The usefulness of pooling of data assumes that trial protocols can be sufficiently aligned and might come at the expense of a loss of statistical power if heterogeneity exists between trials.²⁸

Although SARS-CoV-2 vaccine efficacy against mortality cannot be measured directly in clinical trials, it can be deduced from other endpoints. For example, an RCT efficacy study of a live rotavirus vaccine, including more than 69 000 participants, mostly in high-income countries, showed 95% efficacy against severe disease in vaccinees (notably, this sample size was chosen for safety rather than efficacy endpoints).²⁹ Although this study and others did not provide direct data for vaccine efficacy against mortality,³⁰ this efficacy could be conjectured, and following a WHO recommendation for the widespread introduction of rotavirus vaccination,³¹ substantial declines were noted in mortality from diarrhoeal illness.³²

Asymptomatic SARS-CoV-2 infection is a less important clinical endpoint than is efficacy against mortality. However, if prevention of asymptomatic infection was a surrogate endpoint for efficacy against clinical disease or transmission, this endpoint could allow earlier estimation of the clinically relevant efficacy of a vaccine by use of a decreased sample size. Such an approach is already used for the assessment of pneumococcal vaccines, where the ability of a vaccine to prevent colonisation with vaccine serotypes is increasingly used as a surrogate measure of efficacy of vaccines against pneumococcal disease.³³

As the incidence of SARS-CoV-2 decreases, the time required to meet efficacy endpoints increases. This delay can be countered by increasing the sample size of the study (table 1); however, such a strategy increases the number of individuals exposed to the risks of receiving an investigational product prelicensure. When the incidence is low, data for vaccine efficacy against infection and mild disease can still be collected in outbreak situations. For example, a so-called ring vaccination study design can be used, which relies on tracing and vaccinating contacts, and contacts of contacts, of a confirmed case. Vaccination can be administered immediately or weeks later and can provide a rapid efficacy measure, if appropriately powered.³⁴ This study design was used to show the efficacy of a recombinant vesicular stomatitis virus vectored Ebola vaccine (ie, rVSV-ZEBOV) during the Ebola virus disease outbreak in Guinea in 2015, when no cases were confirmed among contacts vaccinated immediately compared with 16 cases in the delayed-vaccination group.³⁵ However, for this study design to provide a measure of the efficacy of a SARS-CoV-2 vaccine, robust diagnostic and contact-tracing pathways and rapid inducement of protective immunity post vaccination would be needed. Given the respiratory route of transmission and the short incubation period of SARS-CoV-2, it is unlikely that this study design would be a feasible means of assessing the efficacy of SARS-CoV-2 vaccines.

Claims exist that the rate of mutation and phylogenetic diversification of SARS-CoV-2 is slow.³⁶ This suggestion, combined with the selection of conserved antigens for most COVID-19 vaccine candidates,⁴ means that vaccine efficacy detected against a particular circulating variant of SARS-CoV-2 in one region is likely to predict efficacy in other parts of the world.³⁷ However, to support this expectation, and because an efficacious vaccine can itself provide a selective pressure for SARS-CoV-2 mutation, serum samples from vaccinees in efficacy studies should be tested for neutralisation against a range of viral lineages.³⁸ Ongoing surveillance for viral escape from immunity induced by vaccines or mediated by antibodies will also be important.

Efficacy endpoints

To allow comparison of efficacy between vaccine candidates and within differing populations, it is essential that standardised, quantifiable endpoints are applied routinely to clinical trials of COVID-19 vaccines, and that their limitations and potential for bias are understood.

COVID-19 disease severity and mortality

The COVID-19 Clinical Working Group of the Coalition for Epidemic Preparedness Innovations has published guidance suggesting that the primary endpoint for assessment of efficacy should be virologically confirmed COVID-19.³⁹ As understanding of COVID-19 grows, so does appreciation of the heterogeneity of the symptoms and signs associated with SARS-CoV-2 infection.14, 40 The clinical criteria chosen to prompt diagnostic testing needs to carefully balance sensitivity, to ensure all cases are identified, with specificity.³⁹ Symptoms that should prompt contact with the clinical trial team need to be communicated carefully to trial participants since this diagnostic testing is reliant on ad-hoc presentation with symptoms and requires substantial engagement and motivation on the part of the participant. As new information accrues, recognition of additional specific symptoms might require a change to the diagnostic criteria.

Given that not all cases meeting the clinical criteria will be infected with SARS-CoV-2, diagnostic testing to confirm the causative pathogen is important. However, the sensitivity of quantitative RT-PCR, the current gold-standard assay for diagnosis of SARS-CoV-2, is imperfect and influenced by variables such as viral load, sample type, and timing (panel 1 ).⁴³ In clinical settings, a substantial proportion of patients continue to be managed as presumed patients with COVID-19 despite repeated negative quantitative RT-PCR tests, potentially reflecting limitations in diagnostic assays and growing understanding of the clinical presentation and course of SARS-CoV-2 infection. RT-PCR assay specificity is an even more important consideration, especially if the incidence of infection is low and efficacy analysis is powered by a few cases.⁴⁶

Panel 1. Key considerations in the use of SARS-CoV-2 quantitative RT-PCR assays as an efficacy endpoint.

• •Sensitivity and specificity of any quantitative RT-PCR assay for SARS-CoV-2 infection is unknown given the absence of a validated gold standard for diagnosis.⁴¹
• •Numerous quantitative RT-PCR methods using different SARS-CoV-2 genomic targets (including ORF1a or ORF1b, nucleocapsid genes, spike protein genes) have been validated with varying reported sensitivity and specificity.⁴²
• •Reported assay sensitivity is influenced by multiple factors including: assay type, timepoint of infection, sample choice, and duration in transit.⁴³
• •The dynamics of viral shedding in people who are presymptomatic, symptomatic, and recovering from infection varies and is incompletely understood, so timing of sampling is important.⁴⁴
• •Most quantitative RT-PCR assays in use do not distinguish between RNA from live, transmissible virus and non-infectious RNA persisting postinfection.⁴⁵ For this reason, both symptoms and PCR positivity are recommended as a primary outcome by the Coalition for Epidemic Preparedness Innovations.³⁹

An efficacious COVID-19 vaccine could reduce the severity of disease resulting from SARS-CoV-2 infection. To assess this potential, careful data collection is required to evaluate markers of severe disease. Numerous methods of scoring the severity of COVID-19 exist and few are validated (appendix p 1). Classical measures of disease severity, such as admission to hospital, requirement for respiratory support, or admission to intensive care, represent the clinical phenotypes that place the most burden on health-care systems and are important endpoints.³⁹ However, these phenotypes might represent only a proportion of those with disease.⁴⁰

It is unclear whether previous exposure to SARS-CoV-2 provides protection against subsequent infection. Some studies to date have excluded participants who are seropositive for SARS-CoV-2. However, the US FDA recommends that participants in vaccine efficacy studies are not screened or excluded if they have a previous history or laboratory evidence of previous SARS-CoV-2 infection, as prevaccination screening is unlikely to occur in practice with deployment of a licensed COVID-19 vaccine.⁶ Stratification analysis will, therefore, be important to establish the effect of pre-existing immunity on efficacy outputs.

A syndrome of vaccine-associated enhanced respiratory disease has been reported in preclinical studies of some viral vaccines, in which immunised animals had increased likelihood of infection or severe disease when subsequently challenged with the target pathogen. Murine, ferret, and non-human primate animal data for candidate vaccines against severe acute respiratory syndrome coronavirus 1 and Middle East respiratory syndrome have shown this effect, which is thought to be mediated by non-neutralising antibodies or by T-helper-2-cell skewed response.⁴⁷

A risk exists for SARS-CoV-2 vaccine-associated enhanced respiratory disease, and it is unknown how it might manifest in humans.⁶ If vaccine-associated enhanced respiratory disease were to increase the likelihood of severe disease, then its relative risk could be calculated by comparing the incidence of severe disease and mortality between vaccinees and recipients of a comparator vaccine. However, if detection of vaccine-associated enhanced respiratory disease is dependent on the occurrence of disease, then this dependency would provide a null-biased estimate of the relative risk. Regular screening of study participants for asymptomatic infections after vaccination and testing of symptomatic participants allows calculation of the true incidence of infection in groups. This denominator allows accurate calculation of the relative and absolute risk of vaccine-associated enhanced respiratory disease, which are important safety data that are of public interest.

Clinical trials might not be sufficiently powered to detect vaccine-associated enhanced respiratory disease or serious adverse events related to the vaccine, if they are uncommon. The US FDA recommends that follow-up of study participants should continue for as long as is feasible, ideally for at least 1–2 years, to assess the duration of protection and potential for vaccine-associated enhanced respiratory disease as the immune response to the vaccine wanes.⁶ Given that COVID-19 vaccines might be deployed in the early post-marketing period to large populations over a short timeframe, it will be important that robust, ongoing pharmacovigilance is in place post licensure to identify safety signals that large-scale RCTs might not capture.⁶

SARS-CoV-2 infection

SARS-CoV-2 infection can cause only mild, non-specific symptoms in many individuals, which do not result in contact with a health-care professional.¹⁴ Asymptomatic infections are well recognised⁴⁸ but difficult to capture. Serial sampling of vaccinees, for example via diagnostic testing once per week, could ensure that all infected individuals are identified, regardless of symptoms, and give an indication of the duration of infectivity. Ideally, vaccinees would be informed of their test results in real time to allow appropriate isolation. However, it is worth considering that such a system could introduce an unavoidable bias toward clinical endpoints, as participants might be more likely to report symptoms meeting clinical criteria for COVID-19 if they know that they are infected. This monitoring strategy requires a considerable commitment from vaccinees over an extended period, which is likely to result in an ever-reducing number of samples collected. Asking vaccinees to self-sample by, for example, posting self-collected oronasal swabs once per week for quantitative RT-PCR testing could help to increase the number of samples obtained; however, this process could be logistically difficult to implement, with considerable costs and an ever-reducing return rate.

Following infection with SARS-CoV-2, antibody responses are formed against key viral antigens, including the nucleoprotein and spike protein, typically peaking 14–21 days after onset of symptoms.⁴⁹ Most SARS-CoV-2 vaccine candidates seek to induce neutralising anti-spike protein antibodies⁵⁰ and several assays have been described as methods of assessing evidence of infection.⁴³ Vaccine efficacy studies that screen and exclude participants who are seropositive for SARS-CoV-2 could use seroconversion of vaccinees post vaccination as a surrogate for infection (appendix p 2), provided that the antibody assayed is specific to infection and not induced by the candidate vaccine.⁵¹ Seroconversion might allow detection of ongoing or recovered infection in vaccinees with minimal symptoms who do not present for quantitative RT-PCR testing⁵¹ and increase the likelihood of diagnosis when combined with quantitative RT-PCR testing.⁵² Furthermore, given that the duration of seropositivity exceeds that for which RNA can be detected, serological testing offers a substantial operational advantage over quantitative RT-PCR, with a larger time window to capture the endpoint (appendix p 2).⁴³ However, US FDA guidance that participants in vaccine efficacy studies should not be excluded if they have a history or laboratory evidence of previous SARS-CoV-2⁶ means that the value of this endpoint is unclear, particularly in populations with high baseline seropositivity to SARS-CoV-2. Additionally, key caveats further decrease the usefulness of seroconversion as an efficacy endpoint (panel 2 ).

Panel 2. Key considerations for the use of serological markers of infection as a vaccine efficacy endpoint.

• •Sensitivity and specificity of any serological assay for SARS-CoV-2 disease are unknown given the absence of a validated gold standard for diagnosis.⁴¹
• •Reported sensitivity and specificity of antibody tests for SARS-CoV-2 vary widely⁵³ and are most likely influenced by timing of sampling in relation to infection.49, 52, 53
• •Assays need to be able to distinguish antibodies induced by infection from those induced by vaccination.
• •The kinetics of antibody responses following infection are incompletely described and might wane substantially within months.⁵⁴
• •Individuals can be seronegative for antibodies post infection, as evidenced by the detection of memory T-cell responses to SARS-CoV-2 in the absence of antibodies.⁵⁵

Transmission

SARS-CoV-2 is readily transmitted between individuals, predominantly via droplet transmission. However, aerosol and faeco–oral routes of transmission might also take place.⁵⁶ Individuals are also known to transmit SARS-CoV-2 in the asymptomatic and presymptomatic period.⁴⁸ During convalescence, patients can shed viral RNA for many weeks,57, 58 and even longer if immunosuppressed.⁵⁹ However, there is an unclear association between detectable RNA by quantitative RT-PCR and the ability to culture SARS-CoV-2 in vitro.⁶⁰

It is possible that a SARS-CoV-2 vaccine could reduce severity of disease but lead to prolonged shedding of infectious virus, which could have important consequences for public health if shedding resulted in increased transmission. Therefore, it might be important for investigators to consider not only the duration of RNA positivity in regularly collected samples but also whether these samples include replication-competent live virus. Nested quantitative RT-PCR targeting subgenomic RNA that encodes conserved structural proteins has been suggested as a measure of active SARS-CoV-2 replication.⁶⁰ As the transcription of these RNAs is reliant on translation of the ORF1 gene within the host cell and the subsequent assembly of an RNA-dependent RNA polymerase, the detection of subgenomic RNAs can help to identify replication-competent and therefore transmissible virus.⁶¹ However, emerging evidence suggests that subgenomic RNAs might be more stable than previously purported and so could be detectable for a period beyond the disappearance of actively replicating virus.⁶²

Surrogate endpoints

If an immunological correlate of protection is known, then the protective efficacy of a vaccine can be assessed by measuring the proportion of vaccinees who generate a particular immune response, without having to measure clinical outcomes.⁶³ This technique facilitates the rapid screening and deselection of candidate vaccines. A potential surrogate endpoint for a SARS-CoV-2 vaccine would most likely depend on the characteristics of the vaccine, including antigen structure, method of delivery, and antigen processing and presentation in vaccines.⁶

Not all individuals exposed to SARS-CoV-2 become infected⁶⁴ and heterogeneity is seen in clinical outcomes.¹⁴ However, the immunological mechanisms underlying protection or susceptibility to natural infection are unknown. Seroconversion of antibodies against SARS-CoV-2 is a marker of exposure, but whether the presence of neutralising antibodies is sufficient to provide protection against subsequent infection or disease is unclear.⁴⁹ Moreover, if these antibodies are sufficient, we do not know the titre that would be needed for protection or the diverse range of innate immune effector functions that can be relied on for antibody action, such as antibody-dependent complement deposition and antibody-dependent neutrophil phagocytosis.⁶⁵ Cellular immune responses have also been described in response to infection and are likely to be an important component of a protective adaptive immune response.⁶⁶ Indeed, individuals have been described who were seronegative and had T-cell responses to the SARS-CoV-2 spike protein.⁵⁵ However, the particular cellular signature that is required for protection is unknown, and whether protective T cells can be measured in peripheral blood samples is unclear. Additionally, an efficacious SARS-CoV-2 vaccine might provide protection by a mechanism that is distinct from the mechanism induced following natural infection. Distinguishing immunological markers of infection from mechanistic correlates of protection is difficult but important to inform rational design of a vaccine.

If a SARS-CoV-2 vaccine were to show efficacy, a priority would be to identify an in-vitro correlate or surrogate of protection.⁶⁷ Other vaccine candidates could then be deemed efficacious and licensed if they induced similar levels of immune responses in non-inferiority studies, which would circumvent the need for large efficacy studies. Evidence of effectiveness against disease would be needed in post-licensure studies, however, this approach could markedly accelerate development of multiple SARS-CoV-2 vaccines. This approach relies on collaboration and standardisation of in-vitro assays to allow meaningful comparison of immunological outputs from different laboratories.⁶⁸

In the absence of data for humans, animal studies can help to identify potential correlates of protection. Indeed, non-human primate studies are being widely used to understand SARS-CoV-2.⁶⁹ Protection against reinfection with SARS-CoV-2 has been observed in rhesus monkeys, who formed neutralising antibodies on initial exposure,⁷⁰ and a minimum neutralising antibody titre has been proposed.⁷¹ However, since SARS-CoV-2 is a novel pathogen, any surrogate endpoints identified in animal studies would ideally need validation in clinical trials to ensure that they adequately predict efficacy in humans.

When human efficacy studies are unethical or unfeasible, marketing approval can be granted on the basis of “well controlled animal efficacy studies when the results of those studies establish that the drug is reasonably likely to produce clinical benefit in humans” (ie, the so-called animal rule).⁷² For example, the European Medicines Agency recommended marketing approval for an Ebola virus vaccine (ie, Ad26.ZEBOV and MVA-BN-Filo administered in a prime-boost regimen) on the basis of efficacy data that were extrapolated to humans from animal and immunobridging studies.⁷³ If it is impossible to collect human efficacy data, then SARS-COV-2 vaccines might be licensed on the basis of the animal rule, with effectiveness data collected after vaccine roll-out. However, the absence of accepted surrogate endpoints in humans or animals that are reasonably likely to predict the clinical benefit of a SARS-CoV-2 vaccine mean that investigators continue to pursue clinical evidence of vaccine efficacy in studies in humans.⁶

Controlled human infection model

Controlled human infection model (CHIM) studies, in which human volunteers are exposed to infectious pathogens (also known as challenge agents), are an important component of pathology, immunology, and vaccine research. Microbial challenge studies are useful for providing proof of concept for therapeutic interventions and can substantially reduce the time needed to reach phase 3 studies, as has been shown for malaria and typhoid vaccine development.74, 75, 76

Administration of a known dose of SARS-CoV-2 in a carefully controlled setting has been suggested as a model to allow rapid initial assessment of vaccine efficacy and early deselection of vaccine candidates.⁷⁷ A COVID-19 CHIM model has several advantages over studies that are reliant on naturally occurring community transmission, which is difficult to predict and dependent on changes in behaviour and public health interventions (table 2 ).

Table 2.

A comparison of the key factors for clinical trials that are reliant on natural exposure to, or a direct challenge with, SARS-CoV-2

	Natural field infection	Controlled human infection model
Infecting pathogen	Unknown	Sequenced SARS-CoV-2 made to good manufacturing practices
Infecting dose	Unknown	Predetermined and standardised
Timing of infection	Unknown	Predetermined
Risk to participant	No increased risk above population level	Potentially lower*
Numbers of participants required	High	Low
Participant involvement	Minimal	Likely to require an extended stay in study facility
Public health implications	NA	Risk of onward transmission
Confounders	Participant behaviour and risk factors; changes in public health policy; changes in transmission dynamics during the study	NA
Generalisability	Dependant on study size	Unclear, especially with reference to specific groups who are at high risk of severe disease

NA=not applicable. SARS-CoV-2=severe acute respiratory syndrome coronavirus 2.

^*The risk of infection in a controlled human infection model trial could be lower than naturally acquired infection as individuals who are at low risk of severe disease can be selected (eg, aged 18–25 years), the minimum dose of virus needed to acquire infection can be administered, individuals can be carefully monitored, and rescue therapies can be given if needed.

Development of vaccines for COVID-19 has proceeded at an unprecedented rate to date, with some candidates beginning phase 3 studies within 4 months of the start of vaccine development.⁴ If done, SARS-CoV-2 CHIM studies are likely to include carefully selected young volunteers at low risk of severe disease, who are exposed to low doses of SARS-CoV-2 with the aim of establishing only asymptomatic or mild infection. It is unclear whether efficacy shown in such a model will predict the key efficacy measure of protection against severe disease and death in the target older population who are at risk of severe disease.⁷⁸

CHIM studies could provide valuable immunological insights. For example, re-exposing individuals to SARS-CoV-2 who have recovered from naturally acquired infection could help to identify a surrogate marker of protection, which would inform vaccine design. Provided CHIM studies can be done safely, the information gained can be viewed as complementary to traditional RCTs, both to guide resources for large-scale phase 3 studies and in the efficacy evaluation of existing vaccines.

CHIM studies require controlled delivery of a standardised inoculum, ideally made to good manufacturing practices, and meticulous care to prevent community transmission of the challenge strain (appendix p 2).⁷⁹ These studies can be logistically difficult and costly per participant, although the number of participants required is far lower than in large phase 3 studies. Although there are challenges to setting up a CHIM of SARS-CoV-2, there might also be substantial value in doing so, even in the context of a licensed product. Use of this ethically complex and controversial approach for vaccine assessment will require multidisciplinary, international oversight to ensure that outputs are rigorous and justify the potential risks to participants and their communities.78, 79

Conclusion

Assessment of the efficacy of a vaccine is complex for many diseases but particularly so in the case of SARS-CoV-2, where the fundamental understanding of the pathogen is evolving. Multiple vaccines are being tested worldwide in early-phase studies and some vaccine candidates are already in phase 3 studies assessing efficacy.⁴ It is probable that there will not be a single vaccine winner; diverse platforms and technologies can offer different strengths and be relevant in distinct epidemiological contexts. Additionally, there will probably be insufficient supply, at least initially, of a single vaccine. However, collaboration and standardised approaches for assessing different efficacy endpoints will be important to allow meaningful comparison and ensure that the most effective candidates are deployed. Following deployment, well supported pharmacovigilance studies should be established to ensure the ongoing evaluation of vaccine safety.

Capacity to measure vaccine efficacy in field studies is reliant on ongoing SARS-CoV-2 transmission, which is rightly at odds with public health interventions. In the absence of a surrogate of protection, CHIM trials might provide the only means of rapidly assessing vaccine efficacy; however, the relationship between efficacy data from CHIM studies in young individuals and population-level protection is unclear. CHIM studies might help to identify a surrogate of protection. It is probable that any evidence for efficacy against severe disease and mortality in populations that are at risk will only be garnered post licensure via large epidemiological studies.

Finally, the development of SARS-CoV-2 vaccines is under great political and media scrutiny. In keeping with the development of any novel medical intervention, but particularly so in this context, it is imperative that efficacy outcomes for a SARS-CoV-2 vaccine are critically appraised with scientific rigour to understand their generalisability and clinical significance.

Search strategy and selection criteria

References for this Review were identified through a search of PubMed for articles published from Jan 1, 2020, to July 31, 2020, by use of the terms “COVID-19” and “SARS-CoV-2.” Other relevant references were identified from key online sources (eg, WHO) and authors' personal files. Only articles published in English were included.