COVID-19 vaccine efficacy and effectiveness—the elephant (not) in the room

Approximately 96 COVID-19 vaccines are at various stages of clinical development.¹ At present, we have the interim results of four studies published in scientific journals (on the Pfizer–BioNTech BNT162b2 mRNA vaccine,² the Moderna–US National Institutes of Health [NIH] mRNA-1273 vaccine,³ the AstraZeneca–Oxford ChAdOx1 nCov-19 vaccine,⁴ and the Gamaleya GamCovidVac [Sputnik V] vaccine)⁵ and three studies through the US Food and Drug Administration (FDA) briefing documents (on the Pfizer–BioNTech,⁶ Moderna–NIH,⁷ and Johnson & Johnson [J&J] Ad26.COV2.S vaccines).⁸ Furthermore, excerpts of these results have been widely communicated and debated through press releases and media, sometimes in misleading ways.⁹ Although attention has focused on vaccine efficacy and comparing the reduction of the number of symptomatic cases, fully understanding the efficacy and effectiveness of vaccines is less straightforward than it might seem. Depending on how the effect size is expressed, a quite different picture might emerge (figure ; appendix).

Figure.

RRR and NNV with 95% CI ranked by attack rate in the unvaccinated (placebo) group for five COVID-19 vaccines

The lower the NNV and the higher the RRR, the better the vaccine efficacy. Details are in the appendix (p 3). RRR=relative risk reduction. NNV=numbers needed to vaccinate. NIH=US National Institutes of Health.

Vaccine efficacy is generally reported as a relative risk reduction (RRR). It uses the relative risk (RR)—ie, the ratio of attack rates with and without a vaccine—which is expressed as 1–RR. Ranking by reported efficacy gives relative risk reductions of 95% for the Pfizer–BioNTech, 94% for the Moderna–NIH, 91% for the Gamaleya, 67% for the J&J, and 67% for the AstraZeneca–Oxford vaccines. However, RRR should be seen against the background risk of being infected and becoming ill with COVID-19, which varies between populations and over time. Although the RRR considers only participants who could benefit from the vaccine, the absolute risk reduction (ARR), which is the difference between attack rates with and without a vaccine, considers the whole population. ARRs tend to be ignored because they give a much less impressive effect size than RRRs: 1·3% for the AstraZeneca–Oxford, 1·2% for the Moderna–NIH, 1·2% for the J&J, 0·93% for the Gamaleya, and 0·84% for the Pfizer–BioNTech vaccines.

ARR is also used to derive an estimate of vaccine effectiveness, which is the number needed to vaccinate (NNV) to prevent one more case of COVID-19 as 1/ARR. NNVs bring a different perspective: 81 for the Moderna–NIH, 78 for the AstraZeneca–Oxford, 108 for the Gamaleya, 84 for the J&J, and 119 for the Pfizer–BioNTech vaccines. The explanation lies in the combination of vaccine efficacy and different background risks of COVID-19 across studies: 0·9% for the Pfizer–BioNTech, 1% for the Gamaleya, 1·4% for the Moderna–NIH, 1·8% for the J&J, and 1·9% for the AstraZeneca–Oxford vaccines.

ARR (and NNV) are sensitive to background risk—the higher the risk, the higher the effectiveness—as exemplified by the analyses of the J&J's vaccine on centrally confirmed cases compared with all cases:⁸ both the numerator and denominator change, RRR does not change (66–67%), but the one-third increase in attack rates in the unvaccinated group (from 1·8% to 2·4%) translates in a one-fourth decrease in NNV (from 84 to 64).

There are many lessons to learn from the way studies are conducted and results are presented. With the use of only RRRs, and omitting ARRs, reporting bias is introduced, which affects the interpretation of vaccine efficacy.¹⁰ When communicating about vaccine efficacy, especially for public health decisions such as choosing the type of vaccines to purchase and deploy, having a full picture of what the data actually show is important, and ensuring comparisons are based on the combined evidence that puts vaccine trial results in context and not just looking at one summary measure, is also important. Such decisions should be properly informed by detailed understanding of study results, requiring access to full datasets and independent scrutiny and analyses.

Unfortunately, comparing vaccines on the basis of currently available trial (interim) data is made even more difficult by disparate study protocols, including primary endpoints (such as what is considered a COVID-19 case, and when is this assessed), types of placebo, study populations, background risks of COVID-19 during the study, duration of exposure, and different definitions of populations for analyses both within and between studies, as well as definitions of endpoints and statistical methods for efficacy. Importantly, we are left with the unanswered question as to whether a vaccine with a given efficacy in the study population will have the same efficacy in another population with different levels of background risk of COVID-19. This is not a trivial question because transmission intensity varies between countries, affected by factors such as public health interventions and virus variants. The only reported indication of vaccine effectiveness is the Israeli mass vaccination campaign using the Pfizer–BioNTech product. Although the design and methodology are radically different from the randomised trial,² Dagan and colleagues¹¹ report an RRR of 94%, which is essentially the same as the RRR of the phase 3 trial (95%) but with an ARR of 0·46%, which translates into an NNV of 217 (when the ARR was 0·84% and the NNV was 119 in the phase 3 trial). This means in a real-life setting, 1·8 times more subjects might need to be vaccinated to prevent one more case of COVID-19 than predicted in the corresponding clinical trial.

Uncoordinated phase 3 trials do not satisfy public health requirements; platform trials designed to address public health relevant questions with a common protocol will allow decisions to be made, informed by common criteria and uniform assessment. These considerations on efficacy and effectiveness are based on studies measuring prevention of mild to moderate COVID-19 infection; they were not designed to conclude on prevention of hospitalisation, severe disease, or death, or on prevention of infection and transmission potential. Assessing the suitability of vaccines must consider all indicators, and involve safety, deployability, availability, and costs.

This online publication has been corrected. The corrected version first appeared at thelancet.com/microbe on June 11, 2021

We declare no competing interests.