Vaccine Effectiveness Studies in the Field

The original trials of vaccines against infection with severe acute respiratory disease coronavirus 2 (SARS-CoV-2), the virus that causes coronavirus disease 2019 (Covid-19), have clearly shown vaccine efficacy. However, questions crucial to vaccination policy remain, at best, only partially answered. These include the effect of new virus variants, the timing between vaccine doses, the effect of vaccines on asymptomatic infection in contrast to severe disease, the waning of vaccine immunity, and the potentially enhanced effectiveness of mix-and-match strategies that might be used with booster shots.

Wherever possible, vaccine efficacy should be assessed by means of randomized trials. Valid comparison is then straightforward, with considerable advantages for addressing policy questions. Organizational, regulatory, and governance barriers to randomization should be reduced to a minimum in a pandemic. Nevertheless, nonrandomized studies continue to have a place, although their limitations must be recognized. For example, the Early Pandemic Evaluation and Enhanced Surveillance of Covid-19 (EAVE II) database provided important findings on the effectiveness of a single dose of the BNT162b2 or ChAdOx1 nCoV-19 vaccine in Scotland.¹ The possibility of such large-scale cohort studies is relatively easy in countries such as the United Kingdom, where national registration of all vaccinations can be linked to comprehensive data on virus tests, in contrast to the United States. Moreover, a large proportion of test samples in the United Kingdom are genetically sequenced.

However, we cannot randomize variants, so inevitably some aspects of comparing their effects must be observational. Lopez Bernal and colleagues from Public Health England (PHE) now describe in the Journal² an alternative nonrandomized study comparing the effectiveness of the BNT162b2 and ChAdOx1 nCoV-19 vaccines against two variants (B.1.1.7 [alpha] and B.1.617.2 [delta]) after one or two doses. This study used a test-negative design to assess vaccine effectiveness. This design is a form of case–control study in which persons testing positive for a disease after seeking health care (or because of a surveillance test) constitute the cases, whereas controls are selected from persons who test negative in the same situation, with each participant having their vaccination status recorded.³ The odds of vaccination are then compared between cases and controls, with vaccine effectiveness derived from the odds ratio subtracted from 1, expressed as a percentage.

The outcomes in the PHE study occurred between April 5 and May 16, 2021, when the dominant strain in the United Kingdom was shifting from alpha to delta. The key results are encouraging but emphasize the necessity of the second vaccine dose by showing a markedly lower effectiveness against the delta variant than against the alpha variant among persons who received only one dose of either vaccine. Overall, vaccine effectiveness after one dose was lower by approximately 12 to 19 percentage points against the delta variant than against the alpha variant. After two doses, the difference in vaccine effectiveness was notably less, approximately 6 to 8 percentage points, with similar reductions in effectiveness against the delta variant, as compared with the alpha variant, for both vaccines. Other test-negative design studies have recently studied the field effectiveness of one dose of messenger RNA (mRNA) vaccine against the alpha and P.1 (gamma) variants or one or two doses of the CoronaVac vaccine against the gamma variant.^4,5

Overall, the PHE study showed vaccine effectiveness after two doses of the BNT162b2 vaccine of 94% against the alpha variant and 88% against the delta variant. The corresponding percentages with the ChAdOx1 nCoV-19 vaccine were lower, at 74% and 67%. These field estimates are compatible with patterns seen in neutralizing antibody titers in healthy vaccine recipients.⁶

In test-negative design studies, the confounding effects of health care–seeking behavior is reduced; however, other biases will occur.⁷ In comparing the effectiveness of vaccines against virus variants, some biases may be the same for each variant. In the PHE study, the two vaccines were used in different ways over time and were available in different health care settings and in different age groups at different times, thus making valid comparison difficult. In general, comparison between variants is likely to be less subject to bias than the comparison of the vaccines themselves. As more test-negative design studies are implemented, it will be important to distinguish those that are best able to address these biases. Test-negative design studies of variants are only possible with systematic genetic typing of SARS-CoV-2–positive samples. Biases that are unaccounted for probably make uncertainties in vaccine effectiveness greater than shown in confidence intervals, which only account for statistical sampling variation.

Manufacturers of the mRNA-1273 and Ad26.COV2.S vaccines have recently announced positive findings regarding neutralizing antibody activity against the delta variant.^8,9 Similarly positive news about immunity persistence with both mRNA vaccines (mRNA-1273 and BNT162b2) has been reported.¹⁰ However, these results are based on small samples and need confirmation by valid observational data from large samples of vaccinated persons. A recent test-negative design study in Canada showed positive findings regarding vaccine effectiveness with the BNT162b2, mRNA-1273, and ChAdOx1 nCoV-19 vaccines against symptomatic disease with the alpha, beta (B.1.351), gamma, and delta variants.¹¹

Clinically, high effectiveness of vaccines against severe disease and death is critically important. However, from a public health perspective, reducing the incidence of infection is also vital. As of June 19, 2021, Covid-19 vaccines are estimated to have prevented 7.2 million infections and 27,000 deaths in England alone.¹² Similarly, in the United States, an estimated 279,000 deaths and up to 1.25 million hospitalizations have been averted as of the end of June 2021.¹³ As the delta variant affects various countries, including the United States, the current imperative is to vaccinate as many people as possible, as quickly as possible. Underserved and at-risk communities must not be neglected when implementing this strategy.

This editorial was published on July 21, 2021, at NEJM.org.