The value of vaccine programme impact monitoring during the COVID-19 pandemic

Reports of the historic vaccine development successes during the COVID-19 pandemic have highlighted two critical measures of vaccine performance—vaccine efficacy, determined by randomised controlled trials, and vaccine effectiveness, estimated from post-introduction observational studies. Both these statistics describe an individual's risk reduction after vaccination. As immunisation programmes expand globally, more estimates of a third measure of vaccine performance—vaccine impact—are needed. Vaccine impact studies estimate disease reduction in a community.¹ These studies are typically ecological or modelling analyses that compare disease outcomes from pre-vaccine and post-vaccine introduction periods.² Reductions in disease outcomes are realised through direct effects of vaccination in vaccinated people and indirect effects due to reduced transmission within a community. Sometimes other concurrent interventions or phenomena unrelated to vaccine effects, such as changes in risk behaviours or health-care practices, can reduce disease outcomes and confound assessments of vaccine impact.

In The Lancet, Lucy McNamara and colleagues³ report the impressive and early impact of COVID-19 vaccines on the health of older adults in the USA.³ The authors used COVID-19 health outcome and vaccination coverage data from Nov 1, 2020, to April 10, 2021, to compare the relative change in outcomes for pre-introduction and post-introduction periods among people aged 65 years and older, a demographic that received vaccines early, with the relative change in outcomes among younger age groups, who were able to access vaccines later. The outcomes of interest were COVID-19 cases, emergency department visits, hospitalisations, and deaths. Case data and emergency department visits were from datasets that covered a large proportion of the US population, and hospitalisation and death data were national in scope, ensuring gender balance and excellent representation of minority ethnic groups; however, inputs were limited to jurisdictions and age groups with the most complete reporting. Regression models used data aggregated by week, age group, and jurisdiction to compare trends in rate ratios by age groups.

After vaccine introduction, age groups receiving COVID-19 vaccines had large relative decreases in COVID-19 cases, emergency department visits, and hospitalisations compared with a younger reference age group. For example, compared with those aged 50–59 years, the relative decrease in the ratio of pre-vaccine to post-vaccine COVID-19 hospitalisations was 39% (95% CI 29–48) in those aged 60–69 years, 60% (54–66) in those aged 70–79 years, and 68% (62–73) in those aged 80 years and older. The mortality analysis did not follow a similar pattern, as older age groups had no statistically significant differences in pre-introduction to post-introduction mortality ratios compared with a younger reference age group. This finding could have been due to the confounding effects of non-pharmaceutical interventions introduced in long-term care facilities before vaccines, which could have differentially decreased mortality in the older age groups.

Except for the mortality analysis, the disease reductions described in this study align with expectations based on the high level of vaccine protection shown by randomised controlled trials and the rapid uptake of these vaccines in older age groups.4, 5, 6 The strengths of the study include the large geographical scope of the data inputs, and the county-level analyses which, at least partly, accounted for geographically associated variables, such as public health mitigation measures or transmission patterns. The most notable limitation of the study is the ecological nature of its design, which did not allow for individual-level analyses of vaccine effectiveness against disease outcomes or of behaviours that might have affected COVID-19 risk. The authors excluded data from several states because of low quality or incompleteness due to shortcomings in public health data collection and reporting systems in the USA.

Observational assessments of vaccine performance have taken on increased importance as populations targeted for vaccination expand, the epidemiology of the pandemic evolves, and randomised placebo-controlled trials become less feasible and ethical to conduct.7, 8 It will be important to assess public health outcomes in diverse settings, given differences in circulation and emergence of viral variants, the heterogeneity of vaccines and vaccination schedules, and the diversity of populations targeted. Countries with national health-care systems, including Israel and Scotland, have been able to assess vaccine impacts rapidly and comprehensively under changing conditions.9, 10 Future COVID-19 vaccine impact studies in the USA will need to be timely to keep pace with critical public health needs. Finally, as the largest vaccine roll-out in history expands even further, we must ensure that low-income and lower–middle-income countries are supported in their efforts to establish immunisation programme monitoring and evaluation to guide appropriate and context-specific decision making.

© 2022 adamkaz/Getty Images