Commentary: Estimation of vaccine effectiveness using the screening method

Cases of disease in fully vaccinated persons, referred to as vaccine breakthrough cases, may weaken public confidence in vaccines. Breakthrough cases are expected even with highly effective vaccines. As vaccination coverage increases, breakthrough cases will account for increasing proportions of all cases. In 1985, Orenstein and colleagues proposed the use of a simple, rapid screening method for field investigations of measles outbreaks.¹ Applicable to other vaccine-preventable diseases, the screening method was designed to rapidly determine whether vaccines are performing as expected and whether further investigation is warranted. With effective vaccines, the proportion of cases among vaccinated individuals will be lower than the proportion of the general population that is vaccinated.

Based on the basic formula for vaccine efficacy, 1 – (disease incidence or attack rate in the vaccinated/disease incidence in the unvaccinated), the screening method estimates the proportion of cases expected to occur in vaccinated individuals (PCV) at varying proportions of the population vaccinated (PPV) and vaccine efficacy (VE). When any two values are known, the third may be estimated (Figure 1). For example, with 90% vaccine effectiveness, 50% of cases are expected to be breakthrough infections when vaccine coverage reaches 90%. For simplicity, PPV and VE were treated as determinate without confidence bounds.

Figure 1.

Screening Method: The relationship between % of population vaccinated (PPV), estimated vaccine effectiveness (VE) and % of cases vaccinated (PCV).

Source: Orenstein et al, Field evaluation of vaccine efficacy. Bull World Health Organ, 1985.(1).

Adaptation as shown is reproduced from WHO online document available at https://www.who.int/publications/i/item/WHO-2019-nCoV-vaccine_effectiveness-variants-2021.1

As a rapid screen to inform further investigation, treating VE and PPV as determinate was sufficient. However, studies in different settings would be expected to produce estimates within a range of values. In 1993, CP Farrington proposed a solution to estimate confidence bounds for estimates of vaccine effectiveness from observed values.² Farrington formulated the simple equation for the screening method as a likelihood, in which VE = 1 – [PCV/(1 – PCV)]/[PPV/(1 – PPV)] or 1 – odds ratio of vaccination among cases to vaccination in the population. When PCV and PPV are available for multiple strata, such as birth cohort, age or time period, variation in observed vaccine effectiveness may be modelled and quantified.

Farrington noted that the vaccination odds ratio equals the relative risk of disease in the vaccinated compared with the unvaccinated population. Still referred to as the screening method, Farrington’s approach to estimate confidence bounds for vaccine effectiveness is easily applied because the PCV may be obtained from a representative sample of cases in each stratum rather than all cases. PPV may be estimated from immunization coverage surveys and does not require obtaining individual vaccination status in the source population. Standard errors for vaccination coverage estimates may be used to provide more plausible confidence bounds for vaccine effectiveness.³ Farrington predicted that use of the screening method would increase as electronic immunization information systems and district-level estimates of vaccination coverage became more widely available.

Before Farrington’s publication, Orenstein and colleagues provided several practical considerations when applying the screening method to estimate vaccine effectiveness.⁴ Vaccination coverage estimates should match the populations in which cases occurred and include only persons eligible for vaccination. For unbiased estimates of VE among fully vaccinated individuals, partially vaccinated individuals must be excluded from estimation of both PCV and PPV. Cases in which illness onset occurs before complete immune response (e.g. 2 weeks after receipt of the last dose in the series) are excluded. Originally, the screening method assumed stable vaccination coverage during the period when cases occurred. When vaccination coverage is rapidly changing, PPV is measured at least 2 weeks before cases became ill to provide time for vaccine-induced protection.

In several instances, the screening method has provided real-world evidence of effectiveness of newly introduced vaccines. For example, Haemophilus influenza type b (Hib) conjugate vaccine was introduced in the UK’s routine immunization programme as a three-dose infant schedule with no booster dose.⁵ Children of 1–3 years of age were eligible to receive a single dose of the Hib vaccine as part of a catch-up programme. PCV was determined using vaccination records for laboratory-confirmed cases of invasive Hib disease identified through national laboratory surveillance. PPV was obtained from national immunization coverage data on proportions of children who had received three doses by 12 months of age or a single dose after age 12 months. Effectiveness of the complete three-dose infant schedule was >60% but waned 2 years after vaccination. In contrast, a single dose of Hib vaccine after 1 year of age was 97% effective within 2 years of vaccination and remained protective after 2 years. These data obtained from the screening method supported the addition of a booster dose of Hib conjugate vaccine to the infant immunization schedule.

Later, meningococcal group B vaccine (4CMenB) was introduced into the UK’s childhood immunization programme as a reduced two-dose priming schedule with a booster dose at 12 months.^6,7 Evaluation of the effectiveness of the two-dose priming schedule was needed because a three-dose priming schedule had been licensed based on immunogenicity data. PCV was determined using vaccination records from laboratory-confirmed cases of invasive meningococcal disease and PPV was again obtained from national immunization coverage data. More than 90% of children born within 3 months of vaccine introduction had received two doses by 12 months of age. To account for the rapid uptake of the 4CMenB vaccine, the proportion of the eligible population that had received two doses was estimated 2 weeks before the date of illness onset of each case patient. Evidence suggested >50% effectiveness of two priming doses before 13 months of age and two primary doses plus a booster dose after 12 months for prevention of invasive meningococcal disease.⁶

Most recently, the screening method has been applied during the uptake of COVID-19 vaccines to signal reduced vaccine effectiveness following the emergence of SARS-CoV-2 Delta variant (B.1.617.2).^8,9 Higher than expected proportions of breakthrough cases among fully vaccinated persons suggested reduced vaccine effectiveness against COVID-19 disease caused by the Delta variant virus, as well as the possibility of waning protection. An alternative approach to estimating COVID-19 vaccine effectiveness has used counts of fully or partially vaccinated persons from immunization information systems rather than estimating PPV from immunization coverage surveys as is typically done with the screening method.¹⁰ The number of persons remaining unvaccinated have been estimated from census data by subtracting the number of vaccinated individuals. Advances in immunization information systems for COVID-19 vaccination programmes have provided numbers of fully or partially vaccinated persons by vaccine product and district or jurisdiction for product-specific COVID-19 vaccine estimates during vaccine uptake.^9,10 Because the relative risk of COVID-19 cases among vaccinated compared with unvaccinated persons is equivalent to vaccination odds ratios, VE and confidence bounds may be estimated from risk ratios using Poisson regression models.¹⁰ Availability of immunization data for vaccine eligible populations by age, region and time period allows partial control for major confounders and provides more detailed information on vaccine effectiveness than the screening method, but are still crude or unadjusted estimates. However, as Farrington noted, any random errors in values of the population vaccinated reduce the accuracy of VE estimates. These adaptations of the screening method reflect evolution from the idea of a rapid, simple and inexpensive tool to investigate breakthrough cases to providing real-world evidence of vaccine effectiveness.