Improving assessments of population-level vaccine impact

Assessing the population-level impact of any new vaccine is necessary to inform public health decision-making. The most common approach for such assessments is time-series analysis, which compares disease rates and trends before and after vaccine introduction. However, such studies are subject to a number of important biases and confounders, which can lead to either overly optimistic or pessimistic assessments of vaccine programs. These studies can be strengthened by using best practices from epidemiology and biostatistics as well as employing tools from other related fields such as econometrics. We argue that following four basic recommendations could increase the accuracy, reproducibility and comparability of such studies:

1. Report on outcome specificity used in a study in relation to previously published studies. Relative estimates of vaccine effect are expected to rise with increasing outcome specificity.

2. Apply bias-reducing model selection procedures by using data-driven model selection or model averaging strategies.

3. Assess the likelihood of detecting a vaccine effect by performing simple power calculations with simulated time series.

4. Integrate multiple comparison outcomes to address unmeasured bias and confounding using a data-driven weighting of outcomes (each unaffected by the vaccine introduction) into a ‘synthetic control’.

While there are examples of vaccine impact studies that apply some of these suggestions, this is often not the case and it is rarely seen in combination. Following these best practices is particularly important for studies where the relative impact of the vaccine is expected to be modest or where there is a strong possibility that factors other than vaccination could influence disease rates. For example, studies of vaccine impact against influenza, rotavirus, human papillomavirus, and pneumococcus would all benefit from consistent application of these principles.^1–7

To highlight problems that can occur in vaccine impact studies, and to propose potential solutions, we examine assessments of pneumococcal conjugate vaccine as an intervention against pneumonia in detail. Pneumococcus is a major cause of pneumonia and invasive pneumococcal disease globally.^8,9 Pneumococcal conjugate vaccines are associated with a significant and long-term reduction in the incidence of invasive pneumococcal disease in areas of introduction.^10–12 Although this disease is severe and can lead to death and disability, it occurs with substantially lower incidence than pneumonia.⁸ This means that the high cost of the vaccine cannot always be justified based on its impact on invasive pneumococcal disease alone.¹³ Therefore, decisions about the use of pneumococcal conjugate vaccines that rely on cost-effectiveness considerations depend critically on the expected population-level impact against pneumonia, a far more common condition. Several randomized controlled trials (RCTs) have assessed the efficacy of pneumococcal conjugate vaccines against different definitions of pneumonia.^14–18 However, accurate population-level assessments of pneumococcal conjugate vaccines against pneumonia are needed, as these capture both the direct effects of the vaccine and positive and negative population-level effects such as herd immunity and strain replacement, respectively.^19–21 Wide variability in published estimates of vaccine impact against pneumonia highlights the difficulties with the interpretation of such analyses and the need for more robust statistical methods.^22–24

Report on outcome specificity

Assessment of the population-level impact of pneumococcal conjugate vaccines against pneumonia is challenging. While invasive pneumococcal disease is, by definition, caused by pneumococcus, pneumonia can be caused by many different pathogens. Identifying a causative agent for pneumonia is complex and has a high failure rate when attempted, making it difficult to determine the fraction of pneumonia cases caused by pneumococcus.²⁵ This issue substantially complicates both RCTs and population-level studies of vaccine impact against pneumonia.^22,26 Many different clinical endpoints and outcome definitions have been used to assess the efficacy and monitor the impact of pneumococcal conjugate vaccine against pneumonia; however, because these definitions differ in specificity for pneumococcus, one must be cautious when comparing results between studies.²⁶ Pneumonia definitions based on simple clinical diagnostic methods (e.g. assessing history of coughing or difficult breathing, respiratory rate, and presence of lower chest wall indrawing) can capture most pneumonia cases and thus are highly sensitive.²⁷ These definitions, however, are not specific for pneumococcus.²⁶ Definitions requiring X-ray observation of lower airway consolidation, infiltrate or pleural effusion are enriched for bacterial pneumonias and thus are more specific for pneumococcus.^25,26,28

The specificity of the case definition used in an RCT has a substantial influence on the estimated efficacy of pneumococcal conjugate vaccine against pneumonia.²⁶ For instance, an RCT from Latin America estimated an efficacy of 21.8% (95% confidence interval [CI] 7.7%, 33.7%) against WHO-defined consolidated pneumonia, 10.0% (95% CI 1.7%, 17.7%) against radiologically confirmed community-acquired pneumonia, and 8.7% (3.8, 13.4%) against clinically-suspected community-acquired pneumonia.²⁹

Many retrospective analyses of vaccine impact, however, are based on time-series of outcomes defined by diagnostic codes assigned during hospitalization. Such definitions can differ substantially from the endpoints used in RCTs and can differ between settings, making comparisons between studies highly problematic. ^{7,22,23,30,31} Several observational studies of pneumococcal conjugate vaccine impact on pneumonia based on hospitalization data have reported impact estimates notably higher than those from RCTs assessing efficacy against WHO-defined severe pneumonia - a diagnosis warranting immediate hospitalization.^22,26,31 Although indirect effects that occur after widespread use of the vaccine could explain some of this discrepancy, the specificity for pneumococcus of ‘all-cause pneumonia’ definitions used in observational studies is expected to be markedly lower than for the primary endpoints used in RCTs, indicating potential confounding or bias in the observational studies. Thus, to improve comparability between studies, care should be taken to report the specificity of the chosen outcome definition in relation to previously published studies, and, where possible, to perform the same analysis using several different outcome definitions that vary in specificity. For other vaccines, analogous issues surround the definition of diarrhea used for studies of rotavirus vaccine impact, or the grading of cervical lesions in studies of HPV vaccine impact.

Apply bias-reducing model selection procedures

Trend studies of vaccine impact require that an analyst make numerous decisions about how to model the data. These decisions include selecting which years to analyze in the ‘pre’ and ‘post’ vaccine periods, whether to exclude data from the vaccine introduction period, which modeling strategy to employ, how to control for trends unrelated to the vaccine (including seasonality), and whether to estimate the rate of decline of the disease of interest or to just report an average post-vaccine estimate of incidence rate ratio. Many published studies, however, only report results from a single model structure, often with little justification or clarification as to whether alternative models produced different estimates. This is problematic because there will always be different epidemiologically plausible ways to describe the changes in the data that occur around the time of a vaccine introduction, and these alternative models will sometimes yield different results.³² Reporting the results from only one model leaves room for both errors and bias. It does not tell the reader whether other models were considered, what criteria were used for discarding alternatives and how the results from the alternative models differed from those reported. Furthermore, only considering one or very few models means that the sensitivity of the result to the model structure remains untested and allows potential systematic errors to persist.

To address these problems, we suggest a three-step approach for model selection. First, investigators should select, a priori, a range of epidemiologically plausible models for the data. Second, a specific model selection criterion (such as the Akaike Information Criterion) should be chosen to weight the alternative models.³³ Third, results from these alternative models should be reported; if appropriate, estimates from these alternative models should be averaged to obtain a single estimate (such as by using Bayesian model averaging).³⁴ Following these simple guidelines will reduce bias while increasing reproducibility and transparency.

Assess the likelihood of detecting a vaccine effect

Unlike in prospective epidemiologic studies, in retrospective trend studies little thought is typically given to power and sample size considerations. Such analyses can play an important role in determining the likelihood of detecting a true vaccine-related change in disease rates from a given time series. Simple data simulations can help by empirically determining how key characteristics of the time series, such as average numbers of cases, numbers of years of data, and variability between years, influence the accuracy of the estimated effect of the vaccine. Obtaining estimates from simulated time series allows the analyst to determine the variability in estimates that might be expected from a time series with particular characteristics. The exercise also makes the analyst consider a priori what the relationship between epidemiologically plausible vaccine effects and the data available for analysis might be. Simulation studies can also be used to assess the quality of evidence from different published studies, such as when performing meta-analyses or looking for the presence of publication bias.

Integrate multiple comparison outcomes to address unmeasured bias and confounding

Vaccine impact studies attempt to determine how disease rates change after the introduction of a vaccine into a population. However, establishing causality—that the vaccine caused the change—is challenging.^35,36 Even in studies that have appropriate case definitions and have examined alternative model structures, there is still a risk for unmeasured confounding and bias affecting the results and interpretation. It has become common for pneumococcal conjugate vaccine impact studies to attempt to address this problem by analyzing one or a few diseases for comparison. If the outcome of interest (e.g. pneumonia) declines more than the comparison outcome(s), this is often used to strengthen the argument that the difference can be attributed to the vaccine.^22,32

Lipsitch et al. (2010) have described the characteristics by which negative control outcomes can be chosen to effectively detect unmeasured confounding and bias in an observational study.³⁷ They argue that a control outcome should be affected by the same set of causal relations as the main outcome of interest but should not be affected by the exposure of interest. In the case of observational studies of the impact of pneumococcal conjugate vaccines against pneumonia, this would mean that an ideal negative control should be influenced by the same factors as pneumonia but should not be affected by the vaccine.

In practice, applying this principle is very challenging. It is not clear how an investigator can assess whether the comparison outcome and the outcome of interest in fact share the same set of (generally unknown) causal relations, nor is it clear how to tell whether the comparison outcome is itself truly unaffected by the introduction of the vaccine.³⁷ Moreover, it is unclear to what extent the comparison outcomes used in previously published studies approximate the criteria of an ideal negative control. This can be problematic if unmeasured confounding is overlooked or assumed absent as a result of these imperfect comparisons. For example, hospitalizations for dehydration might not share many common risk factors with hospitalizations for all-cause pneumonia. In contrast, bronchiolitis might seem to be a better choice because this outcome shares more causal relations with pneumonia.^22,23 However, there is evidence that pneumococcal conjugate vaccines can affect the risk of hospitalization for bronchiolitis; if this occurs in the study population, using bronchiolitis as the comparison outcome would bias the estimates. ³⁸

We therefore recommend using many comparison outcomes rather than just choosing one or even a few. The different comparison outcomes can then be weighted based on their ability to predict the outcome of interest (e.g. pneumonia) in the pre-vaccine period, forming a single composite outcome termed a ‘synthetic control’.^39–41 Bayesian frameworks exist in econometrics, which can be adapted to vaccine impact studies and which will allow an unbiased weighting of all available comparison outcomes into a synthetic control.³⁵ Naturally, such an approach rests on its own assumptions, but we believe this method has the potential to provide more robust conclusions from comparison outcomes and that it deserves more attention in epidemiology.⁴²

Conclusion

Observational studies of vaccine impact will, by the nature of their respective designs, always be more prone to systematic errors, confounding, and bias than RCTs. These issues are particularly important for assessments of the impact of vaccines against non-specific disease categories such as the impact of pneumococcal conjugate vaccine against pneumonia, rotavirus vaccine impact against diarrhea or human papilloma virus vaccine impact against high-grade cervical lesions. However, the suggestions made here have the potential to improve the accuracy of population-level vaccine impact studies in general.