Estimation of Relative Vaccine Effectiveness in Influenza: A Systematic Review of Methodology

Abstract

Background:

When new vaccine components or platforms are developed, they will typically need to demonstrate noninferiority or superiority over existing products, resulting in the assessment of relative vaccine effectiveness (rVE). This review aims to identify how rVE evaluation is being performed in studies of influenza to inform a more standardized approach.

Methods:

We conducted a systematic search on PubMed, Google Scholar, and Web of Science for studies reporting rVE comparing vaccine components, dose, or vaccination schedules. We screened titles, abstracts, full texts, and references to identify relevant articles. We extracted information on the study design, relative comparison made, and the definition and statistical approach used to estimate rVE in each study.

Results:

We identified 63 articles assessing rVE in influenza virus. Studies compared multiple vaccine components (n = 38), two or more doses of the same vaccine (n = 17), or vaccination timing or history (n = 9). One study compared a range of vaccine components and doses. Nearly two-thirds of all studies controlled for age, and nearly half for comorbidities, region, and sex. Assessment of 12 studies presenting both absolute and relative effect estimates suggested proportionality in the effects, resulting in implications for the interpretation of rVE effects.

Conclusions:

Approaches to rVE evaluation in practice is highly varied, with improvements in reporting required in many cases. Extensive consideration of methodologic issues relating to rVE is needed, including the stability of estimates and the impact of confounding structure on the validity of rVE estimates.

Vaccines have greatly reduced the burden of many infectious diseases. Although randomized trials are required to evaluate a new vaccine, once that vaccine is approved and licensed it will be subject to postmarketing surveillance using observational studies.¹ A variety of observational study designs have been used to monitor vaccine effectiveness (VE). Estimation of causal effects such as VE^2,3 can be challenging in observational studies because of the potential for confounding.⁴ The most commonly used study design for estimation of VE is the test-negative design (TND), in which a single clinical case definition is used for enrollment of participants and laboratory testing is subsequently employed to classify each patient into either the case or control group.^5–7 A recent review identified 348 articles using the TND for monitoring VE of 12 pathogens.⁸

When new vaccine components or platforms are developed, they will typically need to demonstrate noninferiority or superiority over existing products. In these studies, the effectiveness of the new vaccine is to be compared with the existing vaccine to estimate the relative vaccine effectiveness (rVE). After licensure, other relative comparisons may also be of interest, such as VE by time since vaccination if there is a concern over waning VE^9,10; VE by prior vaccination status^11,12; VE by vaccine brand or platform¹³; or VE by genetic clade or subgroup of the pathogen^14,15; all of which will involve estimating a relative effect of one or more vaccines.

As the vaccine development landscape continues to advance, we expect to see an increased focus on relative vaccine comparisons. In particular, the National Institute for Allergy and Infectious Diseases (NIAID) has constructed a strategic plan to support research to develop new and improved vaccines for influenza.¹⁶ Furthermore, as of 18 August 2021, 19 vaccines for coronavirus disease 2019 (COVID-19) have been licensed for either emergency or full use globally,^17–35 with a further 15 currently in phase 3 development,^36–50 including one that will be compared with the conditionally approved vaccine Vaxzevria, rather than placebo,⁵¹ and one that will investigate the effect of the Comirnaty vaccine in those who have already received one dose of Vaxzevria.⁵² Furthermore, studies with multiple relative comparisons are emerging which evaluate effectiveness with respect to prior infection, time since vaccination and different vaccine platforms.⁵³ In the coming months and years, assessment of the relative effectiveness of these vaccines as well as the comparative effect of individual vaccines against different emerging viral variants will be critical. Moreover, quantifying the relative effect of the vaccine at one time point versus another will be a priority for understanding waning in vaccine-derived immunity. Estimates of rVE therefore have a crucial role to play in policy making, most imminently for COVID-19 but also more routinely for seasonal pathogens such as influenza. Consequently, it is essential that rVE estimates are valid and procedures for obtaining these are standardized.

A first step to aid interpretation and make methodologic recommendations for future rVE estimates is identification of how relative vaccine comparisons are conducted in practice. Therefore, the objectives of this study are as follows: to review how rVE is evaluated for influenza vaccines; to identify common categories of rVE comparisons within influenza; to provide a summary of the different estimands and estimation techniques employed; to determine the confounding structures assumed; and to assess the bias and consistency in these estimates. We anticipate the findings will have implications for the ongoing assessment of vaccines for influenza, COVID-19, and other vaccine-preventable diseases, as well as relative comparisons of other interventions.

METHODS

Search Strategy

We followed the PRISMA guidelines when conducting this review and considered studies in any language. A systematic search was carried out on PubMed, Google Scholar, and Web of Science on August 18, 2021, using the following search term:

1. “vaccine” OR “vaccination”

2. “relative effectiveness” OR “relative efficacy”

3. #1 AND #2

4. “relative vaccine effectiveness” OR “relative vaccine efficacy” OR “relative VE”

5. #3 OR #4

6. “vaccine effectiveness” OR “VE”

7. #6 AND “waning”

8. #5 OR #7

We included the “waning” term in the search criteria as an additional option to allow for the inclusion of studies estimating effectiveness of a single vaccine at one time point relative to another that may not be identified using only terms #1 to #6. We also screened the reference lists of retrieved articles to identify any additional eligible studies.

Screening

We initially screened the articles identified in the search strategy to eliminate duplicates and then MMM and HB independently screened the remaining titles for relevance. We defined rVE studies as any providing estimates for the comparative effectiveness or efficacy of two or more vaccine components (e.g., egg-based versus cell-based vaccines), doses, or vaccination schedules directly. We included studies assessing rVE indirectly, for instance through a network meta-analysis, provided they estimated a relative effect.

We excluded studies that estimated rVE for pathogens other than influenza virus, those that focused on treatments rather than vaccines, cost-effectiveness of vaccines, comparative effect of vaccine uptake determinants, articles providing an overview of the research landscape, simulation studies and animal or immunogenicity studies. We excluded studies that only estimated absolute VE or assessed waning by estimating VE of the vaccine against an unvaccinated control group at different time points. We also excluded studies that did not conduct any inference or those that estimated rVE but did not mention the statistical methods used. We included secondary analyses and meta-analyses if the studies provided additional rVE estimates or novel methods for assessing rVE. We excluded any papers conducting interim analyses when the final analysis was available.

Data Extraction and Analysis

MMM and HB extracted data from included articles after the full text screening using a standardized form. We extracted information on season, the types of comparison made, study design, sample size, endpoint, rVE definition, and statistical models or methods used for estimation. We also recorded the rVE estimate, the variables that were adjusted for, or those used for matching or stratification, whether absolute VE estimates were reported, and if any multiple testing corrections were applied (if applicable).

We classified vaccine comparisons according to three main categories: component, dose, and timing or history. This accounted for comparing different vaccine components, the same components with different brand names, the same vaccine with additional booster dose, the same vaccine across a dosing schedule (i.e., the time between vaccinations or cases in which multiple vaccinations are required within in a season), vaccination history, or whether VE was compared in two different time intervals to assess waning; all of which were considered relative effects. MMM and HB carried out a bias assessment using the “risk of bias in nonrandomized studies” (ROBINS-I) tool for observational studies, to assess the risk of bias classified as “low,” “moderate,” “serious,” “critical,” or “no information” across seven domains.⁵⁴ They assessed randomized studies using the “risk of bias in randomized trials” (ROB-2) tool, where bias risk was classified as “low,” “some concerns,” or “high” across five domains.⁵⁵

RESULTS

Included Studies

Our search on PubMed, Google Scholar, and Web of Science (WOS) resulted in 356 articles in the first instance. We removed six duplicates and identified five additional relevant publications from screening the reference lists of published articles. After screening, we identified 63 articles that met criteria for inclusion (Figure 1).^56–118 Relative VE was estimated for either differing vaccine components (n = 38), dosing schedules (n = 17), or vaccination timing or history (n = 9), with one study assessing a range of components and doses, as detailed in the Table. We found 86% (n = 54) of relative VE studies have been conducted in the last decade, with 49% (n = 31) published in the last 2.5 years.

FIGURE 1.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram for studies identified from a PubMed search with search term: (((“vaccine”) OR (“vaccination”)) AND ((“relative effectiveness”) OR (“relative efficacy”))) OR ((“relative vaccine effectiveness”) OR (“relative vaccine efficacy”) OR (“relative VE”)) OR (((“vaccine effectiveness”) OR (“VE”)) AND (“waning”)), which were sorted as indicated.

TABLE.

Relative Comparisons Considered in Included Studies, Classified by Pathogen as Either Vaccine Component, Vaccination Dose, or Vaccination Timing/History

Category	Comparison	Number	Citations
Component	LAIV vs. TIV/IIV	13	71,76,85,91,93,102–104,107–110,115
	Cell cultured vs. egg-based	10	57–60,64,67,72,81,114,116
	Adjuvanted vs. nonadjuvanted	14	60,63,69,72,77,79,95–97,99,100,111–113
	Other	3	90,105,106
Dose	High-dose vs. standard-dose	17	56,61,88,89,92,98,101,117,118,62,66,70,78,79,82,86,87
Timing or history	Semiannual vs. annual	1	73
	One season vs. consecutive seasons	3	75,83,84
	Intraseason waning	5	65,68,74,80,94

Note that some studies have more than one type of comparison.

IIV indicates inactivated influenza vaccine; LAIV, live attenuated inactivated vaccine; TIV, trivalent inactivated vaccine.

Study Design and rVE Definition

We found a range of study designs were used to compare relative effectiveness including retrospective cohort (n = 25), randomized controlled trial (RCT) (n = 22), systematic review and meta-analysis (n = 7),^{56,71,75,84,91,103,111} TND (n = 6),^{61,64,67,76,80,101} case-control (n = 2),^69,85 and prospective cohort (n = 1).⁹⁹ It is important to note that some of the rVE studies included in the review are also incorporated in the seven meta-analyzed estimates. This is owing to the primary aim being to identify rVE methods, including pooled estimands, rather than to draw conclusions on the interventions themselves.

Relative vaccine effectiveness was reported as a percentage in the majority of studies (n = 52) and was otherwise reported as a ratio (n = 11), as shown in Figure 2. The most commonly used definition of relative vaccine effectiveness was (1 – IRR) x100 (n = 27) where IRR denotes the incidence rate ratio for one group versus the other. Other studies reported rVE using only the rate ratio (n = 4),^82,88,99,110 prior event rate ratio (PERR) (n = 3)^62,63,78 or instrumental variable-adjusted (IVadj) rate ratio (n = 1).⁷⁰ The cluster-randomized trial estimated rVE by estimating the IRR of hospitalization for influenza and pneumonia in residents randomized at the facility level (n = 1).⁸² Other definitions included (1 – HR) x100 (n = 7),^{77,93,95,97,115,117,118} (1 – OR) x100 (n = 10),^{57–59,61,64,67,69,76,101,114} OR (n = 5),^{65,74,80,85,94} and $\left(\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$\text{OR}$}\right.\right)\text{X}100$ (n = 1),⁶⁸ where HR is the hazard ratio and OR is the odds ratio. The majority of systematic reviews and meta-analyses reported rVE as a meta-analyzed summary measure, including (1 – RR_pool) x100 (n = 1)⁸⁴; (1 – OR_pool) x100 (n = 1)⁵⁶; OR_pool (n = 1)⁷¹ and pooled change in VE (n = 1).⁷⁵ One study reported a pooled estimate combining ORs and IRRs.¹¹¹

FIGURE 2.

Relative outcome estimated by study design. HR, hazard ratio; IRR, incidence rate ratio; IRR_pool, pooled rate ratio; IV_adj, instrumental variable adjusted; OR, odds ratio; OR_pool, pooled odds ratio; OR&IRR_pool, combined odds ratio and rate ratios pooled; PERR, prior event rate ratio; VE, vaccine effectiveness.

Two studies provided a reference to justify the definition used.^112,116 However, the cited publication had not provided any justification.⁷² Overall, approximately one-third (n = 18, 29%) of the included studies reported absolute VE in addition to the relative vaccine effectiveness (see Figure 3). A total of 83% (n = 5) of test-negative studies reported the absolute VE, compared with only 4% (n = 1)⁵⁹ of retrospective cohort studies.

FIGURE 3.

Number of studies reporting absolute vaccine effectiveness (VE) in addition to relative vaccine effectiveness across a range of identified designs, including randomized controlled trial (RCT), cohort, test-negative design (TND) case-control, and systematic review and meta-analysis.

Approaches to Estimation

Age was the most commonly controlled variable (n = 41, 65%) as both vaccination status and risk of infection change with age. Comorbidities (n = 29, 46%), geographic region (n = 29, 46%), and sex (n = 25, 40%) were also commonly accounted for. Ethnicity (n = 16, 25%), healthcare seeking proxies (n = 13, 21%), calendar time (n = 13, 21%), previous vaccination status (n = 11, 17%), and time of vaccination (n = 8, 13%) were among the other characteristics considered in analyses. These variables were either specified as a covariate in the regression model or were included in the calculation of propensity score matching. However, approximately one-third (n = 18, 29%) of studies either did not include or did not report the covariates considered in analyses.

The estimation of the rate ratio, OR, and HR estimands use a number of different statistical models and techniques. For estimators based on the rate ratio, models used included inverse probability weighted Poisson regression (n = 7),^{60,72,79,88,112,113,116} standard Poisson regression (n = 3),^82,100,110 and nested Poisson regression models (n = 1).⁶⁶ Other approaches included Cox proportional hazards (n = 2),^89,96 Andersen-Gill (n = 3),^103,105,106 log-binomial models (n = 1),⁹¹ logistic regression (n = 1),⁸³ and generalized estimating equations (n = 1).⁹⁹ In studies comparing observed proportions in each group, Fishers exact test was used (n = 6)^{73,90,102,104,108,109} and confidence intervals were constructed using the method of Clopper-Pearson (n = 3),^87,92,98 Blackwelder (n = 1),⁸⁶ Farrington-Manning (n = 1),⁸¹ and Guess et al (n = 1).¹¹⁹ Studies estimating the PERR used a Poisson regression including an interaction term between period and treatment (n = 3),^62,63,88 and multivariable instrumental variable Poisson regression was used to estimate the IV–adjusted IRR (n = 1).⁷⁰ Meta-analyses estimating pooled effects used a random effects model (n = 2),^71,75 with DerSimonian-Laird estimators for the OR (n = 2),^56,111 or a log-binomial model with study included as a fixed effect (n = 1).⁸⁴

Studies basing rVE estimates on the HR employed Cox proportional hazards (n = 4)^93,95,97,118 with treatment as main effect and stratifying covariates as random effects (n = 2)^77,115 or Fine-Gray subdistribution hazard models (n = 1).¹¹⁷ For OR estimators, rVE was estimated using logistic regression (n = 10),^{58,61,64,67,68,76,80,85,94,101} conditional logistic regression (n = 3),^65,69,74 inverse probability weighted logistic regression (n = 1),¹¹⁴ Cox regression (n = 1),⁵⁹ and doubly robust inverse probability of treatment weights (n = 1).⁵⁷ The incidence rates across nursing homes were modeled using marginal Poisson regression with log of resident days as an offset term (n = 1).⁸²

Bias Assessment

We assessed 34 observational studies using the ROBINS-I tool. Overall, we classified 65% (n = 22) studies to be at “moderate” risk of bias, 18% (n = 6)^{60,67,72,88,110,117} as having “serious” risk, and the remaining 18% (n = 6)^{59,63,64,74,93,118} did not provide sufficient information to assess. The individual domain assessments are shown in Figure 4. We deemed all studies to be at low risk of bias with respect to deviations from the planned interventions, and all studies were either at low or moderate risk of bias due to selection, classification of intervention, or reporting of results. We deemed one study to be at serious risk of bias with respect to confounding and measurement of outcomes.¹¹⁰ Bias due to missing data was most common with over a quarter of studies (n = 9)^{59,63,64,68,74,79,93,110,118} not providing sufficient information and we classified bias in 15% (n = 5)^{60,67,72,88,117} of studies as serious.

FIGURE 4.

Bias assessment of (A) 34 nonrandomized studies using ROBINS-I and (B) 19 randomized studies using ROB-2 bias assessment tool.

We assessed 19 of the 22 RCTs included using the ROB-2 tool, as four publications related to different subgroup results from the same overall trial.^83,86,87,92 We found 26% of studies (n = 5)^{73,77,98,108,115} were at low risk of bias, deemed 47% of studies (n = 9)^{82,90,96,100,104–107,109} to have some concerns of bias, and classified the remaining 26% (n = 5)^{81,83,95,97,102} studies as at high risk of bias overall. The measurement of outcome domain was at the highest risk of bias with three studies being classified as high risk.^81,83,95 One study was classified as high risk with respect to deviations from the intervention and missing data.⁹⁷ All domains had studies with some concerns of bias relating to deviations from intended interventions (n = 7)^{96,102,104–107,109}; concerns about the randomization (n = 6)^{82,90,100,102,105,106}; missing data (n = 3)^90,95,107; measurement of outcomes (n = 2)^100,102; and reporting (n = 1).⁸²

Stability of rVE

In the subset of studies presenting both aVE and rVE, 12 studies reported aVE for both vaccines.^{59,61,64,67,71,76,100,101,104,108,109,115} Figure 5 shows the aVE of one vaccine versus aVE of the other vaccine included in the relative comparison, with the majority of studies supporting the hypothesis of proportionality in the vaccine effects. The practical consequences of this can be shown via two scenarios. In scenario 1, assume 3 subjects vaccinated with vaccine A, 5 subjects vaccinated with vaccine B, and 13 subjects unvaccinated, out of a total of 36 subjects. This results in an aVE of vaccine A of 76.9%, an aVE of vaccine B of 61.5%, and an rVE of B versus A of 40%. In scenario 2, assuming 12 subjects are vaccinated with vaccine A, 20 subjects vaccinated with vaccine B, and 23 subjects unvaccinated. This would result in an aVE of vaccine A of 47.8% and an aVE of vaccine B of 13%, which also translates to an rVE of 40%.

FIGURE 5.

Absolute VE of vaccine A on y axis versus absolute VE of vaccine B on x axis in the subset of studies presenting both absolute and relative effects for adjuvanted vs. nonadjuvanted; cell-based vs. egg-based; high-dose vs. standard-dose and live attenuated vs. inactivated vaccine.

DISCUSSION

We identified and reviewed the methodology used to estimate rVE in 63 influenza VE studies. Over half of the identified studies compared the relative effect of two or more vaccine components, one-third focused on comparing doses or dosing schedules of one vaccine, and the remainder estimated the relative effect of vaccine timing or vaccination history. The majority of studies reported relative VE as a percentage. However, we observed substantial variation in the definitions and approaches employed, often with no justification provided for the chosen approach. This reflects the fact that little methodologic consideration has been given to the topic, resulting in a lack of available recommendations, which investigators can use to inform their analysis.

Across all study designs, the majority of rVE estimators were based on either a rate ratio, OR, or HR. Extensions of standard models were used in some studies to address limitations. For example, six studies used inverse probability treatment weighting to address potential biases resulting from issues with confounding and missing data. However, it is worth noting that these methods may not always outperform standard multivariable analysis in dealing with confounding and only balance with respect to observed rather than unobserved covariates.^120,121 One study performed an instrumental variable-adjusted analysis, which aims to estimate a causal effect even if all confounding variables have not been measured and accounted for in the analysis.^122,123 This could be a useful technique for estimating rVE, as the confounding structure is typically not as well understood and problems with self-selection in vaccination can be addressed.¹²⁴ However, identifying an instrumental variable, particularly in observational studies, remains challenging hence limiting its application.^125,126 The use of Cox proportional hazards models was most common for estimation of rVE based on HRs. To obtain valid rVE estimates, it is important for investigators to assess the assumption of proportional hazards in their context. If it is unlikely to hold, we suggest an extension of the model, such as the Andersen-Gill method, which allows for multiple events and time-varying covariates.¹²⁷ This approach may also be preferred for smaller sample sizes, as including only the time of the first event would result in more imprecise effect estimates. In most cases, the best choice of model and framework will be study-dependent and factors such as the population of interest should be carefully considered at the design stage. A summary of the potential considerations is provided in eTable 1 (http://links.lww.com/EDE/B907) in the eAppendix (http://links.lww.com/EDE/B907). Despite many study-dependent considerations, it remains important to standardize the definitions of rVE and reporting of rVE studies, to aid future comparisons and meta-analyses. Furthermore, it is important to ensure considerations relating to confounding structure and other potential biases in the assessment of rVE has been identified and investigated to identify shortfalls in existing methodology and approaches.

We found that only 29% of studies measured or reported a comparison with a nonvaccinated group, that is, by either conducting a primary estimation of absolute VE or reporting the assumed VE estimates, resulting in 71% of studies reporting relative effectiveness only. The omission of a comparison with an unvaccinated group has some important implications for interpretation, which were not highlighted or discussed in the majority of papers. Importantly, relative effect estimates tell us nothing about the absolute effectiveness of each individual vaccine and therefore whether the effect will translate to an impact on public health and policy. For example, two scenarios highlighted in the text result in an rVE of 40%; in scenario one the aVE is 61.5% versus 76.9% as opposed to scenario two, which is 13.1% versus 47.8%. Clearly these two scenarios would have differing effects on public health and so it is vital for policy makers to have both absolute and relative effect estimates. However, in reality, it is likely that many studies cannot feasibly incorporate an unvaccinated control group. For example, the unvaccinated group used in COVID-19 vaccine studies will change considerably over time and will be confounded by prior infection, which may be poorly documented. Hence, where absolute effectiveness cannot be estimated, authors should summarize and report the assumed individual efficacies obtained from previous studies in similar populations to provide the necessary context for the reader.

The aim of VE studies is to derive an estimate of a causal effect, not merely an association.³ When done so in an observational setting, it is well established that obtaining valid estimates of causal effects requires identifying and controlling for confounding variables, such as age and comorbidities, which can be associated with both vaccination status and risk of infection. However, in the case of relative comparisons between two vaccinated groups, the confounding structure is not as well understood. This is reflected in our findings in that only 65% of studies adjusted for age compared with 97% of studies assessing VE with an unvaccinated group as the comparison.⁸ We suggest that accounting for demographics, comorbidities, and other factors relevant to absolute VE estimation is still crucial to ascertain valid estimates of relative effectiveness, as it is plausible that factors, such as age, may still be associated with both the exposure and outcome. For instance, if the comparison of interest is the relative effect of influenza vaccination early versus late in the season in preventing influenza-like illness, then age is likely to be associated with both given that older people are prioritized to receive vaccination early in the season. If a substantial proportion of the 29% of studies not reporting adjustment for covariates did not consider confounding factors, then this may have resulted in substantial biases in the existing literature for relative vaccine effects. However, our bias assessment shows only one study was at serious risk of bias due to confounding.¹¹⁰ Further consideration of the importance of confounding on the validity of causal interpretations of relative vaccine estimates is warranted.

Although we aimed to identify all possible comparisons relating to VE through our specified search criteria, it is possible due to the nonstandardized terminology for assessing rVE that we have missed the inclusion of some studies. One possible subset of studies not identified is the relative effect of a single vaccine on one genetic subgroup versus another. We expect this category of comparison to increasingly feature in the rVE literature, as the effectiveness of emerging vaccines on one SARS-CoV-2 variant versus another will be important to establish. In addition, as the review focused on rVE methods used in practice, we did not include in the Results those methods proposed in simulation or model-based studies. By considering the relevant simulation studies, we identified an additional 10 studies with substantial variability in rVE estimands used. It will be important to incorporate these more novel proposals in future comparisons of statistical properties of methods for rVE.

This review highlighted the lack of consideration given to methodologic and practical implications of relative vaccine effectiveness estimation within the literature. Based on our findings, we recommend better reporting of rVE studies to include the definition of rVE used, absolute VE either estimated within the study or assumed VE stated, discussion of confounding structure and what confounders will be included in the model along with how they are accounted for (i.e., adjusted, matched), and where relevant, multiple testing corrections were included and discussed. Extensive consideration of methodologic issues relating to rVE is needed, including estimand used and the impact of confounding structure on the validity and stability of rVE estimates. These issues should be investigated theoretically and empirically to improve the quality of evidence on rVE.

ACKNOWLEDGMENTS

The authors thank Julie Au for administrative support.