Publishing interim early or mid-season influenza vaccine effectiveness – US Flu VE Network

Abstract

Introduction:

Mid-season influenza vaccine effectiveness (VE) estimates are a useful tool to help guide annual influenza vaccine strain selection, vaccine policy, and public health messaging. We propose using a sample size-driven approach with data-driven inputs for publication of mid-season influenza VE.

Methods:

We used pooled inputs for VE by (sub)type and average vaccine coverage by age groups using data from eight seasons of the US Influenza VE Network to calculate sample sizes needed to estimate mid-season VE.

Results:

We estimate that 135 influenza-positive cases would be needed to detect an overall VE of 40% with 55% vaccine coverage among test-negative controls. Larger sample sizes would be required to produce reliable estimates specifically against influenza A/H3N2 and for older age groups.

Conclusion:

Using an existing network, most of the recent influenza seasons in the US would facilitate valid mid-season VE estimates using the proposed sample sizes for broad age groupings.

Background

In order to optimally contribute to the annual influenza strain selection and to guide vaccination policy, estimates of influenza vaccine effectiveness (VE) must be both timely and reliable. Annual VE estimates may influence two primary decision-making bodies relevant to the United States (US) influenza vaccination program. First, the US Advisory Committee on Immunization Practices (ACIP) reviews emerging data on influenza on an ongoing basis and makes annual recommendations for the prevention and control of seasonal influenza with vaccines. Second, the World Health Organization (WHO) convenes technical consultations each February and September to select vaccine reference strains for the annual northern and southern hemisphere influenza vaccines, respectively¹. Thus, one of the primary aims of Centers for Disease Control and Prevention’s (CDC) US Influenza Vaccine Effectiveness (Flu VE) Network is to calculate and provide to US ACIP and the WHO estimates of VE soon after the influenza season begins (i.e., interim early/mid-season) and again at the end of the season. Ideally, these estimates would be available before the February and September WHO strain selection meetings and at the time of the ACIP meetings that occur in February, June, and October. The timely needs of the strain selection committee and the annual policy recommendations must be balanced with the scientific validity of the effectiveness estimates. We propose use of a sample size-driven approach with data-driven inputs for publication of age- and subtype-specific stratified interim influenza VE estimates.

Methods

We developed a sample-size based approach for analysis and reporting of interim VE estimates from the US Flu VE Network using inputs from historical network data generated during the 2011–12 through 2018–19 influenza seasons.

Data source

Details of the US Flu VE Network have been published previously^2–9. The US Flu VE Network conducts a prospective study of outpatients seeking care for acute respiratory illness (ARI) during the annual influenza season (typically November–April). Annually during this period, the Network sites enroll approximately 8,000–15,000 participants from ~50–66 ambulatory care facilities (comprising primary care clinics, urgent care clinics, and emergency departments) associated with healthcare institutions in five U.S. states: Michigan, Pennsylvania, Texas, Washington, and Wisconsin. Trained study staff enroll consenting persons aged ≥6 months who sought outpatient care for ARI. All participants were tested for influenza virus infection for research purposes by molecular assays, and participants were classified as influenza-positive (cases) or influenza-negative (controls). Study staff collected epidemiological, clinical, vaccination history and vaccine type data from structured interviews, health system medical record extraction, and state immunization information systems.

Pooled vaccine effectiveness estimates

We used previously published end-of-season vaccine effectiveness estimates from the US Influenza Vaccine Effectiveness (Flu VE) Network by influenza season and for influenza A/H3N2, A/H1N1pdm09, and B-lineage viruses to calculate pooled estimates across seasons from 2011–12 through 2018–19^2–9. The “metafor” package was used in R version 3.4.2 to generate pooled VE estimates and plots.

Sample size estimates

We were interested in sample size estimates for 1) VE against medically attended ARI from any influenza virus; 2) VE against medically attended ARI from A/H3N2, A/H1N1pdm09, and B-lineage viruses; and 3) VE for any influenza virus and specific virus types or strains stratified by age. We calculated the number of influenza cases that would be needed to detect VE ranging from 15–70% with corresponding vaccine coverage ranging from 20%–85% using standard formulas^{10, 11}. We set desired power to 80%, alpha to 0.05, and assumed three controls per case with no matching (Supplemental Table 1). Sample sizes were calculated using a SAS Macro (courtesy of Paul Gargiullo, available upon request). We applied the subtype-specific pooled VE for all ages to each age stratum and varied vaccine coverage as observed across seasons. Pooled VE and vaccine coverage among controls were rounded and used as inputs to determine the number of cases required for each stratum. We present sample size estimates (rounded to the nearest multiple of 5 for ease of implementation) for five age groups (i.e., 6 months – 8 years, 9–17 years, 18–49 years, 50–64 years, ≥65 years) that are relevant to US influenza vaccine policy as well as three broader age groups (6 months –17 years, ≥18 years, and ≥50 years) that could be utilized in the event that sample sizes requirements are not met for narrower age groups.

Comparison of interim and final vaccine effectiveness

We compared published interim^12–18 and end-of-season^3–9 VE estimates from the US Flu VE Network from 2011–12 through 2018–19 seasons against sample size estimates based on pooled data. We calculated VE for each season using data on persons enrolled through the week ending after the proposed minimum number (i.e., 135) of influenza-positive patients were enrolled. Methods for calculation (i.e., subject inclusion, model adjustment) mirrored those used for each published estimate.

Results

Pooled VE estimates and vaccination rates

Pooled VE across eight seasons from the US Flu VE Network was 41% (95%CI: 33 to 48) against any influenza, 29% (95%CI: 18 to 38) against A/H3N2, 52% (95%CI: 44, 59) against A/H1N1pdm09, and 51% (95%CI: 46, 56) against influenza B virus (Supplemental Figure 1a–d). Average vaccine coverage among influenza test-negative controls was 55% and varied by age group from 40% among persons aged 9–17 years to 80% among persons aged ≥65 years (Table 1).

Table 1.

Number of influenza-positive cases needed¹ for calculating adjusted vaccine effectiveness (VE) by age group and influenza (sub)type

		Any Influenza		Influenza A/H3N2		Influenza A/H1N1pdm09		Influenza B
Age Stratum	Vaccine Coverage2	Pooled VE	N cases	Pooled VE	N cases	Pooled VE	N cases	Pooled VE	N cases
All ages	55	40	135	30	275	50	75	50	75
6 months–17 years	50	40	140	30	275	50	80	50	80
6 months–8 years	55	40	135	30	275	50	75	50	75
9–17 years	40	40	155	30	300	50	90	50	90
≥18 years	60	40	135	30	275	50	75	50	75
18–49 years	45	40	145	30	285	50	85	50	85
≥50 years	70	40	150	30	305	50	80	50	80
50–64 years	60	40	135	30	275	50	75	50	75
≥65 years	80	40	180	30	380	50	95	50	95

¹Calculated assuming 80% power, α = 0.05, and 3 controls per case

²Average influenza vaccine coverage among influenza test-negative controls over eight influenza seasons of the United States Influenza Vaccine Effectiveness ( Flu VE) Network

Sample size estimates based on pooled data

When considering ARI from any of the influenza viruses, we estimate that 135 influenza cases would be needed to detect an overall VE of 40% with an average 55% vaccine coverage among test-negative controls (Table 1).

Influenza subtype-specific estimates are crucial and vary due to factors such as mismatch between vaccine strains and circulating viruses or host factors. Owing to the lower VE against A/H3N2 viruses during the 8-season study period (i.e., pooled VE of 29%), a larger number of cases (N=275) would be required to generate VE against A/H3N2. In contrast, a smaller sample size would be required for A/H1N1pdm09 (N=75) or influenza B (N=75) viruses (Table 1).

Due to variations in VE and vaccination rates by age, we derived age-stratified sample size estimates. In the oldest age groups of interest, required sample sizes are quite large due to the intersection of lower vaccine effectiveness and much higher vaccine coverage. We estimate that 380 cases would be required to detect VE of 30% against influenza A/H3N2 in those aged ≥65 years (Table 1).

Comparison of past season VE estimates against sample size estimates based on pooled data

The US Flu VE Network published mid-season VE estimates in six out of the eight influenza seasons from 2011–12 to 2018–19 (Figure 1). Mid-season estimates were not published in the 2011–12 or 2015–16 influenza seasons, due to low and late influenza activity and late onset of activity, respectively. In the 2012–13 season, two interim estimates (early and mid-season) were published prior to final estimate publication. The number of cases included in VE analyses ranged from 416–1712 at interim publication. The proportion of total cases in a season included in interim analyses ranged from 17%–69% (median 55%). Broadly, mid-season estimates were similar to final estimates; final estimates were more precise (i.e., confidence interval widths were smaller). In all but one instance (2017–18), the mid-season VE against any influenza was higher than the final estimate. In every mid-season estimate, the number of cases used for calculation of VE exceeded the proposed threshold for VE against any influenza for all persons eligible for vaccination (i.e., 135 cases).

Figure 1.

Comparison of interim and final vaccine effectiveness estimates against any influenza from the United States Influenza Vaccine Effectiveness (Flu VE) Network, 2012–13 through 2018–19 seasons¹.

¹ Mid-season estimates were not published in the 2011–12 or 2015–16 influenza seasons.

On average, the proposed minimum number of cases for estimation of mid-season VE were enrolled approximately 3–4 weeks prior to the actual enrollment cut-off date used for published mid-season estimates (Supplemental Table 2). If VE had been estimated as soon as this criterion was met, estimates would have been similar but less precise than what was published in each season (Supplemental Figure 2). However, in the 2014–15 season, in which genetically distinct A/H3N2 viruses predominated, generating a VE estimate as soon as the proposed minimum sample size was met would have yielded an estimate higher (56%) than what was calculated a month later (23%) with approximately ten times as many cases; 95% confidence intervals for the estimates overlapped.

Discussion

We propose a strategy for publication of interim early or mid-season estimates in the US Flu VE Network using a sample size-based approach with inputs from prior influenza seasons. Pooled influenza (sub)type-specific VE over eight seasons in the US Flu VE Network of 30–50% is consistent with meta-analyses of influenza vaccine effectiveness¹⁹. Lower vaccine effectiveness against influenza A/H3N2 and higher vaccine coverage in the older age groups were two major drivers of larger sample sizes needed to produce reliable estimates. We find that most of the recent influenza seasons in the US would facilitate publication of interim vaccine effectiveness estimates for at least broad age groups prior to pertinent policy meetings in late February using existing research platforms. However, virus-specific estimates among persons aged ≥65 years would require larger samples.

The decision to publish interim influenza VE must consider the balance of timeliness and reliability of estimates. While we show that, within the US Flu VE Network, estimates generated quickly after the minimum sample size for cases were enrolled would have resulted in estimates being published about a month earlier than what occurred, precision and reliability were gained by allowing cases to accrue 3–4 weeks longer. Published interim estimates of VE against any influenza from the US Flu VE Network were similar to final published estimates that included enrollment throughout the influenza season. Our network findings align with previously published reports from other studies comparing interim and end-of-season vaccine effectiveness²⁰. Average vaccine coverage in the US Flu VE Network is higher than other estimates of influenza vaccine coverage in the US, but the enrolled sample represents a group seeking outpatient medical care for acute respiratory illness whereas other published estimates are intended to represent the general US population⁸. Furthermore, we used vaccine coverage observed at the end of the influenza season for our calculations because over 90% of annual influenza vaccinations in the US Flu VE Network occur prior to January²¹. In other studies in which a larger proportion of vaccines are received later in the season, early season vaccine coverage might be a more appropriate input for sample size calculations. While we compared interim VE against any influenza among all persons eligible for influenza vaccination and found consistency with final estimates, stratification of data into age groups, by influenza (sub)types, and/or by type of influenza vaccine would likely be less stable at early/mid-season. Publication of end-of-season estimates will still be necessary to investigate issues related to specific viral genetic groups, age strata, or vaccine types or products, drift after the interim estimates, or late waves.

While we took a sample size-driven approach, other approaches could also be considered. One such approach might be to apply Bayesian methods. Given that the number of influenza VE estimates published globally has increased since the test-negative design became widely utilized, there is now ample data available to apply as priors. Second, given the observation that final, end-of-season estimates tend to be lower than early or interim estimates, one might apply a calculation assuming a time-dependent bias that gradually reduces observed VE over the season resulting in a higher sample size needed for early estimates that more accurately approximate final estimates. Other possible methods, such as a precision-based approach to limit the width of confidence intervals or to rely on an expected number of cases given a certain vaccine effectiveness, could also be utilized in place of what we propose. However, we prefer the sample-size approach simplified methodology while preserving reliability of estimates. Lastly, while we present a sample size-driven approach, situations may require complementing this data-driven approach with practical considerations relevant for public health such as emerging novel influenza viruses with no known prior estimates of VE.

Using the proposed data-driven method that draws on findings from eight recent influenza seasons will strengthen future publications of interim VE generated by the US Flu VE Network and will allow for more transparency in the presentation of findings with decision-making entities such as ACIP and WHO.