Estimating Influenza Vaccine Effectiveness With the Test-Negative Design Using Alternative Control Groups: A Systematic Review and Meta-Analysis

Abstract

One important assumption in case-control studies is that control selection should be independent of exposure. Nevertheless, it has been hypothesized that virus interference might lead to a correlation between receipt of influenza vaccination and increased risk of infection with other respiratory viruses. We investigated whether such a phenomenon might affect a study design commonly used to estimate influenza vaccine effectiveness (VE). We searched publications in MEDLINE, PubMed, and Web of Science. We identified 12 studies using the test-negative design (2011–2017) that reported VE estimates separately derived by 3 alternative control groups: 1) all patients testing negative for influenza (FLU), VE_FLU−; 2) patients who tested positive for other/another respiratory virus (ORV), VE_ORV+; and 3) patients who tested negative for all viruses in the panel (PAN), VE_PAN−. These included VE estimates from 7 countries for all age groups from 2003/2004 to 2013/2014. We observed no difference in vaccination coverage between the ORV-positive and PAN-negative control groups. A total of 63 VE_FLU− estimates, 62 VE_ORV+ estimates, and 33 VE_PAN− estimates were extracted. Pooled estimates of the difference in VE (ΔVE) were very similar between groups. In meta-regression, no association was found between the selection of control group and VE estimates. In conclusion, we did not find any differences in VE estimates based on the choice of control group.

Influenza vaccines are the most effective measure available for reducing the substantial annual disease burden associated with Influenzavirus infections. Influenza vaccines generally have moderate-to-good efficacy, estimated to fall within the range of 60%–70% in children and 50%–60% in adults based on randomized controlled trials (1, 2). However, influenza vaccine efficacy and vaccine effectiveness (VE) can vary from year to year depending on the degree of antigenic match between strains selected for inclusion in the vaccine and circulating strains, as well as the intervals between vaccination and influenza epidemics (3). VE can also vary among subpopulations—persons of different ages, for example (4, 5). These factors may affect both experimental and observational study designs. Thus, annual estimates of influenza VE can provide ongoing evidence on the performance of influenza vaccines in the community.

Although randomized controlled trials are considered the gold standard for measuring vaccine efficacy, for the purpose of making annual VE estimates, observational studies are carried out more commonly because of feasibility, efficiency, and ethics. The test-negative design (TND) has been widely applied for measuring influenza VE on a routine basis in Europe, North America, Australia, and Asia since 2005 (6, 7). In this study design, patients with signs and symptoms meeting predefined clinical definitions (e.g., acute respiratory infection or influenza-like illness) are swabbed and tested for influenza viruses. In some cases, testing may be done against a panel of respiratory viruses. Persons testing positive are defined as cases, while those testing negative for influenza viruses (influenza (FLU)-negative) are classified as controls. Persons who test negative may test positive for another respiratory virus (other/another respiratory virus (ORV)-positive) or may test negative for all viruses in the panel (panel (PAN)-negative). VE is estimated as VE% = 1 − OR_adj, where OR_adj is the odds ratio comparing the odds of vaccination among cases with the odds of vaccination among the controls, adjusted for potential confounders such as age. This design can be embedded within existing surveillance systems to enable timely estimation of VE in both inpatient and outpatient settings at a reasonable cost (8, 9).

Although the number of studies using the TND has been increasing, the theoretical underpinnings and inherent assumptions of the TND need further evaluation (10, 11). As a type of case-control study, the TND should follow the basic principles of control selection: 1) controls should be selected from the same population as cases and 2) controls should be selected independently of exposure, within each stratum of factors included in stratified analysis (12). By restricting cases and controls attending outpatient clinics or hospitals to the same clinical case definition (e.g., influenza-like illness (9)), the TND includes controls from the same source population as cases, and thus reduces bias from differential health-care-seeking behavior.

However, TND studies conducted in inpatient settings could be complicated by acute exacerbation of chronic respiratory conditions, which could be difficult to distinguish from respiratory virus infections requiring hospitalization. Furthermore, if this population were more likely to receive influenza vaccination than the source population, selection bias would be introduced. Nevertheless, in a meta-analysis comparing inpatient and outpatient VE, Feng et al. (13) observed similar VE estimates by setting when the analysis was restricted to the same season, country, and age group. In a simulation study examining the validity of the TND for inpatient settings, Foppa et al. (14) concluded that VE estimates would be biased if chronic respiratory conditions were not well-controlled in the analysis. Under this assumption, vaccination coverage and VE estimates might differ according to different control groups; that is, higher vaccination coverage, and thus higher VE estimates, would be observed with the use of PAN-negative controls.

An assumption of case-control studies is that control selection should be independent of exposure. For influenza VE studies, this assumption may be violated if the risk of infection with a noninfluenza virus is not independent of vaccination status (15, 16). In a randomized controlled trial, Cowling et al. (15) observed increased risk of ORV infection among 2008–2009 trivalent influenza vaccination recipients. Several epidemiologic studies have observed “viral interference,” a phenomenon in which infection by one virus alters susceptibility to infection by another virus. This has been reported, for example, for influenzavirus and rhinovirus (17) and influenzavirus and respiratory syncytial virus (RSV) (18).

The purported underlying mechanism behind these phenomena might involve both nonspecific immunity and influenzavirus-specific immunity (19). After a viral infection, temporary nonspecific immunity against a second infection is induced, which could last for several weeks (20). Given its brief duration, investigators might be unlikely to observe a population or cohort effect. In contrast, the influenzavirus-specific interference may involve T-cell-mediated immunity, which may last for months because of broad heterotypic cross-reactivity. The immune response might vary across different influenza strains and could possibly be more evident during an influenza pandemic wherein specific influenza strains predominate during the season (19). Although the nonspecific and specific immunity against influenza and other infectious diseases has been broadly described, the potential nature and biological mechanisms of virus interference remain unclear (21).

For the TND, virus interference may be important if, by preventing influenza infection through vaccination, nonspecific and influenzavirus-specific immunity is not induced and leaves the vaccinee susceptible to infection by cocirculating viruses he or she might otherwise have been protected from. This would result in higher influenza vaccination coverage among controls with ORVs detected than in controls with no virus detected. Therefore, selection of controls could then lead to biased estimation of VE; that is, the VE estimates derived using the ORV-positive group would be higher than those for the PAN-negative group and the FLU-negative group. Simulation studies have suggested that while this phenomenon can produce biased estimates, that bias is trivial except under extreme conditions (22). However, simulations are often simplistic representations of real studies and may not be able to capture the nuances of immunological phenomena. For example, immunological responses may be less apparent in the elderly than in children or young adults, possibly because of immunosenescence (23). Thus, any evaluation of the potential bias caused by virus interference should consider the age of the population.

In this review, we aimed to assess whether virus interference could affect VE estimates generated from studies using the TND. First, we compared vaccination coverage using alternative control groups. Second, we compared VE estimates according to the choice of control group. Third, we summarized the total difference in VE estimates and assessed deviations from zero. Finally, we assessed whether any differences in VE were equally apparent among all age groups and in different study settings.

METHODS

Study search and selection

We previously reviewed TND studies that estimated influenza VE (6, 8, 13). For the current review, we reassessed all papers derived from our previous searches (last performed on December 28, 2015 (13)) and conducted an online update on April 18, 2017. Following the previous search strategies, papers were searched on PubMed (National Library of Medicine, Bethesda, Maryland), MEDLINE (National Library of Medicine), and Web of Science (Clarivate Analytics, Philadelphia, Pennsylvania) for the following combination of keywords: 1) “influenza” or “flu”; 2) “vaccine effectiveness” or “VE”; 3) “test-negative” or “test negative” or “case-control” or “case control”; and 4) sets 1, 2, and 3. Articles were independently screened by 2 of the authors (S.F. and S.G.S.). Studies estimating influenza VE for any season, any influenza type/subtype, or any type of influenza vaccination by VE were considered. Only articles published in English were considered. Studies or subanalyses of studies which reanalyzed previously published data, reported interim estimates, or did not use the TND were excluded. All studies meeting these inclusion criteria were further screened and were included if influenza VE was estimated using alternative control groups, including FLU-negative, ORV-positive, and/or PAN-negative controls.

Study design information was extracted for each included study using a standardized form. This included: author, publication year, study country, influenza season, population ages, study setting, surveillance system, case definition, time intervals since symptom onset, type of swab, laboratory methods, ORVs tested, vaccination coverage in each control group, covariates included in statistical models, and all VE estimates using alternative control groups.

Assessment of potential influence of virus interference

Differences in influenza vaccination coverage and estimates of VE against influenza A or B according to alternative control groups were compared by paired t test. Adjusted VE estimates obtained using all available control groups were extracted, including the FLU-negative group (denoted VE_FLU−), the ORV-positive group (denoted VE_ORV+), and the PAN-negative group (denoted VE_PAN−). Stratified VE estimates were also extracted by influenza type/subtype for each age group, influenza season, and setting (inpatient or outpatient). For each study, the differences in available VE estimates (ΔVE) were defined as:

$$\mathrm{\Delta }{\mathrm{VE}}_{\mathrm{FLU}-,\mathrm{ORV}+}{=\mathrm{VE}}_{\mathrm{FLU}-}-{\mathrm{VE}}_{\mathrm{ORV}+}.$$

$$\mathrm{\Delta }{\mathrm{VE}}_{\mathrm{FLU}-,\mathrm{PAN}-}{=\mathrm{VE}}_{\mathrm{FLU}-}-{\mathrm{VE}}_{\mathrm{PAN}-}.$$

$$\mathrm{\Delta }{\mathrm{VE}}_{\mathrm{ORV}+,\mathrm{PAN}-}{=\mathrm{VE}}_{\mathrm{ORV}+}-{\mathrm{VE}}_{\mathrm{PAN}-}.$$

We calculated 95% confidence intervals for each ΔVE by bootstrap, using 1,000 resamples (see the Web Appendix, available at https://academic.oup.com/aje). We excluded studies with large uncertainty, as defined by a 95% confidence interval range for VE_FLU− of more than 100 percentage points. The remaining studies were pooled, and estimates of ΔVE_FLU−,ORV+, ΔVE_{FLU−,PAN−}, and ΔVE_ORV+,PAN− against influenza A or B were calculated. Where studies provided both overall estimates and estimates for subgroups, we removed the overall estimate to avoid any overlap. We performed Egger’s test on ΔVE_FLU−,ORV+ and its standard error to assess publication bias. A fixed-effects model was assumed, and heterogeneity was examined by means of the I² statistic and Cochran’s Q test. The inverse of the variance of ΔVE_FLU−,ORV+ was used to weight the studies.

To test our hypothesis that virus interference may vary by age, we further estimated ΔVE_FLU−,ORV+ by age group—that is, children (ages 6 months–17 years), young adults (ages 18–49 years), and older adults (ages ≥50 years). Since inpatient studies may be biased due to recruitment of patients with chronic respiratory disease rather than viral infection, we also examined ΔVE by setting, whenever possible. We evaluated whether pooled ΔVE_FLU−,ORV+ differed from zero among children or young adults more than among elderly adults. To further examine whether any study design feature was associated with VE estimates, we also conducted meta-regression by means of univariate and multivariate random-effects models. The predictors included age group (children, adults, elderly, and all ages), study setting (inpatient/outpatient/mixed), season (single/multiple), restriction of patients to those presenting for health care within 4 days (yes/no), exclusive use of polymerase chain reaction (PCR)/reverse transcription PCR for testing (yes/no), number of ORVs tested, and type of control group (FLU-negative, ORV-positive, or PAN-negative). All analyses were conducted using R, version 3.3.1 (R Foundation for Statistical Computing, Vienna, Austria) and the metafor package.

RESULTS

A total of 120 publications were retrieved from the previous study (6, 8, 13), and 35 articles were obtained from the new search. Based on 155 test-negative studies, 12 articles that met the inclusion criteria were identified (24–35) (Figure 1). Two studies reported estimates from the same population (24, 27). Kelly et al. (27) estimated VE for children recruited from both a hospital emergency department (ED) and general practices in 2008, with estimates reported separately by setting; Blyth et al. (24) conducted analysis for children recruited from the same ED from 2008 to 2012. To avoid this overlap, the ED results reported by Kelly et al. were excluded.

Figure 1.

Identification of eligible studies that used the test-negative design (TND) to estimate influenza vaccine effectiveness (VE) by means of alternative control groups. FLU, influenza; ORV, other/another respiratory virus; PAN, panel.

The first study estimating VE using alternative control groups was a 2011 study from Australia (27). Study characteristics are summarized in Table 1. The studies covered the 2003/2004 to 2013/2014 influenza seasons and 7 countries, including Australia (4 studies (24, 27, 28, 31)), the United States (3 studies (26, 32, 34)), Japan (1 study (33)), China (Hong Kong; 1 study (25)), Portugal (1 study (29)), New Zealand (1 study (30)), and the Netherlands (1 study (35)) (Table 1). Five studies reported estimates for a single year (27, 29–31, 33), while others reported estimates across multiple years (24–26, 28, 32, 34, 35). Among 7 studies reporting pooled estimates across more than 1 year/season, 3 studies also provided season-specific estimates (28, 34, 35).

Table 1.

Design Features of 12 Studies Selected to Analyze Whether Virus Interference Can Affect Vaccine Effectiveness Estimates Generated From Test-Negative Designs, 2011–2017

First Author, Year (Reference No.)	Country	Season	Age Group	Setting	Case Definition	Interval Since Symptom Onset, days	Type of Swab	Laboratory Method(s)	Other Respiratory Viruses Tested
Blyth, 2014 (24)a	Australia	2008–2012b	6–59 months	Outpatient	ILI	≤4	NP	PCR, VC, IF	RSV, PIV 1–4, hMPV, RV, AdV (B–D), CoV, BoV, EV
Cowling, 2014 (25)	China (Hong Kong)	2009–2013	6 months–17 years	Inpatient	ARI	N/A	NP	IF, VC, RT-PCR	RSV, PIV 1–3, AdV
Feng, 2017 (34)	United States	2010–2013	All ages	Outpatient	ARI	≤7	NP, OP, nasal	RT-PCR	RSV, PIV 1–3, hMPV, RV, AdV
Grijalva, 2015 (26)	United States	2010–2012	All ages	Inpatient	Hospital admission with pneumonia	≤3	NP, OP	RT-PCR	RSV, PIV 1–3, hMPV, RV, CoV
Kelly, 2011 (27)a	Australia	2008	6–59 months	Outpatient (general practice and emergency department)c	ILI	≤3	Nasal	VC, RT-PCR	RSV, PIV 1–3, hMPV, RV, EV
Levy, 2014 (28)	Australia	2010–2012	All ages	Outpatient	ILI	≤4	Nasal, throat	VC, PCR, RT-PCR	RSV, PIV 1–3, hMPV, RV, AdV, EV
Nunes, 2014 (29)	Portugal	2012–2013	All ages	Outpatient	ILI	≤7	NP, OP	VC, RT-PCR	RSV, PIV 1–3, hMPV, RV, AdV
Pierse, 2016 (30)	New Zealand	2014	All ages	Inpatient and outpatientc	SARI/ILI	≤7	NP, throat	RT-PCR	RSV, PIV 1–3, hMPV, RV, AdV
Sullivan, 2014 (31)	Australia	2012	All ages	Outpatient	ILI	N/A	Nasal	RT-PCR	RSV, PIV 1–3, hMPV, AdV, EV
Sundaram, 2013 (32)	United States	2004/2005–2009/2010	6 months–5 years, ≥50 years	Outpatient, urgent-care departments, and acute-care hospitals	ARI	<10 and ≤7	Nasal, NP	RT-PCR	RSV, PIV 1–4, hMPV, RV, AdV, CoV
Suzuki, 2014 (33)	Japan	2011–2012	>10 years	Outpatient	ILI + RIDT	≤5	NP	PCRs	RSV, PIV 1–4, hMPV, RV, AdV, CoV, BoV
van Doorn, 2017 (35)	The Netherlands	2003–2014	All ages	Outpatient	ARI or ILI	≤7	Nasal, throat	RT-PCR	RSV, RV, EV (varied by season: PIV 1–4, hMPV, CoV, ADV)

Abbreviations: AdV, adenovirus; ARI, acute respiratory infection; BoV, bocavirus; CoV, coronavirus; EV, enterovirus; hMPV, human metapneumovirus; IF, immunofluorescence; ILI, influenza-like illness; N/A, not applicable; NP, nasopharyngeal; OP, oropharyngeal; PCR, polymerase chain reaction; PIV, parainfluenza virus; RIDT, rapid influenza diagnostic test; RSV, respiratory syncytial virus; RT-PCR, reverse transcription polymerase chain reaction; RV, rhinovirus; SARI, severe acute respiratory infection; VC, viral culture.

^a Results reported by Kelly et al. (27) for children presenting to an emergency department in 2008 were removed from further analysis.

^b Data for 2009 were not included.

^c Separate estimates are provided for each setting.

Patients of all ages were investigated in 7 studies (26, 28–31, 34, 35), children in 4 studies (24, 25, 27, 32), and children, adults, and elderly persons in 2 studies (30, 34). Eight studies were carried out exclusively in outpatient settings (24, 27–29, 31, 33–35), including 1 in an ED (24); 2 were carried out in inpatient settings (25, 26); 1 used both inpatients and outpatients, with estimates broken down by setting (30); and 1 recruited outpatients (including ED patients), urgent-care patients, and acute-care patients (32). Various case definitions were identified. Some studies used an influenza-like illness definition (n = 6 studies), others an acute respiratory infection definition (n = 3); one used either an influenza-like illness or acute respiratory infection definition, while another required hospitalized pneumonia (n = 1); and 1 Japanese study required use of rapid influenza detection tests. The case definitions used are summarized in Web Table 1. All studies included fever or history of fever in the clinical case definition, with the exception of 1 study which enrolled hospitalized pneumonia patients (26). Various restrictions were applied to the inclusion of patients based on the interval between symptom onset and clinical presentation, including presentation within 4 days, 7 days, and 10 days; 2 studies did not specify such a restriction (Table 1).

All studies estimated VE using both FLU-negative and ORV-positive controls. Seven also reported VE estimates obtained using PAN-negative controls (25, 26, 29, 32–35). All studies performed PCR/reverse transcription PCR for influenza diagnosis among all patients or in a subset of patients. Some studies also used immunofluorescence and/or virus culture (24, 25, 27, 28) (Table 1). The number of ORVs included in the respiratory panels ranged from 3 to 8. All included studies tested for RSV. Other commonly included viruses were human metapneumovirus (n = 9 studies), adenovirus (n = 8), and rhinovirus (n = 9) (Table 1). Only 2 studies reported the number of specimens testing positive for each ORV tested (33, 34). In these 2 studies, the most commonly detected ORVs were RSV and rhinovirus, accounting for 66.5% (34) and 73.1% (33) of total ORVs detected, respectively. There was a weak negative correlation between the number of ORVs included in the panel and the proportion of patients identified as PAN-negative (r = –0.37), but this was not statistically significant (P = 0.13). The proportion of controls identified as PAN-negative was lowest among children: 17.3% in 1 US study (32), 26.8% in 1 Australian study conducted in an ED (24), and 21.3% from another Australian study conducted in an outpatient setting (27) (Table 2). The proportion of patients testing positive for ORVs in the FLU-negative group ranged from 13.5% among people aged ≥10 years in Japan (33) to 80.2% among children aged 6–59 months in the United States (32). The interquartile range was 27.5%–45.5% for all studies (Table 2).

Table 2.

Vaccination Coverage Estimated Using Alternative Influenza-Negative Control Groups, 2011–2017

First Author, Year (Reference No.)	Setting	Age Group	Season	Type of Control Group									Difference (ORV+, PAN−), %a	PAN−, %b	ORV+/FLU−, %c
				FLU-Negative			ORV-Positive			PAN-Negative
				No. of Persons Vaccinated	Total No.	% Vaccinated	No. of Persons Vaccinated	Total No.	% Vaccinated	No. of Persons Vaccinated	Total No.	% Vaccinated
Blyth, 2014 (24)	Outpatient	5–59 months	2008–2012d	128	1,200	10.7	85	794	10.7	43	406	10.6	0.1	26.8	66.2
Feng, 2017 (34)	Outpatient	All ages	2010–2011	957	2,784	34.4	422	1,176	35.9	528	1,591	33.2	2.7	37.8	42.2
Feng, 2017 (34)	Outpatient	All ages	2011–2012	721	1,692	42.6	343	701	48.9	375	984	38.1	10.8	45.5	41.4
Feng, 2017 (34)	Outpatient	All ages	2012–2013	907	2,430	37.3	379	958	39.6	526	1,461	36.0	3.6	34.2	39.4
Kelly, 2011 (27)	Outpatient	5–59 months	2008	43	63	68.3	34	47	72.3	9	16	56.3	16.0	21.3	74.6
Levy, 2014 (28)	Outpatient	All ages	2010	71	302	23.5	27	89	30.3	44	213	20.7	9.6	47.5	29.5
Levy, 2014 (28)	Outpatient	All ages	2011	58	246	23.6	11	66	16.7	47	180	26.1	–9.4	51.3	26.8
Levy, 2014 (28)	Outpatient	All ages	2012	177	758	23.4	40	191	20.9	137	567	24.2	–3.3	41.7	25.2
Nunes, 2014 (29)	Outpatient	All ages	2012–2013	38	183	20.8	20	70	28.6	18	113	15.9	12.7	33.7	38.3
Pierse, 2016 (30)	Outpatient	All ages	2014	144	677	21.3	59	299	19.7	85	378	22.5	–2.8	32.8	44.2
Sullivan, 2014 (31)	Outpatient	All ages	2012	218	821	26.6	77	313	24.6	141	508	27.8	–3.2	35.9	38.1
Sundaram, 2013 (32)	Outpatient, UCDs, and ACHs	6 months–5 years	2004/2005–2009/2010	1,014	1,759	57.6	782	1,411	55.4	232	348	66.7	–11.3	17.3	80.2
Sundaram, 2013 (32)	Outpatient, UCDs, and ACHs	≥50 years	2004/2005–2009/2010	937	1,359	68.9	439	659	66.6	498	736	67.7	–1.1	42.3	47.2
Suzuki, 2014 (33)	Outpatient	>10 years	2011–2012	66	193	34.2e	12	26	46.2e	54	167	32.3e	13.9	54.0	13.5
van Doorn, 2017 (35)	Outpatient	All ages	2003–2014	579	2,754	21.0	142	676	21.0	437	2,078	21.0	0	51.3	24.5
Cowling, 2014 (25)	Inpatient	6 months–17 years	2009–2013	428	4,737	9.0	107	1,185	9.0	321	3,552	9.0	0	65.8	25.0
Grijalva, 2015 (26)	Inpatient	All ages	2010–2012	766	2,605	29.4	368	1,196	30.8	398	1,409	28.2	2.6	50.9	45.9
Pierse, 2016 (30)	Inpatient	All ages	2014	267	735	36.3	57	248	23.0	210	487	43.1	–20.1	46.8	33.7

Abbreviations: ACHs, acute-care hospitals; FLU, influenza; ORV, other/another respiratory virus; PAN, panel; UCDs, urgent-care departments.

^a Defined as vaccination coverage in the ORV-positive group minus vaccination coverage in the PAN-negative group.

^b Percentage of participants who were PAN-negative among influenza-negative participants.

^c Percentage of participants who were ORV-positive among influenza-negative participants.

^d Data for 2009 were not included.

^e There were 4 subjects in the ORV-positive control group and 18 subjects in the PAN-negative control group with missing data on vaccination status. Persons with missing vaccination status were not excluded; their vaccination status was categorized as “unknown,” and they were included in the analysis.

Vaccination coverage

Vaccine coverage was reported by all studies for the various control groups considered. In addition, vaccine coverage by age group and control group was calculated for each study, where sufficient detail was provided. Because Pierse et al. (30) reported results by study setting (inpatient and outpatient), Feng et al. (34) and Levy et al. (28) reported results by season, and Sundaram et al. (32) reported results by age group, we were able to calculate estimates for a total of 18 triplets from 12 studies (Table 2). The vaccination coverage estimated from the FLU-negative group ranged from 9.0% among inpatient children aged 6 months–17 years in Hong Kong, China, in 2009–2013 (25) to 68.3% among outpatient children aged 6 months–5 years in Australia in 2008 (27). The differences in vaccination coverage between the ORV-positive and PAN-negative groups ranged from −28.4% to 16.0% (Table 2). However, the paired t test comparing mean vaccination coverage between these groups suggested no statistical difference (P = 0.61). Of 18 differences in vaccination coverage point estimates, 6 were higher than 10%. We did not identify any age- or setting-specific pattern larger or smaller than 10% by univariate analysis.

Difference in VE (ΔVE)

We extracted 63 VE_FLU− estimates, 62 VE_ORV+ estimates, and 33 VE_PAN− estimates for further comparison. The distributions of VE_FLU− and VE_ORV+ estimates are shown in Figure 2A, while the distributions of VE_FLU− and VE_PAN− estimates are shown in Figure 2B. We observed VE estimates to be correlated and mostly distributed close to the identity lines. Among the 63 VE_FLU− estimates, 38 estimates were for persons of all ages, 18 were for children, and 4 were specifically for older adults; 44 were estimates of VE against influenza A or B viruses, while others were calculated for a specific influenza type/subtype (Web Figures 1 and 2). All studies estimated VE after adjusting for potential confounders, including age, and 9 in 12 adjusted for calendar time as a proxy for changing influenza activity. We calculated ΔVE_FLU−,ORV+, ΔVE_{FLU−,PAN−}, and ΔVE_ORV+,PAN− from each available estimate. We did not find evidence of publication bias by assessing ΔVE_FLU−,ORV+ and its standard error (P = 0.64). No statistically significant differences were observed for any of the ΔVE estimates except those reported by van Doorn et al. (35).

Figure 2.

Comparison of influenza vaccine effectiveness (VE) estimated using influenza (FLU)-negative controls (VE_FLU−) with VE estimated using other/another respiratory virus (ORV)-positive controls (VE_ORV+) (A) and comparison of VE estimated using influenza-negative controls (VE_FLU−) with VE estimated using panel (PAN)-negative controls (VE_PAN−) (B) from all available VE estimates. Dotted lines represent the identity line. Ninety-five percent confidence intervals are shown in gray (point estimates and lower confidence limits below −110% are not shown).

After restriction of the VE_FLU− estimates analyzed to those with confidence intervals spanning 100 percentage points or less, 48 VE_FLU− estimates, 48 VE_ORV+ estimates, and 24 VE_PAN− estimates remained for further analysis (Web Figure 1). The 48 point estimates of ΔVE_FLU−,ORV+ ranged from –43% to 18%. Forty of these estimates had an absolute value of less than 10% (Web Figure 1). The VE estimates excluded from meta-analysis are shown in Web Figure 2. In the meta-analysis, we removed pooled ΔVE estimates if ΔVE estimates by age stratum/influenza season were available (n = 21) and estimated pooled ΔVE_FLU−,ORV+ against influenza A or B viruses. I² and Cochran’s Q test implied no concerning heterogeneity (I² = 0, Q(20 df) = 8.4, P = 0.99). The pooled ΔVE_FLU−,ORV+ was –4% (95% confidence interval (CI): –10, 2) as estimated from 21 pairs of differences in VE estimates (ΔVE_FLU− and ΔVE_ORV+), consistent with no substantial difference for VE estimates between the FLU-negative and ORV-positive groups. Similarly, the pooled estimate for ΔVE_{FLU−,PAN−} was –1% (95% CI: –8, 5) (n = 13), and that for ΔVE_ORV+,PAN− was 5% (95% CI: –2, 12) (n = 13). Web Figure 2 shows the confidence intervals of VE_FLU− estimates spanning over 100 percentage points.

We further conducted meta-analysis on ΔVE_FLU−,ORV+ estimates for children (7 ΔVE estimates), adults (3 ΔVE estimates), and elderly (2 ΔVE estimates). The pooled ΔVE_FLU−,ORV+ was 0% (95% CI: –9, 8) for children, –4% (95% CI: –21, 14) for young adults, and 1% (95% CI: –20, 22) for older adults. We were not able to identify any trend among age groups. We also performed sensitivity analysis by restricting VE estimates to specific influenza types/subtypes (H1N1, H3N2, and B) and settings (inpatient or outpatient). The pooled estimate was –2% (95% CI: –19, 15) from 3 ΔVE_FLU−,ORV+ estimates for inpatient settings and –6% (95% CI: –13, 1) from 17 ΔVE_FLU−,ORV+ estimates for outpatient settings. We did not observe a pooled ΔVE_FLU−,ORV+ that was statistically different from zero.

Meta-regression

We performed meta-regression to explore factors that may contribute to VE using univariate and multivariate random-effects models. In univariate models, VE estimates were higher if the estimate was for children (P = 0.001), persons of all ages (P < 0.001), or inpatients (P = 0.04) (Web Table 2). No association was observed between VE estimates and choice of control group. Similarly, we did not observe an interaction between age group and selection of control group (P > 0.05; results not shown). In the multivariate regression, we also did not observe a significant association between the choice of control group and VE (P_(ORV+) = 0.24, P_(PAN−) = 0.93) (Web Table 2).

DISCUSSION

Based on 12 studies identified from 155 TND publications providing 63 VE_FLU− estimates, 62 VE_ORV+ estimates, and 33 VE_PAN− estimates, we did not find any statistical differences in VE by type of control group. The paired t test assessing vaccination coverage also did not demonstrate any difference by choice of control group. Although we observed a ΔVE_FLU−,ORV+ less than zero reported in 1 study, the difference was not consistently observed in each season (35). The pooled ΔVE_FLU−,ORV+ estimate for 21 pairs of observations was –4% (95% CI: –10, 2), suggesting that the choice of control group is unlikely to significantly affect VE estimated using the TND.

The nonspecific immune response to influenzavirus involves the activation of natural killer cells, macrophages, and dendritic cells, the immune functions of which are known to decrease with age (23). Therefore, we assumed that the phenomenon of virus interference may be more prevalent among younger, rather than older, age groups. However, our analysis did not strongly support this hypothesis; although age group was correlated with VE estimates, we did not observe an interaction between any age groups and choices of control groups in multivariate meta-regression. We acknowledge that our sample size was small, and as the number of studies examining this phenomenon increases, evidence for virus interference may arise.

Virus interference may act differently by type/subtype. For example, interference between RSV and influenza A may be more pronounced than interference between RSV and influenza B (36; Dr. Karen Laurie, WHO Collaborating Center for Reference and Research on Influenza at the Peter Doherty Institute for Infection and Immunity, Melbourne, Victoria, Australia, personal communication, 2016). However, we identified no statistically significant ΔVE estimates among types/subtypes. Again, our analysis of these effects was limited by the sample size.

Selection bias may arise from inpatient studies if patients hospitalized due to chronic underlying conditions other than infection are also more likely to be vaccinated. In this study, we identified 3 studies reporting inpatient results and did not observe any such trend. The pooled VE estimates by setting did not differ from zero.

We found that the proportion of PAN-negative controls varied substantially among studies. These differences could partially be explained by the number of ORVs tested. However, we only observed a nonsignificant, weak, negative correlation between the number of ORVs tested and the proportion PAN-negative. The types of viruses included in the panel may also affect the proportion PAN-negative. For example, in the 2 studies reporting the detection rates of each ORV, rhinovirus and RSV were most commonly detected. Only 1 study from Hong Kong (25) did not include rhinovirus, which could explain why this study had the highest proportion PAN-negative. The differences in proportion PAN-negative are likely to also be associated with viral load and shedding. With the exception of the Hong Kong study (25), the studies involving children presented low proportions of PAN-negative subjects among all eligible subjects, which is consistent with observations of higher viral load and shedding from children (37). The heterogeneity of viral shedding by age suggests potential misclassification bias in the PAN-negative group, where results may be more likely to be false-negative among samples taken from older patients. Other reasons for false-negative influenza results may be associated with suboptimal swab quality, imperfect laboratory testing methods, or long intervals from symptom onset to presentation (11). All of these factors could contribute to misclassification of infection status and contribute to larger ΔVE_ORV+,PAN− values. Under this circumstance, the ORV-positive group could be considered the one providing more accurate VE estimates, because it demonstrates that the swab and swabbing method were of sufficient quality to detect virus (28).

In conclusion, based on 12 studies estimating VE using alternative control groups, we did not find evidence of virus interference, suggesting that VE estimates obtained by means of the TND are not biased by virus interference. Using FLU-negative controls is likely to produce VE estimates that are as reliable as those of ORV-positive and PAN-negative controls. From a resource-saving perspective, investigators in surveillance systems or research schemes using the TND to measure influenza VE may consider not testing further for ORVs when estimating influenza VE. Further simulation studies that incorporate multiple sources of bias and examine this phenomenon in different age groups could help confirm or refute our findings.

Abbreviations

CI

confidence interval

ED

emergency department

ORV

other/another respiratory virus

PCR

polymerase chain reaction

RSV

respiratory syncytial virus

TND

test-negative design

VE

vaccine effectiveness