Bias of influenza vaccine effectiveness estimates from test-negative studies conducted during an influenza pandemic

Test-negative (TN) studies have become the most widely used study design for the estimation of influenza vaccine effectiveness (VE) and are easily incorporated into existing influenza surveillance networks. We seek to determine the bias of TN-based VE estimates during a pandemic using a dynamic probability model. The model is used to evaluate and compare the bias of VE estimates under various sources of bias when vaccination occurs after the beginning of an outbreak, such as during a pandemic. The model includes two covariates (health status and health awareness), which may affect the probabilities of vaccination, developing influenza and non-influenza acute respiratory illness (ARI), and seeking medical care. Specifically, we evaluate the bias of VE estimates when (1) vaccination affects the probability of developing a non-influenza ARI; (2) vaccination affects the probability of seeking medical care; (3) a covariate (e.g. health status) is related to both the probabilities of vaccination and developing an ARI; and (4) a covariate (e.g. health awareness) is related to both the probabilities of vaccination and of seeking medical care. We considered two outcomes against which the vaccine is supposed to protect: symptomatic influenza and medically-attended influenza.

When vaccination begins during an outbreak, we found that the effect of delayed onset of vaccination is unpredictable. VE estimates from TN studies were biased regardless of the source of bias present. However, if the core assumption of the TN design is satisfied, that is, if vaccination does not affect the probability of non-influenza ARI, then TN-based VE estimates against medically-attended influenza will only suffer from small (<0.05) to moderate bias (≥0.05 and <0.10). These results suggest that if sources of bias listed above are ruled out, TN studies are a valid study design for the estimation of VE during a pandemic.

1. Introduction

1.1. Background

Since the beginning of the 20th century, five influenza pandemics have occurred, none more devastating than the 1918 ‘Spanish flu’ pandemic. Despite the medical advances (e.g., vaccines and antivirals) and increased public awareness throughout the previous century, influenza pandemics pose a significant public health threat. The most recent 2009 H1N1 pandemic provided a powerful reminder of how dangerous pandemic influenza can be. An estimated 151,700 to 575,400 deaths occurred worldwide during the first year the pandemic strain circulated (1). Due to the continual adaptation of the influenza A virus, future influenza pandemics are inevitable. One important element of pandemic preparedness is the rapid development of a vaccine against the pandemic strain, as vaccination remains the best way to prevent influenza infection (2). It is estimated that as many as 1.5 million cases, 4,000–10,000 hospitalizations, and 200–500 deaths were averted in the United States by the monovalent vaccine during the 2009 pandemic (3).

In the context of a pandemic, estimation of influenza vaccine effectiveness (VE) involves additional challenges compared to the estimation of the effectiveness of seasonal influenza vaccines. During the 2009 pandemic, the monovalent vaccine against the pandemic strain was made available months after the start of the pandemic. The delayed and gradual timing of vaccination may introduce additional bias into estimates of VE compared to seasonal epidemics, where most people get vaccinated before the outbreak. For example, persons infected prior to vaccination are immune to further influenza from the infection and not vaccination. If such people were also more likely to get vaccinated, measurement of effectiveness would be biased toward a higher estimate than true vaccine induced effectiveness.

Since first being used to assess influenza VE in 2005 (4), the TN study design has become the most popular design for assessing annual influenza VE. The TN design is attractive because it can be easily incorporated into existing surveillance systems and it attempts to control for confounding due to propensity to seek medical care because cases and controls are both selected from individuals who seek medical care for ARI (5). Within a TN study, cases are selected from individuals who seek medical care for acute respiratory illness (ARI) and test positive for influenza infection, while controls are individuals who seek care for ARI and test negative for influenza infection.

In this work, we aim to assess the bias of VE estimates from a TN study when a substantial number of vaccinations occur following the beginning of the outbreak, as in a pandemic. We present a dynamic probability model that extends previous models (6–8) to allow vaccination to occur during the outbreak. Using this model we obtain VE estimates in the presence of numerous sources of bias via exact computations of probabilities. Previous investigation into the validity of the TN design has assumed all vaccinated individuals get vaccinated prior to the onset of the outbreak or study (6–15). We will consider assessing VE with respect to two different outcomes of interest: symptomatic influenza (influenza infection resulting in an ARI) and medically-attended influenza (influenza infection resulting in an ARI for which a person seeks medical care). Previous work has shown that estimates of VE may change depending on the outcome of interest (6–8).

2. Methods

2.1. Model Description

Below we briefly outline our dynamic model. Details about the associated variables and the probabilities determining each variable’s distribution (which may depend on variables from previous steps) can be found in Tables 1 and 2, respectively. All variables are defined for each member of the study population. Some variables vary over time (we consider each time unit to be one week). In Figure 1 we present a directed acyclic graph (15, 16) to illustrate the possible sources of confounding and bias present in studies designed to evaluate influenza VE.

Table 1.

Variables in the model.

Variable	Definition	Values
X	Health status	0 – frail person
		1 – healthy person
U	Health awareness (unobserved)	0 – low health awareness
		1 – high health awareness
Vj	Vaccination status in week j	0 – unvaccinated
		1 – vaccinated
K	The week a person is considered effectively vaccinated (two weeks after receiving the vaccine)	K = jv, … , J + 1*
Yj	Infuenza/non-influenza ARI status in week j	0 – no ARI
		1 – non-influenza ARI
		2 – influenza ARI
Mj	Seeking medical care for ARI in week j	0 – no
		1 - yes
Tj	Result of test for influenza infection in week j	0 – negative
		1 – positive

^*K = j_v indicates the first week individuals may become vaccinated and K = J + 1 (one week beyond the final week of the study J) indicates a person who was not vaccinated by the end of the study.

Table 2.

Parameters in the model.

Parameters	Definition	Comments
${\pi }_{xu}$	P(X = x, U = u)
${\alpha }_{jxu}$	P(K = j/X = x, U = u)
${\beta }_{jvx}$	P(Yj = 1/Vj = v, X = x)	${\beta }_{j01}\text{\hspace{0.17em}for\hspace{0.17em}a\hspace{0.17em}}standardperson$
${\theta }_{\beta }$	multiplier for β when Vj = 1	${\beta }_{j11}={\beta }_{j01}\cdot {\theta }_{\beta }$
${\mathrm{\text{φ}}}_{\mathrm{\text{β}}}$	multiplier for β when X = 0	${\beta }_{j00}={\beta }_{j01}\cdot {\phi }_{\beta }$
		${\beta }_{j10}={\beta }_{j01}\cdot {\theta }_{\beta }\cdot {\phi }_{\beta }$
${\gamma }_{jvx}$	P(Yj = 2/Vj = v, X = x)	${\gamma }_{j01}\text{\hspace{0.17em}for\hspace{0.17em}a\hspace{0.17em}}standardperson$
${\theta }_{\gamma }$	multiplier for γ when Vj = 1	${\gamma }_{j11}={\gamma }_{j01}\cdot {\theta }_{\gamma }$
${\phi }_{\gamma }$	multiplier for γ when X = 0	${\gamma }_{j00}={\gamma }_{j01}\cdot {\phi }_{\gamma }$
		${\gamma }_{j10}={\gamma }_{j01}\cdot {\theta }_{\gamma }\cdot {\phi }_{\gamma }$
${\delta }_{1vu}$	P(Mj = 1/Yj = 1, Vj = v, U = u)	${\delta }_{101}\text{\hspace{0.17em}for\hspace{0.17em}a\hspace{0.17em}}standardperson$
${\theta }_{{\delta }_1}$	multiplier for δ1 when Vj = 1	${\delta }_{111}={\delta }_{101}\cdot {\theta }_{{\delta }_1}$
${\mu }_{{\delta }_1}$	multiplier for δ1 when U = 0	${\delta }_{100}={\delta }_{101}\cdot {\mu }_{{\delta }_1}$
		${\delta }_{111}={\delta }_{101}\cdot {\theta }_{{\delta }_1}\cdot {\mu }_{{\delta }_1}$
${\delta }_{2vu}$	P(Mj = 1/Yj = 2, Vj = v, U = u)	${\delta }_{201}\text{\hspace{0.17em}for\hspace{0.17em}a\hspace{0.17em}}standardperson$
${\theta }_{{\delta }_2}$	multiplier for δ2 when Vj = 1	${\delta }_{211}={\delta }_{201}\cdot {\theta }_{{\delta }_2}$
${\mu }_{{\delta }_2}$	multiplier for δ2 when U = 0	${\delta }_{200}={\delta }_{201}\cdot {\mu }_{{\delta }_2}$
		${\delta }_{210}={\delta }_{201}\cdot {\theta }_{{\delta }_2}\cdot {\mu }_{{\delta }_2}$
${\tau }_y$	P(Tj = 1/Yj = y)

A standard person is defined as a person with (V_j = 0, X = 1, U = 1). β_j01 and γ_j01 represent the probabilities of contracting a non-influenza or influenza ARIfor a standard person in week j, respectively, j = 1,…,J. δ₁₀₁ and δ₂₀₁ represent the probabilities of seeking care for non-influenza or influenza ARIfor a standard person, respectively. These probabilities, as well as all π_xu, x = 0, 1; u = 0, 1, all α_jxu, j = 1,…, J; x = 0, 1; u = 0, 1, τ₁, τ₂, and all multipliers (μ, θ, φ), are input parameters. Parameter values are listed in Table C1.

Figure 1: Causal graph of influenza vaccine studies with covariates.

j is the index for week of the study,j = 1, … ,j_v,…, where j=1 is the first week of the study, j=j_v is the first week of vaccination, and J is the total number of weeks in the study. X = health status, (U ) = health awareness (unobserved), V_j = vaccination status in week j, Y_j = ARI status in week j, M_j = seeking medical care for ARI in week j, and T_j = influenza test result in week j. The values of these variables are defined in Table 1.

Step 1: Covariates. We assume that people within the population can be classified with a health status (X) of either “healthy” or “frail” and a health awareness ( U) of either “high” or “low”.

Step 2: Vaccination Individuals can be vaccinated throughout the study. j_v denotes the first week some study participants may be considered vaccinated (accounting for the two week delay in the onset of the effect of the vaccine).

Step 3: Influenza and non-influenza ARI. During the study, a person may become infected with an influenza virus and develop influenza ARI and/or develop one or more non-influenza ARIs. We assume that an individual can only have one influenza ARI during the study period and can have no more than one non-influenza ARI per week, but there is no limit on the total number of non-influenza ARIs. We define a variable Y_j for the illness/infection status in week j, where the distribution of Y_j may depend on the person’s vaccination status (V_j) and health status (X).

Step 4: Seeking medical care for ARI. A person with an ARI in week j may seek medical care (M_j). The probability of seeking medical care depends on Y_j, as only those individuals who have an ARI may seek medical care, and it may be different for influenza ARI and non-influenza ARI patients. This probability may also depend on V_j and U.

Step 5: Testing for influenza infection. We assume that each person who seeks medical care for ARI is tested for influenza infection.

2.1.1. Assumptions

We make several assumptions in the model: (a) vaccination status is determined without error;(b) a person can only have one influenza ARI during the outbreak; (c) a person can have at most one non-influenza ARI per week; (d) the probabilities of influenza ARI and non-influenza ARI do not depend on a person’s health awareness given his/her health status; (e) the probability of seeking medical care does not depend on a person’s health status given his/her health awareness; (f ) every person who seeks medical care is tested for influenza infection; (g) influenza test specificity and sensitivity do not depend on health status, health awareness, or vaccination status, given a person’s influenza infection status; and (f) the vaccine provides leaky protection (i.e., reduces the probability of transmission without rendering a person completely immune to infection).

2.2. True VE

The true VEs against symptomatic influenza and medically-attended influenza may be different (6–8). Therefore, we evaluated the true VE for each of the two outcomes of interest. A person is considered a true case of symptomatic influenza if s/he develops an influenza ARI during the study. Although symptomatic influenza and influenza ARI are identical concepts, the former is considered an outcome of interest (e.g., an outcome against which VE is estimated), while the latter is an outcome we observe in individuals. A person is considered a true case of medically-attended influenza if s/he develops an influenza ARI during the study and seeks medical care for this ARI.

True VE is calculated under the assumption of random vaccination, i.e., the probability that an individual is vaccinated does not depend on their health status or health awareness. Because vaccination status can change during the study, we define true VE as one minus the ratio of the sums of the expected numbers of cases (C_jv) and persons of vaccine status v (N_jv) up to week j:

$$1-\frac{{\sum }_j{C}_{jv}}{{\sum }_j{N}_{jv}},\text{for\hspace{0.17em}}v=0,1.$$

See Supplemental Information sections A.1 and A.2 for expression of true VE in terms of model parameters.

2.3. VE Estimates from TN Studies

A person is classified as a TN case or control at her/his first ARI -related medical visit. This classification does not change, regardless of possible conflicting test results in future visits. Similarly, when a person is classified as a case or control, only their vaccination status at that visit is recorded, regardless of whether it changes in future visits. We also assume a person with no ARI does not seek care and the study begins in week j = j_v, the first week when some of the study participants may be considered vaccinated. Therefore, we define M_j = 0 for j < j_v.

2.3.1. Cases

A person is considered a case in week j, if:

• they did not seek medical care for any ARI prior to week j, so M_k = 0 for every week k = 1, … , j −1 (i.e., M_j−1 = 0)

• they seek medical care for their ARI in week j (i.e., M_j = 1)

• they are diagnosed with influenza ARI in week j (i.e., T_j = 1)

The expected number of vaccinated cases in week j is:

E(vaccinated TN cases in week j) = N ×P(M_j−1 = 0, M_j = 1, T_j = 1, K ≤j),

where N is the s size of the study population, and the expected number of unvaccinated cases in week j is:

E(unvaccinated TN cases in week j) = N ×P(M_j−1 = 0, M_j = 1, T_j = 1, K > j)

See supporting information section B.1 for expressions of expected numbers of cases in terms of the model parameters.

2.3.2. Controls

A TN control in week j is defined in the same way as a case with the exception that an individual tests negative for influenza infection ( T_j = 0). The expected number of vaccinated controls in week j is:

E(unvaccinated TN controls in week j) = N ×P(M_j−1 = 0, M_j = 1, T_j = 0, K ≤j)

and the expected number of unvaccinated controls in week j is:

E(unvaccinated TN controls in week j) = N ×P(M_j−1 = 0, M_j = 1, T_j = 0, K > j)

For expressions of expected number of controls in terms of the model parameters, see supporting information section B.2. To obtain the final 2×2 table for a TN study (Table 3), we sum the expected cell counts over all weeks in the study period j_v, … , J , where j_v is the first week of vaccination. The VE estimate from a TN study is $\widehat{V{E}_{TN}}=1-\widehat{O{R}_{TN}}$, where $\widehat{O{R}_{TN}}$ is the odds ratio from Table 3.

Table 3.

Final 2×2 table for TN study.

	Vaccinated	Unvaccinated
Case	$\sum _{j=j_v}^{J}E\left(\text{vaccinated\hspace{0.17em}cases\hspace{0.17em}in\hspace{0.17em}week\hspace{0.17em}}j\right)$	$\sum _{j=j_v}^{J}E\left(\text{unvaccinated\hspace{0.17em}cases\hspace{0.17em}in\hspace{0.17em}week\hspace{0.17em}}j\right)$
Control	$\sum _{j=j_v}^{J}E\left(\text{vaccinated\hspace{0.17em}controls\hspace{0.17em}in\hspace{0.17em}week\hspace{0.17em}}j\right)$	$\sum _{j=j_v}^{J}E\left(\text{unvaccinated\hspace{0.17em}controls\hspace{0.17em}in\hspace{0.17em}week\hspace{0.17em}}j\right)$

2.3.3. Calculations

We derived exact mathematical expressions of true and expected estimated VE based on our model (supplemental information sections A and B). Using these expressions, we calculated the bias of TN-based VE estimate against each outcome of interest as a function of the model parameters (the various parameters were defined in Table 2). By varying the parameters, we were able to assess TN-based VE estimates under different sources of bias (Table 4) Each source of bias can be attributed to the deviation of a specific parameter from 1, and bias was calculated by varying the value of this parameter. When a source of bias was absent, we kept the corresponding parameter fixed at 1.0.

Table 4.

Sources of bias.

Label	Source of Bias	Parameter	Range
A	Vaccination affects the probability of non-influenza ARI	${\theta }_{\beta }$	0.5–2.0
B1	Frail persons have a higher probability of non-influenza ARI compared to healthy persons	${\phi }_{\beta }$	1.0–2.0
B2	Frail persons have a higher probability of influenza ARI compared to healthy persons	${\phi }_{\gamma }$	1.0–2.0
BS	Frail persons have a higher probability of influenza ARI and non-influenza ARI compared to healthy persons. Health status has the same effect on the probabilities of both types of ARI	${\varphi }_{\beta }={\varphi }_{\gamma }$	1.0–2.0
C	Vaccination lowers the probability of seeking medical care in influenza ARI patients (because of reduced symptom severity)	${\theta }_{{\delta }_2}$	0.5–1.0
D	ARI patients with low health awareness have a lower probability of seeking medical care compared to persons with high health awareness	${\mu }_{{\delta }_1}={\mu }_{{\delta }_2}$	0.5–1.0

Vaccination may increase or decrease the probability of non-influenza ARI, so we varied the ratio of the probability of non-influenza ARI in vaccinated compared with unvaccinated persons, θ_β, from 0.5 to 2.0. We expect healthy persons to have lower probabilities of ARI compared with frail persons, so we allowed the ratios of the probabilities of non-influenza ARI, φ_β, and influenza ARI, φ_γ, in frail compared with healthy persons to vary between 1.0 and 2.0. We expect vaccination to reduce the probability of seeking medical care for influenza ARI compared with non-influenza ARI, thus we varied the ratio of the probabilities of seeking medical care for influenza ARI in vaccinated compared with unvaccinated persons, ${\theta }_{{\delta }_2}$, between 0.5 and 1.0. We expected persons with high health awareness to have a higher probability of seeking medical care for both influenza ARI and non-influenza ARI compared with persons with low health awareness, so assumed the probability ratios of seeking medical care for non-influenza ARI, ${\mu }_{{\delta }_1}$, and influenza ARI, ${\mu }_{{\delta }_2}$, in persons with low health awareness compared with high health awareness to be equal, and varied their common value between 0.5 and 1.0.

For each source of bias separately and in combination we conducted 1,000 Monte Carlo simulations to obtain random values of the relevant parameter(s) (drawn from a triangular distribution with a mode of 1) and to calculate the percentiles of bias. The bias of the VE estimate was calculated for each simulation as the difference between the estimated and true VE. From these Monte Carlo simulations we determined the 5^th, 50^th, and 95^th quantiles of bias and the 50^th and 95^th quantiles of the absolute value of the bias. We derived explicit expressions of the true and expected value of the estimated VE against both outcomes of interest. These expressions are presented in the Supplemental Information, Sections A and B, respectively. All calculations and simulations were performed in R 3.3.1 (19). We developed user-friendly software for calculating the bias from input values of the parameters. The software is available from GitHub: https://github.com/fluvee/Model4_FluSim.git.

2.3.4. Sensitivity Analyses

We performed several sensitivity analyses to determine if different parameter values affected the magnitude of bias. First, we varied vaccination coverage (0.40, 0.60, and 0.80). Next, we varied the value of true VE (0.2, 0.4, 0.6, and 0.8), and finally, we varied the length of the study period (15, 20, 25, 30 weeks). For each sensitivity analysis we held the first week of vaccination fixed at week 24.

3. Results

We evaluated bias of VE estimates from TN studies in the presence of the sources of bias listed in Table 4. VE and biases are presented as fraction between 0 and 1, rather than as percentages. Bias is defined as the difference between the estimate and the true VE. The absolute bias is the value of the bias when the sign is ignored. For example, if the true VE is 0.4 and the estimated VE is 0.5 or 0.3, this would represent biases of 0.1 or −0.1, respectively, and an absolute bias of 0.1 in both cases. In Table 5 we present the 5^th, 50^th, and 95^th quantiles of the bias and the 50^th and 95^th quantiles of the absolute bias under each source of bias from Monte Carlo simulations. We define several terms to aid in our evaluation of the magnitude of bias: little/small or no bias indicates an absolute bias of less than 0.05, moderate bias indicates absolute bias greater than or equal to 0.05 and less than 0.10, substantial bias indicates absolute bias greater than or equal to 0.10 and less than 0.20, and severe bias indicates absolute bias of 0.20 or more.

Table 5.

Bias of TN-bases estimates of VE when first week of vaccination varies.

Source of Biasa	Outcome of Interest	jv	QB(5)b	QB(50)	QB(95)	QAB(50)	QAB(95)
None	Symptomatic Influenza & Medically-Attended Influenza	1c	0.00	0.00	0.00	0.00	0.00
		6	0.05	0.05	0.05	0.05	0.05
		11	0.10	0.10	0.10	0.10	0.10
		15	0.05	0.05	0.05	0.05	0.05
		20	−0.03	−0.03	−0.03	0.03	0.03
		24	−0.04	−0.04	−0.04	0.04	0.04
		28	−0.03	−0.03	−0.03	0.03	0.03
		33	−0.05	−0.05	−0.05	0.05	0.05
		38	−0.07	−0.07	−0.07	0.07	0.07
A	Symptomatic Influenza & Medically-Attended Influenza	1	−0.24	0.00	0.14	0.08	0.24
		6	−0.14	0.05	0.17	0.08	0.18
		11	−0.16	0.09	0.25	0.12	0.26
		15	−0.18	0.04	0.19	0.09	0.22
		20	−0.24	−0.03	0.09	0.06	0.24
		24	−0.16	−0.04	0.04	0.05	0.16
		28	−0.13	−0.03	0.04	0.04	0.13
		33	−0.25	−0.05	0.08	0.07	0.25
		38	−0.26	−0.07	0.05	0.08	0.26
B1	Symptomatic Influenza & Medically-Attended Influenza	1	0.00	0.01	0.03	0.01	0.03
		6	0.05	0.06	0.07	0.06	0.07
		11	0.10	0.10	0.12	0.10	0.12
		15	0.05	0.06	0.07	0.06	0.07
		20	−0.03	−0.02	−0.01	0.02	0.03
		24	−0.04	−0.04	−0.03	0.04	0.04
		28	−0.03	−0.03	−0.02	0.03	0.03
		33	−0.04	−0.04	−0.03	0.04	0.04
		38	−0.07	−0.07	−0.06	0.07	0.07
B2	Symptomatic Influenza & Medically-Attended Influenza	1	−0.02	−0.01	0.00	0.01	0.02
		6	0.03	0.04	0.05	0.04	0.05
		11	0.07	0.09	0.09	0.09	0.09
		15	0.02	0.04	0.05	0.04	0.05
		20	−0.05	−0.04	−0.03	0.04	0.05
		24	−0.05	−0.05	−0.04	0.05	0.05
		28	−0.04	−0.03	−0.03	0.03	0.04
		33	−0.06	−0.05	−0.05	0.05	0.06
		38	−0.10	−0.08	−0.08	0.08	0.10
BS	Symptomatic Influenza & Medically-Attended Influenza	1	0.00	0.00	0.00	0.00	0.00
		6	0.05	0.05	0.05	0.05	0.05
		11	0.10	0.10	0.10	0.10	0.10
		15	0.05	0.05	0.05	0.05	0.05
		20	−0.03	−0.03	−0.03	0.03	0.03
		24	−0.04	−0.04	−0.04	0.04	0.04
		28	−0.03	−0.03	−0.03	0.03	0.03
		33	−0.05	−0.05	−0.05	0.05	0.05
		38	−0.08	−0.08	−0.07	0.08	0.08
Cd	Symptomatic Influenza	1	0.01	0.07	0.16	0.07	0.16
		6	0.05	0.11	0.18	0.11	0.18
		11	0.10	0.17	0.26	0.17	0.26
		15	0.06	0.12	0.21	0.12	0.21
		20	−0.02	0.03	0.11	0.03	0.11
		24	−0.04	0.00	0.05	0.02	0.05
		28	−0.03	0.00	0.04	0.02	0.04
		33	−0.04	0.02	0.09	0.03	0.09
		38	−0.07	−0.01	0.07	0.04	0.07
	Medically-Attended Influenza	1	0.00	0.00	0.00	0.00	0.00
		6	0.03	0.04	0.05	0.04	0.05
		11	0.06	0.08	0.09	0.08	0.09
		15	0.03	0.04	0.05	0.04	0.05
		20	−0.03	−0.02	−0.02	0.02	0.03
		24	−0.04	−0.03	−0.02	0.03	0.04
		28	−0.03	−0.02	−0.02	0.02	0.02
		33	−0.04	−0.04	−0.03	0.04	0.04
		38	-0.07	−0.06	−0.04	0.06	0.07
D	Symptomatic Influenza & Medically-Attended Influenza	1	0.00	0.00	0.00	0.00	0.00
		6	0.05	0.05	0.05	0.05	0.05
		11	0.10	0.10	0.10	0.10	0.10
		15	0.05	0.05	0.05	0.05	0.05
		20	−0.03	−0.03	−0.03	0.03	0.03
		24	−0.04	−0.04	−0.04	0.04	0.04
		28	−0.03	−0.03	−0.03	0.03	0.03
		33	−0.05	−0.05	−0.05	0.05	0.05
		38	−0.08	−0.08	−0.07	0.08	0.08

^aSources of bias: A - vaccination affects the probability of non-influenza ARI, B1 - Frail persons have a higher probability of non-influenza ARI compared to healthy persons, B2 – Frail persons have a higher probability of influenza ARI compared to healthy persons, BS - Frail persons have a higher probability of non-influenza and influenza ARI compared to healthy persons, and health status has the same effect on the probabilities of both types of ARI, C - Vaccination lowers the probability of seeking medical care in influenza ARI patients (because of reduced symptom severity), and D - ARI patients with low health awareness have a lower probability of seeking medical care compared to persons with high health awareness. Bias = estimated VE - true VE. The sign of bias indicates the direction of the difference between the estimated and true VE. A negative sign corresponds to underestimation while a positive bias indicates overestimation,

^bQ_B (5)=5^th quantile of bias, Q_B (50)=50^th quantile (median) of bias, Q_B (95)=95^th quantile of bias, Q_AB (50)=50^th quantile (median) of the absolute bias, QAB (95)=95^th quantile of the absolute bias. Quantiles were determined from 1000 Monte Carlo simulations.

^cAll vaccinated individuals are vaccinated in the first week of the study

^dUnder bias C, bias of estimates of VE against symptomatic influenza and medically-attended influenza differ.

When vaccination occurred during the study, the effect of the first week of vaccination (j_v) was unpredictable. The TN study produced biased VE estimates regardless of the source of bias present, except when all vaccinated individuals were vaccinated in week 1 (baseline) (Table 5). Interestingly, the 95^th quantile of bias of VE estimates from different sources followed a similar pattern when the first week of vaccination varied (Figure 2).

Figure 2: 95^th Quantile of absolute bias of VE estimates by first week of vaccination.

The solid line indicates the incidence of influenza infection in each week during the 2009 pandemic in the United States (18). The 95^th quantile of bias was the same for no sources of bias (None) and biases BS and D; therefore, BS and D are not represented in the figure individually. A is bias resulting when vaccination affects the probability of non-influenza ARI, C - SI and C - MAI is bias resulting when vaccination affects the probability of seeking medical care for influenza ARI and the outcome of interest is symptomatic influenza and medically-attended influenza, respectively. The 95^th quantile of bias follows a similar pattern regardless of the source of bias as the first week of vaccination varies.

When vaccination influenced the probability of non-influenza ARI (bias A), VE estimates suffered from substantial to severe bias regardless of the value of j_v. The largest 95^th quantiles of absolute bias were observed at j_v = 11 and j_v = 38 (Q_AB(95) = 0.26); however, this was only slightly larger than the bias at baseline (Q_AB(95) = 0.24). For several values of j_v, the bias of VE estimates was smaller than baseline. The smallest bias was observed when j_v = 24 (Q_AB(95) = 0.13).

When health status affected the probability of non-influenza ARI (bias B1), VE estimates suffered from little bias at baseline (Q_AB(95) = 0.03) and little to substantial bias for all other values of j_v. The largest bias was observed at j_v = 11 (Q_AB(95) = 0.12). When health status affected the probability of influenza ARI (bias B2), VE estimates suffered from little bias at baseline (Q_AB(95) = 0.02) and little to moderate bias for all other values of j_v, except j_v =38, which suffered from substantial bias (Q_AB(95) = 0.10).

When vaccination lowers the probability of seeking medical care for influenza ARI (bias C) the bias of VE estimates differed by outcome of interest. Estimates of VE against symptomatic influenza suffered from little to severe bias depending on the first week of vaccination. At baseline, VE estimates were substantially biased (Q_AB(95) = 0.16). The largest bias was observed at j_v = 11 (Q_AB(95) = 0.26) and the smallest bias was observed at j_v = 28 (Q_AB(95) = 0.04). When the outcome of interest was medically-attended influenza, VE estimates at baseline were unbiased. For all other values of j_v VE estimates suffered from little to moderate bias (at j_v = 11, Q_AB(95) = 0.09 and at j_v = 28, Q_AB(95) = 0.02). The values of bias were the same when health status had the same effect on both types of ARI (bias BS) and when health awareness had an effect on the probability of seeking medical care (bias D). The largest bias was observed at j_v = 15 (0.10) compared to unbiased estimates at baseline. A summary of the results for each source of bias is shown in Table 6. We also assessed the bias of VE estimates under combinations of sources of bias, and similar patterns of bias were observed (results not shown).

Table 6.

Results summary.

Source of bias	Details
None	Biasissmallto moderate. Largestabsolute biasoccurs at jv = 11.
A	Bias may be severe (> 0.20) and is largest at jv = 11, 38.
B1, B2. BS, C – MAI, D	Pattern of bias very similar to the case of no sources of bias.
C – SI	Bias may be severe (> 0.20) and is largest at jv = 11.

Sources of bias: A - vaccination affects the probability of non-influenza ARI, B1 - Frail persons have a higher probability of non-influenza ARI compared to healthy persons, B2 – Frail persons have a higher probability of influenza ARI compared to healthy persons, BS - Frail persons have a higher probability of non-influenza and influenza ARI compared to healthy persons, and health status has the same effect on the probabilities of both types of ARI, C - Vaccination lowers the probability of seeking medical care in influenza ARI patients (because of reduced symptom severity), and D - ARI patients with low health awareness have a lower probability of seeking medical care compared to persons with high health awareness. SI – symptomatic influenza, MAI – medically attended influenza.

3.1. Sensitivity Analyses

To assess whether deviations from our original input parameters influenced the magnitude of bias when vaccination occurred during the study period, we calculated the bias of TN-based VE estimates when the overall vaccination coverage, true VE, and length of the study period varied. We observed the same pattern of bias between the values of j_v as in the original analysis. Specifically, the 95^th quantile of absolute bias was larger when vaccination started later in the outbreak compared to baseline, except when vaccination influenced the probability of non-influenza ARI (A) and seeking medical care for influenza ARI (C) when the outcome of interest was symptomatic influenza. For a fixed value of j_v, we found that the magnitude of bias increased slightly as vaccination coverage increased, except when the TN-based estimates were unbiased (Table D1). When varying the value of true VE, we found that as true VE increased, the magnitude of bias decreased for all values of j_v (Table D2). For example, when vaccination began in week 38 (jv =38) the absolute value of bias of VE estimates were 0.15 (corresponding to estimated VE=0.05 or 0.35 when true VE=0.2), 0.12 (corresponding to estimated VE=0.28 or 0.52 when true VE=0.4), 0.08 (corresponding to estimated VE=0.52 or 0.68 when true VE=0.6), and 0.04 (corresponding to estimated VE=0.76 or 0.84 when true VE=0.8) when vaccination influenced the probability of seeking medical care for influenza ARI and the outcome of interest was medically-attended influenza. Finally, we found very little difference in the bias of VE estimates when the length of the study period varied (Table D3).

4. Discussion

In this work, we assess the bias of TN-based VE estimates when vaccination begins at different weeks during an influenza outbreak rather than assume all vaccinated individuals were vaccinated prior to the study period (6–14, 20). The assumption of vaccination prior to the study period is reasonable for seasonal epidemics because vaccine campaigns begin prior to the outbreak; however, it is not a reasonable assumption during an influenza pandemic. We use the 2009 pandemic as the motivating example, where a vaccine against the pandemic influenza strain was made available months after the beginning of the outbreak. To determine if a substantial number of vaccinations occurring in the weeks and months following the beginning of an outbreak impacts VE estimates from TN studies, we extend a dynamic probability model (8) to allow vaccination to occur over time. We used a stochastic simulation program to verify that the expected bias derived from our model was the same as the observed bias (Supplemental Information C).

This work highlights that VE estimates from TN studies suffer from bias when vaccination occurs during the study period, regardless of the source of bias present. However, the effect of the first week of vaccination was unpredictable. The largest increase in the magnitude of bias compared to the baseline scenario was 0.10 when there were no systematic sources of bias. VE estimates for all values of the first week of vaccination, j_v, suffered from small to moderate bias under all sources of bias, except when vaccination affected the probability of non-influenza ARI and vaccination reduced the probability of seeking medical care for influenza ARI and the outcome of interest was symptomatic influenza.

The most pronounced bias occurred when vaccination affected the probability of developing a non-influenza ARI. This result is not surprising, as this is a violation of the core assumption of the TN study design (9, 20) because individuals who seek medical care for an ARI and test negative for influenza infection are used as controls. If it is suspected that vaccination impacts the probability of developing a non-influenza ARI, a TN study design should not be used. We found that even when no sources of bias are present, TN-based estimates of VE are biased when vaccination occurs during the study. The delay in vaccination could result in individuals becoming infected prior to vaccination and prior to the study period. This would result in a falsely high number of vaccinated controls (21) and could explain the positive bias observed in VE estimates. Influenza can be asymptomatic or result in mild symptoms, so an individual who is infected may not realize and subsequently be vaccinated, particularly during a pandemic when media coverage and public fear are high.

Conversely, vaccination during an outbreak could result in lower numbers of vaccinated controls biasing VE estimates downwards. This may explain our results where vaccination occurring during the study period mitigated some of the bias observed in when all vaccinations occurred prior to the outbreak. For example, when vaccination affected the probability of non-influenza ARI, baseline VE estimates may suffer from severe bias (Q_AB(95) = 0.24), but when vaccination occurred during the study period, VE estimates suffered only substantial bias (Q_AB(95)= 0.13 when j_v = 24).

A limitation of this work is that we only assessed the bias in unadjusted estimates of VE as the goal of the present work was to determine how different sources of bias impact VE estimates from TN designs, rather than focus on methods to adjust for bias. In practice, statistical methods are used to adjust for known sources of confounding. In the case of a pandemic when a large number of vaccinations occur during the study period, some of the bias may be mitigated by including time as a covariate in multivariable logistic regression or using time as a matching variable (22). Overall, these results are encouraging. If the core assumption of the TN design is satisfied, that is, if vaccination does affect the probability of non-influenza ARI, and VE estimates are not interpreted as VE against symptomatic influenza if it is suspected that vaccination reduces the probability of seeking care for influenza ARI (due to reduced symptom severity (23–25)), then TN-based VE estimates will only suffer from small to moderate bias. These results suggest that if these two sources of bias are ruled out, TN studies are a valid study design for the estimation of VE during a pandemic. However, this work emphasizes the need to make correct statistical adjustments for bias when calculating VE estimates from a TN study when vaccination occurs during the study period.

Highlights:

• When vaccination occurs over time, TN-based VE estimates may suffer from some bias

• The timing of beginning vaccination has an unpredictable effect on bias

• Median bias of TN-based VE estimates are often larger during a pandemic than a seasonal outbreak