A Dynamic Model for Evaluation of the Bias of Influenza Vaccine Effectiveness Estimates From Observational Studies

Abstract

Given that influenza vaccination is now widely recommended in the United States, observational studies based on patients with acute respiratory illness (ARI) remain as the only option to estimate influenza vaccine effectiveness (VE). We developed a dynamic probability model to evaluate bias of VE estimates from passive surveillance cohort, test-negative, and traditional case-control studies. The model includes 2 covariates (health status and health awareness) that might affect the probabilities of vaccination, developing ARI, and seeking medical care. Our results suggest that test-negative studies produce unbiased estimates of VE against medically attended influenza when: 1) Vaccination does not affect the probability of noninfluenza ARI; and 2) health status has the same effect on the probability of influenza and noninfluenza ARIs. The same estimate might be severely biased (i.e., estimated VE – true VE ≥ 0.20) for estimating VE against symptomatic influenza if the vaccine affects the probability of seeking care against influenza ARI. VE estimates from test-negative studies might also be severely biased for both outcomes of interest when vaccination affects the probability of noninfluenza ARI, but estimates from passive surveillance cohort studies are unbiased in this case. Finally, VE estimates from traditional case-control studies suffer from bias regardless of the source of bias.

The Centers for Disease Control and Prevention recommends annual seasonal influenza vaccination for everyone over the age of 6 months (1); therefore, observational studies remain as the only option for estimating influenza vaccine effectiveness (VE) in the United States, most commonly against influenza illness requiring outpatient medical care (2–5). Unlike in randomized clinical trials, in observational studies bias might be introduced into estimates of VE because the exposure (e.g., vaccination) cannot be randomized. Numerous studies have evaluated the bias of influenza VE estimates from case-control studies (6–11), but few have evaluated the bias of VE from cohort studies or compared the bias of VE estimates in case-control and cohort studies.

We considered 3 types of observational studies: passive surveillance cohort (PSC), test-negative (TN), and traditional case-control (TCC). Each study design uses the same definition of cases (i.e., individuals who seek medical care for an acute respiratory illness (ARI) and test positive for influenza) but defines noncases/controls differently (Table 1), potentially resulting in differing estimates of VE. Covariates that affect the probabilities of vaccination, developing an ARI from influenza infection and/or from a noninfluenza source, and seeking medical care for ARI could result in biased estimates of VE (Figure 1). We evaluated the bias of VE estimates under the following sources of bias:

1. Vaccination affecting the probability of noninfluenza ARI: Vaccination might modify the probability of developing noninfluenza ARI (Figure 1, short-dashed line from V to Y_j = 1), resulting in too many or too few vaccinated persons classified as noncases/controls.

2. Confounding bias due to the presence of a covariate (e.g., health status) related to both the probability of being vaccinated and the probabilities of influenza ARI and noninfluenza ARI: Health status might be associated with the probability of being vaccinated (12); for example, frail persons might be more likely to be vaccinated because they are considered at higher risk for influenza infection. Conversely, healthy persons might be more likely to be vaccinated to preserve their good health. Health status might also be associated with the probabilities of developing influenza and noninfluenza ARIs; for example, frail persons might be more likely to have ARI (Figure 1, dotted lines from X to V and X to Y_j).

3. Vaccination modifying the probability of seeking medical care: A person who is vaccinated might have a different probability of seeking medical care for influenza ARI compared with an unvaccinated person, due to a reduction of symptom severity from vaccination (13–15) (Figure 1, long-dashed line from V to M_j).

4. Confounding bias due to the presence of a covariate (e.g., health awareness) related to both the probabilities of vaccination and seeking medical care: A person’s health awareness might be associated with the probability of being vaccinated and seeking medical care, given that a person with high health awareness might be more likely to be vaccinated and seek medical care if they develop an ARI (Figure 1, alternating long-dash and dotted lines from (U) to V and (U) to M_j).

Table 1.

Definitions of Cases and Controls/Noncases for 3 Observational Study Designs Used to Estimate Influenza Vaccine Effectiveness

Study Design	Cases	Noncases/Controls
Passive surveillance cohort	Individuals in the cohort who seek medical care for an ARI and test positive for influenza	All other members of the cohort
Test-negative	Members of the study population who seek medical care for an ARI and test positive for influenza	Members of the study population who seek medical care for an ARI and test negative for influenza
Traditional case-control	Members of the study population who seek medical care for an ARI and test positive for influenza	Randomly selected individuals from the study population who did not develop an ARI throughout the study

Abbreviation: ARI, acute respiratory illness.

Figure 1.

Causal graph of influenza vaccine studies with covariates. X is health status, (U) is health awareness (unobserved), V is vaccination status, Y_j is acute respiratory illness (ARI) status in week j (Y_j = 1 indicates noninfluenza ARI, Y_j = 2 indicates influenza ARI), M_j is seeking medical care for ARI in week j, and T_j is influenza test result in week j, where j = 1, . . . , J and J is the number of weeks in the study. Broken arrows identify possible sources of bias. Short-dashed line: Vaccination (V) affects the probability of developing noninfluenza ARI (Y_j = 1). Dotted line: Health status (X) affects both the probabilities of vaccination (V) and developing ARI (Y_j = 1,2). Long-dashed line: Vaccination (V) affects the probability of seeking medical care (M_j). Alternating long-dash and dotted line: Health awareness (U) affects the probabilities of being vaccinated (V) and seeking medical care (M_j).

Misclassification of influenza infection and vaccination status is another source of bias present in observational studies. Influenza diagnostic tests are not 100% sensitive or specific, resulting in false-positive or false-negative test results, and vaccination status might be misclassified. We assumed that the influenza test has perfect sensitivity and specificity (we later relax this assumption as a sensitivity analysis) and that vaccination status is determined without error.

Using a dynamic model (Figure 1) that extends previously developed static models (7, 10) we evaluated and compared the bias of VE estimates from PSC, TN, and TCC studies. This model provides several advantages over our previous model (10): 1) a time component allowing the intensities of influenza and noninfluenza ARIs to change over time and the possibility of developing more than 1 ARI in a season; 2) incorporation of 2 covariates (health status and health awareness) that might affect the probabilities of vaccination, developing ARIs, and seeking medical care for these ARIs; 3) the ability to assess VE estimates from cohort and other longitudinal studies; and 4) the possibility of vaccination during the study, as in the case of a pandemic.

We used the model to better characterize the conditions under which estimates of VE against symptomatic and medically attended influenza might be biased, evaluated the magnitude and direction of the bias, and compared study designs with respect to bias. Previous work has shown that the bias of VE estimates might change depending on the outcome of interest (10). Although symptomatic influenza and influenza ARI are identical concepts, symptomatic influenza is considered the true outcome (e.g., an outcome against which VE is estimated), whereas influenza ARI is an observed outcome.

All 3 study designs require individuals with influenza ARI to seek medical care to be considered a case; thus, these types of studies provide estimates of VE against medically attended influenza. However, we also assess the bias of estimates from these studies against symptomatic influenza. We are interested in assessing the validity of VE estimates against symptomatic influenza from studies designed to provide estimates of VE against medically attended influenza, because VE estimates might be misinterpreted by the media and public as VE against symptomatic influenza.

METHODS

Model description

We present a dynamic model consisting of 5 steps. Below we define the model steps, the associated variables (Table 2), and the probabilities determining each variable’s distribution (Table 3). All variables are defined for each member of the study population, and we allow some variables to change over time (measured in weeks). Figure 1 illustrates the possible sources of confounding and bias present in studies designed to evaluate influenza VE (15, 16). Model assumptions are shown in Web Table 1 in Web Appendix 1 (available at https://academic.oup.com/aje).

Table 2.

Variables in a Dynamic Model for the Evaluation of Bias of Influenza Vaccine Effectiveness Estimates

Variable	Definition	Values
X	Health status	0 = frail person
		1 = healthy person
U	Health awareness (unobserved)	0 = low health awareness
		1 = high health awareness
V	Vaccination status	0 = unvaccinated
		1 = vaccinated
Yj	Influenza/noninfluenza ARI status in week j	0 = no ARI
		1 = noninfluenza ARI
		2 = influenza ARI
Mj	Seeking medical care for ARI in week j	0 = no
		1 = yes
Tj	Test result for influenza infection in week j	0 = negative
		1 = positive

Abbreviation: ARI, acute respiratory illness.

Table 3.

Definitions of Parameters Used in a Dynamic Model to Evaluate Bias of Influenza Vaccine Effectiveness Estimates

Parametera	Definition	Input Values	Detail
${\mathrm{\pi }}_{xu}$	$P(X=x,U=u)$	${\mathrm{\pi }}_{11}=0.40$
		${\mathrm{\pi }}_{10}=0.40$
		${\mathrm{\pi }}_{01}=0.10$
		${\mathrm{\pi }}_{00}=0.10$
${\mathrm{\alpha }}_{xu}$	$P(V=1|X=x,U=u)$	${\mathrm{\alpha }}_{11}=0.60$
		${\mathrm{\alpha }}_{10}=0.30$
		${\mathrm{\alpha }}_{01}=0.90$
		${\mathrm{\alpha }}_{00}=0.45$
${\mathrm{\beta }}_{jvx}$	$P(Y_j=1|V=v,X=x)$	See Web Table 2	${\mathrm{\beta }}_{j11}={\mathrm{\beta }}_{j01}\cdot {\mathrm{\theta }}_{\mathrm{\beta }}$
${\mathrm{\theta }}_{\mathrm{\beta }}$	Multiplier for $\mathrm{\beta }$ when $V=1$	See Table 5	${\mathrm{\beta }}_{j00}={\mathrm{\beta }}_{j01}\cdot {\mathrm{\phi }}_{\mathrm{\beta }}$
${\mathrm{\phi }}_{\mathrm{\beta }}$	Multiplier for $\mathrm{\beta }$ when $X=0$	See Table 5	${\mathrm{\beta }}_{j10}={\mathrm{\beta }}_{j01}\cdot {\mathrm{\theta }}_{\mathrm{\beta }}\cdot {\mathrm{\phi }}_{\mathrm{\beta }}$
${\mathrm{\gamma }}_{jvx}$	$P(Y_j=2|V=v,X=x)$	See Web Table 2	${\mathrm{\gamma }}_{j11}={\mathrm{\gamma }}_{j01}\cdot {\mathrm{\theta }}_{\mathrm{\gamma }}$
${\mathrm{\theta }}_{\mathrm{\gamma }}$	Multiplier for $\mathrm{\gamma }$ when $V=1$	See Table 5	${\mathrm{\gamma }}_{j00}={\mathrm{\gamma }}_{j01}\cdot {\mathrm{\phi }}_{\mathrm{\gamma }}$
${\mathrm{\phi }}_{\mathrm{\gamma }}$	Multiplier for $\mathrm{\gamma }$ when $X=0$	See Table 5	${\mathrm{\gamma }}_{j10}={\mathrm{\gamma }}_{j01}\cdot {\mathrm{\theta }}_{\mathrm{\gamma }}\cdot {\mathrm{\phi }}_{\mathrm{\gamma }}$
${\mathrm{\delta }}_{1vu}$	$P(M_j=1|Y_j=1,V=v,U=u)$	${\mathrm{\delta }}_{101}=0.25$	${\mathrm{\delta }}_{111}={\mathrm{\delta }}_{101}\cdot {\mathrm{\theta }}_{{\mathrm{\delta }}_{\mathrm{1}}}$
${\mathrm{\theta }}_{{\mathrm{\delta }}_1}$	Multiplier for ${\mathrm{\delta }}_1$ when $V=1$	See Table 5	${\mathrm{\delta }}_{100}={\mathrm{\delta }}_{101}\cdot {\mathrm{\mu }}_{{\mathrm{\delta }}_1}$
${\mathrm{\mu }}_{{\mathrm{\delta }}_1}$	Multiplier for ${\mathrm{\delta }}_1$ when $U=0$	See Table 5	${\mathrm{\delta }}_{110}={\mathrm{\delta }}_{101}\cdot {\mathrm{\theta }}_{{\mathrm{\delta }}_1}\cdot {\mathrm{\mu }}_{{\mathrm{\delta }}_{\mathrm{1}}}$
${\mathrm{\delta }}_{2vu}$	$P(M_j=1|Y_j=2,V=v,U=u)$	${\mathrm{\delta }}_{201}=0.40$	${\mathrm{\delta }}_{211}={\mathrm{\delta }}_{201}\cdot {\mathrm{\theta }}_{{\mathrm{\delta }}_2}$
${\mathrm{\theta }}_{{\mathrm{\delta }}_2}$	Multiplier for ${\mathrm{\delta }}_2$ when $V=1$	See Table 5	${\mathrm{\delta }}_{200}={\mathrm{\delta }}_{201}\cdot {\mathrm{\mu }}_{{\mathrm{\delta }}_2}$
${\mathrm{\mu }}_{{\mathrm{\delta }}_2}$	Multiplier for ${\mathrm{\delta }}_2$ when $U=0$	See Table 5	${\mathrm{\delta }}_{210}={\mathrm{\delta }}_{201}\cdot {\mathrm{\theta }}_{{\mathrm{\delta }}_2}\cdot {\mathrm{\mu }}_{{\mathrm{\delta }}_2}$
${\mathrm{\tau }}_y$	$P(T_j=1|Y_j=y)$	${\mathrm{\tau }}_1=0$
		${\mathrm{\tau }}_2=1$

^a β_j01, γ_j01, δ₁₀₁, and δ₂₀₁ for j = 1, . . ., J, where J is the last week of the study, and π_xu, x = 0, 1, u = 0, 1; α_xu, x = 0, 1, u = 0, 1; τ_y, y = 1, 2; and all multipliers (μ, θ, φ), are input parameters (see Web Table 2).

Step 1: covariates

We assumed that people within the population can be classified with a health status (X) of either “healthy” or “frail” and a health awareness (U) of either “high” or “low”.

Step 2: vaccination

We considered the vaccination scenario where an individual is considered vaccinated if they received the vaccine at least 14 days prior to the study onset (V = 1) or remains unvaccinated throughout the study (V = 0).

Step 3: influenza and noninfluenza ARI

During the influenza season, a person might become infected with an influenza virus and develop influenza ARI. Regardless of influenza infection, a person might develop 1 or more noninfluenza ARIs. We defined a variable Y_j for the illness/infection status in week j as follows: Y_j = 0 for no ARI, Y_j = 1 for noninfluenza ARI, and Y_j = 2 for influenza ARI. If a person has both noninfluenza ARI and influenza ARI in the same week, we consider them to have influenza ARI (i.e., Y_j = 2). The distribution of Y_j might depend on the person’s vaccination (V) and health (X) status.

Step 4: seeking medical care for ARI

A person with an ARI in week j might seek medical care (M_j). The probability of seeking medical care depends on Y_j, because only those individuals who have an ARI might seek medical care, and it could be different for influenza ARI and noninfluenza ARI patients. This probability might also depend on V and U.

Step 5: testing for influenza infection

We assume that each person who seeks medical care for ARI is tested for influenza infection. Let T_j denote the binary test result, where $T_j=\mathrm{1or}0$ for positive or negative, respectively.

True vaccine effectiveness

Bias is defined as the difference between an estimate and its true value. True VE is defined as 1 minus the relative risk of the outcome given vaccination compared with no vaccination when vaccination is random (i.e., the probability of vaccination does not depend on any covariates). We evaluated the true VE for each of the 2 outcomes of interest, symptomatic influenza and medically attended influenza (see Web Appendices 1 and 2 for explicit expressions and derivations of true VE).

True vaccine effectiveness against symptomatic influenza

A person is considered to represent a true case of symptomatic influenza if he or she develops an influenza ARI during the study. True VE against symptomatic influenza (VE_TSI) is

$$V{E}_{TSI}=1-\frac{P(\mathrm{contracting}\mathrm{symptomatic}\mathrm{influenza}|V=1)}{P(\mathrm{contracting}\mathrm{symptomatic}\mathrm{influenza}|V=0)},$$

where

$$\begin{array}{c}P\left(\mathrm{contracting}\mathrm{symptomatic}\mathrm{influenza}|V=v\right)\hfill \\ =\mathbf{\sum }_{j=1}^{J}P\left(Y_j=2|V=v\right),v=0,1;j=1,\dots ,J.\hfill \end{array}$$

True vaccine effectiveness against medically attended influenza

A person is considered to represent a true case of medically attended influenza if he or she develops an influenza ARI during the study and seeks medical care for this ARI. True VE against medically attended influenza (VE_TMAI) is

$$\begin{array}{c}V{E}_{TMAI}\hfill \\ =1-\frac{P(\mathrm{contracting}\mathrm{medically\; attended}\mathrm{influenza}|V=1)}{P(\mathrm{contracting}\mathrm{medically\; attended}\mathrm{influenza}|V=0)},\hfill \end{array}$$

where

$$\begin{array}{c}P\left(\mathrm{contracting}\mathrm{medically\; attended}\mathrm{}\mathrm{influenza}|V=v\right)\hfill \\ =\mathbf{\sum }_{j=1}^{J}P\left(Y_j=2,M_j=1|V=v\right),v=0,1;j=1,\dots ,J.\hfill \end{array}$$

Vaccine effectiveness estimates

In PSC studies, VE is estimated as $\widehat{VE}=1-\widehat{RR}$, where $\widehat{RR}$ is the estimated relative risk of the outcome and is based on sample proportions. In TN and TCC studies, VE is estimated as $1-\widehat{OR}$, where $\widehat{OR}$ is the odds ratio comparing the odds of vaccination in cases and controls (6, 7, 9, 17). We did not consider adjusted estimates of VE because we were interested in characterizing bias rather than in methods to adjust for bias. Current VE estimates are not specifically designed to estimate VE against symptomatic influenza or medically attended influenza. We account for the outcome of interest when we evaluate the bias of these estimates (“Calculations and simulations” section), not when we develop expressions for these estimates (Web Appendix 2). We have written the VE estimates in terms of probabilities; in actual studies, these probabilities are replaced by the corresponding proportions. Therefore, the “VE estimates” we calculated are the expected values of the actual estimates under our assumptions.

For each person, Y_J = (Y₁, . . ., Y_J) and M_J = (M₁, . . ., M_J) are the arrays of Y_j and M_j values from week 1 to week J, respectively.

PSC studies

In a PSC study, a person is considered a case in week j if they have ARI in week j (i.e., Y_j> 0), seek medical care for ARI in week j (i.e., M_j = 1), and test positive for influenza infection (i.e., T_j = 1).

The probability of being a PSC case for a given v is the probability of being a case in at least 1 week:

$$\begin{array}{c}P\left(Case\left(PSC\right)|V=v\right)\hfill \\ =P(\underset{j=1}{\overset{J}{\cup }}[\left\{Y_j=1,M_j=1,T_j=1\right\}\hfill \\ \cup \left\{Y_j=2,M_j=1,T_j=1\right\}]).\hfill \end{array}$$

Thus, the VE estimate from a PSC study is

$$\widehat{V{E}_{PSC}}=1-\frac{\hat{P}(Case(PSC)|V=1)}{\hat{P}(Case(PSC)|V=0)}.$$

TN studies

We assumed that a person is classified as a TN case or control at his or her first ARI-related visit. This classification does not change, regardless of possible conflicting test results in future visits. A person is considered a case in week j if they did not seek medical care for any ARI prior to week j (i.e., M_j−1 = 0), seek medical care for ARI in week j (i.e., M_j = 1), and test positive for influenza infection in week j (i.e., T_j = 1).

The probability of being considered a case for a given vaccination status, $V=v,v=0,1$, is

$$\begin{array}{c}P\left(Case\left(TN\right)|V=v\right)\hfill \\ =\mathbf{\sum }_{j=1}^{J}P(M_j=1,{\mathbf{M}}_{\mathbf{j}-1}=\mathbf{0},T_j=1|V=v).\hfill \end{array}$$

A TN control in week j is defined the same way as a TN case, except they test negative for influenza infection (i.e., T_j = 0). The probability of being considered a TN control for a given vaccination status v is

$$\begin{array}{c}P\left(Control\left(TN\right)|V=v\right)\hfill \\ =\mathbf{\sum }_{j=1}^{J}P(M_j=1,{\mathbf{M}}_{\mathbf{j}-1}=\mathbf{0},T_j=0|V=v).\hfill \end{array}$$

The VE estimate from a TN study is

$$\widehat{V{E}_{TN}}=1-\widehat{O{R}_{TN}},$$

where $\widehat{O{R}_{TN}}=\frac{\hat{P}\left(\mathrm{Case}\left(TN\right)|V=1\right)\hat{P}\left(\mathrm{Control}\left(TN\right)|V=0\right)}{\hat{P}\left(\mathrm{Case}\left(TN\right)|V=0\right)\hat{P}\left(\mathrm{Control}\left(TN\right)|V=1\right)}$.

TCC studies

A TCC case is classified in the same way as a TN case; thus, $P\left(Case\left(TCC\right)\right)=P(Case\left(TN\right))$. A person is considered a TCC control if he or she did not have an ARI during the entire study period (i.e., Y_J = 0). The probability of being considered a TCC control for a given vaccination status v is

$$P\left(Control\left(TCC\right)|V=v\right)=P({\mathbf{Y}}_{\mathbf{J}}=\mathbf{0}|V=v).$$

The VE estimate from a TCC study is

$$\widehat{V{E}_{TCC}}=1-\widehat{O{R}_{TCC}},$$

where $\widehat{O{R}_{TCC}}=\frac{\hat{P}\left(\mathrm{Case}\left(TCC\right)|V=1\right)\hat{P}\left(\mathrm{Control}\left(TCC\right)|V=0\right)}{\hat{P}\left(\mathrm{Case}\left(TCC\right)|V=0\right)\hat{P}\left(\mathrm{Control}\left(TCC\right)|V=1\right)}$.

Calculations and simulations

To evaluate the bias of VE estimates under different sources of bias (Table 4), we derived expressions of true and estimated VE from our model (Web Appendices 1 and 2). Using these expressions and weekly probabilities of influenza and noninfluenza ARIs based on influenza surveillance data (Web Appendix 3, Web Table 2), we calculated the bias of VE estimates under various combinations of sources of bias by varying the values of the corresponding parameters (probability ratios, Table 5). When a source of bias was absent, we kept the corresponding probability ratio fixed at 1.0. Bias was defined as estimated VE minus true VE. When vaccination influenced the probability of noninfluenza ARI, we considered that vaccination might increase or decrease the probability of noninfluenza ARI, so the ratio of the probability of noninfluenza ARI in vaccinated compared with unvaccinated persons, ${\mathrm{\theta }}_{\mathrm{\beta }}$, varied from 0.5 to 2.0. When health status influenced the probability of noninfluenza ARI, influenza ARI, and both noninfluenza and influenza ARI, we allowed the ratios of the probabilities of noninfluenza ARI, ${\mathrm{\phi }}_{\mathrm{\beta }}$, and influenza ARI, ${\mathrm{\phi }}_{\mathrm{\gamma }}$, in frail compared with healthy persons to vary between 0.5 and 1.0, given that we expect healthy persons to have lower probabilities of ARI compared with frail persons. When vaccination influenced the probability of seeking medical care in influenza ARI patients, the ratio of the probabilities of seeking medical care for influenza ARI in vaccinated compared with unvaccinated persons, ${\mathrm{\theta }}_{{\mathrm{\delta }}_2}$, varied between 0.5 and 1.0, given that we expect vaccination to reduce the probability of seeking medical care for influenza ARI compared with noninfluenza ARI. When health awareness influenced the probability of seeking medical care for ARI, we assumed that the probability ratios of seeking medical care for noninfluenza ARI, ${\mathrm{\mu }}_{{\mathrm{\delta }}_1}$, and influenza ARI, ${\mathrm{\mu }}_{{\mathrm{\delta }}_2}$, in persons with low health awareness compared with high health awareness to be equal, and their common value varied between 0.5 and 1.0 because we expected persons with high health awareness to have a higher probability of seeking medical care for both influenza ARI and noninfluenza ARI compared with persons with low health awareness. For each study design and each combination of sources of bias we determined the 5th, 50th, and 95th quantiles of bias from 1,000 Monte Carlo simulations. For each simulation, values for the relevant probability ratio(s) were drawn from independent triangular distributions over the ranges specified in Table 5. The mode of each distribution was assumed to be 1. The true and estimated VE were calculated for each simulation.

Table 4.

Sources of Bias Present in Influenza Vaccine Effectiveness Studies

Label	Source of Bias
A	Vaccination affects the probability of noninfluenza ARI.
B1	Healthy persons have a lower probability of noninfluenza ARI.
B2	Healthy persons have a lower probability of influenza ARI.
BS	Healthy persons have a lower probability of influenza ARI and noninfluenza ARI. Health status has the same effect on the probabilities of both types of ARI.
C	Vaccination lowers the probability of seeking medical care in influenza ARI patients (because of reduced symptom severity).
D	ARI patients with high health awareness have a higher probability of seeking medical care.

Abbreviation: ARI, acute respiratory illness.

Table 5.

Probability Ratios Corresponding to Sources of Bias Present in Influenza Vaccine Effectiveness Studies

Source of Biasa	Probability Ratio	Definition	Parameter	Range
A	PRA	P(NFARI|Vac)/P(NFARI|Unv)	${\mathrm{\theta }}_{\mathrm{\beta }}$	0.5–2.0
B1	PRB1	P(NFARI|Frail)/P(NFARI|Healthy)	${\mathrm{\phi }}_{\mathrm{\beta }}$	1.0–2.0
B2	PRB2	P(FARI|Frail)/P(FARI|Healthy)	${\mathrm{\phi }}_{\mathrm{\gamma }}$	1.0–2.0
BS	PRBS	Common value of PRB1 and PRB2	${\mathrm{\phi }}_{\mathrm{\beta }}={\mathrm{\theta }}_{\mathrm{\gamma }}$	1.0–2.0
C	PRC	P(SMC|FARI, Vac)/P(SMC|FARI, Unv)	${\mathrm{\theta }}_{{\mathrm{\delta }}_2}$	0.5–1.0
D	PRD	P(SMC|Low HA)/P(SMC|High HA)	${\mathrm{\mu }}_{{\mathrm{\delta }}_1}={\mathrm{\mu }}_{{\mathrm{\delta }}_2}$	0.5–1.0

Abbreviations: ARI, acute respiratory illness; FARI, influenza ARI; HA, health awareness; NFARI, noninfluenza ARI; P, probability; PR, probability ratio; SMC, seeking medical care; Unv, unvaccinated; Vac, vaccinated.

^aA: Vaccination affects the probability of noninfluenza ARI. B1: Healthy persons have a lower probability of noninfluenza ARI. B2: Healthy persons have a lower probability of influenza ARI. BS: Healthy persons have a lower probability of noninfluenza ARI and influenza ARI. C: Vaccination lowers the probability of seeking medical care in influenza ARI patients (because of reduced symptom severity). D: ARI patients with high health awareness have a higher probability of seeking medical care. Sources of bias are also listed in Table 4.

A stochastic simulation program was used to validate calculations by simulating data from each type of study design (Web Appendix 4). We performed sensitivity analyses to assess whether our choice of input parameters (value of true VE (Web Table 3); probabilities related to health status, health awareness (Web Table 4), and vaccination (Web Table 5); and values of influenza test sensitivity and specificity (Web Table 6)) affected our results (Web Appendix 5). All calculations and simulations were performed using R, version 3.3.1 (R Foundation for Statistical Computing, Vienna, Austria) (16).

RESULTS

Table 6 shows the 5th, 50th, and 95th quantiles of bias for each study design and source of bias. To aid in our evaluation of the magnitude of bias we defined the following terms for ranges of absolute bias: little/small (0, 0.05), moderate [0.05, 0.10), substantial [0.10, 0.20), and severe ≥0.20.

Table 6.

Bias of Estimates of Vaccine Effectiveness Against Symptomatic Influenza and Medically Attended Influenza in Observational Studies From Monte Carlo Simulations: 5th, 50th, and 95th Quantiles of Bias Under Multiple Sources of Bias

Source of Biasa	Outcome of Interest	True VE	Passive Surveillance Cohort			Test-Negative			Traditional Case-Control
			5th Quantile	Median	95th Quantile	5th Quantile	Median	95th Quantile	5th Quantile	Median	95th Quantile
None	SI and MAI	0.44	0.00	0.00	0.00	0.00	0.00	0.00	0.02	0.02	0.02
A	SI and MAI	0.44	0.00	0.00	0.00	−0.34	0.01	0.22	−0.15	0.02	0.10
B1	SI and MAI	0.44	0.00	0.00	0.00	0.00	0.02	0.05	−0.03	0.00	0.02
B2	SI and MAI	0.44	−0.05	−0.02	0.00	−0.05	−0.01	0.00	−0.03	0.00	0.02
BS	SI and MAI	0.44	−0.05	−0.02	0.00	0.00	0.00	0.00	−0.05	0.00	0.02
C	SI	0.44	0.01	0.10	0.23	0.01	0.10	0.23	0.03	0.12	0.24
C	MAI	Variesb	0.00	0.00	0.00	0.00	0.00	0.00	0.01	0.02	0.02
D	SI and MAI	0.44	−0.10	−0.04	0.00	0.00	0.00	0.00	−0.07	−0.02	0.02
BS, D	SI and MAI	0.44	−0.14	−0.06	−0.01	0.00	0.00	0.00	−0.12	−0.05	0.00
BS, D, A	SI and MAI	0.44	−0.14	−0.07	−0.01	−0.35	0.01	0.22	−0.26	−0.05	0.06
BS, D, C	SI	0.44	−0.07	0.05	0.20	0.01	0.11	0.24	−0.06	0.06	0.21
BS, D, C	MAI	Varies	−0.11	−0.05	−0.01	0.00	0.00	0.00	−0.10	−0.04	0.00
BS, D, A, C	SI	0.44	−0.08	0.05	0.20	−0.19	0.12	0.30	−0.14	0.06	0.23
BS, D, A, C	MAI	Varies	−0.12	−0.05	−0.01	−0.27	0.00	0.18	−0.21	−0.04	0.05

Abbreviations: ARI, acute respiratory illness; MAI, medically attended influenza; SI, symptomatic influenza; VE, vaccine effectiveness.

^a A: Vaccination affects the probability of noninfluenza ARI. B1: Healthy persons have a lower probability of noninfluenza ARI. B2: Healthy persons have a lower probability of influenza ARI. BS: Healthy persons have a lower probability of noninfluenza ARI and influenza ARI. C: Vaccination lowers the probability of seeking medical care in influenza ARI patients (because of reduced symptom severity). D: ARI patients with high health awareness have a higher probability of seeking medical care. Sources of bias are also listed in Table 4.

^b When bias C was present, true VE against MAI varied between 0.720 and 0.437 when the probability ratio ranged from 0.5 to 1.0, respectively.

When no sources of bias (as defined in Table 4) were present, PSC and TN studies produced unbiased estimates of VE. When vaccination affected the probability of noninfluenza ARI, the PSC study produced unbiased VE estimates, while the case-control studies produced VE estimates with a wide range of bias (for TN, 90% interval: −0.34, 0.22; for TCC, 90% interval: −0.15, 0.10). Interestingly, the direction of bias of VE estimates from the TN and TCC estimates was opposite as the probability ratio varied (Figure 2). When frail health status increased the probability of noninfluenza ARI, PSC VE estimates were unbiased, while case-control estimates suffered from small positive bias (for TN, 90% interval: 0.00, 0.05; for TCC, 90% interval: 0.00, 0.02). VE estimates from all 3 studies suffered from small bias when frail health status increased the probability of influenza ARI (for PSC, 90% interval: −0.05, 0.00; for TN, 90% interval: −0.05, 0.00; for TCC, 90% interval: −0.03, 0.02). When health status had the same effect on the probabilities of noninfluenza and influenza ARIs, only TN-based estimates were unbiased, while estimates from the other study designs suffered from small bias. Estimates of VE against symptomatic influenza might be severely biased when vaccination lowers the probability of seeking medical care for influenza ARI (for PSC, 90% interval: 0.01, 0.23; for TN, 90% interval: 0.01, 0.23; for TCC, 90% interval: 0.03, 0.24). Estimates of VE against medically attended influenza were unbiased from PSC and TN studies, while estimates from TCC studies had little bias. When health awareness influenced the probability of seeking medical care, TN-based VE estimates were unbiased, while PSC- and TCC-based estimates might suffer from substantial bias (for PSC, 90% interval: −0.10, 0.00; for TCC, 90% interval: −0.07, 0.02). Our calculated results were validated using our stochastic simulation program (results not shown).

Figure 2.

Plots of vaccine effectiveness (VE) estimates from passive surveillance cohort (PSC), test-negative (TN), and traditional case-control (TCC) studies compared with true VE for each source of bias alone as the probability ratio varies. Each row of plots corresponds to a single source of bias (see Table 4), and each column corresponds to a single study design. If only a solid line is visible, the VE estimate is unbiased. First row: True and estimated VE from PSC (A), TN (B), and TCC (C) studies when the probability ratio of the probability of noninfluenza acute respiratory illness (ARI) in vaccinated compared with unvaccinated persons varies. Second row: True and estimated VE from PSC (D), TN (E), and TCC (F) studies when healthy persons have a lower probability of noninfluenza ARI and influenza ARI compared with frail persons. Third row: True and estimated VE from PSC (G), TN (H), and TCC (I) studies when vaccination lowers the probability of seeking medical care in influenza ARI patients (because of reduced symptom severity), and the outcome of interest is symptomatic influenza. Fourth row: True and estimated VE from PSC (J), TN (K), and TCC (L) studies when vaccination lowers the probability of seeking medical care in influenza ARI patients, and the outcome of interest is medically attended influenza. Fifth row: True and estimated VE from PSC (M), TN (N), and TCC (O) studies when ARI patients with high health awareness have a higher probability of seeking medical care compared with ARI patients with low health awareness. Under sources of bias A, BS, and D (Table 4), the true VE against symptomatic influenza is equal to the true VE against medically attended influenza. Under source of bias C, the true VE against medically attended influenza differs from the true VE against symptomatic influenza when the probability ratio is not equal to 1.

Next, we evaluated the bias of VE estimates when multiple sources of bias were present simultaneously (Table 6). We expected bias in VE estimates due to confounding from an association between the covariates and the likelihood of illness (specifically under the assumption that health status has the same effect on the probabilities of influenza ARI and noninfluenza ARI (BS)) and seeking medical care (D) to occur most often. Bias resulting from vaccination influencing the probability of noninfluenza ARI (A) and seeking medical care for influenza ARI (C) is more controversial. Therefore, BS and D were included in all scenarios, and the presence of A and C was varied. For these reasons, we looked at 4 scenarios: BS and D only; BS, D, and A; BS, D, and C; BS, D, A, and C (Table 6).

In the presence of BS and D only, TN studies produced unbiased estimates, while PSC and TCC studies produced estimates with moderate to substantial bias.

With the addition of bias A (BS, D, A) all study designs produced estimates with bias that ranged from little to severe (for PSC, 90% interval: −0.14, −0.01; for TN, 90% interval: −0.35, 0.22; for TCC, 90% interval: −0.26, 0.06).

In the presence of BS, D, and C, all study designs produced estimates of VE against symptomatic influenza with bias that ranged from little to severe (for PSC, 90% interval: −0.07, 0.20; for TN, 90% interval: 0.01, 0.24; for TCC, 90% interval: −0.06, 0.21). The TN design produced unbiased estimates of VE against medically attended influenza. PSC and TCC studies produced estimates of VE against medically attended influenza that ranged from unbiased to substantially biased (for PSC, 90% interval: −0.11, −0.01; for TCC, 90% interval: −0.10, 0.00).

When all 4 sources of bias were present (BS, D, A, C), estimates of VE against symptomatic influenza and medically attended influenza from all 3 study designs suffered from moderate to severe bias.

DISCUSSION

We have presented a dynamic model that improves upon our previously developed static model (10) by incorporating a time component, which allows for the evaluation of bias from cohort studies in addition to case-control studies. An important finding of this work is that PSC studies produced unbiased estimates of VE even when vaccination affects the probability of developing noninfluenza ARI, while estimates from both types of case-control study might be substantially to severely biased. A core assumption underlying the validity of TN studies is that vaccination does not influence of the probability of developing noninfluenza ARI (6, 17, 18). Our results confirm earlier findings that, when this assumption is violated, TN-based estimates might be severely biased (6, 9, 10). Some studies have found an increased risk of noninfluenza ARI in vaccinated individuals (11, 19), while De Serres et al. (18), using data from clinical trials, and numerous observational studies (4, 20–23), some spanning multiple influenza seasons, found no association between noninfluenza ARI and vaccination. Although a carefully designed clinical trial is needed to definitively determine how influenza vaccination affects the probability of infection from a noninfluenza virus, our results agree with those of Foppa et al. (24), that bias will be severe only in extreme cases if vaccination alters the probability of noninfluenza infection. Thus, in practice, because most scenarios will not be extreme, it is unlikely that this source of bias will have a large impact on TN-based VE estimates.

Additionally, our dynamic model includes an unobserved covariate (health awareness) to account for behaviors that influence risk of developing ARI (e.g., handwashing (25)) and decisions to become vaccinated (e.g., perceived risk (26)) and seek medical care (e.g., access to health care (27)). When high health awareness increases the probability of seeking medical care for ARI, TN-based estimates remain unbiased, highlighting the ability of the TN design to control for differences in propensity to seek medical care in cases and controls (28). However, bias is introduced into estimates of VE from PSC and TCC studies, albeit small (medians, −0.04 and −0.02, respectively).

Most of the results from our dynamic model agreed with our previous results (10); however, we found less bias in case-control studies when persons with frail health status were more likely to develop ARI. This is an important result given that case-control studies, particularly TN studies, are commonly used for estimation of influenza VE (12, 29, 30), and health status is a well-established risk factor of influenza infection (12).

In practice, it is more likely that multiple sources of bias are present simultaneously; thus, we evaluated the bias of VE estimates under several scenarios. We assumed that confounding from health status and health awareness were present in each scenario. When only confounding was present, and the confounder had the same effect on the probabilities of influenza and noninfluenza ARIs, the TN design produced unbiased estimates of VE. When vaccination influenced the probability of noninfluenza ARI in addition to confounding, VE estimates from case-control studies might be severely biased. When vaccination affected the probability of seeking medical care for influenza ARI (due to reduced symptom severity in the event of a vaccine failure (13–15)) in addition to confounding, TN-based estimates were unbiased against medically attended influenza. In this case, estimates from all 3 study designs might be severely biased against symptomatic influenza.

Based on the results presented here and the assumption that multiple sources of bias are present, TN is the preferred study design for estimating VE against medically attended influenza. The TN design is the only study design that adjusts for confounding due to covariates (health status and health awareness). However, in the presence of confounding due to health status, TN-based estimates remain unbiased only when health status has the same effect on the probabilities of both types of ARI. If the core assumption (i.e., vaccination does not affect the probability of noninfluenza ARI) underlying this study design is violated (18, 19, 21), then one should consider a PSC study. Even if this assumption is satisfied, TN-based estimates should not be interpreted as estimates of VE against symptomatic influenza unless the possibility of vaccination influencing the probability of seeking medical care for influenza ARI has been ruled out.

We performed sensitivity analyses to assess the impact of our choice of the value of true VE (Web Figure 1) and probabilities of health status, health awareness (Web Figure 2), and vaccination (Web Figure 3) on the magnitude of bias. We found that bias is dependent on the value of the true VE, particularly for case-control studies (Web Figure 1, Web Table 3), suggesting that more effective vaccines are more robust to sources of bias. When health status and health awareness were correlated (originally, we assumed independence), we found that some of our results differed from those in our original scenario. In this case, VE estimates from TN studies were no longer unbiased against medically attended influenza when vaccination affected the probability of seeking medical care for influenza ARI (Web Figure 2). In fact, when health status and health awareness were correlated and vaccination affected the probability of seeking care for influenza ARI, VE was underestimated by all 3 study designs (when positively correlated, for PSC, 90% interval: −0.12, 0.00; for TN, 90% interval: −0.12, 0.00; for TCC, 90% interval: −0.10, 0.02; and when negatively correlated, for PSC, 90% interval: −0.17, 0.00; for TN, 90% interval: −0.17, 0.00; for TCC, 90% interval: −0.15, 0.02). Finally, we varied the value of influenza test sensitivity and specificity and observed that influenza VE estimates were robust to imperfect test sensitivity but had a larger magnitude of median absolute value of bias when the influenza test had imperfect specificity (Web Table 6). We considered only unadjusted estimates of VE; however, if appropriate adjustments are made, some of the bias might be reduced.

In conclusion, this work highlights the robustness of TN-based estimates of VE against medically attended influenza, especially in the presence of multiple sources of bias. However, caution should be used when interpreting estimates of VE from TN studies if it is suspected that vaccination influences the probability of seeking medical care for influenza ARI because this study design produces biased estimates when health status and health awareness are correlated.

Abbreviations

ARI

acute respiratory illness

PSC

passive surveillance cohort

TCC

traditional case-control

TN

test-negative

VE

vaccine effectiveness