Influenza Vaccination and Mortality: Differentiating Vaccine Effects From Bias

Abstract

It is widely believed that influenza (flu) vaccination of the elderly reduces all-cause mortality, yet randomized trials for assessing vaccine effectiveness are not feasible and the observational research has been controversial. Efforts to differentiate vaccine effectiveness from selection bias have been problematic. The authors examined mortality before, during, and after 9 flu seasons in relation to time-varying vaccination status in an elderly California population in which 115,823 deaths occurred from 1996 to 2005, including 20,484 deaths during laboratory-defined flu seasons. Vaccine coverage averaged 63%; excess mortality when the flu virus was circulating averaged 7.8%. In analyses that omitted weeks when flu circulated, the odds ratio measuring the vaccination-mortality association increased monotonically from 0.34 early in November to 0.56 in January, 0.67 in April, and 0.76 in August. This reflects the trajectory of selection effects in the absence of flu. In analyses that included weeks with flu and adjustment for selection effects, flu season multiplied the odds ratio by 0.954. The corresponding vaccine effectiveness estimate was 4.6% (95% confidence interval: 0.7, 8.3). To differentiate vaccine effects from selection bias, the authors used logistic regression with a novel case-centered specification that may be useful in other population-based studies when the exposure-outcome association varies markedly over time.

How effective is influenza (flu) vaccination in reducing mortality in the elderly? For several decades, vaccines against influenza have been recommended for people aged 65 years or more. It is widely believed that influenza vaccination saves many lives, especially when the epidemic is severe and the vaccine match is good. However, it has not been feasible to conduct randomized trials in the elderly that could yield compelling evidence about vaccine effectiveness, and there is controversy over how to interpret the available research (1).

Observational studies have found that mortality during flu season is much lower in vaccinated elderly people than in those not vaccinated (2–7). However, recently investigators have noticed that morbidity and mortality are relatively low in vaccinees even before the start of flu season (1, 8–15). These reports suggest that much or all of the vaccinated-versus-unvaccinated difference in mortality is attributable to selection bias. This bias can arise if 1) vaccination rates tend to be relatively low in people who are most at risk of death and 2) available data do not permit adequate adjustment for this tendency.

To differentiate vaccine effects from bias, we traced the vaccination-mortality association day by day—before, during, and after flu season—at Kaiser Permanente in Northern California. The usual strategy for minimizing bias is to seek good measures of potential confounders and then adjust for them. However, usually it is not feasible to track weekly changes in frailty and function as they attenuate the propensity to obtain flu shots near the end of life.

Our alternative strategy was to focus on a “difference in differences” (this term and general approach are often used by economists (16)). If the flu vaccine really does prevent deaths, then in a large population there should be a detectable difference between 2 differences: 1) the difference in the odds of prior vaccination between decedents and survivors that is observed on days when flu is circulating and 2) the difference in the odds of prior vaccination between decedents and survivors that would be expected on the same calendar dates if flu were not circulating. To examine such a difference in differences (or the corresponding ratio of odds ratios), we fitted a logistic regression model with a novel case-centered specification.

Our goals were to: 1) examine the propensity to obtain a flu shot in relation to predictors of mortality, 2) estimate the effect of flu shots on mortality, and 3) present and discuss case-centered logistic regression.

MATERIALS AND METHODS

Setting

Kaiser Permanente in Northern California provides comprehensive medical services to a membership which grew from 2.5 million to 3.1 million during the study period: the 9 flu years from September 15, 1996, through September 14, 2005. Each autumn, there is a campaign to deliver flu shots conveniently and at no cost to Kaiser Permanente members. Members are ethnically diverse and similar to the population of California in terms of age, but somewhat underrepresentative of the poor. The current study included everyone aged 65 years or older who was a member of Kaiser Permanente at the start of the flu year (September 15).

Data

Age, sex, and health plan membership were ascertained from Kaiser Permanente administrative databases. Vaccine information was obtained from Kaiser Permanente's Immunization Tracking System. Mortality data were obtained from California death certificate files, and hospital and clinic diagnoses were obtained from Kaiser Permanente clinical and claims databases. Diagnoses made during the 12 months before the flu year were weighted using DxCG software (DxCG, Inc., Boston, Massachusetts) (17, 18), creating an insurance risk score designed to predict costs. This risk score was available only for the last 4 flu years of our study period. Self-reported health status was obtained from a satisfaction survey routinely mailed by Kaiser Permanente to random samples of patients after visits. Twenty-seven percent of the person-time in the study was among people who had responded to this survey during the 12 months prior to the flu year.

Throughout the study period, one of the authors (R. B.) monitored all laboratory tests done within Kaiser Permanente and identified the beginning and end of each flu season, based on the number of influenza tests and whether more than 10 percent were positive. The earliest flu season began on November 9; the latest began on February 16. Every calendar day except January 20–23 fell outside of flu season in at least 2 of the 9 years in the study period.

Statistical analysis

Who gets flu shots?

Vaccine coverage was graphed in relation to age, insurance risk score, and the predicted probability of death at the outset of the flu year. The latter was obtained by logistic regression, regressing death (during the entire flu year) on age and sex (in 12 age-sex groups), the log of the insurance risk score, and dummy variables for diabetes, coronary artery disease, heart failure, and chronic obstructive pulmonary disease. The c statistic summarizing the fit of this model was 0.82.

Estimation of vaccine effectiveness.

Vaccine effectiveness was estimated through case-centered logistic regression. We call it “case-centered” because it focuses on the cases (deaths). Although we used time-varying information from the entire study population, we conducted the logistic regression analysis with a data set that had only as many records as there were deaths. For each day that someone died, we summarized the relevant information on all similar people at risk and included it in the record for the decedent.

The dependent variable was the decedent's vaccination status. It was compared with the expected odds of vaccination for the decedent, which were calculated before conducting the regression analysis. To calculate the expected odds, we found the stratum (or risk set) comprising people who were similar to the decedent on the day of death and calculated the odds of vaccination in the entire stratum including the decedent. These expected odds were included in the model as an offset, which in effect is a denominator variable. With the expected odds as a denominator variable on the right side and the observed odds indicated by vaccination status on the left side, the model could be used to focus on the observed-to-expected ratio, which is an odds ratio. We added measures of time of the year to the right side of the model in order to account for selection effects over time. Then we added the indicator of flu season in order to estimate how much of a difference it made in the odds ratio. This difference is what we sought; it amounted to a difference in differences. It was exponentiated, yielding a ratio of odds ratios, and then subtracted from 1 and multiplied by 100 to obtain the vaccine effectiveness (VE) estimate.

Thus, prior to conducting the regression analysis, we defined 39,444 potential strata (risk sets): 12 age-sex groups × 9 years, each with 365 or 366 days. For each stratum, we calculated vaccine coverage on that day and stored this information in a look-up table. Then, for each decedent, we looked up his or her age-sex group on the day of death and obtained the odds of vaccination. These odds summarize what our expectations would be in the absence of any vaccine effects or selection effects. For example, given a man who died on November 1, 2002, at age 82 years, our expectation came from the proportion of all men his age (80–84 years) who were vaccinated between September 15 and November 1, 2002.

An initial run included on the right-hand side of the model only the offset and an intercept. The intercept coefficient (after exponentiation) is an estimate of the odds ratio for the overall study period. We added polynomial terms for number of days since September 15, days squared, and days cubed in order to examine the trajectory of the odds ratio during the course of the flu year. We added indicators of sex, age group, flu year, and the decedent's receipt of the pneumococcal vaccine (ever vs. never) to refine our examination of the trajectory of the odds ratio. Finally, we added the flu season indicator to find vaccine effectiveness, differentiated from selection effects. We restricted this final analysis to the 61,436 deaths occurring during the period November through April, the 6 months when influenza virus ever circulated.

Similar case-centered logistic regression models were fitted to subgroups defined by age (65–79 years vs. ≥80 years) and cause of death (cardiovascular and respiratory causes vs. all other causes). Vaccine effectiveness should be higher if the cause of death was respiratory or cardiovascular (and therefore more likely to be flu-related) and if the age group was younger (because immune response can decline in the elderly).

Excess mortality during flu season.

Finally, we determined the average amount of excess (flu-attributable) mortality during flu season by fitting a Poisson regression model to data on 39,420 person-day strata (12 age-sex groups × 9 years × 365 calendar days, combining the 2 extra leap days with February 28). The count of deaths was regressed on a flu season indicator and covariates, with the person-time at risk included as an offset term. The covariates included number of days since September 15, days squared, an indicator for each flu year, and an indicator for each calendar month.

RESULTS

The elderly Kaiser Permanente population grew from 273,000 to 387,000 during the study period. There were 115,823 deaths in over 3 million person-years (Table 1). As expected, death was associated with older age, male sex, and a history of diabetes, heart disease, heart failure, or chronic obstructive pulmonary disease. The insurance risk score was a strong predictor of death: 43% percent of decedents scored in the highest (riskiest) 10 percentiles. Self-reported health status was also a strong predictor of death. It had been “fair” or “poor” for 62% of decedents versus only 30% overall. Decedents were slightly less likely than the overall study population to have ever received the pneumococcal vaccine (53% vs. 56%), yet much less likely than the overall study population to have received the flu vaccine in the current year (45% vs. 63%).

Table 1.

Characteristics of Kaiser Permanente Members Aged 65 Years or More, Northern California, 1996–2005

	Study Population at Start of Each Flu Yeara (n = 3,044,531 Person-Years)		Decedents (n = 115,823)
	No.	%	No.	%
Age group, years
65–<70	975,664	32.1	14,864	12.8
70–<80	1,442,172	47.4	44,257	38.2
80–<90	554,927	18.2	43,337	37.4
≥90	71,768	2.4	13,365	11.5
Male sex	1,359,674	44.7	58,802	50.8
Chronic conditions
Chronic obstructive pulmonary disease	122,158	4.0	14,381	12.4
Diabetes	397,151	13.0	23,061	19.9
Coronary artery disease	296,548	9.7	22,574	19.5
Heart failure	124,066	4.1	19,981	17.3
Health statusb
Poor	37,090	4.5	7,323	21.2
Fair	210,292	25.2	14,161	40.9
Good	374,671	45.0	9,962	28.8
Very good	178,786	21.5	2,773	8.0
Excellent	32,547	3.9	391	1.1
Percentile of insurance risk score (predicted cost)c
0–<50th	653,407	49.9	6,805	12.8
50–<90th	524,881	40.1	23,300	43.9
90–<98th	104,877	8.0	15,416	29.1
≥98th	26,197	2.0	7,522	14.2
Pneumonia vaccination, ever	1,700,161	55.8	61,520	53.1
Influenza vaccination in the current flu year	1,913,728	62.9	51,491	44.5

^aMid-September to mid-September.

^bHealth status was self-reported and was available only for patients who were randomly sampled for a visit-based survey about satisfaction with health care that was routinely conducted by Kaiser Permanente. Percentages are percentages of the people with data available.

^cRisk scores were available only for flu years 2001–2002 to 2004–2005. Percentages are percentages of the people with data available.

Flu vaccination began in early October. Of the 1.9 million flu shots delivered in the study population from 1996 to 2005, two-thirds were delivered by November 11, 95% by December 20, and 99% by January 12.

Vaccine coverage increased with age up to age 78 years in women and age 81 years in men (Figure 1). At older ages, coverage decreased. Similarly, vaccine coverage bore a curvilinear relation with the insurance risk score. Vaccine coverage peaked in patients whose predicted costs were in the 80th–90th percentiles (Figure 2). In the highest 10 percentiles of predicted cost (which included 43% of deaths), vaccine coverage decreased substantially. Vaccine coverage peaked in people whose predicted probability of death during the upcoming flu year was 3.0%–7.4% and fell below 50% in patients whose probability of death within a year was over 30% (Figure 3). There was also a curvilinear association of vaccine coverage with self-reported health status: As health declined from “excellent” to “very good” to “good,” vaccine coverage increased from 66% to 72% to 74%; as health status declined further to “fair” or “poor,” coverage decreased to 72% or 66%, respectively.

Figure 1.

Influenza vaccine coverage by age and sex among elderly members of Kaiser Permanente, Northern California, 1996–2005.

Figure 2.

Influenza vaccine coverage in relation to insurance risk score among elderly members of Kaiser Permanente, Northern California, 1996–2005. Intervals on the horizontal axis are spaced unevenly to sharpen the focus on higher-risk patients.

Figure 3.

Influenza vaccine coverage in relation to probability of death among elderly members of Kaiser Permanente, Northern California, 1996–2005. Intervals on the horizontal axis are spaced unevenly to sharpen the focus on higher-risk patients.

Next, we examined mortality week by week after vaccination (Figure 4). Among all vaccinees, there were only 0.5 deaths per 100 person-years on the day of vaccination and the following day, which together comprised week 0; by the eighth week after vaccination, mortality had increased to 3.1 per 100 person-years. In 3 high-risk subgroups defined by older age, chronic conditions, and self-reported health status, there were similar trajectories: Week 8 mortality exceeded week 1 mortality by multiples of 2.3, 2.5, and 3.0 in these 3 high-risk groups, respectively. Mortality among the healthier vaccinees (those with none of these risk factors) is shown on the lowest line of Figure 4: Although it is much lower throughout, it increases by a similar trajectory whereby week 8 mortality is 2.7 times higher than week 1 mortality. Thus, stratification by risk factors did not appear to change the pattern of rising mortality week by week after vaccination. Beyond week 8, mortality increased more gradually in all groups.

Figure 4.

Mortality (deaths per 100 person-years) by week after influenza vaccination among higher-risk vaccinees, lower-risk vaccinees, and all vaccinees (1.9 million flu shots), Kaiser Permanente, Northern California, 1996–2005. The flat reference line shows the average monthly mortality during all unvaccinated time in the study population. COPD, chronic obstructive pulmonary disease.

The vaccination-mortality association throughout the flu year is shown in Figure 5. The triangles show the proportion vaccinated among decedents by week of death, omitting the deaths that occurred when the influenza virus was circulating. The proportion of decedents who were vaccinated was always lower than expected, where the proportion expected comes from vaccine coverage in the decedent's age-sex group on the day of death (as described above). The descending line in Figure 5 traces the “VE” estimate for each week that comes from the corresponding observed and expected proportions shown on the other 2 lines. This descending line is labeled “VE” because it is the estimator of vaccine effectiveness that has been used in many studies, and quotation marks are used because here it can only reflect bias. Because all deaths occurring during flu season were omitted, the level of the “VE” line in Figure 5 cannot be attributed to vaccine effectiveness. Instead we suggest that the “VE” line shows the trajectory of the vaccination-mortality association when it reflects selection bias.

Figure 5.

Observed and expected proportions of decedents with influenza vaccination and the corresponding bias in vaccine effectiveness (VE), Kaiser Permanente, Northern California, 1996–2005. Deaths occurring during the 9 influenza seasons were omitted, so “VE” reflects bias rather than effectiveness.

A smoothed estimate of this trajectory—smoothed by means of case-centered logistic regression—is our point of departure for differentiating vaccine effectiveness from bias. When flu season deaths were included, “VE” in flu season was 4.6% higher than the expected bias.

VE estimates and confidence intervals are shown in Table 2, after adjustment for the bias. Vaccine effectiveness against all-cause mortality during flu season was 4.6% (95% confidence interval: 0.7, 8.3). Vaccination appears to have been more effective (5.3%) at ages 65–79 years than at older ages (3.9%). In addition, the vaccine appears to have been more effective against mortality from cardiovascular and respiratory causes (8.5%) than against mortality from other causes (0.1%). However, these differences in vaccine effectiveness by age and cause of death were not statistically significant (P > 0.10).

Table 2.

Effectiveness of Influenza Vaccination in Preventing Mortality Among the Elderly During Influenza Season, by Cause of Death and Age Group, Kaiser Permanente, Northern California, 1996–2005

Cause of Death	Age Group, years	No. of Deaths	Vaccine Effectivenessa, %	95% Confidence Interval	P Value
All causes	≥65	61,436	4.6	0.7, 8.3	0.0212
All causes	65–79	29,743	5.3	−0.3, 10.6	0.0641
All causes	≥80	31,693	3.9	−1.6, 9.0	0.1606
Cardiovascular or respiratory disease	≥65	31,798	8.5	3.3, 13.4	0.0017
Other causes	≥65	29,638	0.1	−5.9, 5.8	0.9632

^aVaccine effectiveness was estimated by means of case-centered logistic regression.

The analysis of excess (flu-attributable) mortality yielded results consistent with our inference that vaccine effectiveness is the reason why the arrival of influenza strengthened the vaccination-mortality association: During flu season, mortality was higher by 7.8% (95% confidence interval: 5.7, 9.9) than at the same time of the year when influenza virus was not circulating.

DISCUSSION

We found that flu shots reduced all-cause mortality among elderly Kaiser Permanente members by 4.6% during 9 laboratory-defined flu seasons in Northern California. Other researchers have reported that flu shots reduce mortality by much greater amounts. In a meta-analysis of results from 20 cohort and case-control studies, Voordouw et al. (6) found that flu shots reduce winter deaths by 50%, on average; and in a more recent study, Nichol et al. (19) reported a 48% reduction in all-cause mortality among the elderly during flu season. However, Simonsen et al. (11, 12, 20) found that excess mortality attributable to influenza has only been 5%–10% on average during flu seasons in the past several decades. They argued that flu shots could not possibly have prevented more deaths than the 5%–10% of deaths that were flu-related (11–13). Our estimate of excess mortality during flu season was 7.8%, which is consistent with Simonsen et al.’s nationwide estimate but lower than estimates made by others (21–23).

This excess mortality of 7.8% is what we found in a population with over 60% vaccine coverage. Our findings suggest that had none of the elderly been vaccinated, excess mortality during flu season would have averaged about 9.8%. We infer that our 4.6% VE estimate amounts to a 47% reduction (4.6/9.8 = 47%) in the number of flu-attributable deaths that would have occurred had none of the elderly been vaccinated.

Mortality in the Kaiser Permanente elderly population was approximately 3,804 per 100,000 person-years (Table 1). On average, 683 of these 3,804 deaths occurred during a laboratory-defined flu season, including 326 deaths in vaccinees. Our VE estimate of 4.6% implies that in the absence of flu shots, there would have been 342 flu-season deaths (326/0.954 = 342) in vaccinees. Thus, vaccination prevented approximately 16 flu-season deaths per 100,000 person-years (342 − 326 = 16) in the Kaiser Permanente population, which amounted to approximately 25 deaths prevented per 100,000 people vaccinated. The corresponding “number needed to treat” was 4,000; in other words, 1 death was prevented for every 4,000 elderly people vaccinated.

Before estimating vaccine effectiveness, our initial goal was to examine who gets flu shots. Whereas Nichol et al. (19, 24) reported that higher-risk patients were more likely to be vaccinated, Jackson et al. (9) reported that higher-risk patients were less likely to be vaccinated. We found a curvilinear relation between predictors of mortality and vaccination. Perhaps other investigators overlooked the curvilinearity because they considered mainly dichotomous indicators of risk. In our population, as in Nichol et al.’s populations, patients with heart disease, diabetes, or chronic obstructive pulmonary disease were more likely, on average, to get flu shots than patients without these chronic conditions. However, most patients with these conditions had only a moderately elevated risk of death, often in the range where vaccine coverage was highest. In higher-risk patients, who drive mortality rates in the upcoming flu season, the propensity to obtain flu shots waned.

It seems plausible that near the end of life, frailty poses barriers to vaccination, and patients (and providers) may tend to “give up” on preventive measures. However, until then, patients with chronic conditions have more reason and opportunity to get vaccinated than healthy people, because patients with chronic conditions tend to be more vulnerable to influenza and have more contact with providers who encourage vaccination.

Within low-risk subgroups as well as high-risk subgroups, mortality was low soon after vaccination and then increased over time in a pattern suggesting selection bias (Figure 4). It is this rise in mortality with time since vaccination that is especially challenging in the estimation of vaccine effectiveness. One strategy is to strive for better measures of frailty for covariate adjustment and for exclusion of patients known to be near death at the outset of the autumn vaccination campaign. However, Figures 4 and 5 suggest that whatever it is about nearness to death that suppresses vaccination, it varies markedly over time and would be difficult—even with data from charts or interviews—to monitor precisely enough to overcome selection bias.

Rather than seek covariates that might lower the biased “VE” line in Figure 5 and keep it flat at zero outside of flu season, we implemented a difference-in-differences approach: We traced the trajectory of the bias over time and compared the vaccination-mortality association inside flu season with that outside of flu season. What facilitated this approach was: 1) access to data on a large study population over a period of 9 years; 2) substantial year-to-year variation in the calendar dates of flu season ascertained by laboratory data; 3) little year-to-year variation in the calendar dates when flu shots were delivered; and 4) the assumption that real vaccine effectiveness is negligible each year until flu season arrives. The potential confounders of our VE estimate are not the unmeasured aspects of frailty which confounded Nichol et al. (19); instead, confounders would have to be somehow associated with the difference in differences—that is, the difference that the arrival of influenza makes in the vaccination-mortality association.

We examined the difference in differences using case-centered logistic regression. Case-centered logistic regression has several noteworthy features. First, it is closely related to Cox regression in a cohort study. It is equivalent to a stratified Cox model in which death is regressed on a time-varying indicator of vaccination. Each record in the case-centered model summarizes an entire risk set in the corresponding Cox model. The same likelihood is maximized (see the Web Appendix, which is posted on the Journal’s Web site (http://aje.oxfordjournals.org/)). Second, case-centered logistic regression is closely related to matched case-control studies with risk set sampling (also called incidence density sampling). However, there is no sampling: Data are used from all available controls. Third, it simplifies the analysis of changes in the exposure-outcome relation. In effect, it makes the odds ratio the dependent variable, which is then examined in relation to time and other factors. Fourth, case-centered logistic regression reduces computational burdens dramatically. These can be daunting in large studies with time-varying exposures. Fifth, it can minimize privacy concerns in a multisite study. Researchers at the study sites only need to pool aggregated data about each risk set rather than personal data about each person.

Our data and findings have limitations. First, we were missing data on flu shots given outside of Kaiser Permanente if they were never reported to Kaiser Permanente. If we missed flu shots delivered in nursing homes to patients near death, then we exaggerated the bias that we highlighted. Second, Kaiser Permanente's elderly population may differ from other elderly populations. Care-seeking behavior near the end of life may vary across sociocultural settings, and vaccination outreach may vary across practice settings. Third, our VE estimate was conditional on the severity of the flu seasons and the match of the vaccines to circulating strains of the virus. Fourth, we overlooked herd effects. Fifth, we overlooked late effects (if the vaccine prevents complications that increase mortality after flu season). Sixth, the 95% confidence interval around our VE estimate was wide relative to the excess mortality found in flu season: The lower bound (0.7%) was not far from zero, yet the upper bound (8.3%) would amount to the bulk of the excess mortality that would have struck vaccinees. Seventh, our focus on mortality overlooked the impact of vaccination on morbidity.

All-cause mortality is nonspecific. Nevertheless, it is important to consider, especially in the elderly. Although our estimate of 4.6% vaccine effectiveness against all-cause mortality during flu season may seem disappointing, it amounts to approximately 47% of a plausible target: the rise in mortality that would have occurred during flu season had none of the elderly been vaccinated.

Case-centered logistic regression can be a useful way to examine change in the impact of a vaccine or treatment as periods of high risk begin and end. More generally, case-centered logistic regression can be a useful way to examine the exposure-outcome association.

Glossary

Abbreviation

VE

vaccine effectiveness