Controlling Time-Dependent Confounding by Health Status and Frailty: Restriction Versus Statistical Adjustment

Abstract

Nonexperimental studies of preventive interventions are often biased because of the healthy-user effect and, in frail populations, because of confounding by functional status. Bias is evident when estimating influenza vaccine effectiveness, even after adjustment for claims-based indicators of illness. We explored bias reduction methods while estimating vaccine effectiveness in a cohort of adult hemodialysis patients. Using the United States Renal Data System and linked data from a commercial dialysis provider, we estimated vaccine effectiveness using a Cox proportional hazards marginal structural model of all-cause mortality before and during 3 influenza seasons in 2005/2006 through 2007/2008. To improve confounding control, we added frailty indicators to the model, measured time-varying confounders at different time intervals, and restricted the sample in multiple ways. Crude and baseline-adjusted marginal structural models remained strongly biased. Restricting to a healthier population removed some unmeasured confounding; however, this reduced the sample size, resulting in wide confidence intervals. We estimated an influenza vaccine effectiveness of 9% (hazard ratio = 0.91, 95% confidence interval: 0.72, 1.15) when bias was minimized through cohort restriction. In this study, the healthy-user bias could not be controlled through statistical adjustment; however, sample restriction reduced much of the bias.

Nonexperimental studies attributing large benefits to preventive health-care interventions may often be subject to the healthy-user bias (1). This bias arises when patients receiving a preventive medication or vaccination are in better health and/or more likely to engage in healthy behaviors, compared with patients not receiving preventive care (2–5). These differences exaggerate the beneficial association of the preventive intervention under study. The differences between treatment groups are often hard to characterize, especially using typical health-care claims data.

Studies of populations that include individuals in precarious health, such as the elderly or patients with serious comorbid conditions, may be particularly vulnerable to the healthy-user bias (6, 7). In these populations, patients may be at risk of experiencing sudden deteriorations in health status that are not captured in typical health-care data. Confounding by unobserved frailty or functional status is thought to be a common source of bias in studies of preventive interventions conducted in these populations (8). “Functional status,” defined as the level of ease with which a person can perform activities of daily living, can be measured through various instruments but is not easily assessed by using administrative claims data.

One setting where residual confounding has been well documented is in studies of influenza vaccine effectiveness in the elderly and other populations in poor health. In many studies, estimates of influenza vaccine effectiveness have been strongly confounded, suggesting improbably large (∼50%) reductions in all-cause mortality (9–11). Recent studies have found that less biased estimates of influenza vaccine effectiveness can be obtained through the use of alternate study designs, such as case-centered designs or natural experiments (12, 13). However, such designs exploit the unique characteristics of the influenza virus, such as seasonality or vaccine match, and therefore cannot be directly applied to studies of other preventive interventions. More general approaches to confounding control are needed.

In the present study, we explored some alternative approaches to confounding control in a study of influenza vaccine effectiveness in a cohort of patients with end-stage renal disease (ESRD). We linked commonly used administrative claims data with rich clinical data from a large dialysis provider. Because health status can decline rapidly in ESRD patients, we hypothesized that frequently recorded measures of health status obtained during thrice-weekly dialysis sessions might allow for improved control of confounding. We also identified proxies for functional status, such as skilled nursing facility stays and claims for mobility aids (e.g., wheelchair, walker). Using a marginal structural model, we examined how control for these time-varying covariates affected bias in estimates of vaccine effectiveness. We also examined how restriction of the cohort to healthier patients affected bias. Residual confounding was assessed by using the preinfluenza vaccine effectiveness estimate as a negative control.

METHODS

Study design and population

We conducted a cohort study of patients with ESRD who were enrolled in Medicare and belonged to a single large, national dialysis organization. The dialysis provider owns and manages over 1,500 outpatient dialysis facilities located in urban, rural, and suburban areas throughout the United States. We used Medicare claims data from the United States Renal Data System, which includes all ESRD patients in the United States, to measure hospitalizations and outpatient care, some medication use, immunizations, and death. We used linked data from the dialysis provider's clinical research database, which captures detailed clinical, laboratory, and treatment data on patients receiving care at all of the provider's dialysis units. This database provided detailed information on vitamin D dosing, epoetin alfa use and dosing, clinical laboratory values (e.g., hemoglobin, albumin), and the number and frequency of weekly dialysis sessions.

We created a yearly cohort for each of 3 influenza seasons: 2005–2006, 2006–2007, and 2007–2008. Each yearly cohort consisted of adult ESRD patients who had initiated dialysis prior to October 1 of the preceding year (Figure 1). An 8-month window from January 1 to August 31, prior to the start of follow-up, was used to identify insurance status and comorbidities. Patients were required to be on continuous hemodialysis for 3 months prior to the start of follow-up and to receive at least 9 dialysis sessions from the large dialysis provider during the last month of baseline. For example, the cohort identified for the 2005–2006 season initiated dialysis prior to October 1, 2004, had Medicare as a primary payer from January 1 to August 31, 2005, and used continuous hemodialysis from June 1 to August 31, 2005. Vaccination status and time-dependent confounders were assessed beginning on September 1 of each year. We performed an analysis of time to death where cohort members were followed until they died, had a kidney transplant or switch to peritoneal dialysis, were lost to follow-up, or were administratively censored at the end of the influenza season, whichever came first. Inverse-probability-of-censoring weights were not applied in the analysis, as censoring for reasons other than end of the study period was rare (∼12%).

Figure 1.

Study design diagram of inclusion criteria and follow-up time for each yearly cohort of adult patients with end-stage renal disease, United States, 2005–2007.

Exposure and outcome definitions

Influenza vaccination was identified in both the Medicare Part A hospital/outpatient files and the Part B physician/supplier files, as well as from vaccination data provided by the large dialysis provider. A patient was considered vaccinated as of the first date of influenza vaccine administration documented in either data source. We used Current Procedural Terminology codes 90724, 90656, and 90658-60, Healthcare Common Procedure Coding System codes G0008 and G8482, and the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) procedure code 99.52 to identify influenza vaccine. All-cause mortality was identified by the Centers for Medicare and Medicaid Services’ ESRD Death Notification Form.

Time-fixed confounders

Confounders measured during the 8-month baseline period were identified a priori and placed into 3 “blocks” determined by the type of variable and the potential strength of the confounder (refer to Web Appendix 1 available at http://aje.oxfordjournals.org/). The base set of variables consisted of demographic information including the number of years since a patient was diagnosed with ESRD, the number of hospital and skilled nursing days in the last month of baseline (14, 15), infection in the last month of baseline, indicators of frailty including mobility and oxygen use (5, 14, 15), and serious comorbidities, identified by using inpatient and outpatient claims. We searched for specific ICD-9-CM diagnosis codes, and equal weight was given to all diagnosis codes for a given condition (i.e., a comorbidity was determined to be present if a patient had 1 diagnosis code). We also measured body mass index and most recent albumin level using the clinical database. The other blocks of confounders consisted of additional comorbidities (block 2) and preventive services, such as other vaccinations and health screenings (block 3).

Time-dependent confounders

For the main analysis, we updated time-dependent confounders each week, beginning September 1. We constructed variables that were meant to capture changes in frailty status or to indicate severe frailty (Web Appendix 2). The composite ambulatory status and frailty variables were chosen from variables identified as predictors of frailty in the elderly population (5, 14, 15). Some of the variables were not measured every week. If a patient did not have a hemoglobin or albumin lab value for a given week, we imputed a value using the last observation carried forward method. We believe this is a valid method for this situation, as clinicians treating the patient would also use it.

Statistical analysis

Yearly influenza severity was assessed on the basis of the strains in circulation and the level of vaccine match. To estimate the level of vaccine match, we obtained from Centers for Disease Control and Prevention reports the percentage of US virus samples in each strain—A(H1N1), A(H3N2), and B—that antigenically matched the recommended vaccine, and we calculated a weighted average of these percentages according to the proportion of viruses observed to belong to each strain (16–18). A separate model was fit for each influenza season; patients were followed from September 1 until the end of the influenza season. For the crude model, we used pooled logistic models with 1 observation for each person-week to estimate discrete-time approximations (19) of hazard ratios (20), comparing vaccinated with unvaccinated observations within each year. The largest weekly incidence of mortality was 0.5%, which satisfies the rare event requirement for the discrete-time approximation (19). Vaccination was modeled as a time-varying treatment, with all cohort members entering the analysis on September 1 as unvaccinated. Once vaccinated, patients remained in the vaccinated category until they experienced death, were lost to follow-up, or were censored at the end of the influenza season.

The marginal structural model was estimated using inverse-probability-of-treatment weights. These weights create a “pseudo-population” where each observation is weighted by the inverse of the probability of receiving the exposure actually received, conditional on covariates (21). In the pseudo-population, measured confounders should be distributed equally across vaccination groups. Vaccine effectiveness was estimated by comparing vaccinated and unvaccinated observations within this theoretically unconfounded pseudo-population.

Time-varying weights were estimated for each week of follow-up from September 1 until vaccination or the end of the influenza season, whichever came first. To estimate the denominator for each weight, we used the unvaccinated and newly vaccinated person-week observations to fit a pooled logistic model estimating the probability of vaccination. To ensure correct ordering of covariate and exposure data, we used covariate information up through the previous week to predict the current week's vaccination status (22). Time-varying confounders, measured in the previous week, were included along with baseline confounders. The weights were stabilized by using a pooled logistic model to estimate the probability of being vaccinated, conditioning only on baseline covariates. We used robust variance estimates, equivalent to generalized estimating equations with an independent working covariance matrix (23).

We conducted several sensitivity analyses to reduce bias. First, we included an expanded set of baseline confounders. Second, we varied the length of the time window associated with each observation (4-day and 10-day windows instead of weeks). Third, we made the cohort more homogeneous and healthier by requiring survival into the follow-up period, placing 2 sets of restrictions on baseline covariates, and placing restrictions on time-dependent covariates. The survival restrictions required survival for the first several (6, 8, 10, or 12) weeks of follow-up; for example, when we required survival for 6 weeks, follow-up began in week 7. The limited set of baseline restrictions required no hospitalization and ≥95 sessions of hemodialysis during the baseline period. The expanded set of baseline restrictions included the limited baseline restrictions plus no skilled nursing facility care, no infections, and last baseline albumin value >3.3. The time-dependent restrictions amounted to excluding any person who was hospitalized during the required survival period or had fewer than 2 hemodialysis sessions in any week of the required survival period.

Use of the preinfluenza season as a negative control

Because there is no biologically plausible mechanism for the vaccine to prevent illness or death before the influenza virus starts circulating, estimates of vaccine effectiveness during the preinfluenza period should show no association in the absence of confounding. Therefore, preinfluenza-period estimates have been used as a negative control (i.e., an outcome that is known to be causally unrelated to the exposure) to detect residual bias in model estimates (24) and to calibrate vaccine effectiveness models (24–26). To quantify bias in our crude and marginal structural model estimates, we ran the same models during the preinfluenza period (September 1 through the day before the influenza season started). We estimated the start of each influenza season using national influenza surveillance data from the Centers for Disease Control and Prevention. We defined the start of the season as the midpoint of the first week in which more than 10% of submitted respiratory isolates were positive for influenza (16–18). Analyses were conducted with SAS software, version 9.3 (SAS Institute, Inc., Cary, North Carolina). This study was approved by the Institutional Review Board at the University of North Carolina.

RESULTS

The size of the yearly cohorts ranged from 40,620 in 2005 to 46,281 in 2007. Vaccination coverage ranged from 65% to 74%. In general, vaccinated patients were slightly older and more likely to have white race, as well as fewer years of hemodialysis (Table 1). The 2005 and 2006 influenza seasons were slightly less severe than the 2007 season, as multiple strains of influenza (including less severe A(H1N1) and B) were commonly circulating in the community (Table 2). Crude estimates of vaccine effectiveness in preventing all-cause death were similar over all 3 influenza seasons and were biased, as evidenced by a large protective association during the preinfluenza period (Table 3).

Table 1.

Characteristics of the Linked Study Population for Each Influenza Season, United States, 2005–2007

Variable	2005				2006				2007
	Vaccinated (n = 28,030)		Nonvaccinated (n = 12,590)		Vaccinated (n = 27,434)		Nonvaccinated (n = 14,939)		Vaccinated (n = 34,282)		Nonvaccinated (n = 11,999)
	%	Mean (SD)	%	Mean (SD)	%	Mean (SD)	%	Mean (SD)	%	Mean (SD)	%	Mean (SD)
Total, %	69		31		65		35		74		26
Age, years		62.4 (14.4)		60.9 (15.2)		62.4 (14.3)		60.8 (14.9)		62.1 (14.2)		60.6 (15.0)
Male sex	54.3		52.6		54.9		53.3		55.0		53.5
Race
White	51.3		42.6		52.9		44.8		52.0		43.4
Black	42.0		51.7		40.5		49.6		41.6		51.4
Other	6.7		5.7		6.6		5.6		6.4		5.2
Cause of ESRD
Diabetes	45.4		43.7		45.8		44.1		45.0		41.5
Hypertension	30.0		32.3		29.6		31.6		29.7		32.1
Other	24.7		24.1		24.6		24.3		25.3		26.4
Years with ESRD
0	2.0		2.0		1.7		1.6		1.5		1.8
1–3	55.0		53.5		53.5		50.8		50.4		49.2
≥4	43.1		44.6		44.8		47.6		48.1		49.0
Albumin, g/dL		3.9 (0.4)		3.8 (0.4)		3.9 (0.4)		3.8 (0.5)		3.9 (0.4)		3.8 (0.4)
Hemoglobin, g/dLa		12.3 (1.3)		12.2 (1.3)		12.3 (1.3)		12.2 (1.4)		12.2 (1.3)		12.1 (1.4)
Body mass indexb		27.4 (7.0)		26.8 (7.1)		27.7 (7.2)		27.0 (7.2)		27.4 (6.9)		26.9 (6.8)

Abbreviations: ESRD, end-stage renal disease; SD, standard deviation.

^a Sample size was slightly smaller for baseline hemoglobin, which was missing for <0.1% of the sample.

^b Weight (kg)/height (m)².

Table 2.

Characteristics of Specific Influenza Seasons, United States, 2005–2007

Year	Predominant Strain	Vaccine Match, %	Season Start Date	Season End Date
2005	A(H3N2), B	63	12/21/2005	4/19/2006
2006	A(H1N1), A(H3N2)	62	12/20/2006	4/25/2007
2007	A(H3N2)	25	1/9/2008	4/16/2008

Table 3.

Estimates of Vaccine Effectiveness for Preventing Death Under Different Modeling Specifications, United States, 2005–2007

	2005				2006				2007
	Preinfluenza		Influenza		Preinfluenza		Influenza		Preinfluenza		Influenza
	HR	95% CI	HR	95% CI	HR	95% CI	HR	95% CI	HR	95% CI	HR	95% CI
Crude	0.39	0.35, 0.43	0.82	0.76, 0.88	0.32	0.28, 0.36	0.74	0.69, 0.80	0.46	0.43, 0.50	0.83	0.77, 0.90
Basic marginal structural modela	0.43	0.39, 0.47	0.84	0.77, 0.91	0.36	0.32, 0.41	0.79	0.73, 0.85	0.53	0.48, 0.57	0.83	0.77, 0.91
Adding baseline covariates
Block 2	0.43	0.39, 0.47	0.83	0.77, 0.90	0.36	0.32, 0.41	0.79	0.73, 0.85	0.52	0.48, 0.57	0.83	0.76, 0.90
Blocks 2 and 3	0.43	0.39, 0.47	0.84	0.78, 0.92	0.36	0.32, 0.41	0.80	0.74, 0.86	0.53	0.49, 0.57	0.83	0.77, 0.91
Varying time window for time-dependent covariates
4 days	0.45	0.41, 0.51	0.79	0.71, 0.88	0.40	0.35, 0.45	0.79	0.73, 0.85	0.55	0.51, 0.60	0.83	0.76, 0.90
10 days	0.45	0.41, 0.49	0.84	0.78, 0.91	0.41	0.36, 0.46	0.79	0.73, 0.85	0.56	0.51, 0.60	0.84	0.77, 0.91

Abbreviations: CI, confidence interval; HR, hazard ratio.

^a Adjusted for block 1 baseline covariates and time-dependent covariates measured in a 7-day window.

The marginal structural models for all years had similar distributions for the stabilized weights; that is, all years had a median of 1.00, and none of the weights was considered extreme (Web Table 1). The weights balanced observations of vaccinated and unvaccinated persons on baseline and time-dependent covariates (Web Tables 2 and 3). Because estimates from all 3 influenza seasons were similar, we focus our reporting on results for 2005. Adding baseline covariates, including preventive services and an expanded set of comorbidities, did not change the estimate. Similarly, changing the length of the time window associated with each observation did not change results (Table 3).

Restricting covariate values alone had little effect on results (not shown), but progressively applying survival and covariate restrictions reduced bias. Requiring patients to survive at least 12 weeks into the follow-up period moved the preinfluenza estimate from 0.43 to 0.66 (Figure 2). The addition of subsequent baseline and time-dependent restrictions moved the estimate closer to the null. However, this also reduced the sample size, resulting in wider confidence intervals. Additionally, models would not converge when we required 12 weeks of survival and implemented time-dependent variable restrictions.

Figure 2.

Forest plot for estimates of vaccine effectiveness in preventing death in the preinfluenza period under varying levels of cohort restriction, United States, 2005. The full set of baseline restrictions includes no hospitalization, no skilled nursing facility care, no infections, 95 or more sessions of hemodialysis, and last baseline albumin test >3.3. The time-dependent restrictions include no hospitalization and 2 or more hemodialysis sessions in the prior week. The limited set of baseline restrictions includes no hospitalization and 95 or more sessions of hemodialysis. HR, hazard ratio; bars, 95% confidence interval (CI).

Table 4 illustrates the results of the progressive sample restrictions on baseline sample characteristics. As restrictions decreased the sample size, the measured sample characteristics varied little. However, the percentage of the sample that was vaccinated increased from 69% to 75%, suggesting that the more restricted samples were healthier in ways that were not reflected by baseline variables.

Table 4.

Effects of Selected Sample Restrictions on Baseline Sample Characteristics, United States, 2005

Variable	Original Sample (n = 40,620)		6-Week Survival (n = 39,761)		8-Week Survival (n = 39,424)		10-Week Survival (n = 38,995)		12-Week Survival (n = 38,600)		8-Week Survival + Full Baseline Set (n = 26,063)		8-Week Survival + Time-Dependent Set (n = 29,416)		8-Week Survival + Limited Baseline, Time-Dependent Set (n = 24,935)
	%	Mean (SD)	%	Mean (SD)	%	Mean (SD)	%	Mean (SD)	%	Mean (SD)	%	Mean (SD)	%	Mean (SD)	%	Mean (SD)
Vaccinated	69.0		70.3		70.8		71.2		71.4		73.7		73.3		74.5
Age, years		62.0 (14.6)		61.9 (14.6)		61.9 (14.6)		61.8 (14.6)		61.8 (14.6)		62.1 (14.6)		62.0 (14.6)		62.2 (14.5)
Male sex	53.8		53.8		53.8		53.8		53.8		54.8		54.3		54.4
Race
White	49.4		49.2		49.1		49.0		48.9		49.4		49.6		50.0
Black	44.1		44.3		44.4		44.5		44.6		43.8		43.7		43.2
Other	6.5		6.5		6.5		6.5		6.5		6.9		6.7		6.8
Cause of ESRD
Diabetes	44.8		44.8		44.7		44.7		44.6		43.5		44.1		44.2
Hypertension	30.7		30.8		30.8		30.8		30.8		31.9		31.0		31.0
Other	24.5		24.5		24.5		24.5		24.5		24.5		24.9		24.9
Years with ESRD
0	2.0		2.0		2.0		2.0		2.0		1.8		1.9		1.9
1–3	54.5		54.6		54.6		54.6		54.6		53.4		54.7		53.8
≥4	43.5		43.5		43.5		43.5		43.5		44.8		43.4		44.3
Albumin, g/dL		3.9 (0.4)		3.9 (0.4)		3.9 (0.4)		3.9 (0.4)		3.9 (0.4)		4.0 (0.3)		3.9 (0.4)		3.9 (0.4)
Hemoglobin, g/dLa		12.2 (1.3)		12.2 (1.3)		12.2 (1.3)		12.2 (1.3)		12.3 (1.3)		12.4 (1.2)		12.3 (1.2)		12.4 (1.2)
Body mass indexb		27.2 (7.1)		27.2 (7.1)		27.3 (7.1)		27.3 (7.1)		27.3 (7.1)		27.5 (7.0)		27.4 (7.1)		27.4 (7.1)

Abbreviations: ESRD, end-stage renal disease; SD, standard deviation.

^a Sample size was slightly smaller for baseline hemoglobin, which was missing for <0.1% of the sample.

^b Weight (kg)/height (m)².

When we restricted the cohort to 8 weeks of survival, applied a limited set of baseline covariate restrictions, and applied the time-dependent covariate restrictions, vaccine effectiveness during the influenza season was 9% (hazard ratio = 0.91, 95% confidence interval: 0.72, 1.15) (Web Figure 1). We chose to report the estimate from the sensitivity analysis that had the smallest variance (Web Figure 1), among those with confidence intervals crossing the null in the preinfluenza period (Figure 2). Plots of corresponding results for 2006 and 2007 are shown in Web Figures 2–5.

DISCUSSION

We attempted to control the healthy-user bias by using clinically rich data on time-dependent confounders and also through cohort restriction. Results were biased when we used a basic marginal structural model and remained biased when we incorporated an expanded set of baseline covariates and varied the length of the time window associated with each observation. The most effective bias reduction strategy was restricting the cohort to the healthiest people. When we required survival into the preinfluenza period, the estimate of preinfluenza vaccine effectiveness, which should be a hazard ratio of 1.00, moved from 0.43 to 0.66. Further restrictions moved the preinfluenza estimate closer to the null, indicating that the treatment groups were becoming more similar. We estimated vaccine effectiveness during the influenza season at 9% (95% confidence interval: −15, 28) when we applied survival requirements and restricted baseline and time-dependent variables.

Studies that have estimated influenza vaccine effectiveness in preventing death by using alternate study designs have reported similar estimates. One study that used the case-centered design estimated vaccine effectiveness to be 5% (95% confidence interval: 1, 8) among seniors (12). Another study using a natural experiment to compare years with different levels of vaccine match found no benefit of the vaccine in preventing death among patients on hemodialysis (13). It has been estimated that fewer than 10% of wintertime deaths can be attributed to influenza (27); therefore, a vaccine effectiveness estimate of 10% or greater is implausible. Vaccine effectiveness for patients on chronic hemodialysis is likely to be even lower than in the general population because of an overall decrease in immune function (28, 29) and lower immune response to vaccination (30). These facts support our finding of a very small benefit of the vaccine.

The fact that results remained biased when we added baseline covariates and changed the length of the time window associated with each time-dependent observation is unsurprising. It is likely that ICD-9-CM codes cannot be used to identify frailty accurately and are not strongly associated with vaccination (31). Additional studies have suggested that ICD-9-CM codes from administrative data may have insufficient sensitivity for identifying comorbidities, which can result in substantial residual confounding (32, 33).

Covariate adjustment failed to remove bias completely even though our set of covariates included utilization variables related to frailty, such as use of skilled nursing facilities, use of mobility aids, and oxygen use. This finding suggests that additional frailty indicators are necessary. Alternatively, the adult hemodialysis population may be frail enough that this type of measure was insufficiently sensitive to distinguish between hemodialysis patients likely to receive vaccination and those unlikely to receive vaccination because of extreme frailty.

We did find that restricting our cohort to increasingly healthy people reduced the amount of residual bias. Similar results were found in a study of statins and mortality, where restricting the analysis to patients who lacked contraindications and were adherent resulted in an estimate closer to estimates from randomized controlled trials (34). It is likely that eliminating the sicker patients reduced bias through both measured and unmeasured indicators of health status.

In our study, applying survival requirements seemed to reduce bias more, compared with restrictions based on covariate values. Unfortunately, survival requirements cannot be applied a priori to distinguish healthier patients from sicker patients. One possibility would be to develop a risk score that could be used in future studies as a covariate or to restrict the analysis sample. However, such a risk score may not be able to capture frailty or rapidly changing health status. Another option is to estimate a propensity score using pretreatment variables and to conduct sensitivity analyses by 1) checking for variation in the treatment association across propensity score strata and 2) applying increasing amounts of restriction based on the propensity score as recommended by Glynn et al. (35). These procedures can help to detect potential unmeasured confounding, indicated by unexpected treatment associations among people treated contrary to prediction and sensitivity of estimates to the exclusion of observations in the tails of the propensity score distribution.

Restricting the sample does limit interpretation of results and the ability to subsequently generalize results to a wider population. Restriction likely produces an estimate that is closer to what would be seen in a randomized controlled trial of healthy patients. However, because the restrictions we applied were complex and included survival requirements, it would be difficult to find a simple, meaningful description for the population to whom our best estimate of vaccine effectiveness applies. Additionally, applying restrictions involves a trade-off between reducing bias (due to increased homogeneity) and reducing precision (due to decreased sample size). We found that restricting time-dependent variables drastically reduced our sample size, and in some scenarios the reduced sample size prevented us from fitting the final model of preinfluenza vaccine effectiveness. In all the restriction scenarios we explored, precision was limited. It is likely, however, that reducing the strong residual confounding would yield a more valid estimate even in the presence of wide confidence intervals. A detailed statistical discussion of this trade-off is provided by Hanley and Dendukuri (36).

In the assessment of methods to reduce the healthy-user bias, one major advantage of using an influenza vaccine effectiveness study is the availability of the preinfluenza period as a negative control. It has been shown that this period is a reasonable negative control to assess the potential for residual confounding (37). Although other preventive health interventions may not have such a well-defined negative control, it is important in any nonexperimental research setting to assess the possibility of residual bias. Lipsitch et al. (38) discuss how to identify and use other kinds of negative controls in observational studies. Other studies have successfully used a negative control outcome that is causally unrelated to the exposure (24, 39–41).

One limitation of this study is that we may have missed some influenza vaccinations and, therefore, our results could be affected by exposure misclassification. This would likely bias our vaccine effectiveness estimates toward the null because relatively healthy vaccinated individuals would be classified as unvaccinated. Although there is little information about the completeness of influenza vaccine reporting in claims and dialysis clinic data, we have 3 reasons to believe that this occurred infrequently. First, we used vaccination data from 2 sources (Medicare claims and information from the dialysis provider). Second, Medicare covers the cost of influenza vaccine and administration. Therefore, it is unlikely that patients would need to pay out-of-pocket. Third, patients on dialysis have 2–3 opportunities per week to be vaccinated in the dialysis clinic, thus minimizing the need to obtain vaccine from a nontraditional provider that may require payment out-of-pocket.

Using a linked data set with clinical covariates and accounting for time-dependent confounding achieved little reduction in confounding due to the healthy-user bias. It is unlikely that more detailed data would be available to capture functional status in this population and, therefore, restriction may be a more powerful tool to overcome the healthy-user bias. Investigators should consider this strategy in other studies of preventive health interventions.