Evaluating the Case-Positive, Control Test-Negative Study Design for Influenza Vaccine Effectiveness for the Frailty Bias

Abstract

Introduction

Previous influenza vaccine effectiveness studies were criticized for their failure to control for frailty. This study was designed to see if the test-negative study design overcomes this bias.

Methods

Adults ≥50 years of age with respiratory symptoms were enrolled from November 2006 through May 2012 during the influenza season (excluding the 2009–2010 H1N1 pandemic season) to perform yearly test-negative control influenza vaccine effectiveness studies in Nashville, TN. At enrollment, both a nasal and throat swab sample were obtained and tested for influenza by RT-PCR. Frailty was calculated using a modified Rockwood Index that included 60 variables ascertained in a retrospective chart review giving a score of 0 to 1. Subjects were divided into three strata: non frail (≤0.08), pre-frail (>0.08 to <0.25), and frail (≥0.25). Vaccine effectiveness was calculated using the formula [1-adjusted odds ratio (OR)] × 100%. Adjusted ORs for individual years and all years combined were estimated by penalized multivariable logistic regression.

Results

Of 1023 hospitalized adults enrolled, 866 (84.7%) participants had complete immunization status, molecular influenza testing and covariates to calculate frailty. There were 83 influenza-positive cases and 783 test-negative controls overall, who were 74% white, 25% black, and 59% female. The median frailty index was 0.167 (Interquartile: 0.117, 0.267). The frailty index was 0.167 (0.100, 0.233) for the influenza positive cases compared to 0.183 (0.133, 0.267) for influenza negative controls (p=0.07). Vaccine effectiveness estimates were 55.2% (95%CI: 30.5, 74.2), 60.4% (95%CI: 29.5, 74.4), and 54.3% (95%CI: 28.8, 74.0) without the frailty variable, including frailty as a continuous variable, and including frailty as a categorical variable, respectively.

Conclusions

Using the case positive test negative study design to assess vaccine effectiveness, our measure of frailty was not a significant confounder as inclusion of this measure did not significantly change vaccine effectiveness estimates.

Introduction

The paucity of data from randomized clinical trials of influenza vaccine efficacy in older adults has fueled controversy over influenza vaccine effectiveness in this age group. Since all older adults are recommended to receive yearly influenza vaccine, placebo-controlled trials are considered unethical in this age group in the United States. Observational studies of influenza vaccine effectiveness using large administrative databases have often overestimated vaccine effectiveness due to confounding by frailty, which is very difficult to measure using such databases.^1,2 Frailty, in community dwelling older persons, has been shown to be associated with both a decreased likelihood of vaccination and an increased likelihood of hospitalization and/or death, confounding interpretations of vaccine effectiveness^1,2 Frailty is the conceptualization of a phenotype of poor physiologic reserve and poor resistance to stressors and hence is associated with a high risk of morbidity and death.³ Frailty was associated with immune senescence, poor response to vaccination and lower influenza vaccine effectiveness when frail individuals were compared to non-frail, age-matched participants.⁴ In previous studies, frailty scales have predicted vaccine response to polysaccharide pneumococcal vaccine better than age, and have a negative correlation with antibody response to influenza vaccination.⁵

Orenstein et al. used simulation models to evaluate which observational study designs would perform best in measuring vaccine effectiveness, and found that the case-control study design with influenza laboratory-confirmation, was a preferred design.⁶ Specifically, they demonstrated that the test-negative control study design consistently produced a vaccine effectiveness that was closer to the true vaccine effectiveness and that this relationship held true even as the ratio of influenza to non-influenza influenza-like illness (ILI) changed.⁶ In this design, cases are persons with acute respiratory illness and laboratory confirmation of influenza, and controls are persons with acute respiratory illness seeking care in the same site as cases, but who tested negative for influenza. Hence, the term case-positive, test-negative control design effectively described this study design.

The test-negative control study design has been extensively used in both the United States^7,8 and in Europe^9–11 to determine influenza vaccine effectiveness in both outpatients and hospitalized patients. One major assumption when using this approach is that cases and controls are similar with respect to comorbidity or other non-vaccine factors associated with influenza illness. In studies that include older adults, there has been an implicit assumption that frailty is not a confounder, since frailty has not been routinely measured in these studies. Therefore, it is important that this assumption be tested and confirmed in the case-positive, test-negative control study design. Using the study populations from previously reported vaccine effectiveness studies,¹² we collected additional data from comprehensive chart reviews to calculate a frailty index using a standardized measure of frailty, to determine if the case-positive, test-negative study design adequately controlled for frailty.

Methods

Adults hospitalized with respiratory symptoms were prospectively enrolled from November 2006 through May 2012 during influenza seasons to determine rates of hospital admissions and to evaluate vaccine effectiveness in Nashville (Davidson County), TN in one academic and three community hospitals.¹² The 2009–2010 pandemic H1N1 season was excluded as vaccine became available in Nashville after the peak of the epidemic curve. Recruitment occurred two days per week beginning in November until the defined influenza season had arrived at which time recruitment increased to 4–5 days per week. Adults ≥50 years of age with one or more of the following admission diagnoses (International Classification of Diseases, 9^th Revision Number): pneumonia (480–486), upper respiratory infection (465), bronchitis (466), influenza (487), acute exacerbation of chronic obstructive pulmonary disease (490 to 492;496) or asthma (493), viral illness (079.9),¹³ dyspnea (786), acute respiratory failure (518.81), pneumonitis due to solids/liquids (507), or fever (780.6) without localizing symptoms or presenting symptoms of: cough, non-localizing fever, shortness of breath, sore throat, nasal congestion or coryza were eligible for enrollment. Patients or their legally authorized representative provided informed consent. For each participating subject, both nasal and throat swab samples were obtained for RT- PCR. Patient questionnaires and chart review data collection instruments were developed to capture CDC-defined high risk conditions, symptoms, and influenza immunizations.¹²

A person’s exposure was classified as vaccinated if they received an influenza vaccine at least 2 weeks prior to the onset of symptoms, to allow time to mount an immune response. Study nurses obtained vaccine verification from both traditional (primary care physicians) and non-traditional providers (pharmacies, employers, and grocery stores) to determine the duration between vaccination and illness and to verify patient report for both vaccinated and non-vaccinated patients. Patients were excluded if immunization occurred within two weeks of illness onset since some older adults will mount a seroprotective response as early as 7 days¹⁴ and may be inappropriately assigned to the non-vaccinated group.

Influenza testing was performed in a research laboratory where laboratory assistants were unaware of patient’s vaccination status. Influenza positive cases were participants with positive RT-PCR on duplicate testing. Influenza negative controls were participants with an acute respiratory illness who tested negative for influenza by RT-PCR and had an adequate sample demonstrated by evidence of housekeeping genes β-actin or RNase P in the sample. To decrease outcome and exposure misclassification, patients with indeterminate laboratory results, unknown vaccination status, or vaccinated within 2 weeks of presentation were excluded from the analyses. For a sensitivity analysis, patients with more than one admission, only the first influenza positive enrollment or the first enrollment, if none were influenza positive, was included.

Definitions and Covariates

Influenza seasons were defined by the total number of weeks that included all influenza positive specimens from enrolled patients each year. Covariates obtained by self-report or chart review included age in years, sex, race (black, non-black), current smoking (in the past 6 months), home oxygen use, underlying medical conditions (diabetes mellitus, chronic heart or kidney disease, cardiovascular disease, asthma, chronic obstructive pulmonary disease, and asplenia (functional or anatomic), immunosuppression (HIV, corticosteroid use, or cancer), timing of admission relative to the onset of influenza season, and the specific influenza season. All covariates were considered as potential confounding variables.

Frailty

Frailty was calculated using a modified Rockwood Index. The Rockwood Index¹⁵ includes 70 categories of medical problems and functional issues. For each category present, a point is awarded. The total number of points is divided by 70 giving the final index result giving a score of 0 to 1. This index can be effectively shortened if 30 of the original 70 variables are retained.¹⁶ In our hospitals, clinical nurses regularly documented 60 of the original 70 categories. We ascertained these 60 variables in a retrospective chart review (variables listed in the appendix) and hence divided by 60 to obtain the index. Nurses were masked to influenza testing results at time of abstraction. Frailty was categorized into three strata: non frail (Frailty Index ≤0.08), pre-frail (Frailty Index >0.08 to <0.25), and frail (Frailty Index ≥0.25).¹⁷

Analysis

Characteristics of participants with and without frailty data were compared using Pearson Chi-square test for categorical covariates and Wilcoxon rank sum test for continuous variables. Vaccine effectiveness estimates were calculated using the formula [1-adjusted odds ratio (OR)] × 100%.¹⁸ Adjusted ORs for individual years and all years were estimated by multivariable logistic regression models with L1 penalty on all covariates except vaccination status (LASSO).¹⁹ The model outcome was influenza positive cases or negative controls and the exposure of interest was vaccination status while adjusting for the other covariates listed above. Restricted cubic spline function was applied to the variables age and week of influenza season with 3 knots for each variable. Three similar models were run: one model did not include frailty, one included frailty as a continuous covariate and applied restricted cubic splines with three knots, and one model included frailty as a categorical covariate with three levels. We used 10,000 bootstrap samples to construct the 95% confidence interval for the ratio of the two vaccine effectiveness estimates derived from models with and without adjusting for frailty. Using a definition similar to the FDA definition for bioequivalence, we assumed the estimates were equivalent if the ratio of the two estimates fell between 0.85 and 1.25 (the 95% confidence interval).

Results

During the 5 study years, 1023 hospitalized adults ≥50 years of age were enrolled. We were able to retrospectively obtain data on frailty for 978 (95.6%) participants. These 978 participants were white (72%), female (59%), vaccinated (74%), smoked (24%) and had a median age of 68.4 years. Participants for whom we obtained frailty data were similar to those for whom frailty data were missing, except the latter were slightly younger (median 60.5 years vs 68.4 years), more likely to be female (77.8% vs 59.3%) and more likely to be immunosuppressed (49% vs 32%). Of the 978 participants with adequate data to determine frailty, 866 (84.7%) had complete data on immunization status, influenza testing, and covariates. This group included 83 influenza-positive cases and 783 test-negative controls. These participants were 74% white, 25% black, and 59% female. Of the 645 participants immunized, one received a live-attenuated vaccine (0.15%), 16 received the high dose inactivated vaccine (2.4%) and the reminder received the standard dose inactivated influenza vaccine (97.4%).

The mean frailty index was 0.205 ± 0.110 and the median was 0.167 (IQR: 0.117, 0.267); 30.8% (267/866) met the definition for frail and 63.5% (550/866) were prefrail. Figure 1 is a histogram of the frailty scores of the participants showing that almost all patients hospitalized with an acute respiratory illness were either frail or pre-frail. Frailty (median and IQR) for those influenza positive was 0.167 (0.100, 0.233) compared to 0.183 (0.133, 0.267) for those influenza negative (p=0.074). Figure 2 shows the degree of frailty by age group, gender, past medical history, race, immunization history, and influenza status. In univariate analyses increasing age (p<0.001), white race (p=0.003), chronic cardiovascular disease (p<0.001), diabetes mellitus (p<0.001), and immunization with influenza vaccine (p<0.001) were associated with greater frailty.

Figure 1. Histogram of Frailty Index.

This graphic shows the number of participants with each frailty score. The lines are placed at 0.08 and 0.25 to categorize the participants as not frail, pre-frail, and frail.

Figure 2. Median Frailty Value by demographics, comorbidities, and immunization status.

Vaccine effectiveness was estimated without frailty, including frailty as a continuous variable (restricted cubic spline with three knots), and including frailty as a categorical variable. Estimates were 55.2% (95%CI: 30.5, 74.2), 60.4% (95%CI: 29.5, 74.4), and 54.3% (95%CI: 28.8, 74.0), respectively. The ratio of vaccine effectiveness estimates between the model without frailty and the model including frailty as a continuous variable was 1.02 (95% CI: 0.86, 1.25) and the ratio between the model without frailty and the model including frailty as a categorical variable was 0.91(95% CI: 0.83, 1.23). The differences in vaccine effectiveness estimates were not significant and therefore, frailty was not a significant confounder. Vaccine effectiveness estimates were also estimated for each level of frailty and were found to be 38.0% (−32.5, 99.1), 65.8% (39.8. 83.2) and 10.9% (−3.4, 67.7) for not frail, pre-frail, and frail respectively. (Table 1)

Table 1.

Influenza Vaccine Effectiveness by level of Frailty

	Influenza positive/ Total not immunized (%)	Influenza positive/ Total immunized (%)	Vaccine Effectiveness (95% CI)
Overall	37/221 (16.7)	46/645 (7.1)	55.2% (30.5, 74.2)
Not Frail	4/23 (17.4)	3/26 (11.5)	38.0% (−32.5, 99.1)
Pre-Frail	29/149 (19.5)	27/401 (6.7)	65.8% (39.8, 83.2)
Frail	4/49 (8.2)	16/218 (7.3)	10.9% (−3.4 67.7)

Discussion

Our study adds several pieces of knowledge about adults hospitalized with acute respiratory illness, as well as how frailty affects the measurement of influenza vaccine effectiveness using the hospital-based test-negative design. First, it is very clear that the majority of hospitalized patients were either pre-frail or frail. Prior studies have shown that about 22.7% of community dwelling adults age 65 years and older are frail¹⁷ compared to the 72.7% of adults 65 years and older hospitalized with acute respiratory illness in this study.

Second, we were unable to show frailty to be a major confounder in the test-negative design, unlike large cohort studies that compared vaccinated and unvaccinated community dwelling adults. In the test-negative design done in the hospital setting, controls are chosen to represent the population who would require hospitalization if they became ill with influenza. Thus, controls are patients with other acute respiratory illness, who were also hospitalized. We did find a modest difference in the frailty of those immunized and not immunized (0.183 vs 0.150, p <0.001) but the means of both groups were pre-frail and the difference in frailty was not likely clinically significant. Vaccine effectiveness estimates did not change significantly when frailty was added to the test-negative analyses, indicating that it was not a confounder in our population using this study design. It thus appears that frailty is adequately controlled for when both cases and controls are seeking medical care in the hospital setting.

Finally, we evaluated whether frailty was associated with poor vaccine effectiveness. The vaccine effectiveness estimate was 65.8% (39.8, 83.2) for those that were pre-frail. However estimates were imprecise for those not frail and frail. Further studies are needed to determine vaccine effectiveness in those who are frail. Frail individuals are likely to be most impacted by immune senescence.^4,5 Hence vaccines may be less effective in those who are frail. This study is limited by small numbers of those that are frail. The lower proportion of frail with a diagnosis of influenza could be due to higher frequency of other causes of acute respiratory illness, lower exposure to influenza, and/or failure to identify influenza-associated illness in frail individuals.

In conclusion, frailty does not appear to be a significant confounder in the test-negative study design since inclusion of a validated measure of frailty did not substantially change vaccine effectiveness estimates. Our study indicates that the test-negative study adequately controlled for frailty without inclusion of a specific frailty index. However, more studies are needed to determine vaccine effectiveness in frail, older adults.

Highlights.

• We evaluated the test-negative study design for bias due to frailty.

• No evidence of bias due to frailty was seen in the test-negative study design.

• Most hospitalizations due to influenza are in those who are either frail or pre-frail.