Hospital-reported data on the pneumonia quality measure “time to first antibiotic dose” is not associated with inpatient mortality: results of a nationwide cross-sectional analysis

Abstract

Objectives

Significant controversy exists regarding the Centers for Medicare & Medicaid Services (CMS) “time to first antibiotics dose” (TFAD) quality measure. The objective of this study was to determine whether hospital performance on the TFAD measure for patients admitted from the emergency department (ED) for pneumonia is associated with decreased mortality.

Methods

This was a cross-sectional analysis of 95,704 adult ED admissions with a principal diagnosis of pneumonia from 530 hospitals in the 2007 Nationwide Inpatient Sample. The sample was merged with 2007 CMS Hospital Compare data, and hospitals were categorized into TFAD performance quartiles. Univariate association of TFAD performance with inpatient mortality was evaluated by chi-square test. A population-averaged logistic regression model was created with an exchangeable working correlation matrix of inpatient mortality adjusted for age, sex, co-morbid conditions, weekend admission, payer status, income level, hospital size, hospital location, teaching status, and TFAD performance.

Results

Patients had a mean age of 69.3 years. In the adjusted analysis, increasing age was associated with increased mortality with ORs > 2.3. Unadjusted inpatient mortality was 4.1% (95% confidence interval [CI] = 3.9% to 4.2%). Median time to death was five days (25-75^th interquartile range [IQR]: 2-11). Mean TFAD quality performance was 77.7% across all hospitals (95% CI = 77.6% to 77.8%). The risk adjusted odds ratio (OR) of mortality was 0.89 (95% CI = 0.77 to 1.02) in the highest performing TFAD quartile, compared to the lowest performing TFAD quartile. The second highest performing quartile OR was 0.94 (95% CI = 0.82 to 1.08), and third highest performing quartile was 0.91 (95% CI = 0.79 to 1.05).

Conclusions

In this nationwide heterogeneous 2007 sample, there was no association between the publicly reported TFAD quality measure performance and pneumonia inpatient mortality.

Introduction

Pneumonia is a significant cause of mortality in the United States.^1,2 It accounts for 4.3% of all emergency department (ED) admissions, and the cost of treating community acquired pneumonia is estimated at $10 billion dollars per year, with 92% of costs occurring in the inpatient setting.^3-5

The Centers for Medicare & Medicaid Services (CMS) and The Joint Commission (TJC) have defined standardized quality measures that hospitals are required to publicly report and are evaluated on.⁶ One such quality measure relating to pneumonia is the time to first antibiotic dose (TFAD), with the current requirement that the initial antibiotic is given within six hours of hospital arrival (increased from four hours in April 2008).⁷

There has been significant debate regarding the appropriateness of the TFAD measure.^8-11 In holding hospitals to this standard, it is reasonable to expect that achieving this goal would correlate with improved outcome and decreased inpatient mortality. The evidence for the TFAD measure originated in two studies of Medicare patients that demonstrated mortality benefit with earlier initiation of antibiotic therapy.^12,13 But since implementation of the quality measure, subsequent studies have failed to reproduce this mortality benefit.^14-16 Furthermore, there is the potential for unintended consequences of enforcing this quality measure, such as antibiotic overuse, antibiotic misuse, misdiagnosis, and mis-prioritization of patients.^8,9,17-23

The objective of the present study was to determine whether hospital performance on the TFAD quality measure for patients admitted from the ED with a principal diagnosis of pneumonia is associated with decreased inpatient mortality.

Methods

Study Design

We performed a cross-sectional analysis of the 2007 Healthcare Cost and Utilization Project's (HCUP) Nationwide Inpatient Sample (NIS). The Northwestern University institutional review board found this study of de-identified data exempt from informed consent requirements.

Study Setting and Population

The NIS is the largest publicly available all-payer inpatient database in the United States and is provided by the U.S. Agency for Healthcare Research and Quality. Data are weighted to result in a sample that is representative of all admissions to non-federal United States hospitals on an annual basis. Additional detail on the NIS can be found on the HCUP website (http://www.hcup-us.ahrq.gov/). We linked this dataset with the publicly available 2007 CMS Hospital Compare hospital scores for the TFAD measure. The Hospital Compare data report the percentage of pneumonia admissions that achieved TFAD within the quality measure goal of four hours. The ability to link patient-level data from NIS to hospital-level data from Hospital Compare was possible by using the American Hospital Association (AHA) identification number provided within the NIS dataset.

The 2007 NIS dataset contains 8,043,415 inpatient discharge records from 1,044 hospitals located in 40 states. Admissions for pneumonia were identified based on principal diagnosis International Classification of Disease-9th revision (ICD-9) codes. We used similar ICD-9 codes as found in other large pneumonia studies.^13,24 The study sample included patients 18 years or older with an ICD-9 principal discharge diagnosis of 480, 480.0-480.3, 480.8-480.9, 481, 482, 482.0-482.4, 482.8-482.9, 483, 483.0- 483.1, 438.8, 484, 484.0-484.1, 484.3, 484.5-484.8, 485, 486, or 487. Because we wanted to focus on the acute identification and diagnosis of pneumonia, patients were excluded from the analysis if the admission was from a source other than the ED. The AHA identification number linked the NIS patient records to the 2007 CMS Hospital Compare TFAD measure. Therefore, records were excluded if the AHA identification number was not included in the NIS or if the hospital of admission did not report the TFAD quality measure.

Study Protocol

Patient and Hospital Variables: NIS

NIS data included admission source, discharge status (alive or dead), sex, age (divided into quintiles), and race and ethnicity (black, Hispanic, white, other, or missing). Race and ethnicity data were missing for 26.0% of study patients from states where this information was not mandated, and therefore this demographic information was not included in the multivariate analysis. NIS data also included length of stay, weekend admission, primary payer source, and patient zip code median income (divided into quartiles). Individual income level was not available from the NIS. However, we coded patients with a primary payer status of Medicaid or self-pay (uninsured) into a single category as an additional proxy for lower socio-economic status.

To account for confounding due to co-morbid conditions present on admission that may have influenced mortality, we used the Elixhauser ICD-9 secondary diagnosis codes to identify concurrent admission conditions. This co-morbid condition coding scheme has often been used in previously published mortality models to adjust for severity of pre-existing illness for critically ill patients.^25-27 The co-morbidities included in the Elixhauser Index are very similar to those of other risk-adjustment schemes, such as the scheme used by Hospital Compare to account for deaths unrelated to hospital care for their quality measure analysis. The NIS classified patients as having or not having each of 30 Elixhauser chronic co-morbid conditions. For the purpose of our analysis, patients were then categorized by the number of concurrent conditions present at admission (0, 1, 2, 3, 4, 5+).

Hospital-level data in the NIS included bed size (categorized by NIS as small, medium, or large; based on the number of short-term acute beds in the hospital and specific to the hospital region, location, and teaching-status), teaching status (member of the Council of Teaching Hospitals, or not), and location (rural versus urban). To assess whether TFAD performance was an independent predictor of mortality, these hospital characteristics were also included in multivariate models of inpatient mortality.

TFAD Measure

The 2007 CMS Hospital Compare data reported the TFAD measure as the mean percentage of admissions for pneumonia where the first dose of antibiotics was received within four hours of arrival at the hospital. The CMS Hospital Compare also reported the AHA hospital identifier, and this was used to link the mean percentage score to each admission record in the NIS. Records were then divided into balanced quartiles based on mean TFAD percentage. Further information on the 2007 CMS Hospital Compare report is available at http://www.qualitymeasures.ahrq.gov/summary/summary.aspx?doc_id=13224.

Key Outcome Measure

The key outcome measure in this study was overall all-cause inpatient mortality.

Data Analysis

Analyses were conducted with STATA version 10.0 (STATA Corp, College Station, TX). Patient and hospital characteristic data were evaluated across TFAD performance quartiles with chi-square tests for categorical variables, and the analysis of variance (ANOVA) for continuous measures. The unadjusted association of TFAD performance with mortality was evaluated continuously with a t-test and by quartiles with the chi-square test. Significance between TFAD quartiles was conducted by one-way ANOVAs with post-hoc Bonferroni tests. A population-averaged logistic regression model with an exchangeable working correlation matrix was used to analyze the simultaneous effects of age, sex, number of co-morbid conditions, weekend admission, payer status, income level, hospital size, location, teaching status, and TFAD performance on inpatient mortality. This approach accounted for nesting of admissions within hospitals.²⁸ Forward censoring was not used in developing this regression model. This multilevel statistical modeling takes into account both individual, and hospital-level variables, and adjusts for “nesting of admissions,” or similarity of patients presenting to one hospital compared to another hospital. Results are reported as odds ratios (ORs) and 95% confidence intervals (95% CI). Significant associations were interpreted as p < 0.05.

Results

The ED pneumonia patient sample

The final study population included 95,705 patients from 530 hospitals (Figure 1). The overall inpatient mortality for this nationwide sample of admitted patients with acute pneumonia was 4.1% (95% CI = 3.9% to 4.2%). Median time to death was 5 days (25-75^th IQR: 2 to 11 days). Overall, hospital-reported data suggested a relatively high percentage of admissions meeting the TFAD within four hours. The mean TFAD performance score across all hospitals was 77.7% (95% CI = 77.6% to 77.8%). The TFAD performance quartiles were: highest performing quartile 86% to 100%, second highest performing quartile 79% to 85%, third highest performing quartile 72% to 78%, lowest performing quartile 0 to 71%.

Figure 1.

Study Population Sample Selection.

NIS = Nationwide Inpatient Sample; AHA = American Hospital Association

Unadjusted patient and hospital characteristics across TFAD performance quartiles

Table 1 displays patient and hospital characteristics for the 2007 ED pneumonia admission sample across TFAD performance quartiles. The mean age of the study sample was 69.3 years (95% CI = 69.2 to 69.4 years). There was a significant trend towards older patients being more represented in the higher performing quartiles in a post-hoc analysis (p < 0.001). Just over half of the population was female (52.1%). The mean length of stay (LOS) was 5.59 days (95% CI = 5.56 to 5.63 days). The highest performing hospitals had a mean LOS of 5.2 (95% CI = 5.14 to 5.26 days) and the lowest performing hospitals had a mean LOS of 5.84 (95% CI = 5.76 to 5.92 days).

Table 1. Unadjusted association of patient and hospital characteristics with quality measure time to first antibiotic dose (TFAD) performance quartile.

	Overall Sample	Antibiotic Compliance
		1st Quartile	2nd Quartile	3rd Quartile	4th Quartile
Patient Characteristics
Age (in years)*	69.3	67.2	68.8	70.3	70.9
Hospital LOS (days)	5.59	5.84	5.72	5.61	5.20
Race
Missing†	26.08	17.40	24.82	30.88	30.96
White	58.70	59.29	57.90	56.79	60.90
Black	8.18	11.63	9.62	7.45	4.09
Latino	7.04	11.68	7.66	4.88	4.06
Sex
Male	47.88	48.14	47.61	47.28	48.49
Female	52.12	51.86	52.39	52.72	51.51
Number of co-morbid conditions
0	3.92	4.31	3.90	3.57	3.91
1	12.62	13.19	11.95	12.47	12.82
2	20.98	20.77	20.74	20.79	21.62
3	23.53	22.96	23.28	23.79	24.09
4	18.70	18.15	18.67	19.44	18.51
5+	20.24	20.61	21.46	19.95	19.05
Median income
1st quartile (lowest)	28.13	34.44	27.28	29.20	21.46
2nd quartile	26.01	26.57	28.25	23.46	26.12
3rd quartile	24.20	21.42	25.53	24.33	24.57
4th quartile (highest)	21.66	17.57	18.94	22.01	27.85
Medicaid/Uninsured
No	87.24	83.61	85.84	88.58	90.83
Yes	12.76	16.39	14.16	11.42	9.17
Hospital Characteristics
Rural
No	83.04	90.95	91.61	81.03	69.22
Yes	16.96	9.05	8.39	18.97	30.78
Region
Northeast	26.58	21.29	24.23	29.08	31.48
Midwest	14.38	7.06	12.06	20.08	21.90
South	29.06	40.09	33.91	24.85	17.84
West	28.96	31.55	29.80	26.00	28.78
Teaching Status
No	62.34	47.92	59.95	66.81	74.47
Yes	37.66	52.08	40.05	33.19	25.53
Hospital Size
Small	14.94	6.60	10.68	20.27	21.68
Medium	25.46	21.08	29.95	20.39	31.20
Large	59.60	72.32	59.37	59.34	47.12

^*All p values < 0.05 All values are percents unless otherwise stated

^†Total number missing race data: 24,959

1st quartile = lowest performing hospitals, 4th quartile = highest performing hospitals

LOS = length of stay

For states with available data on race and ethnicity (74.0% of the sample), we found a predominantly white population (58.7%), with 8.2% being black and 7.0% being Hispanic. There were a greater proportion of black and Hispanic patients in the lower performing hospitals when compared to higher performing hospitals (p < 0.001). The majority of the population (62.5%) had three or more Elixhauser co-morbid conditions present on admission. There was a heterogeneous mix of median income levels for the patient sample, and lower performing hospitals had a greater proportion of low-income patients. Furthermore, higher performing hospitals had significantly fewer patients with Medicaid or without insurance.

Of the sample, 83.0% were from urban hospitals and 59.6% were from large bed-size hospitals. Over half of ED admissions for pneumonia were to non-teaching hospitals (62.3%). There were a greater proportion of urban hospitals in the two lowest performing quartiles as compared with the two highest performing quartiles. The lowest performing quartile also included a significantly higher percentage of large bed-size hospitals (73.3% versus 47.1% in the highest quartile, p < 0.001). Last, the highest performing quartile hospitals were more likely to be non-teaching hospitals (74.5% versus 47.9% in the lowest quartile, p < 0.001).

Unadjusted Association of TFAD performance quartiles and mortality

There was no significant difference in inpatient mortality between hospitals across the four TFAD performance quartiles, with mortality ranging from 4.01% to 4.08% (p = 0.98) (Figure 2). Mean hospital compliance with the TFAD measure did not significantly differ between patients who died and those who survived their hospital admission (77.7% and 77.8%, respectively; p = 0.71).

Figure 2.

Unadjusted Inpatient Mortality.

Unadjusted inpatient mortality of patients with a principal diagnosis of pneumonia, admitted through the emergency department, by time to first antibiotic dose quality measure performance quartile.

*1st quartile = lowest performing hospitals, 4th quartile = highest performing hospitals

Adjusted Analysis: patient and hospital factors and mortality

Results from adjusted analysis identified increasing age, female sex, and increasing number of co-morbid conditions as predictor variables independently associated with increasing odds of inpatient mortality (Table 2). Patients over the age of 85 were over seven times more likely to die than those 18 to 44 years of age (OR 7.60; 95% CI = 5.98 to 9.65 years). Those with five or more co-morbid conditions were over two times more likely to die than those with no co-morbid conditions (OR 2.4; 95% CI = 1.88 to 3.18).

Table 2. Adjusted analysis of the likelihood of inpatient mortality.

	Inpatient Mortality
	Odds Ratio	95% CI
Patient Characteristics
Age (years)
18-44	Reference
45-64	2.27	1.79-2.88
65-84	4.18	3.30-5.29
> 84	7.60	5.98-9.65
Female	0.82	0.77-0.88
Number of co-morbid conditions
0	Reference
1	1.11	0.84-1.47
2	1.38	1.06-1.80
3	1.51	1.16-1.97
4	1.72	1.32-2.25
5+	2.44	1.88-3.18
Median Income
1st quartile	Reference
2nd quartile	0.99	0.90-1.09
3rd quartile	1.05	0.95-1.16
4th quartile (highest)	1.04	0.93-1.16
Medicaid/self-pay	1.06	0.92-1.22
Hospital Characteristics
Rural	1.10	0.97-1.24
Teaching hospital	0.98	0.88-1.08
Hospital size
Small	Reference
Medium	1.10	0.96-1.27
Large	1.06	0.93-1.21
Weekend admission	1.04	0.97-1.12
TFAD Quality Performance
Lowest performing quartile	Reference
3rd highest performing quartile	0.91	0.79-1.05
2nd highest performing quartile	0.94	0.82-1.08
Highest performing quartile	0.89	0.77-1.02

TFAD = time to first antibiotic dose

None of the hospital characteristics were significant as independent predictors of inpatient mortality. Specifically, hospital compliance with antibiotic timeline guidelines did not predict inpatient mortality. The highest performing TFAD hospital quartile had an OR of 0.89 (95% CI = 0.77 to 1.02) compared to the lowest performing TFAD hospital quartile. The second and third highest quartiles had ORs of 0.94 (95% CI = 0.82 to 1.08) and 0.91 (95% CI = 0.79 to 1.05), respectively.

Discussion

This is the first work to demonstrate a lack of an association between hospital compliance with the time to first antibiotic dose for pneumonia and inpatient mortality in a large cohort of adult patients admitted from EDs across the United States. We demonstrated no significant mortality difference between ED admissions from hospitals performing in the highest performing quartile and ED admissions from hospitals performing in the lowest performing quartile for the TFAD quality performance measure. Furthermore, there was no significant difference in mean hospital compliance with the TFAD measure among patients who died and those who survived their pneumonia hospital admission. There was a statistically significant relationship between hospital length of stay and hospital performance on the TFAD quality measure. However, the clinical significance of 5.20 vs 5.84 days is uncertain, and may not help clarify conflicting evidence on the impact of this secondary outcome.¹⁰

There were several similarities between our study and previous studies. We demonstrated a clear trend towards higher inpatient mortality from pneumonia with increasing age.^15,29-32 We further demonstrated higher inpatient mortality with increasing burden of co-morbid disease.³⁰ Our finding that larger hospital bed-size and teaching hospitals were associated with lower hospital performance on the TFAD measure is consistent with results of other studies investigating pneumonia quality measures and hospital characteristics.^33,34 It is possible that the larger, urban, teaching hospitals care for sicker, more complex patients who have not been captured by our analysis if pneumonia was a secondary diagnosis after sepsis or respiratory failure, thus giving these hospitals a deflated representation of their quality measure performance. In addition, these hospitals generally have much longer wait times than smaller, community hospitals, which could influence their TFAD if compliance with the quality measure does not have a high priority. We also demonstrated that females have a significantly decreased inpatient pneumonia mortality compared to males. The basis of the relationship between sex and inpatient pneumonia mortality has not been specifically investigated in prior studies that we are aware of, but studies have demonstrated sex disparity with the diagnosis of pneumonia.^35,36 Further evaluation of the relationship between and sex and inpatient pneumonia mortality should be pursued.

Previous studies have demonstrated mixed associations between the TFAD quality measure and mortality as reviewed by Yu and Wyer in a meta-analysis of studies comparing inpatient or 30-day mortality among patients receiving early versus delayed antibiotics.¹⁰ Many time-to-antibiotic studies have been limited by small sample sizes, were limited to the Medicare population, or included only patients over the age of 65 years.^{12,13,16,30,37,38} The CMS/TJC quality measure was originally based on studies performed by Houck et al. and Meehan et al., which were limited to a Medicare population of adults over the age of 65 years.^12,13 In practice, the TFAD quality measure is applied to all patients over the age of 18 years, as was included in this study. Our results from a decade after the original study on TFAD quality measure, however, add to the body of literature supporting that there is no association between performance on the TFAD measure and mortality in a general adult population.^10,15,16,30

The TFAD measure has been called a “flawed performance measure.”⁹ Evidence in the literature suggesting possible unintended or harmful consequences of strategizing ED care specifically toward the TFAD quality measure is substantial and growing. Problems have been identified, including misdiagnosis, flawed triage, lack of cost-effectiveness, and inappropriate delivery of antibiotics.^8,9,11,17-23 In 2009, the American Academy of Emergency Medicine issued a literature review and concluded that the TFAD measure should be discontinued as a performance measure.¹¹

Although this study does not investigate the reason for lack of survival improvement with TFAD, we can speculate why we reached different outcomes than the original studies that demonstrated such an improvement. It is possible that there was a much larger disparity in timing of antibiotic delivery to pneumonia patients during the period before and just after the initiation of the quality measure. This wide disparity in care may have led to significant mortality outcome differences in patients as reported in the original studies leading to the quality measure. However, there is now a more uniform time delivery of antibiotics among hospitals, and the pendulum may have swung the opposite direction so that the negative consequences of implementing the TFAD quality measure have superseded any negligible mortality improvements. Implementation of quality measures may require repeated evaluation of their benefit to ensure original intent of the measure is indeed consistent with ongoing medical practice.

Limitations

There were several limitations to this study, many of which are inherent to secondary data analysis of population-based hospital discharge datasets. We selected patients on the basis of principal discharge diagnosis, which relies on accuracy of internal hospital reporting and coding, similar to how patients are presently identified for quality measure performance. The principal diagnosis is generally defined as the diagnosis chiefly responsible for admission to the hospital, but we were unable to confirm that each patient was treated for pneumonia in the ED.³⁹ Moreover, because we relied on discharge diagnosis, and wanted to best capture patients with the primary diagnosis of pneumonia admitted through the ED, we did not include the ICD-9 codes with a primary diagnosis of “respiratory failure” or “sepsis” and a secondary diagnosis of pneumonia, since we believe this would incorporate too many patients who developed pneumonia after admission.

Similarly, the hospital-level data we abstracted from the CMS Hospital Compare website are self-reported data, which may contribute to the narrow range of overall hospital compliance with the quality measure. If all hospitals are reporting similar compliance, the mean performance may have been too uniform to demonstrate an effect on mortality. Furthermore, there may be unmeasured confounding hospital-level data that could influence outcome. We also can not account for the uniformity of data collection between, and even within, hospitals reporting to Hospital Compare.

We were limited by the data provided through the NIS. We were unable to adjust for severity of the specific pneumonia illness, as data required for standard risk stratification means, such as the Pneumonia Severity Index Score's laboratory and vital sign data, were not available.⁴⁰

Our study was a comparison of hospital-level data on TFAD performance and patient-level inpatient mortality. This may somewhat limit the extent to which our findings can be applied to an individual patient; however, it much more accurately approximates the administration of the current performance measure, which is also reported as hospital-level data. We are not able to comment on the association, or lack of association, between the timing of antibiotic administration for pneumonia in a particular patient and individual inpatient mortality. We are also not able to analyze or comment on further outcome measures beyond inpatient mortality, such as 30-day mortality or morbidity measures secondary to delayed antibiotic administration, or inappropriate antibiotic administration. Furthermore, although we are able to identify those patients with inpatient mortality, we are unable to identify cause of inpatient death, and if it was a direct result of pneumonia or involved withdrawal of care.

Despite these limitations, we feel the large sample size and associated power allow for the evaluation of even a small effect size, and provide meaningful data calling into question nationwide application of the TFAD quality measure for pneumonia as a way to improve nationwide pneumonia mortality. Further work in other yearly samples after the change to the six-hour time frame could confirm our findings, as could investigating relationships between quality measure performance data and other meaningful outcomes, such as hospital resource utilization and cost, or patient morbidity.

Conclusions

In this large heterogeneous nationwide sample, there was no association between hospital-reported time to first antibiotic dose quality measure performance and inpatient mortality for patients admitted through the ED for pneumonia. These results raise questions about the quality measure as a benchmark for hospital quality of care in acute pneumonia.