Temporal changes in the influence of hospitals and regional healthcare networks on severe sepsis mortality

Abstract

Background

There is systematic variation between hospitals in their care of severe sepsis, but little information on whether this variation impacts sepsis-related mortality, or how hospitals’ and health-systems’ impacts have changed over time.

Objective

To examine whether hospital and regional organization of severe sepsis care is associated with meaningful differences in 30-day mortality in a large integrated health care system, and to examine the extent to which those effects are stable over time.

Methods

Retrospective analysis of 43,733 patients with severe sepsis in 114 hospitals and 21 geographic regions in the United States Department of Veterans Affairs in 2012, compared to 33,095 such patients in 2008. We used risk- and reliability-adjusted hierarchical logistic regression to estimate hospital- and region-level random effects after controlling for severity of illness using a rich mix of administrative and clinical laboratory data.

Results

The median hospital in the worst quintile of performers had a risk-adjusted 30-day mortality of 16.7% (95% CI: 13.5%, 20.5%) in 2012, compared to the best quintile, which had a risk-adjusted mortality of 12.8% (95% CI: 10.7%, 15.3%). Hospitals and regions explained a statistically and clinically significant proportion of the variation in patient outcomes.

30-day mortality after severe sepsis declined from 18.3% in 2008 to 14.7% in 2012 despite very similar severity of illness between years. The proportion of the variance in sepsis-related mortality explained by hospitals and regions was stable between 2008 and 2012.

Conclusions

In this large integrated healthcare system, there is clinically significant variation in sepsis-related mortality associated with hospitals and regions. The proportion of variance explained by hospitals and regions has been stable over time, although sepsis-related mortality has declined.

INTRODUCTION

Severe sepsis is a leading cause of hospitalization throughout the developed world^1–3, accounting for over 1 million hospitalizations each year in the U.S. alone^4,5. Despite recent improvements, mortality remains unacceptably high. One in five patients with severe sepsis dies during their hospitalization³—one in two with septic shock. Conversely, of patients who die as inpatients of any cause, between one third and one half will die with sepsis^6,7. Much of the existing research on severe sepsis outcomes has focused on the effect of disease biology and individual patient characteristics⁸. However, the type and quality of care delivered to severe sepsis patients varies widely across hospitals⁹, and this variation may influence patient outcomes.

Top-down approaches to improving sepsis care across multi-hospital systems have proven beneficial in select settings, demonstrated by dramatic improvements in processes of care measures and short-term mortality^10–12. However, these initiatives are labor-intensive and often require sustained attention to maintain improvements¹⁰. What is less clear is: in the absence of sustained orchestrated effort, are hospitals, and the regional healthcare systems of which they are a part, a source of variation in sepsis outcomes? Should it matter to patients where they get their sepsis care? And, with the increasing but highly uneven dissemination of sepsis quality improvement efforts, are these hospital and region-level effects growing or shrinking over time?

We sought to understand the extent to which variation in short-term mortality following severe sepsis is explained by the hospital and regional healthcare network where a patient receives care. We tested the following hypotheses about 30-day mortality within the U.S.-nationwide Veteran’s Affairs (VA) health system:

• (H1) even after risk-adjustment, there is statistically significant and clinically meaningful variation in short-term mortality after severe sepsis hospitalization between hospitals;

• (H2) even after adjusting for patient case-mix and hospital-level effects, there is statistically significant and clinical meaningful variation in short-term mortality after severe sepsis hospitalization between regional healthcare networks;

• (H3) the hospital and region-level effects have been growing over time, potentially as a result of variable penetration of quality improvement initiatives for timely and effective treatment of severe sepsis, changes in physician staffing patterns, or other differences in care delivery over time.

METHODS

Organizational Context

In contrast to the U.S. private market, the VA health system has delivered healthcare through 21 geographically organized healthcare networks—Veterans’ Integrated Service Networks—since the mid-1990s¹³. Each network includes dozens of outpatient clinics and approximately 10 VA hospitals, and is charged with providing comprehensive inpatient and outpatient healthcare to Veterans living within a defined geographic catchment area^13,14. The networks are funded through capitated payments based on number of Veterans served and receive financial incentives to meet clinical process and outcome benchmarks¹⁵. While the VA has been a leader in performance measurement, sepsis care was not an area of focus during the study period.

Data and Sample

For our primary analysis, we identified patients’ first hospitalization involving severe sepsis at any of 114 VA Hospitals during 2012. We defined severe sepsis hospitalizations using a commonly employed, validated method that requires ICD-9-CM coding for infection and acute organ dysfunction within the same hospitalization^1,16. Importantly, this is an implicit definition that does not require physician recognition of or coding for sepsis, so is less susceptible to the well-described secular changes in this regard¹⁷. To explore temporal changes in the hospital and region-level effects on severe sepsis mortality, we also identified first severe sepsis hospitalizations in 2008. For each hospitalization, we abstracted age, gender, race, admission source, principal diagnosis (in 221 categories aggregated from ICD-9-CM codes), Elixhauser co-morbidities¹⁸, 11 laboratory values (sodium, blood urea nitrogen, glomerular filtration rate, glucose, albumin, bilirubin, white blood cell count, hematocrit, pH, PaCO2, and PaO2; collected for the first 24 hours after patients met criteria for severe sepsis), and death at 30 days from the VA Inpatient Evaluation Center^19,20.

Statistical Analysis

General Approach

We used 3-level logistic regression models to assess variation in risk- and reliability-adjusted rates of 30-day mortality following admission in the patients’ first severe sepsis hospitalization in a given year. Patients were nested in hospitals, themselves nested within regions. We built separate models for 2012 and 2008 and calculated adjusted 30-day mortality rates for each hospital and region. This research was approved with the Ann Arbor VA Institutional Review Board. We conducted all analyses using Stata software version 13 (StataCorp, College Station, TX).

Severity of Illness Adjustment

We used a severity of illness score to account for the variation in case-mix and illness severity that may confound the relationship between hospitals (or regions) and 30-day mortality; conceptually, this is what APACHE or SAPS scores seek to do²¹. We constructed the score using variables that make up the VA ICU severity score, a validated risk-adjustment measure that includes age, admission diagnosis category, 30 comorbid conditions, and 11 laboratory values drawn in the first 24 hours²⁰. To allow for a flexible nonlinear structure in the severity of illness score, we used a logistic multivariate adaptive regression spline (MARS), a nonparametric spline-and-knot-based form of regression that models the functional forms of covariates, as well as the nonlinearities and higher-level interactions thereof, for an outcome of 30-day mortality²². The MARS model’s predicted probability of 30-day mortality was subsequently used as the severity of illness score and separate models were estimated for each year to allow for changes over time; this avoids confusing changes in model calibration over time with changes in hospital performance. This severity of illness score incorporates chronic comorbidity, acute physiologic dysregulation (worst value in first 24 hours from severe sepsis onset), and age and gender; we refer to models that adjust for the score as “risk-adjusted”.

Reliability Adjustment

We adjusted the hospital and region rates of 30-day mortality to account for the number of severe sepsis hospitalizations cared for at each location. Reliability adjustment is a statistical technique that uses empirical Bayes prediction to provide better estimates of mortality for hospitals with few severe sepsis hospitalizations, where it is difficult to know whether extremely high or low mortality rates are due to chance alone or due to a true difference in the quality of care^23–25. Hospitals with few observations have their mortality rate “shrunk” toward the average, thereby removing the statistical noise that occurs from small sample sizes. The greater the number of hospitalizations, the less the hospital-level mortality rates are changed with reliability adjustment. This approach is used routinely for hospital quality reporting, such as for the outcomes reported on the Center for Medicare and Medicaid Services’ HospitalCompare website²⁶.

Unadjusted Analyses

For unadjusted analyses comparing 2008 and 2012 directly, we used a two-sided t-test. For all proportional statistics, we used a two-sample proportion test.

Quantifying Variation After Risk Adjustment

For each year, we quantified the variation in 30-day mortality across hospitals using the intra-class correlation coefficient (ICC) and median odds ratios (MOR) calculated from each year’s 3-level logistic regression model²⁷. We then directly compared the 2012 ICC and MOR estimates with those from 2008. ICCs are conventionally interpreted as the proportion of variance explained by a given level of aggregation (e.g. hospitals or regions) in linear models²⁸. However, the interpretation of ICC for dichotomous outcomes is limited by the imperfect distinction between individual and cluster (e.g. patient and hospital) level variance in logistic models²⁷. Thus, we measure ICCs to determine the statistical significance of hospital and region effects on sepsis-related mortality. However, we also present MORs, as they provide more interpretable information on the clinical importance of the impact of hospitals and regions on outcomes²⁷.

The MOR represents the increased odds of death that a patient with median baseline risk would have if moving to another hospital (and potentially region) with greater risk. That is, given pairs of randomly selected hospitals, the MOR is the median value of all odds ratios between the hospital with the higher 30-day mortality and the hospital with the lower 30-day mortality²⁹. A MOR of 1.0 implies that the odds of death is equivalent across hospitals; the larger the MOR, the more important the hospital and region-level effects are in driving differences in outcome. Unfortunately, there is no widely implemented approach to generating 95% confidence intervals for MORs at this time.

RESULTS

Patient and Hospital Characteristics

We identified 43,733 hospitalizations with severe sepsis at 114 VA hospitals in 2012 (Table 1). The patients were predominantly white (73%), male (97%), and had a mean age of 70 (SD 12) years. Unadjusted 30-day mortality across all patients was 15% in 2012, down from 18% in 2008 (p<0.001). (In-hospital case mortality was 10% vs. 15% in 2008 [p<0.001]; 90-day mortality was 24% vs. 29% in 2008 [p<0.001]; mortality among mechanically ventilated patients was 32% vs. 34% in 2008 [p=0.002]). Using a severity of illness model that was fit to 2008 and 2012 data, the severity of illness was very similar between years. The predicted risk of 30-day mortality was 16.0% in 2012 versus 16.6% in 2008 (p-value < 0.001).

Table 1.

Characteristics of Patients of Patients with Severe Sepsis

	2008 (N=33,095)	2012 (N=43,733)	p
Age, years ± SD	69.4 ± 12.31	69.7 ± 12.16	<0.001
Gender, N (%)			0.104
Male	32,133 (97%)	42,373 (97%)
Female	962 (3%)	1,360 (3%)
Race, N (%)			<.001
White/Caucasian	23,419 (71%)	32,023 (73%)
Black/African America	5,998 (18%)	8,180 (18%)
Unknown	3,208 (10%)	2,853 (7%)
Other	470 (1%)	677 (2%)
Total Hospital LOS, days ± SD	12.487 ± 14.16	10.896 ± 13.94	<.001
Admission Source, N (%)			<.001
VA Emergency Department	17,205 (52%)	21,176 (48%)
VA Outpatient Clinic	12,735 (38%)	18,972 (43%)
Other	3,155 (10%)	3,585 (9%)
Discharge Status, N (%)			<.001
Outpatient (home)	21,132 (64%)	29,492 (67%)
Nursing Facility	5,247 (16%)	7,296 (17%)
Death	4,841 (15%)	4,420 (10%)
Other/Unknown	996 (3%)	1,409 (3%)
Hospital Transfer	879 (2%)	1,116 (3%)
ICU, N (%)	14,600 (44%)	16,496 (38%)	<.001
Mechanically Ventilated, N (%)	6,761 (20%)	6,657 (15%)	<.001
ICU and Mechanically Ventilated, N (%)	6,539 (20%)	6,502 (15%)	<.001

Of the 114 hospitals included in our analysis, 56 (49%) were teaching hospitals and 59 (52%) provided level 1 (full) ICU services. The hospitals varied in the volume of severe sepsis patients that they cared for in one year, and the median number of severe sepsis patients increased over time (2008 median 266, 75% IQR [110–415]) vs. 2012 median 376, 75% IQR [157–561]).

Hospital and Region-level Effects on Adjusted 30-Day Mortality following Severe Sepsis

After severity-of-illness- and reliability-adjustment of the estimates, variation persisted in 30-day mortality across hospitals (Figure 1)—and to a lesser extent, across regions (Figure 2). The median hospital in the worst quintile had a risk-adjusted mortality of 16.7% (95% CI: 13.5%, 20.5%) in 2012, compared to the median hospital in the best quintile, which had an adjusted mortality of 12.8% (95% CI: 10.7%, 15.3%), p<0.001. The median region in the worst quintile had a risk-adjusted mortality of 16.2% (95% CI: 14.4%, 18.1%) in 2012 (holding constant hospital- and patient-level effects), compared to the median region in the best quintile, which had a risk-adjusted mortality of 13.9% (95% CI: 12.2%, 15.8%), p=0.029.

Figure 1. Variation Between Hospitals in Risk- and Reliability-Adjusted Mortality in 2012.

Error bars from the hospital-level random effects are shown at 1.4 times the standard error, indicating that two hospitals have significantly different performance if their error bars do not overlap. Hospitals are ranked by performance. The range of adjusted mortality among hospitals was 10.9%–13.3% for quintile 1, 13.4%–14.4% for quintile 2, 14.5%–15.0% for quintile 3, 15.0%–16.2% for quintile 4, and 16.2%–20.7% for quintile 5.

Figure 2. Variation Between Regions in Risk- and Reliability-Adjusted Mortality in 2012.

Error bars from the region-level random effects are shown at 1.4 times the standard error, indicating that two regions have significantly different performance if their error bars do not overlap. Regions are ranked by performance.

In the 2012 adjusted hierarchical model, the proportion of variance explained by region was 0.003 (95% CI: .001, .013) – statistically significant (p-value = 0.022) but of small absolute value. The proportion explained by both hospitals and regions was 0.014 (95% CI .009, .023) and statistically significant (p-value < 0.001). The MOR for 30-day mortality across different hospitals within the same region—a more interpretable measure of the hospital effect—was 1.20 in 2012. In other words, if we observe two patients with the same physiology from different VA hospitals in the same region, then we would expect the patient at the worse of the two hospitals to have 20% greater odds of 30-day mortality than the patient at the better hospital. The MOR for 30-day mortality across different hospitals and regions was 1.23 in 2012.

In sum, we find support for (H1) and (H2): the variation is statistically significant, and the magnitude of the differences between hospitals and regions are clinically meaningful.

Temporal Trends in Hospital and Region-Level Effects

To examine whether hospital- and region-level effects on severe sepsis mortality have changed over time (H3), we identified 33,095 hospitalizations involving the first severe sepsis diagnosis at VA hospitals in 2008, in addition to those from 2012 (N=43,733). Patient demographics were similar between 2008 and 2012 (Table 1). Unadjusted 30-day mortality decreased from 18.3% in 2008 to 14.7% in 2012 (p-value < 0.001), despite little change in severity of illness as reported above. Figure 3, Panel A demonstrates the rates of mortality improvement across hospitals.

Figure 3. Changes in Hospital (Upper Panel) and Regional (Lower Panel) Effects on Mortality Between 2008 and 2012.

All results are risk- and reliability-adjusted. Each line segment connects the same organization at the 2 time points. Overall regression lines are depicted in bold, with slope and statistical significance listed in the upper right corner.

The overall strength of association in adjusted 30-day mortality following severe sepsis hospitalization within hospitals and regions was consistent over time. The proportion of variance explained (ICC) by region alone was 0.002 (95% CI: 0.0003, 0.016) in 2008 versus 0.003 (95% CI: 0.001, 0.013) in 2012. The proportion of variance explained by both regions and hospitals was 0.017 (95% CI: 0.011, 0.025) in 2008 compared to 0.014 (95% CI 0.009, 0.023) in 2012. Likewise, the MOR across different hospitals in different regions was 1.25 in 2008, similar to 1.23 in 2012. In both years, the median difference between hospitals was about the same as a 2% absolute difference in baseline risk of death from severity of illness. That is, hypothesis (H3) is refuted.

DISCUSSION

In this large retrospective cohort of hospitalizations within the U.S. VA Healthcare System, we determined the extent to which hospitals and regional healthcare networks explain variation in 30-day mortality following severe sepsis hospitalization. After careful adjustment for case-mix and illness severity using a clinically rich and validated score, we determined that there is statistically significant variation in 30-day mortality due to the hospital or regional healthcare network where a patient receives care. Indeed, there were clinically meaningful differences across hospitals, with a 3.9% absolute mortality difference between median hospitals in the top and bottom quintile.

The finding of statistically and clinically significant variation in 30-day mortality after severe sepsis is consistent with recent studies by Walkey, et al. and Gaiski, et al. demonstrating a volume-outcome relationship in severe sepsis care within non-federal U.S. hospitals^30,31. However, our study did not test which hospital characteristics (e.g. sepsis case volume) may explain the observed variation across institutions. Prior studies of VA populations with other illnesses^32,33 and studies of severe sepsis among integrated healthcare networks demonstrate no volume outcome relationship³⁴. Thus, we hypothesize that factors other than sepsis case volume explain the observed variation, but this needs to be tested explicitly in future studies.

A second major finding of our study is that adjusted 30-day mortality after hospitalization with severe sepsis declined from 18.3% in 2008 to 14.7% in 2012. This mortality improvement occurred in the setting of similar illness severity—with measured risk of 30-day mortality declining only slightly from 16.6% in 2008 to 16.0% in 2012. Thus, the improvements in 30-day mortality are not readily explained by labeling changes and inclusion of less ill patients in more recent years. Rather, our findings imply improved care of severe sepsis patients over time. Furthermore, the rates of ICU use and mechanical ventilation declined by 6.4% and 5.2% between 2008 and 2012—again in the setting of similar illness severity. We interpret this as further evidence of better initial sepsis care may obviate the need for subsequent ICU transfer or mechanical ventilation, in concordance with other studies^11,35.

While 30-day mortality improved and overall variation in mortality across hospitals decreased from 2008 to 2012, the proportion of the variation attributable to hospitals and regions did not change over time. We hypothesize that one reason for the failure of hospital or regional effects to grow over time may be the absence of sustained sepsis-targeted efforts within this large hospital system. If individual VA hospitals or regions had deployed sepsis-targeted quality improvement efforts as successfully as has been done elsewhere¹¹, then those hospitals and regions would likely have disproportionate mortality reductions and thereby increase the proportion of variation explained by the hospital or region.

A secondary, and unexpected, finding of our study was the heterogeneity in individual hospital’s rates of improvement. The finding of overall improved 30-day mortality over time in the VA is consonant with such improvements within ANZICS and other data^3,36. However, we are unaware of previous reports demonstrating the unevenness of such improvements at the hospital level over time. Such heterogeneity emphasizes the essential role of meaningful numbers of sites and concurrent controls in the evaluation of any quality improvement effort. Had an efficacious intervention been evaluated at only one or two of the hospitals that for other reasons had worsening mortality, it would have been discarded. Similarly, consider the harm that could result from a useless or even harmful therapy evaluated at the otherwise improving hospitals, if that evaluation were done only as a pre-/post- test rather than as an appropriate difference-in-differences test with controls. Such a design without concurrent controls, even at multiple hospitals, would have a high risk of confusing the secular trend for an effect of the intervention.

There are some limitations to our study. To adjust for case-mix and severity of illness, we used the VA ICU severity score. While this risk adjustment model has been validated in VA ICU patients and performs similarly to APACHE IV in this population²⁰, it has not been formally validated in patients outside of the ICU--although over half of the cohort used for the published validation had predicted and actual mortality rates of <3%²⁰. It is possible that unmeasured case-mix factors may still influence sepsis-related mortality. Unlike hand-abstracted data like ANZICS, we cannot rule out that the sensitivity of our sepsis case definition may have changed over time. We do note, however, that our algorithm for detecting sepsis cases does not rely on physician coding of sepsis, but rather on the synchronous appearance of infection and acute organ dysfunction, making it more robust to such labeling changes. Further, we note that mortality in our cohort declines despite relatively stable severity-of-illness. Secondly, the statistical power to detect hospital and region-level effects is a complex interaction between the number of hospitals, regions, and patients³⁷. While large number of patients in our study increases the statistical power to detect clinically insignificant hospital and region-level effects, the relatively smaller numbers of hospitals and regions offset this potential concern. Furthermore, we use median odds ratios to assess the clinical significance of the hospital and region-level effects. Finally, while we have shown that hospitals and regions impact short-term sepsis-related mortality, we have not tested institutional or organizational characteristics that may explain this variation.

CONCLUSION

In this study, we found that mortality declined from 2008 to 2012 in patients with severe sepsis in a large integrated health system despite similar risk profiles in both years. Further, we found evidence of consistent hospital, and to a lesser extent, regional influences on outcomes of care.