Hospital-level variation in the development of persistent critical illness

Abstract

Purpose

Patients with persistent critical illness may account for up to half of all intensive care unit (ICU) bed-days. It is unknown if there is hospital variation in the development of persistent critical illness and if hospital performance affects the incidence of persistent critical illness.

Methods

Retrospective analysis of Veterans admitted to the Veterans Administration (VA) ICUs from 2015–2017. Hospital performance was defined by the risk-and reliability-adjusted 30-day mortality. Persistent critical illness was defined as an ICU length of stay of at least 11 days. We used 2-level multilevel logistic regression models to assess variation in risk- and reliability-adjusted probabilities in the development of persistent critical illness.

Results

In the analysis of 100 hospitals which encompassed 153,512 hospitalizations, 4.9% (N=7,640/153,512) developed persistent critical illness. Furthermore, there was variation in the development of persistent critical illness despite controlling for patient characteristics (intraclass correlation: 0.067, 95% CI:0.049–0.091). Hospitals with higher risk-and reliability-adjusted 30-day mortality had higher probabilities of developing persistent critical illness (predicted probability: 0.057, 95% CI:0.051–0.063, p <0.01) compared to those with lower risk-and-reliability adjusted 30-day mortality (predicted probability: 0.046, 95% CI:0.041–0.051, p <0.01). The median odds ratio was 1.40 (95% CI:1.33–1.49) implying that, for two patients with the same physiology on admission at two different VA hospitals, the patient admitted to the hospital with higher adjusted mortality, would have 40% greater odds of developing persistent critical illness.

Conclusion

Hospitals with higher risk-and reliability- adjusted 30-day mortality have a higher probability of developing persistent critical illness. Understanding the drivers of this variation may identify modifiable factors contributing to the development of persistent critical illness.

Patients with long continuous intensive care unit (ICU) stays have increased mortality and utilize a disproportionate number of ICU resources.^1–5 Up to half of ICU bed-days may be occupied by patients who are or will become persistently critically ill.^2,5 The mechanisms leading patients to develop persistent critical illness are unclear. Recent work has focused on physiology and on understanding the cascade of organ failures, which occurs prior to the population onset of persistent critical illness.^6–8

However, patients’ physiologies evolve within hospitals and the type and quality of care differs among hospitals.^9,10 It may be that patients who would previously die earlier during their ICU stay are now surviving their acute illness as a result of newly available superb care, only to remain in the ICU for prolonged periods of time resulting in persistent critical illness—a consequence of the ICU’s own success and “legacies of critical illness”.¹¹ Conversely, it may be that low performing (e.g. unexpectedly high mortality) hospitals also have a higher number of patients with prolonged ICU stays because the patients were initially less optimally managed or more complications occurred during their care, which while not lethal, contributed to a prolonged ICU stay. The direction and magnitude of any association between ICU quality and the development of persistent critical illness is unknown, but would guide future research on what modifiable structural and process of care practices should be explored.¹²

In this study, we sought to understand the variation in the development of persistent critical illness and the extent to which the variation in its development might be explained by hospital factors in a large health care system, the United States (US) Veterans Administration (VA) system. High-performing ICUs are defined in the VA as those with lower risk-adjusted 30-day hospital mortality rates.^13,14, Specifically, we aimed to test the hypothesis that, even after patient-level risk-and reliability-adjustment, there would be statistically significant and clinically meaningful variation in the development of persistent critical illness between hospitals. Moreover, we aimed to test the hypothesis that hospitals with higher risk-and reliability-adjusted 30-day mortality will have a higher probability of persistent critical illness as compared to hospitals with lower risk-and reliability-adjusted 30-day mortality.

Methods

Study Context

The US VA health system is one of the largest integrated health care delivery systems in the world with over 9 million beneficiaries and an electronic medical record which can be leveraged to capture daily data.^15,16

Study Population

Data on Veterans in the VA health system were identified from the Veterans Affairs patient database (VAPD) 2015–2017 and represented over 100 hospitals.¹⁷ As has been previously described in a detailed methodological report, the VAPD includes extensive data and the information is structured at the patient-hospital-day and includes comorbidities, daily vital signs and labs, drug administrations, severity of illness on admission, admission categorizations and is linked to mortality.¹⁷ Analyses from the VA were approved by the IRB of the VA Ann Arbor Health System.

We abstracted data from the VAPD of patients who were admitted to the ICU for a minimum of 11 consecutive days based on multiple studies, which have found the population onset of persistent critical illness to occur at this time. The onset of persistent critical illness in these studies was defined when the in-hospital mortality was no better predicted by the acute admission physiology as compared to the patient’s comorbidities.^2,5 Sensitivity analysis were performed at ICU day 6 and ICU day 16 in light of the heterogeneity of the development of persistent critical illness seen in different populations.^2,5,18

Patients were excluded if they were under the age of 18; documented to have a pre-existing neuromuscular disease (Supplemental Appendix A), which inherently would contribute to prolonged recoveries; were previously in a VA ICU in the preceding 12 months in order to more reliably distinguish early and later time periods of critical illness; or if they had a pre-existing tracheostomy because of the association with the need for prolonged mechanical ventilation.¹ Those with new tracheostomies were included if they remained in ICU. Hospitals were excluded if they had less than 25 ICU hospitalizations a year or had a less-equipped ICU (level 3 or 4 hospital complexity in VA nomenclature).

The VA ICU Severity Score

For internal risk-adjustment, the VA uses an illness severity measure (the VA ICU severity score), which is the predicted 30-day mortality based on several variables (age, admission diagnosis category, 29 comorbid conditions, and 11 laboratory values). We re-calculated the score using these same variables. Normal values were imputed when labs were not clinically measured as has been done in previous severity illness measurements which does not bias the results towards any specific hospital.^13,19 (Supplemental Appendix B) This severity score performs similarly to APACHE IV, with a c-statistic of 0.874 in published work.¹³

Statistical Analysis

We present patient and hospitalization characteristics as counts (percentages), means (standard deviations (SDs)), or medians (interquartile ranges [IQR]) as appropriate. Elixhauser comorbidities were weighted and tabulated as described by using the weighted van Walraven method. ²⁰ We used hospitalization as the unit of analysis, unless otherwise specified. We used two-sided significance testing and considered p <0.05 to be statistically significant.

General Approach

We first used 2-level multilevel logistic regression models to create a hospital performance measure—hospital risk- and reliability- adjusted 30-day mortality. We then used a 2-level multilevel logistic regression to assess variation in risk- and reliability-adjusted probabilities in the development of persistent critical illness.^21–24 We initially calculated the variation in the development of persistent critical illness across hospitals accounting for patient characteristics (demographics and risk factors). A subsequent model was built adjusting for hospital characteristics (complexity and teaching status). A final model was then built adjusting for the hospital risk-and reliability-adjusted 30-day mortality rates. All statistical code is already available at GitHub at https://github.com/CCMRcodes/Hospital-level-variation-in-PerCI. We conducted all analysis with Stata software 15.1 (StataCorp, College Station, TX) and SAS 9.4 (SAS Institute, Cary, NC)

Reliability Adjustments

We used reliability adjustment to avoid overestimating the probability of the development of persistent critical illness at hospitals with low case volume.²⁵ To do this, we used a multilevel model with a random intercept for hospital. The three main advantages of using a random effects model are 1) to reduce the number of parameters estimated, 2) to adjust for hospital level covariates, and 3) to benefit from the property of shrinkage. The shrinkage estimator approach places more weight on a hospital’s point estimate when it is measured reliably but “shrinks” it toward the population mean when there is more error in the measurement (e.g. lower-case volume).^21,23,26 We performed reliability adjustment by generating empirical Bayes estimates.

Hospital Performance Measure: Hospital Risk-and Reliability- Adjusted 30-day Mortality

To account for differences in case-mix, we created a hospital risk- and reliability- adjusted 30-day mortality variable. We used a multilevel logistic model in which the hospitalization was the first level and the hospital was the second level and the dependent variable was 30-day mortality. At the hospitalization level, we adjusted for patient severity of illness (which includes age, admission diagnosis category, 29 comorbid conditions, and 11 laboratory values on admission). The second level only included a hospital random effect. The random effect was then predicted using Bayesian techniques (we generated posterior estimates using predictions of mortality for each hospital). This was added back to the mean log (odds) of mortality and an inverse logit was performed to estimate the risk-and reliability-adjusted 30-day mortality.^21,23

A sensitivity analysis was performed utilizing a hospital risk- and reliability-adjusted ICU mortality variable since persistent critical illness may be more related to ICU factors and not general hospital factors. (Supplemental Appendix C).

Quantifying Variation after Risk-and Reliability-Adjustment for Persistent Critical Illness

We created four multilevel logistic models. (Supplemental Appendix D lists all of the variables in each model.) For each model, we quantified the variation in the probability of developing persistent critical illness across hospitals using the intraclass correlation coefficient (ICC). We then directly compared the ICC for all three models (Model 1 was the empty model with no covariates, Model 2 included patient characteristics (age, gender, Elixhauser comorbidities, severity of illness on admission, diagnosis code on admission); Model 3 included the patient characteristics and hospital characteristics (hospital size, teaching status, complexity level and case-mix); Model 4 included the patient and hospital characteristics and hospital risk- and reliability-adjusted 30-day mortality) to understand how much variation was explained by the hospitals risk-and reliability-adjusted 30-day mortality. Predicted probabilities were calculated using post-estimation commands and were calculated from the final model (Model 4). The median odds ratio (MOR) was calculated from the final model as per the method of Merlo.²⁴ We present MORs, as they provide more interpretable information on the odds ratio scale of the impact of hospitals on the development of persistent critical illness by randomly comparing pairs of hospitals at highest risk to those at lowest risk.²⁴ The MOR quantifies the difference in the rates of persistent critical illness between hospitals. A MOR of 1.0 implies that the odds of persistent critical illness are equivalent across hospitals; the larger the MOR, the more important the hospital-level effects are in driving differences in outcome.²⁷

Results

We identified 4.9% hospitalizations (7,640 of the 153,512 ICU hospitalizations) in the VA from 2015 – 2017 that developed persistent critical illness with a range of 1 to 251 persistent critical illness ICU hospitalizations per hospital. (Figure 1 and Supplemental Appendix Figure 1) The patients were predominately white men with a median age of 67 (IQR: 60, 73). (Table 1) The 100 hospitals were predominately level 1 (highest) complexity and teaching hospitals. (Table 1)

Fig. 1. Flow chart for hospitalizations.

Table 1:

Demographics of the hospitalization from the VA from 2015-2017

	All hospitalizations	Non-persistent critically ill hospitalizations	Persistent critically ill hospitalizations
	N=153,512	N=145,872	N=7,640
Age (years) median (IQR)	67 (60, 73)	67 (60, 72)	68 (62, 73)
Race
White: N (%)	109,587 (71.4)	104,154 (71.4)	5,433 (71.1)
African American: N(%)	32,155 (21.0)	30,499 (20.9)	1,656 (21.7)
Other: N (%)	11,770 (7.6)	11,219 (7.7)	551 (7.2)
Male: N (%)	146,548 (95.5)	139,153 (95.4)	7,395 (96.8)
Elixhauser a: median (IQR)	6 (1, 14)	6 (0, 13)	12 (5, 20)
VA ICU severity score: median (IQR)	2.5 (1.0, 6.6)	2.4 (1.0, 6.2)	6.6 (2.7, 16.1)
ICU length of stay (days): median (IQR)	3 (2, 5)	3 (2, 4)	15 (12, 20)
Hospital length of stay: (days) median (IQR)	6 (4, 10)	6 (3, 9)	22 (16, 32)
In-hospital mortality: N (%)	8,493 (5.5)	6,922 (4.7)	1,571 (20.6)
Discharge Location
Home: N (%)	140,111 (91.2)	134,371 (92.1)	5,740 (75.1)
Facility: N (%)	5,132 (3.4)	4,794 (3.3)	338 (4.5)
Other/Unknown: N (%)	8,269 (5.4)	6,707 (4.6)	1,562 (20.4)
HOSPITAL CHARACTERISTICS
Hospital complexity level b
Level 1: N (%)	143,384 (93.4)	135,976 (93.2)	7,408 (97.0)
Level 2: N (%)	10,128 (6.6)	9,896 (6.8)	232 (3.0)
Bed size: median (IQR)	159 (109, 244)	159 (109, 144)	167 (119, 248)
ICU-case mix
Medical: N (%)	87860 (57.2)	83,262 (57.1)	4,598 (63.9)
Surgical: N (%)	55705 (36.2)	52,947 (36.3)	2,758 (36.1)
Other: N (%)	9947 (6.5)	9,663 (6.6)	284 (3.7)
Teaching facility: N (%)	110,007 (71.7)	104,062 (71.4)	5,945 (77.8)
Region
Midwest: N (%)	33,578 (21.9)	32,091 (22.0)	1,487 (19.5)
South: N (%)	71,404 (46.5)	67,701 (46.4)	3,703 (48.5)
West: N (%)	29,334 (19.1)	27,916 (19.1)	1,418 (18.5)
Northeast: N (%)	19,196 (12.5)	18,164 (12.5)	1,032 (13.5)

^aWeighted score of co-morbidities;

^bWithin the VA, higher level ICUs (1 and 2) are associated with tertiary academic centers.

ICU: intensive care unit

IQR: interquartile range

VA: Veterans Administration

Hospital Risk-and Reliability-Adjustment

After risk-and reliability-adjustment for patient characteristics, there was significant variation between hospitals in the probability of the development of persistent critical illness (ICC: 0.067, 95% CI: 0.049 – 0.091). (Table 2) This implies 6.7% of the variance in the development of persistent critical illness is attributed to hospitals rather than differences in the patients who come to them.

Table 2.

Variance components of the multilevel logistic regression for the development of persistent critical illness.

Model 1: empty model (intercept only)
Variance at the hospital level (standard error)	0.213 (0.037)
ICC of the hospital level (standard error)	0.061 (0.010)
Model 2: patient characteristics (age, gender, comorbidities, severity of illness, admission diagnosis)
Variance at the hospital level (standard error)	0.235 (0.040)
ICC of the hospital level (standard error)	0.067 (0.011)
Percent change in variance compared to Model 1	−9%
Model 3: patient characteristics, hospital characteristics (complex level, teaching status, bed size, case-mix)
Variance at the hospital level (standard error)	0.130 (0.023)
ICC of the hospital level (standard error)	0.038 (0.006)
Percent change in variance compared to Model 1	+31%
Model 4: patient and hospital characteristics, hospital risk-and reliability-adjusted mortality
Variance at the hospital level (standard error).	0.124 (0.022)
ICC of the hospital level (standard error)	0.036 (0.006)
Percent change in variance compared to Model 1	+48%

ICC: Intraclass correlation

Hospital characteristics (hospital size, teaching status, complexity level and case-mix) explained nearly half the residual variation between hospitals in the probability of the development of persistent critical illness (ICC: 0.038, 95% CI: 0.027 – 0.053). In other words by adjusting for hospital characteristics, the variance attributed to hospitals was reduced to 3.8%.

Risk-And Reliability-Adjusted 30-day Mortality at the Hospital Level

Hospitals’ median risk-adjusted 30-day mortality was 0.039 (IQR: 0.035, 0.045). After accounting for the hospitals’ risk- and reliability- adjusted 30-day mortality, the residual variation between hospitals in the probability of the development of persistent critical illness was minimally reduced (ICC: 0.036 95% CI: 0.026 – 0.051). (Figure 2) Hospitals in the worst 20th percentile of risk-and reliability-adjusted 30-day hospital mortality were associated with a higher probability of persistent critical illness (predicted probability: 0.057, 95% CI: 0.051 – 0.063, p < 0.01) as compared to those in the best 20^th percentile (predicted probability: 0.046, 95% CI: 0.041 – 0.051, p < 0.01). The MOR for the development of persistent critical illness was 1.40 (95% CI: 1.33 – 1.49). In other words, for two patients with the same physiology on admission at two randomly chosen VA hospitals, the patient at the lower performing hospital would have, on average, 40% greater odds of developing persistent critical illness than a patient at the higher performing hospital.

Fig. 2. Variation in the development of persistent critical illness.

Hospitals are ranked by hospital risk-and-reliability adjusted 30-day mortality. Bars represent 95% confidence intervals.

Sensitivity Analysis

Hospitals continued to have significant variation in the probability of the development of persistent critical illness even after adjusting for patient characteristics on admission at alternate potential length-of-stay cut-off points: ICU day 6 (ICC: 0.051, 95% CI: 0.038 – 0.067) and at ICU day 16 (ICC: 0.095, 95% CI: 0.067 – 0.132). (Supplemental Appendix C table C1 and C2) Hospital characteristics (hospital size, teaching status, complexity level and case-mix), explained nearly half the variation between hospitals in the probability of the development of persistent critical illness whether defined at cut-offs of ICU day 6 (ICC: 0.038, 95% CI: 0.029 – 0.051) or at ICU day 16 (ICC: 0.052, 95% CI: 0.036 – 0.074).

After accounting for the risk- and reliability- adjusted 30-day mortality, the residual variation between hospitals in the development of persistent critical illness was further reduced when persistent critical illness was defined at the alternate cut-offs of ICU day 6 (ICC: 0.038, 95% CI: 0.028 – 0.050) and ICU day 16 (ICC: 0.046, 95% CI: 0.032 – 0.066). When persistent critical illness was defined at alternative cut-offs, ICU day 6 or 16, the MOR was 1.41 (95% CI: 1.34 – 1.49) and 1.46 (95% CI: 1.37 – 1.59) respectively.

When utilizing the hospitals’ risk-and reliability- adjusted ICU mortality as the quality metric, the residual variation between hospitals in the development of persistent critical illness was reduced (ICC: 0.026 95% CI: 0.018 – 0.037) and the MOR for the development of persistent critical illness was 1.33 (95% CI: 1.27 – 1.40).

Discussion

Key Findings

In this large retrospective national cohort of ICU hospitalizations in the VA, we determined the extent to which hospital (rather than patient) characteristics might explain variation in the development of persistent critical illness. There was clinically meaningful variation as evidenced in the MORs. Using highly granular patient data, after risk- and reliability-adjustment for 30-day mortality, we found that hospitals with higher risk-and reliability-adjusted 30-day mortality had a higher probability of having patients under their care develop persistent critical illness.

Relationship to Previous Studies

Previous work in severe sepsis has found significant variation between hospitals in 30-day mortality.^23,28,29 Additionally, prior studies have suggested that persistent critical illness may be driven by the development of late cardiovascular failure, specifically late sepsis.⁷ Similarly, variation in use of long-term acute care hospitals and tracheostomies have been noted.^30–32 However, no previous studies have evaluated the extent to which hospital factors might contribute to the development of persistent critical illness, prolonged mechanical ventilation, or chronic critical illness.

Study Implications

The development of persistent critical illness is not reliably associated with currently identified patient factors.³³ Neither patient age nor severity of illness on admission are consistently associated with the development of persistent critical illness, ^6,7,34 suggesting other factors may be contributing to the development of persistent critical illness. A patient’s acute pathophysiology is not only driven by the acute illness and premorbid conditions (e.g. frailty) but also by the care the patient receives.^35,36

This may imply that the development of persistent critical illness may share mechanisms with other forms of “failure to rescue”, including death.^37,38 The mechanisms driving these findings may relate to implementation of bundles of care, greater in ICU complication rates, utilization of palliative care, or a culture which does not foster multidisciplinary communication. Each of these factors will need further exploration in future studies to understand the different pathways which result in the development of persistent critical illness.

Strengths and Limitations

Our study has several strengths. We examined a national health system with detailed daily physiologic data collected over a two-year period encompassing 153,512 ICU hospitalizations and 100 hospitals. These data allowed us to risk- and reliability- adjust for 30-day mortality and patient factors. Second, we have shown the extent to which hospitals explain the variation in the development of persistent critical illness.

There are also limitations to our study. First, we used ICUs within the VA healthcare system which may not be generalizable to other hospitals in the U.S. but may be generalizable to other national health care systems. For example, the prevalence of persistent critical illness in the VA is similar to the prevalence reported in Australia and New Zealand.^2,5 However, the mortality in the VA ICU is significantly lower and many patients are discharged home unlike the patients in Australia and New Zealand.² Second, we did not account for transfer to long term acute care hospitals which could potentially lead to lower rates of persistent critical illness. However, the VA does not generally transfer patients to such long-term acute care hospitals. In the VA, there were 828 patients who remained in the ICU for at least 30 days. Additionally, we cannot account for outside hospital ICU transfers to the VA ICUs. Third, while we have shown that hospitals impact the probability of developing persistent critical illness, we have not tested the causal drivers of hospital risk- and reliability-adjusted 30-day mortality. However, the investigation of such factors (e.g. surgical complications, referral to palliative care, nosocomial infections) can be operationalized and we aim to make it the subject to future studies. Fourth, we are not able to account for patients who have limitations of care (do not resuscitate or do not intubate orders) or who transition to comfort care while in the ICU, resulting in these hospitals having higher in-hospital mortality rates. Fifth, we were unable to account for ward capacity issues resulting in prolonged ICU stays and higher rates of persistent critical illness. Additionally, we did not account for other definitions of prolonged ICU stays such as the need for prolonged mechanical ventilation. However, we did use varying time points of an ICU length of stay to define persistent critical illness and found similar results. Lastly, while we focused on the hospital-level of variation in the development of persistent critical illness, clinician variation may also be a mechanism driving its development and will be the subject of future studies.

Conclusion

There is variation between hospitals in the development of persistent critical illness, which is, at least in part, explained by the quality performance of a given hospital, such that higher mortality hospitals have higher rates of persistent critical illness. Given these observations, research aimed at understanding what specific organizational characteristics reduce the likelihood of developing persistent critical illness is now of public health importance.