Abstract
Purpose:
To assess trends in life support interventions and performance of the automated Acute Physiology and Chronic Health Evaluation (APACHE) IV model at mortality prediction compared with Oxford Acute Severity of Illness Score (OASIS) in a contemporary cardiac intensive care unit (CICU).
Methods and Materials:
Retrospective analysis of adults (age ≥18 years) admitted to CICU from January 1, 2007, through December 31, 2015. Temporal trends were assessed with linear regression. Discrimination of each risk score for hospital mortality was assessed with use of area under the receiver operating characteristic curve (AUROC) values. Calibration was assessed with Hosmer-Lemeshow goodness-of-fit test.
Results:
The study analyzed 10,004 patients. CICU and hospital mortality rates were 5.7% and 9.1%. APACHE IV predicted death had an AUROC of 0.82 (0.81–0.84) for hospital death, compared with 0.79 for OASIS (P<.05). Calibration was better for OASIS than APACHE IV. Increases were observed in CICU and hospital lengths of stay (both P<.001), APACHE IV predicted mortality (P=.007), Charlson Comorbidity Index (P<.001), noninvasive ventilation use (P<.001), and noninvasive ventilation days (P=.02).
Conclusions:
Contemporary CICU patients are increasingly ill, observed in upward trends in comorbid conditions and life support interventions. APACHE IV predicted death and OASIS showed good discrimination in predicting death in this population. APACHE IV and OASIS may be useful for benchmarking and quality improvement initiatives in the CICU, the former having better discrimination.
Keywords: Comorbidity, critical care, mortality, quality improvement, renal replacement therapy, ventilation
Introduction
Historically, the function of the coronary care unit was to quickly detect and treat ventricular fibrillation in the clinical setting of acute myocardial infarction. This care model improved survival nearly 5 decades ago [1]. However, over time the patient population in the coronary care unit has changed to include fewer patients with acute coronary syndrome (ACS) and more patients with complex multisystem diseases and comorbidities [2–5]. In response to this change, some centers have relabeled these units the cardiac intensive care unit (CICU). Efforts to optimize patient care would be improved with access to a valid outcome prediction model.
Although outcome prediction models are readily available for the medical intensive care unit (MICU), only the Oxford Acute Severity of Illness Score (OASIS) has been validated for the CICU [6, 7]. However, with the increasing frequency of CICU admission for patients with noncardiac comorbid conditions, there has been interest in assessment of more traditional intensive care unit (ICU) outcome prediction models for the CICU setting. One candidate is the Acute Physiology and Chronic Health Evaluation (APACHE) IV model, which uses data obtained within the first 24 hours of ICU admission to predict in hospital death [8]. Similar to prior iterations of the APACHE score, APACHE IV is the sum of 3 components: an acute physiology score, an age score, and a numerical score for comorbidities. Yet unlike prior versions, APACHE IV incorporates additional data such as disease category, location before ICU admission, use of mechanical ventilation, use of thrombolytic therapy for acute myocardial infarction, the ratio of PaO2 to fraction of inspired oxygen, and Glasgow Coma Scale.
A potential limitation to the use of APACHE IV in general is the complexity of its calculation from 142 variables. Fortunately, programming in the electronic health record at some centers can be used to calculate the score automatically [9]. However, because the APACHE IV was derived from a population of which only 16% were cardiac patients, how well it will perform in the CICU setting is uncertain.
The purpose of this study was to test the hypothesis that the automated version of the APACHE IV model can predict in-hospital death for a contemporary CICU population and to assess CICU trends in life support interventions.
Methods and Materials
This retrospective analysis used an institutional database at Mayo Clinic in Rochester, Minnesota. The Mayo Clinic Institutional Review Board approved the study.
All patients admitted to the CICU were medical patients; none had undergone cardiovascular surgery. Adult patients (age ≥18 years) admitted to the CICU were identified by searching the ICU database covering January 1, 2007, through December 31, 2015. Patients who were excluded from the study were those younger than 18 years, those without Minnesota Research Authorization, and those admitted before or remaining admitted after the defined data collection period.
Variables collected included the following: demographic characteristics, ICU and hospital length of stay (LOS), ICU and hospital discharge status, APACHE IV predicted mortality, APACHE III score, Charlson Comorbidity Index (CCI), last follow-up date, invasive and noninvasive ventilation use, and renal replacement therapy (RRT). The APACHE III score and APACHE IV predicted mortality were generated automatically from data in the electronic health record system with use of a previously validated algorithm [9]. The OASIS was calculated retrospectively using the worst values of 10 variables during the first 24 hours; missing data for OASIS and APACHE scores were imputed as normal [7, 9].
Statistical analysis was performed using RStudio version 3.2.3 (RStudio Inc). Differences in demographic characteristics and the defined variables collected between hospital survivors and nonsurvivors were determined using Kruskal-Wallis tests for continuous variables and Fisher exact tests for categorical variables. Differences between the years were determined with use of Kruskal-Wallis tests for continuous variables and Cochran Armitage test for categorical variables. General linear models were used with the year as an ordinal variable. The discrimination of the APACHE III, APACHE IV, and OASIS models for hospital death was assessed with an area under the receiver operating curve (AUROC) analysis, with differences between scores assessed with the DeLong test. To give a sense of potential clinical performance, we observed the optimal model threshold for APACHE IV predicted mortality, and sensitivity and specificity were calculated. The optimal threshold was considered the point on the receiver operating curve that was closest to the upper left-hand corner (ie, closest to a sensitivity and a specificity of unity). Logistic regression of hospital death was performed for each model to assess calibration using the Hosmer-Lemeshow χ2 goodness-of-fit test. The mean APACHE IV predicted mortality and the observed hospital mortality were used to characterize the observed-to-expected mortality ratios for each decile of APACHE-IV predicted mortality rate.
Results
During the defined study period, there were 12,911 admissions to the CICU. The final statistical evaluation included 10,004 patients. The primary admission diagnoses for all CICU patients included ACS (31%), arrhythmia (20%), heart failure (18%), respiratory failure (16%), sepsis (7%), renal failure (5%), and cardiac arrest (2%). The Table 1 shows the overall sample characteristics and the differences between hospital survivors and nonsurvivors from the CICU cohort. In total, 570 patients died in the CICU, for an overall CICU mortality of 5.7%. In addition, 338 CICU survivors (3.6%) died in the hospital, for an overall hospital mortality score of 9.1%. Patients who died during hospitalization had higher APACHE IV predicted mortality, APACHE III scores, and OASIS, and more frequent use of noninvasive ventilation (22.4% vs 14.1%), invasive ventilation (53.1% vs 12.4%), and RRT (9.1% vs 0.9%) (all P<.001). The median length to follow-up of the 9,096 living patients was 1.6 years from hospital discharge. We observed that 1,440 hospital survivors (35.1%) died during that follow-up period.
Table.1.
Patient Demographic Characteristics Stratified by In-Hospital Deaths
| Patient Value |
||||
|---|---|---|---|---|
| Characteristic | Overall (N=10,004) | Alive (n=9,096) | Dead (n=908) | P Value |
| Age, median (IQR), y | 69.0 (57.8–78.9) | 68.5 (57.1–78.5) | 73.8 (63.6–83.1) | <.001 |
| Male sex, No. (%) | 6,258 (62.6) | 5,716 (62.8) | 542 (59.7) | .07 |
| BMI, median (IQR) | 28.4 (24.9–33.1) | 28.4 (25.1–33) | 28.5 (24.3–33.9) | .95 |
| CICU LOS, median (IQR), d | 1.7 (0.9–2.9) | 1.7 (1.0–2.9) | 1.7 (0.5–3.9) | .22 |
| Hospital LOS, median (IQR), d | 4.6 (2.7–8.9) | 4.7 (2.8–8.9) | 3.7 (1.3–9.2) | <.001 |
| APACHE III score, median (IQR) | 58 (44–73) | 56 (43–70) | 88.5 (69–112) | <.001 |
| APACHE IV predicted mortality score, median (IQR) | 0.09 (0.04–0.21) | 0.08 (0.04–0.18) | 0.41 (0.18–0.69) | <.001 |
| OASIS, median (IQR) | 24 (18–31) | 23 (17–29) | 37 (28–47) | <.001 |
| Charlson Comorbidity Index score, median (IQR) | 5 (3–8) | 5 (3–8) | 5 (3–7) | .08 |
| Invasive ventilation use, No. (%) | 1,607 (16.1) | 1,125 (12.4) | 482 (53.1) | <.001 |
| Invasive ventilation period, median (IQR), d | 1.2 (0.5–2.7) | 1.2 (0.5–2.53) | 1.07 (0.28–2.9) | .01 |
| Noninvasive ventilation use, No. (%) | 1,489 (15) | 1,286 (14.1) | 203 (22.4) | <.001 |
| Noninvasive vent period, median (IQR), d | 0.6 (0.2–1.6) | 0.6 (0.2–1.5) | 0.9 (0.3–2.2) | .001 |
| RRT, No. (%) | 167 (2.0) | 84 (0.9) | 83 (9.1) | <.001 |
Abbreviations: APACHE, Acute Physiology and Chronic Health Evaluation; BMI, body mass index; CICU, cardiac intensive care unit; IQR, interquartile range; LOS, length of stay; OASIS, Oxford Acute Severity of Illness Score; RRT, renal replacement therapy.
Figure 1A shows the AUROC for the APACHE IV predicted mortality as a predictor of hospital death. The automated APACHE IV predicted mortality at 24 hours had a median AUROC of 0.82 (95% CI, 0.81–0.84) for hospital death, consistent with good discrimination of hospital death in this population. The median AUROC for the APACHE III score was 0.81 (95% CI, 0.80–0.83), which was significantly less than for APACHE IV predicted mortality (P=.001). The AUROC for the OASIS was 0.79 (95% CI, 0.78–0.81), which was significantly (P<.01) less than either the APACHE III score or APACHE IV predicted mortality. Both CICU and hospital mortality rates increased overall with each increasing decile of mean APACHE IV predicted death (described as 10% increments of APACHE IV predicted mortality) with the exception of the first decile, in which the predicted death was paradoxically lower than the observed rates (Figure 1B).
Figure 1.
AUROC for APACHE IV Predicted Mortality and CICU Death, Hospital Death, and Mean APACHE IV Predicted Mortality by Deciles. A, The AUROC for the automated APACHE IV predicted mortality. B, The trends of CICU death, hospital death, and median APACHE IV predicted mortality divided into equal number of groups per decile. APACHE indicates Acute Physiology and Chronic Health Evaluation; AUROC, area under the receiver operating curve; CICU, cardiac intensive care unit.
The optimal model performance threshold for the APACHE IV predicted mortality achieved both a sensitivity and a specificity of 0.75, while the optimal cutoff of the OASIS had a sensitivity and specificity of 0.73. Calibration for prediction of hospital death, as determined with the Hosmer-Lemeshow test, was good for OASIS (P=.45) but was suboptimal for both the APACHE III score (P=.01) and the APACHE IV predicted mortality (P<.001).
Over the 9-year study, there were no significant temporal trends in age, sex, body mass index, ICU or hospital death, invasive ventilation, or use of RRT (all P>.05). Trends for CICU and hospital death and for noninvasive and invasive ventilation use are shown in Figure 2. A significant increase occurred in the CICU and hospital LOS (both P<.001), APACHE IV predicted mortality (P=.007) (Figure 3), CCI (P<.001), noninvasive ventilation use (P<.001), and noninvasive ventilation days (P=.02) over the study period. Prevalence of the most common individual comorbid conditions (ie, prior myocardial infarction, prior congestive heart failure, prior chronic kidney disease, prior lung disease, prior diabetes mellitus, and prior cancer) also significantly increased over time (P<.001) (Figure 4).
Figure 2.
Trends in Outcome Measures Over the Study Period. A significant increase occurred in use of invasive ventilator and noninvasive ventilator (P=.006 and P<.001, respectively). No significant change was observed in the cardiac intensive care unit or hospital mortality rate (P=.67 and P=.22, respectively). CICU indicates cardiac intensive care unit.
Figure 3.
Median Acute Physiology and Chronic Health Evaluation IV Predicted Mortality Trend Over the Study Period. The predicted mortality rate significantly increased over the study period. APACHE indicates Acute Physiology and Chronic Health Evaluation.
Figure 4.
Trends in Comorbid Conditions Over the Study Period. The occurrence of all comorbid conditions significantly increased over the study period (P<.001). CHF indicates congestive heart failure; CICU, cardiac intensive care unit; CKD, chronic kidney disease; DM, diabetes mellitus; MI, myocardial infarction.
Discussion
To our knowledge, the present study is the first to explore the application of an automatic APACHE IV predicted mortality system for risk prediction in a contemporary CICU population. The APACHE IV risk calculator is useful for identifying patients at increased risk of hospital death in our CICU population, as has been previously demonstrated for MICU populations [8]. The APACHE IV predicted mortality model had the highest AUROC value for hospital death, followed by APACHE III, and then OASIS, although all these scores showed lower discrimination in this population than in their original derivation and validation cohorts [7, 8, 10].
The poor calibration of the APACHE IV predicted mortality is likely due to the small number of cardiac patients in the derivation cohort or to the fact, or both, that the population used to derive the APACHE IV model was hospitalized during 2002 to 2003, leading to further loss of calibration over time. Despite lower discrimination for hospital death than either the APACHE III or APACHE IV predicted mortality, OASIS had superior calibration in this cohort. The Hosmer-Lemeshow χ2 goodness-of-fit test is sensitive to large sample sizes [11]. This indicates that it is important to consider the observed and predicted values within deciles and the model’s discrimination when the population size is larger, such as ours. The observed mortality rate increased overall with increasing APACHE IV predicted mortality deciles, with the exception of the lowest decile, which underestimated death. This underestimation of death is likely secondary to the small numbers in this decile group and the missing variables given zero weight for the calculation of the APACHE IV predicted mortality scores. In the other groups, the poor calibration of the APACHE IV predicted mortality model led to overestimation of hospital death. This could be secondary to the CICU patients being a lower-risk population or the overestimation of risk in CICU patients in particular.
The APACHE IV predicted mortality is currently used in many MICUs for prognostication, assessment of outcome performance, and benchmarking to identify areas for improvement and serves as the framework for quality improvement initiatives. Our findings suggest that it can be used for these metrics in our contemporary CICU population. However, our finding that the observed to predicted mortality ratio varied as a function of APACHE IV predicted mortality decile implies that this mortality ratio for a given CICU may depend substantially on case mix and baseline risk of the population. This outcome is potentially due to the lower mortality risk in our cohort than in the population from which APACHE IV was derived, suggestive that recalibration of the APACHE IV predicted mortality may be warranted for CICU patients.
In the present study, the APACHE IV predicted mortality was generated from patient data automatically extracted from the electronic health record with use of an automated system that has been created and validated in the Mayo Clinic population to perform as well as the clinical gold standard of physician chart review [9]. The calculation system was based on the proprietary equations made publically available by Cerner Corp, which demonstrated excellent discrimination (AUROC median, 0.87 [95% CI, 0.83–0.92]) of the automated scoring system using a retrospective validation cohort of MICU patients [9]. A limitation of this method is the extraction of the admission diagnosis through a Boolean logic text search to identify the first diagnosis listed on the ICU admission note. This novel approach—to generate the score without the onerous task of manually collecting the 142 variables required for the calculation—improved the feasibility of using this scoring system in our study.
Less than one-third of our CICU population is made up of patients with primary ACS, similar to other recent CICU populations [6]. Although not specifically examining the predictive accuracy of the OASIS, Holland and Moss [6] in their recent study used the OASIS to correct for illness severity in 1,042 CICU admissions and found that the OASIS was an independent predictor of mortality. Our findings validate the use of OASIS in CICU populations, but we suggest use of updated APACHE scores over OASIS, when available, because of superior discrimination.
Our trends analysis expands on prior studies that examined trends in CICU populations. In 2010, Katz et al [4] reviewed the characteristics of their CICU admissions over an 18-year period and showed an increase in noncardiovascular comorbid conditions, such as sepsis, renal failure, acute respiratory failure, and pneumonia, through use of a hospital administrative database. In addition, they found a significant increase in the CCI on admission to the CICU over time. They concluded that this result reflects increasing severity of comorbid disease. We found that the CCI, individual comorbidities, use of mechanical ventilation, and the APACHE IV predicted mortality all increased significantly over time, which supports the notion that the current CICU patients are more medically complex and critically ill.
A recent analysis of the burden of noncardiovascular illness in a tertiary care academic medical center showed that ACS was the primary diagnosis in only 25% of the cases [6], which is consistent with our finding of 31%. Our CICU population had a primary admission diagnosis of sepsis in 7% and renal failure in 5% of patients. Of note, despite increasing trends suggestive that current CICU patients are more critically ill with increasing numbers of comorbidities, we did not find an increase in CICU or hospital death over the study period.
Limitations
A specific advantage of our study was its large sample size, but the single-center nature of this study is an important limitation in the generalization of our results. The relatively rural location of our medical center and our referral base likely affect the generalizability of our findings to other populations. Variations in characteristics of patient populations and medical practices in other medical centers could affect the validity of the scoring system at those centers. Our CICU practice group likely provided more consistent care over time, thus limiting clinical variability to some extent. In addition, we had relatively limited information about admission and discharge diagnoses, which limited our ability to determine whether subpopulations existed within our CICU patients, where the APACHE IV predicted mortality had greater or lesser predictive value. This study is novel in that it showed the feasibility of the use of an automated APACHE IV predicted mortality score. OASIS, while easier to calculate than the APACHE scores, had significantly lower discrimination for hospital death. Importantly, missing data to calculate the OASIS or APACHE scores were imputed as normal, a parsimonious approach that may explain our lower AUROC values than some prior studies [9].
Conclusions
This study shows that the automated APACHE IV predicted mortality has very good discrimination for prediction of hospital death in a contemporary CICU population and showed that patients treated in the modern CICU have increasing numbers of comorbidities and life support interventions. Our data show that the APACHE IV predicted mortality discrimination in predicting hospital deaths in this tertiary care population is superior to simpler scores such as OASIS and APACHE III score, but the calibration is suboptimal. Although recalibration and external validation in other CICUs is needed before general implementation, this tool may be useful for benchmarking and quality improvement initiatives, as well as research advances in the modern CICU.
Highlights.
Modern cardiac intensive care units include patients with more complex disease.
The APACHE IV predicted mortality has not been tested in this population.
The APACHE IV predicted mortality predicts death of CICU patients with good discrimination.
Use of life support interventions is greater in this population.
The APACHE IV predicted mortality may be useful for benchmarking and quality improvement.
Acknowledgments
Funding
This publication was partially supported by Grant Number UL1 TR000135 from the National Center for Advancing Translational Sciences (NCATS). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the National Institutes of Health (NIH).
Role of the Sponsor
The NIH grant was used for initial statistical support and protocol development.
Abbreviations
- ACS
acute coronary syndrome
- APACHE
Acute Physiology and Chronic Health Evaluation
- AUROC
area under the receiver operating curve
- CCI
Charlson Comorbidity Index
- CICU
cardiac intensive care unit
- ICU
intensive care unit
- LOS
length of stay
- MICU
medical intensive care unit
- OASIS
Oxford Acute Severity of Illness Score
- RRT
renal replacement therapy
Footnotes
Conflict of interest: The authors report no conflicts of interest.
Contributor Information
Courtney E. Bennett, Department of Cardiovascular Medicine, Division of Pulmonary and Critical Care Medicine, Mayo Clinic, Rochester, Minnesota. bennett.courtney@mayo.edu.
Scott Wright, Department of Cardiovascular Medicine, Mayo Clinic, Rochester, Minnesota. wright.scott@mayo.edu.
Jacob Jentzer, Department of Cardiovascular Medicine, Division of Pulmonary and Critical Care Medicine, Mayo Clinic, Rochester, Minnesota. jentzer.jacob@mayo.edu.
Ognjen Gajic, Division of Pulmonary and Critical Care Medicine, Division of Pulmonary and Critical Care Medicine, Mayo Clinic, Rochester, Minnesota. gajic.ognjen@mayo.edu.
Dennis H. Murphree, Division of Biomedical Statistics and Informatics, Mayo Clinic, Rochester, Minnesota. murphree.dennis@mayo.edu.
Joseph G. Murphy, Department of Cardiovascular Medicine, Mayo Clinic, Rochester, Minnesota. murphy.joseph@mayo.edu.
Sunil V. Mankad, Department of Cardiovascular Medicine, Mayo Clinic, Rochester, Minnesota. mankad.sunil@mayo.edu.
Brandon M. Wiley, Department of Cardiovascular Medicine, Mayo Clinic, Rochester, Minnesota. wiley.brandon@mayo.edu.
Malcolm R. Bell, Department of Cardiovascular Medicine, Mayo Clinic, Rochester, Minnesota. bell.malcolm@mayo.edu.
Gregory W. Barsness, Department of Cardiovascular Medicine, Mayo Clinic, Rochester, Minnesota. barsness.gregory@mayo.edu.
References
- [1].Killip T 3rd, Kimball JT. Treatment of myocardial infarction in a coronary care unit. A two year experience with 250 patients. Am J Cardiol 1967;20(4):457–64. [DOI] [PubMed] [Google Scholar]
- [2].Valente S, Lazzeri C, Sori A, Giglioli C, Bernardo P, Gensini GF. The recent evolution of coronary care units into intensive cardiac care units: the experience of a tertiary center in Florence. J Cardiovasc Med (Hagerstown) 2007;8(3):181–7. [DOI] [PubMed] [Google Scholar]
- [3].Casella G, Cassin M, Chiarella F, Chinaglia A, Conte MR, Fradella G, et al. Epidemiology and patterns of care of patients admitted to Italian Intensive Cardiac Care units: the BLITZ-3 registry. J Cardiovasc Med (Hagerstown) 2010;11(6):450–61. [DOI] [PubMed] [Google Scholar]
- [4].Katz JN, Shah BR, Volz EM, Horton JR, Shaw LK, Newby LK, et al. Evolution of the coronary care unit: clinical characteristics and temporal trends in healthcare delivery and outcomes. Crit Care Med 2010;38(2):375–81. [DOI] [PubMed] [Google Scholar]
- [5].Ratcliffe JA, Wilson E, Islam S, Platsman Z, Leou K, Williams G, et al. Mortality in the coronary care unit. Coron Artery Dis 2014;25(1):60–5. [DOI] [PubMed] [Google Scholar]
- [6].Holland EM, Moss TJ. Acute Noncardiovascular Illness in the Cardiac Intensive Care Unit. J Am Coll Cardiol 2017;69(16):1999–2007. [DOI] [PubMed] [Google Scholar]
- [7].Johnson AE, Kramer AA, Clifford GD. A new severity of illness scale using a subset of acute physiology and chronic health evaluation data elements shows comparable predictive accuracy. Crit Care Med 2013;41(7):1711–8. [DOI] [PubMed] [Google Scholar]
- [8].Zimmerman JE, Kramer AA, McNair DS, Malila FM. Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today’s critically ill patients. Crit Care Med 2006;34(5):1297–310. [DOI] [PubMed] [Google Scholar]
- [9].Chandra S, Kashyap R, Trillo-Alvarez CA, Tsapenko M, Yilmaz M, Hanson AC, et al. Mapping physicians’ admission diagnoses to structured concepts towards fully automatic calculation of acute physiology and chronic health evaluation score. BMJ Open 2011;1(2):e000216. [DOI] [PMC free article] [PubMed] [Google Scholar]
- [10].Knaus WA, Wagner DP, Draper EA, Zimmerman JE, Bergner M, Bastos PG, et al. The APACHE III prognostic system. Risk prediction of hospital mortality for critically ill hospitalized adults. Chest 1991;100(6):1619–36. [DOI] [PubMed] [Google Scholar]
- [11].Kramer AA, Zimmerman JE. Assessing the calibration of mortality benchmarks in critical care: The Hosmer-Lemeshow test revisited. Crit Care Med 2007;35(9):2052–6. [DOI] [PubMed] [Google Scholar]