Optimized Risk Score to Predict Mortality in Patients With Cardiogenic Shock in the Cardiac Intensive Care Unit

Abstract

Background

Mortality prediction in critically ill patients with cardiogenic shock can guide triage and selection of potentially high‐risk treatment options.

Methods and Results

We developed and externally validated a checklist risk score to predict in‐hospital mortality among adults admitted to the cardiac intensive care unit with Society for Cardiovascular Angiography & Interventions Shock Stage C or greater cardiogenic shock using 2 real‐world data sets and Risk‐Calibrated Super‐sparse Linear Integer Modeling (RiskSLIM). We compared this model to those developed using conventional penalized logistic regression and published cardiogenic shock and intensive care unit mortality prediction models. There were 8815 patients in our training cohort (in‐hospital mortality 13.4%) and 2237 patients in our validation cohort (in‐hospital mortality 22.8%), and there were 39 candidate predictor variables. The final risk score (termed BOS,MA₂) included maximum blood urea nitrogen ≥25 mg/dL, minimum oxygen saturation <88%, minimum systolic blood pressure <80 mm Hg, use of mechanical ventilation, age ≥60 years, and maximum anion gap ≥14 mmol/L, based on values recorded during the first 24 hours of intensive care unit stay. Predicted in‐hospital mortality ranged from 0.5% for a score of 0 to 70.2% for a score of 6. The area under the receiver operating curve was 0.83 (0.82–0.84) in training and 0.76 (0.73–0.78) in validation, and the expected calibration error was 0.9% in training and 2.6% in validation.

Conclusions

Developed using a novel machine learning method and the largest cardiogenic shock cohorts among published models, BOS,MA₂ is a simple, clinically interpretable risk score that has improved performance compared with existing cardiogenic‐shock risk scores and better calibration than general intensive care unit risk scores.

Clinical Perspective.

What Is New?

• Developed using 2 large real‐world data sets (n=8815 for training, n=2237 for validation) and a machine learning method, the 6‐component BOS,MA₂ (blood urea nitrogen ≥25 mg/dL, minimum oxygen saturation <88 %, minimum systolic blood pressure <80 mm Hg, use of mechanical ventilation, age ≥60 years, and maximum anion gap) checklist risk score has improved performance compared with existing models for predicting mortality in patients with cardiogenic shock in the cardiac intensive care unit.

What Are the Clinical Implications?

• The BOS,MA₂ risk score is a simple, clinically interpretable risk score that can guide clinical decision‐making at the bedside for patients with cardiogenic shock in the cardiac intensive care unit.

• This score can be used to assess the impact of treatment strategies on expected mortality, can enable the design of future clinical trials with more homogenous populations, and can serve as a model for developing future risk scores in cardiology.

Nonstandard Abbreviations and Acronyms

CICU

cardiac intensive care unit

ECE

expected calibration error

PLR

penalized logistic regression

RiskSLIM

Risk‐calibrated Super‐sparse Linear Integer Model

Mortality for cardiogenic shock remains high, with fewer than 70% of patients surviving to hospital discharge. ¹ , ² Randomized trials have largely been unsuccessful in identifying strategies to improve mortality for patients with cardiogenic shock, ³ , ⁴ , ⁵ aside from culprit‐vessel revascularization for myocardial infarction (MI). ⁶ Nevertheless, a spectrum of treatment options exists, ranging from emergent mechanical circulatory support to palliation. ⁷ The variation in outcomes for patients with cardiogenic shock may partly stem from differences in illness severity. ¹ In this setting, tools to stratify patients with cardiogenic shock can provide important prognostic information and guide the appropriate triage and selection of therapies.

The Society for Cardiovascular Angiography & Interventions (SCAI) Shock Stage classification system was developed to indicate cardiogenic shock severity ⁸ but represents only 1 component of cardiogenic shock mortality risk prediction and must be used in concert with tools to account for risk modifiers in cardiogenic shock evaluation and prognostication. ⁹ Generic models for predicting mortality in the intensive care unit (ICU) are ill suited to predicting mortality in the cardiogenic shock population, ¹⁰ as they were developed using large heterogeneous ICU patient populations. ¹ Conversely, models to predict mortality specifically for patients with cardiogenic shock are derived from small observational or clinical trial data sets with strict exclusion criteria, which limit their accuracy and generalizability. ¹¹ , ¹² , ¹³ , ¹⁴ , ¹⁵ , ¹⁶ , ¹⁷

To address this knowledge gap, we developed and validated a specialized risk score to predict mortality among patients with cardiogenic shock in the cardiac intensive care unit (CICU). We leveraged 2 large real‐world databases and a machine learning method developed to fit simple additive risk scores. ¹⁸ Such a risk score can quickly be calculated at the bedside and inform shared decision‐making regarding treatment options for critically ill patients.

Methods

Data Sources

We developed and validated our model using independent data sets from 2 publicly available clinical data repositories. Our training data set was derived from the Philips electronic ICU (eICU) database (eICU‐CRD v2.0), which is composed of 200 859 patient encounters for 139 367 unique patients admitted between 2014 and 2015 in 1 of 335 units in 208 hospitals located throughout the United States. ¹⁹ Our validation data set was derived from Medical Information Mart for Intensive Care III (MIMIC‐III), which is composed of 61 532 adult hospital admissions for 53 423 distinct patients admitted to critical care units between 2001 and 2012 at Beth Israel Deaconess Medical Center. ²⁰ Because of the sensitive nature of the data collected for this study, access to these data sets by qualified researchers trained in human subject confidentiality protocols may be obtained by following policies listed on PhysioNet (https://physionet.org/about/database/). Methods to replicate the findings of this study are available from the corresponding author upon reasonable request.

The use of the eICU data set for this study with waiver of informed consent was exempt from institutional review board approval, and the use of the MIMIC‐III with waiver of informed consent was approved by the institutional review board of the Massachusetts Institute of Technology and Beth Israel Deaconess Medical Center.

All data extraction and analyses were conducted using the cloud platform Google BigQuery, Python version 3.7, and R version 4.0.3.

Inclusion Criteria

We included all adult patients (age ≥18 years) admitted to a CICU who met the criteria for SCAI Shock Stage C or greater upon admission. ⁸ The SCAI Shock staging system was designed partly to enable the inclusion of a more homogeneous set of patients for enrollment in clinical trials and retrospective studies. ²¹ , ²² , ²³ , ²⁴ We included patients with SCAI Shock C or higher as those are categories representing patients with overt manifestations signs of shock. ⁸ Specifically, to be classified as SCAI Shock C, patients were required to have evidence of hypoperfusion, defined as a doubling of creatinine within 24 hours, blood lactate level >2.0 mmol/L, or the use of vasoactive medications. Our approach was similar to the operationalization of the SCAI Shock classification scheme in other real‐world data sets that included all patients admitted to CICUs regardless of admission diagnosis. ²¹ , ²⁴

Variables

The outcome of interest was in‐hospital mortality. We considered 39 distinct variables present in both data sets for potential inclusion in our risk model, including demographics (age, sex), an array of comorbidities, vital signs within the first 24 hours (heart rate, systolic blood pressure, respiratory rate, oxygen saturation), laboratory values within the first 24 hours of ICU admission (anion gap, bicarbonate, chloride, glucose, hematocrit/hemoglobin, platelets, potassium, international normalized ratio, sodium, creatinine, blood urea nitrogen, white blood cell count, and red cell distribution width), and use of critical care therapies (renal replacement therapy, mechanical ventilation, intra‐aortic balloon pump, and vasopressors). All variables were directly extracted from the databases except for comorbidities, for which International Classification of Diseases, Ninth Revision (ICD‐9) codes were mapped to comorbidities based on the Clinical Classifications Software categories.

As laboratory results and vital signs were recorded multiple times in the first 24 hours, we considered their minimum and maximum values as potential features in our model. We also evaluated various cutoffs for the dichotomization of quantitative variables as distinct features for our model.

All candidate variables had <25% missing values, and simple imputation using predictive mean matching was applied independently to the training and validation data sets. A complete list of all variables available in both data sets and their corresponding missingness is available (Table S1).

We compared the baseline characteristics of our patient population according to the outcome of interest using standardized differences, with a threshold of at least 10% used to define a meaningful difference. ²⁵

Statistical Analysis and Model Development

We built our risk score using Risk‐Calibrated Super‐sparse Linear Integer Model (RiskSLIM), ¹⁸ a machine learning method designed to fit simple customized risk scores optimized to yield calibrated risk estimates with few terms and small integer coefficients that have been used in clinical applications. ²⁶ , ²⁷

Risk‐Calibrated Super‐sparse Linear Integer Modeling

RiskSLIM uses modern optimization techniques to fit the best logistic regression model with small integer weights and a limited number of risk factors. ²⁸ , ²⁹ Compared with models developed heuristically, this technique can fit risk scores with better risk calibration and area under the receiver operating curve (AUC) by combining logistic regression with feature selection and continuous variable dichotomization techniques in a single step. In this application, the model was constrained to use unit weights to allow for quick computation at the bedside as a checklist.

Comparator Methods

We compared the results of the RiskSLIM models to those developed using conventional penalized logistic regression (PLR) to evaluate the potential loss in accuracy due to feature selection and the use of integer coefficients.

Training Procedure

We trained all models using the eICU data set because it includes patients from a more heterogenous population (ie, across 208 hospitals) ³⁰ and used the MIMIC data set for validation. We evaluated the performance of each model internally using 5‐fold cross‐validation. Platt scaling was employed in the final model to improve the reliability of estimates.

Performance Evaluation

We evaluated all models by rank accuracy and risk calibration. ³¹ We assessed rank accuracy via AUC. We assessed risk calibration by constructing a reliability diagram plotting the observed mortality compared with the predicted mortality and by reporting the expected calibration error (ECE), which reflects how close the predicted mortality risk is to the actual mortality risk.

Comparison to Other Risk Scores

We compared the performance of our RiskSLIM model to that of other published risk scores in our cohorts by comparing the AUC and ECE. We calculated 2 generic ICU scores, the Sequential Organ Failure Assessment score and Oxford Acute Severity of Illness Score, and the cardiogenic shock‐specific CardShock score among all patients with available data. All were computed using data from the first 24 hours of admission.

Supplemental Analysis

To assess the effect of the potential inclusion of a heterogeneous patient population in our cohort, we evaluated the final trained model on various subgroups of the validation set. The 4 subgroups were as follows: patients with a cardiovascular primary ICU admission diagnosis (cardiogenic shock, heart failure, angina, and MI), patients without a cardiovascular primary ICU admission diagnosis, the presence of MI necessitating acute coronary revascularization (percutaneous coronary intervention or coronary artery bypass grafting during hospitalization), and the absence of MI necessitating acute coronary revascularization. Admission diagnoses in the MIMIC‐III were assigned by clinicians and stored as free text.

Primary admission diagnosis was not used to define our study cohort, given that this was not included in prior operationalizations of the SCAI Shock classification scheme in real‐world data sets. In addition, admission diagnoses in eICU and MIMIC are recorded inconsistently. Using them would limit our model's applicability due to the inconsistent capture and accuracy of admission diagnoses in those 2 databases.

Key reporting elements for machine learning analyses in clinical research are summarized in a standardized format in Table S2. ³²

Results

Patient Characteristics

eICU Cohort

Of 29 626 patients admitted to a CICU in the eICU database, 8815 (29.8%) met our inclusion criteria for our final training cohort (Figure 1). The overall in‐hospital mortality rate in the eICU cohort was 13.5% (Table 1).

Figure 1. Study patient flow diagram.

eICU, indicates electronic intensive care unit; ICU, intensive care unit; MIMIC, Medical Information Mart for Intensive Care; and SCAI C, Society for Cardiovascular Angiography & Interventions Shock Stage C.

Table 1.

Baseline Characteristics of eICU Cohort Stratified by Outcome

	Survived to discharge	In‐hospital mortality	Standardized difference
	(n=7627)	(n=1188)
Demographics
Age, y, mean (SD)	63.69 (14.64)	67.80 (13.55)	0.291
Male sex (%)	4543 (59.6)	691 (58.2)	0.028
Comorbidities
Cerebral vascular disease (%)	830 (10.9)	152 (12.8)	0.059
Anemia (%)	40 (0.5)	11 (0.9)	0.047
Atrial fibrillation (%)	1065 (14.0)	224 (18.9)	0.132
Blood malignancy (%)	104 (1.4)	32 (2.7)	0.094
Solid neoplasm (%)	876 (11.5)	192 (16.2)	0.136
Congestive heart failure (%)	1703 (22.3)	323 (27.2)	0.113
Chronic kidney disease (%)	1302 (17.1)	282 (23.7)	0.166
Chronic obstructive pulmonary disease (%)	1192 (15.6)	266 (22.4)	0.173
Coronary artery disease (%)	2099 (27.5)	287 (24.2)	0.077
Diabetes (%)	2575 (33.8)	409 (34.4)	0.014
Valvulopathy (%)	824 (10.8)	98 (8.2)	0.087
Hypertension (%)	4778 (62.6)	718 (60.4)	0.045
Metastatic cancer (%)	92 (1.2)	33 (2.8)	0.113
Prior myocardial infarction (%)	1191 (15.6)	156 (13.1)	0.071
Cardiac ICU therapies
Renal replacement therapy(%)	375 (4.9)	84 (7.1)	0.091
Mechanical ventilation (%)	2099 (27.5)	672 (56.6)	0.616
Intra‐aortic balloon pump (%)	363 (4.8)	80 (6.7)	0.085
≥1 vasopressor (%)	6917 (90.7)	1040 (87.5)	0.101
≥1 inotrope (%)	1156 (15.2)	201 (16.9)	0.048
Vital signs
Heart rate min, mean (SD)	63.19 (13.78)	59.85 (20.53)	0.191
Heart rate max, mean (SD)	113.12 (25.30)	129.80 (27.79)	0.628
Systolic BP min, mean (SD)	88.12 (19.18)	70.67 (20.45)	0.880
Systolic BP max, mean (SD)	158.73 (27.69)	156.33 (33.45)	0.078
Respiratory rate min, mean (SD)	10.44 (4.93)	8.22 (7.22)	0.359
Respiratory rate max, mean (SD)	32.26 (9.37)	36.29 (9.69)	0.423
Oxygen saturation min, mean (SD)	86.31 (13.97)	69.23 (24.93)	0.845
Laboratory results
Glucose min, mean (SD)	117.98 (45.25)	128.73 (68.16)	0.186
Anion gap max, mean (SD)	12.89 (5.85)	17.87 (7.60)	0.734
Bicarbonate min, mean (SD)	22.58 (4.99)	18.97 (6.18)	0.642
Chloride max, mean (SD)	105.98 (6.39)	105.99 (7.92)	0.002
Hematocrit max, mean (SD)	37.15 (6.50)	36.87 (7.41)	0.041
Hemoglobin min, mean (SD)	10.58 (2.37)	10.11 (2.57)	0.188
Platelet min, mean (SD)	181.49 (89.18)	173.15 (98.23)	0.089
Potassium max, mean (SD)	4.61 (0.81)	4.99 (1.00)	0.405
International normalized ratio max, mean (SD)	1.51 (0.88)	2.11 (1.71)	0.439
Sodium min (mean (SD))	135.61 (5.17)	135.33 (6.09)	0.049
Blood urea nitrogen max, mean (SD)	29.70 (22.38)	44.15 (28.00)	0.570
White blood cell max, mean (SD)	14.02 (9.10)	18.61 (16.95)	0.338
Red cell distribution width max, mean (SD)	15.32 (2.34)	16.60 (3.03)	0.472
Creatinine >1.5× baseline (%)	1439 (18.9)	409 (34.4)	0.358
Outcomes
ICU length of stay, d, mean (SD)	1.48 (2.09)	1.77 (2.25)	0.133
Hospital length of stay, d, mean (SD)	9.17 (9.68)	7.35 (8.85)	0.197
Time to death, d, mean (SD)	—	5.39 (6.98)	—

BP indicates blood pressure; and ICU, intensive care unit.

Patients with in‐hospital mortality were more likely to be older (68 versus 64 years) and to have atrial fibrillation (19% versus 14%), solid neoplasm (16% versus 12%), congestive heart failure (27% versus 22%), chronic kidney disease (24% versus 17%), chronic obstructive pulmonary disease (22% versus 16%), or metastatic cancer (3% versus 1%). Additionally, patients with in‐hospital mortality were more likely to be mechanically ventilated (57% versus 28%).

On average, patients with in‐hospital mortality had a higher maximum heart rate (130 bpm versus 113 bpm), lower minimum systolic blood pressure (71 mm Hg versus 88 mm Hg), higher maximum respiratory rate (36/min versus 32/min), and lower minimum oxygen saturation (69% versus 86%) despite the use of all available therapies over the first 24 hours of CICU stay. On average, patients with in‐hospital mortality were also more likely to have hyperglycemia (average minimum glucose 129 versus 118), higher maximum anion gap (18 versus 13), lower minimum bicarbonate (19 versus 23), higher maximum potassium (5.0 versus 4.6), higher maximum international normalized ratio (2.1 versus 1.5), higher maximum blood urea nitrogen (44 versus 30), and higher maximum white blood cell count (19 000 versus 14 000), and were more likely to have a 50% increase in creatinine (34% versus 19%) in the first 24 hours.

MIMIC Cohort

Of the 6802 patients admitted to a CICU in the MIMIC database, 2237 (32.9%) met the inclusion criteria and were included in the final validation cohort (Figure 1). The overall in‐hospital mortality rate in the MIMIC cohort was 22.8% (Table S3).

Cardiogenic Shock Risk Score

After fitting several models using RiskSLIM for model size constraints between 1 and 10 and comparing performance on AUC and calibration (Figure S1), the RiskSLIM model with 6 variables was chosen based on both statistical performance and clinical interpretability. For similar accuracy, the 6‐variable model had better calibration compared with the 4‐, 5‐, and 7‐variable models.

In our final risk score, termed BOS,MA₂, a patient would receive a point for each of the following: maximum blood urea nitrogen ≥25 mg/dL, minimum oxygen saturation <88%, minimum systolic blood pressure <80 mm Hg, any use of mechanical ventilation, age ≥60 years, and maximum anion gap ≥14 mmol/L, based on values recorded during the first 24 hours of ICU stay (Figure 2). Predicted in‐hospital mortality was 0.5% for a score of 0, 1.4% for a score of 1, 3.9% for a score of 2, 10.0% for a score of 3, 23.5% for a score of 4, 46% for a score of 5, and 70.2% for a score of 6. Patients with all risk score values were well represented in both the training and validation cohorts (Figure S2).

Figure 2. BOS,MA₂ risk score for cardiogenic shock mortality in the cardiac intensive care unit.

A patient receives 1 point for meeting each of the criteria as specified in the risk score. The BOS,MA₂ risk score is calculated by summing the number of points. The mortality risk corresponds to the number of points in the reference table. BUN indicates blood urea nitrogen. BOS,MA₂ risk score defined as maximum BUN ≥25 mg/dL, minimum oxygen saturation <88%, minimum systolic blood pressure <80 mm Hg, any use of mechanical ventilation, age ≥60 years, and maximum anion gap ≥14 mmol/L.

Model Validation

The AUC for the BOS,MA₂ risk score model was 0.83 (95% CI, 0.82–0.84) in the training data set and 0.76 (95% CI, 0.73–0.78) in the validation data set (Table 2; Figure 3A). This was only slightly lower than the comparator full PLR model with 51 variables (0.80 [95% CI, 0.78–0.82]), though similar to a parsimonious PLR model including only the same 6 variables in the risk score (0.76 [95% CI, 0.73–0.78]; Table S4). Subsequent inclusion of the BOS,MA₂ risk score itself into comparator full or parsimonious PLR models did not enhance prediction over baseline clinical characteristics alone (Table S4).

Table 2.

BOS,MA₂ Risk Score Model Performance Compared With Other Validated Risk Scores

Cohort	eICU (training)		MIMIC‐III (validation)
Risk score	AUC (95% CI)*	ECE	AUC (95% CI)*	ECE
BOS,MA2	0.83 (0.82–0.84)	0.9%	0.76 (0.73–0.78)	2.6%
CardShock score†			0.66 (0.63–0.69)	11.4%
Sequential Organ Failure Assessmentscore, day 1	0.76 (0.74–0.78)	6%	0.75 (0.73–0.78)	4.7%
OASIS score, day 1			0.77 (0.74–0.79)	4.1%

AUC indicates area under the curve; BOS,MA₂, blood urea nitrogen ≥25 mg/dL, minimum oxygen saturation <88%, minimum systolic blood pressure <80 mm Hg, any use of mechanical ventilation, age ≥60 years, and maximum anion gap ≥14 mmol/L; ECE, expected calibration error; eICU, electronic intensive care unit; MIMIC, Medical Information Mart for Intensive Care; and OASIS, Oxford Acute Severity of Illness Score.

Variables to calculate CardShock and OASIS scores not available in eICU (training) data set.

^*95% CI computed using the DeLong statistic.

^† Calculated based on 1269 patients with complete information available for all variables.

Figure 3. BOS,MA₂ risk score performance.

A, Receiver operating curve of BOS,MA₂ risk score model. B, Calibration plot of BOS,MA₂ risk score model. BOS,MA₂ risk score defined as maximum blood urea nitrogen ≥25 mg/dL, minimum oxygen saturation <88%, minimum systolic blood pressure <80 mm Hg, any use of mechanical ventilation, age ≥60 years, and maximum anion gap ≥14 mmol/L. Black=training cohort, gray=5‐fold cross validation, brown=validation cohort.

The calibration error for the BOS,MA₂ risk score model was 0.9% in the training data set and 2.6% in the validation data set (Table 2; Figure 3B). The calibration error for the BOS,MA₂ risk score model on the validation data set was lower than both the full PLR model and the parsimonious PLR model with the same features (Table S4).

Comparison to Other Risk Scores

In our validation cohort, the AUC for the BOS,MA₂ score (0.76 [95% CI, 0.73–0.78]) was greater than that for the cardiogenic shock‐specific CardShock score (0.66 [95% CI, 0.63–0.69]) and similar to general ICU risk scores (Sequential Organ Failure Assessment 0.75 [95% CI, 0.73–0.78]; Oxford Acute Severity of Illness Score 0.77 [95% CI, 0.74–0.79]; Table 2). However, the validation cohort ECE for the BOS,MA₂ score (2.6%) was lower than all other scores examined (CardShock 11.4%, Sequential Organ Failure Assessment 4.7%, Oxford Acute Severity of Illness Score 4.1%).

Supplemental Analysis

There was no reduction in the model's performance among the validation cohort subgroups examined (Table S5). Notably, among 305 patients with a cardiovascular primary admission diagnosis in the MIMIC data set, the BOS,MA₂ risk score model had similar discrimination (AUC 0.78 [95% CI, 0.73–0.75]) and calibration (ECE 1.05%) compared with when applied to the entire validation cohort. Similarly, among 208 patients with cardiogenic shock associated with MI necessitating acute coronary revascularization and 478 patients with cardiogenic shock without MI necessitating acute coronary revascularization, the model's performance was comparable with an AUC of 0.83 (95% CI, 0.77–0.88) and 0.79 (95% CI, 0.75–0.83), respectively. Notably, the ECE of the BOS,MA₂ risk score was significantly lower than that of the general CICU risk scores in looking at these cardiovascular‐specific subgroups (Table S6).

Discussion

We developed and externally validated a checklist‐based risk score to predict mortality among patients with cardiogenic shock admitted to the CICU using a machine learning algorithm and distinct large real‐world training and validation data sets. The BOS,MA₂ risk score is methodologically robust, generalizable, and readily applicable at the bedside. In addition, this clinical tool has direct practical implications for managing patients with cardiogenic shock.

The BOS,MA₂ risk score model outperforms existing cardiogenic shock and general ICU mortality risk scores in predicting mortality among patients with cardiogenic shock in the CICU. The BOS,MA₂ risk score achieves higher discrimination in external validation than the published CardShock, and IABP‐SHOCK II (Intraaortic Balloon Pump in Cardiogenic Shock II) risk score models (Table 3), ¹¹ , ¹⁷ , ³³ , ³⁴ and performs better than the CardShock score in our own external validation cohort (AUC 0.76 versus 0.66; ECE 2.6% versus 11.4%). Although general ICU risk scores can have high AUCs in external validation in all‐comer ICU cohorts, they have similar discrimination as the BOS,MA₂ risk score in our cardiogenic shock cohort, indicating the challenges of predicting mortality in this complex cardiogenic shock population. Notably, the BOS,MA₂ risk score model has better calibration than Sequential Organ Failure Assessment score and Oxford Acute Severity of Illness Score in our validation cohort, particularly in looking at cardiovascular‐specific subgroups, such as those with MI necessitating acute coronary revascularization. Calibration error in clinical populations in which a score will be used is an important, yet often overlooked, feature of risk score performance in cardiology. ³⁵ Better calibration justifies using the BOS,MA₂ score over general ICU mortality risk scores for patients in cardiogenic shock in the CICU.

Table 3.

Comparison of Established Cardiogenic Shock Risk Scores

	BOS,MA2 score	CardShock score11	IABP‐SHOCK II17	SHOCK trial registry15
Training cohort sample size	8815	219	480	1217
Validation cohort sample size	2237	384	235	None
Validation	External	External	External	None
Area under the curve internal, (external)	0.83 (0.76)	0.85 (0.71)	0.79 (0.73)	0.76 (−)
Total number of predictors	6	7	6	8
Past medical history
History of coronary artery bypass graft		X		X
History of stroke			X
Admission diagnosis
Acute coronary syndrome on admission		X		X
Shock on admission				X
Clinical variables
Age	X	X	X	X
Renal function	X	X	X	X
Glucose >10.6 mmol/L		X	X	X
Anion gap >14	X
End‐organ hypoperfusion		X (lactate)	X (lactate)	X (clinical end point)
Neurologic dysfunction		X (confusion)		X (anoxic brain damage)
Mechanical ventilation	X
Systolic blood pressure	X			X
Oxygen saturation <88	X
Catheterization laboratory and hemodynamic measurements
Thrombolysis in myocardial infarction flow			X
Left ventricular ejection fraction		X

BOS,MA₂ indicates blood urea nitrogen ≥25 mg/dL, minimum oxygen saturation <88%, minimum systolic blood pressure <80 mm Hg, any use of mechanical ventilation, age ≥60 years, and maximum anion gap ≥14 mmol/L; IABP‐SHOCK II, Intraaortic Balloon Pump in Cardiogenic Shock II; and SHOCK trial, Should We Emergently Revascularize Occluded Coronaries in Cardiogenic Shock.

The BOS,MA₂ risk score contains certain elements present in other cardiogenic shock risk prediction models but also includes unique features (Table 3). Age has been consistently associated with an increased risk of death in cardiogenic shock, though cutoffs vary across studies. ¹¹ , ¹³ , ¹⁴ , ¹⁵ , ¹⁶ , ¹⁷ , ³⁶ , ³⁷ , ³⁸ , ³⁹ The RiskSLIM method used in our study enabled us to consider a range of cutoffs simultaneously, and age over 60 was deemed the most discriminant. Kidney dysfunction, as measured by blood urea nitrogen in our study, is also predictive of cardiogenic shock mortality in other studies in the form of creatinine level ¹⁵ , ¹⁶ , ¹⁷ or glomerular filtration rate. ¹¹ Many existing models have also included systolic blood pressure ¹⁵ , ³⁷ or other proxies for hypoperfusion, such as lactate, ¹¹ , ¹⁶ , ¹⁷ , ³⁶ altered mental status, ¹¹ , ¹⁵ and anion gap. ⁴⁰ This is consistent with our model, which also includes both systolic blood pressure and an elevated anion gap.

The BOS,MA₂ risk score includes 2 distinct markers of respiratory failure: mechanical ventilation and hypoxemia (SpO₂ < 88%). Only 1 other model predicting outcomes among cardiogenic shock patients included respiratory failure. However, this model was built on a more heterogenous population of patients admitted to the CICU, and only 14% of patients in the development cohort had shock. ⁴⁰ Most published cardiogenic shock mortality risk scores did not consider mechanical ventilation or oxygen saturation as candidate predictors during development, suggesting that respiratory function may frequently be overlooked despite the pathophysiologic mechanisms linking respiratory and cardiac dysfunction in cardiogenic shock. The use of large real‐world data sets with a wide variety of candidate input parameters enabled the detection of respiratory function inputs as critical predictors of cardiogenic shock mortality using the RiskSLIM method.

Our model has implications for the clinical care patients with cardiogenic shock and for designing future clinically relevant risk scores. The BOS,MA₂ risk score can quickly be calculated at the bedside as a checklist of 6 objective variables that are often readily available, making it easy to remember and implement in clinical decision‐making situations, such as while rounding on patients in the CICU. Additionally, given that it was derived from raw electronic health record data, this risk score can easily be integrated in the electronic health record to facilitate its adoption into clinical practice, which has been an issue with adoption of other clinical risk prediction tools. ⁴¹ The prediction tool can be used in conjunction with the SCAI Shock classification system to help triage patients efficiently, obtain reliable prognostic information, and guide clinical decision‐making for a challenging patient population for which a myriad of potentially high‐risk therapeutic options are available. ⁸ , ⁹ Additionally, this risk score can be used to assess the impact of treatment strategies on expected mortality and can enable the design of future clinical trials with more homogenous populations. Our model also has broader implications for the development of risk scores in clinical cardiology. The machine learning algorithm used in this article can be used in other clinical contexts. Our article thus serves as a model for developing interpretable risk scores in cardiology using the RiskSLIM methodology.

Our study has several key strengths. First, we used 2 large real‐world cohorts (n=8815 for training, n=2237 for external validation), which are the largest cohorts of patients used to develop a cardiogenic shock mortality prediction model to date. Other scores have mainly used much smaller cohorts derived from clinical trials or with narrower inclusion criteria focusing on a single cause of cardiogenic shock or specific mechanical circulatory support, ¹⁴ , ¹⁷ , ³⁹ which may have limited generalizability when applied to real‐world settings. Second, we employed a machine learning algorithm specifically designed to fit simple customized clinically interpretable risk scores optimized to yield calibrated risk estimates. This is in stark contrast to other machine learning methods, which often offer only minor improvements in calibration at the expense of clinical interpretability when applied to generating clinical risk scores, such as in the case of acute MI mortality prediction. ⁴² In comparison to traditional logistic regression methods, we found similar or improved performance despite using only 6 terms as a checklist, which can aid in calculation at the bedside.

Our findings must be interpreted in the context of their limitations. First, the in‐hospital mortality observed in our training cohort is lower than that observed in our validation cohort, likely reflecting the lower severity of patients in the CICUs represented in the eICU cohort, which are primarily community hospitals, relative to those in the singular large tertiary care center represented by the MIMIC database. However, the similar performance of our model across both cohorts reflects the generalizability of our risk score and substantiates its broad applicability across a range of clinical settings. Second, although we included only patients admitted to CICUs, it is possible that we have included patients with septic or mixed shock in our cohorts given that we did not consider admission diagnosis in defining our primary cohort. However, model performance was similar in subgroups of patients with a cardiovascular primary ICU admission diagnosis and with cardiogenic shock with MI necessitating acute revascularization. This suggests that inclusion of such patients was unlikely to affect our results and that our model is valid in these subpopulations. Third, we may not have captured all patients in our real‐world data sets meeting the SCAI Shock C classification criteria due to missing data. However, the mortality rate in our validation cohort was similar to that in another real‐world tertiary care center cohort that operationalized the SCAI shock classification criteria, ²⁴ suggesting that the population of patients included in our study likely reflects the true population of patients with cardiogenic shock in CICUs. Fourth, several clinically relevant variables in other risk scores were unavailable in our data set due to inconsistent capture. Specifically, the presence of mechanical circulatory support beyond the intra‐aortic balloon pump was not documented accurately in the eICU database and was likely rare, given that eICU hospitals were primarily community hospitals. However, we included a comprehensive range of other candidate variables, which captured similar clinical constructs (such as the use of anion gap in place of blood lactate), and our model performance exceeded that of other published models. Fifth, our model is developed based on data from the first 24 hours of CICU admission only. Although other variables may be more critical for prognosis at other time points in a patient's CICU course, we felt that the first 24 hours was the period during which many major triage and therapeutic decisions are made in the CICU. Understanding the relative importance of other variables at different time points in a patient's CICU course remains a rich area for future inquiry. Finally, as the population of patients in the CICU changes over time, it is very likely that this score will need recalibration in the future, which can likely be accomplished using newer versions of the validation data set in the future. ⁴³

Conclusions

In conclusion, we used 2 large real‐world data sets and a machine learning method to develop an externally validated mortality prediction risk score for patients with cardiogenic shock in the cardiac intensive care unit. The BOS,MA₂ risk score is a simple, clinically‐interpretable score that can guide clinical decision‐making for patients with cardiogenic shock. This score can serve as a model for developing future risk scores in cardiology.

Sources of Funding

Dr. Butala is supported by the Boettcher Foundation Webb‐Waring Biomedical Research Grant. Dr. Ustun is supported by the National Science Foundation IIS 2040880.

Disclosures

Dr Butala reports consulting fees and ownership interest in Hilabs, Inc. Dr Yeh reports grant/contracts support from Abiomed, AstraZeneca, BD Bard, Boston Scientific, Cook Medical, Medtronic, and Philips, and consulting fees from Abbott, Boston Scientific, Edwards Lifesciences, Medtronic, and Shockwave Medical outside the submitted work. The other authors report no conflicts.

Supporting information

Tables S1–S6.

Figures S1–S2.