Simplified Mortality Score for the Intensive Care Unit (SMS-ICU): protocol for the development and validation of a bedside clinical prediction rule

Abstract

Introduction

Mortality prediction scores are widely used in intensive care units (ICUs) and in research, but their predictive value deteriorates as scores age. Existing mortality prediction scores are imprecise and complex, which increases the risk of missing data and decreases the applicability bedside in daily clinical practice. We propose the development and validation of a new, simple and updated clinical prediction rule: the Simplified Mortality Score for use in the Intensive Care Unit (SMS-ICU).

Methods and analysis

During the first phase of the study, we will develop and internally validate a clinical prediction rule that predicts 90-day mortality on ICU admission. The development sample will comprise 4247 adult critically ill patients acutely admitted to the ICU, enrolled in 5 contemporary high-quality ICU studies/trials. The score will be developed using binary logistic regression analysis with backward stepwise elimination of candidate variables, and subsequently be converted into a point-based clinical prediction rule. The general performance, discrimination and calibration of the score will be evaluated, and the score will be internally validated using bootstrapping. During the second phase of the study, the score will be externally validated in a fully independent sample consisting of 3350 patients included in the ongoing Stress Ulcer Prophylaxis in the Intensive Care Unit trial. We will compare the performance of the SMS-ICU to that of existing scores.

Ethics and dissemination

We will use data from patients enrolled in studies/trials already approved by the relevant ethical committees and this study requires no further permissions. The results will be reported in accordance with the Transparent Reporting of multivariate prediction models for Individual Prognosis Or Diagnosis (TRIPOD) statement, and submitted to a peer-reviewed journal.

Strengths and limitations of this study.

• Publication of a protocol for an observational study reduces risk of bias and increases transparency.

• Use of prospectively recorded data from international, contemporary, high-quality studies/trials increase data validity and generalisability.

• Use of 90-day mortality as outcome measure will not be affected by intensive care unit (ICU)/hospital discharge practices, as compared with in-ICU or in-hospital mortality.

• The original study/trial databases comprise a limited number of variables because of the pragmatic designs.

• The included participants, interventions and comparators assessed in the original studies/trials differ slightly, but all patients were adults acutely admitted to an ICU.

Introduction

This protocol describes the development of a new clinical prediction rule for use in the intensive care unit (ICU): the Simplified Mortality Score for use in the Intensive Care Unit (SMS-ICU). The protocol describes the derivation and internal validation of the score (phase 1) and the external validation (phase 2). An overview of the steps described in the protocol is provided in figure 1. This manuscript has been prepared according to the Transparent Reporting of multivariate prediction models for Individual Prognosis Or Diagnosis (TRIPOD) statement¹ (checklist included as online supplementary file 1).

Figure 1.

Flow chart illustrating the major steps in the development and validation of the score as described in this protocol. SAPS, Simplified Acute Physiology Score; SMS-ICU, Simplified Mortality Score for the Intensive Care Unit; SOFA, Sequential Organ Failure Assessment.

Background

Severity scores are used in ICUs to assess disease severity, make prognostic predictions, compare the performance of ICUs and in research.^2–4 The performance of severity scores deteriorates over time,^{5 6} and one of the most widely used mortality prediction scores, the Simplified Acute Physiology Score (SAPS) II, is more than 20 years old.⁷

Most mortality prediction scores include many different variables and are thus very complex. This limits clinical use and increases the risk of missing data which is a common challenge when used in research.^8–10 More complex scores do not necessarily perform better, and the performance of the newer and more complex SAPS 3¹¹ is comparable to that of SAPS II.^12–16 The Sequential Organ Failure Assessment (SOFA) score¹⁷ is somewhat simpler than SAPS II, and although it was not developed for this purpose, it appears to be only slightly inferior to SAPS II at predicting mortality.^{18 19} Recently, Dólera-Moreno et al²⁰ presented a simple score that predicts in-ICU mortality, based on data from a single ICU.

A simple and contemporary clinical prediction rule using few readily available variables developed on data from multiple centres and using 90-day mortality as the outcome measure, could prove a valuable tool for use at the bedside in daily clinical practice in the ICU and in research.

Hypothesis and aim

We hypothesise that a new, simple and clinically useful mortality prediction score for use in the ICU can be developed and used to reliably estimate the risk of mortality within 90 days of admission to the ICU. Additionally, we hypothesise that such a score will perform as well as existing and more complex scores (ie, SAPS II).

Accordingly, we aim to develop and validate a simple clinical prediction rule—the SMS-ICU—which can be used at the bedside on ICU admission to reliably estimate the risk of mortality 90 days after ICU admission.

Methods and analysis

As described by Labarère et al,²¹ the development of clinical prediction rules encompasses three consecutive phases: derivation, external validation and impact analysis. This protocol describes the derivation and internal validation (phase 1) and the external validation (phase 2) of the score. If the score performs satisfactory during the validations, we will encourage use of the score in everyday clinical practice and subsequently conduct an impact analysis the details of which will be planned later.

Phase 1: development and internal validation

In the first phase of the project, a mortality prediction model will be developed, evaluated and converted into a simple score that can be used at the bedside. Subsequently, the score will be internally validated.

Study design

An international cohort study comprising patients from the Scandinavian Starch for Severe Sepsis/Septic Shock (6S) trial,⁸ the Transfusion Requirements In Septic Shock (TRISS) trial,⁹ the Conservative versus Liberal Approach to fluid therapy of Septic Shock in Intensive Care (CLASSIC) trial,²² the Stress Ulcer Prophylaxis in the Intensive Care Unit (SUP-ICU) inception cohort study¹⁰ and the Agents Intervening against Delirium in Intensive Care Unit (AID-ICU) inception cohort study.²³

Study participants

All 4247 patients included in the 6S (n=798; 26 ICUs in 4 countries),⁸ TRISS (n=998; 32 ICUs in 4 countries)⁹ and CLASSIC trials (n=151; 9 ICUs in 2 countries),²² and in the SUP-ICU (n=1034; 97 ICUs in 11 countries)¹⁰ and AID-ICU (n=1266; 99 ICUs in 13 countries)²³ inception cohort studies.

The sample consists of adult (≥18 years) critically ill patients acutely admitted to ICUs in university and non-university hospitals. Enrolment of patients in the development sample took place between 23 December 2009 and 30 June 2016. Data collection for the last included patients in the development sample has recently concluded and data are currently undergoing validation. Specific inclusion and exclusion criteria and enrolment dates for the studies are listed in online supplementary file 2.

Data sources

A combined database comprising data from the 6S, TRISS and CLASSIC trial databases, and from the SUP-ICU and AID-ICU inception cohort databases.

Variables

The outcome measure will be 90-day mortality (90 days following inclusion in the trials/studies). Adverse prognostic factors for 90-day mortality will be assessed.

Table 1 contains a complete list of the variables that will be included in the combined database. All variables are known potential predictors of 90-day mortality, and all variables are registered in the 6S, TRISS, CLASSIC and SUP-ICU trials and in the SUP-ICU and AID-ICU inception cohort studies. For the selection of candidate variables from this list, see the Statistics section.

Table 1.

Variables studied

Variable	Data format
Demographic/anamnestic variables
Age	Years (continuous)
Gender	Male/female (binary)
Admission type	Surgical/medical (binary)
Hospital length of stay prior to ICU admission	Days (continuous)
Comorbidity variables
Metastatic cancer	Yes/no (binary)
Haematological malignancy	Yes/no (binary)
AIDS	Yes/no (binary)
Physiological variables
Lowest heart rate*	Beats per minute (continuous)
Highest heart rate*	Beats per minute (continuous)
Lowest systolic blood pressure*	mm Hg (continuous)
Highest systolic blood pressure*	mm Hg (continuous)
Body temperature ≥39° C*	Yes/no (binary)
Biochemical variables
Lowest white cell count*	109/L (continuous)
Highest white cell count*	109/L (continuous)
Lowest potassium*	mmol/L (continuous)
Highest potassium*	mmol/L (continuous)
Lowest sodium*	mmol/L (continuous)
Highest sodium*	mmol/L (continuous)
Lowest bicarbonate*	mmol/L (continuous)
Treatment variables
Use of vasopressors/inotropes†	Yes/no (binary)
Use of mechanical ventilation†	Yes/no (binary)
Use of renal replacement therapy†	Yes/no (binary)
Outcome variable
Vital status 90 days after inclusion	Dead/alive (binary)

For detailed definitions of the variables, please refer to the original publications/protocols.^{8–10 22–24}

All variables registered at inclusion in the different studies/trials except where otherwise noted.

*Worst value registered during the 24 hours prior to inclusion (6S, TRISS, CLASSIC and SUP-ICU trials) or during the first 24 hours in the ICU (SUP-ICU and AID-ICU inception cohorts).

†Treatment initiated or continued on the day of inclusion.

6S, Scandinavian Starch for Severe Sepsis/Septic Shock; AID-ICU, Agents Intervening against Delirium in Intensive Care Unit; ICU, intensive care unit; CLASSIC, Conservative versus Liberal Approach to fluid therapy of Septic Shock in Intensive Care; SUP-ICU, Stress Ulcer Prophylaxis in the Intensive Care Unit; TRISS, Transfusion Requirements In Septic Shock.

Owing to the hard outcome measure, no actions will be taken to blind investigators to the outcome or predictor variables.

Statistics

Baseline and clinical characteristics for all patients in the combined database will be presented as medians with IQRs for continuous data, and numbers (%) for categorical data stratified by 90-day mortality.²⁵ Differences will be assessed by Mann-Whitney U test and χ² test as appropriate.

The score will be developed in five steps:

1. Candidate predictor variables will be selected;

2. Candidate variables with more than 25% missing data will be excluded;

3. For the remaining variables, multiple imputations will be performed for the missing values;

4. A logistic regression model with 90-day mortality as the outcome of interest will be developed;

5. The logistic regression model will be converted into a clinical prediction rule.²⁶

Candidate variables

Candidate variables for the score will be selected based on three criteria:

1. Variables have to be potentially predictive of mortality based on existing literature;

2. Variables have to be readily available on ICU admission in everyday clinical practice;

3. Variables have to be present in the 6S, TRISS and CLASSIC trials databases, in the SUP-ICU and AID-ICU inception cohort study databases, and in the SUP-ICU trial database²⁴ (for external validation, phase 2).

A complete list of candidate variables can be found in table 1.

Missing data and multiple imputation

Candidate variables with more than 25% missing data will be excluded. As data are unlikely to be missing completely at random, multiple imputations for the missing values for the remaining candidate variables will be performed.^{27 28} Multiple imputations will be performed with the fully conditional specification method with 10 imputed data sets and with the inclusion of the outcome measure and the candidate variables remaining after variables with more than 25% missing data having been excluded.

Multiple imputations for missing outcome data will not be performed and patients with missing 90-day mortality data will be excluded from all analyses.²⁷

Logistic regression modelling

Binary logistic regression analysis including the selected candidate variables as independent co-variates and 90-day mortality as the dependent variable will be performed. Continuous co-variates will not be converted to categorical co-variates during this process. Transformation to address possible non-linearity will be considered. Backward stepwise elimination modelling will be used based on the p values for the individual variables obtained from the likelihood ratio test.²¹ To ensure that the score is suitable for use at the bedside, we will aim to limit the number of variables to ensure simplicity and clinical relevance. In order to do this, a prespecified significance threshold for the elimination of variables has not been set. Results will be presented as adjusted ORs with 95% CIs and regression coefficients (β-values).

Conversion into the clinical prediction rule

The logistic regression model will be converted into a clinical prediction rule—the SMS-ICU—by transforming the individual parameter estimates (log(OR)/β-values) of the model into weights (points) according to the method used in the Framingham Heart Study.²⁹ Continuous variables will be converted to ordinal categorical variables based on clinically meaningful thresholds and the lowest adjusted OR/β-value will be used as reference for each ordinal variable. An increasing amount of points will then be assigned according to equivalently spaced adjusted OR/β-values.²⁹ The same method will be used for dichotomous variables.^{29 30} A logistic regression analysis with the total score as the only independent variable and 90-day mortality as the dependent variable will be conducted, and the full regression model will be presented.

Evaluation of the score

Predicted 90-day mortality risks for the different scores along with sensitivity, specificity and observed 90-day mortality in the full derivation sample will be presented.

The score will be evaluated using model diagnostics. The overall predictive performance of the score will be evaluated using Nagelkerke's R² for binary exposure,³¹ which refers to the proportion of variation explained by the model. Discrimination, which refers to the score's ability to distinguish patients that die from patients that live, will be assessed using the area under the receiver operating characteristics curve (AUC).³² Finally, calibration will be assessed as recommended in the literature,^{21 33} via calibration plots and intercept α and slope β from the regression of the binary outcome (90-day mortality) on the predicted values,³⁴ supplemented by the Hosmer-Lemeshow goodness-of-fit Ĉ-statistic.³⁵

Internal validation

The score will be internally validated using bootstrapping.²¹ The principle of bootstrapping is to sample from the empirical distribution from which the data originated (ie, sampling with replacement from the observed data).^{26 36}

The same performance measures will be used during the internal validation process as during the evaluation process, and optimism-corrected performance estimates will be presented.²¹

Sensitivity analysis

The development sample will contain data from the randomised 6S, TRISS and CLASSIC trials. In the TRISS and CLASSIC trials, no statistically significant differences in 90-day mortality was observed between the two treatment groups;^{9 22} however, in the 6S trial, mortality was increased in the group allocated to hydroxyethyl starch (HES) compared with the group allocated to Ringer's acetate.⁸ Consequently, a sensitivity analysis with removal of the patients allocated to HES (n=398) will be conducted.

Analyses will be conducted using SAS V.9.4 or newer (SAS Institute, Cary, North Carolina, USA). All statistical tests will be two-tailed. In general, p values <0.05 will be considered statistically significant and 95% CIs will be presented where appropriate.

Study size and power

The combined database will contain ∼1500 events of death within 90 days.

The SUP-ICU cohort, and the CLASSIC, TRISS and 6S trial databases are static databases and contain 1139 events of death within 90 days. The AID-ICU cohort database will be closed before data are used for the present study, and subsequently be a static database containing ∼380 events of death within 90 days (number of included patients: 1266, expected 90-day mortality: 30%). As a rule of thumb, logistic regression and Cox models should use at least 10 outcome events per predictor to ensure valid predictive modelling.^{37 38} As 22 candidate variables are evaluated, the combined database will contain enough events for the model to have sufficient power.

Phase 2: external validation

In the second phase of the project, the model will be externally validated in a fully independent sample.²¹ For the external validation phase, we will use the full cohort of the SUP-ICU trial²⁴ after its completion.

Study design

A preplanned analysis of data from the SUP-ICU trial.²⁴

Study participants

All patients included in the SUP-ICU trial after the completion of the trial and 90-day follow-up²⁴ (n≈3350; currently recruiting in 30 university and non-university ICUs in five countries with more to be added). The SUP-ICU trial includes adult (≥18 years) acutely ill patients admitted to the ICU. Enrolment started on 4 January 2016 and is expected to conclude by the end of 2017, with follow-up during the first half of 2018. Full inclusion and exclusion criteria are listed in online supplementary file 2.

Data sources

The SUP-ICU trial database will be used after completion of the trial and validation of the database, and will at this time be a static database.

Statistics

Baseline and clinical characteristics for all patients in the SUP-ICU trial²⁴ will be stratified by 90-day mortality and presented as medians with IQRs for continuous data, and numbers (%) for categorical data.²⁵ Differences will be assessed by Mann-Whitney U test and χ² test as appropriate.

Missing data in the SUP-ICU trial database for the relevant variables will be handled using multiple imputations, as recently recommended.³⁹ Multiple imputations with 10 imputed data sets and inclusion of the outcome measure and the respective predictor variables included in each score will be performed.^{27 28} Patients with missing outcome data (90-day mortality) will be excluded from the external validation.²⁷

For the external validation, the same measures of overall performance, discrimination and calibration will be used to evaluate the score as in phase 1 (above). Additionally, the performance of the SMS-ICU will be compared with that of SAPS II⁷ and the initial SOFA score¹⁷ in the external validation sample. For SAPS II, overall performance, discrimination and calibration will be assessed. For the initial SOFA score, only discrimination will be assessed as the SOFA score includes no prediction model.¹⁷ Discrimination of SMS-ICU, SAPS II and SOFA (AUCs) will be compared using the method described by DeLong et al.⁴⁰

Sensitivity analysis

As we do not yet know if 90-day mortality will differ between the two groups in the SUP-ICU trial, the external validation will be conducted using the full study population. Two preplanned sensitivity analyses will be conducted by repeating the analyses in either group.

Analyses will be conducted using SAS V.9.4 or newer (SAS Institute, Cary, North Carolina, USA). All statistical tests will be two-tailed. p values <0.05 will be considered statistically significant and 95% CIs will be presented where appropriate.

Study size and power

Assuming a 25% 90-day mortality rate in the SUP-ICU trial²⁴ there will be ∼840 deaths during follow-up. Generally, at least 100 events and 100 non-events are recommended for external validation studies.^{21 41} The external validation sample is comparable in size with the derivation sample and is thus of sufficient size to evaluate the performance of the score.

Strengths and limitations

Strengths of our study include publication of the protocol including a statistical analysis plan prior to development and validation of the score. Publication of protocols of observational studies, including studies developing and reporting prediction models is currently not mandatory or common practice. Publication of protocols is important, as it increases transparency and decreases the risk of bias, data-driven analyses and post hoc changes in design.⁴² Second, both the development and external validation phases will use contemporary data from international high-quality studies/trials with prospective data collection. This increases data quality and geographical generalisability. Third, our score will use 90-day mortality as the outcome measure. Existing scores generally use ICU or in-hospital mortality as outcome measures; however, score performance is affected by the choice of outcome^{43 44} and longer fixed-time outcome measures are currently recommended.⁴⁵

Our study has limitations too. First, the selection of candidate variables is limited by the availability of these variables across the five original studies used in the development phase. These variables, however, have previously been identified as predictors of mortality and are used in existing mortality prediction scores.⁷ Second, the inclusion and exclusion criteria in the original studies differ slightly which may limit generalisability of the score. However, we believe that the use of five contemporary high-quality studies/trials including less, moderately and severely ill ICU patients (SUP-ICU and AID-ICU inception cohort studies) and more severely ill patients (6S, TRISS and CLASSIC trials) ensures that the score will perform adequately across a large spectrum of illness severities in the ICU. Third, missing data may affect the development and validation of the score, but this is handled using recommended, contemporary methods (multiple imputation).²⁷ Finally, the majority of patients were included in Europe and thus geographical generalisability may be limited outside Europe.

Ethics and dissemination

Ethics and approvals

The 6S, TRISS, CLASSIC and SUP-ICU trials, and the SUP-ICU and AID-ICU inception cohort studies have all been approved by the relevant institutions. Full details regarding ethics and approvals can be found in the original papers/protocols.^{8–10 22–24} No further ethics approvals for the present study are necessary due to the non-interventional (observational) design.

The development of the score will in no way affect patient safety or subject participants to any additional or increased risks.

All original study databases have been approved and are maintained according to Danish laws. Anonymised data sets containing only the relevant variables (table 1) will be extracted from the original study databases and used for the present study, and the deposition of these databases does not require further approvals.

Dissemination

Results will be reported in compliance with the TRIPOD statement¹ and published in an international peer-reviewed journal and presented at relevant academic conferences, regardless of the findings. If the score performs satisfactory during the validations, we will develop an app, a web calculator and/or a pocket guide to facilitate easy calculation of the score at the bedside and encourage its use in ICUs.

Data will be available on request to the corresponding author. If required by the publishing journals, data will be provided with the submitted manuscripts.