Abstract
Background
The Oxford Acute Severity of Illness Score (OASIS) has shown fair prognosis predictive value in critically ill patients, but its predictive value has not been assessed in septic patients.
Objective
The aim of this study was to evaluate the performance of the OASIS for the assessment of mortality in septic patients, especially when compared with the Sepsis-related Organ Failure Assessment (SOFA) score.
Methods
A retrospective cohort study was conducted using data from a public database and septic patients were identified using the Sepsis-3 criteria. The primary outcome was hospital mortality. Data were mainly analyzed using multivariable logistic regression and receiver operating characteristic (ROC) curves. Sensitive analyses were performed in patients with an ICD-9-CM code for sepsis and ROC curves analyses were also conducted in septic patients stratified by the Simplified Acute Physiology Score (SAPS) II as subgroup analyses.
Results
A total of 10,305 septic patients were included. The OASIS was found to be significantly associated with hospital mortality (odds ratio 1.07 per one-point increase, 95% confidence interval [1.06–1.08]), while ROC curves analyses showed the discriminatory power of the OASIS for hospital mortality was statistically significantly lower than that of the SOFA score (area under the ROC curve: 0.652 vs 0.682, p < 0.001). Results of sensitive analyses were consistent, but the significant difference existed only when the SAPS II was higher than 50 according to results of the subgroup analyses.
Conclusions
The OASIS might serve as an initial predictor of clinical outcomes for septic patients, but one should be circumspect when it is applied to severer patients.
Keywords: Sepsis, Severity of illness index, Critical care outcomes, Mortality
Introduction
Scoring systems for outcome prediction have been developed in intensive care medicine for a long time, and some of them have been widely used in the intensive care unit (ICU) (Rapsang & Shyam, 2014). For patients with sepsis, the Sepsis-related Organ Failure Assessment (SOFA) score, which was initially designed to evaluate the severity of organ dysfunction in patients who were critically ill from sepsis (Vincent et al., 1998), has been proposed as a tool to facilitate the identification of patients at risk of dying from sepsis (Singer et al., 2016). The SOFA system uses a few measurements of major organ function to calculate a severity score, which contains some laboratory results such as platelets and bilirubin. Other scoring systems which consisted of more complex parameters such as the Simplified Acute Physiology Score II (SAPS II) (Le Gall, Lemeshow & Saulnier, 1993) were also widely used in ICU practice. However, as we know, clinicians usually prefer an instrument that is not laboratory-dependent and is easy to use. The quick SOFA (qSOFA) score, which consists of only three parameters and can be easily measured at the bedside, has also been proposed as a tool to help identify patients with early sepsis outside of the ICU (Seymour et al., 2016), but it was reported to have poor accuracy for predicting 28-day mortality in critically ill septic patients (Hwang et al., 2018). Johnson, Kramer & Clifford (2013) developed a new reduced severity of illness score using machine-learning algorithms, the Oxford Acute Severity of Illness Score (OASIS), which contained 10 parameters without any laboratory tests and had discrimination and calibration equivalent to more complex existing models. Given that the predictive value of the OASIS was only validated in mixed ICU patient populations, its performance in septic patients remains unknown. In this study, we evaluated the association of the OASIS with outcomes of septic patients in ICU, and examined its predictive value mainly by comparison with the SOFA score.
Methods
Database
The retrospective cohort study was conducted using data from the Medical Information Mart for Intensive Care III (MIMIC-III) database (v1.4). MIMIC-III is a publicly accessible critical care database which consists of de-identified health-related data about over 40,000 patients stayed in the ICU of Beth Israel Deaconess Medical Center between 2001 and 2012 (Johnson et al., 2016). The access to the database has been approved by the institutional review boards of both Beth Israel Deaconess Medical Center and Massachusetts Institute of Technology Affiliates after completing the CITI (Collaborative Institutional Training Initiative) “Data or Specimens Only Research” course (Qingui Chen (ID: 6533812), Record ID: 24321991).
Patients
All adult patients (age ≥ 18 years old) in the database with suspected infection and a SOFA score not less than two points within 24 h after ICU admission were screened for purposes of inclusion, but only those of first hospital admission were considered enrolled. Suspected infection was identified using the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes (See appendix 1 of the report from Angus et al. (2001)). To exclude records of organ donors or potential typographical errors, patients whose length of ICU stay were less than 24 h or whose length of hospital stay were less than length of ICU stay were excluded. Length of ICU stay was determined only by the first ICU stay. No informed consent was required on the de-identified patients.
Data extraction
We extracted data from the database using Transact-SQL and codes from the MIMIC Code Repository (https://github.com/MIT-LCP/mimic-code) (Johnson et al., 2018). Variables extracted included age, sex, admission type, ethnicity, mechanical ventilation on first day, renal replacement therapy on first day, comorbidities (Steiner, Elixhauser & Schnaier, 2002), and parameters of the OASIS (Table S1), the SOFA score, and the SAPS II. For parameters of these three scoring systems, only data within 24 h after ICU admission were extracted and missing components for calculation were treated as normal (usually zero). Length of ICU stay, length of hospital stay, and the Elixhauser Comorbidity Index (SID30) (Thompson et al., 2015) were also calculated. Since the database had date of birth of patients who are older than 89 years old shifted to exactly 300 years before admission to obscure their age, we corrected them (age—300+ 89) before analyses.
Outcomes
The primary outcome was hospital mortality, and the secondary outcomes were ICU mortality and 28-day mortality after ICU admission. The length of ICU stay and length of hospital stay were calculated only for statistical description. ICU mortality and length of ICU stay were determined by the first ICU stay only.
Statistical analysis
Results were presented as median and 25th–75th percentiles for continuous variables and numbers and percentages for categorical variables unless otherwise stated. Continuous and categorical variables were analyzed using Mann–Whitney U and Chi-square tests, respectively, but Fisher’s exact tests were used instead of Chi-squared tests when the expected values in any of the cells of a contingency table are below 10. The associations of OASIS with ICU outcomes were evaluated using multivariable logistic regression or Cox regression analyses. Univariable logistic regression analyses were performed before to examine the determinants of the primary outcome and variables with a p-value less than 0.2 were considered to be included in the multivariable analyses. To examine the linearity assumption of logistic regression, the generalized additive model was used to plot the possibly non-linear relation. Schoenfeld residual plots were also employed to examine the proportional hazard assumption of Cox regression. Discriminatory power was determined by comparing the area under the receiver operating characteristic (ROC) curve (AUC) for each score individually using the method of DeLong, DeLong & Clarke-Pearson (1988). Youden’s index was calculated to determine the best threshold of each scoring system and then patients were grouped by the threshold and Kaplan–Meier curves were drawn to visualize their survival. To test the robustness of the results, we performed sensitive analyses in septic patients identified by ICD-9-CM code for sepsis. To examine potential interaction modifiers of the association between OASIS and the primary outcome, logistic regression analyses across different subgroups were performed. For a variable with a p-value for interaction less than 0.05, ROC curve analysis was conducted stratified by the variable after professional judgement. Statistical tests were two-sided when the option was available. A p-value less than 0.05 was considered to indicate statistical significance, but for multiple comparisons of the AUC of the three scoring systems, Bonferroni correction was conducted by adjusting the critical p-value as 0.05/3. Empower (R) (www.empowerstats.com; X&Y solutions, Inc., Boston, MA, USA) and R software, version 3.4.3 (R Core Team, 2016) were used for all statistical analyses.
Results
Characteristics and clinical outcomes of the patients
A total of 10,305 patients with sepsis were included finally with a median OASIS of 34 (25th–75th percentiles 28–40). The median age of the subjects was 69 years (25th–75th percentiles 56–80 years) and 5,425 of the 10,305 cases (52.64%) were male. Among them, 5,196 (50.42%) patients required mechanical ventilation on first day and 514 (4.99%) patients required renal replacement therapy on first day. The five most common comorbidities were hypertension (52.95%), fluid and electrolyte disorders (41.49%), congestive heart failure (33.91%), cardiac arrhythmias (33.51%), and deficiency anemia (23.19%). Other characteristics of the patients are presented in Table 1. With regard to their clinical outcomes, the hospital mortality was 12.31% with 1,269 non-survivors and 9,036 survivors, the ICU mortality was 5.84% and the 28-day mortality was 14.91%. The length of ICU stay and hospital stay was 3.48 (25th–75th percentiles 1.98–7.62) and 11.67 (25th–75th percentiles 7.00–19.94) days, respectively.
Table 1. Characteristics and comparison between survivors and non-survivors of all patients.
| Variable | All patients (n = 10,305) | Survivors (n = 9,036) | Non-survivors (n = 1,269) | p |
|---|---|---|---|---|
| Age (years) | 69 (56–80) | 68 (55–80) | 74 (61–82) | <0.001 |
| Male | 5,425 (52.64%) | 4,742 (52.48%) | 683 (53.82%) | 0.370 |
| Admission type | <0.001 | |||
| Urgent | 334 (3.24%) | 290 (3.21%) | 44 (3.47%) | |
| Emergency | 9,006 (87.39%) | 7,853 (86.91%) | 1,153 (90.86%) | |
| Elective | 965 (9.36%) | 893 (9.88%) | 72 (5.67%) | |
| Ethnicity | <0.001 | |||
| White | 7,482 (72.61%) | 6,592 (72.95%) | 890 (70.13%) | |
| Black | 847 (8.22%) | 760 (8.41%) | 87 (6.86%) | |
| Asian | 246 (2.39%) | 225 (2.49%) | 21 (1.65%) | |
| Hispanic/Latino | 302 (2.93%) | 277 (3.07%) | 25 (1.97%) | |
| Other | 1,428 (13.86%) | 1,182 (13.08%) | 246 (19.39%) | |
| ICU mortality | 602 (5.84%) | 0 (0.00%) | 602 (47.44%) | <0.001 |
| 28-day mortality | 1,536 (14.91%) | 413 (4.57%) | 1,123 (88.49%) | <0.001 |
| Length of ICU stay (days) | 3.48 (1.98–7.62) | 3.30 (1.94–7.07) | 5.00 (2.43–10.73) | <0.001 |
| Length of hospital stay (days) | 11.67 (7.00–19.94) | 11.49 (6.98–19.61) | 12.90 (7.40–23.20) | <0.001 |
| OASIS on admission | 34 (28–40) | 33 (28–39) | 38 (32–44) | <0.001 |
| SAPS II on admission | 39 (31–48) | 38 (30–46) | 50 (40–59) | <0.001 |
| SOFA score on admission | 5 (3–7) | 5 (3–7) | 7 (4–10) | <0.001 |
| Elixhauser Comorbidity Index (SID30) | 12 (4–23) | 12 (3–21) | 20 (11–29) | <0.001 |
| Sepsis (based on ICD-9) | 1,845 (17.90%) | 1,413 (15.64%) | 432 (34.04%) | <0.001 |
| Mechanical ventilation on first day | 5,196 (50.42%) | 4,510 (49.91%) | 686 (54.06%) | 0.006 |
| Renal replacement therapy on first day | 514 (4.99%) | 423 (4.68%) | 91 (7.17%) | <0.001 |
| Comorbidities | ||||
| Congestive heart failure | 3,494 (33.91%) | 2,989 (33.08%) | 505 (39.80%) | <0.001 |
| Cardiac arrhythmias | 3,453 (33.51%) | 2,948 (32.63%) | 505 (39.80%) | <0.001 |
| Valvular disease | 1,507 (14.62%) | 1,331 (14.73%) | 176 (13.87%) | 0.416 |
| Pulmonary circulation disorder | 815 (7.91%) | 705 (7.80%) | 110 (8.67%) | 0.284 |
| Peripheral vascular disorder | 1,188 (11.53%) | 1,031 (11.41%) | 157 (12.37%) | 0.315 |
| Hypertension | 5,456 (52.95%) | 4,868 (53.87%) | 588 (46.34%) | <0.001 |
| Paralysis | 451 (4.38%) | 413 (4.57%) | 38 (2.99%) | 0.010 |
| Other neurological disease | 1,411 (13.69%) | 1,250 (13.83%) | 161 (12.69%) | 0.266 |
| Chronic pulmonary disease | 2,199 (21.34%) | 1,941 (21.48%) | 258 (20.33%) | 0.349 |
| Uncomplicated diabetes | 2,142 (20.79%) | 1,918 (21.23%) | 224 (17.65%) | 0.003 |
| Complicated diabetes | 761 (7.38%) | 679 (7.51%) | 82 (6.46%) | 0.179 |
| Hypothyroidism | 1,107 (10.74%) | 984 (10.89%) | 123 (9.69%) | 0.197 |
| Renal failure | 1,852 (17.97%) | 1,597 (17.67%) | 255 (20.09%) | 0.035 |
| Liver disease | 1,023 (9.93%) | 791 (8.75%) | 232 (18.28%) | <0.001 |
| Peptic ulcer | 17 (0.16%) | 15 (0.17%) | 2 (0.16%) | 1.000* |
| AIDS | 151 (1.47%) | 131 (1.45%) | 20 (1.58%) | 0.726 |
| Lymphoma | 264 (2.56%) | 195 (2.16%) | 69 (5.44%) | <0.001 |
| Metastatic cancer | 665 (6.45%) | 485 (5.37%) | 180 (14.18%) | <0.001 |
| Solid tumor | 532 (5.16%) | 458 (5.07%) | 74 (5.83%) | 0.250 |
| Rheumatoid arthritis | 346 (3.36%) | 308 (3.41%) | 38 (2.99%) | 0.443 |
| Coagulopathy | 1,798 (17.45%) | 1,417 (15.68%) | 381 (30.02%) | <0.001 |
| Obesity | 595 (5.77%) | 549 (6.08%) | 46 (3.62%) | <0.001 |
| Weight loss | 708 (6.87%) | 584 (6.46%) | 124 (9.77%) | <0.001 |
| Fluid and electrolyte disorders | 4,276 (41.49%) | 3,602 (39.86%) | 674 (53.11%) | <0.001 |
| Blood loss anemia | 276 (2.68%) | 242 (2.68%) | 34 (2.68%) | 0.998 |
| Deficiency anemia | 2,390 (23.19%) | 2,163 (23.94%) | 227 (17.89%) | <0.001 |
| Alcohol abuse | 782 (7.59%) | 688 (7.61%) | 94 (7.41%) | 0.795 |
| Drug abuse | 353 (3.43%) | 330 (3.65%) | 23 (1.81%) | <0.001 |
| Psychoses | 429 (4.16%) | 404 (4.47%) | 25 (1.97%) | <0.001 |
| Depression | 849 (8.24%) | 778 (8.61%) | 71 (5.59%) | <0.001 |
Notes:
Patients were grouped as survivors and non-survivors determined by hospital mortality status. Data are expressed as median (25th–75th percentiles) or n (%) unless otherwise stated. Mann–Whitney U and Chi-square (or Fisher’s exact) tests were used to analyze continuous and categorical variables, respectively. Statistical significance (p < 0.05) is shown in bold. The asterisk indicated that the Fisher’s exact test was used instead of the Chi-squared test.
ICU, intensive care unit; OASIS, Oxford Acute Severity of Illness Score; SAPS II, Simplified Acute Physiology Score II; SOFA, Sepsis-related Organ Failure Assessment score; ICD, International Classification of Diseases, Ninth Revision; AIDS, Acquired Immune Deficiency Syndrome.
Association of the OASIS with clinical outcomes of septic patients
As shown in Table 1, non-survivors had significantly higher OASIS on ICU admission than survivors (38 vs 33, p < 0.001). The distributions of the OASIS and the SOFA score with corresponding hospital mortality are presented in Fig. 1. As each score increased, the hospital mortality of the patients approximately increased accordingly. According to the result of univariable logistic regression analyses for variables associated with hospital mortality which was presented in Table S2, age, admission type, ethnicity, mechanical ventilation on first day, renal replacement therapy on first day, and the Elixhauser Comorbidity Index (SID30) were included in the multivariable regression analyses. Results of curve fitting of the relationship between hospital mortality and several continuous variables were presented in Fig. S1, which indicated that the independent variables were linearly related to the log odds. Schoenfeld residual plots shown in Fig. S2 supported the proportional hazard assumption in Cox models. After adjusting the variables above, the OASIS was significantly associated with hospital mortality (Odds ratio (OR) 1.07 per one-point increase, 95% confidence interval (CI) [1.06–1.08], p < 0.0001), ICU mortality (OR 1.07 per one-point increase, 95% CI [1.06–1.08], p < 0.001), and 28-day mortality (hazard ratio (HR) 1.06 per one-point increase, 95% CI [1.05–1.06], p < 0.001) (Table 2). Results of analyses of the SOFA score and the SAPS II are presented in Tables S3 and S4, respectively.
Figure 1. Hospital mortality by different severity scores on ICU admission among patients with sepsis.
(A) Hospital mortality by OASIS on ICU admission among patients with sepsis; (B) Hospital mortality by SOFA score on ICU admission among patients with sepsis. Abbreviations: OASIS, Oxford Acute Severity of Illness Score; SOFA, Sepsis-related Organ Failure Assessment score; ICU, intensive care unit.
Table 2. Association of OASIS with hospital mortality, ICU mortality, and 28-day mortality.
| Outcomes | OR/HR | 95% CI | p |
|---|---|---|---|
| Hospital mortality | |||
| Non-adjusted | 1.07 | [1.06–1.08] | <0.001 |
| Adjusted | 1.07 | [1.06–1.08] | <0.001 |
| ICU mortality | |||
| Non-adjusted | 1.09 | [1.08–1.10] | <0.001 |
| Adjusted | 1.07 | [1.06–1.08] | <0.001 |
| 28-day mortality | |||
| Non-adjusted | 1.06 | [1.05–1.06] | <0.001 |
| Adjusted | 1.06 | [1.05–1.06] | <0.001 |
Notes:
Associations of OASIS with hospital mortality and ICU mortality were analyzed using logistic regression models. Association of OASIS with 28-day mortality was analyzed using Cox regression models. Model was adjusted for age, admission type, ethnicity, mechanical ventilation on first day, renal replacement therapy on first day, and the Elixhauser Comorbidity Index (SID30). Statistical significance (p < 0.05) is shown in bold.
OASIS, the Oxford Acute Severity of Illness Score; ICU, intensive care unit; OR, odds ratio; HR, hazard ratio; CI, confidence interval.
Discriminatory power of the OASIS in septic patients
As shown in Fig. 2, the AUC of the OASIS for predicting hospital mortality was 0.652 (95% CI [0.636–0.668]), which was significantly lower than that of the SOFA score (AUC 0.682, 95% CI [0.666–0.699], p < 0.001). The best threshold of OASIS was 34.5 with a specificity of 55.80%, a sensitivity of 64.93%, a positive likelihood ratio of 1.47 and a negative likelihood ratio of 0.63. However, the discriminatory power of both scores were all significantly lower than that of the SAPS II (AUC 0.739, 95% CI [0.725–0.753]). Results of comparison of the three scores were similar when they were used to predict ICU mortality (Fig. 2). According to the Kaplan–Meier curves presented in Figs. S3, S4 and S5, patients with higher severity scores had shorter survival time regardless of which scoring system was used.
Figure 2. ROC curves assessing discrimination of different severity scores on ICU admission for predicting hospital mortality and ICU mortality.
(A) ROC curves for hospital mortality by SAPS II, SOFA score and OASIS; (B) ROC curves for ICU mortality by SAPS II, SOFA score and OASIS. Abbreviations: ROC, receiver operating characteristic; ICU, intensive care unit; SAPS II, Simplified Acute Physiology Score II; SOFA, Sepsis-related Organ Failure Assessment score; OASIS, the Oxford Acute Severity of Illness Score; AUC, area under the ROC curve; CI, confidence interval.
Sensitive analyses
Results of the sensitive analyses were consistent. The OASIS was still significantly associated with hospital mortality (OR = 1.03, 95% CI [1.01–1.05], p < 0.001) and 28-day mortality (HR = 1.03, 95% CI [1.01–1.04], p < 0.001) when patients with an ICD-9-CM code for sepsis were included only, but no significant association was found between the OASIS and ICU mortality (OR = 1.02, 95% CI [1.00–1.04], p = 0.064) (See Table 3). Association of the other two scores with outcomes were presented in Tables S5 and S6, respectively. In the sensitive analyses, the OASIS still had the lowest discriminatory power for predicting hospital mortality (AUC 0.586, 95% CI [0.555–0.616]) and ICU mortality (AUC 0.608, 95% CI [0.569–0.646]) when compared with the SOFA score and the SAPS II (Fig. 3).
Table 3. Sensitive analysis of association of OASIS with hospital mortality, ICU mortality, and 28-day mortality.
| Outcomes | OR/HR | 95% CI | p |
|---|---|---|---|
| Hospital mortality | |||
| Non-adjusted | 1.04 | [1.02–1.05] | <0.001 |
| Adjusted | 1.03 | [1.01–1.05] | <0.001 |
| ICU mortality | |||
| Non-adjusted | 1.04 | [1.03–1.06] | <0.001 |
| Adjusted | 1.02 | [1.00–1.04] | 0.064 |
| 28-day mortality | |||
| Non-adjusted | 1.03 | [1.02–1.04] | <0.001 |
| Adjusted | 1.03 | [1.01–1.04] | <0.001 |
Notes:
Only patients diagnosed as sepsis according to ICD-9 codes were included into the sensitive analysis. Associations of OASIS with hospital mortality and ICU mortality were analyzed using logistic regression models. Association of OASIS with 28-day mortality was analyzed using Cox regression models. Model was adjusted for age, admission type, ethnicity, mechanical ventilation on first day, renal replacement therapy on first day, and the Elixhauser Comorbidity Index (SID30). Statistical significance (p < 0.05) is shown in bold.
OASIS, Oxford Acute Severity of Illness Score; ICU, intensive care unit; OR, odds ratio; HR, hazard ratio; CI, confidence interval.
Figure 3. Sensitive analysis of ROC curves assessing discrimination of different severity scores on ICU admission for predicting hospital mortality and ICU mortality.
Only patients diagnosed as sepsis according to ICD-9 codes were included into the sensitive analysis. (A) ROC curves for hospital mortality by SAPS II, SOFA score and OASIS; (B) ROC curves for ICU mortality by SAPS II, SOFA score and OASIS. Abbreviations: ROC, receiver operating characteristic; ICU, intensive care unit; SAPS II, Simplified Acute Physiology Score II; SOFA, Sepsis-related Organ Failure Assessment score; OASIS, the Oxford Acute Severity of Illness Score; AUC, area under the ROC curve; CI, confidence interval.
Subgroup analyses
Associations of the OASIS with hospital mortality were analyzed using logistic regression models across different subgroups, and the result presented in Table S7 found several variables had a p-value of interaction less than 0.05. After professional judgement, only the SAPS II was chosen as a potential interaction modifier and examined in the subgroup analysis further. As shown in Fig. 4 and Table S8, there was no statistically significant difference in the discriminatory power of the OASIS and the SOFA score for predicting hospital mortality when a SAPS II was lower than 50. In addition, the SOFA score had significantly higher AUC than that of the OASIS for predicting ICU mortality only when a SAPS II was higher than 35.
Figure 4. Comparisons of the discriminatory ability of OASIS and SOFA on ICU admission for predicting hospital mortality and ICU mortality stratified by SAPS II.
(A) AUCs for hospital mortality by OASIS and SOFA score across different SAPS II categories; (B) AUCs for ICU mortality by OASIS and SOFA score across different SAPS II categories. Abbreviations: OASIS, the Oxford Acute Severity of Illness Score; SOFA, Sepsis-related Organ Failure Assessment score; ICU, intensive care unit; SAPS II, Simplified Acute Physiology Score II; AUC, area under the ROC curve; ROC, receiver operating characteristic.
Discussion
In the study, we evaluated the performance of the OASIS for predicting clinical outcomes of patients with sepsis in ICU. Results of our study indicated that the OASIS on ICU admission was significantly associated with short-term clinical outcomes of patients with sepsis, but its discriminatory abilities of hospital mortality and ICU mortality were statistically significantly lower than that of SAPS II. However, an interesting finding was that there was no statistically significant difference between the mortality discriminatory power of the OASIS and that of the SOFA score when the septic patients were not that severe (evaluated by their SAPS II). Even when the patients were more severe, it should be noticed that the absolute AUC difference between the OASIS and the SOFA score was rather small and there was also an overlap between the 95% CIs of the two scores, so it should be cautious when interpreting the clinical significance of the results. Nevertheless, as far as we know, this is the first study that assesses the prognostic predictive value of the OASIS in patients with sepsis in ICU, which might help clinicians to make good use of this newly developed scoring system.
Since the prediction of ICU outcomes might contribute to clinical research and individual patient management (Power & Harrison, 2014), many scoring systems for ICU outcome prediction have been developed and some of them have been routinely used. In terms of sepsis, the most famous scoring system is the SOFA score, an indicator of identifying organ dysfunction and part of the diagnosis criteria of sepsis-3 (Singer et al., 2016). The association of the SOFA score with ICU outcomes has been assessed either in mixed ICU patients or septic patients (Ferreira et al., 2001; Jones, Trzeciak & Kline, 2009; Lie et al., 2018), and it has been reported to be a valuable predictor of ICU outcomes. As shown in Table S3 and Fig. 2, results of our study about the SOFA score were consistent. Unlike the SOFA score, the SAPS II is a widely used disease severity scoring system aimed at mixed ICU patients, and it has been reported to perform better for predicting hospital mortality and 90-day mortality than the SOFA score (Granholm et al., 2016). Similar results were also observed in our study that the SAPS II had the best discriminatory power for hospital mortality and ICU mortality among the three scoring systems analyzed. It is not surprising to find the results, since the SAPS II scoring system has more complicated parameters than the other two systems. However, the complexity of the SAPS II scoring system might be a challenge in daily clinical assessment, which impelled us to investigate the performance of the OASIS.
The OASIS system only consists of 10 easily accessible parameters including length of hospital stay prior ICU admission, age, Glasgow Coma Score (Teasdale & Jennett, 1974), heart rate, mean arterial pressure, respiratory rate, temperature, urine output, ventilation, and elective surgery. In fact, the OASIS system was developed from the popular severity scoring systems APACHE IV (Acute Physiology, Age, and Chronic Health Evaluation VI) with the aim of reducing the number of parameters without losing predictive accuracy (Johnson, Kramer & Clifford, 2013). The goal has been achieved to some extent, however, as far as we know, the discriminatory ability of the OASIS for outcomes of septic patients has not been examined especially when compared with the SOFA score, which has similar numbers of items. Before the study we assumed that the OASIS had poorer prognostic predictive value that that of the SOFA score, for the reason that the OASIS does not contain any laboratory parameters. However, surprisingly, we found that the performance of the OASIS was not different from that of the SOFA score when the septic patients were of mild conditions. Although it is impractical to choose and use the OASIS after the SAPS II of a septic patient has been calculated to determine the severity of the patient, we believe that the OASIS is still a promising tool. Clearly, the simple components of the OASIS make it possible to be calculated automatically, which may provide constructive information especially when few laboratory examinations have been done. In addition, in our previous study (Chen et al., 2018) we have found the OASIS might serve as a supplement to the qSOFA score and help to identify more septic patients. Therefore, we reason that perhaps it might also serve as a supplement to the SOFA score to predict outcomes of septic patients. However, further researches are needed to explore the possibilities.
Another issue needed to be discussed is the criteria we used to identify sepsis. In the study, we initially identified septic patients from the MIMIC-3 database using Sepsis-3 criteria. However, this method assumed a causation from infection to organ dysfunction (the SOFA score ≥2) while in a lot of cases there might be only associations, which might lead to an overestimation of case numbers and an underestimation of mortality (Fleischmann-Struzek et al., 2018). Thus, we then performed sensitive analyses using the ICD-9-CM codes to identify sepsis. Results of the sensitive analyses were consistent, indicating the robustness of our conclusions. However, it should be noticed that the two criteria might represent two distinct populations of sepsis, since the Sepsis-3 criteria was developed in 2016. In our study, the patients identified by the ICD-9-CM codes had higher hospital mortality rate (23.41%) with a median SAPS II of 49.5 and a median SOFA score of 6, so it is not strange to find a lower AUC of the OASIS for hospital mortality in the sensitive analyses since they were more severe.
Several limitations of the study should be noticed. First of all, given the observational nature of our study, selection bias was inevitable although we adjusted a few potential confounders. Second, only the discriminatory power of the OASIS was evaluated, we did not assess its calibration. Third, several variables were not considered. The actual admission years of the patients in the database had been shifted to protect patient confidentiality, so we did not include it in the regression models. Since the database included patients from 2001 to 2012, there might be some differences in outcomes of the patients treated in the early 2000s and later, but we did not take therapy as a covariate into consideration. Last but not least, since all the data in the study was treated as a single dataset, the performance of the OASIS was inherently optimistic.
Conclusions
To sum up, the retrospective observational study validated significant associations of the OASIS on ICU admission with short-term outcomes of septic patients, and found that although the discriminatory power of the OASIS for hospital mortality was statistically significantly lower than that of the SOFA score, the significant difference existed only when a SAPS II was higher than 50, which suggesting that the OASIS might serve as an initial predictor of clinical outcomes for septic patients, but one should be circumspect when it is applied to more severe patients.
Supplemental Information
Abbreviations: OR, odds ratio; CI, confidence interval; OASIS, Oxford acute severity of illness score; SAPS II, simplified acute physiology score II; SOFA, Sepsis-related organ failure assessment score.
Notes: Associations of SOFA score with hospital mortality and ICU mortality were analyzed using logistic regression models. Association of SOFA score with 28-day mortality was analyzed using Cox regression models. Model was adjusted for age, admission type, ethnicity, mechanical ventilation on first day, renal replacement therapy on first day, and the Elixhauser Comorbidity Index (SID30). Abbreviations: SOFA, Sepsis-related organ failure assessment score; ICU, intensive care unit; OR, odds ratio; HR, hazard ratio; CI, confidence interval.
Notes: Associations of SAPS II with hospital mortality and ICU mortality were analyzed using logistic regression models. Association of SAPS II score with 28-day mortality was analyzed using Cox regression models. Model was adjusted for age, admission type, ethnicity, mechanical ventilation on first day, renal replacement therapy on first day, and the Elixhauser Comorbidity Index (SID30). Abbreviations: SAPS II, simplified acute physiology score II; ICU, intensive care unit; OR, odds ratio; HR, hazard ratio; CI, confidence interval.
Notes: Only patients diagnosed as sepsis according to ICD-9 codes were included into the sensitive analysis. Associations of SOAF score with hospital mortality and ICU mortality were analyzed using logistic regression models. Association of SOFA score with 28-day mortality was analyzed using Cox regression models. Model was adjusted for age, admission type, ethnicity, mechanical ventilation on first day, renal replacement therapy on first day, and the Elixhauser Comorbidity Index (SID30). Abbreviations: SOFA, Sepsis-related organ failure assessment score; ICU, intensive care unit; OR, odds ratio; HR, hazard ratio; CI, confidence interval.
Notes: Only patients diagnosed as sepsis according to ICD-9 codes were included into the sensitive analysis. Associations of SAPS II with hospital mortality and ICU mortality were analyzed using logistic regression models. Association of SAPS II with 28-day mortality was analyzed using Cox regression models. Model was adjusted for age, admission type, ethnicity, mechanical ventilation on first day, renal replacement therapy on first day, and the Elixhauser Comorbidity Index (SID30). Abbreviations: SAPS II, simplified acute physiology score II; ICU, intensive care unit; OR, odds ratio; HR, hazard ratio; CI, confidence interval.
Notes: Associations of OASIS with hospital mortality were analyzed using logistic regression models across different subgroups to examine potential interaction modifiers. Categories of continuous variables were presented as minimum to maximum. Abbreviations: OASIS, Oxford acute severity of illness score; OR, odds ratio; CI, confidence interval; SAPS II, simplified acute physiology score II; SOFA, Sepsis-related organ failure assessment score; ICD, International Classification of Diseases, Ninth Revision; AIDS, acquired immune deficiency syndrome.
Abbreviations: ROC, receiver operating characteristic; OASIS, Oxford acute severity of illness score; SOFA, Sepsis-related organ failure assessment score; ICU, intensive care unit; SAPS II, simplified acute physiology score II; AUC, area under the ROC curve; CI, confidence interval.
Abbreviations: OASIS, Oxford acute severity of illness score.
Abbreviations: OASIS, Oxford acute severity of illness score.
Abbreviations: OASIS, Oxford acute severity of illness score; ICU, intensive care unit.
Abbreviations: SOFA, Sepsis-related organ failure assessment score; ICU, intensive care unit.
Abbreviations: SAPS II, simplified acute physiology score II; ICU, intensive care unit.
The main codes used for extracting data are presented. More basic codes are available at https://github.com/MIT-LCP/mimic-code.
Acknowledgments
We thank Mr. Jun-rui Zhang from Guangzhou Institute of Standardization for his help in preparation of figures.
Funding Statement
This work was supported by the National Natural Science Foundation of China (grant number 81670066), the Major Science and Technology Planning Project of Guangdong Province, China (grant number 2016A020216009), and the Critical Care Research Funding of the Aesculap Academy (2017). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Additional Information and Declarations
Competing Interests
The authors declare that they have no competing interests.
Author Contributions
Qingui Chen performed the experiments, analyzed the data, contributed reagents/materials/analysis tools, authored or reviewed drafts of the paper, approved the final draft.
Lishan Zhang performed the experiments, analyzed the data, authored or reviewed drafts of the paper, approved the final draft.
Shanhui Ge prepared figures and/or tables, approved the final draft.
Wanmei He prepared figures and/or tables, approved the final draft.
Mian Zeng conceived and designed the experiments, authored or reviewed drafts of the paper, approved the final draft.
Human Ethics
The following information was supplied relating to ethical approvals (i.e., approving body and any reference numbers):
The access of the database has been approved by the institutional review boards of both Beth Israel Deaconess Medical Center and Massachusetts Institute of Technology Affiliates after completing the CITI (Collaborative Institutional Training Initiative) “Data or Specimens Only Research” course (Qingui Chen (ID: 6533812), Record ID: 24321991).
Data Availability
The following information was supplied regarding data availability:
The codes we created are available in the Supplemental File to help readers to replicate the analyses. More codes can be found at https://github.com/MIT-LCP/mimic-code. The raw data itself is from a third-party dataset available from MIMIC-III, a freely accessible critical care database. Reproduction of their data is not permitted according to the Data Use Agreement of the database but access can be requested here: https://mimic.physionet.org/gettingstarted/access.
References
- Angus et al. (2001).Angus DC, Linde-Zwirble WT, Lidicker J, Clermont G, Carcillo J, Pinsky MR. Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Critical Care Medicine. 2001;29(7):1303–1310. doi: 10.1097/00003246-200107000-00002. [DOI] [PubMed] [Google Scholar]
- Chen et al. (2018).Chen QG, Xie RJ, Chen YZ, Zeng M. Clinical value of Oxford acute severity of illness score in identifying quick sequential organ failure assessment-negative patients with sepsis. Zhonghua Jie He He Hu Xi Za Zhi. 2018;41(9):701–708. doi: 10.3760/cma.j.issn.1001-0939.2018.09.010. [DOI] [PubMed] [Google Scholar]
- DeLong, DeLong & Clarke-Pearson (1988).DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44(3):837–845. doi: 10.2307/2531595. [DOI] [PubMed] [Google Scholar]
- Ferreira et al. (2001).Ferreira FL, Bota DP, Bross A, Melot C, Vincent JL. Serial evaluation of the SOFA score to predict outcome in critically ill patients. JAMA. 2001;286(14):1754–1758. doi: 10.1001/jama.286.14.1754. [DOI] [PubMed] [Google Scholar]
- Fleischmann-Struzek et al. (2018).Fleischmann-Struzek C, Thomas-Ruddel DO, Schettler A, Schwarzkopf D, Stacke A, Seymour CW, Haas C, Dennler U, Reinhart K. Comparing the validity of different ICD coding abstraction strategies for sepsis case identification in German claims data. PLOS ONE. 2018;13(7):e198847. doi: 10.1371/journal.pone.0198847. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Granholm et al. (2016).Granholm A, Moller MH, Krag M, Perner A, Hjortrup PB. Predictive performance of the simplified acute physiology score (SAPS) II and the initial sequential organ failure assessment (SOFA) score in acutely Ill intensive care patients: post-hoc analyses of the SUP-ICU inception cohort study. PLOS ONE. 2016;11(12):e168948. doi: 10.1371/journal.pone.0168948. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hwang et al. (2018).Hwang SY, Jo IJ, Lee SU, Lee TR, Yoon H, Cha WC, Sim MS, Shin TG. Low accuracy of positive qSOFA criteria for predicting 28-day mortality in critically Ill septic patients during the early period after emergency department presentation. Annals of Emergency Medicine. 2018;71(1):1–9.e2. doi: 10.1016/j.annemergmed.2017.05.022. [DOI] [PubMed] [Google Scholar]
- Johnson, Kramer & Clifford (2013).Johnson AE, Kramer AA, Clifford GD. A new severity of illness scale using a subset of acute physiology and chronic health evaluation data elements shows comparable predictive accuracy. Critical Care Medicine. 2013;41(7):1711–1718. doi: 10.1097/ccm.0b013e31828a24fe. [DOI] [PubMed] [Google Scholar]
- Johnson et al. (2016).Johnson AE, Pollard TJ, Shen L, Lehman LW, Feng M, Ghassemi M, Moody B, Szolovits P, Celi LA, Mark RG. MIMIC-III, a freely accessible critical care database. Scientific Data. 2016;3:160035. doi: 10.1038/sdata.2016.35. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Johnson et al. (2018).Johnson AEW, Stone DJ, Celi LA, Pollard TJ. The MIMIC code repository: enabling reproducibility in critical care research. Journal of the American Medical Informatics Association. 2018;25(1):32–39. doi: 10.1093/jamia/ocx084. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Jones, Trzeciak & Kline (2009).Jones AE, Trzeciak S, Kline JA. The sequential organ failure assessment score for predicting outcome in patients with severe sepsis and evidence of hypoperfusion at the time of emergency department presentation. Critical Care Medicine. 2009;37(5):1649–1654. doi: 10.1097/ccm.0b013e31819def97. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Le Gall, Lemeshow & Saulnier (1993).Le Gall JR, Lemeshow S, Saulnier F. A new simplified acute physiology score (SAPS II) based on a European/North American multicenter study. JAMA: The Journal of the American Medical Association. 1993;270(24):2957–2963. doi: 10.1001/jama.270.24.2957. [DOI] [PubMed] [Google Scholar]
- Lie et al. (2018).Lie KC, Lau CY, Van Vinh CN, West TE, Limmathurotsakul D. Utility of SOFA score, management and outcomes of sepsis in Southeast Asia: a multinational multicenter prospective observational study. Journal of Intensive Care. 2018;6(1):9. doi: 10.1186/s40560-018-0279-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Power & Harrison (2014).Power GS, Harrison DA. Why try to predict ICU outcomes? Current Opinion in Critical Care. 2014;20(5):544–549. doi: 10.1097/mcc.0000000000000136. [DOI] [PubMed] [Google Scholar]
- R Core Team (2016).R Core Team . R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2016. [Google Scholar]
- Rapsang & Shyam (2014).Rapsang AG, Shyam DC. Scoring systems in the intensive care unit: a compendium. Indian Journal of Critical Care Medicine. 2014;18(4):220–228. doi: 10.4103/0972-5229.130573. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Seymour et al. (2016).Seymour CW, Liu VX, Iwashyna TJ, Brunkhorst FM, Rea TD, Scherag A, Rubenfeld G, Kahn JM, Shankar-Hari M, Singer M, Deutschman CS, Escobar GJ, Angus DC. Assessment of clinical criteria for sepsis: for the third international consensus definitions for sepsis and septic shock (Sepsis-3) JAMA. 2016;315(8):762–774. doi: 10.1001/jama.2016.0288. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Singer et al. (2016).Singer M, Deutschman CS, Seymour CW, Shankar-Hari M, Annane D, Bauer M, Bellomo R, Bernard GR, Chiche JD, Coopersmith CM, Hotchkiss RS, Levy MM, Marshall JC, Martin GS, Opal SM, Rubenfeld GD, Van der Poll T, Vincent JL, Angus DC. The third international consensus definitions for sepsis and septic shock (Sepsis-3) JAMA. 2016;315(8):801–810. doi: 10.1001/jama.2016.0287. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Steiner, Elixhauser & Schnaier (2002).Steiner C, Elixhauser A, Schnaier J. The healthcare cost and utilization project: an overview. Effective Clinical Practice. 2002;5(3):143–151. [PubMed] [Google Scholar]
- Teasdale & Jennett (1974).Teasdale G, Jennett B. Assessment of coma and impaired consciousness. A practical scale. The Lancet. 1974;2(7872):81–84. doi: 10.1016/s0140-6736(74)91639-0. [DOI] [PubMed] [Google Scholar]
- Thompson et al. (2015).Thompson NR, Fan Y, Dalton JE, Jehi L, Rosenbaum BP, Vadera S, Griffith SD. A new Elixhauser-based comorbidity summary measure to predict in-hospital mortality. Medical Care. 2015;53(4):374–379. doi: 10.1097/mlr.0000000000000326. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Vincent et al. (1998).Vincent JL, De Mendonca A, Cantraine F, Moreno R, Takala J, Suter PM, Sprung CL, Colardyn F, Blecher S. Use of the SOFA score to assess the incidence of organ dysfunction/failure in intensive care units: results of a multicenter, prospective study. Critical Care Medicine. 1998;26(11):1793–1800. doi: 10.1097/00003246-199811000-00016. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Abbreviations: OR, odds ratio; CI, confidence interval; OASIS, Oxford acute severity of illness score; SAPS II, simplified acute physiology score II; SOFA, Sepsis-related organ failure assessment score.
Notes: Associations of SOFA score with hospital mortality and ICU mortality were analyzed using logistic regression models. Association of SOFA score with 28-day mortality was analyzed using Cox regression models. Model was adjusted for age, admission type, ethnicity, mechanical ventilation on first day, renal replacement therapy on first day, and the Elixhauser Comorbidity Index (SID30). Abbreviations: SOFA, Sepsis-related organ failure assessment score; ICU, intensive care unit; OR, odds ratio; HR, hazard ratio; CI, confidence interval.
Notes: Associations of SAPS II with hospital mortality and ICU mortality were analyzed using logistic regression models. Association of SAPS II score with 28-day mortality was analyzed using Cox regression models. Model was adjusted for age, admission type, ethnicity, mechanical ventilation on first day, renal replacement therapy on first day, and the Elixhauser Comorbidity Index (SID30). Abbreviations: SAPS II, simplified acute physiology score II; ICU, intensive care unit; OR, odds ratio; HR, hazard ratio; CI, confidence interval.
Notes: Only patients diagnosed as sepsis according to ICD-9 codes were included into the sensitive analysis. Associations of SOAF score with hospital mortality and ICU mortality were analyzed using logistic regression models. Association of SOFA score with 28-day mortality was analyzed using Cox regression models. Model was adjusted for age, admission type, ethnicity, mechanical ventilation on first day, renal replacement therapy on first day, and the Elixhauser Comorbidity Index (SID30). Abbreviations: SOFA, Sepsis-related organ failure assessment score; ICU, intensive care unit; OR, odds ratio; HR, hazard ratio; CI, confidence interval.
Notes: Only patients diagnosed as sepsis according to ICD-9 codes were included into the sensitive analysis. Associations of SAPS II with hospital mortality and ICU mortality were analyzed using logistic regression models. Association of SAPS II with 28-day mortality was analyzed using Cox regression models. Model was adjusted for age, admission type, ethnicity, mechanical ventilation on first day, renal replacement therapy on first day, and the Elixhauser Comorbidity Index (SID30). Abbreviations: SAPS II, simplified acute physiology score II; ICU, intensive care unit; OR, odds ratio; HR, hazard ratio; CI, confidence interval.
Notes: Associations of OASIS with hospital mortality were analyzed using logistic regression models across different subgroups to examine potential interaction modifiers. Categories of continuous variables were presented as minimum to maximum. Abbreviations: OASIS, Oxford acute severity of illness score; OR, odds ratio; CI, confidence interval; SAPS II, simplified acute physiology score II; SOFA, Sepsis-related organ failure assessment score; ICD, International Classification of Diseases, Ninth Revision; AIDS, acquired immune deficiency syndrome.
Abbreviations: ROC, receiver operating characteristic; OASIS, Oxford acute severity of illness score; SOFA, Sepsis-related organ failure assessment score; ICU, intensive care unit; SAPS II, simplified acute physiology score II; AUC, area under the ROC curve; CI, confidence interval.
Abbreviations: OASIS, Oxford acute severity of illness score.
Abbreviations: OASIS, Oxford acute severity of illness score.
Abbreviations: OASIS, Oxford acute severity of illness score; ICU, intensive care unit.
Abbreviations: SOFA, Sepsis-related organ failure assessment score; ICU, intensive care unit.
Abbreviations: SAPS II, simplified acute physiology score II; ICU, intensive care unit.
The main codes used for extracting data are presented. More basic codes are available at https://github.com/MIT-LCP/mimic-code.
Data Availability Statement
The following information was supplied regarding data availability:
The codes we created are available in the Supplemental File to help readers to replicate the analyses. More codes can be found at https://github.com/MIT-LCP/mimic-code. The raw data itself is from a third-party dataset available from MIMIC-III, a freely accessible critical care database. Reproduction of their data is not permitted according to the Data Use Agreement of the database but access can be requested here: https://mimic.physionet.org/gettingstarted/access.