TABLE 2.
Global Open Source Severity of Illness Score-1 Test Set Performance Versus Acute Physiology and Chronic Health Evaluation-IIIj and Acute Physiology and Chronic Health Evaluation-IVa
| Prediction Algorithm (Test Cohort) |
n | Observed Death Rate, % |
Average Predicted Death Rate, % |
Area Under the Receiver Operator Characteristic Curve (95% CI) |
Standardized Mortality Ratio (95% CI) |
Brier Score |
Hosmer- Lemeshow Test Statistic (8 Degrees of Freedom) |
|---|---|---|---|---|---|---|---|
| GOSSIS-1 (overall) | 114,079 | 8.4 | 8.5 | 0.918 (0.915–0.921) | 0.986 (0.966–1.005) | 0.050 | 103.32 |
| GOSSIS-1 (ANZICS-APD) | 74,761 | 8.1 | 8.2 | 0.925 (0.922–0.928) | 0.982 (0.957–1.007) | 0.047 | 87.77 |
| GOSSIS-1 (eICU-CRD)a | 39,318 | 9 | 9.1 | 0.904 (0.9–0.909) | 0.992 (0.959–1.024) | 0.055 | 25.14b |
| APACHE-IIIj (ANZICS-APD) | 74,761 | 8.1 | 13.6 | 0.904 (0.9–0.908) | 0.594 (0.579–0.609) | 0.059 | 2,918.60 |
| APACHE-IVa (eICU-CRD)a | 34,398 | 9.1 | 11.8 | 0.869 (0.863–0.876) | 0.77 (0.743–0.797) | 0.063 | 338.71b |
ANZICS-APD = Australian and New Zealand Intensive Care Society Adult Patient Database, APACHE = Acute Physiology and Chronic Health Evaluation, eICU-CRD = eICU Collaborative Research Database, GOSSIS-1 = Global Open Source Severity of Illness Score-1.
The inclusion criteria for GOSSIS-1 and APACHE-IVa differ slightly due to the imputation approach used in GOSSIS-1. In particular, the GOSSIS-1 and the APACHE-IVa test sets differ by 4,920 patients (all among eICU-CRD patients). The excluded patients—those without an APACHE-IVa prediction were excluded because they had missing data related to non-Acute Physiology Score 3 (e.g., Glasgow Coma Score and chronic comorbidities) related components of the APACHE-IVa prediction equation. When compared on the same set of patients (i.e., those with an APACHE-IVa inhospital mortality prediction), the results remained relatively unchanged, with GOSSIS-1 achieving an area under the receiver operator characteristic curve (AUROC) of 0.906 (95% CI, 0.901–0.911), Brier score of 0.054, and standardized mortality ratio (SMR) of 0.997 (95% CI, 0.962–1.032). In the 4,920 patients without an APACHE-IVa score, there were no significant reductions in discriminative performance (AUROC, 0.893; 95% CI, 0.878–0.907) nor calibration (Brier score, 0.059; SMR, 0.957; 95% CI, 0.866–1.047) of GOSSIS-1.
Calculated on a common set of 34,398 patients who had both GOSSIS-1 and APACHE IVa scores.