Table 1. Overview of unique characteristics and differences between methods.
| Author | Year | Goal | Population location and size | Feature set size | “Gold standard” definition | Model and performance metrics |
|---|---|---|---|---|---|---|
| Carrara et al 8 | 2015 | Mortality prediction in septic shock patients | MIMIC II (ICU) 30,000+ patients |
30 variables | Septic shock: 1991 SIRS criteria, ICD-9 code for septic shock, abnormal interval must exceed 5 h for each feature, SIRS 2 + , SIRS with low SBP despite adequate fluid resuscitation | Multivariate linear regression with Shrinkage Techniques model Mean square error (MSE): 0.03 |
| Danner et al 9 | 2017 | Assess the value of HR-to-systolic ratio in the accuracy of sepsis prediction after ED presentation | Local (ED) 53,313 patients |
9 vitals/variables | Sepsis: Discharge diagnosis of sepsis, evaluated vitals, demographics, chief complaints | Multivariate linear regression model - Accuracy: 0.74 - HR to systolic ratio accounted for 69% of overall predictive ability |
| Capp et al 10 | 2015 | Describe key patient characteristics present within 4 h of ED arrival that are associated with developing septic shock between 4 and 48 h of ED arrival | Local (ED) 1,316 patients |
5 risk factors | Sepsis: manual chart review with SIRS 2 + , evidence of infection (excluded if gastrointestinal bleed) Septic shock: SBP > 90 mm Hg despite appropriate fluid hydration of 30 mL/kg with presence of hypotension for at least 2 h after |
Multivariable logistic regression model Found risk factors associated with progression of sepsis to septic shock between 4 and 48 h of ED arrival: - Female: 1.59 odds ratio (OR) - Nonpersistent hypotension: 6.24 OR - Lactate > 4 mmol/L: 5.30 OR - Bandemia > 10%: 2.60 OR - Past medical of coronary heart disease: 2.01 OR |
| Faisal et al 11 | 2018 | To develop a logistic regression model to predict the risk of sepsis following emergency admission using the patient's first electronically recorded vital signs and blood test results and to validate this novel computer-aided risk of sepsis model, using data from another hospital | Local (ED) 57,243 patients |
12 vitals/variables | Sepsis: ICD-10 codes without organ failure Severe sepsis: ICD-10 codes with 1+ organ failure or septic shock |
Logistic regression models All area under the receiver operator curve (AUROC): 0.79 Sepsis AUROC: 0.70 Severe sepsis AUROC: 0.81 |
| Ho et al 12 | 2012 | Investigate how different imputation methods can overcome the handicap of missing information | MIMIC II (ICU) Sample size not stated |
6 vitals | Sepsis: ICD-9 Septic shock: examined clinical chart records |
- Sepsis: Multivariate logistic regression models - Septic shock: multivariate logistic regression, linear kernel SVM, and regression trees H: Clinical history feature set P: initial physiological state feature set Sepsis AUROC (imputed mean and matrix factorization-based approaches) All H: 0.791 (0.792) Stepwise H: 0.790 (0.791) All H ∪ P: 0.821 (0.822) Stepwise H ∪ P: 0.823 (0.823) Septic shock AUROC: 0.773–0.786 |
| Langley et al 13 | 2013 | Examine clinical features, plasma metabolome, and proteome of patients to predict patient survival of sepsis | CAPSOD (ED) 1,152 individuals with suspected, community-acquired sepsis; Discovery set of 150 patients |
4 vitals/variables | Acute infection + 2+ SIRS | Logistic regression (sepsis prediction) and SVM model (survival and death prediction) Logistic regression AUROC: 0.847 Logistic regression accuracy: 0.851 *best stats occurred at enrollment SVM AUROC: 0.740 SVM accuracy: 0.746 |
| Sutherland et al 14 a | 2011 | Use gene expression biomarkers to prospectively distinguish patients with sepsis from those who experience systemic inflammation from healing of surgery | Local (ICU) 85 patients |
42 biomarkers | Likely enter sepsis cohort if met ACCP/SCCM consensus statement and clinical suspicion of systemic infection Confirmation performed retrospectively |
Classifier: Recursive partitioning, LASSO, logistic regression. Individual genes examined via Bayes-adjusted linear model. MT-PCR diagnostic classifier generated using a LogitBoost ML algorithm (tree-based) PCR Accuracy: 92% AUROC: 0.86–0.92 |
| Gultepe et al 15 | 2014 | Develop a decision support system to identify patients with hyperlactatemia and to predict mortality from sepsis using predicted lactate levels | Local (ED) 741 patients |
7 vitals/laboratories | Sepsis: determined from EHR diagnosis and SIRS criteria | SVM classifier Accuracy: 0.73 AUROC: 0.73 |
| Horng et al 16 | 2017 | To demonstrate the incremental benefit of using free text data in addition to vital sign and demographic data to identify patients with suspected infection in the emergency department | Local (ED) 198,833 control 32,103 cases |
12 vitals/variables | ED ICD-9-CM code | Linear SVM and free text models Bag of words AUROC: 0.86 Bag of words sensitivity: 0.78 Bag of words specificity: 0.79 Topic model AUROC: 0.85 Topic model sensitivity: 0.80 Topic model specificity: 0.75 |
| Thottakkara et al 17 | 2016 | To compare performance of risk prediction models for forecasting postoperative sepsis and acute kidney injury | Local (in-patient) 50,318 patients |
285 variables | Forecast postop sepsis and acute kidney injury AHRQ definition of “post-op sepsis” and organ failure associated with sepsis was identified by ICD-9-CM code for acute organ dysfunction |
Comparison of models that used logistic regression, generalized additive models (GAM), naive Bayes, SVM Naive Bayes performed the worst in the comparison; GAMs and SVMs had good performance; PCA feature extraction (reduced to 5 features) improved predictive performance for all models Severe sepsis AUROC: 0.76–0.91 |
| Vieira et al 18 | 2013 | Proposed a modified binary particle swarm optimization method for feature selection to predict mortality in septic patients | MEDAN (ICU) 382 patients |
Model chooses custom number of features (2–7) | MEDAN data set prelabeled patients for abdominal septic shock | Support vector machine for mortality prediction Modified binary particle swarm optimization (MBPSO): feature selection MBPSO 12 (28) features: No-FS Accuracy: 72.6% (89%) Accuracy: 76.5% (94.4%) |
| Ghosh et al 19 | 2017 | Predict septic shock for ICU patients using noninvasive waveform measurements | MIMIC II (ICU) 1,519 patients |
3 vitals/laboratories | Sepsis: ICD-9 Septic shock: examining clinical chart records |
Coupled hidden Markov models (CHMM) with varying gap interval and observation window sizes CHMM average: 0.85 Multichannel patterns (MCP)-CHMM average: 0.86 |
| Peelen et al 20 | 2010 | Develop a set of complex Markov models based on clinical data to extract meaningful clinical patterns and to provide prediction for sepsis and other diseases | Local (ICU) 2,271 patients |
6 variables | Sever sepsis: SIRS 2+ within 24 h of ICU admission and 1+ dysfunctioning organ system (SOFA) | 3 Markov models (amount of organ failure, type of organ failure, differences between development and persistence of organ failure) ICU death the error rates were 17.7%, 18.1%, and 17.8% and the AUCs were 0.79, 0.79, and 0.80 for models I, II, and III |
| Stanculescu et al 21 | 2014 | Demonstrate that by adding a higher-level discrete variable with semantics sepsis/nonsepsis, can detect changes in the physiological factors that signal the presence of sepsis | Local (NICU) 24 neonates |
Bradycardia, desaturation | Laboratory result of blood culture for neonatal sepsis | Hierarchical switching linear dynamical system (HSLDS) Autoregressive (AR)-HMM AUROC: 0.72 HSLDS deep learning AUROC: 0.69 HSLDS known factors AUROC: 0.62 |
| Stanculescu et al 22 | 2014 | Detect and identify sepsis in neonates before a blood sample is drawn. Furthermore, they wanted to identify which physiological event would contribute most for detecting sepsis | Local (NICU) 24 neonates |
6 vitals/variables | Positive cultures as pathogens: proven sepsis Positive cultures as mixed growth/skin commensal: “suspected sepsis” |
AR-HMM AUROC: 0.74–0.75 AUROC with missing data: 0.72–0.73 AUROC with bradycardia and minibradycardia: 0.79–0.80 AUROC with desaturation: 0.76–0.78 AUROC with all states: 0.79–0.80 |
| Gultepe et al 23 | 2012 | Use a Bayesian network to detect sepsis early | Local (ICU) 1,492 patients |
BN1: 5 variables BN2: 7 variables |
“Sepsis occurrence” | Bayesian network (BN) models BN-1 (vitals) goodness of fit: 15.4 BN-2 (vitals + MAP) goodness of fit: 19.9 Found that lactate is a driver in both models and maybe an important feature for early sepsis detection |
| Nachimuthu and Haug 24 | 2012 | Detect sepsis right after patients are admitted to the ED | Local (ED) 3,100 patients |
11 vitals/variables | Clinician determined “sepsis” during retrospective chart review | Dynamic Bayesian network 3 h after admission AUROC: 0.911 6 h after admission AUROC: 0.915 12 h after admission AUROC: 0.937 24 h after admission AUROC: 0.944 |
| Calvert et al 25 | 2016 | Detect and predict the onset of septic shock for alcohol-use disorder patients in the ICU | MIMIC III (ICU) 1,394 patients |
9 vitals/variables | Septic shock: SIRS 2 + , ICD-9, organ dysfunction, SBP < 90 mm Hg for 1 h, total fluid replacement ≥ 1,200 mL or 20 mL/kg for 24 h |
InSight
Sensitivity: 0.93 Specificity: 0.91 Accuracy: 0.91 F1 score: 0.161 |
| Calvert et al 26 | 2016 | To develop high-performance early sepsis prediction technology for the “general patient population” | MIMIC II (ICU) 29,083 patients |
10 vitals/variables | Sepsis: ICD-9 code, 1991 SIRS for 5 h |
InSight
Sensitivity: 0.90 Specificity: 0.81 AUROC: 0.92 Accuracy: 0.83 |
| Desautels et al 27 | 2016 | To validate InSight with the new Sep-3 definition and make predictions using minimal set of variables | MIMIC III (ICU) 22,583 patients |
8 vitals/laboratories | Sepsis: Sep-3 definition, suspicion of infection equated with an order of culture laboratory draw and dose of antibiotics |
InSight
AUROC: 0.88 APR: 0.60 |
| Mao et al 28 | 2018 | Validate the InSight algorithm for detection and prediction of sepsis and septic shock | MIMIC III (ICU) Local (ED, general) 61,532 stays |
6 vitals/laboratories | Sepsis: ICD-9 + SIRS 2+ (995.91) Severe sepsis: ICD-9 (955.92), organ dysfunction, SIRS 2+ Septic shock: ICD-9 (785.52), SBP < 90 mm Hg (at least 30 min), resuscitated with ≥ 20 mL/kg over 24 h, ≥ 1,200mL in total fluids |
InSight
Detect sepsis AUROC: 0.92 Detect severe sepsis AUROC: 0.87 Detect 4 h before onset sepsis AUROC: 0.96 Detect 4 h before onset severe sepsis AUROC: 0.85 |
| McCoy and Das 29 | 2017 | Aimed to improve sepsis-related patient outcomes through a revised sepsis management approach | Local (ICU) 407 patients |
6 vitals/variables | Severe sepsis: SIRS 2 + , qSOFA score |
Dascena
Sep-3 AUROC: 0.91 Sep-3 sensitivity: 0.83 Sep-3 specificity: 0.96 Severe sepsis AUROC: 0.96 Severe sepsis sensitivity: 0.90 Severe sepsis specificity: 0.85 |
| Shimabukuro et al 30 a | 2017 | Randomized control trial to show lowered mortality and length of stay using a machine learning sepsis prediction algorithm | Local (ICU) 75 controls 67 cases |
7 vitals/laboratories | Severe sepsis: “organ dysfunction caused by sepsis” Random allocation sequence to put patients in groups |
InSight
AUROC: 0.952 Sensitivity: 0.9 Specificity: 0.9 Average length of stay decreased from 13 to 10.3 d In-hospital mortality decreased by 12.3% |
| Henry et al 32 | 2015 | Create and test a score that predicts which patients will develop septic shock | MIMIC II (ICU) 16,234 patients |
54 features | Suspicion of infection: ICD-9 or by presence of clinical note that mentioned sepsis or septic shock Sepsis: suspicion + SIRS Severe sepsis: sepsis + organ dysfunction |
TREWScore (Cox proportional hazards model using the time until the onset of septic shock as the supervisory signal) AUROC: 0.83 Specificity: 0.67 Sensitivity: 0.85 “Patients were identified a median of 28.2 h before shock onset” |
| Lukaszewski et al 31 a | 2008 | Detect and identify septic patients before displaying symptoms for ICU patients | Local (ICU) 92 patients |
7 cytokines | Admission diagnosis upon ICU entry | Neural networks using cytokine and chemokine data Sensitivity: 0.91 Specificity: 0.80 Accuracy: 0.95 |
| Nemati et al 35 | 2018 | Aimed to develop and validate an artificial intelligence sepsis algorithm for early prediction of sepsis | Local (ICU) 33,069 patients |
65 variables | Sepsis: Sepsis-3 | Modified Weibull–Cox proportional hazards model 4 h in advance AUROC: 0.85 |
| Pereira et al 33 | 2011 | Examined different approaches to predicting septic shock with missing data | MEDAN (ICU) 139 patients |
2 sets of 12 and 28 “selected features” | Septic shock: associated with abdominal causes (not clearly defined, data may be prelabeled) | Zero-Order-Hold (ZOH) Fuzzy c-means clustering based on partial distance calculation strategy (FCM-PDS) Performance improvements occur where up to 60% of the data are missing ZOH-FCM-PDS 12 (28) feature AUROC: 0.899 (0.649); FCM-PDS 12 (28) feature AUROC: 0.786 (0.631) |
| Ribas et al 36 | 2011 | Demonstrate that a SVM variant can provide automatic ranking of mortality predictor and have higher accuracy that current methods | Local (ICU) 354 patients |
4 vitals/laboratories | Severe sepsis: organ dysfunction (SOFA) | Relevance vector machine AUROC: 0.80 Error rate: 0.24 Sensitivity: 0.66 Specificity: 0.80 |
| Sawyer et al 38 a | 2011 | Evaluate if implementing an automated sepsis screening and alert system can facilitate in early interventions by identifying non-ICU patients at risk for developing sepsis | Local (Non-ICU) 270 patients |
9 vitals/variables | Intervention group: real-time sepsis alert generated from Clinical Desktop | Recursive partitioning regression tree analysis Within 12 h of sepsis alert, 70.8% of patients in the intervention group received treatment versus 55.8% in control |
| Shashikumar et al 37 | 2017 | Investigates the utility of high-resolution blood pressure and heart rate times series dynamics for the early prediction of sepsis | Local (ICU) 242 patients |
11 vitals/variables | Sepsis: Seymour (Sep-3) at some point during ICU stay | Elastic Net logistic classifier: 3 models: (1) entropy features, (2) EMR + sociodemographic-patient history features, (3) models 1 + 2 Model 1 AUROC (Accuracy): 0.67 (0.47) Model 2 AUROC (Acc): 0.70 (0.50) Model 3 AUROC (Acc): 0.78 (0.61) |
| Taylor et al 39 | 2016 | Compare a machine learning approach to existing clinical decision rules to predict sepsis in-hospital mortality | Local (ED) 4,676 patients |
20 variables | ICD-9 with AHRQ clinical classification software to obtain more exhaustive list of patients | Random forest model AUROC: 0.86 |
Abbreviations: AHRQ, Agency for Healthcare Research and Quality; ACCP/SCCM, American College of Chest Physicians/Society of Critical Care; ED, emergency department; EHR, electronic health record; HR, heart rate; ICD, International Classification of Disease; ICU, intensive care unit; MEDAN, Medical Data Warehousing and Analysis; MIMIC, Medical Information Mart for Intensive Care; ML, machine learning; MT-PCR, multiplex tandem-polymerase chain reaction; NICU, neonatal intensive care unit; PCA, principal component analysis; SBP, systolic blood pressure; SIRS, systemic inflammatory response syndrome; SOFA, sequential organ failure assessment; SVM, support vector machine.
a
Indicates prospective.