Table 4.
Study outcome overview of best and worst area under the curve values
| Study (year) | Hoursa | Data typesb |
Modelsd (NLP)e | AUCf | |
|---|---|---|---|---|---|
| DVLMC | Tc | ||||
| Horng et al.47 (2017) | Identify | DV- - - | CC + NN | RF (BoW) | 0.87 |
| DV- - - | – | NB | 0.65 | ||
| Apostolova and Velez48 (2017) | Identify | - - - - - | NN | SVM (BoW + tf-idf) | – |
| - - - - - | NN | Logistic regression + KNN + SVM (PV) | – | ||
| Culliton et al.49 (2017) | −4 | - - - - - | CN | Ridge regression (GloVe) | 0.64 |
| −8 | - - - - - | CN | Ridge regression (GloVe) | 0.66 | |
| −24 | - - - - - | CN | Ridge regression (GloVe) | 0.73 | |
| −24g | -V- -C | CN | Ridge regression (GloVe) | 0.85 | |
| -V- -C | – | Ridge regression (GloVe) | 0.80 | ||
| Delahanty et al.51 (2019) | +1 | -VL- - | – | GBT | 0.93 |
| +3 | -VL- - | – | GBT | 0.95 | |
| +6 | -VL- - | – | GBT | 0.96 | |
| +12 | -VL- - | – | GBT | 0.97 | |
| +24 | -VL- - | – | GBT | 0.97 | |
| Liu et al.50 (2019) | −7 | -VLM- | CN | GRU (GloVe) | 0.92 |
| −7.3 | -VLM- | CN | GBT (BoW) | 0.91 | |
| −6 | -VLM- | – | GBT | 0.85 | |
| Amrollahi et al.53 (2020) | −4h | -VL- - | PN + NN | LSTM (ClinicalBERT) | 0.84 |
| - - - - - | PN + NN | LSTM (ClinicalBERT) | 0.74 | ||
| Hammoud et al.54 (2020) | −30.6 | DVL- - | CN | Lasso regression (BoW + tf-idf) | 0.89 |
| Goh et al.52 (2021) | Identify | DVLM- | PN | Logistic regression + RF (LDA) | 0.94 |
| DVLM- | PN | dag + Logistic regression (LDA) | 0.92 | ||
| −4 | DVLM- | – | Logistic regression + RF | 0.93 | |
| DVLM- | PN | dag + Logistic regression (LDA) | 0.85 | ||
| −6 | DVLM- | PN | Logistic regression + RF (LDA) | 0.92 | |
| DVLM- | PN | dag + Logistic regression (LDA) | 0.89 | ||
| −12 | DVLM- | PN | Logistic regression + RF (LDA) | 0.94 | |
| DVLM- | – | Logistic regression + RF | 0.79 | ||
| −24 | DVLM- | PN | Logistic regression + RF (LDA) | 0.90 | |
| DVLM- | – | Logistic regression + RF | 0.78 | ||
| −48 | DVLM- | PN | Logistic regression + RF (LDA) | 0.87 | |
| DVLM- | – | Logistic regression + RF | 0.77 | ||
| Qin et al.55 (2021) | −6 to 0i | -VL- - | CN | GBT (ClinicalBERT-sf) | 0.89i |
| -VL- - | – | GBT (ClinicalBERT-m) | 0.86i | ||
Hours: Identify: not detecting hours before or after; –: hours before; +: hours after an event.
Data types: D: demographics; V: vitals; L: laboratory; M: medications; C: codes; T: text; -‘s position in DVLMC indicates which is not used.
Text data types: CC: chief complaints; CN: various types of clinical notes; NN: nursing notes; PN: physician notes; –: no notes.
Machine learning models: dag: dagging (partition data into disjoint subgroups); GBT: gradient boosted trees; GRU: gated recurrent unit; KNN: K-nearest neighbors; LSTM: long short-term memory; NB: Naïve Bayes; RF: random forest; SVM: support vector machines.
Natural language processing (NLP) techniques: BoW: Bag-of-words; ClinicalBERT: Clinical Bidirectional Encoder Representations from Transformers; ClinicalBERT-m: ClinicalBERT from merging all textual features to get embeddings; ClinicalBERT-sf: finetuned ClinicalBERT from concatenating individual embeddings of each textual feature; GloVe: Global Vectors for Word Representation; LDA: Latent Dirichlet Allocation; PV: paragraph vectors; tf-idf: term frequency-inverse document frequency.
Area under the curve (AUC). Apostolova and Velez48 did not provide metrics for AUC.
Culliton et al49 performed 2 experiments, these results are from using a test set instead of 3-fold validation.
Number of hours before onset for Amrollahi et al53 was confirmed through personal communications (with Shamim Nemati on May 27, 2021 and Fatemeh Amrollahi on June 13, 2021).
Qin et al55 AUC values are an average from 0 to 6 h before sepsis, not the specified hours.