Abstract
The quick sepsis-related organ failure assessment (qSOFA) score has been introduced to predict the likelihood of organ dysfunction in patients with suspected infection. We hypothesized that machine-learning models using qSOFA variables for predicting three-day mortality would provide better accuracy than the qSOFA score in the emergency department (ED). Between January 2016 and December 2018, the medical records of patients aged over 18 years with suspected infection were retrospectively obtained from four EDs in Korea. Data from three hospitals (n = 19,353) were used as training-validation datasets and data from one (n = 4234) as the test dataset. Machine-learning algorithms including extreme gradient boosting, light gradient boosting machine, and random forest were used. We assessed the prediction ability of machine-learning models using the area under the receiver operating characteristic (AUROC) curve, and DeLong’s test was used to compare AUROCs between the qSOFA scores and qSOFA-based machine-learning models. A total of 447,926 patients visited EDs during the study period. We analyzed 23,587 patients with suspected infection who were admitted to the EDs. The median age of the patients was 63 years (interquartile range: 43–78 years) and in-hospital mortality was 4.0% (n = 941). For predicting three-day mortality among patients with suspected infection in the ED, the AUROC of the qSOFA-based machine-learning model (0.86 [95% CI 0.85–0.87]) for three -day mortality was higher than that of the qSOFA scores (0.78 [95% CI 0.77–0.79], p < 0.001). For predicting three-day mortality in patients with suspected infection in the ED, the qSOFA-based machine-learning model was found to be superior to the conventional qSOFA scores.
Keywords: qSOFA, infection, sepsis, machine learning, emergency department
1. Introduction
The early recognition and prompt treatment of sepsis in the emergency department (ED) are important to improve patient outcomes [1]. Sepsis has been characterized as a systemic inflammatory response syndrome (SIRS) to infection [2]. The quick sepsis-related organ failure assessment (qSOFA) criteria [3] were introduced by the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). The qSOFA scores can be used outside of the intensive care unit (ICU) for predicting mortality or ICU stay [4,5,6,7,8,9]. However, several meta-analyses of qSOFA scores have shown that these scores had a poor sensitivity for predicting in-hospital mortality [10,11,12,13,14]. Early warning scores [15,16], a tool for identifying hospitalized patients at risk of deterioration, have been proposed for predicting hospital mortality in those with suspected sepsis in the ED [7,17,18,19,20]. However, Hamilton et al. reported that the early warning scores were not accurate in predicting sepsis mortality in the ED (67% of sensitivity and 60% of specificity) [21].
Recently, various machine-learning methods for predicting outcomes more accurately have been implemented in the medical field [22,23,24]. Machine-learning models for the early identification of patients at risk for sepsis have been developed in the ICU [25] and ED settings [26,27,28]. Although these diverse machine-learning models can improve predictive accuracy for sepsis outcomes, they require excessive variables and laboratory results that may not be available in the ED. These factors can lead to poor generalizability of the machine-learning-based prognostic models.
The objective of this study was to develop and validate the prognostic performance of a qSOFA-based machine-learning model for three-day mortality prediction in patients with suspected infection in the ED.
2. Materials and Methods
2.1. Study Design
This retrospective multicenter study was conducted between January 2016 and December 2018 in four hospitals (Dongtan Sacred Heart Hospital, Kangnam Sacred Heart Hospital, Chuncheon Sacred Heart Hospital, and Hallym University Sacred Heart Hospital) of the Hallym University Medical Center (capacity of more than 3000 beds) in the Republic of Korea. Approximately 150,000 patients visited the EDs per year during the study period with a mean of 37,000 (range: 19,000–52,000) annual visits. A total of 447,926 patients visited the ED during the study period. The medical records of 23,587 patients older than 18 years suspected of having an infection were accessed and their data were analyzed.
The Institutional Review Board of the Hallym Medical Center approved the study (approval no. 2019-10-017-001) and waived the requirement for informed consent due to the retrospective nature of the analyses.
2.2. Data Collection and Definition
Clinical data were extracted using the clinical big data analytic solution Smart Clinical Data Warehouse (CDW) from the Hallym University Medical Center, which is based on the QlikView Elite Solution (Qlik, King of Prussia, PA, USA). It analyzes the electronic medical record (EMR) text and integrated fixed data. Using the Smart CDW, we collected the following clinical data of patients with suspected infection: demographic variables (age and sex), diagnoses at the ED, initial vital signs (systolic blood pressure, respiration rate, mental status, body temperature, and heart rate), arterial partial pressure of carbon dioxide, white blood cell count, duration of hospitalization, ICU admission, mechanical ventilation, and mortality. Mental status was evaluated by emergency specialist nurses using the alert, voice, pain, unresponsive (AVPU) scale (Supplementary Table S1). Body temperature was assessed using a tympanic ear thermometer.
Among the patients admitted to the ED, the data of those whose main diagnoses were infection-related were extracted. Infection-related diagnoses were referenced in a previous study by Rodriguez et al. [29]. Two of the authors (pulmonologist and anesthesiologist) reviewed and examined the patient data for eligibility, and ambiguous inclusions were excluded by consensus review (Supplementary Table S2). To avoid recording errors in the EMRs, we excluded systolic blood pressure, heart rate, respiratory rate, and body temperature values that were outside the ranges of 30–300 mmHg, 10–300 beats/min, 3–60 breaths/min, and 30–45°C, respectively [30]. The primary outcome was mortality within 3 days of admission to the ED and secondary outcomes were in-hospital mortality, ICU admission within 3 days of admission to ED, and ICU admission.
The qSOFA criteria consist of systolic blood pressure ≤ 100 mmHg, respiratory rate ≥ 22 breaths/min, and altered mental status [3]. We considered altered mental status as non-alert per the AVPU scale [31]. Patients were assigned one point for each criterion, and a qSOFA score of ≥ 2 was considered indicative of poor outcomes. The SIRS criteria are considered to be met if at least two of the following four clinical findings are present [2]: temperature > 38 °C or < 36 °C, heart rate > 90/min, respiratory rate > 20/min or arterial partial pressure of carbon dioxide < 32 mm Hg, and white blood cell count > 12 × 109/L or < 4 × 109/L or immature band cells > 10%. Additionally, the modified early warning score (MEWS) is widely used for detecting clinical deterioration in patients in the ED [16]; it is also used in prediction assessment for in-hospital mortality or ICU admission (Supplementary Table S1) [32].
2.3. Statistical Analysis and Machine Learning
Data on patient demographics, severity of illness scores (qSOFA, SIRS, and MEWS), diagnoses on admission, and outcome variables were analyzed using descriptive statistics. Continuous and categorical variables are expressed as the median (interquartile range) and number (percentage), respectively. We assessed the discriminatory power of each severity of illness score using the area under the receiver operating characteristic (AUROC) curves. All statistical analyses were performed using the Statistical Package for the Social Sciences (SPSS) Version 24.0 (IBM Corporation, Armonk, NY, USA).
The qSOFA-based machine-learning model was developed using the qSOFA criteria, including systolic blood pressure, respiratory rate, and mental status. A dataset was created with the variables of the 23,587 patients. We divided the dataset into training and validation test sets to prevent the model from overfitting the test set. The test set consisted of data from other hospitals that were not associated with training-validation to test the model. The data from three hospitals (Dongtan Sacred Heart Hospital, Kangnam Sacred Heart Hospital, and Hallym University Sacred Heart Hospital) were used for training-validation (n = 19,353) and the data from one hospital (Chuncheon Sacred Heart Hospital) as the test dataset (n = 4234). The training-validation set was divided into the training and validation sets at a 9:1 ratio. The overall dataset was divided as follows: 74% for the training set, 8% for the validation set, and 18% for the test set. Datasets were standardized by min-max scaling. Regarding the model algorithm, extreme gradient boosting (XGB), light gradient boosting machine (LGBM), and random forest were used. In the training dataset, deaths and ICU inpatients were much fewer than survivors and general ward inpatients, respectively. This data imbalance can bias the machine-learning models and render them inaccurate. To solve this problem, we trained the model after balancing the training dataset through the synthetic minority oversampling technique to a 1:1 ratio in the XGB and LGBM models and modeled the balanced random forest. Three models were trained with the basic hyperparameters and training sets and evaluated with a validation set to select the one with the best performance. We performed 5-fold cross validation on the training dataset and tuned the hyperparameters using grid search. The final model was validated through the test set. We evaluated models with the ROC curve and AUROC using Anaconda (Python version 3.7, https://www.anaconda.com; Anaconda Inc., Austin, TX, USA), the XGBoost package version 0.90 (https: //xgboost.readthedocs.io), the LGBM package version 2.2.3 (https://lightgbm.readthedocs.io/en/latest/Python-Intro.html), and the imbalanced-learn package version 0.5.0 (https://imbalanced-learn.readthedocs.io).
DeLong’s test was used to compare AUROCs between qSOFA scores and qSOFA-based machine-learning models [33].
3. Results
Characteristics of patients with suspected infection in the ED are summarized in Table 1. A total of 23,587 patients suspected of having an infection were enrolled, and their data were analyzed (Figure 1). The median age was 63 years (IQR: 43–78 years) and 46.1% were men (n = 10,862). Approximately 21.9% (n = 5173) of the patients were admitted to the ICU and 941 died in the hospital (4.0%). The most common diagnoses were respiratory (28.6%, n = 6736), intra-abdominal (24.1%, n = 5693), and urinary tract (15.4%, n = 3638) infection. There were significant differences in median age, severity of illness scores, ICU admission, and in-hospital mortality between the test and training-validation datasets.
Table 1.
Characteristics of patients with suspected infection in the emergency department.
| Variable | Total (n = 23,587) |
Training-Validation Datasets (n = 19,353) |
Test Datasets (n = 4234) |
p Value |
|---|---|---|---|---|
| Median age (IQR) | 63 (43–78) | 62 (42–77) | 67 (50–79) | <0.001 |
| Male sex (%) | 10862 (46.1) | 8850 (45.7) | 2012 (47.5) | 0.035 |
| Severity of illness scores (%) | ||||
| qSOFA ≥ 2, | 4698 (19.9) | 1692 (8.7) | 507 (12.0) | <0.001 |
| SIRS ≥ 2 | 12224 (51.8) | 9960 (51.5) | 2264 (53.5) | 0.018 |
| MEWS ≥ 5 | 5857 (24.8) | 4517 (23.3) | 1340 (31.6) | <0.001 |
| Suspected infection source (%) | <0.001 | |||
| Respiratory | 6736 (28.6) | 5437 (28.1) | 1299 (30.7) | |
| Intra-abdominal | 5693 (24.1) | 4622 (23.9) | 1071 (25.3) | |
| Urinary | 3638 (15.4) | 2998 (15.5) | 640 (15.1) | |
| Hepatobiliary | 1871 (7.9) | 1546 (8.0) | 325 (7.7) | |
| Otorhinolaryngological | 1789 (7.6) | 1481 (7.7) | 308 (7.3) | |
| Skin or musculoskeletal | 1132 (4.8) | 944 (4.9) | 188 (4.4) | |
| Gynecological | 430 (1.8) | 393 (2.0) | 37 (0.9) | |
| Central nervous system | 410 (1.8) | 342 (1.8) | 68 (1.6) | |
| Other or unknown | 1888 (8.0) | 1590 (8.2) | 298 (7.0) | |
| Outcomes | ||||
| In-hospital mortality (%) | 941 (4.0) | 795 (4.1) | 146 (3.4) | 0.048 |
| ICU admission (%) | 5173 (21.9) | 4191 (21.7) | 982 (23.2) | 0.029 |
| Hospital length of stay, median (IQR), d | 7 (5–12) | 7 (5–12) | 8 (5–13) | 0.004 |
| Mechanical ventilator use (%) | 1662 (7.0) | 1320 (6.9) | 330 (7.8) | 0.036 |
Values are expressed as median (interquartile range) or number (%). IQR = interquartile range; ED = emergency department; qSOFA = quick Sepsis-related Organ Failure Assessment; SIRS = systemic inflammatory response syndrome; MEWS = modified early warning score; ICU = intensive care unit.
Figure 1.
Flow chart.
Predictive performance of algorithm-specific machine learning models through validation sets are shown in Supplementary Tables S3 and S4 shows the cross-validation results of selected models. The AUROCs showing the performance of the qSOFA-based machine-learning model for outcome prediction are presented in Figure 2. For predicting three-day mortality among patients with suspected infection in the ED, the AUROC of the qSOFA-based machine-learning model was 0.85 (95% CI 0.83–0.86). The prognostic performance of the qSOFA-based machine-learning model for the prediction of several outcomes was as follows: 0.75 (95% CI 0.74–0.76) for in-hospital mortality, 0.79 (95% CI 0.79–0.79) for three -day ICU admission, and 0.79 (95% CI 0.78–0.79) for ICU admission. The AUROCs of the qSOFA-based machine-learning model for three-day mortality in the higher (≥2) and lower (<2) qSOFA groups were 0.76 (95% CI 0.67–0.81) and 0.77 (95% CI 0.71–0.80), respectively (Figure 3).
Figure 2.
The area under the receiver operating characteristic (AUROC) curve of the machine-learning models for predicting outcomes in the test set. ICU = intensive care unit.
Figure 3.
The area under the receiver operating characteristic (AUROC) curve of the machine-learning models for predicting 3-day mortality in the higher (≥2) and lower (<2) qSOFA groups. qSOFA = quick Sepsis-related Organ Failure Assessment.
For predicting three-day mortality, qSOFA scores and MEWS showed better discrimination ability than the SIRS (qSOFA = 0.78 [95% CI 0.68–0.88]; MEWS = 0.77 [95% CI 0.67–0.86]; SIRS = 0.68 [95% CI 0.57–0.79]) (Table 2). The AUROC of qSOFA scores was higher for three-day mortality than for in-hospital mortality, three-day ICU admission, and ICU admission (0.71 [95% CI 0.66–0.75] for in-hospital mortality; 0.73 [95% CI 0.72–0.75] for three-day ICU admission; 0.73 [95% CI 0.72–0.75] for ICU admission).
Table 2.
Area under the receiver operating characteristic curve for outcomes according to the severity of illness scores from the independent test set.
| Variable | AUROC (95% CI) | |||
|---|---|---|---|---|
| 3-Day Mortality | In-Hospital Mortality | 3-Day ICU Admission | ICU Admission | |
| qSOFA | 0.78 (0.68–0.88) | 0.71 (0.66–0.75) | 0.73 (0.72–0.75) | 0.73 (0.72–0.75) |
| SIRS | 0.68 (0.57–0.79) | 0.66 (0.62–0.70) | 0.63 (0.62–0.65) | 0.63 (0.61–0.65) |
| MEWS | 0.77 (0.67–0.86) | 0.65 (0.61–0.70) | 0.69 (0.67–0.71) | 0.69 (0.67–0.70) |
AUROC = Area under the receiver operating characteristics; CI = confidence interval; qSOFA = quick Sepsis-related Organ Failure Assessment; SIRS = systemic inflammatory response syndrome; MEWS = modified early warning score; ICU = intensive care unit.
In the prediction of outcome, the discriminatory abilities of the qSOFA scores and machine-learning models in the test set are shown in Table 3. Prediction performance of the qSOFA-based machine-learning model (0.86 [95% CI 0.85–0.87]) for three-day mortality was significantly higher than the conventional qSOFA scores (0.78 [95% CI 0.77–0.79], p < 0.001). Compared with the qSOFA scores, machine-learning models demonstrated a significantly higher AUROC.
Table 3.
Prediction performance of the qSOFA scores and machine-learning models in the test set.
| Models | qSOFA Scores | qSOFA-Based Machine-Learning Models | |
|---|---|---|---|
| Outcomes | AUROC (95% CI) | p Value | |
| 3-day mortality | 0.78 (0.77–0.79) | 0.86 (0.85–0.87) | <0.001 |
| In-hospital mortality | 0.71 (0.69–0.72) | 0.75 (0.74–0.76) | 0.002 |
| 3-day ICU admission | 0.73 (0.72–0.75) | 0.79 (0.78–0.80) | <0.001 |
| ICU admission | 0.73 (0.72–0.75) | 0.79 (0.77–0.80) | <0.001 |
AUROC = area under the receiver operating characteristics; CI = confidence interval; qSOFA = quick Sepsis-related Organ Failure Assessment; ICU = intensive care unit.
4. Discussion
In this multicenter study, we applied three machine-learning algorithms (extreme gradient boosting, light gradient boosting machine, and random forest) using three qSOFA variables to predict three-day mortality in patients with suspected infection in the ED. The outcome predictive abilities of the qSOFA-based machine-learning model performed in the independent test set was satisfactory. Particularly, prediction performance of the machine-learning model for three-day mortality was superior to the conventional qSOFA score. Furthermore, we developed a qSOFA-based machine-learning model for predicting three-day mortality in the lower (<2) qSOFA groups that showed an acceptable AUROC.
Recently, machine-learning models have been applied for predicting diverse outcomes in the ED, e.g., cardiac arrest prediction [24], ED triage [34,35,36], prediction of hospital admission [37], identification of patients with suspected infection [27], screening of sepsis [28] or septic shock [26], and mortality prediction in patients with sepsis [38] or suspected infection [39]. Our study suggests that the ability of machine-learning models for predicting deterioration within three days of patients with suspected infection are superior to the conventional severity illness scores. Machine-learning models, which can predict sepsis outcome with high accuracy, are important because the sepsis is a medical emergency [40] and there is limited time for patient care in the ED. Horng et al. demonstrated that accurate triggering of clinical decision support will become increasingly more important as clinical decision support becomes more integrated into EMRs [27]. Cho et al. reported that a deep learning-based early warning system, which can be applied with EMRs, accurately predicted the deterioration of patients [41]. We suggest that our qSOFA-based machine-learning model incorporated with real-time clinical variables on the EMR can be utilized by physicians for making clinical decisions for treating sepsis.
The qSOFA criteria consist of systolic blood pressure (≤100 mmHg), respiratory rate (≥22 breaths/min), and altered mental status (three point each) [3]. However, the cutoffs for qSOFA can be potentially arbitrary [42]. On the other hand, our machine learning models using continuous variables may enable finer classification of dataset [43]. Machine leaning approaches are adept at handling high-order interactions between the predictors and non-linear relationships with the outcome [35]. Additionally, ensembles of decision tree methods like gradient boosting can automatically provide estimates of feature importance from a trained predictive model. Feature importance provides a score that indicates how valuable each feature is in the construction of the boosted decision trees within the model. We suggest that these factors contribute to better predictive ability of machine learning models than conventional qSOFA scores.
The qSOFA scores are recognized as a parsimonious tool for other complex and cost consuming way to screen for sepsis outcome [3]. Our findings support that the qSOFA score can be acceptable for predicting acute deterioration in patients with suspected infection (AUROC: three-day mortality = 0.78 and three-day ICU admission = 0.73). However, the results of a recent meta-analysis showed that the qSOFA scores are not useful for predicting in-hospital mortality (AUROC = 0.68) or ICU admission (AUROC = 0.65) [44]. Moreover, because of the poor sensitivity of the qSOFA scores, there is a possibility of delays in sepsis identification [45]. However, novel machine-learning models can accurately predict sepsis onset beforehand [46]. In accordance with the results of previous studies [38,39], our machine-learning models showed higher accuracy rate of mortality prediction in patients with suspected infection than the qSOFA scores. Therefore, machine-learning based prediction models may be beneficial for physicians in management of sepsis in the future.
The major strength of this multicenter study was the use of a large dataset and the successful prediction of mortality in the test set using the machine-learning models. These results are reliable because the model was tested using an independent test set that had patient characteristics different from those in training-validation.
Nevertheless, the study had several limitations due to its retrospective design. Although both authors carefully reviewed the diagnoses, the study population was difficult to be determined because the definition of suspected infection varied among studies. Additionally, we cannot confirm that the attending physician had documented the diagnosis. The indication of ICU admission may vary among different EDs. Therefore, further prospective studies are needed to examine the efficacy of these machine-learning models for predicting mortality in patients with infection in the ED setting.
5. Conclusions
We developed machine-learning models using three qSOFA criteria to predict three-day mortality in patients with suspected infection in the ED. The qSOFA-based machine-learning models are superior to the conventional qSOFA scores. We suggest that our qSOFA-based machine-learning models can assist physicians’ clinical decision-making for treating sepsis in the ED.
Supplementary Materials
The following are available online at https://www.mdpi.com/2077-0383/9/3/875/s1, Table S1: The modified early warning score, Table S2: Infection-related diagnosis, Table S3: Predictive performance of algorithm-specific machine learning models through validation sets, Table S4: Algorithms of selected machine-learning models and results of training set cross-validation.
Author Contributions
M.S.B. and Y.S.K. were responsible for study conception and design and drafting of the article. M.S.B. was responsible for acquisition of the data. All authors were responsible for the analysis and interpretation of data and for critical revision of the manuscript. M.S.B. takes responsibility for the paper as a whole. All authors have read and agreed to the published version of the manuscript.
Funding
This research was supported by the Bio & Medical Technology Development Program of the National Research Foundation (NRF) funded by the Korean government (MSIT) (NRF-2019M3A9E8020536), South Korea.
Conflicts of Interest
The authors disclose that they do not have any conflicts of interest.
References
- 1.Rowan K.M., Angus D.C., Bailey M., Barnato A.E., Bellomo R., Canter R.R., Coats T.J., Delaney A., Gimbel E., Grieve R.D., et al. Early, Goal-Directed Therapy for Septic Shock—A Patient-Level Meta-Analysis. N. Engl. J. Med. 2017;376:2223–2234. doi: 10.1056/NEJMoa1701380. [DOI] [PubMed] [Google Scholar]
- 2.Bone R.C., Balk R.A., Cerra F.B., Dellinger R.P., Fein A.M., Knaus W.A., Schein R.M., Sibbald W.J. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. Chest. 1992;101:1644–1655. doi: 10.1378/chest.101.6.1644. [DOI] [PubMed] [Google Scholar]
- 3.Seymour C.W., Liu V.X., Iwashyna T.J., Brunkhorst F.M., Rea T.D., Scherag A., Rubenfeld G., Kahn J.M., Shankar-Hari M., Singer M., et al. Assessment of Clinical Criteria for Sepsis: For the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) JAMA. 2016;315:762–774. doi: 10.1001/jama.2016.0288. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.April M.D., Aguirre J., Tannenbaum L.I., Moore T., Pingree A., Thaxton R.E., Sessions D.J., Lantry J.H. Sepsis Clinical Criteria in Emergency Department Patients Admitted to an Intensive Care Unit: An External Validation Study of Quick Sequential Organ Failure Assessment. J. Emerg. Med. 2017;52:622–631. doi: 10.1016/j.jemermed.2016.10.012. [DOI] [PubMed] [Google Scholar]
- 5.Askim A., Moser F., Gustad L.T., Stene H., Gundersen M., Asvold B.O., Dale J., Bjornsen L.P., Damas J.K., Solligard E. Poor performance of quick-SOFA (qSOFA) score in predicting severe sepsis and mortality—A prospective study of patients admitted with infection to the emergency department. Scand. J. TraumaResusc. Emerg. Med. 2017;25:56. doi: 10.1186/s13049-017-0399-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Canet E., Taylor D.M., Khor R., Krishnan V., Bellomo R. qSOFA as predictor of mortality and prolonged ICU admission in Emergency Department patients with suspected infection. J. Crit. Care. 2018;48:118–123. doi: 10.1016/j.jcrc.2018.08.022. [DOI] [PubMed] [Google Scholar]
- 7.Goulden R., Hoyle M.C., Monis J., Railton D., Riley V., Martin P., Martina R., Nsutebu E. qSOFA, SIRS and NEWS for predicting inhospital mortality and ICU admission in emergency admissions treated as sepsis. Emerg. Med. J. EMJ. 2018;35:345–349. doi: 10.1136/emermed-2017-207120. [DOI] [PubMed] [Google Scholar]
- 8.Hwang S.Y., Jo I.J., Lee S.U., Lee T.R., Yoon H., Cha W.C., Sim M.S., Shin T.G. Low Accuracy of Positive qSOFA Criteria for Predicting 28-Day Mortality in Critically Ill Septic Patients During the Early Period After Emergency Department Presentation. Ann. Emerg. Med. 2018;71:1–9. doi: 10.1016/j.annemergmed.2017.05.022. [DOI] [PubMed] [Google Scholar]
- 9.Moskowitz A., Patel P.V., Grossestreuer A.V., Chase M., Shapiro N.I., Berg K., Cocchi M.N., Holmberg M.J., Donnino M.W. Quick Sequential Organ Failure Assessment and Systemic Inflammatory Response Syndrome Criteria as Predictors of Critical Care Intervention Among Patients With Suspected Infection. Crit. Care Med. 2017;45:1813–1819. doi: 10.1097/CCM.0000000000002622. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Liu Y.C., Luo Y.Y., Zhang X., Shou S.T., Gao Y.L., Lu B., Li C., Chai Y.F. Quick Sequential Organ Failure Assessment as a prognostic factor for infected patients outside the intensive care unit: A systematic review and meta-analysis. Intern. Emerg. Med. 2019;14:603–615. doi: 10.1007/s11739-019-02036-0. [DOI] [PubMed] [Google Scholar]
- 11.Song J.U., Sin C.K., Park H.K., Shim S.R., Lee J. Performance of the quick Sequential (sepsis-related) Organ Failure Assessment score as a prognostic tool in infected patients outside the intensive care unit: A systematic review and meta-analysis. Crit. Care. 2018;22:28. doi: 10.1186/s13054-018-1952-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Maitra S., Som A., Bhattacharjee S. Accuracy of quick Sequential Organ Failure Assessment (qSOFA) score and systemic inflammatory response syndrome (SIRS) criteria for predicting mortality in hospitalized patients with suspected infection: A meta-analysis of observational studies. Clin. Microbiol. Infect. Off. Publ. Eur. Soc. Clin. Microbiol. Infect. Dis. 2018;24:1123–1129. doi: 10.1016/j.cmi.2018.03.032. [DOI] [PubMed] [Google Scholar]
- 13.Jiang J., Yang J., Mei J., Jin Y., Lu Y. Head-to-head comparison of qSOFA and SIRS criteria in predicting the mortality of infected patients in the emergency department: A meta-analysis. Scand. J. TraumaResusc. Emerg. Med. 2018;26:56. doi: 10.1186/s13049-018-0527-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Tan T.L., Tang Y.J., Ching L.J., Abdullah N., Neoh H.M. Comparison of Prognostic Accuracy of the quick Sepsis-Related Organ Failure Assessment between Short- & Long-term Mortality in Patients Presenting Outside of the Intensive Care Unit—A Systematic Review & Meta-analysis. Sci. Rep. 2018;8:16698. doi: 10.1038/s41598-018-35144-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Smith G.B., Prytherch D.R., Meredith P., Schmidt P.E., Featherstone P.I. The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation. 2013;84:465–470. doi: 10.1016/j.resuscitation.2012.12.016. [DOI] [PubMed] [Google Scholar]
- 16.Subbe C.P., Kruger M., Rutherford P., Gemmel L. Validation of a modified Early Warning Score in medical admissions. QJM Mon. J. Assoc. Physicians. 2001;94:521–526. doi: 10.1093/qjmed/94.10.521. [DOI] [PubMed] [Google Scholar]
- 17.Cildir E., Bulut M., Akalin H., Kocabas E., Ocakoglu G., Aydin S.A. Evaluation of the modified MEDS, MEWS score and Charlson comorbidity index in patients with community acquired sepsis in the emergency department. Intern. Emerg. Med. 2013;8:255–260. doi: 10.1007/s11739-012-0890-x. [DOI] [PubMed] [Google Scholar]
- 18.Corfield A.R., Lees F., Zealley I., Houston G., Dickie S., Ward K., McGuffie C. Utility of a single early warning score in patients with sepsis in the emergency department. Emerg. Med. J. EMJ. 2014;31:482–487. doi: 10.1136/emermed-2012-202186. [DOI] [PubMed] [Google Scholar]
- 19.Geier F., Popp S., Greve Y., Achterberg A., Glockner E., Ziegler R., Heppner H.J., Mang H., Christ M. Severity illness scoring systems for early identification and prediction of in-hospital mortality in patients with suspected sepsis presenting to the emergency department. Wien. Klin. Wochenschr. 2013;125:508–515. doi: 10.1007/s00508-013-0407-2. [DOI] [PubMed] [Google Scholar]
- 20.Vorwerk C., Loryman B., Coats T.J., Stephenson J.A., Gray L.D., Reddy G., Florence L., Butler N. Prediction of mortality in adult emergency department patients with sepsis. Emerg. Med. J. EMJ. 2009;26:254–258. doi: 10.1136/emj.2007.053298. [DOI] [PubMed] [Google Scholar]
- 21.Hamilton F., Arnold D., Baird A., Albur M., Whiting P. Early Warning Scores do not accurately predict mortality in sepsis: A meta-analysis and systematic review of the literature. J. Infect. 2018;76:241–248. doi: 10.1016/j.jinf.2018.01.002. [DOI] [PubMed] [Google Scholar]
- 22.Gupta A., Liu T., Shepherd S., Paiva W. Using Statistical and Machine Learning Methods to Evaluate the Prognostic Accuracy of SIRS and qSOFA. Healthc. Inform. Res. 2018;24:139–147. doi: 10.4258/hir.2018.24.2.139. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Kia A., Timsina P., Joshi H.N., Klang E., Gupta R.R., Freeman R.M., Reich D.L., Tomlinson M.S., Dudley J.T., Kohli-Seth R., et al. MEWS++: Enhancing the Prediction of Clinical Deterioration in Admitted Patients through a Machine Learning Model. J. Clin. Med. 2020:9. doi: 10.3390/jcm9020343. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Ong M.E., Lee Ng C.H., Goh K., Liu N., Koh Z.X., Shahidah N., Zhang T.T., Fook-Chong S., Lin Z. Prediction of cardiac arrest in critically ill patients presenting to the emergency department using a machine learning score incorporating heart rate variability compared with the modified early warning score. Crit. Care. 2012;16:R108. doi: 10.1186/cc11396. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Vieira S.M., Mendonca L.F., Farinha G.J., Sousa J.M.C. Modified binary PSO for feature selection using SVM applied to mortality prediction of septic patients. Appl. Soft. Comput. 2013;13:3494–3504. doi: 10.1016/j.asoc.2013.03.021. [DOI] [Google Scholar]
- 26.Kim J., Chang H., Kim D., Jang D.H., Park I., Kim K. Machine learning for prediction of septic shock at initial triage in emergency department. J. Crit. Care. 2020;55:163–170. doi: 10.1016/j.jcrc.2019.09.024. [DOI] [PubMed] [Google Scholar]
- 27.Horng S., Sontag D.A., Halpern Y., Jernite Y., Shapiro N.I., Nathanson L.A. Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning. PLoS ONE. 2017;12:e0174708. doi: 10.1371/journal.pone.0174708. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Delahanty R.J., Alvarez J., Flynn L.M., Sherwin R.L., Jones S.S. Development and Evaluation of a Machine Learning Model for the Early Identification of Patients at Risk for Sepsis. Ann. Emerg. Med. 2019;73:334–344. doi: 10.1016/j.annemergmed.2018.11.036. [DOI] [PubMed] [Google Scholar]
- 29.Rodriguez R.M., Greenwood J.C., Nuckton T.J., Darger B., Shofer F.S., Troeger D., Jung S.Y., Speich K.G., Valencia J., Kilgannon J.H., et al. Comparison of qSOFA with current emergency department tools for screening of patients with sepsis for critical illness. Emerg. Med. J. EMJ. 2018;35:350–356. doi: 10.1136/emermed-2017-207383. [DOI] [PubMed] [Google Scholar]
- 30.Kwon J.M., Lee Y., Lee Y., Lee S., Park J. An Algorithm Based on Deep Learning for Predicting In-Hospital Cardiac Arrest. J. Am. Heart Assoc. 2018:7. doi: 10.1161/JAHA.118.008678. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Donnelly J.P., Safford M.M., Shapiro N.I., Baddley J.W., Wang H.E. Application of the Third International Consensus Definitions for Sepsis (Sepsis-3) Classification: A retrospective population-based cohort study. Lancet. 2017;17:661–670. doi: 10.1016/S1473-3099(17)30117-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Nannan Panday R.S., Minderhoud T.C., Alam N., Nanayakkara P.W.B. Prognostic value of early warning scores in the emergency department (ED) and acute medical unit (AMU): A narrative review. Eur. J. Intern. Med. 2017;45:20–31. doi: 10.1016/j.ejim.2017.09.027. [DOI] [PubMed] [Google Scholar]
- 33.DeLong E.R., DeLong D.M., Clarke-Pearson D.L. Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics. 1988;44:837–845. doi: 10.2307/2531595. [DOI] [PubMed] [Google Scholar]
- 34.Fernandes M., Vieira S.M., Leite F., Palos C., Finkelstein S., Sousa J.M.C. Clinical Decision Support Systems for Triage in the Emergency Department using Intelligent Systems: A Review. Artif. Intell. Med. 2020;102:101762. doi: 10.1016/j.artmed.2019.101762. [DOI] [PubMed] [Google Scholar]
- 35.Raita Y., Goto T., Faridi M.K., Brown D.F.M., Camargo C.A., Jr., Hasegawa K. Emergency department triage prediction of clinical outcomes using machine learning models. Crit. Care. 2019;23:64. doi: 10.1186/s13054-019-2351-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Choi S.W., Ko T., Hong K.J., Kim K.H. Machine Learning-Based Prediction of Korean Triage and Acuity Scale Level in Emergency Department Patients. Healthc. Inform. Res. 2019;25:305–312. doi: 10.4258/hir.2019.25.4.305. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Hong W.S., Haimovich A.D., Taylor R.A. Predicting hospital admission at emergency department triage using machine learning. PLoS ONE. 2018;13:e0201016. doi: 10.1371/journal.pone.0201016. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Taylor R.A., Pare J.R., Venkatesh A.K., Mowafi H., Melnick E.R., Fleischman W., Hall M.K. Prediction of In-hospital Mortality in Emergency Department Patients with Sepsis: A Local Big Data-Driven, Machine Learning Approach. Acad. Emerg. Med. Off. J. Soc. Acad. Emerg. Med. 2016;23:269–278. doi: 10.1111/acem.12876. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Perng J.W., Kao I.H., Kung C.T., Hung S.C., Lai Y.H., Su C.M. Mortality Prediction of Septic Patients in the Emergency Department Based on Machine Learning. J. Clin. Med. 2019:8. doi: 10.3390/jcm8111906. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Rhodes A., Evans L.E., Alhazzani W., Levy M.M., Antonelli M., Ferrer R., Kumar A., Sevransky J.E., Sprung C.L., Nunnally M.E., et al. Surviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock: 2016. Intensive Care Med. 2017;43:304–377. doi: 10.1007/s00134-017-4683-6. [DOI] [PubMed] [Google Scholar]
- 41.Cho K.J., Kwon O., Kwon J.M., Lee Y., Park H., Jeon K.H., Kim K.H., Park J., Oh B.H. Detecting Patient Deterioration Using Artificial Intelligence in a Rapid Response System. Crit. Care Med. 2020 doi: 10.1097/CCM.0000000000004236. [DOI] [PubMed] [Google Scholar]
- 42.Cortes-Puch I., Hartog C.S. Opening the Debate on the New Sepsis Definition Change Is Not Necessarily Progress: Revision of the Sepsis Definition Should Be Based on New Scientific Insights. Am. J. Respir. Crit. Care Med. 2016;194:16–18. doi: 10.1164/rccm.201604-0734ED. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Dettori J.R., Norvell D.C. The Anatomy of Data. Glob. Spine J. 2018;8:311–313. doi: 10.1177/2192568217746998. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Lo R.S.L., Leung L.Y., Brabrand M., Yeung C.Y., Chan S.Y., Lam C.C.Y., Hung K.K.C., Graham C.A. qSOFA is a Poor Predictor of Short-Term Mortality in All Patients: A Systematic Review of 410,000 Patients. J. Clin. Med. 2019:8. doi: 10.3390/jcm8010061. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Serafim R., Gomes J.A., Salluh J., Povoa P. A Comparison of the Quick-SOFA and Systemic Inflammatory Response Syndrome Criteria for the Diagnosis of Sepsis and Prediction of Mortality: A Systematic Review and Meta-Analysis. Chest. 2018;153:646–655. doi: 10.1016/j.chest.2017.12.015. [DOI] [PubMed] [Google Scholar]
- 46.Fleuren L.M., Klausch T.L.T., Zwager C.L., Schoonmade L.J., Guo T., Roggeveen L.F., Swart E.L., Girbes A.R.J., Thoral P., Ercole A., et al. Machine learning for the prediction of sepsis: A systematic review and meta-analysis of diagnostic test accuracy. Intensive Care Med. 2020 doi: 10.1007/s00134-019-05872-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.