Temporal Differential Expression of Physiomarkers Predicts Sepsis in Critically Ill Adults

Abstract

Background:

Sepsis is a life-threatening condition with high mortality rates. Early detection and treatment are critical to improving outcomes. Our primary objective was to develop artificial intelligence capable of predicting sepsis earlier using a minimal set of streaming physiological data in real-time.

Methods and Findings:

A total of 29,552 adult patients were admitted to the intensive care unit across five regional hospitals in Memphis, TN over 18 months from January 2017 to July 2018. From these, 5,958 patients were selected after filtering for continuous (minute-by-minute) physiological data availability. A total of 617 (10.4%) patients were identified as sepsis cases, using the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) criteria. Physiomarkers, a set of signal processing features, were derived from five physiological data streams including heart rate, respiratory rate, and blood pressure (systolic, diastolic, and mean), captured every minute from the bedside monitors. A support vector machine (SVM) classifier was used for classification. The model accurately predicted sepsis up to a mean and 95% confidence interval of 17.4 $\pm$0.22 hours before sepsis onset, with an average test accuracy of 83.0% (average sensitivity, specificity, and area under the receiver operating characteristics curve of 0.757, 0.902, and 0.781, respectively).

Conclusions:

This study demonstrates that salient physiomarkers derived from continuous bedside monitoring are temporally and differentially expressed in septic patients. Using this information, minimalistic artificial intelligence models can be developed to predict sepsis earlier in critically ill patients.

Introduction

Sepsis is among the leading causes of death in the hospital. Delays in treatment can significantly increase the risk for multi-organ dysfunction and ultimately death (1). Critically ill patients admitted to the intensive care unit (ICU) are monitored for acute physiological deterioration by bedside monitors that produce substantial amounts of data (2), the analysis of which can reveal important ‘physiomarkers’ that predict the onset of sepsis in children and adults (3,4).

Recent research has resulted in a number of predictive algorithms based on electronic medical record data to identify patients at risk for sepsis (5–9). However, these methods often employ time-delayed variables that can take several hours to become available for clinical decision making (10). These time-delayed variables, while effective in retrospective sepsis analyses, present challenges when applied to aid real-time decision support at the bedside. In this retrospective study, we developed a predictive model to identify the onset of sepsis using a minimal set of automatically captured continuous physiological sensor data. We implemented the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) criteria to identify sepsis cases and then used machine learning methods to extract features and apply a Support Vector Machine (SVM) classifier to predict the onset of sepsis earlier than clinical suspicion.

This paper presents one of the first artificial intelligence-based approaches developed using only continuous physiological signals for the prediction of sepsis in critically ill adults. Furthermore, this paper highlights that physiomarkers are both temporally and differentially expressed in critically ill septic patients.

Materials and Methods

In this paper, we implemented a three-step approach to model development and validation. In the first step, we select the optimal machine learning algorithm from a series of binary classifiers. We then perform variable importance analysis to select the optimal number of features where different feature thresholds were investigated to maximize the discriminatory performance between sepsis cases and control cases. In the third step, we select the optimal classification model based on a modified reward function that favors an earlier prediction of sepsis.

Cohort

Continuous physiologic data were collected on 29,552 adult subjects (comprising of 39,185 different patient encounters) admitted to the surgical, medical, neurological, and cardiac ICUs throughout their stay at the Methodist Le Bonheur Healthcare System, Memphis, Tennessee between 1/2017 through 7/2018. Continuous data capture was available in the 58-bed medical ICUs, distributed across five physical locations, therefore of all eligible subjects who were admitted to the medical ICU and met the data availability criteria were included in the study (n=5,958). Of these, a total of 617 subjects met sepsis criteria by the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) criteria (11). The eligible patient chart is summarized in Fig. 1.

Fig. 1:

Consort flow diagram of the patient population

We captured heart rate, respiratory rate, and blood pressures (mean; diastolic; systolic) from the bedside monitor once every minute. In patients without an arterial line, we captured blood pressure measurements obtained from a blood pressure cuff. From this sizable dataset, we derived a number of ‘physiomarkers’ (features described below) that were then fed into classifiers for sepsis prediction. Demographic and clinical data on cases and controls were collected from the electronic medical record using Cerner’s Web Intelligence reporting module.

We applied the Sepsis-3 definition, into a working set of rules based on the work of Nemati et al. (6) to determine patients who met sepsis criteria and also to identify the likely time of sepsis onset. To that effect, we calculated the Sequential Organ Failure Assessment (SOFA) score (11), incorporating clinical and laboratory values, such as creatine, platelet count, and vasopressor dosage. We applied a serial implementation of the SOFA to continuously calculate scores with each update of the streaming blood pressure. The time of initial blood culture or antibiotics administration, whichever came first, was used to define the time of sepsis suspicion (t_suspicion). The time of organ failure (t_onset), was defined as an increase of >2 in the SOFA score at any time up to 24 hours before to 12 hours after t_suspicion. The earlier time between t_suspicion and t_onset was used to define sepsis onset (t_sepsis) (Fig. 2). We hypothesized that, in critically ill patients, physiomarkers identified by machine learning methods could predict the onset of sepsis earlier than the Sepsis-3 definitions’ criteria (t_sepsis).

Fig. 2:

Over the length of a patient’s ICU stay, all timestamps of body fluid cultures and intravenous antibiotic administration were retrieved. The time of initial blood culture or antibiotics administration, whichever came first, was used to define the time of sepsis suspicion (t_Suspicion). If antibiotics were given first, then the cultures must have been obtained within 24 hours. If cultures were obtained first, then antibiotic must have been subsequently ordered within 72 hours. The time of sepsis (t_Sepsis), was defined as an increase of at least 2 in the SOFA score at any time up to 24 hours before to 12 hours after t_Suspicion

This study was performed in compliance with the ethical principles for medical research involving human subjects from the Declaration of Helsinki and was reviewed by the Institutional Review Board at the University of Tennessee Health Science Center, which granted a waiver of written informed consent. All data used in this study was de-identified prior to the analysis.

Methodology

Fig. S1, Supplemental Digital Content 1 provides an overview of our methodology. The original dataset was randomly split into 70% and 30% sets by preserving the class distribution (stratified data partitioning). The 70% was used for model training, while 30% was set aside for a hidden test set and used to report performance statistics. Controls were balanced with cases by using a bootstrap method and randomized across each unique iteration to minimize selection bias. Event times for controls, who did not develop sepsis, were selected at random, provided that at least 24 hours of data preceded the selected timestamp. For both the sepsis and control groups, data was temporally aligned (time-series data split into 2-hour windows starting from 1–23 hours before t_sepsis, or an arbitrary event time (see above) in controls. The sample distribution (training and test sets) for the datasets are given in Table S2, Supplemental Digital Content 2. We extracted a total of 213 physiomarkers using time-frequency domain methods from each of the five physiological data streams, resulting in a total of 1,065 features (see Table S3, Supplemental Digital Content 3) (12). Subsets of these features were then used as input to a series of binary classifiers, such as the random forest, SVM and k-nearest neighbor methods to first identify the optimal machine learning algorithm. The optimal algorithm was selected based on the average five-fold cross-validation performance measures across all temporally aligned datasets (See Text S4, Supplemental Digital Content 4 and results on the classification methods for details). Following the selection of the optimal algorithm, we performed variable importance analysis on the training CV folds to rank the features from most important to least important. To select the optimal number of features included in the trained model, different feature thresholds were investigated to maximize the discriminatory performance between sepsis cases and control cases in the test fold.

To select an optimal model, we developed a method to evaluate model performance in various temporal intervals using a bias factor to reward earlier sepsis predictions, while balancing the overall performance as measured by the F1 Score. We used the F1 metric to adjust for the tradeoff between precision and recall. Hence, for every trained model, the F1 score obtained for each temporal interval’s internal validation set was weighted by a factor of $(1+n*x)$, where n {0%, 5%, 8%, 10%, and 15%} is a bias factor of various magnitudes and $x$ is the number of hours before the definition of sepsis-3 is met. We then summed these weighted F1 scores and normalized them to obtain a metric for model performance evaluation for each bias factor, which we refer to as “normalized weighted F1 score” (See Table S5, Supplemental Digital Content 5).

Measures

Accuracy is the ratio of correctly predicted samples to the total number of test samples. Sensitivity (recall) is the proportion of true positives that are predicted as positive. Specificity is the proportion of true negatives which are predicted as negatives. The F1 score is the harmonic mean of precision and recall. Positive predictive value (PPV) is the proportion of correctly predicted sepsis patients from the total number of sepsis-predicted patients.

Statistical Analysis and machine learning framework

Python Scikit-learn machine learning library (13) was used for calculating descriptive statistical measures, feature extraction, feature selection and building machine learning classifiers. R programming language (14) was used for generating the temporal characteristics heat map. Bootstrap and Bayesian bootstrap (adjusted for weights) were used to assess the predictability of the features generated for predicting sepsis (15). A t-test was performed for determining statistical significance among the demographic characteristics of the dataset.

Results

The incidence of sepsis in this cohort was 10.4%. Age and gender did not differ between sepsis cases and controls, while cases differed from controls by race and ICU-length of stay (Table 1). Cases had an approximately 6-fold increase in mortality compared to controls.

Table 1:

Characteristics of the Study Population

		Sepsis
Characteristics	Overall	Yes	No
Patient, n (%)	5,958 (100)	617 (10.4)	5,341 (89.6)
Male, n (%)	2,932 (49.2)	299 (48.5)	2,633 (49.3)
Mechanical ventilation, n (%)	1,356 (22.8)	490 (74.6)	896 (16.8)
In hospital deaths, n (%)	439 (7.4)	176 (28.5)**	263 (4.9)
Age (yr.) median (IQR)	61.6 (49.9 – 71.8)	63.2 (52.3 – 71.5)	61.5 (49.6 – 71.8)
ICU LOS (d), median (IQR)	4.7 (2.8 – 8.4)	11.2 (6.3 – 19.5)**	4.3 (2.6 – 7.4)
ICU LOS >= 7d, n (%)	1,917 (32.2)	435 (70.5)**	1,482 (27.7)
APACHE II, median (IQR)	8 (5 – 12)	13 ( 9 – 20)	7 (4 – 11)
Self-reported race, n (%)
Black or African American	3,453 (58.0)	387 (62.7)*	3,066 (57.4)
White	2,360 (39.6)	214 (34.7)	2,146 (40.2)
Other/Unknown	104 (1.8)	13 (2.1)	91 (1.7)
Multiple	21 (0.4)	1 (0.2)	20 (0.4)
Asian	19 (0.3)	2 (0.3)	17 (0.3)
Self-reported Ethnicity, n (%)
Not Hispanic or Latino	5,847 (98.2)	605 (98.0)	5,242 (98.2)
Hispanic or Latino	69 (1.2)	8 (1.4)	69 (1.3)
Unknown or Declined	33 (0.6)	4 (0.6)	29 (0.5)

^*Significant at a = 0.01

^**Significant at a = 0.001; LOS: Length of Stay; IQR: Inter Quartile Range

Temporal Expression Profile of the data streams

To illustrate the temporal characteristics identified by the models during the development of sepsis, we analyzed the importance of the physiomarkers using the top 30 features that were identified from a variable importance analysis using the Random Forests Classifier. Fig. 3 illustrates a heatmap of the top 30 ‘physiomarkers’ categorized into four feature groups, for the five data streams from temporally aligned datasets leading to sepsis. The color intensity assigned signals a higher importance for the physiomarkers in the corresponding feature group. Notably, blood pressure played a role in the last hours prior to t_sepsis whereas, heart rate (signal processing) showed the sustained trend as early as 5 hours prior to t_sepsis. Descriptive statistics (See Table S6, Supplemental Digital Content 6) features of heart rate played a role much earlier, as far as 21 hours from t_sepsis, consistent with other studies that suggest an early role that heart rate modulation plays in the development of sepsis (16,17). Finally, within the respiratory rate data stream, ‘change statistics,’ and ‘signal processing’ measures (See Table S6, Supplemental Digital Content 6) suggested the sustained importance of respiratory rate from 10 hours until 1 hour prior to t_sepsis. The list of features for each feature category is given in Table S6, Supplemental Digital Content 6.

Fig. 3: Feature importance in a number of time intervals leading to t_sepsis.

HR: Heart Rate; RR: Respiratory Rate; SBP: Systolic Blood Pressure; DBP: Diastolic Blood Pressure; MBP: Mean Blood Pressure; Color intensity value represents the frequency of features present in each feature category at a particular time period.

Selecting the optimal algorithm

In general, the average k-fold accuracy of SVM was higher than that of random forest, and k-nearest neighbor (Table 2). The SVM model, on the internal validation dataset, achieved an overall five-fold cross-validation accuracy of 77.9% with average sensitivity, specificity, and PPV of 73.8%, 82.0%, and 80.5% respectively. Therefore, we selected the support vector machine algorithm for further analysis.

Table 2:

Average five-fold cross-validation performance measures across all temporally aligned dataset for various machine learning models.

Model	Accuracy [95% CI]	Sensitivity [95% CI]	Specificity [95% CI]	PPV [95% CI]	F1-Score [95% CI]
SVM	77.9 [72.1, 83.7]	73.8 [63.3, 84.4]	82.0 [73.9, 90.0]	80.5 [73.6, 87.4]	76.9 [70.1, 83.6]
RF	76.8 [69.4, 84.2]	73.9 [62.7, 85.2]	79.6 [70.6, 88.7]	78.8 [70.7, 87.0]	76.0 [67.7, 84.3]
kNN	73.8 [67.2, 80.5]	67.8 [55.8, 79.8]	79.9 [71.9, 87.9]	77.3 [69.9, 84.7]	72.0 [63.8, 80.3]

CV: Cross validation, CI: Confidence Interval, PPV: Positive predictive value, SVM: Support Vector Machine, RF: Random Forest, kNN: k-nearest neighbor

We next performed a grid search for the optimal temporal alignment using the selection criterion outlined earlier. Fig. 4 illustrates the maximum normalized weighted F1 score obtained for each time-window leading to t_sepsis. The error bar represents the 95% CI using the various weighted factors. An overall maximum normalized weighted F1 score of 0.718 was obtained for the SVM model trained on data in the observation period 16–18 hours before the definition of sepsis-3 is met when using the top 70 features.

Fig. 4: Maximum normalized weighted F1 score to identify the optimal model for all time intervals leading to t_sepsis.

The error line on the plot illustrates the average and 95% CI between each of the bias functions. The 16h model demonstrates the highest score with minimal errors.

Predictive Power of the Optimal Model

The average test performance metrics using optimal time-period model are given in Table S7, Supplemental Digital Content 7. The SVM model achieved an average test accuracy of 83.0% (95% Confidence Interval (CI): [79.7, 86.3]) with average test sensitivity and test specificity of 75.7% (95% CI: [71.2, 80.1]) and 90.2% (95% CI: [84.5, 95.9]) respectively. The PPV and F1-Score of the balanced test set were 88.7 (95% CI: [82.9, 94.4]) and 81.6% (95% CI: [78.2, 85.1]) respectively.

Fig. 5 illustrates the mean receiver operating characteristic curve with 95% confidence intervals for the optimal model identified, applied to all time intervals for the test set. In addition, the corresponding mean area under the curve (AUC) with 95% CI equals 0.78 $\pm$ 0.01, which illustrates that the model can distinguish between two the diagnostic groups (sepsis/control) with good predictive accuracy.

Fig. 5: Receiver operating characteristic curve with 95% confidence intervals (CI) for the optimal model for all time intervals for the test set.

Also, the mean AUC with 95% CI is illustrated.

Discussion

The key to the construction of accurate machine learning models from physiologic data is the identification of the minimal set of features that are predictive of the event. In this paper, we highlight an automated approach to capture and analyze physiological data streams to predict sepsis in critically ill ICU patients in a large hospital system in the mid-South, USA. While a number of sepsis prediction algorithms have been recently proposed (5,6,9), many features in those models rely on time-delayed laboratory values, in addition to potential human errors introduced in manual data entry. Therefore, an alternative, automated approach, may overcome those limitations. Following the use of a systemic model building approach, a thorough parameter tuning exercise and rigorous validation of the model parameters, we found that sepsis patients were, on average, predicted 17.4 $\pm$ 0.22 hours before t_sepsis. This holds practical value to alert clinicians about the patient’s highly critical status.

The analysis of the importance of data from various time intervals away from t_Sepsis (Fig. 3) suggests an intricate interrelationship between heart rate, blood pressure, and respiratory rate. Notably, heart rate characteristics, such as those derived using signal complexity, time-frequency components and simple descriptive methods were expressed consistently throughout all analysis intervals up to 21 hours before t_sepsis. Marked changes in blood pressures, especially the mean arterial pressure, was observed up to 6 hours earlier. Respiratory rate changes were observed consistently during the intermediate phases of 1 – 16 hours before t_sepsis.

In addition to developing a series of models on temporally aligned datasets to better understand the differences in the parameters used in the corresponding models, we also apply a novel criterion to select the optimal model that predict sepsis earlier and more consistently when applied in an online fashion. We explored various bias factors to evaluate the optimal model using a normalized weighted F1 score (Fig. 4). Our selection criteria consistently selected the 16h model as the optimal model, demonstrating the highest average normalized weighted F1 score with the lowest 95% CI.

To assess the generalizability and validity of the approach, we apply the optimal model in a real-time fashion. The model illustrates good performance in its ability to predict patients who eventually met sepsis criteria. This is evident by the high number of sepsis patients correctly identified out of the total number of sepsis patients, shown in Fig. 6(A) as well as the low number of missed sepsis alarms from the total number of alarms illustrated in Fig. 6(B). In addition, Fig. 6(A) presents the cumulative percentage of correctly identified sepsis patients over time, as part of a retrospective ‘what if’ analysis assuming that the model was applied in an ‘online fashion.’ From the figure, it is seen that a high percentage of patients is correctly predicted throughout the observation period. Furthermore, it is noted that this high percentage of correctly predicted sepsis patients is obtained at a time much earlier than when the sepsis-3 definition is met.

Fig. 6: Temporal model performance and alarms for sepsis and controls.

A) Number of sepsis patients correctly identified out of the total number of sepsis patients, and cumulative percentage of correctly identified sepsis patients over time B) number of missed sepsis alarms from the total number of alarms C) Number of controls correctly identified out of the total number of sepsis controls. D) Number of false alarms from the total alarms.

Furthermore, Fig. 6(C) and 6(D) illustrate that the model does not generate an excessive number of false alarms when distinguishing between sepsis and controls. Lastly, the high value of AUC with very narrow CI width, i.e., 0.78 $\pm$ 0.01, suggests that the optimal model identified has high predictive power.

There are some limitations to our work. This dataset was collected from critically ill patients admitted from the ward to the medical ICUs, across a regional hospital network in the Mid-South, USA, therefore the results may not be widely generalizable. Second, the number of data streams are limited by the monitoring capabilities of the unit, resulting in only 20% of all ICU patients being eligible for analysis. This may indicate that our model is valid among the high acuity group of patients, in whom continuous physiological streams are consistently collected. Third, due to the lack of publicly available external data sources with similar data collection frequency (minute-by-minute vital signs), we were unable to externally validate this model. Fourth, sepsis in patients was developed after admission to the ICU, therefore this model may not apply to sepsis developed in the ward. Fifth, there is an inherent difference in how the data for identifying sepsis cases, control, and t_onset are collected and used versus how the data for the machine learning was collected and used. Sixth, while, most of the vitals were automatically abstracted from the bedside monitor, the continuous body temperature was not always recorded by the bedside monitoring and could not be included in this study. Finally, while this approach indicates that the onset of sepsis can be identified earlier, this study does not address results pertaining to the effectiveness of earlier response. Therefore, additional studies need to be conducted to fully elucidate the interactions between observed physiological features and their underlying biological processes.

The application of artificial intelligence can greatly aid clinicians in identifying critically ill patients likely to develop sepsis. Earlier recognition may lead to earlier therapy, which may improve overall outcomes, although this remains to be seen. In this paper, we focus on the development of a model using only continuous physiological data, absence of laboratory or clinical values. This work suggests that models that use higher-frequency physiological data streams can independently predict the onset of sepsis in the medical ICU up to 17 hours earlier. Further work is required to validate the generalizability of this model using external sites.