OnAI-Comp: An Online AI Experts Competing Framework for Early Sepsis Detection

Abstract

Sepsis has always been a main public concern due to its high mortality, morbidity, and financial cost. There are many existing works of early sepsis prediction using different machine learning models to mitigate the outcomes brought by sepsis. In the practical scenario, the dataset grows dynamically as new patients visit the hospital. Most existing models, being “offline” models and having used retrospective observational data, cannot be updated and improved using the new data. Incorporating the new data to improve the offline models requires retraining the model, which is very computationally expensive. To solve the challenge mentioned above, we propose an Online Artificial Intelligence Experts Competing Framework (OnAI-Comp) for early sepsis detection using an online learning algorithm called Multi-armed Bandit. We selected several machine learning models as the artificial intelligence experts and used average regret to evaluate the performance of our model. The experimental analysis demonstrated that our model would converge to the optimal strategy in the long run. Meanwhile, our model can provide clinically interpretable predictions using existing local interpretable model-agnostic explanation technologies, which can aid clinicians in making decisions and might improve the probability of survival.

1. Introduction

1.1. Backgrounds

SEPSIS is a significant cause of mortality and morbidity among critically ill patients admitted to the intensive care unit (ICU) [1], [2]. Sepsis is “life-threatening organ dysfunction” which will occur due to dysfunctional host response to infection-causing organ failure, tissue damage, or death [3], [4]. A recent epidemiological report suggests that up to a fifth of all global deaths in 2017 were attributed to sepsis, with the heaviest burden falling amongst low-resourced countries [5]. The majority of the deaths reported were immunocompromised, children under the age of 5, and the elderly. Sepsis has long been associated with disparities of health, including among the racial and socioeconomically disadvantaged [6].

In the United States, the Centers for Disease Control and Prevention¹ estimates that there are at least 1.7 million adults who develop sepsis in the United States each year. It is reported that one-third of the people who die in a hospital have sepsis. Moreover, the financial costs caused by sepsis have risen significantly. From 2012 to 2018, the total number of fee-for-service beneficiaries with an inpatient hospital admission (IHA) related to sepsis rose from 811,644 to 1,136,889 [7]. Thus, sepsis remains a common and expensive life-threatening condition with significant mortality for beneficiaries.

1.2. Challenges

Machine learning methods have long been used to improve predictions for sepsis [8], [9]. Indeed, a recent international challenge [4] was conducted to improve modeling strategies and prediction times before clinical recognition of sepsis. A major source of data for sepsis prediction has come from structured data derived from electronic medical records (EMR) [10]–[12], bed-side monitoring [13]–[15], or cytokine/biomarker data [16], [17].

In Ref. [18], S. Nemati et al. proposed an interpretable machine learning model based on Artificial Intelligence Sepsis Expert (AISE) for Sepsis prediction in ICUs. In Ref. [19], Q. Mao et al. applied a machine learning-based algorithm using only six vital signs to detect and predict three sepsis-related gold standards. Their experiments were based on a mixed-ward retrospective dataset. M. Yang et al. [20], [21] proposed an Explainable Artificial-intelligence Sepsis Predictor (EASP) applying XGBoost Learning and Bayesian Optimization, which had won first place in the PhysioNet/Computing in Cardiology Challenge 2019 [4]. In Ref. [22], T. Vicar et al. solved the sepsis prediction problem using a recurrent neural network (RNN).

While these approaches incorporate rich clinical and physiological aspects of the disorder, most of these algorithms use retrospective observational data [23]. In other words, the events to be studied have already occurred before the data collection starts, neglecting valuable information from the new data. In addition, retraining the model with a new dataset will lead to high computational costs. As a result, we need a novel methodology that takes advantage of the new data and avoids retraining the model. One opportunity may rest in integrating online frameworks, such as the multi-armed bandit (MAB) algorithm to augment observational learning with interactive and adaptive training to solve the challenge mentioned above.

MAB is originally a particular type of Reinforcement Learning (RL) model. Due to its wide popularity, it has become a major field of interest. MAB extends RL by ignoring the state and more objectively tries to balance between exploration and exploitation. In general, the goal of RL is to maximize policy, while the optimal policies might not provide you best rewards in each instance.

1.3. Multi-armed Bandit Problems

The MAB problem (or Bandit problem) can be defined as [24]: “A sequential allocation problem defined by a set of actions. At each time step, a unit resource is allocated to an action and some observable payoff is obtained. The goal is to maximize the total payoff obtained in a sequence of allocations.” One interesting thing is that the original motivation to study MAB problems came from a clinical application in which one must decide which treatment policy to adopt for each patient [24], [25]. The payoff gained at each round (i.e., time step) is usually defined as reward. At each round, a parameter called regret is used to evaluate the performance of the algorithm. Specifically, regret is defined as the gap between the optimal reward and the actual reward gained.

Based on the assumptions of the reward (i.e., payoff), the fundamental formalization of MAB problems falls into three different categories: stochastic [26], adversarial [27], and Markovian [28] MAB. In this paper, we focus on the stochastic MAB problem. In a stochastic MAB setting, each arm is associated with an unknown probability distribution restricted on [0,1]. At each round, the reward is drawn independently associated with the selected arm. The stochastic MAB problem by nature has an exploration-exploitation dilemma. Specifically, the decision-maker needs to strike a balance between the exploration of alternative arms that might yield higher rewards and the exploitation of arms that seem to generate the highest rewards based on current empirical knowledge [24], [29], [30]. Upper Confidence Bound (UCB) [29], [31] strategy is one of the widely applied solutions for this dilemma, which can deal with exploration and exploitation synchronously.

1.4. Proposed Solutions

While most algorithms use retrospective observational data to construct models, in this work we apply pre-trained models in an online environment to evaluate performance in the context of predicting sepsis. This is important because, in acute and highly dynamic conditions such as sepsis, there may be significant variation in the clinical status of the patient. Therefore, a static model, which is trained on population level data might not fully capture the saliency in such a critical and highly complex state. In this work, we utilize MAB to model the process of a third trusted party selecting among a group of pretrained Machine Learning (ML) algorithms, which are called AI experts in the framework, i.e., the arms in MAB setting. The system infrastructure could be represented as shown in Fig. 1.

Fig. 1:

System Infrastructure.

As shown in Fig. 2, we model the sepsis prediction process as an online expert competing game. We have a list of candidate artificial intelligence (AI) experts which have been well-trained in prior. Those experts can evaluate the risk of developing sepsis based on the personal information of this patient (i.e., basic demographic information, vital signs monitored and recorded, fluid bolus administered, etc.). As aforementioned, the dataset dynamically accumulates as new patients arrive. Applying UCB in the online learning module, we can learn and update “on the fly” using new data while striking an exploitation-exploration balance (will be explained more in Eq. (4)).

Fig. 2:

System Overview.

Our contribution can be concluded as follows:

• We propose an online learning-based sepsis prediction framework, which can deal with dynamically increasing datasets and strike a balance between exploration and exploitation.

• Our experimental results show that our proposed methodology converges to the optimal strategy in the long run.

• Our model is not limited to sepsis prediction tasks and can be extended to other applications as long as all the experts and performance metrics are normalized appropriately.

2. Problem Formulation

For clarity, we first provide a brief introduction to the MAB problem in this section. In the most basic setting, a MAB problem (i.e., K-armed bandit problem) is defined by a series of random variables ${\text{𝓡}}_{i,n}(1\le i\le K,n\ge 1)$, in which $i$ is the unique ID of an AI expert² (i.e., an arm) and $n$ is the number of rounds. When arm $i$ is played successively at each round, it yields rewards $R_{i,1},R_{i,2},\cdots ,R_{i,T}$, where $T$ is total number of rounds.

In the basic setting, $R_{i,1},R_{i,2},\cdots ,R_{i,T}$ are always assumed to be independent and identically distributed (i.i.d) based on an unknown law of unknown expectation ${\xi }_i$ [29]. However, this is not consistent with the practical scenario, where the reward gained at round $t-1$ and $t$ from the same patient are probably correlated. For example, the cardio output recorded at the moment is usually correlated with the measurement recorded one hour ago. Therefore, as in [29], we do not expect the rewards for each arm to be i.i.d.. Instead, rewards only need to satisfy the following weaker assumption³.

Assumption 1.

For $\forall 1\le i\le K$ and $\forall 1\le t\le T$, we assume

$$\mathbb{E}\left[R_{i,t}\mid R_{i,1},R_{i,1},\cdots ,R_{i,t-1}\right]={\xi }_i\text{ and }{R}_{i,t}\in [0,1].$$ (1)

In other words, $R_{i,t}$ and $R_{j,s}$ might be dependent for $\forall 1\le i,j\le K$ and $\forall 1\le s,t\le T$. This very weak assumption is guaranteed in most applications. Since there is no collaboration among experts, decision made by individual experts will not impact the decisions of other experts.

3. Methods

3.1. System Overview

We can view the whole decision process as an AI expert competing game or an AI experts recommendation system. As shown in Fig. 2, the system is composed of the following components:

• Patient. Suppose that at each round $t$, we have a patient $P_t$ arriving at the hospital/expert recommendation system. We have $t=\{1,2,\cdots T\}$ and each $t$ is associated with an unique timestamp. We view the patient arriving at each round as a unique patient, even if the patient is the same person.

• AI Expert. An AI expert can be any trusted third party that provides ML as a Service (MLaS) aiming to help clinicians analyze EMRs. In this system, there are a group of AI experts ${𝓔}_K=\left\{E_1,E_2,\cdots ,E_K\right\}$, where $K$ is the number of total AI experts. Each AI expert is a well-trained offline model using the same training dataset.

• Trusted third party (TTP). A TTP can be a hospital that has hired a group of AI experts or clinicians to analyze EMRs.

In this work, we assume that all the objects in Fig. 2 is not malicious and will not give away the sensitive information of patients. However, in the practical scenario, other agents in the system might be malicious or might be attacked by malicious attackers. So, a TTP to provide input for all records during the training process is required.

3.2. Online AI Experts Competing Framework (OnAI-Comp)

Reward.

At the end of each round, the system calculates a reward based on the performance of AI experts, which is evaluated by a utility function. For a specific expert $E_k$, let $r_t$ be the reward calculated at round $t$ and $u^k(t)$ be the utility score achieved by AI expert $E_k$ at round $t$. We calculate $r_t$ as $r_t={\max }_{1\le k\le K}{u}^k(t)$, which is the maximum utility score received by AI experts at round $t$.

Let $t_{predict}$ be the specific round associated with the first hour during which $E_k$ reports sepsis, and $t_{sepsis}$ be the round associated with the first hour during which patient $P_t$ actually develops sepsis. If $t_{predict}\in \left[t_{sepsis}-{\text{Δ}}_{early},t_{sepsis}+{\text{Δ}}_{late}\right]$, expert $E_k$ is qualified for positive rewards. Herein, we set ${\text{Δ}}_{early}=12$,${\text{Δ}}_{late}=3$. ${\text{Δ}}_{early}$ and ${\text{Δ}}_{late}$ are based on the parameters provided by the Physionet sepsis challenge [4]. We used the same parameters so that we can apply the standard evaluation function provided by Physionet. There are four different scenarios: false negative (FN), false positive (FP), true positive (TP), and true negative (TN). Similar to [4], we define $u^k(t)$ as follows:

$$u^k(t)=\{\begin{array}{l}0\text{FN for sepsis patient}{P}_t\hfill \\ 0\text{FP for non-sepsis patient}{P}_t\hfill \\ 1\text{TN for non-sepsis patient}{P}_t\hfill \\ \frac{\left|t_{sepsis}-t_{predict}\right|}{{\text{Δ}}_{early}+{\text{Δ}}_{late}}\text{TP for sepsis patient}{P}_t\hfill \end{array}$$ (2)

Regret.

At each round, based on the afterwards feedback from the patients, the system calculates the difference between the optimal strategy and the selected strategy called regret. In this work, we evaluate the performance by average regret (AR). Specifically, we denote ${\text{Δ}}_T$ as the AR up till round $T$:

$${\text{Δ}}_T=\frac{1}{T}\sum _{t=1}^T\left(r_{optimal}-r_t\right),$$ (3)

where $r_{optimal}$ is the reward achieved by the expected optimal strategy. In this work, we formalize all the parameters so that $r_{optimal}=1$.

Algorithm 1.

OnAI-Comp.

Initialization:
$t\leftarrow 1$;
for $j=\{1,2,\cdots ,K\}$ do
${\widehat{\mu }}^j(t)\leftarrow 0$
end for
1:	procedure OnAI-Comp$\left(t,P_t,K,{𝓔}_K\right)$
2:	for $t=1,2,\cdots ,n$ do
3:	Patient data fed to each AI expert;
4:	All the AI experts provide their predictions;
5:	for $j=\{1,2,\cdots ,K\}$ do
6:	$u^j(t)\leftarrow \text{Get\_Utility}\left(t,P_t,E_j\right)$
7:	$\text{Update\_hat\_mu}\left(t,{\widehat{\mu }}^j(t),u^j(t)\right)$
8:	$\text{Update\_UCB}\left(t,\alpha ,{\widehat{\mu }}^j(t),N_j(t)\right)$
9:	$j\ast \leftarrow \text{arg}{\text{max}}_{m}UC{B}^m(t)$
10:	$N_{j\ast }(t)\leftarrow N_{j\ast }(t-1)+1$
11:	end for
12:	$r_t\leftarrow u^{j\ast }(t)$
13:	$\text{𝒩}(t)\leftarrow \left\{N_1(t),N_2(t),\cdots ,N_K(t)\right\}$
14:	$\tilde{\mu }(t)\leftarrow \left\{{\widehat{\mu }}^1(t),{\widehat{\mu }}^2(t),\cdots ,{\widehat{\mu }}^K(t)\right\}$
15:	Calculate AR based on Eq. (3)
16:	end for

Upper confidence bound (UCB).

For each round $t$, we keep a UCB for each expert given by:

$$UC{B}^j(t)={\widehat{\mu }}^j(t)+\sqrt{\frac{\alpha \log (t)}{N_j(t)}},$$

where ${\widehat{\mu }}^j(t)$ is the empirical average utility score achieved by expert $E_j,N_j(t)$ is the number of times expert $E_j$ has been selected up till round $t$, and $\alpha$ is a constant. For each round, the expert with the highest UCB is selected.

The first term in Eq. (4) is the average empirical utility score received by expert $\text{j}$ up till round $\text{t}$, which can evaluate the performance of expert $\text{j}$ based on exploitation of all historical data related to expert $\text{j}$. So, the first term is associated with the exploitation process, in which the experts with higher empirical performance (in terms of utility score) are selected.

The second term in Eq. (4) represents the uncertainty caused by exploration of expert $\text{j}$. Specifically, the less expert $\text{j}$ has been selected before round $\text{t}$ (i.e., the smaller $N_j(t)$ is), the more probable it will be selected at round $\text{t}$. At each round, the expert with the highest UCB value will be selected. So, the second term is related to the exploration process, i.e., the experts that have not been fully exploited are chosen. In this way, the system can strike a better balance between exploitation and exploration. By this way, we could strike a balance between exploitation and exploration. In addition, the information from the new data is utilized to calculate UCB values.

Algorithm 2.

Get_Utility.

1:	function Get_Utility$\left(t,P_t,E_j\right)$
2:	Calculate the utility score based on Eq. (2)
3:	Return $u^j(t)$

Algorithm 3.

Update_hat_mu.

1:	function Update_hat_mu$\left(t,{\widehat{\mu }}^j(t),u^j(t)\right)$
2:	${\widehat{\mu }}^j(t)\leftarrow \frac{(t-1){\widehat{\mu }}^j(t-1)+u^j(t)}{t}$

Workflow.

As shown in Alg. 1, the workflow can be split into following steps: (1) At each round, the TTP processes and feeds the information of the current patient to all AI experts. (2) The experts predict the risk of developing sepsis. (3) The TTP selects $E_{j\ast }$ (i.e., the optimal expert) and shares the predictions of $E_{j\ast }$ with clinicians to assist them with the final decision. (4) The TTP receives the true predicted labels to calculate utility scores and update all variables (i.e., $\text{𝒩}(t),\tilde{\mu }(t)$, UCBs). (5) The TTP calculates the reward and AR. Then, go back to step 1.

4. Regret Analysis

We denote ${\xi }_i$ as the expected reward received by expert $E_i$, ${\xi }^{\ast }={\max }_{1\le i\le K}{\xi }_i$, and $\alpha$ is the same constant as that in Eq. (4). Following the similar steps in [29], we have:

Lemma 1.

For $\forall 1\le i\le K$, the upper bound of the expected number of times expert $E_i$ is selected up till round $t$ is:

$$\mathbb{E}\left[N_i(t)\right]\le \frac{4\alpha \log (t)}{{\delta }_i^2}+1+\frac{{\pi }^2}{3},$$ (5)

where ${\delta }_i={\xi }^{\ast }-{\xi }_i$. Furthermore, we can rewrite the expected AR after $T$ rounds as:

$$\mathbb{E}\left[\sum _{i=1}^T{\text{Δ}}_i\right]=\sum _{j=1}^K{\delta }_j\mathbb{E}\left[N_j(T)\right].$$ (6)

Thus, we have:

Theorem 1 (AR Upper Bound).

For $\forall K>1$, the expected regret after $T$ rounds of OnAI-Comp is bounded as follows:

$$\mathbb{E}\left[\sum _{i=1}^T{\text{Δ}}_i\right]\le \left[4\alpha \sum _{j=1}^K\left(\frac{\log (T)}{{\delta }_j}\right)\right]+\left(1+\frac{{\pi }^2}{3}\right)\left(\sum _j^K{\delta }_j\right).$$ (7)

Theorem 1 shows that the bound of expected AR is depends on $T$ and ${\delta }_i$. Intuitively, the first term of Eq. (7) is caused by uncertainty of the exploration process, while the second term is due to regrets gained in the exploitation process.

Algorithm 4.

Update_UCB.

1:	function Update_UCB $\left(t,\alpha ,{\widehat{\mu }}^j(t),N_j(t)\right)$
2:	$UC{B}^j(t)\leftarrow {\widehat{\mu }}^j(t)+\sqrt{\frac{\alpha \log (t)}{N_j(t)}}$

5. Experimental Results

In the experiments, we use 2019 PhysioNet Computing in Cardiology Challenge dataset⁴ [4], in which sepsis is defined based on the Sepsis-3 guidelines [3]. It contains information on ICU patients from three hospital systems, where the data of each patient is stored separately in a .psv file⁵. The patient features (e.g., Demographics, Vital Signs, .etc.) are recorded on an hourly basis. After a patient leaves the ICU, the system will know if she/he has developed sepsis during the stay (on an hourly basis). Then, the system calculates the utility score based on Eq. (2), updates the corresponding parameters (i.e., UCB values, etc.), and adapts the strategy before next patient arrives.

In total, we have 40,336 patient records and more than 40 features. Four models, i.e., XGBoost⁶ (XGB) [20], [21], Support Vector Machine (SVC), Random Forest (RF) and Logistic Regression (LR) were trained as the AI experts. For fair comparison, we also included random guess (RG) as a dummy expert. We set $\alpha =2$ in Eq. (4). We have open sourced our code⁷.

Regret Analysis.

First, we applied OnAI-Comp using different combinations of AI experts. As shown in Fig. 4 (a) and (b), our proposed OnAI-Comp converges to the expected optimal strategy in the long run. There might be a “cold start” in the beginning, but it converges fast. Comparing the results of different expert combination XGB+RF+SVC and LR+RF+SVC (see Fig. 4 (b)), we can see LR+RF+SVC combination has lower AR. This might happen because XGB [20] (which gains the highest utility score in the challenge) is better trained than LR (which is from the Python library). Thus, when better trained AI experts are used, it is possible to get better results. We also demonstrate the average regret and cumulative regret in TABLE 1 and 2 respectively. As shown in TABLE 1, the model converges with the first 500 time steps no matter what expert combination is used.

Fig. 4:

Average Regret. (a) Average regret gained by different number of experts. (b) Average regret gained by different expert combinations .

TABLE 1:

Average Regret of Different Expert Combinations.

Experts	round = 10	round = 50	round = 100	round = 200	round = 500	round = 1000	round = 1500	round = 2000
XGB	0.1818	0.2431	0.1941	0.1821	0.1960	0.2080	0.2097	0.2105
RG+LR	0.131	0.123	0.1207	0.094	0.0892	0.0928	0.0915	0.0922
RG+SVC	0.1295	0.1019	0.1193	0.0961	0.0855	0.0931	0.0907	0.0906
RG+XGB	0.1974	0.1607	0.1317	0.1402	0.1633	0.1823	0.1876	0.1916
RG+RF	0.0521	0.1466	0.1003	0.0829	0.0778	0.0847	0.0861	0.0871
XGB+RF	0	0.0902	0.099	0.0826	0.0814	0.0885	0.0883	0.0898
XGB+LR	0	0.0902	0.099	0.0826	0.0814	0.0885	0.0883	0.0898
XGB+SVC	0	0.0902	0.099	0.0826	0.0814	0.0885	0.0881	0.0895
RF+LR	0	0.0980	0.1089	0.0896	0.0878	0.0949	0.0939	0.0945
RF+SVC	0	0.0980	0.1089	0.0896	0.0878	0.0949	0.0938	0.0944
LR+SVC	0	0.0980	0.1089	0.0896	0.0878	0.0949	0.0938	0.0944
RG+XGB+RF	0.1019	0.1654	0.1457	0.1123	0.0936	0.0918	0.089	0.0887
RG+XGB+LR	0.1126	0.1442	0.1227	0.0943	0.0811	0.0881	0.0872	0.0886
RG+XGB+SVC	0.0917	0.0963	0.0984	0.0874	0.0804	0.0869	0.0855	0.0865
RG+RF+SVC	0.1325	0.1443	0.1380	0.1000	0.0885	0.0947	0.0919	0.0915
RG+RF+LR	0 7380	0.0865	0.0965	0.0766	0.0824	0.087	0.0868	0.0886
RG+LR+SVC	0 9140	0.1308	0.1185	0.0891	0.0834	0.0897	0.0896	0.0911
XGB+RF+LR	0	0.1137	0.1149	0.0915	0.0870	0.0931	0.0913	0.0921
XGB+RF+SVC	0	0.1137	0.1149	0.0915	0.0870	0.0931	0.0915	0.0921
XGB+LR+SVC	0	0.1137	0.1149	0.0915	0.0870	0.0931	0.0915	0.0921
RF+LR+SVC	0	0.0980	0.1089	0.0896	0.0878	0.0949	0.0938	0.0944
RG+XGB+RF+SVC	0.1922	0.1819	0.1368	0.1065	0.0938	0.0939	0.0915	0.0918
RG+XGB+RF+LR	0.2026	0.1766	0.1438	0.1149	0.095	0.0965	0.0926	0.0926
RG+XGB+LR+SVC	0.1559	0.1653	0.1374	0.1068	0.0871	0.0923	0.0904	0.0904
RG+RF+LR+SVC	0.0762	0.1426	0.1402	0.1085	0.0955	0.0955	0.0944	0.0940
XGB+RF+LR+SVC	0	0.1176	0.1287	0.0985	0.0894	0.0937	0.0922	0.0927
RG+XGB+RF+LR+SVC	0.0322	0.1217	0.1283	0.1030	0.0898	0.0905	0.0879	0.0884

TABLE 2:

Cumulative Regret of Different Expert Combinations.

Experts	round = 10	round = 50	round = 100	round = 200	round = 500	round = 1000	round = 1500	round = 2000
XGB	2.000	12.4000	19.6000	36.6000	98.2000	208.2000	314.8000	421.2000
RG+LR	1.441	6.2746	12.1922	18.8946	44.6685	92.9025	137.2798	184.5152
RG+SVC	1.4247	5.1945	12.0459	19.3167	42.8533	93.1598	136.1482	181.3200
RG+XGB	2.1714	8.1932	13.2985	28.1738	81.7912	182.4437	281.5643	383.4118
RG+RF	0.5727	7.4768	10.1285	16.6660	38.9590	84.7379	129.1850	174.3307
XGB+RF	0	4.6000	10.0000	16.6000	40.8000	88.6000	132.6000	179.6000
XGB+LR	0	4.6000	10.000	16.6000	40.8000	88.6000	132.6000	179.6000
XGB+SVC	0	4.600	10.000	16.6000	40.8000	88.6000	132.2000	179.0000
RF+LR	0	5.0000	11.0000	18.0000	44.0000	95.0000	141.0000	189.0000
RF+SVC	0	5.0000	11.0000	18.0000	44.0000	95.0000	140.8000	188.8000
LR+SVC	0	5.0000	11.0000	18.0000	44.0000	95.0000	140.8000	188.8000
RG+XGB+RF	1.1208	8.4329	14.7169	22.5649	46.9066	91.9052	133.5326	177.4717
RG+XGB+LR	1.2388	7.3562	12.3912	18.9492	40.619	88.1966	130.8845	177.3592
RG+XGB+SVC	1.0092	4.9128	9.9349	17.5769	40.2651	87.0092	128.3862	173.0308
RG+RF+LR	0.8121	4.4132	9.7481	15.3888	41.2942	87.0731	130.2903	177.3375
RG+RF+SVC	1.4572	7.3571	13.9346	20.0921	44.3526	94.7887	138.0153	183.0098
RG+LR+SVC	1.0058	6.6719	11.9658	17.9054	41.7606	89.7950	134.4917	182.2353
XGB+RF+LR	0	5.8000	11.6000	18.4000	43.6000	93.2000	137.0000	184.2000
XGB+RF+SVC	0	5.8000	11.6000	18.4000	43.6000	93.2000	137.4000	184.2000
XGB+LR+SVC	0	5.8000	11.6000	18.4000	43.6000	93.2000	137.4000	184.2000
RF+LR+SVC	0	5.0000	11.0000	18.0000	44.0000	95.0000	140.8000	188.8000
RG+XGB+RF+SVC	2.1145	9.2749	13.8171	21.4056	46.9877	93.9972	137.2963	183.7677
RG+XGB+RF+LR	2.2281	9.0083	14.527	23.0924	47.5925	96.5607	139.0171	185.3276
RG+XGB+LR+SVC	1.7146	8.4278	13.8764	21.4658	43.6148	92.3832	135.6365	180.9075
RG+RF+LR+SVC	0.8380	7.2735	14.1578	21.8025	47.8366	95.6304	141.7602	188.0392
XGB+RF+LR+SVC	0	6.0000	13.0000	19.8000	44.8000	93.8000	138.4000	185.4000
RG+XGB+RF+LR+SVC	0.3542	6.2084	12.9557	20.7061	44.9708	90.5720	131.9569	176.9695

Sometimes, adding an inferior expert (e.g., adding RG to XGB+RF+LR+SVC) can decrease regret after the model converges (see TABLE 1). This might be caused by the fact that when there is a bad-behavior expert such as RG, it is easier for the online-learning module to learn who is the optimal expert. However, adding RG usually causes a severer “cold start” problem (see TABLE 1 and 2).

Comparison with the offline model.

In order to evaluate the effectiveness of our online model, we compare OnAI-Comp with the best standalone offline AI expert [20] in our AI expert list. For fair comparison, we implemented all the experiments in an online environment. As shown in Fig. 3, OnAI-Comp outperforms the standalone XGB model even though OnAI-Comp only has two AI experts and one of them is the dummy RG. This might be caused by the fact that RG can promote the exploration process in MAB setting. From Fig. 3, we can conclude that our proposed model has better performance than existing standalone offline models. Thus, if well-trained ML models with good performance are selected as AI experts, the performance of OnAI-Comp can be guaranteed.

Fig. 3:

with the Offline Model.

Impact of the number of AI Experts.

To evaluate the impact of the number of AI experts on the performance, we compared different combinations of AI experts (see Fig. 4 (a)). First, we made two experts (i.e., SVC and RF) as the primary candidates. Then, we increased the number of experts by adding new experts to the candidate list. We observe that if we add one expert, AR decreases and converges faster than the combination of two experts. Even though the difference of AR seems to be negligible, such tiny improvement, corresponding to several minutes earlier prediction of sepsis, is crucial for saving the lives of septic patients. However, we should choose experts wisely because sometimes adding an inferior expert (e.g., RG) might increase regret (see TABLE 1 and 2).

Model Interpretation.

Using the existing technologies, it is not feasible to interpret the overall system. At each execution time we could only interpret one AI expert. We overcome this challenge by making each AI expert provide the interpretation of their own prediction, thereby allowing the clinician to infer whether the model is reliable based on the interpretation. In this paper, we only interpreted RF.

We used LIME [32], which gives a score to evaluate the impact of each feature on the final prediction, to interpret our models⁸. We demonstrated the impact of each feature on RF for patient #000967 (septic patient) and #000013 (non-septic patient) during the whole ICU stay in Fig. 5 (a) and (b), respectively. Specifically, positive impact values make predicted labels bias to “Septic”, and negative impact values make predicted labels bias to“Non-septic”. In Fig. 5 (a), the black vertical line marks the hour the patient developed sepsis, and the blue vertical line marks the hour of the end of septic symptoms. As shown in Fig. 5, the impact of each feature changes over time. For example, the impact of ICUOS will increase tremendously during the last few hours in ICUs. This observation is consistent with the practical scenario that the possibility of developing sepsis increases as the patient stays in the ICU for a longer period. We also compare the interpretation results of patient #000013 and #000967 within a particular hour in Fig. 6. We observed that Temp, ICUOS, and Resp tend to be essential features for early sepsis prediction.

Fig. 5:

Impact of Features over Time. (a) Patient #000967 (Septic Patient). The black vertical line marks the hour the patient developed sepsis and the blue vertical line marks the hour of the end of septic symptoms. (b) Patient #000013 (Non-septic Patient).

Fig. 6:

Impact of Features within a Particular Hour (RF). (a) Patient #000967 (ICUOS = 36). (b) Patient #000013 (ICUOS = 13).

6. Discussion, Limitations and Conclusion

In this paper, we demonstrate the feasibility of an online learning framework to predict sepsis using an application of contextual experts consisting of several pre-trained offline models. The results of this work suggest that not only can the OnAI-Comp framework outperform the top achieving offline models using MAB, it can also serve to incorporate dynamic information from the real-world with incremental improvements in predictions and contextual awareness of critically ill patients.

The experiments further demonstrate that the OnAI-Comp framework appropriately incorporates information that is found in new data arrivals without retraining an offline model. By dynamically optimizing the online model, we can surpass the performances of offline algorithms alone. Moreover, the resultant model can be easily interpreted as long as the AI experts are interpretable. Indeed, these findings also suggest that OnAI-Comp can be developed as an alternative to offline models, where performance generalization are not achieved.

There are two limitations in OnAI-Comp. First, the AI experts we use in this work are traditional ML models and serve as a pilot demonstration of feasibility; thus, they may be able to demonstrate further improvements with adaptive learning algorithms. In the future, we plan to incorporate more advanced deep learning RNN-based models such as the Long short-term memory (LSTM) model [33]. In addition, as shown in Section 5, though the proposed framework can converge to the optimal strategy in the long run, it suffers from the cold start problem. There is thus a risk that the patients visiting the system during the early rounds might get bad detection results. A possible solution is to silence the results from the early stage and merely use them for training the online module. After the framework converges to the optimal strategy, the clinicians can take advantage of the online learning module. In conclusion, we demonstrate an online learning-based expert selection framework for early sepsis detection for patients in ICUs. Our model can be extended to other applications as long as the utility score and each expert are normalized appropriately. The experimental results show that our model converges to the optimal strategy in the long run.

Biographies

Anni Zhou is a third-year graduate student at Georgia Institute of Technology. She received her Bachelor’s Degree in Electrical and Computer Engineering from Huazhong University of Science and Technology in 2018. She joined Georgia Tech Communications Assurance and Performance Group (CAP) in 2018. Her current interests include the severe sepsis detection and security.

Raheem Beyah is the Dean of the College of Engineering at Georgia Tech and the Southern Company Chair. Dr. Beyah received his Bachelor of Science in Electrical Engineering from North Carolina AT State University in 1998. He received his Master’s and Ph.D. in Electrical and Computer Engineering from Georgia Tech in 1999 and 2003, respectively. Dr. Beyah’s work is at the intersection of the networking and security fields. He leads the Georgia Tech Communications Assurance and Performance Group (CAP). The CAP Group develops algorithms that enable a more secure network infrastructure, with computer systems that are more accountable and less vulnerable to attacks. Through experimentation, simulation, and theoretical analysis, CAP provides solutions to current network security problems and to long-range challenges as current networks and threats evolve.

Rishikesan Kamaleswaran is an Assistant Professor at Emory University, Department of Biomedical Informatics, with secondary appointments in Pediatrics and Emergency Medicine. He earned his Ph.D. in Computer Science from the University of Ontario Institute of Technology in Canada. He was a research fellow at the Division of Neonatology and the Department of Critical Care Medicine at the Hospital for Sick Children (Toronto) where he led efforts on the collection and analysis of physiological data in the Neonatal Intensive Care Unit for multiple clinical conditions including neonatal hypoglyceamia, physiological deterioration, nosocomial infection, and apnoea of prematurity. His current interests include severe sepsis detection and multi-organ dysfunction syndrome.