Dr. Agent: Clinical predictive model via mimicked second opinions

Abstract

Objective

Prediction of disease phenotypes and their outcomes is a difficult task. In practice, patients routinely seek second opinions from multiple clinical experts for complex disease diagnosis. Our objective is to mimic such a practice of seeking second opinions by training 2 agents with different focuses: the primary agent studies the most recent visit of the patient to learn the current health status, and then the second-opinion agent considers the entire patient history to obtain a more global view.

Materials and Methods

Our approach Dr. Agent augments recurrent neural networks with 2 policy gradient agents. Moreover, Dr. Agent is customized with various patient demographics information and learns a dynamic skip connection to focus on the relevant information over time. We trained Dr. Agent to perform 4 clinical prediction tasks on the publicly available MIMIC-III (Medical Information Mart for Intensive Care) database: (1) in-hospital mortality prediction, (2) acute care phenotype classification, (3) physiologic decompensation prediction, and (4) forecasting length of stay. We compared the performance of Dr. Agent against 4 baseline clinical predictive models.

Results

Dr. Agent outperforms baseline clinical prediction models across all 4 tasks in terms of all metrics. Compared with the best baseline model, Dr. Agent achieves up to 15% higher area under the precision-recall curve on different tasks.

Conclusions

Dr. Agent can comprehensively model the long-term dependencies of patients’ health status while considering patients’ demographics using 2 agents, and therefore achieves better prediction performance on different clinical prediction tasks.

INTRODUCTION

OBJECTIVE

Predicting complex disease phenotypes and their outcomes is difficult even for the best human doctors. Patients routinely seek second opinions from other clinical experts to confirm their existing diagnosis to alleviate the issue of high variance. For example, A study from the Mayo Clinic shows that only 12% of patients who seek for second opinion received the same diagnosis as their original one.¹ In those cases, it is clearly beneficial to have multiple clinical experts to evaluate the patient. Can we leverage this idea to improve artificial intelligence agents to perform better in clinical prediction? In particular, can multiple predictive agents work together to improve the prediction accuracy?

In this work, we propose a new multiagent reinforcement learning based model, named Dr. Agent, which trains 2 complementary reinforcement learning (RL) agents to comprehensively extract temporal dependencies in patients’ longitudinal electronic health record (EHR) data. Each agent is designed to focus on certain perspective of patients’ health status. We applied Dr. Agent to the publicly available MIMIC-III (Medical Information Mart for Intensive Care III) database.² We trained Dr. Agent to perform 4 clinical prediction tasks: (1) in-hospital mortality prediction, (2) acute care phenotype classification, (3) physiologic decompensation prediction, and (4) forecasting length of stay (LOS). Improving the accuracy of clinical prediction tasks could help clinicians delaying or preventing risk events and reduce cost.

BACKGROUND AND SIGNIFICANCE

Deep phenotyping

EHR data from millions of patients are now routinely collected across diverse healthcare institutions. The availability of massive EHR data enables the training of complex deep learning models for clinical predictive models.^3,4 Initial success has been demonstrated in estimating length of stay,⁵ patient phenotype identification,⁶ and disease or risk predictions.⁷

Extracting long-term dependencies and multimodal learning are challenges in EHR data analysis. To solve the former challenge, a diagnosis prediction model uses bidirectional recurrent neural networks (RNNs) and attention mechanism to extract the relationships of different visits.⁸ Time-aware long short-term memory networks (T-LSTMs) learn a representation capturing the dependencies between the visits in the presence of time irregularities.⁶ For multimodal learning, both Esteban et al⁹ and Lee et al¹⁰ leveraged hierarchy structures to incorporate different modalities.

Recently, in natural language processing tasks, Gui et al¹¹ proposed an RL-based dynamic skip connections mechanism to model word dependencies. It allows LSTM cells to compute recurrent transition functions based on 1 optimal set of hidden and cell states from the past few states, and thus achieved higher prediction performance compared with traditional attention-based approaches. In this work, we draw the analogy between “words” and “medical codes” and extend the dynamic skip connection mechanism to EHR analysis tasks. Besides, we initialize learning environment with patient demographics. The patient demographic information (eg, age, gender) was incorporated into the learning environment so that the agents can have better contextual information of patients in order to decide which historical visits are more important for prediction.

Multiagent reinforcement learning

Recently, reinforcement learning algorithms have been proposed to identify decision-making strategies for clinical tasks.¹² Nemati et al¹³ leveraged hidden Markov models and deep Q-networks to predict optimal heparin dosing for patients in intensive care units. Wang et al¹⁴ proposed supervised reinforcement learning with RNNs for treatment recommendation. However, all these existing works tried to learn a single model or a single policy for prediction.

Multiagent reinforcement learning is a subfield within reinforcement learning and has shown great success in the realm of Markov decision problems.¹⁵ In this work, we train 2 complementary RL agents to mimic the second-opinion situation. Dr. Agent has 2 policy gradient agents: a primary agent $a_{\mathit{primary}}$ that observes patient current health condition to learn patient representation, while a secondary agent $a_{\mathit{history}}$ summarizes patient historical EHR to learn another patient representation. The different reinforcement agents can summarize long-term health history from complementary perspectives. We also design the reward function that transfers knowledge of disease progression. The learning of the agents is optimized by a long-term reward function related to the accuracy of clinical prediction, which can transfer observed progression trajectory knowledge to help the agents make correct predictions.

MATERIALS AND METHODS

As illustrated in Figure 1, Dr. Agent includes the following components: multiagent skip connection component, demographic information–enriched environment representation, and reward function for disease progression knowledge transfer. Next, we first introduce each component and then provide details of training and inference of Dr. Agent.

Figure 1.

The Dr. Agent model. We use the patient’s demographics to initialize the initial hidden state (ie, ${\mathit{h}}_0$) of the gated recurrent unit (GRU). We use 2 policy gradient agents, $a_{\mathit{primary}}$ and $a_{\mathit{history}}$, to extract long-term history information of patients by using a dynamic skip connections mechanism. At each time step, the 2 agents will generate an optimal state from history hidden states instead of using the last state ${\mathit{h}}_{t-1}$ to compute current state ${\mathit{h}}_t$. We use the model’s output and predict target to compute long-term rewards to optimize the agents’ policy. FC: fully connected.

Multiagent skip connections

We adopt dynamic skip connection mechanism¹¹ with gated recurrent unit (GRU) network¹⁶ to capture long-term dependencies among clinical events in patient EHR data.

Concretely, given the current visit ${\mathit{v}}_t$ at $t$-th timestep, regular GRU cell will use ${\mathit{v}}_t$ and the hidden state of previous timestep ${\mathit{h}}_{t-1}$ to compute the current hidden state ${\mathit{h}}_t$. To achieve dynamic skip connection, instead of using ${\mathit{h}}_{t-1}$, we consider 2 policy gradient agents $a_{\mathit{primary}}$ and $a_{\mathit{history}}$ ($a_p\mathrm{ }$and $a_h$ for abbreviation) to generate an optimal $\stackrel{\sim }{{\mathit{h}}_{t-1}}$ from a history state set ${\mathcal{S}}_h=\left\{{\mathit{h}}_i|i\in t-K,\dots ,t-1\right\},$ where hyperparameter $K$ determines the length of the observation window on the patients’ historical health status. The selection of $K$ depends on tasks. Larger $K$ means that the model can capture longer dependencies. In our experiment, the optimal $K$ is between 5 and 10 for most tasks.

We provide the 2 agents with different environment variables to make the agents observe patients health condition from complementary perspectives.

1. Primary agent: Patients’ current lab tests and diagnosis provide direct information about their health status for clinicians to make treatment plans or prognosis. We use the primary agent $a_p$ to mimic such practice. The environment variable ${\mathit{e}}_p$ of the agent is the health status of patients’ current visit:

   $${\mathit{e}}_p={\mathit{v}}_t$$
2. Second-opinion agent: Accurate diagnoses and informing treatment based on prediction of individual patient risk for a negative health outcome can be difficult, especially for complex or rare diseases, and patients routinely seek second opinions for an understanding of their disease and their risk. Here, we design the second-opinion agent $a_h$ to be able to consider the entire patient history to obtain a more global view via summarizing patients’ historical visits by using historical hidden states of GRU as environment:

   $${\mathit{e}}_h=\mathrm{ }\frac{1}{K}\sum _{i=t-K}^{t-1}{\mathit{h}}_i$$

We hope the 2 agents focus on their own environment and learn more environment-related policies instead of using only 1 agent to process all information. This design can allow agents can comprehensively understand patients’ status and obtain complementary patient embeddings. Experiments show our multiagent strategy outperforms each single agent model.

For each agent $a_{\mathrm{*}}$, after observing the environment ${\mathit{e}}_{\mathrm{*}}$, the agent takes an action $c_{\mathrm{*}}^t$ by selecting an optimal ${\mathit{h}}_{t-k_{\mathrm{*}}}$ in historical hidden state set ${\mathcal{S}}_h$. $k_{\mathrm{*}}$ is sampled from a multinomial distribution ${\pi }_K^{\mathrm{*}}\left\{k\right|{\mathit{e}}_{\mathrm{*}}\}$ as follows:

$$\begin{array}{c}{\mathcal{P}}^{*}=\text{softmax}(\text{MLP}({\mathit{e}}_{*}))\\ {\pi }_K^{*}\{k|{\mathit{e}}_{*}\}=\mathrm{Pr}\{K=k|{\mathit{e}}_{*}\}=\prod _{i=1}^K{p}_i^{[k=i]}\end{array}$$

where $[k=i]$ is 1 if $k=i$, 0 otherwise. $\mathrm{MLP}$ represents a 2-layer perceptron to transform ${\mathit{e}}_{\mathrm{*}}$ to a vector with dimensionality $K$. The softmax is to generate a probability distribution ${\mathcal{P}}^{\mathrm{*}}$, and $p_i$ is the $i$-th element in ${\mathcal{P}}^{\mathrm{*}}$.

Agent $a_p$ and $a_h$ obtain the optimal states ${\mathit{h}}_{t-k_p}$ and ${\mathit{h}}_{t-k_h}$ by sampling $k_p$ and $k_h\mathrm{ }$from the distribution ${\pi }_K^p$ and ${\pi }_K^h$. The actions from 2 agents are combined as follows:

$$\stackrel{\sim }{{\mathit{h}}_{t-1}}=\frac{1}{2}\left({\mathit{h}}_{t-k_p}+{\mathit{h}}_{t-k_h}\right)+\lambda {\mathit{h}}_{t-1}$$

Here, we find that in some tasks, adding information in ${\mathit{h}}_{t-1}$ will have better performance. We use a hyperparameter $\lambda$ to incorporate this situation.

Demographic enhancement

In clinical practice, the demographic information of patients such as gender and age can serve as important baseline data that depict the original health status and subtypes of patients. In naïve RNN models, the hidden state of the cell is initialized to zero by default. We use demographic information ${\mathit{v}}_d$ to initialize the hidden state ${\mathit{h}}_0$ of GRU as:

$${\mathit{h}}_0={\mathit{W}}_{{\mathit{h}}_0}·{\mathit{v}}_d$$

where ${\mathit{W}}_{{\mathit{h}}_0}\in {\mathbb{R}}^{\left|{\mathit{h}}_0\right|\times \left|{\mathit{v}}_d\right|}$. With a more reasonable initial hidden state, our model can better model patients’ follow-up health status representation.

Besides, patients’ demographics can also be an important context about historical health status and help the agents to decide which history visits are more important for prediction. To better incorporate this information, we concatenate the original environment representations of 2 agents with patients’ demographic data:

$$\begin{array}{c}{\mathit{e}}_p={\mathit{v}}_t\oplus {\mathit{v}}_d\\ {\mathit{e}}_h=(\frac{1}{K}\sum _{i=t-K}^{t-1}{\mathit{h}}_i)\oplus {\mathit{v}}_d\end{array}$$

where $\oplus$ refers to the concatenate operation. The 2 agents will observe the demographics enriched environment and generate $\stackrel{\sim }{{\mathit{h}}_{t-1}}$ using the aforementioned equations. Then the GRU cell uses $\stackrel{\sim }{{\mathit{h}}_{t-1}}$ and ${\mathit{v}}_t$ to compute output ${\mathit{h}}_t$ at $t$-th timestep as: ${\mathit{h}}_t=\mathrm{GRU}({\mathit{v}}_t,\stackrel{\sim }{{\mathit{h}}_{t-1}})$.

Finally, we use a 2-layer perceptron to compress the current hidden state and patients’ demographics. The final prediction output $\widehat{y_t}$ is calculated as:

$$\widehat{y_t}=\sigma \left(\mathrm{MLP}\right({\mathit{h}}_t\oplus {\mathit{v}}_d\left)\right)$$

The activation function of the output layer $\sigma$ depends on the specific task. We use a softmax layer for multiclass classification tasks (ie, length of stay), a sigmoid layer for multilabel classification, and binary classification tasks (ie, decompensation, mortality, and phenotyping).

Reward function

The reward function evaluates the quality of the agents’ choice. We hope that the agents choose the optimal states that can help the model make correct prediction. In this study, to predict decompensation, mortality, and length of stay (LOS), we design the reward function $r_t$ at $t$-th timestep as: $r_t={\left(\widehat{y_t}-0.5\right)}^{\mathrm{T}}\mathrm{*}{y}_t$. Because phenotype prediction can be casted as a multilabel classification task, in order to prevent the rewards from being affected by the number of labels, we design the reward function $r_t$ as: $r_t=\left({\left(\widehat{y_{\mathrm{t}}}-0.5\right)}^{\mathrm{T}}\mathrm{*}{y}_t\right)/\sum y_t$, where $\sum y_t$ is the number of the patient’s phenotypes. The design of these reward functions allows agents to get higher rewards when they output a higher prediction for clinical events and get punishment when they fail to predict these events.

Furthermore, we hope that our agents have long-term visions, which means that the selected states are optimal for not only current predictions, but also future predictions. To achieve this, we transfer observed progression trajectory knowledge to help the agents make correct prediction by using long-term reward function. Therefore, instead of using the instantaneous reward $r_t$, the long-term reward $R_t$ is calculated as:

$$R_t=\sum _{i=0}^{T-t}{\gamma }^i{r}_{t+i}$$

where $\gamma$ is the discount factor. It quantifies how much importance we value for future rewards. $\gamma$ varies from 0 to 1. If $\gamma$ is close to 0, the agent will tend to consider only instantaneous rewards. If $\gamma$ is close to 1, the agent will consider future rewards with greater weight. Our experiments show that using long-term reward function allows the model to better extract long-term dependencies in patient data, thus predict more accurately.

Joint learning and policy optimization

Our first task is to optimize the parameters of 2 policy gradient agents $a_p\mathrm{ }$and $a_h$. We should also optimize the parameters of standard GRU and other parameters. For the latter task, the optimization is straightforward. We use the cross-entropy loss for our tasks, denoted as ${\ell }_1$.

To train the agents, we maximize the expected reward under the skip policy distribution. For each agent at $t$-th timestep, we have:

$$\begin{array}{c}{\ell }_{*}={\mathbb{E}}_{\pi (c_{1:t}^{*})}[R_t]\\ \pi (c_{1:t}^{*})=\prod _{i=1}^t\text{Pr}(c_i^{*}|{\mathit{e}}_{*})\end{array}$$

Furthermore, we want to prevent premature entropy collapse and encourage the policy to explore more diverse space. Besides, we also want to reduce the policy variance.¹⁷ We use an entropy term and subtract a baseline to tackle these issues:

$${\ell }_{\mathrm{*}}={\mathbb{E}}_{\pi \left(c_{1:t}^{\mathrm{*}}\right)}\left[R_t-B^{\pi }\left({\mathit{e}}_{\mathrm{*}}\right)\right]-\mathrm{ }{\mathbb{E}}_{\pi \left(c_{1:t}^{\mathrm{*}}\right)}\left[\log \pi \left(c_{1:t}^{\mathrm{*}}\right)\right]$$

$B^{\pi }$ is the on-policy value function.¹⁸

Because of the nondifferentiable of discrete skips, we use the REINFORCE method to optimize ${\ell }_{\mathrm{*}}:$¹⁹

$$\nabla {\ell }_{\mathrm{*}}={\mathbb{E}}_{\pi \left(c_{1:t}^{\mathrm{*}}\right)}\left[\sum _{i=1}^t\nabla \log \mathrm{Pr}\left(c_i^{\mathrm{*}}|{\mathit{e}}_{\mathrm{*}}\right)\mathrm{*}\right(R_t-B^{\pi }\left({\mathit{e}}_{\mathrm{*}}\right)-\sum _{i=1}^t\log \mathrm{Pr}\left(c_i^{\mathrm{*}}|{\mathit{e}}_{\mathrm{*}}\right)-1\left)\right]$$

Then, the final loss $\ell$ is as follows:

$$\ell ={\ell }_1-({\ell }_p+{\ell }_h)$$

Data description and prediction tasks

Source of data, sample size, predictors, and data preprocessing

We use EHR data from the publicly available MIMIC-III database.² We process the raw data according to Harutyunyan et al.²⁰ We use the cohort of 33,678 unique patients. The raw data include 17 physiologic variables as predictors at each timestep, which is transformed into a 76-dimensional vector including numerical and one-hot encoded categorical clinical features. The demographics of each patient is a 12-dimensional vector including ethnicity, gender, age, height, and weight. Details of the predictors, data-preprocessing steps, and sample sizes are provided in the Supplementary Appendix.

Prediction tasks and outcomes

We perform 4 prediction tasks on the MIMIC-III dataset:

1. In-hospital mortality prediction predicts in-hospital mortality from observations recorded early in an intensive care unit admission. We formulated this task as binary classification using observations recorded from a limited window of time (eg, here 48 hours) following admission. The ground truth label is a binary value that indicates whether the patient died before hospital discharge.

2. Acute care phenotype classification is to classify which acute care conditions (of 25 chronic or critical conditions) are present in a given patient record. Because diseases can co-occur, this task is formulated as a multilabel classification problem. The target phenotypes are shown in the Supplementary Appendix.

3. Physiologic decompensation prediction involves the detection of patients who are physiologically decompensating, which means that conditions are deteriorating rapidly. The ground truth of the task is a binary label that indicates whether the patient’s date of death falls within the next 24 hours of the current time point.

4. Forecasting length of stay is to forecast the patient’s LOS. In this task, we predict the remaining LOS once per hour for every hour after admission. We formulate this task as a multiclass classification problem as previous works do.^20,21 We divide the range of values into 10 buckets, 1 bucket for extremely short visits (<1 day), 7 day-long buckets for each day of the first week, and 2 buckets for outliers (one for stays of over 1 week but <2 weeks and one for stays of over 2 weeks).

Model evaluation

Baseline models for performance comparison

We compare Dr. Agent with the following baselines. These models focus on utilizing long-term dependencies in patients’ longitudinal EHR data and also have designed structure to utilize demographic information. It is worth noting that there are lots of state-of-the-art clinical prediction models that utilize extra modules such as convolution layer or de-correlation module to perform better prediction.^7,22 However, their contribution is orthogonal to ours. We focus on better capturing temporal dependencies in EHR data. Our model can be easily combined with other extra modules.

1. Logistic regression (LR): We use LR with l2 regularization based on hand-engineered features described in Lipton et al²³: for each variable, we compute 6 different sample statistic features on 7 different subsequences of a given time series. The per-subsequence features include minimum, maximum, mean, standard deviation, skew, and number of measurements. The 7 subsequences include the full time series, the first 10%/25%/50% of time and the last 50%/25%/10% of time. In total, we obtain 714 features per time series and we also use demographic features. Specifically, we trained a softmax regression model to solve the 10-class bucketed LOS prediction problem.

2. Multilayer perceptron (MLP): We use a 2-layer MLP network with l2 regularization to achieve prediction. The input features are same with LR model. Note that both MLP and LR cannot process time series data because they process each timestep as discrete data sample.

3. GRU: Patients’ demographics are duplicated at each timestep and simply concatenated to the original inputs.

4. Dipole ⁸ : Uses GRU with the additive attention mechanism.

5. T-LSTM ⁶ : Handles irregular time intervals by enabling time decay inside RNN cell, which makes older information less important. The original T-LSTM model is used for unsupervised clustering, and we modify it into a supervised learning model.

6. Simply Attend and Diagnose (SAnD)* ²¹ : Uses self-attention mechanism to predict clinical targets. We use the same hyperparameter settings in the original article. Different from the original article, we modify the padding strategy in SAnD. In order to prevent using future information, we use causal padding to obtain the input embedding.²⁴

The implementation details of these models are shown in the Supplementary Appendix. We have made our codes available on a public repository (https://github.com/v1xerunt/Dr.Agent). We also compare Dr. Agent against its reduced models such as the performance difference between long-term reward function and instantaneous reward function. The results are shown in the Supplementary Appendix.

Evaluation strategy

We truncate the length of samples to a reasonable limit (ie, 400) for all tasks except for in-hospital mortality prediction. We fix a test set of 15% of patients and divide the rest of the dataset into training set and validation set with a proportion of 85%:15%. We fix the best model on the evaluation set within 40 epochs and report the performance in the test set. To estimate a 95% confidence interval, we resample the test set 10 000 times, calculate the score on the resampled sets, and then use 2.5th and 97.5th percentiles of these scores as our confidence interval estimate.

For in-hospital mortality and decompensation prediction tasks, we use the area under the receiver-operating characteristic curve (AUROC), area under precision-recall curve (AUPRC), and Min(Re, P+) as our evaluation metrics. Min(Re, P+) is calculated as the maximum of minimum (recall, precision) on the precision-recall curve, and we provide an illustration example in the Supplementary Appendix. For the phenotype classification task, we consider the following metrics: macro-averaged AUROC, which averages per-label AUROC; micro-averaged AUROC, which computes single AUROC score for all classes together; and weighted AUROC, which takes disease prevalence into account. For the LOS prediction task, we use the Cohen’s linear weighted kappa metric, which measures the interagreement between true and predicted labels. We assign the mean LOS from each bin to the samples assigned to that class, and use mean squared error and mean absolute percentage error as evaluation metrics.

We also assess model calibration visually using a calibration plot and statistically by estimating the calibration slope and intercept,^25,26 except for the LOS task, as the LOS prediction is a multiclass classification task. The closer the slope is to 1, the better the model is calibrated. We compute the average calibration performance of all phenotypes for the phenotyping task.

RESULTS

Table 1 shows the performance of our model and all baseline models. We also report the parameters of each model in the LOS task. Dr. Agent outperforms all baseline models on phenotyping, mortality prediction, and decompensation prediction tasks. On the decompensation task, the relative improvement of Dr. Agent is up to 15% in AUPRC and 6% in Min(Re, P+) compared with the best baseline model T-LSTM and Dipole. On the mortality prediction task, Dr. Agent achieves more than 4% higher AUPRC compared with the best baseline GRU. On the phenotyping task, Dr. Agent achieves 1% relative improvement on AUC compared with the best baseline SAnD. On the LOS task, Dr. Agent achieves 1% higher kappa score and lower mean squared error compared with the best baseline model, while the mean absolute percentage error of Dr. Agent is slightly lower than the GRU (<1%). Among all baselines in Table 1, Dipole and SAnD achieve better performance in most cases because of better handling long-term dependencies. LR and MLP show low performance, as they cannot capture temporal patterns in longitudinal EHR records. The detailed performance about per-phenotype performance in the phenotyping task is shown in the Supplementary Appendix.

Table 1.

Performance comparison for the MIMIC-III tasks

Task 1: Phenotyping
Model	Macro AUC	Micro AUC	Weighted AUC
LR	0.7285 (0.7257-0.7317)	0.7755 (0.7727-0.7786)	0.7242 (0.7209-0.7277)
MLP	0.7371 (0.7342-0.7400)	0.7863 (0.7832-0.7895)	0.7311 (0.7282-0.7340)
GRU	0.7684 (0.7650-0.7719)	0.8174 (0.8147-0.8202)	0.7601 (0.7574-0.7629)
Dipole	0.7484 (0.7453-0.7517)	0.8012 (0.7979-0.8045)	0.7358 (0.7328-0.7389)
T-LSTM	0.7579 (0.7550-0.7609)	0.8091 (0.8062-0.8122)	0.7524 (0.7497-0.7552)
SAnD*	0.7735 (0.7696-0.7776)	0.8232 (0.8205-0.8258)	0.7660 (0.7631-0.7689)
Dr. Agent	0.7821 (0.7792-0.7850)	0.8276 (0.8242-0.8311)	0.7748 (0.7717-0.7779)
Task 2: Mortality prediction
Model	AUROC	AUPRC	Min(Re, P+)
LR	0.8462 (0.8282-0.8641)	0.4797 (0.4413-0.5184)	0.4833 (0.4445-0.5219)
MLP	0.8521 (0.8326-0.8722)	0.4851 (0.4547-0.5162)	0.4849 (0.4517-0.5186)
GRU	0.8600 (0.8401-0.8709)	0.5104 (0.4751-0.5458)	0.4977 (0.4662-0.5293)
Dipole	0.8628 (0.8492-0.8769)	0.4989 (0.4688-0.5293)	0.5026 (0.4709-0.5350)
T-LSTM	0.8617 (0.8480-0.8749)	0.4964 (0.4617-0.5308)	0.4977 (0.4606-0.5337)
SAnD*	0.8382 (0.8188-0.8577)	0.4545 (0.4144-0.4941)	0.4885 (0.4516-0.5246)
Dr. Agent	0.8658 (0.8499-0.8822)	0.5311 (0.4955-0.5662)	0.5054 (0.4724. 0.5393)
Task 3: Decompensation prediction
Model	AUROC	AUPRC	Min(Re, P+)
LR	0.8733 (0.8704-0.8762)	0.2233 (0.2179-0.2295)	0.2374 (0.2301-0.2453)
MLP	0.8779 (0.8741-0.8807)	0.2291 (0.2225-0.2359)	0.2350 (0.2270-0.2428)
GRU	0.9013 (0.8979-0.9048)	0.2611 (0.2536-0.2682)	0.3212 (0.3138-0.3290)
Dipole	0.8970 (0.8933-0.9008)	0.2626 (0.2556-0.2692)	0.3340 (0.3261-0.3415)
T-LSTM	0.8956 (0.8925-0.8987)	0.2670 (0.2591-0.2748)	0.3275 (0.3194-0.3352)
SAnD*	0.8953 (0.8924-0.8988)	0.2353 (0.2277-0.2432)	0.3051 (0.2979-0.3127)
Dr. Agent	0.9071 (0.9037-0.9103)	0.3074 (0.3004-0.3149)	0.3531 (0.3450-0.3611)
Task 4: LOS prediction
Model	Kappa	MSE	MAPE
LR	0.3504 (0.3477-0.3534)	47928 (47557-48335)	237.4 (234.9-239.7)
MLP	0.3577 (0.3545-0.3599)	46804 (46227-47364)	230.0 (227.4-233.1)
GRU	0.3864 (0.3831-0.3898)	39310 (38976-39651)	187.2 (185.4-189.2)
Dipole	0.3965 (0.3935-0.3996)	38971 (38627-39321)	193.2 (191.5-194.6)
T-LSTM	0.3872 (0.3843-0.3902)	40399 (40042-40756)	189.7 (188.6-191.0)
SAnD*	0.3876 (0.3845-0.3908)	39719 (39338-40094)	191.8 (190.5-193.2)
Dr. Agent	0.3988 (0.3955-0.4022)	38446 (38084-38816)	188.8 (186.9-190.5)

AUC: area under the curve; AUROC: area under the receiver-operating characteristic curve; AUPRC: area under precision-recall curve; GRU: gated recurrent unit; LOS: length of stay; LR: logistic regression; MAPE: mean absolute percentage error; MIMIC-III: Medical Information Mart for Intensive Care III; MLP: multilayer perceptron; MSE: mean squared error; SAnD: Simply Attend and Diagnose; T-LSTM: time-aware long short-term memory network.

Figures 2 and 3 show the calibration plot of decompensation and mortality prediction tasks. In the calibration plot, the predictions are grouped into bins based on their predicted probabilities. For each bin, the y-axis indicates the proportion of true outcomes, and the x-axis is the mean predicted probability. Table 2 shows the calibration performance for phenotyping, mortality, and decompensation prediction tasks, and the detailed per-phenotype performance is shown in the Supplementary Appendix. The results show that Dr. Agent achieves the best calibration performance on all 3 tasks. For the baseline models, they are generally well calibrated on the mortality prediction task and also have better calibration performance compared with other tasks. For phenotyping task, Dipole and SAnD* achieve the worst calibration performance, whereas GRU is better calibrated (slope between 0.9 and 1). For decompensation prediction, none of the baseline models are well calibrated (slope < 0.8) and Dipole, LR, and T-LSTM have the worst calibration (slope < 0.7).

Figure 2.

The calibration plot for the decompensation prediction. GRU: gated recurrent unit; LR: logistic regression; MLP: multilayer perceptron; SAnD: Simply Attend and Diagnose; T-LSTM: time-aware long short-term memory network.

Figure 3.

The calibration plot for the mortality prediction. GRU: gated recurrent unit; LR: logistic regression; MLP: multilayer perceptron; SAnD: Simply Attend and Diagnose; T-LSTM: time-aware long short-term memory network.

Table 2.

Calibration performance comparison for the MIMIC-III tasks

Task 1: Phenotyping
Model	Slope	Intercept
LR	0.7149 (0.7062 to 0.7235)	0.0138 (0.0024 to 0.0263)
MLP	0.7155 (0.7132 to 0.7175)	0.0164 (0.0291 to 0.0029)
GRU	0.9136 (0.9085 to 0.9192)	0.0253 (0.0115 to 0.0402)
Dipole	1.6745 (1.6682 to 1.6810)	0.1620 (0.1788 to 0.1485)
T-LSTM	0.8211 (0.8119 to 0.8308)	0.0285 (0.0104 to 0.0467)
SAnD*	0.6773 (0.6652 to 0.6891)	0.0791 (0.0629 to 0.0918)
Dr. Agent	0.9947 (0.9883 to 1.0025)	−0.0080 (-0.0196 to 0.0035)
Task 2: Mortality prediction
Model	Slope	Intercept
LR	0.9125 (0.9048 to 0.9205)	0.0003 (-0.0074 to 0.0082)
MLP	1.1347 (1.1268 to 1.1426)	0.0075 (-0.0010 to 0.0168)
GRU	0.9011 (0.8897 to 0.9144)	−0.0253 (-0.0392 to -0.0127)
Dipole	0.8658 (0.8530 to 0.8777)	0.0014 (-0.0083 to 0.0115)
T-LSTM	1.0744 (1.0612 to 1.0870)	−0.0480 (-0.0569 to -0.0392)
SAnD*	0.9291 (0.9185 to 0.9406)	−0.0191 (-0.0301 to -0.0079)
Dr. Agent	0.9774 (0.9688 to 0.9854)	0.0060 (-0.0015 to 0.0141)
Task 3: Decompensation prediction
Model	Slope	Intercept
LR	0.6762 (0.6738 to 0.6789)	−0.0463 (-0.0588 to -0.0431)
MLP	0.7028 (0.6999 to 0.7057)	−0.0542 (-0.0675 to -0.0429)
GRU	0.7976 (0.7945 to 0.8002)	−0.0902 (-0.1004 to -0.0798)
Dipole	0.5783 (0.5760 to 0.5804)	−0.0659 (-0.0740 to -0.0565)
T-LSTM	0.6925 (0.6897 to 0.6954)	−0.0707 (-0.0793 to -0.0614)
SAnD*	0.7048 (0.7020 to 0.7073)	−0.0495 (-0.0586 to -0.0401)
Dr. Agent	0.9599 (0.9561 to 0.9634)	−0.0632 (-0.0745 to -0.0522)

GRU: gated recurrent unit; LOS: length of stay; LR: logistic regression; MIMIC-III: Medical Information Mart for Intensive Care III; MLP: multilayer perceptron; T-LSTM: time-aware long short-term memory network.

DISCUSSION

Here, we explore how our agents choose optimal historical states. Taking the decompensation task as an example, we extracted all the decompensated visits of all patients and compute the average $k$ of each visit selected by agents. A larger $k$ indicates the agent select a more distant visit as optimal choice and vice versa. Each average $k$ represents the select pattern of agents for each patient. We will observe how different agents make choices. All observations are made on the test dataset.

In Table 3, we compute the average and standard deviation of $k$ chosen by $a_p$ and $a_h$ in Dr. Agent. We can observe that $a_h$ tends to select larger $k$ than $a_p$. This may be because the environment of the 2 agents is different: $a_p$ uses the patient’s current health status as environment, whereas $a_h$ uses the patient’s historical health status. This leads to $a_p$ being more sensitive to variations of patients’ status, so it is more inclined to choose a relatively smaller $k$ as the optimal choice.

Table 3.

Selection patterns of 2 agents in the Dr. Agent model

Agent	Average ± SD of $\mathit{k}$
$a_h$	5.72 ± 2.91
$a_p$	4.97 ± 2.83

We also explore how the settings of different environments and reward functions affect the agents’ choices. The results and discussions are shown in the Supplementary Appendix.

LIMITATIONS OF PROPOSED METHOD

Deep learning–based models have shown great success in different clinical tasks because they have strong ability to process high-dimensional longitudinal EHR data. Though they can achieve better performance compared with traditional statistical methods, interpretability is a major limitation for these deep learning methods including Dr. Agent, as deep learning models are considered as black box. There is recent work focusing on improving interpretability of deep learning models.^7,27 Dr. Agent can potentially combine with those models to provide model explanation in future. Another drawback of Dr. Agent is the cold-start issue (ie, our model may have lower accuracy when applied to patients with very few visits), which is also a common issue for recurrent-based methods.^28,29 According to the benchmark requirements,²⁰ we excluded patients with few visits or high missing rate (detailed exclude rules are shown in the Supplementary Appendix), which may affect model generalizability. In future works, this issue may be solved using transfer learning or introducing expert knowledge in the model.

CONCLUSION

In this work, we propose a reinforcement learning-based clinical predictive model, Dr. Agent. Dr. Agent has 2 policy gradient agents (ie, a primary agent and a second-opinion agent) to mimic the practice that patients seek second options from multiple clinical experts for diagnosis. The 2 agents achieve dynamic skip connection mechanism, which can dynamically choose the optimal hidden states from the past few states of GRU. Dr. Agent can comprehensively model the long-term dependencies of patients’ health status while considering patients’ demographics. Furthermore, we use the long-term rewards function, which makes Dr. Agent have the ability to transfer knowledge of disease progression. In addition, Dr. Agent has a shorter gradient backpropagation path; thus, the model can alleviate the challenges of vanishing gradient. Dr. Agent achieves better performance than state-of-the-art methods on multiple prediction tasks including decompensation, LOS, mortality, and most of the phenotyping tasks on the MIMIC-III dataset. We hope that our model can help physicians identify the patients at high risk to prevent or delay the adverse outcome.

FUNDING

This work was in part supported by National Science Foundation awards IIS-1418511, CCF-1533768 and IIS-1838042 (to JS), and National Institutes of Health awards R01 1R01NS107291-01 and R56HL138415 (to JS).

AUTHOR CONTRIBUTIONS

JG implemented the method and conducted the experiments. All authors were involved in developing the ideas and writing the article.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

The authors have no competing interests to declare.