Assessing Contribution of Higher Order Clinical Risk Factors to Prediction of Outcome in Aneurysmal Subarachnoid Hemorrhage Patients

Abstract

The goal of this study was to investigate the application of machine learning models capable of capturing multiplica tive and temporal clinical risk factors for outcome prediction inpatients with aneurysmal subarachnoid hemorrhage (aSAH). We examined a cohort of 575 aSAH patients from Emory Healthcare, identified via digital subtraction angiog- raphy. The outcome measure was the modified Ranking Scale (mRS) after 90 days. Predictions were performed with longitudinal clinical and imaging risk factors as inputs into a regularized Logistic Regression, a feedforward Neural Network and a multivariate time-series prediction model known as the long short-term memory (LSTM) architecture. Through extraction of higher-order risk factors, the LSTM model achieved an AUC of 0.89 eight days into hospitaliza tion, outperforming other techniques. Our preliminary findings indicate the proposed model has the potential to aid treatment decisions and effective imaging resource utilization in high-risk patients by providing actionable predictions prior to the development of neurological deterioration.

Introduction and Background

Aneurysmal subarachnoid hemorrhage (aSAH) is a grave medical condition, affecting over 30,000 individuals a year in the United States¹. Despite advances in medical care, it remains a major cause of premature mortality, accounting for 27% of all stroke-related years of life lost before the age of 65². Mortality is estimated at 30%, and up to 40% of survivors have long-term neurological deficits resulting in significant loss of quality of life and increased burden and cost on the healthcare system³. Delayed cerebral ischemia (DCI) and cerebral infarction are major complications of aSAH, occurring in 19-46% of patients, and account for a large burden of aSAH-related morbidity4,⁵. Cerebral arterial vasospasm (CVS) is another major complication of aSAH, seen in up to 70% of patients, and is often associated with DCI and cerebral infarction⁶. For this study, CVS is defined as arterial narrowing documented on imaging, specifically computed tomography angiography (CTA) and/or digital subtraction angiography (DSA). As DCI is often difficult to diagnose and predict, the strong association between the DCI and CVS has led to the clinical practice of using CVS detection as a means of early detection of DCI, in an effort to guide early treatment to prevent cerebral infarction. However, less than half of patients with CVS ultimately develop DCI, thus definitive prognostic and therapeutic decision based solely on CVS screening is not ideal and may lead to overtreatment and treatment-related morbidity. Furthermore, while less common, DCI also can occur in the absence of CVS further complicating timely diagnosis and treatment^7,8.

There are a variety of imaging studies used to diagnose CVS, each with its own distinct advantages and pitfalls, in cluding transcranial doppler ultrasound (TCD), computed tomographic angiography (CTA) and CT perfusion (CTP). However, there is conflicting evidence in the literature regarding sensitivity, specificity, and prognostic efficacy. In particular, TCD has been deemed both effective9,¹⁰ and ineffective11,¹² in predicting both vasospasm and poor neuro logic outcomes, using a variety of imaging parameters and outcome definitions. CTA and CTP are additional imaging modalities used to diagnose vasospasm and define brain tissue at risk for DCI. These imaging modalities have been shown to have a higher sensitivity for detection of vasospasm and DCI and have generally been found useful for prog- nostication^13,14. However, the radiation exposure from these techniques must be considered. DSA is considered the gold standard for determination of angiographic vasospasm, but it is an invasive and resource intensive modality with a small, but significant risk of neurologic compromise. A review of the literature to evaluate the diagnostic value of DSA for detection of DCI revealed few studies¹⁴. Furthermore, a recent study found that screening asymptomatic patients with DSA was ineffective, although this remains useful in patients that are either symptomatic or difficult to assess clinically due to poor neurologic status¹⁵.

A wide variety of clinical signs and risk factors for developing both CVS and DCI have been described, including smoking¹⁶, hypertension^16,17, hyperglycemia^16,17, old age18,¹⁹, and level of consciousness^20,21, as well as image-based scoring systems such as modified Fisher score (MFS)^22,23. However, much of the literature is difficult to interpret as outcomes are often poorly defined²⁴. In fact, the terms CVS and DCI are often used interchangeably, further complicating study comparison. Variations in study design and differing outcome definitions underscore the need for a robust, reproducible predictive model with clearly defined outcomes. Additionally, the complex pathophysiology of aSAH, diagnostic challenges of DCI, and risk of poor neurologic outcome emphasizes the need for a prognostic tool to aid in the early identification of aSAH patients at risk for vasospasm, ischemia and stroke prior to neurologic deterioration.

In clinical research, conventional statistical techniques such as linear and logistic regression are the dominant tech niques used for clinical decision support models. These techniques are hypothesis driven and simple to interpret, yet overly simplistic statistical assumptions about data distribution and model fitting, as well as inability to identify higher order interactions among the input variables, limit their applicability in complex data sets. Notably, logistic regression has been used to evaluate the contribution of TCD to predicting outcomes in aSAH patients, and the literature reveals conflicting results, with TCD shown to be both contributory¹¹ and noncontributory^11,25. These findings reinforce the need for an alternative approach to data modeling in aSAH patients.

Machine learning is a branch of artificial intelligence that has been employed in a variety of applications, including medicine, to analyze complex datasets²⁶. While the complexity of clinical problems such as aSAH outcome prediction provides an opportunity for machine learning techniques, the smaller datasets available in healthcare compared to other industries provide a significant challenge to machine learning techniques^27–29.

There have been a few preliminary studies of machine learning models to predict vasospasm in patients with aSAH, which outperformed their respective logistic regression models^30–32. However, two of the models were trained on less than 100 patients, limiting generalizability. Additionally, focusing on vasospasm as an outcome, rather than a measurable clinical outcome, limited the clinical utility of these models. The third model, while trained on a large dataset from multiple sites, did not include several important variables previously identified as prognostic for poor outcome, such as smoking status³².

The superior performance of artificial neural networks (ANN) on complex datasets make them well-suited for clinical applications including longitudinal data and multiplicative risk factors. A Recurrent neural network (RNN) is a special type of ANN tailored towards finding predictive patterns in longitudinal multivariate data, due to their use of internal states (or memory) to process sequences of inputs. This internal memory, and the ability to model trajectories, make RNNs a powerful tool for monitoring of patients staying in hospitals to prevent adverse events³³.

In this work, we compare three predictive modeling approaches, namely a logistic regression model, an ANN model, and an RNN model, to test the hypothesis that multiplicative risk factors (or interaction terms), and temporal features in the data can improve prediction of outcomes in aSAH patients. The rest of this paper is organized as follows. We start with a description of our aSAH patient cohort and the utilized machine learning models, results, and significance of our findings and future directions.

Data and Methods

This investigation was conducted according to Emory University Institutional Review Board approved protocol 66,389. Our preliminary dataset identified approximately 4,191 patients with confirmed aSAH by DSA dating back to January 1998. We used a subset containing 575 patients dating back to January 2009 for this preliminary study. The Emory Healthcare Clinical Data Warehouse (CDW) was queried for the 575 patients in our data subset, with data extraction query for demographic, clinical and imaging data identified in our literature review as associated with aSAH outcome in prior prognostic models. Associated clinical imaging studies and reports included in the CDW retrieval are: TCD, non-contrast head CT head CTA, head CTP, and DSA. Two board-certified neuroradiologists also reviewed each pa tients presenting head CT to determine MFS. Individual chart-review supplemented clinical factors not expected to be captured in the CDW retrieval, including level of consciousness, and World Federation of Neurosurgeons grade. The specific model features extracted were: hypertension^16,17, smoking status¹⁶, hyperglycemia¹⁶, diabetes mellitus¹⁶, increased intracranial pressure¹⁹, leukocytosis³⁴, level of consciousness^20,21, weight¹⁷, World Federation of Neurosur-geons grade^35,36, and blood transfusion³⁷, MFS^22,23, aneurysm treatment (surgical clipping or endovascular coil)³⁸, presence of perfusion mismatch on CTP^13,14,39, TCD vessel elevation > 140cm/sec¹¹, CVS on CTA^14,15, CVS on DSA^14,15, aneurysm location, number of aneurysms and aneurysm size³⁷. Inclusion criteria are patients 18 years of age or older with documented aSAH at admission based on the results of head CTA or DSA. Exclusion criteria are causes of SAH other than aneurysm including but not limited to trauma, infection, vascular malformation, vasculitis, post surgical complication, reversible cerebral vasoconstriction syndrome, and Moya Moya.

Two general types of variables are included in the model; time-variant features which includes laboratory values, vital signs and imaging exams, and time-invariant features which contains the information regarding demographic and patient history. Some variables such as vital signs and laboratory values are updated more frequently comparing to imaging exams. Each patient information was binned with 24-hour time intervals. Sample-and-hold technique was applied on the data. If there were more than one value for imaging exams, we took the median and if there were more than one value for vital signs and laboratory results, we took the mean of the available values. Missing values were imputed with the mean or median of the corresponding feature. Then, numerical features were normalized and transformed if needed.

The outcome labels for this study were the modified ranking scale (mRS) which is a commonly used scale for measur ing the degree of disability or dependence in the daily activities of people who have suffered a stroke or other causes of neurological disability. The mRS score ranges from zero to six, where an mRS of zero corresponds to having no symp toms of disability while an mRS score of six indicates that the patient has expired. We dichotomized the mRS scores into binary outcomes²⁴, corresponding to poor (mRS = 4, 5, 6) and favorable (mRS = 0, 1, 2, 3) outcomes.This measure was used as a surrogate of DCI and cerebral infarction, as this measure is a common secondary outcome used as a surrogate for DCI⁴⁰.

In order to predict patients mRS each day, the data was binned based on 24 hour time intervals. Therefore, for each patient we had a sequence of daily information starting from the end of the first day of ICU admission. In this cohort, each patient had only one outcome which was assigned 90 days after being discharged from the hospital. In order to make daily prediction for each patient, the outcome was replicated⁴¹ and assigned to every day that patient was hospitalized in the ICU.

Our initial model consisted of logistic regression with L1 and L2 regularization (also known as an Elastic Net model) with 10-fold internal cross-validation to optimize the penalty term. We hypothesized that both a neural network model and a recurrent neural network model with long short-term memory (LSTM) would provide more robust prediction in patients with aSAH when compared to logistic regression.

Traditional ANNs were not designed to analyze sequential data, which complicates analysis of longitudinal data sets in clinical settings. An RNN model can be thought of as multiple copies of the same network, each passing a message to a successor. This chain-like structure and the associated memory make the RNN models ideal for use with longitudinal data. To make daily predictions of the outcome of aSAH patients, we utilized a Long Short-Term Memory (LSTM) network, which is capable of learning long-term dependencies in time series data⁴².

The RNN model includes input layer containing sequential data, a LSTM layer with 50 hidden units, a fully connected layer followed by a Softmax layer and a classification layer. Number of hidden units was experimentally assigned, and a fully connected layer is added to capture all the interactions and nonlinear relationships among the features. To train the RNN model, mini batch technique was incorporated with mini batch size of 250, almost the maximum value possible. Adam optimizer was applied and the rest of the parameters in training process are initial learning rate=0.01, Gradient threshold=1, and Learning rate drop period=20. Moreover, a validation set was assigned to apply L2 regularization and prevent overfitting.

In order to make a fair comparison between the RNN model and the artificial neural network, we kept the same setting to train ANN, and only the LSTM layer was excluded from the model.

Results

Three predictive models were applied, namely an Elastic Net model, a feedforward ANN model with a fully connected layer and an LSTM-type recurrent neural network. To present the predictive power of LSTM, we compared the results between those three models. To show generalizability of our findings, 10-fold cross-validation was employed. We split the data into training, validation and testing sets, then used the validation set to tune the hyper-parameters of the models.

The greatest risk of morbidity and mortality occurs during days 2-15 following initial aSAH hemorrhage. Therefore our model prediction focused on days 2 -15 for outcome prediction. The performance measure we selected was area under the receiver operating curve (AUC-ROC or AUC) which is useful for comparing binary outcome classification models. The result of applying the trained models on different days of patient stay to predict the associated mRS is presented in Table 1 and Figure 2. Figure 2 demonstrates the average AUC of applying 10-fold cross-validation while predicting mRS on each day of patient stay in the ICU using the LSTM model. The vertical dotted lines present the error bar which is reasonably bounded. According to Figure 2, the overfitting is negligible and the LSTM model is performing well on both training and testing subsets. This model is capable of capturing non-linearity and temporal information in the data, and therefore in general we see an improvement in performance with inclusion of additional days of data. The best performance of the LSTM model is from day 8 to 13 in which AUC of 0.89 was achieved on testing set. Figure 1 (a) presents the average AUC from applying a NN model with 10-fold cross-validation to assess generalization performance of the model. The feedforward neural network models do not consider the temporal information in the data, but are capable of modeling non-linearity and interactions among the input features. The best performance of the NN model is on days 8, 11, and 12 which is 0.84. Figure 1 (b) shows the AUC of the ElasticNet model, which performed poorly compared to the NN and LSTM models. The main reason is that ElasticNet can neither capture temporal information nor non-linearity in the dataset. As a result, while this model is a regularized version of logistic regression model, ElasticNet is not capable of making meaningful predictions after day 5. In fact, the ElasticNet model severely overfits to the training data and performs poorly on the testing data, which is likely due to the smaller number of patient records available after day 5.

Table 1.

AUC-ROC of applying the trained LSTM, neural network and ElasticNet models to the training and testing sets. 10-fold cross-validation was applied, and the mean AUC across all folds is presented. Number of Samples denotes the available number of training and testing records as the number of days included in the analysis increases (note the drop in the number of records is due to death or discharge).

Number of Days	Number of Samples RNN (LSTM Model) Neural Network ElasticNet Model
	Training	Testing	Training AUC	Testing AUC	Training AUC	Testing AUC	Training AUC	Testing AUC
1	459	116	0.84	0.78	0.80	0.70	0.86	0.71
2	458	115	0.88	0.82	0.84	0.76	0.92	0.76
3	456	115	0.89	0.85	0.86	0.79	0.94	0.80
4	453	114	0.91	0.86	0.87	0.81	0.95	0.81
5	447	112	0.92	0.87	0.88	0.81	0.97	0.84
6	440	110	0.92	0.87	0.89	0.82	0.98	0.83
7	434	109	0.93	0.88	0.89	0.83	0.98	0.81
8	427	107	0.93	0.89	0.90	0.84	0.98	0.80
9	414	104	0.93	0.89	0.91	0.83	0.98	0.79
10	400	100	0.94	0.89	0.91	0.83	0.98	0.78
11	376	94	0.94	0.89	0.92	0.84	0.99	0.78
12	354	89	0.94	0.89	0.92	0.84	0.98	0.75
13	321	81	0.93	0.89	0.90	0.83	0.98	0.74
14	298	75	0.92	0.88	0.89	0.83	0.98	0.74
15	275	70	0.92	0.86	0.88	0.80	0.98	0.72

Figure 2.

AUC-ROC of applying the trained LSTM model on the training and testing sets. 10-fold cross-validation was applied. The two solid lines show the mean AUC of training and testing sets and the dotted lines are the error bars (standard deviations).

Figure 1.

(a) AUC-ROC of applying the feedforward Artificial Neural Network on the training and testing sets (10- fold cross-validated). All patient time series were aligned at admission and risk scores were calculated on a daily basis for up to 15 days. (b) Performance of the L1L2 regularized Logistic Regression model (aka, Elastic Net).

The details of predictive performance associated with the LSTM model, neural networks and ElasticNet model are presented in Table 1. The LSTM model constantly achieved better AUCs compared to the other models. Day 8 has the first best prediction performance for the LSTM model, so by testing the information on day 8 of the patient stay at ICU, we can obtain 0.89 prediction AUC. Time-dependent and time-independent features were employed in training the models. We performed sensitivity analysis in order to find the features which contributed most significantly to the AUC of the LSTM model. This analysis indicated that MRI and DSA features from imaging exams, Ø, Phosphorus, white blood cells, Bilirubin, and PaO2 from laboratory results, admission status, gender, and etiology category were the top features that contributed to the AUC achieved with the LSTM model.

Discussion and Concluding Remarks

Our results confirm our hypothesis that machine learning models can outperform traditional logistic regression in patients with aSAH. We believe that nonlinear interactions between variables, as well as the temporal relationships in longitudinal data are important to incorporate into a prediction model for this complex patient population. Our machine learning model design incorporates daily outcome prediction, which would theoretically allow physicians to more closely monitor patients at highest risk for poor outcome when used prospectively. Additionally, by focusing on an outcome related to DCI, our model does not solely rely on imaging features to potentially guide patient treatment decisions.

While this study was limited to a single institution and will require further validation, our results supports future exploration of the use of multivariate time-series prediction models such as LSTM neural networks on a larger, multi- institutional aSAH dataset. Further work could also explore the incorporation of live patient vital monitoring systems, which provide rich real-time datasets, as well as additional clinical outcomes such as mortality or cerebral infarction on follow up imaging. To eliminate the need for manual image annotation in a real-time implementation, one may build separate machine learning models to automatically annotate the images or use multimodal data fusion using a combination of convolutional neural networks for image feature extraction and recurrent neural networks for analysis of temporal information in time series data. The clinical implications of a successful predictive model for aSAH are profound, with the potential to reduce unnecessary radiation from inappropriate imaging, reduce hospital stay in low risk patients, and reduce morbidity in high risk patients. Further work is this area is warranted.

The results of this preliminary analysis demonstrate that a RNN with LTSM architecture provides robust evidence for clinical outcome prediction in patients with aSAH compared to traditional statistical techniques. We believe this supports further exploration with our full dataset, as well as expansion into a multicenter trial with a more diverse patient population.

Acknowledgments

This study was supported by the Radiology Society of North America (RSNA) Seed Grant.

Figures & Table

Table 2.

p-values from pair-wise comparisons between the AUC-ROCs on the testing sets: 1) RNN versus ElasticNet, 2) NN versus ElasticNet. Three methods of comparing classifier performance were applied, namely Net Reclas sification Improvement (NRI), Integrated Discrimination Improvement (IDI) and pauc⁴³. p-values below 0.05 are considered significant, indicating statistically significant improvement in classification performance using RNN over ElasticNet, etc.

Number of Days	RNN vs. ElasticNet Neural Network vs. ElasticNet
	NRI	IDI	pAUC	NRI	IDI	pAUC
1	0.71	0.53	0.84	0.96	0.91	0.98
2	0.96	0.77	0.98	0.22	0.21	0.52
3	0.21	0.21	0.78	0.14	0.13	0.47
4	0.09	0.08	0.54	0.09	0.08	0.42
5	0.88	0.87	0.94	0.36	0.41	0.60
6	0.04	0.02	0.28	0.03	0.008	0.14
7	0.02	0.006	0.16	0.05	0.03	0.16
8	0.04	0.02	0.25	0.07	0.05	0.30
9	0.07	0.05	0.32	0.13	0.12	0.36
10	0.13	0.12	0.49	0.41	0.54	0.67
11	0.41	0.41	0.56	0.12	0.12	0.35
12	0.41	0.40	0.56	0.18	0.2	0.49
13	0.49	0.50	0.64	0.29	0.41	0.46
14	0.14	0.14	0.43	0.94	0.80	0.97
15	0.14	0.13	0.42	0.33	0.42	0.62