Machine Learning Prediction of Stroke Mechanism in Embolic Strokes of Undetermined Source

Abstract

Background and Purpose:

One-fifth of ischemic strokes are embolic strokes of undetermined source (ESUS). Their theoretical causes can be classified as cardioembolic versus non-cardioembolic. This distinction has important implications, but the categories’ proportions are unknown.

Methods:

Using data from the Cornell Acute Stroke Academic Registry, we trained a machine-learning algorithm to distinguish cardioembolic versus non-cardioembolic strokes, then applied the algorithm to ESUS cases to determine the predicted proportion with an occult cardioembolic source. A panel of neurologists adjudicated stroke etiologies using standard criteria. We trained a machine learning classifier using data on demographics, comorbidities, vitals, laboratory results, and echocardiograms. An ensemble predictive method including L1 regularization, gradient-boosted decision tree ensemble (XGBoost), random forests, and multivariate adaptive splines was used. Random search and cross-validation were used to tune hyperparameters. Model performance was assessed using cross-validation among cases of known etiology. We applied the final algorithm to an independent set of ESUS cases to determine the predicted mechanism (cardioembolic or not). To assess our classifier’s validity, we correlated the predicted probability of a cardioembolic source with the eventual post-ESUS diagnosis of atrial fibrillation.

Results:

Among 1,083 strokes with known etiologies, our classifier distinguished cardioembolic versus non-cardioembolic cases with excellent accuracy (area under the curve, 0.85). Applied to 580 ESUS cases, the classifier predicted that 44% (95% credibility interval, 39–49%) resulted from cardiac embolism. Individual ESUS patients’ predicted likelihood of cardiac embolism was associated with eventual atrial fibrillation detection (OR per 10% increase, 1.27; 95% CI, 1.03–1.57) (c-statistic, 0.68; 95% CI, 0.58–0.78). ESUS patients with high predicted probability of cardiac embolism were older and had more coronary and peripheral vascular disease, lower ejection fractions, larger left atria, lower blood pressures, and higher creatinine levels.

Conclusions:

A machine learning estimator that distinguished known cardioembolic versus non-cardioembolic strokes indirectly estimated that 44% of ESUS cases were cardioembolic.

Introduction

Stroke causes 10% of all deaths as well as a substantial burden of permanent disability.¹ The majority of strokes are ischemic, one-fifth of which lack a clear cause.² Such cryptogenic strokes often appear embolic and can be referred to as embolic strokes of undetermined source (ESUS).³ The natural history of ESUS is not benign, with 1 in 20 patients each year experiencing a recurrent stroke,⁴ which tends to be more severe than the initial stroke.⁵ Despite a substantial amount of investigation,^4,6 better treatments to prevent recurrent stroke after ESUS remain elusive, highlighting the need for a better understanding of the pathophysiology of ESUS.⁷

There are numerous theoretical causes of ESUS.³ Many of these potential mechanisms can be classified as cardioembolic—for example, atrial cardiopathy,⁸ unrecognized myocardial infarction,⁹ or patent foramen ovale¹⁰—in contrast to other common conditions such as nonstenosing atherosclerosis¹¹ or other forms of vasculopathy. This broad distinction between cardioembolic versus non-cardioembolic sources may have important therapeutic implications.^12,13 However, the classification of ESUS into cardioembolic versus non-cardioembolic etiologies remains difficult in the absence of a gold-standard diagnostic test. In such a setting, machine learning may provide novel insight by applying rules extracted from labeled cases (i.e., strokes with a known etiology) to unlabeled cases (i.e., ESUS).

Methods

Design

We developed and tested a machine learning algorithm using data from ischemic stroke patients enrolled in the Cornell AcutE Stroke Academic Registry (CAESAR). We trained the algorithm on known ischemic stroke subtypes in our registry and applied the algorithm to ESUS cases to assess the predicted proportion of cardioembolic versus non-cardioembolic sources (Figure 1). The institutional review board at Weill Cornell Medicine approved this study and waived the requirement for informed consent. Deidentified data and analytic code are available from the corresponding author upon reasonable request.

Figure 1.

Design of Machine Learning Analysis of Mechanism in Embolic Strokes of Undetermined Source.

Patient Population

All patients hospitalized at New York-Presbyterian Hospital/Weill Cornell Medical Center for acute stroke are prospectively enrolled in the American Heart Association’s Get With The Guidelines—Stroke registry. Trained hospital analysts prospectively collect data on demographics, vascular risk factors and comorbidities, stroke severity, and in-hospital treatments and outcomes. CAESAR combines the Get With The Guidelines data plus additional clinical, laboratory, and radiographic data.¹⁴ All clinically diagnosed cases of ischemic stroke are reviewed by a panel of at least three neurologists who adjudicate the etiology per the Trial of Org 10172 in Acute Stroke Treatment (TOAST) classification¹⁵ and the consensus definition of ESUS.¹⁶ The adjudication panel comprises four board-certified neurologists with an average 6 years of post-training experience. Adjudications were performed by group discussion and consensus after review of all available data, including neuroimaging data, from the index hospitalization. In a validation analysis of a subset of 374 patients, double-blind adjudication by two independent reviewers indicated an inter-rater agreement rate of 86.4% and a kappa of 0.81. For this analysis, we included ischemic stroke patients registered in CAESAR from 2011 through 2016. We excluded lacunar strokes because we were interested in better understanding the root causes of ESUS, which is by definition nonlacunar. We excluded patients with an incomplete stroke evaluation. We also excluded patients with multiple potential causes of stroke because the management of these patients often addresses all the known causes, rendering the delineation of the exact cause of stroke less important, and because we wanted to facilitate identification of mechanisms and risk factors unique to individual, well-defined stroke subtypes. Therefore, our analysis included patients with ESUS as well as patients with the following known ischemic stroke etiologies: large-artery atherosclerosis, cardiac embolism, or another determined etiology (e.g., cervical artery dissection). Cases with a known ischemic stroke subtype were labeled as either cardioembolic or non-cardioembolic (i.e., large-artery atherosclerosis or another determined etiology).

Measurements

As features on which to train our machine learning classifier, we used data on patients’ demographics, vascular risk factors and comorbidities, vital signs, laboratory results, and echocardiographic findings (Supplemental Table I). We did not include a history of AF as a training variable since it was essentially completely collinear with a cardioembolic etiology and, by definition, was absent in all ESUS cases. Demographics and risk factors and comorbidities were collected by hospital analysts for the Get With The Guidelines—Stroke registry and were imported directly into the Microsoft SQL server which houses the CAESAR registry. Echocardiographic measurements were imported directly from our hospital’s echocardiographic image management system (Xcelera, Philips Healthcare) into the CAESAR SQL server. Vital signs and laboratory results were imported directly into the CAESAR SQL server from our hospital’s inpatient (Allscripts, Cerner) and outpatient (EPIC) electronic medical record systems. For patients with multiple measurements of a given vital sign or laboratory test, we used the first value after admission.

Statistical Analysis

We estimated the proportion of ESUS arising from occult cardiac sources using a Bayesian targeted machine learning algorithm.¹⁷ The algorithm estimates the proportion in three steps: (i) build two predictive models: one for the probability that the source of stroke is undetermined, and one for the probability of a cardioembolic source among patients with a determined source; (ii) tilt the probability of a cardioembolic source towards a de-biased function in order to achieve the right bias-variance trade-off; and (iii) incorporate prior information using a Bayesian updating step. A uniform distribution in the interval [0, 1] was used as an uninformative prior for the proportion of ESUS or cardiac source. The predictive models in step (i) were fitted using an ensemble machine learning method known as the super learner. The super learner^18,19 algorithm proceeds by splitting the data into 10-fold validation sets. For each validation set, the corresponding training set is given by the remaining 9 folds. Each candidate model is fitted in each training set; hyper-parameters are tuned using 8-fold nested cross-validation. In each training set, the outcome is regressed on the out-of-sample predictions of each candidate model, in order to find weights in an ensemble that minimize the cross-entropy loss function. This ensemble is used to obtain predictions in the corresponding validation sets. The ensemble consisted of gradient-boosted decision trees, random forests, L1 regularized logistic regression, multivariate adaptive regression splines, and vanilla logistic regression. A screening step selecting the 50 variables with the largest univariate correlations with the outcome was used for the latter two algorithms in order to avoid over-fitting. Cross-validation was used to calculate the area under the curve for out-of-sample predictions of known stroke etiologies. These predictive models were then used to compute a Bayesian targeted machine learning estimator for the proportion of ESUS arising from an occult cardioembolic source.

Validation

To assess the validity of our classifier among ESUS patients, we correlated our machine learning estimator’s predicted probability of an occult cardioembolic source with eventual diagnosis of atrial fibrillation (AF) on post-discharge heart-rhythm monitoring tests. After hospital discharge, ESUS patients at our medical center routinely undergo 30 days of continuous heart-rhythm monitoring using an external loop recorder with a highly sensitive automated AF detection algorithm. All automated AF detections are reviewed and adjudicated as true or false AF detections by a cardiac electrophysiologist. AF is defined as sustained AF lasting >30 seconds. We considered eventual post-stroke AF detection to be a reasonable construct for testing the validity of our estimator, since we would expect patients with a higher predicted probability of an occult cardioembolic source to more often eventually manifest AF than patients with a lower predicted probability of an occult cardioembolic source.

Sensitivity Analysis

We performed two sensitivity analyses. First, we included as training features additional data from a radiologist’s interpretation of brain imaging which were available for a subset of patients enrolled during 2011–2015. These brain imaging features were abstracted by a single radiologist blinded to the machine learning classifier’s output, and included the Age-Related White Matter Changes score, the Fazekas score, the number and location of acute infarcts, and the number and location of microbleeds (Supplemental Table I). Second, we excluded patients who were transferred from other hospitals, thereby essentially limiting our population to local patients from neighboring zip codes in Manhattan. Third, we included AF as a training feature.

Results

Among 2,116 ischemic stroke patients registered in CAESAR from 2011 through 2016, we excluded 202 patients with stroke due to small-vessel occlusion, 139 patients with an incomplete evaluation, and 112 patients with two or more potential causes of stroke. Among the remaining 1,663 patients, 1,083 had a known stroke etiology (291 had stroke due to large-artery atherosclerosis, 688 due to cardiac embolism, and 104 due to another determined etiology), while 580 had cryptogenic stroke which was defined per the ESUS consensus definition based on information available at the time of the index hospitalization (i.e., prior to post-discharge AF monitoring) (Table 1).

Table 1.

Baseline Characteristics of Eligible CAESAR Patients, Stratified by Ischemic Stroke Subtype.

Characteristic*	Large-Artery Atherosclerosis (N = 291)	Cardiac Embolism (N = 688)	Other Determined Source (N = 104)	Undetermined Source (N = 580)	P value
Age, mean (SD), yrs	72 (12)	75 (14)	56 (18)	67 (17)	<0.001
Female	115 (39.5)	368 (53.5)	40 (38.5)	310 (53.5)	<0.001
White†	98 (50.5)	290 (65.9)	38 (55.1)	251 (59.5)	0.002
Insurance					<0.001
Commercial	121 (41.6)	298 (43.3)	67 (64.4)	298 (51.4)
Medicare	120 (41.2)	299 (43.5)	17 (16.4)	201 (34.7)
Medicaid	44 (15.1)	82 (11.9)	16 (15.4)	66 (11.4)
Other	6 (2.1)	9 (1.3)	4 (3.9)	15 (2.6)
Hypertension	238 (81.8)	475 (69.0)	52 (50.0)	349 (60.2)	<0.001
Diabetes mellitus	98 (33.7)	138 (20.1)	16 (15.4)	127 (21.9)	<0.001
Active tobacco use	164 (56.4)	328 (47.7)	32 (30.8)	257 (44.3)	<0.001
Coronary artery disease	40 (13.8)	39 (5.7)	11 (10.6)	36 (6.2)	<0.001
Heart failure	54 (18.6)	179 (26.0)	19 (18.3)	86 (14.8)	<0.001
Chronic kidney disease	9 (3.1)	70 (10.2)	1 (1.0)	15 (2.6)	0.90
Peripheral vascular disease	16 (5.5)	44 (6.4)	4 (3.9)	23 (4.0)	0.24
Prior stroke	16 (5.5)	44 (6.4)	4 (3.9)	23 (4.0)	0.004
Onset to arrival, median (IQR), hrs	1.7 (0.3–14.6)	0.9 (0.1–4.9)	1.3 (0.2–9.8)	2.5 (0.4–12.9)	<0.001
NIHSS score on arrival, median (IQR)	4 (1–9)	8 (2–18)	4 (2–14)	3 (1–9)	<0.001
Ejection fraction, mean (SD), %	62 (9)	53 (17)	63 (7)	62 (8)	<0.001
LAVI, mean (SD), mL/m2	31 (11)	50 (33)	32 (14)	34 (14)	<0.001
E/e’ lateral	9.9 (3.7)	11.8 (7.2)	8.5 (3.3)	9.6 (5.1)	0.003
E/e’ medial	13.9 (5.4)	14.5 (7.1)	10.2 (4.3)	13.2 (7.0)	0.05
SBP, mean (SD), mm Hg	157 (31)	145 (30)	152 (48)	150 (30)	<0.001
Serum creatinine, mean (SD), mg /dL	1.2 (0.9)	1.2 (1.0)	1.2 (1.1)	1.1 (0.9)	0.01

Abbreviations: IQR, interquartile range; IV TPA, intravenous tissue plasminogen activator; LAVI, left atrial volume index; NIHSS, National Institutes of Health Stroke Scale; SBP, systolic blood pressure; SD, standard deviation.

^*Data are presented as number (%) unless otherwise specified.

^†Numbers do not sum up to column total because of patients with missing data on race.

Among the 1,663 patients with stroke whose etiology was determined, our predictive model distinguished cardioembolic from non-cardioembolic etiologies with excellent validity (area under the curve, 0.85) (Figure 2; Supplemental Figure I). The Bayesian targeted machine learning estimator which incorporated this predictive model estimated that 44% (95% credibility interval, 39–49%) of ESUS cases were due to an occult cardioembolic source (Figure 3). Compared to ESUS patients whose predicted likelihood of an occult cardioembolic source was in the lowest quartile (<25%), ESUS patients with a predicted likelihood of cardiac embolism in the highest quartile (≥75%) were older and more often white; more often had coronary artery disease, heart failure, and peripheral vascular disease; were less likely to be active smokers; and had more severe strokes, lower ejection fractions, larger left atria, lower blood pressure, and higher serum creatinine levels (Table 2).

Figure 2.

Receiver Operating Characteristics Curve for Bayesian Targeted Machine Learning Estimator of Probability of Occult Cardioembolic Source.

Figure 3.

Distribution of Percentage of ESUS Cases Predicted To Have Occult Cardioembolic Source. Caption: The shaded grey areas indicate the 68% and 95% credibility intervals.

Table 2.

Baseline Characteristics of Patients with ESUS, Stratified by Machine Learning Prediction of a Cardioembolic Source.

Characteristic*	Highest Predicted Probability of Cardiac Embolism† (N = 71)	Lowest Predicted Probability of Cardiac Embolism‡ (N = 149)	P value
Age, mean (SD), yrs	78 (11)	60 (16)	<0.001
Female	38 (53.5)	68 (45.6)	0.27
White§	34 (75.6)	55 (53.4)	0.01
Payment source			0.09
Medicare	32 (45.1)	78 (52.4)
Medicaid	32 (45.1)	43 (28.9)
Private	5 (7.0)	20 (13.4)
Self-pay	2 (2.8)	8 (5.4)
Hypertension	45 (63.4)	97 (65.1)	0.80
Diabetes mellitus	16 (22.5)	36 (24.2)	0.79
Active tobacco use	1 (1.4)	15 (10.1)	0.02
Coronary artery disease	21 (29.6)	13 (8.7)	<0.001
Heart failure	6 (8.5)	1 (0.7)	0.002
Chronic kidney disease	1 (1.4)	3 (2.0)	0.75
Peripheral vascular disease	9 (12.7)	3 (2.0)	0.001
Prior stroke	18 (25.4)	26 (17.5)	0.17
Onset to arrival, median (IQR), hrs	2.1 (1.0–5.5)	2.8 (0.4–20.0)	0.73
NIHSS score on arrival, median (IQR)	8 (3–16)	3 (1–6)	<0.001
Ejection fraction, mean (SD), %	54 (11)	65 (6)	<0.001
LAVI, mean (SD), mL/m2	47 (19)	28 (9)	<0.001
E/e’ lateral	10.3 (3.8)	9.2 (3.6)	0.38
E/e’ medial	14.8 (5.1)	12.4 (4.6)	0.16
SBP, mean (SD), mm Hg	147 (29)	159 (30)	0.01
Serum creatinine, mean (SD), mg /dL	1.5 (1.7)	1.0 (0.8)	0.01

Abbreviations: IQR, interquartile range; IV TPA, intravenous tissue plasminogen activator; LAVI, left atrial volume index; NIHSS, National Institutes of Health Stroke Scale; SBP, systolic blood pressure; SD, standard deviation.

^*Data are presented as number (%) unless otherwise specified.

^†Among the 580 patients with embolic stroke of undetermined source, these data are for the 71 patients to whom our algorithm assigned a predicted probability of a cardioembolic source ≥0.75.

^‡Among the 580 patients with embolic stroke of undetermined source, these data are for the 149 patients to whom our algorithm assigned a predicted probability of a cardioembolic source <0.25.

^§Numbers do not sum up to column total because of patients with missing data on race.

In our validation step, among the 580 patients who were all classified as ESUS after review of all available data including neuroimaging data, our machine learning classifier’s estimated likelihood of an occult cardioembolic source was associated with the eventual detection of AF (OR per 10% increase in predicted probability of cardioembolic source, 1.27; 95% CI, 1.03–1.57) (c-statistic, 0.68; 95% CI, 0.58–0.78); AF was found in 9.1% of those in the top quartile versus 2.9% of those in the bottom quartile of predicted likelihood of a cardioembolic source. In a sensitivity analysis adding as a training feature brain imaging data from the 1,369 patients (82.3%) enrolled during 2011–2015, the predicted proportion of patients with a cardioembolic source remained 44%. Our results were also similar after excluding the 352 patients (21.2%) who were transferred from another hospital, and when including AF as a training feature.

Discussion

In a registry of patients with ischemic stroke, we developed a machine learning classifier that accurately sorted known stroke etiologies into cardioembolic versus non-cardioembolic categories. Among ESUS cases, the predicted probability of an occult cardioembolic source, which was determined using only information available during the initial stroke hospitalization, was associated with the eventual diagnosis of paroxysmal AF after hospital discharge. When this classifier was applied to ESUS cases, it indirectly estimated that slightly fewer than half of currently unclassified strokes were due to an occult cardioembolic source.

Numerous studies have investigated individual conditions which may represent occult mechanisms of ESUS,^9,20–22 and several studies have investigated relationships among various mechanisms.^10,23,24 Such investigations make it increasingly clear that ESUS is a heterogeneous clinical condition with a variety of underlying mechanisms, some of which are cardioembolic and some of which are not.^7,11,25,26 There are few data on the relative proportions of cardioembolic versus non-cardioembolic mechanisms of ESUS, or the characteristics of ESUS patients with presumed cardioembolic versus non-cardioembolic mechanisms. In this context, our study provides several novel findings. First, our results suggest that slightly less than half of ESUS cases may represent occult cardiac embolism. Second, our results suggest that ESUS patients with a likely cardioembolic source have a distinct clinical profile compared to patients with a likely non-cardioembolic source. Those with likely cardiac embolism appear to be an older group with more cardiac comorbidities, worse cardiac function, and more severe strokes, while those with likely non-cardioembolic sources appear to be a younger group with higher blood pressure and a greater prevalence of active tobacco use.

Our study has several limitations. First, our data were from a single academic medical center and thus our algorithm and findings may not be generalizable to other populations. On the other hand, approximately 80% of our stroke patients are admitted through our emergency department and are residents of local neighborhoods in which the majority of strokes are admitted either to our medical center or to other nearby academic medical centers,²⁷ which may mitigate the risk of referral bias. Our findings were essentially identical when we limited our analysis to this local population. Second, in the absence of a gold standard, we cannot verify our machine learning estimator’s prediction that 44% of ESUS cases represent a cardioembolic source. However, our estimator had excellent performance among known cases of cardioembolic and non-cardioembolic stroke, and the validity of its predictions are supported by their association with the eventual diagnosis of AF. Nevertheless, post-stroke AF diagnosis is not definitive proof that the mechanism of the index stroke was cardioembolic, so these findings provide only indirect corroboration. Some cases of post-stroke AF may represent the results of stroke-induced damage to autonomic pathways, rather than a clue as to the underlying etiology of stroke. Third, we lacked pixel-level brain imaging data and it is possible that the inclusion of such granular data would have improved the performance of our classifier and changed its findings when applied to ESUS cases. However, our findings were identical even with the inclusion of detailed data on brain imaging findings which were available for the majority of our cohort. It may be that basic neuroimaging features such as infarct location and white matter disease grade may not contribute much to the algorithm once other features such as echocardiography findings are taken into account. Future studies are required to determine whether more granular neuroimaging features, including imaging of the intracranial and extracranial large arteries and aorta, can improve prediction. Fourth, we lacked data on recurrent stroke and thus could not examine the association between the predicted probability of a cardioembolic source and the risk of stroke recurrence. Fifth, we used cross-validation to obtain an efficient estimate of predictive error. The alternative approach of splitting the data into a training set and a testing set may have resulted in different estimates.

Our study may have several potential implications. First, our findings highlight the phenotypic heterogeneity of ESUS, and support and inform personalized approaches to its treatment. For example, among many other factors, older patients and those with large left atria were over-represented among patients predicted to have a high risk of an occult cardioembolic source. These findings are concordant with secondary analyses of randomized clinical trials which found a benefit with anticoagulant therapy in these two subgroups,^6,13 even though there was no benefit in the overall ESUS population.^4,6 Our study suggests that the findings of these subgroup analyses of the ESUS trials may indicate true therapeutic effects in these selected patient groups who appear to be enriched for underlying cardioembolic sources. Randomized trials of anticoagulant therapy in only ESUS patients with markers of cardiac disease, such as ARCADIA²⁸ and ATTICUS²⁹, are ongoing. If positive, these trials will serve as proof-of-concept of personalized antithrombotic strategies for preventing stroke recurrence after ESUS, but ultimately more refined risk stratification of an underlying cardioembolic source will be necessary to guide trials of not just current oral anticoagulants but novel therapies such as emerging Factor XIa inhibitors. Our study may inform the findings of such future trials by suggesting other clinical selection criteria, such as older age, pre-existing coronary artery disease or peripheral vascular disease, and reduced ejection fraction. Second, our study suggests that machine learning models may have utility not just for predicting events, but for interrogating disease states to shed light on the underlying pathophysiology. This latter application of machine learning has the advantage of easy dissemination via scientific publication, while prediction tools are often difficult to disseminate in the absence of a commercial software product or other platform for inputting the required data.

Conclusions

A machine learning algorithm trained on demographic, clinical, echocardiographic, and laboratory data was able to distinguish known cardioembolic strokes from known non-cardioembolic strokes with excellent accuracy. Among ESUS cases, the predicted probability of occult cardiac embolism was significantly associated with the eventual diagnosis of AF. When applied to patients with ESUS, the algorithm indirectly estimated that slightly fewer than half of these strokes of unknown etiology were due to an occult cardioembolic source. ESUS patients with a high probability of an occult cardioembolic source were older, more often had cardiac disease, and had lower ejection fractions and larger left atria. These findings may shed light on the pathophysiology of stroke and inform the design of future trials on stroke prevention after ESUS.

Non-standard Abbreviations and Acronyms

AF

Atrial fibrillation

CAESAR

Cornell Acute Stroke Academic Registry

ESUS

Embolic stroke of undetermined source

TOAST

Trial of Org 10172 in Acute Stroke Treatment