Abstract
Whether endovascular thrombectomy (EVT) improves functional outcome in patients with large-vessel occlusion (LVO) stroke that do not comply with inclusion criteria of randomized controlled trials (RCTs) but that are considered for EVT in clinical practice is uncertain. We aimed to systematically identify patients with LVO stroke underrepresented in RCTs who might benefit from EVT. Following the premises that (i) patients without reperfusion after EVT represent a non-treated control group and (ii) the level of reperfusion affects outcome in patients with benefit from EVT but not in patients without treatment benefit, we systematically assessed the importance of reperfusion level on functional outcome prediction using machine learning in patients with LVO stroke treated with EVT in clinical practice (N = 5235, German-Stroke-Registry) and in patients treated with EVT or best medical management from RCTs (N = 1488, Virtual-International-Stroke-Trials-Archive). The importance of reperfusion level on outcome prediction in an RCT-like real-world cohort equaled the importance of EVT treatment allocation for outcome prediction in RCT data and was higher compared to an unselected real-world population. The importance of reperfusion level was magnified in patient groups underrepresented in RCTs, including patients with lower NIHSS scores (0–10), M2 occlusions, and lower ASPECTS (0–5 and 6–8). Reperfusion level was equally important in patients with vertebrobasilar as with anterior LVO stroke. The importance of reperfusion level for outcome prediction identifies patient target groups who likely benefit from EVT, including vertebrobasilar stroke patients and among patients underrepresented in RCT patients with low NIHSS scores, low ASPECTS, and M2 occlusions.
Supplementary Information
The online version contains supplementary material available at 10.1007/s12975-022-01040-5.
Keywords: Machine learning, Endovascular thrombectomy, Real-world data, Outcome prediction, Stroke
Introduction
Multiple randomized controlled trials (RCTs) have provided high-level evidence for the efficacy of endovascular thrombectomy (EVT) in patients with acute ischemic stroke caused by large vessel occlusion (LVO) of the anterior circulation [1]. Accordingly, EVT treatment is strongly recommended for patients that comply with RCT inclusion and exclusion criteria [2, 3]. In contrast, the evidence level and class of recommendation are weaker for patients that were underrepresented in RCTs such as those with low symptom burden, medium vessel occlusions, and higher ischemic core volume on baseline imaging [3]. Similarly, whether EVT is beneficial for vertebrobasilar LVO stroke remains uncertain [4, 5]. Yet, these patient groups are considered for EVT in clinical practice, and knowledge whether EVT improves functional outcome in these patients would strongly aid clinical decision-making. Prospective registries depict real-world data including patient groups underrepresented in RCTs but that were treated in clinical practice. Providing information on the outcome of these patients, real-world data complement RCTs in providing evidence for treatments [6, 7, 8] — a potential that has recently been appraised by regulators [9, 10]. However, observational data from prospective registries usually lack a control group, are thus inherently challenging to utilize for the identification of patients that might benefit from treatment, and accordingly provide lower level of evidence and class of recommendation in guidelines.
To overcome this limitation, we here propose and employ a methodological framework to systematically analyze the effect of treatment on functional outcome in real-world data. We followed the premise that only in patients that benefit from EVT treatment the final reperfusion grade (successful vs. unsuccessful) would impact on outcome, whereas in patients without benefit from EVT, information on reperfusion would not impact on outcome. Accordingly, using the value of EVT treatment allocation for outcome prediction in RCT data as a reference, we assessed the value of the final reperfusion grade (final modified Thrombolysis in Cerebral Infarction (mTICI) score) for the prediction of functional outcome in real-world data. To account for dependencies among a larger number of non-linear baseline predictors as observed in stroke patients, we applied a machine-learning algorithm to determine the value of EVT treatment for the prediction of functional outcome in both real-world and RCT patients with LVO stroke. Recently, such variable importance analyses have increasingly been used to rank the predictive values of individual features [11, 12, 13] and to select patients for treatment [14, 15], but have never been applied in the context of patient selection for EVT. Here, we employed the importance of reperfusion level for outcome prediction as a marker to systematically evaluate which patients with anterior and vertebrobasilar stroke might benefit from EVT beyond those complying with RCT criteria.
Methods
Study Samples
German Stroke Registry — Endovascular Treatment
We retrieved data from 6635 patients from the German Stroke Registry — Endovascular Treatment (GSR, ClinicalTrials.gov Identifier: NCT03356392), an ongoing, academic, prospective, multicenter registry in Germany [16]. GSR inclusion criteria were a diagnosis of acute ischemic stroke due to LVO, initiation of EVT, and age > 18 years without any exclusion criteria. Between 2015 and 2019, patients were recruited in 25 centers distributed across Germany. We selected variables prior to arterial puncture (see Supplemental Methods for further details) as well as the final modified Thrombolysis in Cerebral Infarction (mTICI) scale score from the GSR database. We used the mTICI scale as a technical outcome measure of EVT indicating reperfusion level and grouped it into complete (3), substantial (2b), and no or minimal (0–2a) successful reperfusion. We excluded 1400 patients for lack of information and other reasons (see Supplemental Methods and Supplemental Fig. 1 for further details). The study was conducted in accordance with the Declaration of Helsinki and was approved both centrally by the Institutional Review Board (IRB) of the Ludwig-Maximilians-Universität Munich (protocol No 689–15) and by local IRBs.
Virtual International Stroke Trials Archive — Endovascular
We retrieved data from 1615 patients with ischemic stroke due to LVO in the anterior circulation treated with EVT or best medical care from completed RCTs from the Endovascular subsection of the Virtual International Stroke Trials Archive (VISTA-Endovascular, see Supplemental Methods and Supplemental Fig. 1 for database description and exclusion criteria) [17]. If available in the VISTA database, we retrieved the same variables from VISTA-Endovascular that were selected from the GSR database (see Supplemental Methods for exclusion criteria of variables).
Functional outcome was assessed using the mRS score ranging from 0 (no symptoms) to 6 (death) at 90 days. The primary outcome measure was functional independence (mRS 0–2) at 90 days.
Machine Learning and Statistical Analyses
All analyses were performed in “R,” version 4.0.2.
Machine Learning Models
We predicted outcome with gradient boosting machines (GBMs). For each model respectively, data were randomly split into a training set (80%) and a test set (20%) with similar distribution of outcomes between sets. Within the training set, the model was trained and optimized with a repeated (n = 100) fivefold cross-validation. To avoid classification bias in a setting of uneven distribution of outcomes, we down-sampled data within each fold. For each model, we excluded patients with missing data (Supplemental Fig. 1). Predictive performances of machine-learning algorithms were determined in the test set and quantified as the AUC of the receiver operating characteristic (ROC). 95% confidence intervals of ROC curves were calculated using bootstrap replicates. ROC curves were compared with the Delong method. The value of each variable for the prediction of functional outcome was determined in the full datasets by assessing the change in model performance after permuting each variable a hundred times. A larger drop in performance corresponds to higher variable importance. To differentiate between analytical “importance” derived from these analyses and “importance” that interprets the strength of an effect or meaning, we have from here on indicated analytical importance by italic writing.
Results
Added Value of Reperfusion Level for Outcome Prediction
To verify our approach of utilizing the value of a technical outcome measure (final mTICI score) for the prediction of functional outcome as a marker indicating whether patients would benefit from EVT, we first determined the value of EVT treatment allocation for outcome prediction in data from 1488 patients with anterior LVO stroke treated with EVT or best medical care from EVT RCTs (VISTA-Endovascular, Supplemental Fig. 1, Table 1). Information on whether patients underwent EVT was the second most important variable after the NIHSS score for outcome prediction in RCT data (Fig. 1A). Also, adding information on EVT treatment allocation to a classifier based on pre-arterial puncture parameters significantly improved outcome prediction in RCT data (AUC 0.73 [95% CI, 0.66–0.80] vs 0.68 [95% CI, 0.60–0.76], p = 0.007, Fig. 1B, Supplemental Fig. 2). In RCT patients treated with EVT, the mTICI score was the most important variable for outcome prediction (Fig. 1C) and numerically similar to EVT treatment allocation (Fig. 1A). Using these findings as a reference, we next hypothesized that the mTICI score would contribute more to outcome prediction in an RCT-like real-world cohort compared to unselected real-world cohort: we thus determined how the mTICI score contributes to outcome prediction in real-world patients with anterior LVO stroke (data from the German Stroke Registry – Endovascular Treatment [GSR], Table 1) complying with RCT criteria (age 18–80 years, time from symptom onset to arterial puncture < 12 h, pmRS score zero or one) [1]. In this cohort, the mTICI score ranked second among outcome predictors (Fig. 1D), numerically comparable to EVT treatment allocation as well as mTICI score importance in RCT data (Fig. 1A, C), and significantly improved a pre-arterial puncture classifier for outcome prediction (AUC 0.79 [95% CI, 0.75–0.83] vs 0.74 [95% CI, 0.70–0.79], p = 0.001, Fig. 1B, Supplemental Fig. 2). In contrast, the variable importance of the mTICI score for outcome prediction was considerably lower in an unselected real-world cohort (Fig. 1D) as was the numerical improvement of the pre-arterial puncture classifier by adding the mTICI score (AUC 0.84 [95% CI, 0.81–0.87] vs 0.82 [95% CI, 0.79–0.85], p = 0.006: Fig. 1E). Similar findings were obtained when applying a generalized linear model (Supplemental Fig. 3A). To evaluate whether the value of the graded treatment marker mTICI for outcome prediction would be explained by adverse events from EVT or rather the benefit associated with treatment, we performed a sensitivity analysis by excluding patients with adverse events during EVT and importantly found that the mTICI score importance for outcome prediction did not change when excluding these patients (Fig. 1F). Of note, we further found that pre-arterial puncture variables did not have any informative value for predicting the mTICI score (Supplemental Fig. 4).
Table 1.
Baseline characteristics and procedural results of real-world and RCT patients
| Characteristics | GSR-anterior N = 4666 |
GSR-VB N = 569 |
VISTA N = 717 |
|---|---|---|---|
| Age, median (IQR) (years) | 76 (66–83) | 75 (65–82) | 68 (57–76) |
| Female, no. (%) | 2414 (51.7) | 244 (43.0) | 325 (45.3) |
| Medical history, no. (%) | |||
| Hypertension | 3585 (77.6) | 450 (80.5) | 380 (53.2) |
| Diabetes mellitus | 1032 (22.3) | 116 (21.1) | 95 (13.3) |
| Dyslipidemia | 1876 (40.6) | 215 (39.1) | 228 (32.8) |
| Atrial fibrillation | 1987 (43.1) | 188 (33.8) | – |
| Current smoking | 653 (17.1) | 74 (13.0) | 194 (28.4) |
| Drug use, no. (%) | |||
| Oral anticoagulation | 982 (21.5) | 88 (16.2) | – |
| Antiplatelet agents | 1429 (31.3) | 202 (37.2) | – |
| Pre-stroke mRS score, median (IQR) | 0 (0–1) | 0 (0–1) | 0 (0–0) |
| Baseline NIHSS score, median (IQR) | 15 (10–18) | 16 (7–26) | 17 (14–20) |
| Left side, no. (%) | 2414 (51.9) | NA | – |
| Affected vessel, no. (%) | |||
| M1 | 2818 (60.4) | NA | – |
| M2 | 1085 (23.3) | NA | – |
| ICA | 564 (12.1) | NA | – |
| ICA-T | 842 (18.0) | NA | – |
| Basilar artery | NA | 532 (93.5) | – |
| Vertebral artery only | NA | 37 (6.5) | – |
| General anesthesia, no. (%) | 2946 (65.2) | 493 (89.2) | – |
| Ship to interventional center, no. (%) | 1970 (42.2) | 235 (41.3) | – |
| Symptom onset known, no. (%) | 2830 (60.7) | 345 (60.6) | – |
| Out of hour admission, no. (%) | 2772 (59.4) | 324 (56.9) | – |
| High volume center, no. (%) | 2873 (61.6) | 367 (64.5) | – |
| ASPECTS, no. (%) | |||
| High (9–10) | 2205 (52.6) | NA | 349 (48.7) |
| Middle (6–8) | 1593 (38.0) | NA | 301 (42.0) |
| Low (0–5) | 377 (9.0) | NA | 67 (9.3) |
| Intravenous alteplase treatment, no. (%) | 2414 (52.0) | 258 (45.6) | 642 (89.5) |
| Symptom onset to arterial puncture, median (IQR) (min) | 220 (153–334) | 252 (164–399) | 181 (142–237) |
| mTICI, no. (%) | |||
| 3 | 2282 (49.6) | 344 (62.3) | – |
| 2b | 1609 (34.9) | 133 (24.1) | – |
| 0–2a | 714 (15.5) | 75 (13.6) | – |
mRS modified Rankin Scale, IQR interquartile range, NIHSS National Institutes of Health Stroke Scale, M1/2 first/second segment of the middle cerebral artery, ICA internal carotid artery, ICA-T internal carotid artery T, ASPECTS Alberta Stroke Program Early CT Score, min minutes, mTICI modified Thrombolysis in Cerebral Infarction, VB vertebrobasilar
Fig. 1.
Added value of reperfusion level for outcome prediction in RCT and real-world data. A Allocation to EVT treatment was the second most important variable for outcome prediction in RCT data (VISTA, EVT and best medical care group). B Adding information on whether patients underwent EVT improved the pre-arterial puncture model for outcome prediction in RCT data (VISTA, EVT and best medical care group) as did adding the mTICI score in an RCT-like real-world cohort (GSR). C The mTICI score was the most important predictor in RCT data (EVT group only). D Selecting real-world patients that comply with RCT inclusion criteria increased the importance value of the mTICI score for outcome prediction compared to an unselected real-world cohort (GSR). E Adding the mTICI score to pre-arterial puncture variables in an unselected real-world dataset did not considerably improve outcome prediction. F mTICI score importance was similar in RCT-like real-world patients without adverse events during EVT. A, C, D, F Conditional variable importance analyses. Shown are the median and 5% and 95% quantiles the of importance values. ML, machine learning; GSR, German Stroke Registry; VISTA, Virtual International Stroke Trials Archive; HTN, hypertension; DM, diabetes mellitus; pmRS, premorbid modified Rankin Scale; GBM, gradient boosting machine; AUC, area under the curve; VB, vertebrobasilar; NIHSS, National Institutes of Health Stroke Scale; EVT, endovascular treatment; Smok., history of smoking; OAC, oral anticoagulation; SOK, known symptom onset; HL, hyperlipidemia; ASPECTS, Alberta Stroke Program Early CT Score; IVT, intravenous thrombolysis; mTICI, modified Thrombolysis in Cerebral Infarction; RCT, randomized controlled trial; OTP, onset-to-puncture time; OTR, onset-to-randomization time
Added Value of Reperfusion Level in Patient Subgroups from Real-World Data
To identify patients that might benefit from EVT in real world beyond those complying with RCT criteria, we next determined the mTICI score variable importance for outcome prediction across subgroups of age, vessel location, the NIHSS score upon admission, onset-to-arterial puncture time, and the ASPECTS all stratified following approaches from a previous meta-analysis [1] and guidelines [3]. The mTICI score variable importance was higher in patients younger than 65 years compared to older patients, with lower (0–10) compared to higher (21–42) NIHSS scores, with longer (> 360 min) compared to shorter onset-to-arterial puncture times, and in those with lower (0–5) and medium (6–8) compared to higher (9–10) ASPECTS. Similar findings were obtained when applying a generalized linear model (Supplemental Fig. 3B). We further observed a trend for higher mTICI score variable importance in M2 compared to M1 and ICA occlusions. Values for patients with M2 occlusions, lower NIHSS, longer onset-to-puncture times, and lower APSECTS were similar to the value of EVT treatment allocation in RCT data (Fig. 2A). The distribution of the mTICI score was similar across subgroups indicating limited influence on its variable importance for outcome prediction (Supplemental Fig. 5). Considering the lower level of evidence for EVT efficacy in patients with M2 occlusions [18] and lower NIHSS scores [19], we next aimed to identify patients from these subgroups that might show a benefit from effective recanalization by EVT. Younger patients with M2 occlusions showed higher mTICI importance for outcome prediction than older patients, while there was no difference in mTICI importance between patients with higher and lower NIHSS (Fig. 2B). Among patients with lower NIHSS scores, those with lower age, ICA compared to MCA occlusions, and longer onset-to-puncture times showed higher mTICI importance for outcome prediction (Fig. 2C). The mTICI importance for outcome prediction was lower in patients with NIHSS scores between 0 and 5 compared to patients with NIHSS scores between 6 and 10 (Fig. 2D), but comparable to patients with NIHSS scores between 11 and 15 (Fig. 2A), with a similar trend for the prediction of excellent outcome (Fig. 2D).
Fig. 2.
mTICI importance for outcome prediction across baseline strata in real-world data. A mTICI importance values for outcome prediction were higher in younger patients, in M2 occlusions, with lower NIHSS scores, longer onset-to-puncture times, and lower ASPECTS when compared to other subgroups of the respective strata and similar to the variable importance of EVT treatment allocation in RCT data. B In patients with M2 occlusions, mTICI importance was higher in younger patients. C In patients with lower NIHSS scores, mTICI importance was higher in ICA over MCA occlusions and with longer onset-to-puncture times. D Patients with NIHSS scores between 0 and 5 showed lower mTICI importance for outcome prediction compared to patients with NIHSS scores between 6 and 10 with a similar trend observed for the prediction of excellent outcome (mRS 0–1). A–D Conditional variable importance analyses in real-world data (GSR). Shown are the median and 5% and 95% quantiles of importance values. mTICI, modified Thrombolysis in Cerebral Infarction; NIHSS, National Institutes of Health Stroke Scale; OTP, onset-to-puncture time; ASPECTS, Alberta Stroke Program Early CT Score; M1, first segment of the middle cerebral artery; ICA, internal carotid artery; EVT, endovascular thrombectomy; RCT, randomized controlled trial; mRS, modified Rankin Scale
Value of Reperfusion Level for Outcome Prediction in Vertebrobasilar Stroke
The added value of EVT in vertebrobasilar LVO stroke remains uncertain [20]. Here, using real-world data from 569 patients that underwent EVT following vertebrobasilar LVO stroke (Table 1), we found that adding the mTICI score to pre-arterial puncture variables did not significantly improve the pre-arterial puncture classifier (AUC 0.80 [95% CI, 0.70–0.90] vs. 0.78 [95% CI, 0.68–0.89], p = 0.18), also not when selecting real-world patients that comply with BASICS inclusion criteria (age 18–85 years, time from symptom onset to arterial puncture < 6 h, pmRS score < 3, NIHSS score > 9; AUC 0.67 [95% CI, 0.46–0.88] vs 0.68 [95% CI, 0.48–0.88], p = 0.89: Fig. 3A, Supplemental Figs. 1, 2) [21]. In real-world data, the variable importance of the mTICI score for outcome prediction ranked only sixth and was lower than the NIHSS score upon admission, age, the pmRS score, treatment with intravenous thrombolysis (IVT), and the onset-to-puncture time. Its value slightly increased in patients complying with BASICS inclusion criteria (Fig. 3B). While the variable importance of reperfusion by EVT was similar between anterior and vertebrobasilar LVO stroke with patient selection based on RCT inclusion criteria generally leading to higher values, the variable importance of additional IVT treatment was consistently higher in vertebrobasilar compared to anterior LVO stroke (Fig. 3C). To identify subgroups of patients with vertebrobasilar LVO stroke that might benefit from EVT, we determined mTICI score variable importance across different baseline strata and found a greater contribution to outcome prediction in patients older than 75 years, with lower NIHSS scores, and in patients shipped from primary to interventional centers (Fig. 3D).
Fig. 3.
IVT treatment is more important than the mTICI score following EVT in vertebrobasilar LVO stroke. A Adding the mTICI score to pre-arterial puncture variables did not significantly improve the ML model for outcome prediction in both unselected vertebrobasilar LVO stroke patients and a selected cohort of patients complying with RCT inclusion criteria (basilar artery occlusion, NIHSS score > 9, age 18 to 85 years, onset-to-puncture time < 6 h, pmRS score 0 to 2). B Selecting real-world patients that comply with RCT inclusion criteria slightly increased the importance value of the mTICI score for outcome prediction. C While the importance values of the mTICI score for outcome prediction were similar between anterior and vertebrobasilar datasets, the IVT treatment importance value was consistently higher in vertebrobasilar compared to anterior LVO stroke. D The mTICI score importance value was higher in older patients, with lower NIHSS scores, and in patients undergoing interhospital transfer. B–D Conditional variable importance analyses in real-world data (GSR) and RCT data (VISTA, C). Shown are the median and 5% and 95% quantiles of importance values. ML, machine learning; VB, vertebrobasilar; LVO, large-vessel occlusion; RCT, randomized controlled trial; mTICI, modified Thrombolysis in Cerebral Infarction; NIHSS, National Institutes of Health Stroke Scale; OTP, onset-to-puncture time; pmRS, premorbid modified Rankin Scale; IVT, intravenous thrombolysis; OAC, oral anticoagulation; SOK, known symptom onset; EVT, endovascular treatment; ship, interhospital transfer
Discussion
To identify LVO stroke patients that might benefit from EVT but were underrepresented in RCTs and following the premise that only in patients with benefit from EVT successful reperfusion would contribute to outcome prediction, we systematically assessed the value of the level of reperfusion for functional outcome prediction using a machine learning algorithm in data from > 6700 patients from both a large prospective multicenter real-world registry and from RCTs on EVT efficacy. We found that reperfusion level (assessed by the final mTICI score) was of similar importance for the prediction of functional outcome in RCT and RCT-like real-world patients as EVT allocation in RCT patients, which ranked second for outcome prediction in RCT data, in line with the high-level evidence for EVT from RCTs. The contribution of reperfusion level to outcome prediction was higher in an RCT-like compared to an unselected real-world cohort, thus correctly detecting higher EVT efficacy in the RCT-like cohort and endorsing its value as a marker for EVT treatment benefit. We thus utilized the value of reperfusion level for outcome prediction as a marker for treatment benefit in real-world data and identified patients that were largely excluded from RCTs but might benefit from EVT: patients with a lower NIHSS score, M2 occlusions, longer onset-to-puncture times, and a lower ASPECTS as well as older patients with vertebrobasilar LVO. Demonstrating the value of a technical outcome measure for the prediction of functional outcome, our study has the potential to inform future clinical trial design and influence clinical decisions.
While RCTs are indispensable to demonstrate treatment efficacy, they are usually restricted to selected patient populations. In contrast, real-world registry data are more comprehensive and usually include significantly more patients but are observational by nature and lack an untreated control group to allow investigation of treatment efficacy in subgroups excluded from or underrepresented in RCTs. Using a combinatory approach, we here drew on the advantages of both RCTs and real-world registry data to establish and apply the value of reperfusion level for outcome prediction as a marker that indicates benefit from EVT. Our data are in line with studies reporting reperfusion level as a strong predictor of outcome [22, 23]. However, in contrast to the well-known evidence for the link between the degree of reperfusion and functional outcome, we here determined in which patients successful reperfusion impacts functional outcome the most, by applying variable importance analyses. Specifically, in patients with benefit from EVT treatment, variable importance of mTICI is high, whereas in patients without substantial benefit from EVT treatment, information on recanalization (mTICI 2b/3 vs. 0) does not impact functional outcome considerably. Determining variable importance using machine learning has gained increasing attention across medical fields [11, 12, 15], but has never been applied in the context of our study goal. Here, variable importance analyses allowed us to assess the value of reperfusion level in conjunction with all available clinical variables, which is in contrast to the prevailing strategy to adjust only for pre-selected variables. To further consolidate whether the value of reperfusion level for functional outcome prediction could be interpreted as treatment benefit from EVT, future studies might also apply traditional statistical approaches such as logistic regression to compare odds ratios for the association of the level of reperfusion with functional outcome between different patient subgroups.
Following the established evidence for EVT efficacy in patients with anterior LVO stroke [1], there has been high interest to identify patient subgroups excluded from or underrepresented in these RCTs that would also benefit from EVT. Here, we found similar variable importance of reperfusion level for outcome prediction in patients with longer onset-to-puncture times and in patients with lower ASPECTS compared to EVT variable importance in RCT patients. This is in line with RCTs demonstrating EVT efficacy for extended time windows [24, 25] and encourages ongoing trials for lower ASPECTS, respectively [26]. While lower ASPECTS is associated with worse outcome when analyzed cross-sectionally [27], this does not necessarily imply that the treatment effect for this subgroup is smaller than for patients with higher ASPECTS: compared to patients from the pivotal RCT trials forming the HERMES study group with a median ASPECTS of 9 for which EVT increased the likelihood for good outcome at 90 days by 19.5% [1], patients with a large ischemic core (ASPECTS 3–5) from a recent Japanese RCT showed an EVT treatment benefit of 18.3% risk difference for the mRS score 0–3 at 90 days [28]. We also found similar variable importance of the mTICI score in patients with lower NIHSS scores in accordance with a recent meta-analysis indicating similar treatment effects of EVT in this subgroup [19]. Among patients with lower NIHSS scores upon admission, we found higher variable importance of the mTICI score for outcome prediction in younger patients, ICA rather than MCA occlusion, and longer onset-to-puncture times pointing towards treatment benefits in these subgroups. Collectively, these findings might describe a scenario, in which younger patients with mild symptoms show progressive tissue loss if not treated with EVT as their collateral flow, initially sufficient to perfuse most of the affected tissue, breaks down over time. Lastly, we observed a trend for higher importance of reperfusion level for outcome prediction in patients with M2 occlusions compared to M1 occlusions, substantiating the demand for an RCT investigating EVT efficacy in patients with medium vessel occlusion [18]. Such a trial could particularly recruit patients < 75 years, which showed high value of reperfusion level for outcome prediction. It is important to note that we observed overlapping confidence intervals for the importance of the mTICI score for outcome prediction between most subgroups, indicating that more work on other datasets is necessary to validate our findings.
In contrast to anterior circulation LVO stroke, the evidence for EVT efficacy in vertebrobasilar LVO stroke remains uncertain [4, 5, 20]. Analyzing a real-world cohort of 569 patients, we found similar variable importance of reperfusion level for outcome prediction in patients with anterior and vertebrobasilar LVO stroke. Focusing on subgroups of vertebrobasilar LVO stroke patients, we found longer onset-to-puncture times to be linked to higher variable importance of reperfusion level for outcome prediction, which is in line with data from the BASILAR registry [4], which allowed recruitment up to 24 h and showed higher EVT effects than the BASICS study [5]. We also observed that older patients with vertebrobasilar LVO stroke showed high variable importance of reperfusion level for outcome prediction in agreement with a subgroup analysis from the BASICS study [5]. Furthermore, IVT treatment showed a remarkably high variable importance for outcome prediction in vertebrobasilar LVO stroke patients undergoing EVT in line with the BASICS study having the highest IVT rate but lowest absolute risk reduction for poor outcome by EVT compared to the BEST study [20] and the BASILAR registry.
Our study has several strengths. We leveraged data from both a large real-world registry and from several RCTs on EVT efficacy covering > 6700 patients. Our sample size exceeded the sample size of previous studies on machine learning for the prediction of outcome after stroke by a factor of 3–10 [29–31] and allowed to establish a machine learning classifier that was superior to other machine learning models [29, 30, 31] and summative scores 32,33. In contrast, while allowing for the most important subgroup analyses, our sample size was insufficient to further subdivide subgroups such as patients with lower ASPECTS or to study patients with exclusive occlusions of the posterior or anterior cerebral artery. Our study is also limited by a potential selection bias in real-world data that could have influenced variable importance values for outcome prediction, for example by selecting only patients with dominant M2 (over non-dominant) occlusions for EVT and not selecting severely affected patients with ICA occlusion. It is further limited by the potential personal bias on mTICI evaluation and its limited validation for occlusion of the distal anterior or vertebrobasilar circulation. Lastly, a higher predictive treatment value might have also resulted if the treatment was associated with significant adverse events that impact on outcome. Here, we showed that the mTICI score value for outcome prediction is independent of adverse events occurring during EVT indicating that it must rather be the relation of higher mTICI scores with better outcome that drives the value of the mTICI score for outcome prediction.
In conclusion, we here identified several subgroups not well represented in RCTs that might also benefit from EVT including patients with a lower NIHSS score, M2 occlusions, longer onset-to-puncture times, and lower ASPECTS. Our data suggest that IVT treatment has high variable importance for outcome prediction in patients with vertebrobasilar LVO stroke. Our ML-based approach could serve as a blueprint for other medical fields on how to utilize the value of a graded treatment marker for outcome prediction in real-world and RCT data to inform clinical decisions and future RCT design.
Supplementary Information
Below is the link to the electronic supplementary material.
Acknowledgements
We thank Michael D. Hill (Department of Clinical Neurosciences, Hotchkiss Brain Institute, University of Calgary) for the critical revision of the manuscript.
Author Contribution
FQ and ST contributed to the conception and design of the study, acquisition, and analysis of data and drafted a significant portion of the manuscript and figures. FF contributed to the conception and design of the study and acquisition and analysis of data. VIM contributed to the conception and design of the study. GT and CG contributed to the conception and design of the study and drafted a significant portion of the manuscript and figures. All the other authors contributed to acquisition and analysis of data. Composition of the study groups “GSR investigators” and “VISTA-Endovascular collaborators” is detailed in Supplemental Tables I and II.
Funding
Open Access funding enabled and organized by Projekt DEAL. S. T. was supported by a grant from the Corona foundation.
Data Availability
The data that support the findings of this study were obtained from the German Stroke Registry (GSR) and the Virtual International Stroke Trials Archive (VISTA) but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. The R-script used for the current study can be obtained from the first author upon reasonable request.
Declarations
Conflict of Interest
FF reports personal fees from Eppdata. LK has received funding for travel or speaker honoraria from Bayer Vital, Boehringer Ingelheim, Bristol-Meyer-Squibb, Daiichi Sankyo, and Pfizer outside of this study. AH reports a grant from the European Commission Horizon2020 program (PRECISE4Q Grant No. 777 107, coordinator: Dietmar Frey) during the conduct of the study. DF reports grants from the European Commission: Horizon2020 program (PRECISE4Q Grant No. 777 107, coordinator: DF) and grants from the Federal Ministry of Research and Education (GO-Bio Grant No. 031B0154, lead: DF) during the conduct of the study. JF reports grants and personal fees from Anandis, Cerenovus, Microvention, Medtronic, Stryker, and personal fees from Phenox and Penumbra, all outside the submitted work. JF also serves as the CEO of Eppdata. MG reports personal fees from Medtronic, Stryker, Microvention, and Mentice, all outside the submitted work. In addition, MG has a patent on “Systems of acute stroke diagnosis” licensed to GE Healthcare, and a patent on “Systems of intracranial access” licensed to Microvention. CG reports personal fees from Amgen, Boehringer Ingelheim, Daiichi Sankyo, Abbott, Prediction Biosciences, Novartis, and Bayer, all outside the submitted work. GT reports grants and personal fees from Bayer, and personal fees from Acandis, BristolMyersSquibb/Pfizer, Boehringer Ingelheim, Daiichi Sankyo, Portola, and Stryker, all outside the submitted work. ST received funding from the Corona foundation outside of the submitted work. All the other authors declare no competing interests.
Footnotes
Contributor Information
Steffen Tiedt, Email: steffen.tiedt@med.uni-muenchen.de.
the GSR investigators and the VISTA-Endovascular Collaborators:
J. Berrouschot, A. Bormann, G. Bohner, C. H. Nolte, E. Siebert, S. Zweynert, F. Dorn, G. C. Petzold, F. Keil, W. Pfeilschifter, G. F. Hamann, M. Braun, B. Eckert, J. Röther, A. Alegiani, J. Fiehler, C. Gerloff, G. Thomalla, C. Kraemer, K. Gröschel, T. Uphaus, L. Kellert, S. Tiedt, C. Trumm, T. Boeckh-Behrens, S. Wunderlich, A. Ludolph, M. Petersen, F. Stögbauer, U. Ernemann, S. Poli, P. Khatri, M. Bendszuz, S. Bracard, J. Broderick, B. Campbell, A. Ciccone, A. Davalos, S. Davis, A. Demchuk, H. C. Diener, D. Dippel, G. A. Donnan, X. Ducrocq, J. Fiehler, D. Fiorella, G. Ford, M. Goyal, W. Hacke, M. Hill, R. Jahan, E. Jauch, T. Jovin, C. Kidwell, K. R. Lees, D. S. Liebeskind, C. B. Majoie, S. Martins, P. Mitchell, J. Mocco, K. Muir, R. G. Nogueira, J. L. Saver, W. J. Schonewille, A. H. Siddiqui, G. Thomalla, T. A. Tomsick, A. S. Turk, W. H. van Zwam, P. White, S. Yoshimura, and O. O. Zaidat
References
- 1.Goyal M, Menon BK, van Zwam WH, Dippel DW, Mitchell PJ, Demchuk AM, Davalos A, Majoie CB, van der Lugt A, de Miquel MA, et al. Endovascular thrombectomy after large-vessel ischaemic stroke: a meta-analysis of individual patient data from five randomised trials. Lancet. 2016;387:1723–1731. doi: 10.1016/S0140-6736(16)00163-X. [DOI] [PubMed] [Google Scholar]
- 2.Turc G, Bhogal P, Fischer U, Khatri P, Lobotesis K, Mazighi M, Schellinger PD, Toni D, de Vries J, White P, et al. European stroke organisation (eso) - european society for minimally invasive neurological therapy (esmint) guidelines on mechanical thrombectomy in acute ischaemic strokeendorsed by stroke alliance for europe (safe) Eur Stroke J. 2019;4:6–12. doi: 10.1177/2396987319832140. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Powers WJ, Rabinstein AA, Ackerson T, Adeoye OM, Bambakidis NC, Becker K, Biller J, Brown M, Demaerschalk BM, Hoh B, et al. Guidelines for the early management of patients with acute ischemic stroke: 2019 update to the 2018 guidelines for the early management of acute ischemic stroke: a guideline for healthcare professionals from the american heart association/american stroke association. Stroke. 2019;50:e344–e418. doi: 10.1161/STR.0000000000000211. [DOI] [PubMed] [Google Scholar]
- 4.Writing Group for the BG, Zi W, Qiu Z, Wu D, Li F, Liu H, Liu W, Huang W, Shi Z, Bai Y, et al. Assessment of endovascular treatment for acute basilar artery occlusion via a nationwide prospective registry. JAMA Neurol. 2020;77:561–573. [DOI] [PMC free article] [PubMed]
- 5.Langezaal LCM, van der Hoeven E, Mont'Alverne FJA, de Carvalho JJF, Lima FO, Dippel DWJ, van der Lugt A, Lo RTH, Boiten J, Lycklama ANGJ, et al. Endovascular therapy for stroke due to basilar-artery occlusion. N Engl J Med. 2021;384:1910–1920. doi: 10.1056/NEJMoa2030297. [DOI] [PubMed] [Google Scholar]
- 6.O'Leary CP, Cavender MA. Emerging opportunities to harness real world data: an introduction to data sources, concepts, and applications. Diabetes Obes Metab. 2020;22(Suppl 3):3–12. doi: 10.1111/dom.13948. [DOI] [PubMed] [Google Scholar]
- 7.Franklin JM, Patorno E, Desai RJ, Glynn RJ, Martin D, Quinto K, Pawar A, Bessette LG, Lee H, Garry EM, et al. Emulating randomized clinical trials with nonrandomized real-world evidence studies: first results from the rct duplicate initiative. Circulation. 2021;143:1002–1013. doi: 10.1161/CIRCULATIONAHA.120.051718. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Eichler HG, Pignatti F, Schwarzer-Daum B, Hidalgo-Simon A, Eichler I, Arlett P, Humphreys A, Vamvakas S, Brun N, Rasi G. Randomized controlled trials versus real world evidence: neither magic nor myth. Clin Pharmacol Ther. 2021;109:1212–1218. doi: 10.1002/cpt.2083. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.US Food and Drug Administration. Pdufa vi: Fiscal years 2018–2022. 2017. https://www.fda.gov/industry/prescription-drug-user-fee-amendments/pdufa-vi-fiscal-years-2018-2022. Accessed 5 June 2022.
- 10.Bonamici S. H.R.34–21st century cures act. 2016. https://www.congress.gov/bill/114th-congress/house-bill/34. Accessed 5 June 2022.
- 11.Ambale-Venkatesh B, Yang X, Wu CO, Liu K, Hundley WG, McClelland R, Gomes AS, Folsom AR, Shea S, Guallar E, et al. Cardiovascular event prediction by machine learning: The multi-ethnic study of atherosclerosis. Circ Res. 2017;121:1092–1101. doi: 10.1161/CIRCRESAHA.117.311312. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Williams SA, Kivimaki M, Langenberg C, Hingorani AD, Casas JP, Bouchard C, Jonasson C, Sarzynski MA, Shipley MJ, Alexander L, et al. Plasma protein patterns as comprehensive indicators of health. Nat Med. 2019;25:1851–1857. doi: 10.1038/s41591-019-0665-2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Kohoutova L, Heo J, Cha S, Lee S, Moon T, Wager TD, Woo CW. Toward a unified framework for interpreting machine-learning models in neuroimaging. Nat Protoc. 2020;15:1399–1435. doi: 10.1038/s41596-019-0289-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Ubels J, Schaefers T, Punt C, Guchelaar HJ, de Ridder J. Rainforest: a random forest approach to predict treatment benefit in data from (failed) clinical drug trials. Bioinformatics. 2020;36:i601–i609. doi: 10.1093/bioinformatics/btaa799. [DOI] [PubMed] [Google Scholar]
- 15.Ubels J, Sonneveld P, van Beers EH, Broijl A, van Vliet MH, de Ridder J. Predicting treatment benefit in multiple myeloma through simulation of alternative treatment effects. Nat Commun. 2018;9:2943. doi: 10.1038/s41467-018-05348-5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Wollenweber FA, Tiedt S, Alegiani A, Alber B, Bangard C, Berrouschot J, Bode FJ, Boeckh-Behrens T, Bohner G, Bormann A, et al. Functional outcome following stroke thrombectomy in clinical practice. Stroke. 2019;50(9):2500–2506. [DOI] [PubMed]
- 17.Khatri P, Hacke W, Fiehler J, Saver JL, Diener HC, Bendszus M, Bracard S, Broderick J, Campbell B, Ciccone A, et al. State of acute endovascular therapy: report from the 12th thrombolysis, thrombectomy, and acute stroke therapy conference. Stroke. 2015;46:1727–1734. doi: 10.1161/STROKEAHA.115.008782. [DOI] [PubMed] [Google Scholar]
- 18.Goyal M, Ospel JM, Menon BK, Hill MD. Mevo: The next frontier? J Neurointerv Surg. 2020;12:545–547. doi: 10.1136/neurintsurg-2020-015807. [DOI] [PubMed] [Google Scholar]
- 19.Goyal N, Tsivgoulis G, Malhotra K, Ishfaq MF, Pandhi A, Frohler MT, Spiotta AM, Anadani M, Psychogios M, Maus V, et al. Medical management vs mechanical thrombectomy for mild strokes: an international multicenter study and systematic review and meta-analysis. JAMA Neurol. 2020;77:16–24. doi: 10.1001/jamaneurol.2019.3112. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Liu X, Dai Q, Ye R, Zi W, Liu Y, Wang H, Zhu W, Ma M, Yin Q, Li M, et al. Endovascular treatment versus standard medical treatment for vertebrobasilar artery occlusion (best): an open-label, randomised controlled trial. Lancet Neurol. 2020;19:115–122. doi: 10.1016/S1474-4422(19)30395-3. [DOI] [PubMed] [Google Scholar]
- 21.van der Hoeven EJ, Schonewille WJ, Vos JA, Algra A, Audebert HJ, Berge E, Ciccone A, Mazighi M, Michel P, Muir KW, et al. The basilar artery international cooperation study (basics): study protocol for a randomised controlled trial. Trials. 2013;14:200. doi: 10.1186/1745-6215-14-200. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Dargazanli C, Consoli A, Gory B, Blanc R, Labreuche J, Preda C, Bourdain F, Decroix JP, Redjem H, Ciccio G, et al. Is reperfusion useful in ischaemic stroke patients presenting with a low national institutes of health stroke scale and a proximal large vessel occlusion of the anterior circulation? Cerebrovasc Dis. 2017;43:305–312. doi: 10.1159/000468995. [DOI] [PubMed] [Google Scholar]
- 23.Nogueira RG, Liebeskind DS, Sung G, Duckwiler G, Smith WS, Merci, Multi MWC. Predictors of good clinical outcomes, mortality, and successful revascularization in patients with acute ischemic stroke undergoing thrombectomy: pooled analysis of the mechanical embolus removal in cerebral ischemia (merci) and multi merci trials. Stroke. 2009;40:3777–3783. [DOI] [PubMed]
- 24.Nogueira RG, Jadhav AP, Haussen DC, Bonafe A, Budzik RF, Bhuva P, Yavagal DR, Ribo M, Cognard C, Hanel RA, et al. Thrombectomy 6 to 24 hours after stroke with a mismatch between deficit and infarct. N Engl J Med. 2018;378:11–21. doi: 10.1056/NEJMoa1706442. [DOI] [PubMed] [Google Scholar]
- 25.Albers GW, Marks MP, Kemp S, Christensen S, Tsai JP, Ortega-Gutierrez S, McTaggart RA, Torbey MT, Kim-Tenser M, Leslie-Mazwi T, et al. Thrombectomy for stroke at 6 to 16 hours with selection by perfusion imaging. N Engl J Med. 2018;378:708–718. doi: 10.1056/NEJMoa1713973. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Bendszus M, Bonekamp S, Berge E, Boutitie F, Brouwer P, Gizewski E, Krajina A, Pierot L, Randall G, Simonsen CZ, et al. A randomized controlled trial to test efficacy and safety of thrombectomy in stroke with extended lesion and extended time window. Int J Stroke. 2019;14:87–93. doi: 10.1177/1747493018798558. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Ospel JM, Kappelhof M, Kashani N, Menon BK, Campbell BCV, San Roman L, Demchuk AM, Dippel DWJ, Saver JL, Jovin TG, et al. Effect of age and baseline aspects on outcomes in large-vessel occlusion stroke: results from the hermes collaboration. J Neurointerv Surg. 2021;13:790–793. doi: 10.1136/neurintsurg-2020-016621. [DOI] [PubMed] [Google Scholar]
- 28.Yoshimura S, Sakai N, Yamagami H, Uchida K, Beppu M, Toyoda K, Matsumaru Y, Matsumoto Y, Kimura K, Takeuchi M, et al. Endovascular therapy for acute stroke with a large ischemic region. N Engl J Med. 2022;386:1303–1313. [DOI] [PubMed]
- 29.Alaka SA, Menon BK, Brobbey A, Williamson T, Goyal M, Demchuk AM, Hill MD, Sajobi TT. Functional outcome prediction in ischemic stroke: a comparison of machine learning algorithms and regression models. Front Neurol. 2020;11:889. doi: 10.3389/fneur.2020.00889. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Brugnara G, Neuberger U, Mahmutoglu MA, Foltyn M, Herweh C, Nagel S, Schonenberger S, Heiland S, Ulfert C, Ringleb PA, et al. Multimodal predictive modeling of endovascular treatment outcome for acute ischemic stroke using machine-learning. Stroke. 2020;51:3541–3551. doi: 10.1161/STROKEAHA.120.030287. [DOI] [PubMed] [Google Scholar]
- 31.van Os HJA, Ramos LA, Hilbert A, van Leeuwen M, van Walderveen MAA, Kruyt ND, Dippel DWJ, Steyerberg EW, van der Schaaf IC, Lingsma HF, et al. Predicting outcome of endovascular treatment for acute ischemic stroke: potential value of machine learning algorithms. Front Neurol. 2018;9:784. doi: 10.3389/fneur.2018.00784. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Sarraj A, Albright K, Barreto AD, Boehme AK, Sitton CW, Choi J, Lutzker SL, Sun CH, Bibars W, Nguyen CB, et al. Optimizing prediction scores for poor outcome after intra-arterial therapy in anterior circulation acute ischemic stroke. Stroke. 2013;44:3324–3330. doi: 10.1161/STROKEAHA.113.001050. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Ali Raza S, Xiang B, Jovin TG, Liebeskind DS, Shields R, Nogueira RG, Rangaraju S, Trevo2 study g. Pittsburgh response to endovascular therapy score as a pre-treatment prognostic tool: External validation in trevo2. Int J Stroke. 2017;12:494–501. [DOI] [PubMed]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
The data that support the findings of this study were obtained from the German Stroke Registry (GSR) and the Virtual International Stroke Trials Archive (VISTA) but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. The R-script used for the current study can be obtained from the first author upon reasonable request.