This prognostic study investigates the effectiveness of a prognostic tool to predict the functional outcome of individual patients after endovascular treatment for ischemic stroke.
Key Points
Question
Can functional outcome be predicted for individual patients after endovascular treatment (EVT) for ischemic stroke?
Findings
In this prognostic study including 781 and 3260 patients in the derivation and validation cohorts, respectively, a prediction model was developed and validated using data from multiple clinical trials and routine practice of patients who received EVT within 12 hours after stroke onset. The model includes 9 preprocedural and postprocedural characteristics, has excellent discriminative ability and good calibration, and is available online for clinical use.
Meaning
The prediction model for functional outcome of patients after EVT is a simple prognostication tool that can provide patients, family members, and physicians with reliable and accurate outcome expectations 1 day after EVT.
Abstract
Importance
Outcome prediction after endovascular treatment (EVT) for ischemic stroke is important to patients, family members, and physicians.
Objective
To develop and validate a model based on preprocedural and postprocedural characteristics to predict functional outcome for individual patients after EVT.
Design, Setting, and Participants
A prediction model was developed using individual patient data from 7 randomized clinical trials, performed between December 2010 and December 2014. The model was developed within the Highly Effective Reperfusion Evaluated in Multiple Endovascular Stroke Trials (HERMES) collaboration and external validation in data from the Dutch Multicenter Randomized Clinical Trial of Endovascular Treatment for Acute Ischemic Stroke in the Netherlands (MR CLEAN) Registry of patients treated in clinical practice between March 2014 and November 2017. Participants included patients from multiple centers throughout different countries in Europe, North America, East Asia, and Oceania (derivation cohort), and multiple centers in the Netherlands (validation cohort). Included were adult patients with a history of ischemic stroke from an intracranial large vessel occlusion in the anterior circulation who underwent EVT within 12 hours of symptom onset or last seen well. Data were last analyzed in July 2022.
Main Outcome(s) and Measure(s)
A total of 19 variables were assessed by multivariable ordinal regression to predict functional outcome (modified Rankin Scale [mRS] score) 90 days after EVT. Variables were routinely available 1 day after EVT. Akaike information criterion (AIC) was used to optimize model fit vs model complexity. Probabilities for functional independence (mRS 0-2) and survival (mRS 0-5) were derived from the ordinal model. Model performance was expressed with discrimination (C statistic) and calibration.
Results
A total of 781 patients (median [IQR] age, 67 [57-76] years; 414 men [53%]) constituted the derivation cohort, and 3260 patients (median [IQR] age, 72 [61-80] years; 1684 men [52%]) composed the validation cohort. Nine variables were included in the model: age, baseline National Institutes of Health Stroke Scale (NIHSS) score, prestroke mRS score, history of diabetes, occlusion location, collateral score, reperfusion grade, NIHSS score at 24 hours, and symptomatic intracranial hemorrhage 24 hours after EVT. External validation in the MR CLEAN Registry showed excellent discriminative ability for functional independence (C statistic, 0.91; 95% CI, 0.90-0.92) and survival (0.89; 95% CI, 0.88-0.90). The proportion of functional independence in the MR CLEAN Registry was systematically higher than predicted by the model (41% vs 34%), whereas observed and predicted survival were similar (72% vs 75%). The model was updated and implemented for clinical use.
Conclusion and relevance
The prognostic tool MR PREDICTS@24H can be applied 1 day after EVT to accurately predict functional outcome for individual patients at 90 days and to provide reliable outcome expectations and personalize follow-up and rehabilitation plans. It will need further validation and updating for contemporary patients.
Introduction
Since the implementation of endovascular treatment (EVT) for ischemic stroke in daily clinical practice, physicians are confronted with questions from patients and family members about the extent of recovery that they can expect most often quite early after EVT. Although at group level almost one-half of all patients treated with EVT recover to functional independence, outcomes of individual patients remain highly variable and depend on multiple factors.1 For treating physicians, therefore, it is difficult to accurately predict individual outcomes after stroke. Results from previous research suggest that well-validated prognostic models are more accurate in predicting outcome than physicians.2,3
Most prediction models for patients undergoing EVT are based on preprocedural data only and primarily serve to identify patients who may benefit from EVT.4 A clinical prognostic model that can be used after EVT and take into consideration both preprocedural and postprocedural characteristics (eg, age, reperfusion grade, and neurologic status 1 day after EVT) could provide physicians, patients, and family members with more reliable outcome expectations.5,6,7,8 However, 3 externally validated models that were designed for early prognostication after EVT and included postprocedural characteristics were developed before the landmark trials were published.7,9,10 The purpose of the present study was to develop and externally validate a contemporary prognostic model that can be applied 1 day after EVT to predict functional outcome.
Methods
Derivation Cohort
The model was developed with individual patient data from 7 randomized clinical trials (RCTs) on EVT within the Highly Effective Reperfusion Evaluated in Multiple Endovascular Stroke Trials (HERMES) collaboration: Dutch Multicenter Randomized Clinical Trial of Endovascular Treatment for Acute Ischemic Stroke in the Netherlands (MR CLEAN), Endovascular Treatment for Small Core and Anterior Circulation Proximal Occlusion With Emphasis on Minimizing CT to Recanalization Times (ESCAPE), Extending the Time for Thrombolysis in Emergency Neurological Deficits—Intra-Arterial (EXTEND-IA), Solitaire With the Intention for Thrombectomy as Primary Endovascular Treatment (SWIFT PRIME), Randomized Trial of Revascularization With Solitaire FR Device vs Best Medical Therapy in the Treatment of Acute Stroke Due to Anterior Circulation Large Vessel Occlusion Presenting Within 8 Hours of Symptom Onset (REVASCAT), Thrombectomie des Artères Cerébrales (THRACE), and Pragmatic Ischaemic Stroke Thrombectomy Evaluation (PISTE).1,11,12 These RCTs compared EVT—primarily performed with stent retrievers—with standard care in adult patients with ischemic stroke caused by a large vessel occlusion in the anterior circulation and confirmed on computed tomography angiography (CTA) or magnetic resonance angiography (MRA). Inclusion criteria varied between the RCTs.1 All participants provided written informed consent according to each trial protocol, and each RCT was approved by the local ethics committee and included pooling of the data for follow-up studies including this study.
For this study, we included all patients randomly assigned to EVT who underwent arterial puncture within 12 hours of symptom onset for an occlusion of the intracranial carotid artery (ICA), internal carotid artery terminus (ICA-T), or middle cerebral artery (segment M1 or M2). Of the studies used for this analysis, only one (SWIFT PRIME) collected data on race and ethnicity. In this study, 90% of the participants identified as White race. Therefore, we could not analyze race and ethnicity further. This study followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guidelines.
Outcome Measures
The primary outcome was functional outcome at 90 days, assessed with the modified Rankin Scale (mRS). The mRS is an ordinal scale used to measure the degree of disability in daily activities and ranges from 0 (no symptoms) to 6 (death).13 The mRS was modeled as a full ordinal scale, and we subsequently extracted the probabilities for functional independence (defined as mRS 0-2) and survival (defined as mRS 0-5) from the predicted distribution.
Statistical Analysis
Model Development
To identify predictors of functional outcome after EVT, we prespecified 19 preprocedural and postprocedural variables that can be assessed within 1 day after EVT, based on recent literature, expert opinion, clinical relevance, and availability in both the derivation and validation cohort. These variables included the following: age, sex, prestroke disability assessed with the mRS, diabetes (yes/no), hypertension (yes/no), previous stroke (yes/no), baseline stroke severity assessed with the National Institutes of Health Stroke Scale (NIHSS), serum glucose level, systolic blood pressure, intravenous treatment with alteplase (yes/no), baseline location of intracranial large vessel occlusion, baseline collateral score on single-phase CTA (grade 0 indicates no collateral filling; grade 1 indicates collateral filling ≤50% but >0%; grade 2 indicates collateral filling >50% but <100%; grade 3 indicates 100% collateral filling; all scored in the affected middle cerebral artery territory in comparison with the entire contralateral middle cerebral artery territory),14,15 baseline Alberta Stroke Program Early CT Score (ASPECTS), time from symptom onset to arterial puncture, duration of the procedure (arterial puncture to last contrast bolus injection), general anesthesia (yes/no), radiologic reperfusion grade (modified Treatment in Cerebral Infarction [mTICI] score) after EVT on digital subtraction angiography, NIHSS score 24 hours after EVT, and symptomatic intracranial hemorrhage (sICH) 24 hours after EVT (yes/no, as defined in each RCT). The trial was included as a fixed effect to account for possible study-level differences in prognosis in the EVT arms and statistical dependence of outcomes within trials.
We used ordinal logistic regression modeling, which assumes proportional odds, to determine the association of the potential predictors with functional outcome. We tested nonlinearity of the relation between continuous variables (age, systolic blood pressure, glucose level, NIHSS score at baseline, ASPECTS scale, time to arterial puncture, duration of the procedure, NIHSS score at 24 hours) and the log odds of mRS score with restricted cubic spline functions with 3 knots.
We used univariable and multivariable regression analysis with Akaike information criterion (AIC) to balance model fit vs model complexity in a stepwise procedure. Effectively, this implies selection with P < .157 for potential predictors with 1 degree of freedom. The final model was labeled MR PREDICTS@24H for Multivariable Outcome Prediction After Endovascular Treatment for Ischemic Stroke at 24 Hours. Predictor effects were expressed as adjusted (common) odds ratios (ORs) with 95% CIs. To obtain more insight into the relative importance of NIHSS score at 24 hours, we made a model with the NIHSS alone. We also performed additional analyses with final infarct volume (FIV) assessed between 12 hours and 2 weeks, which was not included in the final model because it is not routinely used in clinical practice and not available in the validation cohort. The comparison of the models of NIHSS score at 24 hours alone and FIV alone and the contribution of addition of FIV to the final model on the model fit were tested using the log-likelihood ratio.
We assessed the internal validity of the final model with bootstrap resampling to calculate the degree of optimism in model performance. To correct this optimism, we shrunk the regression coefficients in the final model using penalized regression (penalized function in the rms package for R statistical software [R Project for Statistical Computing]).
Validation Cohort
For external validation, we used data from the MR CLEAN Registry, a nationwide prospective, observational study in 18 centers that provide EVT in the Netherlands.16 Data were collected from consecutive patients with ischemic stroke who had undergone EVT since March 2014, after the last patient was included in the MR CLEAN trial. The medical ethics committee of the Erasmus MC University Medical Center, Rotterdam, the Netherlands, granted permission to carry out the study as a registry, for which no consent was necessary.
We included patients with ischemic stroke who were enrolled in the MR CLEAN Registry and treated with EVT between March 16, 2014, and November 1, 2017, in centers that participated in the MR CLEAN trial. Included patients were 18 years or older, had an occlusion of the ICA(-T) or middle cerebral artery (M1 or M2) on CTA or MRA, and had undergone arterial puncture within 12 hours after symptom onset.
Model Validation and Updating
External validation was performed with the regression coefficients of the final model for outcome prediction after EVT and the model intercept as estimated for the MR CLEAN trial. Model performance was assessed in terms of discrimination and calibration. Discrimination refers to the ability of the model to distinguish between patients with good and poor outcomes. The discriminative ability of the model at internal and external validation was quantified with the concordance statistic (C statistic).17 We calculated the Harrell C statistic for the ordinal mRS and for the predictions of functional independence (mRS 0-2 vs mRS 3-6) and survival (mRS 0-5 vs mRS 6). Calibration refers to the level of agreement between observed outcomes to predicted probabilities. We assessed calibration of predictions graphically by plotting the observed proportion of functional independence or survival in the validation cohort against the predicted probability of functional independence or survival based on the derivation cohort. Calibration was quantified with the calibration slope and calibration intercept of these plots. The calibration slope should ideally be equal to 1, meaning that the effects of predictors are equal in the derivation cohort as compared with those in the validation cohort. The intercept should ideally be 0 and indicates whether the predictions based on the model are systematically too high or too low in the validation cohort.17 The 95% CIs of the C statistic, calibration slope, and calibration intercept were calculated with bootstrap resampling with 2000 replications.
After validation, we refitted the model coefficients of the predictors on the MR CLEAN Registry data. Additionally, we replaced the variable mTICI for extended TICI,18,19 and added a missing category for collateral score because this scoring system might not (yet) be common practice everywhere.
We implemented the updated final model in a web application that provides predictions of functional outcome at 90 days for individual patients with ischemic stroke based on routinely available information 1 day after EVT. It displays the predicted probabilities of functional independence and survival based on the intercept of the MR CLEAN Registry.
Following HERMES policy, patients with missing outcomes were excluded. Missing predictor data were imputed by multiple regression imputation based on relevant study, covariates, intervention, and outcome. According to MR CLEAN Registry policy, missing data, including mRS scores, were imputed by multiple imputation using additive regression based on relevant baseline covariates and outcomes. Statistical analyses were performed with R statistical software, version 3.5.1 (R Project for Statistical Computing).
Results
Derivation Cohort
The HERMES cohort consisted of 781 patients (median [IQR] age, 67 [57-76] years; 414 men [53%]; 367 women [47%]) (eFigure 1 in Supplement 1). Their median (IQR) baseline NIHSS score was 17 (14-21) (Table 1). Substantial reperfusion (ie, mTICI ≥2B) was achieved in 544 of 715 patients (76%). The median (IQR) NIHSS score at 24 hours was 9 (4-16), the median (IQR) mRS score at 90 days was 3 (1-4), 371 patients (48%) achieved functional independence, and 671 patients (86%) survived up to 90 days.
Table 1. Overview of Derivation Cohort and Validation Cohorta.
| Characteristic | Derivation cohort HERMES (n = 781) | Validation cohort MR CLEAN registry (n = 3260) |
|---|---|---|
| Age, median (IQR), y | 67 (57-76) | 72 (61-80) |
| Sex, No./total No. (%) | ||
| Men | 414/781 (53) | 1684/3260 (52) |
| Women | 367/781 (47) | 1576/3260 (48) |
| NIHSS score, median (IQR) | 17 (14-21) | 16 (11-19) |
| Systolic blood pressure, median (IQR), mm Hg | 144 (130-159) | 150 (131-165) |
| Serum glucose level, median (IQR), mmol/L | 6.7 (5.9-7.8) | 6.8 (5.9-8.1) |
| Previous stroke, No./total No. (%) | 89/777 (11) | 544/3233 (17) |
| Hypertension, No./total No. (%) | 426/779 (55) | 1676/3194 (52) |
| Atrial fibrillation, No./total No. (%) | 217/640 (34) | 770/3217 (24) |
| Diabetes, No./total No. (%) | 120/780 (15) | 524/3236 (16) |
| Prestroke mRS score, No./total No. (%) | ||
| 0 | 501/605 (83) | 2160/3188 (68) |
| 1 | 76/605 (13) | 421/31888 (13) |
| 2 | 19/605 (3.1) | 239/3188 (7.5) |
| ≥3 | 9/605 (1.5) | 368/3188 (12) |
| Occlusion location, No./total No. (%) | ||
| ICA(-T) | 198/733 (27) | 818/3121 (26) |
| M1 | 473/733 (65) | 1804/3121 (58) |
| M2 or otherb | 62/733 (8.5) | 499/3121 (16) |
| ASPECTS scale, median (IQR) | 8 (7-9) | 9 (7-10) |
| Collateral score, No./total No. (%) | ||
| 0 | 5/602 (0.8) | 187/3053 (6.1) |
| 1 | 81/602 (14) | 1094/3053 (36) |
| 2 | 268/602 (45) | 1181/3053 (39) |
| 3 | 248/602 (41) | 591/3053 (19) |
| Treatment with IV alteplase, No./total No. (%) | 678/781 (87) | 2445/3248 (75) |
| Time from stroke onset to arterial puncture, median (IQR), min | 240 (185-299) | 195 (150-255) |
| General anesthesia, No./total No. (%) | 227/776 (29) | 775/3063 (25) |
| Duration of the procedure, median (IQR), min | 64 (40-91) | 59 (38-83) |
| Outcome measures | ||
| Reperfusion grade (mTICI), No./total No. (%) | ||
| 0 | 54/715 (7.6) | 531/3173 (17) |
| 1 | 19/715 (2.7) | 94/3173 (3.0) |
| 2A | 98/715 (14) | 592/3173 (19) |
| 2B | 483/715 (68) | 715/3173 (23)c |
| 2C | NA | 339/3173 (11)c |
| 3 | 61/715 (8.5) | 902/3173 (28) |
| NIHSS score at 24 h, median (IQR) | 9 (4-16) | 10 (4-17) |
| sICH at 24 h, No./total No. (%) | 28/770 (3.6) | 192/3245 (5.9) |
| FIV 12 h to 2 wk, median (IQR) mL | 34 (11-103) | NA |
| mRS score at 3 mo, median (IQR) | 3 (1-4) | 3 (2-6) |
| mRS 0-2 at 3 mo, No./total No. (%) | 371/781 (48) | 1235/3047 (41) |
| Survival at 3 mo, No./total No. (%) | 671/781 (86) | 2164/3047 (71) |
Abbreviations: ASPECTS, Alberta Stroke Program Early CT Score; FIV, follow-up infarct volume; HERMES, Highly Effective Reperfusion Evaluated in Multiple Endovascular Stroke Trials; ICA(-T), intracranial carotid artery (terminus); IV, intravenous; M1, middle cerebral artery segment 1; M2, middle cerebral artery segment 2; MR CLEAN, Multicenter Randomized Clinical Trial of Endovascular Treatment for Acute Ischemic Stroke in the Netherlands; mRS, modified Rankin Scale; mTICI, modified treatment in cerebral infarction; NA, not applicable; NIHSS, National Institutes of Health Stroke Scale; sICH, symptomatic intracranial hemorrhage.
SI conversion factor: To convert glucose level to milligrams per deciliter, divide by 0.0555.
Categorical values are presented as No. (%) and continuous values as median (IQR). Missing continuous values in the derivation cohort and validation cohort, respectively, are 0 and 15 for age; 4 and 55 for NIHSS baseline score; 1 and 89 for systolic blood pressure; 26 and 371 for serum glucose; 8 and 109 for ASPECTS scale; 0 and 15 for time from stroke onset to arterial puncture; 174 and 291 for duration of the procedure; and 25 and 333 for NIHSS score at 24 hours.
Other occlusion location (M3 or anterior cerebral artery segment 1 or 2) by core laboratory: in HERMES (n = 1), in MR CLEAN Registry (n = 25).
Extended TICI score MR CLEAN Registry: 2B (n = 715), 2C (n = 339).
Model Development
All prespecified variables, with the exception of treatment with intravenous alteplase and previous stroke, were predictors of outcome in univariable analysis (Table 2). Based on the multivariable analysis, 9 preprocedural or postprocedural variables were included in the final model: age, baseline NIHSS score, prestroke mRS score, presence or absence of diabetes, occlusion location, collateral score, mTICI after EVT, NIHSS score at 24 hours, and sICH (Table 2). The final model had an ordinal C statistic of 0.85 for the ordinal mRS score. The discriminative ability of NIHSS at 24 hours alone was lower than that of the final model (0.83; P < .001), as was FIV alone (0.76; P < .001). By adding FIV to the final model, the discriminative ability increased from 0.85 to 0.86 (P < .001). The internally validated C statistic of the final model, corrected for optimism, was 0.84 (95% CI, 0.84-0.85) for the ordinal mRS, 0.92 (95% CI, 0.91-0.92) for functional independence, and 0.87 (95% CI, 0.85-0.88) for survival (Table 3).
Table 2. Main Effects in Derivation Cohort (HERMES, n = 781) Presented as Common Odds Ratios (ORs) With 95% CIa.
| Characteristic | Univariable models, OR (95% CI) | Multivariable model, OR (95% CI) |
|---|---|---|
| Age, per y | ||
| <65 y | 0.99 (0.97-1.01) | 1.00 (0.98-1.02) |
| ≥65 y | 0.94 (0.92-0.96) | 0.94 (0.91-0.96) |
| Sex | ||
| Men | 1.12 (0.87-1.44) | NA |
| Women | 1 [Reference] | NA |
| Baseline NIHSS score, per point | 0.91 (0.88-0.93) | 1.03 (1.00-1.06) |
| Systolic blood pressure, per 10 mm Hg | 0.87 (0.82-0.91) | 0.97 (0.91-1.03) |
| Glucose, per 30 mmol/L | ||
| <120 mmol/L | 0.55 (0.39-0.79) | 0.95 (0.68-1.34) |
| ≥120 mmol/L | 0.89 (0.81-0.98) | 0.97 (0.89-1.07) |
| Treatment with IV alteplase | 1.07 (0.72-1.60) | NA |
| Previous stroke | 0.84 (0.57-1.25) | NA |
| Hypertension | 0.76 (0.59-0.98) | 0.95 (0.71-1.38) |
| Atrial fibrillation | 0.75 (0.56-0.99) | 1.04 (0.76-1.43) |
| Diabetes | 0.47 (0.33-0.67) | 0.50 (0.33-0.75) |
| Prestroke mRS score, per point | 0.52 (0.40-0.68) | 0.61 (0.46-0.82) |
| Occlusion location | ||
| ICA(-T) | 1.0 [Reference] | 1.0 [Reference] |
| M1 | 1.58 (1.19-2.11) | 1.26 (0.91-1.74) |
| M2 or other | 2.37 (1.42-3.94) | 2.04 (1.16-3.60) |
| Collateral score, per point | 1.78 (1.46-2.17) | 1.24 (0.93-1.65) |
| ASPECTS | 1.35 (1.18-1.53) | 1.00 (0.92-1.10) |
| Time from stroke onset to arterial puncture, per 30 min | 0.95 (0.91-0.99) | 0.97 (0.93-1.01) |
| General anesthesia | 0.71 (0.53-0.95) | 0.98 (0.70-1.37) |
| Postprocedural reperfusion grade (mTICI), per point | 1.73 (1.48-2.01) | 1.20 (1.02-1.41) |
| NIHSS score at 24 h, per point | ||
| <12 points | 0.72 (0.68-0.75) | 0.71 (0.68-0.75) |
| ≥12 points | 0.85 (0.82-0.89) | 0.86 (0.83-0.90) |
| sICH at 24 h | 0.11 (0.05-0.24) | 0.29 (0.11-0.79) |
Abbreviations: ASPECTS, Alberta Stroke Program Early CT Score; ICA(-T), intracranial carotid artery(-terminus); IV, intravenous; M1, middle cerebral artery segment 1; M2, middle cerebral artery segment 2; mRS, modified Rankin Scale; mTICI, modified treatment in cerebral infarction; NA, not applicable; NIHSS, National Institutes of Health Stroke Scale; sICH, symptomatic intracranial hemorrhage.
SI conversion factor: To convert glucose level to milligrams per deciliter, divide by 0.0555.
Common ORs reflect the effect on the reversed mRS (an OR >1 corresponds with better functional outcome). Variables with a P < Akaike information criterion (P < .157 for potential predictors with 1 df) in univariable analysis, were entered into the multivariable model.
Table 3. Performance Measures With 95% CI in Derivation Cohort (N = 781) and Validation Cohort (n = 3260)a.
| Measure | Ordinal mRS | Functional independence (mRS 0-2) | Survival (mRS 0-5) |
|---|---|---|---|
| Internal validation | |||
| C statistic | 0.84 (0.84-0.85) | 0.92 (0.91-0.92) | 0.87 (0.85-0.88) |
| External validation | |||
| C statistic | 0.84 (0.83 to 0.84) | 0.91 (0.90 to 0.92) | 0.89 (0.88 to 0.90) |
| Calibration intercept | NA | 0.61 (0.50 to 0.74) | −0.25 (−0.37 to −0.13) |
| Calibration slope | NA | 0.98 (0.92 to 1.05) | 0.86 (0.80 to 0.94) |
Abbreviations: mRS, modified Rankin Scale; NA, not applicable.
The C statistic is a measure for the ability to distinguish between patients with a low and high probability of good outcome. It can vary between 0.5 for a noninformative model and 1 for a perfectly discriminating model. The calibration intercept reflects the calibration-in-the-large, indicating whether predicted probabilities are systematically too low or too high, and should ideally be equal to 0. The calibration slope reflects the strength of the predictors and should ideally be equal to 1.
Patient Population in Validation Cohort
The MR CLEAN Registry consisted of 3260 patients (median [IQR] age, 72 [61-80] years; 1684 men [52%]; 1576 women [48%]) (eFigure 2 in the Supplement). The median (IQR) baseline NIHSS score was 16 (11-19). Patients in the validation cohort less often had atrial fibrillation (770 of 3217 [24%] vs 217 of 640 [34%]) but more often prestroke disability (mRS of 2 or higher: 607 of 3188 [20%] vs 28 of 605 [4.6%]) and worse collateral scores (collateral score of 0-1: 1291 of 3053 [42%] vs 86 of 602 [15%]) than in the derivation cohort. Fewer patients in the validation cohort had M1 occlusions (1804 of 3121 [58%] vs 473 of 733 [65%]), and time from stroke onset to arterial puncture was shorter (median [IQR] 195 [150-255] minutes vs 240 [185-299] minutes). A total of 2245 of 3248 patients (75%) were treated with intravenous alteplase, compared with 678 of 781 patients (87%) in the derivation cohort. Other baseline patient, baseline imaging, and treatment characteristics were similar between the 2 cohorts (Table 1). Substantial reperfusion (ie, mTICI ≥2B) was achieved in 1956 of 3173 patients (62%). In the validation cohort, the median (IQR) NIHSS score at 24 hours was 10 (4-17), the median (IQR) mRS score was 3 (2-6), 1235 of 3047 patients (41%) achieved functional independence, and 2164 of 3047 patients (71%) survived up to 90 days.
External Validation
In the validation cohort, we found predictor effects similar to those in the derivation cohort (eTable 1 in Supplement 1). The externally validated C statistics were 0.84 (95% CI, 0.83-0.84) for the ordinal mRS, 0.91 (95% CI, 0.90-0.92) for functional independence, and 0.89 (95% CI, 0.88-0.90) for survival (Table 3). Calibration plots showed that the observed proportion of patients achieving functional independence was systematically higher than predicted based on the model (intercept 0.61, slope 0.98). The mean observed probability of functional independence was 41% (1331 of 3260), whereas the model predicted a mean of 34% (1099 of 3260) (Figure 1A). Observed survival was somewhat lower than predicted (intercept −0.25, slope 0.87). The observed probability of survival was 72%, whereas the model predicted 75% (Figure 1B). The regression equation of the final model is available in the eAppendix in Supplement 1. The online tool is accessible for clinical use at www.mrpredicts.com20 (Figure 2).
Figure 1. Calibration Plots.
A, Functional independence (modified Rankin Scale [mRS] score, 0-2) in the validation cohort. The overall observed proportion of patients with mRS score 0 to 2 in the validation cohort was higher than the predicted proportion using our model (41% vs 32%). The linear bar chart shows the distribution of patients with (= 1) or without (= 0) the observed outcome. B, Survival (mRS score, 0-5) in the validation cohort (n = 3260). The overall observed proportion of patients with mRS score 0 to 5 in the validation cohort was similar to the predicted proportion using our model (72% vs 75%). The linear bar chart shows the distribution of patients with (= 1) or without (= 0) the observed outcome. The shading around the flexible calibration line indicates the 95% confidence limits. MR CLEAN indicates Multicenter Randomized Clinical Trial of Endovascular Treatment for Acute Ischemic Stroke in the Netherlands.
Figure 2. Screenshot of the Online Tool.
CTA indicates computed tomography angiography; eTICI, extended thrombolysis in cerebral infarction; ICH, intracranial hemorrhage; M1, middle cerebral artery segment 1; MCA, middle cerebral artery; NIHSS, National Institutes of Health Stroke Scale.
Discussion
The prediction model for functional outcome of patients after EVT can be used within 1 day after EVT for ischemic stroke to accurately predict functional outcome at 90 days after EVT. The model consists of 9 routinely available preprocedural and postprocedural clinical and radiologic characteristics and showed excellent discriminative ability and good calibration. Through validation and updating, the model was optimized for use in contemporary clinical practice.
Several other prognostic models for patients treated with EVT also combined preprocedural and postprocedural characteristics.5,6,7,8,9,10,21,22,23,24,25,26,27 However, some models include rather homogenous patient populations limiting the generalizability of their findings; other models have methodologic shortcomings in model development, such as a small sample size for the amount of tested variables, dichotomization of variables, or no internal validation. Importantly, most models lack external validation. Of the 4 models that have been externally validated, the Symptomatic Hemorrhage, Baseline NIHSS Score, Age, Reperfusion, and Location of Clot (SNARL) score included symptomatic intracranial hemorrhage, NIHSS score at baseline, age, reperfusion grade, and location of the occlusion and concluded that adding postprocedural characteristics would improve outcome prediction.7 The Pittsburgh Outcomes After Stroke Thrombectomy (POST) score was based on age, infarct volume, and hemorrhagic complications,10 whereas the BRANCH scale included baseline blood glucose level, reperfusion grade, age, baseline NIHSS score, change in blood glucose level after 48 hours, and symptomatic intracranial hemorrhage.9 However, these 3 models were derived from cohorts of patients treated before the landmark trials were published, and EVT became the standard of care using second-generation devices. They also did not include NIHSS score or another measure of neurologic status after EVT. The Delayed Functional Independence after Neurothrombectomy (DEFIANT) score was developed and externally validated after these landmark trials and did include (at discharge) NIHSS score in their model, as well as age and any hemorrhage (at discharge).27,28 Although the prognostic tool for outcome prediction after EVT may be used to predict outcome 24 hours after EVT, the DEFIANT score was developed to better understand the phenomenon of delayed functional independence and is meant to be used at discharge for patients who are not functionally independent. Given the sample size of the validation cohort (n = 79), the DEFIANT score needs to be further validated to assess its performance in different populations. We found NIHSS score at 24 hours to be the strongest independent outcome predictor of all 9 included variables. This substantiates previous findings that the postprocedural NIHSS score is a major predictor of long-term functional outcome.6,7,25,29,30,31 Although the ASPECTS scale at baseline was not included in the current model, it might also become an independent predictor of outcome in populations including patients with lower ASPECTS scale resulting from the recent positive results for EVT in patients with large ischemic scores at baseline.32,33,34 Interestingly, time from stroke onset to arterial puncture was not included in the prognostic tool for outcome prediction after EVT. We emphasize that this does not mean that time is not important for the prediction of outcome after EVT but suggest that the well-established effect of time on outcome35 is captured in other (postprocedural) characteristics, such as the NIHSS score at 24 hours. Two models, the POST score and Gender, Age, Diabetes Mellitus History, Infarct Volume, Current Smoker (GADIS) score,10,24 included follow-up infarct volume, as this is known to be a strong independent predictor of outcome after EVT.36 In our study, FIV improved the performance of the model. FIV was assessed on follow-up between 12 hours and 2 weeks; therefore, the observed improvement is probably an overestimation of the improvement with assessment 1 day after EVT. In addition, in many countries, including the Netherlands, patients do not routinely undergo follow-up CT or MRI. As the prognostic tool for outcome prediction after EVT was specifically designed for use in clinical practice within 1 day after EVT, FIV was not considered for inclusion in the model. Moreover, as there is a desire to inform patients and their family members quickly after treatment, other postprocedural characteristics beyond 1 day that may also influence functional outcome, such as the occurrence of pneumonia or (the intensity) of rehabilitation, were not analyzed. In addition, several important prognostic factors, such as socioeconomic status, cultural expectations, and treatment restrictions, were unavailable.
This model may be of use to neurologists, stroke physicians, and rehabilitation specialists, resulting in more homogeneous outcome prediction across different physicians. By providing more objective data of expected outcomes, the model can be used to guide physicians in adapting and personalizing their patients’ follow-up and rehabilitation plans, including discharge destination. The accuracy of outcome prediction by physicians alone has shown to be insufficient for these types of decisions.37 However, treatment decisions such as treatment restrictions or the intensity of rehabilitation probably contributed to the predicted outcome at 90 days. This limitation applies to the large majority of prediction models after stroke. Nonetheless, a more accurate estimation of a patients’ prognosis could also be used in certain other situations, such as assisting families in planning long-term housing arrangements. External factors, such as housing circumstances and social support, become more important when a poor outcome is likely. Furthermore, prediction of outcome provides us with probabilities; even with a 99% predicted probability of survival, the patient might still die. In addition, outcome predictions are most uncertain and potentially inaccurate in patients who are underrepresented in the development population, such as those who are very old, have a high prestroke mRS score, or in uncommon combinations, such as a low NIHSS score at 24 hours despite sICH. As a prognostic model cannot replace clinical judgment, particularly in these patients, the prognostic tool for outcome prediction after EVT should be used as a complementary tool to aid the treating physician.
Strengths and Limitations
A major strength of this study is that it was developed on a large heterogeneous data set generated from patients with ischemic stroke treated with EVT in many different countries throughout Europe, North America, East Asia, and Oceania. It was externally validated in a large Dutch registry, which includes patients treated in clinical practice. This registry is more heterogeneous than RCTs in terms of patient characteristics, representing current clinical practice. This makes the model widely applicable and improves its applicability in daily clinical practice. The increasing experience with EVT over time, resulting in better outcomes, is likely the cause of the suboptimal calibration, which is why the model was updated. Unfortunately for several reasons, prognostic models have been adopted to a limited extent only by the stroke community.38 Models are perceived as being too complicated, too generic, not intuitive enough, or may require information that is not routinely available. Therefore, we aimed to develop and directly externally validate a simple online clinical tool, based on routinely available preprocedural and postprocedural characteristics. Finally, patients who had undergone EVT after 12 hours of symptom onset, those with low ASPECTS scale, or those with posterior circulation stroke were not included in HERMES nor in MR CLEAN Registry. It stands to reason, however, that especially for patients who underwent EVT after 12 hours of symptom onset, predictions of the current model will still be accurate. Yet, to make the prognostic tool for outcome prediction after EVT applicable to a wider range of patients, it needs to be externally validated in data sets beyond the current development and validation data, and then be updated to keep predictions up to date for contemporary patients.
Conclusions
The prognostic tool for outcome prediction after EVT includes 9 preprocedural and postprocedural clinical and radiologic characteristics and can be applied 1 day after EVT to accurately predict functional outcome at 90 days. It provides patients, family members, and physicians with reliable outcome expectations and may assist physicians in personalizing their patients’ follow-up and rehabilitation plans. The model is available online for clinical use and will need further validation and updating to optimally support stroke care.
eTable. Main Effects of Final Model in Derivation Cohort and Validation Cohort Presented as Common Odds Ratios With 95% CIs
eAppendix. R Code (Full Regression Equation)
eFigure 1. Flow of Patients in Derivation Cohort (HERMES)
eFigure 2. Flow of Patients in Validation Cohort (MR CLEAN Registry)
Nonauthor Collaborators. HERMES Collaborators and MR CLEAN Registry Investigators
Data Sharing Statement
References
- 1.Goyal M, Menon BK, van Zwam WH, et al. ; HERMES collaborators . Endovascular thrombectomy after large-vessel ischaemic stroke: a meta-analysis of individual patient data from 5 randomized trials. Lancet. 2016;387(10029):1723-1731. doi: 10.1016/S0140-6736(16)00163-X [DOI] [PubMed] [Google Scholar]
- 2.Ntaios G, Gioulekas F, Papavasileiou V, Strbian D, Michel P. ASTRAL, DRAGON, and SEDAN scores predict stroke outcome more accurately than physicians. Eur J Neurol. 2016;23(11):1651-1657. doi: 10.1111/ene.13100 [DOI] [PubMed] [Google Scholar]
- 3.Saposnik G, Cote R, Mamdani M, et al. JURaSSiC: accuracy of clinician vs risk score prediction of ischemic stroke outcomes. Neurology. 2013;81(5):448-455. doi: 10.1212/WNL.0b013e31829d874e [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Kremers F, Venema E, Duvekot M, et al. ; MR CLEAN Registry Investigators . Outcome prediction models for endovascular treatment of ischemic stroke: systematic review and external validation. Stroke. 2022;53(3):825-836. doi: 10.1161/STROKEAHA.120.033445 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Dekker L, Geraedts VJ, Hund H, et al. Importance of reperfusion status after intra-arterial thrombectomy for prediction of outcome in anterior circulation large vessel stroke. Interv Neurol. 2018;7(3-4):137-147. doi: 10.1159/000486246 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Jeong HG, Kim BJ, Choi JC, et al. Posttreatment National Institutes of Health Stroke Scale is superior to the initial score or thrombolysis in cerebral ischemia for 3-month outcome. Stroke. 2018;49(4):938-944. doi: 10.1161/STROKEAHA.117.020587 [DOI] [PubMed] [Google Scholar]
- 7.Prabhakaran S, Jovin TG, Tayal AH, et al. Posttreatment variables improve outcome prediction after intra-arterial therapy for acute ischemic stroke. Cerebrovasc Dis. 2014;37(5):356-363. doi: 10.1159/000362591 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Weyland CS, Vey JA, Mokli Y, et al. Full reperfusion without functional independence after mechanical thrombectomy in the anterior circulation: performance of prediction models before vs after treatment initiation. Clin Neuroradiol. 2022;32(4):987-995. doi: 10.1007/s00062-022-01166-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Natarajan SK, Dandona P, Karmon Y, et al. Prediction of adverse outcomes by blood glucose level after endovascular therapy for acute ischemic stroke. J Neurosurg. 2011;114(6):1785-1799. doi: 10.3171/2011.1.JNS10884 [DOI] [PubMed] [Google Scholar]
- 10.Rangaraju S, Liggins JT, Aghaebrahim A, et al. Pittsburgh outcomes after stroke thrombectomy score predicts outcomes after endovascular therapy for anterior circulation large vessel occlusions. Stroke. 2014;45(8):2298-2304. doi: 10.1161/STROKEAHA.114.005595 [DOI] [PubMed] [Google Scholar]
- 11.Bracard S, Ducrocq X, Mas JL, et al. ; THRACE investigators . Mechanical Thrombectomy After Intravenous Alteplase vs Alteplase Alone After Stroke (THRACE): a randomized controlled trial. Lancet Neurol. 2016;15(11):1138-1147. doi: 10.1016/S1474-4422(16)30177-6 [DOI] [PubMed] [Google Scholar]
- 12.Muir KW, Ford GA, Messow CM, et al. ; PISTE Investigators . Endovascular therapy for acute ischaemic stroke: the Pragmatic Ischaemic Stroke Thrombectomy Evaluation (PISTE) randomized, controlled trial. J Neurol Neurosurg Psychiatry. 2017;88(1):38-44. doi: 10.1136/jnnp-2016-314117 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.van Swieten JC, Koudstaal PJ, Visser MC, Schouten HJ, van Gijn J. Interobserver agreement for the assessment of handicap in stroke patients. Stroke. 1988;19(5):604-607. doi: 10.1161/01.STR.19.5.604 [DOI] [PubMed] [Google Scholar]
- 14.Tan JC, Dillon WP, Liu S, Adler F, Smith WS, Wintermark M. Systematic comparison of perfusion-CT and CT-angiography in acute stroke patients. Ann Neurol. 2007;61(6):533-543. doi: 10.1002/ana.21130 [DOI] [PubMed] [Google Scholar]
- 15.Tan IY, Demchuk AM, Hopyan J, et al. CT angiography clot burden score and collateral score: correlation with clinical and radiologic outcomes in acute middle cerebral artery infarct. AJNR Am J Neuroradiol. 2009;30(3):525-531. doi: 10.3174/ajnr.A1408 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Jansen IGH, Mulder MJHL, Goldhoorn RB; MR CLEAN Registry Investigators . Endovascular treatment for acute ischaemic stroke in routine clinical practice: prospective, observational cohort study (MR CLEAN Registry). BMJ. 2018;360:k949. doi: 10.1136/bmj.k949 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Steyerberg EW, Vergouwe Y. Toward better clinical prediction models: 7 steps for development and an ABCD for validation. Eur Heart J. 2014;35(29):1925-1931. doi: 10.1093/eurheartj/ehu207 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Almekhlafi MA, Mishra S, Desai JA, et al. Not all “successful” angiographic reperfusion patients are an equal validation of a modified TICI scoring system. Interv Neuroradiol. 2014;20(1):21-27. doi: 10.15274/INR-2014-10004 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Goyal M, Fargen KM, Turk AS, et al. 2C or not 2C: defining an improved revascularization grading scale and the need for standardization of angiography outcomes in stroke trials. J Neurointerv Surg. 2014;6(2):83-86. doi: 10.1136/neurintsurg-2013-010665 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.MDM Test. MR PREDICTS@24H. Accessed July 31, 2023. www.mrpredicts.com
- 21.Cappellari M, Mangiafico S, Saia V, et al. IER-START nomogram for prediction of three-month unfavorable outcome after thrombectomy for stroke. Int J Stroke. 2020;15(4):412-420. doi: 10.1177/1747493019837756 [DOI] [PubMed] [Google Scholar]
- 22.Haranhalli N, Javed K, Boyke A, et al. A predictive model for functional outcome in patients with acute ischemic stroke undergoing endovascular thrombectomy. J Stroke Cerebrovasc Dis. 2021;30(11):106054. doi: 10.1016/j.jstrokecerebrovasdis.2021.106054 [DOI] [PubMed] [Google Scholar]
- 23.Javed K, Qin J, Mowery W, Kadaba D, Altschul D, Haranhalli N. Predicting 90-day functional dependency and death after endovascular thrombectomy for stroke: the BET score. J Stroke Cerebrovasc Dis. 2022;31(5):106342. doi: 10.1016/j.jstrokecerebrovasdis.2022.106342 [DOI] [PubMed] [Google Scholar]
- 24.O’Connor KP, Hathidara MY, Danala G, et al. Predicting clinical outcome after mechanical thrombectomy: the GADIS (gender, age, diabetes mellitus history, infarct volume, and current smoker [corrected]) score. World Neurosurg. 2020;134:e1130-e1142. doi: 10.1016/j.wneu.2019.11.127 [DOI] [PubMed] [Google Scholar]
- 25.van Os HJA, Ramos LA, Hilbert A, et al. ; MR CLEAN Registry Investigators . Predicting outcome of endovascular treatment for acute ischemic stroke: potential value of machine learning algorithms. Front Neurol. 2018;9:784. doi: 10.3389/fneur.2018.00784 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Zhang XG, Wang JH, Yang WH, et al. Nomogram to predict 3-month unfavorable outcome after thrombectomy for stroke. BMC Neurol. 2022;22(1):111. doi: 10.1186/s12883-022-02633-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Jadhav AP, Desai SM, Gupta R, et al. Delayed Functional Independence After Neurothrombectomy (DEFIANT) score: analysis of the Trevo Retriever Registry. J Neurointerv Surg. 2022;neurintsurg-2022-019232. doi: 10.1136/jnis-2022-019232 [DOI] [PubMed] [Google Scholar]
- 28.Desai S, Jha R, Catapano J, et al. O-021 Validation of the Delayed Functional Improvement After neurothrombectomy (DEFIANT) score in the TIGER study. Abstract presented at: Society of NeuroInterventional Surgery 19th Annual Meeting; July 25-29, 2022; Toronto, Canada. [Google Scholar]
- 29.Chalos V, van der Ende NAM, Lingsma HF, et al. ; MR CLEAN Investigators . National Institutes of Health Stroke Scale: an alternative primary outcome measure for trials of acute treatment for ischemic stroke. Stroke. 2020;51(1):282-290. doi: 10.1161/STROKEAHA.119.026791 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Rangaraju S, Frankel M, Jovin TG. Prognostic value of the 24-hour neurological examination in anterior circulation ischemic stroke: a post hoc analysis of 2 randomized controlled stroke trials. Interv Neurol. 2016;4(3-4):120-129. doi: 10.1159/000443801 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Saver JL, Altman H. Relationship between neurologic deficit severity and final functional outcome shifts and strengthens during first hours after onset. Stroke. 2012;43(6):1537-1541. doi: 10.1161/STROKEAHA.111.636928 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Sarraj A, Hassan AE, Abraham MG, et al. ; SELECT2 Investigators . Trial of endovascular thrombectomy for large ischemic strokes. N Engl J Med. 2023;388(14):1259-1271. doi: 10.1056/NEJMoa2214403 [DOI] [PubMed] [Google Scholar]
- 33.Huo X, Ma G, Tong X, et al. ; ANGEL-ASPECT Investigators . Trial of endovascular therapy for acute ischemic stroke with large infarct. N Engl J Med. 2023;388(14):1272-1283. doi: 10.1056/NEJMoa2213379 [DOI] [PubMed] [Google Scholar]
- 34.Yoshimura S, Sakai N, Yamagami H, et al. Endovascular therapy for acute stroke with a large ischemic region. N Engl J Med. 2022;386(14):1303-1313. doi: 10.1056/NEJMoa2118191 [DOI] [PubMed] [Google Scholar]
- 35.Saver JL, Goyal M, van der Lugt A, et al. ; HERMES Collaborators . Time to treatment with endovascular thrombectomy and outcomes from ischemic stroke: a meta-analysis. JAMA. 2016;316(12):1279-1288. doi: 10.1001/jama.2016.13647 [DOI] [PubMed] [Google Scholar]
- 36.Boers AMM, Jansen IGH, Beenen LFM, et al. Association of follow-up infarct volume with functional outcome in acute ischemic stroke: a pooled analysis of seven randomized trials. J Neurointerv Surg. 2018;10(12):1137-1142. doi: 10.1136/neurintsurg-2017-013724 [DOI] [PubMed] [Google Scholar]
- 37.Geurts M, de Kort FAS, de Kort PLM, van Tuijl JH, Kappelle LJ, van der Worp HB. Predictive accuracy of physicians’ estimates of outcome after severe stroke. PLoS One. 2017;12(9):e0184894. doi: 10.1371/journal.pone.0184894 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Counsell C, Dennis M. Systematic review of prognostic models in patients with acute stroke. Cerebrovasc Dis. 2001;12(3):159-170. doi: 10.1159/000047699 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
eTable. Main Effects of Final Model in Derivation Cohort and Validation Cohort Presented as Common Odds Ratios With 95% CIs
eAppendix. R Code (Full Regression Equation)
eFigure 1. Flow of Patients in Derivation Cohort (HERMES)
eFigure 2. Flow of Patients in Validation Cohort (MR CLEAN Registry)
Nonauthor Collaborators. HERMES Collaborators and MR CLEAN Registry Investigators
Data Sharing Statement