Tissue outcome prediction in hyperacute ischemic stroke: Comparison of machine learning models

Abstract

Machine Learning (ML) has been proposed for tissue fate prediction after acute ischemic stroke (AIS), with the aim to help treatment decision and patient management. We compared three different ML models to the clinical method based on diffusion-perfusion thresholding for the voxel-based prediction of final infarct, using a large MRI dataset obtained in a cohort of AIS patients prior to recanalization treatment. Baseline MRI (MRI₀), including diffusion-weighted sequence (DWI) and Tmax maps from perfusion-weighted sequence, and 24-hr follow-up MRI (MRI_24h) were retrospectively collected in consecutive 394 patients AIS patients (median age = 70 years; final infarct volume = 28mL). Manually segmented DWI_24h lesion was considered the final infarct. Gradient Boosting, Random Forests and U-Net were trained using DWI, apparent diffusion coefficient (ADC) and Tmax maps on MRI₀ as inputs to predict final infarct. Tissue outcome predictions were compared to final infarct using Dice score. Gradient Boosting had significantly better predictive performance (median [IQR] Dice Score as for median age, maybe you can replace the comma with an equal sign for consistency 0.53 [0.29–0.68]) than U-Net (0.48 [0.18–0.68]), Random Forests (0.51 [0.27–0.66]), and clinical thresholding method (0.45 [0.25–0.62]) (P < 0.001). In this benchmark of ML models for tissue outcome prediction in AIS, Gradient Boosting outperformed other ML models and clinical thresholding method and is thus promising for future decision-making.

Introduction

In the last two decades, multiple attempts have been made to predict the fate of the ischemic tissue resulting from acute occlusion of an intracranial artery. ¹ Specifically, magnetic resonance (MR) parameters derived from diffusion-weighted imaging (DWI) and perfusion-weighted imaging (PWI) have been shown to correlate well with the voxel-wise defined final infarct. ² Thus, the ischemic core and the at-risk hypoperfused tissue are mapped using the apparent diffusion coefficient (ADC) <620 × 10⁻⁶ mm²/s threshold and the timepoint of the maximum of the residue function (Tmax) >6 seconds threshold, respectively. ³ Thanks to commercially available automated software that use these thresholds, maps of the at-risk tissue can be quickly generated and are now widely used to select optimal candidates for reperfusion therapy. ⁴ However, such uniform thresholding approach has several shortcomings. First, the above fixed thresholds remain debated and alternative thresholds have been proposed for both the ADC^5–7 and Tmax.^8–11 Second, these thresholds rely on rather simplistic physiological assumptions and do not encompass the broad information range obtainable from DWI and PWI that may hold further clues to tissue fate. ¹² Last, spatial information is neglected, i.e., the clinical thresholding method essentially considers all voxels in the same way although gray and white matter have different sensitivity to prolonged hypoperfusion, ¹³ and each voxel is analyzed independently of regional information. One of the challenges with voxel-based prediction is that underlying tissue characteristics likely have an influence on the ultimate fate of the ischemic tissue. ¹⁴

Machine Learning (ML) is a class of computer algorithms that automatically learn to classify observations from data of a training set. Several ML framework have been proposed for tissue outcome prediction to find more flexible decision approaches than hard thresholding rules such as Random Forests classifiers, ¹⁵ , ¹⁶ or Gradient Boosting model. ¹⁷ These models combine image features in nonlinear ways to obtain more flexible decision boundaries. Another type of ML approach, termed Deep Learning, does not require a priori assumptions of what image features are important. Image features are automatically identified and encoded in the network of hidden layers. ¹⁴ U-Net, a class of Deep Learning method, have been proposed for tissue outcome predictions. ¹⁸ , ¹⁹ It enjoys the prestige of recent achievements in the Ischemic Stroke Lesion Segmentation 2017 challenges, ²⁰ in which different Deep Learning architectures were compared for lesion outcome prediction, and which was won by a multiscale U-Net.

The main benefit expected from Deep Learning models for segmentation tasks, such as U-Nets, is their ability to consider global image context for image features extraction. By contrast, other ML models, such as Random Forests or Gradient Boosting classifiers, only incorporate previously defined information and commonly do not consider regional information for predicting the fate of a given voxel. To overcome this limitation, voxel patches surrounding each predicted voxel can be integrated as additional input dimensions. ²¹ However, the voxel patch size is limited by the computational cost that will markedly increase with patch size. Besides the computational cost entailed, the comparison between these different predictive models is complicated because of the various ground truths used in previous studies (i.e., early ¹⁹ or delayed ¹ , ¹⁵ , ¹⁸ follow-up imaging), and the various metrics used for model evaluation (i.e., area under the curve [AUC], ¹ , ¹⁵ , ¹⁸ infarct volume, ¹⁵ , ¹⁸ Dice score ¹⁵ or accuracy ¹⁹ ).

Here, we aim to compare for the first time the accuracy of three ML approaches (namely Random Forests, Gradient Boosting and U-Net) to that of the clinical diffusion-perfusion thresholding paradigm for the prediction of final infarct based on a large pretreatment MR dataset of AIS patients. In order to compare the predictions of final infarct on a voxel basis, we used several of the most popular metrics, including Dice Score, AUC and Absolute Volume Error.

Material and methods

Patients selection

We carried out a retrospective single-center study on consecutive AIS patients who received reperfusion therapy between 2002 and 2019. Inclusion criteria were: i) baseline stroke MRI including DWI and PWI, obtained before reperfusion therapy (MRI₀); and ii) follow-up MRI scheduled around 24 hrs later including DWI (MRI_24h). As per French recommendations, ²² MRI has been implemented in our institution since 2001 as first line diagnostic work-up in candidates to reperfusion, while follow-up MRI is systematically scheduled around 24 hrs after treatment.

Both MRI₀ and MRI_24h include DWI with identical acquisition parameters. All MRIs were acquired on a 1.5-Tesla unit (GE Healthcare, Madison, Wis), using gradient strength of 33 mT/m and an eight-channel head coil. For both MRI₀ and MRI_24h, echo-planar DWI sequences were acquired with 128 × 128 matrix, 24 cm field-of-view, 6-mm thick slices, with b = 0 s/mm² and b = 1000 s/mm² with gradients applied in 3 orthogonal directions, TE = 84 ms, TR = 6675ms, and parallel imaging with acceleration factor = 2, no partial Fourier. ADC maps were computed based on DWI acquisition. On MRI₀, PWI was acquired after injection of a contrast bolus of gadolinium-based contrast agent using an echo-planar T2* sequence with a 96 × 64 matrix, 24 cm field-of-view, 6-mm thick slices, 25 temporal phases, TE = 60 ms, TR = 2000 ms, flip angle = 90°, no acceleration factor.

Patient consent

In accordance with French legislation, patients were informed of their participation in the study, and offered the possibility to refrain from the use of their data. A commitment to compliance (Reference Methodology MR-004 no. 4708101219) was filed to the French national information science and liberties commission (CNIL), in full respect of the General Data Protection Regulation. As the present study only involved retrospective analysis of anonymized data collected as part of routine care, formal approval by an Ethics Committee was not required.

Dataset preprocessing and image analysis

All MRI data were anonymized and exported from the Picture Archiving and Communication System. DWI_24h sequences were registered onto DWI₀, and DWI₀ and PWI were registered onto a reference brain scan in MNI space using 12-parameters affine registration using FSL FLIRT, ²³ , ²⁴ because affine MNI registration ¹⁷ and symmetry features ¹⁵ improve the quality of prediction by providing useful context on voxel location. Registration was conducted using global optimization ²³ with trilinear interpolation, and correlation ratio as similarity metric. Registration results were overlayed onto the MNI mask and manually checked by an experienced neuroradiologist. Whenever needed, the registration procedure was repeated using mutual information metric. Regarding the registration of DWI_24h, a concatenation of the two computed registration matrices was used to prevent doubling the interpolation errors. A brain mask was computed using Otsu-based thresholding on DWI₀ in order to select in-brain voxels. Image intensity normalization was then applied for DWI₀ and DWI_24h on each volume by mean centering and standard deviation scaling, using the mean signal and standard deviation of in-brain voxels contralateral to the ischemic lesion. Perfusion maps were computed using Olea Sphere® (Olea Medical, La Ciotat, France) based on oSVD deconvolution method with an oscillation index threshold of 0.02375. ²⁵ Arterial input function was automatically selected using an algorithm based on a clustering method classifying curves using their area under the curve, their roughness, and their first moment in order to distinguish arterial from tissue signal. ²⁶

ML models inputs and ground truth

Model inputs included DWI, ADC and Tmax maps obtained from MRI₀ (Tmax-only models). We primarily focused on Tmax as the sole surrogate for perfusion imaging because it correlates with clinical ²⁷ and tissue outcome, ¹⁸ and is widely used in clinical trials. ⁴ In order to test the added value of embedding other PWI maps, we also trained Extended-Perfusion models using Mean Transit Time (MTT), Cerebral Blood Flow (CBF) and Cerebral Blood Volume (CBV) as additional inputs. The ADC and PWI maps were not thresholded for ML model training. On MRI_24h, the final infarct was defined as the hyperintense lesion on DWI_24h, and was manually segmented and considered ground truth for our study. ²⁸ Given the high reproducibility of stroke volume measurements on DWI, ²⁹ segmentations were done by the same experienced neuro-radiologist, blinded to all clinical information.

Clinical thresholding model

As done in clinical care, core and tissue at risk were segmented from pretreatment DWI and PWI, respectively. An ADC threshold <620 × 10⁻⁶ mm²/s and Tmax > 6 sec were used to segment the core and the hypoperfused tissue-at-risk from MRI₀, respectively. ⁴ Regions of interest were manually segmented using MANGO software version 4.0.1 using a combination of thresholding and manual drawing tool for suppression of artifacts. ³⁰

The union of core and hypoperfused tissue-at-risk regions of interest on MRI₀, was considered as the predicted final infarct for the clinical thresholding model, as previously proposed by others. ³¹ These segmentations were not used for ML models training (see below).

ML models

Three different supervised ML models were tested: Random Forests and Gradient Boosting (patch-based models), and U-Net (Deep-Learning model). The models were trained to predict final infarct based on model inputs. In order to avoid the arbitrariness of splitting the population into a train and test set, ³² the training was cross-validated with 10 folds, a reasonable compromise between computation time and bias reduction. Cross-validation consisted in equally partitioning the patients in 10 folds. For each fold, the model was trained on 81% patients (training set), hyperparameters were adjusted on 9% of the patients (validation set) and metrics were evaluated on the 10% remaining patients (test set). The cross-validation splits were identical between models in order to ensure comparability.

The U-Net model was a fully convolutional network with 5 contracting blocks and 5 expanding blocks linked together by skip connections and ended with a sigmoid activation layer. It operated slice-by-slice with additional input from adjacent slices (2.5 dimensions). For each given slice, the U-Net provides a parametric output, which can be interpreted as a voxel-wise probability map of infarction on DWI_24h. The training process consisted in learning model weights by comparing the output of the model to the segmented brain infarct maps on DWI_24h. Loss was evaluated on the validation dataset, and the training was early stopped whenever validation loss increased in order to avoid model overfitting. ³³ The architecture of the U-Net model is detailed in online-only Data Supplement and will be published as open source software on https://github.com/NeuroSainteAnne/StrokePrediction.

Patch-based models (Gradient Boosting and Random Forests) combined features from each MRI₀ voxel to predict the final infarct on MRI_24h. The following features were computed for each voxel and each input volume: i) normalized signal intensity issued from 5 × 5 × 3 patches on each input sequence (75 features for each input volume); ii) normalized signal intensity issued from contralateral voxel patches (75 features for each input volume) selected by flipping x-coordinates of ipsilateral patches; iii) voxel coordinates in MNI space. The total feature cardinality was 453. After applying a subsampling factor of 5% in-brain voxels in train dataset, Gradient Boosting and Random Forests were trained to predict DWI_24h segmentation based on these features. Details about models implementation are given in online-only Data Supplement.

Probability maps issued by U-Net, Gradient Boosting and Random Forests were collapsed into a binary outcome prediction map using a 0.5 threshold.

Study endpoints

The main metric used for model evaluation was the patient-level Dice score. The Dice score reflects the amount of overlap between the prediction (i.e., the tissue outcome predicted by ML or clinical thresholding models) and the ground-truth (i.e., the segmented final infarct); it ranges between 0 and 1, with higher figures representing more overlap. It provides information not only on the predicted volume accuracy but also on its location. Other metrics were evaluated to keep comparability with previous studies: 1/Area Under the Curve (AUC); 2/AUC weighted on hypoperfused areas according to Jonsdottir et al. ¹ (AUC₀); and 3/Absolute Volume Estimation Error between predicted and ground-truth segmentation. Patient age, National Institutes of Health Stroke Scale (NIHSS) at admission, type of revascularization therapy (Intravenous Thrombolysis [IVT] and/or Mechanical Thrombectomy [MT]) were recorded. Training time for each model and inference time for each patient were recorded.

Reperfusion status

In the subgroup of patients who underwent MT, we analyzed the effect of adding reperfusion status onto model performance. Reperfusion status was evaluated on angiograms acquired immediately after MT using the modified Thrombolysis In Cerebral Infarction (mTICI) Score, ³⁴ with mTICI 2 b-3 considered successful reperfusion.

For the clinical thresholding model, the final predicted infarct was defined as the combination of infarct core and hypoperfused volumes in patients with unsuccessful reperfusion, given that the lesion typically expands to the boundaries of the initial hypoperfused region. In patients with successful reperfusion, the final predicted infarct was defined as infarct core only, given that most of the initial hypoperfused region should be salvaged. ³¹

For Gradient Boosting and Random Forests, reperfusion status as a binary variable was concatenated along with the other features. For U-Net, reperfusion status variable was tiled on a 256 × 256 map and concatenated along with the other inputs. Each model was trained in two different setups: with or without adding the Reperfusion status and the performance was assessed by 10-fold cross-validation.

Statistical analysis

The patient-level metrics were compared across models using a pairwise Wilcoxon test with p-value adjusted for multiple comparisons with Holm method. Given that large infarct volumes may bias prediction performance, ³¹ we also plotted predicted and true infarct volumes using a Bland-Altman graph and analyzed predictions using two predefined final infarct volume cut-points (50 mL, 100 mL). ³⁵ , ³⁶ Tmax-only models and Extended-Perfusion models were compared using pairwise Wilcoxon test. In the subgroup of patients who underwent MT, Dice scores issued from models with and without taking into account reperfusion status were compared with pairwise Wilcoxon test.

Patient-wise results are expressed as median and Interquartile Range (IQR) or mean ± standard deviation (SD) if needed for comparison with literature. Model overfitting was evaluated by comparing Dice score in the test and training sets. Statistical analyses were computed using R version 3.5.1 with ggplot2 ³⁷ and BlandAltmanLeh ³⁸ packages.

Results

Population

During the study period, of 788 AIS patients who received reperfusion therapy and in whom both MRI₀ and MRI_24h were performed, 394 had complete datasets including DWI₀, PWI and DWI_24h. Patients and stroke characteristics at admission and ≈24 h are summarized in Table 1.

Table 1.

Clinical and radiological characteristics.

Variable	Patients (n = 394)
History and characteristics at admission
Age, years, median (IQR)	70 (58–80)
Male gender	240 (61)
History of hypertension	221 (56)
Current smoking or history of smoking	150 (38)
Stroke causea
– Cardio-embolic	169 (3)
– Atherothrombotic	48 (15)
– Other determined cause	13 (4)
– Undetermined	90 (28)
Known time of symptom onset	368 (93.4)
Initial NIHSS score, median (IQR)	13 (8–26)
Initial imaging and treatment
Stroke onset to admission MRI delayb (mins), median (IQR)	115 (86–155)
Administered treatment
– IVT only	294 (75)
– MT only	46 (12)
– Bridging therapy (IVT + MT)	41 (10)
Stroke onset to-IVTc (mins), median (IQR)	155 (120–200)
Stroke onset-to-MTd (mins), median (IQR)	192 (150–260)
Successful reperfusiond	51 (59)
DWI0 volume, mL, median (IQR)	11 (4–35)
Hypoperfused volume (Tmax >6 sec), mL, median (IQR)	47 (12–108)
24-hours follow-up
MRI0 to MRI24hrs delay (hours), median (IQR)	24.3 (20.3–28.7)
Final infarct volume, mL, median (IQR)	28 (8–88)
– Final infarct volume <50 mL	253 (64)
– Final infarct volume ≥100 mL	83 (21)
24hrs NIHSS score, median (IQR)	8 (3–16)
Modified Rankin Scale score ≤ 2e	190 (53)

Note: Values shown are numbers (percentages), unless stated otherwise.

^aIn patients with available cause in medical file (n = 320).

^bIn patients with known stroke onset time (n = 368).

^cIn patients who received IVT (n = 335).

^dIn patients who underwent MT (n = 87).

^eIn patients with known modified Rankin Scale score (n = 355).

IQR: interquartile range; IVT: intravenous thrombolysis; MT: mechanical thrombectomy; NIHSS: National Institutes of Health Stroke Scale; DWI: diffusion weighted imaging.

Final infarct prediction performance

Gradient Boosting had significantly better Dice score as compared to all other models (P < 0.001), with a median score of 0.53 [IQR 0.29–0.68] (Table 2). Dice scores significantly differed among models, except between U-Net and Random Forests and between U-Net and clinical thresholding model (P = 0.13) (Figure 1). Using the AUC, AUC₀ and Mean Absolute Error, Gradient Boosting and U-Net performed equally well and outperformed Random Forests and clinical thresholding model (P < 0.001). There was no major overfitting for any of the studied models, as shown by the differences between Dice scores obtained in the training and test sets (Table 3). Dice scores were lower for volumes <50mL (0.44 [0.18–0.61] and 0.33 [0.03–0.58] for Gradient Boosting and U-Net, respectively; n = 253) than for volumes ≥100mL (0.70 [0.58–0.80] and 0.74 [0.60–0.80], respectively; n = 83). Adding other PWI maps as inputs did not modify model comparisons and none of the Extended-Perfusion models yielded significantly higher Dice scores than Tmax-only models (p > 0.1) (Table 3).

Table 2.

Final infarct prediction performance for each model.

Model	Dice score Median [IQR]	AUCMedian [IQR]	AUC0Median [IQR]	Volume absolute error (mL) Mean±SD
Clinical thresholding	0.45 [0.25–0.62]	0.66 [0.61–0.72]	0.80 [0.7–0.86]	44.1 ± 56.0
U-Net	0.48 [0.18–0.68]	0.98 [0.95–0.99]	0.94 [0.88–0.97]	28.0 ± 37.8
Random forests	0.51 [0.27–0.66]	0.95 [0.90–0.98]	0.91 [0.85–0.95]	28.8 ± 42.2
Gradient boosting	0.53 [0.29–0.68]	0.98 [0.95–0.99]	0.94 [0.89–0.97]	27.7 ± 40.3

IQR: interquartile range; AUC: area under curve; SD: standard deviation.

Figure 1.

Final infarct volume prediction performance for each model.

Violin plots represent the distribution of patient-wise Dice score for each model. Boxplot centers represent the median value and box lower and upper limits represent the 25th and 75th percentile for Dice scores distribution across the 394 patients. Brackets indicate statistical significance for pairwise comparisons between models. ns=not significant.

Table 3.

Comparison of dice scores between training and test sets.

Models and inputs	Dice score in training setMedian [IQR]	Dice score in test setMedian [IQR]
Tmax-only models
U-Net	0.52 [0.24–0.69]	0.48 [0.18–0.68]
Random forests	0.58 [0.37–0.71]	0.51 [0.27–0.66]
Gradient boosting	0.57 [0.33–0.71]	0.53 [0.29–0.68]
Extended-perfusion models
U-Net	0.50 [0.19–0.69]	0.48 [0.17–0.68]
Random Forests	0.61 [0.42–0.73]	0.52 [0.28–0.67]
Gradient boosting	0.58 [0.37–0.71]	0.54 [0.31–0.68]

Note: Metrics were compared between training and test sets. Random Forests were slightly more prone to overfitting as compared to Gradient Boosting and U-Net, with higher Dice Scores in the training set than in the test set. IQR: Interquartile Range.

Reperfusion status

In the subgroup of patients treated with MT in which reperfusion data were available (n = 87), adding the reperfusion status as input significantly improved the performance of all models: Gradient Boosting (median Dice score 0.55 [IQR 0.28–0.64] vs. 0.47 [0.28–0.63], P = 0.008), Random Forests (0.48 [0.25–0.61] vs. 0.46 [0.23–0.61], P < 0.001), U-Net (0.46 [0.29-0.63] vs. 0.38 [0.12–0.55], P = 0.038), and clinical thresholding (0.49 [0.31–0.63] vs. 0.39 [0.22-0.59], P = 0.002).

Qualitative and quantitative assessment of final infarct volume

Illustrative examples of prediction maps are presented in Figure 2. By simple visual inspection, Random Forests and Gradient boosting yielded sharper and better contrasted but noisier prediction maps than U-Net. Bland-Altman analysis (illustrated only for Gradient Boosting and U-Net in Figure 3) demonstrated an underestimation of predicted volumes (mean difference: −18 mL [limits of agreement: −107-70mL] for Gradient Boosting and −14mL [−101-74mL] for U-Net), which was more prominent for larger volumes.

Figure 2.

Examples of prediction maps with different models.

Four typical examples with pre-treatment MRI₀ (DWI₀ and Tmax maps), ground-truth segmentation of DWI_24h final infarct (black outline) and prediction maps yielded by different models. Corresponding Dice scores (DSC) are given for each prediction map. In patient (a), a small core infarct is seen on DWI₀ with large but moderately hypoperfused area on Tmax. Gradient Boosting performs better and predicts a larger zone of final infarct than other methods. Patient (b) illustrates a small ischemic core on DWI₀ and a small and moderately hypoperfused on Tmax. In patient (c), both large core and large hypoperfusion are seen on MRI₀. In these latter two examples, the models perform well with higher Dice score for high infarct volume. In patient (d), vasogenic edema with mass effect was present on MRI_24h. All models yielded a Dice score >0.5 but underestimated the final infarct volume, mainly by missing the mass effect.

Figure 3.

Bland-Altman plots for the Gradient Boosting and U-Net models.

The solid line represents mean agreement between real and predicted values, and dotted lines represent upper and lower limits of agreement. Greater infarct volumes were associated with larger underestimations.

Computation time

Training time was longer for U-Net (median duration per cross-validation fold: 160 minutes) than for Gradient Boosting (89 minutes) and Random Forests (19 minutes). Inference times for each patient were shorter for U-Net (median duration: 0.3 seconds) than for Gradient Boosting (21 seconds) and Random Forests (2 seconds).

Discussion

In this study, the three most popular ML segmentation models and the clinical thresholding method were compared for their performances in predicting final infarct, using a large single MRI dataset of AIS patients treated with reperfusion therapies. We found that Gradient Boosting model outperformed all other tested models including clinical thresholding method. Thus, Gradient Boosting produced better spatially defined and contrasted prediction maps, and accordingly would seem most suitable for clinical application. Of note, Gradient Boosting and U-Net yielded similar results with respect to absolute volume and AUC. Finally, all models tended to underestimate final infarct volume, especially for large infarcts.

Metrics analysis

Our results highlight the limitations of standard metrics for the evaluation of predictive models. The Dice score, which measures the overlap between predicted and ground-truth volumes, is theoretically the best metric to assess accuracy of both extent and spatial location of infarct prediction, ³⁹ which are important determinants of stroke severity and long-term functional outcome. ⁴⁰ Models with highest Dice score would thus best predict the fate of clinically relevant regions, although this metric in inherently unstable for small-size infarcts, as confirmed in our study.

Alternatively, AUC, although widely used, suffers from a large imbalance between (far more prevalent) healthy brain tissue and infarcted tissue. Hence, this metric carries limited information as all models reach very high AUC. The AUC₀, weighted on hypoperfused areas, is better adapted for small volumes and has the advantage of being threshold independent. ¹ As for the AUC, it was markedly skewed to high values (between 0.89 and 0.98 across our models) and may as such afford lesser discrimination for model selection than Dice scores. Finally, Mean Absolute Error has the advantages of i) straightforward clinical significance given that infarct volume is known to correlate with clinical scores and outcome, ⁴¹ and ii) invariance to spatial registration errors, but does not take into account spatial location errors.

Model comparisons

Overall, we obtained higher Dice scores than previously reported for both Random Forests ¹⁵ or Gradient Boosting. ¹⁷ This may be due to the fact that our models were trained and tested on large datasets with a wide range of infarct volumes, including larger ones (mean 60 mL ± 75 in our study vs 28 mL ± 79 in Grosser et al., ¹⁷ and median 28 mL in our study vs. 7.36 mL in McKinley et al. ¹⁵ ). The overlap between predicted and real volume is indeed expected to increase with larger infarcts, irrespective of the model, as also shown here.

We found that Gradient Boosting outperformed Random Forests, in line with a recent study on 99 AIS patients. ¹⁷ In this latter study, the difference in Dice scores between Gradient Boosting and Random Forests was comparable to ours (0.39 and 0.37 vs 0.53 and 0.51, respectively). This rather small gain in performance has to be put in balance with the additional computational cost, which in our study was 5 times longer with Gradient Boosting.

As compared with U-Net, Gradient Boosting had higher Dice score. This superiority might be due to differences in receptive fields between models. Patch-based models, such as Gradient boosting used here, have small receptive fields (patch: 5 × 5 voxels) as compared to U-Net (here, 32 × 32 voxels). Larger receptive fields should theoretically capture regional perfusion defects at a larger scale, and this may explain the better performance of U-Net for large infarct volumes. However, convolutional architectures use translation equivariance (i.e., the fact that the phenomenon of interest is equally likely to occur in all part of the image) as an inductive bias, which may be inadequate for AIS where lesions are not randomly localized but constrained by arterial territories. Moreover, U-Net prediction method did not outperform clinical thresholding model in our study. Although Deep Learning methods have been previously proposed in literature for tissue outcome prediction, ¹⁸ , ¹⁹ , ³¹ it seems to outperform standard clinical thresholding methods in terms of Dice Score only in a patients with major reperfusion. ³¹ It is however likely that the performance of Deep Learning methods, such as U-Net, being data-greedy, will continue to improve with access to very large amount of data whereas other ML approaches such as Gradient Boosting and Random Forests will tend to plateau despite access to the same amount of data. ¹⁴ Future much larger multicentric datasets may thus be necessary to demonstrate the superiority of Deep Learning over other techniques. Beside its accuracy, the critical point for a clinically compatible software in AIS patients is the speed necessary to obtain prediction maps. Owing to their fast inference once trained, as shown in our study, Deep Learning models that do not require patch segmentation might better translate into clinically usable software.

The best performing models proposed in literature are presented in Table 4. Of note, all models outperformed linear models whenever compared. ¹⁷ , ¹⁸ , ⁴² , ⁴³ This was not unexpected since linear models combine linear transformations of voxel intensities and have limited expressive power. Accordingly, we chose to omit the comparison with linear models in our study.

Table 4.

Comparison with current literature.

Authors	Methoda	AUC	AUC0	Dice Score	Mean volume absolute error (mL)	Number of patients (training/test set)
Scalzo et al. 42	Kernel spectral regression		0.91	–	–	25b
Nielsen et al. 18	Deep learning (SegNet)	–	0.88	–	–	187 (158/29)
McKinley et al. 15	Random forests (FASTER)	–	–	0.34 (±0.22)	30 (±26)	100 (45/55)
Pinto et al. 47	Deep learning (U-Net + Gated recurrent unit)	–	–	0.35	–	75 (43/32)
Winder et al. 43	Random forests	–	–	0.45	–	90b
Grosser et al. 17	Gradient boosting		0.89	0.39	–	99c
Yu et al. 31	Deep learning (U–net)	0.92	–	0.53	–	182d
Our study	Deep learning (U-Net)	0.98	0.94	0.48	28 (±38)	394c
	Gradient boosting	0.98	0.94	0.53	27 (±40)	394c

^aBest performing model for each study is presented.

^bLeave-one-out cross validation.

^c10-fold cross-validation.

^d5-fold cross-validation.

A recent large study reported better performance of a U-Net model than found in our study (Dice score: 0.53 vs 0.48, respectively). ³¹ This could be explained by the use of a U-Net model with attention-gating units, which may improve U-Net performance. More likely, higher Dice score in this study are linked to larger final infarcts (median [IQR] 54 mL [16-117] vs. 28 mL [8-88] here) and more severe strokes that were imaged and treated later than in our study. Nielsen et al. ¹⁸ have investigated another deep convolutional network (SegNet) for infarct prediction but obtained lower AUC₀ values (0.88 ± 0.12 vs. 0.94 for U-Net in our study). The ISLES 2016 and 2017 challenges compared many algorithms for final infarct prediction based on multimodal MRI in smaller populations (75 patients). ²⁰ The best performing Deep Learning model yielded a lower Dice Score (0.32) than all the models tested in our study. However, these comparisons should be interpreted with caution given that population characteristics and sample size differ between studies. The raw metrics indeed appear highly dependent on the population studied, which emphasizes the importance of comparing concurrent models in the same sample. Although it is beyond the scope of the study to compare all existing models, we performed post-hoc supplementary analyses by comparing the performance of our U-Net model to the deep convolutional network proposed by Nielsen et al. ¹⁸ on our data set. Performances were comparable, with similar median Dice scores (0.48 [0.19-0.66] for the network proposed by Nielsen et al vs. 0.48 [0.18-0.68] for our U-Net, P = 0.07). Other Deep Learning models, such as X-Net ⁴⁴ or multi-path U-Net ⁴⁵ have been proposed for acute AIS segmentation and may also be tested for prediction. Taking into account additional information regarding stroke texture might also be valuable. ¹⁵ Furthermore, combining the results of different models with ensemble methods might boost the performances of individual models, as suggested in the ISLES 2017 challenge. ²⁰

Patient heterogeneity and model interpretation

All ML models tested in this study underestimated true final infarct volume. This underestimation was more prominent for large infarcts, in line with a previous study of infarcts imaged in the subacute stage (3–7 days). ³¹ Conversely, volume overestimations were observed in studies using ground truth infarct volume at later time points (1 month ¹⁸ and 3 months ¹⁵ ). These apparent discrepancies are likely explained by the natural evolution of necrotic lesions, which are characterized by peripheral vasogenic edema appearing ∼24hrs after stroke onset and peaking during the first week, followed by lesion shrinkage toward the chronic stage (1-3 months). Accordingly, volume underestimation observed in studies where ground truth is defined soon after treatment, including ours, likely reflects failure to capture vasogenic edema, ³¹ while volume overestimation in studies with delayed ground truth reflects failure to capture lesion shrinkage. ¹⁵ , ¹⁸

Reperfusion effect

Successful reperfusion is associated with limited infarct growth and better functional outcome. ⁴⁶ As expected, integrating reperfusion status in our models improved their performance. This approach has been proposed in previous studies ¹⁵ , ⁴⁷ using different frameworks. In the FASTER algorithm, McKinley et al. trained two separate models for reperfused and non-reperfused subjects. ¹⁵ Pinto et al. integrated reperfusion status into their neural network using gated-recurrent unit. ⁴⁷ We found a similar gain in performance (Dice score increment +0.07 here with Gradient boosting model vs. +0.05 in Pinto et al. ⁴⁷ ) although we chose to concatenate reperfusion status with imaging data to ensure a similar architecture of all models for the sake of comparability. These results emphasize the potential clinical application of such prediction models. Indeed, for a given patient, creating differential prediction maps with and without reperfusion may have a direct application in treatment decision. Moreover, incorporation of other input parameters such as age, admission NIHSS, and glycaemia may improve prediction accuracy, although careful variable selection would be needed to limit model overfitting.

Limitations

Our study has limitations. First, we mainly focused on Tmax because of its known correlation with clinical outcome ²⁷ and final infarct volume. ⁴⁸ Moreover, it was the predominant contributing feature among other perfusion maps in a large ML study predicting tissue outcome in AIS. ¹⁵ We however checked that Extended-Perfusion models did not outperform Tmax-only based models. One might consider other inputs, such as the CBF/Tmax ratio ⁴⁹ which may be a marker of collateral status. An alternative approach entails the use of the source PWI images ¹² instead of the parametric maps, which avoids the use of proprietary deconvolution software and integrates all information contained in the PWI dataset without the need for preliminary perfusion modeling. Second, our clinical thresholding model used predefined ADC and Tmax thresholds as done previously, ³¹ although they are likely imperfect.^5–11 However, they represent the state-of-the-art for estimation of ischemic core and at-risk tissue, and are now used routinely in the clinical setting to select patient for MT beyond 6 hrs after stroke onset. ⁴ Third, final infarct was defined on 24 hr follow-up MRI, an earlier time point than that used in previous studies. ¹⁵ , ¹⁸ This approach was driven by consensus recommendations ²⁸ and has the advantages of allowing more reliable segmentation thanks to the high natural contrast of infarcted tissue on DWI, of minimizing the risk of final infarct overestimation due to vasogenic edema that peaks around day 3-5 ²⁸ and of reducing the odds of both early stroke recurrence and drop-outs. Fourth, we limited our analysis to the effect of reperfusion status in the subgroup of patients who underwent MT, since documentable early reperfusion status was not available in patients treated with IVT only. Fifth, binary outcome prediction maps were generated using an arbitrary fixed threshold (0.5) applied on ML probability maps. However, Dice scores were not significantly improved using a threshold optimization method (see Online-Only Data Supplement, Supplemental Table 1). Sixth, we used a cross-validation scheme, which increases the statistical power but reduces the interpretability of trained models. In order to check the robustness of our findings, we conducted an additional analysis using a single train/validation/test split with a larger test set (25%). Results were similar in terms of Dice scores (see Online-Only Data Supplement, Supplemental Table 2) but statistical power was insufficient to conclude regarding differences between ML models (e.g. 0.54 for Gradient Boosting vs. 0.52 for U-Net, P = 0.06) (see Online-Only Data Supplement, Supplemental Figure 1). Finally, this was a single-center study, and all patients had a standardized stroke MR protocol. While this increases data homogeneity and hence model performance, the trained models may not perform equally well on a more heterogeneous multi-center dataset.

Conclusion

The present work reports a benchmark of ML models for tissue outcome prediction in acute stroke using a large dataset. All models outperformed the thresholding PWI-DWI methods used in clinical practice. Among the three ML models, Gradient Boosting appeared particularly relevant for predicting tissue fate after AIS, followed closely by Random Forests and U-Net. These novel approaches appear promising for predicting tissue outcome and hence decision-making.