Random Expert Sampling for Deep Learning Segmentation of Acute Ischemic Stroke on Non-contrast CT

Abstract

Purpose

To automatically identify the presence and location of ischemic brain tissue on Non-Contrast CT

Materials and Methods

The data set consisted of 260 Non-Contrast CTs from 233 patients of acute ischemic stroke patients recruited in the DEFUSE 3 trial. A benchmark U-Net was trained on the reference annotations of three experienced neuroradiologists to segment ischemic brain tissue using majority vote and random expert sampling training schemes. We used a one-sided Wilcoxon signed-rank test on a set of segmentation metrics to compare bootstrapped point estimates of the training schemes with the inter-expert agreement and ratio of variance for consistency analysis. We further compare volumes with the 24h-follow-up DWI (final infarct core) in the patient subgroup with full reperfusion and we test volumes for correlation to the clinical outcome (mRS after 30 and 90 days) with the Spearman method.

Results

Random expert sampling leads to a model that shows better agreement with experts than experts agree among themselves and better agreement than the agreement between experts and a majority-vote model performance (Surface Dice at Tolerance 5mm improvement of 61% to 0.70±0.03 and Dice improvement of 25% to 0.50±0.04). The model-based predicted volume similarly estimated the final infarct volume and correlated better to the clinical outcome than CT perfusion.

Conclusion

A model trained on random expert sampling can identify the presence and location of acute ischemic brain tissue on Non-Contrast CT similar to CT perfusion and with better consistency than experts. This may further secure the selection of patients eligible for endovascular treatment in less specialized hospitals.

1. Introduction

Stroke is a leading cause of death and disability globally [1, 2]. Timely treatments, such as thrombolysis or thrombectomy, represent effective treatment options to improve clinical outcomes for patients suffering from acute ischemic stroke [3, 4, 5, 6]. Non-Contrast CT (NCCT) is widely used to differentiate ischemic from hemorrhagic stroke. It can be further used to estimate the extent of irreversibly damaged brain tissue as an imaging biomarker for the appropriate triage of patients [7, 8].

Advances have been made to facilitate the delineation of ischemic brain tissue with ruled-based algorithms for perfusion imaging [9]. But resources, specialized technicians, and physicians are needed for interpretations. The most utilized (96–100%) and cheapest imaging modality for acute stroke patients is the NCCT [7, 10].

The ASPECT score is an established semi-quantitative method to evaluate treatment eligibility in ischemic stroke patients based on NCCT by dividing the affected hemisphere into 10 structural regions [3, 4]. However, the extent and relative quantity of damaged brain tissue within a region to be rendered has not been defined [11, 12]. The individual regions cover different amounts of brain tissue which results in their unequal weighing into the score. The location of the stroke has a minor role in the ASPECT score value. Although it has been shown that the location of the stroke estimates the long-term effect on patients’ outcome more precisely [13, 14, 15, 16].

A segmentation tool for acute ischemic brain tissue on NCCT may not only accurately quantify the volume but also allows the mapping of the stroke location to symptoms to better guide treatment decisions. However, a low signal-to-noise ratio limits rule-based algorithms and expert neuroradiologists to segment ischemic stroke on NCCT. An automated pipeline is therefore of high interest [17, 18, 19].

Supervised deep learning models have been widely applied for the segmentation of medical images and show promise for applications in stroke imaging, but rely on accurate expert reference annotations segmentation during training [20, 21]. To enhance the accuracy of ground truth segmentation, multiple expert annotations are commonly fused with a majority or with the probabilistic fusion algorithm STAPLE. The result is a binary ground truth (a voxel is part of the ischemic core or not). Note that STAPLE is not applicable in the context of this study, where sometimes only two, one, or no experts agree on ischemic stroke segmentation [22]. Instead, we use random expert sampling, meaning randomly choosing an expert segmentation as ground truth for each training example. We then compare the model’s performance to the standard majority vote. We hypothesize that randomly sampling experts during training is better in approximating the ground truth of the ischemic core better than a majority vote due to its ability to encode the true probability when using only binary expert labels. For ischemic brain segmentation, the true probability of neurons being irreversibly damaged by ischemia maps to a continuous function of time, collateral blood flow, and clinical variables [23, 24, 25]. If we consider a Bernoulli random variable corresponding to a single pixel (a coin flip with probability p) is defined as 1 with probability p and 0 with probability 1 − p. Consider this variable with p = 50% with three samples corresponding to the annotations of three experts that happen to be (0, 0, 1). If we used a majority vote pre-processing scheme, this would be summarized as a single zero, and any resulting model or estimator would attempt to predict a zero. However, maximizing the cross entropy loss function of the deep learning model (identical to taking the maximum likelihood estimate, (Supplemental Material, Section 12.1.1) would result in an estimate of 1/3. In fact, we’d expect a better estimate in 75% of three-expert datasets. Finally, we correlate the prediction of the random expert model to the clinical outcome and compare it to CT perfusion and follow-up MR imaging.

2. Methods

2.1. Study Design and Data

In this post hoc analysis of the randomized DEFUSE 3 trial, 260 Head NCCT examinations from 233 patients with enrollment between May 2016 through May 2017 were included. The primary trial outcome determined thrombectomy eligibility for patients with acute ischemic stroke with an onset time within 16 hours. Detailed information of the inclusion and exclusion criteria can be found in https://clinicaltrials.gov/ct2/show/NCT02586415 and [9]. Furthermore, 156 consecutive patients without an ischemic stroke from a different institutional database were included to evaluate the image-classification task (relevant stroke vs. no relevant stroke volume). These patients were screened for stroke (from 2011 to 2017) and confirmed to have no stroke with a normal DWI scan 24h after the initial NCCT, but were diagnosed with an alternative condition on the basis of all available clinical and radiologic data available to the treating physician [26]. We included patients from the University Hospital Lausanne (UHL) (1723 patients) as an external generalization cohort. We randomly chose 35 patients of this cohort where each patient was admitted between 2016 through 2019, had an onset time between 3h to 24h and an initial NCCT study. Institutional review board approval was obtained, and consent was waived.

2.2. Image Reference Annotations

Three experienced fellowship-trained neuroradiologists (experts A, B, and C) with 4, 4, and 5 years of post-fellowship experience (A.M., B.F.J.V., J.J.H.). Manual segmentations were done in Horos (Horosproject.org, version 4.0.0). The experts were given the information that each patient had a large vessel occlusion, but the side of the vessel occlusion was not provided. If no abnormal hypodensity was found, experts were offered the option of no segmentation. All manual segmentations were checked for correct coregistration to the NCCT. For the data set of the healthy patient, an empty segmentation mask was generated. The external generalization cohort was segmented by one fellowship-trained neuroradiologists with 2 years of post-fellowship training.

2.3. Data Preparation for Training

The cohort of 233 patients was randomly split into five subgroups for five-fold cross-validation with 208 patients for the training, and 52 for the test sets. If a patient had undergone multiple baseline NCCT (n=27 patients), the additional scans were only added to the corresponding patient NCCT in the training set to prevent information leakage (Figure 1). The 156 healthy cases were randomly and equally added to the test sets (n=33 per test set) with empty ground truth annotations.

Figure 1:

Flowchart of the data partition. 233 patients with their initial NCCT were randomly split into 5 folds of training and test sets. 25 patients had multiple NCCT. Those were only assign to the initial NCCT when in the training set. The external generalization cohort included 35 patients.

2.4. Model Configuration and Training

A preprocessing pipeline and data loader were created to allow on the fly training with majority vote and random expert sampling (Figure 2). A majority vote means to sum the reference masks of all experts, set the voxels with a value greater than half the number of experts to one, and all other voxels to zero. Random expert sampling means choosing an expert for each training example with an equal probability.

Figure 2:

Training scheme pipeline with sampling strategy for random expert sampling and majority vote

A baseline nnUNet architecture and default hyperparameter include a patch size of 28×256×256 and spacing of (3.00, 0.45, 0.45), 7 stages with two 3D convolutions per stage, leaky ReLU as activation function, Dice + Cross-Entropy loss function, a batch size of 2, SGD optimizer with 0.99 Nestov momentum and He initialization (Section 12, Fig. S1). The model was trained with NCCT as input and manual annotations of experts as ground truth to output a predicted segmentation mask of acute ischemic stroke on NCCT [27]. As all models share the same core nnUNet component and for fairness and ease of comparability, we let all models undergo the same training schedule with default hyperparameters to prevent information leakage.

2.5. Model Testing

All analyses have been performed on the aggregated test sets of the five folds. This procedure is explored and validated in previous literature [28]. The results of the aggregated test sets (n=389) were tested for normal distribution with the Shapiro and shown as median and 95% CI estimated by the bootstrapping (R=1000)[29] We tested the best-performing model (overall highest metrics value) on the external generalization cohort.

2.6. Outcome Measures and Statistical Analysis

R (Version 2022.02.3) was used for statistical analysis.

All three expert reference annotations per NCCT were compared to the inter-expert agreement, to the prediction of the majority vote model, and to the prediction of the random expert sampling model (Figure 3). The per-patient median was used for further analysis.

Figure 3:

First two model were trained on majority vote and random expert sampling. Second, the median agreement per case for inter-expert agreement, model-expert agreement for the prediction of majority vote and random expert sampling was the basis to compare random expert sampling to the majority vote and inter-expert agreement.

The model-based predictions were evaluated for segmentation error, image classification, volume classification, correlation to CT ischemic core, 24h-follow-up volume, and clinical outcome prediction.

To access the segmentation and image-classification task a clinically motivated threshold of 1ml was chosen to differentiate between relevant stroke and no relevant stroke volume. For example, after determining no relevant stroke volume on the NCCT, the location, and volume of ischemic brain tissue of 0.5 ml are unlikely to influence the triage of patients and excluded from the segmentation evaluation. Expert A, B, and C had 155, 226, and 206 manual segmentations > 1ml and 78, 7, and 27 manual segmentations < 1ml, respectively.

All NCCTs with a median reference annotation above 1ml were evaluated for segmentation error with the following metrics.

• Volume-based metrics (Volumetric similarity and absolute volume difference [ml])

• Overlap metrics (Dice, Precision and Recall)

• Distance metrics Hausdorff distance with the 95 percentile [mm], and the surface dice at tolerance 5mm

The definition can be found in the supplemental material 12, Table S1 [30].

The Surface Dice at Tolerance measures the distances between each surface voxel of the reference and predicted mask. The chosen tolerance allows a maximum distance between surface voxels on the reference and predicted masks to be considered true positive voxels. We use this metric to account for possible pathophysiological and modality-related lower signals that cause more variability in the outer compared to the inner region of the ischemic stroke [31, 32].

For the image-classification task, the predictions were categorized in stroke volume above or below 1ml including the healthy cases. Sensitivity, specificity, F-score, Correct Classification Ratio, and Area under the curve served as metrics.

We used each metric to evaluate how close the model-based predictions were to experts (accuracy) and how consistent the model-based predictions were across the patient population (precision).

Accuracy was tested for statistical superiority with a one-sided Wilcoxon sign rank test and a lower boundary of 0.0 (p < 0.05, Figure 3). We evaluate for precision with the ratio of variances and standard error (bootstrapped, R=1000). We then tested for statistical superiority with a delta of 0.0.

To evaluate for ischemic core volume size that alters clinical treatment decisions based on ANGEL-ASPECT and SELECT-2 [4, 3], the predicted segmentations were classified in <1ml, <50ml, <100ml and >100ml and compared to the median expert volume with Cohen’s and Fleiss’ kappa.

To put the results into perspective, we computed the Spearman correlation coefficient of the ASPECT and ischemic core volumes estimated by experts, deep learning models, and CT perfusion to the clinical outcomes. We compared the ischemic core volumes of the random expert sampling model and CT perfusion (30% CBF) to the 24h-follow-up DWI (final infarct core) with Bland-Altman plots in patients with full reperfusion (TICI ≥2B). We analyzed differences in correlation coefficients with Fisher’s z-test.

All p-values were adjusted for multiple comparisons with the Holm-Bonferroni method.

2.7. Model and Data Set Availability

The model and data set are available upon request to the corresponding author.

3. Results

3.1. Patient Characteristics

We analyzed 233 randomized and non-randomized patients in the DEFUSE 3 trial (121 women (52%), median (IQR) age, 69 (59–78) years). The expert ischemic core volume and ischemic core (CBF<=30%) and penumbra (Tmax . =6) volume with perfusion imaging were 8[3–26]ml, 11[2–28] and 104[62–157] ml, respectively (median [IQR] ml). The median onset to image time of the 50 patients witnessed was 10[8–12]h. Further patient characteristics are summarized in Table 1.

Table 1:

Patient characteristics

Categories	Characteristic	Randomized-Train1	Non-randomized-Train1	Total-Train1	Total-Test	p-value7
General	Total Number	146	87	233	35
	Age	70 (59–79)	67 (58–76)	69 (59–78)	71 (64–78)	0.4
	Female %	52	52	52	54
Imaging Characteristics	Expert A Volume [ml]	9 (4–21)	22 (6–69)	12 (5–32)
	Expert B Volume [ml]	14 (5–27)	14 (0–60)	14 (4–37)
	Expert C Volume [ml]	3 (1–7)	7 (0–43)	4 (1–12)
	Ischemic Core Volume [ml]	9 (2–27)	18 (0–77)	11 (2–38)
	Tmax6 Volume [ml] 2	117 (78–158)	69 (3–150)	104 (62–157)
	ASPECTS on Baseline CT 3	8 (7–9)	8 (5–10)	8 (7–9)
Process	Witnessed Number	50		50
	Wake-Up Number	77		77
	Unwitnessed Number	19		19
	Onset to Image Time [h]	10 (8–12)		10 (8–12)	10 (5–13)	0.43
Follow-Up	24h DWI Number	146		146
	24h DWI Volume [ml]	39 (24–110)		39 (24–110)
Clinical Outcome	mRS5 at Baseline	0 (0–0)		0 (0–0)	0 (0–0)
	mRS5 at 90 days	4 (2–5)		4 (2–5)	2 (1–3)	0.002

¹Median (1st – 3rd quantile), if not otherwise indicated,

²Time-to-Maximum after 6 seconds,

³Alberta Stroke Program Early CT Score,

⁵modified Ranking Scale,

⁶data for non-randomized patients not available,

⁷double sided Wilcoxon test

3.2. Model Performance

The model-based segmentation with the random expert sampling showed statistically significant better agreement with experts than inter-expert agreement and majority vote training scheme (median ± 95% CI (bootstrapped), Surface Dice at Tolerance 0.7±0.03 vs. 0.56±0.03, 0.68±0.05, Dice 0.50±0.04 vs. 0.31±0.04, 0.29.±0.01, Absolute Volume Difference of 5.36±1.32 ml vs. 10.20±2.09 ml, 7.91±1.77 ml) (Table 2).

Table 2:

Segmentation, Internal Evaluation

		Internal Evaluation					External Evaluation
Categories Metric		Random Sampling5	Interexpert5	p-value random-interexpert6	Majority Vote5	p-value random-majority6	Random Sampling7
Volume	VS	0.71 ± 0.04	0.55 ± 0.06	<0.0001	0.45 ± 0.09	<0.0001	0.49 ±0.08
	AVD [ml]	5.36 ± 0.87	8.85 ± 1.46	<0.0001	7.91 ± 1.74	<0.0001	5.47 ±4.12
Overlap	Dice	0.51 ± 0.04	0.36 ± 0.05	<0.0001	0.45 ± 0.05	<0.0001	0.37 ±0.07
	Precision	0.61 ± 0.07	0.53 ± 0.03	<0.05	0.79 ± 0.04	non-sig	0.90 ±0.06
	Recall	0.60 ± 0.05	0.47 ± 0.03	<0.0001	0.38 ± 0.06	<0.0001	0.24 ±0.06
Distance	HD 95 [mm]	13.60 ± 2.01	17.54 ± 2.01	<0.0001	10.92 ± 1.72	non-sig	18.11 ±7.15
	SDT 5mm	0.71 ± 0.03	0.60 ± 0.05	<0.0001	0.75 ± 0.03	non-sig	0.59 ±0.07

¹VS = Volumetric Similarity,

²AVD = Absolute Volume Difference,

³HD 95 = Hausdorff Distance 95th percentile,

⁴SDT = Surface Dice at Tolerance,

⁵Median ± 95% CI (bootstrapped) compared to Expert A, B, C,

⁶p-values of one-sided Wilcoxon sign rank test,

⁷Median ± 95% CI (bootstrapped) compared to Expert D

The consistency of the model-based segmentation with the random expert sampling across the patient population was significantly better for VS, AVD, and Precision but insignificant elsewhere. However, most non-significant comparisons have a standard deviation that includes variance ratios of 1. For these metrics, no conclusion about the segmentation precision superiority or inferiority can be made and the variance is most likely similar.

The model-based segmentation with the random expert sampling showed a smaller average volume difference and 95% CI when compared to the median expert volume with Bland-Altman plots, while the majority vote tends to underestimate the median expert volume (Figure 4) (mean and 95% CI volume difference of random expert sampling vs. majority vote, 0.8(−25.8–27.5) ml vs. 7.2(−26–42.4) ml).

Figure 4:

Bland-Altman for Random Expert Sampling (blue) and Majority Vote Model Volume (green) estimates compare to Median Expert Volume.

Qualitatively, the probability predictions of random expert sampling correlate better to the fields segmented by the experts than to the predictions for the majority vote which may show overfitting to the single hypodense voxels and image noise (Figure 6).

Figure 6:

Example of a heatmap of expert compared to models trained on random expert sampling and majority vote. Low values are represented by blue to green colors, and high values by yellow to red colors. The prediction of a majority vote underestimates acute ischemic brain tissue shown.

The random expert sampling model indicated similar performance to classifying cases into relevant and not relevant stroke volume vs. inter-expert agreement and the majority vote model, respectively (AUC 0.92±0.02 vs. 0.93±0.02, 0.90±0.02) (Table 3)

Table 3:

Image Classification with 1ml threshold

Categories	Metric	Internal Evaluation			External Evaluation Random Sampling2
		Random Sampling1	Interexpert1	Majority Vote1
Image-level	Sensitivity	0.94 ± 0.02	0.91 ± 0.02	0.65 ± 0.03	0.95 ±0.05
classification	Specificity	0.70 ± 0.03	0.99 ± 0.03	0.97 ± 0.01	0.36 ±0.07
	F-score	0.85 ± 0.02	0.85 ± 0.02	0.77 ± 0.02	0.78 ± 0.05
	AUC	0.92 ± 0.02	0.93 ± 0.02	0.90 ± 0.03	0.74 ±0.09

¹Median ± 95% CI (bootstrapped) compared to Expert A, B, C

²Median ± 95% CI (bootstrapped) compared to Expert D

3.3. Evaluation of NCCT ischemic Core Volume as Imaging Biomarker in Clinical Practice

Cohen’s kappa for volume classification (<1ml, <50ml, <100ml and <100ml) of the random expert sampling model is higher than Fleiss’ kappa for inter-expert agreement and Cohen’s kappa for the majority vote model (0.52 ±0.06 vs. 0.32 ±0.04 and 0.24 ±0.05).

When comparing the initial ischemic core volume to the clinical outcome (mRS after 30 and 90 days), random experts sampling model volume and ASPECTS correlate significantly to the mRS scores after 30 and 90 days and the CTP volume does not (Table 4).

Table 4:

Correlation to the clinical outcome for ASPECTS and volume estimates

Predictor	Rho of mRS 30 days1	p-value	Rho of mRS 90 days1	p-value
ASPECT	−0.18	<0.05	−0.19	<0.05
Ischemic core volume of CTP	0.16	non-sig	0.15	non-sig
Median expert volume	0.17	<0.05	0.16	non-sig
Volume of majority vote model	0.13	non-sig	0.15	non-sig
Volume of random expert sampling model	0.21	<0.01	0.19	<0.05

¹Spearman’s Rho and p-value for correlation

²Alberta Stroke Program Early CT Score

³Computer Tomography Perfusion, Cerebral blood flow (CBF) <30%

Subgroup analysis of patients with full reperfusion (TICI ≥2B, n=51) shows no significant difference when comparing the average volume differences for the random expert sampling volume prediction and CT perfusion volume to the 24h-follow-up DWI volume (Fisher’s z-test p-value = 0.355, Figure 5). Analysis of the whole patient population shows similar results (Supplemental Material, Figure S2).

Figure 5:

Bland-Altman for Random Expert Sampling Model Volume (blue) and CTP Ischemic Core Volume ¡30% (red) compared to 24h DWI-Volume for all reperfusors (TICI≥2B).

4. Discussion

In this study, we found that random expert sampling training of a benchmark deep learning model leads to better agreement with experts than experts among themselves for the segmentation of the ischemic core on NCCT. The model-based ischemic core shows similar volume agreement with the final infarct volume and better correlation to the clinical outcomes as CT perfusion.

The significance of these findings is threefold. First, the model outperforms experts of the highest human expertise when jointly trained by the experts themselves. Second, the model’s volume demonstrates similar clinical predictive value as the ischemic core volume estimates of current rule-based algorithms. Third, we found that contrary to prior deep learning applications, fusion techniques such as a majority vote cause the model to overfit to single voxel values and segment less meaningful clinical information [33, 34, 35, 22].

ECASS I, ESCAPE, DAWN, DEFUSE 3, SELECT-2 and ANGEL-ASPECT are clinical trials that have included the measures of ischemic core on NCCT as inclusion and treatment criteria for patients with acute ischemic stroke [6, 36, 9, 5, 4, 3].

The ECASS I and TWIST trial specified hypodensity consistent with ischemic brain tissue of more than 1/3 of the medial cerebral artery territory as exclusion criteria[37, 38]. Equivalently, ESCAPE and (IMS)-III Trial and retrospective studies suggest Alberta Stroke Program Early CT Score (ASPECTS) of 5, < 6, < 7 or < 8 for estimating a large ischemic infarct and to guide endovascular treatment decisions [36, 39, 38, 40]. However, the benefits of thrombectomy have recently also been demonstrated for patients with ASPECTS 3 − 5 in SELECT-2 and ANGEL-ASPECT [3, 4]. Although ASPECTS is widely used as a predictor for patients to benefit from endovascular treatment using different thresholds, it is limited by inter-rater variability, a modest correlation to ischemic core volumes and location [41, 42, 43, 12].

Besides the ASPECT score, the ASPECT-ANGEL and SELECT-2 trials have also included patients with an ischemic core volume of 70 − 100ml and > 50ml that was quantified with CT perfusion [3, 4].

We proposed a deep learning segmentation method of the ischemic core on NCCT that may alleviate limitations of perfusion imaging and the ASPECTS score.

Different reference standard definitions of the ischemic core exist across various stroke imaging modalities (MR and CT Perfusion, DWI). Sarraj et. al and SELECT investigators have shown that the ischemic core volume on CT perfusion is biased and poorly overlaps with the final infarct volume (median (IOR) absolute volume difference 17.9 (5.6–43.9)ml and Dice score 0.1 (0.0–0.4)) [44].

Diffusion-weighted imaging (DWI) within a short time after the NCCT is used in prior works as ground truth for the segmentation or detection of the ischemic core on NCCT [45, 46, 47, 48]. However, especially in earlier time windows (< 1h) cytotoxic edema is the predominant image abnormality depicted as diffusion restriction on DWI [49]. Vasogenic edema depicted as hypodensity on non-contrast Computed Tomography (NCCT) develops > 1 − 4h after stroke onset and suggests irreversibly damaged brain tissue (ischemic core) [50]. The image correlation of the underlying ischemic core pathology of a DWI lesion on NCCT especially in the earlier time frames seem unclear and voxel-wise comparisons of NCCT and DWI has been scarcely studied [51].

Deep learning medical image segmentation is more often supervised by human experts’ annotations and fusion methods (e.g. majority vote) may approximate an error-free ground truth when collecting annotations from multiple experts. In a segmentation task with reference annotation of uncertainty, small target lesions, or empty segmentations reference annotations require at least highly skilled experts to minimize the errors, and multiple experts’ segmentations are needed to approximate the distribution of interpretations, which is resource-intensive and timely [30]. For acute ischemic stroke segmentation on NCCT, fusion methods for categorical voxel classes may limit the ability of the model to learn segmentation tasks where experts inherently disagree. We found that advanced fusion methods, such as STAPLE or SIMPLE are not applicable and tested majority voting with modest performance [22, 34].

We propose a training requiem for multi-expert training that maximizes the encoded information of the NCCT interpretations from three expert neuroradiologists. We used three experts so that there is a “tie-breaker” between two raters. A larger number of experts is not feasible. The model trained with this combined encoded information (random expert sampling methods) agreed more with experts when compared to the inter-human-reader agreement (Dice score 0.51±0.04 vs. 0.31±0.04 (Table 2)). When comparing the model-based volume and CT Perfusion volume to the final infarct volume we could find no significant difference in final infarct estimation. Neuroimaging is a significant cost driver in acute ischemic stroke patients [52]. In addition, reduced imaging modalities and scan time may increase the net monetary benefit of earlier endovascular treatment ($10,593 for every 10 minutes) [53, 54]. However, additional and prospective studies regarding clinical outcomes are needed before the long-term cost-effectiveness of using NCCT and deep learning tools can be concluded [55].

This study has limitations. First, we only use a patient cohort of one multicenter clinical trial with the primary outcome measure of selecting patients that present within 16h. For the image classification, we do not include stroke mimics or hemorrhagic stroke patients. Further research may test the proposed method in a broader and prospective patient population. Second, we include three experts. The distribution of image interpretation may require a larger set of neuroradiologists which would be resource-intensive but may improve performance. Third, the volume of the stroke is only one imaging feature that correlates with the clinical outcome and may influence decision-making. In future work, deep learning methods may offer voxel-based lesion-symptom mapping for better estimation of clinical outcome prediction.

In clinical practice, our model may identify, quantify, and localize acute ischemic stroke on NCCT comparable to CT perfusion. An accurate and precise deep learning segmentation methodology allows less specialized on-call clinicians to utilize the information for treatment decisions given by NCCT. This possibly enhances the impact of NCCT in endovascular treatment decisions for ischemic strokes as a basic, cheap, and widely available imaging modality.