Automated Prediction of Proximal Middle Cerebral Artery Occlusions in Noncontrast Brain Computed Tomography

Abstract

BACKGROUND:

Early identification of large vessel occlusion (LVO) in patients with ischemic stroke is crucial for timely interventions. We propose a machine learning–based algorithm (JLK-CTL) that uses handcrafted features from noncontrast computed tomography to predict LVO.

METHODS:

We included patients with ischemic stroke who underwent concurrent noncontrast computed tomography and computed tomography angiography in seven hospitals. Patients from 5 of these hospitals, admitted between May 2011 and March 2015, were randomly divided into training and internal validation (9:1 ratio). Those from the remaining 2 hospitals, admitted between March 2021 and September 2021, were designated for external validation. From each noncontrast computed tomography scan, we extracted differences in volume, tissue density, and Hounsfield unit distribution between bihemispheric regions (striatocapsular, insula, M1–M3, and M4–M6, modified from the Alberta Stroke Program Early Computed Tomography Score). A deep learning algorithm was used to incorporate clot signs as an additional feature. Machine learning models, including ExtraTrees, random forest, extreme gradient boosting, support vector machine, and multilayer perceptron, as well as a deep learning model, were trained and evaluated. Additionally, we assessed the models’ performance after incorporating the National Institutes of Health Stroke Scale scores as an additional feature.

RESULTS:

Among 2919 patients, 83 were excluded. Across the training (n=2463), internal validation (n=275), and external validation (n=95) datasets, the mean ages were 68.5±12.4, 67.6±13.8, and 67.9±13.6 years, respectively. The proportions of men were 57%, 53%, and 59%, with LVO prevalences of 17.0%, 16.4%, and 26.3%, respectively. In the external validation, the ExtraTrees model achieved a robust area under the curve of 0.888 (95% CI, 0.850–0.925), with a sensitivity of 80.1% (95% CI, 72.0–88.1) and a specificity of 88.6% (95% CI, 84.7–92.5). Adding the National Institutes of Health Stroke Scale score to the ExtraTrees model increased sensitivity (from 80.1% to 92.1%) while maintaining specificity.

CONCLUSIONS:

Our algorithm provides reliable predictions of LVO using noncontrast computed tomography. By enabling early LVO identification, our algorithm has the potential to expedite the stroke workflow.

Large vessel occlusion (LVO), defined as the occlusion of large, proximal cerebral arteries, has a substantial clinical impact on ischemic strokes, with LVO accounting for up to 46% of ischemic strokes.¹ A computed tomography or magnetic resonance angiography is commonly used to diagnose LVO. However, these modalities have limitations, including restricted accessibility, procedural delays, and the risk of contrast agent–induced anaphylaxis.² Given the importance of early recognition of LVO on timely interventions and improved functional outcomes, there remains an unmet medical need for an accurate and rapid LVO diagnosis.

A few studies have reported that several features on noncontrast computed tomography (NCCT) are indicators of LVO, such as hyperdense arterial signs,^3,4 sulcal effacement,⁵ and loss of gray-white matter differentiation.⁶ While NCCT scans are highly accessible and offer fast acquisition, achieving an accurate diagnosis of LVO from these scans requires expertise in stroke, which is difficult to find in developing countries or hospitals in excessive demand. Previous studies exploring automatic LVO detection using artificial intelligence encountered limitations such as a lack of robust external validation and interpretability in clinical usage.^3,4,6 To date, there have been a limited number of attempts aimed at extracting handcrafted features that hold clinical relevance to LVO, particularly about ischemic density changes and the presence of clots.

This study used modified 4 Alberta Stroke Program Early Computed Tomography Score regions (striatocapsular, insula, M1–M3, and M4–M6)⁷ as a mean of dividing the brain and extracted features of asymmetry in density, volume, and Hounsfield units (HU) distribution between left and right hemispheres from each region, along with the existence of clot sign (Figure 1). Using the handcrafted features, we developed the machine learning (ML) algorithm (JLK-CTL) to predict LVO in NCCT. This model was externally validated using multicenter data.

Figure 1.

Schematic explanation for JLK-CTL. By aligning a standard template with regional masks (striatocapsular, insula, M1–M3, and M4–M6, modified from Alberta Stroke Program Early Computed Tomography Score) onto noncontrast computed tomography scan, we calculated nonequivalence scores (non-eq score) indicating mean differences in Hounsfield units (HU) between corresponding regions on both sides. Furthermore, relative differences in tissue density, volume, and HU distribution between the right and left regions were calculated. These are indicated as net water uptake (NWU), change in volume (ΔVolume), and SD of HU distribution (Std), respectively. The existence of clot sign was included as an additional feature, encompassing 17 features. Outputs of ExtraTrees are represented as large vascular occlusion (LVO) scores. We offer a bar graph that illustrates factors that contribute to an increase (shown in red) or decrease (indicated in blue) in the LVO score.

METHODS

The data that support the findings of this study are available upon request from the corresponding author. The data are not publicly available due to privacy and ethical restrictions. The study was designed to align with STARD (Standards for Reporting of Diagnostic Accuracy Studies).⁸

Study Participants

For the training and internal validation datasets, we retrospectively collected data of patients with ischemic stroke (admitted between May 2011 and March 2015) across 5 comprehensive stroke centers, using the following criteria: (1) age ≥18 years, (2) admission within 7 days of symptom onset, and (3) concurrent computed tomography angiography (CTA) and NCCT evaluations for LVO assessment. For the external validation dataset, patients with ischemic stroke (admitted between March 2021 and September 2021) were retrospectively collected from 2 comprehensive stroke centers, adhering to the aforementioned criteria. In all cases, patients underwent both NCCT and CTA to evaluate LVO at the discretion of the attending physician. Patients were excluded based on the following criteria: (1) the presence of brain tumors, intracranial hemorrhages, ventriculoperitoneal shunts, or extraventricular drainage and (2) NCCT scans indicating errors in the skull-stripping process (Figure S1). The institutional review boards of all participating centers approved the study and waived informed consent due to the anonymous and retrospective study design.

Clinical Data Collection

Using a standardized protocol,⁹ we collected demographic data, medication history, and details about vascular risk factors. Stroke subtypes were determined by consensus among experienced neurologists at each participating center, using a validated magnetic resonance imaging–based algorithm.¹⁰ Admission National Institutes of Health Stroke Scale (NIHSS) scores were evaluated by certified (http://www.stroke-edu.or.kr/) physicians in each hospital.

Identification of LVO

Vascular status dynamically varies throughout the hyperacute stroke phase¹¹; hence, we used CTA, concurrently performed with NCCT, to determine LVO. We defined LVO as a complete contrast filling defect in the intracranial carotid artery or middle cerebral artery M1/M2 segment. An experienced neurologist diagnosed the presence, side, and location of LVO in all cases. Subsequently, we compared these diagnoses with the stroke registry data, which had been independently verified by attending vascular neurologists at each hospital. If the decision was discordant, a consensus was made. All images were analyzed in a central image laboratory.

Data Preprocessing

Alberta Stroke Program Early Computed Tomography Score Region Segmentation

As preprocessing, skull stripping was performed on individual images using an in-house algorithm. Affine transformation and symmetrical image normalization deformation methods were conducted using ANTS software.¹² The resulting transformation was applied to the Alberta Stroke Program Early Computed Tomography Score mask,⁷ labeled by an experienced neurologist, using nearest-neighbor interpolation.¹² The modified mask consists of 4 regions: insula, M1 to M3, M4 to M6, and internal capsule-caudate-lentiform (striatocapsular) regions.

To minimize confounding effects of outliers such as artifacts, hemorrhage, chronic infarcts, and cerebrospinal fluid spaces, additional preprocessing was conducted. A windowing with a center value of 40 HU and a width of 40 HU was applied, followed by the removal of voxels in each region that fell outside the 10 HU margin from the median value.

Clot Sign Segmentation

Clot signs were detected on NCCT images using a convolutional neural network with a U-Net architecture.¹³ The U-Net model was trained on 520 patients, validated on 60 patients, and externally validated on 51 patients. Details on training and validation processes are in the Supplemental Material (Supplemental Methods; Figure S2).

Feature Extraction

Nonequivalence Score (Mean HU Differences)

A statistical test on the mean HU difference between left and right regions was conducted using a 2 one-sided t test.¹⁴ It analyzes whether the observed mean HU difference is statistically significant considering the predefined equivalence margin of ±1 HU. The margin was defined to detect subtle changes of HU in the acute phase, as HU density linearly decreases −0.4 HU/h in patients with ischemic stroke.¹⁵ From the test, a nonequivalence score ranging from 0 to 1 was derived. Lower scores indicate that the observed difference between the 2 hemispheres’ regions is more likely within the margin.

Net Water Uptake

The relative difference in densities of the left and right parts of each region, represented as net water uptake (NWU), was measured as shown in equation (1).¹⁶

$$NW{U}_i=1-\frac{mean(H{U}_i^L)}{mean(H{U}_i^R)}\mathrm{...}$$ Equation (1)

SD Ratio

The relative difference in the SD of the left region to that of its contralateral regions was calculated by equation (2).

$$ST{D}_i=1-\frac{SD(H{U}_i^L)}{SD(H{U}_i^R)}\mathrm{...}$$ Equation (2)

Volumetric Ratio

The relative difference in volume of the left and right parts of each region was calculated through equation (3). This score was included to represent the change in volume of the ischemic brain due to swelling.

$$\Delta V_i=1-\frac{Volume(i^L)}{Volume(i^R)}\mathrm{...}$$ Equation (3)

Presence of Clot Sign

The existence of the clot sign was determined by using the automated clot sign segmentation model. Binary scores of 0 or 1 were appended as features, depending on the predicted clot mask.

Thus, by extracting 4 features from each of the 4 regions and including the presence of a clot sign, 17 features were identified in each NCCT scan.

Model Training and Evaluation

Among the study population, patients from 5 hospitals were randomly split in a 9:1 ratio into training and internal validation datasets. Patients from the remaining 2 hospitals were selected as external validation datasets.

We trained 5 ML classification models, including support vector machine,¹⁷ ExtraTrees,¹⁸ random forest,¹⁹ extreme gradient boosting,²⁰ and multilayer perceptron,²¹ and EfficientNetV2S, a deep learning (DL) model.²² Each model was trained and validated on the same dataset. Subsequently, receiver operating characteristic curves were generated for each model, and the area under the curve (AUC) was computed to quantitatively assess model performance. Also, we assessed the model’s performance depending on onset to image, NIHSS scores, and the presence of atrial fibrillation (AF).

Optimal threshold values of each model maximizing specificity and sensitivity were determined from the internal validation dataset without bootstrapping. Furthermore, the discriminative power of NIHSS scores on LVO prediction was analyzed by incorporating NIHSS scores into the best model as a continuous variable (JLK-CTL+). Details on model training and evaluation were provided in the Supplemental Material, Table S1, and Figure S3.

Feature Importance

To interpret the classification results of ML models, mean decrease in impurity²³ and permutation importance²⁴ were calculated to explain the contribution of each feature to predictions of the model. For the interpretation of an individual prediction case, we implemented Shapley additive explanations.²⁵ Mean decrease in impurity and permutation importance were used to understand the overall importance of features with respect to all cases in datasets, whereas Shapley additive explanations were used to explicitly explain the contribution of features on a single case prediction. The details of feature importance are presented in the Supplemental Methods.

Statistical Analysis

Data were presented as mean±SD, median (interquartile range), or number (percentage). For the comparison of baseline characteristics between training, internal, and external validation datasets, we used ANOVA or Kruskal-Wallis test for continuous variables and the χ² test or Fisher exact test for categorical variables as appropriate. To compare AUC among models, we used the DeLong test.²⁶ All statistical analyses were performed using STATA, version 16.0 (StataCorp, College Station, TX), and P<0.05 was considered statistically significant.

RESULTS

Baseline Characteristics

After excluding 83 patients following exclusion criteria (Figure S1) from 2919 collected patients, 2833 patients with ischemic stroke from seven hospitals were divided into training (n=2463) and internal (n=275) and external validations (n=95). The mean ages of subjects in the training, internal, and external validation datasets were 68.5±12.4, 67.6±13.8, and 67.9±13.6 years, respectively (Table 1). Distributions of sex and vascular risk factors were comparable between datasets. LVO was more frequent in the external validation dataset than in other datasets (17.0%, 16.4%, and 26.0% in training, internal, and external validation datasets, respectively; P=0.045). The slice thickness of NCCT scans ranged from 3.0 to 5.0 mm.

Table 1.

Baseline Characteristics of Datasets

Performance Comparison of Models in Predicting LVO

ML models achieved an AUC of 0.825–0.865 in the internal validation and 0.864–0.896 in the external validation without significant differences (Figure 2A and 2B). The DL model exhibited a superior AUC of 0.951 in the internal validation dataset compared with the ExtraTrees model (P=0.026). However, in the external validation dataset, its performance was significantly compromised, showing worse AUC compared with ML models (all P<0.001). Of note, a marked disparity in the specificity of the DL model was observed between the internal and external validation datasets (Table 2).

Figure 2.

Receiver operating characteristic (ROC) curve of machine learning models and a deep learning (DL) model. A and B, Comparisons of the ROC curves of the DL model with those of machine learning models in the internal and external validation datasets, respectively. C and D, ROC curves of National Institutes of Health Stroke Scale (NIHSS) model, imaging-only model, and imaging plus NIHSS model in internal and external validation datasets, respectively. ExT indicates ExtraTrees; MLP, multilayer perceptron; RF, random forest; SVM, support vector machine; and XGB, extreme gradient boosting.

Table 2.

Performance Metrics of Models in Internal and External Validation Datasets

ML models exhibited similar performances. However, in the internal validation dataset, the ExtraTrees model demonstrated the highest AUC compared with other models; in the external validation dataset, only the ExtraTrees model exhibited a sensitivity >80% with a specificity of 89%. Also, consistency of the performance of the ExtraTrees model between the internal and external validation datasets regardless of the selected threshold was observed (Figure S4). Thus, we selected the ExtraTrees model as our best performing model for further analysis (JLK-CTL). The whole result of JLK-CTL in the external validation dataset is presented in Figure S5.

Better Performance of Imaging Feature Compared With NIHSS Score

The best NIHSS score cutoff value for LVO prediction was 11. Compared with the model solely using NIHSS score (AUC, 0.760 and 0.711 in the internal and external validation datasets, respectively), both models that used imaging (JLK-CTL) or imaging with NIHSS score (JLK-CTL+) showed better performance in predicting LVO (both P<0.001; Figure 2C and 2D; Table 2). Also, on a predefined threshold, incorporating NIHSS scores increased the sensitivity of the model compared with the model using imaging features only. To be specific, to achieve the same sensitivity of JLK-CTL+, the model only using imaging features largely sacrificed specificity (Table 2). This indicates that embedding NIHSS scores increased sensitivity without significantly sacrificing specificity.

Subgroup Analysis: Onset to Image (≤6 Versus >6 Hours), NIHSS Score (≤11 Versus >11), AF (With AF Versus Without AF), and Exhibition of Arterial Stenosis

When subjects were divided into 2 groups by onset to image of 6 hours, AUCs were comparable between 2 groups in internal and external validation datasets (P=0.86 and P=0.07, respectively; Figure 3A and 3B). In internal and external datasets, no significant differences in specificity and sensitivity were observed between 2 groups (Table 3). Adding NIHSS scores to the model increased AUC and sensitivity without sacrificing specificity (Figure S6A and S6B). Stratification by onset to image of 4.5, 4, 3, and 2 hours showed similar results (Figure S7; Tables S2 and S3).

Figure 3.

Subgroup analysis stratified by onset to image (≤6 vs >6 hours) and National Institutes of Health Stroke Scale score (NIHSS score ≤11 vs >11) using JLK-CTL. A and B, Internal and external validations after stratified by onset to image. Red and green lines indicate the receiver operating characteristic curve of the onset-to-image ≤6-hour group and >6-hour group, respectively. C and D, Internal and external validations after stratified by NIHSS score. Red and green lines indicate the receiver operating characteristic curve of the NIHSS score ≤11 group and >11 group, respectively.

Table 3.

Subgroup Analysis of JLK-CTL and JLK-CTL+ on Onset to Image (≤6 Versus >6 Hours) and NIHSS Score (≤11 Versus >11)

After stratified by NIHSS score of 11, no difference in AUC was observed between 2 groups in the internal and external validation datasets (P=0.21 and P=0.08, respectively; Figures 3C and 3D; Table 3). In addition, AUCs were similar between groups with and without AF groups in the internal and external validation datasets (P=0.49 and P=0.40, respectively; Figure S8). The sensitivity tended to be higher in AF-positive subgroups (Table S4).

In the external validation dataset, 6 subjects had relevant stenosis without LVO. LVO scores were not different between subjects with and without relevant stenosis; however, results for those with LVO were markedly different compared with subjects with relevant stenosis but without LVO (Figure S9).

Feature Importance Analysis

Analysis of both mean decrease in impurity and permutation importance metrics revealed that the 3 most influential features in predicting LVO were the nonequivalence scores in the striatocapsular region, the nonequivalence scores in the insula region, and the existence of clot sign (Figure S10). Although not as important as these 3 features, nonequivalence scores in M1 to M3 and M4 to M6 regions and NWU in striatocapsular and insula regions were also identified as important factors in the model’s prediction.

DISCUSSION

In this study, we developed an ML algorithm to predict LVO in NCCT using handcrafted imaging features including mean HU difference (nonequivalence scores), NWU, SD of HU, volumetric ratio, and presence of clot sign. Our model showed robust performance in detecting LVO in both internal and external validation datasets. In addition, compared with the DL model, our model showed better results in the external validation dataset. Adding NIHSS scores to the model improved predictive performance. Feature importance analysis indicated nonequivalence scores in the striatocapsular and insular regions, along with the presence of a clot sign, as crucial features for predicting LVO in NCCT.

Despite promising results of DL models in a variety of diseases,²⁷ they have an inherent overfitting problem due to their high complexity, which reduces model performance on unseen data.²⁸ As indicated by our results, ML algorithms, which have less complex architectures, have a more robust performance when applied to unseen data compared with the DL model. Furthermore, our model allowed the visualization of the tree nodes and important features in the individual case (Figure S11), which is difficult in a DL model.²⁹ Such visualization provided insight into how the classifier used the features during LVO prediction and, thus, may increase the credibility of results.

A previous study³ demonstrated high sensitivity (83%) and specificity (71%) of LVO prediction in NCCT from 1453 patients with stroke in whom 57% had an LVO on CTA. However, unlike our approach, the algorithm in the previous study used outputs of a DL-based model as features, which is prone to overfitting and hard to interpret^30,31; the algorithm was not evaluated on completely new datasets (external validation dataset in our study).

Intriguingly, our model demonstrated similar performance in patients with onset to image ≤4.5 and >4.5 hours. In the external validation dataset, patients with onset to image ≤4.5 hours had higher NIHSS scores compared with those with onset to image >4.5 hours (median, 9 versus 5; P=0.01) although having a comparable frequency of LVO (29% versus 25%; P=0.83). This implies that patients with rapidly evolving stroke are more likely to suffer from more severe symptoms, seek medical attention earlier, and exhibit prominent features associated with ischemic stroke on NCCT, even if they arrive within the first 4.5 hours.

In the external validation dataset, LVO scores were comparable between patients without either LVO or stenosis and patients with stenosis. This finding may be attributed to the small number of patients with stenosis in our study and the variability in infarct patterns associated with stenosis. To explore the association between predicted LVO scores and arterial stenosis more comprehensively, further research involving a broader range of stenosis degrees is warranted.

Feature importance analysis revealed that nonequivalence scores across all regions, clot signs, and NWU in the striatocapsular and insular regions are significant attributes. These results align with our understanding of LVO. The importance of density changes (nonequivalence scores and NWU) in insular and striatocapsular regions reflects the loss of gray-white matter differentiation, occurring at the margins of the insula and lentiform nucleus, respectively.⁷ This raises concerns about performance in cases involving middle cerebral artery bifurcation or M2-middle cerebral artery occlusion. However, in such cases, we observed that HU differences in the insular and M1 to M3 regions emerged as important features for predicting LVO (Figure S12). Additionally, the model’s reliance on the presence of a clot corresponds with the previously established significance of clot signs in detecting LVO.³² Moreover, recent studies have associated NWU with functional outcome,³³ hemorrhagic transformation,³⁴ and early neurological deterioration¹⁶ following ischemic stroke. Although NWU was not as significant as HU differences and clot signs in our study, our findings are consistent with these recent studies.

In the emergency room, NCCT scans are routinely performed on patients presenting with neurological symptoms. Approximately 20% of ischemic stroke cases were initially misdiagnosed on arrival at the emergency room, which leads to a delay in the proper assessment and treatment.³⁵ Furthermore, while endovascular treatment is applicable to 10% of patients with acute ischemic stroke and arriving at the hospital within 24 hours,³⁶ the actual number of patients receiving endovascular treatment is much lower. This discrepancy may be partially attributed to the underdiagnosis of LVO in the emergency department. A recent study³⁷ showed that using an automated platform, compared with radiology residents, halves the time required to process imaging in patients suspected of having LVO. From this perspective, our model has the potential to enhance LVO detection rates and accelerate the workflow for patients eligible for endovascular treatment, thereby improving stroke outcomes.

Our study has strengths, such as large training data labeled through concurrent CTA and available external validation. Nonetheless, a few limitations should be acknowledged. First, our model was trained and validated on the Korean population of patients with stroke. To validate our algorithm in other ethnicities, additional research is required despite our ML model utilized comprehensive imaging features. Second, additional clinical data may improve the performance of the model. However, as information requirements increase, clinical utility decreases. Third, our model showed low positive predictive value, indicating high false positive rates. The best threshold for streamlining the workflow while minimizing unnecessary further examination should be explored in future studies. Additionally, a clot sign prediction algorithm trained on NCCT with a slice thickness ranging from 3 to 5 mm struggles to identify tiny clots. Nevertheless, the utilization of thinner slice thickness may be less feasible in routine clinical settings. Moreover, the small sample size prevents us from adequately evaluating the algorithm’s performance in distinguishing LVO from stenosis. Most importantly, as 96% of patients in datasets had M1 occlusion, our study did not reflect the typical distribution of occluded vessels in patients with acute stroke due to LVO. This would limit the generalizability of our model in real clinical settings. In future studies, analysis of the model based on locations of occlusions is required.

In conclusion, we developed and validated an automated algorithm predicting LVO using NCCT. As the time window for endovascular treatment of LVO strokes is anticipated to increase over time, automated detection of LVO using NCCT may enhance stroke workflow and functional outcomes after LVO strokes.

ARTICLE INFORMATION

Acknowledgments

The authors appreciate the contributions of all members of the Comprehensive Registry Collaboration for Stroke in Korea to this study.

Sources of Funding

This study was supported by the Multiministry Grant for Medical Device Development (grant KMDF_PR_20200901_0098).

Disclosures

P.E. Kim, D. Kim, and Dr Ryu are employees of JLK Inc. The other authors report no conflicts.

Supplemental Material

Supplemental Methods

Tables S1–S4

Figures S1–S12

References 38, 39