Use of quantitative angiographic methods with a data-driven model to evaluate reperfusion status (mTICI) during thrombectomy

Abstract

Background:

Intra-procedural assessment of reperfusion during mechanical thrombectomy (MT) for emergent large vessel occlusion (LVO) stroke, is traditionally based on subjective evaluation of digital subtraction angiographs (DSA). However, semi-quantitative diagnostic tools which encode hemodynamic properties in DSAs, such as angiographic parametric imaging (API), exist and may be used for evaluation of reperfusion during MT. The objective of this study was to use data-driven approaches, such as convolutional neural networks (CNN) with API maps to automatically assess reperfusion in neuro-vasculature during MT procedures based on the modified thrombolysis in cerebral infarction (mTICI) scale.

Methods:

DSAs from patients undergoing MTs of anterior circulation LVOs, were collected, temporally cropped to isolate late arterial and capillary phases and quantified using API peak height (PH) maps. PH maps were normalized to reduce injection variability. A CNN was developed, trained, and tested to classify PH maps into 2 outcomes (mTICI0,1,2a / mTICI2b,2c,3) or 3 outcomes (mTICI0,1,2a / mTICI2b / mTICI2c,3) respectively. Ensembled networks were used to combine information from multiple views (anteroposterior and lateral).

Results:

The study included 383 DSAs. For the 2-outcome classification, average accuracy was 81.0% (95% CI: 79.0%−82.9%) and, area under the receiver operating characteristic curve (AUROC) was 0.86 (0.84–0.88). For the 3-outcome classification, average accuracy was 64.0% (62.0–66.0), AUROCs were 0.85 (0.83–0.87), 0.74 (0.71–0.77) and, 0.78 (0.76–0.81) for the mTICI0,1,2a, mTICI2b, and mTICI2c,3 classes respectively.

Conclusions:

This study demonstrated the feasibility of using hemodynamic information in API maps with data-driven models to autonomously assess intra-procedural reperfusion during MT.

INTRODUCTION

Patients with emergent large vessel occlusion (LVO) acute ischemic strokes (AIS) account for 24% – 46% of all AIS cases [1–3]. While there are many confounding factors, timely recanalization of such blockages is essential for optimal patient outcome [4]. Mechanical thrombectomies (MT) aim at endovascular clot retrieval to restore blood flow to ischemic territories [5, 6]. MTs are reported to have successful reperfusion rates between 75 – 80% [7] compared to 30% early recanalization rate when using intravenous thrombolysis with recombinant tissue-type plasminogen activators (thrombus length < 7 mm) [8]. Thus, MTs have markedly reduced severe disability and mortality compared to intravenous thrombolysis [8], and have been established as the standard care for LVO AISs.

Patients with LVOs undergo computed tomography (CT), magnetic resonance imaging (MRI), CT angiogram, CT perfusion and MR angiogram (MRA) imaging to determine eligibility for the MT procedure, followed by the procedure itself. While pre-procedural AIS imaging has undergone major advancements [9], intra-procedural imaging still lags behind. Currently, intra-procedural MT success is assessed primarily by grading intracranial reperfusion using a cerebral digital subtraction angiogram (DSA). This is done using the thrombolysis in cerebral infarction (TICI) scale as proposed by Higashida et al [10], or the modified TICI scale (mTICI) [11]. mTICI grading systems has received criticisms due to confusing internal inconsistencies [12] as well as inclusion of bias due to grading being conducted solely through direct visual estimations of operators [13]. Clinical trials compared operator scored mTICIs with core lab scored mTICIs and found only 56% proportion of agreement between the two. In 33% of these, the operator scored mTICIs were overestimated compared to those from the core lab [14–16].

Intra-procedural assessment of such endovascular procedures could be improved using quantitative tools similar to CTP. However, such implementation is limited due to the 2D nature of DSA and variability caused by hand injection of contrast. Angiographic parametric imaging (API) has been proposed as an alternative solution. This form of image analysis uses a DSA sequence to semi-quantitatively analyze blood flow through vasculature and angioarchitectures. Intensity at each pixel across the DSA sequence is measured, resulting in a time density curve (TDC) at each pixel. TDCs’ parameterization enables extraction of various parameters such as mean transit time, time to peak, time to arrival, peak height (PH) and area under the TDC. This allows API map generation for each parameter which can be analyzed to understand nature of flow through different vessels and phases in DSAs [17, 18]. Each map encodes one hemodynamic parameter derived from each x-ray pathway, thus making maps less sensitive to subtle flow differences. This suboptimal sensitivity could be improved using a hybrid approach where hemodynamics encoded in API are combined with a data-driven model.

Data-driven models such as convolutional neural networks (CNNs) and other machine learning tools using CT or MRI data have been clinically implemented for automated stroke assessment such as ASPECT score [19]. Following similar trends, tools have been proposed for DSA using API to make predictions regarding treatment success [20]. In this work, we present a study to test feasibility of using CNNs with quantitative angiographic information from API to classify cerebral reperfusion during MT procedures. For the data-driven classification, reperfusion level was assessed using the mTICI scale. However, any other outcomes, including post-procedure MRI or neurologic evaluation, could be used.

METHODS

Data Collection

Retrospective collection and analysis of patient data was conducted at a single center and approved by our Institutional Review Board. Inclusion criteria was any patient with an LVO undergoing a MT. For each patient, baseline, intraprocedural and post-MT DSAs were collected. Anteroposterior (AP) and lateral view DSAs were collected for every scan. Patients with posterior circulation occlusions were excluded. DSAs with image artifacts caused by patient motion during the scan, mainly in cases treated under conscious sedation, were also excluded from the study.

We included 192 patients with 383 angiographic runs in our final analysis. Since angiographic runs from same patients were taken at different time points during MT procedures, they have different levels of reperfusion and can be considered as separate cases. Mean patient age was 68.75 years, initial NIH Stroke Score (NIHSS) was 12, post-procedure NIHSS was 4 and NIHSS shift was −7. Patient demographics, locations of LVOs and summary of mTICI scores are displayed in Figure 2.

Figure 2:

Patient demographic information. p refers to number of patients, n refers to number of DSAs. DSA – Digital Subtraction Angiography, ICA – Internal Carotid Artery.

Acquisition of DSA sequences for all patients were conducted using Canon Infinix biplane systems (Canon Medical Systems Corporation, Otawara, Japan). DSAs were acquired at an average tube voltage of 84.3 ± 5.0 kVp (average ± standard deviation), tube current of 149.4 ± 42.7 mA, pulse width of 84.0 ± 12.1 ms and frame rate of 3 frames per second. Contrast used during acquisitions was iohexol (Omnipaque 350; GE Healthcare, Piscataway, New Jersey).

Overall study workflow is displayed in Figure 1. For each DSA included in analysis, an mTICI label was assigned by two experienced operators (qualified neuro-interventionalists) independently of each other. The operators graded every case and used the full DSA sequences. Operators were not involved in the procedure and were blinded to clinical outcomes and PH maps. Grading was performed according to the following 6 categories: no perfusion (grade 0), partial perfusion beyond initial occlusion but not in distal arteries (grade 1), partial perfusion less than 50% (grade 2a), partial perfusion more than 50% but less than full (grade 2b), complete but delayed perfusion (grade 2c) and complete perfusion (grade 3) [21]. Cases with disagreements in mTICI labels were resolved by consensus decision. This was done to remove any bias in labels used for training the network.

Figure 1:

Workflow of the study. DSA – Digital Subtraction Angiography, PH – Peak Height.

API Map Generation

Reperfusion evaluation with mTICI scores was done based on extent of tissue perfusion as represented by the capillary blush in DSAs [10, 22]. Each DSA was cropped to only include frames where contrast was in the late arterial and capillary phases. Thus, API maps contained a limited number of overlapping structures from early arterial or venous phases. Given the three frames per second acquisitions arterial structure were always present in the final API maps. The temporal cropping was done by an operator with 3 years of experience working with DSAs.

TDCs were extracted at each pixel by tracking flow of contrast across frames in the cropped DSA sequence. PH maps were generated by calculating maximum value from TDCs at each pixel. For the purpose of this feasibility study, only PH maps were considered as it reflects maximum contrast intensity in each pixel across all frames and is thus most reflective of perfusion.

Since hand contrast injection was used for these emergent cases, injection parameters such as concentration, volume, and injection rate were highly heterogeneous between DSA acquisitions. To account for this variability, every pixel value in the PH map was divided by the PH value in the main feeding artery. Thus, normalizing each map to contrast concentration in the respective inlet vessel. The location for normalization was manually conducted by an operator with 3 years of experience working with DSAs and API. The location was chosen on a straight portion of the main feeding artery before splitting into respective branches. Care was taken to avoid curves tortuous or overlapping structures that would affect the quantitative angiographic values as well any regions of image artifacts.

Network Development

CNN was developed using Keras [23] to classify PH maps based on reperfusion level. CNN architecture development was an iterative process based on optimization of metrics such as classification accuracy and receiver operating characteristic (ROC) curves. Final architecture is displayed in Figure 1. Optimizer used during training was Adam optimizer with an initial learning rate of 10⁻³. Loss function used was categorical cross-entropy. Keras callbacks were used to reduce the learning rate as training progressed and automatically terminate training as loss plateaued. In addition, class imbalance between classes was accommodated by implementing a balanced class weighting during training. Thus, the CNN balanced layer weights to ensure equal penalization of under or over-represented classes in the training set. CNNs were trained and tested on a single NVIDIA (Nvidia Corporation, Santa Clara CA) Tesla V100 graphics processing unit.

Following guidelines proposed by Radiology [24] and in order to prevent network overfitting, we split the dataset with 70% (268 cases) reserved for training, 10% (39 cases) reserved as a validation set used for hyperparameter tuning during training, and 20% (77 cases) for testing. Hyperparameter tuning during training was done by tracking loss on the validation set. To test network robustness and ensure that results were not based on specific training-testing splits, a 20-fold Monte Carlo cross validation (MCCV) [25] was conducted. This approach involved randomly splitting the total dataset into training and testing sets 20 times.

In order to combine information contained in both AP and lateral PH maps, we used ensemble networks. This method involves combination of predictions from multiple networks to give a final prediction by assigning a weight to predictions from each network. One network was trained on AP PH maps and another network with the same architecture was trained on lateral PH maps. Weights to assign to predictions from individual networks were calculated using a differential evolution optimization algorithm [26] implemented using SciPy [27]. Once the weights were calculated, they were used to combine predictions from AP and lateral networks to give weighted ensembled predictions.

CNNs can automate outcome predictions and quantitative assessment of lesions such as intracranial aneurysms [20]; however as a self-controlling minimization technique, they don’t allow users to oversee which image features are most important and thus how to improve network predictions. We investigated the use of class activation maps (CAM) to visualize regions of PH maps that trigger the trained algorithm, thus lending insight to whether the CNN makes decisions based on flow or using some other portion of the image which may not be as predictive of reperfusion. CAMs were generated using a method described by Zhou et al [28]. They are obtained by taking outputs from the final convolutional layer and passing it through a global average pooling layer. CAMs are heatmaps where high intensity pixels are features highly weighted towards network classification output.

Statistical Analysis

The CNN was evaluated using five quantitative metrics including classification accuracy, ROC curves, area under the ROC curves (AUROC), sensitivity, specificity, and Matthews correlation coefficient (MCC). Each of these metrics were averaged using results over the 20-fold MCCV. MCC is used in machine learning models to evaluate quality of binary classifications [29]. It has proven to be advantageous as it takes in to account class imbalance and uses every factor in the confusion matrix (true positives, false positives, true negatives and false negatives) [30].

Currently, when classifying intra-procedural DSAs as having sufficient or insufficient reperfusion to determine need for further treatment, clinicians either use a 2-outcome grouping where mTICI 0,1,2a is clinically insufficient reperfusion and mTICI 2b,2c,3 is sufficient reperfusion [31], or a 3-outcome grouping where mTICI 0,1,2a is insufficient reperfusion, mTICI 2c,3 is sufficient reperfusion and mTICI 2b is either sufficient or insufficient reperfusion and the need for further treatment is decided based on other factors [32]. Thus, in addition to a 2-outcome classification between mTICI 0,1,2a and mTICI 2b,2c,3, we also investigated a 3-outcome classification between mTICI 0,1,2a, mTICI 2b, and mTICI 2c,3. In addition, sub group analysis was conducted for using AP and lateral view networks independently and, for using both views combined using the ensemble network. Two-tailed McNemar’s p-test values were calculated in order to evaluate significance of any performance differences (p<0.05). McNemar’s p-test values were also calculated between networks using temporally cropped DSAs and networks using un-cropped DSAs in order to test the effect of temporally cropping out arterial and venous phases from the DSAs prior to PH map generation.

RESULTS

Network Performance

The algorithm takes 0.25 seconds to create and normalize each API map, the AP network took 9.2 minutes to train, the lateral network took 9.3 minutes to train, the ensemble weights were calculated in 0.65 seconds and a single case can be classified using the network in 0.6 milliseconds. Average values for each evaluation metric along with their standard deviations and 95% confidence intervals are displayed in Table 1. Peak network performance was achieved when making a 2-outcome classification using an ensembled network that combined classifications from both, AP and lateral, view networks. While better performance was achieved in terms of accuracy, AUROC, MCC and sensitivity for 2-outcome classifications, better specificity was observed for 3-outcome classifications.

Table 1:

CNN performance in classifying DSAs based on their level of reperfusion based on the mTICI scale. Performance is displayed in the form of average accuracies, area under the receiver operating characteristic curves (AUROC), Matthews correlation coefficients (MCC), sensitivities, and specificities along with their standard deviations and 95% confidence intervals (CI). (A) Two-outcome classification (mTICI grade 0,1,2a versus mTICI grade 2b,2c,3) and (B) Three-outcome classification (mTICI grade 0,1,2a versus mTICI grade 2b versus mTICI grade 2c,3). The 3-outcome classification requires a ROC curve for each outcome, thus there is an AUROC for each outcome in (B). The best results are in bold. The results indicate that best performance is achieved when making a 2-outcome classification using an ensembled network.

Metric	(A) 2 Class: mTICI 0,1,2a versus mTICI 2b,2c,3
	Only AP	Only Lateral	AP and Lateral Ensembled
Accuracy (%)	74.2 ± 3.3 (72.8, 75.7)	76.9 ± 5.9 (74.4, 79.5)	81.0 ± 4.5 (79.0, 82.9)
AUROC	0.83 ± 0.04 (0.81, 0.84)	0.84 ± 0.05 (0.82, 0.87)	0.86 ± 0.04 (0.84, 0.88)
MCC	0.49 ± 0.07 (0.46, 0.52)	0.54 ± 0.11 (0.49, 0.59)	0.62 ± 0.09 (0.58, 0.66)
Sensitivity	0.78 ± 0.1 (0.74, 0.83)	0.84 ± 0.1 (0.79, 0.88)	0.88 ± 0.08 (0.85, 0.92)
Specificity	0.69 ± 0.12 (0.64, 0.75)	0.68 ± 0.14 (0.62, 0.75)	0.72 ± 0.11 (0.67, 0.76)
Metric	(B) 3 Class: mTICI 0,1,2a versus mTICI 2b versus mTICI 2c,3
	Only AP	Only Lateral	AP and Lateral Ensembled
Accuracy (%)	56.0 ± 4.9 (53.9, 58.2)	61.8 ± 4.1 (60, 63.6)	64.0 ± 5.0 (62.0, 66.0)
AUROC - mTICI 0,1,2a	0.81 ± 0.04 (0.79, 0.83)	0.84 ± 0.05 (0.82, 0.86)	0.85 ± 0.04 (0.83, 0.87)
AUROC - mTICI 2b	0.67 ± 0.06 (0.64, 0.70)	0.74 ± 0.06 (0.72, 0.77)	0.74 ± 0.06 (0.71, 0.77)
AUROC - mTICI 2c,3	0.74 ± 0.05 (0.71, 0.76)	0.75 ± 0.08 (0.72, 0.79)	0.78 ± 0.06 (0.76, 0.81)
MCC	0.30 ± 0.09 (0.26, 0.34)	0.40 ± 0.07 (0.37, 0.43)	0.43 ± 0.08 (0.40, 0.47)
Sensitivity	0.66 ± 0.16 (0.59, 0.73)	0.74 ± 0.08 (0.70, 0.77)	0.80 ± 0.1 (0.75, 0.84)
Specificity	0.73 ± 0.08 (0.69, 0.76)	0.74 ± 0.08 (0.70, 0.77)	0.76 ± 0.07 (0.73, 0.79)

Performance was also evaluated with ROC curves that are displayed in Figure 3. Highest AUROC values were achieved when using ensembled networks. While in each plot, ROC curves for each sub-group are similar with overlap of standard deviations, McNemar’s p-test values indicate significant advantage towards using lateral view over AP view (p-value<0.05) and towards using ensembled networks over AP or lateral view networks independently (p<0.05) for both, 2-outcome and 3-outcome classifications.

Figure 3:

Receiver Operating Characteristic (ROC) curves generated from the classifications of the Convolutional Neural Network (CNN). (a) depicts the ROC curves obtained for each sub-group when making a 2-class classification, (b) depicts the ROC curves for each sub-group for the mTICI 0,1,2a class when making a 3-class classification, (c) depicts the ROC curves for each sub-group for the mTICI 2b class when making a 3-class classification, and (d) depicts the ROC curves for each sub-group for the mTICI 2c,3 class when making a 3-class classification. The shaded region around the ROC curve depicts the standard deviations at each point. High AUROC values and thin spread of standard deviations in (a) indicates the best performance is achieved when making a 2-class classification. In all 4 sub-plots, while the ROC curves and the standard deviations overlap between the three sub-groups, the McNemar’s p-test values indicate significant improvement in performance when ensembled networks over AP and lateral view networks.

McNemar’s p-test values between temporally cropped and uncropped AP, lateral and ensembled networks were 0.16, 0.23, and 0.05. Since these values are above or equal to the 0.05 threshold, there is no significant advantage towards using temporally cropped DSAs.

Class Activation Maps

CAMs for two different cases were generated to visualize how the CNN makes its classifications, and are displayed in Figure 4. The CAMs were able to display what PH map regions were being used by the CNN to make the classification decision. The contrast in vasculature activated the network, with larger vessels having a higher activation. Thus, the network is looking at the presence of contrast and vasculature to make classification decisions. Internal carotid artery (ICA) terminus, middle cerebral artery (MCA) presence, and MCA territory, seemed to be greatest contributors towards network classification task.

Figure 4:

(A) and (B) are two examples showing anteroposterior (AP) and lateral digital subtraction angiography (DSA) sequences, the normalized peak height (PH) maps generated from those sequences, the class activation maps (CAMs) as well as the classifications from the convolutional neural network (CNN) for each view and the ensembled CNN for both views. In (A), the AP view CNN incorrectly classifies the PH map as being mTICI 2b, 2c or 3 while the lateral view CNN and ensembled CNN both correctly classify the PH map as mTICI 0,1 or 2a. In (B), the lateral view CNN incorrectly classifies the PH map as being mTICI 0,1 or 2a while the AP view CNN and ensembled CNN both correctly classify the PH map as mTICI 2b, 2c or 3. This shows that misclassifications can occur when either AP or lateral views are used independently, however, when information from both views are combined using an ensemble network, the tool is able to correctly classify the DSA into the appropriate group. In each CNN classification table, the green highlight indicates the network classification. All the CAMs show that the activation occurs in the vessels with the larger vessels causing a higher activation.

DISCUSSION

An objective and unbiased assessment of reperfusion status during MT is critical for estimation of clinical prognosis and documentation for research purposes. In this study we investigated technical feasibility of using a CNN with quantitative angiographic information to assess level of cerebral perfusion for patients undergoing a MT to treat an LVO AIS. We successfully classified PH maps generated from DSAs during an MT into 2-outcome categories (mTICI 0,1,2a and mTICI 2b,2c,3) or 3-outcome categories (mTICI 0,1,2a, mTICI 2b and mTICI 2c,3). This indicates that data-driven models such as CNNs can be used to derive hemodynamic information encoded in API and make decisions regarding nature of cerebral blood flow.

Numerical results for five evaluation metrics used are displayed in Table 1, and ROC curves are displayed in Figure 3. Peak performance was observed when making a 2-outcome classification using ensembled networks where information from both AP and lateral view networks were used to provide a final classification. This was observed considering high average values for each metric along with small standard deviations and tight confidence intervals. The numerical results, intersecting ROC curves, and overlapping standard deviations may indicate similar performance between each sub-group, however, McNemar’s t-test p-values (p<0.05) indicate significant advantage to using ensembled networks over AP and lateral networks independently. In addition to commonly used evaluation metrics, we also calculated MCCs. Since MCC is an application of Pearson correlation coefficient [33], it follows the same patterns in terms of inferring correlation strength between classifications and ground truth [34]. MCC values indicate strong positive relationships for 2-outcome classifications and moderate positive relationships for 3-outcome classifications. Network performance on 3-outcome classifications is lower for each sub-group, however it is still within an acceptable range given this is a feasibility study. The lower performance on 3-outcome classifications can be attributed to lower number of cases in each outcome (169 cases: 140 cases: 74 cases) compared to 2-outcome classification (169 cases: 214 cases) and to the increased task complexity of creating a finer classification. Increasing dataset size and increasing number of cases in each specific class will allow us to achieve higher performance on 3-outcome classifications.

In order to understand which PH map features the CNN uses to make decisions, we generated CAMs. Two specific cases, including input PH maps, CAMs and final classification probabilities from the CNN for those cases were analyzed and displayed in Figure 4. In both cases, the network was able to correctly classify the input map into mTICI 0,1,2a or mTICI 2b,2c,3 outcomes. In all 4 maps, we observed image regions that were activated were vessels, with higher activations in larger vessels. Thus, the network is making its decision based on image intensities in the vasculature. Using this method, we are able to interpret the CNNs output and accept or reject the result if the salient features do not match the clinical experience. CAMs could also be used to optimize input data to improve the network performance [35]. For instance, in Figure 4, some extra-cranial regions activation is observed. Since common sense dictates that those regions should have no contribution to classification, cropping the regions should improve the data-driven model performance. Figure 4 also displays advantages of using an ensembled network which combines information from both AP and lateral view networks. The figure shows misclassifications can occur when either AP or lateral view networks are used independently, however, when information from both networks are combined using an ensemble method, the PH map is correctly classified into the appropriate group.

The level of reperfusion was classified based on the mTICI scale which has its drawbacks [12, 13]. In addition, neurointerventionalists can currently perform the classification themselves by visual assessment of the DSAs. However, an automated process trained using labels provided by a core-lab or experts in the field, could provide an objective tool across many institutions and users. This study is useful as it can be replicated for any other outcome scale such post-op MRI or neurological evaluations, however these are not intra-procedurally acquired. Further investigations need to be conducted using other outcome scales to classify reperfusion levels.

There are some limitations to this study. First, we are only using 383 angiograms (268 for training, 39 for in-training hyperparameter tuning or validation, and 77 for testing) that were all collected from the same center. Thus, we are currently limited to demonstrating only a technical feasibility study of using CNNs to assess reperfusion level. Second, we are currently only using PH maps for this feasibility study, other API maps such as mean transit time, time to peak, and area under the TDC can also be derived and may be used in conjunction with PH maps to boost the performance of the network. Third, we are currently doing only a 2- or 3-outcome classification instead of a full range mTICI scale. This is due to low number of cases per outcome (mTICI 0: 82, mTICI 1: 5, mTICI 2a: 82, mTICI 2b: 140, mTICI 2c: 48 and mTICI 3: 26) which leads to a decrease in performance, as seen when going from a 2- to 3-outcome classification. Fourth, we are currently not identifying location of LVOs, rather just the reperfusion status. Fifth, preprocessing methods such as cropping of DSAs to exclude early arterial and venous phases, and identifying inlet vessels for normalization of PH maps are currently not automated. Lastly, the current normalization process only uses one point from the main feeding artery, we will investigate averaging over multiple points from the main feeding artery, this may provide a more effective normalization.

This study proves feasibility of using CNNs to extract encoded hemodynamic information from API by assessing level of reperfusion during an MT in patients with an LVO AIS. While this study provides neurointerventionalists with a more robust tool to evaluate level of reperfusion during MTs rather than relying solely on subjective assessment of DSAs, it also proved feasibility of using CNNs with API maps and can thus be possibly used for other endovascular interventions.

CONCLUSION

This is a novel attempt at using a data-driven approach to classify DSAs based on nature of flow in the neuro-vasculature by extracting hemodynamic information encoded in quantitative angiographic maps. In this study, we proved feasibility of this approach to make decisions regarding reperfusion status of patients undergoing a MT to treat an emergent LVO AIS. The CNN succeeded in making this assessment with an accuracy of 81.0%, AUROC of 0.86 and MCC of 0.62 when making a 2-outcome classification. When making a 3-outcome classification, the network succeeded with an accuracy of 64.0%, AUROC of 0.85 for mTICI 0,1,2a, AUROC of 0.74 for mTICI 2b, AUROC of 0.78 for mTICI 2c,3, and an MCC of 0.43.

Abbreviations

AIS

acute ischemic stroke

AP

anteroposterior

API

angiographic parametric imaging

AUROC

area under the ROC curve

CAM

class activation map

CNN

convolutional neural network

CT

computed tomography

DSA

digital subtraction angiography

ICA

internal carotid artery

LVO

large vessel occlusion

MCA

middle cerebral artery

MCCV

Monte Carlo cross validation

MCC

Matthews correlation coefficient

MT

mechanical thrombectomy

mTICI

modified TICI

NIHSS

NIH stroke score

TDC

time density curve

TICI

thrombolysis in cerebral infarction

PH

peak height

ROC

receiver operating characteristic