Machine Learning Automated Detection of Large Vessel Occlusion From Mobile Stroke Unit Computed Tomography Angiography

Abstract

Background and Purpose:

Pre-hospital automated large vessel occlusion (LVO) detection in Mobile Stroke Units (MSUs) could accelerate identification and treatment of patients with LVO acute ischemic stroke. Here, we evaluate the performance of a machine learning (ML) model on CT angiograms (CTAs) obtained from two MSUs to detect LVO.

Methods:

Patients evaluated at MSUs in Houston and Los Angeles with out-of-hospital CTAs were identified. Anterior circulation LVO was defined as an occlusion of the intracranial internal carotid artery (ICA), middle cerebral artery (MCA M1 or M2) or anterior cerebral artery (ACA ) vessels and determined by an expert human reader. A ML model to detect LVO was trained and tested on independent datasets consisting of in-hospital CTAs and then tested on MSU CTA images. Model performance was determined using area under the receiver-operator curve (ROC) statistics.

Results:

Among 68 patients with out-of-hospital MSU CTAs, 40% had an LVO. The most common occlusion location was the MCA M1 segment (59%), followed by the ICA (30%) and MCA M2 (11%). Median time from last known well to CTA imaging was 88.0 [interquartile range IQR 59.5, 196.0] minutes. After training on 870 in-hospital CTAs, the ML model performed well in identifying LVO in a separate in-hospital dataset of 441 images with area under ROC curve (AUC) of 0.84 [95% confidence interval, CI 0.80–0.87]. ML algorithm analysis time was under 1 minute. The performance of the ML model on the MSU CTA images was comparable with AUC 0.80 [95% CI 0.71–0.89]. There was no significant difference in performance between the Houston and Los Angeles MSU CTA cohorts.

Conclusions:

In this study of patients evaluated on MSUs in two cities, a ML algorithm was able to accurately and rapidly detect LVO using pre-hospital CTA acquisitions.

Introduction:

Automated detection of large vessel occlusion (LVO) acute ischemic stroke in the pre-hospital setting could substantially accelerate treatment times for endovascular stroke therapy. Many Mobile Stroke Units (MSUs) have the capability to perform both non-contrast head computed tomography (NCHCT) as well as computed tomography angiography (CTA). An automated and accurate algorithm to analyze CTA images for LVO could provide crucial decision support to pre-hospital triage decisions, allowing routing of LVO patients directly to thrombectomy-capable centers and faster post-arrival door-to-puncture times.^1–4 However, the performance of such algorithms on MSU images, which can be of variable quality compared to conventional in-hospital acquisitions and are typically performed earlier after onset, remains undetermined.

We previously developed a machine learning (ML) algorithm to identify LVO from CTA images, which demonstrated exceptional performance using in-hospital conventional CTA acquisitions.⁵ The algorithm leverages brain symmetry information to make decisions on outcome variables. In this study, we explore the hypothesis that an ML algorithm trained and validated on in-hospital CTA images will have suitable performance in LVO detection using CTA images acquired in the field from multiple MSUs in two different cities.

Methods:

Data Availability

The data that support the findings of this study are available from the corresponding author upon reasonable request, after clearance by the local ethics committee. Ethics approval was obtained from the local UTHealth Institutional Review Board, and the need for patient consent was waived (HSC-MS-18–0175). A data transfer agreement between the two institutions (UCLA and UTHealth) was obtained prior to study conduct.

Study Population

The patient population in this study was derived from two prospectively collected MSU cohorts. The first cohort consisted of patients treated in the Houston-based MSU. This MSU evaluates patients suspected of acute ischemic stroke within 4.5 hours of last known normal. Patients are evaluated initially with NCHCT, and eligible patients are treated on scene with intravenous tissue plasminogen activator (IV tPA). On-board CTA is obtained immediately after the start of the tPA infusion in patients using a Ceretom 8 slice CT scanner with an OptiStat hand injector system (4 mL/second). All imaging occurred on scene while the ambulance was flat and stationary following strict radiation safety guidelines. Patients evaluated from October 2017 through March 2020 in the Houston MSU were included in this analysis.

Patients were also included in this analysis from the Los Angeles-based MSU. This MSU evaluates patients for any focal neurological symptoms within 24 hours of onset, and the initial imaging study is a NCHCT using a Ceretom 8 slice CT scanner. Patients with suspicion of LVO based on clinical exam also undergo CTA of the brain using an OptiStat hand injector system (4 mL/second). Patients evaluated from October 2017 through April 2020 were included in this analysis.

From these two combined cohorts, patients were included in this analysis if they underwent CTA in the field on the MSU. Patients were excluded if image quality precluded expert reader determination of LVO (see below).

Study Design

DeepSymNet-v2 LVO Detection

DeepSymNet-v2 is a machine learning algorithm based on a 3-dimensional convolutional neural network that leverages symmetry information in the brain hemispheres to efficiently learn imaging patterns predictive of stroke-related outcomes. In our previous works, we described the first iteration of the algorithm, DeepSymNet that was used to detect LVO,⁶ estimate infarct core⁵ and detect hemorrhagic stroke.⁷ In this work, we employ an updated version, DeepSymNet-v2, that includes improvements in the pre-processing steps and network architecture. Briefly, the image registration pipeline was streamlined by including a CTA-specific template, a multi-resolution pipeline and automatic field of view selection of the anterior circulation. Then, network architecture was improved by using a symmetric and non-symmetric path, reduced the number of parameters by using more efficient building blocks, skip connections⁸ and batch normalization.⁹ The entire DeepSymNet-v2 pipeline was run using a single 2.5GHz computing core and a Tesla V100 Nvidia GPU.

DeepSymNet-v2 was trained and tested on an independent dataset of inpatient-acquired CTA scans. The training dataset consisted of 870 CTA (n=419 with LVO and n=451 without LVO) images. All images were acquired from in-hospital Siemens and General Electric scanners from 4 different sites, with the LVO determination was based on radiology reports. The model was then tested on a separate testing dataset of 441(n=121 with LVO and n=320 without LVO) in-hospital CTAs. The performance of DeepSymNet-v2 on this in-hospital dataset was then reported using area under the receiver-operator curve (AUC ROC) statistics. Then, without any further re-training or parameter tuning, the network was then tested on a dataset of CTA images acquired on the two MSUs.

Reporting guideline of clinical artificial intelligence (AI) modelling research is provided using the MI-CLAIM checklist¹⁰ (Supplemental Table I).

Primary Outcome

The primary outcome of the study was the performance of DeepSymNet at predicting anterior circulation LVO. This outcome was measured by the area under the ROC curve statistic. The output of the DeepSymNet-v2 model is a probability of likelihood of LVO, and this probability was then compared against the gold standard of expert reader interpretation of LVO (binary yes/no) and analyzed using ROC analysis.

LVO Determination

The determination of LVO was made by independent review of all the MSU CTA imaging by a single vascular and interventional neurologist with significant experience in neuroimaging reads (SAS). The reader was blinded to the DeepSymNet-v2 LVO predictions. Patients were included if they harbored an occlusion of the anterior circulation, including the internal carotid artery (ICA), middle cerebral artery (MCA M1 or M2) and/or anterior cerebral artery (ACA). Our cohort was limited to anterior circulation occlusions to be consistent with the indications for guideline-based thrombectomy, as well as other LVO-detection algorithms, which are indicated for use only for anterior circulation. Patients were excluded if the imaging quality was insufficient for the expert reader to make a determination of the presence/absence of LVO. Studies were also excluded if image pre-processing for the ML algorithm including registration and bone-stripping could not be performed. A total of 9 studies were excluded for poor contrast bolus or timing and 1 study was excluded due to significant artifact from large metallic implant.

Statistical Analysis

For the description of the characteristics of the study population, percentages are reported for categorical variables, and medians [interquartile range, IQR] for continuous variables. The Fisher’s exact test and Mann-Whitney U test were used to compare categorical and continuous variables between patient groups, respectively. ML model performance was evaluated using area under the ROC statistics. All statistical tests were 2-sided and conventional levels of significance (α = 0.05) were used for interpretation. Stata 14 (StataCorp LLC, College Station, TX) and Prism 7 (GraphPad, La Jolla, CA) software were used for data analysis.

Results:

Among 78 patients with CTAs performed on MSUs, 68 had adequate imaging quality and met inclusion criteria for the study. Among all patients, median age was 73.9 [IQR 62.4, 82.2] years, and 48.7% were female. The most common vascular risk factors were hypertension (74.4%), diabetes (29.5%) and hyperlipidemia (29.5%). Median time from last known well to CTA acquisition was 86.5 [IQR 56.5, 187.8] minutes. Among the 68 patients, 27 (40%) patients harbored LVO. 65% received IV tPA. The most common occlusion location was the MCA M1 (59%), followed by ICA (30%) and MCA M2 (11%). Demographics and clinical characteristics are detailed in Table 1. The only parameter different among the groups was greater National Institutes of Health Stroke Scale (NIHSS) in LVO patients.

Table 1.

Demographics and Clinical Characteristics for Combined Mobile Stroke Unit Sites

Characteristic		All N=68	LVO N=27	Non-LVO N=41	P-value
Age, median [IQR]		73.2 [62.5, 82.1]	79.0 [63.7 83.9]	71.8 [62.3, 79.5]	0.27
Female, N (%)		35 (51.5)	13 (48.1)	22 (53.7)	0.84
Past Medical History, N (%)
	Prior Stroke	20 (29.4)	9 (33.3)	11 (26.8)	0.87
	Atrial Fibrillation	12 (17.6)	5 (18.5)	7 (17.1)	1.00
	Coronary Artery Disease / Myocardial Disease	10 (14.7)	4 (14.8)	6 (14.6)	1.00
	Diabetes Mellitus	21 (30.9)	8 (29.6)	13 (31.7)	1.00
	Hyperlipidemia	19 (27.9)	7 (25.9)	12 (29.3)	0.87
	Hypertension	50 (73.5)	17 (63.0)	33 (80.5)	0.19
	Smoking	9 (13.2)	5 (18.5)	4 (9.8)	0.57
Initial NIHSS, median [IQR]		10.0 [6.5, 17.5]	17.5 [10.0, 23.0]	7.0 [5.0, 12.0]	<0.01
Receipt of IV tPA, N (%)		44 (64.7)	18 (66.7)	26 (63.4)	0.99
Occlusion Location, N (%)					<0.01
	ICA	8 (11.8)	8 (29.6)	0 (0.0)
	MCA M1	16 (23.5)	16 (59.3)	0 (0.0)
	MCA M2	3 (4.4)	3 (11.1)	0 (0.0)
	No LVO	41 (60.3)	0 (0.0)	41 (100.0)
Imaging Acquisition, median [IQR], minutes
	LKW to CTA	88.0 [59.5, 196.0]	88.0 [65.5, 177.5]	88.0 [55.0, 208.0]	0.88

Abbreviations: Large vessel occlusion (LVO), interquartile range (IQR), National Institutes of Health Stroke Scale (NIHSS), intravenous (IV), tissue plasminogen activator (tPA), internal carotid artery (ICA), middle cerebral artery (MCA), last known well (LKW), computed tomography angiography (CTA)

Inpatient CTA Performance

After training on a dataset of 870 in-hospital acquired CTAs as described above, DeepSymNet-v2 was then tested on an independent dataset of 441 in-hospital CTAs. DeepSymNet-v2 achieved an area under ROC curve (AUC) of 0.84 [95% confidence interval, CI 0.80–0.87] for identifying LVO, as shown in Supplemental Figure I.

MSU CTA Performance

Representative non-contrast CT and CTA images of a patient with a left MCA occlusion acquired in the MSU are shown in Figure 1. The performance of DeepSymNet-v2 at identifying LVO in MSU-based CTAs is shown in Figure 2, with area under the ROC curve of 0.80 [95% CI 0.71–0.89]. ML algorithm running time was < 1 minute per imaging study. Sensitivity and specificity results at different DeepSymNet-v2 probability thresholds are given in Supplemental Table II. There was no significant difference between the DeepSymNet-v2 performance in the Houston and Los Angeles-based cohorts (AUC 0.85 [95% CI 0.73–0.94] vs 0.75 [95% CI 0.52–0.89], p=0.44 with DeLong test for unpaired ROC curves) (Supplemental Figure II).

Figure 1: Representative CT images from MSU.

Ceretom 8 slice CT images acquired in an out-of-hospital setting on mobile stroke unit of a patient with left MCA occlusion. Non-contrast head CT (a), and CTA slices (b and c) are shown. Arrow points to location of the MCA M1 occlusion.

Figure 2. DeepSymNet-v2 performance for LVO detection from MSU CTA.

ROC curve with 95% confidence intervals.

Discussion:

In this study of imaging data from two MSUs, an ML algorithm trained on in-hospital CTA acquisitions was able to detect the presence of anterior circulation LVO using MSU-acquired images in an accurate, automated and rapid fashion. The performance was comparable between two different scanners in the two MSUs studied, and comparable to the performance achieved using in-hospital scanners. These findings suggest that automated LVO detection in MSUs can be achieved using ML algorithms trained primarily on in-hospital acquisitions.

LVO detection in the pre-hospital setting could provide a significant reduction in treatment times for these patients. It is well known that the largest period of time lost in onset to treatment time for patients with LVO is the pre-hospital setting, especially when patients are first brought to a non-thrombectomy center and require secondary transfer to a thrombectomy center.¹¹ LVO detection could allow for accurate triage with direct to comprehensive stroke center delivery, potentially direct to angiography suites, and avoid inter-hospital transfer. In addition, prior studies have shown that even patients that are brought to Comprehensive Stroke Centers have shorter hospital arrival to treatment times when an LVO has been identified in an MSU.³ Moving forward, non-CT based diagnostic testing may broaden the generalizability of such approaches.¹²

One of the challenges with MSU image analyses has been image acquisition and quality. As such, the performance of LVO-detection algorithms on MSU images is unknown. In this study, the performance of our DeepSymNet-v2 algorithm is comparable to performances of several commercially available software packages using in-hospital images (Viz.ai AUC 0.86, Rapid AUC 0.85, MeThinks 0.87).^13,14 The performance reported by commercial packages and DeepSymNetv2 have not been tested head-to-head on the same dataset, and thus all direct comparisons should be taken with care. While there remains room for improvement in MSU-CTA LVO detection, our findings here demonstrating that evaluating an algorithm trained purely on an external dataset of in-hospital CTA performed relatively well on MSU images suggest that further improvement of LVO detection software on in-hospitals scanners will also improve MSU detection. This approach, of training on in-hospital CTAs to improve MSU CTA analysis performance, may be beneficial as imaging datasets of MSU CTAs will be substantially smaller than in-hospital CTA datasets, limiting the capacity for ML algorithms to learn off of MSU CTAs alone.

An additional challenge with MSU CTA analyses is that imaging acquisitions typically occur substantially sooner after last known well than with in-hospital imaging. As a result, MSU scans are likely to be evaluating patients with smaller clot burdens in whom thrombus growth in stagnant arterial segments has not yet occurred and may be evaluating patients with better collaterals before collateral failure occurs.^15,16 The results of the current study indicate that the temporal evolution of these physiologic processes is infrequent or relatively slow, so that a ML trained on in-hospital scans performed well on MSU CTAs.

Our study has several limitations. First, our cohort was limited to patient for whom an LVO determination could be made by a human reader. Future study following this present study comparing ML LVO detection to MSU physician read LVO detection will be needed to determine the accuracy of novel approaches in the prehospital setting. In addition, posterior circulation occlusions and smaller, more distal anterior circulation occlusions were not included in this analysis. This study design, however, is consistent with the evaluations of other LVO-detection ML algorithms.

Conclusion:

In this study of patients evaluated on MSUs in two cities, a ML algorithm was able to accurately detect LVO using out-of-hospital CTA acquisitions. Algorithm performance was comparable to the performance when run on in-hospital CTA acquisitions and when compared against other ML packages.

Abbreviations:

LVO

Large vessel occlusion

MSU

Mobile Stroke Unit

ML

Machine learning

CTA

Computed tomography angiogram

ICA

Internal carotid artery

MCA

Middle cerebral artery

ACA

Anterior cerebral artery

ROC

Receiver-operator curve

IQR

Interquartile range

AUC

Area under ROC curve

CI

Confidence interval

NCHCT

Non-contrast head computed tomography

IV tPA

Intravenous tissue plasminogen activator

NIHSS

National Institutes of Health Stroke Scale

LKW

Last known well