Evaluation of Artificial Intelligence–Powered Identification of Large-Vessel Occlusions in a Comprehensive Stroke Center

The authors' evaluation of Viz LVO shows that the system has the potential for early identification of patients with stroke with large-vessel occlusions, hopefully improving future management and stroke care.

Abstract

BACKGROUND AND PURPOSE:

Artificial intelligence algorithms have the potential to become an important diagnostic tool to optimize stroke workflow. Viz LVO is a medical product leveraging a convolutional neural network designed to detect large-vessel occlusions on CTA scans and notify the treatment team within minutes via a dedicated mobile application. We aimed to evaluate the detection accuracy of the Viz LVO in real clinical practice at a comprehensive stroke center.

MATERIALS AND METHODS:

Viz LVO was installed for this study in a comprehensive stroke center. All consecutive head and neck CTAs performed from January 2018 to March 2019 were scanned by the algorithm for detection of large-vessel occlusions. The system results were compared with the formal reports of senior neuroradiologists used as ground truth for the presence of a large-vessel occlusion.

RESULTS:

A total of 1167 CTAs were included in the study. Of these, 404 were stroke protocols. Seventy-five (6.4%) patients had a large-vessel occlusion as ground truth; 61 were detected by the system. Sensitivity was 0.81, negative predictive value was 0.99, and accuracy was 0.94. In the stroke protocol subgroup, 72 (17.8%) of 404 patients had a large-vessel occlusion, with 59 identified by the system, showing a sensitivity of 0.82, negative predictive value of 0.96, and accuracy of 0.89.

CONCLUSIONS:

Our experience evaluating Viz LVO shows that the system has the potential for early identification of patients with stroke with large-vessel occlusions, hopefully improving future management and stroke care.

Acute ischemic stroke caused by large-vessel occlusion (LVO) contributes disproportionately to stroke-related disability and death.^1,2 It requires emergent detection and treatment ideally by an endovascular approach. Management has changed dramatically during the past few years, most notably due to the numerous clinical trials published in 2015 that indicated that endovascular treatment is superior to tPA alone in the treatment of LVO acute ischemic stroke.^3,4 One of the major contributors to this revolutionary result was the proper selection of eligible patients.⁵ As opposed to earlier trials,^6,7 patients in recent studies were selected primarily by CTA scans. These trials demonstrated the efficacy of mechanical thrombectomy in patients with a limited ischemic core in the setting of moderate-to-severe clinical deficits, which designated such patients as ideal candidates for revascularization therapy. The window for treatment was further extended at the beginning of 2018 to 24 hours,⁸ following 2 trials that demonstrated the efficacy of endovascular treatment for selected patients in timeframes of 6–16 hours⁹ and 6–24 hours.¹⁰ The immediate consequence was an increase in the number of patients eligible for transfer from primary and secondary hospitals to comprehensive stroke centers for endovascular treatment. Thus, fast and accurate recognition of pathology on CT scans has become crucial.

Artificial intelligence algorithms, particularly deep learning, have demonstrated remarkable progress in image-recognition tasks. Methods ranging from convolutional neural networks to variational autoencoders have found great application in the medical image-analysis field, pushing it forward at a rapid pace. Deep learning has the potential to revolutionize entire industries, and given the centrality of neuroimaging in the diagnosis and treatment of neurologic disease, deep learning will likely affect neuroradiologists most profoundly.^11,12

Viz LVO (Viz.ai) is a medical product leveraging a convolutional neural network designed to detect LVOs on CTA scans and notify a neurointerventional specialist within minutes via a dedicated mobile application.

Our aim was to evaluate the detection accuracy of the Viz LVO in real clinical practice at a comprehensive stroke and trauma center.

MATERIALS AND METHODS

A retrospective study was conducted. Viz LVO was installed at the Rambam Health Care Campus in January 2018 for this study. All CTA scans obtained from January 2018 to March 2019 were scanned by the system, including nonacute ischemic stroke cases. The scans were analyzed by the Viz LVO Algorithm, Version 4.1.3, a convolutional neural network using deep learning to detect occlusions from the ICA terminus (ICA-T) to the Sylvian fissure. Analysis of this area would include all occlusions of the M1 segment of the MCA and possibly proximal M2 segment occlusions. Posterior circulation arteries are not assessed by the system.

The results of the system were compared with the formal CTA reading documented in the patients’ files. Each CTA reading was performed by a single reader. The readers were 4 senior neuroradiologists, with 7–25 years of experience. A separate designated pool of 15 examinations was used for evaluating interrater and intrarater reliability among 4 raters. No variation was found between the results given for each CTA examination (intraclass correlation coefficient [ICC] > 0.99).

LVO was considered as either an ICA-T or MCA-M1 occlusion. A second analysis included M2, which was further divided into proximal and distal occlusions using the curve into the Sylvian fissure as an anatomic landmark (Fig 1).

FIG 1.

Division of the M2 segment of the MCA into proximal and distal segments at the curve of the artery into the Sylvian fissure (marked bilaterally by the dashed lines).

Other major pathologies reported in the formal neuroradiologist read were also documented, including cerebral hemorrhage, tumors, and intracranial arterial stenosis. Arterial stenosis was defined as a decrease of more than that in the arterial cross-sectional area calculated by the NASCET formula for ICA or MCA reported in the formal CTA read.

Examinations with metal artifacts (n = 7) as well as those with severe motion or incomplete skull scanning (n = 6) were excluded from the analysis a priori because they are automatically not analyzed by the algorithm. Such examinations are transferred to the server and the mobile application by the system, marked as technically inadequate and classified as negative for LVO. This process is further explained in the Algorithm Description segment and illustrated in Figs 2 and 3.

FIG 2.

Alerts as they appear on the user end of the mobile application, showing the overview screen of examinations with (A) and without (B) a suspected LVO. An overview screen of failed processing is shown in C, in this case, due to metallic artifacts.

FIG 3.

Flow diagram delineating the various steps of the algorithm. App indicates mobile application.

Algorithm Description

The LVO-detection algorithm involves several steps. First, applicable CTA series are identified by inspecting the DICOM metadata. Once an applicable series is identified, the next step is to verify the existence of contrast. The soft matter is extracted by creating a mask of all bone voxels, based on Hounsfield unit thresholding, dilation, and connected component analysis, and removing the bone mask and all voxels external to it. Once the soft matter is extracted, it is inspected for the existence of contrast by counting the total number of voxels with Hounsfield unit values consistent with iodine contrast (100–800 HU). If no contrast is identified, the scan is flagged as a suspected missed bolus and no further processing is conducted.

In the selected examinations, 3D registration of the brain is performed followed by cropping of a 3D cuboid, with dimensions determined so that the ICA-T, M1, and M2 regions are contained within the cuboid. The cuboid is inspected for the presence of metal by looking for voxels with Hounsfield unit values of >3000. If such voxels are identified, the scan is flagged as suspected of containing metallic artifacts and no further processing is conducted. Scans that were not processed due to bad bolus timing or metal artifacts are still available for viewing but are marked by a red frame to notify the user that the algorithm rejected the series. Examples are given in Fig 2, and an illustration of the process is provided in Fig 3.

The 3D cuboid is fed through a 3D segmentation convolutional neural network inspired by the U-Net architecture.¹³ The output of the network is a 3D cuboid of the same dimension as the input, whereby each voxel is assigned a number between 0 and 1 by the network, describing the probability (as estimated by the network) that this voxel is part of the ICA-T or M1 segments. The network was trained on hundreds of manual segmentations of the ICA-T and M1 regions.

Next, the lengths of the left and right segmentations are compared. This step is to identify cases in which due to an ICA occlusion and no retrograde filling, the ICA-T and M1 segments are not visible in the scan. If one of the sides is significantly shorter than the other, an LVO is detected and the system triggers an alert.

If, however, sizable segmentations are available on both sides, these segmentations are extended using another segmentation convolutional neural network of similar architecture that was trained to segment all vessels (not just ICA-T and M1 vessels). The combination of the outputs of both networks is refined to generate the MCA vessel tree. Following this step, end points of the MCA vessels are identified. If the total distance between the ICA-T and the end point is below a predefined threshold, an LVO is detected and the system triggers an alert. The threshold was determined on the basis of the receiver operating characteristic curve to yield approximately equal sensitivity and specificity on the suspected-stroke population and corresponds, roughly, to the beginning of the Sylvian fissure. The process is visualized in Fig 4.

FIG 4.

Overview of the algorithm steps. A, Identification of an applicable scan based on metadata. B, Cropping the head region. Registration (C) and segmentation (D) of ICA-T/M1 regions. E, Additional segmentation of all vessels. Refinement of the segmentations to include only the MCA branches (F) and detection of suspected LVO based on vessel length (G).

If no end point on either side is shorter than the threshold, the algorithm looks for partial occlusions. This is done by computing the centerline of the segmentation and inspecting the average Hounsfield unit value in the vicinity of the centerline. The algorithm is looking for a pattern of a drop in Hounsfield units, followed by an increase (Fig 5). If such a case is identified, an LVO is detected and the system triggers an alert.

FIG 5.

Algorithm processing of a partial occlusion. The cropped scan on the left visualizes a left M1 partial occlusion. The segmentation (on the right) extends through the partial occlusion. However, the average Hounsfield unit value decreases and then increases and a notification is triggered, even though the length of the segmentation exceeds the threshold.

Examples of system identification of both partial and complete occlusions and the matching images sent to the end user by the application during an alert are provided in Fig 6.

FIG 6.

System identification illustration demonstrates stenosis of the M1 segment of the left MCA (A), occlusion of the M1 segment of the left MCA (C), and occlusion of the proximal M2 segment of the right MCA (E), as they appear as preliminary convolutional neural network outcomes (green boxes represent original annotations by the Viz LVO system during identification). The images on the lower row (B, D, and F, respectively) match processed images sent by the system via the application and received by the viewer during an alert.

Statistical Analysis

Statistical analyses were performed using descriptive data analysis, including ranges, means, medians, SDs, and interquartile ranges for continuous variables and frequencies and percentages for categoric variables.

Interrater reliability between system results and the formal read was quantified using an ICC model, namely 2-way random effects, absolute agreement, and single measurement. This model was selected because all ratings were performed by a different set of raters,¹⁴ a scenario that would be expected in routine clinical settings. Thus, this model can be considered a realistic estimate of reliability for this scenario. The interrater ICCs were calculated between the model predictions and senior radiologist reports.

Measures of system performance were examined using sensitivity, specificity, positive predictive value (PPV), negative predictive value, and total accuracy.

In addition, logistic regression models were performed to predict the effect of each factor category— age, sex, and identification of LVO by the Viz LVO system—on LVO detection. ORs and 95% CIs were estimated for each predictor.

To test the additive value of each factor, we entered the variables into receiver operating characteristic (area under the curve) curves one at a time: patient characteristics (age, sex) followed by Viz LVO results. When a logistic regression is fit, receiver operating characteristic curves are routinely used to summarize the model fit and to determine the best cutoff value for predicting whether a new observation is a failure (0) or a success (1).

The receiver operating characteristic curve is the sensitivity or recall as a function of fall-out. Overall, if the probability distributions for both detection and false-positives are known, the curve can be generated by plotting the cumulative distribution function (area under the probability distribution from infinity to the discrimination threshold) of the detection probability in the y-axis versus the cumulative distribution function of the false-positive probability on the x-axis. Ideal prediction produces an area under the curve of 1.00; area under the curve values of 0.70 and higher would be considered strong effects.¹⁵

The level of significance for all statistical analyses was 5%. We analyzed the data using the SPSS, Version 25.0 (IBM). This study was approved by the local Helsinki committee at Rambam Health Care Campus (IRB 0417–17).

RESULTS

A total of 1180 CTAs were scanned by the system and sent to the server and the mobile application during the study period. Thirteen cases had been flagged by the system as technically inadequate and were excluded a priori because they were not analyzed by the algorithm. Of the 1167 cases included in the study, 404 were stroke protocols, with others performed due to trauma, suspected stenosis, and other miscellaneous reasons (Table 1).

Table 1:

Descriptive statistics of the study sample^a

Patients (n = 1167)
Age (mean) [SD]	62.2	19.6
Male	689	59
Stenosis (50%>)	66	5.7
Extracranial ICA	43	3.7
Intracranial	23	2.0
Stroke protocol	404	34.6
Hemorrhage	80	6.8
Tumor	12	1.0
LVO	75	6.4
LVO location (n = 75)
Carotid terminus	28	37.3
M1	47	62.6
Distal occlusion (non-LVO) (n = 44)
Proximal M2	21	47.7
Distal M2–3	23	52.3

^aData are number and percentage unless otherwise indicated.

The interrater ICC for all cases was 0.83 (95% CI, 0.725–0.867). For stroke protocol only, the ICC was higher (0.86; 95% CI, 0.837–0.892). Of 1167 patients, 75 had an LVO as per a senior neuroradiologist’s formal read, representing 6.4% of the cases. Sixty-one of these cases were detected by the system, leaving 14 cases of false-negative results.

The system alerted a possible LVO in 117 examinations, 56 of which did not show occlusion of the ICA-T or MCA M1, defined in our study as an LVO. Nevertheless, in 12 of these false-positive cases an occlusion of a more distal part of the MCA (M2 or M3) was detected. Additionally, 25 more of the false-positive alerts had different major pathologies, such as hemorrhage, tumors, or intracranial stenosis, defined as a decrease of <50% in the arterial cross-sectional area (Table 2).

Table 2:

Pathologies detected in false-positive cases

Pathology	No.	%
Stenosis (>50%)	9	16.1
Distal occlusions	12	21.4
Proximal M2	8	14.3
Distal M2/M3	4	7.14
Hemorrhage	12	21.4
Tumor	4	7.14
No revealed pathology	19	33.9
Overall	56	100

Measures of system performance for the entire group were a sensitivity of 0.81 (95% CI, 0.74–0.91), negative predictive value of 0.99 (95% CI, 0.98–0.99), PPV of 0.65 (95% CI, 0.55–0.74), and accuracy of 0.94 (95% CI, 0.92–0.96).

Logistic regression analysis adjusted for age and sex showed that Viz LVO strongly predicts LVO (OR = 51.75; 95% CI, 28.84–92.84) (Table 3). Further receiver operating characteristic analysis demonstrated an area under the curve of 0.91 (Fig 7).

Table 3:

Prediction of LVO by the Viz LVO system—logistic regression (adjusted for age and sex)

Variable	OR	SE	Sig	95% CI
				Lower	Upper
Suspected LVO	51.75	0.298	.000	28.84	92.84
Age	1.030	0.009	.001	1.013	1.048
Sex	1.474	0.295	.188	0.828	2.626

Note:—SE indicates standard error; Sig, significance.

FIG 7.

Prediction of LVO logistic regression (adjusted for age and sex). The area under the curve is shown to be 0.91. ROC indicates receiver operating characteristic.

In the stroke protocol subgroup, 72 (17.8%) of 404 patients had an LVO acute ischemic stroke. Of the 72 cases, 59 LVOs were identified by the system. Thirteen false-negative cases were encountered. Sensitivity was 0.82 (95% CI, 0.71–0.89); PPV, 0.64 (95% CI, 0.53–0.73); negative predictive value, 0.96 (95% CI, 0.93–0.98); and accuracy, 0.89 (95% CI, 0.86–0.94). Measures of system performance are summarized in Table 4.

Table 4:

Prediction of LVO by the Viz LVO system

System LVO Detection	Sensitivity	95% CI	Specificity	95% CI	NPV	95% CI	PPV	95% CI	Accuracy	95% CI
Entire cohort (n = 1167)	0.81	0.74–0.91	0.96	0.95–0.97	0.99	0.98–0.99	0.65	0.55–0.74	0.94	0.92–0.96
Stroke protocol subgroup (n = 404)	0.82	0.71–0.89	0.90	0.86–0.93	0.96	0.93–0.98	0.64	0.53–0.73	0.89	0.86–0.94

Note:—NPV indicates negative predictive value.

Three non-stroke protocol cases were found to have LVOs and were detected by the system: An elderly lady brought in as a trauma patient due to an automobile collision, a 41-year-old patient referred from another hospital with a suspect mass found to be an infarct, and a man suspected of having carotid artery stenosis, who was found to have complete occlusion of the ICA-T. In all cases, the system alerted the team by identifying an LVO.

DISCUSSION

Computer-aided detection and diagnosis performed using machine learning algorithms can be an important tool in helping physicians interpret medical imaging findings and reducing interpretation times.¹⁶ Imaging analysis has been shown to be the main artificial intelligence medical flagship, with especially promising results in the field of neuroradiology.¹¹ This pairs well with stroke care, in which both timeliness and precision are needed.^17,18 Various artificial intelligence–based systems have been developed for emergent detection of acute ischemic stroke, with Viz LVO being the first to include automatic direct LVO detection from CTA data.¹⁹ Evaluation of the accuracy and sensitivity of the system on a large patient population is imperative for future implementation into common clinical practice. In Rambam Health Care Center, about 150 cases of endovascular treatment for acute ischemic stroke are performed annually, allowing rapid evaluation of the system on a sizeable cohort.

In this retrospective single-center study, we found the Viz LVO detection system to be highly accurate. Similar results were previously reported by Chatterjee et al²⁰ in a study performed using an older version of the software (Viz.ai-Algorithm, Version 4.1.2) exclusively on patients with stroke. A recent study by Barreira et al²¹ showed a sensitivity of 0.90 and accuracy of 0.86 using the Viz.ai Algorithm, Version 4.1.3. Both studies focused on stroke-activation protocols and, therefore, showed high rates of LVOs, 30% of the cohort in the former and 49% in the latter, in contrast to our results of 18% for the stroke protocols and 7% for the entire cohort, regardless of the scan indication.

The system encountered 56 false-positive results, 37 (66%) of which had major pathologies and 19 that had no identified pathology. The high prevalence of pathologic examinations being accidentally flagged as LVOs is related to tissue distortion, resulting in vessels being pushed and changing their course. These results, including identification of 12 M2/3 occlusion cases and 9 cases of stenosis, are difficult to interpret because the inner working of deep learning systems is not completely understood. Future improvements to the algorithms are needed to enable higher accuracy of subtler pathologies on the one hand and exclusion of nonrelevant ones on the other.

The main advantage of using artificial intelligence software in medical analysis is that it can accelerate decision-making, a feature that is especially valuable in situations that demand quick action as in LVO stroke. The system showed suboptimal sensitivity, which prevents it from being used as a diagnostic tool to date. The PPV in our cohort was 0.65. A high PPV is essential to avoid an unacceptable burden on the application end-users due to multiple false-positive alerts.

The main advantage of the system in the clinical setting of acute stroke at this point relies on its ability to accelerate decision-making in cases positive for LVO stroke. This may show great significance in environments in which interventional neuroradiology consultants are less accessible, such as in prehospital advanced imaging used in mobile stroke units, which is a fast-evolving field,²² and in primary care centers.

The study was conducted in the setting of routine clinical practice, unlike previous studies. The patients were not preselected, and the neuroradiologists involved were not notified of the evaluation performed. This feature allowed analysis and assessment of the performance of the system for everyday patients in the emergency department. It accounts for the low rate of LVO acute ischemic stroke in our patient population and the lower PPV found compared with previous publications in stroke-only series. Because the system is being installed currently in multiple medical centers, some without dedicated stroke protocols, it could provide a better reflection of the real impact of the system on the diagnostic and therapeutic flow of patients.

The system uncovered 3 LVOs in patients with a non-stroke protocol that could have been easily missed due to low clinical suspicion. Such alerts could accelerate proper care in this scenario.

This study has several limitations. First, it is not an interventional study. The system was assessed without changing the treatment provided to patients in real-time, due to ethical limitations, thus preventing concrete discussion of improved time and cost with use of the system. Further research is already planned.

Furthermore, the criterion standard for LVO detection relied on a single neuroradiologist read per examination. Although the ICC showed no variation among readers, such evaluation is still subject to mistakes. Data were collected by radiology residents and assessed for possible discrepancies in follow-up examinations and the general clinical course of the patient to minimize such errors. In any case of inconsistency, examinations were marked and reread by a second senior neuroradiologist.

Another point is the exclusion of 13 examinations rejected by the system as technically inadequate, as described above. These examinations were not included in the study because they were not processed by the algorithm for LVO detection.

This study was conducted in a single comprehensive stroke center. One of the most fundamental future applications of the system is in improving notification, assessment, and treatment times for patients arriving at primary stroke centers. Thus, the next step in the evaluation of the system will need to be a multicenter study, comparing treatment timelines.

CONCLUSIONS

Our experience evaluating Viz LVO shows that the system has real potential for early, accurate identification of patients with stroke, hopefully improving workflow and patient care.

ABBREVIATIONS:

ICA-T

ICA terminus

ICC

intraclass correlation coefficient

LVO

large-vessel occlusion

PPV

positive predictive value