Diagnostic accuracy of automated occlusion detection in CT angiography using e-CTA

Abstract

Background and aim

The aim of this study was to assess the diagnostic accuracy of e-CTA (product name) (Brainomix) in the automatic detection of large vessel occlusions in anterior circulation stroke.

Methods

Of 487 CT angiographies from patients with large vessel occlusions stroke, 327 were used to train the algorithm while the remaining cases together with 140 negative CT angiographies were used to validate its performance against ground truth. Of these 301 cases, 144 were randomly selected and used for an additional comparative analysis against 4 raters. Sensitivity, specificity, positive and negative predictive value (PPV and NPV), accuracy and level of agreement with ground truth (Cohen’s Kappa) were determined and compared to the performance of a neuroradiologist, a radiology resident, and two neurology residents.

Results

e-CTA had a sensitivity and specificity of 0.84 (0.77–0.89) and 0.96 (0.91–0.98) respectively for the detection of any large vessel occlusions on the correct side in the whole validation cohort. This performance was identical in the comparative analysis subgroup and was within the range of physicians at different levels of expertise: 0.86–0.97 and 0.91–1.00, respectively. For the detection of proximal occlusions, it was 0.92 (0.84–0.96) and 0.98 (0.94–1.00) for the whole cohort and 0.93 (0.80–0.98) and 1.00 (0.95–1.00) for the comparative analysis, respectively for e-CTA. The range was 0.8–0.97 for sensitivity and 0.97–1.00 for specificity for the four physicians.

Conclusions

The performance of e-CTA in detecting any large vessel occlusions is comparable to less experienced physicians but is similar to experienced physicians for detecting proximal large vessel occlusions.

Introduction

In patients with suspected acute ischemic stroke (AIS), non-enhanced CT and CT angiography are the primary imaging modalities. Non-enhanced CT is important to determine the extent of acute infarction, which can be visually assessed using the Alberta Stroke Program Early CT Score (ASPECTS). Automated calculation of ASPECTS using commercial software packages has gained increasing attention in both research and clinical practice. It has been shown that automatically calculated ASPECTS has similar diagnostic accuracy compared to experienced radiologists and shows good correlation with clinical outcome after endovascular thrombectomy.^1–4

The main purpose of CT angiography is the detection of large vessel occlusions (LVOs) as this determines whether a patient is eligible for mechanical thrombectomy. Detection of LVO is time critical and confident decision making can be challenging for less experienced readers. Fasen et al. have reported that non-neuroradiologists are significantly more likely to miss LVO in CTA compared to neuroradiologists, especially occlusions of the M2 segment of the middle cerebral artery. ⁵ Potential support to improve and accelerate the detection of LVO could come from artificial intelligence driven software tools, of which several have been recently developed and are commercially available.^6–10

In the present study, we aimed to validate e-CTA (e-Stroke Suite, Brainomix Ltd, Oxford, UK) against ground truth and compare its performance with that of physicians with different levels of expertise.

Methods

Study design

This study on diagnostic accuracy is designed and reported to conform to STARD guidelines. ¹¹ The study was approved by the institutional review board. Informed consent was waived.

Participants

All patients undergoing mechanical thrombectomy between January 2010 and December 2016 due to anterior circulation AIS were reviewed. Inclusion criteria were (i) LVO of the terminal internal carotid artery or middle cerebral artery up to the proximal M2 level, (ii) CTA images of sufficient quality, i.e. CT scan primarily in the arterial phase without severe motion artifacts and with a slice thickness ≤1 mm. After exclusion of 29 cases (five cases because of low contrast and three cases due to severe motion artifacts), 487 cases were identified. Of these individuals, a first cohort of 327 cases between 2010 and 2013 was chosen randomly to train the e-CTA software.

The remaining 160 LVO cases were mixed with CTA examinations from 141 consecutive AIS patients without LVO (i.e. in total 301 cases) between January 2014 and December 2017. This cohort was used for validation of the software against standard reference, only (Supplementary Table 1, Supplementary Figure 1).

Of those 301 patients, 144 were again selected randomly and analyzed additionally by four human readers (a board-certified neuroradiologist, a radiology resident and two neurology residents) blinded to any other information, i.e. affected hemisphere, ASPECTS and NIHSS, to ensure a comparative analysis.

Occlusions were classified as either ‘proximal’ comprising terminal internal carotid artery and the proximal M1 segment of the middle cerebral artery or as ‘distal’, comprising distal M1 segment and proximal M2 segments. This simplified classification was specifically chosen for practical reasons. Moreover, there is inconsistent nomenclature of M2 occlusions in the literature. While some authors define the vessel segment beyond the anterior temporal branch as a so called M2 trunk, others regard it a distal portion of the M1 segment.^12,13

Image acquisition and reconstruction

CTA imaging was performed using a variety of multislice CT scanners at stroke centers that participate in a regional stroke network. CT acquisition protocols were locally determined and not identical, reflecting real-world practice. In general, a single contrast bolus was given intravenously, followed by a saline flush. Aortic contrast opacification was monitored using bolus tracking. After a certain threshold was achieved, a caudocranial CT scan was started from the aortic arch to the vertex. For the present study, only axial reformations with a slice thickness between 0.6 and 1 mm were used. These images were de-identified and archived in the institutional Picture Archiving and Communication System.

Test methods

Reference standard was set by a board-certified neuroradiologist with more than 10 years of experience and unrestricted access to all clinical and imaging data, including data on interventional therapy and follow-up.

CTA images in axial reformation were analyzed by e-CTA (Figure 1), a board-certified neuroradiologist, a radiology resident, and two neurology residents. The readers were blinded to any supporting information. Coronal or sagittal reformations of the CTA, non-enhanced CT and CT perfusion were not available to the readers. All readers assessed occlusion location (no occlusion, proximal occlusion, distal occlusion) and the side of occlusion (right, left or not applicable).

Figure 1.

e-CTA demonstrating a distal occlusion of the right middle cerebral artery.

An unblinded reader afterwards determined the coordinates of the occlusion location on CTA images (slice number, X and Y coordinates). These coordinates were also assessed by e-CTA in order to evaluate the proximity of automatically detected occlusion location to the reference standard.

Statistical analysis

Statistical analyses were performed using R (version 3.6.2) and RStudio (version 1.2.5033). Level of agreement of each reader (e-CTA and four human readers) with the reference standard was evaluated for: (i) detection of any occlusion on the correct side, (ii) detection of proximal occlusions on the correct side (distal occlusions were excluded). Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) with 95% confidence intervals were calculated using the pROC package. Accuracy with 95% confidence intervals was calculated using the caret package and e1071 package. Cohen’s Kappa was calculated using the psych package.

Results

Performance of e-CTA

Of the 301 patients analyzed by e-CTA, 79 patients had a right-sided LVO, 81 patients had a left-sided LVO, and 141 patients had no LVO. Of the 160 patients with an LVO, 95 patients had a proximal LVO (i.e. occlusion of the internal carotid artery or proximal M1 segment) and 65 patients had a distal LVO (occlusion of the distal M1 or proximal M2 segment).

e-CTA detected 134 out of 160 (83.2%) occlusions on the correct side. Sensitivity, specificity, PPV, NPV, accuracy, and Kappa of e-CTA in all 301 cases were 0.84, 0.96, 0.96, 0.84, 0.89, and 0.83, respectively for detection of any LVO on the correct side, and 0.92, 0.98, 0.97, 0.95, 0.95, and 0.92, respectively for the detection of a proximal occlusion on the correct side (Table 1).

Table 1.

Detection of any LVO (proximal or distal) on correct side

	Sensitivity	Specificity	PPV	NPV	Accuracy	Kappa
e-CTA (n = 301)	0.84 (0.77–0.89)	0.96 (0.91–0.98)	0.96 (0.91–0.98)	0.84 (0.77–0.89)	0.89 (0.85–0.93)	0.83 (0.77–0.89)
e-CTA (n = 144)	0.84 (0.74–0.92)	0.96 (0.89–0.99)	0.95 (0.87–0.99)	0.87 (0.77–0.93)	0.90 (0.84–0.95)	0.84 (0.76–0.92)
Neuroradiologist (n = 144)	0.97 (0.90–1.00)	0.99 (0.93–1.00)	0.99 (0.92–1.00)	0.97 (0.91–1.00)	0.98 (0.94–1.00)	0.97 (0.93–1.00)
Radiology resident (n = 144)	0.96 (0.88–0.99)	0.92 (0.83–0.97)	0.92 (0.83–0.97)	0.92 (0.83–0.97)	0.94 (0.88–0.97)	0.90 (0.84–0.96)
Neurology resident 1 (n = 144)	0.86 (0.75–0.93)	0.91 (0.81–0.96)	0.90 (0.80–0.96)	0.87 (0.77–0.94)	0.88 (0.82–0.93)	0.81 (0.72–0.89)
Neurology resident 2 (n = 144)	0.91 (0.82–0.97)	1.00 (0.95–1.00)	1.00 (0.94–1.00)	0.93 (0.84–0.97)	0.96 (0.91–0.98)	0.93 (0.88–0.98)

NPV: negative predictive value; PPV: positive predictive value.

Confidence intervals are provided in brackets. The whole cohort (n = 301) was analyzed by e-CTA only. A subgroup (n = 144) was additionally analyzed by four human readers.

Median distance between clot localization by e-CTA and reference standard in all 301 cases was 4 mm (interquartile range 2–12 mm; Supplementary Figure 2).

Comparison with human readers

Of the 144 cases used for comparative analysis, 41 presented with a right-sided LVO and 29 with a left-sided LVO. Of these 70 patients, 40 patients had a proximal LVO and 30 patients had a distal LVO.

e-CTA detected 59 out of 70 (84.3%) occlusions on the correct side. Similar to the neuroradiologist, e-CTA showed a high specificity (0.96) and PPV (0.95) for the detection of any occlusion on the correct side. Sensitivity (0.84) and NPV (0.87), however, were relatively low compared to the radiology resident (0.96 and 0.92, respectively) and one of the neurology residents (0.91 and 0.93, respectively). Accuracy of e-CTA (0.90) and agreement with the reference standard according to Cohen’s Kappa (0.84) were below the readers, except for one neurology resident (Table 1). When excluding distal occlusions, e-CTA showed high accuracy in the detection of proximal occlusions (0.97) with high positive (1.00) and negative (0.96) predictive values, similar to the neuroradiologist (Table 2).

Table 2.

Detection of proximal LVO on correct side

	Sensitivity	Specificity	PPV	NPV	Accuracy	Kappa
e-CTA (n = 236)	0.92 (0.84–0.96)	0.98 (0.94–1.00)	0.97 (0.91–0.99)	0.95 (0.89–0.98)	0.95 (0.92–0.98)	0.92 (0.87–0.96)
e-CTA (n = 114)	0.93 (0.80–0.98)	1.00 (0.95–1.00)	1.00 (0.91–1.00)	0.96 (0.89–0.99)	0.97 (0.93–0.99)	0.94 (0.88–1.00)
Neuroradiologist (n = 114)	0.97 (0.87–1.00)	1.00 (0.95–1.00)	1.00 (0.91–1.00)	0.99 (0.93–1.00)	0.99 (0.95–1.00)	0.98 (0.94–1.00)
Radiology resident (n = 114)	0.95 (0.83–0.99)	1.00 (0.95–1.00)	1.00 (0.91–1.00)	0.97 (0.91–1.00)	0.98 (0.94–1.00)	0.96 (0.91–1.00)
Neurology resident 1 (n = 114)	0.88 (0.73–0.96)	0.97 (0.91–1.00)	0.95 (0.82–0.99)	0.94 (0.85, 0.98)	0.94 (0.88–0.98)	0.86 (0.76–0.96)
Neurology resident 2 (n = 114)	0.80 (0.64–0.91)	1.00 (0.95–1.00)	1.00 (0.89–1.00)	0.90 (0.82–0.96)	0.93 (0.87–0.97)	0.84 (0.73–0.95)

NPV: negative predictive value; PPV: positive predictive value.

Confidence intervals are provided in brackets. The whole cohort after exclusion of distal occlusions (n = 236) was analyzed by e-CTA only. A subgroup (n = 114) was additionally analyzed by four human readers.

Discussion

In the present study, e-CTA, an artificial intelligence based software tool, was validated in a realistic scenario against reference standard and its performance was compared to that of stroke physicians at different levels of expertise. For the detection of proximal vessel occlusions, accuracy of e-CTA was equivalent to experienced physicians. Detection of more distal occlusions was less sensitive, and diagnostic accuracy was below that of a neuroradiologist but comparable to that of physicians in training. e-CTA was able to localize the occlusion within a median distance of 4 mm.

There are two other commercially available software tools for automatic LVO detection on CTA, i.e., RAPID CTA (iSchemaView, Menlo Park, CA) and Viz LVO (Viz.ai, San Francisco, CA, USA), the latter of which is approved by the Federal Drug Administration (FDA) and reimbursed by the American healthcare system. They reported a similar performance with a sensitivity of 90.1% and a specificity of 82.5% for proximal LVOs and 81.4% and 82.2%, respectively, when including more distal occlusions. ⁹

Amukotuwa Shalini et al. have analyzed diagnostic accuracy in LVO detection of another software tool. However, they have not clearly stated the statistical methodology behind their analysis, which makes it difficult to compare their results to the present study. For instance, it is not clear whether the detection of LVO at the correct side was considered in the analysis and no attempt was made to anatomically localize the occlusion. Similarly to e-CTA, accuracy of RAPID CTA is decreased in distal occlusions. ⁸ Identification of distal occlusions is currently a challenge for automated LVO detection tools, as well as for clinicians. Up to 16% of more distal (i.e. distal M1 and proximal M2) but still endovascularly treatable occlusions may be missed by e-CTA, and a “negative” software result needs to be reviewed in the acute situation by a neuroradiologist considering any information including additional techniques such as CT perfusion, if available.

The strength of the present study is that performance data from physicians with a range of experience are available for comparison to e-CTA, and that the anatomical localization was evaluated. However, the study is limited due to a relatively modest number of patients with LVO analyzed by e-CTA against reference standard. Future studies should evaluate e-CTA performance prospectively as a real-time decision aid to support LVO identification by physicians, and the impact on workflow. Also, permanent monitoring and critical evaluation of the performance of such software tools by physicians are mandatory if used in clinical routine.

Currently, LVO detection tools are based on CTA images only. Diagnostic accuracy of decision support software might be further improved by incorporating multimodal imaging such as non-enhanced CT, which can be used to identify occlusion location by the hyperdense artery sign. ¹⁴ Similarly, focal hypoperfusion on CT perfusion may help the reader to scrutinize for, and identify more distal occlusions in this territory. As another minor limitation, about 1.5% of the CTA scans were excluded in the present analysis due to insufficient quality.

The present data show that automated LVO detection using e-CTA has an accuracy at a level between less experienced and expert physicians. Therefore, e-CTA might provide different opportunities to improve patient care under different circumstances. First, e-CTA could prioritize particular cases for evaluation by the responsible radiologist and change the level of urgency within everyday clinical routine. In the time-critical context of LVO stroke, even experienced physicians in comprehensive stroke centers may be able to accelerate the patient pathway using this support. Second, when combined with integration into smartphone application alerts or email notification, this may facilitate speed of remote decision making on on-call service. In many regional stroke networks, teleradiology is used for expert reading of CTA images acquired in peripheral hospitals. Patients are then transferred to a comprehensive stroke center, if an LVO is detected. By adding automated software for detection of LVO to the clinical pathway, this can support the decision for a referral where necessary, by improving physician confidence acting as a decision support tool and thereby accelerate the referral process. In a small study using Viz LVO, this could be demonstrated. ¹⁵ Likewise, e-CTA could be beneficial in mobile stroke units which typically do not have a radiologist on board and rely on image data transfer to a comprehensive stroke center.

Conclusions

e-CTA has a similar performance to physicians in training in detecting LVO on CTA. Automated LVO detection tools may be helpful to accelerate patient pathways, to support less experienced physicians in training, or for use in teleradiology settings. At present the diagnostic accuracy is not sufficient to be used as a standalone diagnostic tool that would replace expert neuroradiologist interpretation.

ORCID iDs

Fatih Seker https://orcid.org/0000-0001-6072-0438

Johannes Alex Rolf Pfaff https://orcid.org/0000-0003-0672-5718

Yahia Mokl https://orcid.org/0000-0002-7587-9684

Anne Berberich https://orcid.org/0000-0002-6991-1334

Simon Nagel https://orcid.org/0000-0003-2471-6647