RapidAI Compared With Human Readers of Acute Stroke Imaging for Detection of Intracranial Vessel Occlusion

Abstract

Background

Rapid detection of intracranial arterial occlusion in patients with ischemic stroke is important to facilitate timely reperfusion therapy. We compared the diagnostic accuracy of neurologists and radiologists against RapidAI (iSchema View, Menlo Park, CA) software for occlusion detection.

Methods

Adult patients who presented to a single comprehensive stroke center over a 5‐month interval with clinical suspicion of ischemic stroke and who underwent multimodality imaging with RapidAI interpretation were included. There were 8 assessors: 1 radiologist, 5 neurologists, and 2 radiology trainees. The reference standard was large‐vessel occlusion (LVO) or medium‐vessel occlusion (MVO) diagnosed by a panel of 4 interventional neuroradiologists. Positive likelihood ratio (LR) and negative LR were used to indicate how well readers correctly classified the presence of intracranial occlusions compared with the reference standard. The positive LR and negative LR for each reader were plotted on an LR graph using RapidAI LRs as comparator.

Results

The assessors read scans from 500 patients (49.6% men). The positive LR of RapidAI for detection of LVO was 8.49 (95% CI, 5.75–12.54), and the negative LR was 0.41 (95% CI, 0.28–0.58). The positive LR for LVO or MVO for RapidAI was 5.0 (95% CI, 3.28–7.63), and the negative LR was 0.66 (95% CI, 0.56−0.79). Sensitivity for LVO (0.65–0.96) and for LVO or MVO (0.62–0.94) was higher for all readers compared with RapidAI (0.62 and 0.39, respectively). Six of 8 readers had superior specificity to RapidAI for LVO (0.75–0.98 versus 0.93) and LVO or MVO (0.55–0.95 versus 0.92).

Conclusions

Experienced readers of acute stroke imaging can identify LVOs and MVOs with higher accuracy than RapidAI software in a real‐world setting. The negative LR of RapidAI software was not sufficient to rule out LVO or MVO.

Nonstandard Abbreviations and Acronyms

CTA

computed tomography angiography

CTP

computed tomography perfusion

LVO

large‐vessel occlusion

MVO

medium‐vessel occlusion

NNR

number needing review

Clinical Perspective

What Is New?

• Few studies have measured the performance of stroke diagnostic artificial intelligence programs in clinical situations or assessed their ability to detect arterial occlusions outside the anterior circulation.

• A strength of this article is the use of likelihood ratio graphs to compare human readers with RapidAI.

• This approach allows us to infer whether readers are better than RapidAI in ruling in and/or ruling out the presence of intracranial occlusion.

What Are the Clinical Implications?

• RapidAI, at its present state of development, should not be solely relied on to triage patients with stroke.

• Reliance on RapidAI could result in a substantial proportion of patients being incorrectly classified as ineligible for reperfusion therapy.

• Artificial intelligence programs, such as RapidAI, can often improve workflow by alerting neurologists and neurointerventional radiologists to possible abnormalities, leading to earlier confirmation of imaging findings and involvement of thrombectomy teams.

Rapid detection of arterial occlusion in ischemic stroke is important to facilitate timely reperfusion therapy. ¹ , ² Patients with large‐vessel occlusions (LVOs) and medium‐vessel occlusions (MVOs) may be eligible for thrombolytic treatment. Thrombectomy is currently indicated for occlusions of large vessels (internal carotid, middle cerebral, and vertebrobasilar arteries), although advances in technology and clinical trial evidence are likely to see these indications extended to more distal, MVOs. Delays in diagnosis of arterial occlusion have major implications for stroke outcome, as mortality and morbidity increase with time to reperfusion. ² , ³ , ⁴ A major hurdle to providing prompt therapy is uncertainty with the interpretation of multimodality stroke imaging. Specialized skills are needed to detect these arterial lesions on computed tomography angiography (CTA). Smaller institutions with limited access to neuroradiology expertise face the greatest challenges.

Automated software, such as RapidAI (iSchema View, Menlo Park, CA), has been used in randomized clinical trials of thrombectomy ⁵ and thrombolytic therapy ⁶ to identify and quantify critically ischemic but potentially salvageable brain tissue. Recently, investigators have extended the repertoire of RapidAI and other automated software to include detection of occluded large vessels. The term “large vessel” originally was used to indicate only the intracranial internal carotid artery and M1 and proximal M2 segments of the middle cerebral artery. More recent studies have also included the basilar artery, intracranial vertebral artery, proximal anterior cerebral artery, and proximal posterior cerebral artery in this definition. ⁷ , ⁸ Although automated detection of occluded intracranial arteries will aid clinicians when dealing with “acute stroke code” situations, most previous evaluations of these tools have occurred in LVO and MVO registry research. ⁹ , ¹⁰ , ¹¹ Few studies have measured their performance in actual clinical practice ¹² or have assessed their accuracy for detecting occlusions outside the anterior circulation. ¹³ The aim of this retrospective study of prospectively collected cases was to compare the diagnostic accuracy of neurologists and radiologists at a high‐volume stroke center with RapidAI software in the detection of intracranial arterial occlusion.

Methods

Data are available on request from the authors. Methods are reported in accordance with Strengthening the Reporting of Observational Studies in Epidemiology guidelines.

Image Selection

Acute stroke imaging studies, performed between April 3, 2021, and August 31, 2021, satisfying all the following criteria, were selected:

• Adult patient aged ≥18 years

• Clinical suspicion of ischemic stroke referrable to any intracranial vascular territory

• Multimodality imaging comprising noncontrast computed tomography (CT) of the brain, CTA of the head and neck, and CT perfusion (CTP), together with RapidAI interpretation, was obtained.

Studies were prescreened by L.A.S. (10 years of experience as an interventional neuroradiologist). This excluded imaging studies if a technique not suitable for RapidAI processing (eg, dual‐energy CT) was used, if imaging was incomplete, or if CT angiography was of inadequate quality because of factors, such as poor contrast bolus injection or patient motion artefact.

Image Acquisition

All CT acquisitions were performed using a 64‐slice GE CT750 HD (General Electric Medical Systems, Milwaukee, WI) or a 256‐slice Philips Brilliance iCT (Philips Healthcare, Cleveland, OH) multidetector CT system. Noncontrast helical CT brain acquisitions (performed at 120 peak kilovoltage [kVp], milliamperes [mA]), adjusted per patient according to automatic exposure control, were reconstructed at 0.625/0.625 mm and 0.8/0.4 mm slice thickness/interval (the z‐axis increment between each image), respectively, with additional 3‐plane 5‐mm slice thickness contiguous multiplanar reformation (MPR). Axial MPRs were orientated to the supraorbital‐meatal line. Axial CTP z‐axis coverage for both systems was 80 mm and was reconstructed at 5 mm for the GE CT750 HD and 10 mm for the Philips Brilliance iCT (80 kVp, 200 mA and 80 kVp, 100 mA, respectively). CTP was performed with intravenous injection of a 60‐mL iodinated contrast bolus (iohexol, Omnipaque 350; GE Healthcare, Milwaukee, WI) followed by a 40‐mL normal saline chaser, both at a flow rate of 6 mL/s. Images were acquired every 2.8 and 2.0 seconds, respectively, for up to 75 seconds after the start of contrast injection. CTA was performed after an injection of 75 mL of iodinated contrast administered at 5 mL/s, followed by 40 mL normal saline chaser at the same injection rate, with acquisition from the aortic arch to skull vertex (kVp [100–120 kVp] and mAs adjusted based on patient size and CT system automatic exposure control). CTA acquisitions were reconstructed into thin slice axial plane images at 0.625/0.625 mm (GE) and 1.0/0.5 mm (Philips) slice thickness/interval with further thick slice maximum intensity projection MPR images separated into cranial and neck vessel portions.

Image Evaluation

Eight human image readers were recruited: 1 diagnostic radiologist (with several years’ experience in advanced stroke imaging), 5 neurologists (none of whom specialized in stroke neurology, although 2 participated regularly in the after‐hours acute stroke service), and 2 junior radiology trainees (both of whom had >12 months’ prior experience with advanced stroke imaging). For each case, readers were provided with the following:

1. Noncontrast CT brain 5‐mm axial images

2. Thin slice axial CTA images

3. Axial, coronal, and sagittal maximum intensity projection MPRs of only the cranial portion of the CTA data, and

4. Perfusion maps (relative cerebral blood volume [rCBV], cerebral blood flow [CBF], mean transit time [MTT], and Tmax)

These could be viewed in any order, although readers were advised to view the perfusion images before the CTA images to assist in localization of occlusions, particularly more distal ones.

All studies were deidentified. They were presented at least 3 months after hospital attendance to reduce the chance of recall should any reader have been involved in the clinical care of the patient. Each reader was provided with a batch of 100 studies and instructed to interpret them within 1 month. The process was repeated until 500 studies were reviewed. The image data were provided via a central picture archiving and communication system with digital imaging and communications in medicine images using a zero footprint image viewer (Vue Motion; Philips Healthcare, Amsterdam, the Netherlands). Digital imaging and communications in medicine functionality included windowing and zooming but not MPR. This enabled the cases to be read anywhere on a personal computer at the convenience of the reader. The digital imaging and communications in medicine viewer was the same as that used in clinical care during acute stroke codes for neurologists at our institution, so readers were familiar with the use of this tool. Radiologists were also familiar with the viewer and used it in their daily work, although not for the primary interpretation of imaging studies. There was no time limit for readers other than the 100 case per month target.

LVO was defined as occlusion of the intracranial internal carotid artery (ICA), proximal segments (M1 and M2) of the middle cerebral artery (MCA), basilar artery (BA), or intracranial vertebral artery. MVO was defined as occlusion of the anterior cerebral artery (A1, A2, A3, or more distal), distal middle cerebral artery (M3, M4, or more distal), or posterior cerebral artery. Site and side of occlusions were recorded on a spreadsheet (Excel; Microsoft Corporation, Redmond, WA) containing a nonidentifying subject number and fields for all arterial segments. Readers recorded the arterial segment with the most proximal extent of an occlusion. Clearly separate distal occlusions in the same vessel or multiple occlusions involving different vessels (eg, right and left sided, anterior, and posterior circulation) were recorded in the spreadsheet when present. If there were tandem distinct occlusions in a single line of arterial supply (eg, intracranial ICA and ipsilateral M2), they were considered as 2 separate occlusions. In the case of arterial obstruction with no distal filling, its most proximal point was taken as the sole site of occlusion. Vascular segments containing nonocclusive thrombus or atheroma were ignored. All readers were blinded to clinical information about site, side, and nature of symptoms. No follow‐up imaging was provided. Readers did not have access to prior imaging or to radiology request or report documents.

Readers were given an introductory videoconference tutorial explaining how to code the data before the first tranche of cases were scored. Readers submitted their image evaluations in batches of 100, which could not later be revised. After 300 had been scored by all readers, a review session was held via videoconference to discuss interpretive errors that had been made by ≥1 readers. Images were discussed without their identifying numbers, and without reference to interpretations by any individual reader. The purpose of this review was educational, to highlight sources of common or recurring errors, both underdiagnosis and overdiagnosis of vascular occlusion, and how these could be mitigated. The final 200 cases were then completed by all readers without further feedback. The average time spent by readers to score a set of images based on the first 20 reads of the first set and last set was 3.6 minutes, comparable to usual viewing practice during stroke codes.

RapidAI Evaluation

A single version of RapidAI (version 5.1) analyzed the same data set as the readers. RAPID LVO is an approved automated algorithm that analyzes CTA images. The algorithm computes vessel density, detecting reduced opacification of intracranial vessels relative to the contralateral brain hemisphere. Corresponding areas of reduced relative vessel density are indicated on the RapidAI CTA output maximum intensity projection (MIP) images of the intracranial vasculature, with colored threshold overlays based on the relative percentage difference. RapidAI produces separate images to indicate suspected LVOs with marked regions of interest.

Although both CTA and CTP images were analyzed by RapidAI algorithms, we used only the CTA module for our comparison with human readers. Findings from the CTP analysis had no influence, as the 2 algorithms function independently. Readers had access to perfusion maps without the calculated perfusion parameters.

Reference Standard for Vessel Occlusion

Three interventional neuroradiologists (H.A. with 10 years, L.A.S. with 10 years, and J.M. with 3 years of experience in interventional neuroradiology) independently assessed all studies to determine the presence and site of arterial occlusions. Whenever this panel was not unanimous, a fourth neurointerventional radiologist (R.C., with 13 years’ experience) made an additional independent assessment, and disagreements were reviewed. A consensus diagnosis was obtained for all cases.

RapidAI Detection of LVO Versus Reference Standard, Accounting for Side of Occlusion

In this analysis, true positive was defined as detection by RapidAI and reference standard of an LVO, with agreement on site and side of the affected artery. The false positive for LVO was defined as detection by RapidAI but not confirmed by reference standard. The false negative for LVO was defined as no detection by RapidAI but detected by reference standard. The true negative for LVO was defined as no detection by RapidAI nor reference standard.

RapidAI Detection of LVO or MVO Versus Reference Standard, Accounting for Side of Occlusion

In this analysis, the above steps for LVO were repeated to include any occlusion (LVO and MVO).

Statistical Analysis

The sensitivity, specificity, and likelihood ratios (LRs) for detection of LVO (analysis 1) and any vessel occlusion (LVO and MVO) (analysis 2) were evaluated for RapidAI and each reader. ¹⁴ The positive LR is the probability that a subject with LVO has a positive test result divided by the probability that a subject without LVO has a positive test result. This was calculated for each reader and for RapidAI. The magnitude of the positive LR indicates how much the pretest odds of occlusion being present are increased by a positive test result. It can be expressed as follows:

$$\mathrm{L}{\mathrm{R}}^{+}=\frac{\mathrm{sensitivity}}{1-\mathrm{specificity}}$$

The negative LR, an indication of the likelihood of disease being absent when the test result is negative, is expressed as follows:

$$\mathrm{L}{\mathrm{R}}^{-}=\frac{1-\mathrm{sensitivity}}{\mathrm{specificity}}.\mathrm{a}$$

We used the recommendation by Jaeschke et al to interpret positive and negative LRs. They suggest that in clinical practice, a diagnostic test with a positive LR >5 and negative LR <0.2 generates moderate change to the pretest odds of disease, whereas a positive LR of >10 or negative LR <0.1 generates large changes to the pretest odds of a condition being present. ¹⁵ Thus, for a diagnostic test to rule in a diagnosis in most circumstances, the positive LR should be >10, and for the test to rule out a diagnosis, the negative LR should be <0.1.

LR graphs were used to assess the overall performance of readers, with the results from RapidAI serving as comparator. ¹⁶ , ¹⁷ The x axis is represented by 1 minus the specificity, and the y axis is represented by the sensitivity of the reader. The LR graph is divided into 4 regions. When the coordinates for the reader are located in the left upper quadrant region (Figure 1), it suggests that the reader is superior to RapidAI for correctly identifying both the presence and absence of disease; in the left lower quadrant, it is superior to RapidAI only for correctly identifying disease when present; in the right upper quadrant, it is superior to RapidAI only for correctly excluding disease when it is absent; and in the right lower quadrant, it is inferior to RapidAI in both domains. ¹⁶ , ¹⁷ , ¹⁸

Figure 1.

Likelihood ratio graph for large‐vessel occlusion. Tool for comparison is RapidAI software. The coordinates of 1–specificity and sensitivity are displayed in red for each clinician reader.

We also evaluated number needing review (NNR) by each rater and RapidAI. ¹⁹ This is a newly proposed metric to assess triage efficiency of LVO detection systems. It was calculated as follows:

$$\mathrm{NNR}=\frac{\mathrm{Total}-\left(\mathrm{True}\mathrm{positive}+\mathrm{True}\mathrm{negative}\right)}{\mathrm{Total}}\mathrm{x}100$$

Total means all scans available for review. NNR is rounded to a whole number to represent the number of scans needing review.

Conduct of this study was approved by our institutional Monash Health Human Research Ethics Committee.

Results

A total of 500 imaging studies meeting the inclusion criteria were collectively analyzed. The median age of patients was 70 (interquartile range, 56−80) years; 49.6% were men. Eighty‐five patients (17%) had intracranial occlusion in ≥1 vessels, and 415 did not. There were 53 patients with single‐site LVO, 9 with >1 site of LVO, 3 with LVO and MVO, 2 with >1 site of MVO, and 18 with a single MVO. In total, 92 anterior circulation and 12 posterior circulation occlusions were identified by the interventional neuroradiologists to form the reference standard. The occlusions were grouped into LVO and MVO categories as defined to allow sensitivity and specificity calculations for each reader and for RapidAI. Occlusion locations are shown in Table 1.

Table 1.

Number of Occlusions at Each Site in 500 Studies Reviewed by a Panel of InterventionalNeuroradiologists

Sites	Right	Left	Middle
ICA	14	16	N/A
A1: ACA	2	1	N/A
A2: ACA	2	2	N/A
A3: ACA	2	0	N/A
A4+: MCA	0	1	N/A
M1: MCA	14	8	N/A
M2: MCA	11	7	N/A
M3: MCA	2	7	N/A
M4+: MCA	1	2	N/A
PCA	2	3	N/A
Vertebrobasilar	N/A	N/A	7

ACA indicates anterior cerebral artery; ICA, internal carotid artery; MCA, middle cerebral artery; and PCA, posterior cerebral artery.

Large‐Vessel Occlusion

There were 77 LVOs in 65 patients. RapidAI’s sensitivity for LVO was 0.62, and its specificity was 0.927, providing a positive LR of 8.49 (95% CI, 5.75–12.54) and a negative LR of 0.41 (95% CI, 0.28–0.58). The readers’ sensitivity, specificity, and LRs are shown in Table 2. All readers had higher sensitivity than RapidAI for diagnosis of LVO when it was present.

Table 2.

Sensitivity, Specificity, Positive LR, Negative LR, and NNR for Each Clinician Reader and RapidAI Software for the Detection of LVO

Reader	Sensitivity	Sensitivity 95% CI	Specificity	Specificity 95% CI	LR+	LR+ 95% CI	LR–	LR– 95% CI	NNR	NNR 95% CI
Reader 1	0.84	0.71–0.93	0.98	0.96–0.99	42.16	21.84–81.36	0.16	0.08–0.30	3	2–6
Reader 2	0.82	0.69–0.92	0.95	0.93–0.97	17.65	11.41–27.30	0.19	0.10–0.34	6	4–9
Reader 3	0.67	0.52–0.79	0.97	0.94–0.98	18.79	11.18–31.57	0.35	0.23–0.57	7	4–12
Reader 4	0.96	0.87–1.00	0.75	0.71–0.79	3.88	3.28–4.59	0.05	0.01–0.20	23	19–27
Reader 5	0.90	0.79–0.97	0.98	0.96–0.99	40.59	21.84–75.41	0.10	0.04–0.23	3	2–6
Reader 6	0.65	0.51–0.77	0.80	0.76–0.84	3.28	2.51–4.3	0.44	0.30–0.63	21	16–28
Reader 7	0.94	0.84–0.99	0.97	0.95–0.98	28.24	17.09–46.66	0.06	0.02–0.18	4	2–7
Reader 8	0.87	0.74–0.94	0.98	0.96–0.99	43.27	22.46–83.35	0.14	0.07–0.27	3	2–6
RapidAI	0.62	0.48–0.75	0.93	0.90–0.95	8.49	5.75–12.54	0.41	0.28–0.58	11	7–16

LR indicates likelihood ratio, LVO, large‐vessel occlusion; and NNR, number needing review.

As demonstrated by the LR graph (Figure 1), 6 of 8 readers were superior to RapidAI in identifying both absence and presence of LVO. These included 3 neurologists (including the 2 with greater acute stroke experience), a radiologist, and 2 radiology trainees. One reader was superior to RapidAI for detecting absence of LVO (sensitivity). One reader had inferior positive and negative LRs compared with RapidAI despite slightly superior sensitivity, attributable to RapidAI’s much greater specificity.

The NNR of RapidAI for LVO was 11, with 2 raters having higher values (Table 2).

LVO and MVO

There were 20 patients who only had MVO. This gave a total of 104 LVOs or MVOs in 85 patients. RapidAI’s sensitivity for LVO and MVO detection combined was 0.39, with a specificity of 0.92, providing a positive LR of 5.0 (95% CI, 3.28–7.63) and a negative LR of 0.66 (95% CI, 0.56–0.79). The readers’ specificity, sensitivity, and LRs for LVO and MVO are shown in Table 3. As demonstrated by the LR graph (Figure 2), all readers were superior to RapidAI for identification of LVO or MVO when it was present (sensitivity). Five readers, 3 neurologists (including the 2 with greater stroke on‐call experience) and 2 radiology trainees, had positive and negative LR superior to RapidAI. Three readers had lower specificity than RapidAI, resulting in lower positive LR for 2 of them. The NNR of RapidAI for LVO was 17, with 2 raters having higher values (Table 3).

Table 3.

Sensitivity, Specificity, Positive LR, Negative LR, and NNR for Each Clinician Reader and RapidAI Software for the Detection of LVO and MVO

Reader	Sensitivity	Sensitivity 95% CI	Specificity	Specificity 95% CI	LR+	LR+ 95% CI	LR–	LR– 95% CI	NNR	NNR 95% CI
Reader 1	0.77	0.67–0.86	0.95	0.93–0.97	15.79	10.24–24.35	0.24	0.16–0.36	8	5–12
Reader 2	0.81	0.71–0.89	0.91	0.87–0.93	8.64	6.31–11.81	0.21	0.14–0.33	11	8–15
Reader 3	0.75	0.64–0.84	0.89	0.86–0.92	6.95	5.15–9.39	0.28	0.20–0.41	13	10–18
Reader 4	0.94	0.86–0.98	0.55	0.51–0.60	2.11	1.87–2.37	0.11	0.05–0.26	39	35–44
Reader 5	0.87	0.78–0.93	0.95	0.93–0.97	18.39	11.89–28.44	0.14	0.08–0.24	6	4–9
Reader 6	0.62	0.51–0.72	0.67	0.62–0.71	1.87	1.51–2.32	0.57	0.43–0.75	34	27–42
Reader 7	0.89	0.80– 0.95	0.94	0.91–0.96	14.62	10–21.38	0.12	0.06–0.22	7	5–10
Reader 8	0.82	0.72–0.90	0.95	0.93– 0.97	17.78	11.46–27.6	0.19	0.12–0.30	7	5–11
RapidAI	0.39	0.28–0.50	0.92	0.89–0.95	5.0	3.28–7.63	0.66	0.56–0.79	17	11–26

LR indicates likelihood ratio, LVO, large‐vessel occlusion; MVO, medium‐vessel occlusion; and NNR, number needing review.

Figure 2.

Likelihood ratio graph for large‐ and medium‐vessel occlusion. Tool for comparison is RapidAI. The coordinates of 1–specificity and sensitivity are displayed in red for each clinician reader.

Discussion

The key finding of this study is that radiologists, radiology trainees, and neurologists with variable amounts of experience in acute stroke imaging at a comprehensive stroke center performed better than RapidAI in detecting LVO and MVO. Against the reference standard of expert neurointerventional radiologists, RapidAI demonstrated a sensitivity of less than two‐thirds for LVO and just over a third for MVO and LVO combined. Overall, RapidAI did not reach negative LR thresholds that would enable it adequately to rule out a diagnosis of LVO or MVO in a real‐world stroke code population. Although RapidAI could play a role before expert human review of CTA studies for ruling in LVO, reliance on RapidAI could result in a substantial proportion of patients being incorrectly classified as ineligible for reperfusion therapy.

Previous studies of RapidAI software have reported sensitivities of 94% to 97% and specificities of 74% to 76% for LVOs in the anterior circulation. ¹⁰ , ²⁰ Differences in performance of artificial intelligence tools on representative clinical cases, compared with clinical trial material or data sets enriched with “atypical” proportions of normal or abnormal scans, highlight the need for monitoring after the clinical implementation of any such application. Other factors that may contribute to variation in performance include differences in hardware, scanning parameters, contrast injection rates, and patient populations. Performance “drift” over time is a further acknowledged problem for artificial intelligence tools with medical and nonmedical tasks. Such drift in diagnostic imaging interpretation may go undetected without regular monitoring against the “ground truth” of expert opinion. ²¹ In practice, this monitoring is costly and time‐consuming in the high‐stakes environment of medical artificial intelligence. For applications that extract image features to predict the future risk of cancer or myocardial infarction, monitoring of performance presents even greater difficulties.

We measured the sensitivity, specificity, and LR of RapidAI against a reference standard and found that it performed well for ruling in LVO (high positive LR) but not sufficiently well for ruling LVO out because its negative LR was not low enough to enable treatment decisions to be based on its interpretation. Ideally, the negative LR should be <0.1 for a test to be useful for ruling out LVO or MVO. ¹⁵ RapidAI was even less sensitive for MVO, although its specificity for MVO and LVO was the similarly high for each (0.92). These findings are consistent with NNR to assess efficiency of vessel occlusion detection systems. ¹⁹ The high NNR for MVO or LVO suggests that RapidAI further review of the scan by an expert is required. ¹⁹

We did not calculate the area under the receiver operating curve for RapidAI detection of arterial occlusion. The test has only 1 threshold, and receiver operating curves are better suited to perform in diagnostic tests with multiple thresholds. Other investigators suggest caution for the validity of using receiver operating curve with single‐threshold diagnostic tests. ²² , ²³ Instead, we used LR to evaluate overall diagnostic performance.

A strength of this article is the use of LR graphs to compare readers with RapidAI. ¹⁶ , ¹⁸ This approach permits simultaneous comparison of the positive LR and negative LR, allowing us to infer whether the reader is superior to RapidAI in ruling in and out the presence of intracranial occlusion, good only for ruling occlusion in, or good only for ruling it out. This can be seen by examining the position of the rater in the 4 quadrants (Figures 1 and 2).

Our study evaluated MVO in addition to LVO because in practice, clinicians need to identify all vascular occlusions. This is an important issue, as those working in hospitals at a distance from stroke treatment centers need to be confident that an artificial intelligence tool can also detect MVO. The identification of an MVO informs decision‐making about thrombolytic therapy, and clinical trials are underway to assess benefit of thrombectomy in MVO. Although studies have shown that neuroradiologists can detect LVO with 89% to 95% sensitivity and 95% to 98% specificity, this level of skill is not universal. ²⁴ , ²⁵ MVO detection is more challenging, for both human readers and RapidAI.

It is important to acknowledge potential advantages of an automated system for detection of LVO and MVO. RapidAI is susceptible to programming‐related systematic errors, but not to errors resulting from fatigue, clinical load, and lapses in concentration. It allows immediate remote access by external readers. This reduces delay in reconstruction, which is 30 minutes on average and can be as long as 50 minutes. ²⁴ The read time for RapidAI averages 5 minutes, faster than a less experienced reader, hastening access to relevant teams and expertise. ¹⁰ , ²⁶ Last, artificial intelligence programs, such as RapidAI, can often improve workflow by alerting neurologists and neurointerventional radiologists to possible abnormalities, leading to earlier confirmation of imaging findings and involvement of thrombectomy teams.

In this analysis, we have used the current version of RapidAI. An advantage of such software is that it can be improved with subsequent versions. Our findings could inform upgrading to improve detection of LVO and MVO. Ideally, one would want a tool with high positive LR and low negative LR ¹⁵ and, more recently, low NNR. ¹⁹ This may not always be possible, but future iterations of RapidAI should focus on reducing negative LR and improving sensitivity for detection of LVO and, particularly, MVO. RapidAI was developed at a time when clinical trials focused on LVO in the anterior circulation. New versions of this and similar software require a broader sensitivity to arterial occlusion, in the vertebrobasilar system and distally in the intracranial carotid territories. The final objective is an automated diagnostic application that gives less experienced clinicians confidence that an LVO or MVO requiring reperfusion therapy will not be overlooked.

Limitations

This study was limited by its retrospective design. Readers were able to review scans at their own pace, which did not fully reproduce the way that these images must be interpreted in real circumstances. We tried to normalize the experience by giving readers access to all the scans that have to be reviewed during acute stroke codes. On the other hand, stroke imaging in practice is assessed in conjunction with clinical information, and its absence could have disadvantaged human readers. Although we aimed to include consecutive cases over a 5‐month period, some cases were excluded because of technical factors or prior contrast exposure (eg, recent coronary angiography).

It is difficult to ascertain from patient records and hospital statistics the true impact of RapidAI software on workflow efficiency and time to reperfusion, although this was not the purpose of this study. We acknowledge that RapidAI is not marketed to detect occlusions in the vertebrobasilar system or of medium‐sized arteries anywhere in the intracranial circulation. With extended indications for thrombectomy currently being explored, it important to highlight the need for new software iterations to have validated detection capacity that encompasses these other vascular occlusions.

As clinical outcomes were not considered, we did not assess the clinical effects of RapidAI interpretations of LVO; these findings from RapidAI had not been available to clinicians at the time of presentation with suspected acute stroke. Clinician readers in this study, despite various backgrounds and years of experience, were likely to have more exposure to CTP and CTA imaging than those working in rural or regional settings, and this is likely to have contributed to their diagnostic performance. The readers also benefited from being able to use information from CTP when determining the likely location of an LVO or MVO, which has been shown to improve detection of these occlusions. ¹¹ Yet, CTP is not always used in acute stroke imaging, particularly when patients present within 6 hours of the onset of symptoms. This may reduce the generalizability of our findings.

Conclusions

Experienced readers of acute stroke imaging can identify LVOs and MVOs with higher accuracy than RapidAI in circumstances that model the real world of acute stroke presentations. Although RapidAI should not currently be used to triage patients with stroke, we expect that future improvements in the software will allow it to perform this role.

Author Contributions

Study design: Lee‐Anne Slater, Stacy Goergen, Ronil Chandra, Thanh G. Phan. Statistical analysis: Thanh G. Phan, Lee‐Anne Slater, Nandhini Ravintharan. Data collection: Nandhini Ravintharan, Lee‐Anne Slater, Ahilan Kuganesan. Manuscript: Lee‐Anne Slater, Stacy Goergen, Ronil Chandra, Ahilan Kuganesan, Sandra Lin, Hamed Asadi, Julian Maingard, Reuben Sum, Victor Gordon, Deepa Rajendran, Yenni Lie, Subramanian Muthusamy, Thanh G. Phan, Peter Kempster.

Sources of Funding

None.

Disclosures

None.