Abstract
Rapid and accurate diagnosis of large vessel occlusions (LVOs) in acute ischemic stroke (AIS) patients using automated software could improve clinical workflow in determining thrombectomy in eligible patients. Artificial intelligence-based methods could accomplish this; however, their performance in various clinical scenarios, relative to clinical experts, must be thoroughly investigated. We aimed to assess the ability of Canon’s AUTOStroke Solution LVO application in properly detecting and locating LVOs in AIS patients. Data from 202 LVO and 101 non-LVO AIS patients who presented with stroke-like symptoms between March 2019 and February 2020 were collected retrospectively. LVO patients had either an internal carotid artery (ICA) (n = 59), M1 middle cerebral artery (MCA) (n = 82) or M2 MCA (n = 61) occlusion. Computed tomography angiography (CTA) scans from each patient were pushed to the automation platform and analyzed. The algorithm’s ability to detect LVOs was assessed using accuracy, sensitivity and Matthews correlation coefficients (MCCs) for each occlusion type. The following results were calculated for each occlusion type in the study (accuracy, sensitivity, MCC): ICA = (0.95, 0.90, 0.89), M1 MCA = (0.89, 0.77, 0.78) and M2 MCA = (0.80, 0.51, 0.59). For the non-LVO cohort, 98% (99/101) of cases were correctly predicted as LVO negative. Processing time for each case was 69.8 ± 1.1 seconds (95% confidence interval). Canon’s AUTOStroke Solution LVO application was able to accurately identify ICA and M1 MCA occlusions in addition to almost perfectly assessing when an LVO was not present. M2 MCA occlusion detection needs further improvement based on the sensitivity results displayed by the LVO detection algorithm.
Keywords: Artificial intelligence, brain, CT angiography, ischemic stroke
Introduction
Computed tomography angiography (CTA) is an imaging modality commonly used to determine the location of an occlusion in acute ischemic stroke patients (AIS). Once the occlusion is located, American Heart Association guidelines are applied to each case, assessing patient specific parameters, for the purpose of determining if a patient is mechanical thrombectomy eligible. 1 Thrombectomy eligible patients meet the following criteria: a National Institute of Health Stroke Scale (NIHSS) score of at least 6, an Alberta Stroke Program Early CT Score (ASPECTS) of at least 6, a large vessel occlusion (LVO) within the internal carotid artery (ICA) or M1 middle cerebral artery (MCA), and meet the requirements for ischemic tissue volumes detailed in the DAWN or Defuse 3 clinical trials when symptom onset is between 6 and 24 hours. If symptom onset is less than 6 hours, computed tomography (CT) perfusion ischemic tissue volumes are not required.1–3 Within the aforementioned trials, infarct and penumbra, defined as unsalvageable and salvageable tissue, respectively, thrombectomy eligible volume requirements vary based on the age, NIHSS score and CT perfusion ischemic tissue ratio for the patient.2–4 Additionally, the American Heart Association guidelines state mechanical thrombectomy can be utilized for patients with an occlusion of the M2 MCA, although benefits for these patients may be uncertain. 1
Within current clinical practice, the site of occlusion in CTA is determined by radiologists who are in charge of reading multiple medical images from all patients who enter their center. Since penumbra tissue can convert to infarct at a rate of 10.1 mL/hour, it is imperative that LVOs are detected as soon as possible to prevent increased loss of neurological function in AIS patients.5,6 However, since potential LVO cases are not prioritized, it can lead to an increased window between the time a patient receives baseline imaging and when their penumbra is salvaged through mechanical thombectomy. Therefore, a more streamlined method needs to be developed to optimize radiologists’ tasks in detecting LVOs, and the answer could lie with artificial intelligence.
Artificial intelligence based methods have shown great potential for localization of aneurysms, segmentation of infarct tissue, and prognosis of vascular lesion and surgical outcomes for delayed aneurysm occlusion.7–10 Recently, an artificial intelligence software for detecting LVOs was developed and implemented into the automation platform by Canon Medical Systems to provide a comprehensive stroke analysis for each patient. To optimize patient triaging to the endovascular intervention suite, this software immediately detects LVOs (ICA, M1 MCA and M2 MCA occlusions) using CTA imaging, and provides the LVO location in both axial and coronal slices for radiologists and neurointerventionalists to interpret.
In this study, we aimed to assess the performance of the AUTOStroke Solution LVO detection application in AIS patients. Additionally, we assessed the algorithm’s performance in comparison with clinical experts in a wide range of anterior circulation occlusions.
Methods
Patient inclusion
For this Health Insurance and Accountability retrospective study, institutional review board approval was obtained and informed consent was waived. In this study, we included patients with AIS symptoms upon comprehensive stroke center arrival. Each patient underwent non-contrast CT imaging to rule out hemorrhage, followed by CTA imaging. A total of 303 consecutive patients between March 2019 and February 2020 were included within this study, with 202 patients being allocated to the LVO cohort and 101 patients being allocated to the non-LVO cohort. Patients included in the LVO cohort had either: a distal ICA occlusion (n = 59), an M1 MCA occlusion (n = 82) or a proximal M2 MCA occlusion (n = 61). Not all LVO patients were required to have undergone mechanical thrombectomy as there may have been other contraindications making them ineligible. For those who underwent mechanical thrombectomy, thrombolysis in cerebral infarction (TICI) scores were recorded as a consensus by a neurosurgery attending and two endovascular fellows not involved in data collection. Non-LVO patients were required to have presented at the comprehensive stroke center with AIS symptoms and undergone AIS protocol imaging, but deemed negative for having any vessel occlusion.
CTA analysis
CTA data was collected using two Aquilion ONE CT units (Canon Medical Systems Corporation, Otawara, Japan). The CT stroke protocol includes the acquisition of non-contrast CT, CTA and CT perfusion scans. CTA volumes were reconstructed at 512 rows by 512 columns with an in-plane resolution of 0.4 mm and a slice thickness of 0.5 mm. Injection of 80 mL of Omnipaque 350 was conducted at a rate of 5 mL/second. Additional protocol parameters utilized were: a tube voltage of 120 kilovolt peak, a tube current ranging from 370 to 600 milliamperes, and a CT dose index of 7.3 milligray. The stroke protocol scans and reconstruction time interval was between 3 and 5 minutes for all patients.
After acquiring the CTA data, an operator with 2.5 years of experience examining contrast-enhanced CT data manually identified the occlusion site in each patient. This was conducted for comparison with the location of the suspected site of occlusion output from the automated LVO detection software. Automated analysis of each patient’s CTA was then conducted using the AUTOStroke Solution LVO application. Canon’s LVO detection application has the following image acquisition requirements to be used successfully: CTA of the head, axial acquisition only, a slice thickness no greater than 1.25 mm, a tube voltage of between 80 and 140 kilovolt peak, a field of view above 170 mm and a 512 by 512 image matrix. Additionally, the algorithm was trained on over 3000 cases and validated on 476 cases, which included distal ICA, M1 MCA and proximal M2 MCA occlusions.11,12 Owing to the occlusion locations used for the training set, the algorithm used is constrained only to the listed regions. Severe motion and metal artifacts are additionally known to impact the performance in being able to detect LVOs. Furthermore, the algorithm may not detect small MCA occlusions in the presence of good collateral flow, or in the event no contrast is visible due to poor timing of the contrast injection during the CTA acquisition protocol.11,12
Once CTA data were pushed to the LVO detection software, automated analysis was conducted, and results were exported to a workstation for interpretation. In the event an occlusion was not detected, the software simply exports the original CTA and a message stating “No findings suggestive of LVO identified.” In the event an LVO was detected, the software exports the original CTA, a resampled CTA with a slice thickness of 10 mm with the slice containing the occlusion stating “suspected LVO identified,” as well as a coronal slice with the suspected LVO. Slices suspected to contain the LVO based on the algorithm were compared with the manually identified occlusion site. In the event the occlusion site (arterial location or hemisphere) differed between the automated algorithm and the manual identification, it was documented that the LVO detection was unsuccessful.
Statistical analysis
Summary statistics for continuous variables and frequency distributions for categorical variables were tabulated for all analyzed patient data as well as for the LVO and non-LVO cohorts. All patient demographics were compared using Student’s t-test and chi-squared and ANOVA statistical testing. Proper launch of the LVO algorithm was documented for each case to ensure clinical workflow would not be hindered by the automated solution. For the LVO cohort, it was documented if the LVO algorithm correctly detected that an occlusion was present. Additionally, it was documented if the slice predicted to contain the occlusion contained the same region as what was indicated by the manual occlusion identification. Disagreements in arterial occlusion location or hemisphere side were documented. For the non-LVO cohort, it was similarly noted if the LVO application launched properly for each case and if an LVO was detected. In the event an LVO was detected, the suspected territory was noted along with any potential image artifacts or other vascular abnormalities, such as a stenosis, that may have caused the false positive result.
Utilizing both the LVO and non-LVO cohorts, accuracy, sensitivity, specificity, positive predictive value, negative predictive value, F1 score and Matthews correlation coefficient metrics were determined for the LVO detection algorithm. Accuracy, sensitivity and specificity represent the proportion of cases correctly labeled in both cohorts, the LVO cohort, and non-LVO cohort, respectively. Positive predictive value represents the proportion of correctly labeled LVO cases compared to the total number of predicted LVO positive cases, while negative predictive value represents the proportion of correctly labeled non-LVO cases compared to the total number of predicted non-LVO cases. F1 score indicates the proportion of correctly labeled LVO cases compared to the total number of true positive, half of the false positive and half of the false negative cases. Matthews correlation coefficient represents the correlation between the predicted and ground truth classifications, with 1 being a perfect correlation, zero being random assignment and −1 being complete disagreement between predictions and true labels. Subgroup analysis was additionally performed within the LVO cohort to assess the performance of the algorithm in detecting ICA, M1 MCA and M2 MCA occlusions individually. Subgroup analyses utilized all of the aforementioned metrics and included all non-LVO cases for calculations. In addition, the following times were documented as 95% confidence intervals: time from pushing the CTA data to the automation platform until the LVO application launched and time from pushing the CTA data to the automation platform until results were exported to the clinical workstation.
Results
Demographics for all patients, including LVO and non-LVO subcategories, are indicated in Table 1. Additionally, indicated in Table 1 are the proportions for each site of occlusion in the LVO cohort and the reperfusion status for the 76.2% (154/202) of LVO patients who underwent mechanical thrombectomy. Intravenous thrombolysis was administered to 40.2% (62/154) of the LVO patients who underwent mechanical thrombectomy at a median time of 115.0 minutes (interquartile range (IQR): 84.0–18.0 minutes) following symptom onset. Statistical significance (p < 0.05) was only seen between the LVO and non-LVO cohorts for the time since onset of symptoms to CTA imaging and NIHSS score variables based on Student’s t-test and chi-squared test.
Table 1.
Characteristics and outcomes of large vessel and non-large vessel occlusion patients.
| Characteristic | All (n = 303) | Large vessel occlusion patients (n = 202) | Non-large vessel occlusion patients (n = 101) |
|---|---|---|---|
| Male sex (%) | 45.2% (137/303) | 48.5% (98/202) | 38.6% (39/101) |
| Age, years, mean standard deviation, [median] (IQR) | 69.8 14.5 [70.0] (60.0–81.0) | 70.6 13.9 [71.0] (61.0–82.0) | 68.2 15.7 [68.0] (60.0–80.0) |
| NIHSS score, mean standard deviation, [median] (IQR) | 11.9 8.2 [11.0] (7.5–15.0) | 15.2 7.0 [15.0] (10.0–20.8) | 5.3 3.2 [4.0] (3.0–7.0) |
| Site of occlusion | |||
| Internal cerebral artery | … | 29.2% (59/202) | … |
| M1 middle cerebral artery | … | 40.6% (82/202) | … |
| M2 middle cerebral artery | … | 30.2% (61/202) | … |
| Time from onset of stroke to CTA imaging, minutes, mean standard deviation, [median] (IQR) | 657.0 2055.8 [212.0] (110.5–644.0) | 404.5 743.4 [186.0] (104.8–461.5) | 1162.0 3356.4 [302.0] (124.0–877.0) |
| Reperfusion cases | … | n = 154 | … |
| Time from onset of stroke to reperfusion, minutes, mean standard deviation, [median] (IQR) | … | 464.6 955.7 [242.0] (144.5–423.8) | … |
| TICI 0 | … | 3.2% (5/154) | … |
| TICI 1 | … | 2.6% (4/154) | … |
| TICI 2a | … | 5.8% (9/154) | … |
| TICI 2b | … | 35.1% (54/154) | … |
| TICI 2c | … | 22.7% (35/154) | … |
| TICI 3 | … | 30.5% (47/154) | … |
Empty cells correspond to non-applicable data based on category or combination of categories. Abbreviations: IQR: interquartile range; NIHSS: National Institute of Health Stroke Scale; CTA: computed tomography angiography; TICI: thrombolysis in cerebral infarction.
Subgroup patient demographics are indicated in Table 2 for patients with ICA, M1 MCA and M2 MCA occlusions. Additionally, Table 2 indicates the reperfusion status for the subgroup of patients with each type of occlusion that underwent mechanical thrombectomy. ANOVA statistical testing between all three subgroups indicated statistical significance only (p < 0.05) for the NIHSS score category. From Table 2 it can be seen that there is a direct correlation between the size of the occluded vessel, with ICA being the largest and M2 MCA being the smallest, and the severity of the NIHSS score calculated.
Table 2.
Characteristics and outcomes of large vessel occlusion patients based on occlusion site.
| Characteristic | ICA (n = 59) | MCA M1 (n = 82) | MCA M2 (n = 61) |
|---|---|---|---|
| Male sex, % | 45.8% (27/59) | 50.0% (41/82) | 49.2% (30/61) |
| Age, years, mean standard deviation, [median] (IQR) | 67.4 14.8 [68.0] (57.0–78.0) | 72.9 12.5 [74.0] (65.0–82.0) | 70.6 14.3 [69.0] (61.0–82.0) |
| NIHSS score, mean standard deviation, [median] (IQR) | 17.1 7.1 [17.0] (12.5–22.5) | 15.4 7.1 [15.0] (11.0–20.0) | 13.0 6.5 [14.0] (8.0–17.0) |
| Time from onset of stroke to CTA imaging, minutes, mean standard deviation, [median] (IQR) | 266.2 249.8 [166.0] (105.5–313.0) | 474.0 587.1 [253.5] (94.3–700.3) | 444.6 1139.6 [156.0] (114.0–373.0) |
| Reperfusion cases | n = 38 | n = 72 | n = 44 |
| Time from onset of stroke to reperfusion, minutes, mean standard deviation, [median] (IQR) | 302.4 249.8 [241.0] (135.5–400.8) | 532.1 624.4 [303.0] (159.3–640.3) | 494.2 1669.4 [200.5] (143.5–366.3) |
| TICI 0 | 7.9% (3/38) | 0.0% (0/72) | 4.5% (2/44) |
| TICI 1 | 5.3% (2/38) | 1.3% (1/72) | 2.3% (1/44) |
| TICI 2a | 0.0% (0/38) | 5.6% (4/72) | 11.4% (5/44) |
| TICI 2b | 31.6% (12/38) | 36.1% (26/72) | 36.4% (16/44) |
| TICI 2c | 21.1% (8/38) | 20.8% (15/72) | 27.3.% (12/44) |
| TICI 3 | 34.2% (13/38) | 36.1% (26/72) | 18.2% (8/44) |
Abbreviations: ICA: internal carotid artery; MCA: middle cerebral artery; IQR: interquartile range; NIHSS: National Institute of Health Stroke Scale; CTA: computed tomography angiography; TICI: thrombolysis in cerebral infarction.
In Table 3, proportions are indicated for the number of times the automated LVO application launched properly and how frequently the application correctly labeled a case as LVO positive or negative for the LVO and non-LVO cohorts. Chi-squared testing indicates statistical significance between the percentage of proper classifications for the LVO and non-LVO cohorts. Table 3 additionally indicates the times from pushing a case to the automated platform until the LVO application is launched and the results are received for interpretation. Student t-testing indicates a significant difference between processing time when there is an LVO present or there is not.
Table 3.
Proportion of instances where the LVO application was correctly launched and correctly classified the input case as LVO positive or negative. Time from pushing each case until the LVO application is launched and results are received are indicated as 95% confidence intervals.
| All (n = 303) | Large vessel occlusion patients (n = 202) | Non-large vessel occlusion patients (n = 101) | |
|---|---|---|---|
| Proper LVO application launch, % | 100.0% (303/303) | 100.0% (202/202) | 100.0% (101/101) |
| Proper classification of LVO presence, % | 81.2% (246/303) | 72.8% (147/202) | 98.0% (99/101) |
| Time from pushing case to LVO application launch, seconds | 26.6 0.7 | 26.3 0.9 | 27.1 0.8 |
| Time from pushing case to receiving LVO results, seconds | 69.8 1.1 | 71.5 1.5 | 66.5 1.1 |
Abbreviation: LVO: large vessel occlusion.
Three example cases of the LVO detection software correctly locating occlusions are shown in Figure 1. The top, middle and bottom rows indicate ICA, M1 MCA and M2 MCA occlusions, respectively, with the red boxes indicating the site of the occlusion in each case. The images in the top row of Figure 2 depict a case where an LVO was not present but the algorithm predicted the presence of one due to the poor timing of the contrast bolus injection. The images in the bottom row of Figure 2 demonstrate a false negative case where an LVO was present within the M1 segment of the MCA, but the software did not detect its presence. The LVO, as indicated by the red box, shows clear disruption to vessel flow.
Figure 1.
Correctly labeled large vessel occlusions (LVOs) using the LVO detection algorithm for the internal carotid artery (top row), M1 middle cerebral artery (middle row) and M2 middle cerebral artery (bottom row). Left and right columns represent axial and coronal views respectively for each case. The site of occlusion is indicated by a red box in each view for each case.
Figure 2.
Incorrectly labeled large vessel occlusion (LVO) and non-LVO cases based on the LVO detection algorithm output. The top row indicates axial and coronal views for a case without an LVO but was predicted to have an occlusion due to the poor contrast visible in the image. The bottom row shows a case where an LVO is present (indicated by red box), but the algorithm did not predict an LVO to have occurred in this patient.
Metrics indicated in Table 4 represent the overall ability of the LVO detection software to accurately label cases as LVO positive or negative. Within each occlusion site category, all 101 negative LVO cases were utilized to determine true negative and false positive totals to calculate the metrics in Table 4. Matthews correlation coefficient shows an increase in classification accuracy for the larger occlusion sites (ICA and M1 MCA) compared to smaller M2 MCA occlusions. Additionally, for all cases where an occlusion was predicted, the occlusion site matched the manually determined site in all but one case. In the aforementioned case, the occlusion was in the M1 MCA territory, but was predicted to be in the ICA territory.
Table 4.
Accuracy, sensitivity, specificity, positive predictive value, negative predictive value, F1 score and Matthews correlation metrics corresponding to the LVO detection algorithm for all and each occlusion site.
| All (n = 303) | ICA (n = 160) | MCA M1 (n = 183) | MCA M2 (n = 162) | |
|---|---|---|---|---|
| Accuracy | 0.81 | 0.95 | 0.89 | 0.80 |
| Sensitivity | 0.73 | 0.90 | 0.77 | 0.51 |
| Specificity | 0.98 | 0.98 | 0.98 | 0.98 |
| Positive predictive value | 0.99 | 0.96 | 0.97 | 0.94 |
| Negative predictive value | 0.64 | 0.94 | 0.84 | 0.77 |
| F1 score | 0.84 | 0.93 | 0.86 | 0.66 |
| Matthews correlation coefficient | 0.67 | 0.89 | 0.78 | 0.59 |
Abbreviations: LVO: large vessel occlusion; ICA: internal carotid artery; MCA: middle cerebral artery.
Discussion
This study provided both a quantitative and qualitative assessment of an artificial intelligence based LVO detection algorithm in AIS patients. Additionally, this study assessed the ability of the LVO algorithm in determining when an occlusion was not present in LVO negative patients. Rapid detection of an LVO is essential in clinical workflow to determine which patients are mechanical thrombectomy eligible based on the American Heart Association’s guidelines.1,13 Studies have shown that complete recanalization within the first 6 hours following symptom onset provides the best chance for good clinical outcome for AIS patients. 14 Being able to streamline clinical workflow through the use of artificial intelligence has the potential benefit of getting patients to endovascular intervention suites more rapidly for mechanical thrombectomy so that they can regain more neurological function. Furthermore, having an algorithm alerts clinicians to the presence and location of an occlusion, which can further streamline the process and potentially act as an educational tool for those in clinical training.
Patient demographic analysis in the study indicated a statistical significance (p < 0.05) in the time since onset of CTA imaging between the LVO and non-LVO cohorts. This is likely due the patient’s stroke-like symptoms being more severe for the LVO cohort causing patients to seek more immediate clinical assistance. Since the non-LVO cohort did not contain any stroke positive patients, symptoms were likely caused by some less serious underlying disease. Furthermore, subgroup analysis indicated the site of the occlusion statistically influenced the NIHSS score with the ICA category having the highest score and M2 MCA category having the lowest. This can be explained because an occlusion within the ICA can lead to the prevention of blood flow to the distal vessels such as the M1 and M2 MCA segments. Therefore, having blood flow cut off to more regions of the brain would lead to more severe stroke symptoms.
Application results from the study indicate the automated platform correctly launched the LVO application for each case loaded, which is essential to prevent the disruption of automated clinical workflow. In addition, the total processing time was deemed to be 71.5 seconds and 66.5 seconds for the LVO and non-LVO cohorts, respectively, with statistical significance indicated between the two cohorts. This significance occurs due to the software generating resampled 10-mm axial view CTAs, and a coronal view as output results when an LVO is present. Even though a significant difference is seen, the time to process a case in both instances is still less (60–78% faster) than the estimated 3-5 minutes required for CTAs to be read by a radiologist. 15
Metrics shown in Table 4 indicate that as the occlusion site vessel decreases in size, so does the ability of the algorithm to detect the presence of the LVO. The overall accuracy of the algorithm (0.81) is significantly decreased by the algorithm’s limitation in detecting M2 MCA occlusions. Furthermore, sensitivity (0.51) and negative predictive values (0.77) are drastically decreased for the M2 MCA category because of the number of occlusions not detected. This decrease in prediction ability for the M2 segment of the MCA is likely due to the decrease in the amount of contrast seen within the voxels containing these vessels, since they encompass a much smaller region compared to the ICA and M1 MCA segment. Since the intensity values in the image, along with location of the intensities corresponding to each other, are some of the main features artificial intelligence algorithms rely on, it is intuitive that an occlusion within a smaller vessel with less contrast would be a more difficult problem for a network to solve. 16 From the aforementioned information, it can then be concluded that more false negatives would occur for the M1 occlusions compared to ICA occlusions since M1 segments take up smaller regions than ICAs due to their size, meaning the network has a larger area of high-intensity values to detect when an ICA occlusion is present from contrast discontinuity. Although improved detection of occlusions within the M2 MCA territory is necessary, treatment guidelines by the American Heart Association only include recanalization of the M2 MCA segmented when symptom onset is less than 6 hours. 17 Furthermore, the guidelines state benefits from an M2 MCA recanalization are uncertain, indicating the ICA and M1 MCA results are of more clinical importance as explicit recanalization guidelines and benefits are listed for these occlusion types.1,13
Another reason the sensitivity for the M2 segments is so low is potentially due to the training split used for the algorithm. If the algorithm was trained mostly on ICA and M1 occlusions, it would make sense that M2 occlusions are tougher to predict the presence of as the algorithm would not have seen as many of them during training. Additionally, since the algorithm was not trained on distal M2 occlusions, the inclusion of this type of occlusion in the training set could aid in the detection of all types of M2 occlusions. Some potential ways to improve the algorithm’s ability to detect these occlusions would be to first increase the training set for the M1 and M2 categories and potentially utilize image pre-processing before training. Thresholding out the skull could improve detection ability of occlusions since both the skull and contrast have high-intensity values compared to surrounding tissue meaning the algorithm may confuse the two at times. Furthermore, adoptive thresholding to isolate the vessel from surrounding brain tissue could be utilized so the network is only training on the vessels and not having to parse through irrelevant brain tissue intensity values. This would be very beneficial for M2 occlusions since visual contrast is less pronounced between the vessel and surrounding tissue, meaning isolation of the vessel would simplify the network’s task. It should additionally be noted that the algorithm used to detect LVOs tracks the vessel from the ICA throughout the MCA. Therefore, if there is not a complete occlusion, the algorithm may not recognize the vessel as occluded, which may occur in the event of a partial occlusion. This is seen as an additional limitation to the algorithm and can be improved by implementing vessel diameter calculation during tracking to make sure the vessel is of substantial width. Additionally, of the 147 cases where an LVO was detected, only one output results in the detected occlusion site being different from the manually determined site. The manually determined site was deemed to be a right M1 MCA occlusion, but the LVO detection algorithm output results indicated the occlusion to be in the ICA territory. Poor timing of the contrast bolus injection is likely the reason for the incorrect region determination by the automated software. As previously mentioned, intensity values within an image are a major feature utilized, which explains the improper location seen.
Previously conducted studies have evaluated the performance of Viz.AI (San Francisco, CA) and RAPID (RapidAI, Menlo Park, CA) artificial intelligence algorithms in detecting LVOs. Both Viz.AI and RAPID grouped ICA and M1 MCA occlusions within their studies. When comparing Viz.AI and RAPID LVO results with the ICA and M1 MCA results from Canon’s automated LVO detection, it was found that all three algorithms had similar sensitivity results for ICA occlusions (∼0.90), while Canon’s software had a slightly lower sensitivity compared to Viz.AI and RAPID for M1 MCA occlusions. Furthermore, Canon’s software and Viz.AI were found to have drastically higher specificity and positive predictive values compared to RAPID (Viz.AI specificity = 0.83, positive predictive value = 0.82, RAPID specificity = 0.76, positive predictive value = 0.43).18,19 These higher specificity and positive predictive value metrics indicate Viz.AI and Canon’s software are more likely to identify cases as LVO negative when they do not have an LVO present. However, Canon’s software is also more likely to label M1 MCA occlusions as LVO negative in comparison with the other software as seen by each software’s negative predictive values (Canon M1 negative predictive value = 0.84, Viz.AI negative predictive value = 0.91, RAPID negative predictive value = 0.98). This is of great importance, since missing the presence of an occlusion is not clinically preferred over falsely identifying an occlusion. This is because a clinician can simply ignore the false positive occlusion prediction as opposed to the software entirely missing the occlusion and not allowing the patient to undergo reperfusion procedures. 18 , 19 In contrast, RAPID appears to show an overabundance of caution by labeling most cases as LVO positive based on the low positive predictive value and high negative predictive value metrics compared to the two other software. Further analysis was additionally conducted by RAPID, analyzing the ability of their software to detect proximal M2 MCA occlusions. The following metrics are seen with a high sensitivity of 0.86, low positive predictive value of 0.14 and high negative predictive value of 0.99. 19 This shows a similar trend in overestimating the number of LVOs present compared to Canon’s software, which tends to underestimate the number of LVOs. Again, this overestimation of number of LVOs present by RAPID software is preferred compared to Canon’s for M2 occlusions, as it indicates RAPID is less likely to miss the presence of an occlusion, meaning more eligible patients will undergo reperfusion procedures.
Limitations of this study include not comparing the specific cases utilized in the study with other commercially available software. As previously stated, Viz.AI and RAPID have had artificial intelligence algorithms developed for them, but these were not available to us for this study. A future comparison of LVO detection ability and how quickly each software streamlines clinical workflow should be conducted. Utilization of only one operator to determine the occlusion site is a second limitation to this study. One operator was utilized to decrease the variability in occlusion site determination compared to multiple operators determining the occlusion site for different portions of the dataset. However, utilizing multiple operators and a consensus system would have decreased the likelihood that any ground truth occlusion site would be labeled incorrectly. Another limitation to this study is no validation cases from an outside institution were used to assess Canon’s automated LVO detection algorithm. An additional limitation of the study is no patients with multiple occlusions were included. Although Canon’s LVO detection algorithm is stated to be capable of detecting multiple occlusions within a single patient, extensive studies with a proper sample size for both bilateral and tandem occlusions should be conducted in the future. The split of approximately two-thirds of the patients being in the LVO cohort and approximately one-third being in the non-LVO cohort is another limitation to the study. This is because the lower number of non-LVO patients could impact the negative and positive predictive value results, since there are fewer chances for true negative and false positives to occur. Although enough patients were included in the non-LVO cohort to ensure statistically significant claims could be made, the reasoning for the lower non-LVO sample size is that there is a greater concern for false negatives in clinical practice compared to false positives. Therefore, since false negatives occur within the LVO cohort, more patients were included for this cohort.
Furthermore, no posterior circulation or medium vessel occlusion patients (MCA M3 segment, anterior cerebral artery A2 or A3 segments) were included in the study, as this artificial intelligence algorithm is highly dependent on the training data utilized. Since this algorithm was only trained on ICA, MCA M1 and MCA M2 occlusions, it would be nearly impossible for it to detect other types of occlusions it has not seen before. However, since it is possible future trials may show benefit to endovascular treatment of medium vessel occlusions, a future study should be conducted to train a neural network on medium vessel occlusion and posterior circulation occlusion patients to automatically predict their presence and streamline patient transfer to endovascular suites. A future multicenter validation study for both training and testing of the algorithm would improve the algorithm, creating a more generalized and robust method. A final limitation to this study is that the degree of head movement was not quantified to see if that had any impact on the results of the study.
Conclusions
Canon’s AUTOStroke Solution LVO application was able to accurately identify ICA and M1 MCA occlusions in addition to nearly perfectly ruling out when an LVO was not present. M2 MCA occlusion detection needs further improvement based on the sensitivity results displayed by the LVO detection algorithm. Furthermore, having an automated method integrated with the onsite CT system provided a rapid stroke solution for comprehensive stroke centers.
Supplemental Material
Supplemental material, sj-pdf-1-neu-10.1177_1971400921998952 for Validation of an artificial intelligence-driven large vessel occlusion detection algorithm for acute ischemic stroke patients by Ryan A Rava, Blake A Peterson, Samantha E Seymour, Kenneth V Snyder, Maxim Mokin, Muhammad Waqas, Yiemeng Hoi, Jason M Davies, Elad I Levy, Adnan H Siddiqui and Ciprian N Ionita in The Neuroradiology Journal
Footnotes
Conflict of interest: Kenneth V. Snyder is consulting for: Canon Medical Systems Corporation, Penumbra Inc., Medtronic, Jacobs Institute; and is a co-founder of Neurovascular Diagnostics, Inc. Max Mokin is a consultant for: Canon Medical Systems, Cerebrotech, Imperative care; and has NIH grant support (R21NS109575). Adnan H. Siddiqui has financial interest/investor/stock options/ownership in: Amnis Therapeutics, Apama Medical, Blink TBI Inc., Buffalo Technology Partners Inc., Cardinal Consultants, Cerebrotech Medical Systems, Inc. Cognition Medical, Endostream Medical Ltd., Imperative Care, International Medical Distribution Partners, Neurovascular Diagnostics Inc., Q’Apel Medical Inc, Rebound Therapeutics Corp., Rist Neurovascular Inc., Serenity Medical Inc., Silk Road Medical, StimMed, Synchron, Three Rivers Medical Inc., Viseon Spine Inc; is a consultant/advisory board for: Amnis Therapeutics, Boston Scientific, Canon Medical Systems USA Inc., Cerebrotech Medical Systems Inc., Cerenovus, Corindus Inc., Endostream Medical Ltd., Guidepoint Global Consulting, Imperative Care, Integra LifeSciences Corp., Medtronic, MicroVention, Northwest University–DSMB Chair for HEAT Trial, Penumbra, Q’Apel Medical Inc., Rapid Medical, Rebound Therapeutics Corp., Serenity Medical Inc., Silk Road Medical, StimMed, Stryker, Three Rivers Medical, Inc., VasSol, W.L. Gore & Associates; and is a principal investigator/steering comment of the following trials: Cerenovus NAPA and ARISE II; Medtronic SWIFT PRIME and SWIFT DIRECT; MicroVention FRED & CONFIDENCE; MUSC POSITIVE; Penumbra 3D Separator, COMPASS, INVEST. Yiemeng Hoi is an employee of Canon Medical Systems USA, Inc. Jason M. Davies has received research grants from the National Center for Advancing Translational Sciences of the National Institutes of Health under award number KL2TR001413 to the University at Buffalo and is a shareholder of RIST Neurovascular. Elad I. Levy owns stock in: NeXtGen Biologics, RAPID Medical, Claret Medical, Cognition Medical, Imperative Care (formerly the Stroke Project), Rebound Therapeutics, StimMed, Three Rivers Medical; a national principal investigator/steering committees for: Medtronic (merged with Covidien Neurovascular) SWIFT Prime and SWIFT Direct Trials; a consultant for: Claret Medical, GLG Consulting, Guidepoint Global, Imperative Care, Medtronic, Rebound, StimMed; on the advisory board for: Stryker (AIS Clinical Advisory Board), NeXtGen Biologics, MEDX, Cognition Medical, Endostream Medical; and is a site principal investigator for: CONFIDENCE study (MicroVention), STRATIS Study—Sub I (Medtronic). Ciprian N. Ionita has received an equipment grant from Canon Medical Systems and from the Cummings Foundation.
Ethical approval: All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee (University at Buffalo Institutional Review Board, IRB ID: MOD00005807) and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Funding: The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was partially funded by Canon Medical Systems USA, Inc.
Informed consent: Informed consent was waived through Institutional Review Board approval.
ORCID iDs: Ryan A Rava https://orcid.org/0000-0001-6456-8445
Ciprian N Ionita https://orcid.org/0000-0001-7049-0592
References
- 1.Powers WJ, Rabinstein AA, Ackerson T, et al. 2018 guidelines for the early management of patients with acute ischemic stroke: a guideline for healthcare professionals from the American Heart Association/American Stroke Association. Stroke 2018; 49: 46–99. DOI: 10.1161/STR.0000000000000158.29203686 [Google Scholar]
- 2.Nogueira RG, Jadhav AP, Haussen DC, et al. Thrombectomy 6 to 24 hours after stroke with a mismatch between deficit and infarct. New Engl J Med 2018; 378: 11–21. [DOI] [PubMed] [Google Scholar]
- 3.Albers GW, Marks MP, Kemp S, et al. Thrombectomy for stroke at 6 to 16 hours with selection by perfusion imaging. New Engl J Med 2018; 378: 708–718. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Miles K, Eastwood JD, Konig M. Multidetector computed tomography in cerebrovascular disease: CT perfusion imaging. UK: CRC Press, 2007. [Google Scholar]
- 5.Guenego A, Mlynash M, Christensen S, et al. Hypoperfusion ratio predicts infarct growth during transfer for thrombectomy. Ann Neurol 2018; 84: 616–620. DOI: 10.1002/ana.25320. [DOI] [PubMed] [Google Scholar]
- 6.Rava R, Snyder K, Mokin M, et al. Assessment of a Bayesian Vitrea CT Perfusion Analysis to predict final infarct and penumbra volumes in patients with acute ischemic stroke: a comparison with RAPID. Am J Neuroradiol 2020; 41: 206–212. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Podgorsak AR, Rava RA, Bhurwani MMS, et al. Automatic radiomic feature extraction using deep learning for angiographic parametric imaging of intracranial aneurysms. J Neurointerv Surg 2020; 12: 417–421. [DOI] [PubMed] [Google Scholar]
- 8.Rava R, Podgorsak A, Waqas M, et al. Use of a convolutional neural network to identify infarct core using computed tomography perfusion parameters. In: SPIE Medical Imaging. San Diego, CA: SPIE, 2021. [DOI] [PMC free article] [PubMed]
- 9.Bhurwani MMS, Waqas M, Podgorsak AR, et al. Feasibility study for use of angiographic parametric imaging and deep neural networks for intracranial aneurysm occlusion prediction. J Neurointerv Surg 2020; 12: 714–719. [DOI] [PubMed] [Google Scholar]
- 10.Rava RA, Mokin M, Snyder KV, et al. Performance of angiographic parametric imaging in locating infarct core in large vessel occlusion acute ischemic stroke patients. J Med Imaging 2020; 7: 016001. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Stephens K. Avicenna. AI, Canon Link Up for AI Stroke Detection. Avicenna, 2020. Available at: https://axisimagingnews.com/market-trends/cloud-computing/machine-learning-ai/avicenna-ai-canon-link-ai-stroke-detection
- 12.CINA User Guide. Avicenna. AI AV-DP-CINA-10-013-SUM-USER-AP-V01-EN, 2020: 8–13.
- 13.Powers WJ, Rabinstein AA. Response by Powers and Rabinstein to letter regarding article, “2018 Guidelines for the early management of patients with acute ischemic stroke: a guideline for healthcare professionals from the American Heart Association/American Stroke Association”. Stroke 2019; 50: e277–e278. [DOI] [PubMed] [Google Scholar]
- 14.Rha J-H, Saver JL. The impact of recanalization on ischemic stroke outcome: a meta-analysis. Stroke 2007; 38: 967–973. [DOI] [PubMed] [Google Scholar]
- 15.Fasen BA, Heijboer RJ, Hulsmans F-JH, et al. Radiology workload in clinical implementation of thrombectomy for acute ischemic stroke: experience from The Netherlands. Neuroradiology 2020; 62: 877–882. [DOI] [PubMed] [Google Scholar]
- 16.Fernandez AGMLS, Bunke RBH, Schmiduber J. A novel connectionist system for improved unconstrained handwriting recognition. IEEE 2009; 31. [DOI] [PubMed] [Google Scholar]
- 17.Powers WJ, Rabinstein AA, Ackerson T, et al. Guidelines for the early management of patients with acute ischemic stroke: 2019 update to the 2018 guidelines for the early management of acute ischemic stroke: a guideline for healthcare professionals from the American Heart Association/American Stroke Association. Stroke 2019; 50: e344–e418. [DOI] [PubMed] [Google Scholar]
- 18.Barreira C, Bouslama M, Lim J, et al. E-108 Aladin study: automated large artery occlusion detection in stroke imaging study–a multicenter analysis. J Neurointerv Surg 2018; 10(Suppl 2). [Google Scholar]
- 19.Amukotuwa SA, Straka M, Smith H, et al. Automated detection of intracranial large vessel occlusions on computed tomography angiography: a single center experience. Stroke 2019; 50: 2790–2798. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supplemental material, sj-pdf-1-neu-10.1177_1971400921998952 for Validation of an artificial intelligence-driven large vessel occlusion detection algorithm for acute ischemic stroke patients by Ryan A Rava, Blake A Peterson, Samantha E Seymour, Kenneth V Snyder, Maxim Mokin, Muhammad Waqas, Yiemeng Hoi, Jason M Davies, Elad I Levy, Adnan H Siddiqui and Ciprian N Ionita in The Neuroradiology Journal