e-ASPECTS software improves interobserver agreement and accuracy of interpretation of aspects score

Abstract

Introduction

There is increased interest in the use of artificial intelligence-based (AI) software packages in the evaluation of neuroimaging studies for acute ischemic stroke. We studied whether, compared to standard image interpretation without AI, Brainomix e-ASPECTS software improved interobserver agreement and accuracy in detecting ASPECTS regions affected in anterior circulation LVO.

Methods

We included 60 consecutive patients with anterior circulation LVO who had TICI 3 revascularization within 60 minutes of their baseline CT. A total of 16 readers, including senior neuroradiologists, junior neuroradiologists and vascular neurologists participated. Readers interpreted CT scans on independent workstations and assessed final ASPECTS and evaluated whether each individual ASPECTS region was affected. Two months later, readers again evaluated the CT scans, but with assistance of e-ASPECTS software. We assessed interclass correlation coefficient for total ASPECTS and interobserver agreement with Fleiss’ Kappa for each ASPECTS region with and without assistance of the e-ASPECTS. We also assessed accuracy for the readers with and without e-ASPECTS assistance. In our assessment of accuracy, ground truth was the 24 hour CT in this cohort of patients who had prompt and complete revascularization.

Results

Interclass correlation coefficient for total ASPECTS without e-ASPECTS assistance was 0.395, indicating fair agreement compared, to 0.574 with e-ASPECTS assistance, indicating good agreement (P < 0.01). There was significant improvement in inter-rater agreement with e-ASPECTS assistance for each individual region with the exception of M6 and caudate. The e-ASPECTS software had higher accuracy than the overall cohort of readers (with and without e-ASPECTS assistance) for every region except the caudate.

Conclusions

Use of Brainomix e-ASPECTS software resulted in significant improvements in inter-rater agreement and accuracy of ASPECTS score evaluation in a large group of neuroradiologists and neurologists. e-ASPECTS software was more predictive of final infarct/ASPECTS than the overall group interpreting the CT scans with and without e-ASPECTS assistance.

Introduction

Patient selection for mechanical thrombectomy in acute ischemic stroke relies heavily on imaging selection, in particular the Alberta Stroke Program Early CT Score (ASPECTS).^1–6 ASPECTS is a semiquantitative scoring system for early ischemic changes on non-contrast CT. In ASPECTS, each of the 10 regions of the middle cerebral artery (MCA) territory of the affected hemisphere are assigned a score of 0 in the presence of early ischemic changes including parenchymal hypoattenuation, loss of grey-white differentiation with or without focal swelling versus a score of 1 if the brain appears normal. ⁷ This score has been extensively validated and has been found to be useful for prognostication of long-term functional outcome following stroke. ⁸

While ASPECTS is now one of the major selection criteria for mechanical thrombectomy in the setting of large vessel occlusion, it does suffer limitations. First, there is a wide range in the reported rates of interobserver and intra-observer agreement in the scoring of ASPECTS.^9–11 Second, chronic infarcts or adjacent white matter disease may confound scoring. Because ASPECTS relies on detection of subtle reductions in parenchymal attenuation, some of which may be beyond the ability of a human reader to consistently detect, grading of ASPECTS is a potentially excellent application of machine learning and artificial intelligence based imaging analysis. Developing algorithms which can accurately detect infarcts for ASPECTS scoring and improve interobserver agreement is important as it would allow for more consistent grading of ASPECTS and more consistent triage of patients. In order to study the potential applicability of such an algorithm to grading of ASPECTS in improving accuracy and interobserver agreement, we performed a multireader study in which readers graded ASPECTS with and without the assistance of an automated algorithm (e-ASPECTS, Brainomix, Oxford, UK). We hypothesized that the use of the algorithm would improve interobserver agreement and accuracy among the reader cohort.

Methods

Study subjects

Following Institutional Review Board approval, we identified all patients who underwent mechanical thrombectomy for acute ischemic stroke at our center from 2015-2018. Among these patients, we identified the following cohort of patients: 1) age ≥18 years, 2) availability of non-contrast CT performed within 30 minutes of groin puncture, 3) presence of anterior circulation large vessel occlusion (i.e. ICA, M1 or M2), 4) TICI 3 revascularization within 30 minutes of groin puncture as determined by two neuroradiologists and 5) availability of non-contrast CT or MRI images at 24–36 hours following mechanical thrombectomy. Exclusion criteria included: 1) NCCTs with excessive artifacts, 2) patients with poor collaterals. Following identification of cases, the neuroradiologist selecting cases identified only those cases in which there was no extension of the infarct from the baseline CT to the final 24 hours CT.

e-ASPECTS

e-ASPECTS scoring (Brainomix, Oxford, UK; version 8) was performed as previously described.^12,13 Briefly, pseudonymized DICOM images were pre-processed and corrected for tilt, rotation and other postitional transformations. Based on a machine learning algorithm, ASPECTS regions were then automatically segmented and classified by the algorithm as ischemic or normal-appearing. When the algorithm detected non-acute hypodensity, it was counted as not-affected for the purposes of this comparison.

Manual ASPECTS scoring

In the same set of NCCT images, ASPECTS was manually determined by 16 physicians who were instructed in the appropriate use of ASPECTS. In this cohort of readers there were six experienced neuroradiologists (>3 years of post-fellowship experience as full-time neuroradiologists), six junior neuroradiologists (<3 years of post-fellowship experience as full-time neuroradiologists) and four neurologists, including 3 vascular neurologists and one critical care neurologist with 1–5 years of post-fellowship experience. The assessors used all available slices rather than just one supraganglionic and one ganglionic as per the original methodology and were able to adjust window-level settings as needed. Early ischemic change was defined as the presence of hypodensity and/or loss of grey-white differentiation with or without cortical swelling. Assessors were blinded to the clinical details other than the presence of unilateral anterior circulation ischemic stroke and suspected laterality.

For the purposes of this study, each of the readers assessed ASPECTS by labeling each affected area. After this, the overall score was tabulated. First, readers interpreted all 60 scans without the assistance of the e-ASPECTS software. After a 2 month waiting period, all the readers again interpreted the scans, but this time with the assistance and availability of the e-ASPECTS software. The readers had access to all features of the software.

Gold standard

Thus, we established the 24 hour CT or MRI final ASPECTS as the gold standard. Because all patients had TICI 3 revascularization within 60 minutes of the non-contrast CT and had moderate to good collaterals we assumed that there would be minimal to no infarct progression. The final ASPECTS was assessed by two readers by consensus. This final ASPECTS was used to establish accuracy of the readers and the e-ASPECTS software.

Statistical analysis

Comparison of performance with and without e-ASPECTS assistance was conducted for all 10 regions separately and for composite ASPECTS scoring. Comparisons were quantified as raw agreement, Fleiss’ Kappa, and intraclass correlation coefficient. With regards to kappa score categorization, <0 indicated no agreement, 0–0.20 indicated slight agreement, 0.21–0.40 indicated fair agreement, 0.41–0.60 indicated moderate agreement, 0.61–0.80 indicated substantial agreement and 0.81–1.0 indicated excellent agreement. We also performed a prevalence-adjusted bias-adjusted kappa (PABAK). We also performed subgroup analyses of interobserver agreement based on level of experience/specialty dividing readers into experienced neuroradiologists, junior neuroradiologists and neurologists. We also assessed the accuracy of overall ASPECTS and per-region ASPECTS for the readers with and without the e-ASPECTS assistance as well as the accuracy of the e-ASPECTS software itself. For the assessment of accuracy, it was determine on a per-region basis. Accuracy was pooled across all readers and calculated as (Number of True Negative Reads+True Positive Reads)/(Total Number of Reads). Accuracy with and without e-ASPECTS assistance was calculated using a chi-squared test. In order to estimate how often the e-ASPECTS software would lead to change in whether or not a patient met AHA guidelines for inclusion in mechanical thrombectomy, we calculated the proportion of cases in which readers changed their read from a dichotomized ASPECTS 0–5 to 6–10 and vice versa. Using a Bonferroni correction, statistical significance was set at P < 0.001 (40 total hypotheses tested). Statistical analysis was performed using R.

Results

Patient population

In our retrospective database of 202 patients, a total of 82 patients had TICI 3 revascularization with thrombectomy. Of these patients, 60 patients met inclusion criteria with prompt revascularization within 30 minutes of groin puncture and moderate to good collaterals on CTA. Mean age was 67.3±16.3 years and 32 patients (53.3%) were female. Thirty-one patients had left sided occlusions and 29 patients had right sided occlusions. Median initial NIHSS was 18 (IQR = 10–22). The median number of passes was 1 (IQR = 1–2) and 35 patients had first pass recanalization (58.3%), 19 patients required two passes (31.7%) and six patients had three passes (10.0%).

ASPECTS agreement with and without e-ASPECTS assistance

In terms of overall ASPECTS score agreement among the 16 reviewers, adjusted intraclass correlation coefficient was 0.48 without the assistance of the e-ASPECTS software and 0.68 with the assistance of the e-ASPECTS software. On a per-region basis, there was improvement in absolute agreement and Fleiss’ Kappa for all regions and no overlap in the confidence intervals for Fleiss kappa for 8 of 10 regions, with the exclusion of M6 and caudate. For example, kappa improved from 0.60 (95%CI = 0.50–0.69) to 0.83 (95%CI = 0.76–0.90) with the assistance of the AI software. For internal capsule kappa improved from 0.62 (95%CI = 0.53–0.72) to 0.82 (95%CI = 0.76–0.89). Prevalence adjusted and bias adjusted kappa values also improved with e-ASPECTS assistance. These data are summarized in Table 1.

Table 1.

Interobserver agreement with and without e-ASPECTS assistance.

	No e-ASPECTS assistance			Yes e-ASPECTS asistance
	% Overall agreement	Kappa (95%CI)	PABAK (95%CI)	% Overall agreement	Kappa (95%CI)	PABAK (95%CI)
M1	78%	0.60 (0.50–0.69)	0.68 (0.66–0.70)	92%	0.83 (0.76–0.90)	0.90 (0.87–0.93)
M2	69%	0.38 (0.29–0.47)	0.47 (0.44–0.50)	83%	0.67 (0.58–0.75)	0.78 (0.75–0.81)
M3	76%	0.52 (0.43–0.61)	0.61 (0.58–0.64)	86%	0.72 (0.63–0.81)	0.81 (0.77–0.85)
M4	84%	0.68 (0.60–0.75)	0.74 (0.72–0.76)	94%	0.88 (0.82–0.93)	0.94 (0.92–0.96)
M5	72%	0.44 (0.36–0.52)	0.50 (0.48–0.52)	83%	0.66 (0.58–0.74)	0.68 (0.66–0.70)
M6	87%	0.74 (0.66–0.83)	0.80 (0.78–0.82)	92%	0.83 (0.76–0.91)	0.92 (0.90–0.94)
Caudate	86%	0.71 (0.63–0.79)	0.76 (0.75–0.77)	89%	0.78 (0.70–0.87)	0.81 (0.80–0.83)
Lentiform	72%	0.44 (0.33–0.54)	0.51 (0.48–0.54)	83%	0.65 (0.56–0.75)	0.78 (0.76–0.80)
Insula	68%	0.35 (0.27–0.44)	0.46 (0.44–0.48)	79%	0.57 (0.47–0.67)	0.65 (0.63–0.67)
Internal Capsule	81%	0.62 (0.53–0.72)	0.72 (0.70–0.74)	91%	0.82 (0.76-0.89)	0.88 (0.87–0.89)

Agreement of raters with e-ASPECTS software

In terms of overall ASPECTS score agreement among the 16 reviewers, overall mean non-weighted agreement statistic between the reviewers and the e-ASPECTS software without the assistance of the e-ASPECTS software was 0.11 (95%CI = 0.02–0.19) and this increased to 0.33 (95%CI = 0.20–0.46) with the assistance of the e-ASPECTS software. Overall weighted agreement statistic between the reviewers and e-ASPECTS software without the e-ASPECTS software assistance was 0.25 (95%CI = 0.20–0.30) and this increased to 0.55 (95%CI = 0.47–0.63) with the assistance of the e-ASPECTS software.

Final infarct determination

Median ASPECTS score was 8 (IQR = 7–10) on the 24 hour CT or MRI. Fifteen patients had an ASPECTS of 10, 11 patients had an ASPECTS of 9, 13 patients had an ASPECTS of 8, 11 patients had an ASPECTS of 7 and 4 patients had an ASPECTS of 6. Six patients had a low ASPECTS (5 or less). The prevalence of infarct by affected region was as follows: M1 (13.3%, 8/60), M2 (25.0%, 15/60), M3 (13.3%, 8/60), M4 (3.3%, 2/60), M5 (13.3%, 8/60), M6 (10.0%, 6/60), Caudate (30.0%, 18/60), Lentiform Nucleus (51.7%, 31/60), Insula (51.7%, 31/60) and Internal Capsule (0.0%, 0/60).

Accuracy

Overall accuracy of the readers improved with the use of the e-ASPECTS software for every region except the caudate and lentiform (P < 0.001 for all estimates). For example, overall accuracy increased from 85.1% to 95.0% for the internal capsule with the use of e-ASPECTS software. Accuracy for the insula increased from 68.4% to 79.1% and accuracy for the M2 territory increased from 69.0% to 83.6%.

When compared to the overall accuracy of the readers with and without the assistance of e-ASPECTS, the e-ASPECTS software alone had higher accuracy for every region except the M4 region, caudate and lentiform nucleus (P < 0.05 for all other estimates). These findings are summarized in Table 2.

Table 2.

Accuracy with and without e-ASPECTS assistance.

	Accuracy
	All raters overall no e-ASPECTS	All raters overall with e-ASPECTS	e-ASPECTS
M1	81%	87%*	90%
M2	69%	84%*	92%ϯ
M3	81%	87%*	93%ϯ
M4	88%	94%*	95%
M5	78%	86%*	93%ϯ
M6	87%	92%*	95%
C	80%	79%	73%
L	67%	72%	72%
In	68%	79%*	85%ϯ
IC	85%	95%*	100%ϯ

*P<0.001 when compared to no e-ASPECTS interpretation.

^ϯ P<0.001 when compared to all raters overall with e-ASPECTS.

Sensitivity analysis by level of experience

Accuracy data by region affected and level of experience/training are shown in Table 3. For senior neuroradiologists overall accuracy was 76.7% without e-ASPECTS software and 84.0% with the use of e-ASPECTS software (P < 0.01). For junior neuroradiologists, accuracy was 81.7% without e-ASPECTS software and 87.1% with e-ASPECTS software (P < 0.01). For neurologists, accuracy was 75.5% without e-ASPECTS software and 84.9% with e-ASPECTS software (P < 0.01). Overall, the regions with the greatest increase in accuracy with the use of e-ASPECTS software were the M2, M3 and internal capsule. There was no improvement in accuracy in the caudate region with the use of e-ASPECTS software.

Table 3.

Accuracy with and without e-ASPECTS assistance by training and level of experience.

	Senior neuroradiologists without e-ASPECTS	Senior neuroradiologist with e-ASPECTS	Junior neuroradiologist without e-ASPECTS	Junior neuroradiologist with e-ASPECTS	Neurologist without e-ASEPCTS	Neurologist with e-ASPECTS
M1	77 (73–81)	87 (83–90)*	82 (78–86)	89 (85–93)	83 (78–88)	86 (81–91)
M2	66 (61–70)	81 (77–85)*	73 (69–77)	86 (82–90)*	68 (63–73)	84 (80–88)*
M3	75 (71–79)	84 (80–88)	87 (83–91)	89 (85–93)	80 (76–84)	89 (85–93)
M4	88 (84–92)	94 (90–98)	89 (85––93)	93 (89–97)	85 (81–88)	94 (90–98)*
M5	78 (74–82)	83 (79–87)	86 (82–90)	91 (87–95)	66 (60–72)	83 (78–88)*
M6	87 (83–91)	91 (88–94)	89 (85–93)	92 (88–96)	84 (80–88)	93 (89–97)*
Caudate	80 (76–84)	77 (73–81)	82 (78–86)	81 (77–85)	77 (72––82)	79 (74–84)
Lentiform	66 (61–71)	70 (65–75)	71 (66–76)	74 (70–78)	61 (57–65)	71 (67–75)*
Insula	68 (64–72)	80 (76–84)*	71 (67–75)	80 (76–84)*	66 (62–68)	77 (73–81)*
Internal capsule	82 (78–86)	93 (90–96)*	88 (84–92)	98 (94–100)*	86 (82–90)	94 (91–97)*

*P<0.001 when compared to “without e–ASPECTS read.”

All values reported as % (95%CI).

Change in dichotomized ASPECTS

For the overall reader cohort, the mean percentage of times in which the estimated ASPECTS changed from 0–5 to 6–10 and vice versa was 11.9% (mean number of 7.125/60 reads, range = 1–18.) The mean percentage of times in which the estimated ASPECTS changed from 6–10 to 0–5 was 2.7% (mean number of 1.625/60 reads, range = 0–6) and the mean percentage of times in which the estimated ASPECTS changed from 0–5 to 6–10 was 9.2% (mean number of 5.5/60 reads, range 1–17).

Based on level of experience and training, the rate of change from ASPECTS 0–5 to 6–10 and vice versa was 15.3% for senior neuororadiologists, 9.7% for junior neuroradiologists and 9.7% for neurologists. The rate of change from ASPECTS 6–10 to ASPECTS 0–5 was 3.1% for senior neuroradiologists, 3.3% from junior neuroradiologists and 1% for neurologists. The rate of change from ASPECTS 0–5 to ASPECTS 6–10 was 12.2% for senior neuroradiologists, 6.4% for junior neuroradiologists and 9.0% for neurologists.

Discussion

Our multi-reader study evaluating accuracy and interobserver agreement of ASPECTS with and without the assistance of a commercial machine learning software program demonstrated a number of interesting findings. First, the use of the Brainomix e-ASPECTS software package resulted in significant improvements in interobserver agreement for assessment of the overall ASPECTS score as well as eight of the 10 individual regions with non-overlapping confidence intervals in the Fleiss’ kappa. Second, the use of e-ASPECTS assistance improved overall reader accuracy for every region with the exception of the caudate nucleus and lentiform nucleus and this was true across all levels of experience. Lastly, and perhaps most surprisingly, we found that the e-ASPECTS software alone had a higher accuracy than the overall reader cohort of neuroradiologists and neurologists. By improving the interobserver agreement and accuracy of neuroradiologists and neurologists in their interpretation of ASPECTS, this automated detection algorithm could potentially help in more accurately and efficiently triaging patients for thrombectomy in stroke. From a clinical decision-making standpoint, CT interpretation with the assistance of the e-ASPECTS software resulted in an overall average rate of change from ASPECTS 0–5 to 6–10 and vice versa of 12% with the vast majority of these changes going from an ASPECTS of 0–5 to one of 6–10.

Previous performance comparisons between older versions of e-ASPECTS and neuroradiologist interpretation of ASPECTS have been performed. Herweh et al evaluated the e-ASPECTS in 34 cases and found it to be superior to trainees and non-inferior to neuroradiologists when compared to a DWI gold standard. ¹⁴ This particular analysis was done on a per-region basis as well as dichotomized (0–5 and 6–10) scores. In a later study, this same research group studied a cohort of 132 patients with follow-up CT as the measure of true ASPECTS and again found the e-ASPECTS to be non-inferior to neuroradiologists. ¹² However, because less than 3% of patients were treated with mechanical thrombectomy and the remainder with tPA, the use of a follow-up DWI, without any knowledge of the timing or extent of recanalization, is a suboptimal reference standard for tissue viability at presentation. Conversely, our study population was enriched with patients in whom near-immediate, complete recanalization was achieved, offering a much more compelling reference standard compared to these prior studies. ¹²

Other, previously published region-based comparisons between e-ASPECTS and manual scoring have found that e-ASPECTS was more sensitive and less specific than human scorers and an alternative software solution (RAPID) in the cortex, but less sensitive and more specific in deep regions. ¹⁵ In multiple prior studies it has been shown that e-ASPECTS did not detect any internal capsule infarcts.^14,16,17 Interestingly, in our study, we found no internal capsule infarcts at 24 hours, which is likely a reflection of the fact that the vascular supply to the internal capsule is primarily from the anterior choroidal artery rather than the middle cerebral artery and even in the case of ICA terminus occlusions, the anterior choroidal artery is usually patent.

To our knowledge, our study is the only one to date to examine how the use of the software package could improve accuracy and interobserver agreement for assessment of ASPECTS across a broad group of neuroradiologists and neurologists. This particular finding is important as many in the artificial intelligence community discuss the potential utility of a workflow model in which clinicians work with the assistance of AI to allow for more accurate, consistent and timely interpretation. This is especially important in settings where radiologists may not be available for image interpretation and reads are dependent on the neurologist taking care of the patient. In our study we found that when using the e-ASPECTS software, neurologists had an equivalent diagnostic accuracy compared to both senior and junior neuroradiologists. Lastly, while we did find that e-ASPECTS by itself was more accurate than the overall cohort of readers for most of the regions in determining final infarct; it may be that with improved understanding of how such algorithms work to detect infarcts, the a model in which interpretations are made by radiologists with the assistance of an AI algorithm (aka a centaur model) could result in improved sensitivity and specificity in ASPECTS quantification. Our study does suggest however that readers should, in general, trust the e-ASPECTS output unless there are clear reasons to doubt the algorithm (i.e. excessive patient potion, significant streak artifact, counting of clearly chronic infarcts, etc). Ultimately, we believe that the centaur model should be used for all AI-based algorithms as even with automated CT perfusion software it is necessary to look at the source images and input functions to see whether the maps can be trusted.

It is important to point out that the use of the e-ASPECTS software did not just improve accuracy and interobserver agreement, but appears to have changed whether a patient would meet AHA criteria (i.e. ASPECTS of 6-10) for mechanical thrombectomy in a substantial proportion of cases. This has major implications for improving treatment decision making for endovascular therapy as standardization of non-contrast CT imaging evaluation and improvement of accuracy is essential to the proper triage of acute stroke LVO patients. The impact is also broad given that non-contrast CT is available at every stroke center. If one assumes that ASPECTS of 0–5 are generally not treated while those of ASPECTS of 6–10 are generally treated, the use of the e-ASPECTS software could be associated with a potential change in clinical management in 12% of cases. Interestingly, most of these changes would have been in favor of treatment as the change from treat to no-treat was 2.7% and the change from no-treat to treat was 9.2%. This was consistent across all levels and types of training.

Our study has limitations. First, our gold-standard is, of course, imperfect. The only perfect gold standard would be if patients received a DWI MRI immediately before or after the non-contrast CT. However, we feel that by restricting the accuracy analysis only to patients who achieved complete revascularization in a timely manner, we could rely on the 24 hour CT or MRI to be consistent with what was actually infarcted at the time of the initial NCCT. It is likely that a 24 hour MRI would provide more detail on infarct extent than a 24 hour CT, however this was not standard of care at our institution. Second, only 10% of the patients included had an ASPECTS of 5 or less suggesting that the majority of our cases had limited ischemic changes. This may limit the generalizability of our findings to the group of patients with low ASPECTS. It is also important to point out that e-ASPECTS is intended to be a decision-making aid and not a replacement for an experienced neuroradiologist. Its main utility is in circumstances in which an experienced neuroradiologist is not available or if there is significant disagreement between clinicians. Further investigations are also needed to see if management decisions change when reads are done with and without the assistance of the e-ASPECTS. Lastly, it is unclear to us why interobserver agreement for some regions was better than others. We suspect that interobserver agreement for cortical regions was lower than deep structures due to the fact that subtle loss of gray-white differentiation in these regions may be difficult to discern. We are also unclear as to why accuracy worsened for senior neuroradiologists in the caudate region. A more in depth study is needed to understand why radiologists interpret imaging the way they do.

Conclusions

The use of e-ASPECTS resulted in significant improvements in interobserver agreement and accuracy across a large cohort of radiologists and neurologists in assessing non-contrast CTs for early ischemic changes in the setting of anterior circulation large vessel occlusion. This was true for both global ASPECTS and on a per region basis. Further studies are needed to validate our findings.