Collateral Automation for Triage in Stroke: Evaluating Automated Scoring of Collaterals in Acute Stroke on Computed Tomography Scans

Abstract

Computed tomography angiography (CTA) collateral scoring can identify patients most likely to benefit from mechanical thrombectomy and those more likely to have good outcomes and ranges from 0 (no collaterals) to 3 (complete collaterals). In this study, we used a machine learning approach to categorise the degree of collateral flow in 98 patients who were eligible for mechanical thrombectomy and generate an e-CTA collateral score (CTA-CS) for each patient (e-STROKE SUITE, Brainomix Ltd., Oxford, UK). Three experienced neuroradiologists (NRs) independently estimated the CTA-CS, first without and then with knowledge of the e-CTA output, before finally agreeing on a consensus score. Addition of the e-CTA improved the intraclass correlation coefficient (ICC) between NRs from 0.58 (0.46–0.67) to 0.77 (0.66–0.85, p = 0.003). Automated e-CTA, without NR input, agreed with the consensus score in 90% of scans with the remaining 10% within 1 point of the consensus (ICC 0.93, 0.90–0.95). Sensitivity and specificity for identifying favourable collateral flow (collateral score 2–3) were 0.99 (0.93–1.00) and 0.94 (0.70–1.00), respectively. e-CTA correlated with the Alberta Stroke Programme Early CT Score (Spearman correlation 0.46, p < 0.001) highlighting the value of good collateral flow in maintaining tissue viability prior to reperfusion. In conclusion, ­e-CTA provides a real-time and fully automated approach to collateral scoring with the potential to improve consistency of image interpretation and to independently quantify collateral scores even without expert rater input.

Introduction

Mechanical thrombectomy is the mainstay of treatment for patients with acute ischemic stroke due to large vessel occlusion, who fulfil guideline criteria [1]. Despite increasing rates of revascularisation, outcomes are not universally positive and are dependent on clinical and imaging characteristics of patients [2]. Collateral cerebral blood flow is a key determinant of both good clinical outcomes and capacity to benefit from thrombectomy, in both early and late presenting patients [3, 4, 5, 6, 7]. Collateral blood flow sustains tissue viability until definitive treatment can be provided, thereby averting infarction of penumbral tissue and improving outcomes [8].

Quantifying collateral cerebral blood flow can be achieved using a variety of scores in a variety of imaging modalities, including computed tomography angiography (CTA), CT perfusion (CTP), and magnetic resonance imaging-based equivalents. CT is the mainstay of acute stroke imaging, with only certain specialist centres using magnetic resonance imaging routinely at the front door. While there has been much interest in CTP to define collateral blood flow, particularly the late time window, CTA is routinely acquired to identify vessel occlusion and available in more centres worldwide. If CTA can provide information on collateral blood flow, it would provide an accessible and routinely acquired biomarker for assessing collateral status.

Several scoring systems exist for quantifying collateral cerebral blood flow on CTA [9, 10, 11]. The most commonly used is the CTA collateral score (CTA-CS) also known as the Tan score [6, 11]. The challenge for the implementation of any manual scoring system in both research trials and clinical practice is inconsistency between scorers, making robust patient characterisation less likely. Solutions to inter-rater variability include fully automated computer scoring, or software assistance to improve consistency between individuals. In this study, we evaluated the e-CTA module of the e-STROKE SUITE (Brainomix Ltd.,), a fully automated calculation of the CTA-CS using a machine learning algorithm.

The objective of this study was to validate the e-CTA software. The results of the e-CTA software were compared to a reference standard consensus opinion derived from 3 expert neuroradiologists (NRs). The agreement between individual NRs, both with and without the automated e-CTA reference was quantified, and hence the effect of e-CTA on inter-rater reliability was assessed. Finally, the agreement between e-CTA and automated Alberta stroke programme early CT score (e-ASPECTS; Brainomix Ltd.,) on non-contrast CT was quantified to explore the relationship between collateral status and maintenance of tissue viability.

Methods

Patients

The Freiburg Ethics commission granted approval for the Study (0171140). Study inclusion criteria were age ≥18 years, anterior circulation ischemic stroke with large vessel occlusion eligible for thrombectomy, and both baseline CT and CTA acquired as part of routine clinical care. Patients were excluded from analysis if there was intracranial haemorrhage on presenting CT imaging, previous territorial infarct or artefacts precluding image analysis as determined by a NR.

Imaging

Imaging was acquired from 2 centres on multiple CT vendor scanners (Toshiba, General Electric, Siemens, Philips). CTA was acquired by using single-phase acquisition, and processed images had a maximum slice thickness of 1 mm.

Collateral Scoring

CTA-CS within the middle cerebral artery (MCA) territory was used to quantify the collateral blood flow status [11]. The CTA-CS comprises a numerical (ordinal) score ranging from 0 to 3: 0, no collateral filling; 1, ≤50% but >0 of the occluded MCA territory; 2, >50% but <100% of the occluded MCA; 3, 100% collateral supply of the occluded MCA territory. CTA-CS was quantified on the CTA scans independently by 3 experienced NRs (P.P., M.E., M.P.). NRs scored the extent of collaterals manually, blinded to automated e-CTA scoring and clinical information. Following initial evaluation, scorers repeated collateral scores with the e-CTA results available. Finally, a consensus score was derived between NRs (reference standard).

Automated Collateral Scoring

Automated software was used to generate the e-CTA score, a module within the e-STROKE SUITE (Brainomix Ltd., Oxford, UK). e-CTA uses a combination of classical image processing techniques and machine learning classifiers. Input DICOM data are resampled into standard resolution, with gantry tilt correction. Fast proprietary registration is used to align data with a standard coordinate system. Major vessels are segmented and labelled in this standard reference frame using a classification framework. Input arteries and output veins are analyzed to estimate acquisition phase of the CTA, to ensure early or late acquisitions are reported to the user. Vessel density is computed throughout the MCA territory, with a probability map showing absence of vessel contrast. The volume of underenhanced vessels is computed relative to the healthy hemisphere. The volume is summarized into the CTA-CS categories to generate the e-CTA score.

Ischemic Damage

Extent of ischemic damage was assessed by a NR (P.P.) on the presenting noncontrast CT using e-ASPECTS software (Brainomix Ltd., Oxford, UK). e-ASPECTS image-processing algorithm follows an artificial intelligence approach, with a combination of traditional 3D graphics and statistical methods, and machine learning classification techniques. The machine learning classifier identifies early and non-acute signs of ischemia to derive a voxel-wise probability map, which generates the e-ASPECTS [12, 13].

Statistical Analysis

The intraclass correlation coefficient (ICC) quantified agreement between scorers or automated outputs. The percentage agreement of CTA-CS was also quantified. CIs for ICCs were calculated using (bias-corrected and accelerated) bootstrap intervals, through 1,000 bootstrap resamples. To determine sensitivity and specificity to collateral status, CTA-CS were categorised into 2 groups: 0–1 (poor collateral flow) and 2–3 (good collateral flow), as has been used to select patients for clinical trials [6]. To correlate the categorical scales of CTA-CS and ASPECTS Spearman's correlation coefficient was estimated.

The primary outcome of the study was the agreement between consensus reference standard and fully automated e-CTA quantified using the ICC and percentage of complete agreement. Additional analyses included proportion of scans where the agreement differed by >1 point, and agreements between NR scorers with and without e-CTA assistance. Sensitivity and specificity for determining favourable CTA-CS by the e-CTA were also calculated. Finally, the correlation of e-CTA with e-ASPECTS was calculated.

Results

Patients

Ninety-eight consecutive patients with acute ischemic stroke eligible for mechanical thrombectomy were prospectively enrolled into the study and underwent CTA. No patient demographics or clinical data could be collected under the Ethics Protocol which specified evaluation of images only.

Representative CT and CTA images from patients together with the CTA-CS heat map and plain CT infarct volume can be seen in Figure 1. e-CTA score was able to successfully process all data.

Fig. 1.

Example of e-CTA. CTA imaging from a representative patient with a right MCA occlusion (e-ASPECTS and e-CTA overlay on lower row). At presentation, the e-ASPECTS is 10 with minimal ischemic change identified by e-ASPECTS infarct volume detection (red mask), but the collateral flow is poor (orange mask, e-CTA: 0). There is extensive final infarction at 24 h consistent with the paucity of collateral flow on presenting imaging. ASPECTS, Alberta stroke programme early CT score; CTA, computed tomography angiography.

NR Scoring

The distribution of CTA-CSs is shown in Figure 2. NRs 1 and 2 generated a similar distribution of CTA-CS, whereas NR 3 tended to give lower scores. The ICC across all 3 raters was 0.58 (0.46–0.67). Between individual NRs ICC ranged from 0.42 to 0.86 and score agreements from 36.2 to 81.6% (Table 1).

Fig. 2.

Distribution of CTA-CSs. Distribution of CTA-CSs (Tan score) for each neuroradiology rater without (a) and with (b) automated CTA-CS assistance. NR, neuroradiologist 1–3.

Table 1.

Inter-rater comparison

	Percentage agreement, blinded, %	Percentage agreement, assisted, %	ICC (95% CI), blinded	ICC (95% CI), assisted
NR 1 vs. 2	81.6	72.7	0.86 (0.79–0.92)	0.80 (0.71–0.88)
NR 1 vs. 3	36.2	64.3	0.42 (0.30–0.55)	0.70 (0.55–0.83)
NR 2 vs. 3	43.6	73.5	0.49 (0.36–0.62)	0.80 (0.71–0.86)

Agreement between 3 NR blind to e-CTA and assisted by the e-CTA result.

ICC, intraclass correlation coefficient; NR, neuroradiologists; CTA, computed tomography angiography.

Addition of automated CTA collateral scoring to guide neuroradiological assessment reduced inter-rater variability and improved overall ICC from 0.58 to 0.77 (0.66–0.85), a difference of 0.19 (0.09–0.31, p = 0.003). Agreements between NRs when assisted ranged from 64.3 to 73.5%, with ICCs from 0.70 to 0.80 (Table 1).

Automated Scoring

Automated e-CTA score produced 90% agreement with the consensus NR scores and an ICC of 0.93 (0.90–0.95). For the remaining 10% of fully automated scores where there was no agreement with the consensus opinion the automated score was within 1 point of the consensus (Fig. 3). Sensitivity and specificity for identifying favourable collateral score compared to consensus were 0.99 (0.93–1.00) and 0.94 (0.70–1.00), respectively.

Fig. 3.

e-CTA versus consensus score. Agreement of the automated Tan score (e-CTA) with the NR consensus Tan score.

When compared to individual NRs blinded to the automated output, the automated assessment agreed strongly with 2 individuals (ICC 0.70 and 0.70), but poorer with the third scorer (0.39). When software assisted, all experts had good agreement with the automated score (0.72, 0.84, 0.93).

CTA-CS versus ASPECTS

Consistent laterality of ischemia was positively identified by both e-ASPECTS and CTA-CS in 95% of cases. In these examinations, the Spearman correlation coefficient was 0.46, p < 0.001, between these 2 markers of ischaemia. A similar correlation was seen between e-ASPECTS and e-CTA with a Spearman correlation coefficient is 0.46, p< 0.001.

Discussion

This study demonstrated a high degree of agreement between the fully automated and objective e-CTA score and a consensus expert CTA-CS. The addition of e-CTA assisted neuroradiological grading of collateral scores and reduced variability between experts. Both the CTA-CS and e-CTA score correlated with e-ASPECTS consistent with the demonstrated deleterious effect of poor collateral blood flow on tissue survival [11].

In this study, e-CTA generated equivalent scores to consensus CTA-CS when used to quantify collateral blood flow. Although current guidelines do not advocate the use of advanced imaging to identify collateral blood flow for patients presenting within 6 h of stroke onset [1], several trials demonstrating the efficacy of early intervention did use collateral blood supply as a selection criterion, identified on either CTP or CTA-CS [6, 7, 14]. Post-hoc analysis of the MR-CLEAN trial, which did not use advanced imaging, confirmed that the treatment effect of thrombectomy was most marked in patients with good collateral status on CTA [5, 15]. For patients presenting beyond 6 h or with an uncertain time of onset, selection using good collateral blood flow has also been shown to identify those who can benefit most from thrombectomy [4], although there is discussion about whether these criteria are overly selective [16].

Widespread identification of patients with favourable imaging on CTP can be challenging as access to CTP is not universal, it increases contrast and radiation doses, and can be more vulnerable to motion artefact [17]. In contrast, CTA is acquired routinely prior to mechanical thrombectomy and is available in most centres. If collateral status on CTA is sufficient to identify patients for revascularisation therapy, this would widen access and streamline treatment pathways for this time critical intervention.

Reliable and accurate scoring is essential if CTA-CS is to be evaluated to select patients for treatment. The 90% agreement of e-CTA with consensus opinion, greater than that seen between NRs, would support the use of an automated e-CTA score to achieve this. Automated collateral scoring with e-CTA negates the demand for real-time neuroradiological input and may help accelerate referral pathways, especially in resource limited settings. This study has demonstrated the high sensitivity and specificity of e-CTA, independent of clinician input, for identifying favourable collateral status. Expert neuroradiology input is rarely available in real time at most hospitals to which stroke patients present, especially at primary stroke centres from which patients need to be transferred to a comprehensive centre. Even when specialist neuroradiology opinion is available, the e-CTA offers the opportunity to improve inter-rater reliability, an essential prerequisite to demonstrating biomarker utility in trials and routine clinical practice.

The correlation of e-CTA score and e-ASPECTS reinforces the pathological significance of collateral blood flow in sustaining tissue viability prior to reperfusion and is consistent with prior work [5, 11]. The use of an automated algorithm offers the potential to generate an objective volume of tissue without collateral blood supply and hence the further refinement of patient selection criteria, in a manner consistent with recent trials using CTP [3, 4]. Further work would be required using paired e-CTA and CTP data to determine the equivalent thresholds for these imaging modalities.

This study has several limitations. The CTA-CS was generated under experimental conditions by specialist NRs without the time pressures of real-world clinical scenarios. In many healthcare settings, ready access to specialist neuroradiology input is not available. As such, the benefit of an automated e-CTA score may be greater than demonstrated here. Although NRs in this study were blinded to clinical data all patients had a confirmed stroke and large vessel occlusion, knowledge of which was available to the expert scorers. The relationship of e-CTA with clinical outcomes and treatment effects is beyond the scope of this study, although this relationship for CTA-CS is well established [5, 11, 15, 18].

Conclusion

e-CTA provides automated and objective measurement of collateral score, which provides similar measures to consensus expert opinion and improves inter-rater reliability of CTA-CS. e-CTA could prove useful for consistent patient selection in clinical trials and translation into clinical practice.

Statement of Ethics

The study was performed in accordance with recommendations guiding physicians in biomedical research involving human subjects adopted by the 18th World Medical Assembly, Helsinki, Finland (1964 and later revisions).

Disclosure Statement

G.H.: is employed part-time by Brainomix. I.Q.G.: has equity interest in and is consultant for Brainomix. All other authors report nothing to disclose.