Interobserver agreement between senior radiology resident, neuroradiology fellow, and experienced neuroradiologist in the rating of Alberta Stroke Program Early Computed Tomography Score (ASPECTS)

Abstract

PURPOSE

The distribution of ischemic changes caused by infarction of the middle cerebral artery (MCA) territories is usually measured using the Alberta Stroke Program Early Computed Tomography Score (ASPECTS). The first interpreter of the brain computed tomography (CT) in the emergency department is the on-call radiology resident. The primary objective of this study was to describe the agreement of the ASPECTS performed retrospectively by the resident compared with expert raters. The second objective was to ascertain the appropriate window setting for early detection of acute ischemic stroke and good interobserver agreement between the interpreters.

METHODS

We identified consecutive patients presenting with hemiparesis or aphasia at the emergency department who underwent brain CT and CT angiography. Each scan was rated using ASPECTS by senior radiology resident, neuroradiology fellow, and later by consensus between two expert raters. Statistical analysis included determination of Cohen’s kappa (κ) coefficient and intraclass correlation coefficient (ICC).

RESULTS

A total of 43 patients met our study criteria. Interobserver agreements for ASPECTS varied from 0.486 to 0.678 in Cohen’s κ coefficient between consensus of two neuroradiologists and a neuroradiology fellow, and from 0.198 to 0.491 for consensus between two neuroradiologists and a senior radiology resident. ICC among three raters (expert consensus, neuroradiology fellow, and senior radiology resident), was very good when 8 HU window width and 32 HU center level setting was used.

CONCLUSION

ASPECTS varied among raters. However, when using a narrowed window setting for interpretation, interobserver agreement improved.

Noncontrast computed tomography (NCCT) is a fast and accurate imaging technique to exclude intracerebral hemorrhage and evaluate acute stroke in suspected cases (1). Nowadays, distribution of the ischemic changes caused by infarction of the middle cerebral artery (MCA) territories are usually measured using the Alberta stroke program early computed tomography score (ASPECTS), which is a semiquantitative scale (2). For detection of acute ischemic stroke by NCCT, early acute ischemic change was defined as parenchymal hypoattenuation or loss of gray-white differentiation (3). Factors that can affect lesion conspicuity and diagnostic accuracy of CT are window width and window level. For obvious detection of acute ischemic change, a narrowed window setting is recommended (4). Currently, intra-arterial thrombectomy (IAT) for large vessel occlusion (LVO) is becoming the standard of care in comprehensive stroke centers harboring multidisciplinary teams and qualified neurointerventionalists; this approach shows good outcome and is also recommended by the Thai Stroke Guidelines (5). Recent studies show that good outcome after IAT can be predicted from the ASPECTS (6–8). In our practice, we found ASPECTS rated by different reviewers to be variable. Therefore, good interobserver agreement of the ASPECTS should be ascertained before use, as this affects the selection of patients for IAT.

According to the literature reviews, most articles compared neuroradiologists and neurologists, or neuroradiology/stroke fellows for evaluation of the ASPECTS (9, 10).

In many academic hospitals, the first provider in the emergency department for interpretation of the brain CT is the on-call radiology resident, whose report is then approved by the attending radiologist. The primary objective of this study was to describe the agreement of the ASPECTS performed retrospectively by the resident compared with an expert rater. The second objective was to determine the appropriate window setting for early detection of acute ischemic stroke and good interobserver agreement between the interpreters.

Methods

We conducted a retrospective data review of 43 patients who presented to the emergency department with suspected acute ischemic stroke (consisting of hemiparesis or aphasia) and underwent NCCT and computed tomography angiography (CTA) of the brain between June 2014 and June 2016. Images of patients with intracranial hemorrhage were excluded from the study. CT scanning was performed with a 128-slice scanner (Aquilion CX) in the emergency department of our hospital using conventional CT technique: 120 KV, 300 mA, 0.75 s scanning time, and 0.5×64 mm scan thickness.

All CT scans were performed with the patients in supine position. Axial images were obtained from the scalp to top of the lamina of the C1 vertebral body. CT scans were performed without intravenous contrast administration. CT images were available in softcopy and were reviewed on a dedicated high-resolution LCD monitor. All CT images were retrospectively reviewed and scored by two experienced neuroradiologists, a neuroradiology fellow, and a senior radiology resident individually, in isolation from the others. The images were viewed separately in three window settings to compare the sensitivity and reliability of agreement of the ASPECTS to detect acute ischemic changes. Three different settings were used in the evaluation of the images (Fig.): the first softcopy image was preset to an 80 HU window width and 20 HU center level (setting 1) (11). Following this, the variable softcopy settings consisted of two initial preset defaults of an 8 HU window width with 32 HU center level (setting 2) and a 20 HU window width with 35 HU center level (setting 3) (11). The final conclusion was made by consensus adjudication. For each patient, the non-neuroradiologist’s final interpretation was compared with the consensus of the two neuroradiologists reports. All raters were blinded to the clinical data findings.

Figure. a–f.

CT softcopy images for evaluation of the ASPECTS are set at an 80 HU window width and 20 HU center level (a, b), an 8 HU window width with 32 HU center level (c, d) and a 20 HU window width with 35 HU center level (e, f).

The ASPECTS is rated from two standard axial cuts, one at the level of the thalamus and basal ganglion, and one just rostral to the ganglionic structures. For ASPECTS, the MCA territory is divided into 10 sections. One point is deducted for each area of early ischemic change, such as parenchymal hypoattenuation or loss of gray-white differentiation as these changes are associated with edema and irreversible injury. A normal brain CT has an ASPECTS of 10 points; hence, an ASPECTS of zero means diffuse brain ischemia in all 10 points of the MCA territory (12). Parenchymal hypoattenuation is evaluated in terms of abnormally low attenuation of brain structures relative to other parts of the same structures or to its contralateral hemisphere.

Patient demographics were collected at the time of admission to the emergency department. The Ethics Committee of our hospital, approved the study. Formal consent was not required as this is a retrospective study.

Statistical analysis

Descriptive statistical analysis of the study population was performed using computer software (SPSS version 14, SPSS Inc.). The categorical data were expressed as percentages with continuous data expressed as mean or median, standard deviation, and range.

Interobserver agreement was evaluated for dichotomized ASPECTS (≤7 and >7). ASPECTS >7 showed better clinical outcome to intra-arterial treatment when compared with ASPECTS ≤7 (6, 7).

Agreement between each interpreter’s ASPECTS was assessed using Cohen’s kappa (κ) coefficient and Intraclass Correlation Coefficient (ICC). The κ values were interpreted as: slight agreement, 0.00 to 0.20; fair agreement, 0.21 to 0.40; moderate agreement, 0.41 to 0.60; substantial agreement, 0.61 to 0.80; or almost perfect agreement, 0.81 to 1.00.

Results

A total of 43 patients were included in this study; 23 were female (53%) with a mean age of 67.1 years. Median time to CT from symptom onset was two hours (range, 20 min to 72 hours). All of the patients were diagnosed with cerebral infarction.

Table 1 shows the baseline demographic characteristics and clinical information of the study patients. For the dichotomized ASPECTS (≤7 and >7), there was moderate to substantial agreement between consensus of the two neuroradiologists and the neuroradiology fellow, while there was slight to moderate agreement between consensus of the two neuroradiologists and the senior resident (Table 2).

Table 1.

Baseline demographic characteristics of the study patients

Characteristics
Female, %	53
Age (years), mean±SD	67.1±13.7
Hemiparesis, %	88.4
Median time of onset (range)	2 h, (20 min to 72 h)
Presence of old infarction, %	44.2

SD, standard deviation.

Table 2.

ASPECTS agreement between different raters

			Cohen κ (95% CI)
Agreement between consensus of 2 neuroradiologists and a neuroimaging fellow for total ASPECTS assessed with Cohen’s κ	Total ASPECTS	Setting 1	0.486 (0.235–0.736)
		Setting 2	0.583 (0.353–0.814)
		Setting 3	0.678 (0.458–0.898)
Agreement between consensus of 2 neuroradiologists and a senior radiology resident for total ASPECTS assessed with Cohen’s κ	Total ASPECTS	Setting 1	0.198 (−0.111–0.507)
		Setting 2	0.491 (0.246–0.737)
		Setting 3	0.443 (0.201–0.684)
Correlation among the three ratings (consensus, neuroradiology fellow and senior radiology resident) for total ASPECTS assessed with ICC	Total ASPECTS		ICC (95% CI)	Interpretation
		Setting 1	0.741 (0.571–0.851)	Good
		Setting 2	0.936 (0.893–0.963)	Very good
		Setting 3	0.780 (0.785–0.925)	Good

Setting 1, 80 HU window width and 20 HU center level; Setting 2, 8 HU window width and 32 HU center level;

Setting 3, 20 HU window width with 35 HU center level.

ASPECTS, Alberta stroke program early computed tomography score; CI, confidence interval; ICC, intraclass correlation coefficient.

Among the three ratings (consensus between the two experienced neuroradiologists, the neuroradiology fellow and the senior radiology resident), ICC for total ASPECTS is shown in Table 2. Results revealed very good concordance and internal consistency/reliability in setting 2 and good concordance in settings 1 and 3.

Discussion

Our findings revealed good correlation for the ASPECTS for NCCT when assessed with ICC. Among the different settings, we found that narrowing the window improved the interobserver agreement. This has important implications concerning acute stroke care, especially considering that residents are often the first clinicians to see these patients at academic institutions.

Interobserver agreement assessment with Cohen’s κ showed better correlation between the expert consensus and the neuroradiology fellow than between the expert consensus and the senior radiology resident. In the binary classification scheme, discrepancy of the ASPECTS by only one point may affect the treatment approach. When the Cohen κ was used in the interpretation, only moderate to substantial agreement was recorded between consensus of the two neuroradiologists and the neuroradiology fellow, with slight to moderate agreement between consensus of the two neuroradiologists and the resident. Our study agrees with the results of Coutts et al. (13) regarding the interobserver agreement of the ASPECTS reaching moderate to substantial agreement among the raters. This issue must be considered since it greatly affects recruiting candidates for mechanical thrombectomy. For example, an ASPECTS 5 patient will be excluded from intra-arterial treatment if the institute policy is strictly attached to the guidelines or from fear of malpractice lawsuits. On the contrary, this patient might be recruited for IAT if the physicians classified the patient to be ASPECTS 6. Knowing that difference of ASPECTS interpretation is marginal within 1–2 points, we propose using ASPECTS with caution, particularly for anyone with CT interpretation just one point below the cutoff point of ASPECTS 6. In our opinion, using a range for patient’s eligibility for IAT may be more appropriate. For example, ASPECTS ≤4 is excluded for IAT, ASPECTS 5–6 is equivocal clinical outcome but not restricted, ASPECTS ≥7 is favorable clinical outcome, instead of the current recommendation using the threshold number for IAT inclusion at ASPECTS ≥6 (14).

The presence of an old infarction is one of the limitations of ASPECTS when using a narrowed window width setting. In our study, 44.2% of patients had old lacunar infarction that could interfere with the interpretation. However, radiologists usually use both narrow window and brain parenchymal window for interpretation. Thus, this limitation would not have much effect in real practice. In difficult cases, diffusion-weighted imaging is the best solution to differentiate between a new infarction and a recent one.

This study had several limitations. First, we offered only two slices of the brain to interpret the ASPECTS, whereas in real practice all scan images are used. Second, we know from prior studies that total time for detection and interpretation of the NCCT for early ischemic change is an important factor (15). Image interpretation in an emergency setting is much more complicated, rushed and confusing than an elective case. In our study, we did not take the reading time into consideration. Finally, we did not have the gold standard in this study and, therefore, cannot interpret the accuracy of the results. Prior studies used magnetic resonance imaging (MRI) of the brain as the gold standard to interpret the ASPECTS (16, 17). However, in our institute, we perform only NCCT and CTA for routine diagnosis of stroke, unless MRI is mandatory.

In future, good interobserver agreement in the selection of patients as candidates for IAT may be obtained through evaluation with other imaging studies or combined with other factors such as collateral score (18). Other imaging studies such as CT perfusion or MRI may help acute stroke evaluation. Neuroimaging plays a pivotal role for rapid identification and triage of acute ischemic stroke patients eligible for mechanical thrombectomy. Faster imaging means more neurons can be saved. The physician will not choose time-consuming imaging tools regardless of whether they give superb accuracy. Ultimately, the fastest, most accessible, most accurate, and less complex modality is optimal.

In conclusion, the ASPECTS varied among raters of different levels of neuroradiology experience. However, interobserver agreement could be improved by employing a narrowed window setting for the interpretation.

Main points.

• The interobserver agreement of ASPECTS can be improved by narrowing the window setting.

• However, ASPECTS may fail to differentiate between an old infarction and an acute infarction when using a narrowed window width setting.

• In the binary classification scheme, discrepancy in the ASPECTS by only one point may affect the patient’s recruitment for mechanical thrombectomy. When the Cohen’s κ was used in the interpretation, only moderate to substantial agreement was recorded between consensus of the two neuroradiologists and the neuroradiology fellow, with slight to moderate agreement between consensus of the two neuroradiologists and the resident.