Interrater variability for CT angiography evaluation between neurologists and neuroradiologist in acute stroke patients

Abstract

Objective:

The diagnosis of arterial occlusion has a considerable impact on the indication of mechanical thrombectomy, and CT angiography (CTA) is recommended in the management of acute stroke. The goal of the present study is to assess the interrater agreement in the diagnosis of occlusion of intracranial arteries on CTA between a neuroradiologist and neurologists.

Methods:

CTA images of 75 acute stroke patients were evaluated for occlusion of intracranial arteries by an experienced interventional neuroradiologist, and stroke and general neurologists.

Results:

75 patients who were treated by intravenous thrombolysis were enrolled in the study. CTA images were available for all 75 patients (34 females; mean age ± SD, 72 ± 14 years; National Institutes of Health Stroke Scale 10; median 8–14; and Alberta Stroke Program Early CT mean 9.7). The agreement between the neuroradiologist and neurologists in evaluation of intracranial artery occlusion was as follows: occlusion of the middle cerebral artery segment M1: observer agreement 77%, kappa (κ) = 0.61 and middle cerebral artery M2: observer agreement 77%, κ 0.48; internal carotid artery: observer agreement 92%, κ 0.84; T occlusion: observer agreement 90.0%, κ 0.33; posterior cerebral artery segments P1 and P2: observer agreement 98%, κ 0.97; basilar artery: observer agreement 96%, κ 0.92; and vertebral artery segment V4: observer agreement 88%, κ 0.48.

Conclusion:

Interrater agreement of CTA evaluation of occlusion between the neurologists and the neuroradiologist was very strong. The ability of the trained neurologists to read an intracranial large vessel occlusion correctly may improve the door-to-needle times in acute stroke.

Advances in knowledge:

In this study, the neurologists were able to recognize occlusion of intracranial arteries. This could accelerate the management of acute stroke care.

INTRODUCTION

Acute occlusion of intracranial arteries is the most common cause of ischaemic stroke. This statement is based on the fact that arterial occlusion is found in up to 70% of patients during the first 6 h after the onset of stroke. The number of occluded arteries decreases during the first 24 h to 54% due to spontaneous recanalization.^1,2

CT examination is essential for the exclusion of the haemorrhagic stroke and for evaluation of the range of an early ischaemic change (Figures 1 and 2). It is essential for the indication of acute recanalization therapy—intravenous thrombolysis (IVT) or mechanical thrombectomy (MT). For the indication of MT, the interpretation of images is complex and includes not only non-contrast CT but also CT angiography (CTA) and CT perfusion or perfusion weighted MR.^3,4 Advanced neuroimaging is recommended mainly for patients in the therapeutic window for MT with ischaemic changes [scored below 6 points by using the Alberta Stroke Program Early CT score (ASPECTS)] or more precisely in patients exceeding 6 h from the onset of symptoms or with an unknown time of onset and with proven large artery occlusion to evaluate the extent of salvageable brain tissue before MT.^4,5 The role of an experienced radiologist in these cases is necessary.

Figure 1.

Haemorrhagic stroke, hypertensive bleeds in basal ganglia (arrow).

Figure 2.

Ischaemic stroke, early ischaemic changes (arrow), Alberta Stroke Program Early CT = 7 points.

Also, the occlusion of intracranial arteries is an important predictor of stroke outcome.² Recently, several studies have shown that MT is a highly effective treatment of stroke if carried out in carefully selected patients.⁶ Patient selection includes the identification of large vessel occlusions, in addition to other imaging features (e.g. the absence of early ischaemic changes and good collateral flow).⁷

The inability of correct and rapid interpretation of imaging findings could be a limiting step in implementing MT into routine clinical practice.

In many centres, a neurologist interprets CT images before the treatment by IVT. In other centres, a radiologist interprets images before IVT only if they are available immediately.

Moreover, in many stroke centres during non-working hours and weekends, non-stroke neurologists take care of acute stroke patients.⁸ Therefore, in our study we intended to assess the interrater agreement of the diagnosis of occlusion of intracranial arteries at CTA between a radiologist, a general neurologist and a neurologist-stroke specialist. Some previous studies confirmed the moderate or strong agreement between a neurologist and a radiologist in case of proximal or distal occlusion of the internal carotid artery (ICA) or in case of CTA perfusion evaluation,^9,10 but the agreement has not been tested in the case of CTA which was performed before recanalization therapy.

METHODS AND MATERIALS

Ethical approval of the study protocol

The study protocol was approved by the Ethics Committee of the University Hospital Ostrava (Ostrava, Czech Republic). All the patients provided written informed consent to participate in the study.

Patients

All consecutive patients with an acute ischaemic stroke (up to 4.5 h from stroke onset, no age limit) who underwent IVT or IVT and MT in the Comprehensive Stroke Centre, University Hospital Ostrava (Ostrava, Czech Republic), within 2013 were assessed. Before recanalization therapy, all patients underwent baseline unenhanced CT and CTA. The intracerebral haemorrhage or tumours, stroke mimics etc. were exclusion criteria. The age, sex, risk factors, early ischaemic changes on CT according to ASPECT score before treatment and the National Institute of Health Stroke Scale score upon hospital admission were recorded. All patients underwent a non-contrast follow-up CT 24–36 h after hospital admission. Also, all patients had an outpatient visit 3 months after the stroke, at which point, the modified Rankin scale score was recorded. Data were collected prospectively in a local and international registry (Safe Implementation of Treatments in Stroke).

Imaging

Single-phase CTA was carried out using the Somatom^® Definition 64 system (Siemens Medical Systems, Erlangen, Germany). After the unenhanced brain CT examination, 50–70 ml of a non-ionic iodine contrast agent (Visipaque^™; GE Healthcare, Piscataway, NJ) was administered by power injector at a rate of 4 ml s⁻¹ followed by a saline chase of 40 ml at a rate of 4 ml s⁻¹ with bolus tracking monitoring in the ascending thoracic aorta. The CTA examined the vasculature from the aortic arch to the cranial vertex. Nominal axial slice thickness acquisition was 0.6 mm, and it was used to calculate maximum intensity projection (MIP) reconstruction images to visualize all vessel segments evaluated in this study. All the post-processing was performed using the same software. Axial 3-mm thick MIP reconstructions, and sagittal and coronal MIP reconstructions with 10-mm thickness were used by both observers for the evaluation of the vessel occlusion site.

Image review

Images were evaluated retrospectively for occlusion of the intracranial arteries by three physicians: an experienced interventional neuroradiologist; an experienced neurologist, stroke specialist (Head of Comprehensive Stroke Centre); and a general neurologist. The latter is subspecialized in epilepsy and does not take care of stroke patients routinely. Both neurologists were trained in CTA emergent evaluation of acute occlusion which comprised of the evaluation of 50 CTAs under supervision of a radiologist. The duration of the training was 1 week. The interventional neuroradiologist, who joined the study, had completed the European Course of Neuroradiology and has been involved in stroke patient diagnostics and endovascular treatment for the past 10 years.

The neuroradiologist and neurologists evaluated all intracranial arteries for large vessel occlusion and for the site of occlusion. Complete occlusion of an intracranial artery was diagnosed if no flow was detected (Figure 3a,b). The middle cerebral artery, M1 and M2 segments; ICA; posterior cerebral artery, P1 and P2 segments; anterior cerebral artery, A1 and A2 segments; basilar artery; vertebral artery (VA) were evaluated. The neuroradiologist and neurologists were blinded to clinical findings and to each other's findings.

Figure 3.

(a) Occlusion of the middle cerebral artery M1 segment (arrow), axial slice. (b) Occlusion of the middle cerebral artery M1 segment (arrow), coronal slice.

Statistical analyses

Patient characteristics are described as numbers and percentages for categorical variables and medians (minimum, maximum) for continuous variables. Data with a normal distribution are the mean ± standard deviation; data with extreme values are the median and interquartile range.

Interrater agreement was assessed using the unweighted Kappa index, adjusted Kappa index and Kappa index for two outcomes and multiple raters. Agreement was considered “poor” with κ < 0.4, “moderate” if 0.41–0.60, “strong” if 0.61–0.80 and “very strong” if >0.81.

Data analyses were performed using the SW Stata^® v. 13 (Stata Corp., College Station, TX).

RESULTS

In 2013, 75 patients were treated by IVT or IVT and MT. The CTA images were available for all 75 patients: age, mean ± SD, 72 ± 14 years; male sex; n (%) = 41(55%). All baseline and risk factors are shown in Table 1.

Table 1.

Patients' characteristics

Patients enrolled	N = 75
Age (years), mean ± SD	72 ± 14
Male sex, n (%)	41 (55%)
Hypertension, n (%)	60 (80%)
Hyperlipidaemia, n (%)	15 (20%)
Diabetes mellitus, n (%)	8 (10%)
Coronary heart disease, n (%)	23 (30%)
NIHSS score, median (IQR)	10 (8–14)
Time from admission to procedure (min), median (IQR)	118 (95–192)
mRankin scale <2 at 3 months; n (%)	41 (55%)
ASPECTS baseline; mean (min–max)	9.73 (8–10)
ASPECTS 24–36 h after IVT; mean (min–max)	9.62 (3–10)
Localization of infarct in anterior circulation n (%)	64 (85%)

ASPECTS, Alberta Stroke Program Early CT score; IVT, intravenous thrombolysis; IQR, interquartile range; max, maximum; min, minimum; mRankin scale, modified Rankin scale; NIHSS, National Institutes of Health Stroke scale; SD, standard deviation.

The total agreement between the neuroradiologist and the neurologists on evaluation of intracranial occlusion was very strong or strong in the following arteries: occlusion of the MCA segment M1: observer agreement 77%, kappa (κ) = 0.61, ICA: observer agreement 92%, κ 0.84; posterior cerebral artery segments P1 and P2: observer agreement 98%, κ 0.97; and basilar artery: observer agreement 96%, κ 0.92. Table 2 shows the correlation boxes and agreements of the free observers separately and in total.

Table 2.

Interobserver agreement of CT angiography between three observers (correlation box)

Parameter (n = 75)	Observer	κa/agreement (%)
		1	2	3
MCA M1	1	–	80.0%	94.7%
	2	0.53	–	80.0%
	3	0.83	0.53	–
	Total	0.61 (strong)/77.3%
MCA M2	1	–	78.7%	96.0%
	2	0.3	–	80.0%
	3	0.86	0.28	–
	Total	0.48 (moderate)/77.3%
ICA	1	–	93.3%	97.3%
	2	0.8	–	93.3%
	3	0.92	0.92	–
	Total	0.84 (very strong)/92.0%
PCA P1b	1	–	98.67%	100.0%
	2	0.97	–	98.67%
	3	1.00	0.97	–
	Total	>0.97 (very strong)/98.7%
PCA P2b	1	–	98.7%	100.0%
	2	0.97	–	98.67%
	3	1.00	0.97	–
	Total	>0.97 (very strong)/98.7%
T occlusionb	1	–	92.0%	97.3%
	2	0.84	–	92.0%
	3	0.79	0.84	–
	Total	0.33 (poor)/90.7%
ACA A1b	1	–	98.7%	100.0%
	2	0.97	–	98.7%
	3	1.0000	0.973	–
	Total	>0.97 (very strong)/98.6%
ACA A2	1	–	100.0%	100.0%
	2	1.00	–	100.0%
	3	1.00	1.00	–
	Total	1.0 (very strong)/100%
BAb	1	–	96.0%	100.0%
	2	0.92	–	96.00%
	3	1.00	0.92	–
	Total	>0.92 (very strong)/96.0%
VA V4	1	–	89.3%	98.7%
	2	0.46	–	88.00%
	3	0.85	0.36	–
	Total	0.48 (moderate)/88.0%

κ, kappa index; ACA A1 and A2, anterior cerebral artery segments A1 and A2; BA, basilar artery; ICA, internal cerebral artery; MCA M1 and M2, middle cerebral artery segments M1 and M2; n, number of observations; Observer 1, general neurologist; Observer 2, interventional neuroradiologist; Observer 3, neurologist-stroke specialist; PCA P1 and P2, posterior cerebral artery segments P1 and P2; VA V4, vertebral artery segment V4.

^aAgreement: κ < 0.4, poor; 0.41–0.60, moderate; 0.61–0.80, strong; κ > 0.81, very strong.

^bPrevalence adjusted bias–adjusted kappa index.

DISCUSSION

Acute occlusion of intracranial arteries is the most common cause of ischaemic stroke. Also, occlusion of intracranial arteries is an important predictor of stroke outcome.² Several studies have clearly shown that MT is a highly effective treatment of stroke if carried out in carefully selected patients. An early recanalization leads to a better outcome in certain patients.⁶ Based on the results of these studies, the guidelines for acute stroke treatment were modified.^4–6 The presence of large intracranial vessel occlusion and the door-to-needle time are the most important criteria for MT. Therefore, the diagnosis of arterial occlusion has a great impact on the method of treatment, and CTA is strongly recommended.^4,5,7 Advanced neuroimaging is recommended for patients in the therapeutic window for IVT or MT with borderline ischaemic changes according to ASPECT or in the case when the time of stroke onset is unknown.^4,7 The role of the radiologist in the diagnostic process is indispensable.

There is a lack of data on interrater reliability in clot detection using CTA. Several studies have demonstrated good interrater agreement of CTA evaluation only between radiologists.^11–13 Some studies confirmed the moderate or strong agreement between the neurologist and radiologist in case of proximal or distal occlusion of ICA or in case of CTA perfusion evaluation.^9,10

In the present study, 75 stroke patients treated by IVT or IVT and MT were analysed. The risk factors and the outcome were comparable with previous studies. It was found out that interrater agreement of CTA evaluation in acute-stroke patients between the neurologists and neuroradiologist was very strong in all territories besides T occlusion and V4 segments of the vertebral arteries. The strong agreement in our study could be explained by the training of both the neurologists. Owing to the low incidence of VA occlusion, only the interrater disagreement of CTA evaluation in anterior circulation in the brain was further re-evaluated during the second reading. In most cases, the observers found arterial occlusion, but they pointed out the site of the occlusion incorrectly (e.g., T occlusion vs MCA M1 occlusion; ICA vs T occlusion; and MCA M1 occlusion vs MCA M2). The raters were able to identify an occlusion generally with very strong reliability, but sometimes, they were not able to identify the specific occluded artery. There was no case of normal evaluation of CTA by the radiologist and arterial occlusion by the neurologists or vice versa.

CTA was normal according to the observers in 16 cases (21%). The normal CTA findings in acute stroke patients were published in previous studies, e.g. Kassem-Moussa and Graffagnino² reported a prevalence of 28% non-occlusion rates. According to the neuroradiologist, 76 occluded and 674 non-occluded arteries were found in all our patients in this present study. Therefore, Kappa agreement does not necessarily mean the diagnostic ability of neurologists to find an occlusion of the intracranial artery. However, both the diagnoses of an artery occlusion and non-occlusion are of equal importance to stroke management care and treatment.

Acute stroke management is crucial for reperfusion therapy. It has been demonstrated that a lack of specialized staff (an expert neuroradiologist and a stroke resident) in hospital during non-working hours reduces the number of carried out IVT, which has a critical outcome for the patient recovery and survival.^8,14,15

The present study has several limitations. Firstly, the CTA evaluation of stroke patients was not compared with healthy control scans. Secondly, the training of the neurologist in CTA reading before the study was not standardized and therefore its reproducibility is partly difficult. Thirdly, our study was carried out at a single centre with a limited number of patients. Fourthly, a great number of negative findings in our study can bias the results of our analysis.

CONCLUSION

Interrater agreement of CTA evaluation in acute-stroke patients between neurologists and the neuroradiologist was very strong in all artery territories besides T occlusion and intracranial VA. CTA is a very good method to evaluate arterial occlusion in patients with acute stroke. The ability to read CTA artery patency by neurologists could improve the management of acute stroke and MT care. It would support faster activation of the thrombectomy team, leading to shorter door-to-needle times and, eventually, better outcomes.