Diagnosis of Recanalization of the Intracranial Artery Has Poor Inter-Rater Reliability

Abstract

BACKGROUND AND PURPOSE:

Recanalization is the important outcome measure for acute stroke therapy. Several methods of recanalization assessment are used in clinical practice, but few studies have addressed their reliability. We, therefore, sought to assess interobserver reliability of the diagnosis of intracranial artery recanalization following intervention by using TIMI criteria.

MATERIALS AND METHODS:

The digital angiography scans of all patients with acute ischemic stroke during 2009 undergoing DSA and endovascular procedures at Ostrava University Hospital were assessed in the study. Images were retrospectively evaluated for intracranial artery recanalization on the TIMI scale by 2 experienced neuroradiologists who were blinded to clinical findings and to each other.

RESULTS:

The angiography scans of 43 patients (16 females; age, 70.5 ± 14 years; median baseline NIHSS score, 15 [IQR, 11–18]) were retrospectively evaluated in our study. At 3 months, 27% of patients had mRS scores ≤ 2 and mortality was 18%. Two radiologists diagnosed TIMI grades as follows: TIMI 0, 16%, and 16%; TIMI 1, 21%, and 8%; TIMI 2a, 32% and 29%; TIMI 2b, 13% and 16%; TIMI 3, 18, and 31%. Interobserver agreement for recanalization was weighted κ = 0.4 (95% CI, 0.2–0.6).

CONCLUSIONS:

The diagnosis of recanalization after interventional procedures was found to have poor interobserver agreement between 2 experienced neuroradiologists. TIMI criteria, therefore, do not permit reliable comparison of the efficacy of recanalization therapy among different studies.

Occlusion of cervical or intracranial arteries is the most common cause of acute ischemic stroke.¹ Recanalization improves outcome after stroke and is, therefore, considered an important surrogate outcome measure for the efficacy of therapeutic intervention in patients with acute ischemic stroke.^1–4 However, there is, at present, no consensus on how intracranial artery recanalization should be evaluated. Various methods of assessment used in clinical practice include TICI and TIMI scales and their variants, but only limited data are available concerning their reliability.^5–7 The aim of the present study was to evaluate interobserver reliability of assessment by using TIMI criteria for intracranial artery recanalization following intervention.

Materials and Methods

All patients with acute ischemic stroke during 2009 undergoing DSA and endovascular procedures at Ostrava University Hospital were enrolled in the study. Eligibility criteria for the endovascular procedure were as follows: time of ≤8 hours from stroke onset, moderate or severe neurologic deficit (NIHSS score, >8), occlusion of a major intracranial artery on CT angiography, and failure of intravenous thrombolysis (nonrecanalization on transcranial Doppler or no clinical improvement after 30 minutes of intravenous thrombolysis infusion).

We recorded and evaluated demographic and clinical data: age, sex, time from symptom onset to the endovascular procedure, baseline NIHSS score, type of recanalization procedure, results of nonenhanced CT and CT angiography scans before the procedure and nonenhanced CT scans after the procedure, and mRS at 3 months after stroke.

DSA images were recorded according to a standard protocol. The baseline and final DSA images were retrospectively evaluated by 2 neuroradiologists for the site of intracranial artery occlusion and for the presence or absence of arterial recanalization. Both neuroradiologists were experienced in performing mechanical thrombectomy procedures, but they did not pass a certified training in TIMI (TICI) classification before the study.

The 2 neuroradiologists were blinded to the clinical findings and to each other. Recanalization of the intracranial artery was assessed by using modified TIMI classification³ (Table 1).

Table 1:

Modified TIMI classification³

Classification
0 = No perfusion
1 = Perfusion past the initial obstruction but limited distal branch filling with little or slow distal perfusion
2a = Perfusion of less than half of the vascular distribution of the occluded artery (eg, filling and perfusion through 1 M2 division)
2b = Perfusion of half or greater of the vascular distribution of the occluded artery (eg, filling and perfusion through 2 or more M2 divisions)
3 = Full perfusion with filling of all distal branches

Data Analyses and Statistical Analyses

Interobserver agreement was assessed by using linear weighted κ statistics. Linear weighting here denotes that the same weights were assigned to the difference between adjacent TIMI categories (eg, TIMI 0 versus 1 or TIMI 2 versus 3). Agreement was considered poor with κ <0.4, good with 0.41–0.75, and excellent with >0.75.

Results

A total of 43 patients were considered candidates for intervention. Of these, 1 patient was excluded because no recanalization therapy was performed after the initial angiography. A further 4 patients were excluded as a result of errors in transferring images to the radiologist located in another hospital via the PACS. The scans of the remaining 38 patients were analyzed. Detailed patient characteristics are shown in Table 2. Eleven (27%) patients had mRS ≤ 2 at 3 months.

Table 2:

Patient characteristics

Characteristics	Data
Patients enrolled	N = 43
Age (mean)	70.5 ± 14
Male sex (%)	27 (63%)
Hypertension	61%
Hyperlipidemia	38%
Diabetes mellitus	24%
Coronary heart disease	42%
NIHSS score (median) (IQR)	15 (11–18)
Time from admission to procedure (median) (IQR)	59 (29–113)
mRS <2 at 3 months	27%
CT angiography before procedure	37
MCA (M1 segment) occlusion	26
MCA (M2 segment) occlusion	4
Intracranical internal carotid artery occlusion	2
Posterior cerebral artery occlusion	1
Basilar artery occlusion	7
Endovascular procedures
Angioplasty	12
Intra-arterial thrombolysis	4
Mechanical embolectomy (Solitaire)a	1
Stenting (Wingspan)b	22
Combined therapy (IVT + endovascular procedures)	22

^aev3, Irvine, California.

^bBoston Scientific, Natick, Massachusetts.

Two radiologists diagnosed TIMI grades as follows: TIMI 0 16% and 16%; TIMI 1, 21% and 8%; TIMI 2a, 32% and 29%; TIMI 2b, 13% and 16%; and TIMI 3, 18% and 31%. Interobserver agreement for recanalization was weighted κ = 0.4 (95% CI, 0.2–0.6) (Table 3).

Table 3:

Interobserver results of recanalization on a modified TIMI scale: cross-tabulation

Observer B	Observer A			TIMI 2b	TIMI 3	Total
	TIMI 0	TIMI 1	TIMI 2a
TIMI 0	4	0	1	1	0	6
TIMI 1	1	1	1	0	0	3
TIMI 2a	0	4	5	1	1	11
TIMI 2b	0	1	2	2	1	6
TIMI 3	1	2	3	1	5	12
Total	6	8	12	5	7	38

For patients with MCA versus non-MCA stroke, the weighted κ was 0.39 (95% CI, 0.2–0.6) and 0.45 (95% CI, 0.2–0.9), respectively.

Discussion

In this study, we report that there was poor agreement in the assessment of recanalization on the TIMI scale by 2 experienced radiologists. Our results, therefore, challenge the reproducibility of recanalization assessment after interventional procedures. As a consequence, it is likely to be difficult or even impossible to compare results based on TIMI scores between different clinical trials, recanalization devices, or case series. Furthermore, our results could explain, in part, why the association between recanalization and outcome differs substantially among different studies.^2–6

Few studies have addressed the reproducibility of recanalization assessment following intervention. The Interventional Management of Stroke II (IMS II) study³ reported that 41% of evaluations of recanalization by TIMI were discordant, mostly due to over-rating of the TIMI score by the clinical site and under-rating by the core laboratory. In the IMS I study, TIMI and arterial occlusive lesion scores had only modest agreement. It has become common practice to compare revascularization trials with historical controls. For these comparisons to be valid, we must use the same revascularization end points.⁸

Several factors could explain the low reproducibility of TIMI scores between different observers. First, the TIMI scale was initially developed for the assessment of reperfusion in patients with myocardial infarction; it is possible that the scale is not well-suited for the assessment of recanalization in stroke. Second, neuroradiologists are insufficiently trained in TIMI grading. Third, radiologists who do not themselves perform the procedure might be less confident in interpreting the images—this feature could explain the most discrepant readings (eg, TIMI 3 or TIMI 2a versus 0, which occurred in 2 cases). Finally, the utility of TIMI could differ between different intracranial territories, though our data suggest that the inconsistency of interobserver agreement did not differ for MCA and non-MCA territories.

A major limitation of this study is its retrospective design; potentially this could have affected the quality and standardization of the images. Our study was also performed at a single center with a limited number of patients.

Conclusions

Interobserver agreement in the evaluation of intracranial artery recanalization following endovascular treatment in our study was poor. A more reliable tool for the assessment of recanalization in stroke is needed. The process of evaluation of recanalization must be unified, and discrepancies in the recanalization rate should be decreased by assessment images in a core laboratory. Training programs for improving the evaluation of recanalization should be established.

ABBREVIATIONS:

CI

confidence interval

IQR

interquartile range

IVT

intravenous thrombolysis

mRS

modified Rankin Scale

TICI

thrombolysis in cerebral infarction

TIMI

thrombolysis in myocardial infarction