Factors Associated with Decreased Accuracy of mTICI Scoring Among Neurointerventionalists During Thrombectomy

Abstract

Background and Purpose:

The modified thrombolysis in cerebral infarct (mTICI) score is used to grade angiographic outcome after endovascular thrombectomy (ET). We sought to identify factors that decrease accuracy of intraprocedural mTICI.

Methods:

We performed a two-center retrospective cohort study comparing operator (n=6) mTICI scores to consensus scores from blinded adjudicators. Groups were also assessed by dichotomizing mTICI scores to 0–2a versus 2b–3.

Results:

130 ET procedures were included. Operators and adjudicators had pairwise agreement in 96 cases (73.8%). Krippendorff’s alpha was 0.712. Multivariate analysis showed ET overnight (Odds Ratio (OR)=3.84, 95% confidence interval (95CI)=1.22–12.1), lacking frontal (OR=5.66, 95CI=1.36–23.6) or occipital (OR=7.18, 95CI=2.12–24.3) region reperfusion, and higher operator mTICI scores (OR=2.16, 95CI=1.16–4.01) were predictive of incorrectly scoring mTICI intraprocedurally. With dichotomized mTICI scores, increasing number of passes was associated with increased risk of operator error (OR=1.93, 95CI=1.22–3.05).

Conclusions:

In our study, mTICI disagreement between operator and adjudicators was observed in 26.2% of cases. Interventions that took place between 22:30–4:00, featured frontal or occipital region non-perfusion, higher operator mTICI scores, and increased number of passes had higher odds of intraprocedural mTICI inaccuracy.

Introduction

Successful reperfusion is a key predictor for better long-term outcomes and less disability following endovascular thrombectomy (ET).^1,2 Percent of reperfusion after ET is most commonly graded using the modified thrombolysis in cerebral infarct (mTICI) score.³ Many patients with high mTICI grades still have variable clinical courses.^1,3–5 Self-reported adjudication of angiographic outcomes tends to provide more favorable assessment of technical success when compared to core lab analysis, possibly resulting in these observed variable clinical courses.^6,7

Factors responsible for under- or overestimation of the degree of reperfusion by operators are largely unknown. We aim to characterize these factors by analyzing the effect of demographic, clinical, and imaging information available to the interventionalist, as well as procedural details.

Materials and Methods:

Data from this study are available from the corresponding author upon reasonable request. This study was approved by the local institutional review boards with a waiver for informed consent. Consecutive cases of ET for anterior circulation strokes from two comprehensive stroke centers between January 2018 and April 2020 were included. Those lacking complete digital subtraction angiograms (DSAs) of the intracranial cavity were excluded. Both participating centers had a similar imaging protocol for patient selection for ET consisting of noncontrasted computed tomography (CT) and CT angiography. For patients presenting outside of the 6-hour window for symptom onset (or last known normal), CT perfusion was also performed. Both centers generally considered patients as suitable candidates for ET based on the following criteria: stroke symptom onset up to 24 hours, NIHSS of 6 and above, and LVO defined as occlusion of the ICA, MCA M1, or M2/3 segment. Patients who were eligible for intravenous thrombolysis received IV recombinant tissue plasminogen activator (rtPA) prior to ET. Intravenous heparin bolus was not administered during or prior to thrombectomy. Baseline, imaging, and procedural characteristics were collected retrospectively from electronic records. Patients were not excluded on the basis of demographics, at-home medication use, or other treatments during hospitalization to present a more heterogeneous and generalizable data set.

Operators graded reperfusion using the mTICI scale. To determine adjudicated assessments of mTICI, DSAs were reviewed by two neurointerventionalist operators, experienced in core lab analysis, blinded to technical details (including procedural mTICI) and clinical data to reduce bias. All ET procedures were originally performed using biplane DSA equipment. These adjudicators graded reperfusion together by comparing final anteroposterior and lateral DSAs to initial anteroposterior and lateral DSAs demonstrating the occlusion until agreement was reached. We followed the definition of reperfusion as described in the consensus statement by Zaidat et al.³ Tissue level cerebral perfusion was evidenced by a capillary blush and classified as following grades: 0 – no reperfusion, 1 – antegrade reperfusion past the initial occlusion but no distal perfusion, 2a – antegrade reperfusion of less than half of the previously occluded target artery territory, 2b – antegrade reperfusion of more than half of the previously occluded target artery territory but still incomplete, 3 – complete antegrade reperfusion of the previously affected target artery territory. 2C grading (near complete antegrade perfusion except for slow flow in a few distal cortical vessels) was not included in our adjudicated classification as this subcategory was not used by the original operators from the participating sites.

Regional location of incomplete reperfusion was not routinely evaluated by the original operators performing ET procedures however this was assessed retrospectively by the adjudicators. Using the territorial distribution of the 12 cortical areas described by Gibo et al., we created a regional template to distinguish between the frontal, parietal, temporal, and occipital regions on lateral DSAs.⁸ Specifically, the frontal region included the orbitofrontal, prefrontal, precentral, and central areas; the parietal region consisted of the anterior and posterior parietal areas; the temporal region included the temporopolar, anterior, middle, and posterior temporal areas; and the occipital region included the angular and temporo-occipital regions (Supplementary Figure I).

Statistical analysis

Statistical analysis was performed using IBM SPSS Statistics Version 25 (IBM Corp., Armonk, NY). P-values ≤0.05 were considered statistically significant. Continuous variables were compared using Student’s t-tests or Welch’s t-tests, and categorical variables using Chi-square tests. Inter-rater reliability was assessed with percentage agreement, the Wilson score, and Krippendorff’s alpha with benchmarks suggested by Krippendorff.^9,10 A receiver operating curve was used to develop time cut-offs most associated with mTICI score disagreements. Multivariate analysis with stepwise logistic regression was performed. These methods were then pursued again in our sample but using a dichotomized design, dividing mTICI into good and bad recanalization (≥2b versus ≤2a) following Heiferman et al’s previous methodology.¹¹

Results:

One-hundred forty ET procedures on anterior circulation large vessel occlusions (LVOs) were identified. After excluding patients with incomplete DSAs, 130 remained. 21.5% of patients received IV rtPA prior to thrombectomy. Of these, there was pairwise agreement between operator and adjudicators in 96 of the 130 patients (73.8%; 95% confidence interval (95CI)=65.7–80.6%). Quantities and relative proportions of mTICI scores are displayed in Figure 1. In 8 cases (6.2%; 95CI=3.1–11.7%), mTICI was underestimated and in 26 cases (20.0%; 95CI=14.0–27.7%), mTICI was overestimated by operators. Krippendorff’s alpha was 0.712 (95CI=0.605–0.811), suggesting a level of agreement acceptable for tentative conclusions (≥0.667) but below Krippendorff’s suggested benchmark of alpha ≥0.800.¹⁰ Of overestimated cases, 26.1% featured <50% of area reperfusion, 13.0% featured 51–75% of area reperfusion, and 39.1% featured 76–90% of area reperfusion.

Figure 1.

Distribution of mTICI scores between pairwise and non-pairwise agreement groups.

Demographics, clinical characteristics, and procedural variables are summarized in Table 1. No demographic was significantly different between the groups; reperfusion to last known well was not significantly different between the groups. ETs performed overnight (between 22:30–4:00 per receiver operating curve analysis) had 2.92-times greater odds of inaccuracy in operator mTICI scoring (95CI=1.12–7.570). If frontal or occipital regions lacked reperfusion (see examples in Figure 2), operators were more likely to incorrectly score mTICI (odds ratio (OR)=2.98, 95CI=1.18–7.53; OR=3.68, 95CI=1.61–8.41; respectively). With parietal region non-reperfusion, operators were more likely to correctly score mTICI intraprocedurally (OR=2.91, 95CI=1.27–6.62).

Table 1.

Demographic and procedural variables

	Number of Patients (% Missing)	No Pairwise Agreement (N=34) (%)	Pairwise Agreement (N=96) (%)	p-value
Age at procedure; Mean±2SE	130(0.0)	67.2±5.09	70.3±3.06	0.303
Sex: Female	130(0.0)	17(50.0)	58(60.4)	0.291
Hypertension	108(16.9)	24(21.5)	59(61.5)	0.298
Diabetes	108(16.9)	10(8.0)	21(23.0)	0.467
Hyperlipidemia	108(16.9)	16(13.2)	35(37.8)	0.274
Atrial fibrillation	108(16.9)	11(11.9)	35(34.1)	0.825
Congestive heart failure	108(16.9)	7(5.2)	13(14.8)	0.396
Current smoker	108(16.9)	6(4.4)	11(12.6)	0.371
Cocaine use	108(16.9)	1(0.8)	2(2.2)	0.598
Coronary artery disease	108(16.9)	9(7.3)	19(20.7)	0.454
Current anticoagulation use	108(16.9)	5(6.0)	18(17.0)	0.790
Current antiplatelet use	108(16.9)	12(9.9)	26(28.1)	0.362
Previous stroke	108(16.9)	7(7.8)	23(22.2)	0.809
Baseline NIHSS; Mean±2SE	130(0.0)	15.1±2.24	15.4±1.358	0.775
Baseline ASPECTS	130(0.0)			0.126
Low(0–5)		8(23.5)	12(12.5)
High(6–10)		26(76.5)	84(87.5)
Laterality	130(0.0)			0.193
Left		14(41.2)	52(54.2)
Occlusion location	130 (0.0)			0.470
Internal Carotid Artery		8 (23.5)	20 (20.8)
M1		20 (58.8)	48 (50.0)
M2		5 (14.7)	26 (27.1)
M3		1 (2.9)	2 (2.1)
Last known well to reperfusion; Mean±2SE	99 (23.8)	441±137	555±119	0.286
Weekend thrombectomy	130(0.0)	8(23.5)	21(21.9)	0.842
Overnight thrombectomy	130(0.0)	10(29.4)	12(12.5)	0.024*
Procedure duration; Mean±2SE	130(0.0)	42.1±10.3	40.4±6.10	0.772
Number of passes; Mean±2SE	130(0.0)	2.44±0.692	2.45±0.342	0.985
Regions lacking perfusion	128(1.5)
Frontal		11(32.4)	13(13.8)	0.018*
Motor		9(26.5)	22(23.4)	0.721
Parietal		18(52.9)	72(76.6)	0.010*
Temporal		6(17.6)	17(18.1)	0.955
Occipital		18(52.9)	22(23.4)	0.001*

SE-Standard Error; NIHSS-National Institute of Health Stroke Scale; ASPECTS-Alberta Stroke Program Early CT Score; M1: Middle Cerebral Artery 1; M2: Middle Cerebral Artery 2; M3: Middle Cerebral Artery 3

^*indicates statistical significance

Figure 2.

Lateral DSA projections demonstrating regions lacking reperfusion.

AB-Demonstration of frontal zone non-reperfusion

CD-Demonstration of occipital zone non-reperfusion

Multivariate analysis found that thrombectomies overnight (OR=3.84, 95CI=1.22–12.1), lacking frontal (OR=5.66, 95CI=1.36–23.6) or occipital (OR=7.18, 95CI=2.12–24.3) region reperfusion, and higher operator mTICI scores (OR=2.16, 95CI=1.16–4.01) were predictive of incorrectly scoring mTICI intraprocedurally whereas parietal non-reperfusion was protective (OR=0.328, 95CI=0.117–0.926).

Dichotomous mTICI scale

There were 113 cases that the operator graded as good recanalization (107; 88.4% had pairwise agreement). Of 17 cases graded as bad recanalization, 14 (11.6%) had pairwise agreement (p=0.095). Krippendorff’s alpha was 0.718 (95CI=0.529–0.874).

No demographic characteristics were significantly different. Frontal non-reperfusion and occipital non-reperfusion were both associated with increased rates of nonagreement (p=0.012, p=0.026, respectively). Increasing number of passes was associated with nonagreement (nonagreement=3.78±1.14 versus agreement=2.35±0.316; p=0.018; supplementary figure II).

Multivariate analysis found increasing number of passes (OR=1.93, 95CI=1.22–3.05), and frontal (OR=13.3, 95CI=11.65–108) or occipital (OR=14.8, 95CI=1.82–122) region non-reperfusion were predictive of incorrectly scoring intraprocedural mTICI whereas temporal region non-reperfusion was protective (OR=0.004, 95CI=0.000–0.111).

Discussion:

Our results are confirmatory of those from Zhang et al. who first reported operators’ tendency to overestimate degree of reperfusion. Though they found overestimation was greatest with mTICI scores of 1 and 2a, in our cohort, mTICI 2b and 3 had the most misinterpretation.¹² Erroneous overestimation of mTICI may influence interventionalists to prematurely terminate ET, fail to achieve optimal reperfusion, and thus reduce chances of good clinical outcome post-thrombectomy.¹² In fact, using the recently published medium vessel occlusion criteria in combination with consensus adjudicator reperfusion percentages suggested that 78.2% of these overestimated cases may have been amenable to further thrombectomy.¹³

Overnight ET (22:30–4:00) increased the odds of the operator misclassifying mTICI. Operator fatigue is one potential explanation. Another is operator bias towards wanting a certain outcome so the procedure could be justifiably stopped. While several publications showed that clinical outcomes of ET during off-hours are similar to on-hours, the definitions of off-hours in these studies were broad (18:00–08:30).^14,15 Our stricter definition of the overnight period was predictive of mTICI error, potentially indicating it as a more realistic representation of when the average neurointerventionalist rests. A recent 10-center prospective study of ET in the U.S. showed 23% of ET were performed between 20:00–04:00, indicating potentially one-in-four ET procedures are at increased risk of operator mTICI overestimation.¹⁶ Increasing numbers of passes was predictive of operator mischaracterizing good versus bad recanalization, possibly due to frustration, fatigue, or operator bias.

Operators were more likely to miss frontal and occipital region non-perfusion and were more likely to accurately recognize parietal and temporal regions non-perfusion. One potential explanation is because the frontal and occipital regions are more variable in perfusion due to various collateral (including leptomeningeal collaterals) supply from the adjacent territories making its accurate assessment more challenging.

Previous studies have analyzed interrater agreement of mTICI. Our interrater mTICI agreement evaluated with Krippendorff’s alpha appears in line with these previous studies.^11,12,17 However, prior studies solely acknowledge these differences exist, whereas we analyzed factors that might predispose to these differences. Similarly, there are no studies evaluating for factors affecting intraprocedural coronary artery reperfusion during percutaneous coronary intervention either.

Identifying factors that are associated with higher likelihood of inaccurate TICI interpretation such as interventions performed at night time or with a higher number of passes can help recognize cases when an additional “quality check” may be warranted. For example, either prompting the operator to perform a second review of DSA or seek feedback from the radiology technologist on the case who normally processes and reviews DSA in real time. Our study further supports the potential need for automated tools capable of accurate and objective grading of cerebral perfusion. Shiraz Bhurwani et al. demonstrated the feasibility of semi-quantitative diagnostic tools which encode hemodynamic properties in DSAs to automatically assess reperfusion during ET procedures based on the mTICI scale classification.¹⁸ The use of data-driven approaches, such as convolutional neural networks opens an opportunity to grade reperfusion during ET in real time, accounting for regional and territorial locations of perfusion deficits or distal embolization alerting the operator and aiding in the decision-making process.

Since both participating centers did not routinely use 2c in mTICI grading in clinical practice, our study did not include it, limiting generalizability of our results. Also, this study is limited in its retrospective nature as well as the limited number of centers included further limiting the generalizability and statistical power of our study. Further, the lack of centralized adjudication likely increased the bias of our study. 113 patients out of 130 were graded by the operator as mTICI 2b or 3. This lack of heterogeneity in categories results in an artificially high estimate of chance agreement and artificially low Krippendorrf alpha. This limitation is similar to in some previous studies such as in the HERMES collaboration.¹¹ Our centers do not reliably save all angiographic runs in a thrombectomy, especially after unsuccessful passes. Future research analyzing the relationship between mTICI accuracy and believed utility of further passes may be warranted.

Conclusions:

In our study, mTICI disagreement between operator and adjudicators was observed in 26.2% of cases. Interventions that took place between 22:30–4:00, featured frontal or occipital region non-perfusion, higher operator mTICI scores, and increased number of passes had higher odds of intraprocedural mTICI inaccuracy.

Non-Standard Abbreviations and Acronyms

ET

Endovascular Thrombectomy

mTICI

modified Thrombolysis In Cerebral Infarct

DSA

Digital Subtraction Angiogram

ASPECTS

Alberta Stroke Program Early CT Score

LVO

Large Vessel Occlusion

95CI

95% Confidence Interval

OR

Odds Ratio

SE

Standard Error