Interobserver Reproducibility of Signal Intensity Ratio on Magnetic Resonance Angiography for Hemodynamic Impact of Intracranial Atherosclerosis

Abstract

Background

Changes of signal intensities (SIs) across intracranial atherosclerosis (ICAS) on magnetic resonance angiography (MRA) may reflect hemodynamic impact of the lesion. We evaluated the interobserver reproducibility of an index termed signal intensity ratio (SIR), developed in a previous study to represent the changes of SIs across ICAS on MRA.

Methods

Symptomatic ICAS on MRA were retrospectively recruited. Two observers respectively evaluated the images and calculated the SIR as follows, blinded to each other’s readings: SIR = (mean poststenotic SI − mean background SI)/(mean prestenotic SI − mean background SI). Statistical analyses were performed to evaluate the interobserver reproducibility of this index.

Results

A total of 102 symptomatic ICASs were enrolled, with 36 (35.3%) lesions of 50%–69% MRA stenoses and others being 70%–99% stenoses or flow void on MRA. Overall, mean SIRs were not significantly different between the 2 observers (.92 ± .17 versus .93 ± .17; mean difference −.006 ±.09; P =.496 for paired t test). Pearson correlation coefficients were >.80 for all analyses, indicating strong linear correlations between SIRs by the 2 observers. Bland–Altman analysis for SIRs of all cases showed no systematic bias between the 2 observers. For different cut-points ranging from .75 to 1.00, the kappa statistics were mostly greater than .6 and interobserver agreements were all greater than 80%, implying substantial agreement between observers.

Conclusions

SIR was demonstrated to be highly reproducible between observers in the present study. Future studies are warranted to further explore the role of this index in comprehensive evaluation and risk stratification of symptomatic ICAS.

Introduction

Intracranial atherosclerosis (ICAS) is the most common cause of ischemic stroke and transient ischemic attack (TIA) in Asian populations and throughout the world.¹ Patients with anatomically severe (70%–99%) ICAS face a high risk of recurrent ischemic stroke in the related territory.² However, those with anatomically moderate (50%–69%) stenosis might also be at considerable risk of recurrent ischemic stroke, in Caucasians and in Asians, according to theWarfarin–Aspirin Symptomatic Intracranial Disease (WASID) trial² and the Chinese Intracranial Atherosclerosis Study.³ Therefore, percentage of maximal luminal stenosis is not the only factor that determines the outcomes of symptomatic ICAS, for which reason evaluating hemodynamic impact of the lesion is important in identifying patients at high risks of recurrent ischemic stroke.

Based on the contrast mechanism of time-of-flight (TOF) magnetic resonance angiography (MRA), which is flow-related enhancement, change of signal intensities (SIs) across an ICAS on TOF MRA could partly reflect hemodynamics of flowing blood.^4,5 In a previous study, we, therefore, developed an index named signal intensity ratio (SIR) on TOF MRA to systematically assess the hemodynamic effects of an ICAS, by calculating the adjusted ratio of post- and prestenotic SIs.⁶ In a following study, we identified a significant negative linear relationship (P =.011) between SIR values and infarct volumes on diffusion-weighted MRI in 36 patients with symptomatic ICAS in the anterior circulation.⁷ Furthermore, in the MRA imaging archive from the Stroke Outcomes and Neuroimaging of Intracranial Atherosclerosis study,⁸ SIR below median (SIR < .9) was found to be related to recurrent stroke in the territory in multivariate analysis (hazard ratio = 10.9, 95% confidence interval = 2.0–58.9; P = .001).⁹

As a noninvasive and easy-to-perform method, measurement of SIR was found to be of high intraobserver reproducibility in 26 stenosed intracranial arteries.⁶ To further validate this method and to facilitate relevant studies in the future, we aimed to evaluate its interobserver reproducibility in the present study.

Methods

Materials and Subjects

A retrospective study was performed using MRA images of patients from Prince of Wales Hospital, Hong Kong SAR, and Tiantan Hospital, Beijing, China, from the Chinese Intracranial Atherosclerosis Study data set.^3,10 Symptomatic ICASs (50%–99%, Warfarin–Aspirin Symptomatic Intracranial Disease criteria¹¹) in one of the following arterial segments identified on TOF MRA were evaluated in the present study: intracranial portion of internal carotid artery (ICA), M1 segment of middle cerebral artery (MCA-M1), or basilar artery (BA).

MRA

MRA was acquired with 1.5 or 3.0 T MR scanners. Parameters for the 3D TOF MRA source images were as follows: slice thickness 0.8–1.2 mm, repetition time 23–40 ms, echo time 7 ms, and flip angle 20–25°, for 1.5 T MR; and slice thickness .6–.9 mm, repetition time 21 ms, echo time 4 ms, and flip angle 18° for 3.0 T MR. Twelve maximum intensity projections (MIPs) with 15° separation were obtained for anterior–posterior and lateral views, respectively. Locations and anatomic severities of the symptomatic stenoses were defined by reviewing radiological reports. Flow void or 70%–99% stenosis was regarded anatomically severe, and 50%–69% stenosis was regarded anatomically moderate.

Measurement and Calculation of SIR

MRA images were separately evaluated by 2 observers in the present study, blinded to each other’s readings. SIR was calculated as follows, adjusted for mean background SI: SIR = (mean poststenotic SI − mean background SI)/(mean prestenotic SI − mean background SI), where mean pre- and poststenotic SIs were measured on the MIP showing the highest percentage of stenosis of the target ICAS. Mean background SI was the mean SI of 2 region-of-interests within the left and right halves of the same MIP. The methodology for measurement and calculation of SIR was the same as what was described in our previous work,⁶ except that in the present study we measured the background SIs on the same MIP that the pre-and poststenotic SIs were measured. For better view of the target vessels, MCA-M1 stenoses were mostly evaluated on anterior–posterior MIPs, and intracranial ICA and BA stenoses were mostly evaluated on lateral MIPs.

The 2 observers used 2 different software programs to evaluate the images, Phillips DICOM viewer R2.6 and OsiriX Imaging Software 5.6, so that the results of the assessment of interobserver reproducibility in this study would be more generalizable. For each lesion, each observer performed SI measurement and SIR calculation twice, blinded to clinical data, and then the average of the 2 measures was considered as the SIR value for the lesion from this observer.

Statistical Analysis

For each lesion, the absolute difference between the 2 observers was defined as the absolute value of difference, and the relative difference was calculated as the ratio of absolute difference and the SIR value from observer 1. Paired t tests were performed to compare mean SIRs by the 2 observers. Pearson correlation coefficient and Bland–Altman analysis were also performed to evaluate the interobserver reproducibility of SIR. Besides, scatter-plots were performed to further reveal the relationships between measures by the 2 observers. Kappa tests were also conducted for several cut points of SIR. For each cut point, it was considered as interobserver agreement on a lesion when the 2 observers agreed on the classification of SIR in terms of “normal” (above the cut point) versus “abnormal” (below the cut point) SIR values. All statistical analyses were performed using PASW Statistics (version 18.0) and MedCalc Software (version 11.4). Two-sided P values less than .05 were considered statistically significant.

Results

A total of 102 symptomatic ICASs on TOF MRA were evaluated in the present study, located in ICA, MCA-M1, and BA in 30 (29.4%), 58 (56.9%), and 14 (13.7%) cases, respectively. Thirty-six (35.3%) lesions were 50%–69% stenoses, and the others were 70%–99% stenoses or flow void on MRA.

Results of paired t tests, mean absolute and relative differences, and Pearson correlation coefficients between measures by the 2 observers, for all cases and for lesions with different locations or different anatomic severities, are shown in Table 1. Overall, mean SIRs were not significantly different between the 2 observers, which were .92 ± .17 and .93 ± .17, respectively, with the mean difference of −.006 ± .09 (P = .496 for paired t test). Besides, SIRs were not significantly different between observers when analyzed categorically by different locations or anatomic severities of ICAS, with all P values greater than .05 for paired t tests (Table 1). And for all the analyses, Pearson correlation coefficients were greater than .80 (Table 1), which indicated strong linear correlations between SIRs calculated by the 2 observers, further shown in Figures 1 and 2. The Bland–Altman plot for SIRs of all cases is shown in Figure 3, which indicates no systematic bias between the 2 observers.

Table 1.

Interobserver reproducibility of SIR measurement from 2 observers

Cases	N	Paired t test		Mean absolute difference*	Mean relative difference (%)†	Pearson correlation coefficient‡
		Mean difference	P value
Overall	102	−.006	.496	.07	8	.847
Anatomic severity of ICAS
50%–69%	36	.007	.604	.06	6	.822
70%–99% or flow gap	66	−.014	.255	.08	8	.856
Location of ICAS
ICA	30	−.023	.118	.07	7	.878
MCA-M1	58	.007	.623	.08	9	.817
BA	14	−.024	.300§	.06	7	.880∥

Abbreviations: BA, basilar artery; ICA, internal carotid artery; ICAS, intracranial atherosclerosis; MCA-M1, M1 segment of middle cerebral artery; SIR, signal intensity ratio.

^*Absolute difference for each case = |(SIR of observer 1 − SIR of observer 2)|.

^†Relative difference for each case = |(SIR of observer 1 − SIR of observer 2)|/SIR of observer 1.

^‡All P values were <.001.

^§P value for Wilcoxon signed-rank test.

^∥Spearman correlation coefficient.

Figure 1.

Scatterplot for SIRs of all cases (N = 102).

Figure 2.

Scatterplots for SIRs of 50%–69% ICAS (n = 36, A) and 70%–99% ICAS or signal void on MRA (n = 66, B).

Figure 3.

Bland–Altman plot for SIRs by observer 1 versus observer 2 shows no systematic bias for the 102 cases evaluated (mean difference = −.006 ± .09, P =.496 for paired t test).

Results of kappa tests for different cut points of SIR are shown in Table 2. For cut points ranging from .75 to 1.00, the kappa statistics were mostly more than .6, and the inter-rater agreement were all more than 80%, indicating substantial agreement between the 2 observers.

Table 2.

Kappa tests for different cut points of SIR

Cut point of SIR	Kappa (95% CI)	Interobserver agreement (%)
.75	.841 (.666–1.000)	97.1
.80	.547 (.338–.757)	86.3
.85	.617 (.451–.784)	84.3
.90	.645 (.492–.799)	83.3
.95	.661 (.514–.808)	83.3
1.00	.636 (.474–.798)	84.3

Abbreviations: CI, confidence interval; SIR, signal intensity ratio.

Discussion

In this study, we evaluated the interobserver reproducibility of SIR, an index developed to reflect hemodynamic impact of ICAS on TOF MRA. We found the index highly reproducible between observers in 102 cases.

Anatomic severity has been used as an important indicator for patient selection in research and the clinical field of stroke as far as ICAS is concerned. For instance, the Stenting and Aggressive Medical Management for Preventing Recurrent Stroke in Intracranial Stenosis trial was targeted at patients with symptomatic 70%–99% stenosis of a major intracranial artery.^12,13 Also, international guidelines have been using the percentage stenosis as an important indicator to guide treatment for patients with symptomatic ICAS.^14,15 However, hemodynamic significance, which is also an important aspect of ICAS, has been paid little attention in research and clinical practice. The situation has been changed in the cardiology area, in which fractional flow reserve, either measured during the performance of coronary angiography or obtained as a simulated, noninvasive fractional flow reserve using coronary computed tomographic angiography, has been found of high diagnostic accuracy of hemodynamic significance of coronary stenosis, and has been used in patient selection of percutaneous coronary intervention and relevant research fields.^16–19

TOF MRA, as a noninvasive and widely used imaging modality to evaluate intracranial arteries, simultaneously possesses flow and anatomic information. Downstream signal reduction or complete signal loss beyond a stenosis on MRA could reflect reduced or turbulent flow. Therefore, in a series of studies, we aimed to develop and validate a method to assess the hemodynamic significance of ICAS on MRA, so as to facilitate its comprehensive evaluation. Based on the mechanisms of TOF MRA, the index SIR could incorporate impact on flow from the lesion itself and from collaterals or other factors. In the present study, SIR of ICA, MCA-M1, or BA stenosis was demonstrated highly reproducible between the 2 observers, with high Pearson correlation coefficients and nonsignificant paired t tests. We did not recruit vertebral artery stenoses in this study because only the distal vertebral arteries were shown on most of the intracranial MRA images from our data set, in which case it would be inaccurate to measure the SIs on the edge of the image. Besides, some of the lesions were at the vertebrobasilar junction, which might also be inaccurate to measure.

Although a previous study had found that an SIR less-than .9 was related to recurrent stroke in the territory,⁹ we do not know the exact cut point to define a “normal” or “abnormal” SIR. So in this study, we performed kappa tests for several cut points of SIR ranging from .75 to 1.00 to evaluate its interobserver agreement. For the cut point of .75, the 2 observers agreed on over 95% of the cases in terms of “normal” or “abnormal” SIRs. For cut points above .80, inter-rater agreements were all approximately 85%. Though the kappa statistics for all cut points were greater than .6, indicating substantial interobserver agreement, there were a considerable percentage of cases on which the 2 observers disagreed with each other. This may partly be explained by the fact that the 2 observers were using different software programs for the measurement, in which case the MRA images appeared to some extent different by the 2 programs in terms of image contrast, brightness, and sharpness.

In conclusion, SIR was a highly reproducible index between observers for evaluating hemodynamic significance of ICAS on TOF MRA. Future studies are warranted to further explore the cut point to define normal and abnormal SIR and to explore the possible relationships between SIR and outcomes in patients with symptomatic ICAS.