Carotid magnetic resonance imaging for monitoring atherosclerotic plaque progression: a multicenter reproducibility study

Abstract

This study sought to determine the multicenter reproducibility of magnetic resonance imaging (MRI) and the compatibility of different scanner platforms in assessing carotid plaque morphology and composition. A standardized multi-contrast MRI protocol was implemented at 16 imaging sites (GE: 8; Philips: 8). Sixty-eight subjects (61 ± 8 years; 52 males) were dispersedly recruited and scanned twice within 2 weeks on the same magnet. Images were reviewed centrally using a streamlined semiautomatic approach. Quantitative volumetric measurements on plaque morphology (lumen, wall, and outer wall) and plaque tissue composition [lipid-rich necrotic core (LRNC), calcification, and fibrous tissue] were obtained. Inter-scan reproducibility was summarized using the within-subject standard deviation, coefficient of variation (CV) and intraclass correlation coefficient (ICC). Good to excellent reproducibility was observed for both morphological (ICC range 0.98–0.99) and compositional (ICC range 0.88–0.96) measurements. Measurement precision was related to the size of structures (CV range 2.5–4.9 % for morphology, 36–44 % for LRNC and calcification). Comparable measurement variability was found between the two platforms on both plaque morphology and tissue composition. In conclusion, good to excellent inter-scan reproducibility of carotid MRI can be achieved in multicenter settings with comparable measurement precision between platforms, which may facilitate future multicenter endeavors that use serial MRI to monitor atherosclerotic plaque progression.

Introduction

The fine delineation of vessel wall morphology by magnetic resonance imaging (MR, MRI) has made cardiovascular MR increasingly popular in longitudinal studies for monitoring atherosclerosis progression [1–10]. Additionally, the combination of contrast weightings enables accurate segmentation of plaques into areas of lipid-rich necrotic core (LRNC), calcification and fibrous tissue [11–13]. Understanding morphological and compositional changes of atherosclerotic plaques under various conditions may facilitate the pursuit of optimal management strategies.

For imaging biomarkers to be used in longitudinal studies, scan-rescan reproducibility provides critical information for study planning. Carotid MRI reproducibility has been primarily explored in single-center studies [14–16]. However, there is a pressing need for contemporary clinical studies to adopt multicenter designs, either to ensure large-scale recruitment or to study clinical conditions with low prevalence. Unique challenges exist for multicenter MRI studies considering that substantial variations in technique instrumentation are typically present among imaging sites. In brain MRI, pooling data from multiple sites have been shown to introduce variations and biases related to scanner platforms and acquisition protocols [17]. In carotid MRI, even with carefully matched imaging parameters, Saam et al. [18] noted systemic measurement differences between platforms at 1.5 T.

In this prospective study, we sought to determine the multicenter reproducibility of MRI and the compatibility of different scanner platforms in assessing carotid plaque morphology and composition. To provide informative data for future studies, subjects with various extent of carotid atherosclerosis were recruited to encompass the full spectrum of plaque severity; standardized MRI protocol and quality assurance procedures were implemented across sites; and the same scanner was consistently used for individual subjects.

Materials and methods

Study population

A multicenter MRI study (NCT00880178 and NCT01178320; http://clinicaltrials.gov) was carried out in parallel to the Atherothrombosis Intervention in Metabolic syndrome with low HDL/high triglycerides: Impact on Global Health outcomes (AIM-HIGH) trial [19]. AIM-HIGH trial participants, who were men and women aged 45 and older, with established atherosclerosis in coronary, cerebrovascular/carotid or peripheral arteries and dyslipidemia [19], were invited to participate in the MRI sub-study at ten imaging centers if they had no contraindications for MRI examination (e.g. metal implants, claustrophobia) or contrast injection (e.g. renal insufficiency). Similar subjects with identical inclusion criteria were also recruited at six non-AIM-HIGH imaging centers to increase the number of centers. Additionally, each center was restricted to enroll no more than six subjects to ensure an adequate sample distribution across centers.

A total of 68 subjects were recruited at 16 imaging centers in the US (n = 10), Canada (n = 3) and China (n = 3) (Supplemental Table 1). Each subject was scheduled for two carotid MRI scans at the same imaging center within 2 weeks. There was no change in medications or clinical status during the time interval; measurement differences were thus considered to primarily reflect scan-rescan variability of MRI examination and image analysis. Institutional review board approval was obtained at all participating sites. Enrolled subjects provided written informed consent.

Carotid MRI

The 16 imaging sites involved two platforms (Supplemental Table 1). All scans were performed at 3.0 T using commercially available carotid phased-array coils (GE: 6-channel, Neocoil LLC, Pewaukee, WI, USA; Philips: 8-channel, Shanghai Chenguang Medical Technologies, Shanghai, China) [20]. A standardized MRI protocol was implemented on both platforms with carefully matched sequences and parameters, which acquires multi-contrast, cross-sectional images around the carotid bifurcation (Supplemental Table 2). Post-contrast T1-weighted images were acquired about 5 min after contrast injection (Magnevist, Bayer Healthcare, Wayne, NJ, USA). Spatial resolution was 0.625 × 0.625 × 2 mm³ before interpolation. Total acquisition time was approximately 45 min.

Standardized quality assurance procedures were followed in image acquisition (Supplemental Fig. 1). All personnel who performed MRI scans for the study were trained by a core lab (University of Washington, Seattle, WA, USA). Phantom and volunteer scans were performed prior to real subject scans. During the study, images were assessed by the core lab within 48 h of acquisition, and the site was notified if the scan needed to be repeated because of inadequate image quality. In subjects with bilateral carotid plaques, the side with larger lesion (visual inspection) was designated as the index side for follow-up imaging.

Image review

Images were transferred and analyzed centrally using a custom-designed image analysis software package (CASCADE, University of Washington, Seattle, WA, USA) that facilitates multiple-series registration and semiautomatic plaque analysis [21]. Analysis was limited to the index side assigned during image quality check to ensure independent sampling. First and repeat scans were aligned using carotid bifurcation as a landmark, which were then separated, randomized, and analyzed independently using a streamlined multi-step approach.

One reader (J.S.) outlined lumen and outer wall boundaries on cross-sectional slices using a computer-assisted contouring tool. A second reader (K.Y.) performed peer-review which involves contour modification and discussion with the first reader as necessary. Lastly, an integrated plaque segmentation tool automatically classified wall area into LRNC, calcification and fibrous tissue, following a previously described algorithm [22]. After finishing all cases, slice-based area measurements were exported and aggregated for volumetric measurements (morphology: lumen, wall and outer wall volumes; composition: LRNC, calcification and fibrous tissue volumes), which indicate plaque progression patterns and are frequently used in previous studies [1–10].

Statistical analysis

Data was presented as mean ± standard deviation (SD) or median (inter-quartile range) for continuous variables as appropriate, and count (percentage) for categorical variables. Measurement reproducibility was summarized using the within-subject SD, the coefficient of variation (CV) and the intraclass correlation coefficient (ICC). Linear mixed models were used to decompose the total variance into within-subject, between-subject and between-site variances to compute reproducibility parameters. CV was defined as 100 % × within-subject SD/mean. ICC was defined as (total variance – within-subject variance)/total variance. Presence or absence of LRNC and calcification for each scan was defined as whether or not the component volume was greater than 0 mm³. Inter-scan agreement on component presence/absence was then quantified using Cohen's kappa [23]. Relationships between measurement value (within-subject mean) and precision (within-subject SD and within-subject CV) were tested using the Spearman correlation coefficient. Platform-specific within-subject SDs were computed and the ratio of these SDs were tested against 1 (no difference in measurement precision between platforms). These ratios were also adjusted for the magnitude of measurement, as the precision tended to be related to the magnitude. Specifically, the within-subject SD was modeled as a power function of the within-subject mean, so the adjusted SD ratio can be interpreted as the ratio of platform-specific SDs for subjects with the same mean value [24]. Confidence intervals (CIs) and corresponding p-values were computed using bootstrap technique, where sites were re-sampled as clusters. Data analyses were performed using R 2.14.1 (R Foundation for Statistical Computing, Vienna, Austria). Statistical significance was defined as p < 0.05 (two-tailed).

Results

All 68 subjects completed two scans with a mean time interval between scans of 12.1 ± 6.8 days (Fig. 1). The average age was 61 ± 8 years and 52 (76 %) were men (Table 1). Despite motion artifacts in some cases, all scans were deemed to have diagnostic image quality during image quality check. Therefore, there were no repeated scans or excluded subjects. The mean scan coverage was 28.8 ± 4.5 mm. Based on the first-scan data, 36 (53 %) and 23 (54 %) had detectable LRNC and calcification, respectively.

Fig. 1.

First and repeat scans on different scanner platforms. a A plaque with calcification (short arrows) and LRNC (long arrows) was scanned twice on GE platform. b A plaque with ulceration (short arrows) and LRNC (long arrows) was scanned twice on Philips platform. Primary weightings are presented to highlight specific features. The last image in each panel illustrates contour-based measurements (yellow contours indicate LRNC). CE-T1W contrast-enhanced T1-weighted, LRNC lipid-rich necrotic core, T1W T1-weighted, T2W T2-weighted, TOF time-of-flight

Table 1. Clinical and imaging characteristics of study population.

	Mean ± SD, median (IQR) or n (%)	Range
Clinical characteristics
Age, years	61 ± 8	45–80
Male sex	52 (76)
Current smokera	17 (25)
Hypertensiona	56 (84)
Diabetes mellitusa	17 (25)
LDL-C, mg/dla	81 ± 34	29–205
HDL-C, mg/dla	35 ± 6	20–56
Triglycerides, mg/dla	169 ± 64	80–388
Imaging characteristicsb
Lumen volume, mm3	1,239 ± 540	489–3,361
Wall volume, mm3	849 ± 264	426–1,694
Total vessel volume, mm3	2,088 ± 746	973–4,707
LRNC volume, mm3	1.2 (0, 27)	0–358
Calcification volume, mm3	0 (0, 3.5)	0–84
Fibrous tissue volume, mm3	751 ± 197	412–1,288

IQR inter-quartile range, LRNC lipid-rich necrotic core, SD standard deviation

^aExcluding 1 subject missing clinical information

^bBased on the first scan

Lumen, wall and outer wall volumes together provide a comprehensive characterization of vessel wall dimensions, all of which demonstrated small measurement errors (CV ranged from 2.5 to 4.9 %) and excellent reproducibility (ICC ranged from 0.98 to 0.99) with repeat scanning (Table 2). Weak but statistically significant correlations (r = 0.24–0.25, p = 0.04–0.05) were noted between within-subject mean and within-subject SD (absolute precision) (Fig. 2). No apparent relationships were noted for within-subject CV (relative precision) with within-subject mean.

Table 2. Reproducibility of MRI measurements on plaque morphology and tissue composition.

	N	Mean (mm3)	Median (mm3)	SD (mm3)	Within-subject SD (mm3)	CV (95 % CI) (%)	ICC (95 % CI)
Morphological measurements
Lumen volume	68	1,221	1,092	545	43	3.5 (2.9, 4.2)	0.99 (0.99, 1.00)
Wall volume	68	849	805	263	41	4.9 (4.1, 5.6)	0.98 (0.96, 0.98)
Total vessel volume	68	2,066	1,903	749	52	2.5 (2.1, 2.9)	0.99 (0.99, 1.00)
Compositional measurements
LRNC volume	46	42	22	74	15	36 (25, 47)	0.96 (0.89, 0.98)
Calcification volume	24	18	12	23	7.9	44 (28, 61)	0.88 (0.59, 0.96)
Fibrous tissue volume	68	750	729	195	37	5.0 (4.3, 5.5)	0.96 (0.94, 0.98)

CV coefficient of variation, ICC intra-class correlation coefficient, LRNC lipid-rich necrotic core, SD standard deviation

Fig. 2.

Bland–Altman plots on plaque morphology. Bland–Altman plots are shown for lumen volume (a), wall volume (b) and total vessel volume (c). A logarithmic scale is used for the x-axis to aid visualization as the measurements spanned a wide range. Note that the actual values shown are on the original scale. Dashed lines indicate the mean difference and dotted lines indicate 95 % limits of agreement

The numbers of subjects where LRNC was detected in both scans, only one scan and neither scan were 32 (47 %), 14 (21 %) and 22 (32 %), respectively (kappa: 0.58; 95 % CI [0.39, 0.78]). For calcification, the numbers were 21 (31 %), 3 (4 %) and 44 (65 %), respectively (kappa: 0.90; 95 % CI [0.79, 1.00]). LRNCs that were inconsistently detected were found to be smaller, with a median volume of 2 (1, 3) mm³, compared to 34 (5, 69) mm³ in those that were consistently detected on both scans. Furthermore, kappa tended to improve when subjects with LRNC volumes <1, 2, or 4 mm³ were excluded (kappa: 0.62, 0.69, and 0.80, respectively).

Variability on quantifying plaque tissue composition was estimated using only subjects that exhibited corresponding components in at least one scan (values were assigned zero if not detected on the other scan). Overall good to excellent reproducibility was observed (ICC ranged from 0.88 to 0.96). Compared to morphological measurements, quantification of compositional structures showed smaller within-subject SD but larger CV (LRNC: 36 %; calcification: 44 %) (Table 2). Notably, within-subject SDs of LRNC and calcification volumes positively correlated with within-subject means (r = 0.77 and 0.74, respectively; Fig. 3), whereas within-subject CVs showed negative correlations (r = −0.76 and −0.65, respectively).

Fig. 3.

Bland–Altman plots on plaque composition. Bland–Altman plots are shown for LRNC volume (a), CA volume (b) and fibrous tissue volume (c). A logarithmic scale is used for the x-axis to aid visualization as the measurements spanned a wide range. Note that the actual values shown are on the original scale. Dashed lines indicate the mean difference and dotted lines indicate 95 % limits of agreement

Pooled inter-scan variability was calculated for each platform separately. Within-subject SD, indicative of measurement precision, was not significantly different between the two platforms studied (Table 3). Furthermore, the SD ratios (See “Statistical analysis” for details) tended to get closer to 1 after adjustment to control for the positive relationship between within-subject SD and mean.

Table 3. Comparison of measurement errors between scanner platforms.

	GE		Philips		SD ratioa
	N	Within-subject SD (mm3)	N	Within-subject SD (mm3)	Crude (95 % CI)	p value	Adjusted (95 % CI)	p value
Morphological measurements
Lumen volume	35	32	33	51	1.6 (0.8, 2.3)	0.158	1.2 (0.8, 2.0)	0.257
Wall volume	35	36	33	47	1.3 (0.8, 1.9)	0.233	1.1 (0.7, 1.7)	0.560
Total vessel volume	35	49	33	52	1.1 (0.7, 1.5)	0.856	1.0 (0.7, 1.4)	0.884
Compositional measurements
LRNC volume	22	13	24	15	1.2 (0.3, 2.9)	0.555	0.9 (0.6, 1.3)	0.500
Calcification volume	10	9.7	14	5.7	0.6 (0.3, 2.9)	0.396	0.9 (0.6, 1.3)	0.515
Fibrous tissue volume	35	35	33	41	1.2 (0.9, 1.6)	0.198	1.0 (0.7, 1.5)	0.900

Abbreviations as in Table 2

^aThe crude SD ratios were computed as the ratio of platform-specific within-subject SDs. The adjusted ratios involved controlling for the within-subject mean of each individual, which tended to be positively associated with within-subject SD (Figs. 2, 3)

Discussion

With a rigorous multicenter design, we prospectively evaluated the reproducibility of MRI and the compatibility of scanner platforms in assessing carotid plaque morphology and composition. Although previous studies have shown that MRI-derived plaque morphological and compositional measurements are promising biomarkers for expanding our knowledge on atherosclerosis progression, fulfilling their potential lies in large-scale multicenter studies that hitherto have been scarce. This systematic examination of multicenter reproducibility of frequently-used MRI biomarkers therefore may provide helpful information for the planning of future clinical studies, particularly regarding sample size determination, quality control, and image analysis.

Multicenter MRI studies usually entail the involvement of multiple scanner platforms. Quantitative measurements are known to be influenced by platform-specific characteristics including hardware, pulse sequence design and image reconstruction algorithms [17, 18, 25]. Although these factors are typically beyond control of clinical investigators, our data show that it is feasible for multicenter studies to employ different platforms and pool serial data acquired at multiple sites if individuals are followed on the same scanner across the study course. Implementation of a standardized protocol with carefully matched pulse sequences, sequence parameters and flow-suppression techniques could be one key factor in obtaining comparable measurement precision between platforms. Moreover, participating sites may have diverse levels of experience in plaque imaging. Quality assurance procedures are recommended throughout the study. This not only includes initial training and case-specific instructions on subject/coil positioning and identifying motion sources, but to a larger extent refers to timely feedback from experienced readers on image quality and directions for improvement even if completed scans are acceptable.

A number of studies have examined inter-scan reproducibility of plaque morphological measurements by carotid MRI [14, 16, 26–29]. The present study, being the largest one and the first multicenter clinical trial, supports the excellent reproducibility of carotid MRI in characterizing plaque morphology. It was demonstrated that measurement precision in multicenter studies can match that reported in single-center studies [14, 16, 26–28], when strict quality assurance procedures are implemented and central review is performed.

Few studies have examined inter-scan reproducibility of plaque compositional measurements [27–29]. Because of the small size of plaque components and complex features that may co-localize with them, measurement variability of plaque components such as the LRNC has been consistently noted to be larger than morphological measurements. Saam et al. [27] summarized measurement errors from four serial scans of subjects treated with placebo. Multiple scans of the same subject were reviewed simultaneously but mixed with those receiving the active treatment, blinding readers from knowing which cases would be used for summarizing reproducibility. CV of LRNC and calcification volumes were found to be 11.1 % (within-subject SD 16 mm³) and 30.8 % (within-subject SD 9 mm³), respectively. Li et al. [28] used a 3T protocol and showed CV of 31.7 % (within-subject SD 15 mm³) and 22.0 % (within-subject SD 9 mm³) for LRNC and calcification volumes. In another study, Wasserman et al. [29] reported a CV of 86 % (within-subject SD 60 mm³) for LRNC volume and identified reader variability as the main contributing factor. In the latter two studies as well as the present one, all images were acquired for measuring reproducibility and therefore paired scans were analyzed separately in order to blind readers.

While our findings suggest the feasibility of multicenter studies, the larger variability of LRNC compared to morphology has additional implications for future studies. Image acquisition (scanning) and image analysis (reading) both contribute to the variability of contour-based plaque measurements. Reader variability is not trivial for small structures, as suggested by Wasserman et al. [29]. Indeed, much smaller measurement variability has been shown by Saam et al. [27] when serial scans were analyzed side-by-side. A sensitive and unbiased reading approach may be that serial scans are analyzed in the same session while readers are blinded to time sequence. With such a design, reader variability in drawing contours is reduced, and artifacts from unsuppressed flow or partial volume are more consistently interpreted. Consequently, biological differences on plaque composition between serial images can be detected more readily [6–8]. Additionally, our data indicated that smaller LRNCs were associated with lower kappa and larger CV, which was not shown in previous studies of smaller sample size. It's conceivable that factors limiting area measurement including the spatial resolution will become more influential in the setting of small LRNCs. Therefore, studies with a particular interest in LRNC progression may benefit from examining relatively large ones. Although this requires a screening step, either utilizing the baseline scan for screening purpose or a separate scan with simplified protocol, the sample size needed to test hypotheses may be substantially reduced.

This study used a standard two-dimensional protocol, whose wide availability, especially in university hospitals, forms a solid basis for planning multicenter studies. Yet the small coverage (32 mm) means that technicians need to carefully position scanning slabs to enable following the same segment during the study. Experience from this study suggests that the chance to exclude subjects due to insufficient coverage is low. As carotid atherosclerosis almost unexceptionally affects the carotid bifurcation, the two-dimensional protocol represents a reasonable solution in planning multicenter longitudinal studies on plaque progression, given previous validation studies and demonstrated successes in single-center reports. On the other hand, lesions outside the current coverage are possible, and the slice thickness (2 mm) is a limiting factor in the reproducibility of volumetric measurements. The development of a multi-contrast protocol based on emerging large-coverage, isotropic 3-dimensional (3D) sequences may afford further advantages. As scan-rescan contrast variations and their impact on plaque measurements in multicenter settings were the main focus of this study, the use of a validated automatic plaque classifier has enabled an unbiased comparison between scanner platforms, and the results are expected to hold independent of readers. In contrast to plaque morphology where excellent reader reproducibility is consistently reported [14–16], previous literature shows that manual segmentation of plaque components suffers from variable reader reproducibility [13, 29–31]. Automatic plaque classifiers, which consistently produce the same results with the same image set, are less subject to reader variability. In terms of accuracy, the automatic plaque classifier has demonstrated good agreement with histology that is non-inferior to expert human reading [21, 22]. Indeed, previous studies have used it to understand the contrast properties of MR sequences as well as changes in plaque composition [5, 32, 33].

This study has several noteworthy limitations. The most important one may be that only two platforms were studied. The feasibility of plaque imaging on a Siemens platform with coils of similar design has been shown [18]. Further efforts are needed to standardize protocols that can be applied across all major MRI manufacturer platforms with comparable performance. Another limitation is that intra-reader and inter-reader reproducibility were not assessed. With a focus on multicenter reproducibility and compatibility of different platforms, we used a standardized workflow including image analysis, which can be readily adopted for future clinical studies but precluded the assessment of reader reproducibility. However, reader reproducibility has been well documented in previous reports [14–16]. Finally, as individual subjects were not scanned on different scanners, we could not estimate inter-scanner variation. Platform changes or upgrades during longitudinal studies might undermine study design and likely reduce reproducibility.

Conclusions

This study demonstrates the feasibility of multicenter MRI studies employing different scanner platforms for monitoring carotid atherosclerotic plaque progression, which can maintain good to excellent inter-scan reproducibility with comparable measurement precision between platforms. Our findings may facilitate future multicenter endeavors that use serial MR plaque imaging to seek optimal clinical management as well as improved understanding of the biology of atherosclerosis.

Supplementary Material

Acknowledgements

Supplemental Fig. 1. Image acquisition workflow including quality assurance procedures for (a) setting up imaging sites (b) and monitoring image quality

Supplemental Table 1. Subject distribution across imaging centers

Supplemental Table 2. Carotid MRI protocol