Volumetric Analysis From A Harmonized Multi-Site Brain MRI Study of a Single-Subject with Multiple Sclerosis

Abstract

Background and Objectives

Magnetic resonance imaging (MRI) can be used to measure structural changes in the brain of people with multiple sclerosis (MS), and is essential for diagnosis, longitudinal monitoring, and therapy evaluation. The North American Imaging in Multiple Sclerosis Cooperative (NAIMS) steering committee developed a uniform high-resolution 3T MRI protocol relevant to the quantification of cerebral lesions and atrophy and implemented it at seven sites across the United States. To assess inter-site variability in scan data, a volunteer with relapsing-remitting MS was imaged with a scan-rescan at each site.

Materials and Methods

All imaging was acquired on Siemens scanners (4 Skyra, 2 TIM Trio, and 1 Verio). Expert segmentations were manually obtained for T1-hypointense and T2 (FLAIR)-hyperintense lesions. Several automated lesion detection and whole-brain, cortical, and deep gray matter volumetric pipelines were applied. Statistical analyses were conducted to assess variability across sites, as well as systematic biases in the volumetric measurements that were site-related.

Results

Systematic biases due to site differences in expert-traced lesion measurements were significant (p<0.01 for both T1 and T2 lesion volumes), with site explaining over 90% of the variation (range = 13.0–16.4ml in T1 and 15.9–20.1ml in T2) in lesion volumes. Site also explained more than 80% of the variation in most automated volumetric measurements. Output measures clustered according to scanner models, with similar results from the Skyra vs. the other two units.

Conclusions

Even in multi-center studies with consistent scanner field strength and manufacturer, after protocol harmonization, systematic differences can lead to severe biases in volumetric analyses.

Introduction

Conventional magnetic resonance imaging (MRI) is an established tool for measuring CNS lesion and tissue compartment volumes in vivo in people with multiple sclerosis (MS). In the brain and spinal cord, inflammatory demyelinating lesions appear hyperintense on T2-weighted (T2) images. Total cerebral T2 lesion volume (T2LV) is a key metric for the longitudinal monitoring of disease severity, as well as a standard outcome in clinical trials of MS therapeutics^1–3. Many T2 lesions exhibit pulse-sequence-dependent hypointensity on T1-weighted images (T1), which has been shown to be associated with more severe (destructive) histopathology and worse clinical outcomes^4–8. MRI is also used to measure cerebral atrophy, a commonly-used supportive outcome measure of the neurodegenerative aspects of the diseases in both relapsing-remitting and progressive forms of MS^9–18. Together, lesion and atrophy measures provide complementary quantitative information about disease progression that are considered central to patient assessment¹⁹.

Unfortunately, differences in acquisition methods have the potential to bias MRI metrics. Factors such as equipment manufacturer, magnetic field strength, and acquisition protocol can affect image contrast and resultant volumetric data. Indeed, several groups have investigated the reliability of volumetric measurements across scanners^20–27, but little is understood about the variability in volumetric measurements of lesions and atrophy in people with MS. Furthermore, many automated segmentation algorithms depend on statistical atlases or models that are built using healthy volunteers or that depend on registration, which can be compromised by the presence of MS pathology²⁸.

The North American Imaging in Multiple Sclerosis (NAIMS) cooperative was established to accelerate the pace of imaging research. As a consortium, our first aim was to facilitate multi-center imaging study by creating harmonized MRI protocols across sites. In this manuscript, we describe initial results from our pilot study, which tested the feasibility of multi-site standardization of MRI acquisition for the quantification of lesion and tissue volumes. We compared inter- to intra-site scan-rescan variability in various MRI output metrics, using consistently acquired 3T acquisitions.

Materials and Methods

Participant

A 45-year-old man with clinically stable relapsing-remitting MS and mild to moderate physical disability was imaged at seven NAIMS sites across the United States (Table 1). He developed the first symptoms of the disease 13 years before study enrollment, and had been relapse-free in the previous year after starting dimethyl fumarate. His last intravenous corticosteroid administration was five years previously. Timed 25-foot walk at study entry was 5.3 seconds. Expanded Disability Status Scale score was 3.5, both at study entry and exit, without any intervening relapses on-study. The participant signed informed consent for this study, which was approved by each site’s institutional review board.

Table 1. 3T brain MRI anatomical acquisition protocols.

Each of the 7 sites used one of 3 different Siemens scanner models (Skyra, Verio and TrioTim), necessitating 3 model-specific protocols for each of the 3 pulse sequences.

	3D T2 FLAIR			3D T1 MPRAGE			2D PD T2
Scanner manufacturer	Siemens Skyra	Siemens Verio	Siemens TrioTim	Siemens Skyra	Siemens Verio	Siemens TrioTim	Siemens Skyra	Siemens Verio	Siemens TrioTim
Operation system version	Syngo MR D13	Syngo MR B17	Syngo MR B17	Syngo MR D13	Syngo MR B17	Syngo MR B17	Syngo MR D13	Syngo MR B17	Syngo MR B17
Coil	20, 32, or 64 Channel*	32 Channel	32 Channel	20, 32, or 64 Channel*	32 Channel	32 Channel	20, 32, or 64 Channel*	32 Channel	32 Channel
Acceleration factor for parallel imaging	2	2	2	2	2	2	3	3	3
Orientation	Sagittal	Sagittal	Sagittal	Sagittal	Sagittal	Sagittal	Axial	Axial	Axial
Field of view (cm)	25.6×25.6	25.6×25.6	25.6×25.6	25.6×25.6	25.0×25.0	25.0×25.0	21.7×24.0	21.7×24.0	21.7×24.0
Matrix size	512×512	512×512	512×512	256×256	256×256	256×256	232×256	232×256	232×256
Number of slices	176	176	176	176	176	176	108	108	108
Repetition time (msec)	4800	4800	4800	1900	1900	1900	3000	3000	3000
Echo time (msec)	353	354	355	2.52	2.52	2.52	11/101	11/101	11/101
Flip angle (degrees)	120	120	120	9	9	9	150	150	150
Voxel size (mm)	0.5×0.5×1.0	0.5×0.5×1.0	0.5×0.5×1.0	1.0×1.0×1.0	0.977×0.977×1.0	0.977×0.977×1.0	0.9375×0.9375×3.0	0.9375×0.9375×3.0	0.9375×0.9375×3.0
Scan time (mins)	6:53	7:00	7:00	4:15	4:16	4:16	1:57	2:15	2:15
Number of signal averages	1	1	1	1	1	1	1	1	1

Key: FLAIR = fluid-attenuated inversion recovery, MPRAGE = magnetization-prepared rapid acquisition gradient echo, PD = proton density.

^*Brigham and Women’s Hospital = 20-channel, University of California San Francisco = 64 channel (the other two Skyra sites = 32 channel).

Scan acquisition

Through consensus agreement in the cooperative, the NAIMS cooperative developed a standardized high-resolution 3T MRI brain scan protocol. All imaging was acquired using Siemens scanners, which at the time of the study were used by the majority of NAIMS sites. Scan-rescan pairs were acquired on these scanners using the acquisition protocol shown in Table 1. At each site, the scan-rescan experiment was performed on the same day, with the participant removed and repositioned between scans. To replicate a “real world” clinical trial setting, none of the participant’s scans were co-registered to each other. The volunteer was also imaged at the National Institutes of Health (NIH) NAIMS site at the beginning and end of the study (five months later) to assess disease stability. Raw MRI scans were distributed to 4 NAIMS sites for post-processing.

Expert lesion tracing

De-identified images underwent manual quantification to assess total cerebral T1-hypointense lesion volume (T1LV) and T2LV from the native 3D FLAIR and T1 images by the consensus of trained observers (FK, FY) under the supervision of an experienced observer (ST). For T2LV, this involved manually identifying all lesions on the FLAIR images. For T1LV, lesions were required to show hypointensity on T1-weighted images and at least partial hyperintensity on FLAIR images. The lesions were then segmented by one observer (GK) using a semi-automated edge-finding tool in Jim (v. 7.0) to determine lesion volumes. Images were presented to the same reading panel for all of the above steps in random order in one batch, and mixed into a stack of 50 other MS images to reduce scan-to-scan memory effects and to preserve blinding.

Automated analysis

Several fully automated pipelines were also used to estimate T2LV, and volume of total brain, normal-appearing white matter (WM), and both cortical and deep gray matter structures. To prevent overfitting, all pipelines were used with their default settings. According to published recommendations for each method separately, where appropriate images were inhomogeneity corrected, rigidly aligned across sequences from each scan session, processed for removal of extracerebral voxels for all processing pipelines, and intensity normalized. For lesional measurements, several algorithms were applied by the laboratories that developed or co-developed the various methods: Lesion-TOADS²⁹, a fuzzy c-means-based segmentation technique with topological constraints, OASIS³⁰, a logistic-regression-based segmentation method leveraging statistical intensity normalization, S3DL³¹, a patch-based dictionary learning multi-class method, and WMLS³², a local support vector machine-based segmentation algorithm developed for vascular lesions that also employs corrective learning. To estimate the volume of gray matter structures, Lesion-TOADS, FSL-FIRST³³ (a Bayesian appearance method), MaCRUISE³⁴ (a combined multi-atlas segmentation and cortical reconstruction algorithm), and MUSE³⁵ (an ensemble multi-atlas label fusion method) were used. The FSL-FIRST³³ analysis was applied directly to the raw T1 images according to common practice, and OASIS³⁰ was applied to the T1, FLAIR, T2, and PD images after preprocessing; all other images were applied to appropriately preprocessed T1 and FLAIR imaging. Not all algorithms measured volumes of the same set of structures. Lesion filling was not performed. Lesion-TOADS, MaCRUISE, and MUSE also yielded estimates for total brain volume.

Statistical Analysis

All statistical analysis was conducted in the R software environment³⁶. To compare estimated volumes within and across sites, mean volumes and standard deviations were computed. T-tests were also used to test for differences in within-site average between scanner platforms. Correlations between these averages across segmentation algorithms were also explored. The proportion of variation explained by site was computed and the association with site was assessed using permutation testing. The coefficients of variation were also estimated across sites. To assess associations between session-average measured total brain and lesional volumes and time of day (morning versus afternoon), Wald testing within a linear model framework was employed both marginally and adjusting for scanner platform.

Results

The participant was found to be stable regarding cerebral lesion load during the study. Comparing images acquired at the NIH at study entry and exit, the manually measured T2LV in the participant was similar (17.9ml in September 2015 versus 17.8ml in February 2016). The T1LV was also stable (15.5ml versus 15.1ml). This imaging stability paralleled his clinical stability (see Methods).

The manually estimated T1LV and T2LV for each scan is shown in Figure 1. Site explained 95% of the variation observed in the estimated T2LV, and 92% of the variation in the estimated T1LV, indicating significant scanner-to-scanner differences despite protocol harmonization, which clearly exceed scan-rescan variability within sites. The range of T2LVs was 15.9ml to 20.1 ml, indicating that differences of up to 25% of the lesion volume were observed across sites. The range of T1LVs was similarly wide, ranging from 13.0ml to 16.4ml. Further inspection of these volumes across platforms indicated that Skyra scanners showed larger lesion volumes compared to other Siemens platforms both on T1LV (Skyra mean T1 = 15.2ml compared with non-Skyra mean T1 =13.8ml, p<0.05) and T2LV (Skyra mean T2 = 18.9ml compared with non-Skyra mean T2 = 16.6ml, p<0.01). A visual example of the segmented lesions across scanners is provided in Figure 2.

FIG 1.

Manually measured T1 (red) and T2 (blue) lesion volumes for scan-rescan pairs at each of seven NAIMS sites. Results from the baseline scan, acquired on the same Skyra scanner and subsequent imaging acquired at NIH, are shown using circles. Points have been slightly offset relative to one another for ease of visualization.

FIG 2.

Comparison of manual segmentation of cerebral T2 hyperintense lesions at four NAIMS sites. 3T MRI scans on Siemens scanners from a single subject with multiple sclerosis showing T2 hyperintense lesions from sagittal fluid-attenuated inversion recovery (FLAIR) sequences from 4 different North American Imaging in Multiple Sclerosis (NAIMS) sites and scanner models: Brigham and Women’s Hospital, Skyra; National Institutes of Health (NIH), Skyra; Oregon Health & Sciences University (OHSU), Trio; Cedars-Sinai, Verio. The upper panel shows the native images. The lower panel shows zoomed and cropped images to illustrate the key findings. The green arrow (lower panel) shows a possible lesion detected and traced on the NIH scan; the red arrow shows the same lesion not detected by the expert procedure on Brigham scan; the purple arrow shows a similar tubular area that was interpreted as a blood vessel on the Cedars-Sinai scan, which was not selected as a lesion by the expert tracing; no lesion was detected on the OHSU scan in this area on this slice or any of the adjacent slices (not shown). The blue arrow shows a different lesion detected and traced on the Brigham, NIH, and Cedars-Sinai scans but not detected by the expert review on the OHSU scan, appearing hazy/subtle (white arrow). The yellow arrow (upper panel) shows a lesion on all scans; however, when adding the tracing of all slices showing the lesion, the 3D volume of the lesion differed among sites: Brigham = 0.059 ml, NIH = 0.053 ml, OHSU = 0.033 ml, Cedars-Sinai = 0.053 ml.

Results from the automated techniques for delineating and measuring T2LV are shown in Figure 3. The automated lesion segmentations showed marked disagreement in the average lesional volume measurements compared with the manually assessed volumes, and all methods showed large site-to-site differences (in some cases up to 7.5ml, or almost 50% of the manually measured lesion volume), except for LesionTOADS (range 10.5ml to 11.0ml) which was more stable. For all methods, site explained more than 50% of the observed variation; 53% of the variation was explained by site (permutation p = 0.36) for S3DL, 54% for Lesion-TOADS (p=0.41), 44% for OASIS (p=0.57), and 83% for WMLS (p=0.002) which clearly was most prone to site-related variation.

FIG 3.

Comparison of manual and automated methods for measuring lesional volume. Scan-rescan imaging is shown using multiple dots for each site and algorithm.

To measure brain structure volumes, several automated methods were used. As an example, results for the thalamus are shown in Figures 4 and 5. While LesionTOADS estimated smaller volumes, MUSE, FIRST, and MaCRUISE yielded similar average measurements. Nonetheless, site was strongly associated with measured thalamic volume, explaining 96% of the LesionTOADS volume variation (p<0.01), 89% of MUSE (p<0.01), 84% of FIRST (p=0.04), and 65% of MaCRUISE (p=0.17). Similar results for the putamen, caudate, cortical gray matter, normal-appearing white matter, and total brain volume were found, as provided in Supplementary Figures 1–5. Summaries of the coefficient of variation give an intuitive measure of the scale of the combined scan-rescan and across-site variation as shown in Figure 6. Finally, the proportion of variation explained by site is shown in Figure 7. Note that, in almost all cases, site explained more than 50% of the variation, with the majority of measurement techniques exhibiting more than 80% variation due to site for all structures assessed.

FIG 4.

FSL-FIRST Automated Segmentation Results: Thalamus. Representative anatomic slice showing segmentation of the thalamus (green) in the single subject. The segmentation maps are overlaid to the original raw 3D T1-weighted images after re-orientation to the axial plane. Segmentation was performed by the fully automated FSL/FIRST pipeline ([FMRIB (Oxford Centre for Functional MRI of the Brain) Software Library Integrated Registration and Segmentation Tool]). The scan site and 3T Siemens model are shown for each image. The first two scans are from the scan/re-scan at Brigham and Women’s Hospital. OHSU= Oregon Health & Sciences University; NIH = National Institutes of Health

FIG 5.

Comparison of automated methods for measuring thalamic volume. Scan-rescan imaging is shown using multiple dots for each site and algorithm.

FIG 6.

Estimated across-site coefficient of variation for each structure using various methods for volumetric measurement. cGM = cortical gray matter; NAWM = normal-appearing white matter; T1LV = T1 lesion volume; T2LV = T2 lesion volume; TBV = total brain volume.

FIG 7.

Estimated proportion of variation explained by site for using various segmentation methods for different structures in the brain. cGM = cortical gray matter; NAWM = normal-appearing white matter; T1LV = T1 lesion volume; T2LV = T2 lesion volume; TBV = total brain volume.

While all images were acquired on 3T Siemens scanners, the model type appeared to influence the results; there was evidence of systematic differences in many measurements between Skyra and non-Skyra scanners. Figure 8 shows the negative log p-value for the comparison of volumes averaged across scan-rescan measurements, with larger values indicating more systematic differences between platforms. The largest platform-associated differences were observed in MaCRUISE measurements of normal-appearing white matter, cortical gray matter, and, consequently, total brain volume. LesionTOADS also showed large differences in total brain volume attributable to cortical gray matter, as did S3DL for T2 lesion volume measurements. MUSE showed major differences in thalamic volume across scanner model, and FIRST showed similar discrepancies in the thalamus and caudate. The correlation between site-averaged measurements varied dramatically, especially for lesional and total brain volume measurements (see Supplementary Figure 6), indicating that site differences resulted in contrasting effects on outputs from the different algorithms. While the other measurements showed less scanner model-related variation, most still showed prominent differences between Skrya and non-Skrya scanners.

FIG 8.

Negative logarithm (base 10) p-value from t-test describing the difference in average volume between Skyra vs non-Skyra platforms explained by site for using various segmentation methods for different structures in the brain. cGM = cortical gray matter; NAWM = normal-appearing white matter; T1LV = T1 lesion volume; T2LV = T2 lesion volume; TBV = total brain volume.

The time of day of scan acquisition was not associated with manually segmented T1 (t=0.45) or T2 lesion volumes (t=0.38), or total brain volume, as measured by any of the automated algorithms (see Supplementary Figures 7 and 8).

Discussion

Clinical MS therapeutic trials have traditionally employed 1.5T MRI platforms to provide metrics on cerebral lesions and atrophy as supportive outcome measures. However, there is a growing interest in the use of high-resolution 3T imaging to assess disease activity and disease severity in MS. Such 3T imaging brings the potential for increased sensitivity to lesions^37,38 and atrophy,³⁹ higher reliability,^39,40 and closer relationships to clinical status,^38,39 when compared to scanning at 1.5T. The purpose of this study was to evaluate the consistency of metrics obtained from a single MS participant using a high-resolution 3T brain MRI protocol distributed to seven sites. The results of our study indicate that even in multi-center acquisitions from the same scanner vendor after careful protocol harmonization, systematic differences in images led to severe biases in volumetric analyses. These biases were present in manually and automatically measured volumes of white matter lesions, as well as in automatically measured volumes of whole brain, gray and white matter structures. These biases were also highly dependent on scanning equipment, which resulted from a significantly higher sensitivity to lesions in newer scanners from the same manufacturer compared with earlier models even at the same field strength.

In comparison to past estimates of reliability of volumetric measurements of brain structures, our findings point to higher between-site variation than previously documented. In particular, Cannon et al.²⁷ reported that between 3% and 26% of the observed variation in global and subcortical volumes was attributable to site; this was a study of 8 healthy participants imaged on two successive days across 8 sites using 3T Siemens and GE scanners. However, it is important to note that the proportion of explained variation has a different interpretation from that reported here. The total variation in Cannon et al. consisted of four contributors to variance: first, across-site differences; second, across-scan differences; third, across-day differences; and fourth, across-subject differences. In our single-participant study, we isolated only the first two variance components which allowed us to compare variation as it is relevant for precision medicine (subject-specific) applications. Previous work indicated that the observed variation attributable to scanning occasion was small^25,27; indeed, Cannon et al. found this to constitute less than 1% of the variation. Thus, we did not scan our participant on subsequent days but rather simply repositioned the participant between scans during the same imaging session. A notable difference between our study and that of Cannon et al. is that we did not use data from a standardized phantom concurrently acquired for correction of between-scanner variations in gradient non-linearity and scaling. Cannon et al. found this correction to improved between-site intra-class correlations and greatly reduced differences between scanner manufacturers. Similarly, Gunter et al.⁴¹ reported the usefulness of a phantom for scanner harmonization and quality control in the Alzheimer’s Disease Neuroimaging Initiative. In future studies, we will focus on applying phantom calibrations across NAIMS sites to extend our current observations. Despite the growing literature on the importance of diurnal variation and hydration status for volumetric analyses ^42–45, we found no significant associations between time of day and measured volumes. This may indicate that in single participant analyses, time of day and day-to-day variation may be of less concern than the much larger source of variation of scanner platform. Interestingly, Cannon et al. also found that measurements acquired using scanners from the same manufacturer and similar receive coils had higher reliability. In our study, we found that even scanner models (i.e. Skyra versus non-Skyra) from the same manufacturer varied markedly in their estimates of lesion volume, highlighting the importance of between-scanner differences for assessing MS-related structural changes.

To assess differences across processing pipelines, we used a variety of techniques for automated segmentation of lesion and white and gray matter volumes. Different segmentation algorithms showed a range of variability in their estimates, as well as their sensitivity to differences between scanners. For example, LesionTOADS showed much less variable lesion measurements than any other technique, and was not as sensitive to differences in scanner platform. Lesion-TOADS was the only unsupervised lesion segmentation technique employed. Contrast differences between the participant data and the training data of the other supervised methods could be associated with the greater sensitivity to scanner differences, and this might be mitigated by specific (albeit potentially laborious) tuning to individual platforms. However, while sensitivity to biological change is generally higher for methods yielding less noisy estimates, as only a single individual was studied here, our data cannot be taken to indicate that LesionTOADS is superior to other methods of estimating thalamic volume, for example. Additionally, both purely intensity-based segmentation algorithms, OASIS and WMLS, appeared to be more sensitive to site differences, which may indicate that methods that rely more on topology, shape, or spatial context may be more stable across scanners. This indicates that across-scanner differences may be driven by contrast differences rather than geometric distortions. Future investigation to extend these findings could involve quantitative contrast-to-noise and signal-to-noise comparisons across scanners. Allowing segmentation parameters to vary across sites could also help stability.

A limitation of this study is its single-subject and single-time-point design, which makes the generalizability of the findings dependent on further investigation. In particular, the degree to which across-site differences might vary by lesion burden and degree of atrophy, as well as demographic variables, requires additional study. Future larger studies of multiple participants across disease stages including longitudinal measurements are necessary for understanding the implications of the biases described in this pilot study. Indeed, such studies would also allow for the assessment of the tradeoff between stability in measures across sites with sensitivity to biological differences. Differences between scanning equipment and scanner software versions have also been noted in past studies of reliability^{23,25,27,46,47}, but their implications for the assessment of pathology remain unclear. In particular, repeated acquisitions on scanners with different receive coils could provide additional insight concerning reliability. In addition, our study was from a single time point across scanners, whereas clinical trials rely on the quantification of intrasubject longitudinal change⁴⁸. Each participant is typically scanned on the same platform, which may limit the variability in on-study change between participant. Further studies are necessary to assess whether scan platform introduces the same level of acquisition-related variability when assessing longitudinal changes.

Given the inter-site differences observed in lesional measurements, across-site inference statistical adjustment for site is clearly necessary when analyzing volumetrics from multi-site studies, even when images are acquired using a harmonized protocol on 3T scanners produced by the same manufacturer. From a single participant, it is unclear what the role of differential sensitivity to lesions might be across people with heterogeneity in lesion location. For example, while lesion detection in the supratentorial white matter might be more straightforward and comparable across people, detection of lesions in the brain stem, cerebellum, and spinal cord may be more sensitive to differences in equipment. New statistical methods for measuring and correcting systematic biases are warranted, especially for studies in which patient populations may differ across sites. Indeed, intensity normalization and scan-effect removal techniques^49–55 (akin to batch-effect-removal methods in genomic studies⁵⁶) are an active area of methodological research and promise to improve comparability of volumetric estimates from automated segmentation methods. After volumes are measured, statistical techniques for modeling estimated volumes from multi-center studies are also rapidly evolving^18,57. These techniques bring the potential to mitigate site-to-site biases in group-level analyses, with better external validity at the cost of increased sample size.

By imaging the same subject with stable relapsing-remitting MS over a period of five months, we assessed scanner-related biases in volumetric measurements at seven NAIMS centers. Despite careful protocol harmonization and the acquisition of all imaging at 3T on Siemens scanners, we found significant differences in lesion and structural volumes. These differences were especially pronounced when comparing Skyra scanners to other Siemens 3T platforms. The results from this study highlight the potential for inter-scanner and inter-site differences that, unless properly accounted for, might confound MRI volumetric data from multi-center studies of brain disorders.

We conclude that our findings raise a key issue with the interpretability of MRI measurements in the context of personalized medicine, even in carefully controlled studies using harmonized imaging protocols.