Global multi‐center and multi‐modal magnetic resonance imaging study of obsessive‐compulsive disorder: Harmonization and monitoring of protocols in healthy volunteers and phantoms

Abstract

Objectives

We describe the harmonized MRI acquisition and quality assessment of an ongoing global OCD study, with the aim to translate representative, well‐powered neuroimaging findings in neuropsychiatric research to worldwide populations.

Methods

We report on T1‐weighted structural MRI, resting‐state functional MRI, and multi‐shell diffusion‐weighted imaging of 140 healthy participants (28 per site), two traveling controls, and regular phantom scans.

Results

Human image quality measures (IQMs) and outcome measures showed smaller within‐site variation than between‐site variation. Outcome measures were less variable than IQMs, especially for the traveling controls. Phantom IQMs were stable regarding geometry, SNR, and mean diffusivity, while fMRI fluctuation was more variable between sites.

Conclusions

Variation in IQMs persists, even for an a priori harmonized data acquisition protocol, but after pre‐processing they have less of an impact on the outcome measures. Continuous monitoring IQMs per site is valuable to detect potential artifacts and outliers. The inclusion of both cases and healthy participants at each site remains mandatory.

1. INTRODUCTION

Neuroimaging has increased our understanding of the neurobiology of obsessive‐compulsive disorder (OCD) (Stein et al., 2019), but studies generally used relatively small samples. International collaborative efforts, such as Enhancing Neuroimaging and Genetics through Meta‐Analysis (ENIGMA), could increase statistical power by combining samples across sites (Thompson et al., 2020). ENIGMA‐OCD meta‐ and mega‐analyses have strengthened international collaboration and investigated whether structural alterations found in OCD reflect neurodevelopmental changes, vulnerability factors, effects of disease (chronicity), or medication (van den Heuvel et al., 2020). However, a methodological limitation of ENIGMA is the lack of prospective harmonization of acquisition protocols before data pooling. Moreover, due to between‐site differences in clinical measures (e.g., course of illness, severity, comorbidity, treatment history, and symptom dimensions), pooling of these data is also limited.

To address this problem, five ENIGMA‐OCD sites from five continents, launched the largest multimodal‐imaging and neurocognitive study in medication‐free OCD patients to date (i.e., the OCD Global Study, entitled ‘Identifying Reproducible Brain Signatures of Obsessive‐Compulsive Profiles’ R01‐MH113250), using harmonized methods for clinical phenotyping, neurocognitive testing and neuroimaging (Simpson et al., 2020). With this collaboration, we aim: (1) to identify reproducible brain signatures that distinguish OCD patients from their unaffected siblings and healthy control participants, and (2) to associate these brain signatures with OCD‐related clinical and neurocognitive profiles. We developed detailed protocols to enhance cross‐site reliability on clinical, neurocognitive, and MRI measures.

This manuscript describes the standardized MRI protocol we developed to acquire structural imaging, resting‐state functional MRI (rsfMRI) and multi‐shell diffusion‐weighted imaging (DWI), within a clinically feasible timeframe (<1 h) and for clinical 3 T scanners from various vendors, and the methods used to monitor scan quality. We describe within and between‐site variability in image quality and standard neuroimaging outcome measures (e.g., whole‐brain morphometry and white matter diffusion) from physical phantoms, traveling human volunteers and 28 healthy participants from each site.

2. METHODS

The OCD Global Study started data collection in 2018 across its five research sites in Brazil, India, the Netherlands, South Africa, and U.S.A (Simpson et al., 2020). Vendors and types of MRI scanners are listed in Table 1.

TABLE 1.

MRI scanners and parameters per site

site	1	2	3	4	5
Scanner	Philips achieva 3.0 T	Philips ingenia 3.0 T CX	GE 3.0 T discovery MR750	Siemens MAGNETOM skyra 3.0 T	GE SIGNA 3.0 T premier
Head coil	32‐Channel	32‐Channel	32‐Channel	32‐Channel	48‐Channel
T1w: 3D sagittal T1‐weighted MPRAGE according to ADNI‐3 protocol
TR (ms) a	6.5	6.5	6.9	2300	2235
TI (ms)	900	900	900	900	900
TE (ms)	2.9	2.9	3	2	2.8
Flip angle (⁰)	9	9	9	9	9
Voxel size (mm)	1 × 1 × 1	1 × 1 × 1	1 × 1 × 1	1 × 1 × 1	1 × 1 × 1
Matrix	256 × 256	256 × 256	256 × 256	256 × 256	256 × 256
Resting‐state fMRI: Gradient‐echo echo‐planar images while subjects are awake and keep their eyes closed; axial ascending slices parallel to line through the pituitary gland and 4th ventricle; 10 min
TR (ms)	2200	2200	2200	2200	2200
TE (ms)	28	28	28	28	28
Flip angle (⁰)	80	80	80	80	80
# Slices	44	44	42	42	42
# Volumes	275	275	275	272	272
Voxel size (mm)	3.3 × 3.3 × 3	3.3 × 3.3 × 3	3.3 × 3.3 × 3	3.3 × 3.3 × 3	3.3 × 3.3 × 3
Slice gap (mm)	0.3	0.3	0.3	0.3	0.3
Matrix	64 × 64	64 × 64	64 × 64	64 × 64	64 × 64
DWI: Multi‐shell single spin‐echo echo‐planar images; parallel imaging factor 2; no multi‐band or simultaneous multi‐slice; axial interleaved slices parallel to line through the pituitary gland and 4th ventricle; 73 directions interleaved 25 b1000, 24 b2000, 24 b3000, 7 b0; sampling scheme according to Caruyer et al. (Caruyer et al., 2013)
TR (ms)	7220	7220	6310	7300	7000
TE (ms)	100	100	81	100	74
Flip angle (⁰)	90	90	90	90	90
# Slices	56	56	56	56	56
Voxel size (mm)	2.5 × 2.5 × 2.5	2.5 × 2.5 × 2.5	2.5 × 2.5 × 2.5	2.5 × 2.5 × 2.5	2.5 × 2.5 × 2.5
Matrix	96 × 96	96 × 96	96 × 96	96 × 96	96 × 96

Abbreviations: DWI, diffusion‐weighted imaging; MPRAGE, magnetization‐prepared rapid acquisition gradient‐echo; T1w, T1 weighted; TE, echo time; TI, inversion time; TR, repetition time.

^a values for TR are highly variable due to different definitions of TR for this pulse sequence.

2.1. Human participants

For details on in‐ and exclusion criteria of the OCD global study, see (Simpson et al., 2020). Written informed consent was obtained from participants according to the Declaration of Helsinki. Study protocols were approved by the five local Medical Ethical Committees.

We selected 28 healthy participants from each site by matching them on age and education using a weighted nearest‐neighbor selection method (Szekér & Vathy‐Fogarassy, 2020).

Two traveling healthy volunteers (not enrolled in the OCD Global study) visited each site and underwent the same MRI protocol within a time frame of 6 months, except at site 5 where only one volunteer was scanned and with a delay of 15 months due to a system upgrade and COVID‐19 restrictions.

2.2. Phantoms

Each site used the ISMRM/NIST system phantom (henceforth: geometry phantom; CaliberMRI, formerly High Precision Devices, Boulder, CO, U.S.A.), to monitor structural geometry. Each site used its own ball‐shaped (sites 1, 3, 5) or cylinder‐shaped (sites 2, 4) agar phantom to monitor the quality of rsfMRI and DWI.

2.3. MRI acquisition

2.3.1. Human participants

See Table 1 for spatial and timing parameters. 3D T1‐weighted structural images were acquired according to the ADNI‐3 protocol (Weiner et al., 2017), including correction of 3D geometric distortion and intensity non‐uniformity. Resting‐state fMRI with eyes closed was acquired for 10 min. Multi‐shell DWI was acquired with 73 diffusion‐weighted directions (25 b1000, 24 b2000, and 24 b3000 s/mm²) and 7 interleaved non‐diffusion‐weighted volumes (b0 s/mm²). We used a reduced spatial resolution version of the DWI sequence of the human connectome project (Sotiropoulos et al., 2013) that can be acquired within 8–10 min on a clinical scanner. For both rsfMRI and DWI, scans with opposite phase‐encoding directions were acquired to correct for susceptibility‐induced distortions. At each site, participants were guided through the MRI session according to a standardized protocol.

2.4. Phantoms

To monitor stability, the agar phantom was scanned twice a month at each site with a 10‐min protocol (supporting methods). This quality control (QC) protocol was based on protocols commonly used in multi‐site fMRI studies (Casey et al., 2018; Friedman & Glover, 2006). To monitor geometry, the geometry phantom was scanned every 2 months as described in the ISMRM/NIST phantom manual and supporting methods. We included all phantom data obtained from start of the study until May 2022, with gaps due to COVID‐19 restrictions.

2.5. Image quality metrics (IQMs)

2.5.1. Human participants

Images were converted from DICOM format to NifTI, and minimally processed prior to calculation of the image quality metrics (IQMs) to monitor data quality.

We used tools from FMRIB Software Library (FSL, version 6.0.1) (Smith et al., 2004), MRI quality control tool (MRIQC) (Esteban et al., 2017), and fMRIPrep (v20.2.3) (Esteban et al., 2019). For an extensive description see supporting methods.

For rsfMRI we designed an additional IQM to describe the temporal variation of WM heterogeneity, because we observed artifacts in several fMRI scans that could not be quantified by measures provided by MRIQC (Figure 1).

FIGURE 1.

Temporal variation in WM heterogeneity. The differences in precision on the y‐axis illustrate the differences between these three examples. (a) Axial slice 19 of a subject from site 1, illustrating severe artifacts on the temporal standard deviation (output from MRIQC). The corresponding time series show the temporal evolution of the SD within a WM mask. The temporal WM heterogeneity equals 7.46%. (b) a subject from site 1 with less severe artifacts, and temporal WM heterogeneity of 0.95%. (c) a subject from site 4 without artifacts, and temporal WM heterogeneity of 0.21%

DWI quality was evaluated using the EDDYQC tool (Andersson & Sotiropoulos, 2016; Bastiani et al., 2019). We used the median sum‐of‐squared‐error (SSE) from DTIFIT per b‐shell, as an additional DWI IQM. IQMs used in the current investigation are briefly described in Table 2. Calculation of motion‐related IQMs and outlier detection in DWI data was performed on raw data, whereas further analysis was performed on motion‐corrected data (Bastiani et al., 2019).

TABLE 2.

Description of image quality metrics

IQM	Explanation	Modality
CJV	Coefficient of joint variation: a Measure of intensity variation between a WM and GM mask proposed by (Ganzetti et al., 2016). Higher values may be indicative of motion or intensity non‐uniformity (INU) artifacts	T1
(t)SNR	Signal‐to‐noise ratio: The amount of real signal relative to the background noise. Because noise in the background is poorly defined with multi‐channel receivers, and because some scanners completely suppress noise in the air background, SNR was calculated using the within tissue variance.	T1, fMRI
CNR	Contrast‐to‐noise ratio: An extension of the SNR that measures the separation of the tissue distributions of GM and WM (Magnotta et al., 2006). Diffusion angular CNR ‐ the diffusion related variance versus the noise variance—separate per b‐value.	DWI
EFC	Entropy‐focus criterion: a Measure of the amount of entropy of the voxels in the image as a marker for ghosting and motion‐related blurring (Atkinson et al., 1997). Lower values are better.	T1, fMRI
INU	Intensity non‐uniformity: Spurious variability in voxel intensity due to imperfections in the acquisition process (Vovk et al., 2007). Location and spread of the bias field produced during the INU correction provides a measure of the data quality. Median values are reported. Values around 1.0 are better	T1
FWHM	Full‐width half maximum: a Spatial distribution of the voxel intensity values in the image as a measure of blurriness. Lower is better.	T1, fMRI
DVARS	Spatial standard deviation of successive difference images: The rate of change of voxel intensity across the entire brain at each volume (Afyouni & Nichols, 2018). Lower is better.	fMRI
FD	Frame‐wise displacement: An index of the amount of frame‐to‐frame displacement during scanning, calculated as the root mean square of the six translation and rotation parameters.	fMRI, DWI
Outliers	Percentage of total number of slices classified by EDDY as outliers due to motion‐related signal drop‐out. Calculated for each b‐shell separately.	DWI
SSE	Sum of squared errors of the diffusion tensor fit: a measure of the accuracy of the tensor fit. The median value within an eroded WM mask is calculated separately for each b‐shell. Lower values are better.	DWI
WMH	Temporal variation of WM heterogeneity: Variation within time‐series of SD/median of signal intensity in WM (after linear detrending).	fMRI

Note: For more information about the image quality metrics derived from MRIqc or eddyqc, see: mriqc.readthedocs.io/en/latest/measures.html or Bastiani et al. (Bastiani et al., 2019), respectively.

2.6. Phantoms

For a detailed QC description, see supporting methods.

Because each site used its own agar phantom, the exact contents may differ between sites. Therefore, SNR values from agar phantoms are intended to evaluate intra‐site stability. fMRI QC was determined based on an analysis for time‐series stability (Friedman & Glover, 2006; Weisskoff, 1996). We determined static SNR (spatial mean within a center ROI), SFNR (signal‐to‐fluctuation‐noise ratio within the ROI), ROI fluctuation, and trend. For DWI QA we checked the consistency of mean diffusivity (MD) within a center ROI across the three b‐values and the three gradient directions using the coefficient of variation (SD/mean). We determined within‐site variation in MD at room temperature.

In images of the geometry phantoms, we determined the position of the 56 fiducial spheres and compared this with phantom specifications. Deviations in R‐L, A‐P, and S‐I directions were expressed as percentage of the expected distance, and the largest deviations were used as QC parameters.

2.7. Outcome measures

We extracted global morphometric measures from each T1‐weighted image using FreeSurfer 7.1.1: volumes of whole brain (without ventricles), total cortical gray and white matter, and mean cortical thickness (Dale, Fischl, & Sereno, 1999). We determined both raw volumes and volumes normalized for estimated total intracranial volume by multiplying with 1948 ml/eTIV (http://www.freesurfer.net/fswiki/eTIV (accessed May 20, 2022)).

Independent component analysis (ICA) was performed to identify the spatial consistency of resting‐state networks between sites. We conducted group ICA (FSL) on the denoised fMRI scans, separately for each site, excluding the traveling volunteers. We applied a low model order ICA (6 components) to identify large ‘standard’ intrinsic resting‐state networks and prevent them from breaking down into smaller subnetworks. For each site we identified the default mode network (DMN), somatomotor network (SMN) and visual network (VN) based on their spatial likeness to the Yeo 7‐Network parcellation (Yeo et al., 2011). We quantified the spatial overlap (cross‐correlation) between these resting‐state networks across sites and with the Yeo parcellation.

Using DTIFIT, we calculated fractional anisotropy (FA), MD, axial diffusivity (AD) and radial diffusivity (RD) for each b‐shell (b1000, b2000, b3000), and compared these between sites using tract‐based spatial statistics (TBSS) (Smith et al., 2006). To register the DWI scans to a common space, we used DTI‐TK (Zhang et al., 2006), which utilizes the full tensor orientation information (Bach et al., 2014; Wang et al., 2011). We determined median diffusion measures of the whole brain skeleton (thresholded at FA 0.2), and of the skeleton voxels within the forceps major (through the splenium of the corpus callosum) and forceps minor (genu corpus callosum). We selected the forceps ROIs as examples of robust tracts with mostly parallel fibers. These ROIs were taken from the JHU‐ICBM tracts (Hua et al., 2008) with 25% threshold.

2.8. Statistics

We calculated means and standard deviation (SD) of IQMs and outcome measures for the 28 participants per site. For between‐site comparisons, Cohen's f was computed as a standardized effect size. Conventionally, we consider f = 0.1, f = 0.25, f = 0.4 as small, medium, large effect, respectively. Inter‐ and intra‐site variability of IQMs and outcome measures were visualized using raincloud plots (Allen et al., 2019). Data from the two traveling volunteers were displayed on the same plots.

3. RESULTS

3.1. Human participants

Mean age of the 140 healthy participants (73 females) was 27.3 (SD = 6.0) years, with on average 15.7 (SD = 2.1) years of education. Demographics per site are shown in Table 3. Between sites, participants were well matched on age, sex and education.

TABLE 3.

Demographics

site	1	2	3	4	5	Statistics
Age (years)	28.4 ± 5.2 [21–38]	26.9 ± 4.9 [19–37]	27.0 ± 6.3 [18–45]	27.3 ± 6.9 [19–42]	27.1 ± 6.8 [20–49]	H (4) = 1.90
						P = 0.75
Sex (M/F)	14/14	15/13	14/14	12/16	16/12	Χ2 (4) = 1.03
						P = 0.91
Education level (years)	15.8 ± 2.9 [7–20]	15.9 ± 1.7 [11–18]	16.0 ± 2.4 [11–21]	15.1 ± 1.9 [12–21]	16.1 ± 1.6 [13–20]	H (4) = 4.072
						P = 0.396

Note: Age and education level are shown as mean ± SD and [range].

3.2. Human IQMs

Human IQMs are shown in Table 4. For most IQMs of structural MRI, within‐site variation was smaller than between‐site variation (Figure 2). Effect sizes were large, and between‐site variation was also reflected as intra‐subject variation of the traveling volunteers.

TABLE 4.

Human image quality measures (IQMs) (mean ± SD) for 28 subjects per site for T1w structural scans, rsfMRI and diffusion‐weighted imaging (DWI)

T1	1	2	3	4	5	Cohen's f
CJV	0.35 ± 0.03	0.29 ± 0.02	0.39 ± 0.04	0.33 ± 0.02	0.34 ± 0.02	1.41
EFC	0.68 ± 0.01	0.72 ± 0.01	0.71 ± 0.01	0.62 ± 0.02	0.68 ± 0.02	2.59
FWHM	3.85 ± 0.13	4.22 ± 0.15	3.21 ± 0.1	3.66 ± 0.09	3.65 ± 0.1	2.96
INU	11.74 ± 0.75	14.16 ± 0.76	10.86 ± 0.75	10.53 ± 0.42	12.37 ± 0.84	1.89
SNR GM	1.06 ± 0.03	1.04 ± 0.03	1.03 ± 0.05	1.19 ± 0.05	0.96 ± 0.05	1.82
rsfMRI
EFC	0.44 ± 0.03	0.43 ± 0.03	0.49 ± 0.03	0.50 ± 0.03	0.48 ± 0.03	1.40
FWHM	2.40 ± 0.18	2.18 ± 0.34	2.37 ± 0.11	2.64 ± 0.15	2.39 ± 0.11	0.76
FD (mm)	0.15 ± 0.09	0.12 ± 0.04	0.13 ± 0.05	0.14 ± 0.07	0.14 ± 0.07	0.20
DVARS	27.7 ± 6.2	23.0 ± 3.6	25.4 ± 2.3	23. 8 ± 3.8	24.7 ± 6.1	0.37
SNR	4.62 ± 0.44	5.45 ± 0.84	3.06 ± 0.41	2.31 ± 0.4	3.26 ± 0.38	2.26
tSNR	51.4 ± 13.3	62.3 ± 9.6	48.5 ± 8.6	60.4 ± 7.0	47.6 ± 10.3	0.64
WMH	1.85 ± 2.65	0.83 ± 0.32	0.51 ± 0.48	0.29 ± 0.09	0.58 ± 0.25	0.46
DWI
CNR b0	31.6 ± 5.95	29.0 ± 4.2	36.6 ± 5.1	42.2 ± 6.8	36.8 ± 6.3	0.83
CNR b1000	2.27 ± 0.57	1.93 ± 0.32	1.88 ± 0.33	3.30 ± 0.96	3.27 ± 0.96	0.96
CNR b2000	5.6 ± 2.32	4.08 ± 1.64	2.67 ± 0.48	6.22 ± 2.43	6.14 ± 3.52	0.62
CNR b3000	3.65 ± 0.73	2.74 ± 0.63	2.16 ± 0.31	4.24 ± 1.19	5.33 ± 2.06	1.01
Outliers b1000 (%)	1.71 ± 0.68	1.52 ± 0.91	0.57 ± 0.36	1.90 ± 0.77	1.02 ± 0.47	0.76
Outliers b2000 (%)	0.12 ± 0.26	0.12 ± 0.17	0.14 ± 0.30	0.10 ± 0.23	0.06 ± 0.09	0.13
Outliers b3000 (%)	0.14 ± 0.23	0.12 ± 0.16	0.07 ± 0.10	0.14 ± 0.17	0.07 ± 0.10	0.19
SSE b1000	0.06 ± 0.01	0.06 ± 0.01	0.09 ± 0.02	0.05 ± 0.01	0.04 ± 0.01	1.43
SSE b2000	0.18 ± 0.02	0.21 ± 0.04	0.29 ± 0.06	0.17 ± 0.02	0.14 ± 0.03	1.44
SSE b3000	0.38 ± 0.04	0.46 ± 0.09	0.64 ± 0.13	0.39 ± 0.04	0.31 ± 0.07	1.48
FD (mm)	0.32 ± 0.07	0.34 ± 0.08	0.31 ± 0.09	0.36 ± 0.10	0.28 ± 0.08	0.33

Note: Effect sizes are given with Cohen's f. IQMs are described in more detail in Table 2.

Abbreviations: (t)SNR, (temporal) signal‐to‐noise ratio; CJV, coefficient of joint variation; CNR, contrast‐to‐noise ratio; DVARS, spatial standard deviation of successive difference images; EFC, entropy‐focus criterion; FD, framewise displacement; FWHM, full‐width half maximum; INU, intensity non‐uniformity; SSE, sum‐of‐squared error; WMH, temporal variation of WM heterogeneity.

FIGURE 2.

Image Quality Metrics (IQMs) of structural T1w MRI. The variation per site is indicated by the raincloud plots. The traveling volunteers A and B are indicated with the color corresponding to each of the sites. CJV, coefficient of joint variation; EFC, entropy focus criterion; FWHM, full width at half maximum; SNR GM, signal‐to‐noise ratio of gray matter; INU, intensity non‐uniformity

IQMs of rsfMRI were variable (Figure 3) and showed medium to large effect sizes. Temporal WM heterogeneity was low in sites 3, 4 and 5 (median values below 0.5%, with an incidental outlier corresponding to a higher value of framewise displacement). Temporal WM heterogeneity was higher in sites 1 and 2 (median 0.77% and 0.75%, respectively). Scans with very high values could also be identified in maps provided by MRIQC, displaying the temporal SD (Figure 1). Temporal WM heterogeneity was not clearly related to other human IQMs, although static SNR values were notably higher for the Philips scanners (sites 1 and 2).

FIGURE 3.

Image Quality Metrics (IQMs) of resting‐state functional MRI (fMRI). The variation per site is indicated by the raincloud plots. The traveling volunteers A and B are indicated with the color corresponding to each of the sites. DVARS, rate of change of voxel intensity across the entire brain at each volume (t‐); EFC, entropy focus criterion; FD, framewise displacement; FWHM, full width at half maximum; SNR, (temporal) signal‐to‐noise ratio; WMH, temporal variation in WM heterogeneity expressed in %

Like for rsfMRI, motion‐related DWI IQMs (i.e., framewise displacement) were similar across sites (Figure 4). At high b‐values, lower signal intensity makes outlier detection more difficult, resulting in a lower percentage outliers and smaller effect sizes than at b1000. Effect sizes of CNR for all b‐values were large and within‐site variation was larger in sites with a high median CNR. High values of CNR were partly reflected by lower SSE values.

FIGURE 4.

Image Quality Metrics (IQMs) of diffusion‐weighted imaging (DWI). The variation per site is indicated by the raincloud plots. The traveling volunteers A and B are indicated with the color corresponding to each of the sites. CNR: contrast‐to‐noise‐ratio for b0 and per b‐shell, SSE: sum‐of‐squared error of diffusion tensor fit, FD: framewise displacement. For site 5 FD was not determined, because volumes consist of slices that are acquired at different times. On this scanner the design of the DWI sequence used maximal interleaving of b‐values for subsequent slices, which were reconstructed by the scanner into volumes with one b‐value

3.3. Human outcome measures

Human outcome measures are shown in Table 5. Effect sizes for all metrics were still medium to large, but were markedly smaller than for the IQMs.

TABLE 5.

Human outcome measures (mean ± SD) for 28 subjects per site for T1w structural scans and diffusion‐weighted imaging (DWI)

T1	1	2	3	4	5	Cohen's f
Brain volume (ml)	1134 ± 108	1110 ± 114	1204 ± 122	1149 ± 124	1154 ± 125	0.32
GM volume (ml)	649 ± 58	647 ± 60	708 ± 57	655 ± 66	678 ± 66	0.47
WM volume (ml)	458 ± 55	438 ± 57	470 ± 68	467 ± 60	449 ± 60	0.23
NBV (ml)	1479 ± 59	1518 ± 59	1469 ± 41	1559 ± 63	1488 ± 55	0.47
Norm. GM (ml)	847 ± 47	885 ± 40	866 ± 36	889 ± 42	876 ± 35	0.38
Norm. WM (ml)	596 ± 33	597 ± 39	571 ± 38	633 ± 37	577 ± 37	0.52
Cort. Thickness (mm)	2.526 ± 0.066	2.582 ± 0.103	2.509 ± 0.075	2.471 ± 0.07	2.541 ± 0.069	0.49
DWI WM skeleton
FA b1000	0.509 ± 0.018	0.469 ± 0.018	0.520 ± 0.017	0.500 ± 0.02	0.488 ± 0.015	1.04
FA b2000	0.497 ± 0.016	0.472 ± 0.016	0.507 ± 0.015	0.489 ± 0.019	0.479 ± 0.012	0.84
FA b3000	0.477 ± 0.015	0.46 ± 0.014	0.495 ± 0.015	0.470 ± 0.018	0.465 ± 0.012	0.87
MD b1000 a	0.703 ± 0.021	0.717 ± 0.016	0.706 ± 0.018	0.720 ± 0.021	0.724 ± 0.020	0.43
MD b2000	0.589 ± 0.018	0.602 ± 0.014	0.602 ± 0.017	0.609 ± 0.018	0.611 ± 0.016	0.46
MD b3000	0.495 ± 0.015	0.505 ± 0.012	0.511 ± 0.014	0.512 ± 0.016	0.515 ± 0.014	0.53
RD b1000	0.486 ± 0.023	0.517 ± 0.020	0.482 ± 0.020	0.503 ± 0.026	0.512 ± 0.020	0.66
RD b2000	0.409 ± 0.019	0.428 ± 0.016	0.413 ± 0.017	0.426 ± 0.021	0.432 ± 0.016	0.53
RD b3000	0.349 ± 0.016	0.362 ± 0.013	0.353 ± 0.014	0.363 ± 0.017	0.367 ± 0.013	0.50
AD b1000	1.140 ± 0.022	1.127 ± 0.016	1.164 ± 0.024	1.161 ± 0.021	1.157 ± 0.024	0.66
AD b2000	0.954 ± 0.019	0.953 ± 0.014	0.985 ± 0.021	0.977 ± 0.019	0.970 ± 0.020	0.69
AD b3000	0.793 ± 0.016	0.796 ± 0.014	0.834 ± 0.017	0.814 ± 0.015	0.814 ± 0.017	0.97
DWI forceps major skeleton
FA b1000	0.661 ± 0.044	0.648 ± 0.052	0.658 ± 0.043	0.669 ± 0.050	0.622 ± 0.036	0.38
FA b2000	0.663 ± 0.041	0.664 ± 0.045	0.654 ± 0.043	0.675 ± 0.046	0.621 ± 0.033	0.47
FA b3000	0.650 ± 0.043	0.651 ± 0.042	0.649 ± 0.045	0.656 ± 0.042	0.617 ± 0.034	0.36
MD b1000	0.753 ± 0.032	0.744 ± 0.026	0.75 ± 0.028	0.777 ± 0.029	0.781 ± 0.024	0.56
MD b2000	0.634 ± 0.028	0.624 ± 0.021	0.638 ± 0.025	0.655 ± 0.023	0.656 ± 0.024	0.54
MD b3000	0.532 ± 0.023	0.524 ± 0.018	0.54 ± 0.021	0.549 ± 0.018	0.551 ± 0.022	0.52
RD b1000	0.416 ± 0.05	0.424 ± 0.051	0.423 ± 0.039	0.429 ± 0.050	0.464 ± 0.031	0.38
RD b2000	0.352 ± 0.038	0.349 ± 0.038	0.361 ± 0.033	0.361 ± 0.036	0.390 ± 0.024	0.44
RD b3000	0.302 ± 0.032	0.299 ± 0.031	0.307 ± 0.029	0.312 ± 0.028	0.330 ± 0.021	0.40
AD b1000	1.455 ± 0.058	1.419 ± 0.048	1.463 ± 0.055	1.516 ± 0.066	1.468 ± 0.065	0.56
AD b2000	1.23 ± 0.052	1.212 ± 0.037	1.245 ± 0.055	1.287 ± 0.056	1.239 ± 0.067	0.50
AD b3000	1.004 ± 0.035	0.99 ± 0.028	1.045 ± 0.045	1.045 ± 0.037	1.034 ± 0.053	0.61
DWI forceps minor skeleton
FA b1000	0.555 ± 0.029	0.507 ± 0.031	0.573 ± 0.029	0.552 ± 0.034	0.541 ± 0.027	0.75
FA b2000	0.545 ± 0.027	0.506 ± 0.029	0.552 ± 0.025	0.543 ± 0.032	0.529 ± 0.024	0.63
FA b3000	0.526 ± 0.026	0.489 ± 0.027	0.537 ± 0.026	0.523 ± 0.033	0.511 ± 0.025	0.63
MD b1000	0.726 ± 0.029	0.740 ± 0.026	0.731 ± 0.029	0.732 ± 0.031	0.749 ± 0.031	0.29
MD b2000	0.606 ± 0.023	0.620 ± 0.020	0.628 ± 0.022	0.623 ± 0.025	0.631 ± 0.026	0.37
MD b3000	0.505 ± 0.020	0.516 ± 0.015	0.532 ± 0.019	0.524 ± 0.02	0.530 ± 0.021	0.53
RD b1000	0.474 ± 0.032	0.510 ± 0.034	0.468 ± 0.031	0.482 ± 0.038	0.500 ± 0.033	0.49
RD b2000	0.399 ± 0.025	0.426 ± 0.026	0.406 ± 0.022	0.411 ± 0.029	0.422 ± 0.024	0.40
RD b3000	0.339 ± 0.020	0.361 ± 0.021	0.349 ± 0.019	0.352 ± 0.023	0.360 ± 0.019	0.40
AD b1000	1.227 ± 0.039	1.203 ± 0.031	1.274 ± 0.053	1.236 ± 0.044	1.265 ± 0.049	0.60
AD b2000	1.020 ± 0.037	1.008 ± 0.027	1.065 ± 0.041	1.041 ± 0.038	1.046 ± 0.045	0.54
AD b3000	0.842 ± 0.029	0.829 ± 0.022	0.887 ± 0.035	0.861 ± 0.031	0.862 ± 0.037	0.66

Note: Effect sizes are given with Cohen's f.

^a MD, RD and AD are given in units of 10⁻³ mm²/s.

Normalized volumes were more comparable within sites than raw volumes (Figure 5), but effect sizes were similar. Importantly, normalization based on eTIV may introduce additional variability (see traveling volunteer B). Further, eTIV was initially underestimated for 7 participants, and overestimated in one; this could be solved (see Supporting Methods). Cortical thickness showed some variation between sites. Between‐site variation in raw volumes of the traveling volunteers was lower than within‐ or between‐site variation of the 140 healthy participants.

FIGURE 5.

Structural outcome measures from T1w scans. The variation per site is indicated by the raincloud plots. The traveling volunteers A and B are indicated with the color corresponding to each of the sites

Spatial correlations on the fMRI‐derived resting‐state networks showed high consistency across sites 3, 4, and 5, while overlap between sites 1 and 2 and other sites was slightly lower (Figure 6). This might reflect differences in fMRI IQMs between sites. Generally, overlap with the Yeo parcellation was smaller.

FIGURE 6.

Spatial correlations between sites for three selected resting‐state networks: the default mode network (DMN), the somatomotor network (SMN) and the visual network (VN). Correlations are shown between each of the five sites and between all sites and the YEO parcellation (lower row)

Diffusion measures are shown in Figure 7, Figure 8, and Figure 9. For each b‐value separately, we observed only a small variation of MD values within and between sites, although effect sizes were still large. Effect sizes for ROIs (skeletonized Forceps Major and Forceps Minor, indicated in Table 5) were smaller than for the full WM skeleton. MD, AD, and RD values decreased with increasing b‐value, as expected for non‐Gaussian diffusion. The traveling volunteers showed little between‐site variation in diffusion measures.

FIGURE 7.

Fractional anisotropy (FA), mean diffusivity (MD), radial diffusivity (RD), and axial diffusivity (AD) of the WM skeleton. The variation per site is indicated by the raincloud plots. The traveling volunteers A and B are indicated with the color corresponding to each of the sites. The unit of diffusivity is 10⁻³ mm²/s. MD, AD, and RD values decrease with increasing b‐value, as expected for non‐Gaussian diffusion. For instance, MD of the skeleton (mean over all 140 subjects) decreased from 0.71 (b1000) to 0.60 (b2000) to 0.51 × 10⁻³ mm²/s (b3000)

FIGURE 8.

Diffusion measures of skeleton voxels within forceps major. The variation per site is indicated by the raincloud plots. The traveling volunteers A and B are indicated with the color corresponding to each of the sites. The unit of diffusivity is 10⁻³ mm²/s

FIGURE 9.

Diffusion measures of skeleton voxels within forceps minor. The variation per site is indicated by the raincloud plots. The traveling volunteers A and B are indicated with the color corresponding to each of the sites. The unit of diffusivity is 10⁻³ mm²/s

3.4. Phantoms

Deviations in the geometry phantom varied per site and direction, but remained below 0.50%, and were systematic, as shown by small SD's, although small distinct changes in deviations at site 3 and 5 could be explained by gradient re‐calibration, and a software upgrade, respectively (Table 6).

TABLE 6.

Image quality measures (IQMs) phantom data

Site	1	2	3	4	5
Geometry
Deviation x (%)	−0.46 ± 0.03	−0.24 ± 0.07	−0.26 ± 0.16	−0.14 ± 0.12	−0.14 ± 0.27
Deviation y (%)	−0.50 ± 0.03	−0.33 ± 0.05	−0.01 ± 0.02	−0.23 ± 0.13	0.35 ± 0.06
Deviation z (%)	−0.32 ± 0.03	−0.23 ± 0.05	−0.45 ± 0.14	−0.17 ± 0.16	−0.13 ± 0.37
Agar SNR
NEMA SNR COV (%)	6.5	7.0 (14.5) a	3.9	2.8	13.9 a
Agar fMRI
ROI SNR COV (%)	11.5	10.9	4.5	12.4	7.2
ROI FFNR	0.99 ± 0.08	1.04 ± 0.07	1.02 ± 0.03	1.08 ± 0.04	1.01 ± 0.03
Trend (%)	0.87 ± 0.43	0.62 ± 0.51	0.50 ± 0.20	1.07 ± 0.45	0.39 ± 0.22
Fluctuation (%)	0.34 ± 0.06	0.10 ± 0.02	0.05 ± 0.02	0.03 ± 0.00	0.08 ± 0.01
Agar DWI
MD COV x‐y‐z b (%)	1.36	1.29	1.28	1.25	1.11
MD COV b‐shells b (%)	1.04	1.13	1.13	0.34	1.05

^a At site 2 higher COV was related to temporary change of head coil, and at site 5 higher COV was related to an upgrade of reconstruction software (see Figure S1).

^b Within‐session COV of MD (mean diffusivity) between directions x‐y‐z, and between b‐shells b400‐800‐1200.

Within sites, normalized SNR values obtained from the agar phantom were stable over time, with COV between 2.8% and 13.9%. We detected outliers in SNR at site 2, due to a temporary change of head coil (that needed repair), and an SNR increase at site 5, due to an upgrade in reconstruction software (Figure S1).

Compared to these standard NEMA SNR, for most sites fMRI IQMs of the agar phantom yielded higher COVs for SNR, between 4.5% and 12.4%. FFNR, the ratio between SNR and SFNR, had mean values per site between 0.99 and 1.08. Fluctuation was below 0.1% for 4 sites, and 0.34% for site 1. Whether this relates to the larger temporal WM heterogeneity in human IQMs needs future investigation. Mean trend varied between 0.39% and 1.07%, with relatively large SD, showing variation both within and between sites.

DWI IQMs of the agar phantom showed variation of MD over time (Figure S2). Fluctuations were present in all sites, likely due to temperature fluctuations between sessions, and typically random. Only in site 2 we observed a sudden jump in May 2019, coinciding with a software update of the scanner. Within sessions, MD was comparable for the three b‐values, with mean within‐session COV between 0.34% and 1.13% (Table 6). We noticed a small systematic effect of gradient direction, which differed by site (Figure S2), with mean within‐session COV between directions between 1.11% and 1.36%.

4. DISCUSSION

We described the image quality monitoring pipeline of our global multi‐center multi‐vendor study on OCD (Simpson et al., 2020). We showed the importance of collecting regular phantom scans to monitor stability, and noticed relationships between some IQMs of human and phantom scans, but these need to be confirmed in the final cohort. Overall, within‐site variations were smaller than between‐site variations, and effect sizes of IQMs were typically larger than effect sizes of outcome measures. Thus, despite efforts to harmonize scan protocol across sites, between‐site variations persist that can stem from variation between participants other than age, sex, and education or from differences between MRI scanners. Indeed, outcome measures from the traveling volunteers still showed variation, although limited. This highlights the importance of collecting data of matched control subjects at each site, which is part of this study design.

Despite protocol standardization with respect to spatial and timing parameters, between‐site variation in IQMs of the human structural, rsfMRI and DWI images persisted, likely due to factors such as general scanner performance, pulse sequence implementation, and image reconstruction. For rsfMRI the cause of higher temporal WM heterogeneity in two sites is unknown. For DWI, the high between‐site variation in CNR at higher b‐values is unclear, as is the observation that sites with high mean CNR also showed a large within‐site variation; both observations need further investigation in the final cohort. The variation in IQMs had no direct relationship, however, with outcome measures. Outcome measures of human structural scans showed similar between‐site variation for raw volumes compared with normalized volumes. This was especially clear for the two traveling volunteers, who demonstrated consistent raw volumes at all sites, while normalized volumes were more variable. It has been noted that scaling with eTIV might lead to over‐correction (Klasson et al., 2018), and we noticed that eTIV values may depend on whether or not the neck has been removed from the input images. Differences in cortical thickness between sites, also for the two traveling volunteers, are possibly due to subtle tissue contrast differences. DWI outcome metrics were also more comparable between sites, despite the large variation in the corresponding IQMs. Of note, motion‐related IQMs will play only a minor role in DWI outcome measures, because of efficient outlier replacement (Bastiani et al., 2019). Similar as for structural outcome measures, DWI metrics of the traveling subjects were highly comparable between sites, although different from most control participants. Finally, the between‐site variation in functional IQMs might have had a small effect on the spatial consistency of the identified resting‐state networks. Statistical techniques may be employed to reduce site effects further. ComBat, for example, has been shown to be an effective multi‐site harmonization method for DWI (Fortin et al., 2017), fMRI (Yu et al., 2018), and structural MRI data (Fortin et al., 2018; Radua et al., 2020), without removing true effects. Although such techniques may further improve the quality of the data, our study is based on the assumption that prospective harmonization of input data remains preferable. In future analyses we will be able to compare the power of our study with studies based on retrospectively collected imaging data, such as ENIGMA. Geometry phantom IQMs showed systematic but small deviations (0.50% or < 1 mm over 180 mm distance). Relative SNR, an IQM of the agar phantom, remained stable over time for most sites, but was sensitive in detecting the temporary coil replacement at site 2, and a change in reconstruction software at site 5. The timing of this change, and of the gradient re‐calibration at site 3, could be included in structural analyses of the final cohort. IQMs of the agar phantom showed between‐site variation in fMRI fluctuation. Whether this relates to the larger temporal WM heterogeneity in human IQMs in two sites needs to be investigated in the future. Within‐ and between‐site variations in fMRI trend may depend on vendor‐specific implementation of stabilization, and on timing of the experiment (e.g., measuring after scanner start‐up or shortly after extensive gradient use). DWI IQMS of the agar phantom showed good correspondence between MD values at different b‐values, and only small systematic differences along different gradient orientations. These differences are well below the variations observed in a previous large multi‐center study (Belli et al., 2016).

Limitations of this study include the fact that a traveling volunteer could not be scanned at one site due to COVID‐19 restrictions. We only assessed DWI outcome measures based on multiple single‐shell data, while the multi‐shell dataset is valuable for tractography and analyses like NODDI or estimation of free water content (Pasternak et al.,  2012; Zhang et al.,  2012). We also used different agar phantoms, not necessarily from the same batch, and measured DWI at room temperature for pragmatic reasons. Only with a phantom in ice‐water can the temperature be held stable between sessions and sites (Chenevert et al., 2011). Instead, we choose a short pragmatic agar protocol, which can regularly be repeated on clinical scanners, while still providing information about MD stability at different b‐values and gradient directions.

In conclusion, this paper described how we perform the image processing and monitor scan quality in our worldwide multi‐center multi‐vendor study that utilizes clinical scanners and an a priori harmonized data acquisition protocol. The results suggest that phantom and human IQMs can be used to monitor scan quality within sites across the period of participant recruitment. Still, the field may benefit from additional benchmark studies to recommend limits for (in)sufficient data quality (similar to the cut‐off for framewise displacement during rsfMRI). The between‐site variability in outcome measures emphasizes the importance of including patients and matched control subjects at each site. Although it will remain important to reduce remaining site effects in the final dataset using statistical techniques, the development of standardized acquisition and analysis protocols could help the neuroimaging field in psychiatry move toward greater reproducibility and rigor.

CONFLICT OF INTEREST

Dr. van den Heuvel received consultancy fee from Lundbeck and a stipend from Elsevier for serving as associate editor of Journal of Obsessive‐Compulsive and Related Disorders.

Dr. Simpson has received research support from Biohaven Pharmaceuticals, royalties from UpToDate Inc and Cambridge University Press, and a stipend from the American Medical Association for serving as Associate Editor of JAMA Psychiatry.

Dr. Stein has received research grants and/or consultancy honoraria from Johnson & Johnson, Lundbeck, Servier, and Takeda.

Dr. Vriend is listed as an inventor on a patent licensed to General Electronic.

(WO2018115148A1).

None of the remaining authors has any disclosures.

Supporting information

Supporting Information S1

Pouwels, P. J. W. , Vriend, C. , Liu, F. , de Joode, N. T. , Otaduy, M. C. G. , Pastorello, B. , Robertson, F. C. , Venkatasubramanian, G. , Ipser, J. , Lee, S. , Batistuzzo, M. C. , Hoexter, M. Q. , Lochner, C. , Miguel, E. C. , Narayanaswamy, J. C. , Rao, R. , Janardhan Reddy, Y. C. , Shavitt, R. G. , Sheshachala, K. , … van den Heuvel, O. A. (2023). Global multi‐center and multi‐modal magnetic resonance imaging study of obsessive‐compulsive disorder: Harmonization and monitoring of protocols in healthy volunteers and phantoms. International Journal of Methods in Psychiatric Research, 32(1), e1931. 10.1002/mpr.1931

DATA AVAILABILITY STATEMENT

Data available on request from the authors.