Bootstrap approach for meta-synthesis of MRI findings from multiple scanners

Abstract

Background-

Neuroimaging data from large epidemiologic cohort studies often come from multiple scanners. The variations of MRI measurements due to differences in magnetic field strength, image acquisition protocols, and scanner vendors can influence the interpretation of aggregated data. The purpose of the present study was to compare methods that meta-analyze findings from a small number of different MRI scanners.

Methods-

We proposed a bootstrap resampling method using individual participant data and compared it with two common random effects meta-analysis methods, DerSimonian-Laird and Hartung-Knapp, and a conventional pooling method that combines MRI data from different scanners. We first performed simulations to compare the power and coverage probabilities of the four methods in the absence and presence of scanner effects on measurements. We then examined the association of age with white matter hyperintensity (WMH) volumes from 787 participants.

Results-

In simulations, the bootstrap approach performed better than the other three methods in terms of coverage probability and power when scanner differences were present. However, the bootstrap approach was consistent with pooling, the optimal approach, when scanner differences were absent. In the association of age with WMH volume, we observed that age was significantly associated with WMH volumes using the bootstrap approach, pooling, and the DerSimonian-Laird method, but not using the Hartung-Knapp method (p <0.0001 for the bootstrap approach, DerSimonian-Laird, and pooling but p=0.1439 for the Hartung-Knapp approach).

Conclusion-

The bootstrap approach using individual participant data is suitable for integrating outcomes from multiple MRI scanners regardless of absence or presence of scanner effects on measurements.

Graphical Abstract

Introduction

Access to multiple magnetic resonance imaging (MRI) facilities is indispensable when recruiting volunteers for research scans in large community-based studies in order to reduce participant burden. However, pooling data from multiple MRI scanners has potential limitations because MRI measurements are sensitive to vendor, magnetic field strength, and image acquisition parameters (Takao et al., 2011,Takao et al., 2013,Barrio-Arranz et al., 2015).

White matter hyperintensities (WMHs), changes in white matter as seen on T2-weighted MRI scans, are thought to reflect vascular disease (Wardlaw et al., 2015,Chao et al., 2013,Dickie et al., 2016) and Alzheimer’s Disease (AD) risk (Brickman et al., 2009,Alosco et al., 2018,Kandel et al., 2016,Arfanakis et al., 2020). Total WMH or lesion volume is reported to vary by image acquisition parameters, magnetic field strength, vendors and automatic segmentation tools (Guo et al., 2019,Shinohara et al., 2017,Tudorascu et al., 2016). For this reason, while pooling data increases statistical power, the validity of pooling MRI measures acquired from different scanners and protocols has been questioned (Ashburner., 2009,Focke et al., 2011,Stonnington et al., 2008,Bauer et al., 2010,Marchewka et al., 2014).

Meta-analysis has been utilized for synthesizing results from multiple studies or multi-center studies, where the study effect is often treated as a random entity. Application of meta-analysis in biomedical research has increased dramatically over the past decade. A search with the keyword meta-analysis in PubMed showed about 151,000 publications between 2008 and 2018, more than five times the 27,000 publications from the previous decade (1998–2008). Random-effects meta-analysis has been employed for neuroimaging studies (Costafreda., 2009,Berlingeri et al., 2019,Tench et al., 2017,Fusar-Poli, P. et al., 2009,Fusar-Poli, Paolo., 2012). However, random effects meta-analysis that incorporates variance across scanners/centers in a population might be overly conservative when inferences are about the average effect with data from a small number of MRI scanners.

In order to measure the effects of interest without undue variation that comes from combining data across scanners, we propose a meta-analysis using bootstrap resampling with individual participant data within each scanner, where the variance of combined summary statistics across scanners is estimated by a bootstrap method. Here we compared this approach to two common random effects meta-analysis methods, DerSimonian and Laird (DerSimonian and Laird., 1986) and Hartung and Knapp (Hartung and Knapp., 2001b,Hartung and Knapp., 2001a), and a pooling method that combines individual-level data from different MRI scanners. We simulated data under scenarios with and without scanner effects on measurements and compared coverage probability and power of four methods. In addition, the performance of all four methods was compared using WMH volumes from 787 participants from four ongoing cohort studies of aging and dementia (Bennett et al., 2018,Barnes et al., 2012,Barnes et al., 2015,Schneider et al., 2009) at the Rush Alzheimer’s Disease Center to replicate the well-established association of age with WMH (Gunning-Dixon et al., 2009).

2. Methods

2.1. Meta-analysis

We analyzed the association of WMH volume with age after controlling for sex, race, clinical diagnosis, and global cognition at each scanner. The association of age with WMH from each scanner is represented by the regression coefficient of age (${\widehat{\beta }}_i$, i=1, …, K; number of scanners, K) in a multiple linear regression on the WMH volumes. In random effects meta-analysis, the association of age estimated from the data obtained from the i^th MRI scanner, denoted by β_i, is considered a random perturbation from the population-level effect (β) so that β_i= β +ε_i. A synthesized estimate $\left(\widehat{\beta }\right)$ is estimated as a weighted sum of individual estimates, ${\widehat{\beta }}_i$ (i=1,…,K).

$${\widehat{\beta }}^R=\frac{\sum _{i=1}^K{w}_i^R{\widehat{\beta }}_i}{\sum _{i=1}^K{w}_i^R},$$ (1)

where ${\text{W}}_i^R=1/\left({\widehat{\sigma }}_i^2+{\widehat{\tau }}^2\right)$ is the weight for i^th study using ${\widehat{\sigma }}_i^2$ the sampling variance estimated from the regression model from the i^th scanner, and ${\widehat{\tau }}^2$ is the between-scanner variance of the ${\widehat{\beta }}_i$ across the K estimated coefficients.

Estimation of the variance of the weighted sum, ${\widehat{\beta }}^R$, differs by methods. We investigated two commonly applied meta-analytic approaches: DerSimonian-Laird (DerSimonian and Laird., 1986) and Hartung-Knapp (Hartung and Knapp., 2001b,Hartung and Knapp., 2001a). In DerSimonian-Laird, the variance of the weighted sum is $1/\sum _{i=1}^K{\text{W}}_i^R$, whereas Hartung-Knapp adopts a more empirical approach, a weighted expansion of empirical variance of ${\widehat{\beta }}_i$ (Hartung and Knapp., 2001a,Hartung and Knapp., 2001b). The Hartung-Knapp approach is often preferred because its coverage probability is closer to the nominal level than DerSimonian-Laird approach (Partlett and Riley., 2017,Guolo and Varin., 2017). Additionally, while DerSimonian-Laird approach uses a normal distribution for tests, the Hartung-Knapp approach uses a t-distribution with degrees of freedom (df) being one less than the number of studies. When the number of studies is small, the t-test can be conservative (Higgins et al., 2008,Guolo and Varin., 2017).

2.2. Bootstrap approach with individual participant data

In this paper, we proposed a bootstrap approach to deal with scanner effects in meta-analyzing data. In contrast to the random effects meta-analysis in Eq. (1), we did not incorporate the variance across scanners $\left({\widehat{\tau }}^2\right)$ into the weight terms because we performed inferences on the average effect from the observed data. Therefore, we set the weights as ${\text{W}}_i^B=1/\left({\widehat{\sigma }}_i^2\right)$. And the weighted sum in the approach is given as

$${\widehat{\beta }}^B=\frac{\sum _{i=1}^K{w}_i^B{\widehat{\beta }}_i}{\sum _{i=1}^K{w}_i^B}.$$ (2)

This formation is also referred to as common effect meta-analysis. In this study, we adopt an empirical approach for estimating the variance of the weighted sum in Eq. (2). Common effect meta-analysis adopts theoretical variance based on an unrealistic assumption that weights (or inverse of variance) are known prior to study. It has been reported that the theoretical variance underestimates the true variance (Li et al., 1994,Hedges and Olkin., 1985,Domínguez Islas and Rice., 2018). Our method requires individual participant data to estimate the variance of the weighted sum in Eq. (2). We estimate the variance by employing bootstrap resampling (Efron., 1979).

Our procedure consists of three steps, as specified below. We denote the data set for each MRI scanner as (D_i; i =1,...,K), and the j^th bootstrap replicate of the data as $D_i^{(j)}$. We repeat Steps 1 and 2 below M times (j=1, …, M).

Step 1: Within-scanner estimation

Both the estimated coefficient and its estimated sampling variance, (${\widehat{\beta }}_i^{(j)}$, ${\widehat{\sigma }}_i^{(j)}$), are estimated from $D_i^{(j)}$ (i=1, …, K).

Step 2: Integration of estimates across scanners

The weighted sum, ${\widehat{\beta }}^{B(j)}$, with the j^th bootstrap replicate of the data ($D_i^{(j)}$: i=1,...,K) is given as

$${\widehat{\beta }}^{B(j)}=\frac{\sum _{i=1}^K{w}_i^{B(j)}{\widehat{\beta }}_i^{(j)}}{\sum _{i=1}^K{w}_i^{B(j)}},$$ (3)

where the weights are given as $w_i^{B(j)}=1/({({\widehat{\sigma }}_i^{(j)})}^2)$.

Step 3: Construction of the confidence interval

Finally, we construct confidence level by using quantiles from normal distribution as suggested in (DiCiccio and Efron., 1996):

$${\widehat{\beta }}^B\pm z_{1-\frac{\alpha }{2}}\sqrt{{\text{V}}^B({\widehat{\beta }}^{B(j)})},$$

where ${\text{V}}^B({\widehat{\beta }}^{B(j)})$ is the variance estimated with the bootstrap samples of ${\widehat{\beta }}^{B(j)}$ (j=1, …, M) in Eq. (3), i.e., ${\mathbf{V}}^B({\widehat{\beta }}^{B(j)})=\frac{1}{M-1}\sum _{j=1}^M{({\widehat{\beta }}^{B(j)}-{\widehat{\beta }}^{boot})}^2$ and ${\widehat{\beta }}^{boot}=\frac{1}{M}\sum _{j=1}^M{\widehat{\beta }}^{B(j)}$. A diagram of the proposed bootstrap approachis shown in Figure 1. In Figure 2, procedures of the bootstrap approach, two random effects approaches, and pooling were compared. R code for implementing the proposed bootstrap algorithm is provided in Supplementary material 1.

Figure 1. Diagram of the Bootstrap approach.

At each bootstrap resampling (j=1,…,M), the weighted sum, ${\widehat{\beta }}^{B(j)}$, is estimated, and the variance is obtained as the final outcome. 95% CI is estimated using quantiles from the standard normal distribution.

Figure 2.

Procedures from the four methods.

2.3. Simulation design

We conducted simulations under two scenarios: presence and absence of scanner differences. Parameters applied in the simulations included the number of scanners and the number of participants at each scanner. The number of scanners (K) tested was small (2, 5); the number of participants (n) at each scanner was tested with small to moderate (30, 50, 100) sample sizes. Individual participant data at each scanner without scanner differences was generated as y_il = α + βx_l + e_il, with e_il ~ N(0,σ²) (i=1, …, K; l=1, …, n). We incorporated scanner-specific level effect and variance in the presence of scanner effects on measurements as y_il = α_i+ βx_l + e_il with $e_{il}~N\left(0,{\sigma }_i^2\right)$, where α_i = α +δ_i and ${\sigma }_i^2={\sigma }^2{v}_i^2$. Two random components $\left({\delta }_i,v_i^2\right)$ were sampled from δ_i~U(−5,5), and $v_i^2~{\chi }^2$(5). The coefficient of determination was 0.09 on average per dataset $\left(r_i^2=0.09\right)$ in the absence of scanner effects and 0.018 with the presence of scanner effects $\left(r_i^2=0.018\right)$. We compared all four methods based on coverage probability and power. Coverage probability is defined as the probability that the confidence interval includes the true value, and is estimated as the proportion of confidence intervals in simulations that cover the pre-specified true value. At (1-p)×100% confidence interval, we expected to cover the true value with (1-p) probability, where p refers to the type I error rate when the null hypothesis is true. Higher than the expected coverage probability indicates a higher type II error, and lower coverage probability a higher type I error. Accuracy of a method is thus determined by closeness to the target coverage probability. Power is the probability of rejecting the null hypothesis when an alternative hypothesis is true. Thus, higher power is preferred.Details of the simulation design are shown in Supplementary material 2.

2.4. MRI Acquisition

Scanner A is a 3 Tesla Siemens Trio (Erlangen, Germany) MRI scanner. High-resolution T1-weighted anatomical data were obtained using a 3D MPRAGE sequence. T₂–weighted FLAIR data were collected on all participants using a 2D fast spin-echo sequence. Details of acquisition parameters for the 3D MPRAGE and 2D fast spin-echo sequence are shown in Table 1.

Table 1.

MRI acquisition parameters

Scanner	Scanner A	Scanner B
Vendor	Siemens Trio	Philips Achieva
Field strength	3T	3T
	T1-weighted 3D MPRAGE
TE (msec)	2.98	3.7
TR (msec)	2300	8
TI (msec)	900	955
flip angle (radian)	9	8
FOV (cm2)	25.6×25.6	24.0×22.8
Sagittal slices (n)	176	181
Slice thickness (mm)	1	1
Acquisition matrix	256×256	240×228
Parallel Imaging Acceleration Factor	2	2
	T2-weighted 2D FLAIR
TE (msec)	150	90
TR (sec)	9	9
TI (sec)	2.49	2.5
FOV (cm2)	22×22	22×22
Axial slices (n)	35	35
Slice thickness (mm)	4	4
Acquisition matrix	256×256	256×201
Parallel Imaging Acceleration Factor	2	1.6

Scanner B is a 3 Tesla Philips Achieva (Best, The Netherlands) MRI scanner. High-resolution T1-weighted anatomical data were obtained using a 3D MPRAGE sequence. T₂–weighted FLAIR data were collected on all participants using a 2D fast spin-echo sequence. Details of acquisition parameters for the 3D MPRAGE and 2D fast spin-echo sequence are shown in Table 1.

2.5. WMH assessment

The total volume of WMH was quantified for each participant using an automated segmentation of WMHs: the brain intensity abnormality classification algorithm (BIANCA)(Griffanti et al., 2016). The T1-weighted MPRAGE data was first spatially registered to the T2-weighted FLAIR data using affine registration (FLIRT, FMRIB, University of Oxford, UK). The brain was extracted from the coregistered MPRAGE and FLAIR image volumes (BET, FMRIB, University of Oxford, UK). WMHs were then automatically segmented for each participant’s native space using BIANCA with T₂-weighted FLAIR image. For each participant, maps of WMHs were generated, and the total volume of WMHs was measured and normalized by the intracranial volume (ICV) derived from CAT12 (http://www.neuro.uni-jena.de/cat/). WMH volume, which is represented as percentage of ICV, was then log10-transformed to reduce skewness.

2.6. Participants

Participants were older adults enrolled in ongoing cohort studies of aging and dementia including the Religious Orders Study (ROS), the Rush Memory and Aging Project (MAP) (Bennett et al., 2018), the Minority Aging Research Study (MARS) (Barnes et al., 2012), and the Rush Clinical Core (Barnes et al., 2015,Schneider et al., 2009). Eligibility for each study requires age ≥65 years, absence of known dementia, and agreement to annual clinical evaluations. The 3T MRI sub-study began in 2012. Of about 1900 participants eligible for and consenting to MRI scans since 2012, 1127 participants completed at least one MRI scanning session. All four studies and the MRI substudy were approved by the Institutional Review Board of Rush University Medical Center. Written informed consent was obtained from each participant. De-identified data employed in this study can be requested at the Research Resource Sharing Hub of the Rush Alzheimer’s Disease Center (www.radc.rush.edu).

2.7. Demographic and clinical variables

Date of birth, sex, years of education, and race were self-reported at study entry; age at scan was calculated. Race categories were based on the 1990 U.S. Census as previously described (Barnes et al., 2012). Vascular risk factors such as smoking, diabetes and hypertension have been shown to be associated with WMH volumes, with a higher number of vascular risk factors being associated with a higher volume of WMH (Williamson, Lewandowski et al. 2018, Gebeily, Fares et al. 2014). Self-report of smoking status and histories of hypertension and diabetes were obtained through self-report and inspection of medications. The number of these factors reported was a summary of vascular risk (0 – 3).

Participants underwent a comprehensive neuropsychological assessment. A core battery of 17 tests was used to make a composite score for global cognition, as previously described (Wilson et al., 2015,Wilson et al., 2005). To create the composite, raw scores from the individual cognitive scores were converted to z-scores using the baseline mean and standard deviation of all participants enrolled in the parent cohorts. Each participant’s standardized z-scores were then averaged to yield a composite global cognition score. Data were reviewed by a neuropsychologist to determine cognitive status at each visit. Participants were evaluated by a clinician who used all cognitive and clinical data available to provide a diagnosis. Dementia and related conditions were diagnosed using the guidelines of the joint working group of the National Institute of Neurological and Communicative Disorders and Stroke and Alzheimer’s Disease and Related Disorders Association (McKhann et al., 1984). Diagnosis of MCI was applied to individuals who had cognitive impairment but did not meet criteria for dementia.

2.8. Association of age with WMH volume

We analyzed the association of age with WMH volume after controlling for sex, education, race, vascular risk, clinical diagnosis, and a measure of global cognition. The association of age with WMH volume was estimated from each scanner, and then meta-analyzed using the four methods described above: the bootstrap approach with individual participant data, pooling, DerSimonian-Laird, and Hartung-Knapp. We compared coverage probability and power of four methods using simulated data with and without scanner differences.

3. Results

In simulation with absence of scanner differences, the bootstrap approach performed better than two random effects meta-analyses, with its coverage probability being closer to the nominal level, 95% (Bootstrap 0.94, DerSimonian-Laird 0.99, Hartung-Knapp 1.0, Pooling 0.95 with the number of scanners (K) =2 and the number of participants (N) = 50) (Table 2, Figure 3). Moreover, the coverage of the bootstrap approach was similar to that of pooling, which is the optimal approach in the absence of scanner differences (Table 2, Figure 3). In the presence of scanner differences, the bootstrap approach performed better than two random-effects meta-analyses and pooling (Table 3, Figure 3). While pooling performed the best in the absence of scanner differences, its performance was poorer in the presence of scanner differences (Bootstrap 0.94, DerSimonian-Laird 0.99, Hartung-Knapp 1.0, Pooling 0.92 with K=2 and N=50). Coverage probabilities of all four methods were more accurate when the number of scanners was five and as the number of subjects per scanner increased, whether scanner differences were present or absent.

Table 2.

Coverage probability in the absence of scanner differences.

	Bootstrap	Pooling	DerSimonian-Laird	Hartung-Knapp
N	Number of scanners (K) = 2
30	0.945	0.950	0.985	1
50	0.941	0.945	0.985	1
100	0.956	0.959	0.989	1
	Number of scanners (K) = 5
30	0.941	0.942	0.969	0.998
50	0.950	0.950	0.978	1
100	0.948	0.949	0.976	0.998

Number of people within each scanner (n)

The nominal coverage probability was set at 0.95.

Figure 3.

Coverage probability. Coverage probability (y-axis) versus the number of participants (N= 30, 50, 100) in x-axis from four methods, bootstrap, pooling, DerSimonian-Laird, and Hartung-Knapp, are shown. Simulation results without scanner differences are shown in (a) and (b), and those with scanner differences in (c) and (d). The number of scanners is two in (a) and (c), and five in (b) and (d). In each plot, red line is for the nominal coverage probability (0.95), the blue solid line connects the simulation results for the bootstrap method, the black solid line for the pooling approach, the black dotted line for Dersimonian-Laird, and the black dot-dashed line those for Hartung-Knapp.

Table 3.

Coverage probability in the presence of scanner differences.

	Bootstrap	Pooling	DerSimonian-Laird	Hartung-Knapp
N	Number of scanners (K) = 2
30	0.927	0.921	0.982	1.000
50	0.942	0.917	0.987	1.000
100	0.942	0.908	0.987	1.000
	Number of scanners (K) = 5
30	0.944	0.920	0.975	0.996
50	0.944	0.937	0.976	0.998
100	0.941	0.927	0.981	0.998

Number of people within each scanner (n)

The nominal coverage probability was set at 0.95.

The bootstrap approach showed higher power than the two random-effects meta-analyses in the absence of scanner differences in simulation (Table 4, Figure 4). Again, the power of bootstrap approach was similar to that of pooling, the optimal approach in the absence of scanner differences (Bootstrap 0.89, DerSimonian-Laird 0.60, Hartung-Knapp 0.0, Pooling 0.90 with K=2 and N=50). (Table 4, Figure 4). In the presence of scanner differences, the bootstrap approach showed higher power than two random-effects meta-analyses and pooling (Table 5, Figure 4). While pooling performed the best in the absence of scanner differences, its performance was again poorer when scanner differences were present (Bootstrap 0.45, DerSimonian-Laird 0.20, Hartung-Knapp 0.0, Pooling 0.41 with K=2 and N=50). Power of all four methods, was higher as the number of subjects increased and when the number of scanners was five, whether scanner differences were present or not.

Table 4.

Power in the absence of scanner differences.

	Bootstrap	Pooling	DerSimonian-Laird	Hartung-Knapp
N	Number of scanners (K) = 2
30	0.738	0.773	0.456	0.000
50	0.890	0.897	0.602	0.000
100	0.998	0.999	0.930	0.000
	Number of scanners (K) = 5
30	0.998	0.973	0.911	0.679
50	0.999	0.999	0.993	0.940
100	1.000	1.000	1.000	1.000

Number of people within each scanner (n)

The coefficient of determination simulated was 0.09 on average per scanner.

Figure 4.

Power. Power (y-axis) versus the number of participants (N= 30, 50, 100) in x-axis from four methods, bootstrap, pooling, DerSimonian-Laird, and Hartung-Knapp, are shown. Results without scanner differences are shown in (a) and (b), and those with scanner differences in (c) and (d). The number of scanners is two in (a) and (c), and five in (b) and (d). In each plot, blue solid line is for the bootstrap method, black solid line for the pooling approach, black dotted line for Dersimonian-Laird, and black dotdashed line for Hartung-Knapp.

Table 5.

Power in the presence of scanner differences.

	Bootstrap	Pooling	DerSimonian-Laird	Hartung-Knapp
N	Number of scanners (K) = 2
30	0.311	0.308	0.127	0.000
50	0.445	0.414	0.205	0.000
100	0.680	0.615	0.364	0.000
	Number of scanners (K) = 5
30	0.590	0.470	0.417	0.154
50	0.866	0.748	0.699	0.398
100	0.980	0.933	0.917	0.712

Number of people within each scanner (n)

The coefficient of determination simulated was 0.018 on average per scanner.

We examined four methods with WMH data from 787 participants described above. There were 409 participants scanned in scanner A and 378 scanned in scanner B. The mean age was 79.6 years (SD=6.7) and 77.4 years (SD=6.8) for scanners A and B, respectively, and the mean education was about 16 years (SD=3.2), and 15.8 (SD=3.6), respectively. Over 70% of participants were female at each scanner. The Scanner B group was younger, had a higher percentage of African Americans, higher vascular risk scores, lower global cognitive scores, and a higher proportion of cognitively intact participants (Table 6).

Table 6.

Characteristics of study participants by Scanner.

	Scanner A (N=409)	Scanner B (N=378)	P-value for difference
Age
Mean (SD)	79.6 (6.7)	77.4 (6.8)	P < 0.0001
Men (%)	96 (23.5)	70 (18.5)	P = 0.0968
Education
Mean (SD)	16.0 (3.2)	15.8 (3.6)	P = 0.3423
Race
AA (%)	90 (22.0)	275 (72.8)	P < 0.0001
Global cognition
Mean (SD)	0.2048 (0.61)	0.1181 (0.62)	P = 0.0337
Clinical diagnosisa
NCI (number,%)	316 (77.5)	318 (85.5)	P = 0.0044
MCI (number,%)	87 (21.3)	48 (12.9)
Dementia (number,%)	5 (1.2)	6 (1.6)
Vascular risk factors	1.2	1.4	P= 0.0002
Smoke (number, %)b	166 (41.4)	155 (41.3)	P= 1.0000
Diabetes (number, %)c	63 (15.5)	99 (26.3)	P= 0.0002
Hypertension (number, %)d	243 (59.7)	268 (71.1)	P= 0.0010

AA, African American (AA); NCI, no cognitive impairment; MCI, mild cognitive impairment (MCI)

^aClinical diagnosis was not available for seven participants.

^bHistory of smoking was not available for eleven participants.

^cHistory of diabetes was not available for three participants

^dHistory of hypertension was not available for three participants

Note: Participants from two scanners showed difference in age, racial distribution, cognitive score, clinical diagnosis, and vascular risk factors.

In a meta-analysis for the association of age with WMH volumes after controlling for demographics, vascular risk, clinical diagnosis, and global cognition, the proposed bootstrap approach, DerSimonian-Laird and pooling all indicated that older age was associated with increased WMH volume (all Ps <0.0001), but the Hartung-Knapp approach (P=0.1439) was not, as shown in Table 7. The association of age with WMH volumes by MRI scanner is shown in Figure 5.

Table 7.

Association of age with WMH volume.

	Coefficient	S.E.	P-value
Scanner A	0.0198	0.0029	P < 0.0001
Scanner B	0.0106	0.0022	P < 0.0001
Bootstrap approach	0.0139	0.0017	P < 0.0001
Pooling	0.0154	0.0018	P < 0.0001
DerSimonian-Laird	0.0146	0.0034	P < 0.0001
Hartung-Knapp	0.0146	0.0034	P = 0.1439

Note: 1. WMH volume = log10 (WMH volume in mm³ as a percentage of ICV) 2. One year of additional age is associated with 1.05% greater WMH volume at Scanner A, and 1.02% at Scanner B. 3. In these analyses, the DerSimonian-Laird approach applies the standard Gaussian distribution for inference, while Hartung-Knapp approach applies the t-distribution with 1 degree of freedom.

white matter hyperintensity (WMH); intracranial volume (ICV).

Figure 5.

Association of age with WMH volume by scanner. Blue dots show participants from Scanner A, and turquoise dots from Scanner B. WMH volume = log10 (WMH volume in mm³ as a percentage of ICV).

4. Discussion

Large epidemiological studies involving MRI often utilize multiple scanners. Estimated lesion volume has been reported to vary by image acquisition parameters, magnetic field strength, and vendors as well as automatic segmentation tools (Shinohara et al., 2017,Tudorascu et al., 2016,Guo et al., 2019). We sought a bootstrap integration of data from different MRI scanners with individual participant data. We also examined pooling and two random effect meta-analysis methods: DerSimonian-Laird and Hartung-Knapp. In simulations with an absence of scanner differences, the bootstrap approach performed better than two random effects meta-analyses in terms of coverage probability and power, and was similar to pooling, the optimal approach. However, when scanner differences were present, the bootstrap approach performed better than the other three methods. In testing the association of age with WMH, we found the association was supported with the bootstrap, pooling, and DerSimonian-Laird approaches, but not with the Hartung-Knapp approach.

While meta-analysis is a standard approach for synthesis of findings, random effects meta-analysis incorporating variance across scanners could be overly conservative when inferences are about the average effect in data from a small number of MRI scanners. We demonstrated this using simulated data and actual WMH volume data. In simulations, random effects meta-analysis showed higher coverage probability and lower power for either presence or absence of scanner effects. In the data analysis of the association of age with WMH volume, Hartung-Knapp approach was overly conservative and failed to show an association (p-values < 0.0001 with all methods except Hartung-Knapp (p-value=0.1439)). We also found that meta-analysis results varied depending on the method chosen. Out of the two methods, Hartung-Knapp was more conservative. Of note, the underlying distribution for testing statistics is a normal distribution for DerSimonian- Laird, compared to a t-distribution for Hartung-Knapp (df = the number of studies-1; df=1 with two studies). While differences between normal and t-distributions become negligible when the number of scanners increases, hypothesis testing using t-distributions has very low power when the number of studies is small, as was the case in the current study (Higgins et al., 2008,Guolo and Varin., 2017).

While pooling had properties similar to those of the bootstrap approach in simulations with absence of scanner effects, pooling was poorer when scanner effects were present. Potential bias and increased variation from multiple scanners have been reported (Barrio-Arranz et al., 2015,Takao et al., 2011,Takao et al., 2013), therefore, conclusions from pooling can differ from those from the bootstrap approach and can often be misleading. We thus discourage the pooling approach unless scanner differences are found to be negligible. We simulated scanner-specific level and variance. Further study would be warranted to understand which causes of scanner differences have more deterimental influence on scanner differences. We demonstrated our bootstrap approach with case resampling (Efron., 1979). However, our method can be combined with other types of resampling methods as well, for example, residual or parametric resampling. Residual resampling is a semi-parametric approach that resamples residuals from the regression model with the whole data under assumption that the regression estimates are close to the truth, while parametric resampling makes assumptions on both the shape of regression and distribution of errors. Since case resampling requires no assumptions, it is suitable for situations where large uncertainty in the true model is expected, and gives wider confidence intervals than does residual or parametric resampling (Davison and Hinkley., 2013). Case resampling is typically recommended when the sample size is relatively small.

Since the bootstrap approach resamples data with replacement, the number of all possible resampling is nⁿ at each scanner. We randomly selected 10000-bootstrap samples. While larger M gives better approximation, we confirmed that the estimate was fairly stable as shown in Supplementary Figure S.3. The bootstrap approach was constructed based on the assumption that there exists a common effect across scanners. If the common effect assumption of the bootstrap method is suspect, we suggest using random effects meta-analysis which allows random variation in the effect of interest.

We acknowledge that there are other methods that adjust MRI intensity as opposed to meta-analysis, such as the ComBat (Johnson et al., 2007). It is noteworthy however, that to estimate scanner effects with the ComBat method, multiple MRI measurements from the same scanner are required, such as FreeSurfer (Fischl and Dale., 2000) measurements from all subfields. Our bootstrap method is therefore favored when a single measure per person is of interest, as in the present WMH volume analyses. Furthermore, the bootstrap approach can be applied to metrics from other MRI modalities such as Freesurfer measurements, diffusion tensor imaging and magnetic susceptibility mapping. Our finding replicates the well-established association between older age and greater WMH volume. We chose this example to demonstrate that random effects meta-analysis is conservative when the average effect across scanners is of interest and the number of scanners is small. Out of two meta-analysis methods employed, Hartung-Knapp method was more conservative and failed to show the WMH/age association.

This study has several strengths. Our study is based on four well-defined community-based cohorts that have relatively large sample sizes and well-characterized cognitive performance, health-related factors, and MRI data. Using a range of validated meta-analytic approaches, we compared the utility of each to the bootstrap approach with individual participant data.

This study also has several limitations. First, participants are not randomized to scanner. We therefore did not test for differences in WMH volumes between the MRI scanners because the scanners differed in the demographic characteristics of participants as well as in image acquisition parameters and vendors. Second, the proposed approach is computationally intensive. It is well known that resampling algorithms (e.g. permutation, Jackknife, and bootstrap) are computationally intensive. However, given recent advances in computing power, 10,000 bootstrap resampling with WMH volume in this study was completed in about 30 secs with 3.0 GHz CPU and 8.0 GB RAM PC. Further reduction of computation time may be obtained by adopting a parallel computation algorithm. Efficiencies and computational power will enhance the use of the proposed method to address important questions in the future.

Highlights.

• Random effects meta-analysis is not suitable when the number of MRI scanners is small.

• Pooling performs poorly when there is undue variation due to scanner differences.

• The proposed approach performs better than random effects meta-analysis and pooling.