Reliability correction for functional connectivity: Theory and implementation

Abstract

Network properties can be estimated using functional connectivity MRI (fcMRI). However, regional variation of the fMRI signal causes systematic biases in network estimates including correlation attenuation in regions of low measurement reliability. Here we computed the spatial distribution of fcMRI reliability using longitudinal fcMRI datasets and demonstrated how pre‐estimated reliability maps can correct for correlation attenuation. As a test case of reliability‐based attenuation correction we estimated properties of the default network, where reliability was significantly lower than average in the medial temporal lobe and higher in the posterior medial cortex, heterogeneity that impacts estimation of the network. Accounting for this bias using attenuation correction revealed that the medial temporal lobe's contribution to the default network is typically underestimated. To render this approach useful to a greater number of datasets, we demonstrate that test‐retest reliability maps derived from repeated runs within a single scanning session can be used as a surrogate for multi‐session reliability mapping. Using data segments with different scan lengths between 1 and 30 min, we found that test‐retest reliability of connectivity estimates increases with scan length while the spatial distribution of reliability is relatively stable even at short scan lengths. Finally, analyses of tertiary data revealed that reliability distribution is influenced by age, neuropsychiatric status and scanner type, suggesting that reliability correction may be especially important when studying between‐group differences. Collectively, these results illustrate that reliability‐based attenuation correction is an easily implemented strategy that mitigates certain features of fMRI signal nonuniformity. Hum Brain Mapp, 2015. © 2015 Wiley Periodicals, Inc.

INTRODUCTION

Intrinsic functional connectivity magnetic resonance imaging (fcMRI) has emerged as a powerful tool for mapping large‐scale networks in the human brain [Biswal et al., 1995; Buckner et al., 2013; Craddock et al., 2013; Fox and Raichle, 2007]. It has provided insight in features of human brain organization and has shown potential to identify alterations of network structure in neuropsychiatric and neurodegenerative disorders including Alzheimer's disease, schizophrenia, and autism [Buckner et al., 2008; Fox and Greicius, 2010]. fcMRI utilizes correlations in spontaneous fluctuations in the blood oxygenation level‐dependent (BOLD) signal to assess functional coupling between brain regions [Biswal et al., 1995]. In general, correlations are known to be systematically biased downward due to measurement instability, a phenomenon termed attenuation [Fan, 2003]. fcMRI estimates are therefore likely to be influenced by measurement instability of the BOLD signal that arises from technical constraints like magnetic susceptibility artifacts [Ojemann et al., 1997] as well as physiological functions such as respiration and cardiac activity [Birn et al., 2008; Chang et al., 2009]. Other sources of instability may arise from neurally meaningful variations in network dynamics and configurations [Cole et al., 2013; Hutchison et al., 2013; Krienen et al., 2014; Shirer et al., 2012]. Since some brain regions are more prone to these effects than others, the instability of the functional MRI signal is nonuniformly distributed across the cortex.

Systematic mis‐estimation of signal correlation can bias many second‐order network measures derived from BOLD signal. Examples include graph theoretical representations of the networks such as the default network (DN) that is implicated in AD and schizophrenia [Achard and Bullmore, 2007; Andrews‐Hanna et al., 2010; Buckner et al., 2008; Liu et al., 2008; Sanz‐Arigita et al., 2010; Stam et al., 2007; Supekar et al., 2008], and the local and distant connectivity that are often examined when studying neural development [Fair et al., 2009] and aging [Rowe et al., 2006]. Some previous studies have implemented strategies to mitigate this bias, e.g., Yeo et al. normalized the connectivity profiles of all brain vertices to unit length prior to clustering the vertices into different networks, so that the clustering was less influenced by the inhomogeneity of signal‐to‐noise ratio (SNR) [Yeo et al., 2011].

Building on these prior explorations we suggest that the connectivity mis‐estimation due to measurement instability should be generally addressed and can be mitigated by applying attenuation correction [Muchinsky, 1996; Spearman, 1910; Zimmerman and Williams, 1997]. Theoretically, correlation coefficients of the BOLD signal between two brain regions can be corrected once the exact measurement reliability of these two regions is available. While the reliability of BOLD signal measurement cannot be directly assessed, we can obtain the connectivity profile of a specific region and compute its test‐retest reliability. This estimate of reliability can be used to construct a correction term for attenuation.

A series of studies have quantified the reliability of fcMRI measures in several functional networks including the default [Meindl et al., 2010; Shehzad et al., 2009; Van Dijk et al., 2010b], attention [Van Dijk et al., 2010b], and motor networks [Mannfolk et al., 2011; Shehzad et al., 2009; Van Dijk et al., 2010b]. However, these prior explorations have mainly focused on pre‐selected networks or regions of interest [Honey et al., 2009; Shehzad et al., 2009; Zuo et al., 2010]. The spatial distribution of functional connectivity reliability on a whole brain level remains to be systematically quantified.

Here we analyzed two independent longitudinal data sets to quantify fcMRI reliability and investigated how reliability‐based attenuation correction might alter connectivity profiles of brain regions, especially those with low tSNR. As a test case, we focused on the medial temporal lobe (MTL) and its coupling to other regions distributed across the default network to demonstrate the effects of reliability‐based attenuation correction. This choice was made for several reasons. First, the MTL is prone to susceptibility artifacts [Ojemann et al., 1997]. Second, the DN including the hippocampal formation is one of the most frequently investigated networks in functional connectivity research with important implications in memory function and Alzheimer's disease [Buckner et al., 2008]. Third, there is a discrepancy between the strong anatomical connections between the posteriomedial cortex (PMC) and MTL components of the DN [Kobayashi and Amaral, 2003, 2007] and their rather weak functional coupling. For example, MTL involvement in the DN was not observed in the first study exploring the network using fcMRI [Greicius et al., 2003] but was subsequently observed in follow‐up analyses where SNR properties were better in the MTL [Greicius et al., 2004]. We suspect that this discrepancy is partially caused by technical constraints of fcMRI, particularly the relatively low reliability in the MTL.

The first objective of this work was to provide a comprehensive picture of reliability of functional connectivity estimates and its influencing factors including scan length, scanner type, and characteristics of the study population such as age and neuropsychiatric state. The second objective was to investigate how the information of reliability can be used to improve the validity of fcMRI measurements. We propose to employ reliability maps as a correction term for disattenuation of functional connectivity estimates. As enrolling the same subjects for repeated scan sessions is not always feasible, we investigated if reliability can be estimated based on data from a single session.

MATERIALS AND METHODS

Participants and Data Collection

Two independent groups of healthy adults were used as discovery and replication data sets. Both datasets were acquired using the same type of scanner and sequence but the two groups differed in ethnicity and age. A third dataset was analyzed to explore whether reliability distribution is influenced by young age, neuropsychiatric disease, and scanner type.

The first dataset included 25 healthy subjects (age 51.8 ± 6.99, nine female, two left handed, all Asian), each of whom underwent five scanning sessions within six months (7, 14, 30, 90, and 180 days from enrollment). These data have been reported previously [Mueller et al., 2013b]. Participants were screened to exclude individuals with a history of neurologic or psychiatric conditions as well as those using psychoactive medications. Participants provided written informed consent in accordance with guidelines set by the institutional review board of Xuanwu Hospital.

All participants performed two or three rest runs per session (6 m 12 s per run) to estimate intrinsic functional connectivity. After quality control, 23 subjects who had at least two good runs [temporal SNR >100, mean relative head displacement <0.05 mm; Van Dijk et al., 2012] in each session were included in this study (mean = 2.02 runs). Temporal SNR of each voxel's time series was estimated by averaging the signal intensity across the whole run and dividing it by the standard deviation over time. All data were acquired on a 3 T TimTrio system (Siemens, Erlangen Germany) using the 12‐channel phased‐array coil supplied by the vendor. Functional data were obtained using a gradient echo‐planar pulse sequence (TR, 3,000 ms; TE, 30 ms; flip angle, 90°; 3 mm isotropic voxels, transverse orientation, 47 slices fully covering the cerebral cortex and cerebellum). Subjects were instructed to stay awake and keep their eyes open; no other task instruction was provided. Structural images were acquired using a sagittal MP‐RAGE three‐dimensional T1‐weighted sequence (TR, 1,600 ms; TE, 2.15 ms; flip angle, 9°; 1.0 mm isotropic voxels; FOV, 256 × 256).

The second dataset consisted of 91 young healthy subjects (age 25.8 ± 14.86, 44 female, nine left handed). The data were collected as part of the Brain Genomics Superstruct Project of Harvard University and the Massachusetts General Hospital [Holmes et al., 2015]. These data are described in prior publications [Buckner et al., 2011; Van Dijk et al., 2012; Yeo et al., 2011]. Each subject underwent two scanning sessions within a time range between 2 days and 379 days (mean = 90 ± 95 days). After quality control, 91 subjects who had at least one good run (temporal SNR >100, mean relative head displacement <0.05 mm) in each session were included in this study (mean = 1.57 runs). Data were acquired on one of three different matched 3 T TimTrio scanners using the vendor‐supplied 12‐channel phased‐array coil. Imaging parameters and inclusion criteria were the same as those for the first data set. Participants provided written informed consent in accordance with guidelines set by the institutional review boards of Harvard University and Partners Healthcare.

The third dataset was employed to explore whether reliability distribution is influenced by young age, neuropsychiatric disease, and scanner type. This dataset was made publically available by the Autism Brain Imaging Data Exchange (ABIDE) Initiative [Di Martino et al., 2013]. To test for potential effects of young age we analyzed data of 26 children (age 10.18 ± 1.77, eight females, all right‐handed) and 26 young adults (age 23.71 ± 4.48, seven females, all right handed) who were scanned on a 3 T Allegra scanner (Siemens, Erlangen Germany) at the Child Study Center of New York University Langone Medical Center. To avoid a potential confound of head motion (which is likely to be higher in children), we selected probands from a larger cohort to ensure that head motion (measured by the mean relative displacement in mm, see [Van Dijk et al., 2012]) was not significantly different across the two groups (mean relative motion children = 0.063 ± 0.023, mean relative motion young adults = 0.067 ± 0.038, P = 0.71). For further information on the ABIDE data sets including scanning protocols please refer to Supporting Information Table S1 and the ABIDE website (http://fcon_1000.projects.nitrc.org/indi/abide/).

To gain insight into whether neuropsychiatric diagnosis might have an impact on the reliability distribution of functional connectivity estimates, 25 subjects with autism were analyzed from the ABIDE sample (age 23.47 ± 6.4, all male, two left‐handed) and their 25 matched controls (age 23.31 ± 6.29, all male, two left‐handed), scanned on a 3 T TimTrio system (Siemens, Erlangen Germany) at the University of Utah. Again, subjects were matched for head motion (mean relative motion autism = 0.079 ± 0.037, mean relative motion controls = 0.075 ± 0.023, P = 0.66).

To explore potential differences in reliability distribution between scanner types, data from 17 children (age 9.96 ± 1.67, three females, all right‐handed) scanned on a 3 T GE Signa scanner (General Electric, Milwaukee, WI) at Stanford University were contrasted with data from 17 children (age 9.95 ± 1.71, two females, all right‐handed) scanned on a 3 T Allegra scanner (Siemens, Erlangen Germany) at the Child Study Center in New York. Again, the two groups were matched for head motion (mean relative displacement GE Signa = 0.077 ± 0.041, mean relative displacement Siemens Allegra = 0.076 ± 0.019, P = 0.96).

Data Preprocessing

Resting‐state fMRI data were processed using procedures [Buckner et al., 2011; Yeo et al., 2011] that were adapted from Fox et al. [2005] and Van Dijk et al. [2010a]. The following steps were performed: (1) slice timing correction (SPM2, Wellcome Department of Cognitive Neurology, London, UK); (2) rigid body correction for head motion with the FSL package [Jenkinson et al., 2002; Smith et al., 2004]; (3) normalization for global mean signal intensity across runs; and (4) low‐pass temporal filtering, head motion regression, and ventricular and white matter signal regression. Whole brain signal regression was also included in the processing stream, which can improve the correction of motion related artifacts [Satterthwaite et al., 2013; Yan et al., 2013]. Structural data were processed using the FreeSurfer version 4.5.0 software package (http://surfer.nmr.mgh.harvard.edu). Surface mesh representations of the cortex from each individual subject's structural images were reconstructed and registered to a common spherical coordinate system [Fischl et al., 1999]. The structural and functional images were aligned using boundary‐based registration [Greve and Fischl, 2009]. The resting‐state BOLD fMRI data were then aligned to the common spherical coordinate system via sampling from the middle of the cortical ribbon in a single interpolation step. See Yeo et al. [2011] for details. In this study, a symmetric surface template of the cerebral cortex was constructed using FreeSurfer [Greve et al., 2013; Wang et al., 2013]. fMRI data of Each individual's fMRI data were registered to this template using resampling with a mesh of 1,284 vertices.

Estimating Functional Reliability

Functional connectivity maps were computed by taking each of the 1,284 vertices on the brain surface as the seed, resulting in 1,284 maps for each subject and session. The functional connectivity map based on a given seed vertex i can be denoted as ${\mathrm{F}}_{\mathrm{i}}\left(s,t\right), \mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e} i=\mathrm{1,2},\dots ,\mathrm{1,284},\mathrm{a}\mathrm{n}\mathrm{d} {\mathrm{F}}_{\mathrm{i}}$is a 1 × 1,284 vector, s indicates the subject, and t indicates the session.

Reliability of connectivity maps based on a particular seed vertex is determined by the signal reliability at the seed vertex as well as the mean reliability of all vertices in the brain. Connectivity reliability was estimated using the maps derived from the five scanning sessions (or two sessions for the second data set) of each subject:

$${\mathrm{R}}_{\mathrm{i}}\left(s\right)=E\hspace{0.17em}\left[\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{r}({\mathrm{F}}_{\mathrm{i}}\left(s,t_m\right),{\mathrm{F}}_{\mathrm{i}}\left(s,t_n\right))\right], \mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e} m,n\hspace{0.17em}=\hspace{0.17em}\mathrm{1,2},\dots ,5; m\ne n$$ (1)

Connectivity reliability was then averaged across subjects (e.g., 23 subjects for the first data set or 91 subjects for the second data set) and assigned to the seed vertex i (see Fig. 1):

$${\mathrm{R}}_{\mathrm{i}}\hspace{0.17em}=\hspace{0.17em}E\left[{\mathrm{R}}_{\mathrm{i}}\left(s\right)\right], \mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e} s\hspace{0.17em}=\hspace{0.17em}\mathrm{1,2},\dots ,23;$$ (2)

Figure 1.

Test–retest reliability of intrinsic functional connectivity is distributed nonuniformly across the cerebral cortex. (A) Test–retest reliability of functional connectivity was quantified in 23 Asian subjects collected in Beijing, China. Reliability was estimated using the five scanning sessions of each subject and then averaged across all subjects. Reliability is highest in the posteromedial cortex, the inferior parietal cortex, and the visual cortex (reliability above the global mean is shown in warm colors). Low reliability is observed in the medial temporal lobe and medial prefrontal cortex, both of which are affected by magnetic susceptibility noise. (B) Test–retest reliability was explored in an independent replication data set (n = 91) of young subjects collected in Boston, USA. Each subject was scanned twice. The overall pattern of reliability distribution was highly similar to the map derived from the initial data set (Spearman rank correlation r = 0.89). [Color figure can be viewed in the online issue, which is available at http://wileyonlinelibrary.com.]

Note that this reliability is the product of measurement reliability values at vertex i and all vertices in the brain instead of the measurement reliability of seed vertex i itself.

Correcting Correlation Attenuation Using Reliability

Resting‐state functional connectivity is estimated based on the correlation coefficient between the BOLD signals extracted from different regions. It has been well established that correlation values are systematically biased downward due to measurement instability, and such bias may be corrected based on the estimate of reliability, a procedure known as disattentuation [Muchinsky, 1996; Spearman, 1904, 1910; Zimmerman and Williams, 1997]. Thus correlation coefficients between two variables can be corrected if the exact measurement reliability of these two variables is available. The corrected correlation between two variables i and j is provided by:

$$\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}\left(i,j\right)\hspace{0.17em}=\hspace{0.17em}\frac{\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}(\widehat{i},\widehat{j})}{\sqrt{M_i}\sqrt{M_j}}$$ (3)

where ${\mathrm{M}}_{\mathrm{i}}$ and ${\mathrm{M}}_{\mathrm{j}}$ represent the measurement reliability of i and j, respectively. $\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}\hspace{0.17em}(\widehat{i},\widehat{j})$is the correlation coefficient before correction. The BOLD signal measurement reliability $M_i$ and $M_y$ cannot be directly assessed but can be estimated from the connectivity reliability $R_i$as shown in Eq. (2). For a given dataset, if we assume the mean signal reliability values across all vertices is a constant $\alpha$, then $R_i\approx \alpha \hspace{0.17em}\times \hspace{0.17em}{M}_i.$ Therefore, the distribution of the connectivity reliability is proportional to the distribution of the measurement reliability.

Correlation values between any two vertices can then be corrected according to Eq. (3). Since $\alpha$ is unknown, before attenuation correction, the entire reliability map R was multiplied by a constant to force the maximal reliability to be 1. This operation avoids over‐correction in regions of low reliability, but at the cost of minimal or no correction for regions of higher reliability. To ensure that only vertices with a meaningful signal were included in the correction analyses, vertices with a mean reliability below a pre‐selected cut‐off (mean reliability <0.1 in the current study) were excluded from further analyses.

Correcting ROI‐Based Correlation Maps

Posterior cingulate cortex (PCC) and inferior parietal lobe (IPL) ROIs were created by projecting the center of predefined volume ROIs [Buckner et al., 2008] to the FreeSurfer spherical surface model and constructing a circle (radius = 8 mm, defined as the arc length on the sphere) around the peak vertex on the sphere. For both ROIs, correlation maps were generated by calculating the correlation coefficients between the seed ROI and all other brain vertices. These correlation maps were corrected for reliability attenuation by applying the correction term to each ROI‐vertex pair. The resulting ROI‐based correlation maps were standardized by z‐score conversion so that maps across participants could be averaged and compared.

Correcting Region‐to‐Region Correlation Strength Within the Default Network

To investigate how attenuation correction affects the connectional architecture of the default network, we conducted a graph analytical representation of the correlation strength among 11 predefined DN ROIs in the brain volume, analogous to Buckner et al. [2008]. Network profiles were derived before and after correcting for correlation attenuation. Correlation strength for each ROI pair was calculated with and without attenuation correction. The correlation matrices of both approaches were then graphically represented using the Pajek software package version 2.04 (http://pajek.imfm.si/). Only significant correlations at P < 0.001 were included in the analysis. The positioning of nodes is based on a spring‐embedding algorithm (Kamada–Kawai algorithm) that positions strongly connected regions close to one another and weakly connected regions farther away while lines depict correlation strengths.

Visualization

All results were visualized using the CARET software [Van Essen, 2005].

RESULTS

Reliability is Distributed Nonuniformly Across the Cortex

Reliability was estimated at each vertex of the brain surface based on the intrasubject test–retest reproducibility of its connectivity profile. In the first data set of 23 subjects, reliability was estimated based on the five scanning sessions of each subject and showed a highly heterogeneous distribution across the brain surface (maximum = 0.69, mean = 0.51) (Fig. 1A). The frontoparietal association regions, the PMC, and the visual cortex displayed especially high reliability. Low reliability regions fell within the temporal lobe (extending into MTL and temporal pole) and medial and orbital prefrontal cortex (Fig. 1A)—regions known to be affected by magnetic susceptibility artifacts.

To replicate this finding we investigated another group of healthy adults imaged under similar technical conditions. Ninety‐one subjects (second data set) underwent two scanning sessions on separate days (mean interval = 90 days). This group was different from the first group in terms of age, but both datasets were collected from a Siemens 3 T TimTrio scanner with the same sequence. The spatial distribution of reliability observed in the first group was preserved in the second group (Fig. 1B and Supporting Information Fig. S1, Spearman rank correlation between the two maps is r = 0.89), although the mean reliability was slightly higher in the second data set (maximum = 0.75, mean = 0.58). These results demonstrate that under similar technical conditions (i.e. same type of scanner and sequence), reliability distribution in healthy adults is stable. BOLD signal measurement reliability can then be estimated using the reliability of connectivity profile (see Methods), and used for the attenuation correction.

When comparing reliability distribution (Fig. 1) to a map of normalized tSNR values (see Supporting Information Fig. S2), a set of brain regions showing relatively high tSNR values but low reliability can be identified. These regions include the pre‐SMA, the midcingulate cortex, and the right dorsal insula. Although the reliability and tSNR maps show a significant correlation (r = 0.61), the discrepancy suggests that reliability is not simply dominated by tSNR. Low reliability in high tSNR regions may be caused by technical noise not explained by tSNR as well as biological variability.

Attenuation Correction Enhances Specific Connections in the Default Network

As reliability of resting‐state functional connectivity is nonuniformly distributed across the brain, its impact on connectivity estimates differs across brain regions. We computed the raw (uncorrected) functional connectivity of DN regions and then disattenuated connectivity estimates by applying reliability as a correction term. Correlation maps based on a PCC seed region were identified before and after reliability correction. The uncorrected correlation map revealed classic default network components such as the medial prefrontal and posteromedial cortex, the inferior parietal lobule, and the lateral temporal lobe (Fig. 2A, top row). After reliability correction, most of these components remained stable, but the correlation strength between the PCC seed region and the MTL increased (Fig. 2B, top row, PCC‐MTL correlation before correction: r = 0.2950, after correction: r = 0.5566, P = 0.0023) The analysis was repeated using a seed region placed in the IPL (Fig. 2, bottom row) and also revealed a connectivity increase between the IPL and MTL after reliability correction (Fig. 2B, bottom row, IPL‐MTL correlation before correction: r = 0.1769, after correction: r = 0.3377, P = 0.0057). This analysis furthermore showed that before correction, the correlation between PCC and IPL (r = 0.5260) was significantly stronger than the correlation between PCC and MTL (r = 0.2950, P = 0.00003), but this difference was almost evened after correction (PCC‐IPL after correction: r = 0.6243, PCC‐MTL after correction: r = 0.5566, P = 0.4129).

Figure 2.

Functional connectivity maps before and after attenuation correction. (A) A functional connectivity map derived from a posterior cingulated cortex (PCC) seed (left panel) and an inferior parietal lobe (IPL) seed (right panel) retrieved classic default network components such as the medial prefrontal and posteromedial cortex, the inferior parietal lobe, and the superior and inferior temporal gyri. (B) After reliability correction, most of these components remained stable, but the correlation strength between the PCC seed and the medial temporal lobe (MTL) increased (left panel). The same analysis performed with a seed placed in the IPL also indicated an increased connectivity between the IPL and the MTL after correction (right panel). (C) The effect of reliability correction is depicted as the difference between corrected and uncorrected maps. [Color figure can be viewed in the online issue, which is available at http://wileyonlinelibrary.com.]

For a more detailed quantification of functional connectivity before and after correction, the correlation strengths among 11 predefined DN seed regions [Buckner et al., 2008] were calculated and visualized using a graph analytical representation. The network architecture derived from uncorrected correlation matrices largely replicated the default network architecture known from previous studies [Andrews‐Hanna et al., 2010; Buckner et al., 2008] with MTL structures forming a circumscribed, slightly distant sub‐system within the default network (Fig. 3A). Analogous to the seed based approach shown in Figure 2, the relative coupling of most DN components remains stable after reliability correction. However, MTL components increased their connectivity strength among each other and to other DN components after attenuation correction. For example, after correction, the MTL components were closer to IPL, vMPFC and PCC in the graph. Although attenuation correction boosted correlation strength of low reliability regions, it has to be noted that it did not change the relative coupling patterns of the DN per se. For example, the dMPFC, a region that showed very weak or even negative correlation to MTL structures before correction, remains uncoupled with the MTL even after correction (Fig. 3B). A graphical depiction of the correlation strength change after correction, as well as significance values of the correlation change for each seed pair, are provided in Supporting Information Figure S3.

Figure 3.

Reliability‐based attenuation increases MTL connectivity within the default network. A graph analytical representation of the correlation strength among 11 predefined default network seeds replicated the previously described DN architecture, where MTL components form a circumscribed and slightly distant subsystem (left panel). Reliability‐based attenuation preserved this relative correlation profile of the DN, but increased the correlation strength of MTL components among each other and to other DN components, especially the PCC, the vMPFC, and the IPL. Notably, regions that showed very weak or even negative correlation to MTL structures before correction, like the dMPFC, remained uncoupled with the MTL even after correction. The table depicts correlation values of each seed pair before and after reliability correction. Asterisks indicate significant increase after correction. [Color figure can be viewed in the online issue, which is available at http://wileyonlinelibrary.com.]

Reliability Distribution can be Estimated From Repeated Runs Within One Session

As repeated sessions are not always feasible, especially in case–control studies, we tested whether repeated resting state runs within the same scanning session can be used as test and retest data to obtain a good approximation of the reliability distribution. The reliability map derived from this “split‐session” approach applied to one of the five sessions of data set 1 showed a high degree of similarity to the original reliability map obtained from five sessions (Spearman rank correlation r = 0.81, P < 0.0001), although the reliability estimated from the single session was slightly higher (maximum = 0.74, mean = 0.57, Fig. 4 and Supporting Information Fig. S1). We then applied the “split‐session” reliability estimates as the correction term to PCC‐MTL, IPL‐MTL, and PCC‐IPL connections (Supporting Information Fig. S4). Analogous to the results described above, the correlation strength between the PCC seed region and the MTL increased significantly (PCC‐MTL correlation before correction: r = 0.2950, after correction: r = 0.4393, P = 0.0370). The analysis was repeated using the IPL seed (Supporting Information Fig. S4, bottom row) and a significant connectivity increase between the IPL and MTL was found after reliability correction (IPL‐MTL correlation before correction: r = 0.1769, after correction: r = 0.2683, P = 0.0477).

Figure 4.

Reliability distribution can be estimated from the data acquired in a single session. The reliability map derived from two runs of the same session (A) shows a highly similar spatial distribution (r = 0.81, P < 0.0001) to the reliability map derived from five sessions (B). This suggests that reliability could be estimated and applied as a correction term even in studies where no repeated sessions were conducted. [Color figure can be viewed in the online issue, which is available at http://wileyonlinelibrary.com.]

Reliability Increases With Acquisition Length While its Spatial Distribution is Relatively Stable

To further investigate the relationship between scan length and reliability estimates, we concatenated all first runs of the five scanning sessions of data set 1 and treated it as a 30‐min test data set. All second runs of the five scanning sessions were concatenated and treated as the 30‐min retest data. We first computed mean reliability (averaged across the whole cortical surface) using 30 min of test data and 30 min of retest data. Reliability was then estimated based on different scan length and its spatial distribution was compared with the reliability map derived from 30 min of scan length (Fig. 5). To avoid that reliability was estimated using data from the same session only, we shuffled the test and retest data separately using 2 min blocks (100 randomizations). We found that reliability increased with scan length, especially between 1 and 15 min of scan length where mean reliability increased from 0.1915 to 0.6101. Beyond 15 min the change of reliability became rather asymptotic with much less increase, e.g., reliability increased from 0.6609 to 0.7042 when scan length increased from 20 to 30 min. However, the spatial distribution of reliability is relatively stable even at short scan lengths. For example, the reliability map derived from 5 min of scan length is similar to the map derived from 30 min (correlation >0.8).

Figure 5.

Global reliability increases with acquisition length while its spatial distribution is relatively stable even at short scan lengths. Data of 23 subjects who underwent a total acquisition time of 1 h was iteratively divided into split halves of 1 to 30 min. Global reliability (averaged across the whole cortical surface) and spatial correlation of reliability distribution were calculated at each scan length. Global reliability values drastically increased with acquisition length (A), especially between 1 to 15 min of scan length (r = 0.1915 and r = 0.6101, respectively). Beyond 15 minutes the increase in absolute reliability becomes asymptotic with much less increase between 20 and 30 minutes for example (r = 0.6609 and r = 0.7042, respectively). The spatial distribution of reliability is very stable even at very short scan length (B). For example, the reliability map derived from 5 min of scan length is similar to the map derived from 30 min (correlation >0.8). [Color figure can be viewed in the online issue, which is available at http://wileyonlinelibrary.com.]

Reliability Distribution is Influenced by Age, Neuropsychiatric Diagnosis, and Scanner Type

The observation that reliability distribution can be roughly estimated based on short scan lengths indicated that it is possible to estimate the reliability distribution in various subject cohorts even if repeated scan sessions are not available. Here we applied the “split‐session” approach to a third dataset to investigate how reliability distribution might vary in subjects of young age, in neuropsychiatric patients, and data collected using different types of scanners. As these data consisted of one run only, this single run was evenly split into two halves. Applying a two‐sided t‐test at a significance level of P < 0.05 (FDR‐corrected for multiple comparisons) we found a different reliability distribution in children (under 13) compared with young adults (17 and over), with children showing lower reliability in the anterior and posterior midline, the lateral prefrontal cortex, and the inferior parietal cortex, but higher reliability in the insular and motor cortex (Fig. 6A). Similarly, reliability differed between probands with autism and their matched controls, with autism subjects showing lower reliability in right medial prefrontal and inferior frontal cortex, and in the left inferior parietal cortex (Fig. 6B). Finally, when comparing reliability maps derived from data collected using a Siemens versus a GE scanner, reliability of Siemens data seemed higher in the left and right midline, whereas reliability of GE data was higher in the insular and sensory‐motor cortex (Fig. 6C). Sample and scanner specific effects coexisted with stable properties of reliability nonuniformity that are presumably caused by constant technical factors such as susceptibility artifacts. For example, the MTL was consistently among the brain regions showing the lowest reliability values in all study groups tested.

Figure 6.

Reliability is influenced by age, neuropsychiatric state, and scanner type. Reliability distribution differs between children and young adults (A), autism subjects and matched controls (B), and between Siemens and GE data (C). Between‐group reliability differences are FDR‐corrected and shown on the cortical surface. These results suggest that different study groups exhibit a specific reliability distribution that should be accounted for. [Color figure can be viewed in the online issue, which is available at http://wileyonlinelibrary.com.]

DISCUSSION

The reliability of functional MRI measurements is nonuniformly distributed across the cortex, which can lead to systematic bias of connectivity estimated from signal correlation. Here we showed that the reliability distribution is relatively consistent in healthy adults collected under similar technical conditions. Investigating different scan lengths we found that global reliability strongly depends on acquisition time, although reliability distribution is stable even at relatively short scan lengths. In general, reliability is particularly low in regions of susceptibility artifact, suggesting that a substantial portion of the reliability variance is likely due to nuisance technical factors. Importantly, reliability estimated from connectivity profile, which has the same spatial distribution as measurement reliability, can be used as an effective correction term to account for the systematic bias caused by measurement instability. Applying reliability‐based attenuation correction to connectivity profiles of the DN revealed that MTL structures might play a more central role in the network than previously considered. To render the approach useful to a greater number of studies, where repeated scans might not always be available, we further demonstrated that the spatial reliability distribution can be estimated based on repeated runs within a single scanning session. Finally, we showed that the spatial reliability distribution is influenced by age and neuropsychiatric diagnosis, suggesting the necessity to account for its impact on functional connectivity estimates when comparing subjects from different cohorts.

Reliability of Resting State fcMRI is Influenced by Multiple Factors

Reliability of fcMRI measurements may be influenced by multiple factors, including imaging‐related artifacts and biological variability. Imaging‐related, technical artifacts are a main factor influencing reliability. The present study demonstrated that, across all data sets, regions showing low reliability were primarily located in the orbitofrontal cortex and in the inferior temporal lobe extending into the MTL. These regions are known to be most affected by susceptibility artifacts due to high magnetic field inhomogeneities [Ojemann et al., 1997], suggesting that the spatial nonuniformity of fcMRI reliability is strongly influenced by this technical artifact. While the stability of the BOLD signal itself cannot be directly measured, there have been previous attempts to quantify the quality of the signal by calculating the signal to noise ratio (SNR) of each voxel's time series by averaging the signal intensity across the whole run and dividing it by the standard deviation over time [Yeo et al., 2011], which is a useful, yet not very precise approximation of SNR. While this tSNR map showed some similarity to our reliability map, there were some marked discrepancies, i.e. regions with high tSNR and yet low reliability, indicating that fcMRI reliability is not only driven by tSNR, but influenced by multiple factors, including technical noise not explained by tSNR and probably some biological variability.

Biological variability can originate from sources such as cardiac and respiratory cycles that ideally would be minimized by adequate data preprocessing [Van Dijk et al., 2010b). The influence of head motion on rs‐fcMRI is complicated and has been vigorously investigated [Power et al., 2014; Satterthwaite et al., 2013; Van Dijk et al., 2012; Yan et al., 2013; Zeng et al., 2014]. Other sources of biological variability include neuronal sources. For example, degree of consciousness [Greicius et al., 2008; Horovitz et al., 2008], cognitive [Waites et al., 2005], emotional state [Harrison et al., 2008], and task [Cole et al., 2013; Krienen et al., 2014; Shirer et al., 2012] can modulate specific interregional functional connections. This variability is juxtaposed with stable functional connectivity properties that presumably reflect neuronal constraints of anatomical connectivity [Lu et al., 2011] and that persist during sleep [Horovitz et al., 2008], light sedation [Greicius et al., 2008], and even anesthesia [Vincent et al., 2007].

With these multiple biological and technical influences, drawing a clear picture of reliability of rs‐fcMRI is challenging. The present study has leveraged two longitudinal datasets to show that the spatial reliability distribution is relatively stable in healthy adult populations under similar technical conditions but is influenced by age, neuropsychiatric diagnosis, and acquisition parameters. Therefore, when comparing children with adults, patients with healthy controls, or data sets collected from different types of scanners or sequences, it may be necessary to consider the potential impact of reliability distribution.

Importantly, reliability distribution can be robustly estimated even at relatively short scan lengths, e.g. using a split‐session approach when repeated sessions are not available. However, while reliability distribution is relatively robust, global reliability values strongly depend on acquisition time. In fact, the relationship is almost linear between 1 and 10 min and only becomes more asymptotic beyond 15 min. In order to obtain a mean reliability of over 0.6 a minimum of 15 min of acquisition time is required.

Reliability can be Used as a Correction Term for Disattenuation

Our results demonstrate that under certain technical conditions, like the same type of scanner and sequence, reliability distribution is a characteristic property of fcMRI that is relatively robust. This predictability means that reliability may be used as the correction term to account for correlation attenuation. Applying this correction emphasizes the prominence of the MTL within the DN, which might have been underestimated by earlier approaches. Connectional anatomy in nonhuman primates indicates that MTL structures have both afferent and efferent projections to specific posteriomedial, temporal, parietal, and medial frontal regions that are estimated to be components of the DN [Kobayashi and Amaral, 2003, 2007; Lavenex et al., 2004]. However, functional connectivity studies repeatedly find that coupling between the MTL and cortical DN regions is relatively weak as compared with corticocortical coupling, although regions within the DN are the most strongly coupled regions when the MTL itself is the initiating point for fcMRI analysis [Kahn et al., 2008; Vincent et al., 2006]. Conventional graph theoretical representations of the DN have suggested that MTL structures form a separated sub‐unit with rather weak connections to certain other DN nodes, at least insofar as its relative coupling strength compared with cortical regions of the DN [Andrews‐Hanna et al., 2010; Buckner et al., 2008]. This discrepancy is likely to be due to technical constraints of fcMRI. Our results suggest that after reliability‐based attenuation correction these MTL components show stronger connectivity to other DN components like IPL, vMPFC, and PCC—a profile more consistent with our knowledge about its anatomical connections [Kobayashi and Amaral, 2003, 2007]. Notably, DN components without known anatomical connections to the MTL, such as the dMPFC, remain uncoupled even after correction.

Importance

In theory there are several scenarios how the reliability bias might affect study results. First, connectivity of low reliability brain regions like the MTL and MPFC (see above) might be generally underestimated, leading to smaller effect sizes in case–control or longitudinal studies. Second, group comparisons of connectivity might be distorted because patient and control groups are likely to have different reliability maps. Our data indicate that some brain regions like the MTL and MPFC show low reliability across data sets and scanners, but other regions like the PMC and the insular cortex show variable reliability depending on the age and neuropsychiatric diagnosis of the specific study sample, and the acquisition procedure used. Therefore it is advisable to obtain group‐specific reliability maps, to gain an impression of whether the underlying reliability distributions differ. If reliability distributions substantially differ between study groups, and/or the ROIs to be investigated fall within regions of low reliably, disattenuation correction might improve results. The correction‐related changes in DN architecture indicate that the relative correlation strength of low reliability regions like the MTL and MPFC might have been systematically underestimated by conventional approaches and might benefit from attenuation correction. The procedure will likely similarly influence results of other procedures applied to fcMRI data with spatially varying reliability [Beckmann et al., 2005; Spreng et al., 2013; Stevens and Spreng 2014; Wig et al., 2014]. The fact that reliability can be estimated in repeated resting state runs of a single scanning session will increase the feasibility of reliability‐based attenuation correction in future studies.

Clinical Relevance

These results suggest that reliability‐based attenuation correction might enable a more accurate interpretation of BOLD‐derived MRI measures, including functional connectivity and BOLD activation, especially in regions of low reliability. This is especially important as major regions of interest in neuropsychiatric disorders fall within brain areas that consistently display low reliability across data sets and scanners, like the MTL and MPFC/orbitofronatal cortex. MTL connectivity for example plays a crucial role in memory research, where hippocampal connectivity to the PMC predicts memory performance in healthy individuals [Wang et al., 2010] and the disruption of connectivity between these regions has been observed in patients with MCI and mild AD [Greicius et al., 2004; Wang et al., 2006]. Other implications of MTL connectivity have been demonstrated in autism spectrum disorders (ASD), where altered MTL‐PMC connectivity has been linked to social symptom severity [Lynch et al., 2013; Monk et al., 2009; Weng et al., 2010], and in schizophrenia where MTL‐PMC connectivity is also reduced [Zhou et al., 2008]. MPFC connectivity also has important implications in autism research, where reduced MPFC connectivity within the DN is one of the most robust and replicable findings [Assaf et al., 2010; Kennedy and Courchesne, 2008; Mueller et al., 2013a; von dem Hagen et al., 2013; Weng et al., 2010] which is associated with genetic risk for ASD [Rudie et al., 2012] and typical symptoms of the disorder [Assaf et al., 2010; Weng et al., 2010]. In addition, intrinsic MPFC connectivity plays an important role in schizophrenia research where MPFC connectivity within the DN is increased in individuals with high genetic risk for schizophrenia [Whitfield‐Gabrieli et al., 2009] and in individuals at ultra‐high risk of becoming psychotic [Shim et al., 2010], but decreased in patients with chronic schizophrenia [Camchong et al., 2011; Öngür et al., 2010]. Our data indicate that reliability differs between patient and healthy groups and a reliability‐based correction may improve the specificity of many clinical findings.

Relation Between Reliability and Interindividual Differences

Reliability, defined as the test–retest reproducibility within a subject, is an important prerequisite for the successful exploration of individual differences in connectivity architecture. Unless an imaging measure is highly reproducible within a subject the variance observed across individuals cannot be fully attributed to individual differences as the observed intersubject variance involves both the true individual difference as well as the measurement instability. To determine brain regions that exhibit high sensitivity to discover true interindividual differences we defined variability‐based SNR (vSNR) as the ratio between interindividual variability and intra‐individual variability (defined as 1‐reliability). This vSNR could serve as a measure of sensitivity to interindividual differences in a certain brain region. Brain regions more sensitive to individual differences than intra‐subject variance will offer greater power in finding associations between imaging markers and behavioral or genetic variants. vSNR at the vertex level is depicted in Supporting Information Figure S5.

Limitations

When applying reliability correction there are several limitations one needs to be aware of. The first limitation pertains to the issue of “over‐correction.” In our approach we apply a scaling factor that forces the maximal reliability to be one and scales up low reliability regions accordingly. This procedure avoids overcorrection at the cost of minimal or no correction for regions of higher reliability. However, extremely low variability values might not be scaled up sufficiently by this scaling factor approach. This might for example be the case for scans with very short acquisition time where signal quality and therefore reliability are insufficient. In our study we addressed this potential problem by introducing a cut‐off of minimal reliability of 0.1 for each vertex at the group level. The rationale behind this cutoff is that vertices with a group‐level reliability below 0.1 probably do not provide enough meaningful signal to be included in connectivity analyses. In our case, we did not have to exclude any vertices based on this cut‐off in any of the described study samples.

A second limitation pertains to potential under‐correction. If reliability is estimated from a single session, i.e., either from repeated runs within one session or from a split single run, it is typically higher than reliability estimated from real test retest data acquired from different sessions. To visualize this effect of over‐estimation of reliability of “neighboring” test–retest data, we plotted reliability over time for real test‐retest reliability as compared with split session reliability (Supporting Information Fig. S6). Especially in short data, the split session approach leads to an over‐estimation of reliability, which in turn results in potential under‐correction. This phenomenon of a weaker correction effect when applying split‐session reliability maps as the correction term can be seen in our results, where for example MTL‐PCC connectivity strength almost doubled after normal reliability correction, but only increased by about 50% when applying the split‐session reliability as the correction term.

Another limitation of the current study is that it is restrained to functional connectivity estimations based on correlation coefficients, which is one of the most broadly employed methods. There are however various other ways of quantifying functional connectivity (for a comprehensive review see [Smith et al., 2011]), that are not taken into account here. It is also important to recognize that the present notion of functional connectivity as implemented with a single correlation value is a static measure that only constitutes a statistical estimate of the likelihood that two regions co‐vary during the time window being investigated. However, functional correlations measured using short time windows suggest the relations are temporally dynamic [Hutchison et al., 2013; Liu and Duyn, 2013], meaning that the resting‐state of the human brain is not a single static state but consists of multiple states that dynamically emerge and dissolve [Hutchison et al., 2013; Liu and Duyn, 2013]. It is presently unclear to what degree temporally evolving correlation patterns reflect shifts in neuronal configurations or dynamic sensitivity to confounding factors. The present notion of reliability, as implemented with a static correlation analysis, implicitly defines reliability to be the reproducibility of the totality of all possible states exhibited by the brain. In other words, the present notion cannot be meaningfully applied to individual states but only to the full set of these states. While it might improve static correlation measures, and may therefore provide a more accurate estimation of the static functional architecture of the brain, it similarly sacrifices the dynamic characteristics of neural activity, that itself may carry biologically meaningful information about the individual [Grady and Garrett, 2014]. However, in a setting where the static estimate of functional connectivity is targeted and related to other static traits such as diagnosis or genotype, attenuation correction may improve the association analyses. In the future it might however be interesting to also investigate the intrasubject variability of whole brain dynamics, e.g. the frequency of appearance of certain states, using advanced methods like neural‐mass modeling of whole‐brain dynamics [Deco et al., 2008].

CONCLUSION

The present study demonstrated that reliability is heterogeneously distributed across the cortex, which causes connectivity estimates in different brain regions to be differently biased. The reliability distribution is stable in similar subject groups collected under similar technical condition, but differs in young children, in patients and varies across different scanners. The bias can be accounted for by applying reliability‐based attenuation correction. Using group‐specific reliability maps as a correction term might therefore be a valuable approach to improve the validity of fMRI estimates and their neurophysiological interpretation. When repeated scans are not available, reliability distribution may be conveniently estimated from split data of a single session, as reliability distribution is relatively stable even at short scan length. Absolute reliability values, however, strongly depend on acquisition length.

Supporting information

Supporting Information