The effect of resting condition on resting-state fMRI reliability and consistency: A comparison between resting with eyes open, closed, and fixated

Abstract

Resting-state fMRI (rs-fMRI) has been demonstrated to have moderate to high reliability and produces consistent patterns of connectivity across a wide variety of subjects, sites, and scanners. However, there is no one agreed upon method to acquire rs-fMRI data. Some sites instruct their subjects, or patients, to lie still with their eyes closed, while other sites instruct their subjects to keep their eyes open or even fixating on a cross during scanning. Several studies have compared those three resting conditions based on connectivity strength. In our study, we assess differences in metrics of test–retest reliability (using an intraclass correlation coefficient), and consistency of the rank-order of connections within a subject and the ranks of subjects for a particular connection from one session to another (using Kendall's W tests). Twenty-five healthy subjects were scanned at three different time points for each resting condition, twice the same day and another time two to three months later. Resting-state functional connectivity measures were evaluated in motor, visual, auditory, attention, and default-mode networks, and compared between the different resting conditions. Of the networks examined, only the auditory network resulted in significantly higher connectivity in the eyes closed condition compared to the other two conditions. No significant between-condition differences in connectivity strength were found in default mode, attention, visual, and motor networks. Overall, the differences in reliability and consistency between different resting conditions were relatively small in effect size but results were found to be significant. Across all within-network connections, and within default-mode, attention, and auditory networks statistically significant greater reliability was found when the subjects were lying with their eyes fixated on a cross. In contrast, primary visual network connectivity was most reliable when subjects had their eyes open (and not fixating on a cross).

Introduction

Resting-state functional Magnetic Resonance Imaging (rs-fMRI) can provide an estimate of functional connectivity by computing the correlation of low-frequency (<0.1 Hz) blood-oxygen-level-dependent (BOLD) signal fluctuations in the resting brain, to infer neuronal activity, between regions of interest (Biswal et al., 1995). Fluctuations during this resting-state are thought to represent both spontaneous neuronal activation as well as unconstrained mental activity, such as “day dreaming” or “mind wandering” (Fox et al., 2005; Mason et al., 2007). Although this unconstrained mental activity may be affected by the resting conditions, the instructions given to subjects vary widely across studies. The most common instructions are to rest with either eyes closed, eyes open, or eyes fixating on a cross. Inconsistencies in the resting condition across studies could create difficulties replicating results from one research site to the next. In addition, it is unclear which of these resting conditions provide the most reliable estimates of functional connectivity. The application of resting state functional connectivity using MRI (rs-fcMRI) to clinical studies and the investigation of individual differences in functional connectivity critically depend not only on a consistent spatial pattern of connectivity but also on a repeatable measure of the strength of connectivity. The goal of our current study is to compare the reliability of functional connectivity estimates, as well as the consistency of those estimates, derived from three different resting conditions: eyes closed (EC), eyes open (EO), and eyes fixating (EF).

Several previous studies have examined differences in functional connectivity across these three resting conditions. Feige et al. demonstrated the average BOLD signal intensity in visual processing areas to be higher for EO when compared to EC (Feige et al., 2005); and a study by Yan et al. reported that although the functional connectivity maps of the default mode network (DMN, see Raichle et al., 2001 for details on DMN) appear similar across different resting conditions, the strength of connectivity was lower for EC when compared to EF (Yan et al., 2009). Similarly, a study by Van Dijk et al. found lower correlations within the default mode and attention networks for EC when compared to both EF and EO (Van Dijk et al., 2010). Other studies have found the amplitude of low frequency fluctuation to be greater for EC when compared to both EO and EF as well as that the BOLD signal power is greater for EC (McAvoy et al., 2008; Yan et al., 2009).

One potential source for these differences is that the physiological state of the subject (breathing and cardiac cycles) depends on the resting state condition, and that this, in turn, causes differences in the rs-fMRI data. However, it has been shown that the resting condition has no significant effect on those physiological cycles (McAvoy et al., 2008). Rather, the observed differences were hypothesized to arise from discrepancies in BOLD oscillations depending upon the resting condition itself (McAvoy et al., 2008). This could be due to changes in the alpha rhythm, the most prominent rhythmic electrical potential fluctuation, which is greater when the subject has his/her eyes closed compared to when the eyes are opened (Berger, 1929, 1930). Prior studies have shown that the amplitude of alpha rhythms is positively correlated to the BOLD signal (Goldman et al., 2002; Moosmann et al., 2003), although, it should be noted that Laufs et al. failed to reproduce this result (Laufs et al., 2003). The differences in functional connectivity during EO and EC conditions could also originate from the possibility that while subjects have their eyes open the increase in visual information gathered and evaluated may contribute to an increase in mind wandering or daydreaming, thus making the activity within the DMN more coherent, a possibility suggested by Yan et al. (2009).

While this prior research provides important insights into the effect of different resting conditions, it does not indicate which of these conditions provide the most reliable estimates of functional connectivity. We therefore compare not only the strength of functional connectivity but also examine the test–retest reliability of functional connectivity and its consistency under the three different resting conditions.

Materials and methods

In this study, we use three different measures to assess reliability, consistency, and overlap of repeated estimates of connectivity matrices and maps. The intraclass correlation coefficient compares within- to between-subject variance and was used to assess test–retest reliability. The Kendall's W is also a measure of repeatability, but compares the rankings of subjects or connections between sessions. To distinguish this from test–retest reliability measured with the ICC, we refer to this term as the consistency. These terms are defined in greater detail below.

Participants and fMRI data acquisition

Written informed consent was obtained from subjects prior to each scanning session in accordance with a University of Wisconsin-Madison IRB approved protocol. Twenty-five healthy adults (10 females; 35.5 ± 17.7 years of age on average) with no history of neurological or psychological disorders were scanned three times for each of the three resting conditions: EC, EO, and EF. Six of these nine resting-state scans (i.e., two scans for each of the three conditions) were acquired on the same day whereas the remaining three scans were acquired an average of three months later. The order of the scans was counterbalanced across scanning sessions and subjects to ensure that differences observed were due to differences in resting conditions and not scan order. All the scans were acquired using a 3 T GE scanner (MR750, General Electric Medical Systems, Waukesha, WI). Each functional scan was 10 min in length and acquired with the same echo planar imaging (EPI) sequence (TR = 2.6 s, TE = 25 ms, flip angle = 60°, FOV = 224 mm × 224 mm, matrix size = 64 × 64, slice thickness = 3.5 mm, number of slices = 40). Subjects were instructed to relax and lie still in the scanner while remaining calm and awake. For each resting condition, subjects were additionally instructed to either keep their eyes closed (EC), to keep their eyes open (EO), or to fixate their gaze on a cross (EF) back-projected onto a screen via an LCD projector (Avotec, Inc., Stuart, FL). For both the EO and EF conditions, subjects were allowed to blink if necessary. T1-weighted structural images were acquired before the functional images using an MPRAGE sequence with the following parameters: TR = 8.13 ms, TE = 3.18 ms, flip angle = 12°, FOV = 256 mm × 256 mm, matrix size = 256 × 256, slice thickness = 1 mm, and number of slices = 156.

Data preprocessing

All functional data was processed in AFNI (Cox, 1996). Preprocessing steps included rigid-body volume registration to reduce the influence of subject motion, implementation of RETROICOR to reduce the physiological noise introduced by the respiration and cardiac cycles (Glover et al., 2000), and slice timing correction. EPI time series were aligned to their T1-weighted anatomical (Saad et al., 2009). The anatomical scan was then skull stripped and segmented into white matter (WM) and cerebrospinal fluid (CSF) masks using FSL's FAST routine (Smith et al., 2004; Woolrich et al., 2009; Zhang et al., 2001). The average signal time course from the eroded WM and CSF masks (2 and 1 voxels in each direction, respectively), together with their first derivatives and the 6 rigid body motion parameters were used as covariates of no interest and regressed out of the EPI time series data. The residuals were then transformed to Talairach Atlas Space (Talairach and Tournoux, 1988) and resampled to 3 × 3 × 3 mm³ voxels. Global signal regression was not performed because it has been observed that this processing step can induce negative correlations and affect group results (Saad et al., 2012). One reason why this happens is that global signal, by definition, includes signal from regions of interest (i.e. gray matter). In order to ensure that the signal of interest is not included (or to minimize its contribution) in our regressors of no interest, we eroded the white matter mask by two voxels and the CSF mask by one voxel in every direction. This reduced the probability of getting voxels containing heterogeneous tissue (i.e. WM and GM or CSF and GM) to only keep voxels containing only WM or CSF tissue (get rid of or minimize partial voluming effects from the segmentation). Finally, EPI time series were temporally filtered (band-pass: 0.001 Hz < f < 0.01 Hz) and spatially smoothed with a 3D Gaussian kernel (FWHM = 4 mm).

Motion

Resting-state functional connectivity measures can be significantly affected by subject motion (Power et al., 2012; Satterthwaite et al., 2012; Van Dijk et al., 2012). We therefore assessed whether there were significant differences in subject motion across the three resting conditions. We examined both the volume-to-volume displacement calculating translation and rotation separately (Power et al., 2012; Satterthwaite et al., 2012; Van Dijk et al., 2012) as well as combined (similar to Power et al., 2012, but combining the sum of the squares rather than the sum of the absolute values). Specifically, the formula used for the volume-to-volume translational motion is:

$$\text{MRD}=\sqrt{{({\mathrm{x}}_{\mathrm{j}}-{\mathrm{x}}_{\mathrm{i}})}^2+{({\mathrm{y}}_{\mathrm{j}}-{\mathrm{y}}_{\mathrm{i}})}^2+{({\mathrm{z}}_{\mathrm{j}}-{\mathrm{z}}_{\mathrm{i}})}^2}$$

while the rotational motion was calculated using the Euler angle formula and then averaging the absolute value of this angle across subjects (Van Dijk et al., 2012) and scans for each resting condition:

$$\text{EulerAngle}={cos}^{-1}\left(\right(cos\left(\mathrm{\alpha }\right)cos\left(\mathrm{\beta }\right)+cos\left(\mathrm{\alpha }\right)cos\left(\mathrm{\gamma }\right)+cos\left(\mathrm{\beta }\right)cos\left(\mathrm{\gamma }\right)+sin\left(\mathrm{\alpha }\right)sin\left(\mathrm{\beta }\right)sin\left(\mathrm{\gamma }\right)-1)/2)$$

The formula used for the sum of squared differences (SSD) measurements combining both translation and rotation is:

$$\text{SSD}=\sqrt{{({\mathrm{x}}_{\mathrm{j}}-{\mathrm{x}}_{\mathrm{i}})}^2+{({\mathrm{y}}_{\mathrm{j}}-{\mathrm{y}}_{\mathrm{i}})}^2+{({\mathrm{z}}_{\mathrm{j}}-{\mathrm{z}}_{\mathrm{i}})}^2+{({\mathrm{\alpha }}_{\mathrm{j}}-{\mathrm{\alpha }}_{\mathrm{i}})}^2+{({\mathrm{\beta }}_{\mathrm{j}}-{\mathrm{\beta }}_{\mathrm{i}})}^2+{({\mathrm{\gamma }}_{\mathrm{j}}-{\mathrm{\gamma }}_{\mathrm{i}})}^2.}$$

where i and j are two consecutive time points (epi volumes), and x, y, z, α, β, and γ are the six parameters of motion (3 translations and 3 rotations). Rotations were converted from degrees to millimeters by computing the displacement of a point on a sphere of 57 mm radius, which is approximately the distance from the center of the brain to the cortex (Jones et al., 2008; Kennedy and Courchesne, 2008). Time points were censored out whenever the SSD estimates exceeded 0.2 mm (Power et al., 2012). Here, censored out refers to the process of ignoring time points in the correlation calculations for which the SSD was larger than or equal to 0.2 mm.

A two way analysis of variance (ANOVA) was run for each of the methods of estimating motion, also including the number of time points censored out. The two ways correspond to subjects and resting conditions. No significant differences were observed between the three resting conditions (more detail on the statistics is provided in the Results section) and the means across subjects along with the corresponding standard deviations are presented in Table 1.

Table 1.

Average motion estimation across subjects using the mean sum of squared differences (SSD) of realignment parameters, the relative displacement (MRD), and the Euler angle of rotation for each resting condition. The last column includes the number of time points removed (average ± standard deviation and the corresponding range % of time point censored).

Resting condition	MRD (mm)	Euler angle (degree)	SSD (mm)	RMS (mm)	Total censored points (% of time points removed)
EO	0.053 ± 0.028	0.0094 ± 0.009	0.068 ± 0.034	0.25 ± 0.11	7.8 ± 13.8 (0–9.5%)
EC	0.053 ± 0.029	0.0086 ± 0.0088	0.065 ± 0.035	0.24 ± 0.12	11.3 ± 15.5 (0–11.8%)
EF	0.052 ± 0.030	0.012 ± 0.014	0.069 ± 0.037	0.26 ± 0.12	9.3 ± 12.6 (0–9.6%)

Functional connectivity: ROI based analysis

Functional connectivity measures were computed using a region of interest (ROI) based approach (Biswal et al., 1995). Eighteen ROIs, or seeds, from the default mode, dorsal attention, motor, visual, and auditory networks with radiuses of 4 mm were generated based on the coordinates provided in the 2010 study by Van Dijk et al. (for more information on these seeds see (Buckner et al., 2008; Corbetta and Shulman, 2002; Fox et al., 2005, 2006; Fransson and Marrelec, 2008; Greicius et al., 2003; Raichle et al., 2001). Briefly, the default mode network seeds include the posterior cingulate (PC), the medial prefrontal cortex (mPFC), the left and right lateral parietal cortices (LatPar), as well as the left and right hippocampal formations (HF). The dorsal attention network seeds include the left and right frontal eye fields (FEF), the left and right medial temporal areas (MT), as well as the left and right intraparietal sulci (IPS). The auditory network seeds include the left and right primary auditory cortices (Aud); the motor network seeds include the left and right primary motor cortices (Mot); and the visual network seeds include the left and right primary visual cortices (Vis).

The pre-processed resting-state fMRI signal was averaged over each of these seeds. Functional connectivity matrices were generated for each of the subject's nine functional scans by computing regressions pair-wise over all of the seeds. The Pearson correlation coefficient from each regression was converted into a more normally distributed variable via the Fisher Z transform (Zar, 1996) to create an 18 × 18 matrix of connectivity strength (Z-scores) of which the upper right triangle represents the 153 unique connections. The eighteen average time-courses were also used in a voxel-wise ordinary least squares regression (AFNI's 3dDeconvolve) to generate connectivity maps. The Pearson correlation coefficients in the resulting statistical parametric maps were converted to normally distributed variables via the use of the Fisher Z transform (Zar, 1996). These maps were used to visualize the connectivity networks for each seed and to assess differences between resting conditions (using a paired T-test of the difference).

Test–retest reliability

We define reliability as the ability of a method, here rs-fMRI, to reproduce results for a subject such that differences observed between subjects are stable. In other words, a reliable measure requires a small within-subject variance compared to the between-subject variance. The test–retest reliability of fMRI activation and connectivity has previously been assessed using the Intraclass Correlation Coefficient (ICC) introduced by Shrout and Fleiss in 1979 (Braun et al., 2012; Li et al., 2012; Shehzad et al., 2009; Thomason et al., 2011; Zuo et al., 2010). A higher ICC reflects greater reliability while a lower ICC reflects lower reliability. For each of the 153 unique connections, an n × k matrix of connectivity strength was created with rows as subjects (n = 25) and columns as scans (k = 2). ICCs were calculated for matrices with “intra-session” scans (i.e., scans 1 and 2, which were obtained during the same scanning session) as well as with “inter-session” scans (i.e., scans 1 and 3 and scans 2 and 3, which were obtained during different scanning sessions). Let MSb be the mean square between subjects and MSw the mean square within-subject, then the ICC based on single measurements using a random effects model is defined by:

$$\text{ICC}=\frac{\text{MSb}-\text{MSw}}{\text{MSb}-(\mathrm{k}-1)\text{MSw}}.$$

A measure of reliability (the ICC) is obtained for each connection.

Shehzad et al. found significant differences in ICCs of functional connectivity depending on the significance of the connection (Shehzad et al., 2009). In theory, the connectivity between two unconnected regions should be approximately zero in all subjects and in all scans with the only variations being due to noise. This would result in a between subject variance that is approximately equal to the within-subject variance and thus an ICC of zero. Consequently, a non-zero ICC for a non-significant connection suggests there exists consistent noise within a subject that is reliably different from another subject. While it is unlikely that two areas are completely unconnected (e.g. see Gonzalez-Castillo et al., 2012), the ICC of weakly connected regions are more heavily influenced by noise. Consequently, we examine the ICCs of five established networks with known significant connections: the default mode, dorsal attention, auditory, motor, and visual network connections. Differences in ICCs between the different conditions were assessed using jackknife resampling assessed at a significance level of p < 0.05 (see below). The ICC calculations were carried out using MATLAB (2010b, The MathWorks, Natick, Massachusetts).

Consistency of connectivity strength for connections and subjects

The consistency of the rank order of connection strengths has previously been assessed using Kendall's coefficient of concordance, or W, which ranges from 0 being no agreement to 1 being complete agreement (Guo et al., 2011; Kendall and Gibbons, 1990; Kendall and Smith, 1939; Legendre, 2005; Li et al., 2012; Shehzad et al., 2009; Thomason et al., 2011; Zang et al., 2004). The Kendall's W was calculated for both an intra-individual case (i.e., ranking different connections by strength of connectivity and comparing across scans for each subject), as well as an inter-individual case (i.e., ranking different subjects by strength of connectivity and comparing across scans for each connection). More specifically, the intra-individual Kendall's W was computed for each subject's n × k matrix of connectivity strength, where rows are connections (n = 153), columns are scans (k = 2), and ranking occurs within columns. The inter-individual Kendall's W was computed for each connection's n × k matrix, where rows are subjects (n = 25), columns are scans (k = 2), and ranking occurs within columns. Both intra-individual and inter-individual Kendall's W were calculated on matrices with “intra-session” scans (i.e., scans 1 and 2) as well as with “inter-session” scans (i.e., scans 1 and 3 and scans 2 and 3) according to the equation:

$$\mathrm{W}=\frac{12\times \sum _{\mathrm{i}=1}^{\mathrm{n}}{({\mathrm{R}}_{\mathrm{i}}-\overline{\mathrm{R}})}^2}{({\mathrm{k}}^2\times ({\mathrm{n}}^3-\mathrm{n}))-(\mathrm{k}\times \mathrm{T})}$$

where T is the tie correction factor (Legendre, 2005) given by

$$\mathrm{T}=\sum _{\mathrm{k}=1}^{\mathrm{m}}({\mathrm{t}}_{\mathrm{k}}^3-{\mathrm{t}}_{\mathrm{k}})$$

where m is the number of unique rank values and t is the number of ties for each unique rank.

We consider all the connections for the intra-individual Kendall's W because we are trying to determine the consistency of rankings for the upper right triangle of the connectivity matrix for different scans of the same subject. In other words, this metric would assess, for instance, whether the connectivity between the left and right motor cortices is consistently different than the connectivity between the left motor cortex and the left visual cortex across scans for a given subject. For the inter-individual Kendall's W, we once again focus on within-network connections, as non-significant connections should be null (i.e., Z = 0) making the ranking of such connections a characterization of noise.

Statistical analysis

A Jackknife resampling method was used for determining the variance of the ICC and Kendall's W for each resting condition in an unbiased fashion. In this leave-one out statistical resampling method introduced by Quenouille and furthered by Tukey (Quenouille, 1949; Tukey, 1958), the ICC and intersubject Kendall's W were each recomputed 25 times, once for leaving out each subject. The implementation is justified for our study since it assumes independence in the variable (here subject—coming from a random sample). These values were directly compared across resting conditions, which created a distribution of differences. This distribution was used to determine whether differences between two resting conditions were significant at a 0.05 significance level. This was only used on the ICC and the inter-individual Kendall's W, because only a random variable (e.g., subjects) should be left out. Connection type (the variable in the intra-individual Kendall's W) does not satisfy the assumption of independence.

Results

Functional connectivity

We generated connectivity maps from 18 different seeds for each subject and for each of the three different resting state conditions. Fig. 1 displays the group connectivity maps for the left primary visual cortex seed (lVis) for each condition (p-threshold of 1 * 10⁻¹⁵). A T-test map was generated to assess the differences. Connectivity between the visual seeds was significantly modulated by the resting condition. The EC condition resulted in greater negative connectivity between the left visual cortex and both bilateral thalami and anterior cingulate cortex, and greater positive connectivity between the left visual cortex and higher-order visual cortical regions or visual association areas, superior parietal cortex, and supramarginal gyrus. The blue voxels from Fig. 1 are located close to the edge of the bilateral thalamic cluster. Fig. 2 shows the connectivity maps for each resting condition for the lIPS seed. Overall, the maps are similar across resting conditions showing the same pattern of connectivity (significant connectivity to the right IPS as well as the precentral frontal sulcus); however, a t-test revealed significantly greater connectivity within the dorsal attention network during the EC condition. Fig. 3 displays the connectivity maps of the PC seed, representing the DMN. Functional connectivity maps are highly similar for all conditions, with the PC showing significant connectivity to medial prefrontal cortex and bilateral parietal areas. A t-test revealed no significant differences in functional connectivity of the PC seed among these regions of the default mode network for different resting conditions. However, significantly greater negative connectivity was found between the PC and bilateral insula for the EF condition compared to EC.

Fig. 1.

Connectivity maps of the left visual seed for each resting state condition. The maps are thresholded at a p-value of 1 * 10⁻¹⁵.

Fig. 2.

Connectivity maps of the left intraparietal sulcus seed (showing the dorsal attention network) for each resting state condition. The maps are thresholded at a p-value of 1 * 10⁻¹⁵.

Fig. 3.

Connectivity maps of the posterior cingulate seed (showing the default mode network) for each resting state condition. The maps are thresholded at a p-value of 1 * 10⁻¹⁵.

Table 2 shows the mean connectivity strength for the connections in each of the different networks studied here as well as combined within-network connections and all connections. A 5 × 3 ANOVA (networks × resting condition) on all connections examined here found a significant effect of network, but no significant effect of resting condition. Paired t-tests comparing the connectivity strength for EO, EC, and EF conditions found significantly greater connectivity strength in the auditory cortex for the EC compared to the EF condition (p < 0.00012 for scans 1 and 2, p < 0.009 for scans 1, 2 and 3). No significant differences were found between conditions in the motor, visual, default mode, and attention networks after correcting for multiple comparisons.

Table 2.

Average connectivity strength (Fisher Z-score) for each network studied for each resting condition and each session. For the mean Z, values in bold indicate that they are the highest of the three resting conditions. Significance testing was done using a two-tailed paired t-test to assess which condition was significantly higher than the other conditions between the different resting conditions for each network studied here. Cells in bold indicate significant differences after multiple comparison correction. For comparisons that were significantly different, the condition that resulted in the higher reliability is indicated in the parentheses.

	Condition	Mean Z	Comparison	P-value
All within-network
Intrasession	EO	0.24	EO vs. EC	0.67
	EC	0.24	EO vs. EF	0.23
	EF	0.24	EC vs. EF	0.2
Intersession	EO	0.21	EO vs. EC	0.67
	EC	0.21	EO vs EF	0.0374 (EF)
	EF	0.22	EC vs. EF	0.0313 (EF)
Attention
Intrasession	EO	0.19	EO vs. EC	0.51
	EC	0.18	EO vs. EF	0.59
	EF	0.19	EC vs. EF	0.88
Intersession	EO	0.16	EO vs. EC	0.21
	EC	0.16	EO vs EF	0.89
	EF	0.17	EC vs. EF	0.36
DMN
Intrasession	EO	0.21	EO vs. EC	0.92
	EC	0.21	EO vs. EF	0.59
Intersession	EF	0.25	EC vs. EF	0.53
	EO	0.17	EO vs. EC	0.85
	EC	0.17	EO vs EF	0.032 (EF)
	EF	0.21	EC vs. EF	0.047 (EF)
Auditory
Intrasession	EO	0.51	EO vs. EC	0.046 (EC)
	EC	0.58	EO vs. EF	0.032 (EF)
	EF	0.43	EC vs. EF	0.00012 (EC)
Intersession	EO	0.54	EO vs. EC	0.093
	EC	0.58	EO vs EF	0.22
	EF	0.49	EC vs. EF	0.009 (EC)
Motor
Intrasession	EO	0.4	EO vs. EC	0.7
	EC	0.41	EO vs. EF	0.12
	EF	0.35	EC vs. EF	0.053
Intersession	EO	0.39	EO vs. EC	0.5
	EC	0.39	EO vs EF	0.99
	EF	0.37	EC vs. EF	0.58
Visual
Intrasession	EO	0.81	EO vs. EC	0.44
	EC	0.86	EO vs. EF	0.14
Intersession	EF	0.73	EC vs. EF	0.025 (EC)
	EO	0.86	EO vs. EC	0.17
	EC	0.88	EO vs EF	0.34
	EF	0.78	EC vs. EF	0.039 (EC)

Test–retest reliability

Fig. 4 shows the relationship between the ICC and the mean connectivity strength (Fisher-Z transformed correlation coefficient) for all 190 ROI pairs (connections). The connections with the highest connectivity strength have a high reliability (between approximately 0.5 and 0.9 for the intrasession and 0.1 and 0.75 for the intersession ICC). However, the converse is not necessarily true. Areas with weaker connectivity have a wide range of ICCs, from roughly −0.5 to about 0.6 for the intersession ICC and from about −0.2 to roughly 0.9 for the intrasession ICC.

Fig. 4.

Connectivity strength (Z) versus test–retest reliability (ICC) for all the connections and the three resting connections for the intrasession (left) and intersession (right) comparisons.

Fig. 5 shows the ICC averaged for all within-network connections, as well as broken down for different networks. Error bars represent the standard deviation of the ICCs across different connections within the specified network. No error bars are shown for the motor, auditory, and visual networks, since only one connection was assessed in each of these networks. As expected, the intra-session reliability is higher than inter-session reliability. On average for all within-network connections, the EF condition was significantly more reliable than the other two conditions for both intra- and inter-session comparisons (cf. Table 3).

Fig. 5.

Average intra- and inter-session ICC values for each resting condition. Results are presented for all within-network connections, as well as broken down by networks. Differences in ICCs between the different conditions were assessed using a paired Jackknife method. Significant differences are indicated. * = 0.05 > p > 0.01, ** = 0.01 > p > 0.001, and ***p < 0.001.

Table 3.

Results of the Jackknife analysis determining the significance of the ICC differences for each network and for intra- and inter-session comparisons.

ICC	Network	Significant difference	p-Value
Intrasession	DMN	EO > EC	≪0.0001
		EF > EO	0.0072
		EF > EC	≪0.0001
	Attention	–
	Within-network	EO > EC	0.012
		EF > EO	≪0.0001
		EF > EC	≪0.0001
	Auditory	EC > EO	0.0296
		EF > EO	≪0.0001
		EF > EC	≪0.0001
	Motor	EO > EC	0.0056
		EF > EO	0.012
		EF > EC	≪0.0001
	Visual	EO > EF	0.0072
		EC > EF	0.0117
Intersession	DMN	EF > EO	0.0392
		EF > EC	0.0392
	Attention	EF > EO	≪0.0001
		EF > EC	0.0344
	Within-network	EF > EO	0.0168
		EF > EC	0.0056
	Auditory	–
	Motor	EO > EF	≪0.0001
		EC > EF	≪0.0001
	Visual	EO > EC	≪0.0001
		EO > EF	≪0.0001
		EC > EF	0.0264

When the connections are broken down by networks, other differences emerge. In the DMN the ICC is significantly higher for EF compared to EC and EO (cf. Table 3). Similarly, for the auditory and attention networks, EF resulted in the highest reliability, although this was only statistically significant for intra-session ICC in the auditory cortex and inter-session ICC in the attention network. In the visual cortex, the reliability was highest in the EO condition and lowest in the EF condition. The connectivity between the left and right motor cortices was most reliable in the EF condition for intrasession comparisons, and the EO condition for intersession comparisons.

Consistency of connectivity strength for connections and subjects

Fig. 6 shows the mean intraindividual Kendall's W (i.e. the consistency of ranking connections) across subjects for all the connections, all sessions, and all conditions. All the results for the different tests of significance are summarized in Table 4. The error bars reflect the standard deviation in the intra-individual Kendall's W across subjects. This Kendall's W was quite similar across resting conditions, but there were some subtle differences. For both the intra-session and intersession comparisons, the EC condition yielded higher Kendall's W values. A Wilcoxon signed-rank test showed that this difference was significant only for the EC versus the EO condition.

Fig. 6.

Average intraindividual Kendall's W for each resting condition and each session. The Kendall's W values were averaged across subject. The intraindividual Kendall's W were calculated based on all the connections. Differences in Kendall's W between the different conditions were assessed using Jackknife method at a level of significance of 0.05. * = 0.05 > p > 0.01, ** = 0.01 > p > 0.001, and ***p < 0.001.

Table 4.

p-Values from a two tailed Wilcoxon Signed-Rank Test assessing differences in intraindividual Kendall's W values between the different resting conditions for all the connections. Cells in bold indicate significant differences (p < 0.05) after multiple comparisons.

Intraindividual Kendall's W	All connections
	Comparison	Corrected p-value
Intrasession	EO vs. EC	>0.05
	EO vs. EF	>0.05
	EC vs. EF	>0.05
Intersession	EO vs. EC	0.0048
	EO vs. EF	< 0.05
	EC vs. EF	>0.05

Fig. 7 shows the consistency of individual differences in the strength of functional connectivity (i.e. the consistency of ranking subjects), as computed by the interindividual Kendall's W. The error bars represent the standard deviation of the Kendall's W across connections within that network. On average for all within-network connections, the EF condition had higher Kendall's W for both intra- and inter-session indicating an overall higher consistency in individual rankings (i.e. subjects with the highest connectivity strength in one scan also have the highest connectivity in other scans) (see Table 5). Once we broke the analysis into different networks we observed that, for the intra-session comparison, the EF condition yielded significantly higher interindividual Kendall's W than both other conditions in the DMN, as well as the auditory and motor connectivity. Of note, for inter-session comparisons, the EF condition was found to be significantly more consistent in the DMN and the attention network; while the EO condition resulted in significantly higher Kendall's W for the motor and visual connectivity.

Fig. 7.

Average interindividual Kendall's W for each resting condition and each session. The Kendall's W values were averaged across connection. The interindividual Kendall's W were calculated based on the within-network connections. Differences in Kendall's W between the different conditions were assessed using Jackknife method at a level of significance of 0.05. * = 0.05 > p > 0.01, ** = 0.01 > p > 0.001, and ***p < 0.001.

Table 5.

Results of the Jackknife analysis determining the significance of the inter-individual Kendall's W differences for each network and for intra- and inter-session comparisons.

Inter-individual Kendall's W	Jackknife	Intrasession	p-Value
	Network	Significant difference
Intrasession	DMN	EO > EC	≪0.0001
		EF > EO	0.0072
		EF > EC	≪0.0001
	Attention	EF > EO	0.0024
	Within-network	EF > EO	≪0.0001
		EF > EC	≪0.0001
	Auditory	EC > EO	≪0.0001
		EF > EO	≪0.0001
		EF > EC	≪0.0001
	Motor	EO > EC	0.02
		EF > EO	0.0088
		EF > EC	≪0.0001
	Visual	EO > EC	≪0.0001
		EO > EF	≪0.0001
Intersession	DMN	EF > EO	0.0392
		EF > EC	0.0424
	Attention	EF > EO	≪0.0001
		EF > EC	0.0056
	Within-network	EF > EO	0.0072
		EF > EC	0.00152
	Auditory	EC > EO	0.012
		EF > EO	0.0088
	Motor	EO > EC	0.0072
		EO > EF	≪0.0001
		EF > EC	≪0.0001
	Visual	EO > EC	≪0.0001
		EO > EF	≪0.0001
		EC > EF	0.0392

Motion

Table 1 shows the average motion across subjects and scans for each condition studied here as well as p-values from the different statistical comparisons. No significant difference was observed in the amount of subject motion during the imaging run between EO, EC and EF conditions (p = 0.55 for the translational MRD, p = 0.64 for the rotational motion as measured by the Euler angle, p = 0.36 for SSD, and p = 0.76 for the amount of time points censored).

Discussion

Functional connectivity

Consistent with results from other studies (Fox et al., 2005; Fransson, 2005; Yan et al., 2009), we found that default mode network connectivity maps are largely similar across different resting conditions. That is, in all three of the resting conditions we assessed, the PC has significant positive connectivity to the medial prefrontal cortex and bilateral parietal regions. However, a statistical comparison between resting conditions reveals a decreased connectivity between the PC node of the default mode network and the insula, part of the salience (or cingulo-opercular) network, in the EF condition. We found no significant differences in the voxel-wise connectivity maps of the lIPS seed of the attention network across resting conditions. For the visual connectivity maps (lVis seed), we found clusters of greater negative correlation between the left visual seed and areas covering a large portion of the left and right thalami in the EC condition. This might be due to an increase in alpha wave fluctuation during the EC condition, which has been found to originate in the thalamic regions (Schreckenberger et al., 2004).

After examining the connectivity strength at the ROI level, rather than voxel-wise maps, and breaking our results down into connectivity strength averaged across scans within the same session (average connectivity strength for scan 1 and scan 2) and across different sessions (average connectivity for scans 1 and 3 and for scans 2 and 3) we observed significant differences only in the auditory network. There was a trend towards stronger default mode network connectivity in the EF condition (p < 0.03, uncorrected), which is consistent with prior studies (Van Dijk et al., 2010; Yan et al., 2009), but this did not survive multiple comparison correction.

Test–retest reliability

With respect to previous studies, our ICC values were higher than those reported by some studies (Shehzad et al., 2009), but consistent with other studies (Braun et al., 2012). These differences in ICC may be coming from differences in preprocessing steps, age of the sample (mean and range), acquisition parameters, or equipment differences (e.g., head coil, MR scanner). Our study examined subjects over a wide age range, which may have resulted in greater between-subject variance and thus greater ICC values. The ICC values confirm that rs-fMRI is a reliable method, particularly when evaluating connectivity within-networks. The lower ICC values observed across sessions might be coming from the fact that the subjects were in a different state of mind when they came for the second visit (e.g., different stress levels, anxiety levels, overall mood, satiety, somnolence, etc.), and may thus reflect true neuronal differences rather than variations due to noise. Future studies could thus focus on the impact of these state-related changes on the test–retest reliability of rs-fMRI.

Significant differences in reliability were found between the three resting conditions, with the EF condition having the highest reliability overall and as well as when connections were broken down into networks (in particular in the DMN, attention, auditory, and motor networks). Further analysis showed that this resulted from a decrease in the within-subject variability within a session (increasing intra-session reliability) and an increase in the between-subject variability (for the inter-session reliability). The higher within-subject variability in the EO and EC conditions might be due to higher motion of the eyes during scanning, but additional studies are needed to further explore this theory. In the DMN, attention, and motor networks, the EC condition was the least reliable. Further examination of the within- and between-subject variances indicates that this was due to an increased within-subject variability for EC relative to EO and EF.

Within the visual network, the EF condition yielded the lowest ICC values. This may be due to a lower amplitude and coherence of BOLD fluctuations in the visual cortex due to a reduced variability in the amount of incoming visual information, or a greater consistency in the visual activity across subjects. Further analysis showed that this reduced ICC in the EF condition resulted primarily from a decrease in between-subject variability. Finally, for the EC condition, higher BOLD signal amplitudes due to an increase in alpha rhythms may have impacted the ICC values.

Consistency of connectivity strength for connections and subjects

Intra-individual Kendall's W values reported in Fig. 6 are very similar to values reported by Shehzad et al. (2009). The high Kendall's W values show that the connectivity measures obtained via the rs-fMRI technique are consistent within-subjects. That is, for each subject, areas of the brain that are the most significantly connected in one session are still the most significantly connected when that session is repeated. This measure, however, is not sensitive to variations in global noise, which could result in an increase or decrease in the strength of all connections. Thus, while the overall pattern of connectivity remains largely the same across sessions, the strength of connectivity may still be affected by resting-state conditions.

Inter-individual Kendall's W were found to be very high as well. Averaged for all within-network connections, the EF condition yielded significantly more consistent results for both intra- and inter-session comparisons. In other words, in the EF condition, subjects with the highest connectivity strength in scan 1 were more likely to also have the highest connectivity strength in scans 2 and 3. This result holds for the DMN, the attention network, the auditory, and motor connectivity.

Motion

The levels of motion in our study are similar to those found in previous studies (Satterthwaite et al., 2012; Van Dijk et al., 2012). We did not find any significant differences in the amount of motion in different resting conditions, measured by the mean relative displacement. However, given the significant effect of motion on resting data acquisition, if it is known or expected that a population tends to move less in one of the resting conditions, it would be advisable to use that particular condition. Furthermore, future studies should focus on the impact of head motion on test–retest reliability as well as the impact of different methods of restricting and correcting for head motion on test–retest reliability.

Limitations

Some of the differences with respect to other studies (i.e. connectivity strength results) could be coming from the recruitment of the sample and differences in the various preprocessing steps. A potential limitation of this study is that the protocol did not keep the time of day that the scans were acquired consistent from session one to session two. However, all sessions started between 1 pm and 5 pm for each subject. We therefore believe that differences due to the time at which the scan was acquired are relatively minor. It should also be emphasized that our study assessed the test–retest reliability, specifically the consistency of ranking subjects (with the Kendall's W) and the comparison of within-subject/across-scan variations to between-subject variations. However, we cannot determine whether the reliable differences in connectivity between subjects are in fact neuronal in origin. Furthermore, variations in the connectivity within an individual across sessions may reflect differences in neuronal connectivity (e.g. different mental states). Such state-related differences may be of great interest, e.g. for clinical or diagnostic purposes. If the EF condition reduces session-to-session variations within typical subjects but preserves session-to-session variations of interest (e.g. in a patient population), then this condition would be advantageous for clinical studies. However, since we only scanned typical subjects, we cannot determine whether session-to-session differences of interest within a patient population are affected by resting condition in the same way as session-to-session variations within a typical population.

Conclusion

This study confirmed the reliability and consistency of the rs-fMRI technique as a tool to observe and characterize functional connectivity of the human brain. Functional connectivity maps were consistent within-subjects and the strength of connectivity within default mode, attention, motor, visual, and auditory networks were reliable across subjects. More particularly, having subjects keep their eyes open (and not asking them to fixate) resulted in the highest reliability in the visual network. In contrast, asking subjects to keep their eyes open and fixate on a cross resulted in the highest reliability (compared eyes open and eyes closed conditions) when examining all the within-network connections as well as the DMN, the attention network, and auditory connectivity.