The Healthy Brain Network Serial Scanning Initiative: a resource for evaluating inter-individual differences and their reliabilities across scan conditions and sessions

Abstract

Background: Although typically measured during the resting state, a growing literature is illustrating the ability to map intrinsic connectivity with functional MRI during task and naturalistic viewing conditions. These paradigms are drawing excitement due to their greater tolerability in clinical and developing populations and because they enable a wider range of analyses (e.g., inter-subject correlations). To be clinically useful, the test-retest reliability of connectivity measured during these paradigms needs to be established. This resource provides data for evaluating test-retest reliability for full-brain connectivity patterns detected during each of four scan conditions that differ with respect to level of engagement (rest, abstract animations, movie clips, flanker task). Data are provided for 13 participants, each scanned in 12 sessions with 10 minutes for each scan of the four conditions. Diffusion kurtosis imaging data was also obtained at each session.

Findings: Technical validation and demonstrative reliability analyses were carried out at the connection-level using the Intraclass Correlation Coefficient and at network-level representations of the data using the Image Intraclass Correlation Coefficient. Variation in intrinsic functional connectivity across sessions was generally found to be greater than that attributable to scan condition. Between-condition reliability was generally high, particularly for the frontoparietal and default networks. Between-session reliabilities obtained separately for the different scan conditions were comparable, though notably lower than between-condition reliabilities.

Conclusions: This resource provides a test-bed for quantifying the reliability of connectivity indices across subjects, conditions and time. The resource can be used to compare and optimize different frameworks for measuring connectivity and data collection parameters such as scan length. Additionally, investigators can explore the unique perspectives of the brain's functional architecture offered by each of the scan conditions.

Data description

An extensive literature has documented the utility of functional MRI (fMRI) for mapping the brain's functional interactions through the detection of temporally correlated patterns of spontaneous activity between spatially distinct brain areas [1–7]. Commonly referred to as intrinsic functional connectivity (iFC), these patterns are commonly studied during the “resting state,” which involves the participant quietly lying awake and not performing an externally driven task. Resting state fMRI (R-fMRI) has gained popularity in clinical neuroimaging due to its minimal task and participant compliance demands. R-fMRI has also demonstrated good test-retest reliability for commonly used measures [8–12], and utility in detecting brain differences associated with neuropsychiatric disorders [13,14]. Despite these successes, a growing body of work is questioning the advantages of resting state, given reports of higher head motion, decreased tolerance of the scan environment (e.g., boredom, rumination), and increased likelihood of falling asleep compared to more engaging task-based fMRI paradigms [15–18]. This is particularly relevant for studies of pediatric, geriatric, and clinical populations, all of which are characterized by lower tolerance of the scanner environment.

A number of less challenging scan conditions have been proposed as alternatives for estimating iFC. Particularly intriguing are “naturalistic viewing” paradigms [15,19,20]. It has been shown that the mental state (i.e., emotional state, performing a task, etc.) of the participant during scanning can affect iFC patterns; recent work suggests that low engagement states (e.g., computer animations with limited cognitive content) may come close to mimicking rest from a neural perspective [21]. Several studies have illustrated the ability to relate trait phenotypic variables to inter-individual differences in iFC across conditions, even if extrinsically driven signals (i.e., task stimulus functions) are not removed [21–27]. However, comprehensive comparisons of the relative impact of scan condition on detection of inter-individual differences in iFC, and the test-retest reliability of these differences, are needed before these paradigms can fully supplant R-fMRI.

Here we describe a dataset that was generated as part of a pilot testing effort for the Child Mind Institute Healthy Brain Network, a large-scale data collection effort focused on the generation of an open resource for studying child and adolescent mental health. The primary goal of the data collection was to assess and compare test-retest reliability of full-brain connectivity patterns detected for each of four scan conditions that differed with respect to level of engagement. Specifically, 13 participants were scanned during each of the following four conditions on 12 different occasions: 1) rest, 2) free viewing of computer-generated abstract shapes with music designed to have minimal cognitive or emotional content (i.e., “Inscapes”, [15]), 3) free viewing of highly engaging movies [19], and 4) performance of an active task (i.e., an Eriksen flanker task [28], with no-Go trials included). For each of the non-rest conditions, three different stimuli were used, with each being repeated four times across the 12 sessions to enable the evaluation of repetition effects. Given the focus on naturalistic viewing, an additional scan session containing a full viewing of “Raiders of the Lost Ark” (Lucasfilm Ltd., 1981) was included to facilitate interested parties in the exploration and evaluation of hyper alignment approaches, which offer increasingly popular and unique solutions to overcoming anatomical variability when attempting to match functional systems across individuals [29].

Although not a primary focus of the data collection, additional structural imaging data was collected, which are being shared as well: 1) MPRAGE [30], 2) diffusion kurtosis imaging [31,32], 3) quantitative T1/T2 anatomical imaging (single session) [33], and 4) magnetization transfer (single session) [34] (see Table 1).

Table 1.

HBN-SSI experimental design.

	Shared imaging data
Session #	Session type	Description
1	Baseline characterization	• Multiecho MPRAGE
		• DKI
		• Quantitative T1/T2 mapping
		• Magnetization transfer ratio
		• FLAIR
		• fMRI: rest (10 min)
2-7, 9-14	Repeat scanning	• Multiecho MPRAGE
		• DKI
		• fMRI: rest (10 min)
		• fMRI: naturalistic viewing: inscapes (10 min)
		• fMRI: naturalistic viewing: movie clips (10 min)
		• fMRI: flanker task (10 min)
8	Full feature movie	• fMRI: Raiders of the Lost Ark (20 min × 6)

Methods

Participants and procedures

Thirteen adults (ages 18–45 years; mean age: 30.3; 38.4% male) recruited from the community participated in the Healthy Brain Network's Serial Scanning Initiative. Each participant attended 14 sessions over a period of 1–2 months; see Table 1 for the breakdown of data acquired across sessions. All imaging data were collected using a 1.5T Siemens Avanto equipped with a 32-channel head coil in a mobile trailer (Medical Coaches, Oneonta, NY). The scanner was selected as part of a pilot initiative being carried out to evaluate the capabilities of a 1.5T mobile scanner when equipped with a state-of-the-art head coil and imaging sequences. All research performed was approved by the Chesapeake Institutional Review Board, Columbia, MD [35].

Experimental design

As outlined in Table 1, each participant attended a total of 14 separate imaging session; these included: 1) a baseline characterization session containing a variety of quantitative anatomical scans; 2) 12 serial scanning sessions, each using the same imaging protocol consisting of four fMRI scan conditions (10 min per condition), diffusion kurtosis imaging, and a reference MPRAGE anatomical scan; and 3) a Raiders of the Lost Ark movie viewing session.

Functional MRI scan conditions included in serial scanning

The following four functional scan conditions were selected to sample a range of levels of engagement, presented in ascending order of level of engagement (see Fig. 1):

Figure 1.

Shown here are sample stimuli from each of the four scan conditions included in the present work. These included: 1) resting state, (far left); 2) inscapes (middle left); 3) movie clips (e.g., the Matrix; middle right); and 4) flanker task (with no-go trials; far right).

Rest

The participant was presented a white fixation cross in the center of a black screen and instructed to rest with eyes open. Specific instructions were as follows: “Please lie quietly with your eyes open, and direct your gaze towards the plus symbol. During this scan let your mind wander. If you notice yourself focusing on a particular stream of thoughts, let your mind wander away.”

Inscapes

Inscape is a computer-generated animation comprised of abstract, non-social, technological-looking 3D forms that transition in a slow, continuous fashion without scene cuts. Visual stimulation is accompanied by a piano composition based on the pentatonic scale with a slow tempo (48 bpm), which was intended to be calming and to harmonize with the noise generated by EPI sequences [15]. Three unique 10-min sequences were created using the original 7-min Inscapes, and were presented across the 12 repeat scanning sessions. These clips are available for download from the HBN-SSI web page [36].

Movie

Three unique 10-min movie clips were presented across the 12 repeat scanning sessions. To ensure a high level of engagement, three Hollywood movie clips (American versions) were selected, each representing a different movie genre and containing a narrative arc that fit into the 10-min clip. The specific clips selected were: Wall-E (Walt Disney Productions, 2008, time codes 00:02:03:13 to 00:12:11:05), The Matrix (Warner Bros., 1999, 00:25:23:10 to 00:35:19:20), and A Few Good Men (Columbia Pictures,1992, 01:58:13:01 to 02:08:11:18). Due to copyright issues, these clips could not be shared.

Flanker

The Eriksen Flanker task consisted of presenting a series of images containing five arrows. For each image, the participant was asked to focus on the center arrow and indicate if it is pointing left or right by pushing a button with their left or right index finger. The flanking arrows could be pointing the same way (congruent) or the opposite way (incongruent). Also built into the task were a neutral stimulus and a go/no-go aspect. The neutral task contained diamonds instead of flanking arrows, making the central arrow direction more obvious. The no-go stimuli contained x's instead of flanking arrows, indicating that the subject should not push either button. See Fig. 1 for a visualization of the stimuli. The stimuli and timing of their presentation are available for download from the HBN-SSI web page [36].

Counter-balancing

Order effects are an obvious concern when comparing four functional scan conditions. To minimize these effects, we ensured that for each participant; 1) each scan type occurred an equal number of times in each of the four scan slots across the 12 sessions, and that 2) each scan type had an equal frequency of being preceded by each of the other three scan types. We made use of three exemplars of each non-rest stimuli to enable the examination of repetition effects. For movies, this involved having three 10-minute clips, each from a different movie; for Inscapes, this involved three different animation sequences and for the flanker task, three different stimulus orderings were used. We guaranteed that across the 12 scan sessions, each exemplar occurred one time across every three scan sessions. Specific ordering of exemplars was varied across odd and even numbered participants. For each participant, individual-specific ordering information is provided in the release.

Imaging protocols (See Table 2 for scan protocol details)

Table 2.

MRI acquisition parameters for scans included in the HBN-SSI

Image	Structural								Functional Rest/Movie/Inscapes/Flanker
	Whole brain T1	Inversion recovery	Whole brain T2	Magnetization transfer	ME-MPRAGE RMS	T2 FLAIR	DWI	DKI
Manufacturer	Siemens
Model	Avanto
Head coil	32 Channel
Field strength	1.5T
Sequence	3D Despot 1	IR-SPGR / 3D Despot 1	3D Despot 2	3D FLASH	ME-MPRAGE/ 3D TFL	FLAIR	EPI	EPI	EPI
Flip angle(s) [Deg]	2.66;3.55;4.44;5.33;6.22;8.0;11.55;16.0	5	10.0;13.33;16.66;19.99;23.33;30.0;43.33;60.0	15	7	150	90	90	55
Phase cycling [Deg]	NA	NA	0;180	NA	NA	NA	NA	NA	NA
Inversion time [ms]	NA	400	NA	NA	1000	2500	NA	NA	NA
Echo time [ms]	2.4	2.4	2.7	11	1.64	89	76.2	93.8	40
Repetition time [ms]	5.2	5.3	5.4	30	2730	9000	3110	4500	1450
Bandwidth per voxel (readout) [Hz/Px]	350	350	350	350	651	190	1628	1628	2374
Parallel acquisition	None	None	None	GP2	GP2	GP2	MB3	None	MB3
Partial Fourier	P6/8 S7/8	S6/8	P6/8 S7/8	P6/8 S6/8	None	None	P6/8	P6/8	None
Slice orientation	S	S	S	S	S	T	T	T	T
Slice phase encoding direction	AP	AP	AP	AP	AP	RL	AP/PA	AP	AP
Slice acquisition order	SA	SA	SA	IA	IA	IA	IA	IA	IA
Slice gap [%]	20	20	20	20	50	30	0	0	0
Field of view [mm]	220 × 220	220 × 220	220×220	256×256	256×256	201×230	192×192	192×192	192×192
Reconstructed image matrix	128 × 128 × 96	128 × 128 × 48	128 × 128 × 96	256 × 256 × 176	256 × 256 × 176	448 × 512 × 25	96 × 96 × 72	96 × 96 × 72	78 × 78 × 54
Reconstructed resolution [mm]	1.72 × 1.72 × 1.8	1.72 × 1.72 × 3.6	1.72 × 1.72 × 1.8	1.0 × 1.0	1.0 × 1.0 × 1.0	0.45 × 0.45 × 6.5	2.0 × 2.0 × 2.0	2.0 × 2.0 × 2.0	2.46 × 2.46 × 2.5
Number of measurements	8	1	16	1	4	1	1	1	420
Acquisition time [min:sec]	05:00	00:53	08:38	06:41	06:32	02:44	00:16	09:59	10:18
Fat suppression	None	None	None	None	None	On	On	On	On
Number of directions	NA	NA	NA	NA	NA	NA	64	64	NA
Number of B zeros	NA	NA	NA	NA	NA	NA	1	1	NA
B value (s) [s/mm2]	NA	NA	NA	NA	NA	NA	0	0;1000;2000	NA
Averages	NA	NA	NA	NA	1	NA	NA	NA	NA

AP: anterior posterior; GP: GRAPPA in the phase direction; MB: multiband; PA: posterior anterior; RL: right left; IA: interleaved ascending; SA: sequential ascending; S: saggital; T: transverse.

• fMRI (sessions 1–14): For all fMRI scans, the multiband EPI sequence provided by CMRR [37] was employed to provide high spatial and temporal resolutions (multiband factor 3, voxel size: 2.46 × 2.46 × 2.5mm; TR: 1.46 seconds).

• MEMPRAGE (sessions 1–7, 9–14): Across all sessions (except the full-movie session), we obtained a multi-echo MPRAGE sequence for the purposes of anatomical registration [38]. Within a given scan, four echoes are collected per excitation and combined using root mean square average. This enables the images to be acquired with a higher bandwidth to reduce distortion, while recovering SNR through averaging. The added T2* weighting from the later echoes also helps differentiate dura from brain matter.

• Diffusional Kurtosis Imaging (DKI): Leveraging the capabilities of the CMRR multiband imaging sequence, we were able to acquire 64 directions at 2 b-values (1000 and 2000 s/mm²). This enables diffusion kurtosis-specific metrics to be calculated from the data, in addition to standard DTI metrics, and can improve tractography [31].

• Quantitative Relaxometry MRI (Quantitative T1, T2, and My- elin Water Fraction): DESPOT1 and DESPOT2 sequences were used to characterize microstructural properties of brain tissue. These innovative acquisition strategies enable quantitation of T1 and T2 relaxation constants, which can be combined to calculate myelin water fraction [39].

• Magnetization Transfer: High-resolution T1-weighted structural images were acquired with a FLASH sequence, with and without a saturation RF pulse. The magnetization transfer ratio is calculated from the resulting images, which is purportedly a sensitive marker of myelination [34].

Limitations

A limitation of the described resource is that the data were collected using a 1.5T scanner platform, rather than 3T. While we do not expect the overall results obtained with data from the 1.5T and 3T platforms should be fundamentally different, there is generally better SNR and temporal resolution with the 3T scanner platform. To mitigate these differences, 1) the system was upgraded to 32 receive channels to take advantage of the latest head-coil technologies for increasing SNR, and 2) simultaneous multi-slice imaging was used to improve the spatial and temporal resolution.

DATA records

Data privacy

The HBN-SSI data are being shared via the 1000 Functional Connectomes Project and its International Neuroimaging Data-sharing Initiative (FCP/INDI) [40]. Prior to sharing, all imaging data were fully de-identified by removing all personally identifying information (as defined by the Health Insurance Portability and Accountability) from the data files, including facial features. The removal of facial features was performed using the “mri_deface” software package developed by Bischoff-Grethe et al. [41]. All data were visually inspected before release to ensure that these procedures worked as expected.

Distribution for use

Imaging data

All MRI data can be accessed through the Neuroimaging Informatics Tools and Resources Clearinghouse [36] and FCP/INDI's Amazon Web Services public Simple Storage Service (S3) bucket. In both locations, the imaging data are stored in a series of tar files that can be directly downloaded through a HTTP client (e.g., a web browser, Curl, or wget). The data are additionally available on S3 as individual NifTI files for each scan, which can be downloaded using a HTTP client or S3 client software such as Cyberduck [42].

All imaging data are released in the NIfTI file format; they are organized and named according to the Brain Imaging Data Structure (BIDS) format [43].

Phenotypic data

Partial phenotypic data will be publicly available without any requirements for a data usage agreement. This includes age, sex, handedness, the internal state questionnaire, and the New York Cognition Questionnaire [43]. These data are located in a comma separated value (.csv) file accessible via the HBN-SSI web site and are included with the brain imaging data structure-organized imaging data as tab separate values files. The remainder of the phenotypic data (see Table 3), including the PANAS [44] and results from the ADHD Quotient system [45], will be made available to investigators following completion of the HBN Data Usage Agreement. The HBN Data Usage Agreement is modeled after that of the NKI-Rockland Sample and is intended to prevent against data re-identification; it does not place any constraints on the range of analyses that can be carried out using the shared data, or place requirements for co-authorship. Following submission and execution of the data usage agreement, users can access the phenotypic data through the COINS Data Exchange (an enhanced graphical query tool, which enables users to target and download files in accord with specific search criteria) [46].

Table 3.

Questionnaires and physical measures collected.

Questionnaires
Internal State Questionnaire (pre-scan, post-scan)	3-item self-report questionnaire assessing hunger and thirst. Participants respond on a visual analogue scale ranging from “I am not hungry/thirsty/full at all” to “I have never been more hungry/thirsty/full”. Responses are rated from 0 to 100. Participants complete this questionnaire before and after each scan.
New York Cognition Questionnaire (NYC-Q) (post-scan)	31-item self-report questionnaire that asks participants about the different thoughts and feelings that they may have had while in the MRI scan. Participants are asked to indicate the extent to which their thinking or experience corresponded to each item on a 9-point scale.
PANAS (post-scan)	The PANAS-S is a self-administered, 20-item Likert scale assessment that measures degree of positive or negative affect. Users are asked to rate 10 adjectives that measure positive feelings such as joy or pleasure, and 10 adjectives that measure negative feelings, such as anxiety or sadness, on a scale of how closely the adjective describes them in the present moment or over the past week. Items are rated on a five-point scale.
Physical Measures Vitals	Participant vitals (blood pressure, heart rate, blood glucose level, first day of last menstrual cycle) were collected prior to each scan using standard measurement devices in a laboratory environment.
Voice data samples	Audio samples of participant speech were recorded prior to scanning. Each sample consisted of 10 sentences with 5 different implicit emotions (neutral, happy, sad, angry, fearful), 10 non-words, and 2 min of free speech. For each sample different sentences were drawn from the same set of emotions; the non-words also differed in each sample but had similar characteristics (ie number of syllables, chunks). Stimuli were presented on a laptop computer screen. Completion of the sample took up to 15 min.
Quotient ADHD System	Quotient is a computer based task designed to assess three core symptoms of ADHD: hyperactivity, attention, and impulsivity. Participants respond to stimuli presented with random timing and random placement on a screen. Completion of the task takes up to 30 min.
GeneActiv Actimetry Device	Between scanning sessions, participants wore a non-invasive actimetry sensor that recorded heart rate and indices of physical activity and sleep. The device was placed on participants’ non-dominant wrist and data was collected at each scanning session.

Technical validation

Quality assessment

Consistent with the established FCP/INDI policy, all completed datasets contributed to HBN-SSI are made available to users regardless of data quality. Justifications for this decision include the lack of consensus within the imaging community on what constitutes good or poor quality data, and the utility of “lower quality” datasets for facilitating the development of artifact correction techniques. For HBN-SSI, the inclusion of datasets with significant artifacts related to factors such as motion are particularly valuable, as it facilitates the evaluation of the impact of such real-world confounds on reliability and reproducibility.

To help users assess data quality, we calculated a variety of quantitative quality metrics from the data using the Preprocessed Connectome Project Quality Assurance Protocol (QAP) [47]. The QAP includes a broad range of quantitative metrics that have been proposed in the imaging literature for assessing data quality [48].

For the structural data, spatial measures include: signal-to-noise ratio (SNR) [49], contrast-to-noise ratio (CNR) [49], foreground-to-background energy ratio (FBER), percent artifact voxels (QI1) [50], spatial smoothness (FWHM) [51], and entropy focus criterion (EFC) [52]. These are shown for different participants in Fig. 2. Spatial measures of fMRI data include (Fig. 3): EFC, FBER, FWHM, as well as ghost-to-signal ratio (GSR) [53].

Figure 2.

Subset of QAP spatial anatomical measures for each participant (horizontal axis). Depicted are the following measures: CNR, SNR, EFC, FBER, spatial smoothness (FWHM), and percent artifact voxels (QI1). Each point indicates the measure calculated for an individual scan; for each participant, the data across scan conditions and sessions are depicted using a single color.

Figure 3.

Subset of QAP spatial functional measures for each participant (horizontal axis). Depicted are the following measures: GSR, SNR, EFC, FBER, and spatial smoothness (FWHM). Each point indicates the measure calculated for an individual scan; for each participant, the data across scan conditions and sessions are depicted using a single color.

Temporal measures of fMRI data include (Fig. 4): mean frame-wise displacement (mean FD) [54], median distance index (quality) [55], standardized DVARS (DVARS) [56], outliers detection [55], and global correlation (GCOR) [57]. See Figs. 2–4 for a subset of the metrics; the full set of measures are included on the HBN-SSI website in .csv format for download. Review of the QAP profiles led us to exclude three participants based on excessively high mean FD from the illustrative analyses presented in the next section. Although not a focus of the current work, visual inspection of the figures points to the potential value of this dataset for establishing the reliability of QAP measures. The impact of scan condition on each of the functional QAP measures was examined using a one-way ANOVA. No significant differences were found for any of the measures. In addition, the test-retest reliability of each QAP measure, for each condition, was assessed using the intra-class correlation coefficient (ICC). The results are shown in Table 4.

Figure 4.

Subset of QAP temporal functional measures for each participant (horizontal axis). Depicted are the following measures: Outliers detection (Outliers), GCOR, Quality, mean frame-wise displacement (Mean FD), and DVARS. Each point indicates the measure calculated for an individual scan; for each participant, the data across scan conditions and sessions are depicted using a single color.

Table 4.

ICC values representing the test-retest reliability of QAP measures, for each scan condition

Measure	Rest	Inscapes	Movie	Flanker
EFC	0.90	0.91	0.93	0.92
FBER	0.84	0.84	0.84	0.83
FWHM	0.58	0.60	0.74	0.76
GSR	0.56	0.56	0.61	0.62
SNR	0.92	0.91	0.93	0.92
Outliers	0.08	0.18	0.06	0.50
GCOR	0.11	0.09	0.16	0.04
Quality	0.94	0.94	0.93	0.95
Mean FD	0.30	0.39	0.40	0.68
DVARS	0.42	0.49	0.47	0.49

FMRI analyses

A broad range of analyses, including but not limited to evaluations of test-retest reliability, can be performed using the present HBN-SSI dataset. Here, we provide a few illustrative analyses to demonstrate the technical validity and utility of these data; they are not intended to be exhaustive.

Data preprocessing

Prior to image processing, Freesurfer [58] was used to combine the 12 available MPRAGE images into an MRI robust average image for each individual participant. A non-rigid registration between MPRAGE images and a 2-mm MNI brain-only template (FSL's MNI152_T1_2mm_brain.nii.gz, [59]) was calculated using ANTs [60]. Further anatomical processing included skull stripping using AFNI's 3dSkullstrip [61] (to include any voxels in the ventricles incorrectly removed by this utility, the brain mask was augmented using a ventricle mask that was generated by reverse transforming the ventricles included in the MNI atlas into native space for each participant). Next, data was processed using a development version of the open-source, Nipype-based [62] Configurable Pipeline for the Analysis of Connectomes [1] (C-PAC version 0.4.0 [63]). See here for image preprocessing configuration file [64].

Following resampling of the fMRI data to RPI orientation, image preprocessing in C-PAC consisted of the following steps: 1) motion correction, 2) boundary-based registration [65], 3) nuisance variable regression (1st and 2nd order polynomial, 24-regressor model of motion [66], mean WM mask signal, mean CSF mask signal). We then extracted representative time series for each ROI in the CC200 atlas [67] (by averaging within-ROI voxel time series). All possible pairwise correlations were calculated amongst ROI time series to generate a ROI-to-ROI connectivity matrix for each scan in each session for each subject. To facilitate ease of presentation and interpretation of our findings, the connections were sorted by intrinsic connectivity network membership, as defined by Yeo et al. [68].

Fingerprinting

Prior work by Finn et al. [22] demonstrated the ability to “fingerprint” individuals based on their functional connectivity matrices. Specifically, they found that the level of correlation between connectivity matrices for data obtained from the same participant on different occasions was markedly higher than that observed for connectivity matrices obtained from different participants; this was true regardless of whether functional connectivity was based on resting state or task activation data. Consistent with their work, we found a dramatically higher degree of correlation, using Pearson's R, between connectivity matrices obtained from the same individual on differing sessions (mean: 0.599, SD: 0.083, 95% CI: 0.598–0.600), when compared to differing individuals (mean: 0.445, SD: 0.065, 95% CI: 0.444–0.445) (Fig. 5). Also consistent with their findings, we found this to be true regardless of the scan condition employed.

Figure 5.

Similarity of full-brain connectivity matrices across participants (green), sessions (blue), and scan conditions (yellow), as measured using Pearson correlation coefficients (red). Also depicted in the bottom right are the distributions of correlation coefficients when comparing scans from the same subject (within subject), and scans from different subjects (between subject). In the right column are the values for scans from the same subject, and in the left are scans from different subjects. The median, first, and third quartiles are also depicted with horizontal lines.

Connection-wise reliability for the four states

A key question is how much variation among scan conditions (i.e., between-condition reliability) impacts reliability as opposed to between-session reliability (i.e., test-retest reliability). To address this question, we analyzed the 12 sessions obtained for the 10 participants with minimal head motion using a hierarchical linear mixed model (note: three subjects were missing the flanker task from one session each; these were treated as missing values in our analyses). The hierarchical linear mixed model allows for the estimation of reliability by providing estimates of variance between participants, across the four conditions (for the same participant) and between sessions within each condition.

(1)

For a given functional connectivity measurement ν, iFC_ijk(ν) is the modeled iFC for the i-th session, for the j-th condition of the k-th participant, taking into account condition and session effects. The equation is composed of an intercept μ₀₀₀, a random effect between sessions for the j-th condition of k-th participant γ_jk, a random effect for the k-th participant δ_k, and an error term ε_ijk.γ_jk, δ_k, and ε_ijk are assumed to be independent, and follow a normal distribution with zero mean. The total variances of iFC can be decomposed into three parts: 1) variance between participants ( = Var[δ]), 2) variance between conditions for the same participant ( = Var[γ]), and 3) variance of the residual, indicating variance between sessions (= Var[ε]). The reliability of the iFC across conditions can be calculated as intra-class correlation coefficients as follows (Fig. 6, left):

(2)

and across sessions as follows Fig. 6, right)):

(3)

Figure 6.

ICCs quantifying between-condition reliabilities (left) and between-session reliabilities at the connection-level. ICC values were obtained using a hierarchical linear mixed model. These connection-level values are grouped on the vertical and horizontal axes based membership of ICN. No overlap indicates that the voxel did not spatially overlap with any ICN.

Findings revealed an impressively high degree of between-condition reliability for most connections (percentiles: 50th: 0.854, 75th: 0.955, 95th: 1), as opposed to between-session (i.e., test-retest) reliability, which was notably lower (percentiles: 50th: 0.270, 75th: 0.355, 95th: 0.507). Of interest, between-condition reliability tended to be lowest in the visual and somatosensory networks, each of which would be expected to vary in a systematic way across conditions due to differences in visual stimulation (movie > inscapes > flanker > rest) and motor demands (flanker > all other conditions).

Regarding test-retest reliability, follow-up analyses also looked at connection-wise ICC for each of the stimulus/task conditions separately using a linear mixed model (as implemented in R) (see Fig. 7), finding similar ranges of ICC scores across conditions, though with some notable differences (e.g., higher ICC for visual network in movies and inscapes; higher frontoparietal ICCs in flanker task and rest). Table 5 gives a breakdown of the summary statistics for each scan condition, within network conditions, and between network connections. Additionally, we used image-wise correlation coefficient (I2C2) [69] to look at functional networks and their interactions from a multivariate perspective. As can be seen in Fig. 7, a high degree of correspondence was noted between the strength of the reliability for a given network (i.e., I2C2) and the strengths of the reliabilities for the individual edges in the network (i.e., ICC).

Figure 7.

Connection-wise ICC values across all subjects, sessions, and scan conditions (top), as well as network-wise calculations of test-retest reliability carried out using the I2C2, again across all subjects, sessions, and scan conditions (bottom).

Table 5.

Displayed here are summary statistics of the distribution of ICC values from the test-retest reliability analysis of each scan condition. Shown are the mean, standard deviation, and 95% CI of ICC values for within network and between network connections.

	Within network				Between network
	Mean	SD	95% CI		Mean	SD	95% CI
Rest	0.349	0.148	0.345	0.352	0.272	0.130	0.268	0.276
Inscapes	0.332	0.152	0.328	0.336	0.218	0.127	0.214	0.222
Movie	0.356	0.151	0.352	0.360	0.261	0.125	0.257	0.265
Flanker	0.366	0.178	0.362	0.371	0.277	0.148	0.272	0.282

Finally, to gain insights into the effects of scan duration on test-retest reliabilities, we repeated ICC and I2C2 analyses using 10, 20, and 30 min of scan data across 4 pseudo-sessions (i.e., for 20 min, we combined data from 2 sessions; for 30 min, we combined data from 3 sessions). Consistent with prior reports, our analyses revealed notable improvement of ICC and I2C2 values with longer scans, particularly when increasing from 20 to 30 min (see Figs 8 and 9).

Figure 8.

Impact of scan duration on test-retest reliability at the connection level. We randomly sampled sessions, and concatenated the time series temporally to create pseudosessions of 10, 20, and 30 min of data. For each of the pseudosession durations, we depict ICCs obtained for each scan condition. Note: across durations, the number of pseduosessions was held constant at four.

Figure 9.

Impact of scan duration on test-retest reliability at the network level. We randomly sampled sessions to create pseudosessions of 10, 20, and 30 min of data. For each of the pseudosession durations, we depict I2C2 obtained for each scan condition. Note: across durations, the number of pseduosessions was held constant at four.

Concluding remarks

These illustrative analyses highlight the value of these data for addressing questions regarding between-condition and between-session reliability. Beyond quantifying reliabilities for connectomic indices, the data available can also be used by investigators to answer questions regarding minimum data requirements (e.g., number of timepoints) and optimal image processing strategies. Finally, it is worth noting that the availability of naturalistic viewing states (Inscapes, movie clips) in the resource will give resting state fMRI-focused investigators an opportunity to explore the added value of these states for calculating iFC and more (e.g., exploration of inter-subject correlation and inter-subject functional connectivity [23, 70]).

Software and availability

The Configurable Pipeline for the Analysis of Connectomes (C-PAC) was employed to carry out the image processing for the analyses included in the text and can be downloaded from the C-PAC web page [63]. Additionally, the configuration file containing the settings for C-PAC is available for download [64]. C-PAC is a python-based software, which can run on Unix-based platforms. Windows is not supported. Note, not all of C-PAC's dependencies are python based. A list of the dependencies can be found on the C-PAC web page under the installation section [71]. C-PAC operates under a BSD 3-Clause license. Snapshots of the C-PAC code and other supporting metadata are openly available in the GigaScience repository, GigaDB [72].

Availability of supporting data

The HBN-SSI is available online [36]. Further supporting metadata are openly available in the GigaScience repository, GigaDB [72].

Abbreviations

AFNI, analysis of functional neuroimages; ANT, Advanced Normalization Tool; CPAC, Configurable Pipeline for Analysis of Connectomes; DKI, diffusion kurtosis imaging; EFC, entropy focus criterion; FBER, foreground-to-background energy ratio; FSL, FMRIB Software Library; FWHM, full width half maximum; GCOR, global correlation; GSR, ghost-to-signal ratio; HBN, Healthy Brain Network; I2C2, image intra-class correlation coefficient; ICC, intra-class correlation coefficient; ICN, Intrinsic Connectivity Network; iFC, intrinsic functional connectivity; Mean FD, mean frame-wise displacement; MNI, Montreal Neurological Institute; MPRAGE, Magnetization Prepared Rapidly Acquired Gradient Echo; QAP, quality assurance protocol; QI1, percent artifact voxels; R-fMRI, resting state functional magnetic resonance imaging; SNR, signal-to-noise ratio; SSI, Serial Scanning Initiative.

Ethics approval and consent to participate

All experimental procedures were performed with approval of the Chesapeake Institutional Review Board and only after informed consent was obtained.

Consent for publication

All participants consented to have their data shared.

Competing interests

The authors declare that they have no competing interests.

Funding

This work was supported by The Healthy Brain Network (http://www.healthybrainnetwork.org), and its supporting initiatives are supported by philanthropic contributions from the following individuals, foundations, and organizations: Lee Alexander, Robert Allard, Lisa Bilotti Foundation, Inc., Margaret Billoti, Christopher Boles, Brooklyn Nets, Agapi and Bruce Burkhard, Randolph Cowen and Phyllis Green, Elizabeth and David DePaolo, Charlotte Ford, Valesca Guerrand-Hermes, Sarah and Geoffrey Gund, George Hall, Joseph Healey and Elaine Thomas, Hearst Foundations, Eve and Ross Joffe, Anton and Robin Katz, Rachael and Marshall Levine, Ke Li, Jessica Lupovici, Javier Macaya, Christine and Richard Mack, Susan Miller and Byron Grote, John and Amy Phelan, Linnea and George Roberts, Jim and Linda Robinson Foundation, Inc, Caren and Barry Roseman, Zibby Schwarzman, David Shapiro and Abby Pogrebin, Stavros Niarchos Foundation, Nicholas Van Dusen, David Wolkoff and Stephanie Winston Wolkoff, and the Donors to the Brant Art Auction of 2012. MPM is a Randolph Cowen and Phyllis Green Scholar.

Author contributions

Conception and experimental design: JE, LP, MPM, RCC, SC, SG, and TV. Implementation and logistics: DOC, NVP, RCC, SC, and TV. Data collection: AB, MK, NGV, and YO. Data informatics: DOC, JP, and RCC. Data analysis: DOC, LA, MPM, and TX. Drafting of the manuscript: DOC, MPM, RCC, and TX. Critical review and editing of the manuscript: all authors contributed equally to the critical review and editing of the manuscript.

Author details

¹Child Mind Institute Healthy Brain Network, New York, NY. ²Center for Biomedical Imaging and Neuromodulation, Nathan S. Kline Institute for Psychiatric Research, Orangeburg, NY. ³Yale University, New Haven, CT. ⁴City College of New York, New York, NY. ⁵The Graduate Center of the City University of New York, New York, NY. ⁶Massachusetts Institute of Technology, Cambridge, MA.