Characterization of Task-free and Task-performance Brain States via Functional Connectome Patterns

Abstract

Both resting state fMRI (R-fMRI) and task-based fMRI (T-fMRI) have been widely used to study the functional activities of the human brain during task-free and task-performance periods, respectively. However, due to the difficulty in strictly controlling the participating subject's mental status and their cognitive behaviors during R-fMRI/T-fMRI scans, it has been challenging to ascertain whether or not an R-fMRI/T-fMRI scan truly reflects the participant's functional brain states during task-free/task-performance periods. This paper presents a novel computational approach to characterizing and differentiating the brain's functional status into task-free or task-performance states, by which the functional brain activities can be effectively understood and differentiated. Briefly, the brain's functional state is represented by a whole-brain quasi-stable connectome pattern (WQCP) of R-fMRI or T-fMRI data based on 358 consistent cortical landmarks across individuals, and then an effective sparse representation method was applied to learn the atomic connectome patterns (ACP) of both task-free and task-performance states. Experimental results demonstrated that the learned ACPs for R-fMRI and T-fMRI datasets are substantially different, as expected. A certain portion of ACPs from R-fMRI and T-fMRI data were overlapped, suggesting some subjects with overlapping ACPs were not in the expected task-free/task-performance brain states. Besides, potential outliers in the T-fMRI dataset were further investigated via functional activation detections in different groups, and our results revealed unexpected task-performances of some subjects. This work offers novel insights into the functional architectures of the brain.

1. INTRODUCTION

Functional magnetic resonance imaging (fMRI) techniques have been widely used to study the functional activities and cognitive behaviors of the brain in recent years. Generally, fMRI studies can be differentiated into various categories based on the stimulus used, e.g., resting state fMRI (R-fMRI) (task-free) (Raichle et al., 2001; Fox and Raichle, 2007) and task-based fMRI (T-fMRI) (task-performance) (Linden et al., 1999; Heeger and Ress, 2002; Calhoun et al., 2001; Koshino et al., 2005; Gailard et al., 2004). For these fMRI studies, the quality of fMRI data is vital because it strongly influences the reliability of conclusions inferred from the fMRI data (Simmons et al., 1999; Stocker et al., 2005). During fMRI scans, there are several factors which may affect the fMRI data qualities (Stocker et al., 2005), such as fMRI hardware related factors, experimental designs, and participating subject's issues (e.g., motion, lack of attention or any other unexpected cognitive behaviors which are not related to the experimental designs). A variety of fMRI data quality control studies have focused on fMRI imaging quality, which already made significant contributions to the quality assurance of fMRI data (Simmons et al., 1999; Foland and Glover, 2004; Stocker et al., 2005; Friedman and Glover, 2006). Furthermore, there were many studies that aimed to optimize and improve experimental designs, especially in event-related task fMRI studies (Anders 1999; Wager and Nichols, 2003; Savoy, 2005; Amaro and Barker, 2006). These task-based experimental designs were expected to provide a statistically meaningful contrast between the neuronal activity at task-performance and the background condition. In addition, the reliability and variability of the results based on fMRI data were investigated and analyzed in a variety of papers (McGonigle et al., 2000; Specht et al., 2003; Schuyler et al., 2010).

An important but underexplored issue in T-fMRI/T-fMRI is how to ascertain the performance of the participating subject's functional brain behaviors during fMRI scans. It is an ideal case that researchers design experimental fMRI paradigms appropriately such that participating subjects collaboratively pay close attention and strictly responds to stimulus events. However, it is difficult to strictly control every subject's mental status and cognitive behaviors all the time during fMRI scan sessions. As a consequence, the analysis results derived from fMRI data based on the assumption that every participating subject was strictly complying with the experimental design could be doubtful to some extent, due to the critical lack of effective approaches that can accurately assess the performance of the participant during fMRI scans. For instance, if a participating subject's brain was actively thinking during R-fMRI scans, how different this R-fMRI data will be from other strict resting state fMRI data acquired during task-free states? Similarly, if a participating subject's brain was in resting state, that is, not following the administered task-performance paradigm, how different this T-fMRI data will be from other strict task-based fMRI data acquired during task-performance states? If these differences are substantial, can we quantitatively characterize and automatically differentiate those unreliable R-fMRI/T-fMRI data from strict R-fMRI/T-fMRI data? The answers and possible solutions to these questions can significantly enhance our understanding of the function mechanisms of the brain and help us better monitor and control the quality of R-fMRI/T-fMRI data in the subsequent quantitative analysis, e.g., inference of resting state networks (RSNs), functional connectivity analysis, and task-based functional region localization.

In response to the above unanswered questions, this paper presents a novel computational framework to characterize the brain's task-free and task-performance functional states by learning from both R-fMRI and T-fMRI datasets. Our computational pipeline is composed of three major components. First, the structural connectome of each subject is constructed via our recently developed and validated 358 Dense Individualized and Common connectivity-based Cortical Landmarks (DICCCOL) (Zhu et al., 2012) based on DTI data. Second, a sliding window approach was employed to construct each subject's temporally varying functional connectomes based on the structural connectome and coincident fMRI data, which was further interactively divided into quasi-stable segments. Third, we represent the brain's functional status by a set of whole-brain quasi-stable connectome patterns (WQCP), and then apply the Fisher discriminative dictionary learning (FDDL) sparse coding approach (Yang et al., 2011) to learn the atomic connectome patterns (ACP) of both task-free and task-performance states from large-scale temporally segmented WQCPs. Essentially, the integration and pooling of many WQCPs from different brains are enabled by the DICCCOL system (Zhu et al., 2012), which provide intrinsic structural and functional correspondences across different individuals and populations. Consequently, the WQCPs from the different temporal segments of multiple brains can be readily pooled and effectively compared via sparse coding and representation methods, which will learn the most descriptive atomic patterns in forming a combined meaningful dictionary to represent and discriminate those WQCPs. Therefore, the major methodological novelties of this paper lie in the DICCCOL-based structural/functional connectome construction and the sparse coding and representation of functional brain states.

The computational pipeline has been applied on two separate multimodal DTI/R-fMRI/T-fMRI datasets of twenty-six healthy adolescents and thirty-seven healthy adults. Our experimental results demonstrated that the learned ACPs for R-fMRI and T-fMRI datasets are substantially different, as expected, and that the ACPs learned from independent R-fMRI datasets of healthy adolescents and adults are quite reproducible. Importantly, a certain portion of ACPs were overlapped between the two datasets, suggesting that some participating subjects were not in the expected task-free/task-performance states during the R-fMRI/T-fMRI scans and should be considered as potential outliers in the following steps of data analysis. As examples, some potential outlier WQCP segments from the T-fMRI dataset within resting state ACP patterns were further examined. Our activation detection results on T-fMRI datasets demonstrated that the subjects with outlier resting state ACPs have almost no group activation regions, while the subjects without outlier resting state ACPs exhibit consistent task-related activations. This result suggests that the ACP patterns could be potentially used to infer whether the participating subjects were following the administered experimental tasks or not during T-fMRI scans. In general, our experimental results revealed interesting phenomena of the regularity, diversity and dynamics of functional connectomes in task-free and task-performance states. Notably, an early short version of this methodology was presented in the MICCAI 2012 conference (Zhang et al., 2012).

2. MATERIALS AND METHODS

2.1. Overview

The flowchart of the proposed computational framework is summarized in Fig. 1. First, the 358 consistent DICCCOL landmarks that have been discovered and validated in our recent study (Zhu et al., 2012) are located in the DTI data of each brain (green bubbles in the left panel of Fig. 1) via an effective functional landmark prediction approach (Zhang, et al., 2011; Zhu et al., 2012). After pre-processing (Zhu et al., 2011b; Zhu et al., 2012), both R-fMRI and T-fMRI images are co-registered into the DTI space using FSL FLIRT, and the representative R-fMRI/T-fMRI time series in each DICCCOL were extracted (Fig. 1(1)). Second, by using a sliding time window approach (Li et al., 2012c), the dynamic functional connectivity time series between each pair of DICCCOLs are measured and thus the time-varying functional connectomes are constructed (Zhu et al., 2012). Furthermore, the cumulative functional connectivity strength of each landmark with all other DICCCOLs at each time point is summed, and the functional connectome is thus compactly represented by a column as shown in Fig. 1(2). Third, as extensive observations show that the functional connectome strengths are relatively stable in a continuous time period, therefore, they are interactively segmented into quasi-stable segments (called WQCP above), which form a set of WQCP training samples (Fig. 1(3)). Fourth, the WQCP samples from both R-fMRI and T-fMRI datasets were pooled together for sparse representation and classification via the Fisher discriminative dictionary learning (FDDL) method (Fig. 1(4)) (Yang et al., 2011) and a set of representative ACPs were obtained. Finally, each WQCP segment is classified to one ACP and the distributions of ACPs can be examined at the individual and population levels, as illustrated in Fig. 1(5).

Fig. 1.

The flowchart of our computational framework. (1) R-fMRI/T-fMRI signal extraction for each DICCCOL; (2) Construction of functional connectomes; (3) Interactive segmentation of WQCP training samples; (4) FDDL sparse learning and classification. (5) Examination of the distributions of ACPs at the individual and population levels.

2.2. Data Acquisition and Pre-processing

Two populations including twenty-six healthy adolescents (ages 11~17) and thirty-seven healthy adults (ages 23~46) were recruited in Sichuan, China, under the IRB approvals of the Second Xiangya Hospital and the Central South of University. Multimodal DTI/R-fMRI/T-fMRI datasets for each participant were acquired on a 3T MRI scanner in West China Hospital, Huaxi MR Research Center, Department of Radiology, Chengdu, China. Acquisition parameters were as follows: DTI: 256×256 matrix, 3mm slice thickness, 240mm FOV, 50 slices, 15 DWI volumes, b-value=1000; fMRI: 64×64 matrix, 4mm slice thickness, 220mm FOV, 30 slices, TR=2s. The pre-processing of DTI data included brain skull removal, motion correction and eddy current correction (Zhu et al., 2011b). After that, streamline fiber tracking was conducted via the MedINRIA (http://www-sop.inria.fr/asclepios/software/MedINRIA/). Then, white matter (WM) and grey matter (GM) tissue segmentation were performed on DTI data via the approaches in Liu et al., 2007. The cortical surface was reconstructed based on the WM tissue map using the marching cubes algorithm (Liu et al., 2008). Pre-processing steps of the R-fMRI data include brain skull removal, motion correction, spatial smoothing, temporal pre-whitening, slice time correction, global drift removal, and band pass filtering (0.01Hz~0.1Hz) (Li et al., 2010). Notably, we used the DTI image space to define connectome nodes and fMRI data was mapped to the DTI space. Our rationale is that both fMRI and DTI use EPI (echo planar imaging) sequences, and their geometric distortions tend to be similar (Li et al., 2010). As a result, the misalignment between DTI and fMRI images is much less than that between T1 and fMRI images (Li et al., 2010). To ensure a good mapping of fMRI data to DTI space, we applied the anatomical and connectional constraints (Li et al., 2010) in the mapping process.

In addition to the above DTI and R-fMRI datasets, visual block-based T-fMRI data for the adolescent group and auditory block-based T-fMRI data for the adult group were acquired using similar scan parameters as those in R-fMRI. The visual task paradigm was designed as follows (Fig. 2a). Each block contains a 30s rest period and a 20s task period when video stimuli were presented to the scanned subject (Zhang et al., 2011). Here, the “fix” represents fixation on a “+” sign for ten seconds, Nu represents neutral video, Ne represents negative video showing earthquake scenes, Po represents positive video showing the rescue and encouraging scenes of the earthquake rescue, and R indicates resting state for 30 seconds (including 8 seconds in the beginning as one sentence on the screen for informing the subject to make a choice and 22 seconds for fixation on a “+” sign). Totally, this paradigm lasts 440 seconds.

Fig. 2.

Illustrations of block-based task paradigm designs. (a) Visual task paradigm. (b) Auditory task paradigm.

The paradigm design of the auditory task is illustrated in Fig. 2b. There are three blocks in total and in each block the subject listened to either neutral story (the first and third block) or earthquake story (the second block), and then followed by 30 seconds imagination according to which story they had just listened to and then another 60 seconds of resting state period. The pre-processing steps of T-fMRI data are similar to those in Zhu et al., 2011b.

2.3. DICCCOL-based Structural Connectome

In order to construct functional connectomes, we used the dense map of 358 consistent DICCCOL landmarks reported in Zhu, et al., 2012 to construct structural connectomes. The visualizations of all of these 358 DICCCOLs have been released online at: http://dicccol.cs.uga.edu. One example of 358 DICCCOLs distributed on a cortical surface is shown here in Fig. 3. As the 358 DICCCOLs are very consistent and reproducible, e.g., replicated across over 240 brains (Zhu et al., 2012), they can be quite accurately predicted in the DTI data of individual brains. For a new subject with DTI data, we first warp the brain to the template via FSL FLIRT and obtain the initial locations of 358 DICCCOL landmarks in the subject. Similarly, we extract the white matter fiber bundles emanating from the small regions around the neighborhood of each landmark, and each neighbor region will serve as the candidate for landmark optimization. Also, the fiber bundles of the candidate landmarks of the subject will be represented via the trace-maps (Zhu et al., 2012). For each landmark to be optimized in the testing brain, the trace-map distances between the candidate landmarks and the DICCCOL landmarks in the template subjects are measured. The predicted location of each DICCCOL in the new brain is determined by selecting one candidate landmark that has the least group-wise fiber connection variance (Zhu et al., 2012). Thus, based on the DTI data of each subject, we can predict 358 DICCCOL landmarks, which offer the structural substrates for the construction of structural connectomes. Here, the fiber connection strength between any pair of 358 DICCCOLs is used as the connectome edge (Zhu et al., 2012), which is shown in Fig. 3b as an example. It should be noted that we used the same number of DICCCOL landmarks in both groups of R-fMRI and T-fMRI datasets in order to ensure that their connectome patterns can be integrated and compared in the following sections.

Fig. 3.

(a) An example of 358 DICCCOL landmarks on a cortical surface. (b) An example of DICCCOL-based structural connectome.

2.4. Whole-brain Quasi-stable Connectome Patterns (WQCPs)

After all of the 358 cortical landmark locations were predicted in each subject's brain, the fMRI time series for each landmark can be extracted from both of the R-fMRI and T-fMRI datasets (Section 2.2). To investigate the temporal dynamics of the large-scale functional connectomes in task-free and task-performance states, a sliding time window approach (Li et al., 2012c) was applied here. Specifically, for each time point t, the functional connectivity (FC) between each pair of DICCCOLs (i and j) is defined as follows:

$$F{C}_{i,j}=PearsonCorrelation(W_i,W_j),F{C}_{i,j}=0\text{if}i=j$$ (1)

$$\begin{array}{cc}\hfill W_i=& [s_{i,t},s_{i,t+1},\dots ,s_{i,t+l-1}]\hfill \\ \hfill W_j=& [s_{j,t},s_{j,t+1},\dots ,s_{j,t+l-1}]\hfill \end{array}$$ (2)

where W_i and W_j represent the fMRI time series with l-time-points window length of DICCCOL i and DICCCOL j respectively, where l was set to 14 volumes in this study; s_i,t and s_j,t are the corresponding fMRI signal strengths of DICCCOL i and DICCCOL j at the time point t; l is the length of the time window. FC is measured using the absolute value of Pearson correlation coefficient between the two windowed fMRI signals W_i and W_j at each time point. Then, the dynamic functional connectivity time series between each pair of landmarks are obtained, which can be represented by a 3D 358×358×time matrix.

To have a compact representation of the functional connectome, the cumulative connectivity strength of each landmark is measured by summing the absolute value of all the functional connectivities between this landmark and all of the other landmarks. This operation converts the functional connectome matrix (358×358) at each time point into a representative connectome vector (358×1). Thus, we obtained a 2D temporally dynamic functional connectome matrix for each participant (see Fig. 4). In Fig. 4, the horizontal axis represents time points and the vertical axis is the cumulative functional connectivity strength of each landmark, which is color-coded by the color-bar on the right. From this figure, it can be clearly observed that the connectome strength keeps relatively stable in a continuous time period. Therefore, the dynamic connectome matrix can be temporally segmented into a series of whole-brain quasi-stable connectome patterns (WQCP), which were implemented interactively in this study. The manual segmentations were performed and cross-validated by two experts. Since the functional connectome pattern within each WQCP is quasi-stable, each WQCP segment is averaged among the time axis, resulting in a single WQCP vector. Then, all the WQCP segments and vectors from both R-fMRI and T-fMRI data of each subject were pooled together as training samples, and then were represented using sparse coding methods in Section 2.5. Notably, in this paper, the above methodologies were first applied on the R-fMRI and visual T-fMRI dataset of the adolescent participants, and then on the adult R-fMRI and auditory T-fMRI dataset, respectively.

Fig. 4.

The dynamic functional connectivity strength time series and WQCP segments (separated by dash black lines) from two fMRI datasets. (a) A resting state fMRI case. (b) A visual T-fMRI case. The horizontal axis of each image represents time points and the vertical axis represents the cumulative functional connectivity strength time series of each DICCCOL.

2.5. Fisher Discriminative Dictionary Learning for Sparse Representation of WQCP

In recent years, sparse representation of signals has attracted significant attention in applications such as signal separation (Li et al., 2004; Starck et al., 2005), signal denoising (Elad and Aharon, 2006), signal classification (Huang and Aviyente, 2006; Wright et al., 2009; Zhang and Li, 2010; Ramirez et al., 2010; Yang et al., 2010) and image restoration (Mairal et al., 2008; Ranzato et al., 2006). It has been widely recognized that sparse coding exhibits very good performance in image analysis, especially in image classification (Wright et al., 2009). Basically, sparse coding aims to search for the most compact representation of the signal in terms of a sparse linear combination of atoms in an over-complete dictionary. In sparse representation, the performance heavily depends on the quality of the dictionary. A critical issue is that the number of atoms of a dictionary can be very big, which increases the coding complexity and may not fully exploit the discriminative information hidden in the training samples (Yang et al., 2011). Many dictionary learning methods (e.g., Aharon et al., 2006; Rodriguez and Sapiro, 2007; Mairal et al., 2008; Pham and Venkatesh, 2008; Zhang and Li, 2010; Ramirez et al., 2010; Yang et al., 2010) have been proposed for the purpose of learning an over-complete dictionary from the training samples, over which the given signal could be well represented. Actually, to reconstruct the signal accurately is not enough for signal classification, during which the discrimination of the representation for the given signal classes is also very important (Huang et al., 2006). Thus, both the reconstruction and discrimination terms were considered in the recent sparse coding based classification studies. This paper adopted an effective Fisher discriminative dictionary learning (FDDL) based sparse representation methodology (Yang et al., 2011) and tailored it for extracting atomic connectome patterns (ACP) of functional brain states from the original combined WQCP samples. In general, the FDDL employs the Fisher discrimination criterion to learn a structured dictionary composed of small class-related sub-dictionaries. During the learning process, the sparse coding coefficients were trained to not only have small within-class scatter but also have large between-class scatter. Each sub-dictionary was formed finally to well represent the training samples from the corresponding class but have poor representation ability for samples from other classes. Thus, both the reconstruction error and the coding coefficient will be discriminative when finished.

Here, the learned dictionary is denoted by D = [D₁, D₂,..., D_c] , where D_i is the sub-dictionary corresponding to the class i, and c is the total number of classes learned. D_i has the dimension of 358×N_i, where N_i is the size of the i^th sub-dictionary which varies across sub-dictionaries. The sum of the size of all sub-dictionaries is equal to the total number of WQCP samples. An example visualization of the sub-dictionaries is shown in Supplemental Fig. 5. A = [A₁, A₂,..., A_c] represents all training WQCP vector samples, where A_i is the sub-set of the training WQCP vector samples belonging to the class i. In addition, X = [X₁, X₂ ,..., X_c] represents the coding coefficient matrix of A over D. The sub-dictionaries is initialized by the results from K-means clustering on the WQCPs for a rough estimation of the data distribution. Then the FDDL model is represented as follows:

$$J_{(D,X)}=\arg {\min }_{(D,X)}\{r(A,D,X)+{\lambda }_1{\Vert X\Vert }_1+{\lambda }_{2}f\left(X\right)\}$$ (3)

FDDL aims to generate a comprehensive over-complete dictionary through updating the dictionary and the coding coefficients. It tries to minimize the total sum of three items: the first term r(A,D, X) is called the discriminative fidelity term, the second term ∥X∥₁ is the sparsity constraint and the last term f(X ) is a Fisher discrimination constraint imposed on the coefficient matrix. λ₁ and λ₂ are scalar parameters for trade-off between discriminative fidelity, sparsity, and discrimination capability. The value of them (0.01 and 0.02) are experimentally determined by minimizing the overall error variance during the training stage. The detailed information of the parameter selection is listed in Supplemental Table 1. The first discriminative fidelity term can be written by the following:

$$r(A_i,D,X_i)={\Vert A_i-DX\Vert }_F^2+{\Vert A_i-D_i{X}_i^i\Vert }_F^2+\sum \begin{array}{c}c\hfill \\ i=1\hfill \\ j\ne i\hfill \end{array}{\Vert D_j{X}_i^j\Vert }_F^2$$ (4)

where $X_i=[X_i^1;\dots X_i^j;\dots ;X_i^c]$ and $X_i^j$ is the coding coefficient of A_i over D_j. The first term on the right of Eq. (4) indicates that the dictionary D should have good representation ability of A_i. Meanwhile it is expected to well represent A_i by D_i (the second term) but not by D_j (j ≠ i) (the last term).

$$f\left(X\right)=tr\left(S_W\left(X\right)\right)-tr\left(S_B\left(X\right)\right)+\eta {\Vert X\Vert }_F^2$$ (5)

$$S_W\left(X\right)=\sum \begin{array}{c}c\hfill \\ i=1\hfill \end{array}{\sum }_{x_k\in X_i}(x_k-m_i){(x_k-m_i)}^T$$ (6)

$$S_B\left(X\right)=\sum \begin{array}{c}c\hfill \\ i=1\hfill \end{array}{n}_i(m_i-m){(m_i-m)}^T$$ (7)

The discriminative coefficient term f(X) is introduced here to make the dictionary discriminative for the training samples and it is achieved by minimizing the within-class scatter S_W(X) and maximizing the between-class scatter S_B(X) of X, and the parameter η is set to 1 for both algorithm convex and maximizing the discriminability as described in Yang et al., 2011. The expressions of S_W(X) and S_B(X) are also given in Eq. (6) and Eq. (7), where m_i and m are the mean vectors of X_i and X respectively, and n_i is the number of samples in A_i. The last term ${\Vert X\Vert }_F^2$ is an elastic term used to make f(X) convex and stable and ɿ is a parameter. The detailed optimization procedures of FDDL model and the discussion of local minimum were presented in Yang et al., 2011.

Considering that the FDDL methodology takes account of both reconstruction capability and discriminative capability of the training samples, both the reconstruction error and the coding coefficient discriminative power should also be considered in the classification scheme to obtain more accurate results. Specifically, when D is learned, there are two classifiers that can be used: global classifier (GC) and local classifier (LC) via coding the given samples over the learned dictionary. This study adopted the GC to perform the sparse coding learning and classification of WQCP samples. For one input WQCP vector sample y, first, the sparse coding coefficients can be obtained by solving the following:

$$\stackrel{⌢}{\alpha }=\arg {\min }_{\alpha }\{{\Vert y-D\alpha \Vert }_2^2+\gamma {\Vert \alpha \Vert }_1\}$$ (8)

where $\stackrel{⌢}{\alpha }=[{\stackrel{⌢}{\alpha }}_1;{\stackrel{⌢}{\alpha }}_2;\dots ;{\stackrel{⌢}{\alpha }}_c]$ and ${\stackrel{⌢}{\alpha }}_i$ is the coefficient vector linked to the D_i. γ is a constant parameter. Then, the sample y is attributed to the class, the sub-dictionary associated with which has the minimum representation error defined by:

$$e_i={\Vert y-D_i{\stackrel{⌢}{\alpha }}_i\Vert }_2^2+w\cdot {\Vert \stackrel{⌢}{\alpha }-m_i\Vert }_2^2$$ (9)

where the first term is the reconstruction error using class i; and the second term is the distance between $\stackrel{⌢}{\alpha }$ and m_i, which is the learned mean vector and w is a constant.

This study used and tailored the above FDDL methodology to train the combined task-free and task-based WQCP vector samples to achieve a compact and meaningful dictionary. In particular, each sub-dictionary or class is associated with an ACP. The total number of ACPs (and thus sub-dictionaries) is experimentally set to be 16. The number was determined based on the experiments of trying different numbers (9-20) of ACPs used in the training stage. In each experiment with a unique number of ACPs, the Bayesian Information Criteria (BIC) (Schwarz, 1978) was obtained by the following equation:

$$BIC=n\ln \left({\sigma }_e^2\right)+k\ln \left(n\right)$$ (10)

where ${\sigma }_e^2$ is the summed variance of each sub-dictionary, n is the total number of segments used for the training, and k is the number of sub-dictionaries. As the error variance is monotonically decreasing as more sub-dictionaries are used for the training, the number of sub-dictionaries could be determined by the optimal k that minimizes the BIC value. The BIC values could be found in Supplemental Fig. 3.

Afterward, we used the GC to classify the WQCP vector samples into 16 ACP patterns and acquired the distributions of all WQCPs for both task-free and task-performance samples in each ACP. Finally, each ACP pattern is described and represented by the averaged center of all of the WQCP segments and vectors belonging to this pattern.

3. RESULTS

In this section, we applied the methods in Sections 2.3-2.5 on the datasets in Section 2.2, and qualitatively and quantitatively examined the ACPs derived from R-fMRI and visual T-fMRI datasets in Sections 3.1-3.2. The potential outliers detected in the quantitative analyses were further examined via both group-wise and individual activation detections from T-fMRI datasets in Section 3.3. The reproducibility of the learned ACPs and their distributions in the first dataset was assessed via an independent auditory T-fMRI/R-fMRI dataset in Section 3.4. Finally, comparison of the FDDL method with non-discriminative dictionary learning model is provided in Section 3.5.

3.1. Visual Evaluation of ACPs in R-fMRI and T-fMRI Datasets

Both R-fMRI and visual T-fMRI data of 26 adolescents were analyzed following the steps in Sections 2.3-2.5. First, by using the method in Section 2.3, the 358 consistent DICCCOL landmarks for each subject were predicted and located based on DTI data. Then, according to the methods in Section 2.4, the dynamic functional connectome strength matrices were calculated and interactively segmented, and thus we obtained a collection of totally 1149 WQCP segments and corresponding averaged WQCP vectors. Specifically, there were 474 samples from R-fMRI data and 675 samples from visual T-fMRI data. Subsequently, all of the 1149 samples were trained and classified by the FDDL method in Section 2.5. Finally, a structured combined dictionary containing 16 sub-dictionaries corresponding to 16 ACPs was learned and each WQCP sample was assigned into one ACP. The visualizations of these 16 sub-dictionaries are shown in Supplemental Fig. 5.

We visualized all of these 16 ACPs using their averaged WQCP centers in Fig. 5a. For each WQCP vector sample (358×1), it is associated with an averaged 358×358 functional connectome matrix by averaging all the 358×358 functional connectome matrices in the corresponding WQCP segments. The functional connectome strength of each ACP was calculated by averaging all the 358×358 functional connectivities of each WQCP sample associated with it, i.e., each WQCP center. Fig. 5a shows the visualization result of the 16 ACP patterns represented by sixteen 358×358 functional connectome matrices, and the top 100 high functional connectivity pairs between DICCCOLs in each ACP in Fig. 5a are visualized on the cortical surface in Fig. 5b. It should be noted that the ACPs #1-#9 in the red frame in Fig. 5a contain the WQCPs dominantly from the visual T-fMRI data, while the ACPs #11-#16 in the blue frame primarily include the WQCPs from R-fMRI data. It is evident that the ACPs learned from visual T-fMRI and R-fMRI datasets are remarkably different. This result suggests that it is feasible to employ functional connectomes to characterize and differentiate the functional brain activities in task-performance and task-free states, which is the underlying hypothesis of this work. Notably, though the ACPs in visual T-fMRI data are quite variable (Fig. 5a), the top high functional connectivities in these the ACPs tend to localize in the visual cortex, as highlighted by the red arrows in Figs. 5a-5b. This result is quite reasonable given that the visual task administered to the subjects is expected to significantly stimulate the vision system. Furthermore, the variable ACPs (#1-#9 in Figs. 5a-5b) also suggest the complex patterns of functional connectomes during visual task performance, which merits further investigation in the future.

Fig. 5.

Sixteen ACP patterns in both R-fMRI and T-fMRI datasets. (a) Representation of each ACP by the averaged 358×358 functional connectomes. It is a compact representation of task-free and task-performance states. (b) Visualization of the top 100 high functional connectivity pairs between two DICCCOLs on the cortical surface.

Interestingly, the ACPs in the resting state (#11-#16 in Figs. 5a-5b) are even more variable and dynamic than those in task-performance state. For instance, in ACP #11 the whole functional connectome is strongly connected (yellow and red colors in Fig. 5a), while in ACP #15 the whole functional connectome is weakly connected (cyan and blue colors in Fig. 5a). In contrast, in ACP #12 and #14, certain subnetworks are more strongly connected (Fig. 5a). This observation is consistent with recent literature reports that resting state functional networks undergo remarkable temporal dynamics as seen in R-fMRI data (e.g., Chang and Glover, 2010; Li et al., 2012c; Smith et al., 2012), although the characterizations and mechanisms of those temporal dynamics are unknown to date. Despite the variable patterns of functional ACPs in resting state, the top high functional connectivities within the ACPs of task-free state tend to be localized in the default mode network (Raichle et al., 2001; Fox and Raichle, 2007), as illustrated by the yellow circles in Fig. 5b.

To further explore the neuroscience interpretation of the ACPs in Fig. 5a, the functional connectivities with strength over 0.5 in each ACP were visualized on the cortical surfaces in Supplemental Fig. 1. In Supplemental Fig. 1, 358 DICCCOL landmarks are represented by an inner ring of color-coded nodes, connections are represented by edges, and 22 most commonly used functional networks are represented by 22 outlier rings of colored nodes. Each colored node in each ring of 358 nodes stands for a functional network. The visualizations in Supplemental Fig. 1 further confirm the findings in Fig. 5, as discussed above. In particular, these results demonstrate that functional connectome is an effective approach to representation and characterizations of functional brain activities in both task-free and task-performance states, which is one of the major methodological contributions of this work.

3.2. Quantitative Analysis of ACPs in R-fMRI and T-fMRI Datasets

To quantitatively describe the ACPs, the distributions of both task-free and visual task-performance WQCP samples classified to those 16 ACPs are shown in Fig. 6. The horizontal axis indexes the 16 ACPs discriminated by the learned FDDL dictionary. Numbers 1~16 corresponds to the sixteen patterns of ACP in Fig. 5. The vertical axis represents the percentages of WQCP samples distributed in each ACP for task-free (indicated by blue circles in Fig. 6) and visual-task-performance (indicated by red squares in Fig. 6) datasets respectively. From Fig. 6, it can be clearly observed that the task-free and task-performance periods show quite different distribution patterns. In total, there are 94.2% of task-performance WQCP samples distributed in ACPs #1~#9, while there are 97.5% task-free WQCP samples distributed in the ACPs #11~#16, which implies that ACPs #1~#9 (in the red frame in Fig. 5) contain dominantly the task-performance patterns and ACPs #11~#16 (in the blue frame in Fig. 5) primarily include task-free ACP patterns. For ACP #10 (Fig. 5a), the task-free and task-performance WQCP samples overlap, as highlighted by the black arrow in Fig. 6. The percentages of ACP #10 in both resting-state and task-performance WQCPs are less than 3%. It is interesting that the functional connectivities in ACP #10 are globally high, as shown by Fig. 5a(10). The neurobiological basis of this highly active functional connectome pattern remains to be elucidated in the future. From a technical perspective, these overlapping WQCP samples impose significant difficulty to decide which functional state (task-free or task-performance) they belong to, and thus we consider them as an uncertain pattern in the following steps of data analysis.

Fig. 6.

The distributions of R-fMRI and T-fMRI WQCP samples in 16 ACPs. The horizontal axis represents each ACP pattern. The vertical axis represents the corresponding percentages of R-fMRI or T-fMRI WQCP samples distributed in each ACP.

In terms of the distributions of ACP #10 in individual subjects, we found 11 WQCP samples in 8 resting state subjects and 2 WQCP samples in 2 visual task-based subjects which were classified into the uncertain ACP #10 pattern. Fig. 7 shows four examples with ACP #10 WQCP segments from both T-fMRI and R-fMRI datasets. The top two are from visual T-fMRI subjects and the bottom two come from R-fMRI subjects. Similar to Fig. 4, the horizontal axis of Fig. 7 is time points and the vertical axis represents the functional connectivity strength of the 358 DICCCOL landmarks. It should be noted that each WQCP segment corresponding to ACP #10 is marked by the black dash line box in Fig. 7. These patterns showcase the co-existence of ACP #10 in both task-free and task-performance states in Fig. 6, and further confirm the quite high global functional connectivity among different DICCCOLs in Fig. 5a(10). Interestingly, the time points of ACP #10 occurrences in Figs. 7a-7b are quite different even though the subjects are expected to follow the same block-based task paradigm. This result demonstrates that it is difficult to control the participating subject's mental status and/or cognitive behaviors during task-performance and that it is risky to assume that the participant would strictly follow the administered task paradigm, which is the underlying hypothesis of this work. Instead, the data-driven functional connectome modeling and analysis proposed in this work can be very helpful to reveal and characterize the functional brain states and their dynamics in either task-performance or task-free conditions.

Fig. 7.

WQCP samples with ACP pattern #10 in task-performance ((a) and (b)) and resting state ((c) and (d)). The ACP pattern #10 is highlighted by the dashed black boxes in (a)-(d).

In addition to the shared ACP pattern #10 among task-free and task-performance states, we further investigated the potential outliers in both resting state and task-performance WQCP samples. For instance, there is one WQCP sample in a task-free subject, but it was clustered into the ACP #4, which is considered as one task-performance ACP. In total, we found 37 WQCP segments (out of totally 675 task-performance WQCP samples) in 17 subjects that were clustered into various task-free ACPs i.e. ACP#11~16, as shown by the red squares in the right side of Fig. 6 (highlighted by the yellow arrows). Fig. 8 shows examples of the detected potential outlier WQCP segments of the 17 subjects with patterns #11~#16, as marked in black boxes in Fig. 8. The quantitative summaries of these potential outlier patterns are provided in Table 1 and the ACP index distribution of the outlier WQCPs is provided in supplementary Table 3. These results imply that the participants in our experiments exhibited relatively good resting-state performance for high quality R-fMRI data, but they did not perform equally well in visual task experiments, in that 17 of them exhibited resting state ACP patterns during the task-performance scans. This result suggests that these subjects might not follow the administered visual tasks quite well in certain periods during the T-fMRI scans, which is one of the underlying premises of this work. Thus, we should take additional caution when analyzing the task-based fMRI datasets of these 17 subjects, e.g., in T-fMRI activation detections, which will be further discussed in Section 3.3.

Fig. 8.

The detected potential outliers in task-performance subjects with ACPs #11~#16, totally 17 subjects. The horizontal axis of each image is time points and the vertical axis is the functional connectivity strength of 358 DICCCOLs. The WQCP segments with ACPs #11~#16 were marked by different black lines as shown in the last image.

Table 1.

The numbers and percentages of subjects with detected outlier ACP patterns for task-free and task-performance respectively.

ACP index	Rest Number/Percent	Task Number/Percent
Pattern #1	0	-
Pattern #2	0	-
Pattern #3	0	-
Pattern #4	1/3.8%	-
Pattern #5	0	-
Pattern #6	0	-
Pattern #7	0	-
Pattern #8	0	-
Pattern #9	0	-
Pattern #10	8/30.8%	2/7.7%
Pattern# 11	-	1/3.8%
Pattern #12	-	4/15.4%
Pattern #13	-	4/15.4%
Pattern #14	-	3/11.5%
Pattern #15	-	11/42.3%
Pattern #16	-	3/11.5%

3.3. Examination of Outlier WQCPs and T-fMRI Data

According to our analysis results in Section 3.2, there are certain WQCP segments in 17 subjects that were classified to the resting state ACP patterns. We hypothesize that this discrepancy might be due to the fact that these subjects were not well following the administered task paradigms and/or were performing different task-unrelated cognitive behaviors during the scans. In this case, if we use these subjects’ T-fMRI datasets to conduct activation detection according to the designed task paradigm, it is very likely to obtain unexpected activation results, such as very little or even no group active regions due to their unexpected task-performance during scans. To test this hypothesis, the methodology of T-fMRI activation detection using the FSL FEAT tool was applied on our visual T-fMRI dataset as follows. First, according to our analysis results in Section 3.2, the two subjects who had the uncertain pattern ACP #10 were discarded and the other datasets were divided into two groups: 8 subjects exhibiting expected task-performance patterns (ACPs #1~#9 in Fig. 5a) and 16 subjects showing unexpected task-performances detected by our methods (those have WQCP segments in ACPs #11~#16). Then, the activation detection experiments were performed on the two groups separately. Meanwhile, the T-fMRI activation detection results for individual subjects were also acquired via FSL FEAT. For comparison purpose, the whole-group activation detection result containing all 24 subjects was also obtained. Therefore, each individual activation detection result and three group-wise activation detection results (two divided groups and one whole-group) were obtained for comparison. It should be mentioned that three contrasts including NU (watch neutral videos), NE (watch negative videos of earthquake) and PO (watch positive rescue scenes after earthquake), as mentioned in Section 2.2, were considered for T-fMRI activation detections. The activation detection results are shown in Fig. 9. In Fig. 9, (a), (b), and (c) show three group activation detection results using the NU contrast for the whole 24-subject group, the 8-subject group with expected task-performance, and the 16-subject group with unexpected task-performances, respectively. It can be easily seen that the 16-subject group (Fig. 9c) with outlier WQCP segments has no activation regions at all, which demonstrates that there are no corresponding and consistent visual activations detected among these subjects. However, in the other 8-subject group's and 24-subject group's activation results, certain group-wise consistent activation patterns were detected, as shown in Figs. 9a and 9b. In addition, two sample t-tests were performed on the GLM regression results (beta value) from the 16-subjects in the with-outlier group and the 8-subjects in the without-outlier group, where the samples space is the voxels collection. The test result showed that there is significant difference between the with-outlier group and without-outlier group with p-value=0.05, indicating that the group activation results are significantly different. These T-fMRI activation detection results strongly support our hypothesis that the 16 subjects with outlier resting state ACP patterns might not strictly follow the administered visual task paradigms during their scans, compared to other 8 subjects. Figs. 9d and 9e (above the yellow dash line) provide two typical individual activation detection results from the 16-subject group, and Figs. 9f and 9g (below the yellow dash line) show two individual activation results from 8-subject group. It can be easily observed that the two subjects from the 16-subject group show completely different activation results, e.g., the top one is almost not activated (very little red regions) while the bottom one is almost activated in the whole brain (large-scale red and yellow regions). These huge variations from one to another across the 16-subject group with outlier WQCP segments lead to no group activation results at all (Fig. 9c). However, the subjects from other expected task-performance groups have consistent activation regions, which finally show meaningful group-wise activation results (e.g. the regions marked by the green circles in Figs. 9a and 9b and the green arrows in Figs.9f and 9g). The results in Fig. 9 demonstrate that functional connectomes and the derived ACPs/WQCPs are effective approaches to representing and characterizing the functional brain activities in task-free and task-performance states. Interestingly, similar results were found in the T-fMRI activation detection results using the contrast NE (very little group-wise activation regions in the 16-subject group in Supplemental Fig. 2) and contrast PO (no group-wise activation regions inthe16-subject group in the Supplemental Fig. 2).

Fig. 9.

The activation detection results of different groups and individuals, voxels are color-coded by their z-value according to the color bars. (a) The whole-group result with 24 subjects; (b) The 8-subject group with expected task-performances; (c) The 16-subject group with potential outliers in task-performances detected by our methods; (d) and (e) show two example individual activation results from the 16-subject group; (f) and (g) show two example individual activation results from the 8-subject group.

This result further demonstrates that those participating subjects with outlier WQCP segments exhibited substantially more variable brain activities that much less follow the administered task paradigms, and that the functional connectomes and ACP/WQCP are effective in characterizing and differentiating the functional brain activities in task-performance and task-free states. Furthermore, the experimental results in this section strongly support our hypothesis that participating subjects do not necessarily follow the administered tasks and inclusion of their T-fMRI data could substantially alter the group-wise activation detection results.

3.4. Reproducibility Study

To examine the reproductive of our methods, a separate R-fMRI and auditory T-fMRI datasets from 37 adult participants (Section 2.2) were analyzed using the same methods in Section 2.3~2.5. First, 656 resting state WQCP segments and 807 auditory task based WQCP segments were obtained, which were then pooled together for the FDDL training and learning (Section 2.5). Here, the number of ACPs is also set to be 16 (the same as the visual T-fMRI/R-fMRI dataset in Sections 3.1-3.3) for comparisons. Similarly, we obtained the distribution histograms of both resting state samples and auditory task-based WQCP samples in the 16 ACP patterns (Fig. 10). In Fig. 10, the horizontal axis represents each ACP, which was generated by the learning and classification procedures of FDDL from the combinations of task-free and auditory task-based WQCP training samples. The vertical axis represents the percentages of both resting state and auditory task-based WQCPs in each pattern. From this figure, it can be clearly seen that the resting state and auditory task based learning results exhibit similar ACP pattern distributions as those R-fMRI and visual T-fMRI results (Fig. 6) in Section 3.2. The numbers and percentages of subjects with outlier ACP patterns are provided in Supplemental Table 2.

Fig. 10.

The distributions of R-fMRI and auditory T-fMRI WQCPs in 16 ACPs. The meaning of the horizontal axis and the vertical axis is similar to that in Fig.6.

Since the visual tasks and auditory tasks are quite different, the reproducibility study here only focuses on the resting state ACPs for detailed analysis. In Fig. 10, there are also totally 6 ACP patterns #11~#16 that correspond exactly to the resting state patterns, as illustrated by the blue dots and curve. This result suggests the good reproducibly of our methods and FDDL clustering results in two independent R-fMRI/T-fMRI datasets of different age populations (adolescents and adults). Notably, it is interesting that there are about 13.1% of the resting state WQCPs correspond to ACP #1 that belongs to the auditory task-performance. The visualization of the functional connectome and highly connected pairs are provided in Supplemental Fig. 6. It turns out that some DICCCOLs in the auditory cortex are highly connected in ACP #1, suggesting that the participating subjects might have sporadic auditory responses even during resting state because of the constant external sound stimulus in fMRI scans.

Furthermore, the 6 resting state ACP patterns were also visualized by the 358×358 functional connectome matrix of each ACP center (see images in Fig. 11, labeled by “adult”). Interestingly, we found that the two sets of resting state ACP patterns acquired from the adolescent group (Section 3.1, labeled by “adolescent” in Fig. 11) and the adult group (this section, labeled by “adult” in Fig. 11) look very similar, and each pattern here corresponds to one resting state ACP in Fig. 5a (#11~#16). These results provide strong evidence that the methods presented in this paper have good reproducibility in different R-fMRI/T-fMRI datasets. The corresponding relationships of each resting state ACP between two different R-fMRI datasets were summarized in Fig. 11 and the percentages of each pattern are also marked below each sub-figure. Notably, the minor difference between the R-fMRI histograms in Fig. 6 and Fig. 10 is that the bins of ACPs #15-#16 and ACPs #12-#13 are swapped in these two histograms, suggesting the minor difference of WQCP percentages in two R-fMRI datasets (please see the exact percentage provided at the bottom of each sub-figure in Fig. 11). This minor difference might be due to the normal population variation or the difference in the numbers of training samples, which merits future investigations using larger scale datasets.

Fig. 11.

The correspondences between resting state ACPs obtained from the adolescent participants (Section 3.1, labeled as “adolescent”) and the adult participants (this section, labeled as “adult”), respectively. The percentages of ACPs in each population are also provided at the bottom of each sub-figure.

3.5. Comparison with Non-discriminative Dictionary Learning Model

In order to further demonstrate the effectiveness of the discriminative dictionary learning algorithm and the validity of the ACPs, we applied another dictionary learning and sparse coding model on the same dataset. The Online Dictionary Learning model, introduced in Mairal et al. 2010, has the similar iterative learning process as in the FDDL model used in this work. One key difference between these two models is that the cost function to be minimized in the online dictionary learning model does not have the discriminative fidelity term r as in the FDDL model. Thus, the non-discriminative model would not aim to maximize the difference between the learned sub-dictionaries. We used the non-discriminative dictionary learning model to identify 6 ACPs on the resting-state WQCPs from the adolescent subjects using the similar process described in Section 2.5. The patterns of the six ACPs were then compared with the ACPs #11 to #16, which were also supposed to be the representative patterns of resting-state data from the adolescent subjects. The visualization of the comparison is shown in Fig. 12. It could be observed from the visualizations that the patterns of ACPs identified by non-discriminative dictionary learning method are in rough correspondence with those by the FDDL method. However, the ACPs identified by the nondiscriminative model are similar to each other, especially between ACP #1 and #3. Moreover, the ACP corresponding to the first sub-dictionary obtained by the non-discriminative model takes a dominant proportion over the whole dataset (77%), indicating the possibility that there could be certain WQCPs mislabeled to the ACP #1 due to the similarity across the ACPs. This comparison experiment justifies the usage of discriminative dictionary learning method in this work.

Fig. 12.

The correspondences between ACPs obtained by the FDDL model used in this work and the nondiscriminative dictionary learning model. The first and third rows showing the FDDL results are the same as the visualizations in Fig. 11.

4. DISCUSSION AND CONCLUSION

This paper presents a novel computational framework for quantitative characterization of task-free and task-performance functional brain states via sparse representation of a set of whole-brain quasi-stable connectome patterns (WQCP). Our methodological contributions lie in the following three aspects. First, the large-scale consistent, robust and reproducible DICCCOL landmarks, which possess structural and functional correspondence across individuals and populations, offer a common and individualized brain reference system for functional connectome analysis. As a result, the different snapshots of temporally varying patterns of functional connectomes within a short period of fMRI scans (6-8 minutes) from different brains can be aggregated, modeled and characterized at the population level. Without the inter-subject correspondences established by the DICCCOL system, it would be very challenging to perform quantitative modeling and analysis of functional connectomes at the population level. Second, the temporally dynamic functional connectomes were segmented into large number of quasi-stable WQCPs, which are deemed to better describe the temporal behaviors of the brain's functional activities during R-fMRI/T-fMRI scans. The WQCPs effectively deal with the natural dynamics of brain functions, while ensuring that the ACP patterns are learned and clustered from relatively homogeneous training samples. This methodology warrants that the derived ACPs are both descriptive and representative. Third, the FDDL spare coding methodology was adopted and employed to effectively learn the ACP patterns and to represent the temporal WQCPs of both resting state and task-performance datasets through learning a combined and comprehensive dictionary from original WQCP training samples. The learned ACPs not only possess compact sparse representation of functional connectome patterns, but also have better discrimination capability across those patterns. In general, these methodologies provide a general framework to assess the functional connectomes and their dynamics in different R-fMRI/T-fMRI studies and for different populations in the future.

The interesting phenomena and novel neuroscience insights obtained from our experimental results lie in the following four aspects. First, the learned ACPs for R-fMRI and T-fMRI datasets are substantially different, which reveals the dissimilar, intrinsic functional connectome states of the human brain in the task-free and task-performance conditions. The quantitative characterization and visualization of those ACPs offered novel insights into the functional brain interactions and dynamics. Second, a certain portion of overlapping ACPs between R-fMRI and T-fMRI datasets were detected and examined, and the results suggested that the participating subjects with outlier ACPs might not be in the expected task-free/task-performance states during R-fMRI/T-fMRI scan sessions. These results provide a starting point for future elucidations of the mechanisms of different functional connectome patterns in various functional brain states. Third, the participating subjects showing resting state ACP patterns (potential outliers) during task scans almost have no group-wise activation regions compared to other groups, which strongly supports our hypothesis that subjects do not necessarily follow the administered task paradigms. These results alert researchers that additional caution should be paid to perform group-wise T-fMRI activation detections and according quality assurance procedures should be considered. Fourth, similar ACP patterns learned from the large-scale WQCP segments of two independent R-fMRI/T-fMRI datasets of two age populations confirmed the reproducibility of our analysis pipeline and results. These results suggest that the ACPs truly reflect the common functional connectome patterns and distributions, and effectively characterize the possible functional brain states. This study provides important implications in deciphering the temporal transition patterns of those ACPs and the underlying mechanisms in the future.

The current studies reported in this paper could be potentially enhanced and expanded in the following directions. First, the current DICCCOL system only contains 358 consistent cortical landmarks, which is still far from covering all functional brain regions and representing all possible functional connectome patterns. One of our ongoing works is to significantly expand the current DICCCOLs by adding anatomy guidance during the landmark initialization and optimization, thus achieving potentially many more consistent landmarks across individuals and populations. Then, the structural and functional connectomes (Zhu et al., 2012) could be substantially enhanced. Second, current WQCP segments were averaged along the time domain to obtain a WQCP vector, for the purposes of the normalization of temporal domains and the reduction of complexity in ACP learning. In the future, we plan to further examine the temporal characterizations of the WQCPs and their dynamical transition patterns. Therefore, more details about the temporal dynamics of functional connectomes will be elucidated. Third, due to the lack of effective automated algorithms to detect the temporal change points of functional connectome patterns, our current studies were still based on interactive segmentation of WQCPs. Supplemental Fig. 7 shows a pilot study for comparison of automatic and manual segmentation of WQCPs. It is clear that the automatic segmentation and manual segmentation are largely in agreement in abrupt change points such as those highlighted by the red arrows. However, automatic segmentation tends to have more subtle and ambiguous change points such as those highlighted by the black arrows. In the future, we plan to develop and validate novel accurate change point detection algorithms that can effectively determine the temporal transition points of those functional connectomes along the temporal domain. As a result, the proposed ACP learning and WQCP segmentation procedures could be possibly applied on large-scale multimodal DTI/R-fMRI/T-fMRI datasets to further replicate and validate the findings reported in this work. Finally, the current WQCPs and ACPs were learned from healthy brains and we have not done any studies in datasets from neurological or psychiatric diseases. We plan to apply the methods in this paper to multimodal DTI/R-fMRI/T-fMRI datasets of brain diseases such as Alzheimer's disease and Schizophrenia.

In summary, it has been shown in this study that structural connectomes constructed from DTI data can provide the structural substrates from the derivation of functional connectomes from T-fMRI/R-fMRI datasets. The functional connectomes and their representative patterns have provided important insights into the functional activities of the human brain in task-performance and task-free states based on multimodal R-fMRI/T-fMRI datasets. Our studies have demonstrated that structural and functional connectomes are powerful approaches to studying the human brain function. We envision that the public release of connectome and connectomics tools (e.g., Zhu et al., 2012; Li et al., 2012a; Li et al., 2012b) will offer unique and exciting opportunities for the neuroimaging field to examine the structures and functions of the brain and their relationships. Furthermore, the structural and functional connectomes will provide enabling approaches to examining the large-scale functional interactions and dynamics (Sun et al., 2012) in the healthy and diseases brains in the near future.

Highlights.

1. Representation of structural connectomes by DICCCOL based on DTI data

2. Construction of whole-brain quasi-stable functional connectomes from fMRI data

3. Learn atomic connectome patterns (ACP) via effective sparse representations

4. Characterization of task-performance and task-free states by ACPs

5. Novel insights into the functional architectures of the brain