Quantifying functional connectivity in multi‐subject fMRI data using component models

Abstract

Functional magnetic resonance imaging (fMRI) is increasingly used to characterize functional connectivity between brain regions. Given the vast number of between‐voxel interactions in high‐dimensional fMRI data, it is an ongoing challenge to detect stable and generalizable functional connectivity in the brain among groups of subjects. Component models can be used to define subspace representations of functional connectivity that are more interpretable. It is, however, unclear which component model provides the optimal representation of functional networks for multi‐subject fMRI datasets. A flexible cross‐validation approach that assesses the ability of the models to predict voxel‐wise covariance in new data, using three different measures of generalization was proposed. This framework is used to compare a range of component models with varying degrees of flexibility in their representation of functional connectivity, evaluated on both simulated and experimental resting‐state fMRI data. It was demonstrated that highly flexible subject‐specific component subspaces, as well as very constrained average models, are poor predictors of whole‐brain functional connectivity, whereas the best‐generalizing models account for subject variability within a common spatial subspace. Within this set of models, spatial Independent Component Analysis (sICA) on concatenated data provides more interpretable brain patterns, whereas a consistent‐covariance model that accounts for subject‐specific network scaling (PARAFAC2) provides greater stability in functional connectivity relationships between components and their spatial representations. The proposed evaluation framework is a promising quantitative approach to evaluating component models, and reveals important differences between subspace models in terms of predictability, robustness, characterization of subject variability, and interpretability of the model parameters. Hum Brain Mapp 38:882–899, 2017. © 2016 Wiley Periodicals, Inc.

INTRODUCTION

Functional magnetic resonance imaging (fMRI) is a brain imaging technique that measures fluctuations in blood‐oxygen associated with neuronal activity. These measurements can be used to quantify voxel‐wise functional connectivity relationships [Biswal et al., 1995, 2010; Lowe et al., 1998], which are commonly defined in terms of linear covariance estimates, or equivalently, correlation for standardized time‐series. As fMRI measurements are often noisy and exhibit considerable inter‐subject variability, the empirical estimation of covariance (or correlation) can be highly unreliable. This prohibits the analysis and interpretation of measured functional connectivity structure, particularly when trying to estimate generalizable group‐level connectivity. This issue is further intensified when correcting for interdependency, for example, when estimating partial correlations [Marrelec et al., 2006].

To address this issue, two approaches are commonly employed to reduce noise and inter‐subject variability: reducing the connectivity analysis to anatomically pre‐defined brain regions by averaging voxels [Achard et al., 2006; Bassett and Bullmore, 2006; Biswal et al., 1995; Fox et al., 2005; Greicius et al., 2003; Smith et al., 2010], and filtering noise by compressing the high dimensional spatial representation into a low dimensional subspace using multivariate component models [Calhoun et al., 2001b; Friston et al., 1993; McKeown et al., 1998]. In this article, we focus on the latter approach, which identifies sets of brain regions with similar temporal dynamics. Although it is the more challenging modeling approach, there are significant benefits to “learning” the salient features of functional connectivity, without being biased by a priori assumptions regarding the significance and functional organization of cortical regions.

A variety of different component models have been developed for modeling multi‐subject fMRI data [Beckmann and Smith, 2005; Calhoun et al., 2001a, 2009; Damoiseaux et al., 2006; Esposito et al., 2005; Guo and Pagnoni, 2008; Schmithorst and Holland, 2004]. Typical approaches include Principal Component Analysis (PCA) which maximizes explained variance, and Independent Component Analysis (ICA) models, where component bases are usually linear rotations of the PCA space that maximize the independence of higher‐order moments [Bell and Sejnowski, 1995]. These models can be used for both individual analysis and the modeling of subject groups. For groups, the extracted spatial subspace is assumed to be consistent across subjects, and components are estimated on data that are temporally concatenated across subjects (e.g., “group ICA” and “concat‐ICA” approaches [Calhoun et al., 2001b]). As a compromise between individual and group‐level models, dual regression techniques [Filippini et al., 2009] use group‐level spatial components to estimate subject‐specific time‐courses, which are then used as regressors to estimate the within‐subject brain maps that best approximate the group‐level spatial components.

When modeling functional connectivity relationships in multi‐subject data, it may be useful to assume that interactions between components in the extracted subspace are consistent across subjects, to improve model generalization and simplify the interpretation of model parameters. This is the assumption behind the multi‐way PARAFAC2 model [Harshman, 1972; Kiers et al., 1999]. This model identifies a component basis that maximizes the explained data variance, under the constraint of a fixed connectivity network, with subject‐dependent network scaling parameters. The PARAFAC2 model has been widely used in chemometrics [Bro et al., 1999], with recent applications in EEG [Weis et al., 2010] and integration of EEG‐fMRI data [Ferdowsi et al., 2013], although it has not been previously used to quantify whole‐brain functional connectivity structure in fMRI.

Other data‐driven techniques have also been developed to compress fMRI data and summarize structure in functional data. For example, clustering models are among the most widely used alternatives to component modeling. Unlike component models, which represent voxels as a linear combination of independent sources, these techniques identify homogeneous brain regions that constitute functional units [Cohen et al., 2008; Goutte et al., 1999]. In recent years clustering methods have also been extended to perform group‐level modeling in neuroimaging data, for example, Craddock et al. [2012]. However, to date, there has been little comparison between component‐based and clustering models, in terms of their ability to robustly model group‐level functional connectivity.

An important goal when modeling functional brain networks is to identify reliable network structure in multi‐subject data, by leveraging shared structure across subjects. This is a non‐trivial challenge, and there has been little investigation into which component models are best suited for this purpose. In particular, component models can be defined with varying levels of flexibility, ranging from fully subject‐specific PCA and ICA models, to moderately constrained PARAFAC2 models (e.g., consistent covariance but some flexible parameters), to highly constrained representations of “average” covariance in the group. The optimal model will depend on which level of flexibility best captures functional connectivity features within the group.

To evaluate the effectiveness of component models at defining whole‐brain functional connectivity, we may ask how generalizable the results are. That is, how well do they predict between‐voxel functional connectivity values in independent datasets? To address this question, we implemented a cross‐validation framework that performs within‐subject splitting, to estimate prediction accuracy of whole‐brain connectivity for individual subjects [Hansen et al., 1999; Strother et al., 2002; Thirion et al., 2014]. We used three different measures of covariance generalization, and compared a range of different component models including: within‐subject analysis, concatenated group‐level models, dual regression, and constrained‐covariance models based on the PARAFAC2 framework. In addition, we compared the performance of the group‐level clustering model developed by [Craddock et al., 2012]. These models were evaluated using both simulated and experimental resting‐state fMRI datasets, thereby highlighting how well different component models quantify whole‐brain connectivity.

As a secondary issue, researchers are often concerned with the interpretation of parameter estimates produced by component models, in particular the anatomical regions represented in spatial components, and the functional connectivity between individual components. These parameter estimates must be stable for the population of interest, otherwise they do not allow for a meaningful interpretation of functional structure, as there is no reason to choose one parameter representation over another. For experimental data, we also evaluated the reproducibility and interpretability of both the spatial components and between‐components connectivity measures within our cross‐validation framework [Himberg et al., 2004; Strother et al., 2002; Thirion et al., 2014]. This was evaluated for the two classes of models that optimally predict voxel‐wise functional connectivity, including PARAFAC2 and group PCA/ICA. These analyses demonstrate that models with comparable generalization may still significantly differ in the reproducibility and interpretability of model parameters.

MATERIALS AND METHODS

Component Models for the Quantification of Multi‐Subject Functional Connectivity

Consider an fMRI data matrix ${\mathit{X}}_s\in {\mathbb{R}}^{V\times T}$, consisting of $V$ brain voxels, sampled at $T$ time‐points for subject $s$. If the voxel (row) means are subtracted, total functional connectivity in the brain is given by the empirical “outer” $V\times V$ covariance matrix $\frac{1}{T^{*}}{\mathit{X}}_s{\mathit{X}}_s^{\top }$, where $T^{*}$ is the effective degrees of freedom in ${\mathit{X}}_s$; typically, $T^{*}<T$ due to the subtraction of row means and regression‐based preprocessing (e.g., controlling for residual motion artifact and physiological noise). Component‐based models seek a linear reduced‐rank representation of the data, to improve the generalizability of covariance estimates, and reduce the data to a smaller, more interpretable set of features.

The singular value decomposition (SVD) provides the optimal representation of data in terms of explained variance. If we decompose a single subject's data by SVD, we obtain

$${\mathit{X}}_s\approx {\mathit{U}}_s^{V\times C}{\mathit{\Sigma }}_s^{C\times C}{\mathit{V}}_s^{T\times C^{\top }},$$

for $C\le T^{*}$, where ${\mathit{U}}_s$ and ${\mathit{V}}_s$ are orthonormal matrices. We thereby model subject‐specific voxel‐wise covariance by $\frac{1}{T^{*}}{\mathit{X}}_s{\mathit{X}}_s^{\top }\approx \frac{1}{T^{*}}{\mathit{U}}_s{\mathit{\Sigma }}_s^2{\mathit{U}}_s^{\top }.$

Neuroscience studies often use component models to estimate the most consistent functional connectivity structure within a group of subjects. This is often done by concatenating the data of each subject into the data matrix $[{\mathit{X}}_1,\dots {\mathit{X}}_s]$ and performing SVD:

$$\left[{\mathit{X}}_1,\dots {\mathit{X}}_s\right]\approx \mathit{U}\mathit{\Sigma }\left[{\mathit{V}}_1^{\top },\dots ,{\mathit{V}}_s^{\top }\right].$$

In this case, the modeled covariance $\frac{1}{T^{*}S}\left[{\mathit{X}}_1,\dots {\mathit{X}}_s\right]{\left[{\mathit{X}}_1,\dots {\mathit{X}}_s\right]}^{\top }\approx \frac{1}{T^{*}S}\mathit{U}{\mathit{\Sigma }}^2{\mathit{U}}^{\top }$ forms a single “consensus” estimate of voxel covariance across all subjects, whereas the covariance of each subject based on this model typically differ since $\frac{1}{T^{*}}{\mathit{X}}_s{\mathit{X}}_s^{\top }\approx \frac{1}{T^{*}}\mathit{U}\mathit{\Sigma }{\mathit{V}}_s^{\top }{\mathit{V}}_s\mathit{\Sigma }{\mathit{U}}^{\top }\ne \frac{1}{T^{*}}\mathit{U}\mathit{\Sigma }{\mathit{V}}_{s\prime }^{\top }{\mathit{V}}_{s\prime }\mathit{\Sigma }{\mathit{U}}^{\top }\approx \frac{1}{T^{*}}{\mathit{X}}_{s\prime }{\mathit{X}}_{s\prime }^{\top }$. The inequality follows from ${\mathit{V}}_s^{\top }{\mathit{V}}_s\ne {\mathit{V}}_{s\prime }^{\top }{\mathit{V}}_{s\prime }$ in general. If we wish to correctly estimate a consensus representation of subject covariance, we must constrain ${\mathit{V}}_1^{\top }{\mathit{V}}_1={\mathit{V}}_2^{\top }{\mathit{V}}_2=\dots ={\mathit{V}}_s^{\top }{\mathit{V}}_s=\mathit{I}$, for example,

$$\left[{\mathit{X}}_1,\dots {\mathit{X}}_s\right]\approx \mathit{U}\mathit{\Sigma }\left[{\mathit{Q}}_1^{\top },\dots ,{\mathit{Q}}_s^{\top }\right],$$

where all ${{\mathit{Q}}_s}^T{\mathit{Q}}_s=\mathit{I}$. This model enforces a consistent covariance representation across subjects that maximizes explained variance, under the constraint that components be orthogonal in both space and time, parameterized by spectral decomposition $\mathit{U}{\mathit{\Sigma }}^2{\mathit{U}}^{\top }$. We can relax this strict constraint by permitting consistent but non‐orthogonal covariance between components ${{\mathit{V}}_s}^T{\mathit{V}}_s={\mathit{R}}^T\mathit{R}$. In this case, we re‐parameterize the model by adding a consistent network term $\mathit{R}$:

$$\left[{\mathit{X}}_1,\dots {\mathit{X}}_s\right]\approx \mathit{U}\mathit{\Sigma }{\mathit{R}}^{\top }\left[{\mathit{Q}}_1^{\top },\dots ,{\mathit{Q}}_s^{\top }\right],$$

Although more flexible, the above representation does not permit components of the underlying network to vary in strength across subjects. This can be alleviated by replacing $\mathit{\Sigma }$ with a subject‐specific (non‐negative) diagonal matrix ${\mathit{\Sigma }}_s$, so that we approximate the sth subject's data matrix by:

$${\mathit{X}}_s\approx \mathit{U}{\mathit{\Sigma }}_s{\mathit{R}}^{\top }{\mathit{Q}}_s^{\top }.$$

This latter model has been well studied in the psychometics and chemometrics literature, where it is denoted the PARAFAC2 model [Bro et al., 1999; Harshman, 1972; Kiers et al., 1999]. We note that $\mathit{U}$ is considered orthonormal in all discussed models for consistency; however, this is not assumed in the most general formulation of the PARAFAC2 model.

There are thus a number of different approaches to modeling group functional connectivity in fMRI, with tradeoffs between model simplicity and flexibility of fitting individual subjects. In Table 1, we define the above five different modeling approaches to quantify subject specific voxel‐wise functional connectivity matrices ${\mathit{M}}_s$ in group data, ordered by increasing flexibility in their ability to represent subject‐specific effects.

Table 1.

The five different models considered for parameterizing the de‐noised covariance ${\mathit{M}}_s$ as well as their numbers of free parameters $F$)

	Parameterization of voxel‐wise covariance		Number of model parameters $F$, as well as subject specific estimated correlation ${\mathit{C}}_s$ between estimated components
Average	${\mathit{M}}_s=\frac{1}{T^{*}}\mathit{U}{\mathit{\Sigma }}^2{\mathit{U}}^{\top }$	$F=K\left(V+T^{*}S+1\right)-\left(S+1\right)K(K+1)/2+S$ $F_{\mathrm{I}\mathrm{C}}=K\left(T^{*}S+1\right)-SK(K+1)/2+S$ ${\mathit{C}}_s=\mathit{I}$
All subject covariance matrices are approximated by the same spectral decomposition, based on the model $\left[{\mathit{X}}_1,\dots ,{\mathit{X}}_s\right]\approx \left[{\mathit{Z}}_1,\dots ,{\mathit{Z}}_s\right]=\mathit{U}\mathit{\Sigma }[{\mathit{Q}}_1^{\top },\dots ,{\mathit{Q}}_S^{\top }]$
Consistent	${\mathit{M}}_s=\frac{1}{T^{*}}\mathit{U}\mathit{\Sigma }\left({\mathit{R}}^{\top }\mathit{R}\right)\mathit{\Sigma }{\mathit{U}}^{\top }$		$F=K\left(V+T^{*}S+1\right)+K(K-1)/2-\left(S+1\right)K(K+1)/2+S$ $F_{\mathrm{I}\mathrm{C}}=K\left(T^{*}S+1\right)+K(K-1)/2-SK(K+1)/2+S$ ${\mathit{C}}_s={\mathit{R}}^{\top }\mathit{R}$
All subject covariance matrices are approximated by assuming the covariance between the extracted subject specific time‐series are consistent based on the model $\left[{\mathit{X}}_1,\dots ,{\mathit{X}}_s\right]\approx \left[{\mathit{Z}}_1,\dots ,{\mathit{Z}}_s\right]=\mathit{U}\mathit{\Sigma }{\mathit{R}}^{\top }[{\mathit{Q}}_1^{\top },\dots ,{\mathit{Q}}_S^{\top }]$
Scaled consistent	${\mathit{M}}_s=\frac{1}{T^{*}}\mathit{U}{\mathit{\Sigma }}_s\left({\mathit{R}}^{\top }\mathit{R}\right){\mathit{\Sigma }}_s{\mathit{U}}^{\top }$		$F=K\left(V+T^{*}S+S\right)+K(K-1)/2-\left(S+1\right)K(K+1)/2 +S$ $F_{\mathrm{I}\mathrm{C}}=K\left(T^{*}S+S\right)+K(K-1)/2-SK(K+1)/2+S$ ${\mathit{C}}_s={\mathit{R}}^{\top }\mathit{R}$
Relaxes the constraint of fixed component scaling across subjects based on the model ${\mathit{X}}_s\approx {\mathit{Z}}_s=\mathit{U}{\mathit{\Sigma }}_s{\mathit{R}}^{\top }{\mathit{Q}}_s^{\top }$ corresponding to PARAFAC2.
Common subspace	${\mathit{M}}_s=\frac{1}{T^{*}}\mathit{U}\mathit{\Sigma }{\mathit{V}}_s^{\top }{\mathit{V}}_s\mathit{\Sigma }{\mathit{U}}^{\top }$		$F=K\left(V+T^{*}S+1\right)-K(K+1) +S$ $F_{\mathrm{I}\mathrm{C}}=K\left(T^{*}S+1\right)-K(K+1)/2+S$ ${\mathit{C}}_s={\mathit{V}}_s^{\top }{\mathit{V}}_s$
Covariance between components is individual defined for each subject based on the group PCA/ICA model $\left[{\mathit{X}}_1,\dots ,{\mathit{X}}_s\right]\approx \left[{\mathit{Z}}_1,\dots ,{\mathit{Z}}_s\right]=\mathit{U}\mathit{\Sigma }[{\mathit{V}}_1^{\top },\dots ,{\mathit{V}}_S^{\top }]$
Individual	${\mathit{M}}_s=\frac{1}{T^{*}}{\mathit{U}}_s{\mathit{\Sigma }}_s^2{\mathit{U}}_s^{\top }$		$F=S(K\left(V+T^{*}+1\right)-K(K+1) +1)$ $F_{\mathrm{I}\mathrm{C}}=S(K\left(T^{*}+1\right)-K(K+1)/2+1)$ ${\mathit{C}}_s=\mathit{I}$
Each subject data is modelled individually by PCA/ICA, based on the model ${\mathit{X}}_s\approx {\mathit{Z}}_s={\mathit{U}}_s{\mathit{\Sigma }}_s{\mathit{V}}_s^{\top }$

$K$, $V$, $S$, and $T^{*}$ respectively denote the number of components, voxels, subjects and effective degrees of freedom in the time‐series. ${\mathit{Z}}_s$ and ${\mathit{M}}_s$ denotes the models' de‐noised estimates of the signal and covariance, respectively, for subjects. The columns of $\mathit{R}$ and $\mathit{V}$ are unit normalized. $F_{\mathrm{I}\mathrm{C}}$ denotes the number of free parameters used in calculating the Bayesian/Akaike information criteria, when considering the spatial mode $\mathit{U}$ (largest dimensionality) marginalized in the training likelihood.

Shown in Table 1, $\mathit{R}$ has effectively $K(K-1)/2$ free parameters, as ${\mathit{R}}^{\top }\mathit{R}$ can be defined in terms of an upper triangular matrix representation for $\mathit{R}$ given by the Cholesky decomposition, reduced by $\mathit{K}$ parameters, as all the diagonal elements of ${\mathit{R}}^{\top }\mathit{R}$ are fixed to one. Thus, the Scaled Consistent model (PARAFAC2) has the same number of free parameters as the Common Subspace model (group PCA/ICA) when $S=2$, and becomes more constrained as $S$ increases. As the voxel mode is much larger than the number of subjects and time points, the Individual model has substantially more parameters than the other models; the spatial mode ${\mathit{U}}_s$, is defined individually with a total of $SKV$ parameters.

We obtained solutions for the different models, computed in Matlab, as follows: Common Subspace and Individual models were computed using a standard SVD, which provides the PCA solution; we obtained both spatial and temporal ICA (sICA and tICA) by applying the FastICA algorithm with “pow3” nonlinearity and the deflation approach to decorrelate components (code from: http://research.ics.aalto.fi/ica/fastica/) to spatial and temporal components, respectively [Hyvärinen, 1999]. We estimated the Scaled Consistent PARAFAC2 solution using the software provided at (http://www.models.life.ku.dk/algorithms), which uses the modified Alternating Least Squares (ALS) solution proposed in [Bro et al., 1999]. The Average and Consistent solutions were estimated using modified implementations of the PARAFAC2 solution. The five component models were also compared with dual regression (Dualreg), which was implemented in Matlab. This article examines dual regression on a PCA basis, as ICA methods did not show a significant difference in generalization. We also evaluated the clustering model of [Craddock et al., 2012] (Clustering), with results shown for the temporal‐model clustering, as the spatial‐mode clustering did not show significant differences in generalization. Cluster‐model networks were obtained using the Common Subspace approach, but replacing component maps with the unit‐normalized binary clustering maps. When evaluating the performance of the clustering approach, it should be noted that clustering is inherently less flexible as it does not allow scaling of the voxel contributions.

In this section, we have defined three features of component‐based models that are important for characterizing functional connectivity in group data: (1) the “outer” $V\times V$ covariance matrix, which captures all functional connectivity relationships in the brain; (2) the spatial orthonormal basis $\mathit{U}$ of brain patterns showing reliable covariance relationships in the data; and (3) the “inner” correlation matrix ${\mathit{C}}_s$, which defines the functional connectivity between individual components of $\mathit{U}$.

A Cross‐Validation Framework for Evaluating Component Models

To compare between component models, we must quantify their ability to detect generalizable features of covariance in fMRI data. With increasing model flexibility, we obtain a richer description of group functional connectivity, but potentially at the expense of decreased model stability and generalizability. In the absence of a “ground truth” for experimental fMRI data, we have developed a cross‐validation approach based on the NPAIRS framework [Strother et al., 2002], that uses multiple performance metrics. The following sections review the testing framework, and key features of the metrics. Of primary interest is the ability of the component representation of the $V\times V$ covariance matrix to predict between‐voxel connectivity in independent test data. However, we also consider metrics quantifying the reproducibility of the spatial basis $\mathit{U}$, and of the correlation matrix ${\mathit{C}}_s$ between time series of extracted components. Because model performance may depend on the chosen metric of prediction and reproducibility, we tested multiple different prediction and reproducibility measures.

Prediction of the Voxel‐Wise Covariance

We evaluated prediction accuracy of the fitted component model, based on how well the full $V\times V$ subject covariance matrix generalizes to independent test‐data covariance ${\frac{1}{T^{*}}\mathit{X}}_s^{*}{{\mathit{X}}_s^{*}}^{\top }$. Covariance is expected to vary across subjects; our goal is to determine what level of model flexibility gives the most accurate prediction of individual‐subject covariance. To perform model evaluation, we defined a within‐subject cross‐validation framework for group data. For the dataset ${\{\mathit{X}}_s\} (s=1\dots S)$, each matrix was split into two split‐halves containing an equal number of time‐points. The network model was constructed using split‐half 1, and the estimated parameters were used to predict covariance ${\frac{1}{T^{*}}\mathit{X}}_s{{\mathit{X}}_s}^{\top }$ of split‐half 2. We then exchanged splits and performed modeling on split‐half 2 and prediction on split‐half 1, before computing average prediction across splits. The following three measures of prediction were evaluated.

Indirect fitting of voxel covariance (IF)

The IF is given by the sum of squared error (SSE) between the training model and test data, over all elements of the $V\times V$ voxel covariance matrix. For subject $s$, we computed the Frobenius distance of the modeled covariance matrix ${\mathit{M}}_s$, relative to test data ${\mathit{X}}_s^{*}$:

$${\mathrm{S}\mathrm{S}\mathrm{E}(\mathit{X}}_s^{*})=‖\frac{1}{T^{*}}{\mathit{X}}_s^{*}{{\mathit{X}}_s^{*}}^{{}^{\top }}-{\mathit{M}}_s‖{}_F^2$$

Total model prediction error was then computed as the sum of subject error terms $\mathrm{I}\mathrm{F}=\sum _s{\mathrm{S}\mathrm{S}\mathrm{E}(\mathit{X}}_s^{*})$. This metric is based on the indirect fitting objective developed for PARAFAC2. [Harshman, 1972; Kiers, 1993]. Although a statistical theory has not been developed around IF, it has the advantage of minimal modeling assumptions, and is robust in the case of ill‐posed covariance matrices, where $V\gg T^{*}$. Because the IF measures the squared error of variance, it is expected to be highly sensitive to mis‐specification of variance scaling. This metric has been previously examined as a goodness‐of‐fit estimator in the context of structural equation modeling [Bentler and Bonett, 1980; MacCallum, 1996] and economic forecasting [Ledoit and Wolf, 2004], but has not yet been applied to functional neuroimaging. Note that this metric includes the main diagonal terms of the covariance matrix which are related to the signal power. To test whether this had an effect, we also calculated the IF while excluding the main diagonal, which led to values that were indistinguishable from the original IF metric.

Covariance generalization error (GE)

This is a probabilistic estimate of generalization error, similar to [Hansen et al., 1999], but used to directly estimate covariance matrices. For subject data matrix ${\mathit{X}}_s$ (with row means subtracted), we consider the $K$‐dimensional signal subspace representation ${\mathit{M}}_s$ with eigen‐decomposition ${\mathit{M}}_s=\mathit{S}\mathit{\Lambda }{\mathit{S}}^{\top }$, and assume that noise ${\sigma }_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}^2$ is isotropic Gaussian, that is, ${\sigma }_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}^2=\frac{1}{V-K}\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{e}(\frac{1}{T^{*}}{\mathit{X}}_s{\mathit{X}}_s^{\top }-\mathit{S}\mathit{\Lambda }{\mathit{S}}^{\top })$. Given total modeled covariance ${\widehat{\mathbf{\Sigma }}}_{tot}={\mathit{M}}_s+{\mathit{I}}_V{\sigma }_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}^2$, we can measure the Wishart likelihood of the test‐data scatter matrix ${\mathit{X}}_s^{*}{{\mathit{X}}_s^{*}}^{{}^{\top }}$ with T* degrees of freedom. Because the test scatter matrix is singular, model generalization was inferred by the negative log‐likelihood of the singular Wishart distribution [Uhlig et al., 1994]:

$$L_{\mathrm{G}\mathrm{E}}\left({\mathit{X}}_s^{*}\right)=\frac{1}{2}\left(V{T}^{*}-{T^{*}}^2\right)\log \left(\pi \right)+\frac{V{T}^{*}}{2}\log \left(2\right)+\log {\Gamma }_{T^{*}}\left(\frac{T^{*}}{2}\right)-\frac{1}{2}\left(T^{*}-V-1\right)\log {\left|{\mathit{X}}_s^{*}{\mathit{X}}_s^{*\top }\right|}_{+}+\frac{T^{*}}{2}\left|{\widehat{\mathbf{\Sigma }}}_{tot}\right|+\frac{1}{2}\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{e}\left({{\widehat{\mathit{\Sigma }}}_{tot}}^{-1}{\mathit{X}}_s^{*}{\mathit{X}}_s^{*\top }\right)$$

where ${\left|{\mathit{X}}_s^{*}{\mathit{X}}_s^{*\top }\right|}_{+}$ denotes the pseudo‐determinant, given by the product of all non‐zero singular values, and the following terms are defined as per [Hansen et al., 1999]:

$$\left|{\widehat{\mathit{\Sigma }}}_{tot}\right|=\sum _{k=1}^K\log \left({\lambda }_k\right)+\left(V-K\right)\mathrm{l}\mathrm{o}\mathrm{g}({\sigma }_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}^2)$$

$$\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{e}\left({{\widehat{\mathit{\Sigma }}}_{tot}}^{-1}{\mathit{X}}_s^{*}{\mathit{X}}_s^{*\top }\right)=\frac{{\sum }_{t=1}^T{\mathit{x}}_s^{*}\left(t\right){{\mathit{x}}_s^{*}\left(t\right)}^{\top }-{\sum }_{k=1}^K\frac{{\lambda }_k-{\sigma }_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}^2}{{\lambda }_k}{\sum }_{t=1}^T{\left[{{\mathit{x}}_s^{*}\left(t\right)}^{\top }{\mathit{s}}_k\right]}^2}{{\sigma }_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}^2}$$

Model prediction error was then obtained by summing subject‐specific log‐likelihood terms: $\mathrm{G}\mathrm{E}=\sum _s{L}_{\mathrm{G}\mathrm{E}}\left({\mathit{X}}_s^{*}\right)$. This model is cast in a multivariate probabilistic framework, which is consistent with standard fMRI network modeling assumptions. However, it makes the simplifying assumption of isotropic noise variance over voxels, which may be unrealistic for fMRI data.

Heteroscedastic covariance generalization error (HGE)

As an alternative to GE, we may assume that the component signal space is independent of noise, and that noise variance is voxel‐specific (heteroscedastic). Consider again the modeled $K$‐dimensional subspace representation of data covariance ${\mathit{M}}_s$ with eigen‐decomposition ${\mathit{M}}_s=\mathit{S}\mathit{\Lambda }{\mathit{S}}^{\top }$, but now with voxel‐wise noise variance $\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left({\mathit{\sigma }}_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}^2\right)$, estimated on the residual of the training data after subtracting the signal subspace ${\mathit{X}}_{\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{i}\mathrm{d}}={\mathit{X}}_s-\mathit{U}{\mathit{\Sigma }}_s{\mathit{V}}_s^{\top }$. We again measured negative log‐likelihood of the test‐data scatter matrix ${\mathit{X}}_s^{*}{{\mathit{X}}_s^{*}}^{{}^{\top }}$ with T* degrees of freedom, based on the singular Wishart distribution, and the $L_{\mathrm{H}\mathrm{G}\mathrm{E}}\left({\mathit{X}}_s^{*}\right)$ takes the same form as $L_{\mathrm{G}\mathrm{E}}\left({\mathit{X}}_s^{*}\right)$ above. However, we total covariance is now defined as ${\widehat{\mathit{\Sigma }}}_{tot}={\mathit{M}}_s+\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left({\mathit{\sigma }}_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}^2\right)$, with the following substitutions:

$$\left|{\widehat{\mathit{\Sigma }}}_{tot}\right|=\left|{\mathit{I}}_K+{\mathit{\Lambda }}^{1/2}{\mathit{S}}^{\top }\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left({\mathit{\sigma }}_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}^{-2}\right)\mathit{S}{\mathit{\Lambda }}^{1/2}\right|+\sum _{v=1}^V{\mathit{\sigma }\left(v\right)}_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}^2$$

$$\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{e}\left({{\widehat{\mathit{\Sigma }}}_{tot}}^{-1}{\mathit{X}}_s^{*}{\mathit{X}}_s^{*\top }\right)=\sum _{t=1}^T{{\mathit{x}}_s^{*}\left(t\right)}^{\top }\left[\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left({\mathit{\sigma }}_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}^2\right)-\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left({\mathit{\sigma }}_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}^{-2}\right)\mathit{S}\left({\mathit{\Lambda }}^{-1}+{\mathit{S}}^{\top }\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left({\mathit{\sigma }}_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}^{-2}\right)\mathit{S}\right){\mathit{S}}^{\top }\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}\left({\mathit{\sigma }}_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}^{-2}\right)\right]{\mathit{x}}_s^{*}\left(t\right)$$

The above expressions make use of Woodbury's matrix inversion lemma as well as the matrix determinant lemma. The model prediction error is again obtained by summing over subject‐specific negative log‐likelihood terms: $\mathrm{H}\mathrm{G}\mathrm{E}=\sum _s{L}_{\mathrm{H}\mathrm{G}\mathrm{E}}\left({\mathit{X}}_s^{*}\right)$. This probabilistic model employs a more flexible spatially‐varying noise estimate than GE, under the assumption that the estimated signal subspace is noise‐independent.

Total prediction error was plotted for each metric and all component models, as a function of model order $K$. For all prediction curves, we also generated confidence bounds on total prediction error by Bootstrapping on subjects and computing standard error on the distribution of IF, GE, and HGE values.

Information Criteria

In addition to the generalization error, we evaluated model evidence via the Bayesian information criterion (BIC) [Schwarz, 1978] and the Akaike Information Criterion (AIC) [Akaike, 1974]. To obtain the training likelihood for this purpose we consider all the models represented in the form $\stackrel{\sim }{\mathit{X}}\approx \mathit{U}\mathit{L}$, where $\stackrel{\sim }{\mathit{X}}$ is a $V\times \left(T^{*}S\right)$ concatenated matrix of subject data in a compressed temporal representation, $\mathit{U}$ is the orthonormal voxel‐projection matrix, and $\mathit{L}$ is the associated low‐dimensional data representation. The full‐rank representation ( $T^{*}$ temporal dimensions per subject) can be obtained as a set of subject‐specific SVDs, concatenated on the temporal dimension. We approximate this data representation with the generative model:

$$\begin{array}{c} { \mathit{u}}_v \sim \mathcal{N}(0,V^{-1}\mathit{I})\\ \stackrel{\sim }{\mathit{x}}\left(v\right) \sim \mathcal{N}({\mathit{u}}_v\mathit{L},\mathit{\Psi })\end{array}$$

where $\mathit{\Psi }$ is a diagonal noise variance matrix with $T^{*}S$ elements; the $T^{*}$ elements corresponding to subject s are given by residual noise variance ${\sigma }_s^2$, averaged across all voxels and temporal dimensions.

The marginalized likelihood is obtained by integrating out ${\mathit{u}}_v$, using Woodbury's matrix inversion lemma:

$$\begin{array}{cc}P\left(\stackrel{\sim }{\mathit{X}}|\mathit{L}\right)& =\prod _v^{}\int \mathcal{N}\left(\stackrel{\sim }{\mathit{x}}\left(v\right)|{\mathit{u}}_v\mathit{L},\mathit{\Psi }\right)\mathcal{N}\left({\mathit{u}}_v|0,V^{-1}\mathit{I}\right)d{\mathit{u}}_v\\ & =\prod _v^{}\mathcal{N}\left(\stackrel{\sim }{\mathit{x}}\left(v\right)|0,\mathit{\Psi }+V^{-1}{\mathit{L}}^{\top }\mathit{L}\right)\end{array}$$

This likelihood can then be used to obtain the BIC, which is a large‐sample approximation to the model evidence:

$$\mathrm{B}\mathrm{I}\mathrm{C}=V\mathrm{l}\mathrm{o}\mathrm{g}\left|\mathit{\Psi }+V^{-1}{\mathit{L}}^{\top }\mathit{L}\right|+\sum _v\stackrel{\sim }{\mathit{x}}{\left(v\right)}^{\top }{\left(\mathit{\Psi }+V^{-1}{\mathit{L}}^{\top }\mathit{L}\right)}^{-1}\stackrel{\sim }{\mathit{x}}\left(v\right)+F_{\mathrm{I}\mathrm{C}}\log \left(SV{T}^{*}\right).$$

Similarly for AIC, we obtain:

$$\mathrm{A}\mathrm{I}\mathrm{C}=V\mathrm{l}\mathrm{o}\mathrm{g}\left|\mathit{\Psi }+V^{-1}{\mathit{L}}^{\top }\mathit{L}\right|+\sum _v\stackrel{\sim }{\mathit{x}}{\left(v\right)}^{\top }{\left(\mathit{\Psi }+V^{-1}{\mathit{L}}^{\top }\mathit{L}\right)}^{-1}\stackrel{\sim }{\mathit{x}}\left(v\right)+{2F}_{\mathrm{I}\mathrm{C}},$$

where the number of free parameters, $F_{\mathrm{I}\mathrm{C}}$, corresponds to the number of free parameters in $\mathit{L}$ and ${{\sigma }_s}^2$. These numbers are given in Table 1. Note that for the Individual model, likelihood is computed by marginalizing with respect to ${\mathit{u}}_v$ for each subject, and thus the information criteria are calculated individually, and then summed across subjects. Model evidence was plotted for all component models, as a function of model order $K$.

Reproducibility of Spatial Parameters

We further examined the reproducibility of the estimated network spatial basis $\mathit{U}$, since a common goal of component modeling is to interpret spatial component features. Components must be stable in order to have a valid interpretation which, at minimum, requires that there be no model ambiguity. For PCA‐based models, this is fulfilled as long as all singular values of its eigen‐decomposition are distinct. For models enforcing consistent group covariance, uniqueness is less clear. However, uniqueness conditions exist for the PARAFAC2 model [Berge and Kiers, 1996; Harshman et al., 1996] and our model is further constrained by spatial orthonormality. In general, we do not expect reproducibility to be hampered by non‐identifiability issues.

When model ambiguity is not a concern, reproducibility may be evaluated separately from prediction: due to bias‐variance tradeoffs in model parameterization, it is possible to have a highly predictive model with unstable voxel features. Reproducibility was evaluated using the cross‐validation procedure in the previous sub‐section, and we compared fitted parameters for the component models trained on split‐half 1 relative to split‐half 2. We evaluated two different metrics of spatial reproducibility:

Amari distance metric (AD)

This metric was developed and is widely used in the context of comparing ICA source‐separation models [Amari et al., 1996]. For two split‐half matrices $\mathit{X}$ and $\mathit{Y}$ with columns corresponding to the spatial component vectors, we define cross‐product matrix $\mathit{Q}=\mathit{X}{\mathit{Y}}^{-1}$ (where ${\mathit{Y}}^{-1}$ denotes the pseudo‐inverse of $\mathit{Y}$ for rectangular matrices). We then compute the “distance” between subspaces estimated for the two split‐halves by:

$$\mathrm{A}\mathrm{D}\left(\mathit{X},\mathit{Y}\right)=\frac{1}{K}(\sum _{i=1}^K\left[\left(\sum _{j=1}^K\left|Q_{ij}\right|/\underset{j}{\max }\left|Q_{ij}\right|\right)-1\right]+\sum _{j=1}^K\left[\left(\sum _{i=1}^K\left|Q_{ij}\right|/\underset{j}{\max }\left|Q_{ij}\right|\right)-1\right])$$

Because this is a non‐symmetric measure, we computed the average of $\mathrm{A}\mathrm{D}(\mathit{X},\mathit{Y})$ and $\mathrm{A}\mathrm{D}(\mathit{Y},\mathit{X})$. This metric quantifies the consistency of the basis vectors in $\mathit{X}$ and $\mathit{Y}$. Thus, for two matrices spanning the same subspace, it will assign a lower AD value if the individual components of matrices $\mathit{X}$ and $\mathit{Y}$ have high pairwise correlation.

RV coefficient (RV)

The RV coefficient, first established by [Escoufier, 1973], is an alternative metric of reproducibility between spatial subspaces. It is the multivariate generalization of the Pearson correlation to square, positive semi‐definite matrices and is applied to rectangular matrices $\mathit{X}$ and $\mathit{Y}$, by measuring the RV coefficient on scatter matrices ${\mathit{S}}_x=\mathit{X}{\mathit{X}}^{\top }$ and ${\mathit{S}}_y=\mathit{Y}{\mathit{Y}}^{\top }$. The RV coefficient is computed as:

$$\mathrm{R}\mathrm{V}\left(\mathit{X},\mathit{Y}\right)=\frac{{\text{trace}\{{(\mathit{X}}^{\top }\mathit{Y})}^2\}}{\sqrt{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{e}{\left\{{(\mathit{X}}^{\top }\mathit{X}\right)}^2\}}\sqrt{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{e}{\left\{{(\mathit{Y}}^{\top }\mathit{Y}\right)}^2\}}} .$$

This metric is of particular interest because most multivariate analysis models can be defined as maximizing the RV coefficient, subject to additional constraints, including PCA, partial least squares, and canonical correlation analysis [Robert et al., 2007]. Unlike AD, this similarity metric is rotationally invariant, and does not depend on the orientation of the individual components in $\mathit{X}$ and $\mathit{Y}$ but rather the overlap of their spanning subspace.

Reproducibility of the Functional Network Model

A second parameter set of interest is the connectivity between components; this feature may be as important as the spatial components in studies of functional neuroimaging. In this article, we focused on estimation of both the average between‐cluster connectivity $\mathit{C}$ and the partial correlation matrix $\mathit{P}$, which fully accounts for dependencies between variables [Marrelec et al., 2006]. In general, connectivity values cannot be directly compared between the split‐halves, as they are estimated for non‐identical component bases. Instead, we developed a within‐split bootstrapping approach. For this procedure, we initially fit the component model of interest on all subject data, to obtain spatial basis $\mathit{U}$. Subjects were then sampled with replacement for $b=1, \dots , \mathrm{1,000}$ iterations, and for each sample, we re‐estimated the average between‐cluster connectivity ${\mathit{C}}_b$ on the fixed basis $\mathit{U}$. For the Common Subspace model, ${\mathit{C}}_b$ was obtained by taking the average of all subject connectivity matrices ${\mathit{C}}_s={\mathit{V}}_s^{\top }{\mathit{V}}_s$ in the bootstrap sample scaled to unit diagonal to reflect correlations. For the Scaled Consistent model this was done by re‐fitting the PARAFAC2 model on the bootstrap sample with fixed $\mathit{U}$, to generate a new estimate ${\mathit{C}}_b={{\mathit{R}}_b}^{\top }{\mathit{R}}_b$. For each connectivity element ( $i,j$), we converted the bootstrapped correlations into normally‐distributed variables via Fisher's Z‐transform, defined as ${c_{ij,b}}^{\mathit{*}}=\frac{1}{2}\mathrm{l}\mathrm{o}\mathrm{g}\left((1+c_{ij,b})/(1-c_{ij,b})\right)$. Using the transformed values ${\left\{{c_{ij,b}}^{\mathit{*}}\right\}}_{b=1}^{1000}$, we computed the bootstrap ratio as the mean divided by the standard error, which provides z‐scored measure of reliability for all elements of $\mathit{C}$ [Efron and Tibshirani, 1986], and allows us to estimate P‐values based on a two‐tailed normal distribution. A similar approach was used to estimate z‐score significance for all elements of $\mathit{P}$, by converting bootstrapped ${\mathit{C}}_b$ matrices into ${\mathit{P}}_b$.

We obtained a robust measure of overall reliability for the connectivity matrices, by taking the median of (absolute) bootstrap ratio values for all connectivity elements ( $i,j$). The median bootstrap ratio was then compared for component models as a function of model order $K$. We also compared the reliability of lagged models of maximum connectivity, as suggested in Arbabshirani et al. [2013], by projecting subject data onto the fixed spatial basis ${\mathit{V}}_s={{\mathit{X}}_s}^T\mathit{U}$, then selecting the time‐lag that maximizes each pairwise correlation between columns of ${\mathit{V}}_s$ within a ±2 TR window. The bootstrapping procedure was then repeated for time‐lagged estimates of subject connectivity matrices ${\mathit{C}}_s$.

Visualization and Interpretation of Component Models

While the metrics of prediction and reproducibility are used to guide model selection, the ultimate goal of group functional connectivity studies is the visualization and interpretation of the results in the context of neuroscience research. To this end, we must be able to display spatial component maps $\mathit{U}$ and between‐component correlation matrix $\mathit{C}$, and assess them in terms of statistical significance.

Representing spatial components

We obtained consensus spatial maps that are Z‐scored based on their reproducibility, using the procedure established by [Strother et al., 2002]. Because the individual spatial components of matrices ${\mathit{U}}^{\left(1\right)}$ (split‐1) and ${\mathit{U}}^{\left(2\right)}$ (split‐2) may have arbitrary ordering and sign, we first matched components between splits using a restricted Procrustes matching procedure (i.e., admitting only re‐ordering and sign‐flipping of components) to minimize a total sum‐of‐squares cost function. For each pair of matched components ${{\mathit{u}}_k}^{\left(1\right)}$ and ${{\mathit{u}}_k}^{\left(2\right)}$, we normalized them to mean zero and variance one, and constructed the scatterplot of ${{\mathit{u}}_k}^{\left(1\right)}$ versus ${{\mathit{u}}_k}^{\left(2\right)}$ pairwise voxel values. The projection ${\mathit{u}}^{\left(e\right)}=({\mathit{u}}_d^{\left(1\right)}+{\mathit{u}}_d^{\left(2\right)})/\sqrt{2}$ (1st PC of the scatterplot) measures reliable voxel values across splits, and ${\mathit{u}}_d^n=({\mathit{u}}_d^{\left(1\right)}-{\mathit{u}}_d^{\left(2\right)})/\sqrt{2}$ (2nd PC of the scatterplot) measures uncorrelated, non‐reproducible signal. This was used to obtain spatially Z‐scored maps for each component as ${\mathit{z}}_d={\mathit{u}}_d^{\left(e\right)}/{\sigma }_{{\mathit{u}}_d^{\left(n\right)}}$, where ${\sigma }_{{\mathit{u}}_d^{\left(n\right)}}$ is a single global noise estimate.

Representing functional connectivity

We employed the bootstrapping procedure defined above in “Reproducibility of the functional network model,” in order to visualize reliable functional connectivity values. Based on bootstrap ratio values, empirical P‐values were obtained for elements of the partial correlation matrix $\mathit{P}$, and between‐component connectivity plots were then displayed for the different component models, after correcting for multiple comparisons using Bonferroni correction over all possible connections.

Simulation Data

To evaluate component models in the presence of ground truth, we simulated 2D “brain slices,” of dimensions $55\times 80$ pixels. Slices consisted of 36 square functional parcels ranging in size from $5\times 5$ to $20\times 20$ pixels, where each parcel has relative scaling from 0 (edge) to 1 (center) defined by a cosine basis function. Data were generated for 20 simulated subjects, with 100 time‐points per split‐half. We used PARAFAC2 (“scaled consistent” model in Table 1) as a generative model for simulation signal, as this can be converted into the other subspace models by altering parameter constraints and/or noise contributions.

For a fixed $K=20$ component subspace, spatial basis $\mathit{U}$ was generated by first producing 20 maps via random Gaussian weightings on the 36 parcels. An orthonormal basis U was then obtained by computing an SVD on the maps and taking the resulting 20 eigen‐images. Connectivity matrix $\mathit{R}$ was generated as a random normal matrix (with unit‐normalized columns), and orthonormal subject basis $\mathit{Q}s$ obtained by generating random normal vectors and convolving with AFNI's “SPMG1” function (http://afni.nimh.nih.gov/pub/dist/doc/program_help/3dDeconvolve.html; for TR = 2.0s) to simulate a smooth hemodynamic response. Linear independence between vectors was enforced via Graham‐Schmidt orthogonalization. Realistic network scaling matrices ${\mathit{\Sigma }}_s$ were simulated by applying SVD to each experimental subject dataset (described in the next section), and taking the average 20‐component eigenspectrum.

Noise and variability were simulated at three different levels of the model. At the voxel level, we added Gaussian noise to each pixel and time‐point, with noise amplitude ${\mathit{v}}_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}$ defined as the ratio of noise variance relative to signal variance, at each pixel. Between‐split, within‐subject network variability was simulated by adding uniformly‐distributed noise to elements of ${\mathit{\Sigma }}_{\mathit{s}}$, sampled independently per split. Between‐subject variability was also simulated, by adding the same uniformly‐distributed noise to elements of ${\mathit{\Sigma }}_{\mathit{s}}$ for both splits. For these variability measures, the sampling range is defined as the fraction of the largest eigenvalue ${\mathit{v}}_{\mathrm{s}\mathrm{p}\mathrm{l}\mathrm{i}\mathrm{t}}$ and ${\mathit{v}}_{\mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}}$. For example, ${\mathit{v}}_{\mathrm{s}\mathrm{p}\mathrm{l}\mathrm{i}\mathrm{t}}=0.5$ indicates additive noise sampled from the interval [0, 0.5σ₁] is added to each element of ${\mathit{\Sigma }}_{\mathit{s}}$. Results are shown for combinations of the parameter settings ${\mathit{v}}_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}=\{0.10, 0.50\}$, ${\mathit{v}}_{\mathrm{s}\mathrm{p}\mathrm{l}\mathrm{i}\mathrm{t}}=\left\{0.00, 0.50\right\},$ and ${\mathit{v}}_{\mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}}=\{0.00, 0.50\}$, to explore model performance under different levels of noise and signal variability. We compared generalization error of the component models in Table 1, as well as their ability to correctly identify optimal model order $K=20$.

We also examined the impact of spatial and temporal noise autocorrelation on model prediction, as a possible explanation for observed differences between experimental and simulated data. For spatial autocorrelation, each 2D slice image had its map of Gaussian noise smoothed by convolving with a 2D Gaussian kernel with full width at half maximum (FWHM) of ±1 pix. For temporal autocorrelation, at each pixel the Gaussian noise time‐series was convolved with the same “SPMG1” hemodynamic response function. These results are briefly discussed in Results section “Prediction of experimental whole‐brain functional connectivity,” and depicted in Supporting Information Figure S1.

Experimental Data and Preprocessing

The component models were evaluated on a set of resting‐state fMRI data that were used in a previous study [Andersen et al., 2014]. This dataset consisted of 30 healthy subjects (median age 45, range 22–69; 15 female). The Data were acquired using EPI (Echo‐planar imaging), with TR = 2.49 s, TE = 30 ms, FA = 90°; 42 slices, acquired in a 64 × 64 matrix, with 3mm isotropic voxels on a 3T (Siemens Magnetom Trio) MR scanner. For each subject run, 480 time points (∼20 min of resting‐state data) were acquired.

Data were preprocessed using the following steps in the SPM8 package (revision 5236): rigid‐body motion correction to volume mean, slice‐timing with sinc interpolation, normalization to an EPI template using affine warping and a DCT basis [Ashburner and Friston, 1999], and spatial smoothing with a isotropic 6 mm FWHM Gaussian kernel. We then corrected for head motion spikes using the algorithm established in [Campbell et al., 2013]: we obtained a sliding‐window estimate of signal change at each brain volume and identified outliers (P < 0.05, fitted gamma distribution). These volumes were replaced by interpolated values from adjacent runs, to avoid temporal discontinuities. We removed global signal by regressing out the first PC of each run [Carbonell et al., 2011], which is a moderately conservative approach that only removes global signal orthogonal to other BOLD fluctuations. We also performed motion parameter regression, by applying a PCA to the six rigid‐body motion parameters [Woods et al., 1999] and removing the first two PCs (consistently account for >85% of variance), and we removed low‐frequency drifts by regressing out third‐order Legendre polynomial basis (chosen using AFNI's selection heuristic; refer to: http://afni.nimh.nih.gov/pub/dist/doc/program_help/3dDetrend.html). Finally, we removed vascular, CSF and white matter signal using the PHYCAA+ algorithm [Churchill and Strother, 2013]. For analysis purposes, each subject's 480 time‐point run was split into two equal‐size blocks of 240 scans. All regression‐based preprocessing was performed separately for the two splits, to maximize independence between training/test splits and to avoid potential processing‐dependent model biases.

RESULTS AND DISCUSSION

Prediction of Simulated Whole‐Brain Functional Connectivity

Figure 1 displays prediction curves of simulated data based on measures of IF, GE, and HGE, for all component models. For all simulations and prediction measures, error decreases until saturating near the true data dimensionality of K = 20. Going from left to right, column‐1 depicts low‐noise data. All models have comparable performance at low K, but the highly flexible Individual and Dualreg models show consistently greater error at high K, whereas group‐level models retain comparable generalization. Column‐2 shows increased prediction error for higher voxel‐wise noise ( ${\mathit{v}}_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}=0.5$). The prediction curves now saturate at a lower K, indicating a tendency to under‐estimate model order in the presence of voxel‐level noise. The IF metric is particularly sensitive to noise effects, saturating near K = 12 components. In comparison, Column‐3 shows that the effect of adding between‐split network variability ( ${\mathit{v}}_{\mathrm{s}\mathrm{p}\mathrm{l}\mathrm{i}\mathrm{t}}=0.5$) has a relatively small effect on prediction error compared with voxel noise. Finally, Column‐4 shows that adding inter‐subject variability ( ${\mathit{v}}_{\mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}}=0.5$) has the greatest impact on relative model performance. The constrained Average and Consistent models produce higher prediction error than the Scaled Consistent and Common Subspace models. In addition, Individual and Dualreg models now exhibit “cross‐over” behavior, where they outperform group‐level models at low K, but are consistently worse at higher K. The prediction measures diverge in one key respect: for IF, the Average/Consistent models exhibit higher error than Individual and Dualreg models, whereas the opposite is seen for GE and HGE measures.

Figure 1.

Prediction error curves for simulated resting‐state data. Average and Consistent models are combined due to near‐identical error curves. Each column shows prediction values for a set of fixed simulation parameters, including voxel noise scaling $v_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}$, between‐split network variability $v_{\mathrm{s}\mathrm{p}\mathrm{l}\mathrm{i}\mathrm{t}}$, and between‐subject network variability $v_{\mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}}$. [Color figure can be viewed at http://wileyonlinelibrary.com]

Figure 2 plots information criterion curves for simulated data, showing similar curves for both BIC and AIC measures. In general, results are consistent with cross‐validation measures, with all models saturating near the true K = 20 dimensionality, although for group‐level models the BIC begins to increase again beyond K = 20. This is a consequence of BIC having a larger penalty weighting on the number of free parameters $F_{\mathrm{I}\mathrm{C}}$, compared to AIC.

Figure 2.

Information criterion curves for simulated resting‐state data, including Bayesian Information Criterion (BIC) and Akaike Information Criterion (AIC). Average and Consistent models are combined due to near‐identical curves. Each column shows information criterion values for a set of fixed simulation parameters, including voxel noise scaling $v_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}$, between‐split network variability $v_{\mathrm{s}\mathrm{p}\mathrm{l}\mathrm{i}\mathrm{t}}$, and between‐subject network variability $v_{\mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}}$. [Color figure can be viewed at http://wileyonlinelibrary.com]

As with cross‐validation measures, higher voxel‐wise noise ( ${\mathit{v}}_{\mathrm{n}\mathrm{o}\mathrm{i}\mathrm{s}\mathrm{e}}=0.5$) tends to increase both BIC and AIC values and leads to under‐estimation of model order, whereas adding between‐split network variability ( ${\mathit{v}}_{\mathrm{s}\mathrm{p}\mathrm{l}\mathrm{i}\mathrm{t}}=0.5$) has a minimal effect on model evidence. The addition of inter‐subject variability ( ${\mathit{v}}_{\mathrm{s}\mathrm{u}\mathrm{b}\mathrm{j}\mathrm{e}\mathrm{c}\mathrm{t}}=0.5$) also increases between‐model differences, with Average/Consistent models showing worse performance than Scaled Consistent and Common Subspace models. As expected, the probabilistic information criterion curves more closely resemble the probabilistic GE and HGE metrics, compared with the non‐probalistic IF measure.

Prediction of Experimental Whole‐Brain Functional Connectivity

Figure 3 displays prediction curves of experimental data based on measures of IF, GE, and HGE, for all component models. Highly constrained Average/Consistent models show consistently greater error than the more flexible Scaled Consistent and Common Subspace models. The latter two models show consistently optimal prediction, where Scaled Consistent has lowest error for the IF metric, and Common Subspace has lowest error for GE and HGE metrics. The Clustering model shows consistently higher error than all group‐level component models. Conversely, the highly flexible Individual and Dualreg models exhibit cross‐over behavior seen in simulations (Fig. 1), where they outperform group‐level models at low $K$, but are consistently worse at higher $K$. The Dualreg approach shows the benefits of group‐level component estimation, as it has consistently better prediction than Individual. Thus, the Scaled Consistent and Common Subspace approaches tend to provide optimal prediction of functional connectivity.

Figure 3.

Prediction error curves for experimental resting‐state data, along with bootstrapped standard error bars. Average and Consistent models are combined due to near‐identical error curves. [Color figure can be viewed at http://wileyonlinelibrary.com]

In general, experimental results resembled simulation curves with between‐subject network variability (Fig. 1, Column‐4). However, unlike simulated data, models do not consistently saturate for experimental data, within the tested range of 1–150 components. While IF consistently saturates for all models by $K=25$, only highly flexible Individual and Dualreg models show saturation of GE and HGE within the tested range of components. In addition, for experimental data, Individual and Dualreg models appear much more heavily penalized for GE and HGE metrics at high $K$. Because these metrics are based on an assumption of independent, identically distributed noise, the discrepancies may be due to an inaccurate error model. In Supporting Information Figure S1, the random noise model (Supporting Information Fig. S1A) is compared with models with spatially and temporally autocorrelated noise. Supporting Information Figure S1B shows that for spatially smooth noise, GE and HGE fail to saturate at the correct K = 20 dimensionality, for group‐level models. Conversely, Supporting Information Figure S1C shows that for temporally smooth noise, GE and HGE are much larger for Individual and Dualreg models. These results demonstrate tradeoffs between the different prediction models: IF (which assumes no error model) is more robust to the effects of noise autocorrelation, whereas GE and HGE are less prone to under‐estimating dimensionality in the presence of noise (shown in Fig. 1).

In Figure 4, we plot the information criterion curves for experimental data. As with the simulations, both BIC and AIC curves are similar to probabilistic generalization error curves GE and HGE of Figure 3. The highly constrained Average and Consistent models show highest error for group‐level models, followed by Scaled consistent, and Common subspace models. In addition, the flexible Individual model shows initially high generalization, but quickly becomes worse than group‐level models with increasing K. However, two distinctions are seen from cross‐validated error. The first is that Scaled consistent and Common subspace models show a greater difference in performance, whereas they are near‐indistinguishable for GE and HGE measures. The second difference is that Individual models show optimal BIC and AIC for up to K = 50 components, whereas they become sub‐optimal at K = 30 for GE and HGE. Finally, we note that the shapes of BIC and AIC curves are generally similar, with the exception of Average and Consistent models that saturate at approximately K = 60 for the BIC metric. This indicates that the choice of information criterion has a minimal impact when evaluating models at the tested range of K.

Figure 4.

Information criterion curves for simulated resting‐state data, including Bayesian Information Criterion (BIC) and Akaike Information Criterion (AIC). Average and Consistent models are combined due to near‐identical curves. [Color figure can be viewed at http://wileyonlinelibrary.com]

Reliability of Model Parameters

Figure 5 plots reproducibility of the spatial component bases for the Scaled Consistent (i.e., PARAFAC2) and Common Subspace (i.e., PCA and ICA) models, which have lowest prediction error in Figure 2. For both reproducibility metrics, we see comparable performance between component models for low dimensionalities of $K<25$, but beyond this point they diverge in performance. The AD metric shows optimal performance for PARAFAC2 followed by sICA, PCA, then tICA. As this metric is highly sensitive to pairwise matching of components, this indicates that PARAFAC2 provides more stable individual component patterns at high model dimensionalities. Conversely, the RV coefficient shows near‐identical performance between PCA and sICA, but lower performance for PARAFAC2, particularly at higher subspace dimensionalities. This indicates that overall subspace overlap between independent split‐halves is higher for the Common Subspace models.

Figure 5.

Reproducibility of spatial component bases for the optimal group‐level models, identified based on generalization error (Fig. 3) and information criteria (Fig. 4), including Scaled Consistent and Common Subspace models. For the Common Subspace approach, we also show different component representations including Principal Component Analysis (PCA), and temporal and spatial Independent Component Analysis (tICA and sICA, respectively). As indicated by the arrow in the bottom of each plot, note that for the Amari index lower values indicate better reproducibility, whereas higher values are associated with better reproducibility for the RV metric. [Color figure can be viewed at http://wileyonlinelibrary.com]

Figure 6 depicts the reliability of the between‐component functional connectivity estimates. We compared Scaled consistent against the sICA Common Subspace approach, since PCA and tICA models assume temporal independence between components. Comparisons included both standard network model and the lagged models of maximal connectivity. For fixed connectivity matrices, Scaled Consistent consistently estimates more reliable connectivity values, for both correlation (Fig. 6A) and partial correlation (Fig. 6B) estimates. However, the reliability of Scaled Consistent decreases for lagged estimation, becoming comparable with the Common Subspace model. This we attribute to the fact that the PARAFAC2 model specifically optimizes for consistency of the zero‐lag consistency of the group covariance, while ICA does not explicitly consider the group covariance.

Figure 6.

Bootstrapped reliability of between‐component functional connectivity matrices for the optimal group‐level models, identified based on generalization error (Fig. 3) and information criteria (Fig. 4), including Scaled Consistent and Common Subspace models. For the Common Subspace method, we depict spatial Independent Component Analysis (sICA), as it is the only tested method that does not assume temporal independence between components. Lagged connectivity matrices are also estimated by maximizing subject‐specific connectivity over a ±2 TR window. Results are shown for both (A) correlation and (B) partial correlation matrices. For each model, median (absolute) bootstrapped reliability is measured over all connectivity elements, as a function of model order $K$. [Color figure can be viewed at http://wileyonlinelibrary.com]

Figures 7, 8, 9, 10 display both spatial component patterns and functional connectivity relationships between components, based on partial correlation estimates; results are shown for the optimal $K=25$ model dimensionality (estimated from the IF prediction metric). For visualization, each of the spatial maps were thresholded to retain the 5% voxels that showed the largest absolute values. For temporally‐independent PCA and tICA models, no between‐component connectivity structure can be displayed, as the between‐component correlations are uniformly zero. Both the scaled consistent (PARAFAC2) and common subspace (sICA) models show numerous highly‐significant partial correlations between components.

Figure 7.

Scaled Consistent (PARAFAC2) spatial components and network connectivity. The figure shows the spatial components. Each map is displayed as sagittal, coronal and axial projections in neurological convention and are arranged in a circle ordered. For visualization the maps were thresholded to retain the 5% of voxels with largest absolute magnitude, positive super‐threshold voxels are displayed in red, whereas negative super‐threshold voxels are displayed in blue. The lines that connect each spatial map represent partial correlations as indicated by the colorbar to the right, and the thickness of each line indicate the statistical significance of the connection calculated based bootstrap ratios of each correlation value and corrected for multiple comparisons over all connections by Bonferroni correction. [Color figure can be viewed at http://wileyonlinelibrary.com]

Figure 8.

Spatial ICA components and network connectivity. This figure displays the components extracted by spatial ICA based on the Common Subspace model, the formatting of the figure is consistent with Figure 7. [Color figure can be viewed at http://wileyonlinelibrary.com]

Figure 9.

Temporal ICA components. The figure displays the spatial components extracted by temporal ICA based on the Common Subspace model. Spatial maps are displayed and thresholded consistently with Figures 7 and 8, but here the maps are arranged in two lines because the orthogonal property of the model does not enable estimation of between component correlations. [Color figure can be viewed at http://wileyonlinelibrary.com]

Figure 10.

PCA components. This figure displays spatial components extracted by the common subspace PCA model, formatting is consistent with Figure 9. [Color figure can be viewed at http://wileyonlinelibrary.com]

The spatial patterns for the PCA and tICA model are difficult to interpret due to the global and distributed nature of the components. Both the sICA and PARAFAC2 models identify several components that are restricted within certain brain regions, allowing easier interpretation. Quantitatively, the sICA model appears to be more interpretable in that it generally captures more spatially sparse brain regions, whereas components estimated by PARAFAC2 model are less interpretable due to less skewed component distributions (i.e. more mixed sign in the components). This was anticipated due to the super‐Gaussian source distribution of the sICA model, which enforces spatially sparse and thereby interpretable components.

Figure 11 shows a comparison between matched components for the Scaled consistent and sICA Common subspace models. We identified components reflecting the default‐mode network, primary visual network and motor network which had the highest absolute correlation with spatial reference templates. For the default‐mode network we used the template from [Franco et al., 2009] provided in the Group ICA toolbox (http://mialab.mrn.org/software/gift), whereas the motor templates were created by combining Brodmann areas 4a and 4p [Geyer et al., 1996] provided in the SPM Anatomy toolbox [Eickhoff et al., 2005]. Similarly, the visual template was created by combining Brodmann areas 17 and 18 [Amunts et al., 2000]. All three matched components show some overlap indicating that they likely reflect similar networks. However, as also stated above the components identified using sICA appear to be slightly more sparse, leading to easier interpretation.

Figure 11.

Comparison of Scaled consistent and sICA Common subspace components. This figure shows selected axial slices of spatial components matched by absolute spatial correlation across Scaled consistent (PARAFAC2) and sICA models. The top row (marked “DMN”) displays the network showing the largest correlation with the default‐mode template, whereas the second row (marked “Motor”) displays the component showing the largest correlation with the template motor template. The bottom row (marked “Visual”) shows the component exhibiting the largest correlation with the visual template. Spatial maps are displayed in neurological intervention, as marked by the L‐Left and R‐Right markers, at axial MNI‐space slices according to the displayed z‐coordinate. Spatial maps are thresholded at a Z‐score of 3, red indicates that the region is above threshold in the Scaled consistent model, whereas blue indicates that the region was above threshold for the sICA model. Purple indicates that both models were above threshold. [Color figure can be viewed at http://wileyonlinelibrary.com]

CONCLUSION

We have established a cross‐validation framework for comparing different component subspace models in group fMRI data, which predicts between‐voxel functional connectivity relationships. This framework was used to analyze five different component models, with varying levels of flexibility in parameterizing subject‐specific variability. The Average model ensures that all subject covariance matrices are approximated by the same spectral decomposition, whereas for the Consistent model the covariance between the extracted component time‐series is consistent across subjects. The Scaled consistent model relaxes this assumption by allowing scaling of the components across subjects. For the Common subspace model, the covariance between components is individually defined for each subject forming temporal concatenated group PCA/ICA models. Finally, in the Individual models each subject is modeled by a separate spectral decomposition (per subject PCA/ICA).

We demonstrated consistent results for three different predictive measures (IF, GE, and HGE), including uniformly poor generalization for highly constrained Average and Consistent models. We attribute this poor generalization to substantial inter‐subject variability, which these models were unable to account for. The superior generalization of the more flexible Scaled consistent and Common subspace models indicates that inter‐subject differences in covariance are reliable across independent data splits, and can thus be consistently estimated.

In contrast, the highly flexible Individual model was optimal only for small dimensionalities, and rapidly increased in error with number of estimated components K. This is due to its inability to share information about covariance structure across subjects, producing a noisy, over‐fitted structure with poor generalization. Moreover, the dual regression (Dualreg) approach which uses group‐level spatial components to estimate subject‐specific time‐courses, improves prediction, but remains highly similar to the Individual model in performance. We attribute the lack of improved predictability using dual regression to overfitting, due to the additional parameters added in the dual regression step. Improved reproducibility of spatial maps was observed by [Zuo et al., 2010] when using dual regression. However, we note that our prediction metrics are not directly comparable to the reproducibility of the spatial networks as measured by spatial correction. In addition, a limitation of our within session splitting procedure is that the long resting state sessions (20 min) might be associated with limited reproducibility, whereas [Zuo et al., 2010] used three separate sessions.

With respect to the prediction metrics investigated, the best performing models were the Common subspace and Scaled consistent models, which were both able to account for subject variability, while exploiting shared structure across subjects. This supports the well‐established group‐PCA (and equivalently, group‐ICA) approaches as viable models for multi‐subject fMRI data. We also demonstrated no significant difference in generalization between the Common subspace model (group PCA/ICA) and the constrained Scaled consistent model (PARAFAC2). Therefore, we can define a consistent connectivity model across subjects with strong generalization, simply by accounting for subject variability in terms of relative scaling of the components.

Our results also show that the Scaled consistent and Common subspace models, which have comparable prediction, have significant tradeoffs in stability (and interpretability) of model parameters. While the Common subspace model produced greater overlap of spatial subspaces, (measured by the RV coefficient), the Scaled consistent model maximized the stability of individual spatial component patterns (measured via AD) and extracted the most consistent subspace network structure ${\mathit{C}}_s$ _. Thus, the added constraints of consistency in PARAFAC2 makes component estimation more stable across independent data splits, potentially providing more viable interpretations of individual component maps and their interactions over less constrained common subspace models. We also expect that reproducibility may be improved by further constraining common subspace modeling approaches, either by regularizing using prior information such as functional templates [Schultz et al., 2014] and atlases [Dhillon et al., 2014], or by imposing additional constraints on inter‐subject dependence, as in independent vector analysis [Lee et al., 2008] or group information guided ICA [Du and Fan, 2013]. Similarly, we expect the performance of the Individual models may be improved using regularization such as penalizing the deviation from a common subspace [Varoquaux et al., 2011].

When comparing the extracted spatial maps of the group‐ICA model to the group‐PCA and scaled consistent (PARAFAC2) models, we find that spatial ICA provides sparser, more localized components. This is more interpretable in terms of connectivity relationships between brain regions, compared with group‐PCA, PARAFAC2 and temporal ICA. The latter models result in a mixture of highly global and highly sparse components, making it more difficult to interpret brain regions implicated in network interactions. We displayed components for a dimensionality of K = 25 (i.e., optimizing the IF metric), as the whole component space can be directly visualized in Figures 7, 8, 9, 10. At this dimensionality, the individual components contain several non‐contiguous regions and may be regarded as networks themselves [Kiviniemi et al., 2003]. In the analysis of resting‐state fMRI data, a much higher number of components are often estimated using group ICA, which typically creates component maps with small, isolated brain regions. Although this enables easier interpretation of network interactions [Kiviniemi et al., 2009], our results in Figures 5 and 6 indicate that both the component maps and functional connectivity values become significantly less stable.

To our knowledge, this is the first article to systematically evaluate generalization and reproducibility of component models for multi‐subject fMRI analysis. We believe that this validation framework is a powerful approach for investigating component models in multi‐subject fMRI data. Current results point toward group‐PCA/ICA and PARAFAC2 as viable frameworks for modeling subject variability, while at the same time identifying consistent generalizable structure. In particular, it seems the more constrained scaled consistent representation offered by the PARAFAC2 modeling framework provides both high model prediction, and the most reproducible results. One remaining issue for PARAFAC2 is component interpretation, which is hampered by the lack of localization, when compared with spatial group ICA. Further research is required to establish extensions of this framework that incorporate spatial constraints, without compromising prediction and network reproducibility. In the current manuscript we considered only resting‐state fMRI data, which was a deliberate choice as the models investigated make no assumption about generalization of time‐series across subjects, making them particularly suitable for analysis of resting‐state fMRI data. Nonetheless, it is an important area of research to investigate whether the studied models are also able to successfully extract task‐related patterns of brain activity.

Robust quantification of whole‐brain functional connectivity in multi‐subject data is important, to accurately and reliably determine intrinsic functional network structure in the brain. We believe that the proposed framework can be used to quantify how well component models account for both group‐level and subject‐specific covariance relationships between brain regions in multi‐subject fMRI datasets. This also has general importance for analyses of covariance in the brain, including sparse inverse estimation [Marrelec et al., 2006; Smith et al., 2010] and time‐evolving dynamics [Hutchison et al., 2013].

Supporting information

Supporting Appendix A