Combining Multiple Connectomes Improves Predictive Modeling of Phenotypic Measures

Abstract

Resting-state and task-based functional connectivity matrices, or connectomes, are powerful predictors of individual differences in phenotypic measures. However, most of the current state-of-the-art algorithms only build predictive models based on a single connectome for each individual. This approach neglects the complementary information contained in connectomes from different sources and reduces prediction performance. In order to combine different task connectomes into a single predictive model in a principled way, we propose a novel prediction framework, termed multidimensional connectome-based predictive modeling. Two specific algorithms are developed and implemented under this framework. Using two large open-source datasets with multiple tasks—the Human Connectome Project and the Philadelphia Neurodevelopmental Cohort, we validate and compare our framework against performing connectome-based predictive modeling (CPM) on each task connectome independently, CPM on a general functional connectivity matrix created by averaging together all task connectomes for an individual, and CPM with a naive extension to multiple connectomes where each edge for each task is selected independently. Our framework exhibits superior performance in prediction compared with the other competing methods. We found that different tasks contribute differentially to the final predictive model, suggesting that the battery of tasks used in prediction is an important consideration. This work makes two major contributions: First, two methods for combining multiple connectomes from different task conditions in one predictive model are demonstrated; Second, we show that these models outperform a previously validated single connectome-based predictive model approach.

1. Introduction

Advanced functional magnetic resonance imaging (fMRI) techniques, particularly functional connectivity analyses, are revealing robust individual differences in patterns of neural activity that predict continuous phenotypic measures (Dubois and Adolphs, 2016; Rosenberg et al., 2018). Predictive modeling of the associations between phenotypic measures and the functional organization of an individual’s brain improves generalization of results to novel individuals and increases their eventual clinical utility. Recent work has used functional connectivity matrices, or connectomes, to predict a wide range of phenotypic measures, including fluid intelligence (Finn et al., 2015), brain maturity (Dosenbach et al., 2010), and sustained attention (Rosenberg et al., 2015). However, most of the current state-of-the-art algorithms only build predictive models based on a single connectome for each individual (Dadi et al., 2018). This approach neglects the complementary information contained in connectomes from different sources and reduces prediction performance.

While functional connectivity is usually calculated from data acquired during rest, task conditions better reveal individual differences (Finn et al., 2017; Vanderwal et al., 2017) and improve phenotypic prediction (Greene et al., 2018; Rosenberg et al., 2015). Further, the observed improvement in predictive power appears to be task specific, suggesting that task conditions are likely better at generating models of phenotypes related to the circuits they perturb (Greene et al., 2018; Rosenberg et al., 2015). Overall, it is unlikely that a single task can be developed that is optimal for all phenotypes. Instead, methods that incorporate functional connectivity information from a spectrum of tasks into a single predictive model may yield the best performance and most generalizable method for predicting phenotypic measures from connectomes.

In order to combine different task connectomes into a single predictive model in a principled way, we propose a novel prediction framework, termed multidimensional connectome-based predictive modeling. Two algorithms, each with their own strengths and limitations, are provided to illustrate the advantage of utilizing multiple connectomes. Both are based on the previously validated connectome-based predictive modeling (CPM) method (Shen et al., 2017). The first utilizes canonical correlation analysis (CCA), while the second utilizes ridge regression. CCA combines multiple task connectomes by finding the projection direction which maximizes correlation between the combined connectomes and behavioral measure(s) to be predicted. In contrast, ridge regression directly incorporates the large number of edges in multiple connectomes through regularization.

Using two large open-source datasets with multiple tasks—the Human Connectome Project (HCP) (Van Essen et al., 2013) and the Philadelphia Neurodevelopmental Cohort (PNC) (Satterthwaite et al., 2016), we validate and compare our two algorithms against performing CPM on each task connectome independently, CPM on a general functional connectivity (GFC) (Elliott et al., 2019) matrix created by concatenating time series from all task conditions to create a single connectome for an individual, ridge regression on GFC and CPM with a naive extension to multiple connectomes where each edge for each task is selected independently. Preliminary versions of this work were presented at the 2018 IEEE International Symposium on Biomedical Imaging (Gao et al., 2018a) and the 21st International Conference on Medical Image Computing and Computer Assisted Intervention (Gao et al., 2018b).

In all, our contribution in this paper is two-fold. First, we propose the combination of multiple connectomes from different task conditions in one predictive model. Second, we developed two algorithms for dealing with multiple connectomes and show that they outperformed the validated single connectome predictive model.

2. Methods

2.1. Connectome-based Predictive Modeling (CPM)

CPM (Shen et al., 2017) is a validated method for extracting and pooling the most relevant features from connectivity data in order to construct linear models to predict phenotypic measures (Figure 1a). Briefly, edges of connectivity matrices that are significantly correlated with the phenotypic measure of interest are selected. The selected features are then pooled (e.g. averaged) and linear regression is used to predict the phenotypic measure in novel individuals. It is designed for single connectome-based prediction, but can be easily extended to multiple-connectome scenarios as each edge is selected independently. However, this simple approach is not equipped to efficiently incorporate the increasing number of features introduced by multiple connectomes.

Figure 1. Algorithm flow chart for three major models mentioned.

a) The original CPM flow chart b) cCPM extends CPM to handle multiple connectomes per individual by replacing the correlation step in CPM with a canonical correlation analysis (CCA) step. c) rCPM extends CPM to handle multiple connectomes per individual by replacing the pooling (i.e. averaging) and linear regression step with a ridge regression step.

2.2. Multidimensional Connectome-based Predictive Modeling

Although CPM can be extended to leverage multiple connectomes, a specially designed framework will better utilize the complementary information in different brain connectivity patterns driven by corresponding task conditions. Here, we present two realizations of this framework to illustrate the feasibility and advantage of combining multiple connectomes for prediction.

2.2.1. CCA Connectome-based Predictive Modeling (cCPM)

Canonical Correlation Analysis (CCA):

For two sets of observation matrices X and Y, assuming that the variables are correlated, CCA seeks linear combinations of the columns of these two matrices that maximize correlation between them. In other words, we want to find vectors a and b such that the random variables Xa and Yb maximize the correlation. Assuming that X and Y are centered such that each column of either matrix has mean zero, the correlation to be maximized can be expressed by the following equation:

$$\rho =\frac{{\left(Xa\right)}^{\text{T}}\left(Yb\right)}{\sqrt{\left[{\left(Xa\right)}^{\text{T}}\left(Xa\right)\right]\left[{\left(Yb\right)}^{\text{T}}\left(Yb\right)\right]}}$$

CCA Connectome-based Predictive Modeling:

The cCPM pipeline consists of six steps (Figure 1b). In the first step, individuals are divided into training and testing sets using 10-fold cross-validation. We denote the number of individuals for training as N^train and number of individuals for testing as N^test. In the second step, edges are combined. For the k-th edge, we have a matrix E_k ∈ R^Ntrain*M containing edge strength of the k-th edge for all the individuals in the training set. Each row of the matrix E_k denotes each training individual’s k-th edge’s different strengths under m different tasks. Using CCA, we can find the canonical coefficients w_k ∈ R^M for each edge. As each edge matrix E_k corresponds to the observation matrix X in the above definition equation for CCA, these coefficients w_k correspond to the vector a, and the observation matrix Y will store the behavioral measures. We then combine connectomes from all tasks into a total connectivity matrix using different canonical correlations, $E_k^{\mathrm{total}}={\sum }_{m=1}^M{E}_k\left(:,m\right)w_k\left(m\right)$, where the m-th column of E_k is denoted as E_k(:,m). Within each single task, each edge is demeaned across different individuals so that each column of E_k has mean 0. In the third step, we assign the combined edges that are significantly correlated with the behavioral measures to the “correlated network” (CN). The significance of the correlation is found from the CCA. Here, we assume that CCA always maximizes the positive correlation between combined edge strength and behavioral measure as the sign of the canonical coefficients can trivially be changed to maximize the positive correlation. Various significance thresholds for feature selection can be used. In the fourth step, we calculate “network strength” s^CN by pooling (i.e. summing) the strength of all CN edges in each individual’s total connectivity matrix, yielding a summary value s^CN(i) for each individual:

$$s^{{}^{\mathrm{CN}}={\sum }_{\text{k}}\left(B\left(\text{k}\right){\sum }_{\text{m}}{E}_{\text{k}}\left(:,\text{m}\right)w_{\text{k}}\left(\text{m}\right)\right)}$$

where s^CN is the vector of summary values, B(k) is the indicator of whether the k-th edge passes the thresholding for CN. In the fifth step, we use linear regression (y=β₀ + β₁s^CN) to model the association between “network strength” and the phenotypic measure in N^train individuals. In the sixth step, the “network strength” is calculated for the excluded N^test individuals, and is submitted to the corresponding regression model to generate phenotypic measure estimates for those testing individuals. This process is repeated iteratively, with different individuals in the training and testing sets.

2.2.2. Ridge regression Connectome-based Predictive Modeling (rCPM)

Ridge regression:

In ordinary least-squares (OLS) regression, a greater number of independent variables compared to the number of observations leads to an ill-posed problem and overfitting. To solve this ill-posed problem, regularization on regression coefficients can be applied to shrink the coefficients. Ridge regression shrinks the regression coefficients by imposing a L²-norm penalty on their size. Compared with OLS regression, the coefficients from ridge regression minimize a penalized residual sum of squares,

$${\widehat{\beta }}^{\mathrm{ridge}}=\underset{\beta }{\arg \min }\left\{{\sum }_{i=1}^N{\left(y_i-{\beta }_0-{\sum }_{j=1}^p{x}_{ij}{\beta }_j\right)}^2+\lambda {\sum }_{j=1}^p{\beta }_j^2\right\}$$ (1)

where λ is the complexity parameter that controls the shrinkage strength: λ = 0 gives rise to the unregularized OLS, while increasing λ shrinks the coefficients towards zero. If we write the criterion in equation (1) in matrix form, RSS(λ) = (y − Xβ)^T (y − Xβ) + λβ^Tβ, the ridge regression solutions can be solved by

$${\widehat{\beta }}^{\mathrm{ridge}}={\left(X^{\text{T}}X+\lambda I\right)}^{-1}{X}^{\text{T}}y,$$

where I is the identity matrix. Compared with the solution for OLS, ${\widehat{\beta }}^{\mathrm{OLS}}={\left(X^{\text{T}}X\right)}^{-1}{X}^{\text{T}}y,$, adding a positive constant to the diagonal of X^TX before inversion makes the problem nonsingular, even if X^TX is not of full rank.

Ridge regression CPM:

Based on ridge regression, we modify the original CPM framework to better suit the high-dimensional nature of connectivity data (Figure 1c). Specifically, due to the positive semi-definite nature of a functional connectivity matrix, the edges are not independent. Ridge regression is more robust than OLS in this case.

Instead of summing selected edges and fitting a one-dimensional OLS model, we directly fit a ridge regression model with training individuals using the selected edges from all the tasks and apply the model to testing individuals in the cross-validation framework. λ parameter in the ridge regression is chosen by another inner 10-fold cross-validation which uses only the training individuals. The largest λ value that has a mean squared error (MSE) within one standard error of the minimum MSE is chosen. In the Results (Section 3.3), we show that rCPM is not sensitive to λ.

2.3. Experiment setup

Datasets:

We applied all algorithms (see Competing methods below) to the Human Connectome Project (HCP) 900 Subject Release and the Philadelphia Neurodevelopmental Cohort (PNC) first study release. These data releases were the only releases available at the time that this work began.

Phenotypic measure:

In both datasets, a matrix reasoning test—a measure of fluid intelligence (gF)—was used as the phenotypic measure for prediction. In the HCP dataset, a 24-item version of the Penn Progressive Matrices test was used; this test is an abbreviated form of Raven’s standard progressive matrices (Bilker et al., 2012). In the PNC dataset, 24- and 18-item versions of the Penn Matrix Reasoning Test were used (Bilker et al., 2012; Moore et al., 2015). Integer scores indicate number of correct responses (HCP: PMAT24_A_CR, range = 5–24, mean = 17.53, s.d. = 4.45, median = 19; PNC: PMAT_CR (phv00194834.v1.p1.c1), range = 0–23, mean = 12.27, s.d. = 4.04, median = 12).

HCP participants:

From this dataset, we restricted our analyses to those individuals who participated in all nine fMRI conditions (seven task, two rest), whose mean frame-to-frame displacement was less than 0.1 mm and whose maximum frame-to-frame displacement was less than 0.15 mm (see HCP imaging parameters and preprocessing), and for whom gF measures were available (n = 515; 241 males; ages 22–37). This conservative threshold for exclusion due to motion was used to mitigate the substantial effects of motion on functional connectivity; following this exclusion, there was no significant correlation between motion and gF for most conditions (all p > 0.05, Bonferroni corrected) except the Social task, right-left (RL) phase encoding run (r_s = −0.16 (p = 0.00017)), the Relational task, left-right (LR) phase encoding run (r_s = −0.15 (p = 0.0008)), and the Emotion task, RL phase encoding run (r_s = −0.14 (p = 0.0017)).

PNC participants:

From this dataset, we used behavioral, structural imaging, and functional imaging data. We restricted our analyses to those individuals who participated in all three fMRI runs (two task, one rest), on whom registration was successful (nine individuals were excluded for failed registrations), whose mean frame-to-frame displacement was less than 0.1 mm and whose maximum frame-to-frame displacement was less than 0.15 mm (as for the HCP dataset, and with the same motivation), and for whom gF measures were available (n = 571; 251 male, ages 8–21). Following exclusion for motion, there was no significant correlation between motion and gF for any condition (all p > 0.05, Bonferroni corrected).

fMRI processing:

fMRI data were processed with standard methods and parcellated into 268 nodes using a whole-brain, functional atlas defined in a separate sample (see (Greene et al., 2018) for more details). Task functional connectivity was calculated based on the “raw” task timecourses, with no regression of task-evoked activity: the mean timecourses of each node pair were correlated and correlation coefficients were Fisher transformed. Matrices were generated for both the LR and RL phase encoding runs in the HCP data, and these matrices were averaged for each condition, thus generating one 268×268 connectivity matrix per individual per task condition. Nodes that have missing coverage during any individual’s scan were excluded from all individuals (9 nodes in HCP and 18 nodes in PNC were excluded). These matrices were used to generate cross-validated predictive models of gF.

Competing methods:

We compared cCPM and rCPM to four simpler CPM-based approaches. For the first approach, we performed CPM on each task independently as previously demonstrated (Greene et al., 2018). For the second approach, we performed CPM on a general functional connectivity (GFC) matrix created by averaging brain connectivity information across all task conditions (Elliott et al., 2019). As time courses are z-score normalized before creating connectomes, averaging connectomes is similar to first, concatenating time courses and, then, correlating them. However, as the time length of each task is different, by concatenating time series first avoids biasing FC estimates toward the shortest tasks. Here, we generated GFC as suggested in the original paper by first concatenate and then correlate time courses. For the third approach, we compared ridge regression with CPM on GFC matrices. This is similar to performing rCPM on GFC matrices. However, we have chosen not to use the “rCPM” term here to avoid confusion between using GFC and our more direct way of combining multiple connectomes. Finally, for the fourth approach, we used a naïve extension to CPM, where all task connectomes were vectorized and concatenated to create a feature space that contained all task data for CPM. In contrast to the cCPM and rCPM approaches, this naive implementation does not consider any shared or unique information offered by each task. Corrected resampled t-tests (Bouckaert and Frank 2010) were used to compare competing methods.

Internal validation:

10-fold cross-validation was used to train all models. In 10-fold cross-validation, the sample was randomly divided into 10, approximately equal-sized groups; on each fold, the model was trained on 9 groups and tested on the excluded 10th group. This process was iteratively repeated 10 times, with each group excluded once. This procedure was repeated for 100 random divisions. CPM was performed with a range of p-value edge selection thresholds from 0.001 to 0.5. Model performance was evaluated by the cross-validated R²,

$${\text{R}}_{\mathrm{CV}}^2=1-\frac{{{\sum }_{\text{i}=1}^{\text{n}}\left({\text{y}}_{\text{i}}-\widehat{\text{y}}\right)}^2}{{{\sum }_{\text{i}=1}^{\text{n}}\left({\text{y}}_{\text{i}}-\overline{\text{y}}\right)}^2}$$

${\text{R}}_{\text{CV}}^2$ can be negative (Scheinost et al., 2019) and negative values were set to 0. In this paper, $\sqrt{{\text{R}}_{\text{CV}}^2}$ is reported as it is comparable to, but less biased than, the normally used Pearson correlation between observed and predicted measures when using cross-validation. $\sqrt{{\text{R}}_{\text{CV}}^2}$ is averaged over the cross validation folds.

External validation:

Additionally, we trained models using one of the datasets (either HCP or PNC) and applied the model to the other dataset. For external validation, we only used the Emotion and Working Memory tasks from HCP for consistency with the available task data from the PNC. To fairly compare between models, CPM was performed with the 50%, 5% and 1% of edges with lowest p-values. Model performance was evaluated by Pearson correlation coefficient (r_Pearson) between the predicted and observed gF measures.

Quantification of task contribution:

To quantify the contribution of each task to a given predictive model, we calculated the m-th task’s average weight (labeled W_m) to the model as

$$W_m={\sum }_{k}B\left(\text{k}\right){\beta }_k^{m}std\left(E_k\left(:,m\right)\right)$$

where B(k) indexes whether the k-th edge is selected, std(E_k(:, m)) represents the standard deviation of the k-th edge in the m-th task and ${\beta }_k^m$ represents the weight learned by cCPM or rCPM for the k-th edge in the m-th task. To make the results more interpretable, W_m are then normalized to have sum 1, Σ_mW_m = 1, so that it represents each task’s contribution proportion in the whole model.

Similarly, as certain tasks may contain redundant information for prediction, we adopted forward feature selection to select the optimal combination of tasks. Forward feature selection finds the optimal combination of tasks by adding each of the tasks in a stepwise way; in each step, the task that improves prediction the most will be added to the selected task list. The optimal combination is found when any additional task won’t lead to further improvement in prediction. We found the optimal task combination for HCP with both cCPM and rCPM. Both of the two algorithms are performed with a p-value threshold of 0.1.

Sensitivity to hyperparameters:

Although all of the tested approaches are relatively simple in terms of choosing hyperparameters before training the model, all are dependent on the chosen p-value threshold for edge selection. To evaluate the sensitivity of each approach to this hyperparameter, we repeated analyses with p-value thresholds of 0.001, 0.005, 0.01, 0.05, 0.1, and 0.5. The rCPM approach has an additional hyperparameter: the penalty weighting parameter λ. To show that the rCPM models are also not sensitive to the choice of λ, we fixed λ at the average chosen value in cross-validation and varied it in 10% steps to test whether fixing and perturbing λ changes the prediction performance of rCPM.

Data and code availability statement:

The HCP data used in this study are publicly available on the ConnectomeDB database (https://db.humanconnectome.org). The PNC data used in this study are publicly available on the database of Genotypes and Phenotypes (dbGaP accession code phs000607.v1.p1); a data access request must be approved to protect the confidentiality of individuals. MATLAB scripts to run the cCPM and rCPM analyses can be found at (https://github.com/YaleMRRC/CPM). BioImage Suite tools used for analysis and visualization can be accessed at (https://bioimagesuiteweb.github.io/webapp/connviewer.html). MATLAB scripts written to perform additional post-hoc analyses are available from the authors upon request.

3. Results

3.1. Combining multiple connectomes improves prediction accuracy compared with single connectome-based prediction

As shown in Figure 2, all models that incorporate task data significantly predicted fluid intelligence, whereas the models based only on rest did not predict fluid intelligence better than simply predicting the population mean (i.e. ${\text{R}}_{\text{CV}}^2$ did not differ from zero). In both datasets, rCPM (HCP: 0.436±0.0072, PNC: 0.356±0.0078) outperformed (HCP: p = 7.37 * 10⁻⁴, PNC: p = 0.0721, corrected resampled t-test comparing rCPM to the next best performing method, GFC-ridge) all competing approaches. GFC-ridge (HCP:0.387±0.0068, PNC: 0.329±0.151) and cCPM (HCP: 0.386±0.0084, PNC: 0.301±0.0122) performed similarly to each other (HCP: p = 0.43, PNC: p = 0.18) in both the two datasets and outperformed the other competing methods. The naïve CPM implementation (HCP: 0.354±0.0094, PNC: 0.293±0.0130) and CPM using a GFC matrix (HCP: 0.333±0.0086, PNC: 0.263±0.0151) had similar performance to models built from the best-performing single task, working memory (HCP: 0.322±0.0134, PNC: 0.288±0.0144). We also tested the performance of ridge regression on single task connectomes. While the overall prediction performance increases when using ridge regression, rCPM still significantly outperforms ridge regression on single task connectomes (HCP: p = 0.0012, PNC: p = 0.0146, comparing rCPM to the next best performing single task with ridge regression).

Figure 2. Comparison of the predictive modeling approaches’ ability to predict an individual’s gF.

a) HCP dataset. b) PNC dataset. Purple box plots show results from CPM on a single task. The orange, green, red, and blue box plots show results from combining multiple task connectomes using GFC-CPM, GFC-ridge, CPM, cCPM, and rCPM, respectively. Box plots show cross-validated $\sqrt{{\text{R}}_{\text{CV}}^2}$ with the error bars representing the 25^th and 75^th percentiles, respectively. Values below (or above) the 25th (or 75th percentiles) are shown as *. The best results across different edge selection thresholds are shown. Task acronyms: GAM: Gambling, LAN: Language, MOT: Motor, REL: Relational, SOC: Social, WM: Working Memory, EMO: Emotion.

3.2. Different tasks contribute differentially to the model

As shown in Figure 3, different tasks contribute different numbers of edges to the final model. Tasks that are more predictive by themselves (e.g. Working Memory) tend to contribute more edges to the model, while less predictive tasks (e.g. Emotion) contribute fewer edges (Figure 3a). Similarly, in terms of contribution W_m to the regression model, tasks that are predictive by themselves contribute more to the overall predictive model while less-predictive tasks contribute less (Figure 3b). Additionally, tasks appear to select different edges in different networks for prediction (Figure 3c). In the figure, percentage of edges in each network pair that are selected by the model are shown. While the presented results are mostly for rCPM (Figure 3ac), the same trend is observed for cCPM (Figure S2). As shown in Figure S3, the predictive utility of a task is not dependent on the length of the task.

Figure 3. Different tasks’ contributions to the model.

a) Visualization of the selected edges for different tasks in the model. Top row represents 2% and bottom row represents 20% of total number of selected edges. 81.7% and 99.8% of feature contribution (combined sum of each feature’s regression coefficient times its standard deviation) in regression are possessed respectively by those networks. Anatomical acronyms: PFC = Prefrontal, MOT = MotorStrip, INS = Insula, PAR = Parietal, TEM = Temporal, OCC = Occipital, LIM = Limbic, CER = Cerebellum, SUB = Subcortical, BSM = Brainstem. b) Different tasks’ average contribution fraction to the cCPM and rCPM model. c) Different tasks’ contributions to the model, summarized at the network level. Network acronyms: MF=Medial Frontal, FP=Frontoparietal, DMN=Default Mode Network, MOT=Motor Cortex, V1=Visual I, V2=Visual II, VA=Visual Association, SA=Salience.

Finally, as shown in Figure 4, using stepwise forward task selection, the second-best-performing individual task was not added to the combined model until step 5 or 6 (for rCPM and cCPM, respectively). Similarly, the best-performing model did not use all tasks. Together, these results suggest that certain tasks may contain redundant information for prediction and that an optimal combination of tasks with complementary information is needed to maximize prediction performance.

Figure 4. Forward task selection for cCPM and rCPM.

a) shows the results for cCPM while b) shows the results for rCPM. The optimal task combinations for the two algorithms are both using 6 tasks, where cCPM excludes the Language task while rCPM excludes the Emotion task. However, the algorithm performance of using all 7 available tasks is not significantly worse than using 6 tasks (cCPM: p = 0.38, rCPM: p = 0.40), and overall, including more tasks significantly improves prediction.

3.3. Evaluation of hyperparameters on model performance

As shown in Figure 5ab, the performance of all approaches varies as a function of the p-value threshold in the edge-selection step. The p-value threshold controls the number of edges retained in the models. A lower p-value threshold represents more stringent edge selection and fewer edges will enter the final model. The GFC-CPM, GFC-ridge, the naïve CPM and cCPM all exhibit a decrease in performance as the p-value threshold is increased (i.e. more retained features), except for cCPM and GFC-ridge on the HCP dataset. In contrast, rCPM exhibits better performance as the p-value threshold is increased. As a result, the improvement in prediction performance offered by rCPM over the competing approaches is at its maximum at a higher p-value threshold (i.e. p = 0.1 or p = 0.5). However, the extra computation cost induced by more features should also be considered. As shown in Figure 5c, results from rCPM are insensitive to the choice of λ over the tested range.

Figure 5. Models’ performance with various hyperparameters.

a) Varying edge selection threshold for the HCP dataset. b) Varying edge selection threshold for the PNC dataset. c) Varying penalty weighting parameter for rCPM. In a) and b) edge selection threshold=1.0 represents no edge selection. Horizontal line indicates prediction accuracy with λ chosen by inner cross validation.

3.4. Models trained on one dataset can be generalized to another dataset

Figure 6 shows that our models generalized to independent, external datasets. By showing the results under different edge sparsity levels, we validate that model generalizability does not decrease with more edges as more features often lead to overfitting. Actually, models based on more edges still outperform models based on fewer edges. GFC’s generalizability is also tested here as it can potentially utilize all the available task scans in HCP dataset and apply it to PNC, which has fewer tasks. However, we didn’t see improvement in generalizability by including more tasks.

Figure 6. Different models’ generalizability to independent, external datasets.

a) Trained on HCP and applied to PNC. b) Trained on PNC and applied to HCP. b) Trained on PNC and applied to HCP. The results are presented as Pearson correlation between predicted and actual measures. Models trained on either HCP or PNC datasets can significantly predict gF in the other dataset.

3.5. Exploratory comparison of ridge regression to lasso (least absolute shrinkage and selection operator) and elastic net

Given the improved performance of rCPM, we explored the performance of CPM approaches based on two other regularized regression approaches, lasso and elastic net. Instead of imposing a L²-norm penalty on regression coefficient β as in ridge regression, lasso uses a L¹-norm penalty $\lambda {\sum }_{j=1}^p|{\beta }_j|$. This encourages coefficients to be set to zero, while ridge regression only shrinks the size of the coefficients. Thus, lasso is often preferred for feature selection when sparsity is preferred. However, lasso tends to select only a small number of variables when the sample size is small. To overcome this limitation, elastic net regularization combines both the L¹-norm penalty and L²-norm penalty. An α hyperparameter is used to balance between the weight of the two kinds of penalty. So the overall objective function for elastic net can be written as,

$${\widehat{\beta }}^{\mathrm{EN}}=\begin{array}{c}\arg \min \\ \mathbf{\beta }\end{array}\left\{{\sum _{i=1}^N\left(y_i-{\beta }_0-\sum _{j=1}^p{x}_{ij}{\beta }_j\right)}^2+\lambda \left(\frac{\left(1-\alpha \right)}{2}\sum _{j=1}^p{\beta }_j^2+\alpha \sum _{j=1}^p|{\beta }_j|\right)\right\}$$

For this objective function, the bigger α is, the more lasso-type shrinkage will be put on the coefficients. When α = 1, it is the same as lasso; when α = 0, it is the same as ridge regression. Implementations of these approaches are identical to rCPM with the exception that the ridge regression step is replaced with either lasso or elastic net.

In our experiments, neither elastic net nor lasso performed as well as ridge regression (HCP: p = 4.4 * 10⁻⁵, PNC: p = 0.0035). For both datasets, α = 0 always produces the best prediction (Figure 7), suggesting—again—the importance of including a large number of edges in a predictive model to best reflect distributed patterns of functional connectivity.

Figure 7. Comparison of ridge regression with Elastic Net and Lasso.

α, the weighting parameter between ridge and lasso-type regularization, is varied across different values. α = 0 is the same as ridge regression while α = 1 is the same as lasso. Ridge regression generates the most accurate prediction in both the HCP and PNC datasets.

4. Discussion

In this paper, we proposed cCPM and rCPM, two general connectome-based prediction frameworks that utilize complementary information in different task connectomes to improve phenotypic prediction. We tested the two algorithms on two open-source datasets, HCP and PNC, to predict fluid intelligence using all the available task connectomes. rCPM shows superior performance in prediction for within-sample prediction compared to the competing methods, including cCPM, though all methods performed similarly for out-of-sample prediction. By looking at the contribution of specific tasks to the final prediction and the stepwise forward optimal task combination selection, we found that different tasks contribute differentially to the final predictive model. In contrast to other competing methods, rCPM performed better when the number of features included in the ridge regression step was large and included features only weakly correlated with intelligence. Last but not least, although the model takes in a large number of features, we validated its generalizability and robustness by using a range of hyperparameters and testing models on external datasets. Overall, cCPM and rCPM provide a powerful framework to combine all available functional imaging data into a single predictive model.

The major contribution of our paper is showing that combining connectomes in an appropriate manner improves prediction. Although task-based connectomes improve prediction performance compared to resting-state connectomes (Greene et al., 2018), approaches to combine multiple task-based connectomes into a single predictive model are limited. The GFC method is one approach to combine task-based connectomes (Elliott et al., 2019). However, by simply averaging over all conditions, GFC loses a large amount of task-specific information, which can be used to increase prediction performance, and performs worse than the other methods that combine multiple connectomes for predictive modeling.

Several regression approaches exist to shrink regression coefficients in the case of highly correlated features, which is common in connectome-based predictive modeling. Two common approaches are principal components regression (PCR), and partial least squares (PLS). Ridge regression shrinks both the high- and low-variance directions of the features, but applies greater shrinkage to the low-variance directions (Krishnan et al., 2011; Mwangi et al., 2014; Zhong et al., 2009) In contrast, PCR does not shrink the high-variance directions and simply discards the lower-variance directions (Frank and Friedman, 1993). PLS also shrinks the low-variance directions, but may inflate the higher variance directions (Frank and Friedman, 1993). Based on these theoretical considerations and additional experimental observations (see for example (Dubois et al., 2018; He et al., 2018)), ridge regression may be preferred for minimizing prediction error because of its smooth shrinkage (Frank and Friedman, 1993), though we did not test these other approaches in the context of connectome-based predictive modeling. Those related approaches will be tested in future work.

Similarly, ridge regression generated better predictions than approaches that perform shrinkage and feature selection, like lasso and elastic net. While popular in neuroimaging machine learning (Dadi et al., 2019), lasso has several limitations compared to ridge regression (Zou and Hastie, 2005). When the number of features is greater than the sample size, lasso limits the number of nonzero features to be the same as the sample size, even though additional features may be associated with the phenotype of interest. Similarly, lasso tends to retain only one feature from any set of highly correlated features, shrinking the other features to zero. In cCPM and rCPM (and other connectome-based predictive modeling approaches), the number of features (i.e. edges) is typically greater than the number of individuals and features are highly correlated. While elastic net regularization attempts to combine the strengths of ridge regression and lasso (Zou and Hastie, 2005), we did not observe any improvements in prediction performance over ridge regression with elastic net.

Despite both approaches incorporating complementary task information, rCPM significantly outperformed cCPM. As only one phenotypic measure was used in these analyses, cCPM simplifies to linear regression. Given this, we chose to follow the CPM design of pooling selected features, rather than constructing a single large matrix with all selected features for linear regression. This single matrix would be rank deficient, resulting in unstable solutions. Ridge regression is a natural answer to this problem as the regularization term allows for stable solutions that minimize the effect of noisy edges on model performance.

RCPM demonstrated improved prediction performance when more features were included in the model. Although rCPM incorporates a modest feature selection step, retaining nearly 60% of the edges (edge number retained with p < 0.5) during this step produced the best prediction performance. We attribute these results to ridge regression’s ability to shrink noisy and correlated features, thereby, reducing their influence on the model. Moreover, these results suggest that even edges that are not strongly correlated with the behavioral measures still help with prediction, phenotypic information is encoded in spatially distributed connectivity patterns, and (almost) all available information should be used for prediction. These results also align with recent work in “double descent” test risk of predictive models, which suggests a lower test risk when large numbers of features—more than the number of samples—are included (Belkin et al., 2019).

A natural question that arises is: if each task differentially contributes to the final predictive model, what tasks should be included when designing a study? When using a single task, results suggest that tasks and phenotypes of interest should be matched such that the selected task perturbs brain circuits relevant to the phenotype (Greene et al., 2018; Rosenberg et al., 2015). Yet, when combining multiple tasks, including tasks with complementary information appears to be the most beneficial. For example, in the forward feature selection result (Figure 4), the second task added is the motor task, which by itself is a poorer predictor of fluid intelligence than either the gambling or language task. However, the motor task likely provides more complementary information than the gambling or language task and, thus, provides the maximum gain in prediction power when added to the model. To generalize, we hypothesize that a battery of tasks that perturbs multiple and complementary brain circuits will yield better predictive models than a battery of tasks that perturbs a single brain circuit with different, subtle manipulations. While it may not be possible to collect a single battery of tasks that is optimal for all phenotypic information, we suspect that a standard battery could be developed that is good enough in most cases. Future work should develop and compare different batteries of tasks in terms of predictive modeling.

This work has some limitations. First, currently for cCPM and rCPM, all individuals are required to have complete data from all the tasks. As such, using cCPM and rCPM increases the likelihood of removing individuals from the analysis due to missing data. As the utility of predictive models is often dependent on sample size (Cui and Gong, 2018; Varoquaux et al., 2017), cCPM and rCPM may only be applicable to larger datasets that can support removing individuals due to missing data. Future work includes extending cCPM and rCPM to support data imputation methods to handle missing data. Second, cCPM and rCPM may make model interpretation and visualization more challenging. While cCPM and rCPM retain CPM’s ability to simply map features back to the brain, the mappings from both of the two algorithms are task specific. Ultimately, the task at hand will determine if the added task information reduces interpretability of the model (see Rule #10 from (Scheinost et al., 2019) for a greater discussion). RCPM exhibited increased prediction performance when the number of features was larger, and while better prediction is good, the large number of edges can be difficult to visualize and interpret. It should be noted however that this may be a more accurate depiction of the complex systems involved in executing cognitive functions, and our tendency to reduce findings to one or two brain regions is likely grossly oversimplifying this complex system. In other words, the circuits involved in cognition may not lend themselves to easy visualization. Third, cCPM and rCPM can make external validation harder. The external dataset needs to have similar tasks for the model to be applicable. For example, when using external validation between the HCP and the PNC datasets, we limited the tasks from the HCP to only two for compatibility with the PNC dataset. Fourth, cCPM and rCPM did not show improvement for out-of-sample prediction. This result could suggest that cCPM and rCPM overfit for within-sample prediction or that the HCP and PNC datasets differ in important aspects, such as age, that limit generalizability of any model (see Rule #9 from (Scheinost et al., 2019)). Fifth, similarly, due to the large number of edges (features) involved in the prediction models compared with the sample size, overfitting is hard to eliminate completely (Whelan and Garavan, 2014) and this causes the difference between our results with the previous works (Finn et al., 2015). However, with the use of both cross-validation and independent dataset generalization, our evaluation results are less-prone to suffer from overestimation. Finally, while we focus on connectivity data derived from multiple fMRI tasks, cCPM and rCPM are agnostic to the type of input data and can easily incorporate structural connectivity data from DTI or from other functional modalities like EEG. In future work, we will explore the inclusion of other measures to further improve prediction.

In summary, we present cCPM and rCPM, two extensions to CPM to handle connectomes from multiple sources. Our results suggest that prediction of phenotypic measures can be improved by including multiple task conditions in computational models, that different tasks provide complementary information for prediction, and that cCPM and rCPM provide two principled methods for modeling such data.