A Method to Compare the Discriminatory Power of Data-driven Methods: Application to ICA and IVA

Abstract

Background:

The widespread application of data-driven factorization-based methods, such as independent component analysis (ICA), to functional magnetic resonance imaging data facilitates the study of neural function and how it is disrupted by psychiatric disorders such as schizophrenia. While the increasing number of these methods motivates a comparison of their relative performance, such a comparison is difficult to perform on real fMRI data, since the ground truth is, relatively, unknown and the alignment of factors across different methods is impractical and imprecise.

New Method:

We present a novel method, global difference maps (GDMs), to compare the results of different fMRI analysis techniques on real fMRI data, quantify their relative performances, and highlight the differences between the decompositions visually.

Comparison with Existing Methods:

We apply this method to compare the performances of two different factorization-based methods, ICA and its multiset extension independent vector analysis (IVA), for the analysis of fMRI data from 109 patients with schizophrenia and 138 healthy controls during the performance of three tasks.

Results:

Through this application of GDMs, we find that IVA can determine regions that are more discriminatory between patients and controls than ICA, though IVA is less effective at emphasizing regions found in only a subset of the tasks.

Conclusions:

These results demonstrate that GDMs are an effective way to compare the performances of different factorization-based methods as well as regression-based analyses.

1. Introduction

Due to its high spatial resolution and non-invasive nature, functional magnetic resonance imaging (fMRI) data has become one of the most popular means of understanding normal neural function as well as how it is disrupted by disorders, such as schizophrenia (Logothetis, 2008). The data processing strategies for fMRI data can be roughly grouped into two schemes: hypothesis-driven and data-driven (X. Zhao et al., 2004). Hypothesis-driven methods exploit actual or supposed a priori knowledge about brain activity and, generally, study neurological relationships across a few regions or with respect to specific stimuli. Data-driven methods, on the other hand, offer a less targeted and more holistic approach, often by decomposing the observed data into a set of factors. Such techniques include: principal component analysis (PCA) (Thirion & Faugeras, 2003; Zhong et al., 2009), independent component analysis (ICA) (Mckeown et al., 1998; Calhoun, Adali, Pearlson, & Pekar, 2001; Stone, Porrill, Porter, & Wilkinson, 2002; Wang, 2011; Adali, Levin-Schwartz, & Calhoun, 2015), dictionary learning/sparse coding (DL) (Abraham, Dohmatob, Thirion, Samaras, & Varoquaux, 2013; Abolghasemi, Ferdowsi, & Sanei, 2015; S. Zhao et al., 2015), non-negative matrix factorization (NMF) (Lohmann, Volz, & Ullsperger, 2007; Ferdowsi, Abolghasemi, & Sanei, 2010), tensor-based methods (Davidson, Gilpin, Carmichael, & Walker, 2013; Kuang et al., 2013), and independent vector analysis (IVA) (J.-H. Lee, Lee, Jolesz, & Yoo, 2008). The performance of each of these factor models depends on the validity of their modeling assumptions for the dataset being analyzed and, thus, motivates a comparison of different factor models on the same dataset. However, it is difficult to compare the performance of different factor models on real data, since the ground truth is not known and each method typically produces multiple factors.

In order to avoid this issue, many papers that compare different factorization techniques focus on their performance on simulated data, see e.g., (X. Zhao et al., 2004; Degras & Lindquist, 2014; Kuang et al., 2015; Engberg, Andersen, Mørup, & Madsen, 2016). However, these artificial datasets are usually simple when compared with real fMRI data (Welvaert & Rosseel, 2014; Eklund & Nichols, 2017). When comparing the performance of different factor methods on real fMRI data, most papers align factors from different methods and then rely on a visual comparison, see e.g., (Sui, Adali, Pearlson, Clark, & Calhoun, 2009; Erhardt et al., 2011; K. Lee, Tak, & Ye, 2011; Moeller, LeVan, & Gotman, 2011; Schöpf, Windischberger, Kasess, Lanzenberger, & Moser, 2010). However, aligning even a subset of the total number of factors from multiple techniques can be time consuming, due to the potentially large number of factors from each method. Additionally, each method exploits different properties of the signal (Lahat, Adali, & Jutten, 2015) and such comparisons are inherently subjective, since they rely on visual interpretation. Another metric for the comparison of different factorization methods is reproducibility or generalizability, i.e., their ability to produce similar factors across different subjects and sessions (Strother et al., 2002; Afshin-Pour, Hossein-Zadeh, Strother, & Soltanian-Zadeh, 2012; Rasmussen, Abrahamsen, Madsen, & Hansen, 2012; Roels, Bossier, Loeys, & Moerkerke, 2015). However, focusing solely on reproducibility ignores how informative the extracted factors are for a given task. In the case where multiple groups, such as those affected by a psychiatric disease and those who are healthy are analyzed, the ability of a factor to differentiate between the groups can be used to determine the performance of different factorization methods, see e.g., (Sui et al., 2010; Ramezani, Marble, Trang, Johnsrude, & Abolmaesumi, 2015; Levin-Schwartz, Calhoun, & Adali, 2017). The use of this criterion as a measure of performance is well motivated and exploits the knowledge that there should be some brain-related differences between the groups. However, the technique does not solve the fundamental problem of alignment, since often there are multiple discriminatory factors for each method.

In this paper, we present a novel model comparison technique, global difference maps (GDMs), and demonstrate how they can be used to visually highlight the differences between factorization methods and quantify the discriminative or relational power of a dataset within a decomposition. We apply this technique to highlight the differences between individual analyses, using ICA, and a joint analysis, using IVA, of three fMRI tasks: an auditory oddball (AOD) task, a Sternberg item recognition paradigm (SIRP) task, and a sensorimotor (SM) task. Through this application, we show how GDMs can be an effective way to compare the performances of different factorization-based methods. Results show that IVA can determine regions that are more discriminatory between patients and controls than ICA, however, this improvement in discriminatory power comes at the cost of not emphasizing some of the regions found using ICA in a subset of the tasks.

2. Materials and Methods

2.1. Feature Extraction

Since the timing of the stimuli in each task is different, it is difficult to jointly analyze multi-task fMRI data. Rather, for each subject, a simple linear regression is run on the data from each voxel using the statistical parametric mapping toolbox (SPM) (SPM5, 2011), where the regressors are created by convolving the hemodynamic response function (HRF) in SPM with the desired predictors for each task. The resulting regression coefficient maps are used as features for each subject and task. This reduction, using a lower-dimensional though still multivariate representation of the data, facilitates exploration of the associations across the features from multiple tasks, see e.g., (Calhoun et al., 2006; Ramezani et al., 2014), as well as enabling the identification of intrinsic functional networks (Calhoun & Allen, 2013).

2.2. FMRI Tasks and Features

The datasets used in this study are from the Mind Research Network Clinical Imaging Consortium Collection (Gollub et al., 2013) (publicly available at http://coins.mrn.org). These datasets were obtained from 247 subjects, 109 patients with schizophrenia and 138 healthy controls. Next, we introduce the tasks as well as the associated multivariate features analyzed in this study.

2.2.1. Auditory Oddball Task

This auditory task required the subjects to listen to three different types of auditory stimuli in a pseudo-random order: standard (1 kHz tones occurring with a probability of 0.82), novel (complex sounds occurring with a probability of 0.09), and target (1.2 kHz tones occurring with a probability of 0.09, to which a right thumb button press was required) (Kiehl & Liddle, 2001; Gollub et al., 2013). Each run of the task consisted of 90 stimuli, each with a 200 ms duration and a randomly changing interstimulus interval of between 550 and 2,050 ms. The order of novel and target stimuli was changed between runs, thus ensuring that the responses were not dependent on the stimulus order (Gollub et al., 2013). For this task, the regressor was created by modeling both the target and standard stimuli as delta functions and convolving this sequence of delta functions with the default SPM HRF in addition to their temporal derivatives (Michael et al., 2009). Subject averaged contrast images between the target versus the standard tones were used as the feature for this task.

2.2.2. Sternberg Item Recognition Paradigm Task

This visual task required the subjects to remember a set of 1, 3, or 5 randomly chosen numbers between 0 and 9. The task paradigm consisted of: a 1.5 second learn condition, a blank screen for 0.5 seconds, a 6 second encode condition, where the whole sequence of digits was presented together, and a 38 second probe condition, where the subject was shown a sequence of integers and had to indicate, with a right thumb button press, whether or not it was a member of the memorized set (Gollub et al., 2013). Each probe digit was presented for up to 1.1 seconds in a pseudo-random fashion within a 2.7 second interval (Gollub et al., 2013). A total of 84 probes, consisting of 42 targets and 42 foils was obtained per scan and the prompt-encode-probe conditions were run twice for each set size in a pseudo-random order (Gollub et al., 2013). For this task, the regressor was created by convolving the probe response for the three digit set with the default SPM HRF (Michael et al., 2009). This was done for both runs of the probe response and then the average map was computed and used as the feature for this task.

2.2.3. Sensory Motor Task

This auditory task required the subjects to listen to a sequence of auditory stimuli that consisted of 16 different tones each lasting 200 ms and ranging in frequency from 236 Hz to 1,318 Hz with a 500 ms inter-stimulus interval (Gollub et al., 2013). The first tone presented was at the lowest pitch and each subsequent tone was higher in pitch than the previous one until the highest tone was reached, then the order of the tones was reversed (Gollub et al., 2013). Each tonal change required a right thumb button press. A run consisted of 15 increase-and-decrease blocks, alternated with 15 fixation blocks, with each block lasting 16 seconds in duration (Gollub et al., 2013). For this task, the regressor was created by convolving the entire increase- and-decrease block with the default SPM HRF (Michael et al., 2009). This was done for both runs for each subject and then the average map was used as the feature for this task.

2.3. ICA

For an fMRI feature dataset from M subjects, $\mathbf{X}\in {ℝ}^{M\times V}$, where the mth row of X is formed by flattening the feature of V voxels from the mth subject, the noiseless ICA model can be written as

$$\mathbf{X}=\mathbf{AS},$$ (1)

where the C spatially independent latent sources or neural activation patterns, $\mathbf{S}\in {ℝ}^{C\times V}$, are linearly mixed by the mixing matrix, $\mathbf{A}\in {ℝ}^{M\times C}$. Given this model, ICA seeks to determine a demixing matrix, W, such that the estimated components, the rows of $\widehat{S}=\mathbf{WX}$, are as independent as possible. Since we seek to maximize independence between the estimated components, ${\widehat{s}}_1,\dots ,{\widehat{s}}_C$, estimation of the demixing matrix can be accomplished through the minimization of the mutual information among the components, written as

$${\mathcal{I}}_{\mathrm{ICA}}(\mathbf{W})=\sum _{i=1}^{C}H({\widehat{\mathbf{s}}}_i)-\log \mid \text{det}(\mathbf{W})\mid -H(\mathbf{X}),$$ (2)

where H(·) is the entropy.

Since fMRI data is of high dimensionality and quite noisy, i.e., is in a space consisting of signal as well as noise, dimension-reduction using PCA is a crucial preprocessing step in order to avoid over-fitting in subsequent analyses. Determining the appropriate order—the size of the signal subspace—for this PCA step using real fMRI data is an active area of research, see e.g, (Cordes & Nandy, 2006; Y.-O. Li, Adali, & Calhoun, 2007; X. Xie, Cao, Weng, & Jin, 2009; Chen et al., 2010; Yourganov et al., 2011; Hui et al., 2013). Despite its fairly extensive study, the majority of order estimation techniques are appropriate for single subject or single task analyses, which limits their applicability to the joint analysis of multiple tasks, multiset fusion. The reason for this is that these techniques ignore the complementary information between the datasets, which is the main motivation for performing a joint analysis on such data. In this work, we use the order estimated by a modification of the procedure in (Levin-Schwartz, Song, Schreier, Calhoun, & Adali, 2016). The reason for this is because, to the best of our knowledge, it is the only method that has shown desirable performance for the task of order estimation in both the sample-poor and sample-rich regimes inherent to the fusion of medical imaging data. The technique, named PCA and canonical correlation analysis (PCA-CCA), estimates the size of the signal subspace shared by two datasets, i.e., the common order, through a sequence of binary hypothesis tests. The process begins by assuming that the common order is 0 and that the null hypothesis is that the current common order is appropriate. If the null hypothesis is rejected, the common order is increased by 1 and the test is repeated, until the null hypothesis cannot be rejected or the common dimension is equal to half the number of subjects, M/2. In this application, we estimate the order for each pairwise combination of tasks and then select the highest estimated order to enable the retention of the most complementary information across the datasets.

We should note that although there are many ICA algorithms, in this work, we used the ICA by entropy bound minimization (EBM) algorithm, due to the fact that it has shown superior performance in both simulated and real neurological data when compared with the popular Infomax algorithm (Bell & Sejnowski, 1995), see e.g., (X.-L. Li & Adali, 2010b; Adali, Anderson, & Fu, 2014; Adali et al., 2015). This improved performance derives from the fact EBM does not assume a fixed form for the distribution of the latent sources and instead attempts to upper bound their entropy through the use of measuring functions (X.-L. Li & Adali, 2010b). The use of these measuring functions enables the description of a wide variety of distributions, including those that are: unimodal, bimodal, symmetric, and skewed (X.-L. Li & Adali, 2010b), thus, generally, improving the estimation of all sources within the mixture.

2.4. IVA

FMRI data from subjects performing multiple tasks has been increasingly gathered during the same study, since each task is expected to provide related information regarding neural function and how it is impacted by neurological diseases, such as schizophrenia, see e.g., (Calhoun et al., 2006; Fitzgerald et al., 2008; Michael et al., 2009; Mijovic et al., 2012; Ramezani et al., 2015). In order to address this scenario, we extend the model in (1) to K datasets, or tasks, as

$${\mathbf{X}}^{[k]}={\mathbf{A}}^{[k]}{\mathbf{S}}^{[k]},1\le k\le K.$$ (3)

Due to the scaling and permutation ambiguities inherent to ICA, running a separate ICA individually for each task and aligning the results is both impractical and suboptimal, since it would ignore the complementary information that each task provides (Lahat et al., 2015). This motivates the performance of a joint analysis, such as using IVA, a recent multiset extension of ICA, which exploits similarities across datasets to achieve a successful decomposition (J.-H. Lee et al., 2008).

The mutual information cost function for IVA can be written as,

$${\mathcal{I}}_{\mathrm{IVA}}({\mathbf{W}}^{[k]})=\sum _{c=1}^{C}H({\widehat{\mathbf{S}}}_c)-\sum _{k=1}^K\log \mid \text{det}({\mathbf{W}}^{[k]})\mid -\sum _{k=1}^{K}H({\mathbf{X}}^{[k]}),$$ (4)

where ${\widehat{S}}_c={\left[{\widehat{s}}_c^{[1]},\dots ,{\widehat{s}}_c^{[k]}\right]}^{\mathrm{T}}\in {ℝ}^{K\times V}$ is the cth source component matrix (SCM), formed by concatenating the cth estimated component from each of the datasets. Note that the difference between (2) and (4) is that we are now minimizing the mutual information between SCMs and not components. This cost function reduces to the sum of separate ICAs performed on each dataset individually if the sources in each of the datasets are mutually independent of each other (Adali et al., 2014). In this work, IVA via a multivariate Gaussian and Laplacian algorithm (Ma, Calhoun, Phlypo, & Adali, 2014) is used, since it has proven to be an effective IVA algorithm for analysis of fMRI data, see e.g., (Ma et al., 2014; Gopal et al., 2016).

2.5. GDMs

As discussed in Section 2.3, since the data from each subject within a task has been reduced to a feature, the columns of the estimated mixing matrix, A, provide the weight of each component across the subjects. This means that the ith column of the estimated mixing matrix, a_i, represents the subject-specific weights of the ith source estimate, s_i. Thus, if the goal of the study is to determine a set of explanatory factors that can be used to assess differences between two groups, as is the primary focus for this work where the groups are patients with schizophrenia and healthy controls, the weights can be used to determine if any component is expressed differently across the two groups. In order to statistically determine components that have different expressions, on average, between patients and controls, a two-sample t-test is performed on columns of A, where one group is represented by the weights corresponding to the patients with schizophrenia and the other by the weights corresponding to the healthy controls (Calhoun & Adali, 2009). The formula for the two-sample t-test is presented below

$$t=\frac{{\stackrel{‒}{\mu }}_1-{\stackrel{‒}{\mu }}_2}{\sqrt{\frac{{\sigma }_1^2}{n_1}+\frac{{\sigma }_2^2}{n_2}}},$$ (5)

where μ_i, ${\sigma }_i^2$, n_i are the mean, variance, and number of samples in group i. Note that when the difference between the weights of the two groups is statistically significant, the component describes functional differences between patients and controls and is referred to as a possible biomarker of disease. The t-statistics derived from these tests can serve as the basis for the construction of the GDMs.

Before describing how we construct GDMs, we should note that, since an important focus of this study is differentiating patients with schizophrenia from healthy controls, we focus on how GDMs can highlight the brain regions that differentiate between two groups. However, as we describe at the end of this section, GDMs can also be constructed such that they reflect the regions that are correlated with certain behavioral variables of interest, highlighting associations between psychological tests and neural function.

Due to the scaling ambiguity of ICA, prior to the construction of GDMs, Z-scores are independently computed from the individual biomarkers using the mean and standard deviation computed using all voxels. Let N be subset of the C components estimated using separate ICAs or IVA for each dataset with weights that are statistically significantly different, at p < 0.05, between patients and controls. Denoting these components as ${\widehat{s}}_n^{[k]},1\le n\le N$, we construct the GDM for that method and dataset, ${\widehat{s}}_{\mathrm{GDM}}^{[k]}$, as follows

$${\widehat{s}}_{\mathrm{GDM}}^{[k]}=\sum _{n=1}^N\frac{t_n}{\sum _{m=1}^N{t}_m}{\widehat{s}}_n^{[k]},1\le k\le K,$$ (6)

where t_n is the t-statistic, calculated as described in Section 2.3, for the nth component that is positive or made to be positive by multiplying the corresponding subject covariation and component by −1. Note that due to the sign ambiguity associated with ICA and IVA, multiplying both the component as well as the subject covariation by −1 will produce the same solution (Hyvärinen, Karhunen, & Oja, 2001). The physical interpretation of such a transformation is that, for example, rather than having patients have increased activation in a certain region over controls, controls have increased deactivation over patients. The GDM can be seen as a summary map that describes only the regions that activate/deactivate significantly differently between patients and controls for a given decomposition and dataset within that decomposition. Each biomarker is scaled by the value of its corresponding t-statistic, so has more weight if the component is better able to differentiate between patients and controls. Note that though we describe the construction of GDMs for factors extracted using ICA and IVA, they could also be constructed using the results of any factor analysis, such as PCA, NMF, or DL. Additionally, though beyond the scope of this work, GDMs can also be defined based upon regression models, which are useful when there are less clearly defined groups. Though we should note that using general linear models where the regressor is the disease state of the subjects is, generally, less robust than factor models in this case (Calhoun, Adali, Stevens, Kiehl, & Pekar, 2005).

We can quantify the discriminative power of a GDM, and thus, indirectly, the whole decomposition, by generating component weights in a nearly identical manner to the GDM spatial maps and performing a two-sample t-test on the resulting weights. The corresponding component weights for the GDM are given by

$${\widehat{a}}_{\mathrm{GDM}}^{[k]}=\sum _{n=1}^N\frac{t_n}{\sum _{m=1}^N{t}_m}{\widehat{a}}_n^{[k]},1\le k\le K,$$ (7)

where ${\widehat{a}}_n^{[k]}$ is the subject covariation, obtained using ICA or IVA, corresponding to ${\widehat{s}}_n^{[k]}$ and, just as for ${\widehat{s}}_n^{[k]}$, is multiplied by −1 if the sign of t_n is negative. This construction is expected to result in component loadings that show greater discriminatory power than the original ${\widehat{a}}_n^{[k]}$, since ${\widehat{a}}_{\mathrm{GDM}}^{[k]}$ is constructed from only those component loadings that are statistically significant. Additionally, the ${\widehat{a}}_n^{[k]}$ can be modeled as unit step functions with weights corresponding to their disease status and corrupted with Gaussian noise, where higher values of t_n imply lower variances for the noise. Therefore, the summation of multiple ${\widehat{a}}_n^{[k]}$, weighted as in (7), will tend to result in a lower value for the variance of the noise and, therefore, a more discriminative subject covariation. Additionally, note that the t-tests performed on the columns of the mixing matrices may be seen as a feature extraction step, thus corrections for multiple comparisons should only account for the number of GDMs calculated.

We should note that, by construction, the intensity of each voxel in a GDM is dependent on a linear combination of voxel values across components. For this reason, the GDMs produced by algorithms that have fewer overlapping voxels across components are expected to be more easily interpretable. This is true for methods that promote sparsity, such as: Infomax, IVA-L (J.-H. Lee et al., 2008), or dictionary learning (Abraham et al., 2013). This is also true for methods that make few assumptions about the distribution of the latent sources, such as: EBM or SparseICA-EBM (Boukouvalas, Levin-Schwartz, Calhoun, & Adali, 2018), in addition to methods that account for sample dependence between voxels, such as: weights-adjusted second-order blind identification (Yeredor, 2000) as well as ICA and IVA using entropy-rate bound minimization (X.-L. Li & Adali, 2010a; Fu, Anderson, & Adali, 2015).

A fundamental goal of neuroimaging studies is to understand the underlying neural basis for psychological measurements (Shen et al., 2014; Wan et al., 2014; Meng et al., 2017). GDMs lend themselves to this goal in a straightforward manner and through their application can facilitate the discovery of which neuroimaging datasets most effectively explain the results of psychometric tests. This extension can be done by using the t-statistics derived from the correlations of the subject weights with the behavioral variables of interest, rather than those derived using the two-sample t-test. This means that the functional form of (6) and (7) would not change, instead t_n would be defined based on the sample correlation and ${\widehat{a}}_n^{[k]}$ would be selected based on the significance of their correlation to the test score of interest. In this study, the behavioral variables were derived from the measurement and treatment research to improve cognition in schizophrenia (MATRICS) consensus cognitive battery (MCCB), which is widely recognized as a valuable tool that provides a comprehensive evaluation of cognitive function of schizophrenia within clinical trials (Nuechterlein et al., 2008; August, Kiwanuka, McMahon, & Gold, 2012; Sui et al., 2015).

2.6. Simulation Setup

For real brain data, information about the true latent sources and subject weights is, generally, not known apriori. Therefore, in order to demonstrate the efficacy of GDMs, we generate 5 simulated sources, shown in Figure 1. We then linearly mix the simulated latent sources with a simulated mixing matrix whose columns have a step-type characteristic corrupted by additive i.i.d. Gaussian noise, simulating a latent difference between two groups of subjects, each with 150 subjects in order to make the scenario similar to the real fMRI data used in this study. Thus, the equation for the columns of the mixing matrix is given by

$${\mathbf{a}}_i={\mathbf{C}}_i+{\mathbf{n}}_i,1\le i\le 5,$$ (8)

where n_i is i.i.d. white Gaussian noise, generated separately for each column, and c_i is the step profile of size 1 for c₁, 0.5 for c₂, 0.26 for c₃, 0.23 for c₄, and 0 for c₅. The differing values of the step height correspond to different average relative differences between the groups and range from high to no difference. Note that since we know the value of the mean difference between the groups as well as the standard deviation of the samples (subjects), we are able to determine theoretical values for the t-statistics for each column of A. We then reduce the dimension of the dataset from 300 to 5 using PCA, perform ICA, and estimate a GDM.

Figure 1:

The true sources, true GDM, and GDMs calculated using the results of EBM and FastICA. Note that the t-statistic from the GDM derived using EBM is closer to the value of the t-statistic from the true GDM compared with the GDM derived using FastICA.

The purpose of this simulation is twofold. First, we seek to demonstrate the utility of GDMs to visually summarize the results of a decomposition. Second, to show how it is possible to use GDMs to relate the performance of different factorization methods. Towards this second aim, in this simulation, we compare the performance of two ICA methods, EBM and FastICA (Hyvärinen, 1999) using the hyperbolic tangent nonlinearity. We selected FastICA primarily because it is a popular method for the analysis of fMRI data, see e.g. (Kiviniemi, Kantola, Jauhiainen, Hyvärinen, & Tervonen, 2003; Ge et al., 2016; J. Xie, Douglas, Wu, Brody, & Anderson, 2017). However, FastICA uses a fixed nonlinearity and so it may not able to fit a very wide variety of distributions meaning that the performance of FastICA may suffer when the distribution of the latent sources is different from the one implied by its nonlinearity. In contrast, as already mentioned in Section 2.3, EBM is able to fit a wide variety of distributions. We present the results of this analysis in Figure 1. Note that in order to test the consistency of the results, those presented here are the result of 1,000 independent runs.

3. Results and Discussion

3.1. Simulation Results

From Figure 1, we can see that, as expected, the brightness of the region in the GDM corresponds to the significance of the latent subject weights. Regions with more significant weights are brighter than those regions with less significant weights. This highlights the ability of a GDM to summarize the results of a decomposition visually as well as how the significance of a component is incorporated into the result. We note that though the GDMs produced using the results of FastICA and EBM look similar, the t-statistic derived using the results from EBM matches the i-statistic from the true GDM more closely than the one derived using the results from FastICA. In order to probe this result, we computed the average Frobenius norm difference between the true GDM and the ones produced using the two ICA algorithms, divided by the square-root of the product of their individual Frobenius norms. We found that the average normalized difference was 0.737 for the GDM produced using FastICA, while it was 1.67 × 10⁻³ for the GDM produced using EBM. The close alignment between the GDM produced using EBM and the true GDM compared with the result using FastICA likely stems from the fact that the distribution of the latent sources in this simulation is bimodal. This means that, the distribution of the latent sources is not a close match to the one assumed by FastICA (Hyvärinen, 1999) but can be more accurately modeled using the flexible density model of EBM (X.-L. Li & Adali, 2009).

3.2. Detection of Group-level Differences

In order to investigate the ability of a GDM to summarize the discriminatory power of a decomposition, we estimate 25 components, corresponding to the maximum order that we find by applying PCA-CCA to the pairwise combinations of the three tasks. We display the t-statistics of the subject weights that were found to be statistically significant for each dataset and the corresponding GDMs for the separate ICAs and IVA in Figure 2.

Figure 2:

Comparison of the t-statistics from the GDMs compared with the t-statistics of the original biomarkers found using (a) separate ICAs and (b) IVA for each task. The t-statistics of the GDMs are shown in red, while the original biomarkers are shown in blue. Note that the t-statistics of the GDM are larger than the t-statistics from the original biomarkers, meaning the t-statistics from the GDMs can be used to summarize the separation power of a decomposition.

From Figure 2, we see that the t-statistics from the GDMs are an upper-bound for the t-statistics from their constituent biomarkers. This seems to imply that the t-statistics from the GDMs may summarize the discriminative power of a decomposition of a given dataset and can therefore be used to compare the performance of different analysis techniques on the same data. We also note from Figure 2(a) that the AOD and SM datasets seem to provide more discriminatory information than the SIRP dataset, since the significance of the t-statistic the GDM from the SIRP dataset is lower than those of the AOD and SM datasets. In Figure 2(b), we see that when the datasets are combined in a joint analysis using IVA, the significance of the SIRP dataset increases, while the significance levels of the AOD and SM datasets decrease. This seems to indicate that through a joint analysis of the multi-task data, datasets that are not as discriminative by themselves, like the SIRP, can be made more discriminative if exploiting related information across different tasks. However, we do note a drop in significance for the two datasets that are more significant individually, namely the AOD and the SM.

We next investigate whether the trends that we note using the t-statistics of the GDMs are accurate and that no spurious relations are introduced. This is done by comparing the p-value corresponding to the t-statistic from the GDM with the p-value obtained using Hotelling’s T²-test (Hotelling, 1947). Hotelling’s T²-test, the multivariate counterpart to the two-sample t-test assess the validity of the hypothesis that a two multivariate distributions have the same mean vectors against the alternative hypothesis that they do not. For this application, we evaluate whether the vector containing the subject loadings of all of the biomarkers from the controls is different from the vector containing the corresponding subject loadings from the patients. The results of this comparison are shown in Table 1.

Table 1:

Significance in terms of p-values for each combination of dataset and analysis method measured using a 2-sample t-test run on the subject loadings of the GDM as well as Hotelling’s T²-test. Note that the trends of the significance of each combination of dataset and analysis method is consistent across the two tests.

	AOD (ICA)	AOD (IVA)	SIRP (ICA)	SIRP (IVA)	SM (ICA)	SM (IVA)
GDM	1.65 × 10−13	3.74 × 10−12	4.54 × 10−7	4.20 × 10−8	7.52 × 10−16	4.53 × 10−9
T2-test	8.19 × 10−10	2.48 × 10−9	3.95 × 10−5	3.06 × 10−6	1.15 × 10−14	1.00 × 10−6

Based on the results shown in Table 1, we can see that the relative significance for the t-statistics of the GDMs and the Hotelling’s T²-test is the same. This seems to imply that the GDMs are effectively summarizing the discriminative power of a dataset within a decomposition and not introducing any artificial relations. The reason for this is due to the fact the Hotelling’s T²-test is the classical measure of the total difference between the subject covariations associated with the patients and those associated with the controls. However, it is important to note that the advantage of summarizing the discriminatory power through the use of GDMs rather than through the use of Hotelling’s T²-test is that the GDMs, i.e., the spatial maps, themselves can be related. This enables the identification of regions that differentiate patients and controls in different decompositions and a comparison of these regions. In order to explore the differences between individual ICAs and IVA in terms of the regions that are found to differentiate between patients and controls, we compare the GDMs derived from each of the methods for each of the three tasks after thresholding the maps at |Z| > 1.96. Using such a technique, we can visualize how much joint discriminative information each region possesses across tasks. We display the results in Figure 3.

Figure 3:

GDMs generated from the (a) AOD, (b) SIRP, and (c) SM tasks. For each plot, only those voxels that had an absolute Z-score above 1.96 are displayed. The red shows regions that are common to both the GDM extracted from the IVA results as well as the GDM extracted from the separate ICAs. The cyan shows the regions that are in the GDM extracted from IVA results but not in the GDM extracted from the separate ICAs. The yellow shows the regions that are in the GDM extracted from separate ICAs but not in the GDM extracted from the IVA results.

Figure 3 indicates that the discriminatory regions found using IVA but not when using separate ICAs—shown in cyan—correspond, mainly, to the sensorimotor region of the brain. Such a result seems reasonable, since this region is expected to be consistently activated across the three tasks and have been shown to differentiate patients with schizophrenia from healthy controls in these tasks, see e.g., (Liu et al., 2009; Sui et al., 2010; Deserno, Sterzer, Wiistenberg, Heinz, & Schlagenhauf, 2012). We also see regions, such as some of the auditory activation in the AOD and SM datasets as well as the visual activation in the SIRP, which are found using separate ICAs, but not when performing a joint analysis using IVA—shown in yellow. This highlights the ability of a joint analysis using IVA to emphasize similarities across datasets and also highlights the need to compare the results of a joint analysis to those of single dataset analyses, since some of the regions associated with a subset of the tasks may not be as strongly emphasized using IVA (Svensén, Kruggel, & Benali, 2002). Additionally, we can investigate the extent to which IVA emphasizes similar regions across all tasks when compared with individual ICAs by looking at the regions in the GDMs that are found to activate at |Z| > 1.96 for all tasks. Such an analysis is important, since a fundamental reason for analyzing multi-task fMRI data is to determine the common information provided by the tasks, which may be useful for identifying dysfunctional regions associated with brain disorders (Plis et al., 2014). The results of this comparison are presented in Figure 4.

Figure 4:

GDMs generated from (a) indivudal ICAs and (b) IVA. For each plot, only those voxels that have an absolute Z-score above 1.96 are displayed. The red shows regions that are common to all three tasks, the number of these voxels is 252 for separate ICAs and 286 for IVA. The cyan shows the regions that two of the three tasks, the number of these voxels is 1,420 for separate ICAs and 2,123 for IVA. The yellow shows the regions that are only activated in one of the three tasks, the number of these voxels is 9,469 for separate ICAs and 8,376 for IVA.

From Figure 4, we can see that the number of voxels that are common to two or more tasks is higher when analyzing the data using IVA than when separate ICAs are performed on each of the datasets. Such a result seems reasonable, since IVA is able to exploit the joint information across tasks, whereas individual ICAs cannot. This again highlights the improved power of IVA over separate ICAs when there are similarities across datasets.

3.3. Association with Psychological Measures

In order to investigate the ability of a GDM to summarize the association of components within a decomposition with a behavioral variable of interest, we display the t-statistics corresponding to the Pearson correlation between the subject weights and MCCB scores for the GDMs as well as its constituent components and display the results in Figure 5.

Figure 5:

Comparison of the t-statistics derived from the correlation between the subject weights and MCCB scores from the GDMs compared with those from the original components. The t-statistics of the GDMs are shown in red, while the original components are shown in blue. (a) and (b) show this relationship for the speed of processing score for ICA and IVA, respectively. (c) and (d) show this relationship for the verbal learning score for ICA and IVA, respectively. (e) and (f) show this relationship for the verbal working memory score for ICA and IVA, respectively. (g) and (h) show this relationship for the nonverbal working memory score for ICA and IVA, respectively.

From Figure 5, we see that, just as they may be seen to summarize the discriminatory power of a decomposition, GDMs can be used to summarize the associative power between components within a decomposition and behavioral variables of interest. Using this fact, we explore the use GDMs to assess the value of performing a joint analysis on multi-task fMRI using IVA rather than performing individual analyses using ICA. We present these results in Table 2.

Table 2:

Estimated t-statistics derived from the subject loadings of the GDMs as well as significance of the correlation between the subject loadings for the GDMs and MCCB scores. No correction for multiple comparisons was performed, however we highlight the entries that are not significant after a Bonferroni correction—p > 2.02 × 10⁻³ for a p-value of 0.05—in red. Note that the only significance values that do not survive the Bonferroni correction, and the lowest t-statistic value, are those corresponding to the ICA decomposition of the SIRP dataset.

	AOD (ICA)	AOD (IVA)	SIRP (ICA)	SIRP (IVA)	SM (ICA)	SM (IVA)
t-statistic	7.81	7.31	5.18	5.66	8.64	6.08
Speed of Processing	4.25 × 10−7	2.50 × 10−5	5.91 × 10−3	7.86 × 10−6	1.02 × 10−5	9.14 × 10−4
Verbal Learning	1.06 × 10−6	6.45 × 10−6	2.63 × 10−4	3.03 × 10−6	4.70 × 10−9	1.73 × 10−7
Verbal Working Memory	7.04 × 10−4	1.23 × 10−3	3.03 × 10−3	1.95 × 10−5	1.40 × 10−4	5.19 × 10−4
Nonverbal Working Memory	5.54 × 10−6	7.56 × 10−5	3.58 × 10−3	7.99 × 10−5	6.73 × 10−8	7.10 × 10−5

From Table 2, we see that the GDM found using ICA on the SIRP dataset does not correlate significantly, after a Bonferroni correction, with the majority of behavioral scores. This means that the results of a unimodal factorization of this dataset cannot be strongly associated with the measured clinical outcomes. However, the GDM created from the IVA decomposition of that dataset does correlate significantly, even after a conservative correction for multiple comparisons. This trend is similar to the one observed in Figure 2, i.e., that the factors extracted from datasets, which may not, inherently, be strongly associated with behavioral variables of interest, using a joint analysis can have a stronger association with the behavioral variables. However, this may come at the cost of reducing the associative power of datasets that are strongly associated with that variable by themselves, in this case the AOD and SIRP datasets. This motivates estimating the amount of joint information that exists between fMRI tasks prior to performing an analysis, see e.g., (Löfstedt & Trygg, 2011; Alter, Brown, & Botstein, 2003; van Deun, Smilde, Thorrez, Kiers, & van Mechelen, 2013; Levin-Schwartz et al., 2016), as well as comparing the results of a joint analysis with those from individual analyses (Svensén et al., 2002). It is important to note that since a subset of the possible contrasts that could be derived from each task were used in this work, the present study should be seen as an initial example to demonstrate the power of GDMs to relate different factorization methods on real fMRI data rather than a general statement about the tasks themselves.

4. Conclusions

The increasing use of fMRI data to study neural function and its disruption due to psychiatric conditions, such as schizophrenia, has lead to the rise of a wide variety of methods to analyze such data. This leads to the issue of how to objectively compare the performance of different methods, without the need for a tedious factor alignment step. In this paper, we have presented a novel technique, GDMs, to relate the results of different fMRI analysis methods based upon either their ability to detect differences between patients and controls or to correlate behavioral variables as well as to highlight which regions differ between the methods. We applied this technique to the decompositions of fMRI data from three tasks derived using individual analyses using ICA as well as a joint analysis using IVA. Our results show that GDMs are an effective technique to compare the results of different factorization methods on real fMRI data. Additionally, through our comparison of the results using individual ICAs and IVA, we find that a joint analysis using IVA is able to determine regions that are more similar across tasks and can be more discriminative between patients and controls than those found using individual analyses using ICA. It is important to note that though we construct GDMs based upon highlighting group differences and associations with behavioral variables, GDMs could also be constructed from the output of a regression analysis, which can account for the effects of confounding variables and be used when there are less clearly defined groups. We should also note that though developed in the context of comparing different decompositions of multi-task fMRI data, GDMs can be applied to data from different neuroimaging modalities, such as electroencephalogram or diffusion tensor imaging, or even in the context of multimodal fusion.

• *GDMs effectively compare the results of factorization methods on real fMRI data
• *IVA determines more related and discriminative networks than ICA
• *ICA is more effective at emphasizing task-specific networks than IVA