Sample-Poor Estimation of Order and Common Signal Subspace with Application to Fusion of Medical Imaging Data

Abstract

Due to their data-driven nature, multivariate methods such as canonical correlation analysis (CCA) have proven very useful for fusion of multimodal neurological data. However, being able to determine the degree of similarity between datasets and appropriate order selection are crucial to the success of such techniques. The standard methods for calculating the order of multimodal data focus only on sources with the greatest individual energy and ignore relations across datasets. Additionally, these techniques as well as the most widely-used methods for determining the degree of similarity between datasets assume sufficient sample support and are not effective in the sample-poor regime. In this paper, we propose to jointly estimate the degree of similarity between datasets and their order when few samples are present using principal component analysis and canonical correlation analysis (PCA-CCA). By considering these two problems simultaneously, we are able to minimize the assumptions placed on the data and achieve superior performance in the sample-poor regime compared to traditional techniques. We apply PCA-CCA to the pairwise combinations of functional magnetic resonance imaging (fMRI), structural magnetic resonance imaging (sMRI), and electroencephalogram (EEG) data drawn from patients with schizophrenia and healthy controls while performing an auditory oddball task. The PCA-CCA results indicate that the fMRI and sMRI datasets are the most similar, whereas the sMRI and EEG datasets share the least similarity. We also demonstrate that the degree of similarity obtained by PCA-CCA is highly predictive of the degree of significance found for components generated using CCA.

1. Introduction

The collection of data from multiple modalities has become common in neurological studies, since different modalities are expected to provide complementary views of complicated systems, such as the study of brain activity [18]. Thus, full utilization of all common information forms the fundamental goal of performing a joint analysis on multimodal data. Since little is known about the intermodality relationships, it is important to minimize the underlying assumptions placed on the data and let the data “speak for itself.” Because of this fact and their ability to treat separate modalities in a symmetric manner, multivariate data-driven methods have proven to be quite popular for the fusion of multimodal neurological data, see e.g., [2, 7, 18]. To this end, canonical correlation analysis (CCA), which maximizes the correlation of sources across datasets [16], has proven to be an effective multivariate and data-driven fusion method, see e.g., [2, 11, 14, 34]. However, if the covariances are unknown and must be estimated from the samples, then CCA requires sufficient sample support [27]. This is particularly an issue when performing multimodal data fusion, since the number of samples, i.e. subjects, is typically much less than the dimension of the neurological data that is used. Thus, special attention must be paid both before performing an analysis, i.e., when determining the similarity between datasets and their order—the dimension of the signal subspace—and while performing the analysis itself.

In this paper, we define the similarity between two datasets as the number of common components that both datasets share, i.e., those components that are correlated across datasets, raising the issue of how to determine this number when the sample size is limited. One of the most popular exploratory techniques to estimate the number of common components between two datasets is based on the canonical correlation coefficients (CCCs) calculated using CCA [16] and defining a threshold for the level of the correlation, see e.g., [6, 15, 19, 23]. Other methods for estimating the number of common and distinctive sources include: orthogonal n-way partial least squares (OnPLS) [24], generalized singular value decomposition [4], and distinctive and common components with simultaneous-component analysis (DISCO-SCA) [37]. These methods all assume sufficient sample support and thus perform poorly when the number of samples is not significantly greater than the number of observations. CCA, in particular, suffers greatly in the sample-poor regime, where all CCCs are significantly misestimated [32] and the highest CCCs, usually of greatest interest, may saturate at 1 [27], meaning that they provide no information about the true relationship between the datasets.

Since multimodal data is often quite noisy and of high dimensionality, dimension reduction using principal component analysis (PCA) is a crucial preprocessing step for avoiding the problem of over-fitting in subsequent analyses. However, the effectiveness of PCA is intimately tied to the problem of order selection. For a single dataset, the most popular order selection methods define the order based on information theoretic criteria (ITC) [38], i.e, by using a function of the estimated eigenvalues of the data and the number of model parameters. These methods include: Akaike’s information criterion (AIC) [3], minimum description length (MDL) [29] or Bayesian information criterion (BIC) [31], and extensions of those methods, see e.g., [22]. Though these methods have found widespread application in multimodal fusion, they are not directly applicable for two major reasons. The first is that almost all of these eigenvalue-based methods, with the notable exception of [26], assume sufficient sample support. If this is not true, such as for multimodal fusion using CCA, where the number of subjects is much less than the dimension of the data, the performance of these methods deteriorates rapidly because the eigenvalues cannot be estimated accurately [26]. Additionally, these methods only report on the sources that have greatest energy in each dataset individually. Since we are interested in common components that are linked across datasets, the use of methods that focus solely on a single dataset is not a desirable solution to the question of order selection for multimodal fusion. This provides the incentive to consider the problems of determining the degree of similarity and order jointly. Though not used in the context of medical imaging, there are methods that consider these two problems jointly, see e.g., [17, 40], however these techniques are heuristic and will fail in the sample-poor regime [30].

In this paper, we discuss an effective method, PCA and CCA (PCA-CCA) along with the order selection rule from [32], for jointly determining the number of common sources for datasets and their order, in the sample-poor regime and demonstrate its importance as a preliminary step for multimodal fusion. To the best of our knowledge, this method is the only one that addresses the issues of common source detection and order selection for the sample-poor case encountered when using CCA for multimodal fusion. The versatility and high performance of this technique are first demonstrated through simulations. We then apply this new method to the pairwise combinations of functional magnetic resonance imaging (fMRI), structural magnetic resonance imaging (sMRI), and electroencephalogram (EEG) data drawn from 14 patients with schizophrenia and 22 healthy controls performing an auditory oddball (AOD) task and relate these results to the pairwise fusion results obtained using CCA. Through this application, we demonstrate a strong correlation between the number of common components estimated using PCA-CCA, i.e., the similarity between datasets, and number of statistically significant components estimated during the fusion analysis. This technique of investigating the pairwise combinations of datasets drawn from the same subjects provides unique insight into the degree of complementarity between related data of different modalities.

2. Materials and Methods

2.1. Theory

2.1.1. Traditional and Sample-poor Hypothesis Test

Let us assume that we have M independent and identically distributed (i.i.d.) paired samples of x^[1] ∈ ℝⁿ and x^[2] ∈ ℝ^m from the two-channel measurement model [32],

$${\mathbf{x}}^{[1]}={\mathbf{A}}^{[1]}{\mathbf{s}}^{[1]}+{\mathbf{n}}^{[1]}$$

$${\mathbf{x}}^{[2]}={\mathbf{A}}^{[2]}{\mathbf{s}}^{[2]}+{\mathbf{n}}^{[2]},$$ (1)

where s^[k] ∈ ℝ^d+f, k = 1, 2, are zero-mean jointly Gaussian random vectors with cross-covariance matrix, R_s1s2 = E{s^[1](s^[2])^T}, given by

$${\mathbf{R}}_{{\mathbf{s}}_1{\mathbf{s}}_2}=\left[\begin{array}{cc}\hfill \mathbf{\text{diag}}({\rho }_1{\sigma }_1^{[1]}{\sigma }_1^{[2]},\dots ,{\rho }_d{\sigma }_d^{[1]}{\sigma }_d^{[2]})\hfill & \hfill {\mathbf{0}}_{d\times f}\hfill \\ \hfill {\mathbf{0}}_{f\times d}\hfill & \hfill {\mathbf{0}}_{f\times f}\hfill \end{array}\right],$$

where ${\sigma }_i^{[k]}$ is the unknown standard deviation of signal component $s_i^{[k]}$ and ρ_i is the correlation coefficient between $s_i^{[1]}$ and $s_i^{[2]}$. Thus, both s^[1] and s^[2] have d correlated signals and f uncorrelated signals. Without loss of generality, we assume the auto-covariance matrices of s^[1] and s^[2] to be diagonal. The noise terms n^[1] and n^[2] are independent of each other, independent of the signals, and zero-mean Gaussian with unknown covariance matrices. Additionally, without loss of generality we assume that A^[1] and A^[2] are of full column rank and, like the dimensions d and f, are fixed but unknown.

We collect the M sample pairs into data matrices ${\mathbf{X}}^{[1]}=[{\mathbf{x}}_1^{[1]},\dots ,{\mathbf{x}}_M^{[1]}]$ and ${\mathbf{X}}^{[2]}=[{\mathbf{x}}_1^{[2]},\dots ,{\mathbf{x}}_M^{[2]}]$. When performing CCA when M < m+n, at least m+n−M of the sample canonical correlation coefficients, k̂_i, i = 1, …, q, q = min(m, n), will be identically 1 regardless of the values of ρ_i and thus do not provide any information about the relationship between s^[1] and s^[2] [27]. Moreover, even in the case where M is greater, but not significantly greater, than m+n, the sample canonical correlations may significantly overestimate the population canonical correlations [32]. This result provides the incentive to estimate a suitable rank, r, in order to reduce the dimensions of X^[1] and X^[2], thus allowing accurate estimation of the number of correlated signals.

A classical way of estimating d is by assuming that the sources are drawn from a multivariate Gaussian distribution and applying a sequence of binary hypothesis tests [5, 21]. The test begins with s = 0 and compares the two hypotheses H₀ : d = s and H₁ : d > s. If the null hypothesis is rejected, then s is increased by one and the test is repeated, until either the null hypothesis is not rejected or s = q. This test is based on the Bartlett-Lawley statistic [5, 21], which is given by

$$C(s)=(M-s-\frac{m+n+1}{2}+\sum _{i=1}^s{\hat{k}}_i^{-2})\text{ ln}\prod _{i=s+1}^q(1-{\hat{k}}_i^2),$$ (2)

and is asymptotically distributed under H₀ as χ² with (m−s)(n−s) degrees of freedom. The fact that the test statistic is distributed according to the χ² distribution enables the determination of a threshold, T(s), to meet a given probability of false alarm, P_FA, for the test. A major constraint of the traditional framework is the assumption of sufficient samples, i.e., that the k̂_i’s are accurate estimates of the true k_i’s, making it not applicable for the sample-poor regime.

As proposed in [32], the sample-poor version of the classical hypothesis test selects

$$\hat{d}=\underset{r\in ℛ}{\text{max}}\underset{s\in 𝒮}{\text{min}}\{s:C(s,r)<T(s,r)\},$$ (3)

where ℛ = {1, …, r_max}, 𝒮 = {0, …, r − 1}, and r_max = min(M/2, q) and the test statistic is

$$C(s,r)=(M-s-r-\frac{1}{2}+\sum _{i=1}^s{\hat{k}}_i^{-2}(r))\text{ ln}\prod _{i=s+1}^r(1-{\hat{k}}_i^2(r)).$$

Note in the above expression that the estimated canonical correlations are functions of the reduced rank, r; if r is too big, then k̂_i will be over-estimated, while if it is too small, then k̂_i will be under-estimated [32].

By limiting the dimension through r_max, we can ensure that none of the estimated canonical correlations will saturate at 1. Note that to guarantee that the rank-reduced statistic under H₀ is still asymptotically distributed as χ² with (r−s)² degrees of freedom, we must pick r_max “sufficiently” smaller than M/2. This is critical in order to compute the test threshold, T(s, r), for a given probability of false alarm. Based on our numerical simulations, a good value for r_max seems to be ⌊M/3⌋, where ⌊·⌋ is the floor function [32]. However, note that there is a certain degree of robustness of the results to the selection of the value of r_max observed in real data. The value for r that leads to d̂ in (3) is chosen as the PCA rank. Greater detail about the implementation of PCA-CCA to determine the order and similarity between multimodal datasets is found in Appendix B.

2.1.2. CCA for Multimodal Fusion

Due to the inherent dissimilarities between different modalities, direct fusion of multimodal data is difficult. Therefore, rather than jointly analyzing the multimodal datasets directly, it is often beneficial to reduce each modality to a feature, a lower-dimensional multivariate representation of the data, for each subject within the study. See Section 3.2 for greater detail about the features used in this study. This reduction enables the exploration of associations across feature sets through the analysis of variations across individuals [7, 9, 10]. Such an investigation of variations between subjects through fused features provides a natural way to explore associations across modalities, since it constructs a common dimension for datasets of otherwise different dimensionality [13, 35]. In addition, due to the dissimilarities between modalities, the subject covariations provide the dimension of coherence in multimodal fusion using CCA, since similarity between covariations across modalities is expected. Figure 1 depicts the model for multimodal fusion using CCA following spatial dimension reduction through PCA; the implementation of CCA for fusion is presented next.

Figure 1.

Generative model for multimodal fusion using CCA. Note that this model is obtained after spatial dimension reduction through PCA, i.e., X̃^[1] and X̃^[2] are both r × M matrices.

As explained in detail in Appendix B, we apply spatial dimension reduction through PCA, where the order, r, is selected using (3), to obtain dimension reduced feature datasets X̃^[1] and X̃^[2]. These datasets can be decomposed into the combination of subject covariations, S̃^[1] and S̃^[2], and associated components, Ã^[1] and Ã^[2], in the following manner,

$${\tilde{\mathbf{X}}}^{[k]}={\mathbf{Ã}}^{[k]}{\tilde{\mathbf{S}}}^{[k]},k=1,2.$$ (4)

By using this decomposition, the relationship across modalities can be evaluated based on the correlation of the subject covariations between the two modalities. We assume that the subject covariations are uncorrelated within each modality and are correlated with at most one subject covariation from the other modality. This enables the exploration of the associations across the modalities. Additionally, note the lack of a noise term in (4), which emphasizes that this analysis occurs following dimension reduction through PCA.

By exploiting the simple form of (4), we can use CCA to identify relations between the subject correlations [14]. CCA seeks to transform these two sets of data in order to maximize their correlation. This can be expressed in the following maximization problem, where in the first stage we obtain the first rows of P^[1] and P^[2], ${\mathbf{P}}_1^{[1]}$ and ${\mathbf{P}}_1^{[2]}$, respectively,

$$\underset{{\mathbf{p}}_1^{[1]},{\mathbf{p}}_1^{[2]}}{\text{max}}\text{ corr }({\mathbf{p}}_1^{[1]}{\tilde{\mathbf{X}}}^{[1]},{\mathbf{p}}_1^{[2]}{\tilde{\mathbf{X}}}^{[2]})=\frac{{\mathbf{p}}_1^{[1]}{\hat{\mathbf{R}}}_{{\mathbf{X}}_1{\mathbf{X}}_2}{({\mathbf{p}}_1^{[2]})}^{\mathrm{T}}}{\sqrt{{\mathbf{p}}_1^{[1]}{\hat{\mathbf{R}}}_{{\mathbf{X}}_1{\mathbf{X}}_1}{({\mathbf{p}}_1^{[1]})}^{\mathrm{T}}{\mathbf{p}}_1^{[2]}{\hat{\mathbf{R}}}_{{\mathbf{X}}_2{\mathbf{X}}_2}{({\mathbf{p}}_1^{[2]})}^{\mathrm{T}}}},$$ (5)

where R̂_XiXj = X̃^[i] (X̃^[j])^T/M denotes a sample (cross-) covariance matrix and the operation corr(·, ·) refers to the sample correlation between the columns of the given matrices. Subsequent rows of P^[1] and P^[2] are estimated analogous to (5) with the additional constraint that

$${\mathbf{p}}_i^{[k]}{\tilde{\mathbf{X}}}^{[k]}{\left({\mathbf{p}}_j^{[k]}{\tilde{\mathbf{X}}}^{[k]}\right)}^{\mathrm{T}}=0,\forall j\ne i,k=1,2.$$

The estimated subject covariations are given by,

$${\mathbf{Ŝ}}^{[k]}={\mathbf{P}}^{[k]}{\tilde{\mathbf{X}}}^{[k]},k=1,2.$$ (6)

Greater detail about the implementation of CCA to perform source separation for multimodal data is found in Appendix B.

The rows of Ŝ^[k] provide the loadings of each component across subjects. Thus, the p-th row of Ŝ^[k], ${\mathbf{ŝ}}_p^{[k]}$, represents the relative weights of the p-th source estimate, ${\mathbf{â}}_p^{[k]}$, for all subjects [2]. Since each dataset has been reduced to a feature for each subject, to look for differences in the expression of components between two groups, a two-sample t-test can be performed on the subject covariations, where one group is represented by the subject covariations from the patients with schizophrenia and the second by the subject covariations from the healthy controls [7, 9, 10]. This enables the detection of components that can differentiate patients from controls, providing insight into how brain disorders affect both neurological structure and function.

2.2. Simulation Setup

For real brain data, the ground truth, i.e., the true latent sources and subject covariations, is not available. Thus, in order to test the performance of PCA-CCA, we generate simulated data according to (1) and examine the probability of correctly estimating the true number of correlated components as the dimension of the datasets, m and n, increases from 10 to 10,000. We compare the performance of PCA-CCA to that of two classical ITC-based methods, AIC and MDL. These methods use the AIC and MDL framework in [38] in the PCA step to select r, and then use the AIC and MDL framework described in [33] to estimate the common order. For these simulations, we have M = 50 samples and d = 2 pairs of correlated components each with variance of 2 and correlation coefficients, ρ₁ = 0.8 and ρ₂ = 0.7. The number of uncorrelated components in s^[1] and s^[2], f, is 3. The variances of the uncorrelated signals and the white noise are both 1. These parameters were selected to align with properties of real neurological data. The entries of the mixing matrices A^[k] are independently drawn from uniformly distributed random numbers between 0 and 1. Additionally, we consider two values, 0.001 and 0.005, for probability of false alarm, P_FA. We have also used a minimum description length (MDL) based method, described in Appendix A, which does not require the selection of a probability of false alarm. The plots presented here are the average result of 1000 independent Monte-Carlo simulations, where, for each simulation, every method was used on the same data.

We repeated this simulation three times, changing only the distribution from which the sources, s^[k], are drawn. In the first case, we consider sources drawn from a zero mean Gaussian distribution, which aligns with the form typically used in all derivations [5, 21, 22, 32, 38]. The subsequent case considers sources that are uniformly distributed with zero mean and the same variance as the previous case. Such sub-Gaussian or platykurtic sources are a good approximation of the subject covariations, since if there is a difference between the two groups then the distribution describing the subject covariations will be bi-modal and hence sub-Gaussian. Thus, the performance of PCA-CCA in this case is of great interest. The final case considered is where the sources are Laplacian distributed with zero mean and the same variance as the previous distributions. Due to the ability of the Laplacian distribution to provide good approximations of empirical fMRI source distributions [8, 25], studying the performance of PCA-CCA in this regime is of interest for the fusion of two fMRI datasets.

3. Results

3.1. Simulation Results

Figure 2 shows the high level of performance obtained using PCA-CCA together with the order selection rule in [32] for the three source distributions considered. The results indicate that PCA-CCA is robust to model mismatch in the latent source distribution. In all cases, the performance of PCA-CCA improves as the dimensionality of the datasets increases. The reason for this is that as the dimension of the datasets increases, the signal eigenvalues of ${\mathbf{R}}_{{\mathbf{x}}_i{\mathbf{x}}_i}={\mathbf{A}}^{[i]}{\mathbf{R}}_{{\mathbf{s}}_i{\mathbf{s}}_i}{\left({\mathbf{A}}^{[i]}\right)}^T+{\sigma }_n^2{\mathbf{I}}_N,i=1,2$, where R_sisi is the population covariance matrix of the correlated and uncorrelated signals, N = m = n is the dimension of the system, ${\sigma }_n^2$ is the variance of the noise, and I_N is a N-dimensional identity matrix, increase, whereas the noise eigenvalues remain unchanged. Thus, the per-component signal-to-noise ratio, i.e, the ratio of signal eigenvalue to the noise eigenvalue, increases as the dimension increases, thus improving the probability of detection. We see that the performance of the classical methods, AIC and MDL, deteriorates rapidly as the datasets become increasingly sample-poor because, in this regime, the eigenvalues cannot be estimated accurately. Finally, note that the MDL-based method consistently has the highest performance, indicating the benefits of its use over that of an arbitrary selection of a probability of false alarm.

Figure 2.

Probability of correctly detecting the number of common components versus the dimension of the datasets, m and n, when the latent sources are (a) Gaussian, (b) uniformly, and (c) Laplacian distributed. Note that for these simulations, m = n.

3.2. Auditory Task and Data Preprocessing

The fMRI (m = 60, 261), EEG (m = 451), and sMRI (m = 306, 626) data used in this study were obtained from 22 healthy controls and 14 patients with schizophrenia, meaning that M is 36. The first two modalities, fMRI and EEG, were obtained individually while the subjects listened to three different types of auditory stimuli: standard (500 Hz occurring with probability 0.80), novel (nonrepeating random digital sounds with probability 0.10), and target (1 kHz with probability 0.10, to which a button press was required). These two modalities provide complementary views of functional neural activity, with fMRI observing the changes in blood-oxygenation levels and EEG measuring electrical activity of neurons through the scalp. The third modality, sMRI, provides structural information about the brain by describing three main tissue types: gray matter, white matter, and cerebrospinal fluid. Including this final modality permits the exploration of the degree to which neural structure underlies brain function. For more information on the task and the preprocessing of the data, see [14].

FMRI

fMRI preprocessing steps include: slice timing correction to account for the sequential acquisition of the slices, registration to correct subject motion in the scanner, spatial filtering to reduce noise, and spatial normalization to allow comparisons of fMRI data across different individuals. The features used for fMRI data are task-related spatial activity maps calculated by the general linear regression approach [14].

EEG

EEG preprocessing steps include: amplifying the detected signals to correct the minor observed changes and then digitizing the resultant signals. The extracted features, event-related-potentials (ERPs), are formed by averaging epochs (recording periods) of a single channel of the EEG data time matched to the sensory stimulus of the AOD task [14]. For this study, we used the channel corresponding to the midline central (Cz) position because it was the best single channel for the detection of both anterior and posterior sources for the task.

SMRI

sMRI preprocessing steps include: bias field corrections, which correct for intensity changes caused by radio frequency or main magnetic field inhomogeneities, spatial filtering to reduce noise, and spatial normalization to allow comparisons of brains across different individuals. We use non-modulated probabilistically segmented gray matter images as the sMRI features for this fusion study [14].

3.3. AOD Fusion Results

The fusion results using CCA are depicted in Figure 3. These components are those that were found to be both biologically meaningful and statistically significant, i.e., they had activations corresponding to biologically meaningful areas and p-values in terms of group differences smaller than 0.05. Biological meaning is assessed based on prior information about the spatial and temporal properties of the components. For fMRI and sMRI components, smooth and focal regions are expected, while ERP components should display a smooth response with peaks corresponding to those in the group response. When using the fMRI and sMRI data together, one component is found. This component corresponds to frontal activation in the fMRI section and motor activation in the sMRI. When the fMRI and ERP data are used in the analysis, a single one component is found that is both statistically significant and physically meaningful. This component corresponds to sensory-motor and auditory activation in the fMRI part and corresponding N2\P3 complex in the ERP. These regions correspond to areas that have been previously noted as affected in schizophrenia. We see a similar N2\P3 complex in the ERP component generated using the sMRI and ERP data. Note that these discriminative components can be leveraged to classify a new subject as either a patient with schizophrenia or a healthy control [28]. By regressing the discriminative components onto the new subject’s feature data, we obtain a set of subject covarations that can be classified using either k-means or a support vector machine.

Figure 3.

Meaningful and statistically significant components generated using CCA with pairwise combinations of the three modalities. The uncorrected significance (p-values) for the components are (a) 0.027 for the fMRI-sMRI component, (b) 0.003 for the fMRI-ERP component, and (c) 0.018 for the sMRI-ERP component. All spatial maps are Z-maps threshold at Z=3.5.

Using PCA-CCA and the MDL criterion described in Appendix A, we can estimate the number of common components for each of the pairwise combinations of the fMRI, sMRI, and ERP data. This estimation of the number of common components allows us to measure the degree of similarity between the datasets in a meaningful way, by defining greater similarity in terms of a greater number of common components. The results of this analysis are displayed in Table 1, where we also show the number of statistically significant components found for each combination of modalities. Components are declared significant if they pass a two-sample t-test run on the rows of the estimated subject covariations, Ŝ^[k], k = 1, 2, calculated using CCA. We see that the fMRI and sMRI data have the greatest degree of similarity since they share the greatest number of common components estimated using PCA-CCA. This result makes sense since both modalities are MRI data of spatial nature and thus they would be expected to have a greater degree of similarity. The result also corresponds to the number of significant components found after performing fusion using CCA. Since the fMRI-sMRI combination has the greatest number of significant components, there is a greater number of chances of finding common significant components. This explains why we observe that greatest number of meaningful components using the combination of fMRI and sMRI. Using PCA-CCA, we found that the combination of fMRI and ERP data produced the second greatest number of common components. This result again makes sense since both the fMRI and ERP data describe functional changes in the brain, so they exhibit a fairly large degree of similarity. The result correlates to the results obtained in terms of number of significant components found after performing fusion using CCA on the modality pair of fMRI-ERP. The least similar modalities are the sMRI and ERP. This result is confirmed by the results obtained using PCA-CCA, since the sMRI-ERP combination produced the fewest number of common components. This result makes sense since the sMRI data is structural whereas the ERP data is functional. Thus, using PCA-CCA, we can measure the degree of similarity between datasets and the results can be predictive of the fusion results using CCA.

Table 1.

Number of significant components at two p-values, PCA-CCA estimated common components, d, and estimated order, r, for the three pairwise combinations of modalities. Statistical significance is assessed by a two-sample t-test run on the rows of the estimated subject covariations, Ŝ^[k], k = 1, 2, calculated using CCA.

	fMRI-sMRI	fMRI-ERP	sMRI-ERP
Number of significant components, p = 0.05	2	1	1
Number of significant components, p = 0.1	5	4	2
Number of common components using PCA-CCA	4	3	2
Estimated order using PCA-CCA	17	17	12

4. Discussion

The successful application of PCA-CCA to determine the degree of similarity between pairwise combinations of multimodal datasets naturally leads to the issue of extensions beyond two datasets. For fusion using CCA, the straightforward extension is the use of multiset CCA (MCCA) [20], with one of the five cost functions selected. This technique has been successfully applied to the fusion of three mulimodal datasets, see e.g., [13, 36]. The results of the application of MCCA, using only the generalized variance (GENVAR) cost for brevity, to the fMRI, sMRI, and ERP data in this study is displayed in Figure 4. As expected, we can see a great degree of correlation between the CCA results and those achieved using MCCA. The motor, auditory, and visual activation in the second component bear striking resemblance to those of the fMRI-ERP component. In addition, the N2\P3 activation observed in the ERP components that are generated using CCA is similar to that of the ERP component generated using MCCA.

Figure 4.

Meaningful and statistically significant components generated using MCCA with GENVAR. The uncorrected significance (p-values) for the components are (a) 0.014 for the first component and (b) 0.018 for the second component. All spatial maps are Z-maps thesholded at Z=3.5.

However, a similar extension of PCA-CCA to more than two datasets remains an open problem. This is due to the fact that in the case of two datasets the number of non-zero canonical correlations defines the degree of correlation between the two sets. However, for more than two datasets, the canonical correlations cannot be used to define the number of correlated components.

To understand the reason for this, consider the case where we have three datasets with four independent signals in each. Only the first two sources in dataset 1 are correlated with the first two sources in dataset 2. Similarly, only the first and third sources in dataset 2 are correlated with the first and third sources in dataset 3. Finally, only the first and fourth sources in dataset 1 are correlated with the first and fourth sources in dataset 3. In this example, there are two correlated signals in each pair of datasets but there is just one signal, i.e., the first signal in each dataset that is correlated across all the datasets. In order to identify this common signal, we expect a non-zero correlation between all three pairwise combinations of the first signal components of the three datasets only. However, MCCA would generate non-zero correlations for all pairwise combinations of the four signals from all datasets, regardless of the cost used, indicating that all of the components are common to all datasets, which is not the case.

Though a true multiset extension of PCA-CCA is preferable, an interim solution is the following heuristic method, which can estimate the number of correlated components among all three datasets. Note that this method can be extended to an arbitrary number of datasets, though it is only presented for three datasets here.

$${\mathbf{x}}^{[1]}={\mathbf{A}}^{[1]}{\mathbf{s}}^{[1]}+{\mathbf{n}}^{[1]},$$

$${\mathbf{x}}^{[2]}={\mathbf{A}}^{[2]}{\mathbf{s}}^{[2]}+{\mathbf{n}}^{[2]},$$

$${\mathbf{x}}^{[3]}={\mathbf{A}}^{[3]}{\mathbf{s}}^{[3]}+{\mathbf{n}}^{[3]}.$$

For datasets x^[1] and x^[2], by applying the PCA-CCA approach, we can estimate the number of correlated signals between the first and second datasets, d̂^[1,2], and the optimal rank, r^[1,2], that PCA should keep for x^[1] and x^[2]. With these estimates, the canonical sources, ${\mathit{\omega }}_1^{[1,2]}$ and ${\mathit{\omega }}_2^{[1,2]}$, associated with the d̂^[1,2] largest canonical correlations may be obtained from the r^[1,2]-approximation of x^[1] and x^[2], respectively. The above procedure for x^[1] and x^[2] can be analogously applied for the pairwise combinations x^[2] and x^[3], as well as x^[1] and x^[3]. Due to the fact that the canonical source, e.g., some row in ${\mathit{\omega }}_1^{[1,2]}$, is equal to the corresponding original source, e.g., some row in s^[1], up to a scaling factor, we can find three canonical sources, ${\mathit{\omega }}_1^{[1,2]},{\mathit{\omega }}_2^{[2,3]},{\mathit{\omega }}_3^{[1,3]}$ whose cross-correlation matrices, $E\left\{{\mathit{\omega }}_1^{[1,2]}{\left({\mathit{\omega }}_2^{[2,3]}\right)}^T\right\},E\left\{{\mathit{\omega }}_1^{[1,2]}{\left({\mathit{\omega }}_3^{[1,3]}\right)}^T\right\},E\left\{{\mathit{\omega }}_2^{[2,3]}{\left({\mathit{\omega }}_3^{[1,3]}\right)}^T\right\}$, are all of rank d, where d is the number of common signals among three datasets. Then, the ITC-based detection scheme described in [39] can be used. Applying this method to the fMRI, sMRI, and ERP data, one common signal can be identified. The result using this method corresponds to what is observed with MCCA, where there is only one component that is common to all three modalities.

The collection of multimodal data has become increasingly common in medical studies, raising the issue of how best to combine data that are, fundamentally, different. In this paper, we proposed the use of PCA-CCA and the order selection rule of [32] to determine the degree of similarity between multimodal neurological data. The results using PCA-CCA indicate that this method can not only measure the degree of similarity between dissimilar datasets but also provides an accurate prediction of the significance of components generated by CCA. This predictive capability of PCA-CCA enables the determination of meaningful combinations of modalities prior to analysis. There are several possible directions available for future consideration.

• A true multiset extension of PCA-CCA. Though the proposed heuristic method may be used for three datasets, it is not effective for detecting the number of common components in a large number of datasets. A possible solution may lie in tensor decompositions, which have recently been applied to the processing of medical data, see e.g., [12].

• An extension of PCA-CCA enabling the detection of components based on higher-order dependence. The limitation of PCA-CCA to detect similar components based solely on correlation has obvious limitations when information about higher-order dependence is available. Exploring the possibility of kernelizing PCA-CCA or using independent vector analysis (IVA) could prove fruitful in exploiting this information [1].

• Application of PCA-CCA to multitask data drawn from the same modality. In the same way that PCA-CCA can be used to determine the similarity between modalities, PCA-CCA can be applied to multi-set data, i.e., data of the same type such as multi-task data, in order to determine the degree of similarity between different datasets.

Highlights.

• ►Signal subspace or order estimation is crucial for studies using neurological data.
• ►We use PCA-CCA to find the order and quantify similarity for multimodal datasets.
• ►PCA-CCA has high performance in the sample-poor regime faced in multimodal fusion.
• ►Similarity between multimodal data is predictive of data fusion results with CCA.

Appendix A

A fundamental issue with hypothesis tests is the need to set a probability of false alarm, which determines the test threshold. For different problems, the ideal value for this probability of false alarm and thus the threshold changes, and cannot, in general, be determined for real data. This provides the incentive to use a threshold that is based on ITC. The general MDL expression for detecting the number of correlated components in a two channel model is given by [33]

$$I_{\mathrm{M}\mathrm{D}\mathrm{L}}(s)=M\prod _{i=1}^s(1-{\hat{k}}_i^2)+\text{log}(M)s(m+n-s),$$ (7)

where the number of correlated components is determined by the value of s which minimizes (7).

We can modify (7) in a similar manner as (2) to account for the PCA step, resulting in the final MDL criterion [32]

$$I_{\mathrm{M}\mathrm{D}\mathrm{L}}(s,r)=M\prod _{i=1}^s(1-{\hat{k}}_i^2(r))+\text{log}(M)s(2r-s).$$ (8)

The decision rule (3) is thus replaced with

$$\hat{d}=\underset{r\in ℛ}{\text{max}}\underset{s\in 𝒮}{\text{argmin}}{I}_{\mathrm{M}\mathrm{D}\mathrm{L}}(r,s),$$ (9)

and, because of its desirable performance, as demonstrated in the simulation results, is used to compute the number of common components for the multimodal data considered in this study.

Appendix B

In this section, we describe the pipeline developed in this paper to fuse multimodal data. Consider two sets of multimodal feature data X^[1] and X^[2] gathered from the same M subjects, whose dimensions are m × M and n × M. In this study, these datasets could be any pairwise combination of the fMRI, sMRI, and EEG data.

The first stage in the pipeline is the calculation of the joint order and degree of similarity between X^[1] and X^[2]. This is done by performing PCA-CCA on the two datasets using the decision rule specified by (9) and a value of r_max specified in Section 2.1.1. This preliminary analysis produces the joint order of X^[1] and X^[2], r, as well as the degree of similarity between X^[1] and X^[2], d. This value of r is used as a preprocessing step for the second stage, where multimodal data fusion is performed using CCA.

Prior to performing data fusion using CCA, dimension reduction is first performed using PCA, with the order specified by r, on each dataset separately, in order to reduce X^[1] and X^[2] to X̃^[1] and X̃^[2], respectively, such that

$${\tilde{\mathbf{X}}}^{[k]}={\mathbf{F}}^{[k]}{\mathbf{X}}^{[k]},k=1,2,$$ (10)

where F^[k] are data reduction matrices and are equal to the eigenvectors corresponding to the highest r eigenvalues of the sample covariance matrix of X^[k]. Note that both X̃^[1] and X̃^[2] have the same dimensionality, i.e, r × M. Next, CCA is performed on X̃^[1] and X̃^[2], producing estimated subject covariations, Ŝ^[1] and Ŝ^[2], according to (6). Then using (4), the estimated dimension-reduced sources, Ã̂^[1] and Ã̂^[2], are found. Finally, the full estimated sources, Â^[1] and Â^[2], are found through back-reconstruction of Ã̂^[1] and Ã̂^[2] using

$${\mathbf{Â}}^{[k]}={\left({\mathbf{F}}^{[k]}\right)}^{\mathrm{T}}{\widehat{\mathbf{Ã}}}^{[k]},k=1,2.$$ (11)

Note that when fusing three datasets, the PCA-CCA stage is skipped and the joint order is specified by the mean order found in each of the pairwise combinations of modalities. Additionally, when fusing three datasets, the CCA step is replaced with MCCA using the GENVAR cost [20].