Discriminating Schizophrenia and Bipolar Disorder by Fusing FMRI and DTI in A Multimodal CCA+ Joint ICA Model

Abstract

Diverse structural and functional brain alterations have been identified in both schizophrenia and bipolar disorder, but with variable replicability, significant overlap and often in limited number of subjects. In this paper, we aimed to clarify differences between bipolar disorder and schizophrenia by combining fMRI (collected during an auditory oddball task) and diffusion tensor imaging (DTI) data. We proposed a fusion method, “multimodal CCA+ joint ICA’, which increases flexibility in statistical assumptions beyond existing approaches and can achieve higher estimation accuracy. The data collected from 164 participants (62 healthy controls, 54 schizophrenia and 48 bipolar) were extracted into “features” (contrast maps for fMRI and fractional anisotropy (FA) for DTI) and analyzed in multiple facets to investigate the group differences for each pair-wised groups and each modality. Specifically, both patient groups shared significant dysfunction in dorsolateral prefrontal cortex and thalamus, as well as reduced white matter (WM) integrity in anterior thalamic radiation and uncinate fasciculus. Schizophrenia and bipolar subjects were separated by functional differences in medial frontal and visual cortex, as well as WM tracts associated with occipital and frontal lobes. Both patients and controls showed similar spatial distributions in motor and parietal regions, but exhibited significant variations in temporal lobe. Furthermore, there were different group trends for age effects on loading parameters in motor cortex and multiple WM regions, suggesting brain dysfunction and WM disruptions occurred in identified regions for both disorders. Most importantly, we can visualize an underlying function-structure network by evaluating the joint components with strong links between DTI and fMRI. Our findings suggest that although the two patient groups showed several distinct brain patterns from each other and healthy controls, they also shared common abnormalities in prefrontal thalamic WM integrity and in frontal brain mechanisms.

INTRODUCTION

Schizophrenia (SZ) is a psychotic disorder characterized by altered perception, thought processes, and behaviors ; and bipolar (BP) illness is a mood disorder involving prolonged states of depression and mania (Goodwin and Jamison 2007). These two brain diseases have overlapping symptoms (for example reports suggest that 60% of bipolar 1 patients have psychotic features (Goes, et al. 2007; Guze, et al. 1975)), persistent neurocognitive deficits (Glahn, et al. 2004), similar risk genes (Bahn 2002), and co-occurrence within relatives (Lichtenstein, et al. 2009) ; however, the common and distinct neural mechanisms underlying these disorders remain unclear.

Many brain imaging techniques including functional MRI (fMRI), structural MRI (sMRI), EEG/MEG and diffusion tensor imaging (DTI) provide information on different aspects of the brain (e.g. blood flow, integrity of gray matter, integrity of white matter tracts). FMRI can assess functional differences related to rest or neurocognition and DTI can additionally provide information on structural connectivity among brain networks. Numerous studies have been published directly comparing schizophrenia to bipolar disorder in one imaging modality, such as fMRI (Calhoun, et al. 2007; Hamilton, et al. 2009; Pearlson 1997), sMRI (Altshuler, et al. 2000; McIntosh, et al. 2005; Strasser, et al. 2005), and genetics (Bousman, et al. 2009; McIntosh, et al. 2009). Diverse brain alterations have been identified in these two patient groups, but with limited replicability (Fornito, et al. 2009) and generally small sample sizes.

Although functional and structural brain studies have identified quantitative alterations in schizophrenia and bipolar disorder, such findings have not yet provided a diagnostic measure that is both sensitive and specific. It is likely in part that the lack of consistent findings is because most models favor only one data type or do not combine data from different imaging modalities effectively, thus missing potentially important differences which are only partially detected by each modality (Calhoun, et al. 2006a). Combining modalities may thus uncover previously hidden relationships that can unify disparate findings in brain imaging.

Existing multimodal studies in these two patient groups have combined EEG with MRI data (McCarley, et al. 2008; McIntosh, et al. 2006); however, to our knowledge no report has combined fMRI and DTI data to investigate both commonalities and differences between SZ and BP; although within each illness, these modality combinations have been examined. Specifically, Schlosser observed a direct correlation in schizophrenia between frontal fractional anisotropy (FA) reduction and fMRI activation in regions in prefrontal and occipital cortices (Schlosser, et al. 2007). Wang identified abnormal perigenual anterior cingulate and amygdala functional connectivity in bipolar disorder when combining fMRI and DTI data (Wang, et al. 2009), suggesting that disrupted white matter connectivity may contribute to disturbances in coordinated functional processing. All of the latter applications were based on univariate analysis in regions of interest. Although these initial studies combined DTI and fMRI data in a rather simplistic fashion, they demonstrated the feasibility and potential of a multimodal fusion approach in addressing various theoretical and applied issues of structure-function relationships (Rykhlevskaia, et al. 2008).

The complexities of SZ and BP make these diseases an ideal test bed for exploration using joint information derived from functional-structural data fusion, although more advanced statistical and analytical models are required to do so. Data from different imaging methods are typically analyzed separately; however such approaches do not enable examination of cross-information between modalities. An alternative approach, called data integration, combines modalities only at the moment of interpreting the result, thus not allowing for any interaction between the data-types (Olesen, et al. 2003; Seghier, et al. 2004), e.g. using seed areas. Such methods based on voxel-wise comparison of structural and functional maps cannot be statistically informative, hence limiting inferences based on these data.

Another strategy based on simultaneous analysis of multimodal data is data fusion, which includes “blind” data-driven approaches such as joint independent component analysis (jICA) (Calhoun, et al. 2006b), multimodal canonical correlation analysis (mCCA) (Correa, et al. 2010a; Correa, et al. 2010c), partial least squares (PLS) (Grady, et al. 2006; Martinez-Montes, et al. 2004), linked ICA (Groves, et al. 2010) and other adaptive approaches such as parallel ICA (Liu, et al. 2009) and coefficients-constrained ICA (Sui, et al. 2009). All of the above are based on linear mixture models and provide complimentary perspectives on data fusion via different optimization assumptions.

According to many previous findings in brain connectivity studies which also combined function and structure (Camara, et al. 2010; Olesen, et al. 2003; Rykhlevskaia, et al. 2008), it is plausible to assume the components decomposed from each modality have some degree of correlation between their mixing profiles among subjects. Therefore, we are motivated to propose a data-driven model that is optimized for this situation and also to have excellent performance for achieving both flexible modal association and source separation. Among the existing joint analysis models, joint ICA maximizes the independence of joint sources of two or more datasets, thus the decomposed source maps are distinct from each other, but it only allows one mixing matrix for all modalities (e.g. modalities are highly correlated, r=1); On the other hand, mCCA allows a different mixing matrix for each modality and links two datasets by maximizing the correlation of the inter-subject mixing profile, but the associated source maps may not differ sufficiently in some cases, especially when the canonical correlation coefficients are close in value (Sui, et al. 2010a). Hence, we are motivated to combine the complementary mCCA and jICA approaches to improve the performance of the joint source extraction. In other words, mCCA first relates two datasets with flexible linkage (correlation), thus giving us more confidence to perform joint ICA on the associated maps, which allows both highly and weakly correlated joint components. Figure 1 depicts four blind data fusion methods, including the above three and our proposed scheme: mCCA+jICA. Note that the proposed mCCA+jICA approach is distinct from our previous work in (Sui, et al. 2010b), where spatial CCA (sCCA) +jICA is used to analyze datasets of the same type, such as multitask fMRI data. The difference between sCCA and mCCA is illustrated in Figure 1 for clarification.

Figure 1.

presents a summary of various blind data-driven methods for data fusion. Note that both mCCA and sCCA belong to CCA, but maximize the correlation in different parts of the data. In mCCA+jICA, mCCA automatically links two datasets via correlation between mixing matrices; while jICA further decomposes the remaining mixtures in the associated maps.

The basic strategy of mCCA+jICA is as follows: mCCA is first adopted to link two or more modalities by correlation maximization of the mixing profiles, see Figure 1. The two mixing matrices are so-called canonical variants (CVs) and their associated maps (C_i, i=1, 2) may still contain mixtures of real independent sources. Next we perform a joint ICA on the concatenated maps [C₁, C₂] and further decompose the remained mixtures to [S₁, S₂] by maximizing the joint independence. Hence, mCCA and jICA are complementary to one another and if used together can relax the limitations of each. In our work we extract two mixing matrices (A₁, A₂), with variable A₁–A₂ correlation for each component, which provides both shared (highly-correlated) and unique (weakly-correlated) information between modalities. In DTI-fMRI fusion, the structure-function link is reflected by this A₁–A₂ correlation, which can be seen as a multivariate generalization of published reports which correlate an FA value in one ROI for each subject to all the fMRI contrast voxels (Avants, et al. 2010; Olesen, et al. 2003)”.

In order to reduce the redundancy of the large scale brain data and facilitate the identification of relationships between modalities, each modality is first reduced to a “feature” for each subject, e.g. an fMRI contrast map from the general linear model and DTI measures such as fractional anisotropy (FA). The fMRI contrast map depicts the cortical regions representing the task effect in relation to an experimental baseline. FA refers to the coherence of the orientation of water diffusion, independent of direction. It is calculated as the fraction of total diffusion that can be attributed to anisotropic diffusion, with higher values corresponding to a more consistent diffusion orientation (McIntosh, et al. 2008a). Therefore, a breakdown in white matter integrity results in a lower FA. Our goal was to simultaneously extract the group-discriminating brain activations in fMRI contrasts as well as the abnormal white matter tracts reflected in the FA maps.

Therefore, in this paper we highlighted both similarities and differences between two modalities across three diagnostic groups, which had not yet been attempted for a relatively large number of subjects. Our aim was to better understand the complex distinctions at a brain level between bipolar disorder and schizophrenia. We applied the proposed model to 164 subjects comprising 62 healthy controls (HCs), 54 SZs and 48 BPs, who were recruited at the Olin Neuropsychiatric Research Center, Hartford and were scanned by both fMRI (while performing an auditory oddball task) and DTI. Results were examined from several perspectives. We found not only group-discriminating regions for each modality and each pair-wise groups, but also spatial variations in temporal lobe and different group relationships with regard to age effects on our dependent measures. Most importantly, we are able to visualize an underlying function-structure network by evaluating the joint components which have strong DTI-fMRI links. Note that some of the specific findings we report were not obtainable from separate analysis of each modality and required a joint analysis. The potential benefits of our approach were demonstrated both in a simulation and an application to multimodal human brain data.

MATERIALS AND METHODS

Assumptions

We assume that the multimodal dataset X_k is a linear mixture of sources S_k and the nonsingular mixing matrix A_k, k = 1, 2,

$${\mathbf{X}}_k={\mathbf{A}}_k{\mathbf{S}}_k$$ (1)

where X_k is in form of subjects by voxels, as shown in Figure 2; A_k is in dimension of subjects by number of sources M_k. The columns of A₁ and A₂ (mixing profile of each modality) have higher correlation only on their corresponding indices,

$$E\{{\mathbf{A}}_1^T{\mathbf{A}}_2\}={\mathbf{R}}_{1,2}\approx \mathit{\text{diag}}(r_1,r_2\dots r_M)M=\text{min}(M_1,M_2)$$ (2)

where r₁, r₂…r_M can be either common or different from each other. This assumption is more flexible compared to that of both joint ICA and mCCA. Because joint ICA requires A₁ and A₂ to be exactly the same; and mCCA achieves complete source separation only when r₁, r₂…r_M are sufficiently distinct (Li, et al. 2009a). This constraint is not always easily satisfied, especially when the number of components M are large (e.g. >10) or the number of datasets used in the analysis is small (e.g., two as in this case). Furthermore, due to the potential common correlation values among r₁, r₂…r_M, applying individual ICA within each dataset may introduce ambiguity in feature matching via cross-correlation, as we shown in (Sui, et al. 2010b). By contrast, the proposed mCCA+jICA model alleviates these limitations: because mCCA makes the jICA job more reliable by providing a closer initial match via correlation; while jICA further decomposes the remaining mixtures in associated maps and relaxes the constraint that mCCA has on the distinctiveness of r₁, r₂…r_M.

Figure 2.

Flow chart of the mCCA+jICA analysis process for the clinical fMRI and DTI data fusion

Theory Development

Given that there are N subjects, typically, the number of voxels L in X_k is much larger than N. Due to the high dimensionality and high noise levels in the brain imaging data, order selection is critical to avoid overfitting the data. Using the minimum description length (MDL) criterion as in (Li, et al. 2007), M_k, the number of independent components are estimated for each dataset, M_k < N. Dimension reduction is then performed on X_k using singular value decomposition (SVD), a scheme where small singular values of the matrix are treated as noise/redundancy are discarded, given

$${\mathbf{X}}_k^T={\mathbf{E}}_k{\mathbf{U}}_k{\mathbf{F}}_k=[{\mathbf{E}}_k^{\prime }{\mathbf{E}}_k^{″}]{\mathbf{U}}_k{\mathbf{F}}_k,k=1,2$$ (3)

where ${\mathbf{E}}_k^{\prime }$ in dimension of L × M_k, contain the eigenvectors corresponding to the significant eigenvalues sorted from high to low in U_k; E″ contain the eigenvectors which are treated as noise and hence omitted from the next steps of the analysis. CCA is thus performed on the dimension-reduced datasets Y_k in size of N × M_k, given by ${\mathbf{Y}}_k={\mathbf{X}}_k{\mathbf{E}}_k^{\prime }$, generating two transformed variants, D₁ D₂, the so-called canonical variants (CV) (Correa, et al. 2008). D₁ and D₂ have maximum correlation between each other, while the vectors within each are uncorrelated and whitened. Their correlation vector c₁, c₂…c_M is called the canonical correlation coefficient (CCC), given by

$$E\{{\mathbf{D}}_1^T{\mathbf{D}}_2\}=E\{{\mathbf{D}}_2^T{\mathbf{D}}_1\}=\mathit{\text{diag}}(c_1,c_2\dots c_M)M=\text{min}(M_1,M_2)$$ (4)

$$E\{{\mathbf{D}}_1^T{\mathbf{D}}_1\}=E\{{\mathbf{D}}_2^T{\mathbf{D}}_2\}=\mathbf{I}$$ (5)

Based on the linear mixture model, we simultaneously obtain the associated components C₁ and C₂ via

$${\mathbf{X}}_k={\mathbf{D}}_k·{\mathbf{C}}_{k}k=1,2$$ (6)

However, the performance of mCCA for blind source separation (BSS) may suffer when c₁, c₂…c_M are very close in values, which might occur frequently in applications using real brain data, since the multimodal connection among components usually are not very high and could be similar in value(Correa, et al. 2010b). Therefore, C₁ and C₂ will typically be two sets of sources that do not represent a completely decomposed set of components.

Joint ICA is then implemented on the concatenated maps, [C₁, C₂], to maximize the independence among joint components by reducing their second and higher order statistical dependencies, as in equation (7). ICA as a central tool for BSS has been studied extensively and many algorithms have been developed based on different patterns of cost function (Bell and Sejnowski 1995; Cichocki A, et al. 2007; Hyvarinen 2001). We utilized COMBI (Tichavsky, et al. 2008) in our work due to its flexibility in adapting to different source distributions and its fast speed.

$$[{\mathbf{S}}_{\mathbf{1}},{\mathbf{S}}_{\mathbf{2}}]=\mathbf{W}·[{\mathbf{C}}_{\mathbf{1}},{\mathbf{C}}_{\mathbf{2}}]$$ (7)

Thus two joint independent components S₁ and S₂ are achieved, with their corresponding mixing profile linked via correlation. According to (1), (6), and (7), the proposed mCCA+jICA scheme can be summarized as

$${\mathbf{X}}_k=({\mathbf{D}}_k·{\mathbf{W}}^{-\mathbf{1}})·{\mathbf{S}}_k,{\mathbf{A}}_k={\mathbf{D}}_k·{\mathbf{W}}^{-\mathbf{1}}$$ (8)

We then correlate A₁ with A₂, generating a correlation profile, i.e. r₁, r₂…r_M in (2), which could contain identical values.

Simulated data

By simulating two types of image sources as two features, we investigate the joint BSS performance of mCCA+jICA on simulated data and compare it to that of joint ICA and mCCA. As shown in Figure 4, eight sources are generated for each feature to simulate images (256×256) and one-dimensional signals (1×2000) respectively, resulting in true sources S₁ (in dimension of 8×65536) and S₂ (in dimension of 8×2000). Note that the source type in 1D, 2D or 3D does not matter, the relevant factor is the final vector length. Here S₁ and S₂ are designed on purpose with greatly different vector length (65536 vs. 2000). This has multiple benefits over our previous multitask fusion paper (Sui, et al. 2010b) in which we used spatial CCA instead of multimodal CCA which requires S₁ and S₂ to have the same length. The mixing matrices of each feature, i.e., A₁ and A₂ (in dimension of 100×8), have diverse correlations between their corresponding columns, A₁–A₂ correlation = [0.99 0.07 0.36 0.63 0.20 0.23 0.79 0.36], as the ground truth listed in Figure 3. One hundred noisy mixed images X_k are generated for each modality under each of the 11 noisy conditions via X_k = I_k + N_k = A_k S_k + N_k, k=1, 2; where I_k is pure signal mixture and N_k is random Gaussian noise. The corresponding mean peak signal-to-noise ratios (PSNR) are in range of [−1 20] dB. The PSNR is a most commonly used measure of image quality after corruption or recovery, which is defined as (9) for the j^th mixture of feature k at every noisy condition. Typical PSNR value for the acceptable image quality is about 30 dB; the lower the value, the more degraded the image (Thomos, et al. 2006).

$$\mathit{\text{PSNR}}(k,j)=10{\text{log}}_{10}\left[\frac{\underset{i\in [1,l]}{\text{max}}{({\mathbf{I}}_k(i))}^2}{\frac{1}{l}{\displaystyle \sum _{i=1}^l}|{\mathbf{X}}_k(i)-{\mathbf{I}}_k(i){|}^2}\right]j=1,2\dots 100,l=\{\begin{array}{c}65536,k=1\hfill \\ 2000,k=2\hfill \end{array}$$ (9)

Figure 4.

simulation results of comparing separation performance of 3 methods 8 sources for each simulated modality: S₁ (left) and S₂ (right) are designed on purpose with greatly different final vector length (65536 vs. 2000, implying multimodal data); 100 mixtures are generated for each PSNR, here we display PSNR=6dB. The correlations of mixing coefficients between corresponding sources of each modality are listed in the middle, so do their estimations. See jICA separate sources accurately except the 3^rd S₂ signal, while mCCA estimates the modal connection accurately except that it can not decompose images quite well. mCCA+jICA combine both advantages and improve the performance remarkably.

Figure 3.

illustrates the whole simulation results for 3 factors and in 2 ways. The 3 factors are source estimation (Ŝ), mixing matrix estimation (Â) and modal linkage shown by correlation between mixing coefficients of each modality(corr(A₁(:,i),A₂(:,i))), which are displayed in three rows. For each factor, we compared 3 algorithms in different noise conditions (left column) and for each source index (right column). Under each noise condition (PSNR), we illustrate the average of all sources’ estimation. For each specific source, we plotted its mean estimation and standard derivation across all noisy conditions. It is indicated that mCCA+jICA is robust for noise and source type, and its source decomposition performance is the best.

Three joint BSS models: jICA, mCCA and mCCA+jICA were implemented on simulated datasets respectively under every PSNR for 10 runs. The decomposed components are paired with the true sources via cross-correlation automatically within each feature. We adopted three metrics to estimate the joint BSS performance:

1. the average correlation of the estimated components Ŝ with true sources S;

2. the average correlation of the estimated mixing profiles  with the ground truth A;

3. the mean square error of the estimated A₁–A₂ correlation compared to the truth.

For each metric, we compared the three algorithms in two ways, i.e., under different noise conditions and at different source indices.

Human brain data

Participants 62 healthy controls (HC, age 38±17, 30 females), 54 patients with schizophrenia (SZ, age 37±12, 22 females) and 48 patients with bipolar disorder (BP, age 37±14, 26 females) were recruited at the Olin Neuropsychiatric Research Center and were scanned by both fMRI and DTI. All subjects gave written, informed, Hartford hospital IRB approved consent. Schizophrenia or bipolar disorder was diagnosed according to DSM-IV-TR criteria in the on the basis of a structured clinical interview (First 1995) administered by a research nurse and review of the medical file. All patients were stabilized on medication prior to the scan session in this study. Healthy participants were screened to ensure they were free from DSM-IV Axis I or Axis II psychopathology (assessed using the SCID (Spitzer 1996)) and also interviewed to determine that there was no history of psychosis in any first-degree relatives). All subjects were urine-screened to eliminate those who were positive for abused substances. Patients and controls were age and gender matched, with no significant differences among 3 groups, where age: p=0.93, F=0.07, DF=2. Sex: p=0.99, χ² =0.017, DF=2. All participants had normal hearing, and were able to perform the oddball task successfully during practice prior to the scanning session.

The Auditory oddball task involved subjects encountering three frequencies of sounds: target (1200 Hz with probability, p=0.09), novel (computer generated complex tones, p = 0.09), and standard (1000 Hz, p=0.82) presented through a computer system via sound insulated, MR-compatible earphones. Stimuli were presented sequentially in pseudorandom order for 200 ms each with inter-stimulus interval (ISI) varying randomly from 500 to 2050 ms. Subjects were asked to make a quick button-press response with their right index finger upon each presentation of each target stimulus; no response was required for the other two stimuli. Two runs of 244 stimuli were presented (Kiehl, et al. 2001).

Imaging parameters Scans were acquired at the Institute of Living, Hartford, CT on a 3T dedicated head scanner (Siemens Allegra) equipped with 40mT/m gradients and a standard quadrature head coil. The functional scans were acquired using gradient-echo echo planar imaging (EPI) with the following parameters: repeat time (TR) = 1.5 sec, echo time (TE) = 27 ms, field of view = 24 cm, acquisition matrix = 64×64, flip angle = 70°, voxel size = 3.75×3.75×4 mm³, slice thickness = 4mm, gap = 1 mm, number of slices = 29; ascending acquisition. Six dummy scans were carried out at the beginning to allow for longitudinal equilibrium, after which the paradigm was automatically triggered to start by the scanner. DTI images were acquired via a single-shot spin-echo echo planar imaging (EPI) with a twice-refocused balance echo sequence to reduce eddy current distortions, TR/TE= 5900/83 ms, FOV= 20 cm, acquisition matrix=128×96, reconstruction matrix 128×128, 8 averages, b=0, and 1000 s/mm² along 12 non-collinear directions, 45 contiguous axial slices with 3 mm slice thickness.

FMRI preprocessing fMRI data were preprocessed using the software package SPM5 (http://www.fil.ion.ucl.ac.uk/spm/software/spm5/)(Friston, et al. 2005). Images were realigned using INRIalign, a motion correction algorithm unbiased by local signal changes (Freire, et al. 2002). Data were spatially normalized into the standard MNI space (Friston, et al. 1995), smoothed with a 12 mm³ full width at half-maximum Gaussian kernel. The data, originally 3.75×3.75×4 mm, were slightly subsampled to 3×3×3 mm, resulting in 53×63×46 voxels.

FMRI Feature Extraction a GLM approach using SPM5 was used to find task-associated brain regions, labeled as contrast maps which were then used as fMRI features within our mCCA+jICA analysis. Specifically, a GLM analysis consisted of a univariate multiple regression of each voxel's time course with an experimental design matrix, generated by the convolution of the task onset times with a hemodynamic response function. This resulted in a set of beta-weight maps associated with each parametric regressor. The subtraction of target beta-weight map with the standard beta weight map was referred to as a “target contrast” map, which represented the task effect in relation to an experimental baseline. We utilized the average target effect for the AOD task.

DTI preprocessing consists of the following steps:

1. DTI Quality Check. The DTI data quality is checked for a) signal dropout due to subject motion, producing striated artifacts on images; b) excessive background noise in the phase encoding direction, due to external RF leakage in the MRI scan room or to subject motion; and c) large amounts of motion in the absence of signal dropout. If for a specific gradient direction any slice was found to have a problem, we decided to exclude the whole volume rather than some specific slices. Our assumption was that the presence of a large head movement in one slice compromised the entire volume. This resulted in different number of gradients being used for different subjects. The average number of gradient directions dropped was 12% for healthy controls and 14% for patients with schizophrenia and bipolar disorder (being similar for the two patient groups). FA may tend to increase as number of gradient directions is reduced, but the bias is negligible if the reduction in the number of gradient directions is no more than 15%, as reported in (Ling, et al. 2010), which examined this problem via simulations. In our study, the motion artifacts were not significantly larger in patients group.

2. Motion and eddy current correction. After the above data pruning we had one 4D DTI volume and a table of corresponding b-values and gradient direction vectors. Next we registered all the images to a b=0 s/mm² image. Twelve degrees of freedom, affine transformation with mutual information cost function was used for image registration.

3. Adjusting the diffusion gradient direction. Two corrections were applied to the diffusion gradients. The nominal diffusion gradients directions are prescribed in the magnet axis frame. We rotated them to correspond to the image slice orientation. No correction is required if the imaging slice is pure axial. A second correction accounted for any image rotation during the previous motion and eddy current correction step. The rotation part of the transformation found previously was extracted, and each gradient direction vector corrected accordingly. All the image registration and transformations were done with the FLIRT (FMRIB’s Linear Image Registration Tool) program (FMRIB Software Library [FSL]; www.fmrib.ox.ac.uk/fsl), and the detection of outliers and data pruning was done with a custom program written in IDL (www.ittvis.com).

DTI Feature Extraction. We used Dtifit, a tool in FSL to calculate the diffusion tensor and the fractional anisotropy (FA) maps. The FA image was aligned to a MNI FA template with a nonlinear registration algorithm FNIRT (FMRIB’s Nonlinear Image Registration Tool; FSL) and was co-registered with fMRI contrast via SPM, resulting in a final 53×63×46 matrix with the voxel size of 3×3×3 mm. The FA maps were then smoothed using SPM5 with a 12 mm³ full width at half-maximum Gaussian kernel (Silver, et al. 2011).

Analysis Pipeline After feature extraction, the 3D image of each subject was reshaped into a one-dimensional non-zero vector and stacked one by one, forming a matrix with dimensions of 164×[number of voxels] for each modality. Then the feature matrix was normalized to have the same average sum-of-squares (computed across all subjects and all voxels for each modality). The normalization was needed because the AOD and FA data had different ranges. A single normalization factor was used for each data type; thus, following normalization, the relative scaling within a given data type was preserved, but the units between data types were the same (in a least-squares sense). After normalization, the data was processed via the pipeline shown in Figure 2, i.e., dimension reduction-> multimodal CCA-> joint ICA-> component analysis.

RESULTS

Simulations

Figure 3 illustrates the simulation results for three evaluation metrics on the whole, displayed in three rows. For each metric, we compared three algorithms in different noisy conditions (PSNRs, left column) and source distributions (source index, right column). Under each noise condition (PSNR), we illustrate the averaged estimation accuracy on sources, mixing profiles and the modal linkage respectively in figure (a), (c) and (e). It is evident that mCCA+jICA is quite robust to noise, and its BSS performance is consistently the best in all noise conditions. Consequently, joint ICA is the second best in source estimation and mCCA is the second best in mixing profile estimation; mCCA+jICA also overperforms mCCA on estimation of modal connection and their estimation errors were not influenced much by the noise. Note that when PSNR=−1dB, i.e., noise is bigger than signal, all three methods can still have the estimation accuracy of Ŝ and  higher than 0.55.

For each specific joint source, we plotted the mean correlation and its standard derivation across all noisy conditions via bars in figure 3 (b) and (d). In figure 3 (f), the true A₁–A₂ correlation is given via a yellow bar for every source, while the mean square error and its standard derivation of the link estimation were plotted in red for mCCA and in green for mCCA+jICA. Note that both very high (0.99) and low (0.07) correlation values exist in modal connection, representing both shared and distinct parts of two features. Some sources have very close (5 and 6, r =0.20, 0.23) or the same (3 and 8, r=0.36) low correlation values, which is quite ordinary in real applications of brain data fusion and mCCA+jICA consistently performs better.

We next focus on one noisy case (PSNR=6dB) in order to directly investigate the joint BSS performance in this context. Results are presented in Figure 4, where true sources and true modal connection are shown on the left. Joint ICA separates almost all sources accurately especially for sources 1, 4, and 7 since their A₁–A₂ correlation >0.6, but failed to decompose the 3^rd source for feature 2 whose A₁–A₂ correlation is lower (r=0.36). Multimodal CCA can track the modal connection more precisely than jICA, but cannot completely decompose image sources in feature 1, particularly sources 3–6. The proposed mCCA+jICA combine advantages of both methods and improve the performance substantially. It succeeds in separating sources and linking them correctly in a less-constrained condition, where there is no stringent requirement for the A₁–A₂ correlation.

Human brain data

Next, mCCA+jICA was applied to real fMRI and DTI data collected from 164 subjects including three groups. Our goal was to identify the aberrant functional-structural brain relationships in schizophrenia and bipolar disorder. Ten components were estimated for each feature according to an improved MDL criterion (Li, et al. 2007). We summarize the results in the following four aspects:

1) Group differences in loading parameters

Two sample t-tests were performed for each IC on its mixing coefficients between each of the possible pair-wised groups, as shown in Figure 5, where the components with significant group differences were summarized and the PT means both SZ and BP. For each component, we chose 4 representative slices to display and use different color frames for discrimination. If two independent components have the same frame color in both modalities they are joint components. The p values displayed with yellow text passed a Bonferroni correction for multiple comparison (p<0.005), others passed an uncorrected significance level (p<0.05).

Figure 5.

Summary of components with significant group differences in four pair-wise group combinations. Two ICs with the same frame color in two modalities represent joint ICs. Those p values displayed with yellow text pass the Bonferroni correction for multiple comparison (p<0.005), others pass the uncorrected significance level (p<0.05).

The specific identified regions of the above group-discriminating components and their abbreviations are summarized in Table 1 for fMRI (Talairach labels) and Table 2 for DTI (white matter tracts) respectively, where the ICs with ^# were joint group-discriminating components, i.e., it can differentiate groups in both AOD task and FA. The ICs with * had highly significant p values, which passed a Bonferroni correction for multiple comparison. To summarize the white matter results, we used the Johns Hopkins white-matter tractography atlas (from FSL), from which 20 structures were identified (Hua, et al. 2008); mostly these are large bundles. In Table 2, the WM tract labels, the identified volume (cc) and the percentage that indicates the overlap of the identified regions in each WM tract are listed in detail.

Table 1.

MNI Table of group-discriminating fMRI components (R/L)

IC 1#		BA	Vol	Zmax (MNI)
Postivie
POG	Postcentral Gyrus	1–3, 5, 40, 43	9.2/11.8	4.3 (−45, −20, 56)/6.0 (42, −20, 56)
PRG	Precentral Gyrus	4, 6	9.7/11.3	4.4 (−36, −23, 62)/5.9 (39, −17, 59)
IPL	Inferior Parietal Lobule	40	0.9/3.5	3.4 (−45, −32, 57)/5.3 (45, −35, 57)
SFG/MFG	Superior/Middle Frontal Gyrus	6	1.4/3.8	3.1 (−39, −3, 55)/4.3 (27, −8, 64)
MeFG	Medial Frontal Gyrus	6, 32	5.4/4.8	3.2 (0, −6, 50)/3.2 (3, −6, 53)
CG	Cingulate Gyrus	24	1.0/0.4	2.9 (0, −3, 47)/2.8 (3, −1, 47)
PCL	Paracentral Lobule	4, 6, 31	1.5/1.3	2.7 (0, −24, 51)/2.7 (3, −29, 65)
Negative
IFG	Inferior Frontal Gyrus	47	1.9/0.5	2.6 (−48, 23, −9)/2.2 (39, 17, −8)
PCun	Precuneus	7	0.6/0.2	2.3 (−3, −74, 42)/2.2 (3, −71, 39)
IC 2#*		BA	Vol	Zmax (MNI)
Positive
IFG/MFG	Inferior/Middle Frontal Gyrus	9, 11, 44–46	1.2/1.3	2.3 (−53, 10, 19)/2.3 (45, 29, 7)
Negative
PC	Posterior Cingulate	29, 30	0.6/0.3	5.1 (−3, −46, 8)/4.1 (3, −46, 8)
LG	Cuneus/Lingual Gyrus	17–19	5.7/5.9	4.5 (−9, −85, −11)/4.9 (3, −96, 2)
FUSG	Fusiform Gyrus	18, 19, 37	1.9/1.7	4.3 (−24, −85, −13)/3.8 (21, −88, −13)
PHG	Parahippocampal Gyrus	28, 30, 34, 35	2.3/1.3	4.1 (−9, −38, 5)/2.9 (15, −6, −12)
SFG	Superior Frontal Gyrus	6, 10	2.8/0.4	4.0 (−3, 3, 66)/2.5 (27, 61, 5)
THA	Thalamus		1.5/0.8	3.7 (−3, −14, 9)/3.3 (3, −14, 9)
STG	Superior Temporal Gyrus	38	1.7/0.0	3.1 (−53, 14, −13)/NA
IC 6		BA	Vol	Zmax (MNI)
Postivie
STG	Superior Temporal Gyrus	22	2.6/0.7	3.8 (−59, 6, 0)/2.5 (56, 3, 3)
MTG	Middle Temporal Gyrus	19	0.0/1.6	NA/3.3 (39, −72, 26)
THA	Thalamus		0.8/1.3	3.1 (−6, −2, 8)/3.1 (6, −5, 11)
IFG	Inferior/Medial Frontal Gyrus	9–11, 44– 47	1.1/1.2	2.4 (−3, 58, 3)/2.8 (24, 28, −19)
PCun/AG	Precuneus/Angular Gyrus	7, 19, 31, 39	0.4/2.4	2.2 (−15, −60, 25)/ 3.0 (36, −71, 31)
PRG	Precentral Gyrus	6, 44	1.7/1.4	2.6 (−59, 0, 6)/2.6 (56, 3, 8)
AC	Anterior Cingulate	25	0.1/0.1	2.6 (−3, 6, −3)/2.2 (3, 6, −3)
Negative
Cun/LG	Cuneus/Lingual Gyrus	17, 18, 19	2.9/7.3	3.9 (−3, −93, 5)/4.8 (3, −96, 2)
S/MFG	Superior/Middle Frontal Gyrus	6, 8–10, 46	4.6/5.1	4.0 (−39, 59, 8)/4.0 (36, 58, 3)
PHG	Parahippocampal Gyrus	28, 34, 35	2.2/0.0	4.0 (−12, −1, −15)/NA
CG	Cingulate Gyrus	23	1.0/0.4	2.9 (−3, −28, 26)/2.7 (3, −28, 26)
IC 8#		BA	Vol	Zmax (MNI)
Positive
Cun	Cuneus	7, 17–19, 23, 30	14.7/11.8	5.4 (−6, −67, 9)/5.0 (3, −75, 15)
PC	Posterior Cingulate	18, 23, 29– 31	4.2/3.8	5.4 (−6, −69, 12)/4.6 (6, −69, 12)
LG	Lingual Gyrus	17–19	8.1/8.8	5.2 (−15, −58, 3)/4.8 (12, −64, 3)
PCun	Precuneus	7, 18, 19, 31	3.9/1.9	4.6 (−3, −72, 23)/4.2 (3, −74, 26)
PHG	Parahippocampal Gyrus	18, 19, 30	2.0/0.9	4.4 (−21, −52, 3)/3.1 (12, −47, 2)
FUSG	Fusiform Gyrus	19, 37	1.1/0.3	3.0 (21, 59, 7)/2.5 (21, 62, 7)
Negative
MFG	Middle/Medial Frontal Gyrus	10, 11	1.0/2.1	2.5 (−3, 49, −15)/2.7 (3, 46, −12)
IFG	Inferior Frontal Gyrus	11, 47	0.1/0.3	2.5 (−27, 31, −19)/2.7 (24, 28, −19)
AC	Anterior Cingulate	32	0.0/0.2	NA/2.2 (6, 40, −10)
THA	Thalamus		0.2/0.3	2.2 (−3, −11, 12)/2.1 (3, −17, 12)

Table 2.

The white matter tract label of the group-discriminating FA components

IC1#		WM tracts		vol(cc)		percentage	Zmax(R/L)
Pos	ATR	Anterior thalamic radiation		2.0/2.1		4%/4%	4.7(22,24,6)/3.6(28,25,6)
	FM	Forceps minor/Forceps major		4.7/3.1		8%/6%	4.6(27,47,21)/3.8(17,16,22)
	SLFt	SLF (temporal part)		1.8/0.1		39%/5%	4.5(13,22,19)/2.1(40,24,23)
	CST	Corticospinal tract		2.3/1.6		6%/5%	4.3(21,25,6)/4.1(34,24,6)
	SLF	Superior longitudinal fasciculus		1.3/2.5		1%/2%	4.3(12,22,19)/2.6(40,34,27)
	ILF	Inferior longitudinal fasciculus		2.0/1.3		5%/3%	4.2(14,22,19)/3.4(42,30,13)
	IFO	Inferior fronto-occipital fasciculus		2.5/0.1		5%/0%	3.9(14,24,19)/2.5(36,45,25)
	CGC	Cingulum (cingulate gyrus)		0.1/0.9		1%/4%	2.2(25,43,26)/3.0(29,43,26)
Neg	ATR	Anterior thalamic radiation		2.8/3.6		6%/7%	4.1(22,29,23)/4.0(33,27,22)
IC2#		WM tracts		vol(cc)		percentage	Zmax(R/L)
Pos	ATR	Anterior thalamic radiation		5.4/5.5		11%/10%	4.9(25,34,23)/4.5(29,35,23)
	SLF	Superior longitudinal fasciculus		3.0/1.8		3%/2%	3.7(14,21,23)/3.3(46,36,11)
	SLFt	SLF (temporal part)		0.7/0.2		16%/12%	3.6(14,21,22)/2.5(44,28,14)
	ILF	Inferior longitudinal fasciculus		1.1/1.8		2%/4%	3.4(15,20,23)/3.5(45,34,11)
	FM	Forceps minor/Forceps major		2.0/2.9		4%/6%	3.1(33,49,21)/3.4(16,18,24)
	IFO	Inferior fronto-occipital fasciculus		1.8/0.9		3%/2%	3.4(16,18,23)/2.5(36,17,24)
	CG	Cingulum		0.6/0.7		2%/2%	2.6(20,24,19)/2.8(33,18,30)
Neg	SLF	Superior longitudinal fasciculus		2.5/2.7		3%/3%	3.3(17,38,28)/3.2(37,29,29)
IC7*		WM tracts		vol(cc)		percentage	Zmax(R/L)
	Positive
Pos	CGC	Cingulum (cingulate gyrus)		0.6/0.5		3%/2%	2.7(21,20,33)/3.7(29,41,27)
	CST	Corticospinal tract		0.7/1.0		2%/3%	2.4(20,33,35)/3.6(34,32,35)
	ATR	Anterior thalamic radiation		0.8/1.1		2%/2%	3.3(26,26,6)/3.2(34,33,34)
	SLF	Superior longitudinal fasciculus		1.5/2.2		1%/2%	2.9(9,21,16)/3.4(40,36,32)
	UF	Uncinate fasciculus		0.1/0.1		1%/1%	2.6(14,48,24)/3.3(40,50,21)
	IFO	Inferior fronto-occipital fasciculus		0.5/0.8		1%/2%	2.7(14,48,23)/3.0(41,13,25)
	FM	Forceps minor/Forceps major		0.9/0.3		2%/1%	2.9(24,56,29)/2.7(30,9,28)
	ILF	Inferior longitudinal fasciculus		0.2/0.5		1%/1%	2.7(18,33,16)/2.9(30,12,32)
Neg	ILF	Inferior longitudinal fasciculus		1.8/0.5		4%/1%	4.4(13,24,17)/4.0(44,30,10)
	IFO	Inferior fronto-occipital fasciculus		2.6/0.3		5%/1%	4.4(14,22,18)/2.8(36,11,14)
	FM	Forceps major		2.0		4%	4.3(16,18,21)
	SLF	Superior longitudinal fasciculus		1.6/2.8		2%/3%	4.3(13,23,18)/4.1(39,28,28)
	CGH	Cingulum (hippocampus)		0.6/0.1		6%/1%	3.2(18,20,18)/2.6(32,14,19)
	ATR	Anterior thalamic radiation		0.8/1.7		2%/3%	2.7(25,33,15)/3.2(30,31,24)
IC8#		WM tracts		vol(cc)		percentage	Zmax(R/L)
Pos	SLF	Superior longitudinal fasciculus		3.4/3.0		4%/3%	4.3(10,27,12)/4.5(45,27,12)
	SLFt	SLF (temporal part)		1.6/0.1		35%/7%	4.4(12,24,17)/3.2(44,27,13)
	ILF	Inferior longitudinal fasciculus		1.8/2.1		4%/4%	4.2(13,24,17)/3.1(43,29,11)
	IFO	Inferior fronto-occipital fasciculus		2.8/1.2		5%/3%	3.8(15,22,22)/3.0(38,19,23)
	FM	Forceps minor/Forceps major		7.3/2.7		13%/5%	3.6(31,47,24)/3.7(16,21,23)
	ATR	Anterior thalamic radiation		3.5/4.0		7%/8%	3.2(27,29,10)/3.4(34,48,23)
	CGC	Cingulum (cingulate gyrus)		0.8/1.7		4%/7%	2.7(23,23,26)/3.3(30,46,24)
IC9		WM tracts		vol(cc)		percentage	Zmax(R/L)
Pos	SLF	Superior longitudinal fasciculus		7.5/5.6		8%/6%	4.2(11,23,17)/3.4(37,29,30)
	CST	Corticospinal tract		3.1/3.5		9%/10%	3.3(18,30,30)/3.3(36,29,31)
	ILF	Inferior longitudinal fasciculus		1.1/0.5		3%/1%	3.3(17,19,29)/3.2(40,20,18)
	IFO	Inferior fronto-occipital fasciculus		1.0/0.3		2%/1%	3.2(14,21,19)/3.1(40,21,18)
	ATR	Anterior thalamic radiation		0.5/0.4		1%/1%	2.6(21,34,31)/2.9(34,27,31)
	CGC	Cingulum (cingulate gyrus)		0.2/0.5		1%/2%	2.5(21,25,30)/2.5(33,28,31)
Neg	UF	Uncinate fasciculus		1.6/0.7		15%/6%	4.2(23,41,13)/4.7(30,43,14)
	ATR	Anterior thalamic radiation		4.0/2.1		8%/4%	4.5(23,39,17)/4.0(28,30,13)
	SLF	Superior longitudinal fasciculus		0.6/0.6		1%/1%	3.8(9,24,12)/4.0(44,25,12)
	ILF	Inferior longitudinal fasciculus		0.6/1.4		1%/3%	2.7(16,14,15)/3.3(40,24,12)
	CST	Corticospinal tract		0.4/0.7		1%/2%	2.8(26,29,9)/2.9(29,28,10)
	IFO	Inferior fronto-occipital fasciculus		0.8/0.5		2%/1%	2.6(21,43,16)/2.5(33,39,17)

2) Group differences in spatial maps

The back-reconstruction of spatial maps for each group was performed based on the linear mixture model as in (10),

$${\mathbf{S}}_k={\left[\begin{array}{c}{\mathbf{A}}_{\mathit{\text{HC}},k}\hfill \\ {\mathbf{A}}_{\mathit{\text{SZ}},k}\hfill \\ {\mathbf{A}}_{\mathit{\text{BP}},k}\hfill \end{array}\right]}^{-1}·\left[\begin{array}{c}{\mathbf{X}}_{\mathit{\text{HC}},k}\hfill \\ {\mathbf{X}}_{\mathit{\text{SZ}},k}\hfill \\ {\mathbf{X}}_{\mathit{\text{BP}},k}\hfill \end{array}\right],{\mathbf{S}}_{\mathit{\text{HC}},k}={[{\mathbf{A}}_{\mathit{\text{HC}},k}]}^{-1}·{\mathbf{X}}_{\mathit{\text{HC}},k},{\mathbf{S}}_{\mathit{\text{SZ}},k}={[{\mathbf{A}}_{\mathit{\text{SZ}},k}]}^{-1}·{\mathbf{X}}_{\mathit{\text{SZ}},k},{\mathbf{S}}_{\mathit{\text{BP}},k}={[{\mathbf{A}}_{\mathit{\text{BP}},k}]}^{-1}·{\mathbf{X}}_{\mathit{\text{BP}},k},k=1,2$$ (10)

where S_k denotes the IC maps derived by mCCA+jICA for all subjects; k represents the modality, [·]⁻¹ represents the pseudo-inverse of the matrix; S_HC,k, S_SZ,k and S_BP,k are the back-reconstructed spatial maps for HC, SZ and BP respectively. To investigate the group variation in spatial maps, we correlated S_HC,k/S_SZ,k/S_BP,k with S_k individually for both modalities. The FA maps did not show large differences among 3 groups with relatively higher correlations (r>0.8); while the fMRI maps exhibited more variations. We listed all correlation values between the reconstructed group maps with the IC maps for fMRI in Table 3. For display, we chose the IC that had correlation values higher than 0.5 for all groups (to make sure that the spatial maps of all groups were consistent to some degree) and controls were obviously different from patients in correlation values (±0.1). Hence IC 9 was selected based on the above criteria, which shows strong activation in both superior temporal gyrus and motor cortex.

Table 3.

Similarity between the reconstructed group-maps with the fMRI IC map

IC Num	1	2	3	4	5	6	7	8	9	10
corr(SHC,1,S1)	0.78	0.75	0.85	0.43	0.63	0.61	0.39	0.90	0.89	0.64
corr(SSZ,1,S1)	0.77	0.63	0.64	0.61	0.10	0.50	0.55	0.81	0.78	0.69
corr(SBP,1,S1)	0.92	0.78	0.82	0.72	0.42	0.62	0.54	0.46	0.75	0.76

Figure 6 displays the variation of functional spatial maps among groups. IC9 derived from mCCA+jICA for all groups is shown on the left. On the right, the back-reconstructed group sources S_BP, S_SZ, S_HC and S_{g common} (the common activation of S_BP, S_SZ and S_HC) are overlaid one by one from bottom to top. For display, the components are converted to Z-values (by dividing by the standard deviation of the source) and thresholded at |Z|>2.

Figure 6.

Variation of functional spatial maps (AOD_IC9) among groups. Left: The AOD_IC9 derived from mCCA+jICA. Right: the back-reconstructed group sources S_BP, S_SZ and S_{g common} (the common activation of S_BP, S_SZ and S_HC) are overlaid one by one from bottom to top. For display, the components were converted to Z-values and thresholded at |Z|>2.

3) Group differences in age effects

The derived mixing coefficients also provide a way to investigate the relationship between the identified regions and subjects’ behavioral data, such as age. We correlated participants’ ages with the mixing profile for each IC and three were found to be significant: AOD_IC1, FA_IC1 and FA_IC9. The correlation values (r) and significance level (p) were listed in Table 4 for all subjects and each group. We also computed the 95% BCa (Bias corrected and accelerated percentile method) bootstrap confidence interval(ci) for all the correlations by using 100 bootstrapping samples for each group. Multiple comparisons for 40 correlations were performed using FDR correction and the significant p values were marked as bold in Table 4. Figure 7 demonstrated the scatter plot of age versus ICA weights, and the group linear trends in different colors and markers for AOD_IC1 and FA_IC9; specifically, HC trend in red line, SZ trend in blue line, BP trend in green line and trend of all subjects in black line. For FA_IC1, all groups showed significant anti-correlation between age and FA weights, thus were plotted using one marker.

Table 4.

Correlation with ages and the corresponding p values and confidence intervals (ci)

		All Subjects	HC	SZ	BP
AOD_IC1	r	0.30	0.50	0.15	0.27
	p	7e-05	4e-05	3e-02	6e-02
	ci	0.20~0.43	0.34~0.65	−0.09~0.36	0.09~0.43
FA_IC1	r	−0.42	−0.49	−0.36	−0.35
	p	2e-08	1e-05	1e-03	2e-03
	ci	−0.56~−0.31	−0.66~−0.38	−0.59~−0.15	−0.58~−0.11
FA_IC9	r	−0.36	−0.25	−0.53	−0.40
	p	2e-06	5e-02	4e-05	5e-03
	ci	−0.51~−0.24	−0.48~−0.05	−0.70~−0.39	−0.64~−0.15

Figure 7.

demonstrates the scatter plots and linear trends between subjects’ age and loading parameters. Specifically, HC in red line, SZ in blue line, BP in green line and trend of all subjects in black line. For FA_IC1, All groups have very significant correlation, thus were plotted using one marker.

4) Functional-structural network

We calculated the A₁–A₂ correlations for all components, as listed in Table 5, which provide a measure that describes the modal linkage, i.e., the functional-structural connection. Note that most of the correlation values are quite significant with p<=0.005, except for IC4, which was a ventricular CSF artifact. We choose 3 representative ICs for display, whose A₁–A₂ correlations are moderate to high. Among these, IC 8 had the highest A₁–A₂ correlation (r=0.37, p=1e-6) and was the only component showing SZ versus BP differences in their loadings. FA_IC7 shows the most group-discriminative loadings and FA_IC9 shows a significant correlation with age. In addition, both AOD_IC7 (default mode network) and AOD_IC9 (temporal and motor activation) include brain regions that are likely activated by the auditory oddball task.

Table 5.

The functional-structural mixing profile correlation

Comp Num	1	2	3	4	5	6	7	8	9	10
A1–A2 Correlation	0.26	0.29	0.35	0.16	0.26	0.23	0.21	0.37	0.27	0.28
p value	8e-04	2e-04	4e-06	4e-02	7e-04	3e-03	5e-03	1e-06	5e-04	4e-04

Figure 8 shows the AOD component maps on the left, the FA component in the middle and a high-level functional-structural network diagram on the right. Our goal is to evaluate the intersection of the results with 1) for FA, known tracts, and 2) for fMRI, known gray matter regions, and to identify which known tracts are both intersected by the regions of FA changes and touch the regional fMRI changes. The functional lobes with large volume activations are bordered by red solid frames; by contrast, the lobe with small volume activations is bordered dotted blue line. Only the FA fiber tracts with more than 1 cm³ volume identified (R+L, positive+negative) were displayed. The specific Talairach and WM labels of the three joint components are listed in table 6 for reference.

Figure 8.

This figure attempts to verify that the “linked”(joint) components do correspond to FA changes in known tracts and functional changes in distant regions connected to that tract. Note that we did not perform actual fiber tractography but provided a type of summary statistic. Three joint components are selected with their A₁–A₂ correlation from moderate to high. On the right diagrams, functional region with a red solid line frame indicates that a major portion of it is activated. Region with dotted line frame indicates that only small part of it is activated. We plotted only FA fiber tracts with more than 1 cm³ volume(R+L) activated. Abbreviations are defined below, F: frontal lobe, P: parietal lobe, T: temporal lobe, O: occipital lobe, PH: parahippocampal gyrus, SLF: Superior longitudinal fasciculus, CGC: Cingulum, CST: Corticospinal tract, UF: Uncinate fasciculus IFO: Inferior fronto-occipital fasciculus, ILF: Inferior longitudinal fasciculus, FMIN: Forceps minor, FMAJ: Forceps major, ATR: Anterior thalamic radiation.

Table 6.

Validation of linkage----labels of three joint components

FMRI (R/L)						DTI (R/L)
IC 7		BA		Vol	Zmax	IC7		Vol	Zmax
Positive						Positive
IPL		40		1.9/4.4	2.2/2.5	CGC		0.6/0.5	2.7/3.7
POG		2, 5, 40		0.5/1.2	2.1/2.4	SLF		1.5/2.2	2.9/3.4
PCun		7		0.6/0.6	2.4/2.2	CST		0.7/1.0	2.4/3.6
STG		22, 38, 42		0.8/0.6	2.3/2.3	ATR		0.8/1.1	3.3/3.2
Negative						IFO		0.5/0.8	2.7/3.0
CG		31		4.3/3.5	5.8/5.5	FMIN		0.9	2.9
PCun		7, 19, 23, 31, 39		7.8/7.4	5.6/5.6	ILF		0.2/0.5	2.7/2.9
PC		23, 29, 30, 31		5.1/5.3	5.5/5.5	Negative
MeFG		6, 8–11		7.8/8.2	4.3/4.2	ILF		1.8/0.5	4.4/4.0
AG/AC		10, 32,39		2.6/3.3	3.6/3.9	IFO		2.6/0.3	2.8/4.4
MFG/SFG		8, 9, 10		5.0/7.0	3.8/3.8	FMAJ		2.0	4.3
Cuneus		7		0.2/0.2	3.8/2.9	SLF		1.6/2.8	3.9/4.1
STG/MTG		39,22		2.1/3.1	3.0/3.6	CGH		0.6/0.1	3.2/2.6
SmG		40		1.7/1.0	3.2/3.4	ATR		0.8/1.7	2.7/3.2
IC 8		BA		Vol	Zmax	IC8		Vol	Zmax
Positive						Positive
Cuneus		7, 17–19, 23, 30		14.7/12	5.4/5.0	SLF		3.4/3.0	4.3/4.5
PC		18, 23, 29–31		4.2/3.8	5.4/4.6	SLFt		1.6/0.1	4.4/3.2
LG		17, 18, 19		8.1/8.8	5.2/4.8	ILF		1.8/2.1	4.2/3.1
PCun		7, 18, 19, 31		3.9/1.9	4.6/4.2	IFO		2.8/1.2	3.8/3.0
PH		18, 19, 30		2.0/0.9	4.4/3.1	FMAJ		2.7	3.7
FUSG		19, 37		1.1/0.3	3.0/2.5	FMIN		7.3	3.6
Negative						ATR		3.5/4.0	3.2/3.4
IFG/MFG		10, 11		1.1/2.4	2.5/2.7	CGH		0.8/1.7	2.7/3.3
Thalamus				0.2/0.3	2.2/2.1	CST		0.2/0.3	2.7/2.9
IC9		BA		Vol	Zmax	IC9		Vol	Zmax
Positive						Positive
STG/MTG		13,21,22, 38,41,42		12/14.7	3.1/4.7	SLF		7.5/5.6	4.2 /3.4
TTG		41, 42		0.7/1.7	2.8/4.5	CST		3.1/3.5	3.3/3.3
PRG		4, 6, 13, 43, 44		0.7/8.1	2.4/4.2	ILF		1.1/0.5	3.3/3.2
Insula		13, 22, 40		1.2/8.2	2.3/4.1	ATR		0.5/0.4	2.6/2.9
POG/IPL		2, 3, 40, 43		1.9/8.7	2.7/4.1	CGC		0.2/0.5	2.5/2.5
SFG/MFG		6, 32		2.8/2.4	3.7/3.3	FMIN		0.3	2.4
CG		24, 32		1.7/1.3	3.3/3.3	Negative
Negative						UF		1.6/0.7	4.2/4.7
SPL		7		1.5/2.5	3.9/4.6	ATR		4.0/2.1	4.5/4.0
PCun		7, 19, 39		3.2/5.1	3.6/4.2	SLF		1.1/1.4	3.8/4.0
IPL		7, 39, 40		6.5/5.1	3.7/3.9	ILF		0.8/1.4	2.7/3.3
MFG		6, 8–10, 46		7.3/6.0	2.8/3.1	CST		0.3/0.6	2.8/2.9
SFG/MeFG		6,8, 9, 10		3.7/2.7	2.7/2.7	IFO		0.8/0.5	2.6/2.5
IFG		9, 10, 45, 46		0.8/1.6	2.4/2.6

DISCUSSION

In this paper, we highlighted patterns of both similarities and differences in two modalities (fMRI and DTI) across three diagnostic groups (HC, SZ, BP), which have not been attempted previously for a relatively large number of subjects. Our aim was to clarify differences between bipolar disorder and schizophrenia using functional-structural data fusion employing an advanced analytical model. We propose a multimodal fusion method called “mCCA+jICA”, which can work with less stringent assumptions than the previously proposed approaches. It can also achieve higher decomposition accuracy and identify valid links between features. Note that the mCCA+jICA approach does not increase the computational load appreciably; however it achieves the best performance in simulations designed to be similar to real-world brain imaging data fusion.

One of the major strengths of this model is that the combination of mCCA and joint ICA improves the performance of joint BSS substantially under a less-constrained condition. Specifically, in Figure 3 and 4, sources 1, 4 and 7 have higher A₁–A₂ correlation values, thus joint ICA works well, in accordance with its hypothesis; consequently, the performance of mCCA suffers from ambiguity and misinterpretation in sources 3&8, 5&6 due to the requirement of sufficiently distinct canonical correlations; by contrast, the proposed model mitigates the performance deficits of mCCA by using a further ICA decomposition. In addition, the use of mCCA makes the job of jICA easier and more reliable by providing a closer initial match. We can also define links among data via the A₁–A₂ correlation values. As shown in Figure 3 (f), for sources with lower or closer A₁–A₂ correlations (sources 2,3,6,8), which could be the practical cases in brain imaging data fusion, mCCA+jICA consistently performs better, and has no stringent requirements imposed on the A₁–A₂ correlation.

Based on the merits of the proposed model, we applied it to a real brain data set, with 3 groups and 2 modalities. There are at least three major advantages to combining DTI and fMRI data:

1. FMRI as a well-established neuroimaging technology, can act as a reference framework verifying the validity of conclusions derived from the relatively newer DTI method (Reinges, et al. 2004).

2. Such a combination may provide insights into the relationships/connectivity between brain structure and function (Guye, et al. 2008; Skudlarski, et al. 2010).

3. Combining structural and functional brain imaging data allows for constructing more informative, biologically plausible models and may have utility for neurosurgical applications.

Therefore, the large dataset and the novel method together demonstrated the potential to detect relevant brain differences between bipolar disorder and schizophrenia.

Amplitude difference

Three joint components (IC 1, 2, 8) and three modality-specific components (AOD_IC6, FA_IC7 and FA_IC9) were identified as group-discriminating ICs, as shown in Figure 5. We next discuss the findings for each of these in detail.

AOD_IC2 is the most significant functional component discriminating both SZ and BP from HC, which includes dorsolateral prefrontal cortex (DLPFC), thalamus and occipital cortex. DLPFC plays an important role in the integration of sensory and mnemonic information, executive function, planning and regulation of cognitive function and action. Researchers have frequently reported dysfunction and lack of functional connectivity of this region in patients with schizophrenia (Badcock, et al. 2005; Hamilton, et al. 2009) and bipolar disorder (Curtis, et al. 2001; Glahn, et al. 2010). The thalamus has multiple functions and is believed to act as a relay station between many subcortical areas and the cerebral cortex. It was also reported to be impaired in both SZ and BP (Cerullo, et al. 2009; Danos 2004). Our results are consistent with the above findings and suggest that these deficits might be related to shared risk factors and disease mechanisms common to both disorders. The joint FA_IC2 indicates an FA reduction in BP versus HC in the anterior thalamic radiation (ATR) that projects from anterior and medial regions of the thalamus to frontal lobe, in agreement with (Anand, et al. 2009).

FA_IC7 is the most significant DTI component where the identified WM fiber tracts mostly connect frontal and occipital lobes, including ATR, inferior fronto-occipital fasciculus (IFO), cingulum and superior longitudinal fasciculus (SLF). This finding is consistent with several previous reports of DTI abnormalities in schizophrenia (Kubicki, et al. 2005; Kubicki, et al. 2003; Schlosser, et al. 2007) and bipolar disorder (Wang, et al. 2009; Yurgelun-Todd, et al. 2007), suggesting that disruptions in white matter connectivity may contribute to coordinated brain dysfunction, especially in the frontal lobe, which is frequently thought of as “disconnected” from other brain regions in both disorders (Heng, et al. 2010; Williams, et al. 2004).

AOD_IC8 and FA_IC8 are joint components that can distinguish two patient groups, though not at a high level of significance (p=0.04). In functional maps, the parahippocampal gyrus (PH), medial frontal cortex, thalamus and visual cortex are emphasized, confirming the results in (Hall, et al. 2009), where two brain disorders were separated by differences in hippocampal and prefrontal cortical activation. Moreover, visual cortex impairments in SZ versus HC are also observed, consistent with (Brenner, et al. 2009; Demirci, et al. 2009). In FA maps, fibers associated with occipital and frontal lobes were identified, connecting the identified regions in AOD_IC8 in an organized network (see Figure 8), especially SLF, forceps minor (FMIN, that connects lateral and medial frontal lobes via the genu of the corpus callosum), and forceps major (FMAJ, that connects the occipital lobes via the splenium of the corpus callosum). These patterns of white matter pathology offer a potential diagnostic distinction between the two disorders. Furthermore, a similar function–structure relationship in the visual system was reported in healthy controls by (Toosy, et al. 2004), where the mean FA of the optic radiations correlated significantly with fMRI measures of visual cortex activity. Together with our results, these findings support the hypothesis that cortical fMRI response may be constrained by the external anatomical connections of white matter fibers.

Spatial variation

The back-reconstructed components present a direct view of the spatial distribution for each group (which can vary) and the intensities of the image (Z scores) provide a relative strength of the degree to which the group source contributes to the overall data (Beckmann and Smith 2005). If S_HC,k, S_SZ,k and S_BP,k are more similar to each other, they are likely to have more overlaps with S_k; if there are deviations, it likely means spatial group variability exist in such regions. In practice, it is hard to determine exactly to what extent the deviations between group sources can occur, however, simulations tell us that general back-reconstruction is quite capable of handling variations between the groups in the regions which contribute to the extracted source, an important feature given that we don’t expect the groups to have the same pattern. (Calhoun, et al. 2001; Erhardt, et al. 2010)

As shown in Figure 6, all three groups show higher consistence in parietal lobe (BA 7 19 39 40), primary and supplementary motor cortex (BA 3 4 6), and middle frontal gyrus (BA 10 32 46), implying that the spatial distribution of the identified regions implicated in movement and sensory integration is similar for all subjects (Andersen and Buneo 2002).

On the other hand, more variations were observed in temporal gyrus (BA 21 22 41 42) which is responsible for processing of auditory information, suggesting that aberrant patterns of coherence in temporal lobe may be a cardinal abnormality in both schizophrenia and bipolar disorder (Calhoun, et al. 2008; Chance, et al. 2008; Pearlson 1997). SZ activated more in superior and middle temporal gyri, supported by (Calhoun, et al. 2004). BP exhibits additional activations in insula, which has a role in emotional regulation, in accordance with findings in (Kempton, et al. 2009; McIntosh, et al. 2008c; Pessoa and Adolphs 2011), where insula also differentiates BP from HC and SZ. We also observed spatial deviations occurred between individual group map and IC map. For example, BP alone shows clear bilateral thalamic activation, while the left panel shows only the left thalamus for all subjects, suggesting that there may be diminished prefrontal modulation of subcortical and medial temporal structures within the anterior limbic network (e.g., amygdala and thalamus) that results in mood dysregulation, such as bipolar illness (Blumberg, et al. 2003; Strakowski, et al. 2005).

Age effects

Three ICs show significant correlation between subjects’ age and loadings. AOD_IC1 presents activations mainly in motor cortex (BA 1–6), accompanied by a functional asymmetry with left dominance, see Figure 7(a), consistent with the fact that AOD task needs participants push the button with their right fingers. Controls have a very significant correlation r = 0.5, p=4e-5, while patient groups do not (p does not pass multiple comparison), implying that the motor regions of HC are normally more involved in the task with age increased (Bennett, et al. 2010), whereas patients have no such trend due to motor system deficits (Rogowska, et al. 2004). FA_IC1, as a joint component of AOD_IC1, also demonstrates significant (p=2e-08), but anti-correlation with age. Note that all groups have low p values, suggesting all subjects have a general age-related decrease in FA values, representing a breakdown in white matter integrity, as shown in Figure 7 (b) and Table 4. This finding is in agreement with multiple prior reports from DTI aging research (Grieve, et al. 2007; Sullivan and Pfefferbaum 2003). Furthermore, as illustrated in Figure 7(c) and Table 4, SLF, corticospinal tract (CST), ATR and uncinate fasciculus (UF) are indentified in FA_IC9; interestingly, both patient groups revealed negative correlations with age r<−0.4 while HC do not, suggesting that white matter density in these tracts decreases faster in patients, especially for schizophrenia. Similar abnormalities in SZ were reported by (McIntosh, et al. 2008b).

Function-structure network

Imaging structural and functional brain connectivity has the potential to reveal the complex network of brain organization, which may reveal the pathway of information segregation and integration, and may also underlie the clinical consequences of alterations in mental illnesses. In Figure 8 and Table 6, we selected 3 joint components for illustration, attempting to verify that the “linked” components do indeed correspond to FA changes in known tracts and functional changes in distant regions connected to that tract. Note that we did not perform the WM fiber tractography but provided a type of summary statistic. FA and fMRI are showing different things and a strength of our method is that it can detect complicated FA/fMRI relationships without requiring a detected direct link.

The anterior thalamic radiation is identified in all three FA components (IC7,8,9) in Figure 8, constituting a potential anatomical biomarker for these two mental illnesses. ATR carries association fibers from anterior and medial regions of the thalamus to prefrontal cortex. This pathway is a part of several functionally segregated thalamofronto-striatal loops(Alexander, et al. 1990) and implicated in the pathophysiology of both BP and SZ (Sussmann, et al. 2009). These findings are also supported by neuro-pathological studies (Lewis 2000) showing evidence of decreased thalamic projection neurons and prefrontal thalamic inputs.

AOD_IC7 demonstrates negative activations in the typical default mode network (Raichle, et al. 2001) and positive activations in STG and IPL, which are potentially related by the tracts shown in FA_IC7. In FA_IC7, except ATR, IFO passes backward from frontal lobe into occipital and lateral temporal lobes; CST originates from the motor cortex to lower motor neurons in the spinal cord, cingulum projects from the cingulate gyrus to the entorhinal cortex(an important memory-related area in brain, that is a major input to the hippocampus) and SLF connects all brain lobes with each other.

AOD_IC9 reflects main fMRI activations in temporal and frontal lobe, consequently, FA_IC9 localize the breakdown of WM integrity in UF (carrying association fibers between the medial prefrontal cortex and the anterior temporal lobe, including amygdala) and ATR for patients, more so for bipolar. Our results support the DTI findings in (McIntosh, et al. 2008a) and (Sussmann, et al. 2009), where patients with SZ and BP show consistent reduced integrity in the UF and ATR, implying a considerable overlap in white matter pathology, possibly relating to risk factors common to both disorders.

Future extension

We would like to highlight additional aspects of our study. Besides the application to fMRI-DTI data fusion, mCCA+jICA can be used to analyze various combinations of multimodal data too, such as fMRI with EEG/MEG, or genetic data. Furthermore, this approach is not limited to two-way fusion, but can potentially be extended to 3-way or N-way fusion of multiple data types by replacing the multimodal CCA with multi-set CCA (Li, et al. 2009b); e.g., combining fMRI, sMRI and genetic data to construct a broad function-structure-genetics network, thus arguing for widespread utilization of the proposed method.

Another interesting future direction is to further examine the subgroup differences within patients, e.g., BPs with/without psychotic features, by applying the proposed model, which might optimally separate a larger-sample size of BPs, though in current case we did not find very significant differences between these two subgroups.

A possible limitation is that mCCA+jICA works on extracted features of the original imaging data. In our case, a “feature” is a distilled dataset representing the interesting part of each modality and tends to be more tractable than working with the large-scale original data due to the reduced dimension, e.g. 4D fMRI data (Calhoun and Adali 2009). Hence the main reason to use features is to provide a simpler space in which to link the data. The trade-off is that some information may be lost. However there is considerable evidence that the use of features is quite useful. For example, for task-related fMRI data, the contrast maps depict the cortical regions representing the task effect in relation to an experimental baseline, which already utilize the temporal information. In a recent PNAS paper (Smith, et al. 2009), which used ICA on rest data and on task blobs (in the case of task blobs, even more information was lost than we used) and they found nearly identical resting fMRI patterns from the task blobs, motivating the use of features.

Our approach is also suitable for resting-state fMRI, we can perform an ICA on the original 4D data and then use the ICA components as input. This has been done for default mode in the joint ICA context in multiple papers including (Franco, et al. 2008). Similar work has also been done by (Smith, et al. 2009).

In conclusion, studies featuring combination of DTI and fMRI information prove to be fruitful for a more informative understanding of bipolar disorder and schizophrenia. In this paper, we developed an mCCA+jICA model that takes advantage of two multivariate approaches and enables more flexibility in statistical assumptions, promising for wider applications in neuroimaging community. Our application highlighted data from two illnesses and two modalities, and was analyzed carefully to uncover multiple types of group differences in the brain, including amplitude, spatial distribution, age effects and functional-structural relationships. We identified several group-discriminating aspects that replicated previous reports, implying that while SZ and BP show differences in regions including parahippocampal gyrus and visual cortex, they also share several common abnormalities in frontal brain mechanisms and in prefrontal thalamic white matter tracts. Such observations add to our understanding of the neural correlates of both disorders and may have utility as potential brain illness biomarkers.