Multimodal Fusion with Reference: Searching for Joint Neuromarkers of Working Memory Deficits in Schizophrenia

Abstract

Multimodal fusion is an effective approach to take advantage of cross-information among multiple imaging data to better understand brain diseases. However, most current fusion approaches are blind, without adopting any prior information. To date, there is increasing interest to uncover the neurocognitive mapping of specific behavioral measurement on enriched brain imaging data; hence, a supervised, goal-directed model that enables a priori information as a reference to guide multimodal data fusion is in need and a natural option. Here we proposed a fusion with reference model, called “multi-site canonical correlation analysis with reference plus joint independent component analysis” (MCCAR+jICA), which can precisely identify co-varying multimodal imaging patterns closely related to reference information, such as cognitive scores. In a 3-way fusion simulation, the proposed method was compared with its alternatives on estimation accuracy of both target component decomposition and modality linkage detection. MCCAR+jICA outperforms others with higher precision. In human imaging data, working memory performance was utilized as a reference to investigate the covarying functional and structural brain patterns among 3 modalities and how they are impaired in schizophrenia. Two independent cohorts (294 and 83 subjects respectively) were used. Interestingly, similar brain maps were identified between the two cohorts, with substantial overlap in the executive control networks in fMRI, salience network in sMRI, and major white matter tracts in dMRI. These regions have been linked with working memory deficits in schizophrenia in multiple reports, while MCCAR+jICA further verified them in a repeatable, joint manner, demonstrating the potential of such results to identify potential neuromarkers for mental disorders.

I. Introduction

Noninvasive neuroimaging has provided remarkable new insights into human brain structure and function in both health and disease. There is increasing evidence that instead of using a single imaging modality to study its relationship with physiological or cognitive features, people are paying more attention to multimodal fusion, an approach that is able to capitalize on the strength of multiple imaging techniques, since it can uncover the hidden relationships that might be missed from separate unimodal imaging studies [1–4]. Compelling evidence has confirmed that neuropsychiatric disorders reflect fundamental differences in brain structure and function. By jointly analyzing rich types of data and taking advantage of the cross-information, multimodal fusion can help better reveal the potential functional-structural covariations. For example, how brain structure shapes brain function, and to what degree brain function is related to the underlying brain anatomy [5]. Increasingly, studies are focusing on identifying the intrinsic functional or structural brain patterns that may ultimately drive a specific domain of human cognition or behavior, whereas most existing fusion models are purely data-driven. Hence in this work, we are motivated to develop a supervised multimodal fusion model that is able to discover the co-varying imaging features particularly related to a referred measurement more precisely and robustly.

Existing multivariate data-driven multimodal fusion methods have in most cases been based on blind source separation (BSS) techniques [6]. Specifically, multi-set canonical correlation analysis (CCA) and sparse CCA maximize the inter-modality covariance across multiple data sets, thus enable the identification of the co-varying multimodal components with similar individual variabilities, but their associated spatial maps may not be sufficiently unique [1]. Joint independent component analysis (ICA), and linked ICA perform well in spatial decomposition by maximizing the joint independence, but all modalities share a common profile. Combining the strengths of MCCA and jICA, Sui et al developed “MCCA+jICA” [7, 8], a blind fusion algorithm, which successfully captures both multimodal interactions and spatial components at high accuracy to study brain diseases. Other data fusion approaches like independent vector analysis (IVA) generalizes ICA to multiple data sets using the mutual information rate, achieving a similar performance to MCCA+jICA [9]. Similarly, Liu et al proposed parallel ICA [10] to specially deal with imaging genetic data, by maximizing both inter-modality correlation and the independence within each modality.

Although the above existing fusion methods can optimize joint source separation among multiple modalities in different aspects, investigators may also be interested in discovering multimodal associations with a specific reference varying across subjects, e.g., a cognitive score or a psychotic symptom. However, all above mentioned approaches are unsupervised, without inducing any prior reference [1, 5, 11–13]. Hence they may not be optimal when investigating the brain patterns related to a specific measure of interest. By contrast, a supervised learning model can be more goal-directed, since it takes advantage of the prior knowledge to guide the fusion analysis and is able to pinpoint a particular component of interest from a large complex dataset. One example is pICA-R [14] (parallel ICA with reference) which used candidate genes as spatial priori to improve investigation of the relationships between brain imaging and specific genetic attributes. While currently utilizing neuroimaging data to identify cognitive biomarkers has been a hot topic, mining multimodal co-alterations linking with a specific cognitive domain, e.g. working memory remains unexplored.

All above motivate us to improve the existing method and propose a supervised, fusion with reference model, i.e., “MCCAR+jICA” (multi-set CCA with reference + joint ICA). While keeping the original excellent performance on joint separation, the proposed method will further enable detection of the co-varying multimodal features that have significant correlations with the reference, which may not be achieved by blind N-way multimodal fusion approaches.

On the other hand, in neuropsychological studies, evidence has been accumulated that many mental disorders including schizophrenia (SZ) are associated with significant impairment in cognitive functioning, in which working memory is one of the most severely affected cognitive domains [15]. While most existing studies on schizophrenic working memory deficits are based on unimodal analysis [16, 17], multi-modal fusion become a natural option to provide more clues by exploiting the relationship between the enriched imaging data and the cognitive ability across individuals. In this paper, based on the proposed model, we aim to adopt working memory performance as the reference to guide a 3-way function+diffusion+structure MRI fusion, and ultimately identify the abnormal brain patterns correspond to the working memory impairment in schizophrenia. Two independent human data cohorts with 147 HC/147SZ and 44HC/39SZ respectively were used. Results show similar brain co-variations identified by MCCAR+jICA, with substantial overlap between the two cohorts for the executive control networks in fMRI, salience network in sMRI, and major white matter tracts in dMRI. These regions have previously been suggested to be associated with working memory deficits in multiple reports [18], but never in a multimodal analysis across multiple cohorts in the same study. To the best of our knowledge, this is the first attempt to estimate cognitive biomarkers by jointly mining three types of MRI data under the guidance of a particular cognitive domain score.

The remaining of the paper is organized as follows: section II presents the algorithm development of MCCAR+jICA as well as the used data. Section III is the results of simulation. In section IV, the real human brain data application with working memory were detailed. Section V includes a discussion of the results and a conclusion.

II. Methods and Materials

The main idea of MCCAR+jICA is straightforward: while maintaining the performance of MCCA+jICA, we hope to optimize specific subject-level correlations with a reference. To do this, a prior reference such as a cognitive score is incorporated to guide joint source separation, as shown in Figure 1.

Fig. 1.

Flowchart of the supervised 3-way data fusion strategy of MCCAR+jICA. As an example, the feature matrix of each modality is decomposed as 5 independent components (ICs), M=5. (a) Data preprocessing and stacked to 2D matrix for each modality. (b) represents multi-site canonical correlation analysis with reference (MCCAR) and its objective function. (c) illustration of MCCAR when M=5 and the 3^rd component is linked with reference, resulting in 3 canonical variants A_k, whose pair-wise covariation of is a diagonal matrix as shown in (d). (e) indicate the joint independent component analysis (jICA).

A. MCCAR

Assume that there are n multimodal datasets X_k, k = 1,2, …, n, each is a linear mixture of components C_k with a nonsingular mixing matrix A_k, k denotes modality.

$${\mathit{X}}_k={\mathit{A}}_k{\mathit{C}}_{k}k=1,2,\dots ,n$$ (1)

where X_k is a subjects-by-voxels feature matrix, A_k is in dimension of subjects by number of components M. MCCA with reference (MCCAR) imposes an additional constraint upon the MCCA framework to maximize not only the covariations among mixing matrices of each modality, but also the top column-wise correlations between A_k and the reference signal, as shown in Figure 1(c) and equation (2).

$$max{\mathrm{\sum }}_{\mathrm{k},\mathrm{j}=1}^{\mathrm{n}}\left\{{\Vert \text{corr}\left({\mathit{A}}_{\mathrm{k}},{\mathit{A}}_{\mathrm{j}}\right)\Vert }_2^2+\mathrm{\lambda }·{\Vert \text{corr}({\mathit{A}}_{\mathrm{k}},\text{ref})\Vert }_2^2\right\}$$ (2)

where corr(A_k, A_j) is the column-wise correlations between A_k and A_j, k, j = {1,2,3 … n}, k ≠ j.

The basic strategy of MCCAR is as follows: consider that there are N subjects, dimension reduction is first performed on X_k, thus the signal subspace given by Y_k = X_kE_k are determined. MCCAR is thus performed on Y_k, generating the canonical variants A_k by maximizing the sum of squared correlations (SSQCOR) among canonical variants (CVs) as well as the SSQCOR between each CVs and the reference signal. We can summarize the optimization procedure of MCCAR as below. Consider that the CVs A_k given by A_k = Y_kw_k were jointly decomposed into M components, then the canonical coefficient vectors w_k are updated by two stages:

• Stage 1:

  $$\{w_1^{(1)},w_2^{(1)},\dots ,w_n^{(1)}\}=\mathit{arg}\underset{w}{max}\left\{{\mathrm{\sum }}_{k,j=1}^n{\mid r_{k,j}^{(1)}\mid }^2+\lambda ·{\mathrm{\sum }}_{k=1}^n{\mid r_{k,\mathit{ref}}^{(1)}\mid }^2\right\}$$ (3)
• Stage 2: for i = 2: M

  $$\{w_1^{(i)},w_2^{(i)},\dots ,w_n^{(i)}\}=\mathit{arg}\underset{w}{max}\left\{{\mathrm{\sum }}_{k,j=1}^n{\mid r_{k,j}^{(i)}\mid }^2+\lambda ·{\mathrm{\sum }}_{k,j=1}^n{\mid r_{k,\mathit{ref}}^{(i)}\mid }^2\right\}\text{s.t.}{\mathrm{w}}_{\mathrm{k}}^{(\mathrm{i})}\perp \left\{{\mathrm{w}}_{\mathrm{k}}^{(1)},{\mathrm{w}}_{\mathrm{k}}^{(2)},\cdots ,{\mathrm{w}}_{\mathrm{k}}^{(\mathrm{i}-1)}\right\},\mathrm{k}=1,2,\cdots ,\mathrm{n}$$ (4)

• end

where ${\mathrm{w}}_{\mathrm{k}}^{(\mathrm{i})}(\mathrm{i}=1,\dots ,\mathrm{M})$ is the i^th column of the w matrices, ${\mathrm{r}}_{\mathrm{k},\mathrm{j}}^{(\mathrm{i})}=\text{corr}({\mathit{A}}_{\mathrm{k}}^{(\mathrm{i})},{\mathit{A}}_{\mathrm{j}}^{(\mathrm{i})})$ is the correlation between the i^th column of A_k and A_j, ${\mathrm{r}}_{\mathrm{k},\text{ref}}^{(\mathrm{i})}$ is the correlation between the i^th column of A_k and the reference signal, which has the same length of subject numbers. λ is the regularization parameter balancing the weight of two objective functions: ${\mathrm{\sum }}_{\mathrm{k},\mathrm{j}=1}^{\mathrm{n}}{\mid r_{k,j}^{(i)}\mid }^2$ and ${\mathrm{\sum }}_{\mathrm{k}=1}^{\mathrm{n}}{\mid r_{k,\mathit{ref}}^{(i)}\mid }^2$ and controls the convergence of the optimization. Stage 1 is solved by first calculating the partial derivative function of the SSQCOR cost with respect to each ${\mathrm{w}}_{\mathrm{k}}^{(1)}$ and equating it to zero to find the stationary point. Since the SSQCOR cost is a quadratic function of each ${\mathrm{w}}_{\mathrm{k}}^{(1)}$, the partial derivative is a linear function of ${\mathrm{w}}_{\mathrm{k}}^{(1)}$ and hence, the closed form solution can be derived. Stage 2 and higher stages are solved in a similar manner with the cost function replaced by a Lagrangian multiplier incorporating the orthogonality constraints on the canonical coefficient vectors.

In Figure 1, k ≠ j, k, j = 1,2,3, an example is when M = 5, as shown in Figure 1(b) and (c), based on the above optimization, we can obtain A₁, A₂, A₃ simultaneously as shown in Figure 1(d), which satisfies:

$$\mathrm{E}\{{\mathit{A}}_{\mathrm{k}}^{\mathrm{T}}{\mathit{A}}_{\mathrm{k}}\}=\mathrm{I},\mathrm{E}\{{\mathit{A}}_{\mathrm{k}}^{\mathrm{T}}{\mathit{A}}_{\mathrm{k}}\}\approx \text{diag}\left({\mathrm{r}}_{\mathrm{k},\mathrm{j}}^{(1)},{\mathrm{r}}_{\mathrm{k},\mathrm{j}}^{(2)},\dots ,{\mathrm{r}}_{\mathrm{k},\mathrm{j}}^{(\mathrm{M})}\right)$$ (5)

k ≠ j, k, j ∈ {1,2,3}; i.e., the covariation of A_k will be a diagonal matrix. In addition, one or more joint components, here the 3^rd joint component, will be the target component that has significant correlations with the reference.

B. MCCAR+jICA

Due to the potential marginally significant correlations between ROI and reference in each modality, applying correlation with reference within each dataset may introduce ambiguity in feature matching [19]. While MCCA focus only on maximizing inter-modality covariance as shown in Fig. 1(b); therefore, the proposed MCCAR solves two problems in one processing, resulting in unique and robust cross-modality correspondence as well as a CV that is highly correlated with reference in each modality. Although MCCAR can provide a useful decomposition in many cases, the associated maps C_k may still not be unique in some cases. In order to keep the joint target component and maximize the spatial independence, we further apply jICA on the concatenated maps [C₁, C₂, …, C_n] to obtain the final independent components (ICs)S_k, as well as their corresponding mixing matrices D_k:

$$\mathit{W}[{\mathit{C}}_1,{\mathit{C}}_2,\cdots ,{\mathit{C}}_{\mathrm{n}}]=[{\mathit{S}}_1,{\mathit{S}}_2,\cdots ,{\mathit{S}}_{\mathrm{n}}]$$ (6)

Combined with equation (1),

$${\mathit{X}}_{\mathrm{k}}={\mathit{D}}_k·{\mathit{S}}_{\mathrm{k}}=({\mathit{A}}_{\mathrm{k}}·{\mathit{W}}^{-1})·{\mathit{S}}_{\mathrm{k}}k=1,2,\cdots ,n$$ (7)

Therefore, compared to blind N-way decomposition, MCCAR+jICA ensures both spatial independence and better correspondence with the reference in the mixing profile. This means in human brain imaging applications, we could simultaneously explore interested multimodal covariation and associate it with specific clinical measures.

C. Simulated Data

We next simulated multimodal MRI data to compare the proposed method with its alternatives for its capability to extract accurate spatial maps and correspondence between multiple modalities and with the reference. Eight brain networks were simulated using the simTB [20] for fMRI and sMRI. DMRI was generated using the Johns Hopkins University (JHU) white matter atlas in which we selected 8 typical fiber bundles, as shown in Figure 2 (the red boxed source maps are designed to be correlated with reference). Loading matrices for each modality, A₁, A₂, A₃ were constructed in size of 300×8, resulting in 300 samples with 21025, 40000 and 65536 voxels for fMRI, dMRI and sMRI feature matrices respectively by linear combination. Here, we used a real cognitive score of 300 subjects as a reference and carefully designed one component for each modality (in different order, 7^th, 4^th, 1^st for fMRI, sMRI and dMRI respectively) to be significantly correlated with the reference, as shown in Supplementary Table I. We then applied MCCAR+jICA to the simulated datasets and compared its performance with 4 alternative methods under 16 noise levels. The noise levels were simulated with peak signal-to-noise ratio (PSNR) ranges from 1 dB to 34dB. Typical PSNR value for the acceptable image quality is about 30 dB; the lower the value, the more degraded the image [21].

Fig. 2.

The simulated 8 source maps for 3 modalities. The red boxed source maps are those designed to be correlated with the reference. Note that these component order are different between modalities, a strength of the supervised fusion model is able to extract them together as a joint component.

D. Human Brain Data

In this study, we used two independent data cohorts. One is recruited from the Function Biomedical Informatics Research Network (FBIRN) phase III datasets including 147 SZs (39.5 ± 11.7) and 147 HCs (37.4 ± 11) that were matched for age, gender, handedness, and race distributions. Demographics of all subjects are shown in Supplementary Table II. All subjects were collected from seven FBIRN consortium sites (University of California Irvine, University of California Los Angeles, University of California San Francisco, Duke University, University of North Carolina, University of New Mexico, University of Iowa, and University of Minnesota). Each dataset including diagnosis, age at time of scan, gender, illness duration, symptom scores, and current medications when available, were shared by each research group according to their site’s protocols. Inclusion criteria required all participants to be adults between the ages of 18 and 65 years. Diagnosis of schizophrenia was confirmed by trained raters using the Structured Clinical Interview for DSM-IV (SCID) [22]. All patients were on a stable dose of antipsychotic medication either typical, atypical, or a combination for at least 2 months. Current symptom severity was rated using the Positive and Negative Syndrome Scale (PANSS) [23]. All SZs were clinically stable at the time of scanning. In addition, HC participants were excluded for current or past psychiatric illness based on SCID assessment or for having a first-degree relative with a diagnosis of an Axis-I psychotic disorder. Written informed consent was obtained from all study participants under protocols approved by the Institutional Review Boards at each study site. Demographic information for subjects of each sites are provided in Supplementary Table III.

Another cohort consisting of 39 SZs (35.6±13.1) and 44 HCs (36.3±12.5) were collected from the University of New Mexico (UNM). Details of the demographic information, inclusion and exclusion criteria could be found in Supplementary Table IV.

1) Image Parameters and Preprocessing

The resting state fMRI data were collected on six 3T Siemens TIM Trio scanners and one 3T MR750 General Electric (GE) scanner. The imaging protocol for the resting state scans at all sites was a T2*-weighted AC-PC aligned echo planar imaging sequence (TR/TE 2 s/30 ms, flip angle 77 degrees, 32 slices collected sequentially from superior to inferior, 3.4 × 3.4 × 4 mm with 1 mm gap, 162 frames, 5:38 min). For the resting scan, subjects were instructed to lie still with eyes closed.

DMRI were acquired on six 3T Siemens Tim Trio System and one 3T GE Discovery MR750 scanner. All parameters for these two scanners were the same except for TE (Siemens 84ms/GE 81.7ms). The rest of the parameters for both Siemens and GE were as follows: TR = 9000ms; acquisition matrix = 128 × 128; field of view (FOV) = 256×256 mm; slice thickness = 2 mm; number of slices = 72; slice gap=2 mm; voxel resolution 2×2×2 mm; flip angle = 90; number of diffusion gradient directions = 30, b = 800 seconds/mm2, and 5 measurements with b = 0; number of excitations (NEX) =1. All images were registered to the first b = 0 image by FMRIB Linear Image Registration Tool (FLIRT: http://fsl.fmrib.ox.ac.uk/fsl/fslwiki/FLIRT).

High-resolution structural brain scans were also acquired on six 3T Siemens Tim Trio System and one 3T GE Discovery MR750 scanner using standardized sequences. Siemens MP-RAGE scan parameters were: TR/TE/TI=2300/2.94/1100 ms, flip angle=9°, resolution =256×256×160. GE IR_SPGR scan parameters were: TR/TE/TI=5.95/1.99/45 ms, flip angle =12°, resolution =256×256×166. All scans covered the entire brain with FOV=220 mm², voxel size= 0.86×0.86×1.2 mm³, sagittal scan plane, GRAPPA/ASSET acceleration factor=2, and NEX=1.

The fMRI data was preprocessed using the automated analysis pipeline[24], whose steps are conducted in SPM 5 (http://www.fil.ion.ucl.ac.uk/spm) as follows: Motion correction to the first image using INRIalign; slice timing corrected to the middle slice; and normalization to MNI space, including reslicing to 3 × 3 × 3 mm voxels. We further regressed out six motion parameters, white matter, and cerebrospinal fluid in denoising. Data were then spatially smoothed with an 8 mm full width half max (FWHM) Gaussian filter. To calculate fractional amplitude of low frequency fluctuations (fALFF) [25], the sum of the amplitude values in the 0.01 to 0.08Hz low-frequency power range was divided by the sum of the amplitudes over the entire detectable power spectrum (range: 0–0.25Hz) [see more details in [26]].

DMRI data were preprocessed using the FMRIB Software Library (www.fmrib.ox.ac.uk/fsl) and consisted of the following steps: 1) quality check with any gradient directions with excessive motion or vibration artifacts identified and removed; 2) motion and eddy current correction; 3) correction of gradient directions for any image rotation done during the previous motion correction step; and 4) calculation of diffusion tensor and scalar measures such as fractional anisotropy (FA), which were then smoothed using 8 mm FWHM Gaussian filter.

Using the unified segmentation methods of SPM5, the sMRI was normalized to MNI space, resliced to 3 × 3 × 3 mm, and segmented into gray matter (GM), white matter, and cerebral spinal fluid (CSF). Then, the GM images were smoothed with a FWHM of 8 mm Gaussian filter. Subject outlier detection was further performed using a spatial Pearson correlation with the template image to ensure that all subjects were properly segmented [details can be found in [27, 28]].

2) Normalization and Site Effect Correction

After preprocessing, the three-dimensional brain images of each subject were reshaped into a one-dimensional vector and stacked, forming a matrix (N_subj×N_voxel) for each of the three modalities. These three matrices were then normalized to have the same average sum of squares (computed across all subjects and all voxels for each modality) to ensure all modalities had the same ranges.

Multivariate analysis of covariance (MANCOVA) was performed on the normalized feature matrices. Gender, age and site were set as covariates, along with their interactions were all regressed out from fALFF, FA and GM matrices respectively, to minimize their potential impact on the imaging data, the resulting data were then ready for fusion analysis.

3) The Working Memory Scores

We use one cognitive domain measure, working memory (WM) score, as the reference for supervised multimodal MRI fusion. For fBIRN data, the WM score was measured by the Computerized Multiphasic Interactive Neuro-cognitive System (CMINDS) [29]. As listed in Supplementary Table II, CMINDS involves 6 domains, i.e., speed of processing, attention/vigilance, working memory, verbal learning, visual learning and reasoning/problem solving. For the validated data cohort from UNM, working memory score is measured by the MATRICS Consensus Cognitive Battery (MCCB) system, which is launched by the National Institute of Mental Health and contain one more domain, social cognition than CMINDS, see supplementary Table IV. Both cognitive measurement systems include computerized neuropsychological tasks that are structurally and functionally similar, and the neurocognitive domain z-scores were computed from these tests. As reported by [29], CMINDS is very similar to MATRICS on measuring cognitive deficits in schizophrenia.

III. Results of Simulation

The proposed method MCCAR+jICA together with its 4 alternatives (MCCA1, MCCA2, MCCAR, MCCA+jICA) were tested on the carefully designed simulated data, in which MCCA1 denotes MCCA optimized by the maximum variance method, MAXVAR [30], MCCA2 represent MCCA optimized by the sum of squared correlations method, SSQCOR [31]. For each method, the decomposed components are paired with the true sources via cross-correlation automatically within each modality. We compared their fusion performance across multiple aspects.

One important property is that whether the supervised method is able to detect the target component (significantly correlated with reference) accurately and with right correspondence across modalities. Figure 3(a) shows the ability for different fusion methods in getting priority-set reference signals as one joint component under one noise level (PSNR =7). It is clear that only supervised method MCCAR (blue) and MCCAR+jICA (magenta) could obtain multi-modal target components with right correspondence (same component order [1,1,1], all correlated with reference), i.e., the component linked with the reference in multiple modalities will come out as one joint component. While unsupervised algorithms may lose such correspondence info in noisy conditions, though the correlation value with reference could be captured. Figure 3(b) and (c) displays the boxplot of the estimation accuracy of the identified target component across 16 noise levels. The estimation accuracy used here is defined as the correlation between the true source(s)/mixing matrices and the estimated component(s)/mixing matrices. It is evident that MCCA+jICA (yellow) and MCCAR+jICA (magenta) exhibit outperform others on both source and mixing matrix for 3 modalities. Figure 3(d) displays the estimation of inter-modality correlation (black is the true value) of the identified target component and Figure 3(e) shows the mean absolute error of the estimated multimodal correlation. It is also evident that the estimation accuracy of MCCAR+jICA is closest to the ground truth. More importantly, even compared with 3 separate ICA in each modality, MCCAR+jICA also achieves the best modal-connection estimation with minimum absolute error, validating the advantages of the supervised, goal-directed model on extracting the target component more precisely.

Fig. 3.

Comparison of 5 approaches in a simulated 3-way data fusion, including MCCA1 (red), MCCA2 (green), MCCAR (blue), MCCA+jICA (yellow), and MCCAR+jICA (magenta). (a) Only two supervised method MCCAR and MCCAR+jICA are able to extract multi-modal target components with right correspondence as a joint component (same IC order). (b) and (c) show that MCCAR+jICA outperform others on estimation accuracy of both source and mixing profile of target ICs. (d) and (e) validate that MCCAR+jICA achieves best estimation of inter-modality correlation with minimum absolute error.

Besides the target component, for decomposition of other components, as shown in Figure 4, it is evident that MCCAR+jICA is quite robust to noise and its source separation performance is consistently the best in all noise conditions among 5 approaches, MCCA+jICA is the second best in source and mixing matrix estimation. Moreover, the performance of MCCAR is not as good as MCCAR+jICA, since MCCAR+ jICA achieves more precise source estimation than MCCAR, which demonstrates the necessity of applying jICA. Finally, the accuracy for different component numbers to identify target components is shown in Figure 5. The higher values of the accuracy, the brighter color is in the squares in Figure 5. It is clear that when estimated component number is equal or higher than the true value (M ≥ 8), MCCAR+jICA achieved the highest or close to highest accuracy among 5 methods in most cases.

Fig. 4.

The comparison of estimation accuracy of all the components under 16 level noises. MCCA1 (red), MCCA2 (green), MCCAR (blue), MCCA+jICA (yellow), and MCCAR+jICA (magenta). The latter two methods combining ICA and CCA achieved best or equivalent best performance for the whole decomposition.

Fig. 5.

Performance comparison when using different number of component M for decomposition. True source number =8, here M varies from 6 to 12. The color of squares represent the correlation between the estimated target IC and the ground truth on either source maps (top) or mixing coefficients (bottom) for different methods. The brighter square color, the higher estimation accuracy. Modality 1, 2, 3 denotes fMRI, dMRI and sMRI respectively. MCCAR+jICA achieves best estimation when M>= true source number.

When determining the value of λ that balances the weight of ${\mathrm{\sum }}_{\mathrm{k},\mathrm{j}=1}^{\mathrm{n}}{\mid r_{k,j}^{(i)}\mid }^2$ and ${\mathrm{\sum }}_{\mathrm{k}=1}^{\mathrm{n}}{\mid r_{k,\mathit{ref}}^{(i)}\mid }^2$ in (3)/(4), we performed 5-fold cross validation on 300 simulated data for 50 times. 4/5 of the data was trained by MCCAR+jICA to be decomposed into A_train and S, where S is further used in the left out 1/5 testing data to decompose it into A_test and S. Then we tested the correlation between the reference and the target component of A_test (with the same IC order of the target component derived from A_train) for 5×50=250 times on each modality. As shown in Figure 6, the mean and standard derivation of correlations of all iterations for 3 modalities were calculated. When λ= 0.8, the mean correlation between estimated target IC and the reference reaches its maximum value, thus we set λ=0.8 in simulation under PSNR=7. For real human MRI data, we adopted the same strategy to determine the value of λ.

Fig. 6.

Correlation of the identified components and reference signal across multiple cross-validations under PSNR=7. When λ is 0.8, the mean correlation between estimated target IC and the ground truth of all modalities and tests reaches its maximum value. The black line, yellow patch and blue line represent mean, standard error of the mean (SEM) and the standard deviation (SD) of correlations between target IC and reference.

IV. Results of Human Brain Data

A. FBIRN Results

In human data application, we first applied the proposed method on FBIRN Phase III multimodal data consisting of 294 subjects (147 HC, 147 SZ), who were measured by CMINDS [29] with a working memory score. Three representative MRI features, i.e., fALFF from resting-state functional MRI, GM volume from sMRI, and FA from dMRI were extracted and combined by MCCAR+jICA, in which the working memory domain score was set as the reference. Similar cross-validation method (as shown in Figure 6) was used to determine λ in FBIRN data, and we finally set λ = 0.8. We aim to discover the joint ICs that are not only significantly correlated with CMINDS working memory scores, but also joint group-discriminative. Here the joint IC means components of the same index across modalities. 20 components were estimated for each feature according to an improved MDL criterion [9]. We then performed two-sample t-tests on mixing coefficients of each IC for each modality.

Among the 20 derived ICs, the 6^th IC was found to be the component of interest. It is both correlated with working memory scores (r=0.296, 0.241, 0.301), and significantly group-discriminating (p=7.4×10⁻⁶, 1.0×10⁻³, 7.0×10⁻⁹ FDR corrected) for fMRI, dMRI and sMRI, respectively. Figure 7 displayed the spatial maps (transformed into Z scores, visualized at |Z| >2) and showing HC>SZ for all modalities on the mean of loading parameters, as the boxplot shown in Figure 7 (b), so that the positive Z-values (red regions) indicate higher contribution in HC than SZ and the negative Z-values (blue regions) indicate higher contribution in SZ than HC. The identified regions in IC6 are summarized in Supplementary Table V for fALFF and GM components as well as FA (white matter tracts, from John Hopkins Atlas). Figure 7 (c) indicates the positive correlation between loadings of IC6 and the working memory scores in three modalities (HC: red dots, SZ: blue dots); the higher loadings correspond to better working memory performance. Additionally, the identified IC6 is also anti-correlated with PANSS negative scores (r=−0.229, −0.276, −0.240) for fMRI, dMRI and sMRI, respectively. No significant correlation was found with PANSS positive scores.

Fig. 7.

Joint components that are significantly correlated with working memory domain score of CMINDS and also indicate significant group differences in all modalities. (a) The spatial maps visualized at |Z|>2; the positive Z-values (red regions) means HC>SZ and the negative Z-values (blue regions) means HC<SZ. (b) Correlations between loadings of component and the CMINDS working memory scores (HC: red dots, SZ: blue dots); the higher loadings correspond to better working memory performance. (c) Boxplot of the loading parameters for each component, with the p values of two sample t-test between HC and SZ. In (c), the black line, gray patch and yellow patch represent mean, SD and SEM of the group loadings.

Regarding the identified brain regions, both fMRI_IC6 and sMRI_IC6 identified dorsolateral prefrontal cortex (DLPFC), where patients indicated higher fALFF values but lower GM volume. There are very consistent evidences that SZ have difficulty with processes attributed to the central executive component of working memory. Two meta-analyses on working memory related brain activation in schizophrenia provide consistent evidence for altered activity in DLPFC [15, 18, 32], indicating that the link between DLPFC dysfunction and disrupted working memory is a prominent feature of leading cognitive neuroscience models of schizophrenia. Several lines of evidence also suggest working memory and the DLPFC component in particular, as a critical domain of dysfunction in the pathophysiology of schizophrenia. In sMRI studies, the DLPFC is a key cortical region in which gray matter is reduced in volume in schizophrenia, changes that are correlated with negative symptom severity in patients. Overall, our results suggest that higher fALFF but lower GM volume in DLPFC relates to worse working memory in schizophrenia [33]. Commonly in fMRI_IC6 and sMRI_IC6, SZ also showed lower values in thalamus, which is believed to be the mediator of attention under the notable contextual and leading influence of the neocortex [34]. Thalamus has dense reciprocal projections with cerebral neocortex and with limbic structures, and forms a key part of the pathway for transmission of sensory information to cortex [35]. The thalamus and the basal ganglia are key structures linked to the prefrontal cortex and are known to be involved in working memory [36] and it also participate in the thalamic–cortical–striatal circuitry subserving working memory [37]. An alternative account of the present findings, which emphasizes the fractional similarity network analysis results, posits the DLPFC/anterior cingulate cortex (ACC)/thalamus triad as a core deficit, with the dysfunction elsewhere in the network as a downstream functional consequence of working memory disturbance [15].

For fMRI only, patients indicated higher fALFF values in superior frontal gyrus, medial frontal gyrus, superior temporal gyrus (STG), and inferior parietal lobule. Both fronto-temporal and front-parietal circuits’ abnormalities were observed in fALFF. Fronto-temporal dysconnectivity has been proposed as a mechanism leading to the psychotic symptoms, especially auditory hallucinations, in schizophrenia. Disrupted fronto-parietal circuit may account for the impaired executive function and cognitive control in schizophrenia, especially the working memory deficit [38].

For GM only, patients showed lower GM volume in ACC, insular, which consist of the salience network [39], as well as subcortical areas including caudate, amygdala, hippocampus. Insular is a cortical structure with extensive connections to many areas of the cortex and limbic system. It integrates external sensory input with the limbic system and is integral to the awareness of the body’s state (interoception) [40]. Recent studies report that the dorsal anterior insular and dACC, core regions of the salience network, play a critical role in mediating the interaction between emotion perception and executive control involving in emotional working memory processing [41]. Although the DLPFC lesion is reported consistently in patients with schizophrenia relative to healthy subjects, abnormal activation patterns are not restricted to this region. Two previous meta-analysis about working memory deficits in schizophrenia has pointed out that apart from DLPFC dysfunction, the ACC and bilateral insular (main parts of the salience network) also participate in working memory performance in SZ [18, 42]. We successfully replicated the previous studies that a relatively greater degree of reduction in prefrontal and salience network in schizophrenia has been extracted.

For dMRI, the co-occurring FA values in superior longitudinal fasciculus (SLF), inferior longitudinal fasciculus (ILF), forceps minor (FMIN) and forceps major (FMAJ) were lower in SZ. The FMIN linking DLPFC region is traversed by tracts interconnecting the frontal lobe, providing evidence for disrupted anatomical connections in the fronto-limbic circuitry [43]. FA changes in the SLF, the major white matter connection between prefrontal and parietal cortices, relate to verbal working memory performance [17]. As reported, the integrity of this physiological connection predicted performance on a verbal working memory task, indicating that this structural change may have important functional implications. According to the macrocircuit theory, specific white matter tracts are disrupted either as a cause or a consequence of a disorder in the gray matter regions they connect. Consistent with previous reports, the current study also found moderate correlations between the severity of negative symptom and diffusion measurements of certain tracts, including FMAJ, FMIN, SLF, and ILF [43].

B. Independent Cohort Validation

Due to the interferential effects of varying measurement conditions and demographic distributions, few studies have been strictly validated for independent cohorts. Here we further tested the stability of our proposed method using another independent validation cohort, data from UNM, including 39 SZs and 44 HCs. No overlapped subjects between UMN and FBIRN. A similar working memory domain score from a cognition measurement system: MCCB, was used as reference. 24 components were estimated for each feature according to an improved MDL criterion [9]. Among the 24 derived ICs, IC1 was found to be both correlated with MCCB working memory scores (r=0.284, 0.3, 0.224) and significantly group-discriminating (p=4.0×10⁻⁴, 1.7×10⁻², 4.8×10⁻²) for all 3 modalities, as displayed in Figure 8. The identified regions in IC1 are summarized in Supplementary Table VI for fALFF, FA, and GM, respectively.

Fig. 8.

Joint components that are significantly correlated with CMINDS working memory scores and group-discriminating in all modalities. (a) The spatial maps visualized at |Z|>2. (b) Correlations between loadings of component and the CMINDS working memory scores (HC: red dots, SZ: blue dots). (c) Boxplot of the loading parameters for each component, with the p values of two sample t-test between HC and SZ. In (c), the black line, gray patch and yellow patch represent mean, SD and SEM of mixing coefficient for each group respectively.

Figure 9 indicates the comparison of working memory (WM) related components obtained from 2 independent cohorts. Note that the working memory performance was calculated from two systems, though they are not completely identical in WM tasks, we still obtained very similar brain patterns related to WM performance between two independent data sets. Particularly, sMRI and DTI get the highest overlapped brain regions for both FBIRN and UNM. For sMRI, decreased GM values in salience network (including dorsal ACC and insular) and executive control network (including DLPFC and STG), and subcortical areas (including caudate, putamen, thalamus) were detected for both cohorts. For DTI, the major tract including the SLF, inferior fronto-occipital fasciculus [44], ILF, FMAJ [45] and FMIN, ATR, which are summarized by a recent meta-analysis [46], were almost the same for both cohorts. While for fMRI, the prefrontal areas and posterior DMN are the most consistent brain areas for both FBIRN and UNM. Previous studies showed that the DMN was generally divided into two subsystems: anterior part and posterior part (pDMN). The former is mostly related to self-referential mental thought and the latter engages in episodic memory retrieval [47]. In our results, we recognized pDMN (including posterior cingulate cortex (PCC), superior frontal gyrus, middle frontal gyrus, posterior inferior parietal lobule and middle temporal gyrus [48]) in fMRI, which showed higher fALFF values in SZ. Previous task-related neuroimaging studies have indicated that the PCC increases activation in successful retrieval of episodic memory [49]. Overall, all the above mentioned brain networks (the salience network, pDMN, executive control network and prefrontal areas) were replicated in another independent cohort.

Fig. 9.

Independent cohort validation of the identified joint target component. The working memory domain scores measured by CMINDS and MCCB systems were used as references for MCCAR+jICA for FBIRN and UNM data respectively. The spatial maps were visualized at |Z|>2; the positive Z-values (red regions) means HC>SZ and the negative Z-values (blue regions) means HC<SZ.

Furthermore, to verify the effectiveness of the identified regions, we also searched the meta-analysis results from Neurosynth (http://neurosynth.org/) using working memory as search terms. According to the inference map we obtained, the bilateral anterior insular, middle frontal and inferior frontal areas are the brain regions most associated with working memory performance, while we successfully extract then in two sites on both GM and fALFF. Note in Neurosynth, if a study used working memory as the searched keywords, we obtained statistical inference maps displaying z-scores according to the probability of a region’s being activated. In other words, these maps displayed the brain regions that were consistently activated in studies that are loaded highly on a specific term. These maps were threshold at a false discovery rate (FDR) of 0.01. We also noticed that the subcortical regions in fMRI are different between two sites, this may be due to the discrepancy of the working memory tasks included in two cognitive measurement systems.

In sum, we successfully replicated the modality-covarying network particularly related to WM deficits in schizophrenia. The highly consistent functional-anatomical-structural co-alterations between two sites verified the effectiveness of the proposed fusion with reference model. More importantly, the identified executive control network, salience network, and the posterior default mode network have previously been suggested to be associated with WM deficits in multiple reports [16, 18, 42, 50] but never in a multimodal analysis across multiple cohorts in the same study, which suggests a great potential for the use of the proposed method.

V. Discussion and Conclusion

In this study, we proposed a novel supervised fusion with reference model, MCCAR+jICA, which is able to extract interested components associated with a specific prior reference, e.g., cognitive scores. Compared to the blind (unsupervised) fusion approaches, e.g. MCCA, jICA, etc. as summarized in Sui [5, 11, 12], the proposed model is more goal-directed by taking advantage of a priori to guide the fusion, which is also shown to be more stable in various noise levels and even when the estimation of component number is not accurate.

In simulation, we compared MCCAR+jICA with its alternatives on the performance of getting right target joint components. Results indicate that MCCAR+jICA is able to extract the particular component of interest with improved accuracy on both mixing coefficients and source maps. Moreover, source separation performance of all other components is also kept as the higher level. In the real-world fusion application, we combined data from brain function, white matter tracts and gray matter volume from schizophrenia and healthy controls. A joint component was identified that are not only correlated with working memory domain scores from CMINDS, but also show significant group differences between schizophrenia and controls. Specifically, we successfully extracted robust modality-common covarying networks including the executive control network and prefrontal areas (both detected in fALFF and GM), and the modality-specific networks including pDMN (only detected in fALFF) and the salience network (only detected in GM). Most importantly, the identified working memory related, brain functional and structural networks can be replicated in another independent cohort and both were impaired in SZ. This means for brain imaging applications, we can simultaneously explore multimodal inter-subject covariation linked with a specific clinical measure.

In addition to working memory dysfunction, other cognitive domains could also be studied using our method, e.g. composite cognitive scores, one of the most widely reported cognitive deficit in schizophrenia [51, 52], which would appear in our another work. Furthermore, MCCAR+jICA can be applied straightforwardly to study other brain diseases (such as psychotic major depression disorder (MDD), or non-psychotic bipolar disorder (BP)). In addition, the choice of cognitive measure is flexible. Other references that could be used include symptom severity scores, age, or behavioral measures. Finally, apart from the above mentioned cognitive measures, genetic data could also be used as reference, e.g. a microRNA, to explore brain structure and function associated with genetic variants, presenting a new direction of future fusion work on mental disorders [14].

A possible limitation of this work is that MCCAR+jICA operates on extracted features, rather than the original imaging data, (e.g. using fALFF instead of 4DfMRI data). Although some of the temporal information is lost using this method, a “feature” tends to be more tractable than working with the large-dimensional original data [53] and provides a simpler space in which to link the data [54]. While in future work, we can include temporal features, like dynamic states, or functional network connectivity matrices [55] as fusion input for fMRI, to capture both temporal and spatial co-alterations. Furthermore, most participants were receiving antipsychotic and/or mood stabilizing medication at the time of scanning (medication information can be found in Supplementary Table VII), which may result in potential structural and functional brain alterations [56].

In summary, this study proposed a novel supervised fusion model MCCAR+jICA, and provided proof-of-concept of its application in brain imaging data. To our knowledge, this is the first attempt to use cognitive score as a reference to guide multimodal imaging data fusion for seeking potential multimodal neuromarkers of working memory deficits in schizophrenia. Based on the proposed model, we not only identified co-varying brain regions that were suggested linked with core schizophrenic deficits of working memory in multiple reports, but also verified the results in a repeatable manner in an independent cohort, promising a widely use of the proposed method in detection of potential biomarkers for mental disorders.