Linked 4-Way Multimodal Brain Differences in Schizophrenia in a Large Chinese Han Population

Abstract

Multimodal fusion has been regarded as a promising tool to discover covarying patterns of multiple imaging types impaired in brain diseases, such as schizophrenia (SZ). In this article, we aim to investigate the covarying abnormalities underlying SZ in a large Chinese Han population (307 SZs, 298 healthy controls [HCs]). Four types of magnetic resonance imaging (MRI) features, including regional homogeneity (ReHo) from resting-state functional MRI, gray matter volume (GM) from structural MRI, fractional anisotropy (FA) from diffusion MRI, and functional network connectivity (FNC) resulted from group independent component analysis, were jointly analyzed by a data-driven multivariate fusion method. Results suggest that a widely distributed network disruption appears in SZ patients, with synchronous changes in both functional and structural regions, especially the basal ganglia network, salience network (SAN), and the frontoparietal network. Such a multimodal coalteration was also replicated in another independent Chinese sample (40 SZs, 66 HCs). Our results on auditory verbal hallucination (AVH) also provide evidence for the hypothesis that prefrontal hypoactivation and temporal hyperactivation in SZ may lead to failure of executive control and inhibition, which is relevant to AVH. In addition, impaired working memory performance was found associated with GM reduction and FA decrease in SZ in prefrontal and superior temporal area, in both discovery and replication datasets. In summary, by leveraging multiple imaging and clinical information into one framework to observe brain in multiple views, we can integrate multiple inferences about SZ from large-scale population and offer unique perspectives regarding the missing links between the brain function and structure that may not be achieved by separate unimodal analyses.

Introduction

Multimodal brain imaging techniques are playing increasingly important roles in elucidating structural and functional properties in normal and diseased brains, as well as providing the conceptual glue to bind together data from multiple types or levels of analysis. It’s clear that joint analysis of multimodal brain imaging measures is able to capture more views of brain aspects and covariations between modalities, but the merit of fusion methods has not been fully recognized by the community due to the difficulty of theory and data availability.^1–3 Most current studies analyze each modality independently,⁴ which may ignore the multimodal cross-information. Therefore, the complexity of the human brain coupled with incomplete measurements provided by existing imaging technology makes multimodal fusion an essential tool to mitigate against the misdirection,^5–7 which hopefully can provide a key to finding the missing link(s) in complex mental illness, such as schizophrenia (SZ).

We have previously successfully developed several multivariate fusion algorithms^8,9 and applied them to SZ^10–12 and bipolar disorder (BP).¹³ These approaches have also been adopted by others to study aberrant intermodality differences in major depression¹⁴ or obsessive-compulsive disorder.¹⁵ Similar methods were also used to investigate covarying multimodal brain patterns in Autism.^16,17 Given the fact that data-driven fusion model is able to complement and exploit the richness of multiple neuroimaging features together (for details, see review¹⁸), in this study, based on a large, pure Chinese Han population (307 SZ patients and 298 age and gender-matched healthy controls [HCs], single ethnicity), we aim to combine 4 types of magnetic resonance imaging (MRI) features together to investigate the linked functional–structural–anatomical alternations in SZ. Besides the basic imaging features used for fusion input such as morphometric measures in structural MRI (sMRI), or fractional anisotropy (FA) maps derived from diffusion MRI (dMRI), here, we uniquely include functional features such as functional network connectivity (FNC)¹⁹ and regional homogeneity (ReHo)²⁰ extracted from resting-state fMRI. By combining these 4 measures, we may elucidate the links between brain structural measures and temporal coherence of blood oxygen level dependent (BOLD) signal fluctuations or spontaneous neural activity.²¹

As a core symptom of psychotic disorders,²² auditory verbal hallucination (AVH) is experienced by 75% of patients diagnosed with SZ²³ and is generally acknowledged as the first class symptom of SZ. Although there is still not a consistent conclusion on AVH, both structural and functional brain disruptions involved in AVH in SZ are mainly located in the prefrontal cortex (PFC), including dorsolateral prefrontal cortex (DLPFC) and medial prefrontal cortex (mPFC), thalamus, sensorimotor cortex, and temporal lobe (especially Heschl’s gyrus).²⁴ Obviously, these brain regions are involved in language and speech monitoring processes, though the role of these regions in AVH is not still clear. Currently, 2 popular inferences generating AVHs are the bottom-up and top-down processes, which are opposed to each other. Based on the existed findings, Hugdahl²⁵ proposed a model that combines the 2 processes: hyperactivation of the bottom-up process (neural hyperexcitation in the temporal lobes produces the actual initiation of AVHs) and hypoactivation of the top-down process (impaired frontal lobes fail to inhibit the attentional focus on the voices) together produces AVHs. Therefore, psychotic syndrome ratings such as Auditory Hallucinations Rating Scale (AHRS),²⁶ the Positive and Negative Syndrome Scale (PANSS),²⁷ and cognitive task performance (digit span [DS] tasks to measure working memory ability) were also used to identify the imaging component of interests and to examine how they are impaired in the SZ. Finally, to demonstrate the reliability of our results, we also conducted a validation test in a separate Chinese sample (in total 106 subjects, including 40 SZs and 66 age and gender-matched HCs).

To the best of our knowledge, this is the first attempt to combine 4 diverse MRI features to unravel the coaltered brain patterns of multiple facets for SZ patients, in particular for a large Chinese Han population. By taking advantage of the 4-way MRI cross-information, we may not only reveal co-occurring functional–structural–anatomical brain features impaired in SZ but also provide a multiview of brain patterns specifically involved in hallucination or working memory impairment, which may deepen our understanding of the schizophrenic pathophysiology.

Methods

Participants

All subjects were within the 18–45 age range, right handed, who were screened for ethical clearance, with only Chinese Han people included in the study. As the discovery cohort, 605 Chinese Han participants (307 SZs and 298 age and gender-matched HCs) were recruited from 4 hospitals in China, with the same enrollment criteria, including Peking University Sixth Hospital (Site 1); Beijing Huilongguan Hospital (Site 2); Xijing Hospital (Site 3); and Henan Mental Hospital (Site 4). All of them were scanned by Siemens scanner with same protocols and compatible scanning quality. For validation dataset, 106 Chinese Han participants (40 SZs and 66 demographically matched HCs) scanned by a GE scanner were also recruited from the Site 4 from the same research study. Written informed consent was obtained from all study participants under protocols approved by the Institutional Review Boards at each site.

All HCs (298 in the discovery and 66 in validation dataset) were recruited by advertisement from the same geographical areas as patients, with no current or past axis I or II disorders (DSM-IV-TR) as screened by the SCID-nonpatient version. Additional exclusion criteria for all subjects included current or past neurological illness, substance abuse or dependence, pregnancy, and prior electroconvulsive therapy or head injury resulting in loss of consciousness.

All SZ patients were assessed using the Structured Clinical Interview for DSM disorders (SCID) and diagnosed by experienced psychiatrists according to the criteria of DSM-IV-TR for SZ. About 197 of 307 SZs in the discovery dataset and all 40 SZs in validation dataset received antipsychotic medication using multiple antipsychotic drugs before/during the scan. Table 1 lists the demographic and clinical information of all subjects in both cohorts (details for each site in the discovery experiment are listed in supplementary table S2).

Table 1.

Demographic and Clinical Information

Measure	The Discovery Cohort			The Validation Cohort
	Healthy Control (HC; n = 298)	Schizophrenia (SZ; n = 307)		Healthy Control (HC; n = 66)	Schizophrenia (SZ; n = 40)
	Mean (±SD)	Mean (±SD)	P-value	Mean (±SD)	Mean (±SD)	P-value
Age	27.77 (±6.74)	27.14 (±7.09)	.266	31.36 (±6.96)	28.61 (±7.93)	.065
Sex (M/F)	157/141	160/147	.889	32/34	25/15	.161
Medication (chlorpromazine equivalent)	NA	412.57 (±210.95)	NA	NA	336.25 (±160.52)	NA
First-episode	NA	145/307	NA	NA	14/40	NA
Positive and Negative Syndrome Scale (PANSS)
Positive	NA	23.91 (±4.20)	NA	NA	25.03 (±3.70)	NA
Negative	NA	19.44 (±6.01)	NA	NA	24.10 (±6.28)	NA
Total	NA	82.14 (±12.07)	NA	NA	89.87 (±11.37)	NA
Auditory Hallucinations Rating Scale (AHRS)	NA	24.09 (±7.33)	NA	NA	25.93 (±4.54)	NA
Digit span
Digit forward (DF)	8.78 (±1.38)	7.94 (±1.55)	9.34E−11	8.14 (±1.51)	6.51 (±1.57)	8.23E−07
Digit backward (DB)	6.52 (±1.90)	5.21 (±1.83)	7.19E−15	5.18 (±1.68)	3.97 (±1.44)	3.00E−04

Note: Chlorpromazine equivalent denotes standardized total dose of antipsychotic drugs.

Imaging Acquisition

All subjects were scanned by fMRI, dMRI, and sMRI, in which the discovery cohort data were collected on a 3.0 T Siemens Trio Tim Scanner (Siemens; Site 1, 2 & 3) or a 3.0 T Siemens Verio Scanner (Siemens; Site 4) with an 8-channel SENSE head coil, and the validation test data were collected on a 3.0 T Signa HDx GE Scanner (General Electric). To ensure equivalent and high-quality data acquisition, the scanning protocols for all the 4 sites were set up by the same experienced expert to achieve a harmonized imaging quality. The imaging parameters are listed in supplementary file S1.

Psychotic Syndrome Ratings

We used PANSS²⁷ scores to measure the disease severity of SZ on both positive and negative symptoms. In addition, for those SZ patients who have P3 value (subitem in PANSS measuring hallucination) higher than 3 (121 out of 307 SZs in the discovery cohort and 15 out of 40 SZs in validation cohort), we further performed a descriptive measurement of AVH on them using a 7-item AHRS.²⁶

Working Memory Performance

Digit span tasks are regarded as a mainstay in psychological assessment and are frequently used to study working memory in both healthy and diseased populations.²⁸ We used both digit forward (DF) and digit backward (DB) scores to identify component that is highly correlated with working memory performance. DB task is thought to be a more complex span task, which relies heavily on working memory processing,^28,29 because it requires information storage as well as concurrent processing essential to mentally reordering the information. All participants completed a DF span task followed by a DB span task (see supplementary file S1 for detailed descriptions).

Data Preprocessing

For fusion purposes, each modality is first reduced to a “feature” for each subject, providing a simpler, more tractable space to link the high-dimensional neuroimaging data.^30,31 In this study, for each subject, we extracted 4 imaging features, ie, segmented gray matter (GM) image from sMRI, FA map from dMRI, ReHo from resting-state fMRI, and a 50 × 50 FNC matrix that is computed based on correlation of time series of 50 selected components resulted from group independent components analysis (gICA) (supplementary figure S1). Note that although ReHo and FNC both are resulted from fMRI data, they capture differential functional perspectives. ReHo measures local functional homogeneity on a voxel-wise basis and reflects coordination in regional neural activity,³² while the FNC is defined as average connectivity among different functional intrinsic networks that even located remotely in the whole brain.^19,33 Rigorous quality control is performed for each feature; for more details, please see supplementary file on specific preprocessing steps. For the validation dataset, except for FNC, the other 3 features are similarly preprocessed as detailed in supplementary file. In the validation data, we used same steps to generate ReHo, FA, and GM maps. For FNC, we used the selected group ICs in discovery dataset as references to back-construct all corresponding ICs and time courses for each subject by using spatially constrained ICA³⁴ via GIG-ICA³⁵ (http://mialab.mrn.org/software/gift), which were then used to calculate the FNC matrix, please see more details in supplementary file.

Normalization and Site Effect Correction

After obtaining 4 types of extracted MRI features, the 3D brain images or 2D FNC matrices of each subject were reshaped into a 1D vector and stacked along subjects, forming a matrix with dimensions of no. subjects × [no. voxels or no. functional connectivity (FC)] for each of the 4 features. Then site label was used as covariate to be regressed out to minimize the site effect for each feature. Next, the 4 feature matrices were normalized to have the same average sum of squares (computed across all subjects and all voxels/locus for each feature), to ensure all features to have the same ranges and contribute equally in fusion. Note that the validation data were acquired from one site, so we do not need to regress out the site effect.

Fusion Analysis

The preprocessed multimodal features were jointly analyzed by multiset canonical correlation analysis plus joint independent component analysis (mCCA + jICA),^8,9 which had been successfully applied to many clinical cases.^1,9,15 The code for the mCCA + jICA algorithm has been released in the Fusion ICA Toolbox (FIT, http://mialab.mrn.org/software/fit), which is available for downloading and usage by the scientific community. The flowchart of the 4-way fusion is shown in supplementary figure S2, in which mCCA + jICA takes advantages of the 2 blind source separation (BSS) models: CCA and ICA. First, mCCA was employed to find the connections among features by maximizing correlations among loading parameters, and then, jICA was used to separate spatial maps precisely, all of which have been successfully applied to discriminate psychotic disorders.^10,36 In the discovery cohort, the component number was estimated using modified minimum description length³⁷ to be 15, 12, 16, and 12 for ReHo, FA, GM, and FNC, respectively. We thus chose an estimated component number of M = 16 for the subsequent analysis because we have found that a slight overestimation of the component number may improve decomposition accuracy in simulation.⁸ Consequently, a considerable amount of variance is retained when M = 16, ie, 98%, 99%, 98%, and 76% for ReHo, FA, GM, and FNC, respectively. Similarly, we chose M = 13 as estimated component number in the validation test, and 98%, 99%, 99%, and 76% variance was retained for ReHo, FA, GM, and FNC, respectively.

Post Hoc Statistical Analysis

After applying the mCCA + jICA framework to the human brain data, independent component (IC) $S_k$ and the mixing matrices $A_k$ for each feature (k = 1, 2, 3, 4) were generated, providing a variety of ways to analyze the intercorrelation between features as well as the group differences.⁸ In this study, we are most interested in the following:

• 1. Covarying functional and structural abnormalities: 2-sample t tests were performed on mixing coefficients (loadings) of each IC for each feature between SZs vs HCs for each cohort, resulting in the group-discriminative components. If the components of the same index show group differences in all features, they are called joint group-discriminative ICs.

• 2. Impact of clinical and cognitive measures: The mixing coefficients $A_k$ provide a way to identify components of interests that are specifically correlated with patient symptom severity (PANSS,²⁷ AHRS²⁶) and working memory performance (Digital Span). For DS scores that were recorded for all subjects, we further performed a partial correlation to regress out-group effect before calculating its correlation with IC loadings.

Results

Group Differences in Human Brain Data

Figure 1 indicates one joint IC (IC12) in the discovery cohort that shows significant group-different loadings, with P = 7.7e−15*, 1.0e−14*, 0.0019*, and 1.3e−12* for ReHo, GM, FA, and FNC, respectively (* means false discovery rate [FDR] corrected for multiple comparisons, the same below). In addition, the top 3 pairwise intermodality correlations existed in FA-GM: r = .27, P = 2.6e−11*; FA-ReHo: r = .25, P = 4.1e−10*; and ReHo-GM: r = .18, P = 5.6e−06*.

Fig. 1.

The identified joint group-discriminative components (IC12) in the discovery cohort. (A) Regional homogeneity (ReHo) component; (B) gray matter (GM) component; (C) fractional anisotropy (FA) component; The spatial maps (top) of ReHo, GM, and FA were visualized at |Z| >2.5, with the positive Z scores shown in red. The boxplot and loadings of each component are shown below with schizophrenia (SZ) in red and healthy control (HC) in blue. (D) Functional network connectivity (FNC) component: the FNC matrix (left) was transformed into Z scores and thresholded at |Z| >3 (middle, red nodes), which is displayed through the BrainNet Viewer toolbox (http://www.nitrc.org/projects/bnv/). The red key node size denotes FNC degree.

The identified group-discriminative brain regions in joint IC are listed in supplementary tables S3–S6. Particularly, ReHo and GM components were both impaired in SZ in hippocampus, anterior cingulate cortex (ACC), and fusiform gyrus. SZ patients indicated higher ReHo values in basal ganglia networks (BGNs) including caudate, putamen, and globus pallidus, as well as DLPFC (figure 1A). For GM, SZs had lower GM volume in the salience network (SAN, insular, and ACC), superior temporal gyrus (STG), and inferior parietal lobule (IPL) than HCs (figure 1B). Interestingly, the key nodes of FNC component (for more details see supplementary figures S1 and S3 and supplementary table S6) in figure 1D were highly overlapped with the abnormal regions shown in ReHo component (figure 1A), such as putamen, ACC, fusiform gyrus. For FA, SZ patients expressed lower integrity on white matter (WM) tracts including anterior thalamic radiation (ATR), forceps minor (FMIN)/forceps major (FMAJ), right cingulum, and right superior longitudinal fasciculus (SLF) (figure 1C).

A replicable functional and structural covariation was found in a joint IC (with P = 0.0071*, 0.0047*, 0.022, 1.4e−05* on 2-sample t tests between HC and SZ for ReHo, GM, FA, and FNC, respectively, as shown in figure 2) in the validation data, see anatomical details in supplementary tables S7–S10. Results showed SZ patients had higher ReHo values in BGNs, visual areas, and DLPFC, lower GM volume in the SAN (insular and ACC) and temporal lobe, and lower WM integrity in tracts of ATR, FMIN/FMAJ, right cingulum, and right SLF than HCs. In addition, the key nodes in the identified joint FNC component showed a spatial consistence between 2 cohorts on the basal ganglia-sensory motor network. More importantly, joint functional ICs (ReHo and FNC) indicated high coherence too, ie, the peak regions in ReHo component also serve as the key nodes in the FNC component in both cohorts. The voxel/node-wise spatial correlation of the identified joint group-discriminative ICs between discovery and validation cohorts are r = .36, .38, .24, and .5 for ReHo, GM, FA, and FNC, respectively, with P < 1.0e−16, P < 1.0e−17, P < 1.0e−12, and P < 1.0e−20. All the above suggest the relatively high replicability of the identified 4-way functional–structural coalteration in SZ.

Fig. 2.

The identified joint group-discriminative components (IC12) in the validation test. (A) Regional homogeneity (ReHo) component; (B) gray matter (GM) component; (C) fractional anisotropy (FA) component; The spatial maps (top) of ReHo, GM, and FA were visualized at |Z| > 2.5, with the positive Z scores shown in red. The boxplot and loadings of each component are shown below with schizophrenia (SZ) in red and healthy control (HC) in blue. (D) Functional network connectivity (FNC) component: the FNC matrix (left) was transformed into Z scores and thresholded at |Z| >3(middle, red, and black nodes), in which red nodes and red lines are overlapped with those key nodes and lines discriminated in the joint independent component (IC) in the discovery cohort.

Association With the AHRS and PANSS

Three components were identified to be correlated with AHRS scores (ReHo_IC8_D: r = −.21, P = .02; GM_IC13_D: r = −.25, P = .005*; FNC_IC1_D, r = −.18, P = .0045*), as shown in figure 3. Note that the spatial maps of ReHo, GM, and the key FNC nodes showed high consistence; the coaltered regions include PFC, thalamus, insula, and STG, which were typical regions involved in auditory hallucination.^24,38,39Figure 3C demonstrates overlay of the nodes and edges of the FNC_IC1_D upon the surface mapping of ReHo_IC8_D, in which red nodes were those FNC nodes retained by transforming FNC_IC1_D into z scores and thresholded at |Z| > 3. Red edges mean SZs have lower FC strength than HCs, while blue edges mean SZs have higher FC strength. Red mapping denotes SZs have lower ReHo values than HC, and blue mapping is otherwise. Namely, the lower ReHo, lower GM volume, and weaker FC in PFC correspond to more severe auditory hallucinations (higher AHRS scores) and vice versa in thalamus and STG. In the validation test, we did not find components significantly correlated with AHRS scores. This is probably due to only 15 out of 40 SZs with severe auditory hallucinations exist in validation cohort, thus hard to result in a significant correlation.

Fig. 3.

Spatial maps and scatter plot of the 3 components significantly correlated with Auditory Hallucinations Rating Scale (AHRS). (A) Regional homogeneity (ReHo) IC8; (B) gray matter (GM) IC13; the spatial maps (top) of ReHo and GM were visualized at |Z| >2.5, with the positive Z scores shown in red. (C) Functional network connectivity (FNC) IC1: the FNC component was also transformed into Z scores and visualized by BrainNet Viewer, in which red nodes are key nodes thresholded at |Z| >3 (the node size denotes FNC degree), and red lines means healthy controls (HCs) have higher functional connectivity (FC) strength than schizophrenia (SZ), while blue lines means HCs have lower FC strength than SZs. These edges and nodes of FNC_IC1 were mapped upon the surface mapping of ReHo IC8 in (C), indicating a high coherence between ReHo and FNC component associated with AHRS, while all 3 components demonstrate high spatial consistence.

Among the derived joint component (IC12), GM_IC12_D also indicated significant correlation with negative PANSS scores (GM: r = −.17, P = .0032*) as shown in supplementary figure S4. The lower GM volume in SAN,⁴⁰ STG, and hippocampus, the more severe negative symptom exhibited in SZ patients.

Association With Working Memory Performance

Correlation analysis with DS results also indicated that 2 components have positive correlations with DB scores (GM_IC13_D：r = .14, P = .00096*; FA_IC5_D: r = .15, P = .00062*; as shown in figures 4A and 4B), in which higher DB number represents better working memory performance. These correlations were still significant after regressing out the diagnosis (group label), ie, GM_IC13_D: r = .13, P = .004*; FA_IC5_D： r = .175, P = 5.9e−05*. Note that this GM component was the same IC anticorrelated with AHRS, involving DLPFC, mPFC, and thalamus, while the FA component covered WM tracts such as corticospinal tract (CST), SLF, and ATR that pass through the regions identified in GM_IC13_D. The higher WM integrity in these tracts corresponds to better working memory performance.

Fig. 4.

The identified components associated with working memory ability from 4-way fusion analysis within each cohort separately. All correlations are significant after controlling group effect as shown above. (A, B): gray matter (GM) and fractional anisotropy (FA) component from discovery cohort. (C, D): GM and FA components from validation cohort. Note that (A) and (C), (B) and (D) demonstrate a relatively high spatial consistence (overlap), with significant 3D spatial correlation r = .64 and r = .16 for GM and FA, respectively, P < 1.0e−10.

In the validation data, we also found 2 components correlated with DB scores (GM_IC2_V: r = .26, P = .0076*; FA_IC5_V: r = .20, P = .039; as shown in figures 4C and 4D). Similarly, these correlations were still significant after regressing out the diagnosis (group label), ie, GM_IC2_V: r = .24, P = .014; FA_IC5_V: r = .22, P = .04. Interestingly, spatial consistence was achieved between 2 cohorts on the identified brain regions associated with working memory ability. Namely, the voxel-wise spatial correlation between components in figures 4A vs 4C and figures 4B vs 4D are r = .64 (P < 1.0e−20) and r = .16 (P < 1.0e−10), respectively, implicating the functional and structural covariation in SZ was generally replicable.

Discussion

To the best of our knowledge, this is the first attempt to combine 4 aspects of brain features to characterize schizophrenic impairment by using a data-driven, multimodal fusion approach⁹ for a large pure Chinese Han population (307 SZs/298 HCs). We identified linked changes in functional–structural measures spanning multiple imaging modalities, which were further investigated by evaluating their associations with the severity of auditory hallucination and working memory performance.

Functional and Structural Covariation

Consistent with the hypothesis that SZ is a syndrome with a functionally and anatomically widespread brain deficits and disconnectivity,^41,42 we identified widely distributed brain networks impaired in SZ that covary in 2 or more of the 4 imaging features. Specifically, subcortical regions including putamen, caudate, and hippocampus indicated severe deficits and were involved in multiple imaging features. We found increased ReHo in the striatum in SZ, consistent with the meta-analysis reported in Xu et al,⁴³ coaltered with the lower FNC in SZ in the basal ganglia (putamen)-sensorimotor network,⁴⁴ as well as the lower WM integrity in cortical spinal tract that subserves motor control. This neuronal loop serves as the basis of various functions of the basal ganglia, including motor, cognitive control, motivation, and emotional processing,⁴⁵ of which dysfunction in SZ has been frequently reported.^46–48 Similar findings were reported in an adolescent-onset SZ study,⁴⁹ in which motor cortices and WM CST tracts were simultaneously detected. Furthermore, we also identified abnormal changes of SZ in the frontoparietal network (FPN, comprising DLPFC and IPL), co-occurring in both ReHo and FNC features, as well as lower FAs in the dorsal component of SLF,^50,51 which originates in the superior and medial parietal cortex, passes around the cingulate sulcus, and terminates in the dorsal and medial frontal lobe. Recent findings have demonstrated striatal dopamine receptors can mediate information transfer in FPN,⁵² which plays a central role in cognitive impairment in SZ.^53,54 All above multimodal findings shared from both cohorts (supplementary tables S3–S10) together provide indirect support for the inference⁵⁵ that disruptions in the basal ganglia-prefrontal network are significant in SZ, which may be associated with the corresponding dysfunction of SZ in cognitive control/working memory.^56,57 Moreover, our study further elucidates this by revealing significant pairwise intermodality correlations between the subject loading parameters (FA-ReHo: r = .25, P = 4.1e−10*; FA-GM: r = .27, P = 2.6e−11*; ReHo-GM: r = .18, P = 5.6e−06* in discovery cohort), which depicts a closely linked functional–anatomical system.

In addition, multimodal hippocampal abnormalities in SZ emerged with the increased hippocampal ReHo and decreased GM volume compared with HCs, which may be associated with impaired memory and emotional function in SZ,⁵⁸ consistent with the meta-analysis results.^43,59–61 Higher ReHo in fusiform and visual cortex (V2–V5) combined with lower FA in FMAJ and IFO tracts that passing through these regions were also identified in SZ.^62,63 Such hippocampal–prefrontal–occipital coupling in both brain function and structure suggests that they mutually contribute to a wide array of cognitive processes perturbed in SZ.⁶⁰

Regarding the identified cortical regions, the SAN⁶⁴ that is associated with cognitive deficit in many psychosis studies^65,66 was identified in 3 features in both cohorts, indicating lower GM volume,⁶⁷ higher ReHo in insular,^68–71 and hypoconnectivity between the PFC and the anterior insula in SZ, again supported by many unimodal and multimodal studies.^65,72 Moreover, the GM volume in the SAN (GM_IC12_D) is also negatively correlated with the negative PANSS score also linked with cognitive impairment (supplementary figure S4).

Covarying Regions Related to Working Memory Impairment

Digit span tasks have been widely used to measure the immediate verbal recall, attentional capacity, and working memory in neuropsychological research and clinical evaluations,⁷³ comprising 2 forms: DF and DB, in which DB requires information storage and concurrent processing essential to mentally reordering the information, thus heavily relying on working memory.^28,29,74 Several studies have found that DS tasks can activate the phonological loop (verbal memory) and involve the central executive.^73,75 Here, we further validated that the higher GM in DLPFC, mPFC, STG, IPL (BA 40), and thalamus, which are key parts of FPN,⁵² the better working memory performance of the participants.^76,77 On the other hand, a FA component including CST, SLF, and ATR has also been identified to be positively correlated with digit backward scores, which is still significant after controlling for group diagnosis. Interestingly, the identified large fiber tracts in the present study connect the frontal cortex to different subcortical and/or cortical regions, eg, ATR connects to the thalamus and SLF connects to the temporal and parietal lobe, suggesting that the GM deficits in the frontal lobe might be impacted by the relevant associative WM tracts damage in SZ. Note that all the identified tracts in the present study, with exception of the SLF, have a right-larger or right-lateralized damage⁷⁸ in SZ, consistent with recent studies on lateralization.^79–81 In the validation test, we also found 2 components correlated with DB scores (GM_IC2: r = .24, P = .014; FA_IC5: r = .22, P = .04; as shown in figures 4C and 4D). Interestingly, spatial consistence was achieved between 2 cohorts on the brain regions (spatial correlation, P < 1.0e−10) associated with working memory ability. The 3D spatial correlation between figures 4A vs 4C and figures 4B vs 4D is r = .64 and r = .16, respectively, with P < 1.0e−10 for both, implicating the functional and structural covariation in SZ was generally replicable.

Multimodal Covariation Related to Auditory Hallucinations

Many studies have reported both structural and functional brain disruptions involved in AVH in SZ separately,^24,82–87 including the PFC (including DLPFC and mPFC), thalamus, sensorimotor cortex, and temporal lobe (mainly located in Heschl’s gyrus). Our results integrate the above findings in a multivariate, data-driven manner, demonstrating a high spatial coherence of brain regions correlated with AVH in ReHo, GM, and FNC features. However, unlike the consistent reductions in GM, in which lower GM volume in the above-mentioned regions associate with more severe AVH symptoms (r = −.25, P = .005*), functional components reflected a reverse tendency between frontal and temporal lobe. Namely, lower ReHo and weaker FC strength (red edge) in prefrontal lobe are correlated with higher AHRS scores in SZ patients, whereas higher ReHo in STG/thalamus and the stronger FCs (blue edges) of SZ in thalamotemporal network correspond to more severe hallucinations (higher AHRS scores). In sum, figure 3 demonstrated a synchronous change in both functional and anatomical areas impacted by AVH, but it should also be noted that the loading of the ReHo_IC8_D is only moderately correlated with AHRS scores and did not pass FDR multiple comparison correction. Our results suggest PFC mediate inversely with STG/thalamus on generation of AVH, while all of them are disrupted in structure.⁸⁸ This is supported by the ERC “VOICE” model as proposed in Hugdahl et al^25,89 in which AVH is considered to be resulted from abnormality of 2 cognitive networks/systems: prefrontal hypoactivation that leads to a failure of executive control and inhibition and temporal lobe hyperactivation that contributes to the process of AVH. The identified regions constitute a thalamocortical perceptual circuit, in which the temporal cortex leads to the perceptual misrepresentations, and the parietal cortex shifts attention on the “voice,” following by the failure of executive inhibition and response suppression due to the attenuated FC within the prefrontal cortices, altogether contributing to the generation of hallucinations.^25,89 Note that no components were found to be correlated with AHRS scores in validation cohort. This is probably due to only 15 out of 40 SZs with severe auditory hallucinations are recorded in this dataset; thus, it’s hard to result in a significant correlation.

Limitation and Future Directions

A limitation to the current study is that information on antipsychotic or mood stabilizing medications for part of the patients were unavailable, which makes it difficult to assess the medication effect that may result in certain structural and functional changes,^90,91 though some of the patients were in their first episode. Another point worth noting is that we did not collect tobacco or alcohol use for all the participants during the rest fMRI session. Furthermore, with regard to the FNC feature, several brain regions in particular subcortical nuclei, eg, thalamus, commonly implicated in SZ were not shown in the selected ICs. To address this issue, either a seed-based approach or the atlas-based functional connectivities^92–94 could be a choice of features for fusion in future work.

In addition, from the perspective of methodology, several issues should be mentioned. First, in our article, 4 different features were extracted from 3 different modalities as the input of the 4-way mCCA + jICA fusion framework, of which the ReHo and FNC maps were filtered within the range of 0.01–0.08 Hz in the preprocessing step. However, real connectivity in the human brain may take place in the subsecond range. To address this issue, integrating MEG or EEG measurements may be an alternative strategy. Second, our analyses are focusing on the macroscale level, not linking with microscale analyses. In the featured research, the integration of translational research, postmortem analyses, or integration of myelin measurements are the promising trends and direction. Finally, non-neuronal activity, eg, heart rate variability (HRV),⁹⁵ microRNAs (miRNAs),⁹⁶ glucocorticoids,⁹⁷ and glial cells (including astrocyte, oligodendrocyte, and microglia)⁹⁸ may play an important role in several psychiatric diseases and is not easily monitored. Future methods based on supervised learning^96,99 that are able to incorporate such non-neuronal activity as references to guide imaging fusion may provide more insight into the underlying mechanism of psychiatric diseases.

More importantly, the strength of multimodal fusion is its ability to find or validate new discovery that may not be detected by separate unimodal analyses.^18,100 For example, some previous findings^24,82 suggested that decreased N-acetyl-aspartate (NAA)/choline ratios¹⁰¹ and increased activation^85,102 in thalamus may be associated with the severity of hallucinations, but no evidence support the inference that the thalamus is directly involved in AVH. By contrast, our data-driven analysis (figure 3) provided proof of concept that reduced GM volume, higher regional synchronous activity, and higher FC within thalamus in SZ were all correlated with higher AVHS (more severe hallucination), supporting that thalamus is actively involved in the regulation process of prefrontal hypoactivation and temporal hyperactivation on auditory hallucinations. This promises a new direction on future analysis of multisite, high-resource data in an integrated manner—capitalizing on the relative strengths of each modality and providing results/prediction synergistically.

In sum, we are just beginning to unlock the potential of multimodal imaging, which offers unprecedented opportunities to further deepen our understanding of brain disorders.⁴ To the best of our knowledge, this is the first attempt to comprehensively utilize 4 types of MRI features: ReHo, GM, FA, and FNC, in a joint analysis to investigate multiple impairments of SZ on a large, Chinese Han population. Our results concluded that a widely distributed network disruption is evident in SZ, especially involving the BGN and FPN, which may be associated with their corresponding dysfunction in working memory performance. Moreover, results demonstrate high spatial coherence among ReHo, GM, and FNC features related to auditory hallucination and further provide evidence for the “VOICE” hypothesis²⁵ that prefrontal hypoactivation and temporal hyperactivation in SZ may lead to failure of executive control and inhibition. By providing multiple views within one analysis integrating diverse imaging and clinical information, we may minimize incorrect conclusions about mental illness and offer unique perspectives on the links between the brain and mental disorders.¹⁰³