Unraveling schizophrenia replicable functional connectivity disruption patterns across sites

Abstract

Functional connectivity (FC) disruption is a remarkable characteristic of schizophrenia. However, heterogeneous patterns reported across sites severely hindered its clinical generalization. Based on qualified nodal‐based FC of 340 schizophrenia patients (SZ) and 348 normal controls (NC) acquired from seven different scanners, this study compared four commonly used site‐effect correction methods in removing the site‐related heterogeneities, and then tried to cluster the abnormal FCs into several replicable and independent disrupted subnets across sites, related them to clinical symptoms, and evaluated their potentials in schizophrenia classification. Among the four site‐related heterogeneity correction methods, ComBat harmonization (F1 score: 0.806 ± 0.145) achieved the overall best balance between sensitivity and false discovery rate in unraveling the aberrant FCs of schizophrenia in the local and public data sets. Hierarchical clustering analysis identified three replicable FC disruption subnets across the local and public data sets: hypo‐connectivity within sensory areas (Net1), hypo‐connectivity within thalamus, striatum, and ventral attention network (Net2), and hyper‐connectivity between thalamus and sensory processing system (Net3). Notably, the derived composite FC within Net1 was negatively correlated with hostility and disorientation in the public validation set (p < .05). Finally, the three subnet‐specific composite FCs (Best area under the receiver operating characteristic curve [AUC] = 0.728) can robustly and meaningfully discriminate the SZ from NC with comparable performance with the full identified FCs features (best AUC = 0.765) in the out‐of‐sample public data set (Z = −1.583, p = .114). In conclusion, ComBat harmonization was most robust in detecting aberrant connectivity for schizophrenia. Besides, the three subnet‐specific composite FC measures might be replicable neuroimaging markers for schizophrenia.

This study reported that ComBat harmonization was most robust in detecting aberrant connectivity while controlling for the false discoveries for schizophrenia. Besides, the identified three subnet‐specific composite functional connectivity measures might be considered replicable neuroimaging markers for schizophrenia.

1. INTRODUCTION

Schizophrenia is a debilitating neuropsychiatric disorder characterized by a series of positive and negative symptoms and cognitive impairment (Gustavsson et al., 2011; Whiteford et al., 2013). As a noninvasive neuroimaging modality, functional magnetic resonance imaging (fMRI) allows detecting the coupling of spontaneous neural activity between brain areas termed functional connectivity (FC). Many fMRI studies have corroborated that schizophrenia suffered severe FC disturbance across widely distributed brain areas (van den Heuvel et al., 2010; Y. M. Wang et al., 2020).

Nevertheless, the reported impaired FC patterns in schizophrenia were inconsistent across existing studies (Karbasforoushan & Woodward, 2012; Sheffield & Barch, 2016). Both hyper‐connectivity and hypo‐connectivity of the prefrontal cortex (Yoon et al., 2013; Zhou et al., 2015), default mode network (DMN) (F. M. Fan et al., 2013; Whitfield‐Gabrieli et al., 2009), striatum (A. Li et al., 2020; Quidé et al., 2013; Salvador et al., 2007), and thalamus (Ferri et al., 2018; Gong et al., 2019) were reported in schizophrenia. The reported FC heterogeneity may arise from biological sampling variability and systematic bias (Yamashita et al., 2019; Yu et al., 2018), for example, patients variability in illness course, disease status, and medication (Anhøj et al., 2018; T. Li et al., 2017; Sharma et al., 2018), and sites variability in acquisition scanners, parameters, and operators (Noble et al., 2017). The heterogeneous patterns reported across sites severely hindered the generalization of FC as a potential biomarker for schizophrenia diagnosis and treatment evaluation.

Recently, several strategies have been proposed to remove the systematic heterogeneities in neuroimaging studies across sites (Dewey et al., 2019; Garcia‐Dias et al., 2020; Huynh et al., 2019; Joo et al., 2021; S. Li et al., 2019; Pinto et al., 2020; Pomponio et al., 2020). Among these methods, meta‐analysis is a traditional and effective tool that systematically ensembles the summary statistics from several selected studies (or sites) to develop a single consensus statistical inference with greater power (Dennis et al., 2018; Schmaal et al., 2017; van Erp et al., 2018). In contrast, mixed‐effect mega‐analysis (ME‐Mega) ensembles multisite individual raw data into a single statistical model by considering sites as random nuisance covariates, demonstrating potentials in increasing power and reducing site heterogeneities (Favre et al., 2019; Radua et al., 2020). Moreover, ComBat harmonization (ComBat), a popular batch adjustment method originally developed for genomics data (Johnson et al., 2007), has recently been used in neuroimage studies to boost statistical power and replication by adjusting site effect (Radua et al., 2020; Yu et al., 2018; Zavaliangos‐Petropulu et al., 2019). To our knowledge, most of the previous studies focused on comparing performance between different correction methods in removing multisite biases of healthy humans (Fortin et al., 2017; Fortin et al., 2018). Therefore, it is interesting and crucial to know which correction method is most effective in eliminating the site‐related heterogeneity in detecting schizophrenia aberrant brain functional and structural damage. Although one pioneer study showed that ComBat could increase statistical power compared to other site‐related heterogeneity correction methods in one schizophrenia structural magnetic resonance imaging (MRI) study (Radua et al., 2020), it is unknown which methods would be more effective in characterizing the aberrant FCs for schizophrenia while controlling for site effects.

With the advances in machine learning (ML) techniques, substantial progress has been made in discovering connectivity biomarkers for schizophrenia individual diagnosis (Du et al., 2018; A. Li et al., 2020; Yoshihara et al., 2020). However, most prior FC‐based classification models were trained on small, single‐site participants, thus having poor generalization caused by heterogeneities between sites (Arbabshirani et al., 2013; Cao et al., 2020; Du et al., 2018; Gheiratmand et al., 2017; Jo et al., 2020; Rashid et al., 2016). Moreover, although a few studies have sought to detect the FC biomarkers by pooling data from multiple sites (Cai et al., 2020; Cheng et al., 2015), they did not try to remove the between‐site FC bias that may discount the classification performance. One of the exceptions is that a recent study tried to minimize the effects caused by sites and other nuisance variables using a data‐driven L1‐norm regularized sparse canonical correlation analysis (CCA; Yoshihara et al., 2020). Another challenge for schizophrenia diagnosis based on neuroimaging biomarkers is the balance between model complexity and interpretability: on the one hand, fewer intelligible neuroimaging biomarkers are preferred for psychiatrists' comprehension; on the other hand, most of the state‐of‐art ML models are “black boxes,” which contain hundreds to millions of parameters that severely hinder the model interpretability (Santana et al., 2020). To disentangle the “black boxes” issue of ML, a few methods have recently been proposed to help interpret the ML predictions while preserving the performance (S. M. Lundberg et al., 2020; Lundberg & Lee, 2017; Ribeiro et al., 2016; Štrumbelj & Kononenko, 2014; Xie et al., 2022). Early studies have indicated the aberrant connectivities of schizophrenia may be network‐specific, which usually involves the somatosensory network (SMN) (S. Li et al., 2019; Sharma et al., 2018; Yu et al., 2017; Zhang et al., 2019), DMN (F. M. Fan et al., 2013; Whitfield‐Gabrieli et al., 2009), striatum, and thalamus (Anticevic et al., 2014; D. Dong et al., 2018; Karcher et al., 2019; A. Li et al., 2020). Herein, we attempted to categorize the abnormal connectivities into several independent subnets using an unsupervised clustering; moreover, we tried to apply the subnet‐specific composite FC measures in ML to discriminate schizophrenia from the normal controls (NC). Simplifying a large amount of disrupted FC features into several independent composite measures may reconcile classification validity and interpretability for schizophrenia.

This study aimed to unravel the replicable and independent subnet‐specific FC measures for schizophrenia. We first compared the robustness of four commonly used site‐related heterogeneities correction methods on the local discovery set and public validation set separately, including random‐effects meta‐analysis (Meta_r), fixed‐effects meta‐analysis (Meta_f), ME‐Mega, and ComBat. Then, based on early reports of network‐specific involvement in schizophrenia (Giraldo‐Chica & Woodward, 2017; Karbasforoushan & Woodward, 2012; A. Li et al., 2020; Zhou et al., 2015), we tried to cluster these abnormal FCs into several replicable and independent subnets based on the statistics of the best correction model (ComBat) and related the subnet‐specific composite FCs to clinical symptoms. Finally, we hypothesized that ML using the subnet‐specific composite FC measures could discriminate the schizophrenia patients (SZ) from NC with comparable performance compared to that using the fully identified FC features.

2. MATERIALS AND METHODS

2.1. Participants and data sets

Six data sets with 369 SZ and 355 NC were enrolled in this study, including three locally obtained data sets and three publicly available data sets (Supplementary Table 1). The three local data sets collected from Tianjin Medical University General Hospital, including the Local1, Local2, and Local3, were used as discovery set to identify the abnormal FCs, to cluster these FCs into several independent subnets, and to train the classification model. Their acquisition was approved by the Ethics Committee of Tianjin Medical University General Hospital, and written informed consent was obtained from each participant. Structured Clinical Interview for the Diagnostic and Statistical Manual of Mental Disorders IV was used to diagnose schizophrenia and confirm the absence of psychiatric illnesses in NC. The exclusion criteria included: (1) age lower than 18 years old; (2) left handedness; (3) any MRI contraindications; and (4) histories of neurological disorders, severe systemic illnesses, or substance abuse. In addition, the Positive and Negative Syndrome Scale (PANSS) was obtained for each subject. The publicly available data sets were obtained from SchizConnect (http://schizconnect.org/) (L. Wang et al., 2016), including the BrainGluSchi (Bustillo et al., 2017), COBRE (Aine et al., 2017), and NMorphCH (Alpert et al., 2016), which were used as validation set to test the reproducibility of the identified aberrant FC subnets and their composite FC measures, to evaluate the connectivity‐symptom associations, and to test the classification performance. For the NMorphCH data set, we only comprised the baseline data, and it was further separated into NMorphCH1 and NMorphCH2 subsets due to different scanning parameters (Supplementary Table 2). The detailed information of the public data sets is available on the SchizConnect website (http://schizconnect.org/).

FMRI data were obtained on six distinct 3.0‐Tesla scanners using gradient‐echo echo‐planar imaging sequences. Besides, high‐resolution T1‐weighted structural MRI images were also included for spatial normalization. The detailed acquisition parameters are summarized in Supplementary Table 2. After a series of quality control steps, 340 SZ and 348 NC were finally included in the formal analyses (Supplementary Method).

2.2. Image preprocessing and FC calculation

The fMRI data were preprocessed using the following pipeline described in Supplementary Method: slice timing, realign, spatial normalization using segmentation + DARTEL method nuisance covariates regression, and band‐pass filtering (0.01–0.1).

Brainnetome atlas was used to parcellate the cerebral cortex and subcortical nuclei into 246 regions (L. Fan et al., 2016). GRETNA software version 2.0.0 (https://www.nitrc.org/projects/gretna/) was then employed to extract the mean blood oxygen level‐dependent time series of these cortical regions and to calculate a Fisher‐Z transformed Pearson correlation coefficient (also termed FC) between each pair of regions, obtaining 30,135 unique connectivity features (246 × 246 connectivity matrix).

2.3. Intergroup FC comparisons using different site‐related heterogeneity correction methods

A linear regression model was first applied to the FC for each site to remove the age and sex effects separately. We chose the site‐specific covariates regression because: (1) we assumed that the effects of age and sex varied across sites and (2) to make covariates controlling comparable across different correction methods (ComBat, ME‐Mega, Meta_f, and Meta_r).

Then, four site‐related heterogeneity correction methods (including ComBat, ME‐Mega, Meta_f, and Meta_r) were employed to remove site effects while comparing FC differences between the SZ and NC.

2.3.1. Raw statistics without correction

We referred to intergroup FC comparisons without adjusting for site effects as “Raw” statistics. A general linear model (GLM) was performed with each FC as the dependent variable and group (SZ or NC) as the independent variable. The model can be written as:

$$y_{\mathit{nv}}=a_v+X_n{\beta }_v+{\epsilon }_v$$ (1)

where $y_{\mathit{nv}}$ is the vth FC of the nth individuals from all sites, $a_v$ is the intercept estimate of the vth FC, $X_n$ is the group identifier (1 = SZ, 0 = NC) of the nth individuals, ${\beta }_v$ is the estimate of the coefficient (effects) of the vth FC, and $\epsilon$ is the error term.

2.3.2. ComBat and statistics

ComBat is a popular batch adjustment method originally developed for genomics data and has recently been applied to harmonize neuroimaging data across sites (Fortin et al., 2017; Fortin et al., 2018; Radua et al., 2020). ComBat uses Bayesian regression to discover and correct systematic differences among multivariate data collected from diverse batches (or sites in this study). This tool can simultaneously remove additive bias (mean difference) and multiplicative bias (variance distribution difference) among sites while preserving the biological variation of interest. The model is formulated as (Radua et al., 2020):

$$y_{\mathit{inv}}={\alpha }_v+X_{\mathit{\text{in}}}{\beta }_v+{\gamma }_{\mathit{iv}}+{\delta }_{\mathit{iv}}{\epsilon }_{\mathit{inv}}$$ (2)

where $y_{\mathit{inv}}$ is the vth FC of the nth individual in the ith site, $a_{\nu }$ is the overall mean of the vth FC, $X_{\mathit{\text{in}}}$ are the covariates of interest of the nth individual in the ith site (e.g., age and gender). Because we removed the site‐related age and gender effects in advance, $X_{\mathit{\text{in}}}$ is empty in this study. ${\beta }_v$ is the effect of the covariates of interest of the vth FC. Additionally, ${\gamma }_{\mathit{iv}}$ and ${\delta }_{\mathit{iv}}$ indicate the addition and multiplication effects of ith site for the vth FC.

As described in Johnson et al. (2007), the ComBat estimates ${\gamma }_{\mathit{iv}}$ and ${\delta }_{\mathit{iv}}$ of the site effect parameters using conditional posterior means. The final ComBat‐harmonized FC values are defined as:

$$y_{\mathit{inv}}^{\text{Combat}}=\frac{y_{\mathit{inv}}-a_v-X_{\mathit{\text{in}}}{\beta }_v-{\gamma }_{\mathit{iv}}}{{\delta }_{\mathit{iv}}}+a_v+X_{\mathit{\text{in}}}{\beta }_v$$ (3)

Finally, these ComBat‐harmonized FC data underwent group comparison using the same GLM model as described in Equation (1).

2.3.3. ME‐Mega statistics

ME‐Mega has been used in many recent studies to account for site effects in large‐scale studies (Radua et al., 2020). The idea of ME‐Mega is to estimate an overall effect by merging the raw individual participant data from all sites into one linear mixed model with the sites as a random effect to control the average effect of each site. In our study, the ME‐Mega controlled the inter‐site variation by considering the “sites” as additional random intercepts within the GLM and simultaneously estimated the intergroup differences in FC between the SZ and NC, which can be expressed as the following.

$$y_{\mathrm{inv}}={\alpha }_v+X_{\mathit{\text{in}}}{\beta }_v+{\gamma }_{\mathit{iv}}+{\epsilon }_v$$ (4)

in which $y_{\mathrm{inv}}$ represents the vth FC of the nth individuals from site $i$, ${\alpha }_v$ is the intercept estimate of vth FC, $X_{\mathit{\text{in}}}$ is the group identifier (1 = SZ, 0 = NC) of the nth individuals from site $i$, ${\beta }_v$ is the estimate of the coefficient (effects) of the group differences of vth FC, ${\gamma }_{\mathit{iv}}$ is the random intercepts for site $i$ of vth FC, and ${\epsilon }_v$ is the error term.

2.3.4. Fixed‐effects meta‐analyses (Meta_f)

We conducted the fixed‐effects meta‐analyses in two steps. In the first step, we estimated the intergroup difference between SZ and NC for each site using the GLM as described in Equation (1).

In the second step, the effects (${\beta }_i$) of each FC of all sites were then pooled into one meta‐effect ($\widehat{\beta }$) by weighing the within‐site variance of each site ($w_i$) (Radua & Mataix‐Cols, 2012), where the weights $w_i$ are calculated as the inverse of the within‐site variance ($v_i$) of ${\beta }_i$ (Borenstein et al., 2010):

$$w_i=\frac{1}{v_i}$$ (5)

Then, the meta‐effect ($\widehat{\beta }$) was pooled with Equation (6):

$$\widehat{\beta }=\frac{{\sum }_{i=1}^s{w}_i{\beta }_i}{{\sum }_{i=1}^s{w}_i}$$ (6)

And the standard error (SE) of the meta‐effect ($\widehat{\beta }$) is estimated as Equation (7):

$$\mathrm{SE}=\sqrt{\frac{1}{{\sum }_{i=1}^s{w}_i}}$$ (7)

Finally, a Z‐statistics was introduced to estimate the significance of the pooled effects by Equation (8):

$$Z=\frac{\widehat{\beta }}{\mathit{SE}}$$ (8)

2.3.5. Random‐effects meta‐analyses (Meta_r)

The steps of random‐effects meta‐analyses were similar to those of fixed‐effects ones, except that the weights ($w_i$) for the site i is determined by both within‐site variance ($v_i$) and between‐site variance (${\tau }^2$) with Equation (9) (Borenstein et al., 2010):

$$w_i=\frac{1}{v_i+{\tau }^2}$$ (9)

The pooled effect, SE, and statistical Z value for each comparison are estimated using the same algorithms described in Equations ((6), (7), (8)).

It should be noted that all these site‐effect correction methods and intergroup FC comparisons were carried out in the local discovery set and public validation set separately.

Moreover, because the GLM models return a t‐statistic for each comparison, while meta‐analyses return a z‐statistic, to make comparisons between different models feasible, we converted the t‐values for Raw, ComBat, and ME‐Mega into z‐statistics. This step was realized using MatLab's “tcdf” and “norminf” functions.

2.4. Performance evaluation of different site‐effect correction methods

We evaluated the sensitivity and false discovery rate (FDR) of each site‐effect correction method in detecting abnormal FCs (p = .05/30135, Bonferroni corrected) and compared the performance among different methods in the local discovery set and public validation set separately. Because there is no golden standard to determine which FC is “truly” disrupted in schizophrenia, we defined the “true” disrupted FC at four levels based on their repeatability among different site‐effect correction methods to make performance comparisons available. For the first level, the “true” disrupted FCs refer to those that survived at least one of the four site‐effect correction methods. Similarly, for the second, third, and fourth levels, the “true” disrupted FCs refer to those who survived at least two, three, and four site‐effect correction methods, respectively. As indicated, a higher “golden standard” level results in more reliable while more conservative statistics. It should be noted that although the present multi‐level defined “true” aberrant FCs sound more reliable than those only based on one preassigned correction method, the “true” aberrant FCs defined in the present study can only be understood as “replicable” across correction methods. However, there has still been no ideal solution to this issue. Ultimately, we calculated the FDR, sensitivity, and F1 score at four “golden standard” levels for each site‐effect correction method with the following equations:

$$\mathrm{FDR}=\frac{\mathrm{FP}}{\mathrm{TP}+\mathrm{FP}}$$ (10)

$$\text{Sensitivity}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$ (11)

$$\mathrm{F}1=2\times \frac{\left(1-\mathrm{FDR}\right).\text{Sensitivity}}{\left(1-\mathrm{FDR}\right)+\text{Sensitivity}}$$ (12)

where TP, FP, and FN denote true positive, false positive, and false negative, respectively. Here, the FDR refers to the ratio of false discovered abnormal connectivity to all discovered abnormal connectivity for a certain method. The sensitivity represents the discovered true abnormal connectivity relative to all true abnormal connectivity defined by the “golden standard.” The F1 score combines the precision (1‐FDR) and sensitivity and is an objective integrated measure for model performance.

2.5. Aberrant FC patterns clustering

We further explored whether the aberrant FCs of schizophrenia could be clustered into several independent subnets. A hierarchical clustering model was trained based on the edge‐wise z‐statistic matrix of intergroup differences estimated by the best statistical model (ComBat) in the local discovery set. Detailed steps were as follows: (1) On the original 246 × 246 z‐statistic matrix, we selected the top N most impacted ROIs by iteratively removing the minimally affected ROI with the smallest sum of absolute z‐statistic between this ROI and the remaining. This iterative strategy removed ROIs that contributed less to the connectivity disruption and preserved the top N most impacted ROIs. (2) Then, a hierarchical clustering method was applied on the z‐statistic matrix of the top N impacted ROIs (N × N) to separate them into independent clusters with similar FC disruption patterns within each cluster while different between them. Specifically, a similarity matrix was calculated based on the z‐statistic matrix using the Euclidean distance; then, a “farthest distance” method was used to iteratively link pairs of ROIs in closest proximity, forming progressively larger clusters in a hierarchical tree. (3) An automatic grid searching procedure was applied on the clustering model to determine both the optimal cluster numbers and ROI numbers (N) based on Calinski–Harabasz criterion. Specifically, we defined the grid with the ROI numbers with N from 10 to 100 (10 intervals) and cluster numbers from 2 to 10. We iteratively calculated the Calinski–Harabasz index for each parameter combination. Finally, the results suggested top 40 ROIs and 2 clusters were the best hyperparameters for hierarchical clustering (best Calinski–Harabasz score = 203.866) (Supplementary Figure 1).

2.6. Evaluation of aberrant FC patterns across data sets and sites

Hierarchical clustering separated the top 40 severely damaged brain regions into two clusters. Cluster1 mainly contained brain regions within the SMN and visual network (VN), and cluster2 comprised regions within the striatum, thalamus, and ventral attention network (VAN). The two clusters further separate the aberrant FCs into three independent subnets: Net1 was constituted by FCs between regions within cluster1. Net2 was constituted by FCs between regions within cluster2. Net3 was constituted by FCs between cluster1 and cluster2 regions.

We further explored whether these three abnormal FC patterns were replicable across data sets and sites. The subnet‐specific composite FC measure $\left({\mathrm{FC}}_{\mathrm{Net}}\right)$ was calculated for each subject in the local discovery set and public validation sets, respectively, by the following equation:

$${\mathrm{FC}}_{\mathrm{Net}}=\frac{{\sum }_1^k{\mathrm{FC}}_i\times \mathrm{abs}\left(z_i\right)}{k}$$ (13)

in which k represents the number of links in this subnet, ${\mathrm{FC}}_i$ represents the ith connectivity value after ComBat harmonization $\left(i\in \left[1k\right]\right)$, and $z_i$ represents the z‐statistic of intergroup differences revealed by the ComBat statistical model in the local discovery set.

Finally, a GLM was used to test the intergroup differences in subnet‐specific composite FC measures in the local discovery and public validation set, respectively (p < .05, Bonferroni correction).

In addition, to test the validity of clustering, we compared the statistical power of subnet‐specific composite FC with and without clustering. The composite FC without clustering was calculated based on two strategies: (1) we only focused on the top 40 most involved brain regions used for hierarchical clustering, and the composite FC of the ComBat harmonized 40 × 40 FC matrix was calculated by weighting the absolute z‐scores obtained from local discovery set and (2) we combined all the ComBat harmonized 246×246 FC matrix into one composite FC by weighting the absolute z‐scores obtained from local discovery set. And GLM was used to test the intergroup differences in composite FC without clustering (p < .05, Bonferroni correction).

Finally, to test if the ComBat‐corrected FCs can improve the statistical power compared with the uncorrected ones, the subnet‐specific composite FC without site‐effect correction were also extracted and compared between groups using GLM (p < .05, Bonferroni correction).

2.7. Association between the subnet‐specific FCs and clinical symptoms

A CCA was introduced to test the association between the subnet‐specific composite FC measures and the 30 symptom domains of PANSS (both with and without ComBat adjustment) (Wilk's lambda test, p < .05). Furthermore, the relationships between the subnet‐specific composite FC measures and illness duration and chlorpromazine (CPZ) equivalent dose (Gardner et al., 2010) (mg eq/day) were assessed by Spearman correlation (p < .05). Because subnet‐specific composite FC measures were derived from the weights (absolute z‐statistics) of local discovery set, to avoid double dipping, we carried the FC‐symptom association in only the public validation set.

2.8. Schizophrenia classification

Six commonly used ML models were applied to evaluate the classification performances under two different feature selection conditions: the three subnet‐specific composite FC measures and all connectivity features. These models were carried out using Scikit‐learn version 0.20.1 (https://scikit-learn.org/stable/) from python (version 3.7.1 https://www.python.org/). The models were trained on the local discovery set. The hyperparameters of each model were automatically determined by a grid searching procedure, and a tenfold cross‐validation method was used to tune the optimized hyperparameters and construct the predictive model. After model training, we evaluated the prediction performance of each model based on different features in the public validation set. Finally, the area under the receiver operating characteristic curve (AUC) was calculated, and DeLong's test was used to compare the performance between two different feature selection conditions (p < .05).

3. RESULTS

3.1. Demographics and clinical characteristics

The demographic and clinical characteristics of the subjects are summarized in Supplementary Table 1. There were no significant differences in either sex or age between the SZ and NC in each site (p > .05). In addition, there showed no statistical difference in mean FD between the SZ (0.18 ± 0.11) and NC (0.17 ± 0.10) (t = −1.02, p = .31), indicating head motion could not contribute to the potential FC differences between the two groups.

3.2. Robustness of different site‐effect correction methods in detecting schizophrenia aberrant FCs

Among the four site‐effect correction methods, ME‐Mega discovered the highest number of aberrant FCs (280), followed by Meta_f (271), ComBat (224), and Meta_r (128) when summing the results of the local and public data sets (p < .05/30135, Bonferroni correction). Raw statistics also identified 267 aberrant FCs. (Figure 1a, Supplementary Table 3).

FIGURE 1.

Performance of five statistical methods in detecting aberrant FCs of schizophrenia. (a) The Venn diagram depicts the overlapping pattern of identified significant aberrant FCs by different statistical methods in the local discovery set (left panel), public validation set (middle panel), and all data sets (right panel). The line plots demonstrate the FDR (b) and sensitivity (c) at four reference levels. (d) Depicts average F1 scores for each statistic method. Error bars indicate the standard deviation. FC, functional connectivity; FDR, false discovery rate; ME‐Mega, mixed‐effect mega‐analysis; Meta_f, fixed‐effects meta‐analyses; Meta_r, random‐effects meta‐analyses

For the local discovery set, we found that Meta_r achieved the overall best control for FDR (0.113 ± 0.121), followed by ComBat (0.191 ± 0.276), Meta_f (0.235 ± 0.286), and ME‐Mega (0.317 ± 0.282) (Figure 1b, left panel). ME‐Mega owed the overall best sensitivity among the four site‐effect correction methods (0.964 ± 0.036), followed by Meta_f (0.921 ± 0.095), ComBat (0.848 ± 0.134), and poorest for Meta_r (0.526 ± 0.277) (Figure 1c, left panel). When considering the balanced performance as revealed by F1 score, Meta_f owed the overall best performance (0.787 ± 0.201), followed by ComBat (0.776 ± 0.165), ME‐Mega (0.756 ± 0.222), and Meta_r (0.602 ± 0.136) (Figure 1d, left panel).

For the public validation sets, we found that Meta_r also achieved the overall best control for FDR (0.152 ± 0.157), followed by ComBat (0.167 ± 0.263), ME‐Mega (0.218 ± 0.250), and Meta_f (0.261 ± 0.267) (Figure 1b, middle panel). Meta_f owed the overall best sensitivity among the four site‐effect correction methods (0.967 ± 0.040), followed by ComBat (0.890 ± 0.116), ME‐Mega (0.878 ± 0.095), and Meta_r (0.600 ± 0.233) (Figure 1c, middle panel). Finally, ComBat had the overall best F1 score (0.819 ± 0.164), followed by Meta_f (0.801 ± 0.198), ME‐Mega (0.787 ± 0.150), and Meta_r (0.657 ± 0.079) (Figure 1d, middle panel).

When summarizing the results of the local and public sets, we found Meta_r also achieved the overall best control for FDR (0.125 ± 0.131), followed by ComBat (0.171 ± 0.252), Meta_f (0.237 ± 0.263), and ME‐Mega (0.272 ± 0.254) (Figure 1b, right panel). Meta_f owed the overall best sensitivity among the four site‐effect correction methods (0.939 ± 0.072), followed by ME‐Mega (0.925 ± 0.061), ComBat (0.868 ± 0.127), and Meta_r (0.586 ± 0.242) (Figure 1c, right panel). Finally, ComBat had the overall best F1 score (0.806 ± 0.145), followed by Meta_f (0.803 ± 0.179), ME‐Mega (0.776 ± 0.173), and Meta_r (0.656 ± 0.099). Uncorrected raw data had the lowest F1 score (0.627 ± 0.109) (Figure 1d, right panel).

In summary, among the four site‐effect correction methods, ComBat had the overall best performance in considering the balance between FDR and sensitivity. Thus, we chose the ComBat correction for further analysis.

3.3. Schizophrenia replicable aberrant FC patterns across data sets and sites

Among the top 40 most severely damaged brain regions, both hyper‐ and hypo‐FCs were identified in multiple systems in both the local discovery set (Figure 2a) and public validation set (Supplementary Figure 2), including regions in the VAN, VN, SMN, striatum, and thalamus (Supplementary Table 4). Hierarchical clustering recognized the top 40 severely damaged brain regions into two independent clusters. Cluster1 mainly contained brain regions within the SMN and VN, and cluster2 comprised nodes within the striatum, thalamus, and VAN. The two clusters further separate the aberrant FCs into three independent subnets: Net1 was constituted by FCs between regions within cluster1 and demonstrated hypo‐connectivities. Net2 was constituted by FCs between regions within cluster2 and also demonstrated hypo‐connectivities. Net3 was formed by FCs between cluster1 and cluster2 regions and generally showed hyper‐connectivities (Figure 2b, left panel). The three independent abnormal FC patterns can also be replicated in the public validation set (Figure 2b, right panel).

FIGURE 2.

Schizophrenia aberrant functional connectivity (FC) patterns across data sets and sites: (a) Map of top 40 impaired cerebral regions with abnormal hypo‐ (left) and hypo‐FCs (right) in schizophrenia in local discovery set. The color bar represents the sum of z‐statistics of aberrant FCs. (b) Hierarchical clustering of the top 40 involved cerebral regions in local discovery (left) and public validation (right) data sets. Heatmap represents the z‐statistic of FC differences between the schizophrenia patients (SZ) and normal controls (NC). (c) Subnet‐specific composite FC distributions for SZ (green) and NC (red) in local (left) and public (right) data sets. (d) Forest plots of effect size and 95% confidence interval of the three aberrant FC subnets in each site and their meta‐analysis statistics for local (left) and public data sets (right)

Moreover, GLM found abnormally lower composite FC for Net1 and Net2 (FC_Net1, FC_Net2), while abnormally higher composite FC for Net3 (FC_Net3) in both the local and public sets (p < .05, Bonferroni correction) (Figure 2c). We also tested if these subnet‐specific FCs (FC_Net1, FC_Net2, FC_Net3) were replicable in each site. Forest plots showed that all three subnet‐specific FCs could be stably replicated in each site of the local discovery set. Furthermore, in the public validation set, all sites had the same effect direction as the discovery statistic, and 5/12 statistics can pass the statistical significance, indicating relatively high repeatability in the public sites for these subnet‐specific FC measures (Figure 2d).

Furthermore, the composite FCs without clustering derived either from all 246 ROIs or top 40 ROIs demonstrated weak or no statistical differences between the SZ and NC in both the discovery and validation data sets (Supplementary Table 5), indicating that the subnet‐specific composite FCs derived from clustering is robust in representing the aberrant FC patterns of schizophrenia. Finally, for both the local discovery and public validation sets, the statistical significances (absolute T or p values) of the three subnet‐specific FC measures after ComBat were higher than those without harmonization, further supporting that ComBat harmonization can improve the statistical power (Supplementary Table 6).

3.4. Association of schizophrenia subnet‐specific FCs with clinical variables

In the public validation sets, there was a significant positive correlation between the first canonical variable of the subnet‐specific FC measures and that of ComBat‐corrected PANSS 30 items (r = .590, p = .048) (Figure 3a). Furthermore, the FC_Net1 contributed dominantly to the first canonical variable (Figure 3b). Besides, the positive symptom hostility and the general symptom disorientation mainly contributed to the first canonical variable of PANSS (Figure 3c). Post hoc Spearman correlation analysis further identified a negative correlation between FC_Net1 and the two PANSS symptom scores: hostility and disorientation (p < .05) (Figure 3d). There was no association between the three subnet‐specific FCs and CPZ equivalent dose and illness duration (p > .05) (Supplementary Figure 5).

FIGURE 3.

Correlation between schizophrenia subnet‐specific functional connectivity (FC) measures and clinical symptoms in public data sets. (a) The correlation between the first canonical variable of the subnet‐specific composite FCs and that of 30 PANSS items, and the canonical loadings for (b) the three subnet‐specific composite FC measures and (c) the 30 PANSS items. Colors represent the canonical loading coefficients. (d) Scatterplots of the Spearman correlation between the composite FC measure of Net1 and two representative symptoms. PANSS, Positive and Negative Syndrome Scale

It should be noted that 17 out of 30 PANSS items had significant inter‐site differences before harmonization (p < .05/30, Bonferroni correction, Supplementary Figure 3a), and these heterogeneities were diminished after ComBat (p > .05, Supplementary Figure 3b), suggesting harmonization of PANSS scores can also reduce inter‐site heterogeneities. Furthermore, the correlation between the composite FC measure of Net1 and ComBat‐adjusted symptoms was slightly stronger than the unadjusted PANSS items (Supplementary Figure 4).

3.5. Performance of subnet‐specific FCs in schizophrenia classification

The public validation set was used to test the out‐of‐sample performance of each predictive model. Among the six ML classifiers, XGBoost achieved the best classification performance using the three subnet‐specific composite FC measures as the features (AUC = 0.728), while the SVM classifier achieved the best performance based on the full‐connectivities features (AUC = 0.765). DeLong's test did not find any statistical differences between the two AUCs (Z = −1.583, p‐value = .114) (Figure 4b). In addition, there were no statistical differences in AUCs between the two different feature selection strategies in any ML classifier (p > .05), except that the AUC of the SVM using the three subnet‐specific FCs was slightly lower than that using the full‐connectivities features (Z = −2.536, p‐value = .011) (Figure 4a, Supplementary Table 7).

FIGURE 4.

Performance of different connectivity selection strategies for schizophrenia classification. (a) AUCs in the out‐of‐sample public data sets for six classifiers according to three subnet‐specific composite functional connectivity (FC) features (blue) and full connectivity features (orange). Asterisk represents statistically significant between the two feature selection strategies. (b) The receiver operating characteristic curves of two feature selection strategies with their best classifiers. AUC, area under curve; DT, decision tree; KNN, k‐nearest neighbor; LR, logistic regression; RF, random forest; SVM, support vector machine; XGBoost, eXtreme gradient boosting

4. DISCUSSION

This study depicted that ComBat was most robust in removing the measurement heterogeneity across sites in unraveling the aberrant FCs in schizophrenia. Besides, three replicable disrupted subnets were discovered in schizophrenia across different data sets and sites. Furthermore, FC within sensory areas was negatively correlated with the positive symptom “hostility” and general symptom “disorientation.” Finally, ML using three subnet‐specific composite FC measures could powerfully discriminate schizophrenia from NC with comparable performance to all identified FC features. Therefore, we speculated that the identified three subnet‐specific composite FC measures derived from replicable aberrant FC patterns could be considered generalizable neuroimaging biomarkers for schizophrenia.

Prior studies have demonstrated that data set heterogeneities between sites can introduce severe systematic bias in neuroimaging quantification (Chen et al., 2014; Clark et al., 2006; Fortin et al., 2017; Fortin et al., 2018; Garcia‐Dias et al., 2020). The popular solutions for inter‐site variability included meta‐analysis (Dennis et al., 2018; Schmaal et al., 2017; van Erp et al., 2018), ME‐Mega (Group et al., 2021; Haukvik et al., 2020; Koshiyama et al., 2020; Radua et al., 2020). In addition, with the excellent performance of ComBat in removing the batch effects on genomic data (Johnson et al., 2007), this method was recently applied to harmonize neuroimaging data and demonstrated great potential (Fortin et al., 2017; Fortin et al., 2018; Ingalhalikar et al., 2021; Yu et al., 2018). However, little attention has been paid to which site‐effect correction methods were more robust in discovering the aberrant FC of schizophrenia. The present study demonstrated that ComBat is more robust than ME‐Mega, fixed‐effect meta‐analysis, and random‐effect meta‐analysis in detecting aberrant FC for schizophrenia while controlling the false positive rate. Consistent with previous studies, ME‐Mega's higher FDR may be attributed to its assumption that the error terms follow the same distribution across sites, which is rarely the case as sites usually have different error variances (Radua et al., 2020). Besides, although random‐effect meta‐analysis had the lowest FDR, its sensitivity was lower than half of any other site‐effect correction methods, suggesting too conservative to unravel the aberrant connectivity in schizophrenia, supported by a recent comparative study on different image‐based meta‐analysis approaches (Maumet & Nichols, 2018). Finally, although the sensitivity of ComBat harmonization was relatively lower than the fixed‐effect meta‐analysis and ME‐Mega, it is the second best in controlling the false discoveries (only lower than the random‐effect meta‐analysis). In considering the balance between false discovery controlling and sensitivity, ComBat had the overall best performance among the four site‐effect correction methods.

Based on the unraveled stable aberrant FCs of schizophrenia revealed by ComBat, we further depicted three independent replicable aberrant FC patterns in schizophrenia across different sites based on hierarchical clustering. First, we found hyper‐connectivity between the thalamus and sensory processing areas (SMN and VN). Many studies stably replicated the hyper‐connectivity between the thalamus and SMN (Anticevic et al., 2014; Baran et al., 2019; Ferri et al., 2018; Gong et al., 2019; Woodward & Heckers, 2016). The hyper‐connectivity of the thalamus‐VN in schizophrenia could also be validated by early reports (Dong et al., 2019; Yamamoto et al., 2018). Second, the hypo‐connectivity within the sensory areas built up the second stable pattern for schizophrenia, which was also supported by recent studies (S. Li et al., 2019; Sharma et al., 2018; Yu et al., 2017; Zhang et al., 2019). We further observed that the hypo‐connectivity within sensory areas was negatively correlated with positive symptom hostility and general symptom disorientation, respectively. The correlation between FC of sensory areas and disorientation was in line with an early study that related the abnormal FC with working memory (Kang et al., 2011). The third remark pattern was the hypo‐connectivity among the thalamus, striatum, and VAN, which was validated by many prior studies (Anticevic et al., 2014; D. Dong et al., 2018; Karcher et al., 2019; A. Li et al., 2020). The striatum and thalamus are key regulation targets of the dopaminergic system, and dopaminergic hyperfunction of these regions has been widely acknowledged as the pathophysiology of schizophrenia (Weinstein et al., 2017). Striatum‐VAN dysconnectivity in schizophrenia was also indicated to be regulated by dopamine system dysfunction (Solé‐Padullés et al., 2016). Thus, we speculated that this thalamic‐striatal‐VAN dysconnectivity pattern might be considered an indicator of subcortical dopaminergic dysregulation.

Recent advances in ML techniques substantially promote the neuroimaging biomarkers for schizophrenia classification with reported accuracies from 51.65 to 96% (Arbabshirani et al., 2013; Cao et al., 2020; Du et al., 2018; Gheiratmand et al., 2017). However, model generalization and result interpretation are still two big obstacles for clinical practice. The first obstacle is mainly caused by heterogeneities across sites (Arbabshirani et al., 2013; Cao et al., 2020; Du et al., 2018; Gheiratmand et al., 2017; Jo et al., 2020; Rashid et al., 2016), while the second is mainly determined by model complexity (Santana et al., 2020). Based on the aberrant FC patterns present above, we argued that most FC features within the same subnets are redundant for schizophrenia classification. Consistently, we found that ML using only the three ensembled subnet‐specific FCs could achieve a powerful classification performance comparable to thousands of all identified FCs. In summary, the three replicable aberrant subnets portrayed the major connectivity dysfunction of schizophrenia, and their derived subnet‐specific FC measures could be considered generalizable and interpretable biomarkers for schizophrenia.

One of the major limitations of this study is that most SZ patients enrolled in the present study were chronic and underwent medication. Although we did not find any association between the aberrant FCs and disease duration and CPZ equivalent dose, it remains possible that the cumulative medication effects or disease course contribute to the aberrant FC patterns, which should be validated in first‐episode, drug‐naïve patients. Second, there is no golden standard to determine which FC is “truly” disrupted in schizophrenia since then. Thus, although the present multi‐methods defined “true” aberrant FCs sound more reliable than those only based on one preassigned correction method, the “true” aberrant FCs should be verified in the future. Finally, although site‐effect correction methods such as ComBat could remove systematic bias (such as rater and scanner effects), but might also remove important biological information (such as disease duration and symptoms). Thus, the balance between removing systematic bias and preserving biological information should be considered with caution.

5. CONCLUSIONS

In summary, this study suggests that ComBat harmonization was most robust in detecting aberrant connectivity while controlling for the false discoveries for schizophrenia. Besides, the identified three subnet‐specific composite FC measures might be considered replicable neuroimaging markers for schizophrenia.

CONFLICT OF INTEREST

The authors declare no conflict of interest.

Supporting information

SUPPLEMENTARY FIGURE 1 Calinski‐Harabasz criterion values for hierarchical clustering under different parameter combinations for local discovery data set.

SUPPLEMENTARY FIGURE 2. Map of top 40 involved cerebral regions nodes with abnormal hyper‐ and hypo‐FCs in schizophrenia of public data sets. Color bar represents the sum of z‐scores of aberrant FCs.

SUPPLEMENTARY FIGURE 3. Inter‐site differences in PANSS item scores before and after Combat. A one‐way analysis of variance was used to test the inter‐site differences. Asterisk represents statistical significance (p < 0.05/30, Bonferroni correction).

SUPPLEMENTARY FIGURE 4. Correlation between the composite FC measure of Net1 and PANSS symptoms before and after harmonization. (a) Correlation using Combat‐adjusted PANSS scores and (b) using un‐adjusted PANSS scores.

SUPPLEMENTARY FIGURE 5. Association of the subnet‐specific FCs with chlorpromazine equivalent dose and illness duration.

SUPPLEMENTARY TABLE 1: Demographic information of recruited data sets.

SUPPLEMENTARY TABLE 2: Scanning information for fMRI data.

SUPPLEMENTARY TABLE 3: Summary of identified aberrant functional connectivity for each statistical method.

SUPPLEMENTARY TABLE 4: Annotation for the top 40 schizophrenia‐involved brain nodes with the most abnormal connectivity features.

SUPPLEMENTARY TABLE 5: Comparison of the subnet‐specific composite FC with and without clustering

SUPPLEMENTARY TABLE 6: Comparison of the subnet‐specific composite FC before and after Combat.

SUPPLEMENTARY TABLE 7: AUC statistical test of six classifiers with two feature selection strategies in public data sets.

Du, X. , Wei, X. , Ding, H. , Yu, Y. , Xie, Y. , Ji, Y. , Zhang, Y. , Chai, C. , Liang, M. , Li, J. , Zhuo, C. , Yu, C. , & Qin, W. (2023). Unraveling schizophrenia replicable functional connectivity disruption patterns across sites. Human Brain Mapping, 44(1), 156–169. 10.1002/hbm.26108

DATA AVAILABILITY STATEMENT

This study involved two large data sets (Tianjin Dataset and SchizConnect Dataset). The SchizConnect Dataset is publicly available at http://schizconnect.org/. The Tianjin Dataset supporting the findings of this study are available from the corresponding author upon reasonable request. All codes included in this study are also available upon reasonable request by contacting the corresponding author.