Morphometric Integrated Classification Index: A Multisite Model-Based, Interpretable, Shareable and Evolvable Biomarker for Schizophrenia

Abstract

Background and Hypothesis

Multisite massive schizophrenia neuroimaging data sharing is becoming critical in understanding the pathophysiological mechanism and making an objective diagnosis of schizophrenia; it remains challenging to obtain a generalizable and interpretable, shareable, and evolvable neuroimaging biomarker for schizophrenia diagnosis.

Study Design

A Morphometric Integrated Classification Index (MICI) was proposed as a potential biomarker for schizophrenia diagnosis based on structural magnetic resonance imaging data of 1270 subjects from 10 sites (588 schizophrenia patients and 682 normal controls). An optimal XGBoost classifier plus sample-weighted SHapley Additive explanation algorithms were used to construct the MICI measure.

Study Results

The MICI measure achieved comparable performance with the sample-weighted ensembling model and merged model based on raw data (Delong test, P > 0.82) while outperformed the single-site models (Delong test, P < 0.05) in either the independent-sample testing datasets from the 9 sites or the independent-site dataset (generalizable). Besides, when new sites were embedded in, the performance of this measure was gradually increasing (evolvable). Finally, MICI was strongly associated with the severity of schizophrenia brain structural abnormality, with the patients’ positive and negative symptoms, and with the brain expression profiles of schizophrenia risk genes (interpretable).

Conclusions

In summary, the proposed MICI biomarker may provide a simple and explainable way to support clinicians for objectively diagnosing schizophrenia. Finally, we developed an online model share platform to promote biomarker generalization and provide free individual prediction services (http://micc.tmu.edu.cn/mici/index.html).

Introduction

Schizophrenia is one type of severe chronic mental disorders that causes heavy economic and social burdens on patients, families, and the community.^1,2 Magnetic resonance imaging (MRI) has been frequently used to identify schizophrenia brain structural damage for its high spatial resolution and multi-dimensional measures. Early studies have reported extensive structural abnormalities in schizophrenia,^3–5 suggesting MRI-derived structural measures are potential biomarkers for the diagnosis, treatment evaluation, and prognosis prediction of schizophrenia.^6–8 In addition, recent studies combining MRI and machine learning (ML) methods have shown promising performance for schizophrenia classification.^9–11

Previous MRI-based studies have shown great diversities in either brain structural damages and classification performance for schizophrenia,^12–14 which lead to poor generalization for single-site studies.^15–17 Recent advances in neuroimaging and ML have shown that multisite big data can minimize the effects of heterogeneities across sites and thus increase prediction generalization.^18–21 However, multisite raw MRI data sharing is becoming a challenge by many factors.^17,22,23 For example, massive storage, networking, and computing resources are required, leading to huge human and resource costs for research and clinical institutions. Besides, raw MRI data contain sensitive, personally identifiable information; even some de-identification techniques are applied to them.^24–26 A candidate solution is sharing anonymous intermediate summary data rather than sensitive raw data. A successful case of intermediate neuroimaging data sharing is the Enhancing Neuro Imaging Genetics Through Meta-Analysis (ENIGMA) Consortium, which ensembles tens of thousands of neuroimaging data from sites globally by sharing their MRI statistical results for meta-analysis (http://enigma.ini.usc.edu/).²⁷ However, it is still challenging to find an effective way to ensemble multisite intermediate data to achieve a prediction performance comparable to or higher than those using the original MRI data.

Another challenge of using structural MRI data for schizophrenia diagnosis is the poor interpretability of the multivariable ML models.^28,29 Interpretability of biomarker is critically important for clinical practice, as it can increase understanding of the biological and clinical correlations between the biomarker and the disease, and provide insight into how a model can be improved.³⁰ With the rapid progress of machine learning and deep learning, many models may effectively discriminate schizophrenia patients from healthy controls^11,31; however, most of them are “black boxes” whose algorithms and parameters are too complicated for a psychiatrist to comprehend, which is especially true for deep learning models that are composed of millions of nonlinear hyper-parameters.³² To disentangle the “black boxes” issue, several methods have recently been proposed to help users interpret the ML predictions.^33–36 For example, a recent study constructed a neuroimaging biomarker “FSA score” based on the classification distance to the separating hyperplane of support vector machine classifier and showed convincing results on both biological meanings and classification for schizophrenia.³⁷ Recently, a SHAP (SHapley Additive exPlanations) framework had been proposed to assign each feature an additive importance value for a particular prediction in a single model.³⁶ However, the SHAP framework is designed to interpret a single-mode, and its expansion to multisite models remains to be explored. Besides, its biological and clinical relevance also needs to be further discovered.

This study tried to extend the SHAP framework into multisite ML models to construct a generalizable and explainable neuroimaging biomarker named Morphometric Integrated Classification Index (MICI) (see flowchart in figure 1). Based on structural MRI (sMRI) data of 1270 subjects (588 schizophrenia patients and 682 healthy controls) from 10 sites (Supplementary Tables S1, S2, Supplementary figure S1), our findings highlighted the generalizable, interpretable, evolvable, and shareable of the proposed MICI measure for schizophrenia diagnosis.

Figure 1.

Flowchart of the study design. Abbreviations: MICI, Morphometric Integrated Classification Index; NC, normal control; SHAP, SHapley Additive exPlanation; SZ, Schizophrenia.

Methods

Detailed descriptions of methods were provided in Supplementary materials.

Results

Construction of the MICI and performance evaluation

In the present study, a total of 10 datasets were used in this study, including three local datasets and seven public neuroimaging datasets.^38–43 Schizophrenia was diagnosed according to the Diagnostic and Statistical Manual of Mental Disorders-IV (DSM-IV). After quality control (Supplementary figure S1), 588 SZ and 682 NC were finally included and the distribution of Contrast to Noise Ratio and Signal to Noise Ratio in each site were shown in Supplementary figure S2. The detailed sources and scanning parameters for the sMRI data and demographic information for finally included participants are shown in the Supplementary tables S1, S2.

A total of 484 sMRI-derived morphometric features were chosen to construct the ensembling model and its derived MICI measure, including 444 cortical features derived from the Destrieux atlas (average cortical thickness [148], sum of cortical surface area [148] and cortical volume [148]), 33 subcortical features derived from Gaussian Classifier Atlas, and 7 whole-brain features (Supplementary table S3). First, we tested previously applied MRI-based machine learning models for schizophrenia to identify the best classification model for a single site (Supplementary table S4).^44–50 Among the 11 candidate classifiers, the XGBoost achieved the highest performance in both single-site and merged models (Delong test, P < 0.05) (figure 2A, Supplementary figure S3A, B). Thus, the XGBoost classifier was chosen for multisite model ensembling.

Figure 2.

Classification performance of the multisite ensembling model and MICI prediction. (A) The classification performance of different single-site classifiers. (B) The classification performance of different multisite models. * The AUC of an independent-site dataset (Local3). (C) The intra-class correlation of AUCs between the MICI prediction and the SWEM in independent-sample datasets. (D) and (E) The average AUCs of the SWEM (D) and the MICI prediction (E) increasing as the increase of the embedded numbers of sites. The box margins represent the 25% and 75% percentiles, and the line margins represent the 1.5 times interquartile range. (F) The intra-class correlation of AUCs between the SWEM and the MICI prediction for the 100 times iterations. (G) and (H) The average AUCs of the SWEM (G) and the MICI prediction (H) in an independent testing site increasing as a function of the embedded number of sites. The gray background represents one standard deviation. (I) The intra-class correlation of the prediction probability between the MICI and the SWEM in the independent-site dataset. Abbreviations: AUC, area under curve; MICI, Morphometric Integrated Classification Index; SWEM, sample-weighted ensembling model.

Then we proposed a sample-weighted ensembling model (SWEM) to integrate the multiple “single-site” pre-trained classifiers into a single prediction considering the sample size used for training in each site. To evaluate the performance of our proposed ensembling model, we first compared the classification performance of the SWEM with the traditional single-site models (satellite model) and multisite raw data-based model (merged model) in the independent-sample testing datasets from the 9 sites and an independent-site testing dataset. We found that SWEM achieved comparable area under the curve (AUC) (independent-sample: 0.82 ± 0.08; independent-site: 0.83) with the merged model (independent-sample: 0.81 ± 0.11; independent-site: 0.82) (Delong test, P > 0.82), and higher AUC than the satellite models (independent-sample: within-site AUC = 0.77 ± 0.07, cross-site AUC = 0.70 ± 0.08; independent-site: 0.72 ± 0.06) (Delong test, P < 0.05) (figure 2B and Supplementary tables S5–S7). To explore if the classification performance increases with the embedment of new sites for the proposed SWEM (evolvability), we randomly embedded the sites into the ensembling model with 100 times iterations. The average classification AUC of the independent-sample testing datasets was gradually increasing from 0.72 (1 site) to 0.82 (9 sites) when new sites were added into this framework (figure 2D), and this increasing pattern was reliably validated in each testing dataset (Supplementary figure S4), and this pattern was replicated on an independent-site testing dataset (AUC = [0.73:0.83]) (figure 2G).

A measure named MICI was created from the SWEM to increase the interpretability. The MICI values were calculated based on the SHapley Additive exPlanation (SHAP) method,³⁶ which is based on the cooperative game theory⁵¹ that splits the final prediction into individual additive contributions (SHAP values) of each valid feature. In the present study, we extended the SHAP value onto a multisite scenario by ensembling multiple “single-site” SHAPs into a scalar named MICI value for each feature (MICI_F) or each subject (MICI_P), in considering the sample size of the pre-trained model of each site (see the Method part for detail). We found that the proposed MICI_P measure performed comparable performance (independent-sample AUC = 0.82 ± 0.08, independent-site AUC = 0.83) with the SWEM and the merged model (Delong test, all P > 0.82), and had higher performance than the satellite model (Delong test, all P < 0.05) (figure 2B, Supplementary figure S3C, D). Besides, as shown in figure 2C, the classification performances between the SWEM and MICI were highly consistent across testing sites (ICC, R ≈ 1.00, P < 0.001). To test if the CNR or SNR affect the classification performance, we also calculated the MICI after regressing out the CNR and SNR from the morphometric features. The classification performance of the new MICI showed no significant difference compared to the one without CNR/SNR regression (Delong test, P = 0.28 for independent-sample testing datasets, P = 0.54 for independent-site dataset) (Supplemental figure S5).

We also evaluated the evolvability of the MICI prediction. On the independent-sample testing datasets, we found the performance of MICI was monotonically increasing when new sites were gradually embedded (AUC = [0.73:0.82]) (figure 2E). Moreover, a high similarity was observed between the SWEM and MICI performance (ICC = 0.99, P < 0.001) (figure 2F). The evolvability of MICI was replicated in the independent-site dataset (AUC = [0.73:0.83], ICC = 0.99, P < 0.001) (figure 2H, I).

MICI differences between the SZ and NC groups

To explore the potential biological meaning of the proposed MICI value, we first compared the differences in participant-wise MICI (MICI_P) and feature-wise MICI values (MICI_F) between the SZ and NC groups using a two-sample t-test. We found that the proposed MICI_P in the SZ was significantly higher than those in the NC (T = 22.34, P = 1.77E-93), and this difference can be replicated in each site (figure 3A). On the feature level, compared to the NC group, the SZ group demonstrated increased MICI_F values derived from cortical volume and thickness in the insula, prefrontal, sensory, and parietal cortex, and from subcortical volume in the putamen, pallidum, and lateral ventricle, and so on (P < 0.05, FWE correction for independent comparisons, figure 3B). Furthermore, we compared the inter-group differences in raw morphometric features and found significantly increased gray matter volume in subcortical structures such as the lateral ventricle, putamen, and pallidum, and found widely decreased thickness and volume in the cerebral cortex, such as the insula, prefrontal, sensory, and parietal cortex (P < 0.05, FWE correction, figure 3C).

Figure 3.

Group differences in MICI and its association with schizophrenia morphometric abnormalities. (A) The histogram distributions of participant-wise MICI values in SZ and HC and their differences. (B) and (C) The intergroup difference in MICI values (B) and brain morphometric measures (C) between the SZ and HC groups (P < 0.05, FWE correction). Colorbar represents the T values. (D) the Spearman correlations in schizophrenia-related absolute changes between MICI and brain morphometric measures (P < 0.05). Abbreviations: HC, healthy controls; MICI, Morphometric Integrated Classification Index; SZ, schizophrenia.

Associations between MICI and brain morphometric changes

On the association between the MICI and gray matter morphometry, we found a significantly negative correlation between the MICI_F and cortical morphometric features (especially the cortical volume and cortical thickness) in both the SZ and NC groups (Spearman correlation, P < 0.05, Bonferroni correction) (Supplementary figure S6), indicating the potential of MICI_F in the representation of the gray matter morphometry. Moreover, there was a significant positive correlation between SZ-related brain morphometric abnormality and MICI_F change levels across features (R = 0.75, P = 1.81E-81). The associations between SZ-related MICI_F changes and morphometric abnormalities were also identified in different types of morphometric features, including cortical surface area (R = 0.56, P = 1.41E-12), cortical thickness (R = 0.77, P = 3.44E-28), cortical volume (R = 0.61, P = 2.69E-15), and subcortical volume (R = 0.83, P = 2.17E-19) (figure 3D). Finally, these associations were replicated in each site (P < 0.05, Bonferroni correction, Supplementary figure S7). These findings indicate that MICI reflects the severity of schizophrenia brain morphometric abnormality.

Association between MICI and clinical measures

At the participant level, a significant positive correlation was identified between MICI_P values and the total scores of Scale for Assessment of Negative Symptoms (SANS) (R = 0.38, P = 6.50E-10) and between MICI_P values and Scale for Assessment of Positive Symptom (SAPS) scores in schizophrenia patients (R = 0.24, P = 1.30E-4) (figure 4A). To further elucidate which MICI features contribute to the symptoms of schizophrenia, Spearman correlations identified three features’ MICI_F values that were positively associated with SANS scores (P < 0.05/484, Bonferroni correction), including the cortical volume of the superior frontal sulcus and planum temporale of the superior temporal gyrus (STG), and the cortical thickness of middle occipital gyrus (MOG); we also identified three features’ MICI_F values that were positively associated with SAPS scores (P < 0.05/484, Bonferroni correction), including the cortical volume of the anterior transverse collateral sulcus and the cortical surface of medial occipitotemporal and lingual sulci, and intraparietal and transverse parietal sulci (figure 4B). Besides, the MICI_P values showed a weak positive correlation with duration of schizophrenia (R = 0.18, P = 0.012, uncorrected) (figure 4A). We did not find correlation between MICI_P and total chlorpromazine equivalent dose in schizophrenia patients (R = 0.099, P = 0.38).

Figure 4.

Associations between MICI values and clinical measures in schizophrenia. (A) The Spearman correlation of participant-wise MICI values with disease duration, SANS scores, and SAPS scores in schizophrenia patients. (B) The Spearman correlation of feature-level MICI values derived from cortical features with SANS and SAPS scores. Colorbar represents the -log(p) of correlation, and the significant brain areas were contoured by white lines. Abbreviations: MICI, Morphometric Integrated Classification Index; SANS, scale for the assessment of negative symptoms; SAPS, scale for the assessment of positive symptoms.

Association between MICI and brain expression of risk genes of schizophrenia

Among the 196 schizophrenia-associated risk genes, Spearman correlation identified 27 genes whose brain mRNA expression levels were significantly associated with the MICI_F differences between schizophrenia patients and HC (P < 0.05, permutation-based FWE correction), including 19 genes with the MICI_F derived from cortical thickness, seven with the MICI_F from cortical surface area, and one with the MICI_F from cortical volume (figure 5A, Supplementary table S8). Hierarchical clustering analysis grouped these 27 genes into two differential-expressed clusters with the highest Calinski-Harabasz score. Cluster 1 contained nine genes that mainly showed a negative correlation with the MICI_F changes, among which CNTN4 showed the strongest negative correlation (R = -0.35, P < 0.05, FWE correction based on permutation). Cluster 2 included 18 genes that were positively correlated with the MICI_F changes, among which TAF5 showed the strongest positive correlation (R = 0.41, P < 0.05, FWE correction based on permutation) (figure 5B). Brain lifespan expression profile of gene CNTN4 demonstrates a low expression level in the brain at the early embryonic stage, increasing expression to peak at the middle embryonic stage, and staying stable. In contrast, the TAF5 shows the highest expression at the early embryonic stage, decreasing rapidly to the bottom at postnatal one year and remaining stable afterward (figure 5C).

Figure 5.

Associations between MICI changes and brain transcription of schizophrenia risk genes. Spearman correlations between the average T-value of MICI changes and AHBA brain mRNA expression levels of 196 schizophrenia risk genes across brain areas were performed to identify genes whose expression levels were associated with the MICI changes (P < 0.05, permutation-based FWE correction). (A) The hierarchical clustering heatmap of identified 27 genes. Colorbar represents the Spearman correlation coefficients. (B) The scatter plot of the correlation between MICI changes and the representative genes of the 2 clusters, CNTN4 (upper) and TAF5 (bottom). (C) The human brain gene expression profiles throughout the lifespan of the two representative genes according to the Human Brain Transcriptome database (http://hbatlas.org/).

Discussion

This work proposed a potential biomarker named Morphometric Integrated Classification Index to discriminate schizophrenia patients from normal controls based on sMRI data of 1270 subjects from 10 sites. This measure is derived from an XGBoost plus sample-weighted ensembling framework based on ensembling anonymous single-site model parameters rather than original private MRI data, thus helping to protect patients’ privacy and saving computation resources. The MICI measure achieved better classification performance than the traditional machine learning models trained with raw MRI data on several independent-site datasets. Moreover, a new site is easily included in this ensembling framework, and the classification performance of the MICI is increasing as a new site is embedded. Finally, the derived MICI measure was associated with brain structural abnormalities, clinical symptoms of schizophrenia, and brain mRNA expressions profile of schizophrenia risk genes. Thus, these findings highlighted the generalizability, shareability, evolvability, and interpretability of the proposed MICI for schizophrenia diagnosis using structural neuroimaging data acquired from different sites.

Although black-box machine learning models have been proved to outperform traditional clinical practices in many fields, the poor interpretability severely restricts their clinical applications.^28,29 It is a dilemma on the complexity and interpretability of ML models: on the one hand, the model is expected to be as simpler as people can understand but may sacrifice performance; on the other hand, the model is expected to be more powerful but maybe at the cost of increased complexity (and thus sacrifice interpretability). Multisite model-ensembling strategies proposed in the present study further complicated the situation of model interpretability. Thus another challenge for machine learning is the tradeoff between model interpretability and model flexibility to make model-agnostic be explained.⁵² To solve this issue, we proposed a single scalar named morphometric integrated classification index—MICI—from the ensembling framework based on the theorem of SHAP values. SHAP fits a linear additive model to split the final prediction into individual additive contributions (SHAP values) for each valid feature, meaning that the sum of the SHAP values of all valid features is approximately equivalent to the model prediction.^36,51 However, the SHAP approach is initially designed for a single model scenario. In this work, we extended it into a multisite scenario by ensembling the SHAP values of multiple sites into a single MICI value accounting for the sample size of each model. Benefiting from the “local accuracy” characteristic of SHAP value,³⁶ our results showed that the MICI value achieved comparable classification performance and evolvability as the multi-variates ensembling model and merged model based on raw data. Moreover, the MICI values demonstrated preferable interpretability on brain structural changes and clinical features of schizophrenia, and brain transcriptional profiles of schizophrenia risk genes.

Although mounting studies have shown that multisite data ensembling can greatly promote generalization and ML prediction performance,^{18–20,53,54} multisite raw MRI data sharing is severely restricted by computing resources, personal privacy, and other effects.^22–26 To solve these barriers during MRI data sharing, we presented an XGBoost plus sample-weighted ensembling framework that integrates the parameters of anonymous single-site models rather than original private MRI data to make a neuroimaging biomarker. Like other model ensembling approaches,^19,55 our presented biomarker owed several advantages. First, it diminished ethical and privacy issues brought by raw data sharing and thus improved data security. Second, the shared model-based data are much smaller than the raw MRI data and thus greatly save storing and networking resources during sharing. Finally, the present model-ensembling method does not need further re-training, thus saving computation resources.¹¹ These features can accelerate the drive toward multisite data sharing and promote clinical applications.

Specifically, we found that MICI values in schizophrenia patients were consistently higher than those in healthy controls across sites, and the MICI values in schizophrenia were significantly higher in many cortical and subcortical structures, such as the prefrontal, insula, sensory cortex, and lateral ventricle, putamen, and pallidum, which was consistent with previous reports about the abnormal cerebral structures in schizophrenia.^56–58 This is substantiated by the significant correlations between MICI changes and brain structural changes in schizophrenia, indicating the powerfulness of MICI values in representing schizophrenia’s brain structural abnormalities. It should be noted that the direction of MICI changes was not always consistent with the morphometric changes. For example, the increment of cortical MICI was accompanied by a decrement of cortical thickness and volume, while the increment of subcortical MICI was accompanied by a consistent increment of subcortical volume, suggesting that MICI may be considered an abnormality severity measure rather than an abnormality direction measure. We also demonstrated a positive association between MICI and positive and negative symptoms of schizophrenia, indicating higher MICI may reflect more severe clinical symptoms. The MICI-symptom association was also identified at the area level. For example, negative symptoms were significantly positively associated with the MICI of the STG, which was consistent with an early finding reporting a positive association between the GMV of this area and negative symptoms.⁵⁹ Besides, positive symptoms were significantly associated with the MICI of the MOG. Similarly, the association between the thickness of MOG and hallucination had been reported in first-episode psychosis patients.⁶⁰ These findings indicate that MICI values can reflect the severity of clinical symptoms in schizophrenia.

Finally, we identified 27 schizophrenia risk genes whose brain mRNA expression levels were significantly associated with the MICI changes in schizophrenia. These genes can be classified into two clusters with opposite associations with the MICI. As a representative gene for cluster 1, CNTN4 showed the strongest negative association with MICI increment; and its brain expression level increases rapidly from the early embryonic to the middle embryonic stage and keeps stable after. CNTN4 gene is an axonal glycoprotein associated with cell adhesion molecules and plays important roles in axon arborization and neuronal development.^61,62 Early reports had indicated it is a risk gene for schizophrenia and autism.^63,64 Furthermore, overexpression of CNTN4 resulted in a significant decrease in head size of zebrafish embryos, indicating an important role of this gene in regulating brain development.⁶⁵ On the contrary, the representative gene for cluster 2, TAF5 demonstrated the strongest positive association with MICI increment, shows the highest expression at the early embryonic stage, and whose expression decreases rapidly down to the bottom at postnatal one year. TAF5 is related to DNA-binding transcription factor activity⁶⁶ and was also reported as a candidate risk gene for schizophrenia.⁶⁷ These findings indicate that the proposed MICI measure can provide biologically meaningful information about schizophrenia.

However, several limitations should be mentioned when interpreting our findings. Firstly, the generalization of the MICI still needs to be validated and improved with more datasets, especially using drug-naïve first-episode schizophrenia dataset. Therefore, we developed an online free-share platform and wish more and more sites share their models to validate and improve the biomarker’s performance (http://micc.tmu.edu.cn/mici/index.html). In addition, this cloud platform also provides free individual prediction services for the public, which not only provide the predictions based on uploading data, but also the MICI can be obtained directly to perform further scientific research and clinical assistance. Second, we established MICI only based on sMRI data; however, this measure is theoretically modality-free. Different neuroimaging modalities (such as diffusion MRI, functional MRI [fMRI], and PET) may provide unique information about schizophrenia diagnosis and treatment. Thus, it is preferred to apply this framework to other neuroimaging modalities to obtain different types of ensembling biomarkers for schizophrenia in the future. Moreover, in consideration that the symptoms of schizophrenia patients often change over time, thus it is preferred to combine the fMRI with sMRI to construct MICI, which may increase the power in characterizing the state of schizophrenia and promote prognosis prediction.

Funding

This work was supported by the National Key Research and Development Program of China (2018YFC1314300), National Natural Science Foundation of China (81971599, 82030053, and 81971694), Tianjin Key Project for Chronic Diseases Prevention (2017ZXMFSY00070), Science&Technology Development Fund of Tianjin Education Commission for Higher Education (2018KJ082), Tianjin Applied Basic Research Diversified Investment Foundation (21JCYBJC01490).