Joint multi-site domain adaptation and multi-modality feature selection for the diagnosis of psychiatric disorders

Highlight

• •Proposed a joint multi-site and multi-modality framework for psychiatric disorder diagnosis.
• •Developed a hypergraph-based multi-source domain adaptation (HMSDA) method that enabled the transformation of source domain subjects into a common target domain.
• •Introduced a triplet-based local ordinal structure for effective feature selection across different modalities.
• •Demonstrated superior classification performance in diagnosing schizophrenia and autism spectrum disorder compared with 6 DA, 10 multi-modality feature selection, and 8 multi-site and multi-modality methods.

Abstract

Identifying biomarkers for computer-aided diagnosis (CAD) is crucial for early intervention of psychiatric disorders. Multi-site data have been utilized to increase the sample size and improve statistical power, while multi-modality classification offers significant advantages over traditional single-modality based approaches for diagnosing psychiatric disorders. However, inter-site heterogeneity and intra-modality heterogeneity present challenges to multi-site and multi-modality based classification. In this paper, brain functional and structural networks (BFNs/BSNs) from multiple sites were constructed to establish a joint multi-site multi-modality framework for psychiatric diagnosis. To do this we developed a hypergraph based multi-source domain adaptation (HMSDA) which allowed us to transform source domain subjects into a target domain. A local ordinal structure based multi-task feature selection (LOSMFS) approach was developed by integrating the transformed functional and structural connections (FCs/SCs). The effectiveness of our method was validated by evaluating diagnosis of both schizophrenia (SZ) and autism spectrum disorder (ASD). The proposed method obtained accuracies of 92.2 %±2.22 % and 84.8 %±2.68 % for the diagnosis of SZ and ASD, respectively. We also compared with 6 DA, 10 multi-modality feature selection, and 8 multi-site and multi-modality methods. Results showed the proposed HMSDA+LOSMFS effectively integrated multi-site and multi-modality data to enhance psychiatric diagnosis and identify disorder-specific diagnostic brain connections.

1. Introduction

Computer-aided diagnosis (CAD) approaches are of great interest for analyzing high-resolution medical images, such as functional magnetic resonance imaging (fMRI) and structural magnetic resonance imaging (sMRI) (Qi et al., 2022, Rashid and Calhoun, 2020, Zhu et al., 2022b) to improve the clinical characterization of psychiatric disorders. Network neuroscience principles can be used to estimate brain functional networks (BFNs) or brain structural networks (BSNs), where vertices and edges are defined as brain functional connections (FCs) or structural connections (SCs) (Raj and Powell, 2018). Such work provides the basis for researchers to further understand the functional and structural organization of the brain. Utilizing brain networks derived from multi-site and multi-modality has emerged as a promising biomedical strategy for CAD of psychiatric disorders, including schizophrenia (SZ) and autism spectrum disorder (ASD). However, most techniques commonly assume that multi-site data are drawn from the same distribution (Abraham et al., 2017); and thus ignore the challenge of data heterogeneity arising from inter-site disparities in scanners, image contrasts, resolutions, and noise levels across different sites (Heinsfeld et al., 2018). This has led to unsatisfactory performance in many studies focused on CAD of psychiatric disorders using multi-site data (Hu et al., Wachinger et al., 2016, Zhao et al., 2024).

Domain adaptation (DA) can reduce the data distribution difference between the source domains and the target domains for cross-domain knowledge transferring (Kouw and Loog, 2019). DA aligns data from the source domains and the target domains to approximate a shared distribution, leveraging labeled data from the source domains to train a model and directly utilize the learned model to predict the labels of the unlabeled data from the target domains. Wang et al. (Wang et al., 2020a) projected the data from the source domains and the target domains into the common subspace to learn the domain-invariant features based on low-rank representation (LRR), resulting in data from the two domains exhibiting similar distributions in the new feature space. Ding et al. (Ding et al., 2022) introduced a low-rank structure and an inter-class difference constraint DA method for the diagnosis of ASD. Kunda et al. (Kunda et al., 2022) assessed the statistical dependence between acquisition sites and FC features, and adopted a DA method to reduce the site dependence of FC features to improve ASD classification performance. Wang et al. (Wang et al., 2020b) proposed a novel multi-source DA to transform source subjects to follow the same distribution with the target subjects and developed a multi-class classifier by effectively fusing multi-view features to perform the diagnosis of the target subjects. Although existing DA methods have achieved promising results, one disadvantage is that they only consider the pairwise relationships between labels and samples in terms of geometric structure preservation (Courty et al., 2016), which ignores the high-order relationships among subjects. Multi-site subjects are likely more complex and such relationships could be significant for inducing more discriminative multi-site features.

Furthermore, current DA methods are limited to a single-modality. However, it is clear that different modalities provide complementary information and numerous studies demonstrated that multi-modality fusion is more effective in improving the accuracy of CAD than single modality (Calhoun and Sui, 2016). Feature selection has emerged as a critical step in multi-modality based methods due to the relatively high feature dimensionality across different modalities, which may contain redundant features and hurt subsequent classification performance. Zhu et al. (Zhu et al., 2015) combined two subspace learning methods, linear discriminant analysis and locality preserving projection to perform multi-modality feature selection. Jie et al. (Jie et al., 2015) proposed a manifold regularized multi-task feature selection method which used both multi-task learning and Laplacian regularization to preserve the inherent correlations among multiple modalities and the data distribution information in each modality. Ning et al. (Ning et al., 2021) proposed a relation-induced shared multi-modality representation learning framework for CAD by integrating the representation learning, dimension reduction, and classifier modeling into a unified framework. Hao et al. (Hao et al., 2023) proposed a multi-modality self-paced locality-preserving learning framework to preserve the inherent structural relationships within the original data that facilitated a sample selection process from simple to complex. However, the intrinsic structure information among subjects is not accurately preserved in existing multi-modality feature selections. The use of pairwise similarity measures or other simple metrics to represent complex structural relationships among subjects results in the loss of topological structure information. Besides, due to intra-modality heterogeneity, including individual differences, data distribution variations, and label inconsistencies (Zhu et al., 2022a), brings challenges in selecting discriminative features. Moreover, these methods overlook the challenges of the use of multi-site data.

In this paper, we developed a novel HMSDA+LOSMFS (Hypergraph based Multi-Source Domain Adaptation & Local Ordinal Structure based Multi-task Feature Selection) method for multi-site and multi-modality fusion. The BFNs and BSNs from different sites were used as the individual target domain and the remaining sites served as the source domains to formulate a multi-source DA problem. Meanwhile, both BFNs and BSNs provide complementary insights for characterizing subjects, which is treated as a multi-task learning problem. Accordingly, the proposed HMSDA+LOSMFS method involves two stages. Firstly, HMSDA focused on transforming the source domain subjects into the target space, incorporating hypergraph-based regularization to explicitly model the high-order relationships in the source domains after transforming. Secondly, each transformed modality was treated as a single learning task, and multi-modality feature selection was considered as a multi-task learning problem with Laplacian ordinal projection to preserve the inherent differences and similarities within each modality's data and select discriminative features. Finally, multi-kernel support vector machine (SVM) was used to fuse the selected features from multi-modality data for the final classification. The proposed HMSDA+LOSMFS was validated on the diagnosis of both SZ and ASD.

2. Methods and materials

2.1. Overview

Fig. 1 gives the overview of the proposed method, which mainly includes the following steps: (1) Constructing BFNs and BSNs of the source domains and the target domains, in which the upper triangle of the FC/SC matrices for each subject was reshaped into a feature vector (Fig. 1a). (2) A novel HMSDA method was proposed to transform the source domain subjects into the target domain feature space. Both low-rank and hypergraph regularizations were introduced in HMSDA (Fig. 1b). (3) A novel LOSMFS method based on local ordinal structure was proposed to select the multi-modality features, a triplet-based regularization was introduced to enforce the selected features to preserve ordinal locality of the original data (Fig. 1c). (4) The multi-kernel SVM was used for the diagnosis of psychiatric disorders (Fig. 1d).

Fig. 1.

Flowchart of the proposed HMSDA+LOSMFS for psychiatric disorders’ diagnosis. (a) BFNs and BSNs of the source domains and the target domains were constructed based on fMRI and sMRI, and the upper triangle of the FC/SC matrices for each subject was reshaped into a feature vector. (b) HMSDA was proposed to transform the source domain subjects into the target domain feature space. (c) LOSMFS based on local ordinal structure was proposed to select features. (d) The multi-kernel SVM was used for psychiatric disorders’ diagnosis.

2.2. Construction of BFNs and BSNs

The Anatomical Automatic Labeling (AAL) atlas (90 brain regions, without cerebellum) (Tzourio-Mazoyer et al., 2002) was used as the template to construct BFNs and BSNs. Specifically, the regional averaged time series were extracted according to the AAL atlas and the adjacency matrix (90 × 90) of the BFN was constructed by calculating the Pearson’s correlation (PC) between the mean time series from each pair of brain regions. BSNs were constructed by Jensen divergence to quantify the similarity of gray matter volume between two brain regions (Li et al., 2021, Joshi et al., 2011). Specifically, we first extracted gray matter volume (GMV) for all the voxels within each brain region, and the probability distribution function (PDF) of the corresponding brain region was calculated using kernel density estimation (Duong, 2007). Subsequently, the Jensen divergence was calculated between a pair of brain regions in their PDFs as SCs. Due to the symmetry of BFNs and BSNs, the upper triangular elements of each network were assembled into one column to generate a feature vector ($\frac{90\times \left(90-1\right)}{2}=4005$ dimensions) and FCs and SCs of all participants were concatenated to obtain two aggregated feature matrices, respectively.

2.3. Hypergraph based multi-source DA

Suppose we have a set of subjects including M modalities, i.e., ${\mathit{S}}_i=\left[{\mathit{S}}_i^1,\cdots {,\mathit{S}}_i^M\right]$, where ${\mathit{S}}_i^m\in {\mathbb{R}}^{D_m\times N_{S_i}},i=1,...,I,m=1,\cdots ,M$ denotes the feature matrix of the m-th modality in the i-th source domain, D_m represents the dimension of the m-th modality and $N_{S_i}$ represents the number of subjects in the i-th source domain. Another set of subjects ${\mathit{T}}^m\in {\mathbb{R}}^{D_m\times N_T}$ represents the data of the m-th modality in the target domain, where N_T represents the subject number in the target domain. The goal of multi-source DA is to find the transformation matrices ${\mathit{W}}_i^m\in {\mathbb{R}}^{D_m\times D_m}$ that transform the feature distribution of subjects in multiple source domains into the target domain to reduce the data distribution difference between source and target domains (Jhuo et al., 2012). The multi-source DA based on LRR is proposed to improve the robustness of models to undesirable noise and the objective function is formulated as:

$$\underset{{\mathit{W}}_i^m,{\mathit{Z}}^m,{\mathit{E}}^m}{\min }rank\left({\mathit{Z}}^m\right)+\tau {\Vert {\mathit{E}}^m\Vert }_{2,1}s.t.[{\mathit{W}}_1^m{\mathit{S}}_1^m,...,{\mathit{W}}_i^m{\mathit{S}}_i^m]={\mathit{T}}^m{\mathit{Z}}^m+{\mathit{E}}^m$$ (1)

where rank(.) represents the rank of a matrix, ${\Vert .\Vert }_{2,1}$ represents the l_2,1-norm, τ is a parameter to balance the two terms, ${\mathit{Z}}^m\in {\mathbb{R}}^{N_T\times N_S}$ represents the reconstruction coefficient matrix, ${\mathit{E}}^m\in {\mathbb{R}}^{D_m\times N_S}$ represents the sparse error matrix between the transformed features of all source domains and the reconstructed target domain data, ${\mathit{W}}_i^m{\mathit{S}}_i^m\in {\mathbb{R}}^{D_m\times N_{S_i}}$ represents the transformed matrix sets of all source domains, and N_S represents the total number of subjects across all source domains. Since rank minimization is an NP-hard problem and the nuclear norm serves as an effective surrogate for its relaxation, we minimize Eq. (2) to replace Eq. (1):

$$\underset{{\mathit{W}}_i^m,{\mathit{Z}}^m,{\mathit{E}}^m}{\min }{\Vert {\mathit{Z}}^m\Vert }_{\ast }+\tau {\Vert {\mathit{E}}^m\Vert }_{2,1}s.t.[{\mathit{W}}_1^m{\mathit{S}}_1^m,...,{\mathit{W}}_i^m{\mathit{S}}_i^m]={\mathit{T}}^m{\mathit{Z}}^m+{\mathit{E}}^m$$ (2)

where ${\Vert .\Vert }_{\ast }$ represents the nuclear norm of a matrix. Z^m is determined by jointly finding a low-rank encoding of the transformation coefficient matrix from all sites, so that DA can be solved across multiple sites. Minimizing ${\Vert {\mathit{E}}^m\Vert }_{2,1}$ can make features more robust to noise or outliers.

Besides reducing the data distribution difference among multi-site data via Eq. (2), we aim to further discover the intrinsic data structure of different sites. Most methods assume that subjects with pairwise similarities have identical labels or phenotypes, incorporating this relationship as a regularization term in DA. However, the oversight of high-order relationships involving three or more subjects neglects potential discriminative information beneficial to DA. Therefore, we introduced HMSDA by incorporating a hypergraph regularization into Eq. (2). Denote a hypergraph as $\mathcal{G}=\left(\mathcal{V},\mathcal{E},\mathit{A}\right)$, where $\mathcal{V}=\left\{v_1,v_2,\cdots ,v_{N_V}\right\}$ represents the vertex set and $\mathcal{E}=\left\{e_1,e_2,\cdots ,e_{N_H}\right\}$ represents the hyperedge set. N_V denotes the number of vertices and N_H denotes the number of hyperedges. $\mathit{A}={\left(a\left(e_1\right),a\left(e_2\right),\cdots ,a\left(e_{N_H}\right)\right)}^T$ represents the hyperedge weight set with each hyperedge e_q being assigned the weight a(e_q). Unlike simple graph edges, which involve pairs of vertices, hyperedges include arbitrarily sized sets of vertices. The binary vertex-edge relations are represented by an incidence matrix $\mathit{M}\in {\mathbb{R}}^{N_V\times N_H}$, each element of M can be defined as:

$$M_{\mathit{pq}}=\left\{\begin{array}{c}1,if{v}_p\in e_q\\ 0,if{v}_p\notin e_q\end{array}\right)$$ (3)

For M, the degree of each vertex v and each hyperedge e can be formulated as:

$$d\left(v_p\right)=\sum _{e_q\in \mathcal{E}}a\left(e_q\right)M_{\mathit{pq}},1\le p\le N_V$$ (4)

$$\delta \left(e_q\right)=\sum _{v_p\in \mathcal{V}}{M}_{\mathit{pq}},1\le q\le N_H$$ (5)

In order to capture the high-order correlation among different subjects after DA, the method proposed by Zhou et al. (Zhou et al., 2006) is adopted to define the hypergraph Laplacian matrix as following:

$${\mathit{L}}^h=\mathit{I}-\mathit{\Theta }$$ (6)

where ${\mathit{L}}^h\in {\mathbb{R}}^{N_V\times N_V}$ is the hypergraph Laplacian matrix, I is the identity matrix and $\mathit{\Theta }={\mathit{D}}_v^{-\frac{1}{2}}\mathit{M}\mathit{A}{\mathit{D}}_e^{-1}{\mathit{M}}^T{\mathit{D}}_v^{-\frac{1}{2}}$. D_v and D_e denote diagonal matrices comprising the vertex degrees and hyperedge degrees, respectively. The k-nearest neighbors (KNN) algorithm is adopted to construct the hypergraph based on the source domain subjects. It selects the k nearest neighbors based on attribute similarities, enriching the original hypergraph structure with more informative connections (Purkait et al., 2016). Specifically, by taking each vertex in a hypergraph as the center node, Euclidean distances between the center node and other vertices are computed, connecting each center node to its k nearest neighbors and obtaining the incidence matrix M. The objective function of HMSDA can be formulated as follows:

$$\begin{array}{c}\underset{{\mathit{W}}_i^m,{\mathit{Z}}^m,{\mathit{E}}^m}{\min }{\Vert {\mathit{Z}}^m\Vert }_{\ast }+\tau {\Vert {\mathit{E}}^m\Vert }_{2,1}+\frac{\lambda }{2}.\frac{1}{2}\sum _{p=1}^{N_S}\sum _{q=1}^{N_S}{\left(z_p^m-z_q^m\right)}^2{\Theta }_{\mathit{pq}}^m\hfill \\ =\underset{{\mathit{W}}_i^m,{\mathit{Z}}^m,{\mathit{E}}^m}{\min }{\Vert {\mathit{Z}}^m\Vert }_{\ast }+\tau {\Vert {\mathit{E}}^m\Vert }_{2,1}+\frac{\lambda }{2}tr\left({\mathit{Z}}^m{\mathit{L}}^h{\left({\mathit{Z}}^m\right)}^{\prime }\right)\hfill \\ s.t.[{\mathit{W}}_1^m{\mathit{S}}_1^m,...,{\mathit{W}}_i^m{\mathit{S}}_i^m]={\mathit{T}}^m{\mathit{Z}}^m+{\mathit{E}}^m\hfill \end{array}$$ (7)

where $z_p^m$ and $z_q^m$ are the p-th and q-th column vectors in Z^m. Eq. (7) can be solved via the augmented Lagrange multiplier (ALM) algorithm (Perone et al., 2019) and the solution of HMSDA is shown in Supplementary Table 3. Once obtaining the optimal solution ${\mathit{W}}_i^m,{\mathit{Z}}^m,{\mathit{E}}^m$, we used ${\mathit{W}}_i^m{\mathit{S}}_i^m$ to transform the source domain subjects into the target domain. The transformed source domain samples can be integrated with the target domain training samples to form the feature matrix ${\mathit{X}}^m\in {\mathbb{R}}^{D_m\times N},N=N_T+N_S$ and the corresponding label vector ${\mathit{y}}^m\in {\mathbb{R}}^N$ for subsequent experiments.

2.4. Local ordinal structure based multi-task feature selection

Although HMSDA aids in reducing heterogeneity across sites, there may still be heterogeneity in intra-modality. In addition, subjects have high-dimensional features with significant redundancy. Therefore, we aimed to enhance discriminability among multi-site subjects in multi-modality feature selection, which can be represented as the optimization result of the ranking information of the original data. Local ordinal structure describes the inherent hierarchical relationships between neighboring samples and the ranking of these neighboring samples (Guo et al., 2017). It is useful for identifying discriminative features that demonstrate reproducibility and stability across heterogeneous data. Assuming a triplet $\left({\mathit{x}}_i^m,{\mathit{x}}_j^m,{\mathit{x}}_k^m\right)$ consisting of ${\mathit{x}}_i^m\in d_{\mathit{min}}$; its neighbors ${\mathit{x}}_j^m\in d_{\mathit{max}}$ and ${\mathit{x}}_k^m$ in the m-th modality. p^m is the projection vector of ${\mathit{x}}_i^m$ and ${\mathit{r}}_i^m={{\mathit{p}}^m}^T{\mathit{x}}_i^m$ in the m-th modality, and the corresponding mapped triplet becomes $\left({\mathit{r}}_i^m,{\mathit{r}}_j^m,{\mathit{r}}_k^m\right)$. Define dist(.,.) as the distance metric, feature selection preserves local ordinal structure under the following condition: if $dist\left({\mathit{x}}_i^m,{\mathit{x}}_k^m\right)\le dist\left({\mathit{x}}_i^m,{\mathit{x}}_j^m\right)$, $dist\left({\mathit{r}}_i^m,{\mathit{r}}_k^m\right)\le dist\left({\mathit{r}}_i^m,{\mathit{r}}_j^m\right)$, which is beneficial to the classification of ${\mathit{x}}_i^m$ and this structure is needed to preserved in the projection space. Integrating above definition and the rearrangement inequality, the local ordinal structure over a collection of triplets is optimized via the following objective function:

$$\underset{{\mathit{R}}^m}{\max }\sum _{i=1}^n\sum _{j=1}^n\sum _{k\in {\mathcal{N}}_i}{F}_{j,k}^i\left[dist\left({\mathit{r}}_j^m,{\mathit{r}}_k^m\right)-dist\left({\mathit{r}}_i^m,{\mathit{r}}_k^m\right)\right]$$ (8)

where ${\mathcal{N}}_i$ represents the k neighbors sample sets of ${\mathit{x}}_i^m$ and R^m represents the selected feature matrix, F^m is an antisymmetric matrix and its elements can be calculated by:

$$F_{j,k}^i=\left\{\begin{array}{c}{E}_{j,k}^{i}sgn\left(E_{j,k}^i\right),if{\mathit{x}}_k^m\in {\mathcal{N}}_i,{\mathit{x}}_i^m,{\mathit{x}}_k^m\in d_{\mathit{min}},{\mathit{x}}_j^m\in d_{\mathit{max}}\\ -E_{j,k}^{i}sgn\left(E_{j,k}^i\right),if{\mathit{x}}_k^m\in {\mathcal{N}}_i,{\mathit{x}}_i^m\in d_{\mathit{min}},{\mathit{x}}_j^m,{\mathit{x}}_k^m\in d_{\mathit{max}}\\ 0,otherwise\end{array}\right)$$ (9)

where $E_{j,k}^i=dist\left({\mathit{x}}_j^m,{\mathit{x}}_k^m\right)-dist\left({\mathit{x}}_i^m,{\mathit{x}}_k^m\right)$ and sgn(.) represents the sign function. A distance matrix C^m can be calculated from F^m as:

$$C_{i,k}^m={\sum }_j{F}_{k,j}^i$$ (10)

Then, due to F^m is an antisymmetric matrix and $F_{j,k}^i=-F_{k,j}^i$. Eq. (8) can be redefined as:

$$\underset{{\mathit{R}}^m}{\max }\left\{\begin{array}{c}-{\sum }_{j=1}^n{\sum }_{k=1}^n{\sum }_{i=1}^n{F}_{i,k}^{j}dist\left({\mathit{r}}_j^m,{\mathit{r}}_k^m\right)\\ -{\sum }_{i=1}^n{\sum }_{k=1}^n{\sum }_{j=1}^n{F}_{j,k}^{i}dist\left({\mathit{r}}_i^m,{\mathit{r}}_k^m\right)\end{array}\right\}=\underset{{\mathit{R}}^m}{\max }\left\{\begin{array}{c}-{\sum }_{j=1}^n{\sum }_{k=1}^n{C}_{j,k}^{m}dist\left({\mathit{r}}_j^m,{\mathit{r}}_k^m\right)\\ -{\sum }_{i=1}^n{\sum }_{k=1}^n{C}_{i,k}^{m}dist\left({\mathit{r}}_i^m,{\mathit{r}}_k^m\right)\end{array}\right\}$$ (11)

Following, Eq. (12) can be obtained by subtracting the two equations in Eq. (11), which can be written in ultimate form as Eq. (13) using Euclidean distance:

$$\underset{{\mathit{R}}^m}{\min }\sum _{i=1}^n\sum _{j=1}^n{C}_{i,j}^{m}dist\left({\mathit{r}}_i^m,{\mathit{r}}_j^m\right)$$ (12)

$$\underset{{\mathit{R}}^m}{\min }\sum _{i=1}^n\sum _{j=1}^n{C}_{i,j}^m{\Vert {\mathit{r}}_i^m-{\mathit{r}}_j^m\Vert }_2^2=\underset{{\mathit{R}}^m}{\min }tr\left({{\mathit{R}}^m}^T{\mathit{L}}^m{\mathit{R}}^m\right)=\underset{{\mathit{p}}^m}{\min }tr\left({{\mathit{p}}^m}^T{{\mathit{X}}^m}^T{\mathit{L}}^m\mathbf{X}{}^m{\mathit{p}}^m\right)$$ (13)

where ${\mathit{L}}^m={\mathit{D}}^m-\frac{{\mathit{C}}^m+{{\mathit{C}}^m}^T}{2}$, D^m is a diagonal matrix and $D_{\mathit{ii}}^m={\sum }_{j=1}^N\frac{C_{\mathit{ij}}^m+C_{\mathit{ji}}^m}{2}$. The above optimization problems were combined with multi-task feature selection. An l_2,1-norm regularization term was added for ${\mathit{p}}^m$ to minimize redundant information across modalities (Zhang et al., 2011). The final objective function of LOSMFS presented in Eq. (14):

$$\underset{{\mathit{p}}^m}{\min }\frac{1}{2}\sum _{m=1}^M{\Vert {\mathit{y}}^m-{\mathit{X}}^m{\mathit{p}}^m\Vert }_F^2+\alpha {\Vert \mathit{P}\Vert }_{2,1}+\beta tr\left({{\mathit{p}}^m}^T{\mathit{X}}^m{\mathit{L}}^m{{\mathit{X}}^m}^T{\mathit{p}}^m\right)$$ (14)

where $\mathit{P}=\left[{\mathit{p}}^1,\cdots ,{\mathit{p}}^M\right]$ denotes the weight matrix including all p^m. The accelerated proximal gradient (APG) algorithm (Jie et al., 2015) was used to solve Eq. (14) and the optimization procedure for selecting features from M modalities is shown in Supplementary Table 4. The feature matrix R^m after feature selection can be obtained by multiplying X^m and p^m.

3. Classification

Multi-kernel SVM (Zhang et al., 2012) was adopted for the classification task after feature selection. First, a kernel matrix $K^m\left({\mathit{r}}_i^m,{\mathit{r}}_j^m\right)={\varphi }^m\left({\left({\mathit{r}}_i^m\right)}^T\left({\mathit{r}}_j^m\right)\right)$ was generated for each modality after feature selection. The M kernel matrices were linearly combined with $K\left({\mathit{r}}_i,{\mathit{r}}_j\right)={\sum }_m{\delta }_m{K}^m\left({\mathit{r}}_i^m,{\mathit{r}}_j^m\right)$, where δ_m is a combination coefficient with constraint of ${\sum }_m{\delta }_m=1,{\delta }_m\ge 0$. The optimal δ_m was determined via a grid-search method through cross-validation on the training sets. The trained multi-kernel SVM model was used to predict the category of the testing sets. In particular, we trained a linear kernel SVM using the LIBSVM toolbox (Chang and Lin, 2011) for classification; as it inherently employs a feature weighting mechanism, wherein the absolute values of components in the normal vector of the SVM's hyperplane serve as weights for features. Patients (SZ/SAD) and normal controls (NCs) were considered as positive and negative samples, respectively.

3.1. Validation datasets

Function Biomedical Informatics Research Network (FBIRN) phase III datasets (Keator et al., 2016) including 154 SZs and 152 normal controls (NCs) that were matched for age, gender, handedness and race distributions were used as validation dataset 1 (University of California Irvine, UCI, n = 45; University of California San Francisco, UCSF, n = 61; Duke University/University of North Carolina, DU/UNC, n = 56; University of New Mexico, UNM, n = 28; University of Iowa, UI, n = 57 and University of Minnesota, UM, n = 59). Autism Brain Imaging Data Exchange (ABIDE) I (Di Martino et al., 2014) datasets including 100 ASDs and 100 NCs were chosen as validation dataset 2 (NYU, n = 50; PITT, n = 50; USM, n = 50 and Yale, n = 50). The demographic information of SZ and ASD are summarized in Supplementary Table 1-2. Detailed scanning parameters and preprocessing steps can be found in Supplementary “Multi-modality imaging parameters and preprocessing” section.

4. Results

4.1. Experimental setup

A 5-fold cross-validation strategy was used to evaluate the classification performance. Specifically, the target domain subjects were randomly divided into 5 subsets (with each subset having a roughly equal size of subjects), and each time one subset was selected as the testing sets, while other subjects in the remaining subsets and all source domain subjects were used as the training sets. There were four hyper-parameters in our model, i.e., τ, λ, α and β. They were determined by 5-fold cross-validation, employing a grid-search strategy, and were varied within the range of [1e^-5, 1e^-4, 1e^-3, 1e^-2, 1e^-1], respectively. It is time-consuming and inefficient to tune all parameters simultaneously. Therefore, we tuned τ and λ each time by fixing α and β, and tuned α and β while fixing τ and λ in different classification tasks. We empirically set the number of hyperedges as 3 and the number of the k neighbors as 3. Moreover, in multi-kernel SVM, C was set as 1 and the kernel combination coefficients δ_FC, δ_SC were chosen from 0.1 to 1.0 with step 0.1 and constrained with δ_FC+δ_SC=1. We calculated the mean value and standard deviation (STD) of each metric after 5 times of 5-fold cross-validation as the final results. Seven metrics were used to evaluate the classification performance of different methods, including accuracy (ACC), sensitivity (SEN), specificity (SPE), balanced accuracy (BAC), positive predictive value (PPV), negative predictive value (NPV) and area under the curve (AUC). Higher values for these metrics indicate better classification performance.

4.2. Comparison with 6 DA methods

To intuitively demonstrate the effectiveness of our HMSDA method, we conducted experiments on synthetic data generated by Gaussian distribution, including two source domains and one target domain. As shown in Supplementary Figure 1, our method aligned the data distributions across all domains, demonstrating its effectiveness in reducing the difference among domain distributions.

To verify the effectiveness of the proposed HMSDA method, we compared it with alternative DA methods, including: (1) LRR (Liu et al., 2013), aiming to find a low-dimensional representation for multiple domains, (2) multi-site adaption framework via low-rank representation decomposition (maLRR) (Wang et al., 2020a), learning the common component from specific projection matrices of multiple source domains as the projection matrix of the target domain, (3) transfer subspace learning via low-rank and sparse representation (TSL_LRSR) (Xu et al., 2016), projecting the data of the two domains into the common subspace to learn the domain invariant feature representation, (4) geodesic flow kernel (GFK) (Gong et al., 2012), leveraging low-dimensional data structures to address the dissimilarity in data distributions between the source and target domains, (5) low-rank and class discriminative representation (LRCDR) (Liu et al., 2023), simultaneously learning class-discriminative representations from multiple source domains and the target domain, (6) multi-source low-rank domain adaptation (MSLRDA) (Wang et al., 2020b), adopting the low-rank coding of subjects’ features to preserve the mutual relations in their labels. The regularization parameters for LRR, maLRR, TSL_LRSR, LRCDR and MSLRDA were selected from the set [1e^-5, 1e^-4, 1e^-3, 1e^-2, 1e^-1], and the parameter (i.e., dimensionality of the subspace) in GFK was chosen from [5, 10, …, 50] via 5-fold cross-validation. It is worth noting that for maLRR, TSL_LRSR, GFK and LRCDR, we followed an identical experimental setup as Liu et al. (Liu et al., 2023) to perform dimensionality reduction for obtaining the optimal results. We conducted experiments separately on the FBIRN and ABIDE I datasets, each domain was selected as the target domain in turn, and the remaining domains were treated as the source domains. After DA, features were selected by LOSMFS and multi-kernel SVM was used for fusion and classification.

The classification performance in distinguishing SZ/ASD and NC was displayed in Table 1, Table 2 and Supplementary Table 5-6. The proposed HMSDA achieved the best average classification performance and consistently achieved the highest accuracy for different given target domains, in which the highest accuracy was observed when using UCI and USM sites as the target domains. The declined performance when using other target domains may be caused by the increased complexity of the multi-site data. HMSDA exhibited superior classification performance when comparing with LRR and MSLRDA, indicating that the inclusion of hypergraph regularization enables the utilization of high-order relationships among subjects. Thus, the proposed method can effectively alleviate inter-site heterogeneity, and achieve more robust and stable classification performance.

Table 1.

Average classification performance (STD) of different DA methods across all sites in classifying SZ and NC.

Method	ACC	SEN	SPE	BAC	PPV	NPV	AUC
LRR (Liu et al., 2013)	0.8471 (0.0277)	0.8372 (0.0716)	0.8629 (0.0353)	0.8481 (0.0225)	0.8571 (0.0490)	0.8432 (0.0466)	0.6172 (0.0632)
maLRR (Wang et al., 2020a)	0.8146 (0.0405)	0.8347 (0.0449)	0.8106 (0.0438)	0.8226 (0.0423)	0.8083 (0.0449)	0.8290 (0.0416)	0.5937 (0.0701)
TSL_LRSR (Xu et al., 2016)	0.7926 (0.0218)	0.8493 (0.0778)	0.7311 (0.1159)	0.7902 (0.0265)	0.7644 (0.0698)	0.8116 (0.0580)	0.5680 (0.0529)
GFK (Gong et al., 2012)	0.7846 (0.0453)	0.7980 (0.0888)	0.7712 (0.0723)	0.7846 (0.0455)	0.7772 (0.0579)	0.8009 (0.0702)	0.6134 (0.0671)
LRCDR (Liu et al., 2023)	0.8321 (0.0247)	0.8445 (0.0733)	0.8167 (0.0601)	0.8165 (0.0494)	0.8014 (0.0515)	0.8366 (0.0645)	0.5948 (0.0592)
MSLRDA (Wang et al., 2020b)	0.8530 (0.0269)	0.8540 (0.0562)	0.8465 (0.0432)	0.8486 (0.0270)	0.8422 (0.0292)	0.8729 (0.0557)	0.6249 (0.0780)
Ours	0.8686 (0.0378)	0.8651 (0.0891)	0.8788 (0.0428)	0.8719 (0.0447)	0.8827 (0.0444)	0.8716 (0.0730)	0.6247 (0.0665)

Table 2.

Average classification performance (STD) of different DA methods across all sites in classifying ASD and NC.

Method	ACC	SEN	SPE	BAC	PPV	NPV	AUC
LRR (Liu et al., 2013)	0.7430 (0.0871)	0.7299 (0.1202)	0.7625 (0.1143)	0.7462 (0.0866)	0.7650 (0.1152)	0.7549 (0.0857)	0.5888 (0.1953)
maLRR (Wang et al., 2020a)	0.7060 (0.0678)	0.7211 (0.0630)	0.7032 (0.0856)	0.7122 (0.0641)	0.7175 (0.0669)	0.7326 (0.0583)	0.5688 (0.1904)
TSL_LRSR (Xu et al., 2016)	0.6775 (0.0804)	0.7821 (0.1176)	0.4924 (0.2018)	0.6599 (0.0533)	0.6739 (0.1427)	0.7375 (0.0467)	0.4715 (0.1961)
GFK (Gong et al., 2012)	0.6450 (0.0500)	0.7000 (0.2129)	0.5900 (0.2126)	0.6450 (0.0500)	0.6463 (0.0884)	0.6978 (0.1097)	0.6060 (0.3201)
LRCDR (Liu et al., 2023)	0.7060 (0.0602)	0.7192 (0.0434)	0.6150 (0.1276)	0.6897 (0.0756)	0.6966 (0.0941)	0.7005 (0.0386)	0.5670 (0.1609)
MSLRDA (Wang et al., 2020b)	0.7420 (0.0654)	0.7209 (0.0438)	0.7761 (0.1114)	0.7485 (0.0627)	0.7723 (0.1095)	0.7399 (0.0499)	0.6056 (0.1735)
Ours	0.7653 (0.0722)	0.7793 (0.0848)	0.7689 (0.0866)	0.7741 (0.0744)	0.7751 (0.0805)	0.7720 (0.0700)	0.6192 (0.1608)

4.3. Comparison with 10 multi-modality feature selection methods

To verify the effectiveness of the proposed LOSMFS method, we compared it with several multi-modality feature selection methods, including: (1) FC: t-test2 with p < 0.05; (2) SC: t-test2 with p < 0.05; (3) t-test: multi-kernel with t-test; (4) Lasso: multi-kernel with Lasso; (5) Multi-task multi-modality feature selection (M3L) (Zhang et al., 2012): multi-kernel with multi-task; (6) Manifold regularized multi-task multi-modality feature selection (M2TFS) (Jie et al., 2015): multi-kernel with manifold regularization and multi-task; (7) Hypergraph based multi-task multi-modality feature selection (HMTFS) (Shao et al., 2020): multi-kernel with hypergraph-based regularization and multi-task; (8) Discriminative multi-task multi-modality feature selection (DMTFS) (Ye et al., 2016): multi-kernel with discriminative regularization term and multi-task; (9) Clustered multi-task multi-modality feature selection (CMTFS) (Jacob et al., 2008): multi-kernel with a new spectral norm and multi-task; (10) Adaptive-similarity-based multi-modality feature selection (ASMFS) (Shi et al., 2022b): multi-kernel with a similarity matrix learned from different modalities and multi-task. The regularization parameters of Lasso, M3L, M2TFS, HMTFS, DMTFS, CMTFS and ASMFS were selected from [1e^-5, 1e^-4, 1e^-3, 1e^-2, 1e^-1], the parameter for discriminative regularization term in DMTFS and the number of neighbors in ASMFS were chosen from [0, 0.1, 0.2, 0.3, 0.4] and [1, 3, 5] via 5-fold cross-validation, respectively. The neighbor size in the k nearest vertices in HMTFS was 7 and the neighbor size in CMTFS was 5.

The proposed LOSMFS was compared with other multi-modality feature selection methods without DA procedures. As shown in Fig. 2, LOSMFS achieved the best accuracy in two classification tasks. FC and SC obtained the lowest accuracy since only single modality was used for classification. Other methods utilized inter-modality information from different modalities greatly improving the accuracy, indicating the necessity of multi-modality fusion for classification. M2TFS, HMTFS, DMTFS and ASMFS resulted in high classification performance in the two classification tasks, indicating that maintaining inter-modality information in different ways is effective for feature selection.

Fig. 2.

Comparison of 10 multi-modality feature selection methods for classifying SZ/ASD and NC without DA procedures.

Then, the combination of HMSDA and different multi-modality feature selection methods was used in classifying SZ/ASD and NC (Fig. 3, Fig. 4). The proposed method improved the accuracy by 2 % comparing with the best competing DMTFS (Fig. 3f) and achieved a 1.47 % increase comparing with the best competing M2TFS (Fig. 4a). The incorporation of local ordinal structure proves to be highly beneficial in improving the classification performance. The 4.37 % (Fig. 4a), 4.5 % (Fig. 4b), 15.7 % (Fig. 4c) and 9.7 % (Fig. 4d) increase in accuracy were observed compared with Fig. 2, demonstrating the superiority of jointly utilizing DA and multi-modality feature selection in multi-site SZ/ASD diagnosis.

Fig. 3.

Comparison of 10 multi-modality feature selection methods for SZ and NC classification. Each domain was alternatively used as the target domain, while the remaining domains were regarded as source domains.

Fig. 4.

Comparison of 10 multi-modality feature selection methods for ASD and NC classification.

4.4. Multi-modality single-site classification

We further investigated the effect of multi-modality single-site classification by running the proposed feature selection method on each site (Supplementary Table 7-8). It is evident that the classification performance varied across different sites, with the best results achieved in UI and USM datasets. This observation supports the notion that data heterogeneity among various sites poses challenges in improving classification performance using multi-site data.

4.5. Comparison with 8 state-of-the-art multi-site and multi-modality methods

The proposed HMSDA+LOSMFS was also compared with 8 state-of-the-art multi-site and multi-modality methods: including traditional machine learning methods (1) Dempster Shafer DA-optimal transport (DS-OT) (Shi et al., 2022a), (2) optimal transport-based pyramid graph kernel (OTPGK) (Ma et al., 2023); and deep learning methods (3) Convolutional neural network (CNN) (Sherkatghanad et al., 2020), (4) denoising autoencoder (DAE) (Heinsfeld et al., 2018), (5) variational autoencoder (VAE) (Zhang et al., 2022), (6) deep transfer learning neural network (DT-LNN) (Li et al., 2018), (7) privacy-preserving multi-source DA (PPMDA) (Han et al., 2022) and (8) multi-level FC fusion classification framework (MFC) (Liang and Xu, 2022). ABIDE I dataset on the USM site was used to make comparisons (Table 3). It is clear that HMSDA+LOSMFS generally outperformed the 8 state-of-the-art multi-site and multi-modality methods.

Table 3.

Comparison with 8 state-of-the-art multi-site and multi-modality methods. (HC: High-order connections).

Modality	Methods	Subjects	Feature type	ACC	SEN	SPE	AUC
Single	DS-OT (Shi et al., 2022a)	38ASDs 23NCs	FC	0.7541	0.7391	0.7632	−
	OTPGK (Ma et al., 2023)	38ASDs 22NCs	FC	0.7830	−	−	0.7560
	CNN (Sherkatghanad et al., 2020)	46ASDs 25NCs	FC	0.7700	0.8000	0.7200	0.6900
	DAE (Heinsfeld et al., 2018)	46ASDs 25NCs	FC	0.6400	0.6900	0.5900	−
	VAE (Zhang et al., 2022)	46ASDs 25NCs	FC	0.8169	−	−	−
	DT-LNN (Li et al., 2018)	38ASDs 23NCs	FC	0.7040	0.7250	0.6700	0.7300
	PPMDA (Han et al., 2022)	33ASDs 19NCs	FC	0.6785	−	−	−
Multi	MFC (Liang and Xu, 2022)	46ASDs 25NCs	FC+HC	0.7610	0.7610	0.7600	−
	Ours	25ASDs 25NCs	FC+SC	0.8480	0.8316	0.8942	0.7611

4.6. Parameter tuning

There are four hyperparameters (i.e., τ, λ, α and β) to be tuned in the proposed HMSDA+LOSMFS. To study the influence of these parameters on the classification performance, we further implemented experiments for the classification of SZ/ASD and NC under different combinations of parameters. Using each site as the target domain and the remaining ones as source domains, we conducted the planned experiments. To do this we independently varied the values of τ, λ, α and β from the range [1e^-5, 1e^-4, 1e^-3, 1e^-2, 1e^-1], where the corresponding accuracy was the averaged value calculated under the 5-fold cross-validation strategy. Supplementary Fig. 2-11 showed the classification results in FBIRN and ABIDE I dataset, where we fixed one parameter and varied the values of the other three parameters. As shown in Supplementary Fig. 2-11, the classification performance slightly fluctuated within a small range (2 %-6.06 % in SZ vs. NC and 2.48 %-5.72 % in ASD vs. NC). In most cases, the classification results of the proposed HMSDA+LOSMFS were stable with respect to different parameters, demonstrating that our method was fairly insensitive to hyperparameters. In addition, we also used 100, 200, 400, 800, 2000 and 4005 (no dimensionality reduction) to discuss the performance of the proposed method. As shown in Supplementary Table 9-10, the classification performance achieved the highest when no dimensionality reduction was applied (i.e., when the feature dimension is 4005). It sounds reasonable that dimensionality reduction may lead to the loss of relevant information related to the classification task, resulting in decreased performance.

4.7. The identified disorder-specific diagnostic connections

The discriminability of the identified brain connection is ranked by the regression coefficient matrix W. Fig. 5, Fig. 6 showed the top 10 connections in SZ/ASD and NC classification. In SZ vs. NC classification, several brain connections were consistently identified over different target domains, including left fusiform gyrus-to-right lenticular nucleus, pallidum (FFG.L-PAL.R), FFG.L-to-left thalamus (FFG.L-THA.L), left lenticular nucleus, putamen-to-left precuneus (PUT.L-PCUN.L), left inferior temporal gyrus-to-left inferior occipital gyrus (ITG.L-IOG.L), right middle frontal gyrus, orbital part-to-right inferior occipital gyrus (ORBmid.R-IOG.R), right lingual gyrus-to-right calcarine fissure and surrounding cortex (LING.R-CAL.R), and left angular gyrus-to-right lenticular nucleus, putamen (ANG.L-PUT.R). In ASD vs. NC classification, the consistently identified brain connections across different target domains, including left hippocampus-to-left superior occipital gyrus (HIP.L-SOG.L), left rolandic operculum-to-CAL.R (ROL.L-CAL.R), left anterior cingulate and paracingulate gyri-to-left Supplementary motor area (ACG.L-SMA.L), left supramarginal gyrus-to-right gyrus rectus (SMG.L-REC.R), left superior frontal gyrus, dorsolateral-to-ANG.L (SFGdor.L-ANG.L), left temporal pole: superior temporal gyrus-to-left paracentral lobule (TPOsup.L-PCL.L), and PUT.L-to-ACG.L.

Fig. 5.

Brain discriminative connections identified in SZ vs. NC classification. These results were generated by our method using different target domains.

Fig. 6.

Brain discriminative connections identified in ASD vs. NC classification. These results were generated by our method using different target domains.

5. Discussion

In this paper, we proposed a joint multi-site and multi-modality framework, named HMSDA+LOSMFS, for psychiatric disorders’ diagnosis. Our method includes two novel aspects: (1) a novel HMSDA was proposed to transform subjects from various source domains into a common target feature space, aligning the source subjects to have the same distribution as the target subjects; (2) a novel LOSMFS was developed for feature selection. The experimental results on the diagnosis of both SZ and ASD showed that our method outperformed 6 DA, 10 multi-modality feature selection and 8 multi-site and multi-modality methods. This approach can improve the multi-site detection of multi-modality biomarkers for brain disorders.

One major contribution is that a hypergraph-based regularization was introduced to explicitly depict the high-order relationship in source domains after transforming in HMSDA. Existing multi-source DA methods (Wang et al., 2020b, Zhang and Zhang, 2016) only considered the pairwise relationship between labels and samples. However, such pairwise relationship does not reflect the complex connections of multi-site subjects. Hence, we employed hypergraph learning and defined a hypergraph similarity matrix to characterize the high-order relationships among multi-site subjects. As demonstrated in Table 1, Table 2, incorporating such relationships can greatly improve the classification performance.

Another contribution is that a triplet-based local ordinal structure regularization was added to reveal the underlying ranking information of multi-site subjects from different classes, and jointly selected the common features in different modalities. Most of the previous methods focused on the relationships between labels and samples or the simple dependence between samples after mapping the original feature space into a new space, ignoring the discriminative information among samples (Zhu et al., 2022a). Our method accurately preserved intrinsic structural information among subjects in each modality by retaining ranking information within each sample's neighborhood. Moreover, it adaptively maintained compactness within the same distribution while extracting discriminative features. As shown in Fig. 3, Fig. 4, preserving the local ordinal structure of data in feature selection yielded competitive performance when compared to multi-modality feature selection methods.

Besides the performance improvement in classification, we found that the brain connections selected by our method were consistent with exiting studies. The FCs within temporal-subcortical have been reported as biomarkers for classification of SZ (Huang et al., 2022, Zhu et al., 2019, Yu et al., 2023). Note that the temporal cortex has consistently been shown to play an important role in discriminating SZ patients (Rashid et al., 2016). The FCs contributing most to the classification of SZ were identified within subcortical-parietal connection (Zhuang et al., 2019). In addition, the frontal-parietal connection captured the functional distinctiveness in the diagnosis of ASD (Bi et al., 2018, Guo et al., 2017). Discriminative SC patterns in identification of infants at high-risk for ASD are temporal-occipital and frontal-occipital connections (Jin et al., 2015). These results indicate a degree of consistency in the discriminative connections identified across multi-site data using our method.

Despite its promising performance, our method still has some limitations. First, the high dimension of features and the singular value decomposition will increase the computational burden for HMSDA. This can be addressed through parallel computing and the current advanced computing server clusters. Second, the DA and multi-modality feature selection were optimized in two separate steps, leading to the local optimal solution for HMSDA+LOSMFS. A globally optimized framework that integrates DA, feature selection and classifier training need to be considered in future improvements. Third, our study mainly focused on neuroimaging data for the classification, although the inclusion of demographic and other clinical information could potentially enhance the feature selection domains.

In this paper, we proposed a novel multi-site and multi-modality fusion method, HMSDA+LOSMFS, for classifying psychiatric disorders. The effectiveness of our method was validated on the diagnosis of both SZ and ASD. The proposed method obtained the highest accuracy for the diagnosis of SZ and ASD, respectively, comparing with 6 DA, 10 multi-modality feature selection and 8 multi-site and multi-modality methods. Collectively, the proposed HMSDA+LOSMFS effectively integrates the multi-site and multi-modality data to enhance psychiatric diagnosis and identify disorder-specific diagnostic brain connections.

Availability of data and material

The FBIRN multi-modal data used in the present study can be accessed upon request to the corresponding authors. The ABIDE I multi-modal data used in the present study can be accessed in https://fcon_1000.projects.nitrc.org/indi/abide/abide_I.html.

Funding

This work was supported by the National Natural Science Foundation of China (62376124, 62136004, 62276130), the Natural Science Foundation of Jiangsu Province, China (BK20220889), the National Key Research and Development Program of China (2023YFF1204803) and the Key Research and Development Plan of Jiangsu Province, China (BE2023668, BE2022842).

CRediT authorship contribution statement

Yixin Ji: Writing – original draft, Software, Methodology, Conceptualization. Rogers F. Silva: Writing – review & editing, Supervision. Tülay Adali: Writing – review & editing. Xuyun Wen: Writing – review & editing. Qi Zhu: Writing – review & editing. Rongtao Jiang: Writing – review & editing. Daoqiang Zhang: Writing – review & editing, Supervision. Shile Qi: Writing – review & editing, Supervision, Funding acquisition. Vince D. Calhoun: Writing – review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following are the Supplementary data to this article:

Data availability

Data will be made available on request.