Progressive Graph-Based Transductive Learning for Multi-modal Classification of Brain Disorder Disease

Abstract

Graph-based Transductive Learning (GTL) is a powerful tool in computer-assisted diagnosis, especially when the training data is not sufficient to build reliable classifiers. Conventional GTL approaches first construct a fixed subject-wise graph based on the similarities of observed features (i.e., extracted from imaging data) in the feature domain, and then follow the established graph to propagate the existing labels from training to testing data in the label domain. However, such a graph is exclusively learned in the feature domain and may not be necessarily optimal in the label domain. This may eventually undermine the classification accuracy. To address this issue, we propose a progressive GTL (pGTL) method to progressively find an intrinsic data representation. To achieve this, our pGTL method iteratively (1) refines the subject-wise relationships observed in the feature domain using the learned intrinsic data representation in the label domain, (2) updates the intrinsic data representation from the refined subject-wise relationships, and (3) verifies the intrinsic data representation on the training data, in order to guarantee an optimal classification on the new testing data. Furthermore, we extend our pGTL to incorporate multi-modal imaging data, to improve the classification accuracy and robustness as multi-modal imaging data can provide complementary information. Promising classification results in identifying Alzheimer’s disease (AD), Mild Cognitive Impairment (MCI), and Normal Control (NC) subjects are achieved using MRI and PET data.

1 Introduction

Alzheimer’s disease (AD) is the most common neurological disorder in the older population. There is overwhelming evidence in the literature that the morphological patterns are observable by means of either structural and diffusion MRI or PET [1–3]. However, morphological abnormal patterns are often subtle, compared to high inter-subject variations. Hence, sophisticated pattern recognition methods are of high demand to accurately identify individuals at different stages of AD progression.

Medical imaging applications often deal with high dimensional data and usually less number of samples with ground-truth labels. Thus, it is very challenging to find a general model that can work well for an entire set of data. Hence, GTL method has been investigated with great success in medical imaging area [4, 5], since it can overcome the above difficulties by taking advantage of the data representation on unlabeled testing subjects. In current state-of-the-art methods, graph is used to represent the subject-wise relationship. Specifically, each subject, regardless of being labeled or unlabeled, is treated as a graph node. Two subjects are connected by a graph link (i.e., an edge) if they have similar morphological patterns. Using these connections, the labels can be propagated throughout the graph until all latent labels are determined. Many current label propagation strategies have been proposed to determine the latent labels of testing subjects based on subject-wise relationships encoded in the graph [6].

The assumption of current methods is that the graph constructed in the observed feature domain represents the real data distribution and can be transferred to guide label propagation. However, this assumption usually does not hold since morphological patterns are often highly complex and heterogeneous. Figure 1(a) shows the affinity matrix of 51 AD and 52 NC subjects using the ROI-based features extracted from each MR image, where red dot and blue dot denote high and low subject-wise similarities, respectively. Since the clinical data (e.g., MMSE and CDR scores [1]) is more related with clinical labels, we use these clinical scores to construct another affinity matrix, as shown in Fig. 1(c). It is apparent that the data representations using structural image features and clinical scores are completely different. Thus, there is no guarantee that the learned graph from the affinity matrix in Fig. 1(a) can effectively guide the classification of AD and NC subjects. More critically, the affinity matrix using observed image features is not even necessarily optimal in the feature domain, due to possible imaging noises and outlier subjects. Many studies take advantage of multi-modal information to improve discrimination power of transductive learning. However, the graphs from different modalities might be different too, as shown in the affinity matrices using structural image features from MR images (Fig. 1(a)) and functional image features from PET images (Fig. 1(b)). Graph diffusion [5] is recently proposed to find the common graph. Unfortunately, as shown in Fig. 1, it is hard to find a combination for the graphs in Fig. 1(a) and (b) that can lead to the graph in Fig. 1(c), which is more related with final classification task.

Fig. 1.

Affinity matrices using structural image features (a), functional image features (b), and clinical scores (c).

To solve these issues, we propose a pGTL method to learn the intrinsic data representation, which could be eventually optimal for label propagation. Specifically, the intrinsic data representation is required to be (a) close to subject-wise relationships constructed by image features extracted from different modalities, and (b) verified on the training data and guaranteed to be optimal for label classification. To that end, we simultaneously (1) refine the data representation (subject-wise graph) in the feature domain, (2) find the intrinsic data representation based on the constructed graphs on multi-modal imaging data and also the clinical labels of entire subject set (including known labels on training subjects and also the tentatively-determined labels on testing subjects), and (3) propagate the clinical labels from training subjects to testing subjects, following the latest learned intrinsic data representation. Promising classification results have been achieved in classifying 93AD, 202 MCI, and 101NC subjects, each with MR and PET images.

2 Methods

Suppose we have N subjects {I₁, . . ., I_P, I_P+1, . . ., I_N}, which sequentially consist of P training subjects and Q(= N − P) testing subjects. For P training subjects, the clinical labels F_P = [f_p]_p=1,...,P are known, where each f_p ∈ [0, 1]^C is a binary coding vector indicating the clinical label from C classes. Our goal is to jointly determine the latent labels for Q testing subjects based on a set of their continuous likelihood vectors F_Q = [f_q]_q=P+1,...,N, where each element in vector f_q indicates the likelihood of the q-th subject belonging to one of C classes. For convenience, we concatenate F_P and F_Q into a single label matrix F^N×C = [F_PF_Q].

2.1 Progressive Graph-Based Transductive Learning

Conventional Graph-Based Transductive Learning

For clarity, we first extract single modality image features from each subject I_i (i = 1, . . ., N), denoted as x_i. In conventional GTL methods, the subject-wise relationships are computed based on feature similarity, which is encoded in an N × N feature affinity matrix S. Each element s_ij (0≤s_ij ≤1, i, j = 1, . . ., N) represents the feature affinity degree between x_i and x_j. After constructing S (based on feature similarity), conventional methods determine the latent label for each testing subject I_q by solving a classic graph learning problem:

$${\widehat{\mathbf{F}}}_q=arg{min}_{{\mathbf{F}}_q}{\sum }_{i,j=1}^N{\Vert {\mathbf{f}}_i-{\mathbf{f}}_j\Vert }_2^2{s}_{ij}.$$ (1)

As shown in Fig. 1, the affinity matrix S might not be strongly related with the intrinsic data representation in the label domain. Therefore, it is necessary to further design a graph based on the labels matrix, rather than solely using the graph constructed by the features. However, the labels on testing subjects are not determined yet. In order to solve this chicken-and-egg dilemma, we propose to construct a dynamic graph which progressively reflects the intrinsic data representation in the label domain.

Progressive Graph-Based Transductive Learning on Single Modality

We propose three strategies to remedy the above issue. (1) We propose to gradually find an intrinsic data representation T = [t_ij]_i,j=1,...,N which is more relevant than S to guide the label propagation in Eq. (1). (2) Since only the training images have their known clinical labels, exclusively optimizing T in the label domain is an ill-posed problem. Thus, we encourage the intrinsic data presentation T also respecting the affinity matrix S, where image features are complete in the feature domain. (3) In order to suppress possible noisy patterns and outlier subjects, we allow the intrinsic data representation T to progressively refine the affinity matrix S in the feature domain. In this way, the estimations of S and T are coupled, thus bringing a dynamic graph learning model with the following objective function:

$$\begin{array}{c}arg{min}_{\mathbf{S},\mathbf{T},\mathbf{F}}{\sum }_{i,j=1}^N\left\{\mu {\Vert {\mathbf{f}}_i-{\mathbf{f}}_j\Vert }_2^2{t}_{ij}+{\Vert {\mathbf{x}}_i-{\mathbf{x}}_j\Vert }_2^2{s}_{ij}+{\lambda }_1{s}_{ij}^2+{\lambda }_2{\Vert s_{ij}-t_{ij}\Vert }_2^2\right\}\\ s.t.0\le s_{ij}\le 1,{\mathbf{s}}_i^{\prime }1=1,0\le t_{ij}\le 1,{\mathbf{t}}_i^{\prime }1=1,\mathbf{F}=[{\mathbf{F}}_P{\mathbf{F}}_Q]\end{array}$$ (2)

where μ is the scalar balancing the data fitting terms from two different domains (i.e., the first and second terms in Eq. (2)). Suppose s_i ∈ R^N×1 and t_i ∈ R^N×1 are vectors with the j-th element as s_ij and t_ij separately. In order to avoid trivial solution, l₂-norm is used as the constraint on each element s_ij in affinity matrix S. λ₁ and λ₂ are two scalars to control the strengths of the last two terms in Eq. (2).

Progressive Graph-Based Transductive Learning on Multiple Modalities

Suppose we have M modalities. For each subject I_i, we can extract multi-modal image features ${\mathbf{x}}_i^m$, m = 1, . . .,M. For m-th modality, we optimize the affinity matrix S^m. As shown in Fig. 1(a) and (b), the affinity matrices across modalities could be different. Thus, we require the intrinsic data representation T to be close to all S^m, m = 1, . . .,M. It is straightforward to extend our above pGTL method to the multi-modal scenario:

$$\begin{array}{c}arg{min}_{{\mathbf{S}}^m,\mathbf{T},\mathbf{F}}\sum _{i,j=1}^N\left\{\mu {\Vert {\mathbf{f}}_i-{\mathbf{f}}_j\Vert }_2^2{t}_{ij}+\sum _{m=1}^M\left[{\Vert {\mathbf{x}}_i^m-{\mathbf{x}}_j^m\Vert }_2^2{s}_{ij}^m+{\lambda }_1{\left(s_{ij}^m\right)}^2+{\lambda }_2{\Vert s_{ij}^m-t_{ij}\Vert }_2^2\right]\right\}\\ s.t.0\le {\mathrm{s}}_{\text{ij}}^{\mathrm{m}}\le 1,{\left({\mathbf{s}}_{\mathrm{i}}^{\mathrm{m}}\right)}^{\prime }1=1,0\le {\mathrm{t}}_{\text{ij}}\le 1,{\mathbf{t}}_i^{\prime }1=1,\mathbf{F}=[{\mathbf{F}}_{\mathrm{P}}{\mathbf{F}}_{\mathrm{Q}}]\end{array}$$ (3)

It is worth noting that, although the multi-modal information leads to multiple affinity matrices in the feature domain, they share the same intrinsic data representation T.

2.2 Optimization

Since our proposed energy function in Eq. (3) is convex to each variables, i.e. S, T, F, we present the following divide-and-conquer solution to optimize one set of variables at a time by fixing other sets of variable. We initialize $S=exp(-{\Vert x_i-x_j\Vert }_2^2/2{\sigma }^2)$, σ is an empirical parameter, $\mathbf{T}={\sum }_{m=1}^M{\mathbf{S}}^m/M$, F_Q = {0}^{Q × C}.

Estimation of Affinity Matrix S^m for Each Modality

Removing the unrelated terms w.r.t. S^m in Eq. (3), the optimization of S^m falls to the following objective function:

$$arg{min}_{{\mathbf{S}}^m}{\sum }_{i,j=1}^N{\Vert {\mathbf{x}}_i^m-{\mathbf{x}}_j^m\Vert }_2^2{s}_{ij}^m+{\lambda }_1{\left(s_{ij}^m\right)}^2+{\lambda }_2{\sum }_{i,j=1}^N{\Vert s_{ij}^m-t_{ij}\Vert }_2^2$$ (4)

where (0 ≤ s_ij ≤ 1, ${\left({\mathbf{s}}_i^m\right)}^{\prime }1=1$). Since Eq. (4) is independent of variables i and j, we further reformulate Eq. (4) in the vector form as below:

$$arg{min}_{{\mathbf{s}}_i^m}{\Vert {\mathbf{s}}_i^m+\frac{{\mathbf{d}}_i}{2r_1}\Vert }_2^2$$ (5)

where ${\mathbf{s}}_i^m$ is the i-th column vector of affinity matrix S^m, d_i = [d_ij]_j=1,...,N is a vector with each $d_{ij}={\Vert {\mathbf{x}}_i^m-{\mathbf{x}}_j^m\Vert }^2-2{\lambda }_2{t}_{ij}$, and r₁ = λ₁ +λ₂. The problem in Eq. (5) is equivalent to project onto a simplex, which has a closed-form solution in [7]. After we solve each ${\mathbf{s}}_i^m$, we can obtain the affinity matrix S^m.

Estimate the Intrinsic Data Representation T

Fixing S^m and F, the objective function w.r.t. T reduces to:

$$arg{min}_{\mathbf{T}}{\sum }_{i,j=1}^N\mu {\Vert {\mathbf{f}}_i-{\mathbf{f}}_j\Vert }_2^2{t}_{ij}+{\lambda }_2{\sum }_{m=1}^M{\sum }_{i,j=1}^N\left({\Vert s_{ij}^m-t_{ij}\Vert }_2^2\right)$$ (6)

Similarly, we can reformulate Eq. (6) by solving each t_i at a time:

$$arg{min}_{{\mathbf{t}}_i}{\Vert {\mathbf{t}}_i+\frac{{\mathbf{h}}_i}{2r_2}\Vert }_2^2$$ (7)

where h_i = [h_ij]_j=1,...,N is a vector with each element $h_{ij}=\mu {\Vert {\mathbf{f}}_i-{\mathbf{f}}_j\Vert }_2^2-2{\lambda }_2{\sum }_{m=1}^M{\mathbf{s}}_i^m$, and r₂ = Mλ₂ is a scalar.

Update the Latent Labels F_Q on Testing Subjects

Given both S^w and T, the objective function for the latent label F_Q can be derived from Eq. (3) as below:

$$arg{min}_{\mathbf{F}}{\sum }_{i,j=1}^N{\Vert {\mathbf{f}}_i-{\mathbf{f}}_j\Vert }_2^2{t}_{ij}\Rightarrow arg{min}_{\mathbf{F}}\text{Trace}({\mathbf{F}}^{\prime }\mathbf{LF}),$$ (8)

where Trace(.) denotes the matrix trace operator, L = diag(T) − (T′+T)/2 is the Laplacian matrix of T. By differentiating Eq. (8) w.r.t. F and letting the gradient LF = 0, we obtain the following equation: $\left[\begin{array}{ll}{\mathbf{L}}_{PP}\hfill & {\mathbf{L}}_{PQ}\hfill \\ {\mathbf{L}}_{QP}\hfill & {\mathbf{L}}_{QQ}\hfill \end{array}\right]\left[\begin{array}{l}{\mathbf{F}}_P\hfill \\ {\mathbf{F}}_Q\hfill \end{array}\right]=0$, where L_PP, L_PQ, L_QP, and L_QQ denote the top-left, top-right, bottom-left, and bottom-right blocks of L. The solution for F_Q can be obtained by F̂_Q = −(L_QQ)^{− 1}L_QPF_P.

Discussion

Taking MRI and PET modalities as example, Fig. 2(a) illustrates the optimization of Eq. (3) by alternating the following three steps. (1) Estimate each affinity matrix S^m, which depends on the observed image features x^m and the currently estimated intrinsic data representation T (red arrows); (2) Estimate the intrinsic data representation T, which requires the estimations of both S¹ and S² and also the subject-wise relationship in the label domain (purple arrows); (3) Update the latent labels F_Q on the testing subjects which needs guidance from the learned intrinsic data representation T (blue arrows). It is apparent that the intrinsic data representation T links the feature domain and label domain, which eventually leads to the dynamic graph learning model.

Fig. 2.

(a) The dynamic procedure of the proposed pGTL method, (b) Classification accuracy as a function of the number of training samples used.

3 Experiments

Subject Information and Image Processing

In the following experiments, we select 93 AD subjects, 202 MCI subjects, and 101 NC subjects from ADNI dataset. Since MCI is a highly heterogeneous group, we further separate them into 55 progressive MCI subjects (pMCI), who will finally develop into AD patients within the next 24 months, and 63 stable MCI subjects (sMCI), who won’t convert to AD after 24 months. The remain MCI subjects included a group not converted in 24 months but converted in 36 months and another group with observation information in baseline but missing information in 24 months. Each subject has both MR and 18-Fluoro-DeoxyGlucose PET (FDG-PET) images.

For each subject, we first align the PET image to MR image. Then we remove the skull and cerebellum from MR image and segment MR image into white matter, gray matter and cerebrospinal fluid. Next, we parcellate each subject image into 93 ROIs (Regions of Interest) by registering the template (with manual annotation of 93 ROIs) to the subject image domain. Finally, the gray matter volume and the mean PET intensity image in each ROI are used and form a 186-dimensional feature vector.

Experiment Settings

First, we evaluate our proposed pGTL method, with comparison to classic classification methods, such as Canonical Correlation Analysis (CCA) [8] based SVM (denoted as CCA in the following context), Multi-Kernel SVM (MKSVM) [9], and a conventional GTL method, since these methods are widely used in AD studies. In order to demonstrate the overall performance of our method in several classification tasks, i.e. AD vs NC, MCI vs NC, and pMCI vs sMCI, in each experiment, we use 10-fold cross-validation strategy, with 9 folds of data as training dataset and the remaining 1 fold as testing dataset. Second, we compare our proposed method with three recently published state-of-the-art classification methods: (1) random-forest based classification method [10], (2) multi-modal graph-fusion method [4], and (3) multi-modal deep learning method [11]. It is worth noting that we only use the classification accuracy reported in their papers, in order for fair comparison.

Parameter Settings

In the following experiments, we use the same greedy strategy to select best parameters for CCA, MKSVM and our proposed method. For example, we obtain the optimal values for μ, λ₁ and λ₂ in our method by exhaustive search in the range from 10⁻³ to 10³ in a small portion of training dataset.

Comparison with Classic CCA, GTL and Multi-kernel SVM (MKSVM)

The classification accuracies by CCA, MKSVM, GTL and our method are evaluated in three classification tasks (AD vs NC, MCI vs NC, and pMCI vs sMCI), respectively. The averaged classification accuracy (ACC), sensitivity (SEN), and specificity (SPE) with 10-fold cross-validation are summarized in Table 1. It is clear that our proposed method beats other competing classification methods in three classification tasks, with significant improvement under paired t-test (p < 0.001, designated by ‘*’ in Table 1).

Table 1.

Comparison of classification performance by different methods.

Classification Task	Method	ACC (%)	SEN (%)	SPE (%)
AD vs NC		89.1 ± 1.57	87.6 ± 2.02	90.5 ± 1.25
		90.0 ± 1.03	89.1 ± 1.53	90.7 ± 1.28
		88.0 ± 1.27	90.4 ± 1.59	85.8 ± 1.90
		92.6 ± 0.65	92.2 ± 1.34	92.9 ± 1.37
MCI vs NC		68.3 ± 1.95	78.8 ± 1.92	47.3 ± 3.16
		72.6 ± 1.87	84.5 ± 1.70	48.9 ± 3.57
		71.9 ±0.94	92.8 ±0.93	29.9 ± 2.31
		78.6 ± 1.19	85.1 ± 2.18	65.4 ± 3.35
pMCI vs sMCI		64.7 ± 1.39	41.5 ± 3.30	84.9 ± 3.20
		67.5 ±2.79	57.6 ± 5.75	76.2 ± 5.42
		67.7 ± 1.27	57.8 ± 2.85	76.5 ± 2.33
		76.0 ± 1.19	75.6 ± 2.30	76.3 ± 3.00

Furthermore, we evaluate the classification performance w.r.t. the number of training samples, as shown in Fig. 2(b). It is clear that (1) our proposed method always has higher classification accuracy than both CCA and MKSVM methods; and (2) all methods can improve the classification accuracy as the number of training samples increases. It is worth nothing that our proposed method achieves large improvement against MKSVM, when only 10 % of data is used as the training dataset. The reason is that supervised methods require a sufficient number of samples to train the reliable classifier. Since the training samples with known labels are expensive to collect in medical imaging area, this experiment indicates that our method has high potential to be deployed in current neuroimaging studies.

Comparison with Recently Published State-of-the-Art Methods

Table 2 summarizes the subject information, imaging modality, and average classification accuracy by using state-of-the-art methods. These comparison methods represent four typical machine learning techniques. Since the classification between pMCI and sMCI groups are not reported in [4, 10, 11], we only show the classification results for AD vs NC, and MCI vs NC tasks. Our method achieves higher classification accuracy than both random forest and graph fusion methods, even though those two methods use additional CSF and genetic information.

Table 2.

Comparison with the classification accuracies reported in the literatures (%).

Method	Subject information	Modality	AD/NC	MCI/NC
Random forest [10]	37AD + 75MCI + 35NC	MRI + PET + CSF + Genetic	89.0	74.6
Graph fusion [4]	35AD + 75MCI + 77NC	MRI + PET + CSF + Genetic	91.8	79.5
Deep learning [11]	85AD + 169MCI + 77NC	MRI + PET	91.4	82.1
Our method	99AD + 202MCI + 101NC	MRI + PET	92.6	78.6

Discussion

Deep learning approach in [11] learns feature representation in a layer-by-layer manner. Thus, it is time consuming to re-train the deep neural-network from scratch. Instead, our proposed method only uses hand-crafted features for classification. It is noteworthy that we can complete the classification on a new dataset (including greedy parameter tuning) within three hours on a regular PC (8 CPU cores and 16 GB memory), which is much more economic than massive training cost in [11]. Complementary information in multi-modal data can help improve the classification performance, therefore, in order to find the intrinsic data representation, we combine our proposed pGTL with multi-modal information.

4 Conclusion

In this paper, we present a novel pGTL method to identify individual subject at different stages of AD progression, using multi-modal imaging data. Compared to conventional methods, our method seeks for the intrinsic data representation, which can be learned from the observed imaging features and simultaneously validated on the existing labels of training data. Since the learned intrinsic data presentation is more relevant to label propagation, our method achieves promising classification performance in AD vs NC, MCI vs NC, and pMCI vs sMCI tasks, after comprehensive comparison with classic and recent state-of-the-art methods.