Multi-task Dictionary Learning based on Convolutional Neural Networks for Longitudinal Clinical Score Predictions in Alzheimer’s Disease

Abstract

Computer-aided diagnosis (CAD) systems for medical images are seen as effective tools to improve the efficiency of diagnosis and prognosis of Alzheimers disease (AD). The current state-of-the-art models for many images analyzing tasks are based on Convolutional Neural Networks (CNN). However, the lack of training data is a common challenge in applying CNN to the diagnosis of AD and its prodromal stages. Another challenge for CAD applications is the controversy between the requiring of longitudinal cortical structural information for higher diagnosis/prognosis accuracy and the computing ability for processing varied imaging features. To address these two challenges, we propose a novel computer-aided AD diagnosis system CNN-Multitask Stochastic Coordinate Coding (MSCC) which integrates CNN with transfer learning strategy, a novel MSCC algorithm and our effective AD-related biomarkers–multivariate morphometry statistics (MMS). We applied the novel CNN-MSCC system on the Alzheimers Disease Neuroimaging Initiative (ADNI) dataset to predict future cognitive clinical measures with baseline Hippocampal/Ventricle MMS features and cortical thickness. The experimental results showed that CNN-MSCC achieved superior results. The proposed system may aid in expediting the diagnosis of AD progress, facilitating earlier clinical intervention, and resulting in improved clinical outcomes.

1. Introduction

AD and its early stage, Mild Cognitive Impairment (MCI), are becoming the most prevalent neurodegenerative brain diseases in elderly people worldwide [1]. Mini Mental State Examination (MMSE) and Alzheimers Disease Assessment Scale cognitive subscale (ADAS-Cog) are well-known AD assessment scales [2, 3]. Clinicians and researchers have used them to evaluate currently individual cognitive decline led by AD factors. It is valuable to predict future cognitive progression for an early intervention or prevention.

Brain changes due to AD occur even before amnestic symptoms appearing [4], AD Studies on magnetic resonance (MR) images have shown that objective brain structure measures, such as hippocampal, ventricle structures and cortical thickness, could identify significant cortical structural deformations related with AD pathology [5, 6] even before observing obviously lower MMSE/ADAS-Cog scores [7–9]. Our previous studies proposed a novel 3D surface measure, multivariate morphometry statistics (MMS), consisting of multivariate tensor-based morphometry (mTBM) and radial distance (distances from the medial core to each surface point) and demonstrated that MMS of ventricles and hippocampi are potential preclinical AD imaging biomarkers [10, 11]. Our previous work [12, 13] applied cortical thickness measures to predict future clinical scores related with AD symptoms. Seldom existing CAD systems considered the inherent correlations among the above effective brain structure measures that may be useful for robust and accurate predictions of MMSE/ADAS-Cog scales. So this work expect to develop a CAD system using these three structure measures to predict future MMSE/ADAS-Cog scales, and then by referring the scale trends, the clinicians can provide early intervention or prevention to slow or even stop the degenerate trends.

Deep learning models [14, 15] are capable of learning the hierarchical structure of features extracted from real-world images. Convolutional Neural Networks (CNNs) are a class of multi-layer, fully trainable models that are able to capture highly nonlinear mappings between inputs and outputs [16]. Recently, CNNs have been successfully applied to the domain of brain imaging analysis, including image classification [17], segmentation [18], and autistic diagnosis [19]. But there is still little CNNs studies on AD diagnosis, a key challenge is a lack of training data. Transfer learning (TF) can help feature learning in the data-scarce target domain by transferring knowledge from similar data-rich source domain, it is potential to address this challenge [15, 20].

After using CNNs with TF, we confront another challenge that is high dimensional feature maps derived from small number of individual MR images in AD research domain. To address this so called large p, small n problem, dictionary learning was proposed to use a small number of basis vectors termed dictionary to represent high dimensional features effectively and concisely [21–23]. However, most existing studies on dictionary learning focus on the prediction of target on a single brain structure measure [24, 25]. In general, a joint analysis of tasks from multiple cortical measures is expected to improve the performance but remains a challenging problem.

Recently, Multi-Task Learning (MTL) has been successfully used on regression under the different cortical structure measures [26]. Further, there have been studies to combine TF and MTL together to solve the issue of small sample size. Zhang et al. integrated CNN, transfer learning and MTL for biological image analysis [27]. These studies indicated that integrating abnormal features among different cortical structures performed better than using single type of structural measures. Based on MTL, we made a multi-task Stochastic Coordinate Coding (MSCC) algorithm to partition the dictionaries into the common and individual parts and more suitable for the situation of individual features of varied structural measures. And we proposed a CAD system CNN-MSCC which utilized TF strategy to get one initial CNN model pre-trained from millions of images in ImageNet dataset [28, 29], employed MSCC to refine and fuse features from varied structural measures, and performed the Lasso [30] to predict future MMSE/ADAS-Cog scales representing AD progression. The proposed algorithm CNN-MSCC aims to resolve the CNN application challenges of the limited sample size, high dimension feature maps refining and multiple sources of features integrations.

Our main contributions can be summarized as follows:

• –We employ transfer learning and CNN to explore whether the transfer learning property of CNN can be enhanced to generate features from geometry meshes of biological images since the current bottleneck for CNNs to be applied to many biological problems is the limited amount of available labeled training data. We pre-train the deep neural network on the ImageNet data and transfer the knowledge of natural images to generate the neuroimaging features for the real world application.
• –We considered the variance of subjects from different cortical structure measures and proposed a novel unsupervised dictionary learning method, termed Multi-task Stochastic Coordinate Coding (MSCC), learning the different tasks simultaneously and utilizing shared and individual dictionary to encode both consistent and varied imaging features. To the best of our knowledge, it is the first deep model to integrate multi-task learning with dictionary learning research for brain imaging analysis.
• –We tested CNN-MSCC on three baseline brain structure measures to better predict the future clinical cognitive scores. Specifically, we used multiple baseline structure measures as multiple tasks input to predict three future time points clinical scores. Our new approach is able to boost the performance of diagnoses ranging from cognitively unimpaired to AD.

2. Multi-task Dictionary Learning based Convolutional Neural Networks

Our first goal here is to explore whether this transfer learning framework of CNN can be generalized to biological image studies. Specifically, we pre-train the CNN model using ImageNet data [28], containing millions of labeled natural images with thousands of categories to obtain initial parameters and subsequently generate the features on the longitudinal data for each task. In the experiments, we apply Alexnet [17], which contains 7 layers, including convolutional layers with fixed filter sizes and different numbers of feature maps. We employ rectified non-linearity, max-pooling on each layer in our CNN model. We pretrain the CNN model on the ImageNet dataset, then remove the last fully-connected layer (this layer’s outputs are the 1000 class scores for a different task like ImageNet). Finally, we treat the rest of the CNN as a fixed feature extractor for the publicly available Alzheimer’s Disease Neuroimaging Initiative (ADNI) database [31].

We further propose to use multi-task learning strategy to boost the future clinical score regression accuracy. The entire pipeline of our method is illustrated in Fig. 1. To be specific, we train the deep CNN model on the Imagenet dataset firstly. Then we employ the pretrained network as a feature extractor for the ADNI dataset from multiple baseline structure measures. The AlexNet has a seven layer structure deep neural network. As a result, we generate seven-deep output features for each structure measure. We further employ MSCC to conduct the multi-task learning simultaneously, generating the sparse features and dictionaries from the deep features of different structure measures. In MSCC, we utilize shared and individual dictionaries to encode both consistent and varied imaging features along multiple structure measures. In the end, we employ the sparse codes generated from MSCC to perform the Lasso [30] and predict the future AD progression. MSCC is one kind of online learning methods and the advantage of online learning method is to solve the cases that the size of the input data might be too large (sample size up to 2867562 in this work) to fit into memory or the input data comes in a form of a stream.

Fig. 1.

The streamline of our proposed framework. We pre-train the deep CNN model on the Imagenet dataset and use the pre-trained model as a feature extractor for the ADNI dataset. We employ the extracted features from three cortical structure measures to conduct the multi-task dictionary learning for AD progression prediction, generating the sparse features for different structure measures. Finally, we use Lasso regression on the learnt features to predict future MMSE and ADAS-Cog scores.

3. Multi-task Stochastic Coordinate Coding

3.1. Dictionary Learning

Given a finite training set of signals $X=\left(x_1,\dots ,x_n\right)$ where $X\in {ℝ}^{p\times n}$. Each x_i is an image patch and $x_i\in {ℝ}^p$. Dictionary learning aims to learn a dictionary D where $D\in {ℝ}^{p\times l}$ and a sparse code matrix $Z,Z\in {ℝ}^{l\times n}$. The original signals X is modeled by a sparse linear combination of D and Z as X ≈ DZ. Given one image patch x_i, we can formulate the following optimization problem:

$$\underset{D\in \Psi ,z_i\in {ℝ}^l}{\min }f\left(D,z_i\right)=\frac{1}{2}{\Vert x_i-D{z}_i\Vert }_2^2+\lambda {\Vert z_i\Vert }_1,$$ (1)

where $\Psi =\left\{D\in {ℝ}^{p\times l}:\forall j\in 1,\dots ,l,||D_j|{|}_2\le 1\right\}$, D_j denotes the jth column of D. λ is the positive regularization parameter. z_i is the learnt sparse codes for x_i and $Z=\left(z_1,\dots ,z_n\right)$.

The optimization of Eq.1 can be decomposed into an alternative learning process in the Online Dictionary Learning methods (ODL) [22]. Given each image patch x_i, ODL keeps the D fixed and learn z_i, then keep z_i fixed and learn D. The learning process runs κ (a fixed constant) iterations until there are no more changes on D and Z.

3.2. The Proposed Algorithm

Given features from T different tasks: $\left\{X_1,X_2,\dots ,X_T\right\}$, our objective is to learn a set of sparse codes $\left\{Z_1,Z_2,\dots ,Z_T\right\}$ for each task where $X_t\in {ℝ}^{p_t\times n},Z_t\in {ℝ}^{l_t\times n}$ and $t\in \{1,\dots ,T\}$. p_t is the image patch number of X_t (i.e. cortical measure),n is the number of subjects for X_t and l_t is the dimension of each sparse code in Z_t. When employing the ODL to learn the sparse codes Z_t by X_t individually, we obtain a set of dictionary $\left\{D_1,\dots ,D_T\right\}$ but there is no correlationship between learnt dictionaries. Another solution is to construct the features $\left\{X_1,\dots ,X_T\right\}$ into one matrix X to obtain the dictionary D. However, if there is no latent common information shared by the same subject with different cortical structure measures, only one dictionary D is not enough to show the variation among features from different structure measures. Such fact is supposed to be easily revealed in the variance of dictionary atoms and the sparsity of their corresponding sparse code matrices. To address this challenge, we integrate the idea of multi-task learning into the online dictionary learning method. We propose a novel dictionary learning algorithm, termed as Multi-task Stochastic Coordinate Coding (MSCC), to learn the sparse codes of subjects from different structure measures. In this work each task corresponds one special structure measure

Algorithm 1.

Multi-task Sparse Coordinate Coding

Require: Samples from different structure measures: $\left\{X_1,X_2,\dots \mathrm{..}{X}_T\right\}$ and for each $X_t,X_t\in {ℝ}^{p\times n_t}$

Ensure: Dictionaries and sparse codes for each structure measure: $\left\{D_1,\dots ,D_T\right\}$ and $\left\{Z_1,\dots ,Z_T\right\}$

1: for k = 1 to κ do

2:  for t = 1 to T do

3:   for i = 1 to nt do

4:    Get an image patch xt(i) from sample Xt.

5:    Update ${\widehat{D}}_t^k:{\widehat{D}}_t^k=\Phi$.

6:    Update $z_t^{k+1}(i)$ and index set $I_t^{k+1}(i)$ by a few steps of CCD:

7:      $\left[z_t^{k+1}(i),I_t^{k+1}(i)\right]=CCD\left({\widehat{D}}_t^k,{\overline{D}}_t^k,x_t(i),I_t^k(i),z_t^k(i)\right)$.

8:    Update the ${\widehat{D}}_t$ and ${\overline{D}}_t$ by one step SGD:

9:     $\left[{\widehat{D}}_t^{k+1},{\overline{D}}_t^{k+1}\right]=SGD\left({\widehat{D}}_t^k,{\overline{D}}_t^k,x_t(i),I_t^{k+1}(i),z_t^{k+1}(i)\right)$.

10:    Normalize ${\widehat{D}}_t^{k+1}$ and ${\overline{D}}_t^{k+1}$ based on the index set $I_t^{k+1}(i)$.

11:    Update the shared dictionary $\Phi :\Phi ={\widehat{D}}_t^{k+1}$.

12:   end for

13:  end for

14: end for

For the subjects’ feature matrix X_t of a particular task, MSCC learns a dictionary D_t and sparse codes Z_t. D_t is composed of two parts: $D_t=\left[{\widehat{D}}_t,{\overline{D}}_t\right]$ where ${\widehat{D}}_t\in {ℝ}^{p_t\times \widehat{l}},{\overline{D}}_t\in {ℝ}^{p_t\times {\overline{l}}_t}$ and $\widehat{l}+{\overline{l}}_t=l_t$. ${\widehat{D}}_t$ is the same among all the learnt dictionaries $\left\{D_1,\dots ,D_T\right\}$ while ${\overline{D}}_t$ is different from each other and only learnt from the corresponding subjects’ feature matrix X_t. Therefore, objective function of MSCC can be reformulated as follows:

$$\begin{gathered} \underset{\left(D_1,\dots ,D_T\right)}{\min }\sum _{t=1}^T\frac{1}{2}{\Vert X_t-\left[{\widehat{D}}_t,{\overline{D}}_t\right]Z_t\Vert }_F^2+\lambda \sum _{t=1}^T{\Vert Z_t\Vert }_1: \\ \text{subject}\text{to}{\widehat{D}}_1=\cdots ={\widehat{D}}_T\text{and}{D}_t\in {\Psi }_t \end{gathered}$$ (2)

where ${\Psi }_t=\left\{D_t\in {ℝ}^{p\times l_t}:\forall j\in 1,\dots ,l_t,{\Vert {\left[D_t\right]}_j\Vert }_2\le 1\right\}$ and ${\left[D_t\right]}_j$ is the jth column of D_t.

Fig. 2 illustrates the framework of MSCC with features of ADNI from three different tasks (i.e. cortical structure measures), which represents as X₁, X₂ and X₃, respectively. Through the multi-task learning process of MSCC, we obtain the dictionary and sparse codes for features from each task t: D_t and Z_t. In MSCC, a dictionary D_t is composed by a shared part ${\widehat{D}}_t$ and an individual part ${\overline{D}}_t,{\widehat{D}}_1={\widehat{D}}_2={\widehat{D}}_3$. For the individual part of dictionaries, MSCC learns a different ${\overline{D}}_t$ only from the corresponding feature matrix X_t. We vary the number of columns ${\overline{l}}_t$ in ${\overline{D}}_t$ to introduce the variant in the learnt sparse codes Z_t. As a result, the dimensions of learnt sparse codes matrix Z_t are different from each other.

Fig. 2.

Illustration of the learning process of MSCC.

The initialization of dictionaries in MSCC is critical to the entire learning process. We propose a random patch method to initialize the dictionaries from different tasks. The main idea of the random patch method is to randomly select l image patches from n subjects $\left\{x_1,x_2,\dots ,x_n\right\}$ to construct D where $D\in {ℝ}^{p\times l}$. It is a similar way to perform the random patch approach in MSCC. In MSCC, the way we initialize ${\widehat{D}}_t$ is to randomly select $\widehat{l}$ subjects’ feature from features’ matrices across different tasks $\left\{X_1,\cdots ,X_T\right\}$ to construct it. For the individual part of each dictionary, we randomly select $\overline{l}$ subjects’ feature from the corresponding matrix X_t to construct ${\overline{D}}_t$. After initializing dictionary D_t for each task, we set all the sparse code Z_t to be zero at the beginning. The key steps of MSCC are summarized in Algorithm 1.

In algorithm 1, k denotes the epoch number where $k\in [1,\kappa ]$. Φ represent the shared part of each dictionary D_t which is initialized by the random patch method. For each subject’s feature x_t(i) extracted from X_t, we learn the ith sparse code $z_t^{k+1}(i)$ from Z_t by several steps of Cyclic Coordinate Descent (CCD) [32]. Then we use learnt sparse codes $z_t^{k+1}(i)$ to update the dictionary ${\widehat{D}}_t^{k+1}$ and ${\overline{D}}_t^{k+1}$ by one step Stochastic Gradient Descent (SGD) [33]. Since $z_t^{k+1}(i)$ is very sparse, we use the index set $I_t^{k+1}(i)$ to record the location of non-zero entries in $z_t^{k+1}(i)$ to accelerate the update of sparse codes and dictionaries. Φ is updated in the end of kth iteration to ensure ${\widehat{D}}_t^{k+1}$ is the same among all the dictionaries.

The learning process of sparse code $z_t^{k+1}(i)$ is shown in algorithm 2. At first, we generate the non-zero index set $I_t^{k+1}$ by one step of CCD to record the non-zero entry of $z_t^{k+1}(i)$. Then we perform S steps CCD to update the sparse codes only on the non-zero entries of $z_t^{k+1}(i)$, accelerating the learning process significantly. Ω is a sparse matrix multiplication function that has three input parameters. Take $\Omega (A,b,I)$ as an example, A denotes a matrix, b is a vector and l is an index set that records the locations of non-zero entries in b. The return value of function Ω is defined as: $\Omega (A,b,I)=Ab$. When multiplying A and b, we only manipulate the non-zero entries of b and corresponding columns of A

Algorithm 2.

Updating sparse codes $z_t^{k+1}(i)$

Require: The image patch $x_t\text{(}i\text{)}$, dictionaries ${\widehat{D}}_t^k$ and ${\overline{D}}_t^k$, sparse codes $z_t^k(i)$ and index set $I_t^k(i)$

Ensure: The updated sparse code $z_t^{k+1}(i)$ and the index set $I_t^{k+1}(i)$.

1: for j = 1 to lt do

2:     $g={\left[{\widehat{D}}_t^k,{\overline{D}}_t^k\right]}_j^T\left(\Omega \left(\left[{\widehat{D}}_t^k,{\overline{D}}_t^k\right],z_t^k(i),I_t^k(i)\right)-x_t(i)\right)$

3:     $z_t^{k+1}{(i)}_j={\Gamma }_{\lambda }\left(z_t^k{(i)}_j-g\right)$

4:  if $z_t^{k+1}{(i)}_j\ne 0$ then

5:      Put j into the index set $I_t^{k+1}(i)$.

6:  end if

7: end for

8: for s = 1 to S do

9:  for every element μ in the index set $I_t^{k+1}(i)$ do

10:      $g={\left[{\widehat{D}}_t^k,{\overline{D}}_t^k\right]}_{\mu }^T\left(\Omega \left(\left[{\widehat{D}}_t^k,{\overline{D}}_t^k\right],z_t^{k+1}(i),I_t^{k+1}(i)\right)-x_t(i)\right)$

11:      $z_t^{k+1}{(i)}_{\mu }={\Gamma }_{\lambda }(\left(z_t^{k+1}{(i)}_{\mu }-g\right)$

12:  end for

13: end for

Algorithm 3.

Updating dictionaries ${\widehat{D}}_t^{k+1}$ and ${\overline{D}}_t^{k+1}$

Require: The image patch $x_t\text{(}i\text{)}$, dictionaries ${\widehat{D}}_t^k$ and ${\overline{D}}_t^k$, sparse codes $z_t^{k+1}(i)$ and index set $I_t^{k+1}(i)$.

Ensure: The updated dictionaries ${\widehat{D}}_t^{k+1}$ and ${\overline{D}}_t^{k+1}$

1:   Update the Hessian matrix $H_t^{k+1}:H_t^{k+1}=H_t^k+z_t^{k+1}(i)z_t^{k+1}{(i)}^T$.

2:    $R=\Omega \left(\left[{\widehat{D}}_t^k,{\overline{D}}_t^k\right],z_t^{k+1}(i),I_t^{k+1}(i)\right)-x_t(i)$

3:   for j = 1 to p do

4:    for every element μ in the index set $I_t^{k+1}(i)$ do

5:        ${\left[{\widehat{D}}_t^{k+1},{\overline{D}}_t^{k+1}\right]}_{j,\mu }={\left[{\widehat{D}}_t^k,{\overline{D}}_t^k\right]}_{j,\mu }-\frac{i}{H_t^{k+1}(\mu ,\mu )}{z}_t^{k+1}{(i)}_{\mu }{R}_j.$

6:    end for

7:   end for

based on the index set I, speeding up the calculation by utilizing the sparsity of b. Γ is the soft thresholding shrinkage function [34] and the definition of Γ is given by: ${\Gamma }_{\phi }(x)=sign(x)(|x|-\phi )$.

The procedure of updating dictionaries is shown in Algorithm 3. We perform one step SGD to update the dictionaries: ${\widehat{D}}_t^{k+1}$ and ${\overline{D}}_t^{k+1}$. The learning rate is set to be an approximation of the inverse of the Hessian matrix $H_t^{k+1}$, which is updated by the sparse codes $z_t^{k+1}(i)$ in kth iteration. For the μth column of dictionary, we set the learning rate as the inverse of the diagonal element of the Hessian matrix, which is $1/H_t^{k+1}(\mu ,\mu )$. Since $D_t\in {\Psi }_t$ in equation (2), it is necessary to normalize the dictionaries ${\widehat{D}}_t^{k+1}\text{and}{\overline{D}}_t^{k+1}$ after updating them. We can perform the normalization on the corresponding columns of non-zero entries from $z_t^{k+1}(i)$ because the dictionaries updating only occurs on these columns. Utilizing the non-zero information from $I_t^{k+1}(i)$ can accelerate the whole learning process.

4. Experiments

AD and its early stage, Mild Cognitive Impairment (MCI), are becoming the most prevalent neurodegenerative brain diseases in elderly people worldwide [1]. To this end, there have been a lot of studies [5, 7, 11, 35] on investigating the underlying biological or neurological mechanisms and also discovering biomarkers for early diagnosis of AD and MCI. Data for testing the performances of our proposed framework were obtained from the ADNI database (adni.loni.usc.edu). [36], which has been considered as the benchmark database for performance evaluation of various methods for AD diagnosis. The ADNI was launched in 2003 as a public-private partnership, led by Principal Investigator Michael W. Weiner, MD. The primary goal of ADNI is to test whether biological markers such as serial MRI and positron emission tomography (PET), combined with clinical and neuropsychological assessment can measure the progression of MCI and early AD. The structural MR images were acquired from 1.5T scanners. The raw MR images and MMSE/ADAS-Cog scales were downloaded from the public ADNI website (www.loni.ucla.edu/ADNI). The information about subjects criteria are in www.adni-info.org. In the experimental dataset, there are 837 baseline subjects between 68–82 years of age.

4.1. Image Patches of Three Cortical Structure Measures

We utilized three baseline structural measures of brain, which are hippocampal multivariate morphometry statistics (MMS), lateral ventricle MMS and cortical thickness. For the hippocampal surface features, we used FIRST software [37] and marching cube method [38] to automatically segment and reconstructed hippocampal surfaces for each brain MR image. Then, we registered and computed hippocampal MMS [39]. For each subject, we obtained a 120,000 dimensional features of the hippocampal surfaces and we used a 50 × 50 window to obtained a collection of image patches, then we get 220968 baseline hippocampal image patches.

For the ventricular surface features we did the following. First, we segmented images of the lateral ventricles to build the ventricular structure surface models using a level-set based topology preserving method [40]. Then we computed surface registrations using the canonical holomorphic one-form segmentation method [41]. Finally, MMS [39] were computed and obtained as a 308,247 dimensional features of the ventricular surfaces for each subject. The cortical thickness was computed by FreeSurfer [42] which deforms the white surface to pial surface and measures deforming distance as the cortical thickness. The spherical parameter surface and weighted spherical harmonic representation [43] are used to register pial surfaces across subjects, which means each subjects have the same dimension (161,800) cortical thickness. After preprocessing the data with the 50 × 50 patch window size, we have 2867562, 1504926 image patches for ventricle and cortical thickness measures, respectively.

4.2. CNN-MSCC Modeling and Evaluation

We build a prediction model on these multiple task geometry surface features from 837 baseline subjects of AD, MCI and cognitively unimpaired (CU) categories, we expect to predict future MMSE/ADAS-Cog scales (from 6th month to 24th month). In this study, we took Alexnet structure [17] as the initial CNN model, which contains 7 layers, including convolutional layers with fixed filter sizes and different numbers of feature maps and the architecture of our CNN is shown in Tab. 1. We pretrained the CNN model on the ImageNet dataset [28], containing millions of labeled natural images with thousands of categories then removed the last fully-connected layer (this layer’s outputs are the 1000 class scores for a different task like ImageNet). Finally, the rest of the CNN was transferred and used to extract feature maps from MMS image patches of special cortices and cortical thickness [36].

Table 1.

The architecture of CNN used in this work.

Deep Layer	Function	# of neurons
1	Convolutional Layer	253440
2	Pooling Layer	186624
3	Convolutional Layer	64896
4	Convolutional Layer	64896
5	Convolutional Layer Pooling Layer	43264 9216
6	Fully Connected Layer	4096
7	Fully Connected Layer	4096

We implemented our CNN model using the Caffe toolbox [44]. The network was trained on a Intel (R) Xeon (R) 48-core machine, with 2.50 GHZ processors, 256 GB of globally addressable memory and a single Nvidia GeForce GTX TITAN black GPU. In the experimental setting of MSCC, the sparsity λ = 0.1. Also, we selected 10 epochs with a batch size of 1 in Algorithm 1 and 3 iterations of CCD in Algorithm 2 (P is set to be 1 and S is set to be 3) in all the experiments. After we get the MSCC features, we used Max-Pooling for further dimension reduction. Therefore, the feature dimentsion of each subject is a 1 × 2000 vector. To predict future clinical scores, we used Lasso regression following CNN-MSCC. For the parameter selection, 5-fold cross validation is used to select model parameters in the training data (between 10⁻³ and 10³).

In order to evaluate CNN-MSCC model, we randomly split the data (image patches and their corresponding MMSE/ADAS-Cog scores) into training and testing sets using an 8:2 ratio and used 10-fold cross validation to avoid data bias. Lastly, we evaluated the overall regression performance of our proposed system using normalized mean square error (nMSE), weighted correlation coefficient (wR) and root mean square error (rMSE) for task-specific regression performance measures [45]. The three measures are defined as follows:

$$nMSE(Y,\widehat{Y})=\frac{{\sum }_{i=1}^t{\Vert Y_i-{\widehat{Y}}_i\Vert }_2^2/\sigma \left(Y_i\right)}{{\sum }_{i=1}^t{n}_i},$$

$$wR(Y,\widehat{Y})=\frac{{\sum }_{i=1}^{t}Corr\left(Y_i,{\widehat{Y}}_i\right)n_i}{{\sum }_{i=1}^t{n}_i},$$

$$rMSE(y,\widehat{y})=\sqrt{\frac{\Vert y-\widehat{y}{\Vert }_2^2}{n}}.$$

For nMSE and wR, Y_i is the ground truth of target of task i and ${\widehat{Y}}_i$ is the corresponding predicted value, σ(Y_i) is the Standard deviation of Y_i, Corr is the correlation coefficient between two vectors and n_i is the number of subjects of task i. For rMSE, y is the ground truth of target at a single task and $\widehat{y}$ is the corresponding prediction by a prediction model. The smaller nMSE and rMSE, as well as the bigger wR mean the better results. nMSE and wR are used to evaluate the overall performances of the proposed system across three times points, rMSE is used to evaluate CNN-MSCC performance of each time point. We reported the mean and standard deviation based on 40 iterations of experiments on different splits of data.

4.3. Performance Analysis

After We constructed CNN-MSCC model with the image patches from the combined three kinds of features, we used Lasso to individually predict 6-month (M06), 12-month (M12) and 24-month (M24) MMSE and ADAS-cog scores with 8:2 ratio on training and testing data sets. The prediction results are reported in Fig. 3(a) and Fig. 3(b). Fig. 3(a) shows the overall nMSE and wR measures of CNN-MSCC to predict MMSE and ADAS-Cog scores across three time points (M06, M12 and M24), for MMSE and ADAS-Cog score predictions, CNN-MSCC achieves nMSE of 0.274±0.051 and 0.762±0.012 respectively, it also achieves wR of 0.751±0.083 and 0.862±0.045, respectively.

Fig. 3.

MMSE and ADAS-Cog prediction performances in terms of nMSE and wR (a) and in terms of rMSE (b) using proposed CNN-MSCC model.

Fig. 3(b) shows the rMSE measures for MMSE and ADAS-Cog scores at M06, M12 and M24, respectively, for MMSE and ADAS-Cog scores at M06, CNN-MSCC achieves rMSE of 2.198±0.062 and 4.322±0.269, for MMSE and ADAS-Cog scores at M12, the proposed method achieves rMSE of 2.211±0.459 and 4.930±0.912, for MMSE and ADAS-Cog scores at M24, we achieves rMSE of 2.290±0.601 and 5.521±0.816. We can observe that the rMSE of predicting M06, M12 and M24 scores of MMSE and ADAS-Cog are stable across all three time points. Especially there is an obvious improvement of the proposed CNN-MSCC for later time points (12, 24-month). This may be due to the data sparseness in later time points, as the proposed sparsity-inducing models are expected to achieve better prediction performance.

Zhou et al. proposed similar multi-task learning models to predict future MMSE and ADAS-Cog scores related with AD progression using baseline MRI features, MRI data were from 648 subjects of ADNI dataset, 5 categories of MRI features were used, they were cortical thickness, volumes or surface measures [45]. The nMSE, wR and rMSE results estimated from our proposed system are very comparable to the best results reported in [45]. There are three main reasons about the outperformances of the proposed multi-task model: advanced cortical structure measure MMS was introduced, this effective biomarker had been well studied in our previous work and outperformed volume measure [10, 11]; since deep model is good at modeling the complex cortical structure in the medical imaging field [46], pretrained CNN was used to generate hierarchical structure features from high-dimension MMS and cortical thickness measures; we developed a novel unsupervised learning method MSCC to improve the computing ability for processing hierarchical structure features of multi-tasks.

5. Conclusions and Future Work

In this work, we proposed a deep learning model, CNN-MSCC, to model multiple cortical structure features, for predicting future MMSE/ADAS-Cog scores. The proposed model is validated by extensive experimental studies and shown to be more efficient than similar studies. In future work, we will optimize our method and investigate its capability on brain multimodality imaging datasets.