A review of feature reduction techniques in neuroimaging

Abstract

Machine learning techniques are increasingly being used in making relevant predictions and inferences on individual subjects neuroimaging scan data. Previous studies have mostly focused on categorical discrimination of patients and matched healthy controls and more recently, on prediction of individual continuous variables such as clinical scores or age. However, these studies are greatly hampered by the large number of predictor variables (voxels) and low observations (subjects) also known as the curse-of-dimensionality or small-n-large-p problem. As a result, feature reduction techniques such as feature subset selection and dimensionality reduction are used to remove redundant predictor variables and experimental noise, a process which mitigates the curse-of-dimensionality and small-n-large-p effects. Feature reduction is an essential step before training a machine learning model to avoid overfitting and therefore improving model prediction accuracy and generalization ability. In this review, we discuss feature reduction techniques used with machine learning in neuroimaging studies.

1.0 Introduction

Conventional group-level neuroimaging data analysis techniques (e.g. voxel-based morphometry) have been used in neuroimaging research for decades. However, these techniques have only been able to identify average between-group differences and unable to make predictions on individual subjects. The inability to make predictions from individual subjects neuroimaging scan data may have greatly hampered the ability to translate neuroimaging research results into clinical practice (Brammer, 2009; Linden, 2012). Recently though, machine learning (ML) techniques also variously known as multivariate pattern analysis (MVPA) or pattern recognition (PR) techniques, have been used to decode or predict individual subjects’ brain states using neuroimaging scan data. These studies have increased significantly and equally become successful in decoding brain states which raises the possibility of being deployed for potential clinical use and particularly in making personalized clinical decisions (Linden, 2012; Mwangi et al. 2012a).

Machine learning applications in neuroimaging can be divided into two broad categories namely classification and regression. In classification, neuroimaging data with corresponding categorical labels (e.g. Healthy Controls vs Patients or Treatment responders vs Non-responders) are used to develop a predictive classifier. The resulting classifier is used to make predictions on subjects’ data not present during the training stage. Previous classification studies include; Alzheimer’s disease (AD) (Kloppel et al. 2008; Magnin et al. 2009; Zhang et al. 2011b), Major Depressive Disorder (MDD) (Mwangi et al. 2012a; Zeng et al. 2012), Autism Spectrum Disorder (ASD) (Ecker et al. 2010; Ingalhalikar et al. 2011),Schizophrenia (Koutsouleris et al. 2009), Mild Cognitive Impairment(MCI) (Haller et al. 2010b) and Attention Deficient Hyperactivity Disorder (ADHD) (Lim et al. 2013; Sato et al. 2012; Zhu et al. 2005). Conversely, in regression neuroimaging data with corresponding continuous targets (e.g. clinical scores or age) are used to ‘train’ a regression model. Similar to above, the model is used to make predictions on novel individual subjects’ data not present during training. These techniques have recently been used to predict individual subjects’ age (Brown et al. 2012; Dosenbach et al. 2010; Franke et al. 2012; Franke et al. 2010; Mwangi et al. 2013) and clinical scores in MDD, AD and chronic pain (Marquand et al. 2010a; Mwangi et al. 2012b; Stonnington et al. 2010; Wang et al. 2010).

Notably, previous studies in both classification and regression have used neuroimaging scan data from multiple modalities such as T₁-weighted magnetic resonance imaging (MRI), functional MRI (fMRI), diffusion tensor imaging (DTI), positron emission tomography (PET) and single-photon emission computed tomography (SPECT). In this review, we generalize our discussions to all imaging modalities and assume that all scans have been subjected to standardized pre-processing routines (e.g. spatial normalisation, segmentation and or smoothing). Neuroimaging data pre-processing frameworks in different modalities are discussed elsewhere (Ashburner, 2007; Ashburner, 2009; Johansen-Berg and Behrens, 2009; Penny et al. 2007). In this review, pre-processed neuroimaging scans’ voxels are referred to as ‘features’ or ‘predictor variables’. In addition, we generalize our feature reduction discussion to all machine learning techniques such as support vector machines (SVMs) (Mwangi et al. 2012a; Orru et al. 2012; Vapnik, 1999), relevance vector machines (RVMs) (Mwangi et al. 2012a; Tipping, 2001) and Gaussian process classifiers (GPCs) (Marquand et al. 2010b; Rasmussen and Williams, 2006). Discussions of these methods with respect to neuroimaging data are also given elsewhere (Johnston et al. 2012; Misaki et al. 2010; Orru et al. 2012).

1.1 Rationale for feature reduction

In many neuroimaging studies, the sample size (number of subjects or observations) is often less than 1000. In comparison, pre-processed brain scans may contain (>100,000) non-zero voxels. As a result, the numbers of features (voxels) greatly outnumber the number of observations (sample size). This is a common problem in machine learning literature known as the curse-of-dimensionality (Bellman, 1961) or small-n-large-p (Fort and Lambert-Lacroix, 2005). Consequently, without pre-selecting the ‘most relevant’ features and effectively discarding redundant features plus noise, a predictive machine learning model has a marked risk of ‘overfitting’ (Guyon and Elisseeff, 2003; Hua et al. 2009). Overfitting implies that the model training process yields a machine learning model with poor generalization ability which is interpreted as inability to make accurate predictions on novel subjects’ data (Guyon and Elisseeff, 2003; Hua et al. 2009; Kohavi and John, 1997).

In view of the above, there is a general consensus from previous neuroimaging machine learning studies that feature reduction is a fundamental step before applying a predictive model (e.g. SVM or RVM) to neuroimaging data (Bray et al. 2009; Cheng et al. 2012; De Martino et al. 2008; Duchesnay et al. 2007; Duchesnay et al. 2004; Duff et al. 2011; Franke et al. 2010; Liu et al. 2012; Lohmann et al. 2007; Mitchell et al. 2004; Mwangi et al. 2012a; Norman et al. 2006; Pereira et al. 2009; Rizk-Jackson et al. 2011; Schrouff et al. 2013; Valente et al. 2011; Van De Ville and Lee, 2012). Other than mitigating the curse-of-dimensionality effect as above, the feature reduction process may also facilitate a deeper understanding of the scientific question of interest. For example, in a highly accurate predictive classifier able to predict patient treatment responders against non-responders, brain regions identified during feature reduction may offer an insight into different neural substrates of non-responders. In this review we divide feature reduction techniques into two broad categories namely; supervised and unsupervised techniques respectively.

2.0 Supervised feature reduction techniques

Supervised feature reduction techniques use highly dimensional neuroimaging data and the required outcome labels (e.g. +1 treatment responders, −1 treatment non-responders) to select relevant features and discard redundant features plus noise. However, as in the wider machine learning literature, we further subdivide these techniques into three categories namely; ‘filter’, ‘wrapper’ and ‘embedded’ methods, respectively (Guyon and Elisseeff, 2003; Saeys et al. 2007). A clear distinction between these categories exists. First, filter techniques such as t-tests, Anova and Pearson correlation coefficient use simple statistical measures (e.g. mean, variance, correlation coefficients) to rank features according to their relevance in detecting group-level differences. Second, wrapper techniques use an objective function from a classification or regression machine learning model to rank features according to their relevance to the model. Lastly, embedded methods select relevant features as ‘part’ of the machine learning process by enforcing certain ‘penalties’ on a machine learning model thus yielding a small subset of relevant features.

In passing, we note that relevant features are usually selected using training data only to avoid double-dipping which entails using both training and testing data partitions to select relevant features (Kriegeskorte et al. 2010; Kriegeskorte et al. 2009). In this review, we assume both (training and testing datasets) have been adequately separated using a cross-validation procedure such as hold-out or leave-one-out which are explored elsewhere (Johnston et al. 2012; Pereira et al. 2009; Strother et al. 2002; Theodoridis and Koutroumbas, 2009). Figure 1 illustrates a feature reduction process without double dipping. Conversely, Figure 2 illustrates a feature reduction procedure with double dipping - which should be avoided.

Figure 1.

Feature reduction without double-dipping. Training and testing datasets are separated before the feature reduction process.

Figure 2.

Feature reduction with double-dipping. Features are selected from the same set of training and testing data.

2.1 Filter techniques

2.1.1 Pearson correlation coefficient

The Pearson correlation coefficient (PCC) ranks features by calculating linear correlations between individual features and class labels in classification or continuous targets in regression (Guyon and Elisseeff, 2003). Here, we assume a two group classification problem with predictor variables x and diagnostic labels y_i, (e.g. Patients-1 vs Controls -2). The Pearson correlation coefficient between predictor variables and diagnostic labels is calculated as:

$$P_i=\frac{\sum _i(x_i-\stackrel{‒}{x})(y_i-{\stackrel{‒}{y}}_i)}{\sqrt{\sum _i{\left(x_i-\stackrel{‒}{x}\right)}^2\sum _i{\left(y_i-\stackrel{‒}{y}\right)}^2}}$$ (1)

Where x_i stands for the feature value of the i^th sample and $\stackrel{‒}{x}$ is the mean of these feature values. y_i are diagnostic labels and $\stackrel{‒}{y}$ is the mean of all y_i in the sample (Fan et al. 2007; Guyon and Elisseeff, 2003). The relevance of a feature in separating both classes is evaluated by considering the absolute value of the correlation coefficient P_i with higher values indicating the feature’s greater relevance in discriminating between classes. Lastly, relevant features above a user-defined threshold of correlation coefficients are selected for subsequent machine learning analyses. However, to select the optimal user-defined threshold, cross-validation procedures (e.g. k-fold or leave-one-out) are used to iteratively evaluate a range of threshold values and the threshold with a low generalization error selected.

Numerous studies have applied PCC filters to select relevant features. For example, in MCI classification (Wee et al. 2011), cocaine exposed individuals classification (Fan et al. 2007), AD classification (Dai et al. 2012; Davatzikos et al. 2008; Grana et al. 2011) and in gender (Men vs Women) classification (Fan et al. 2006).

However, although PCC filters are applicable to both multi-group (>2) classification and regression tasks, they are only able to detect linear dependencies between features and corresponding targets (Guyon and Elisseeff, 2003). This is a major setback for PCC filters especially in tasks involving selecting relevant features from high-dimensional data with multivariate relationships (Wang et al. 2010).

2.1.2 T-tests and analysis of variance (ANOVA)

Statistical hypothesis testing techniques such as t-tests and ANOVA have extensively been used to detect average group-level differences in neuroimaging studies. Recently, though, these techniques have been used for supervised feature selection in neuroimaging machine learning studies with success. Here, we assume a two class classification task of categorising (treatment responders +1 vs. treatment Non-responders −1). The assumption that predictor variables are from pre-processed neuroimaging scans still holds. As a result, a two-sample t-test evaluates the null hypothesis that the two-group (Responders, Non-responders) sample means are equal given observed variance. Let ${\stackrel{‒}{x}}_p$ and ${\stackrel{‒}{x}}_c$ be voxel means for patient and control groups respectively and s_p,s_c denote corresponding standard deviations. The t-score of a feature v is calculated as (Sheskin, 2004);

$$t=\frac{\mid {\stackrel{‒}{x}}_p-{\stackrel{‒}{x}}_c\mid }{\sqrt{\frac{{(V_p-1s)}_p^2+{(V_c-1s)}_c^2}{N_p+N_c-2}}•\left[\frac{1}{N_p}+\frac{1}{N_c}\right]}$$ (2)

Where N_p and N_c denote number of subjects in each group. Once this calculation is performed, a threshold of significance (e.g. p-value) which represents the probability of obtaining a statistic greater in magnitude than t under the null hypothesis is defined (Ashburner and Friston, 2000; Penny et al. 2007; Theodoridis and Koutroumbas, 2009). Subsequently, an optimal user-defined threshold of significance (p-value) representing relevant features is selected through a cross-validation process and relevant features used for subsequent machine learning analyses (Mwangi et al. 2012a). Several classification studies have recently used t-tests to select relevant features for machine learning in neuroimaging. For example, AD (Chaves et al. 2009; Chu et al. 2012; Hinrichs et al. 2009; Zhang et al. 2011a), ASD (Duchesnay et al. 2011), MDD (Mwangi et al. 2012a), gender (Duchesnay et al. 2007), schizophrenia (Kovalev et al. 2003), MCI (Wee et al. 2012) and in other non-clinical studies (Balci et al. 2008; Haynes and Rees, 2005). Applications of t-tests in feature selection are computationally fast and easy to implement and scale well to high dimensional data meaning they are able to select a small subset of relevant features from the original high-dimensional feature set (Hua et al. 2009; Mwangi et al. 2012a; Saeys et al. 2007).

However, t-tests are also weighed down by a number of shortcomings. First, they are univariate, meaning they do not take into account interactions between multiple features and spatial patterns (non-multivariate). Second, t-tests are only able to detect two-group differences, although this shortcoming is compensated by the equivalent analysis of variance (ANOVA) technique.

The ANOVA technique is used to select relevant features in multiple (>2) groups by generalizing a t-test to more than two groups (Cohen, 1998). However, similar to t-tests, ANOVAs are univariate but have also been used in selecting relevant features in neuroimaging classification tasks. For example, Costafreda and colleagues (Costafreda et al. 2011) recently used ANOVA for feature selection in classifying patients with bipolar disorder (BD), schizophrenia and healthy controls. Other neuroimaging classification studies using ANOVA for feature reduction include; ASD (Coutanche et al. 2011), MDD (Costafreda et al. 2009), schizophrenia (Yoon et al. 2008) and fMRI visual activation classification task (Cox and Savoy, 2003). In passing, we note that ANOVAs offer the same benefit as t-tests, but an optimal threshold of relevant features is selected using a cross-validation process with the training data. Most notably, we note that the multivariate analysis of covariance (MANCOVA) is a multivariate extension of ANOVA and has recently been used in a number of neuroimaging feature reduction tasks (Friston et al. 1996, Calhoun et al. 2011, Allen et al. 2011). Lastly, Wilcoxon test (De Martino et al. 2008), signal-to-noise ratio (Ingalhalikar et al. 2011), Gini-contrast (Langs et al. 2011) and relief (Haller et al. 2010a) are other filter techniques, albeit with few applications in neuroimaging machine learning studies so far.

However, whilst these supervised filter techniques have extensively been used in mitigating the small-n-large-p or curse-of-dimensionality problems in neuroimaging, they collectively suffer from a significant setback by not considering interactions between multiple features or spatial patterns (not multivariate).

2.2 Multivariate wrapper techniques

Multivariate wrapper techniques use an objective-function from a multivariate machine learning model (e.g. classification or regression) to rank features according to their relevance or importance to the model (Guyon and Elisseeff, 2003; Kohavi and John, 1997). Multivariate wrapper approaches are further subdivided into two sub-categories namely; forward selection and backward elimination (Kohavi and John, 1997). In forward selection, the search for relevant features begins with an empty set and features are iteratively added in ‘small’ pre-defined steps until an optimal number of features are found. On the contrary, in backward elimination, the search begins with all features in the training set which are iteratively removed in ‘small’ pre-defined steps until an optimal number of features are found. Recursive feature elimination (RFE) (Guyon et al. 2001), which is popularly used in neuroimaging studies is an example of a backward elimination technique.

2.2.1 Recursive feature elimination

We assume a two-group classification task with a set of features x_i, and corresponding labels y_i. Equally, we assume training data is now sub-divided into ‘training’ and ‘evaluation’ subsets respectively. A machine learning algorithm (e.g. linear support vector machine or linear relevance vector machine) is ‘trained’ resulting into observation weights. As a result, feature or voxel relevance weights α_i are calculated as:

$$W=\sum _{x_i\in cf}{\alpha }_i{y}_i{x}_i$$ (3)

Where cf stands for observations or subjects with non-zero weights (e.g. support vectors in SVM or analogous relevance vectors in RVM). Subsequently, absolute values of the weights W are ranked in order of importance (low-less relevant, high-most relevant) and a user-defined percentage (e.g. 2%) of the lowest ranking features removed. In the next step, a new model is trained minus the non-relevant features and the new model’s generalization error (or accuracy) on the evaluation subset is assessed. This process is repeated until a termination criterion is reached or until the feature set is empty. Lastly, features leading to the best generalization ability (high accuracy) are selected for training the final machine learning model and the rest of the features discarded. Figure 3 illustrates a recursive feature elimination process.

Figure 3.

Flow diagram illustrating the recursive feature elimination (RFE) process.

However, RFE requires definition of two parameters. 1) Backward-eliminations termination or stopping criteria. The first possibility is to remove low ranking features iteratively until the feature set is empty and the iteration resulting to the best generalization ability or high accuracy is selected. The second possibility is to terminate the procedure when the model performance of the current iteration is not significantly better than the previous step, as explored by De Martino and colleagues (De Martino et al. 2008). 2) Another user-defined parameter is the percentage of features removed at every backward-elimination step. Previous neuroimaging studies using RFE have variably used this parameter, for example 8% (De Martino et al. 2008), 10% (Craddock et al. 2009) and 2% (Mwangi et al. 2013). It is not currently well understood how the choice of this parameter affects the overall model performance and this remains an open research question. However, removing a very small percentage of features at every iteration is computationally expensive, whilst removing a higher percentage may result into inclusion of non-relevant features (Craddock et al. 2009; De Martino et al. 2008).

RFE offers several benefits by first, considering multivariate interactions between entire spatial patterns in the data. Second, the technique uses a predictive model to remove non-relevant or redundant features which may result in a better generalization ability (Guyon and Elisseeff, 2003). However potential setbacks should be noted. First, this technique is computationally intensive as it performs a complete heuristic search of the feature input space (Saeys et al. 2007). Second, as cross-validation methods are used to avoid a biased feature selection process, this may result in different features being selected in every cross-validation iteration (Craddock et al. 2009). This problem has recently been addressed by Dosenbach and colleagues (Dosenbach et al. 2010) by recommending reporting of a consensus discrimination map which aggregates features selected in all cross-validation iterations.

Previous feature selection applications using RFE in neuroimaging classification tasks include; ASD (Calderoni et al. 2012; Duchesnay et al. 2011; Ecker et al. 2010; Ingalhalikar et al. 2011), MDD (Craddock et al., 2009), schizophrenia (Castro et al. 2011b), psychosis (Gothelf et al. 2011), object recognition in a fMRI task (Hanson and Halchenko, 2008), fragile X syndrome (Hoeft et al. 2011), ADHD (Marquand et al. 2011), MCI (Nho et al. 2010), fMRI spatial patterns (De Martino et al. 2008), mood disorders (Mourao-Miranda et al. 2012) and AD (Davatzikos et al. 2008; Mesrob et al. 2008). An interesting variant of RFE, which involves backward-elimination of voxel clusters rather than individual features, has recently been explored (Deshpande et al. 2010). In passing, we note that although the majority of RFE applications in neuroimaging are largely in predictive classification, recently RFE has been used for regression tasks (Fan et al. 2010; He et al. 2008; Mwangi et al. 2013).

Remarkably, although previous RFE applications have largely utilized multivariate wrappers in input spaces (e.g. linear SVM), continuing efforts in solving the ‘pre-image problem’ (Kwok and Tsang, 2004) have recently allowed extraction of feature/voxel weights in non-linear feature spaces (e.g. non-linear SVM) (Kjems et al. 2002; LaConte et al. 2005; Mwangi et al. 2013; Rasmussen et al. 2011).

2.2.2 Searchlight

The searchlight technique was introduced by Kriegeskorte and colleagues (Kriegeskorte et al. 2006) for multivariate feature reduction in neuroimaging data. This technique selects relevant features as follows. First, 3-dimensional (3D) spherical volumes of pre-defined radius or ‘searchlight’ (e.g. 4mm) are centered at every voxel and populated through neuroimage volumes in the training data. Second, a machine learning classifier (e.g. SVM) is trained at every searchlight volume using only voxels within the spherical searchlight volume and classifier accuracies from searchlight volumes centered at every voxel recorded. Through permutation methods (Kriegeskorte et al. 2006; Pereira et al. 2009), the searchlight accuracies at every voxel are converted into p-values, which are subsequently thresholded to remove non-relevant voxels. However, the searchlight technique requires a user to determine the spherical regions-of-interest radius (e.g. 4mm), and according to Kriegeskorte and colleagues (Kriegeskorte et al. 2006), there should be a balance between the size of neuroimaging scans and the spherical searchlight volume.

Previous applications of feature reduction using searchlight in neuroimaging machine learning tasks include: episodic memory decoding (Chadwick et al. 2010), scene representation decoding (Bonnici et al. 2011) and attention shifting cortical activation decoding (Greenberg et al. 2010). We note that majority of these studies have largely been task-based fMRI decoding solutions.

Notably, the searchlight method assumes that any discriminative information lies within the searchlight radius and according to (Formisano et al. 2008), this technique may be unreliable in detecting ‘distant pattern’ differences (e.g. bilateral cerebral regions). However, we note that this is not the case with other multivariate wrapper methods such as RFE, which are able to detect multivariate interactions across ‘distant patterns’ as above.

2.3 Embedded feature reduction techniques

In this section, we discuss three of the most popular embedded methods namely; least absolute shrinkage and selection operator (LASSO) (Tibshirani, 1996; Tibshirani, 2011), the Elastic Net (Zou and Hastie, 2005) and the partial least square (PLS) method (MacIntosh and Lobaugh 2004). The Elastic Net and LASSO techniques combine both machine learning and feature reduction steps by enlisting a regularization framework (e.g. L₁ and L₂ norm regularization) resulting to a reduced subset of relevant features (Zou and Hastie, 2005). On the contrary, PLS selects relevant features by establishing or analyzing associations between the independent and dependent variables (e.g. brain activity or structure and behavior) (Krishnan et al. 2011).

2.3.1 Least absolute shrinkage and selection operator (LASSO)

We assume a two-group classification task with a set of features x_ij, and corresponding target labels y_i, where i = 1,2,..N represents observations (subjects) and j = 1,2,...,P represents the number of features (predictor variables). Additionally, we assume that the predictor variables are normalized by subtracting sample mean and dividing by the standard deviation. As a result, the LASSO computes model coefficients $\widehat{\beta }$ by minimizing the following function (Tibshirani, 1996; Tibshirani, 2011).

$$\sum _{i=1}^N{\left(y_i-\sum _j{x}_{ij}{\beta }_j\right)}^2+\lambda \sum _{j=1}^P\mid {\beta }_j\mid$$ (4)

λ is a user-defined parameter which controls the balance between the model having few non-zero coefficients $\widehat{\beta }$ (sparsity) and high prediction accuracy or generalization ability. Interestingly, as λ approaches 1, the model becomes increasingly sparse meaning few ‘relevant’ features whilst as λ approaches 0, the model is less sparse meaning more ‘relevant’ features (Bunea et al. 2011). To arrive at an optimal value for, cross-validation procedures (e.g. k-fold or leave-one-out) are used to test a range of λ parameters with pre-defined steps and the parameter resulting to high model accuracies is selected. The LASSO function is solved using an optimization procedure such as the coordinate descent algorithm as explored elsewhere (Friedman et al. 2010; Tseng and Yun, 2009). A solution to this optimization problem is provided by Friedman and colleagues (Friedman et al. 2010) in R and Matlab (The Mathworks, Inc) routines.

The benefits of this method in a feature reduction process are two fold. First, the LASSO yields a small set of model coefficients $\widehat{\beta }$ with majority of coefficients set to zero and corresponding features discarded from the subsequent machine learning process. Second, the LASSO is able to cope with situations where there are a large number of predictor variables (voxels) and fewer observations (subjects) as is the case in majority of neuroimaging studies (Bunea et al. 2011). Previous applications of feature selection using LASSO in neuroimaging machine learning tasks include: AD classification (Casanova et al. 2011; Kohannim et al. 2012b; Rao et al. 2011; Vounou et al. 2011; Yan et al. 2012), prediction of video stimulus scores in fMRI (Carroll et al. 2009), ASD classification (Duchesnay et al. 2011), prediction of brain characteristics using genetic data (Kohannim et al. 2012a; Kohannim et al. 2012b), prediction of pain stimuli in fMRI (Rish et al. 2010) and Gender classification (Casanova et al. 2012).

The LASSO technique has been successful in mitigating small-n-large-p or curse-of-dimensionality problems in neuroimaging albeit with several setbacks. First, if predictor variables in a group are highly correlated, the LASSO selects only one variable from the group and ignores the rest (Bunea et al. 2011; Zou and Hastie, 2005). Second, the number of selected relevant features may not exceed the number of observations or samples before the model begins to saturate (Zou and Hastie, 2005). To overcome these two limitations, Zou and Hastie (Zou and Hastie, 2005) introduced the Elastic Net technique, which is relatively similar to LASSO, albeit with a few modifications, as explored below.

2.3.2 Elastic Net

The Elastic Net is formulated in a similar manner as the LASSO, but with an additional quadratic term (Zou and Hastie, 2005). We assume a two-class classification problem with similar observations and predictor variables and a preceding normalization step, as above. As a result, the Elastic Net computes model coefficients $\widehat{\beta }$ by minimizing the following objective function (Bunea et al. 2011; Ogutu et al. 2012; Zou and Hastie, 2005).

$$\sum _{i=1}^N{\left(y_i-\sum _j{x}_{ij}{\beta }_j\right)}^2+{\lambda }_1\sum _{j=1}^P\mid {\beta }_j\mid +{\lambda }_2\sum _{j=1}^P{\beta }_j^2$$ (5)

Contrary to LASSO, the Elastic Net requires definition of two user-defined model regularization parameters (λ₁ and λ₂), which control the degree of penalization. The L₁ penalty ${\sum }_{j=1}^P{\beta }_j^2$ promotes sparsity in the solution, resulting in few features with non-zero weights, whilst L₂ penalty encourages stability in the solution and acts as a bound on the number of features selected (Bunea et al. 2011; Kohannim et al. 2012a; Ogutu et al. 2012; Zou and Hastie, 2005). These parameters are often selected using an objective parameter grid-search process which evaluates a ‘range’ of parameters in two-dimensions (grid-search) and parameters giving the best performance selected. However, the grid-search process can be computationally intensive (Bunea et al. 2011). Similar to LASSO, the Elastic Net function is solved using an optimization procedure such as the coordinate descent algorithm also explored elsewhere (Friedman et al. 2010). An implementation of the Elastic Net solution is freely provided in both R and Matlab (The Mathworks, Inc) by Friedman and colleagues (Friedman et al. 2010).

Previous applications of feature reduction using Elastic Net in neuroimaging machine learning tasks include: AD classification (Rao et al. 2011; Shen et al. 2011; Wan et al. 2011) and treatment response predictions in ADHD (Marquand et al. 2012).

2.3.3 Partial Least Squares

The partial least square (PLS) feature reduction method is divided into two major categories namely; partial least squares correlation (PLSC; McIntosh et al. 1996, Krishnan et al. 2011) and partial least squares regression (PLSR; Wold et al 2001). Here, we discuss PLSC which is by far the most popular variant of the partial least squares method used in neuroimaging feature reduction tasks (Krishnan et al. 2011).

We recall our two-group classification task with a set of features x_ij stored in a matrix X and corresponding target labels y_i, stored in a matrix Y where i 1,2,..N represents observations (subjects) and j 1,2,…,P represents the number of features (predictor variables). Additionally, we assume that the predictor variables are normalized by subtracting sample mean and dividing by the standard deviation (z-scores). PLSC begins by computing the cross product of the predictor variables and target vectors as below;

$$M=Y^{T}X$$ (6)

The resulting matrix M is decomposed using singular value decomposition (SVD) (Krishnan et al. 2011, McIntosh and Misic 2013). The SVD matrix decomposition process is explored elsewhere (Bishop 2006, Krishnan et al. 2011). The SVD decomposition yields the following.

$$M=US{V}^T$$ (7)

The above decomposition results into a set of singular vectors (U-left, V-right) whilst S is a diagonal matrix representing the ‘singular values’ (McIntosh and Misic 2013). The left singular vectors U contains the weights or coefficients identifying the variables or voxels in matrix X making the highest contribution in explaining the relationship between the predictor variables (e.g. brain scans) and the targets (e.g. diagnostic labels or behavioural scores) (McIntosh and Misic 2013). Lastly, a set of latent variables for both the predictor variable matrix X and target labels Y are reconstructed by computing the dot product of the singular vectors and the original data as below and also explored elsewhere (Krishnan et al. 2011).

$$L_x=XV$$ (8)

Where L_x is a reduced set of latent variables representing the original predictor variables in X (Krishnan et al. 2011). On the contrary, L_y represents the latent variables for the target variables and computed as follows (Krishnan et al. 2011).

$$L_y=YU$$ (9)

At this point, the original high-dimensional predictor variables (e.g. brain scans) are now represented by a set of low dimensional latent variables. Lastly, bootstrap and permutation tests are used to identify the optimal number of latent variables as explored elsewhere (Krishnan et al. 2011, Ziegler et al. 2013, McIntosh and Misic. 2013). An implementation of the PLSC method for neuroimaging data analysis is freely provided in Matlab (The Mathworks, Inc) by Shen and colleagues at (research.baycrest.org/pls/source/). A rigorous introduction to the PLS method and accompanying pseudocodes are also provided elsewhere (Krishnan et al. 2011). Notably, as PLSC correlates predictor variables (e.g. brain scans) and target variables (e.g. diagnosis or behavioral design), prediction tasks are handled by it’s variant - partial least squares regression (PLSR) (Wold et al. 2001). The PLSR is explored elsewhere (Wold et al. 2001, Krishnan et al. 2011).

Previous applications of feature reduction using the partial least square method in neuroimaging machine learning tasks include: Age classification (Young vs Old) (Chen et al. 2009) and prediction of cognitive behavioral scores (Ziegler et al. 2013, Menzies et al. 2007, Nester et al. 2002). Notably, PLS has also been used in multimodal feature reduction tasks (Sui et al. 2012, Martinez et al. 2004).

3.0 Unsupervised feature reduction techniques

Unsupervised feature reduction techniques also known as dimensionality reduction or feature extraction techniques construct relevant features through linear or non-linear combinations of the original predictor variables (features) (Lee and Verleysen, 2007). In this review, we restrict our discussion to two of the most popular unsupervised dimensionality reduction techniques in neuroimaging namely, principal component analysis and independent component analysis.

3.1 Principal component analysis

Principal component analysis (PCA) constructs relevant features by linearly transforming correlated variables (e.g. raw voxels in a brain scan) into a smaller number of uncorrelated variables also known as principal components (Joliffe, 2002). The resulting principal components are essentially linear combinations of the original data capturing most of the variance in the data (Joliffe, 2002; Mourao-Miranda et al. 2005; Mourao-Miranda et al. 2006; Zhu et al. 2005). Construction of principal components from high-dimensional data begins by first, normalizing the original features by subtracting sample mean and dividing by the standard deviation. Second, Eigen decomposition of the covariance matrix from the standardized data is performed resulting in eigenvalues and eigenvectors of the covariance matrix. Third, eigenvalues are sorted in a decreasing order effectively representing decreasing variance in the data (Joliffe, 2002). Lastly, principal components are constructed by multiplying the originally normalized data with the ‘leading’ eigenvectors whose exact number is a user-defined parameter. The ‘leading’ eigenvectors explain most of the variance in the data. As a result, highly-dimensional neuroimaging scans with potentially many correlated voxels are now represented by relatively few uncorrelated principal components which are later used for machine learning analyses. A detailed and rigorous mathematical derivation of PCA is given elsewhere (Joliffe, 2002).

PCA has been successful in extracting relevant features in neuroimaging classification studies. For example in Schizophrenia (Caprihan et al. 2008; Radulescu and Mujica-Parodi, 2009; Yoon et al. 2007), AD (Lopez et al. 2011), Psychosis (Koutsouleris et al. 2009), MDD (Fu et al. 2008), ADHD (Zhu et al. 2008) and face recognition in fMRI (Mourao-Miranda et al. 2005). Most notably principal components have also been used in neuroimaging machine learning regression studies. For example, AD clinical scores prediction (Wang et al. 2010) and age predictions (Franke et al. 2010).

PCA offers two major benefits to dimensionality reduction in neuroimaging machine learning studies. First, the technique is easy to implement and computationally efficient. Second, the technique is un-supervised- meaning it does not require corresponding categorical or continuous labels or targets to extract relevant features. However, PCA also suffers from several setbacks. First, the user is required to define the number of principal components although Hansen and colleagues (Hansen et al. 1999) propose using the generalization error from a cross-validation process to select an optimal number of principal components. Second, as principal components are linear combinations of the original features, they may not be easily interpretable (Bunea et al. 2011). Third, whilst constructing relevant components, PCA may ignore the required outcome (e.g. discrimination of disease vs healthy) (Bunea et al. 2011). Lastly, as principal components are constructed through a linear transformation, this process may not adequately detect more complex non-linear feature interactions (Bishop, 1995), such as those that may occur in neuroimaging scan data. This setback has led to the formulation of kernel-pca- which in brief is the application of PCA in a feature space created through a kernel function (Scholkopf and Smola, 2002). Several studies have recently explored the application of kernel-pca to dimensionality reduction problems in neuroimaging (Rasmussen et al. 2012; Sidhu et al. 2012; Thirion and Faugeras, 2003; Wang et al. 2011).

3.2 Independent component analysis

Independent component analysis (ICA) is a multivariate data-driven technique which belongs to the broader category of blind-source separation methods used to separate data into underlying independent information components (Stone, 2004). ICA separates a set of ‘mixed signals’ (e.g. raw data from an fMRI scan) into a set of independent and relevant features (e.g. paradigm-related signals in fMRI). To achieve this goal, ICA assumes the source signals are statistically independent from an unknown but linear mixing process (Calhoun et al. 2009).

To elucidate this, first we consider a i × j matrix X where (i = number of fMRI scans in a study and j = number of voxels from the pre-processed scans). Second, we consider a n × j matrix S with rows representing (n) number of ‘expected’ independent components. Additionally, we represent A as a i × n ‘mixing’ matrix with columns containing associated time-courses of the n components. Effectively, this becomes a ‘blind-source separation’ problem (Stone, 2004) which can be represented by a linear model X = AS (Calhoun and Adali, 2006). In addition, we consider the function Y = WX. Consequently, the goal of an ICA algorithm is to estimate the j × i ‘unmixing’ matrix W such that Y becomes a good approximation of the true signal sources S. ICA algorithms use high-order statistical methods to solve for independent components (ICs) and a gentle introduction to these methods is given elsewhere (Calhoun and Adali, 2006; Stone, 2004).

There are two broad variants of ICA applications in fMRI. The first variant depends on the dimension of priority (e.g. spatial dimension-spatial ICA, temporal dimension-temporal ICA). Majority of ICA dimensionality reduction studies in fMRI have mostly extracted relevant independent components from the spatial dimension (Calhoun et al. 2009) and a detailed evaluation of both spatial and temporal options is given elsewhere (Calhoun and Adali, 2006). The second category of ICA further sub-divides the technique into either individual-subject ICA or group-level ICA (Calhoun et al. 2001; Calhoun and Adali, 2006). In individual-subject ICA, each subject’s data are entered into a separate ICA analysis while in group-level ICA one set of components for the groups are estimated and back-reconstructed from an aggregate mixing matrix to obtain individual-subject independent components and discriminative maps. The group-level ICA back-reconstruction step is explored elsewhere (Erhardt et al. 2011). A Matlab (The Mathworks, Inc) implementation of group ICA for fMRI is made freely available elsewhere (Calhoun, 2011).

Benefits of employing ICA in dimensionality reduction tasks should be noted. First, unlike univariate methods, ICA does not require an investigator to specify a regressor of interest-which may require prior knowledge and assumptions about the experiment (e.g. paradigm or model regressor in fMRI) (Calhoun and Adali, 2006). Second, ICA has proved to be successful in disentangling otherwise mixed brain signals (e.g. separating physiological, motion and scanner related components) (Calhoun and Adali, 2006). However, potential setbacks of ICA should also be noted. First, ICA algorithms are computationally intensive (Correa et al. 2007). Second, according to Birn and colleagues (Birn et al. 2008), current ICA algorithms may not adequately separate respiration and default mode network signals in fMRI.

Notable neuroimaging machine learning studies using ICA in dimensionality reduction include (Castro et al. 2011a; Chai et al. 2010; De Martino et al. 2007; Douglas et al. 2011; Duff et al. 2011; Ince et al. 2008; Sato et al. 2012; Tagliazucchi et al. 2012; Toussaint et al. 2012; Yang et al. 2010). A review of ICA applications in multi-modal feature reduction tasks is given elsewhere (Sui et al. 2012). Lastly we note that a significant methodological difference between PCA and ICA exists. In PCA a set of possibly correlated variables are converted into a set of uncorrelated features, whilst in ICA original variables are transformed into a set of nearly statistically independent features (Calhoun and Adali, 2006). However, the main commonality between ICA and PCA and possibly a distinctive characteristic that separates them from filter, wrapper and embedded feature reduction techniques is that the former are unsupervised while the latter are supervised.

3.3 Coordinate-based meta-analysis (CBMA) techniques

In the previous sections, we have mostly focused on ‘data-driven’ feature reduction techniques. Here though, we discuss techniques which may rely on existing ‘domain knowledge’ for feature reduction. Meta-analysis techniques have previously been used to model, analyze and report brain activation or group-level differences across neuroimaging studies (Eickhoff et al. 2009). Popular meta-analysis techniques include; activation likelihood estimation (ALE) (Laird et al. 2005b), kernel-density estimation (Scott, 1992) and multi-level kernel density estimation (Wager et al. 2007). A more detailed discussion of these techniques in relation to the wider meta-analysis reporting in neuroimaging is given elsewhere (Eickhoff et al. 2009; Laird et al. 2005a; Laird et al. 2005b).

Recently though, CBMA techniques have been used in modeling distributions of reported foci from fMRI activation studies and resulting regions-of-interest used as input features for machine learning analyses. For instance, Yarkoni and colleagues (Yarkoni et al. 2011) recently applied a CBMA technique to select relevant features in classifying working memory, emotion and pain using fMRI. The same group has recently developed a CBMA feature reduction framework known as Neurosynth (Mitchell, 2011; Yarkoni et al. 2011). Neurosynth is a tool for automated synthesis of fMRI data as reported in published studies (Yarkoni et al. 2011). Durkat and colleagues (Dukart et al. 2012) have recently applied a CBMA technique in classifying AD subjects in multicenter studies with high generalization ability. Other studies exploring CBMA techniques in feature reduction have also been reported. For example, Dosenbach and colleagues (Dosenbach et al. 2010) used regions-of-interest (ROIs) derived from a fMRI meta-analysis as relevant features for predicting individual subject’s age. Doyle and colleagues (Doyle et al. 2013) used existing domain knowledge to select the thalamus, anterior cingulate cortex and occipital cortex as relevant regions to predict the effect of a drug (ketamine) on brain activity. Recently, Chu and colleagues (Chu et al. 2012), reported that features from ROIs selected using a priori domain knowledge (e.g. hippocampal degeneration in Alzheimer’s disease) resulted to better generalization ability as compared to features selected from a data-driven approach such as t-test or RFE.

Notably, as CBMA techniques pool activation foci coordinates from numerous neuroimaging studies, this approach may improve a-posteriori certainty, increase statistical power and therefore make neuroimaging studies less susceptible to type II errors (Wager et al. 2007). The latter is a common problem in neuroimaging studies with relatively small sample sizes. However, CBMA techniques may also suffer from information loss as they represent data in published studies with a high degree of sparseness (Salimi-Khorshidi et al. 2009).

4.0 Summary and Discussion

There is general consensus within the neuroimaging machine-learning community that feature reduction is an important process before training a machine learning model. The main benefits of this process are two fold. First, to remove any redundant features (voxels) plus noise a process which may improve prediction accuracy or generalization ability as well as support interpretability of study results. The latter may in turn help in generating post-hoc inferences.

In this review, we have noticeably distinguished feature reduction methods into two major categories namely; supervised and unsupervised techniques respectively. We have further subdivided supervised techniques into three subcategories namely; filter, wrapper and embedded techniques. In summary, filter techniques use statistical feature ranking criterions (e.g. labels vs feature correlations in PCC) to discard redundant features. However, these techniques have two common setbacks. First, they are not multivariate and as such they do not take into account interactions between multiple features and spatial patterns. Second, these techniques need a user to define an optimal relevant feature threshold value (e.g. absolute correlation coefficient in PCC or p-value in t-test), although previous studies have prevailed over this impediment by optimizing this parameter through cross-validation procedures. On the other hand, wrapper techniques are multivariate although computationally intensive. Notably, De Martino and colleagues recently attempted a hybrid combination of both filter and wrapper techniques (t-test and RFE) which had superior prediction accuracy as compared to both techniques alone. Lastly, we note that the performance of embedded feature reduction methods (LASSO and Elastic Net) strongly depends on the choice of penalization parameters which should be chosen through a cross-validation process (Bunea et al. 2011; Shi et al. 2007).

We note that a significant difference between supervised and unsupervised techniques exists. Unsupervised techniques construct features independent of the outcome of interest whilst supervised techniques choose relevant features based on their ability to detect group-level differences.

Pertinent issues that in general may determine the performance of many feature reduction techniques at a high-level should be noted. First, the need to set a threshold of optimal number of features. Second, In the event one is using a cross-validation process to train and test the model (e.g. leave-one-out), relevant features may differ from fold-to-fold. Notably, although feature reduction methods ultimately help in removing redundant data and noise, equally important and relevant features may be inadvertently removed during feature selection (Guyon and Elisseeff, 2003).

In conclusion, feature reduction techniques are frequently being used in neuroimaging machine learning studies to mitigate the curse-of-dimensionality or small-n-large-p problems and maximize prediction accuracies. Lastly, whilst there are a number of studies empirically comparing different feature reduction approaches (Craddock et al. 2009; De Martino et al. 2008; Ryali et al. 2010) in neuroimaging studies, none of these studies recommend any technique as the best in all neuroimaging machine learning tasks.

Table 1.

A tabular summary of both supervised and unsupervised feature reduction techniques.

Supervised	Unsupervised
a) Filter techniques -T-test, Anova, pearson correlation coefficient	a) Data driven -Principal component analysis -Independent component analysis
b) Wrapper techniques -Recursive feature elimination -Searchlight
c) Embedded techniques -Least absolute shrinkage and selection operator -Elastic Net -Partial least squares method	b) Domain Knowledge driven -coordinate-based meta-analysis