Abstract
Background
In the typical setting of gene-selection problems from high-dimensional data, e.g., gene expression data from microarray or next-generation sequencing-based technologies, an enormous volume of high-throughput data is generated, and there is often a need for a simple, computationally-inexpensive, non-parametric screening procedure than can quickly and accurately find a low-dimensional variable subset that preserves biological information from the original very high-dimensional data (dimension p > 40,000). This is in contrast to the very sophisticated variable selection methods that are computationally expensive, need pre-processing routines, and often require calibration of priors.
Results
We present a tree-based sequential CART (S-CART) approach to variable selection in the binary classification setting and compare it against the more sophisticated procedures using simulated and real biological data. In simulated data, we analyze S-CART performance versus (i) a random forest (RF), (ii) a fully-parametric Bayesian stochastic search variable selection (SSVS), and (iii) the moderated t-test statistic from the LIMMA package in R. The simulation study is based on a hierarchical Bayesian model, where dataset dimensionality, percentage of significant variables, and substructure via dependency vary. Selection efficacy is measured through false-discovery and missed-discovery rates. In all scenarios, the S-CART method is seen to consistently outperform SSVS and RF in both speed and detection accuracy. We demonstrate the utility of the S-CART technique both on simulated data and in a control-treatment mouse study. We show that the network analysis based on the S-CART-selected gene subset in essence recapitulates the biological findings of the study using only a fraction of the original set of genes considered in the study's analysis.
Conclusions
The relatively simple-minded gene selection algorithms like S-CART may often in practical circumstances be preferred over much more sophisticated ones. The advantage of the “greedy” selection methods utilized by S-CART and the likes is that they scale well with the problem size and require virtually no tuning or training while remaining efficient in extracting the relevant information from microarray-like datasets containing large number of redundant or irrelevant variables.
Availability
The MATLAB 7.4b code for the S-CART implementation is available for download from https://neyman.mcg.edu/posts/scart.zip.
Background
Variable selection, often also known as feature selection, feature reduction, attribute selection or variable subset selection, is the machine learning technique of identifying a subset of relevant features for building robust learning models. In modern molecular biology the technique is also often known as discriminative gene selection, devised to detect influential genes based on data from DNA microarray experiments. By removing most irrelevant and redundant features from the data, feature selection helps improve the performance of learning models by alleviating the effect of the curse of dimensionality, improving the predictability and interpretability of the model as well as speeding up the learning process.
Due to its importance in many areas of statistical inference, various methods of variable selection have been proposed in the literature over last two decades with a noticeable tendency to favor algorithms increasingly conceptually complex and computationally expensive (e.g., SSVS method of [1]). However, as discussed e.g., in [2] more and more evidence seems to indicate that in many variable selection problems involving complicated stochastic structure (e.g., genes or proteins selection), the computationally intense methods may loose grounds against their simpler alternatives. This is especially so in the so-called large p small n problems, i.e., when the amount of the information in the data is far outpaced by the number of parameters and their complicated underlying interactions. See, for instance, [3] for a good discussion and an overview. The purpose of the current paper is to show, by means of examples based on simulated and real biological data, that at least in the context of “gene selection”-type problems, i.e., under the assumptions often made when analyzing DNA microarray data (e.g., under the LIMMA Bayesian model assumptions, see [4]), the powerful and sophisticated methods for variable selection may be often outperformed by relatively simple, sequential feature-selection algorithms. Furthermore, these simple methods preserve the biological information when applied to real biological datasets.
Herein we consider a set of numerical simulations where we compare some of the most sophisticated variable selection methods currently available to a simple sequential classification-and-regression-tree algorithm (throughout dubbed ‘S-CART’). In the next section (Methods) we present the general setup of the experiment and describe the other selection algorithms used for comparison with S-CART procedure as well as describe the hierarchical model used for deriving the numerical data. In Results section we summarize and discuss our findings based on both the simulated data as well as the real biological data from the mice microarrays experiments described in [5]. For the benefit of the readership less familiar with the statistical learning methods applied for variable selection, we also offer a brief overview of these techniques in the Appendix.
Methods
In this section we briefly describe the variable selection methods compared as well as give an overview of the setup for the data generation for the numerical comparisons experiment. For the sake of simplicity and to make our points stand out more clearly, we restrict ourselves herein to the binary classification problem only, i.e we consider only two classes to discriminate between.
Variable Selection Procedures
The competitors to be compared with the sequential classification and regression tree algorithm (S-CART) were picked so as to represent different respective classes of methods. And so for (i) ensemble classifiers we consider the well-known Breiman random forest algorithm [6], (ii) for Bayesian stochastic searches we consider a method developed by [1] and finally (iii) for multiple-testing based selection methods we consider a gene-specific, multiplicity adjusted t-test criterion which has recently been shown to have many near-optimality properties from the viewpoint of the statistical decision theory (??Dudoit).
Sequential classification-and-regression-tree variable selection
Our sequential classification and regression tree algorithm (S-CART) is based on the following procedure. Let F = {1,…, p} be an initial finite collection of variables of interest in the variables selection problem for the purpose of binary classification. In our sequential variable selection approach we use the CART procedure (see appendix) with the Gini index criterion for the algorithm termination and the usual CART-type complexity criterion for the final pruning of the set of variables. Thus, we proceed iteratively according to the following:
Algorithm 1: S-CART Variable Selection
Step 1 Follows the usual CART algorithm and choose split at some variable jk; label it “significant variable”. Compute the corresponding Gini index Gjk (cf. (A.1) in Supplemental Materials)
Step 2 Remove the variable found in step 1 so the new set of variables becomes F = F \ {jk}.
Step 3 Continue alternating between steps 1 and 2 until the relative Gini index Gjk/Gjk+1 falls below a given threshold ∈ > 0 or F = Ø.
Step 4 For the selected set of “significant variables” run the usual CART pruning based on the cost-complexity criterion (cf. (A.2) in Appendix).
This algorithm in essence prunes the feature space of “significant variables”, by sequentially running CART, until the first available split, on the progressively smaller set of variables. The final removed set of features is then declared “significant” and subsequently pruned again in the last step of the algorithm to remove highly correlated / redundant ones. We note that in the numerical simulations described in the next section, in order to control the missed discovery rate, we use a simplified version of the algorithm when the stopping criterion is dictated by the number of true features and the selected variables are not pruned (Step 4 is not performed). A somewhat similar approach has been utilized with random forest method through iteratively constructing an ensemble of trees, and at each iteration eliminating variables considered “non-significant” [7].
Random forest selection
Random forest (RF) is a classification method in many ways similar to CART, but unlike the latter, RF is derived from an entire ensemble of decision trees run on the bootstrapped data, where final classification is based on a “majority-vote” of the predicted membership of the ensemble of trees. With p variables and n observations, we randomly sample the set of n observations with replacement to create N the so-called bootstrap samples where N is a sufficiently large number to allow for the inference on their distribution (in our simulations below we took N = 500). We call the observations left out of any bootstrap sample (by an elementary calculation, one can see that on average the proportion of 1/ exp(1) of the original sample is left out) the “out of bag” (OOB) data. For each bootstrap sample we grow the classification tree in a CART-like fashion (without pruning), except that at each splitting node, we make a split based on a random m-subset of the p variables, where typically . This is done for reducing the correlation between trees and improving the accuracy of classification. For each tree grown on the bootstrap sample, we classify OOB and, as with CART, we use the Gini index for measuring node impurity. The mean decrease in nodes impurity, as measured by the Gini index when splitting on a variable, is then used for the variables selection. That is, all the variables are ranked by their mean decrease in node impurity as measured by the Gini index. The top ranked ones are then taken as “significant.” This particular criterion for the variable selection procedure is convenient since the appropriate Gini index calculations are provided in the R-software implementation of RF (available from CRAN as a package called randomForest written by A. Liaw and M. Wiener [8]). The RF-based feature selection method of similar flavor was recently applied to mass spectrometry data in breast cancer patients [3].
Stochastic search selection
Recently, [1] implemented a fully parametric Bayesian variable selection methodology that builds upon the stochastic search variable selection technique (SSVS, see e.g., [9]). The idea of SSVS is to imbed a simple linear regression model [10] into a hierarchical Bayes model in which the regression coefficients are given mixture-type priors [11]. [12] extended this approach to the general multivariate case for probits and multinomial classification. This is accomplished through the use of latent (unobserved) variables representing the class labels or categorical response [12]. A key idea for variable selection is construction of a latent binary vector that indexes all possible variable subsets. Through appropriate prior elicitation and integration of parameters out of the joint posterior, the posterior distribution of all possible subsets given the data can be derived, albeit in non-explicit form. See [13] for details of prior specification. This posterior space is searched via the Metropolis-Hastings algorithm using add/delete/swap-type moves. This Markov chain Monte Carlo procedure (MCMC) allows variables to be selected via marginal or joint posterior probability. A key attractive feature of this technique is that it assesses the joint effects of the variables and thus possibly identifies sets of combined predictors which in separation could predict poorly. Because variable subsets are selected on the basis of maximal joint posterior density, SSVS inherently attempts to avoid correlation of included variables, as inclusion of an additional highly-correlated variable (at the expense of a potentially valuable predictor) into a model brings no additional predictive power, leading to no increase in joint posterior probability. This stochastic search methodology is ideal for data of dimension much larger then the number of observations (p ≫ n). Specifically, SSVS works well with data that inherently has substructure [14], and it has also been demonstrated to work well with data having a dependency structure, i.e. correlated covariates [15,16].
LIMMA selection
In the binary classification setting, variable selection is achieved using the moderated t-test statistic from the hierarchical Bayesian regression model implemented in the R LIMMA package [4]. More details on LIMMA are provided in the next section.
Data Generation
In this subsection we provide a general description of our simulation experiment and the underlying Bayesian model used to generate the data. The model used here is a simplified version of the hierarchical model applied to microarray data settings by various researchers and implemented in a Bioconductor package LIMMA [4]. In our experiments we generated under various model parameters sets of n = 10 independent copies of mutually independent (given all model parameters) p-dimensional random vectors and . This data followed a model, where k components of the vectors were taken as (marginally) different densities so as to create the “significantly” marginally different variables. More precisely, we took the components of the to be independent, conditionally on the parameters, under the following Bayesian structure
| (1) |
for l = 1, 2 (number of classes), i = 1,…,p (number of variables), and j = 1,…, n = 10 (number of samples). Here P(x|μ, σ2) is the normal density N(μ, σ2), and we assume that the density of μ, is where as the density of σ2, P(σ2|γ, τ) is the inverse gamma, IG(γ, (γ − 1)τ). The structure of the hyper-parameter values applied under all the scenarios was as follows
The particular numerical values of the hyper-parameters were calibrated so that they would result in a good range of the partial FDR values for various algorithms in order to facilitate their comparisons (see the next section). However, in our experience even relatively large departures from these numerical values did not seem to effect substantially the outcomes of the simulation study.
The 14 model scenarios were created by varying the values of the first “significant” k/2 components mean in both groups (i.e., by adjusting the size of μ1) as well as the number of variables p and a corresponding number of significant variables k on the following numerical grid
Detailed specifications for each of the scenarios are given in Table 1. For each of the 14 sets of parameters we have generated 500 random realizations of the two groups corresponding to the two classes with 10 samples in each one.
Table 1. Parameters specification for each of the 14 scenarios used for the hierarchical model.
| Scenario no | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | |
| p | 100 | 100 | 500 | 500 | 500 | 500 | 1000 | 1000 | 1000 | 1000 | 5000 | 5000 | 5000 | 5000 |
| k | 10 | 10 | 10 | 10 | 50 | 50 | 20 | 20 | 100 | 100 | 100 | 100 | 500 | 500 |
| μ1 | 1 | 0.25 | 1 | 0.25 | 1 | 0.25 | 1 | 1 | 0.25 | 1 | 0.25 | 1 | 0.25 | 1 |
The in our Bayesian model above were, given model parameter values, independent draws. However, for completeness, we have also extended the Bayesian model above so as to include scenarios with dependency. This was done by adding an additional normal variate to certain groups of variables. More precisely, assuming all variances are unity, we added the normal variable aY to some number of to create a cluster of correlated variables. The correlation of two cluster members was
Thus, by manipulating the value of a, we could obtain any positive correlation value desired for a cluster. Assuming all variances to be unity, we adjusted the density of σ2, P(σ2|γ,τ), to be inverse gamma, and set various values of a so as to obtain the independent (given model parameters) clusters of correlated variables with intra-cluster correlation values ρ = {0, 0.1, 0.5, 0.7, 0.9}. The number of variables included in each of these clusters was taken as fraction of the total of p variables, {0.2p, 0.2p, 0.2p, 0.2p, 0.1p} totaling 0.9p. These 0.9p variables constituted the non-significant variables. For the remaining k = {0.10p} significant variables, we specify half, or 0.05p, to be correlated with ρ = 0.5 and the other 0.05p to be independent.
The Comparison Criteria
In the seminal paper [17] introduced the false discovery rate (FDR) as the measure of accuracy for the multiple selections procedure:
| (2) |
Here R is the total number of variables declared significant, V is the number of “false” declarations, and it is assumed that V = 0 when R = 0. FDR gives us a convenient way of the selection algorithms comparison. Since each of the algorithms considered here provides an ordering to the selected variables in terms of their “importance”, the partial false discovery rate calculated as the function of the first top i selected variables (denoted as FDRi) allows us to see the efficacy of the ranking methods used by the various selection algorithms. And thus if Vi is the number of false discoveries among top i variables declared as significant, define the corresponding partial FDR as
| (3) |
For the purpose of our comparisons below we approximate FDRi with its empirical counterpart based on 500 runs of each set of samples in each scenario for S-CART, RF, and t-test. FDRi for SSVS is approximated using 30 runs of each set of samples in each scenario due to computational expense. Note that with k being the true number of significant variables the FDRk gives as also the ‘missed discovery rate’, i.e., identifies the average number of significant variables which were missed by the selection algorithm.
The Implementation
The algorithms used for the variable selections procedures compared herein were implemented in Matlab version 7.4b and R 2.10. The computations were performed on an Apple Quad Core Mac Pro workstations with dual 3.2 GHz Quad-Core Intel Xeon CPU and 8GB of memory. For details of implementation, including pre-processing procedures for the methodologies, prior calibration, etc., see the Appendix.
Results and Discussion
The results of our simulation experiments are summarized in Table 2 in terms of the overall FDRk (which in this case is also a missed discovery rate, cf. Section). Tables 3, 4, and 5 summarize the results our the simulations with dependency. Plots of some selected partial FDRs are presented in Figures 1, 2, and 3, corresponding to scenario 9 without correlated variables, scenario 9 with correlated variables, and scenario 13 without correlated variables, respectively. Figure 2 displays the partial FDRs corresponding to Table 3. Figure 3 displays the partial FDRs for the challenging scenario of relatively large k/p ratio but small μ1. All figures include the usual 95% confidence interval in dotted lines (calculated from the empirical percentiles).
Table 2. The comparison of the various false (or missed) discovery rates FDRk for the four algorithms in 14 scenarios considered in our numerical experiment with independent draws from Bayesian model.
| Scenario no | Parameters | FDRk | ||||||
|---|---|---|---|---|---|---|---|---|
| p | k | μ1 | S-CART | RF | j-SSVS | m-SSVS | LIMMA | |
| 1 | 100 | 10 | 0.25 | 0.51 | 0.52 | 0.67 | 0.65 | 0.56 |
| 2 | 100 | 10 | 1.00 | 0.16 | 0.17 | 0.67 | 0.65 | 0.17 |
| 3 | 500 | 10 | 0.25 | 0.67 | 0.69 | 0.87 | 0.86 | 0.68 |
| 4 | 500 | 10 | 1.00 | 0.26 | 0.29 | 0.67 | 0.64 | 0.25 |
| 5 | 500 | 50 | 0.25 | 0.51 | 0.55 | 0.75 | 0.75 | 0.56 |
| 6 | 500 | 50 | 1.00 | 0.18 | 0.23 | 0.62 | 0.61 | 0.19 |
| 7 | 1000 | 20 | 0.25 | 0.68 | 0.71 | 0.88 | 0.87 | 0.69 |
| 8 | 1000 | 20 | 1.00 | 0.25 | 0.29 | 0.76 | 0.76 | 0.24 |
| 9 | 1000 | 100 | 0.25 | 0.52 | 0.58 | 0.78 | 0.78 | 0.56 |
| 10 | 1000 | 100 | 1.00 | 0.18 | 0.26 | 0.66 | 0.65 | 0.18 |
| 11 | 5000 | 100 | 0.25 | 0.70 | 0.77 | - | - | 0.73 |
| 12 | 5000 | 100 | 1.00 | 0.28 | 0.41 | - | - | 0.30 |
| 13 | 5000 | 500 | 0.25 | 0.52 | 0.72 | - | - | 0.56 |
| 14 | 5000 | 500 | 1.00 | 0.18 | 0.52 | - | - | 0.20 |
Table 3.
The comparison of the various false (or missed) discovery rates FDRk for the four algorithms in the Bayesian model with dependencies for all significant variables.
| Scenario no | Parameters | FDRk | ||||
|---|---|---|---|---|---|---|
| p | k | μ1 | S-CART | RF | LIMMA | |
| 9 | 1000 | 100 | 0.25 | 0.47 | 0.52 | 0.48 |
| 10 | 1000 | 100 | 1.00 | 0.29 | 0.35 | 0.28 |
Table 4.
The comparison of the various false (or missed) discovery rates FDRk for the four algorithms in the Bayesian model with dependencies for {0.05p} independent (uncorrelated) significant variables only.
| Scenario no | Parameters | FDRkρ=0 | |||||
|---|---|---|---|---|---|---|---|
| p | k | kρ=0 | μ1 | S-CART | RF | LIMMA | |
| 9 | 1000 | 100 | 50 | 0.25 | 0.35 | 0.41 | 0.41 |
| 10 | 1000 | 100 | 50 | 1.00 | 0.20 | 0.27 | 0.23 |
Table 5.
The comparison of the various false (or missed) discovery rates FDRk for the four algorithms in the Bayesian model with dependencies: {0.05p} correlated significant variables only.
| Scenario no | Parameters | FDRkρ≠0 | |||||
|---|---|---|---|---|---|---|---|
| p | k | kρ≠0 | μ1 | S-CART | RF | LIMMA | |
| 9 | 1000 | 100 | 50 | 0.25 | 0.59 | 0.62 | 0.55 |
| 10 | 1000 | 100 | 50 | 1.00 | 0.37 | 0.44 | 0.33 |
Figure 1. Variable Selection for 1000 Independent Variables.

Comparison of partial FDR for p = 1000 with k = 0.1p in Bayesian model with uncorrelated variables.
Figure 2. Variable Selection for 1000 Variables with Dependance Structure.

Comparison of partial FDR for p = 1000 with k = 0.1p in Bayesian model with correlated variables.
Figure 3. Variable Selection for 5000 Variables with Dependance Structure.

Comparison of partial FDR for p = 5000 with k = 0.1p in Bayesian model with correlated variables.
The overall performance comparison across the models shows that S-CART and LIMMA outperformed random forest fairly consistently for all p and all three tree-based methodologies outperformed the SSVS method in all scenarios. This rank performance holds for correlated and independent variables in the Bayesian model. The S-CART uniformly had the lowest partial FDR of all the selection algorithms for the Bayesian model with independent variables for μ1 = 0.25. In the Bayesian model with independent variables for μ1 = 1, S-CART generally outperformed LIMMA, RF, and SSVS, but S-CART and LIMMA had similar performance in scenarios 4, 8, and 10. This indicates that S-CART is better able to identify significant variables than LIMMA when the mean difference between groups of independent variables is smaller.
For the Bayesian model with dependencies (through introduction of correlated variables), S-CART and LIMMA outperformed RF in every scenario. SSVS was not attempted in the Bayesian model with dependencies due to its poor performance in the independent draws model and slow running time. Interestingly, S-CART had superior performance to LIMMA in FDRkρ=0 for independent significant variables (Table 4, while LIMMA had better performance to S-CART in FDRkρ≠0 for dependent significant variables (Table 5).
Of the tree-based methods, the random forest technique had the poorest performance in terms of FDRk. In our numerical comparisons RF grows a large number (500) of bootstrapped trees. However, it is clear from the simulations that a fixed and unvarying number of trees is not appropriate, as random forest's error significantly increases and displays very different qualitative behavior for p = 5000, k = 500, and μ1 = 0.25 and 1. For these large datasets, FDRk stayed very low for the first k = 100 or so variables, but FDRk has a “bump” in its FDRk. Figure 3 shows this clearly. For μ1 = 1, there is another large “bump” in the FDRk around 275 variables, where FDRk rapidly increases (see Appendix). We believe this to be due to an insufficient number of bootstrapped trees grown (500). Because of this, random forest could not get adequate sampling or coverage of the input space. A much larger number of trees should be grown for random forest for high-dimensional data. In the context of gene-expression studies where microarrays are used, often having 25,000 to 50,000 variables, random forest does not appear to a be a good algorithm for detection of simple differential gene-expression differences, as it will necessitate a large number of trees and result in unacceptably slow algorithm. Also, this illustrates the fact of deteriorating performance of the tree-based methods as one moves further down a tree. The tree organization at lower levels of the tree is highly sensitive to changes and misclassifications at the higher level of the tree. This may be due to perturbations in the sampling data, due to insufficient coverage of the input space, resulting in an overall higher variance.
It is interesting to note the performance of S-CART and LIMMA in the Bayesian model with independent variables, where S-CART generally outperforms LIMMA. This is not surprising, as through the Gini index node impurity criterion, S-CART essentially picks off variables in a greedy fashion that allow for maximal discriminative classification efficacy, i.e. sequentially picking best hyperplanes. The greedy nature of the S-CART implementation gives it an extremely fast running time, eclipsing the execution speed of all the tests in this paper. These results also suggest that the S-CART node impurity measure (Gini index) utilized in this paper is highly efficient.
Introduction of correlated variables in the Bayesian model resulted in more performance parity between S-CART and LIMMA. However, both S-CART and LIMMA outperformed random forest in all scenarios. From our simulations the overall poor performance of the SSVS is apparent. In all the scenarios SSVS has, by far, the largest FDRk and a comparatively large standard deviation, i.e. low consistency. The much larger variance for SSVS is most likely due to the comparatively smaller number of runs for SSVS (30) in comparison to the tree-based methods (500). It is interesting to note that variable selection via marginal posterior density outperformed selections with joint posterior density. Recall that joint posterior density uses the variables in a linear model to account for the discriminative change in the two experimental groups and thus, any model chosen via joint posterior density considers the joint effects of the model's constituitive variables. Because marginal posterior density outperformed joint posterior criteria, it is assumed that for this experiment, single variable models are preferred, as there was no benefit to increasing the complexity of the model with additional variables.
There are several reasons for this surprisingly poor performane of the SSVS. First and foremost, convergence was almost certainly not achieved. Despite a 200,000 iteration burn-in and additional 100,000 run, the model continued to displayed highly erratic performance. This means the sampler did not efficiently explore the posterior space, and probably did not converge to the equilibrium distribution when remaining 100,000 samples were taken to approximate the posterior. Another reason for poor performance is that multiple chains were not run. Running multiple chains and pooling MCMC output after discarding burin-in generally improves performance because the sampler starts from different initial conditions on its way to convergence to the posterior. Thus, more of the posterior space can be explored. This was not done in this experiment due to time/computational limitations. Another important reason for poor performance is the size of the dataset. To the best of our knowledge, the SSVS is generally run following an ANOVA or t-test dimension reduction retaining only relatively small subset of significant variables of size p < 1000 [14]. Because our current experiment utilized up to 5,000 variables, it is likely that the prior specifications and the calibrations suggested in the literature, while appropriate in the context of tested and published conditions [1], were not suitable for the current datasets. A contributing reason to poor performance in the binary classification setting is the sophistication of the technique itself. A linear model implemented as a mixture model with a varying number of components is perhaps overly complex for our relatively simple set of scenarios. In particular, SSVS attempted to adjust for the complicated dependence structure of the variables [14] which was simply not present in our data. We found it interesting that in our experiments SSVS always started with a very low FDRk, which then rapidly increased with k. Thus, SSVS seems well suited for finding a small number of discriminating variables. For microarray gene-expression analyses where the user is interested in only a small number of discriminating variables, e.g. k ≤ 10, SSVS remains a very good algorithm. However, we should note that, in comparison with other methods, SSVS's algorithm execution is extremely slow due to its complexity.
Application to Microarray Data
We demonstrate the usefulness of the S-CART algorithm as a microarray gene-reduction tool that preserves biological information with a small number of genes. This analysis is based on [5], who studied oral creatine (Cr) supplementation on aging in C57Bl/6J mice. [5] collected RNA for microarray profiling (Gene Chip Mouse Genome 430 2.0 Arrays (Affymetrix UK Ltd.)) from the brain hemispheres of six mice fed ad libitum on 1 Cr supplemented pellets for 6 months and seven mice fed on equicaloric pellets without Cr (Altromin). Strictly from gene expression analysis, the authors determined that neurogenesis, energy, protein biosynthesis, and behavior pathways were principally affected by Cr supplementation. We downloaded the microarray data of this paper from NCBI GEO. Pre-processing consisted of removing genes with missing entries, resulting in 42,991 total genes. In MATLAB, Algorithm 1 (S-CART) was run on this dataset (with no further pre-procesing), where we specified the procedure to make a single greedy search through the gene space. S-CART returned a subset of 250 significantly differentially-expressed genes. Figure 4 is a heatmap visualization of this gene subset and clearly shows the genes classify all 13 arrays correctly in binary fashion, e.g., into control (1-7) and treatment groups (8-13). To explore the biological relationships of this gene subset using Ingenuity Pathways Analysis (Ingenuity®Systems, http://www.ingenuity.com) (IPA), we calculated p-values (control vs. treatment) for these 250 genes using the moderated t-test statistic from the LIMMA package in R [4] and uploaded these into IPA. From this 250 gene subset, IPA generated a number of networks, and the top biological functions (all adjusted p-values < 0.05) of these networks were identified as: cellular development, nervous system development and function, neurological disease, protein synthesis, and nucleic acid metabolism. For illustration, Figure 5 depicts the highest scoring network (e.g., most relevant) for the 250 gene subset, with “neurodegeneration” disease process relationships included. The top biological functions and disease processes for this network were cellular development, nervous system development and function, and neurological disease. As may be seen in Figure 5, most of the connections in the network involved the gene CREB1 (down-regulated in Cr treated mice, p-value = 5.56e-05), which had been known to have a positive relationship with neurodegenerative processes, implying a neuroprotective role. These finding are very consistent with those reported by [5] based on a much larger collection of identified significant genes.
Figure 4. Heatmap from S-CART Analysis.

Heatmap of 250 genes returned from S-CART on data from [5]
Figure 5. IPA Top Scoring Network.

Top-scoring network returned from IPA based on S-CART gene selection.
Conclusions
In our comparative numerical analysis the FDR plots of ranked variables are monotonically increasing for all the selection algorithms, indicating that the procedures have efficiency in variable selection under our experimental scenarios. It is interesting to note that the overall variability of FDR for all the procedures, as measured by the 95% confidence envelopes remains fairly constant throughout the scenarios starting at approximately fifty significant variables (k = 50) indicting that the procedures are fairly consistent in their performance. Although the tree-based methods (RF and S-CART) did outperform the SSVS methodology, the SSVS did not do poorly given the large dataset sizes, small number of significant variables, and comparatively small number of runs (30 versus 500 for tree-based methods). However, our results suggest that SSVS technique requires extensive calibration and does not scale well on datasets well above 1,000 variables.
The robust performance of a simple-minded method like S-CART is of practical interest in the context of gene discovery, because in most microarray applications the number of variables (i.e. genes) p ≥ 20, 000 is much larger than number of samples (n). Microarrays measure mRNA transcript abundance (from which we infer gene expression), and thus often have a rich differential response reflected in gene-space, given some stressor [18]. Because of the complex interdependencies of genetic networks, and of biological systems in general, it is probable that in most settings many genes will undergo small and significant changes in gene expression. In such scenarios, k/p is relatively high and μ1 relatively small. According to our simulation analysis, in such cases the simple S-CART is seen to be a more nimble and sensitive method than either random forest, LIMMA, or SSVS. The other advantages of S-CART, besides its simplicity, are its non-parametric nature and no need for data-driven tuning, in contrast to having the necessity of specifying a number of trees in random forest (requiring prior knowledge and testing of the dataset) and the prior calibration in SSVS. As demonstrated using a real biological dataset S-CART can also quickly reduce the dimensionality of a microarray dataset without loosing biological information. The dataset considered here was derived from a control-treatment group mouse study investigating the neuroprotective effect of creatine [5]. Analyzing the genes returned by S-CART using IPA software, the top scoring (e.g., most relevant) pathways concerned cellular development, nervous system development and function, neurological disease, protein synthesis, and nucleic acid metabolism. Despite utilizing only a fraction of genes identified in [5], we see that the findings from our analysis of these top pathways returned by S-CART are identical with the conclusions of the original study.
Supplementary Material
Acknowledgments
This research was partially funded under NIH grants R01DE019243 and R01CA152158 to GR and K08DE021430 to CB.
Appendix
CART models
For the reader's convenience we present here a very brief description of a classification and regression tree (CART) learning method. CART is a tree-based method that partitions the feature space into a set of M regions R1, R2,…, Rm and then fits a constant cm (m = 1,…, M) into each of them (c.f., e.g., Hastie 2001) under some prescribed loss function. Albeit herein we use CART for the binary classification, it is easier to describe the method first in the regression setting. To this end consider the input variables X1i, X2i,…, Xpi with associated continuous response Yi for observations i = 1…, n. The fitted response, putting a constant cm in each region, is:
where I(·) denotes the usual indicator function. The constants cm are typically fitted under square loss function, i.e., by minimizing in which case it is immediate that the solutions ĉm are weighted averages of yi's in regions Rm, m = 1…, M,
where Nm is the number of element in the above sum (if Nm = 0 we take ĉm = 0). The CART algorithm proceeds to make successive binary partitions at some split variable Xj (j = 1…, p) and a split point s. Starting with all the data, selecting the best pair (Xj, s) of split variable and split point is generally infeasible, so a greedy algorithm is utilized that scans through X1 to Xp as follows. Consider a splitting variable Xj and a split point s in the support of Xj, and define the pair of half-spaces
We search for the splitting variable at j and a corresponding split point s that solve the minimization problem
Note that for any j and s, the inner minimization is achieved by:
so split point s for a split variable Xj can be determined quickly, and therefore by scanning through all the inputs X1, X2,…, Xp the finding of the pairs (j, s) is computationally feasible. Having found the best split, we partition the data into the two resulting half-spaces and repeat the splitting process in each of the regions and so on with the binary partitioning inducing a corresponding binary tree T0 in the space of all binary trees corresponding to all possible spits. The binary partitioning is utilized here because multiway splits could fragment the data too quickly, reducing the data available for the subsequent splits [19, chapter 9].
Typically, we are interested in a “pruned” tree, that is, a subtree T ⊂ T0 that can be obtained by collapsing non-terminal nodes (internal nodes) of T0. Define
and a cost-complexity criterion
where |T| is the number of terminal nodes in T. For given α ≥ 0 one is typically interested in a subtree Tα ⊂ T0 minimizing Cα (T). This formulation of the problem quantifies a trade-off between goodness-of-fit and a tree size through the parameter α ≥ 0, where α = 0 results in the full tree T0.
For K-classification problems where Yi ∈ {1,…, K}, with K ≥ 2 some modifications are needed in the above approach. Instead of ĉm, we estimate the proportion of observations from class k in node m as
To classify observations in region (node) m, we consider a majority vote, that is, assign all observations in node m to class k(m) = argmaxkp̂mk. In the classification problems, instead of the previously discussed least-squares criteria for splitting variables and defining regions Rm, other more convenient criteria of tree node impurity are typically used. They include misclassification error, the Gini index, and cross-entropy or deviance [19, chapter 9]. In the current paper, we use the Gini index, Gm where
| (4) |
Note that if instead of using a majority vote at node m we were to use a random classification rule assigning to class k (k = 1…, K) with probability p̂mk then the Gini index may be interpreted as the probability of misclassification in this random rule. The previous complexity criterion in our current setting may be modified as
| 5 |
Stochastic search variable selection
The paper [20] presents an implementation of a fully parametric Bayesian variable selection methodology that builds upon the stochastic search variable selection technique (SSVS). The main idea of SSVS is to imbed a linear regression model into a hierarchical Bayes model, where regression coefficients are given mixture priors. [20] extended the SSVS model to the general multivariate case for probits and multinomial classification. This was accomplished through the use of latent (unobserved) variables, representing the categorical response, and a latent binary vector to index all possible variable subsets. Through appropriate prior elicitation (e.g., conjugate) and integration of parameters out of the joint posterior, the posterior distribution of all possible subsets given the data can be derived, albeit in non-explicit form. This is accomplished by the usual Markov chain Monte Carlo (MCMC) procedure where the posterior space is searched via Metropolis-Hastings using add, delete and swap Metropolis moves. This SSVS implementation allows variables to be selected via marginal or joint posterior probability [21]. An attractive feature of Vannucci's SSVS technique is that it assesses the joint effects of the variables. Two variables independently may not be descriptive given the observed data, but in conjunction, may be valuable predictors. Because variable subsets are selected on the basis of maximal joint posterior density, the subsets will inherently attempt to avoid correlation of included variables, as inclusion of an additional highly-correlated variable (at the expense of a potentially valuable predictor) into a model brings no additional predictive power, leading to no increase in joint posterior probability. Some formal highlights are presented below and are based off [22] as well as [21].
Let (Z,X) be the observed data, where Zn×1 is the categorical multinomial response vector taking values 0, 1,…, J − 1 and Xn×p is the predictor matrix. Please note that the (variable, observation) indexing is used elsewhere in the paper, while (observation, variable) indexing is used in this subsection. A data augmentation approach uses latent unobserved data Yn×q, with q = J − 1 and defines
relating yi and the binary probit outcome zi which is assumed known.
It is assumed that Y comes from a multivariate normal distribution with common covariance
with Yi = (yi,1,…, yi,J−1) being the row vector of Y that corresponds to the ith subject. The relationship between unobserved Yi and zi becomes
In our current case considered in the paper J = 2, resulting in binary responses and univariate latent model.
[21] use the notation of [23] to cast the standard multivariate normal regression model as
where Y is an n × p random matrix, 1an n × 1 vector of 1′s, X the n × p design matrix (fixed), and B the p × q matrix of regression coefficients.
The prior of (α, B, Σ) is decomposed in [22] as
Conditional independence of α and B on Σ results in
Variable selection is achieved through the introduction of a latent binary p-vector γ in the π(B|Σ) prior, whose jth element γj takes values 0 or 1, corresponding to exclusion or inclusion of the jth variable in the model
The prior of γ is simplest to set as multivariate Bernoulli. Each γj is independent and Prob(γj = 1) = ωj and Prob(γj = 0) = 1 − ωj. The hyperparameters ωj must be specified for j = 1,…, p. The prior on the number of variables in a model m (number of entries γj = 1 in γ vector) is set through the expectation of prior of γ, i.e. , so that .
Conjugate priors are adopted for α, B and Σ, following [22], as
where IW is the inverse Wishart distribution described below. The key point here is that the prior covariances for α and B “are dependent on Σ in a way that directly extends the univariate regression natural conjugate prior distributions.” That is, through the embedding of γ into these priors that are conditional to Σ, γ directly influences the matrix p-variate distribution of B by deleting rows of B (columns of H). In this setting Σ takes now the inverse Wishart distribution (IW) with n degrees of freedom and q = J − 1 dimensions, and with shape parameter δ = n − J. If γj = 0, then the jth row of B has zero variance and its prior at this zero vector is degenerate. If γj = 1, the jth row of B has non-zero variance. [21,22] derived the posterior distribution of γ given (Y, X) by integrating α, B, and Σ from the joint posterior.
The model's observed data are (Z,X) and parameters are (α, B, Σ, γ). Posterior inference is complicated due to the use of latent variables. The posterior distribution is sampled by constructing a Markov Chain using the Metropolis-Hastings algorithm, where add, delete, and swap Metropolis moves explore the posterior space of variable subsets as indicated by the composition of γ. The resulting posterior distribution of γ ”encapsulates what we need to know about the effectiveness of different explanatory variables in explaining the variation in the q-responses γ.” As already noted, the main advantages of this implementation are that the subset selections assess the joint effect of variables in attempting to account for the discriminative changes seen in the response data and that this procedure truly determines how “valuable” a variable is. Disadvantages are computational demands due to the large posterior space (defined on all possible subsets, 2P i.e. {0, 1}P). It may not be possible to efficiently search over this space in extremely large datasets, even with extensive prior calibration. Also, there is risk that the MCMC procedure will not converge to the posterior distribution within a reasonable timeframe. However, this can be minimized by running multiple chains from different initial conditions for long periods, discarding all but the last iterations and pooling the MCMC.
Additional Details on Implementation
Data generated from the Bayesian model was used without preprocessing for S-CART and RF. S-CART was implemented in MATLAB using the Statistics Toolbox. For each CART iteration, we enabled “pruning” option to obtain a subtree T ⊂ T0 by collapsing non-terminal nodes (internal nodes) of T0. random forest was implemented in R using the package randomForest. An ensemble of 500 trees was grown for each random realization of the Bayesian model, with variable splits based on a random m-subset of the p variables, where .
SSVS was implemented in MATLAB using software available from [20]. Data for this technique had be pre-processed by rescaling by variable range and mean-centering the variables (columns). This was done so that rank(X) ≤ min{n − 1, p}. In microarray applications, the data is frequently quantile-normalized, so that the distribution of gene expression is similar across arrays. For the purposes of a large scale simulation, running and pooling long chains is impractical, and was not attempted here. A single chain was run for 300,000 iterations, where the first 200,000 were discarded as burn-in, for each dataset. Prior specification is also an issue, although the authors give procedures for calibration of the priors from the data and from theoretical constraints. In this experiment, author guidelines [1] as well as specification in the spirit of empirical Bayes are utilized and results compared.
Hyperparameters controlling model shrinkage, prior probabilities of variable inclusion, and model expectation size (the average number of variables to be included in the linear model) are all specified. Prior probabilities of variable inclusion ωj are assigned independent Bernoulli trials. In the spirit of empirical Bayes, we assign ωj to be (1-p value) from a two-tailed t-test (calculated from rescaled dataset). Model size is given an expectation of 10, so that the MCMC procedure tends to explore the space of smaller models. That is,
The prior distribution for B given γ depends on the matrix Hγ. We choose to give H = cI, and c must be carefully calibrated from the data. This parameter relates the prior to posterior precision ratio. In general, we choose c such that
where λ̄ = mean(Diag{1/λ1,…, 1/λr}) and Diag{1/λ1,…, 1/λr}) is proportional to 1/Diag{cov(X)}. Thus, λ̄ = mean(1/Diag{cov(X)}). Because of the high correlations in the columns of X, we can calculate c based on the lower decile of the n − 1 nonzero eigenvalues per [1], i.e. where 10 percent of eigenvalues are less. This results in
where λ0.1 = quantile0.1{1/Diag(cov(X))}. For simplicity, we take
This value of c is highly robust to model construction, and typically ranges from 1 to 10, and for our model, generally falls around 5. The hyperparameter h is set h = 106. The starting γ vector is taken with 10 randomly selected γj = 1, with all other γj = 0.
Additional Experimental Results
The panel of plots in Figures 6 and 7 summarize the partial FDR's for scenarios 5, 6, 9, 10, 13, 14 (see Table 1 above) and the model with correlated variables. These have the usual 95 percent confidence intervals plotted in dotted lines. SSVS has exorbitant computational expense with scenarios 10-14 (where p = 1,000). Because of this, we were not able to generate a sufficient number of runs (e.g., 30) to establish, with any statistical precision, SSVS's FDR for p = 5,000 variables. However, because of the relatively poor performance of SSVS in comparison to the tree-based methods in the previous scenarios (and especially those where p = 1,000), we believe SSVS would have similar results as SSVS performed in previous scenarios.
Figure 6. Additional FDR Plots (I).

Comparison of partial FDR for p = {1000, 500}, with k = 0.1p in Bayesian model with uncorrelated variables.
Figure 7. Additional FDR Plots (II).

Comparison of partial FDR in Bayesian model with uncorrelated (top panels, p = 5000) and correlated (bottom panels, p = 1000) variables. In both cases k = 0.1p.
The bottom right plot in Figure 7, where p = 5000, k = 500, and μ1 = 1 demonstrates the effect of insufficient coverage of the input space for Random Forest, as there is a noticeable “bump” in FDRk-around 270 variables.
Footnotes
Authors contributions: GR has developed the main idea and CB has implemented the S-CART method and performed all numerical simulations. Both authors have designed the experiments and wrote the manuscript.
Contributor Information
Caleb D. Bastian, Email: cbastian@math.princeton.edu.
Grzegorz A. Rempala, Email: grempala@mcg.edu.
References
- 1.Sha N, Vannucci M, Tadesse MG, Brown PJ, Dragoni I, Davies N, Roberts TC, Contestabile A, Salmon M, Buckley C, Falciani F. Bayesian variable selection in multinomial probit models to identify molecular signatures of disease stage. Biometrics. 2004;60:812–819. doi: 10.1111/j.0006-341X.2004.00233.x. [DOI] [PubMed] [Google Scholar]
- 2.Marbach D, Prill R, Schaffter T, Mattiussi C, Floreano D, Stolovitzky G. Revealing strengths and weaknesses of methods for gene network inference. Proceedings of the National Academy of Sciences. 2010;107(14):6286. doi: 10.1073/pnas.0913357107. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Datta S. Classification of breast cancer versus normal samples from mass spectrometry profiles using linear discriminant analysis of important features selected by random forest. Stat Appl Genet Mol Biol. 2008;7 doi: 10.2202/1544-6115.1345. Article7. [DOI] [PubMed] [Google Scholar]
- 4.Smyth GK. Linear models and empirical bayes methods for assessing differential expression in microarray experiments. Stat Appl Genet Mol Biol. 2004;3 doi: 10.2202/1544-6115.1027. Article3. [DOI] [PubMed] [Google Scholar]
- 5.Bender A, Beckers J, Schneider I, Hölter S, Haack T, Ruthsatz T, Vogt-Weisenhorn D, Becker L, Genius J, Rujescu D, Irmler M, Mijalski T, Mader M, Quintanilla-Martinez L, Fuchs H, Gailus-Durner V, de Angelis M, Wurst W, Schmidt J, Klopstock T. Creatine improves health and survival of mice. Neurobiol Aging. 2008;9:1404–11. doi: 10.1016/j.neurobiolaging.2007.03.001. [DOI] [PubMed] [Google Scholar]
- 6.Breiman L. Tech rep. University of California; Berkeley: Random Forests; p. 2001. [Google Scholar]
- 7.Diaz-Uriarte R, de Andres SA. Variable selection from random forests: application to gene expression data. 2005 http://www.citebase.org/abstract?id=oai:arXiv.org:q-bio/0503025.
- 8.Liaw A, Wiener M. Classification and Regression by randomForest. R News. 2002;2(3):18–22. http://CRAN.R-project.org/doc/Rnews/ [Google Scholar]
- 9.George EI, McCulloch RE. Variable selection via Gibbs sampling. Journal of the American Statistical Association. 1993:88–881. [Google Scholar]
- 10.Tanner MA, Wong WH. The calculation of posterior distributions by data augmentation. J Amer Statist Assoc. 1987;82(398):528–550. With discussion and with a reply by the authors. [Google Scholar]
- 11.George EI, McCulloch R. Approaches to Bayesian variable selection. Statistica Sinica. 1997;7:339–373. [Google Scholar]
- 12.Albert JH, Chib S. Bayesian analysis of binary and polychotomous response data. J Amer Statist Assoc. 1993;88(422):669–679. [Google Scholar]
- 13.Brown PJ, Vannucci M, Fearn T. Multivariate Bayesian variable selection and prediction. J R Stat Soc Ser B Stat Methodol. 1998;60(3):627–641. [Google Scholar]
- 14.Swartz MD, Mo Q, Murphy ME, Lupton JR, Turner ND, Hong MY, Vannucci M. Bayesian Variable Selection in Clustering High-Dimensional Data With Substructure. Journal of Agricultural, Biological, and Environmental Statistics. 2008;13(4):407–423. http://pubs.amstat.org/doi/abs/10.1198/108571108X378317. [Google Scholar]
- 15.Swartz MD, Kimmel M, Mueller P, Amos CI. Stochastic search gene suggestion: a Bayesian hierarchical model for gene mapping. Biometrics. 2006;62(2):495–503. doi: 10.1111/j.1541-0420.2005.00451.x. [DOI] [PubMed] [Google Scholar]
- 16.Swartz MD, Shete S. The null distribution of stochastic search gene suggestion: a Bayesian approach to gene mapping. BMC Proc. 2007;1(Suppl 1):S113. doi: 10.1186/1753-6561-1-s1-s113. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J Roy Statist Soc Ser B. 1995;57:289–300. [Google Scholar]
- 18.Stoughton RB. Applications of DNA Microarrays in Biology. Annual Review of Biochemistry. 2005;74:53–82. doi: 10.1146/annurev.biochem.74.082803.133212. http://dx.doi.org/10.1146/annurev.biochem.74.082803.133212. [DOI] [PubMed] [Google Scholar]
- 19.Hastie T, Tibshirani R, Friedman J. Elements of Statistical Learning: Data Mining,Inference, and Prediction. New York: Springer-Verlag; 2001. [Google Scholar]
- 20.Vannucci M. Matlab Code for Bayesian Variable Selection. ISBA Bulletin. 2000;7:3. [Google Scholar]
- 21.Vannucci M, Colla V. Meaningful discretization of continuous features for association rules mining by means of a som. Proceedings of the European Symposium on Artificial Neural Networks; Citeseer. 2004; pp. 489–494. [Google Scholar]
- 22.Brown P, Vannucci M, Fearn T. Multivariate Bayesian variable selection and prediction. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 1998;60(3):627–641. [Google Scholar]
- 23.Dawid AP. Some matrix-variate distribution theory: notational considerations and a Bayesian application. Biometrika. 1981;68:265–274. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
