Local Sparse Bump Hunting

Abstract

The search for structures in real datasets e.g. in the form of bumps, components, classes or clusters is important as these often reveal underlying phenomena leading to scientific discoveries. One of these tasks, known as bump hunting, is to locate domains of a multidimensional input space where the target function assumes local maxima without pre-specifying their total number. A number of related methods already exist, yet are challenged in the context of high dimensional data. We introduce a novel supervised and multivariate bump hunting strategy for exploring modes or classes of a target function of many continuous variables. This addresses the issues of correlation, interpretability, and high-dimensionality (p ≫ n case), while making minimal assumptions. The method is based upon a divide and conquer strategy, combining a tree-based method, a dimension reduction technique, and the Patient Rule Induction Method (PRIM). Important to this task, we show how to estimate the PRIM meta-parameters. Using accuracy evaluation procedures such as cross-validation and ROC analysis, we show empirically how the method outperforms a naive PRIM as well as competitive non-parametric supervised and unsupervised methods in the problem of class discovery. The method has practical application especially in the case of noisy high-throughput data. It is applied to a class discovery problem in a colon cancer micro-array dataset aimed at identifying tumor subtypes in the metastatic stage. Supplemental Materials are available online.

1 INTRODUCTION

We introduce an exploratory method for mode or class discovery of a (continuous or discrete) regression function of many continuous variables. In a typical setting that we have in mind, the number of input variables (predictors) is much larger than the number of measurements (so called p ≫ n case), such as in high-throughput data.

The question of how to identify intrinsic structure in high dimensional data is still largely unanswered. It is common to treat the task of identifying structure in the data with unsupervised approaches. Finding isolated data structures in an input space of several variables can be seen as a density estimation problem of joint probability densities, or as a clustering problem, or alternatively, as a pattern recognition problem. Although these notions are distinct, they are somewhat linked, and can be cast into the “mode/bump hunting” framework. Historically, “mode/bump hunting” studies using non-parametric methods has a long tradition in density estimation, where a number of approaches exist. Unfortunately, because of the difficulty of the problem, most of these non-parametric approaches focus on univariate or bivariate cases at most, with limited or no applicability in high dimension. Therefore, most if not all of these approaches (whether non-parametric or not) will not be applicable in high dimensional settings, which is the primary focus of this study.

Only a few non parametric methods have been proposed for testing multivariate modality (Burman and Polonik 2008; Hartigan and Mohanty 1992; Polonik 1995; Rozal and Hartigan 1994). Among them is the supervised non-parametric procedure for bump hunting in regression function, known as the Patient Rule Induction Method (PRIM) by Friedman & Fisher (Friedman and Fisher 1999). PRIM was originally intended for exploration of modal regions in multivariate datasets of continuous variables. The procedure, algorithmic in nature, makes few statistical assumptions and does not explicitly state a model, although one can be formulated (Wu and Chipman 2003). Apart from numerous successful applications of conventional PRIM in many fields (geology, marketing, finance, medicine, bioinformatics, process optimization) there has been only a few extensions of the original work, including a Bayesian model-assisted formulation of PRIM (Wu and Chipman 2003), a boosted version of PRIM based on Adaboost (Pei Wang et al. 2004), an extension of PRIM to censored responses (LeBlanc et al. 2002), and to discrete variables (Il-Gyo and Chi-Hyuck 2008).

In contrast to decision tree-based methods such as CART (Breiman et al. 1984), the Patient Rule Induction Method does not attempt to build a functional model of the target function over the entire input space to then find the bumps of local maxima (or minima) from the estimated target function (a formidable task in high dimension!). Instead, PRIM is designed to directly estimate the bumps of the target function by seeking a (possibly disjoint) domain of it, over which its response average is relatively high (or low).

One of the critiques made in the original work of Friedman & Fisher (Friedman and Fisher 1999) was the lack of measures of significance regarding modal regions. Also, although PRIM is intrinsically multivariate, it was uncertain from the original work how the algorithm would perform in very high dimension. As pointed out by their original designers, a conventional PRIM-based bump hunting procedure will fail or at least perform poorly in the cases of high collinearity and/or high correlation between variables (Friedman and Fisher 1999). These important but hard questions received attention only recently in the work of Polonik & Wang that shade light on its basic statistical properties and shortcomings in high dimension (Polonik and Wang 2007). Apart from the recent theoretical analysis of PRIM properties by Polonik & Wang, none of the previous extensions of PRIM specifically extend the method to make it work in very large multivariate settings, and especially when the number of variables dominates the number of observations (p ≫ n paradigm).

The paper is organized as follows. Section 2 exposes the problem with the help of a motivating example. Using simulated data, we show the difficulty of uncovering the real structure with conventional techniques. Section 3 lays out the overall method. Section 4 demonstrates the adequacy of our procedure and its performance in comparison to competitive methods. Also, we show how to apply the procedure to a large dataset from a colon cancer micro-array study aimed at investigating the question of phenotypic diversity and tumor heterogeneity in colon tumors. Section 5 ends off with some discussion. Supplemental Materials (4.6.5) will be available online at http://www.amstat.org/publications/JCGS.

2 MOTIVATING EXAMPLE

2.1 Goal & Simulation Setup

Suppose our task is to identify isolated classes of a target function without assuming their numbers, and locate their corresponding support(s) in an input space of variables. To show how standard or naive data mining techniques may be inadequate for this task (even in low dimension when p ≪ n), we set up the following simulated example. Consider a supervised problem with a p-dimensional random variable X of p input variables $\mathbf{X}={[{\mathbf{x}}_j]}_{j=1}^p$ and a discrete output variable y, taking on d = 1, …, D (unorderable) categorical values or class labels. Let’s assume for the moment that X is distributed as a D-component mixture of overlapping (or non-linearly separable) p-variate Gaussian distributions: $\mathbf{X}\sim {\Sigma }_{d=1}^D{\pi }_d{\phi }_p({\mathit{\mu }}_d,{\mathbf{\Sigma }}_d)$, where π_d denotes the mixing weights and φ_p(μ_d, Σ_d) denotes the probability density of a p-variate Gaussian distribution with mean μ_d and covariance matrix Σ_d for d = 1, …, D. Let the data consist in n i.i.d. simultaneous observations of input and output variables: ${\{{\mathbf{x}}_i,y_i\}}_{i=1}^n$. Suppose that each individual component is labeled with a distinct class label d for d = 1, …, D and that we only observe G < D of them. For illustration purposes, we let D = 3, G = 2, p = 100, and n = 2, 400. In this setup, the samples are labeled group G1 and group G2, within which we suspect that two hidden subgroups exist, later termed sub-group #1 and sub-group #2. The goal is to discover the correct number of (unobserved) class labels and match them to the correct components.

Basic distributional assumptions of our simulation stem from the nature of high-throughput data it aims at dealing with. We chose in the simulation setup a Gaussian mixture model with correlation for the input variables X. In sum, we assume that potential bump supports of transformed data are likely to sit within samples of correlated variables with near Gaussian distributions. Also, because unidentified sub-groups are in essence undistinguishable, it is likely that these sub-groups are correlated and differ little in location and dispersion. Values of the distribution parameters were as follows (rounded up to the third decimal):

$$\begin{array}{l}{\mathit{\mu }}_1={\left(\begin{array}{c}2.981\\ 4.565\\ 2.006\\ 1.000\pm 0.01\\ ⋮\\ 1.000\pm 0.01\end{array}\right)}_{(100\times 1)}{\mathbf{\Sigma }}_1={\left(\begin{array}{cccccc}0.945& 0.018& 0.016& \approx 0.001& \cdots & \approx 0.001\\ 0.018& 0.657& 0.024& \approx 0.001& \cdots & \approx 0.001\\ 0.016& 0.024& 0.105& \approx 0.001& \cdots & \approx 0.001\\ \approx 0.001& \approx 0.001& \approx 0.001& 0.010& \cdots & \approx 0.001\\ ⋮& ⋮& ⋮& \ddots & \ddots & ⋮\\ ⋮& ⋮& ⋮& & & \approx 0.001\\ \approx 0.001& \approx 0.001& \approx 0.001& \cdots & \approx 0.001& 0.01\end{array}\right)}_{(100\times 100)}\\ {\mathit{\mu }}_2={\left(\begin{array}{c}4.408\\ 3.095\\ 1.490\\ 1.000\pm 0.01\\ ⋮\\ 1.000\pm 0.01\end{array}\right)}_{(100\times 1)}{\mathbf{\Sigma }}_2={\left(\begin{array}{cccccc}0.304& 0.185& 0.008& \approx 0.001& \cdots & \approx 0.001\\ 0.185& 0.279& 0.005& \approx 0.001& \cdots & \approx 0.001\\ 0.008& 0.005& 0.112& \approx 0.001& \cdots & \approx 0.001\\ \approx 0.001& \approx 0.001& \approx 0.001& 0.010& \cdots & \approx 0.001\\ ⋮& ⋮& ⋮& \ddots & \ddots & ⋮\\ ⋮& ⋮& ⋮& & & \approx 0.001\\ \approx 0.001& \approx 0.001& \approx 0.001& \cdots & \approx 0.001& 0.01\end{array}\right)}_{(100\times 100)}\\ {\mathit{\mu }}_3={\left(\begin{array}{c}5.233\\ 2.007\\ 0.996\\ 1.000\pm 0.01\\ ⋮\\ 1.000\pm 0.01\end{array}\right)}_{(100\times 1)}{\mathbf{\Sigma }}_3={\left(\begin{array}{cccccc}0.410& 0.311& 0.016& \approx 0.001& \cdots & \approx 0.001\\ 0.311& 0.413& 0.020& \approx 0.001& \cdots & \approx 0.001\\ 0.016& 0.020& 0.096& \approx 0.001& \cdots & \approx 0.001\\ \approx 0.001& \approx 0.001& \approx 0.001& 0.010& \cdots & \approx 0.001\\ ⋮& ⋮& ⋮& \ddots & \ddots & ⋮\\ ⋮& ⋮& ⋮& & & \approx 0.001\\ \approx 0.001& \approx 0.001& \approx 0.001& \cdots & \approx 0.001& 0.01\end{array}\right)}_{(100\times 100)}\end{array}$$

2.2 Comparative Analysis

This class discovery problem could be cast in a standard unsupervised problem (e.g. clustering or density estimation), where one would try to identify D = 3 clusters or densities, and hope that (at least) two of them would be found in group G2. In this simulation, we compared the efficiency of class discovery by common supervised data mining algorithms: namely classification by support vector machine (SVM) (Cortes and Vapnik 1995), classification by tree (CART) (see 4.6.5 and (Breiman et al. 1984)), the Patient Rule Induction Method for mode hunting (PRIM) (see 4.6.5 and (Friedman and Fisher 1999)), as well as unsupervised ones: partitioning clustering (K-means), density estimation by one-class-classification SVM. In the latter method, the model tries to find the support of a distribution and thus allows for outlier/novelty detection. An attractive feature of PRIM is that it can handle categorical outcomes and recast a classification problem into a bump-hunting problem (see 4.6.5) and (Friedman and Fisher 1999; Hastie et al. 2001) and search for instance for hidden sub-class(es) and their support(s). Clearly, tree-based or SVM-based classification methods, which are designed to classify observations when the number of classes is fixed or known in advance, are typically not designed for a class discovery task, and are expected to perform poorly. CART is useful here as a comparative greedy rule induction method to the patience of PRIM.

Briefly, the Patient Rule Induction Method (PRIM) is as follows (see 4.6.5) and (Friedman and Fisher 1999; Polonik and Wang 2007). Consider a continuous p-random vector X with support S and an univariate output random variable y. Denote by p(X) the joint p.d.f. and by f_X(x) the regression (target) function: f_X(x) = E(y|X = x). The goal is to find a domain D ⊂ S, within which the average value f̄_D of f_X(x) is expected to be significantly larger than its average value f̄_S over the rest of the input space. In addition, one wishes that the support (size) of D, say β_D be greater than a minimal support threshold, say β. Algorithmically, PRIM partitions the input space in a sequence of boxes ${\{B_m\}}_{m=1}^M$ that collectively covers the domain D, where values of the target function f_X(x) are expected to have local maxima. The strategy employs a patient top-down peeling, followed by repeated bottom-up pasting. Two important meta-parameters control the box induction algorithm: (i) the peeling fraction 0 < α < 1 that controls the degree of patience used in the generation of each box, and (ii) the minimal box support threshold 0 < β < 1 expressed as a fraction of the whole data that is used in the stopping criterion. Each box B_m, m = 1, …, M, is described by a sub-rule involving the value-subsets s_j,m of each individual input variable x_j. In the case of real-valued inputs, the entire input space S ⊂ ℝ^p, s_j,m is a contiguous interval of the form $s_{j,m}=[t_{j,m}^{-},t_{j,m}^{+}]$, and each box is a p-dimensional hyper-rectangle in ℝ^p of the form $B_m={\otimes }_{j=1}^p[t_{j,m}^{-},t_{j,m}^{+}]$. Then each sub-rule becomes ${\mathfrak{R}}_{\text{m}}=\{\mathbf{x}\in B_m\}=\{{\cap }_{j=1}^p({\mathbf{x}}_j\in [t_{j,m}^{-},t_{j,m}^{+}])\}$ and the solution domain D is fully described by a disjunctive rule of M conjunctive sub-rules of the form: $\mathfrak{R}={\cup }_{m=1}^M{\cap }_{j=1}^p({\mathbf{x}}_j\in [t_{j,m}^{-},t_{j,m}^{+}])$.

The gap statistic is a method used for estimating the true number of clusters in a set of data, a notoriously difficult and central problem in cluster analysis (Supplemental Materials 4.6.5) and (Tibshirani et al. 2001). In this example, the gap statistic was tested over a range of 1–10 clusters of samples. We use the stopping rule of the gap statistic paper to estimate the optimal number k̂ of clusters, that is the smallest value of k for which the estimated gap statistic is maximal up to 1 standard deviation (see Supplemental Materials 4.6.5). Supplemental Figure 8 shows the resulting number of clusters optimizing the gap statistic using the objective gap statistic stopping rule. The optimal number of clusters of samples found by optimization of the gap statistic in the entire space was k̂ = 2 (Supplemental Figure 8).

Denote ${\{P_r\}}_{r=1}^R$ the partitions of the input variable space generated by a CART partition algorithm. Let’s further denote a tree T and a terminal node by r, representing a partition P_r of cardinal n_r = |P_r| for r = 1, …, R, such that ${\Sigma }_{r=1}^R{n}_r=n$. The recommended strategy for determining the optimal size of a tree in regression and classification problems is to grow an overly large tree, then prune it back using a cost-complexity pruning criterion to get a more parsimonious model. We used a pruning algorithm based on the minimization of the cross-validated residual mean deviance statistic as a function of the number of partitions or tree size r = 1, …, R (Ripley 1996). It is difficult to set a general stopping rule because the minimum of the deviance statistic is not always available and one usually has to look for a ’kink’ in the curve in that case. We found general to take as a stopping rule the average value of the tree sizes (rounded up to the nearest integer), which are less than their mean value, and for which the deviance statistic values are less than their mean value (see Supplemental Materials 4.6.5). Supplemental Figure 8 shows the resulting cross-validated number of terminal nodes as determined by growing and pruning the tree to its optimal size using the objective deviance statistic stopping rule. The number of partitions of the entire input space found by the deviance statistic was r̂ = 4 (Supplemental Figure 8).

2.3 Summary

Figure 1 summarizes graphically our simulation results in high dimension (p = 100) and for non-linearly separable distributions. Neither of the two statistics tested were able to detect the correct number of modes in our simulated dataset. Surprisingly, traditional unsupervised procedures (clustering and density estimation), which would naturally come to mind to address this problem, equivalently fail to uncover the three underlying distributions. On the other hand, traditional supervised approaches such as classification procedures by binary decision tree (CART), or support vector machine (SVM), or even ordinary bump hunting by patient rule induction method (PRIM), which one would imagine have an advantage because they use the class information of the response, were not able to efficiently uncover the three underlying distributions (when run in the entire space with default parameter values). Actually, tree-based or SVM-based classifiers, which are designed to work when the number of classes is fixed or known in advance, are typically not designed for a class discovery task, and should be expected to perform poorly. Interestingly, ordinary bump hunting by patient rule induction method (PRIM) did also perform poorly, giving a spurious number of modes (6) see Figure 1. Notice that in all cases these poor performances stem from a consistent failure to detect the two underlying sub-groups in group G2.

Figure 1.

Simulation setup and nature of the problem. All plots are projections from (X₁, ···, X_p) into subspace (X₁, X₂). Upper left: target groups from 3 overlapping target distributions (D = 3) with 2 class labels (G = 2) in the original input space (p = 100). 95% confidence ellipses and Ordinary Principal Components are shown. Upper middle: K-means clustering (2 clusters). Upper right: Support Vector Machine (2 classes). Bottom left: Density estimation by one-class SVM (2 densities). Bottom middle: CART partitioning (4 partitions). Bottom right: PRIM bump hunting (6 boxes).

So, current procedures, when used in an inappropriate or “naive” way, are likely to fail in this type of problem, even in very low dimension when p ≪ n. What this class discovery motivating example indicates is how even sophisticated tests and methods lack of guidance in this task. For instance, at the time of this writing there are no generally-accepted methods for choosing optimal PRIM meta-parameters. Next section introduces our methodology and how it can substantially improve the bump hunting efficiency when (i) an appropriate strategy is employed, (ii) modality is statistically tested, and (iii) an adequate parameter/tuning estimation is done altogether.

3 METHODS

3.1 Algorithm

The Patient Rule Induction Method (PRIM) was predicted to not perform well in very high dimension and when the variables are highly correlated or highly collinear (Friedman and Fisher 1999; Polonik and Wang 2007). To address these shortcomings, especially in the p ≫ n situation, we devised a local and sparse bump hunting procedure based on a derivation of PRIM. The idea is to use a “Divide & Conquer” strategy to search for bumps recursively and locally in each partition of the input space. The approach bears some similarities to Ooi (Ooi 2002) who used trees for density visualization and mode hinting. The idea also likens itself to Minotte (Minotte 1997) and Burman (Burman and Polonik 2008) in that we take advantage of testing the multi-modality locally. In addition, our premise is that a bump hunting will be facilitated if run in a local sub-space that is rotated in the directions of maximum variance of the data in that sub-space. Implicit is the assumption that a local rotation will be more relevant and favorable to the bump search than a global rotation of the entire space.

A natural solution of breaking the problem down is to use a supervised rule induction method that will create input space partitions, wherein modes or classes are assumed to exist. Tree-based algorithm like CART (Breiman et al. 1984) is a good candidate at this stage. As for determining the adequate rotation of the local space, the new coordinate axis directions will easily be given by a Principal Component Analysis (PCA). In sum, after our procedure identifies the partitions of interest, it recursively searches for bumps using PRIM in each of their corresponding Principal Component (PC) sub-space, where input variables now display maximum variance along the new coordinate axes, now parallel or orthogonal to PRIM boxes. Due to the high dimensionality of the input variable space, combined with the fact that variables act in concert with one another and that not all of those surveyed might be involved, we make use of a modified version of PCA, called Sparse PCA (SPCA) (Jolliffe et al. 2003; Zou et al. 2006), which provide a much more optimal and biologically faithful transformed space within which to search for bumps. To impose sparsity in the eigenvector loadings and to compute Principal Components in situations where p ≫ n, we employ the penalized regression formulation of PCA, based on the Elastic Net regularization (Zou and Hastie 2005; Zou et al. 2006).

We devised the following algorithm, called Local Sparse Bump Hunting (LSBH) (see Supplemental Materials 4.6.5 for notations). In the outer loop of the following algorithm, dependency on subscript r is later dropped but understood.

We show in the next sections and in Supplemental Materials (4.6.5) that the use of SPCA addresses several shortcomings of PRIM in high dimension. First, the use of SPCA actually amounts to a twofold dimension reduction technique in that it not only imposes great parsimony in the number of selected principal components, but also sparsity in their loadings (Subsections 3.2, 4.2 and 4.8.3). Next, it appropriately deals with correlations between input variables (Subsection 3.2): since the search for modes/classes is run in a space whose components are nearly uncorrelated (Zou et al. 2006), this will substantially improve the efficiency of the PRIM step (Friedman and Fisher 1999). Third, it resolves some theoretical trade-off issues during PRIM meta-parameters estimation (Subsection 3.4). Fourth, it reduces the number of PRIM descriptive rules of the bumps, thereby resulting in great simplification and aided interpretation (Subsection 4.3). Finally, by virtue of the elastic net regularization (Zou and Hastie 2005), SPCA automatically selects whole groups of correlated variables once one variable among the group is selected (see Supplemental Materials 4.6.5 and Subsection 3.2). Altogether, we expect the method to substantially optimize the mode search, otherwise not possible or efficient if run directly in the input space and in a global way.

In general, the number of partitions of interest may be any number r ∈ {1, …, R}. For simplification, we focused later on a single partition of interest that we denoted P_r0 with cardinal n_r0 = |P_r0|. Let’s also denote by X^(r₀) = {x_i: x_i ∈ P_r0 ⊂ S} the (n_r0 × p) sub-matrix of X corresponding to the data-subset of partition P_r0 in the original input space, and similarly y^(r₀) = {y_i: x_i ∈ P_r0 ⊂ S} for the response.

3.2 Sparse Principal Component Analysis

At this stage, we suppose that we have selected a partition of interest P_r0 in which we have determined that there exists at least two modes or classes. The idea of imposing sparsity in the eigenvector loadings was first introduced by Joliffe et al. with his modified PCA based on the LASSO regularization (Jolliffe et al. 2003). It was recently extended to p ≫ n situation by Zou et al. (Zou et al. 2006) after the Elastic Net regularization became available (Zou and Hastie 2005).

Consider the (n_r0 × p) matrix ${\mathbf{X}}^{(r_0)}={\left[{\mathbf{x}}_j^{(r_0)}\right]}_{j=1}^p$, where each input variable ${\mathbf{x}}_j^{(r_0)}$ is assumed to be suitably normalized on n_r0 independent observations. Ordinary Principal Component Analysis (PCA) can be computed via the Singular Value Decomposition (SVD) of the covariance matrix Σ of X^(r₀). Denoting the SVD of X^(r₀) = UDV^T, and assuming w.l.o.g. that E[X^(r₀)] = 0, we can get the SVD of an estimate of the covariance matrix by $\widehat{\mathbf{\Sigma }}=\frac{1}{n_{r_0}}{\mathbf{X}}^{(r_0)T}{\mathbf{X}}^{(r_0)}=\mathbf{V}(\frac{1}{n_{r_0}}\mathbf{D}{\mathbf{D}}^T){\mathbf{V}}^T=\mathbf{V}\mathbf{\Delta }{\mathbf{V}}^T$. The sample ordinary Principal Components (PCs) are then easily obtained by Z^(r₀) = X^(r₀)V. Similarly to the idea of Principal Component Regression (PCR), one can derive a regression-based formulation of PCA as follows. If we denote ${\mathbf{z}}_j^{(r_0)}$ for j = 1, …, p the n_r0-vector corresponding to the j^th PC, then ${\mathbf{z}}_j^{(r_0)}={\mathbf{X}}^{(r_0)}{\mathbf{v}}_j$ for the j^th PC and its loadings v_j can be estimated by regressing ${\mathbf{z}}_j^{(r_0)}$ on the p variables of X^(r₀). This regression scheme is repeated for each PC for j = 1, …, p. To ensure an exact PCA solution is reproduced when p > n_r0, an L2 constraint, or Ridge regularization, is added (Zou et al. 2006). Furthermore, a sparse approximation to the exact PCA can be found by adding an L1 constraint, or LASSO regularization (Tibshirani 1996), generating altogether Elastic Net (EN) solutions (Zou and Hastie 2005):

$${\widehat{\mathbf{\gamma }}}_j^{EN}={\text{argmin}}_{{\mathbf{\gamma }}_j}\left[{\mid {\mathbf{z}}_j^{(r_0)}-{\mathbf{X}}^{(r_0)}{\mathbf{\gamma }}_j\mid }^2+{\lambda }_{1,j}\mid {\mathbf{\gamma }}_j\mid +{\lambda }_2{\mid {\mathbf{\gamma }}_j\mid }^2\right]$$ (1)

The estimated eigenvector v̂_j of each PC is given by the normalized version of the EN solution: ${\widehat{\mathbf{v}}}_j=\frac{{\widehat{\mathbf{\gamma }}}_j^{En}}{\mid {\widehat{\mathbf{\gamma }}}_j^{EN}\mid }$ to (1). The normalized version of ${\widehat{\gamma }}_j^{EN}$ approximates v_j since some of the elements (i.e. loadings) may have been shrunken to 0. The resulting PCs are thusly termed sparse PCs (SPCs). Zou et al. derived what they termed a “self-contained” version of this approach and an alternating minimization algorithm for estimation of the SPCs. For p > n_r0, a computationally efficient version of the estimation algorithm was provided by letting λ₂ → ∞ (Zou et al. 2006). Note the dependency of the shrinkage parameter λ_1,j on j.

Notably, the EN regularization overcomes important limitations of the LASSO encountered in the p > n_r0 case. In any case, the LASSO selects at most n_r0 variables before it saturates, because of the nature of the convex optimization problem (Tibshirani 1996). This seems to be a limiting feature for a variable selection method. The ridge penalty was conveniently used to get (ridge) solutions to parameter estimates in p ≫ n problems (Hastie and Tibshirani 2003; Zou and Hastie 2005). In addition, if there is a group of variables among which the pairwise correlations are very high, then the LASSO tends to select only one variable from the group and does not care which one is selected. The Elastic Net regularization overcomes these problems in the p > n_r0 case, in that (i) it can select more than n_r0 variables if necessary, (ii) and by automatically selecting into the model whole groups of correlated input variables once one variable among the group is selected (see 4.6.5) and (Zou and Hastie 2005).

Similarly as in PCA, the dimensionality q of the local SPC space is determined by the selected number q ≪ p of leading SPCs. Let’s denote by ${\widehat{\mathbf{Z}}}^{(r_0)}={\left[{\widehat{\mathbf{z}}}_j^{(r_0)}\right]}_{j=1}^q$ the sample SPCs spanning the (rotated) partition P_r0 of the SPC space (where the dependence on λ₁,_j is suppressed but understood). Note that the SPCs are no longer constrained to be uncorrelated like ordinary PCs. However, it is always possible to completely decorrelate the components by orthogonalizing the SPC space with the precaution that it should be carried out in an appropriate space of reduced dimensionality s ≤ p to not lose the sparsity. The exact manner in which to calibrate the dimensionality (q) and the degree of sparsity (s) imposed by (1) on the PCs will be discussed next.

Let ${\widehat{\mathbf{Z}}}^{(r_0)}={\left[{\widehat{\mathbf{z}}}_j^{(r_0)}\right]}_{j=1}^q$ be the sample SPC directions. While the ordinary principal components are uncorrelated, the regular SPCs are not. Thus, classical measures of fit for the number of SPCs (for example, scree plots based on the Cumulative Percent of Explained Variance (CPEV ) derived from tr(Ẑ^(r₀)TẐ^(r₀))) are no longer possible due to the slight correlation between components. As a fix to the above decorrelation procedure, Zou et al. (Zou et al. 2006) propose an alternate measure of fit that takes into account the correlation in Ẑ^(r₀). The idea is to use regression projection to remove the linear dependence between correlated components. Given q ≪ p SPC directions ${\widehat{\mathbf{z}}}_j^{(r_0)}$ for j = 1, …, q, let ${\widehat{\mathbf{z}}}_{j.1,\dots ,j-1}^{(r_0)}$ be the remainder of ${\widehat{\mathbf{z}}}_j^{(r_0)}$ after regressing out the effects of ${\widehat{\mathbf{z}}}_1^{(r_0)},\dots ,{\stackrel{\sim }{\mathbf{z}}}_{j-1}^{(r_0)}$ captured by: ${\widehat{\mathbf{z}}}_{j.1,\dots ,j-1}^{(r_0)}={\widehat{\mathbf{z}}}_j^{(r_0)}-{\mathbf{H}}_{1,\dots ,j-1}{\widehat{\mathbf{z}}}_j^{(r_0)}$, where H_{1, …,j−1} is the projection matrix of regressing ${\widehat{\mathbf{z}}}_j^{(r_0)}$ on ${\widehat{\mathbf{z}}}_1^{(r_0)},\dots ,{\stackrel{\sim }{\mathbf{z}}}_{j-1}^{(r_0)}$. Then, the adjusted variance of ${\widehat{\mathbf{z}}}_j^{(r_0)}$ is ${\mid {\widehat{\mathbf{z}}}_{j.1,\dots ,j-1}^{(r_0)}\mid }^2$, which is a function of the individual shrinkage parameter λ_1,j. Denote the j = 1, …, q shrinkage parameters by a shrinkage vector λ₁ = (λ_1,1, …, λ_1,q)^T. Then, the adjusted CPEV is a function of the shrinkage vector λ₁ and of the number of q ≪ p of SPC directions, given by:

$$\mathit{CPEV}({\mathbf{\lambda }}_1,q)=\underset{j=1}{\overset{q}{\Sigma }}{\mid {\widehat{\mathbf{z}}}_{j.1,\dots ,j-1}^{(r_0)}\mid }^2\times 100,$$ (2)

Note that When SPCs are uncorrelated, (2) amounts to the usual definitions derived from the ordinary PCs. Similarly, we define the Cumulative Percent of Non-Zero Loadings (CPNZL) as a function of the shrinkage vector λ₁ and of the selected number q ≪ p of SPC directions. Let J denotes the indicator p-vector of cumulative non-zero loadings of the (p×q) loading matrix ${\widehat{\mathbf{V}}}^{(r_0)}={(v_{lj})}_{l=1,j=1}^{l=p,j=q}$, each component J_l of which being the row-wise logical “OR” on the q-row vector of loadings: $J_l={\bigvee }_{j=1}^q\text{I}(v_{lj}\ne 0)$, for l = 1, …, p. Note that if we let $s(j)={\Sigma }_{l=1}^p\text{I}(v_{lj}\ne 0)$ be the number of non-zero loadings of the j^th eigenvector ${\widehat{\mathbf{v}}}_j^{(r_0)}$, each SPC can be written as follows (taking into account its individual sparsity): ${\widehat{\mathbf{z}}}_j^{(r_0)}=\mathbf{X}{\widehat{\mathbf{v}}}_j^{(r_0)}={\Sigma }_{k=1}^{s(j)}{v}_{kj}{\mathbf{x}}_k$, for j = 1, …, q. Now, let $s={\Sigma }_{l=1}^p{J}_l$ be the cumulative number of non-zero loadings for the first q eigenvectors, which is evidently a function of q and λ₁, then CPNZL is also a function of the shrinkage vector λ₁ and of the number of q ≪ p of SPC directions, given by:

$$\mathit{CPNZL}({\mathbf{\lambda }}_1,q)=\frac{1}{p}\underset{l=1}{\overset{p}{\Sigma }}\underset{j=1}{\overset{q}{\bigvee }}\text{I}(v_{1j}\ne 0)\times 100$$ (3)

The CPNZL reflects the proportion of total non-zero loadings, which is the complement measure of sparsity. It is a direct function of the amount of shrinkage λ₁ and of how many q leading SPCs are selected. Clearly, the dependencies on λ₁ and q are important for performing an appropriate SPCA. The quantities CPEV and CPNZL are both important in this respect and we later show in simulated and real datasets (Subsections 4.2 and 4.8.3) how the number q of selected leading SPCs and each one’s respective amount of shrinkage λ₁ is optimized using (2) and (3) simultaneously.

Since SPCs are no longer restrained to be uncorrelated, it might be desirable to decorrelate them prior to applying the bump hunting procedure. However, to preserve the sparsity in the orthogonalized SPCs, we want to carry out this orthogonalization only (after projecting the data) in an appropriate space of reduced dimensionality s ≪ p. If V̂^(r₀) is the (p × q) loading matrix obtained from the SVD of Σ̂, we do not lose any information by removing the all-zero rows of loadings from it (J_l = 0). The resulting (s × q) matrix is essentially the product of a (s × p) projection matrix P by V, denoted Ṽ = PV.

Decorrelated SPCs are obtained by orthogonalizing their corresponding eigenvectors in the loading matrix Ṽ. The QR factorization of Ṽ gives QR = Ṽ, where Q is the new matrix of orthogonal eigenvectors. After normalizing the column vectors of Q, we get our new (s × q) matrix of orthogonal eigenvectors, denoted $\mathbf{W}={[{\mathbf{w}}_j]}_{j=1}^q$. The matrix W is eventually used to generate the final (n_r0× q) matrix of local and uncorrelated SPCs: Z̃^(r₀) = X̃^(r₀)W, where X̃^(r₀) = X^(r₀)P^T is the corresponding (n_r0 × s) projected matrix of partition P_r0.

In fact, a complete SPCA procedure would amount to a projection-orthogonalization-rotation of the original data in the input space S. Denote this transformation by , the entire SPC space by (S), and the transformed partition P_r0 by (P_r0). The transformed data of partition P_r0 can be written as the (n_r0 × q) sub-matrix Z̃^(r₀) and denoted , where typically q ≪ p and s ≪ p.

3.3 Testing Multi-modality

In this section, we introduce a formal hypothesis testing framework for testing multi-modality by bootstrap hypothesis testing. The idea is borrowed from the problem of testing the number of modes in a density estimation problem, where a kernel density estimator is used with a smoothing parameter (Efron and Tibshirani 1993). Consider testing the number of modes of the target function f_Z̃^(r0)(z̃^(r₀)), or equivalently the number of boxes found by PRIM over the transformed partition (P_r0) of the SPC space. The parameters at play are the PRIM meta-parameters α and β. Consider a series of one-sided sequential tests for testing the number m₀ of modes (boxes) under the null hypothesis, where m₀ ∈ {1, …, M}.

$$\{\begin{array}{l}{H}_0:\#\text{modes}[f_{{\stackrel{\sim }{\mathbf{z}}}^{(r_0)}}({\stackrel{\sim }{\mathbf{z}}}^{(r_0)})](\alpha ,\beta )=m_0\hfill \\ H_1:\#\text{modes}[f_{{\stackrel{\sim }{\mathbf{z}}}^{(r_0)}}({\stackrel{\sim }{\mathbf{z}}}^{(r_0)})](\alpha ,\beta )>m_0\hfill \end{array}$$ (4)

To carry out the hypothesis test we must specify (i) a null distribution for the number of modes under H₀, (ii) and a test statistic t(z̃^(r₀)). As the meta-parameter values increase, the number of modes (PRIM boxes) do not necessarily decrease monotonically, but vary jointly and in a complex manner. Denote Ω the bi-dimensional meta-parameter space. For each number of modes corresponds a contour in meta-parameter space Ω = {(α, β): α ∈ , β ∈ }, delineating a region of meta-parameter pairs (α, β), where domains ⊆ (0, 1) and ⊆ (0, 1) are chosen to cover a range as wide as possible, in accordance with reasonable boundaries instructed by PRIM designers (Friedman and Fisher 1999). Since the variance of the estimate of the box is proportional to 1/α (Friedman and Fisher 1999), the value of α cannot be made too small. Likewise, β is lower bounded by the minimal number of observation in a box. Denote by Ω₀ the meta-parameter sub-space of Ω of all (α, β) pairs giving exactly m₀ modes, i.e. formally:

$${\mathrm{\Omega }}_0=\{(\alpha ,\beta )\in \mathrm{\Omega }:\#\text{modes}[f_{{\stackrel{\sim }{\mathbf{z}}}^{(r_0)}}({\stackrel{\sim }{\mathbf{z}}}^{(r_0)})](\alpha ,\beta )=m_0\}$$ (5)

and define ${\pi }_0=\frac{\mid {\mathrm{\Omega }}_0\mid }{\mid \mathrm{\Omega }\mid }$ as the proportion of meta-parameter value pairs mapping the number of modes of the target function f_Z̃^(r0)(z̃^(r₀)) in (P_r0) under H₀. Recall that the mode supports are described by the set of m₀ rules of the box sequence ${\{B_m\}}_{m=1}^{m_0}$. It seems reasonable to use as the null distribution of m₀ all modes estimated from f_Z̃^(r0)(z̃^(r₀)) based on Ω₀ and the proportion π₀ (Efron and Tibshirani 1993). As for the test statistic, chose the estimated proportion t(z̃^(r₀)) = π₀. A small value of π₀ indicates that few parameters values are possible to create an estimate of f_Z̃^(r0)(z̃^(r₀)) with m₀ modes and is therefore evidence against H₀.

Our statistical test for H₀ is based on estimating an achieved significance level or p-value p(m₀) using the bootstrap. Let ${\stackrel{\sim }{\mathbf{z}}}^{(r_0)\ast }={\{{\stackrel{\sim }{\mathbf{z}}}_i^{(r_0)\ast }\}}_{i=1}^{n_{r_0}}$ be a bootstrap sample of size n_r0, drawn from the null distribution, and ${\pi }_0^{\ast }$ be the corresponding estimated proportion of parameter values under the null, p(m₀) is given by:

$$p(m_0)={Pr}_{H_0}^{\ast }[t({\stackrel{\sim }{\mathbf{z}}}^{(r_0)\ast })>t({\stackrel{\sim }{\mathbf{z}}}^{(r_0)})]={Pr}_{H_0}^{\ast }[{\pi }_0^{\ast }>{\pi }_0]$$ (6)

To estimate the p-value, generate B bootstrap datasets z̃^(r₀)*1, …, z̃^(r₀)*B and compute the B replicates ${\{t({\stackrel{\sim }{\mathbf{z}}}^{(r_0)\ast b})\}}_{b=1}^B$, that is ${\{{\pi }_{m_0}^{\ast b}\}}_{b=1}^B$. A bootstrap estimate of p(m₀) is then easily derived as:

$$\widehat{p}(m_0)=\frac{1}{B}\underset{b=1}{\overset{B}{\Sigma }}\text{I}[t({\stackrel{\sim }{\mathbf{z}}}^{(r_0)\ast b})>t({\stackrel{\sim }{\mathbf{z}}}^{(r_0)})]=\frac{1}{B}\underset{b=1}{\overset{B}{\Sigma }}\text{I}[{\pi }_{m_0}^{\ast b}>{\pi }_0]$$ (7)

The test procedure is sequentially repeated for each candidate number of mode m₀, where m₀ ∈ {1, …, M}. Finally, we propose to choose between two simple stopping rules to estimate the number of modes in the data: (i) either by taking the value of m̂₀ maximizing p̂(m₀); (ii) or by taking the minimum value m̂₀ for which we cannot reject the null hypothesis at a certain significance level, say η, formally:

$$\begin{array}{l}\mathit{rule}\#1:{\widehat{m}}_0={\text{argmax}}_{m_0}\{\widehat{p}(m_0)\}\\ \mathit{rule}\#2:{\widehat{m}}_0={min}_{m_0}\{\widehat{p}(m_0)>\eta \}\end{array}$$ (8)

On one hand, if one is on a discovery path, one would prefer a more liberal rule (rule #1). On the other hand, one could prefer a more conservative rule (rule #2). We favored the second rule (#2) as Efron et al. (Efron and Tibshirani 1993) to lower the risk of over-fitting, and will use it from now on.

3.4 Tuning PRIM Meta-parameters

In this section we propose a method to estimate the PRIM meta-parameter α and β to successfully optimize the accuracy of PRIM rules. Our estimation procedure benefits from an estimation of the number of modes/classes within the data and a theoretical justification. Any selection of meta-parameter (α, β), even by conditioning on the correct number of modes, does not guarantee accurate and optimal PRIM rules. In fact, Friedman et al. noted that the variance of the estimate of the box is proportional to 1/α (Friedman and Fisher 1999), so that the value of α cannot be made too small. More recently, Polonik et al. indicated that under regularity conditions pα has to be kept small (pα_n = o(1) as n → ∞) for a given β in order to obtain consistency and good rates of convergence (Polonik and Wang 2007). Hence, the tuning of the meta-parameter α should also depend on the dimension p.

To ensure that both statistical properties and accuracy of peeling will hold simultaneously, one must (jointly) estimate the meta-parameters α and β in a space of reduced dimensionality, thusly allowing α to take on larger values. In the following high dimensional dataset example (Subsection 4.8), we show how one can greatly reduce the dimensionality (from p = 2068 to q = 8) without loosing too much explained variance. By reducing the dimensionality of the space to q ≪ p, we will allow α to take on larger values, thusly favoring the accuracy of the bumps without sacrificing the consistency. Overall, our tuning/estimation strategy trades-off between the goals of accuracy of peeling, consistency, and goodness of fit after dimension reduction.

We propose to get estimates based on bootstrap confidence intervals of the meta-parameter pairs (α, β) that would minimize an objective function, conditioning on the estimated number of modes m̂₀ in the partition (P_r0). To resolve the aforementioned trade-off, it seems reasonable to use as an objective function some variability measure of the outcome boxes of PRIM.

Given the sequential nature of the PRIM-induced box sequence ${\{B_m\}}_{m=1}^{{\widehat{m}}_0}$ (see Supplemental Materials 4.6.5), it is natural to focus on the first box B₁. Recall that each individual box B_m found in ((P_r0)) is at this stage a q-dimensional random variable. To proceed, we derive a simple summary statistic of the variance of B_m, based on the input value-subsets $s_{j,m}=[t_{j,m}^{-},t_{j,m}^{+}]$ for fixed m, where B_m is of the form $B_m={\otimes }_{j=1}^q{s}_{j,m}$ in ℝ^q. We define a “Box Coefficient of Variation” (BCV) of a particular box B_m (for fixed m) as the sum of the coefficients of variation of interval boundaries in each SPC direction:

$$\mathit{BCV}(B_m)=\underset{j=1}{\overset{q}{\Sigma }}[\text{cv}(t_{j,m}^{-})+\text{cv}(t_{j,m}^{+})]$$ (9)

where cv(.) denotes the coefficient of variation. Since each individual box B_m is a function of (α, β), BCV can be denoted BCV (α, β). The estimated value pair (α̂, β̂) is found by minimizing BCV (α, β), given that m̂₀ modes are to be found exactly in the partition (P_r0) of the SPC space. The minimizers are found by scanning over the lattice of value-pairs (α, β) ∈ Ω₀. Formally:

$$(\widehat{\alpha },\widehat{\beta })={\text{argmin}}_{(\alpha ,\beta )\in {\mathrm{\Omega }}_0}[\mathit{BCV}(\alpha ,\beta )]$$ (10)

In practice, we get bootstrap estimates of BCV_m(α, β) for each value pair (α, β), and we build percentile confidence intervals (Efron and Tibshirani 1993) of (α̂, β̂) based on percentiles of their bootstrap distributions. To proceed, generate B′ bootstrap datasets z̃^(r₀)*1, …, z̃^(r₀)*B of size n_r0 and derive B bootstrap replications ${\{({\widehat{\alpha }}^{\ast b},{\widehat{\beta }}^{\ast b})\}}_{b=1}^B$ of (α̂, β̂) as follows:

$$({\widehat{\alpha }}^{\ast b},{\widehat{\beta }}^{\ast b})={\text{argmin}}_{(\alpha ,\beta )\in {\mathrm{\Omega }}_0}\left[{\widehat{\mathit{BCV}}}^{\ast b}(\alpha ,\beta )\right]$$ (11)

where ${\widehat{\mathit{BCV}}}^{\ast b}(\alpha ,\beta )$ denotes the estimated “Box Coefficient of Variation” obtained from the b^th bootstrap sample z̃^(r₀)*b, (b = 1, …, B) and for each value pair (α, β). Note that the estimate of BCV^*b(α, β) is obtained by carrying out a nested bootstrap with B″ < B′ bootstrap datasets generated from each parent bootstrap sample z̃^(r₀)*b. The bootstrap distributions of α̂^* and β̂^* consist in the B′ bootstrap replications ${\{{\widehat{\alpha }}^{\ast b}\}}_{b=1}^{B^{\prime }}$ and ${\{{\widehat{\beta }}^{\ast b}\}}_{b=1}^{B^{\prime }}$ respectively from which the percentile intervals are derived.

The 100 · (1 − 2θ)% percentile intervals of α and β are defined by the θ^th and (1 − θ)^th quantiles of the cumulative distribution function of α̂ ^* and β̂^* respectively. Since by definition the inverse of the cumulative distribution function of θ is the θ^th quantile of the bootstrap distribution, we can write the estimated percentile intervals of α and β as:

$$\{\begin{array}{l}[{\widehat{\alpha }}_{lo},{\widehat{\alpha }}_{up}]=\left[{\widehat{\alpha }}^{\ast (\theta )},{\widehat{\alpha }}^{\ast (1-\theta )}\right]\hfill \\ \left[{\widehat{\beta }}_{lo},{\widehat{\beta }}_{up}\right]=\left[{\widehat{\beta }}^{\ast (\theta )},{\widehat{\beta }}^{\ast (1-\theta )}\right]\hfill \end{array}$$ (12)

where x̂^*(θ) and x̂^*(1−θ) represent respectively the θ^th and (1 − θ)^th quantiles of of the bootstrap distribution of x̂^*. In practice, since we can only have a finite number B′ of replications, the θ^th empirical quantile x̂^*(θ) of x̂^* is taken to be the B′ · θ^th value in the ordered list of the B′ replications of x̂^*. Likewise, the (1 − θ)^th empirical quantile x̂^*(1−θ) of x̂^* is taken to be the B′ · (1 − θ)^th value in the ordered list of the B′ replications of x̂^* (Efron and Tibshirani 1993). Bootstrap estimates (α̂, β̂) are then easily derived by taking for instance the median values of the 100 · (1 − 2θ)% percentile confidence intervals of the pair (α, β) at a certain confidence level, say θ = 0.05.

4 VERIFICATION

4.1 Encapsulation and Sparsity Efficiency - Simulated Dataset

We tested our LSBH algorithm (1) in the initial synthetic dataset of our motivating example (section 2) with three overlapping distributions (D = 3) in a p = 100 dimensional input space. As already mentioned, CART partitioning produced four partitions in this dataset, labeled ${\{P_r\}}_{r=1}^4$ (Figure 1 and Supplemental Figure 8). Further results in Supplemental Materials (Supplemental Figure 9) illustrate the efficiency of the “encapsulation” step of the LSBH procedure and how this translates into greater interpretability (Supplemental Figure 10). As mentioned in the methodology section, the choice of CART for recursive partitioning seems natural. It is clear that the greediness of CART is an advantage over the patience of PRIM to partition the data. This choice is further discussed in our conclusion.

Algorithm 1.

Local Sparse Bump Hunting.

Partition the input space into ${\{P_r\}}_{r=1}^R$ partitions, using e.g. CART. While r ∈ {1, …, R}: If Gr > 1 (i.e. test eligibility for bump hunting: is #class labels in Pr > 1?) Standardize the input variables (optional). Run a local SPCA: Estimate the local Sparse Principal Components (SPCs), select a first few of them j = 1, …, q, where q ≪ p, and chose an optimal amount of shrinkage/sparsity for each of them, resulting in an individual number of non-zero loadings s(j). (Optionally, carry out a complete decorrelation of the SPCs) Rotate the local space according to corresponding SPCs main directions. Denote the transformed partition in the SPC space by (Pr). Test local multimodality m̂0 within transformed partition (Pr) If m̂0 > 1 Conditioning on m̂0, estimate PRIM meta-parameters α and β, in (Pr). Run a local and tuned PRIM-based bump hunting within (Pr) and get descriptive rules of the bumps in the SPC space of the form ${\mathfrak{R}}_r={\cup }_{m=1}^{{\widehat{m}}_0}{\cap }_{j=1}^q\{{\mathbf{z}}_j\in [t_{j,m}^{-},t_{j,m}^{+}]\}$. Rotate the local rules ℜr back into the input space to get rules in the form of “sparse linear combinations”: r ← r + 1 Collect the local rules from all partitions to get a global rule of the form: $\mathfrak{R}={\cup }_{r=1}^R{\mathfrak{R}}_r^{\prime }$ giving a full description of the estimated bumps in the entire input space.

We assume at this point that we have tested the eligibility of a partition P_r for bump hunting (G_r > 1 - see algorithm 1). Here, results are shown in a specific partition of interest (r₀ = 2). We show how SPCA induces sparsity and selects only relevant loadings in high dimension. Since most of the variance was setup w.l.o.g. in the input subspace (X₁, X₂, X₃) (see section 2), we expect the shrinkage to occur in any subspace of the complement space (X₄, X₅, …, X₁₀₀) depending on the imposed amount of shrinkage. To determine an optimal degree of sparsity, an appropriate step-by-step selection of the amount of shrinkage was done as described in the method section (3) and in details in Supplemental Figure (12). Here, as expected from the model, all input variables (X₄, X₅, …, X₁₀₀) were found to be shrunk to 0 (Figure 2).

Figure 2.

Top Left: Superimposed CPEV and CPNZL scree plots as a function of shrinkage for each leading SPC (first three shown) for the data in partition P_r0 (synthetic dataset with r₀ = 2). Top Right: Summary of CPEV and CPNZL scree plots as a function of the number of selected leading SPCs (3). The goodness of fit (CPEV=58.5%) and complement of sparsity (CPNZL=60.0%) are shown for the chosen amount of shrinkage and number of selected SPCs. Bottom: loading plots of the first three selected SPCs. Here, the resulting sparsity (100% − 60.0% = 40.0%) is moderate, but in fact all input variables except (X₁, X₂, X₃) are shrunk to nearly 0.

4.2 Superiority of the Bootstrap Test Statistic - Simulated Dataset

Next, we tested the mode finding ability of our method in the transformed partition of interest (P_r0), and compared with others found in the initial synthetic dataset of our motivating example (section 2). Table 1 reports the estimated number of clusters, partitions, and modes as determined by the gap, the deviance, and the bootstrap test statistics respectively, conducted in each partition, using our objective stopping rules as described in sections 3, and Supplementary Materials 4.6.5.

Table 1.

Estimated number of clusters (k̂), partitions (r̂) and modes (p-values p̂ at η = 0.01, B = 1000) for the number of modes m₀, tested under H₀ in each transformed partition of the SPC space in synthetic dataset.

m0	(P1)			(P2)			(P3)			(P4)
	k̂	r̂	p̂	k̂	r̂	p̂	k̂	r̂	p̂	k̂	r̂	p̂
1	yes	no	0.031	no	no	<0.001	yes	no	0.026	yes	no	0.247
2	no	yes	0.906	no	no	0.494	no	no	0.835	no	yes	0.966
3	no	no	0.156	yes	no	0.587	no	yes	0.153	no	no	0.829
4	no	no	0250	no	yes	0.634	no	no	NA	no	no	0.739
5	no	no	<0.001	no	no	0.387	no	no	NA	no	no	0.392
6	no	no	0.062	no	no	0.340	no	no	NA	no	no	0.010
7	no	no	<0.001	no	no	0.128	no	no	NA	no	no	0.030
8	no	no	NA	no	no	0.034	no	no	NA	no	no	0.032

As shown in Table 1, (P_r0) is the only partition where the unimodal hypothesis is rejected at the η = 0.01 significance level, but not the bimodal one. So, the only partition of interest is in this case (P_r2), where the number of modes is correctly estimated at m̂₀ = 2.

We assume at this point that we have selected the transformed partition of interest (P_r0) (here r₀ = 2), where we have carried out a complete SPCA, and within which we now characterize the modality. Restricting ourselves to Z̃^(r₀), i.e. to the transformed data-subset of (P_r0) in the SPC space, we show in Figure 3 the results of the multi-modality analysis. In general, domains and can be entertained as follows: = {0.01, …, 0.50} and = {0.01, …, 0.90}. Figure 3 shows the contours of the number of modes as a function of PRIM meta-parameters values (α, β), and the estimated number of modes in a side-by-side comparison of the gap, deviance, and bootstrap test statistics profiles.

Figure 3.

Testing Multimodality in synthetic dataset. Left: Contours of the number of modes in the meta-parameter space Ω for the target function in partition (P_r0) of the SPC space (synthetic dataset with r₀ = 2). Note the importance of the patience (α₀) in controlling the number of modes for a fixed minimal box support (β₀). Right: Multimodality tests profiles in partition (P_r0) of the SPC space: bootstrap test statistic (step function of p-values for each mode tested under H₀), gap statistic, deviance statistic (LOO CV). Red arrows indicate stopping rules results.

To account for sampling variability, we computed the empirical estimate of the probability of finding (m ∈ {1, …, 6}) modes in (P₂) by repeating the mode finding procedure on 100 replicated synthetic datasets. Table 2 reports the resulting probability estimates. In contrast to other employed statistics, where none of them could give the correct estimated number of distributions (including in the partition P_r0 - (Figure 1 and Supplemental Figure 8), the bootstrap test statistic could find with near perfect consistency the correct number of modes m̂₀ = 2 in (P₂). In addition, table 2 suggests that a partition clustering algorithm would tend to underfit the data, whereas a binary decision tree would tend to overfit it, even after pruning.

Table 2.

Probability estimates of finding m modes in partition (P_r0) of the SPC space.

m	Gap Stat.	Dev. Stat.	Boot. Test Stat.
1	54	0	3
2	2	3	97
3	44	13	0
4	0	52	0
5	0	24	0
6	0	8	0

4.3 Evaluation of the Classification Accuracy - Simulated Dataset

At this stage, we suppose that we have carried out a complete LSBH procedure in the selected partition of the simulated data. In the following sections, we checked the adequacy of our method for estimating meta-parameters α and β, using standard accuracy evaluation procedures. What we need to do first is turn the bump hunting output of our LSBH procedure into a discrete classifier output, where class labels are inferred for each instance (see indirect application of PRIM in Supplemental Materials 4.6.5). By virtue of the rule induction nature of PRIM, it is straightforward to use the box definition rule as the classification rule. Likewise, the box majority class rule can serve as the class decision rule. This framework allows employing the traditional measures of classification performance. In binary class domains, a common metric for assessing the performance of any classifier is prediction accuracy, namely through the true- and false-positive rates (TPR and FPR), also known as sensitivity and 1-specificity. Recall that the end result of the bump hunting procedure are boxes, where the output variable is assumed to be maximal (e.g. (y_g = 1) with a y_g ∈ {0, 1} binary coding). By definition, the true- and false-positive rates for fixed class g are defined as follows with our notations (with r₀ = 2 in our case):

$$\begin{array}{l}{\mathit{TPR}}_g(\alpha ,\beta )=Pr[\widehat{\mathbf{y}}(\alpha ,\beta )=1\mid {\mathbf{y}}_g=1]\\ {\mathit{FPR}}_g(\alpha ,\beta )=Pr[\widehat{\mathbf{y}}(\alpha ,\beta )=1\mid {\mathbf{y}}_g=0]\end{array}$$ (13)

A common practice for assessing classifiers and visualizing their performance is by minimizing combined error rates of true and false positives, which can be done by Receiver Operating Characteristics (ROC) analysis. The ROC technique has been widely used in diagnostic testing and disease classification and more recently in micro-array studies. ROC curves plot (TPR_g(α, β) versus FPR_g(α, β)), where each point on the curve corresponds to a different classification rule, based on a threshold value of some observation score. In the ROC space, classification rules that have (FPR_g, TPR_g) close to (0, 1) indicate good discriminatory performance as opposed to those with (FPR_g, TPR_g) near the identity line that corresponds to all possible performances of a random classifier. The performance of classification is naturally assessed by measuring the accuracy of prediction, whereas the performance of ranking is commonly measured by taking the Area Under the (ROC) Curve (AUC) (Klement and Flach 2008).

The estimation procedure with B′ = 200 and B″ = 100 yielded the value-pair estimates: (α̂ = 0.44; β̂ = 0.40). Figure 4 maps the distribution of meta-parameter values in the meta-parameter space Ω and in the ROC space by number of modes. This plot confirms the multi-modality test result in that the best classifiers, including the optimal one, are found under two modes (Figure 4). Note that the unimodal meta-parameter values (rejected by the multi-modality test) yields among the worst classification results. Also, although some classifiers seemingly perform as well or even better under three and more modes, they should not be considered because we showed that higher than two modes-procedures result in an over-fitting (see Methods section 3). Moreover, Figure 4 unveils the optimal meta-parameter value-pair (α_opt = 0.46; β_opt = 0.30) corresponding to the best performance achievable by the LSBH procedure. We confirm that the method of meta-parameter estimation (Subsection 3.4) selects a value-pair (α̂ = 0.44; β̂ = 0.40), which (i) falls under the correct number of mode (2), and (ii) approximates very well the optimal value-pair. These results hold for both target sub-groups #1 and #2 (Figure 4). Finally, notice how the classification performance can degrade almost as bad as a random classifier (even under two modes in sub-group #1) in the absence of meta-parameters estimation (fine tuning).

Figure 4.

Mapping of meta-parameter values in meta-parameter space and in ROC space (simulated dataset). Left: mapping of estimated vs. optimal meta-parameter values in meta-parameter space Ω. Middle and right: scatter-plots of discrete classifier performances in the ROC space based on all possible meta-parameter value-pairs by number of modes for each target sub-group #1 or #2. Green and blue dots respectively represent the actual estimated value-pair (α̂ = 0.44; β̂ = 0.40) and the optimal value-pair (α_opt = 0.46; β_opt = 0.30) of meta-parameters (i.e. the farthest point from the identity line in the ROC space, or best (FPR, TPR) trade-off) under two modes for both sub-groups. The blue dotted lines are the identity line of all possible performances of a random classifier, and the shortest segment to it, pointing to the best classifier.

To assess the importance of sampling variability in our method, and the performance relatively to competitors, we repeated the whole procedure of finding the hidden target sub-groups #1 and #2 in 100 replicated datasets from the original simulated data in comparison to previous competitive methods. The following table (Table 3) reports the classification performance results in terms of Area Under the Curve (AUC) and Sensitivity/Specificity analysis in the simulated dataset in comparison to unsupervised competitive methods: K-means clustering, one-class SVM-based density estimation; as well as supervised ones: CART classification, SVM classification, or a direct use of PRIM bump hunting. Each competitor was used with its own default parameter values. By “default” in the case of our method, we mean without fine tuning as described in 3, i.e. by simply taking e.g. median values of meta-parameter ranges for (α, β) ∈ Ω₀ i.e. (α̂ = 0.21; β̂ = 0.51). The performance analysis showed that both ROC(t₀) statistic and pAUC(t₀) statistic ranked LSBH models better for any values of t₀ ∈ (0, 1) (Table 3).

Table 3.

Comparative empirical Area Under the Curve ( $\widehat{\mathit{AUC}}$), Sensitivity ( $\widehat{1-\mathit{FPR}}$) and Specificity ( $\widehat{\mathit{TPR}}$) reported in % for each target sub-group #1 or #2 (#1 and #2: simulated dataset). Comparison between K-means partitioning clustering algorithm (CLUS), one-class SVM-based density estimation (DENS), tree-based partitioning (CART), SVM-based classification (SVM), direct Patient Rule Induction Method (PRIM) (all with default parameter values), and our Local Sparse Bump Hunting Method (LSBH) (with default or tuned meta-parameter values). Standard errors in parenthesis.

Measure		CLUS	CART (default)	SVM (default)	DENS (default)	PRIM (default)	LSBH (default)	LSBH (tuned)
$\widehat{1-\mathit{FPR}}$	#1	00.00 (0.00)	82.33 (3.35)	00.00 (0.00)	52.09 (0.53)	86.67 (0.61)	92.75 (0.46)	90.99 (0.36)
$\widehat{\mathit{TPR}}$		100.00 (0.00)	24.45 (3.12)	100.00 (0.00)	55.85 (0.47)	65.69 (1.48)	50.20 (1.24)	77.36 (0.81)
$\widehat{\mathit{AUC}}$		50.00 (0.00)	53.39 (0.24)	50.00 (0.00)	53.97 (0.12)	76.18 (0.55)	71.47 (0.76)	84.17 (0.38)
$\widehat{1-\mathit{FPR}}$	#2	46.02 (0.06)	54.26 (0.39)	49.45 (0.02)	05.67 (1.62)	85.26 (0.72)	82.31 (0.67)	95.93 (0.25)
$\widehat{\mathit{TPR}}$		100.00 (0.00)	97.57 (0.09)	100.00 (0.00)	94.83 (1.48)	77.79 (1.18)	88.97 (0.83)	85.53 (0.55)
$\widehat{\mathit{AUC}}$		73.01 (0.03)	75.91 (0.19)	74.72 (0.01)	50.25 (0.07)	81.52 (0.42)	85.64 (0.67)	90.73 (0.29)

Figure (5) is the corresponding ROC scatterplot. In sum, AUCs, $\widehat{1-\mathit{FPR}}\text{s},\widehat{\mathit{TPR}}\text{s}$ and ROC scatterplots confirm the improved classification accuracy performance of LSBH (for both target sub-groups #1 or #2) in comparison to previous competitive methods. Also, we observed that increasing amounts of overlap between the two sub-groups #1 or #2 (maximum showed here in Figure 5 and Table 3) resulted in total collapse of performance of K-means clustering and decreased performance of PRIM, while our procedure (tuned or not) remains robust to it. Finally, notice the definite advantage of fine tuning of the meta-parameters in our LSBH procedure in subgroup #1, which amounts to a specificity/sensistivity trade-off in subgroup #2.

Figure 5.

Comparative ROC scatterplots in replicated datasets for each target sub-group #1 or #2 between K-means partitioning clustering algorithm (CLUS), one-class SVM-based density estimation (DENS), tree-based partitioning (CART), Support Vector Machine (SVM), or direct Patient Rule Induction Method (PRIM) (all with default parameter values), and our Local Sparse Bump Hunting Method (LSBH) (with default or tuned meta-parameter values). Open circles are centroids averaged over the 100 replicated datasets for each method. Because performances by methods overlay in some coordinates (e.g. in (100,100)), some results mask each other on the scatter plot).

4.4 Evaluation of the Prediction Accuracy - Simulated Dataset

Finally, to further check the adequacy of our method, we compared the overall classification performances of our method (LSBH) to previous competitors in the same simulated data. In the case of SVM-or CART-based methods, regular rules of class prediction were used. In the case of LSBH and PRIM, predicted classes were generated by turning the output into discrete class labels (see above section), where the box definition rules served as the classification rule, and the majority class rule served as the class decision rule. In the case of K-means clustering, predicted classes of instances were generated by nearest cluster centroid. Table 4 reports the results of the Average Percentage Error Rate (APER), as well as the Cross-Validated Prediction Error Rate (PER) of the entire procedure. In the latter case, a five-fold cross-validation was carried out on 100 repeated random splits between a training and test sets from the original simulated dataset.

Table 4.

Comparative empirical Average Percentage of Error Rates ( $\widehat{\mathit{APER}}$) and cross-validated Prediction Error Rates ( $\widehat{\mathit{PER}}$) (all in %) in the synthetic dataset between all methods.

Measure	CLUS	CART (default)	SVM (default)	DENS (default)	PRIM (default)	LSBH (default)	LSBH (tuned)
$\widehat{\mathit{APER}}$	34.96	33.29	33.33	61.92	18.83	12.54	8.54
$\widehat{\mathit{PER}}$ (training)	34.75	36.32	32.99	65.49	17.39	13.97	8.08
$\widehat{\mathit{PER}}$ (test)	33.73	35.85	31.96	61.06	21.76	16.28	9.94

Overall, LSBH outperforms competitors in terms of prediction accuracy (smaller $\widehat{\mathit{PER}}\text{s}$) and ranking/classification accuracy (larger $\widehat{\mathit{AUC}}\text{s}$, sensitivities $\widehat{1-\mathit{FPR}}\text{s}$, and specificities $\widehat{\mathit{TPR}}\text{s}$). Also, notice the definite advantage of fine tuning of the meta-parameters in the LSBH procedure and over a direct PRIM approach in the entire input space.

4.5 Visual Performance - Simulated Dataset

Figure 6 illustrates graphically the overall Local Sparse Bump Hunting (LSBH) procedure. The resulting efficiency of mode discovery by our method is shown when LSBH is run (i) locally in the transformed partition (P_r0) of the SPC space, (ii) after testing the multimodality in it, and finally (iii) with the appropriate tuning of meta-parameters.

Figure 6.

Graphical illustration of Local Sparse Bump Hunting (LSBH) on simulated dataset. All plots are projections from (X₁, …, X_p) into input subspace (X₁, X₂) or SPC subspace (Z₁, Z₂) (with superimposed perspective effect). Top Left: target groups from 3 overlapping target distributions (D = 3) with 2 class labels (G = 2) in the original input space (p = 100) with PC’s and 95% CE. Top Right: identification of the CART partition of interest (P_r0) with its SPCs. Bottom Left: two bumps found (dark green vs light green), shown in the SPC subspace with delineation by one side of the bump boundary (vertical solid line). Bottom Right: final 2 bumps shown back in the original input space.

4.6 Application in High Dimensional Data - Micro-array Dataset

4.6.1 The Dataset

LSBH was applied to a large micro-array gene expression dataset, initially generated from a group of sporadic MSS colon cancer tumor samples. The samples were staged according to Astler-Coller-Duke’s staging system (Cohen et al. 1997). They ranged from stage “B” primary tumors, to stage “C” representing a progressive worsening of the disease with spread of secondary tumors, to stage “D” representing metastases spread to distant organs, and finally “METS”, representing the most advanced stage of metastases. The 4 staged sample sizes were labeled as follows: “B”: 25, “C”: 21, “D”: 35 and liver “METS”: 23, summing to a total of n = 104 samples. The differentially expressing variables (genes) across the tumor types were selected by a Bayesian ANOVA model (Ishwaran and Rao 2003, 2005). We thusly selected p = 1500 such genes that were further used in our analysis.

4.6.2 Partitioning the Input Space

After partitioning the input (gene) space, we found a total of four (cross-validated) partitions, i.e. R = 3 terminal nodes (Supplemental Figure 11). According to our LSBH algorithm, the procedure asks to go recursively over each partition. Biologists were interested in finding whether there is any sample heterogeneity specifically among the “METS” samples, based on the assumption that these are the samples for which sample heterogeneity is most likely. Therefore, we focused our attention on those “METS”-majority class partitions only (i.e. terminal nodes labeled “M”). We show the results with this restriction, but it is clear that it can be alleviated. In this data, it turns out that the “METS” partition of interest in the original space was unique, further denoted: P₃ (r₀ = 3) (Supplemental Figure 11).

4.6.3 Determining an Optimal Degree of Sparsity

A complete SPCA analysis was carried out and SPCs were estimated locally within P₃. We now detail how to select the number q of leading SPCs, as well as how to determine the optimal individual amount of shrinkage λ_1;j for each of them using (2) and (3) when a very large number of variables are present (p = 1500). Supplemental Materials (4.6.5) and Supplemental Figures 12 & 13 describe the process in detail for the data in partition P₃. The general idea is to do for each SPC a stepwise and conditional selection of the amount of shrinkage by optimizing the variance-sparsity trade-off. The total number q of selected leading SPCs is dictated by the amount of Cumulative Percentage of Explained Variance (CPEV) (2) and the amount of Cumulative Percentage of Non-Zero Loadings (CPNZL) (3) desired. We decided to select q = 11 leading SPCs because beyond this point, further SPCs give diminishing returns in terms of explained fit/sparsity tradeoff (Supplemental Figures 12 & 13). Clearly, more sparsity can be obtained by allowing larger values of λ_1;j for each SPC in the selection process, resulting in a heavier shrinkage, but at the price of losing some fit (explained variance). In this example a medium sparsity has been imposed using a medium amount of shrinkage, resulting in a total of s = 632 non-zero loadings (genes) with q = 11 leading SPCs.

4.6.4 Testing Multimodality in Partition (P₃) of the SPC Space

Using our stopping rule (rule #2), the estimated number of modes in the transformed data subset Z̃⁽³⁾ of partition (P₃) of the SPC space was found to be m̂₀ = 1, i.e., we would apparently not assume any sub-group in the “METS” observations and the analysis would stop here (Supplemental Figure 14). However, given how conservative rule # 2 is, and given the estimation error attached to these quantities due to sampling variability, one may want to entertain both stopping rules. So, for exploratory purposes, we show what the results are, should we use rule # 1, i.e. with m̂₀ = 2.

4.6.5 Sparse Bump Hunting in Partition (P₃) of the SPC Space

The goal is to search for sub-class(es) in the transformed data subset Z̃⁽³⁾ of partition (P₃) of the SPC space. The partition encompasses n₃ = 22 observations, among which the proportion of “METS” samples (majority samples or prototypes) was enriched as compared to the full complement of samples (“D”). Here, we considered one dummy output variable y_M, such that {y_M = I(class =\M″)}, where y_M = 1 now represents a majority class of interest, say with respect to all the other classes combined, where y_M = 0. Assuming two modes, LSBH modeled two bona fide bumps (domains of two sub-classes of “METS”), splitting the observations in data Z̃⁽³⁾ into two (Figure 7). The corresponding rule involved a single component: : z̃₄ < 2562.

Figure 7.

Graphical illustration of Local Sparse Bump Hunting (LSBH) on micro-array dataset. Scatter plots of observations in data Z̃⁽³⁾ of the partition (P₃) projected into SPC subspace (Z₃,Z₄). Two bumps shown with bump boundary and membership: red and green labels denote the first and second bumps respectively.

There is indication that this bump is not found by clustering, and at best inaccurately by a CART binary decision tree (data not shown). Generally speaking, any rule induction method might be equivalently used in the SPC space. However, as stressed out earlier, CART algorithm would tend to overfit the data, even after pruning, probably due to its inherent greediness (Subsection 4.4), making CART rule less accurate than PRIM. By controlling the patience in LSBH through the α and β meta-parameters (see Supplemental Materials 4.6.5), we generally reduce the risk of overfitting and optimize the bump definition accuracy. This is a well-known advantage of PRIM over CART in situations where patience is needed (Friedman and Fisher 1999).

5 CONCLUSION

Throughout the paper, we assumed without loss of generality that the regression function is non-negative f_X(x) ≥ 0, and while PRIM (and our LSBH procedure) are designed to find bump(s) of a continuous or discrete target function assuming either mode(s) or class(es) respectively (Friedman and Fisher 1999), we focused on a class discovery problem, where the output variable assumes (unorderable) categorical values. Also, while PRIM and our LSBH procedure are designed to be applicable for both discrete and continuous input variables, we focused on the continuous X-variable case.

Using the bump hunting framework recasting scheme (Supplemental Materials 4.6.5) and (Friedman and Fisher 1999; Hastie et al. 2001), we showed how to turn a class discovery problem into this framework. In bump hunting, the goal is to identify those bumps or class domains where the corresponding dummy target function f_g(.) is larger than that of any other class (Supplemental Materials 4.6.5). Those bumps or class domains are those within which an observation is most likely to be from one of the classes. Our study relies on the underlying premise that the class discovery problem will be more efficient if cast into the bump hunting framework: it is more favorable to search for class domains or denser domains of the higher-output observations, within which these observations are most likely to be from one of the classes, than to find boundaries to separate groups of observations based on some measures of purity/impurity (e.g. classification) or similarity/dissimilarity (e.g. clustering).

Since our LSBH procedure successively uses two rule induction methods, one may ask if and how they could be used interchangeably. Empirical evidences indicate that better results were obtained with a greedy algorithm like CART to determine the partition(s) of interest in which the bump hunting is done. This indicates that greediness is important at this stage. Conversely, once in the local SPC space, we observed that by controlling the patience, we generally reduce the risk of overfitting in determining the bumps. Friedman et al. pointed out the advantage of PRIM over CART when patience was needed (Friedman and Fisher 1999). Therefore, patience seems more important at this stage. In conclusion, our premise is to use adequately the greediness and patience of these rule induction methods. Note that bump hunting and classification differ in objective. In classification, the goal is to find a pre-specified number of regions of the input space in which the distribution of the outcome variable is as different as possible from one region to another (Hastie et al. 2001). In contrast, in bump hunting, the goal is to find the so-called bump(s) or class domain(s) of the input space within which the target function is relatively high, without their number being assumed nor known in advance.

A question relates to the adequacy of using the space of maximum variance of the inputs to search for modes in the output variable. We anticipate that a possible extension of our approach would be to treat the inputs simultaneously with the output variable, where the correlation to the response itself would enter into equation. This is the spirit of the methods introduced in high dimensional data by Nguyen et al. (Nguyen and Rocke 2002) and Ghosh (Ghosh 2002), where the Partial Least Square (PLS) components are constructed so that the sample covariance between the response and a linear combination of the p input variables is maximum. One may argue that the PLS criterion is more sensible since there is no a priori reason why constructed components having large input variable variation should be correlated to the output variable. The problem, however, is that PLS is really designed to handle continuous only output response variables and especially for models that do not really suffer from conditional heteroscedasticity as it is the case for binary or multinomial data. In addition, unlike the SPC components whose loadings can be as sparse as needed, the PLS components are not. Clearly, a definite extension of our LSBH procedure, which would benefit from both advantages, would be to construct sparse PLS components. Another alternative would be the recent idea of Supervised Principal Component Analysis (S-PCA) (Bair et al. 2006). In S-PCA, rather than performing the PCA using all of the inputs, one uses only those input variables with the strongest estimated correlation with the output variable. By doing so, a PCA based on this initial subset of inputs will generate components guaranteed to be correlated to the response (with good predictive value). S-PCA is a potential alternative in high dimensional situations because not only are the S-PCA features correlated to the output variable, but they also have a reduced number of loadings by construction.

While the group of metastatic tumors appears clinically homogeneous, our procedure revealed two metastatic subgroups with distinct genetic/molecular profiles, the first of which being close to the most advanced stage of primary tumors, while the second represents a more terminal and aggressive metastatic stage (Dazard et al. 2010). A novel bump gene expression signature was derived, which appears to divide colon cancer into two populations: a population whose expression pattern can be molecularly encompassed within the bump, and an outlier population that cannot be (Dazard et al. 2010). Because of large metastatic patient survival heterogeneity, these subtypes have been suspected to exist for some time ago and are potentially of great interest to colon cancer clinicians and biologists.

Supplementary Material

1. Appendix.

Supplemental Text including introduction to formal setup and details of the bump hunting framework, the gap statistic for clustering, the deviance statistic for recursive partitioning, summary of advantages, and considerations on computational complexity of the LSBH method. Supplemental Figures & Tables. (Appendix.pdf - pdf file).

2. Computer Code.

Two R code files with two README flat text files for cluster setup and R session usage. (Computer Code.zip - zip file).