Optimized Ranking and Selection Methods for Feature Selection with Application in Microarray Experiments

Abstract

In microarray experiments, the goal is often to examine many genes, and select some of them for additional investigation. Traditionally, such a selection problem has been formulated as a multiple testing problem. When the genes of interest are genes with unequal distribution of gene expression under different conditions, multiple testing methods provide an appropriate framework for addressing the selection problems. However, when the genes of interest are a set of genes with the largest difference in gene expression under different conditions, multiple testing methods do not directly address the selection goal and sometimes lead to biased conclusions. For such cases, we propose two methods based on the statistical ranking and selection framework to directly address the selection goal. The proposed methods have an inherent optimization nature in that the selection is optimized according to either a pre-specified minimum correct selection ratio (r*-selection) or probability of making a correct selection (P*-selection). These methods are compared with the multiple testing method which controls the tail probability of the proportion of false positives. Both simulation studies and real data applicaitons provide insight into the fundmental difference between the multiple testing methods and the proposed methods in the way of addressing different selection goals. It has been shown that the proposed methods provide a clear advantage over the multiple testing methods when the goal is to select the most significant genes (not all the significant genes). When the goal is to select all the significant genes, the proposed methods perform equally well as the current multiple testing methods. Another advantage provided by the proposed methods is their ability to detect noisy data and therefore suggest no sensible selection can be made.

1 Introduction

Technology has reached a point that massive datasets, such as microarray data, have established themselves as a permanent part of the statistical landscape. In microarray experiments, the goal is often to examine many genes and select some of them for additional investigation. This problem has traditionally been viewed through the prism of multiple testing, in which the selection of a set of genes is formulated as a series of hypothesis tests on the equal distribution of gene expression for all genes under examination. Since tens of thousands of genes are tested simultaneously, the multiplicity problem arises, i.e., the overall Type I error rate of multiple tests will be drastically inflated. Early proposals for dealing with the multiplicity problem focused on the control of family-wise error rate (FWER) or false discovery rate (FDR) (Benjamini and Hochberg, 1995). For more details, see the review by Dudoit et al. 2003. The FWER was quickly recognized as being too conservative for the analysis of microarray data and was extended to a less stringent error rate: generalized family-wise error rate (gFWER) (Dudoit et al., 2004). gFWER is the probability that k (k ≥ 1) or more false positives occur when multiple tests are conducted. Recently some procedures have been proposed to control gFWER in microarray data analysis (Korn et al., 2004, Van der Laan et al., 2004a,b, Lehmann and Romano, 2005, Xu and Hsu, 2007, Romano and Wolf, 2007). Van der Laan et al., (2004b) also suggested the control of the tail probability of the proportion of false positives (TPPFP) among significant findings. On the other hand, the FDR is also recognized as a useful measure of the false positive error rate in microarray studies (Allison et al., 2006). The FDR can be interpreted as the expected proportion of false positives among significant findings. A few multiple testing procedures have been proposed for FDR control under independent or dependent structure of test statistics (Benjamini and Hochberg, 1995, Yekutieli and Benjamini, 1999, Benjamini and Hochberg, 2001). Storey (2002) suggested that the positive false discovery rate (pFDR) is a more reasonable error rate to use in the analysis of microarray data than FDR, which can be interpreted as the Bayesian posterior probability that a significant finding is a false positive. He also devised a multiple testing procedure to control pFDR which has greater statistical power. Conditional false discovery rate (cFDR) (Tsai et al., 2003, Pounds and Cheng, 2004), the total error criterion (Genovese and Wasserman, 2002) and profile information criterion (Cheng et al., 2004) have also been proposed as alternatives to FDR. See Pounds (2006) and Farcomeni (2008) for the most recent reviews.

When the goal of microarray analysis is to discover differentially expressed genes for intense further study, for which case genes of interest can be considered as genes with unequal distribution of gene expression between the two groups, it is appropriate to use p-value based multiple testing procedures with the appropriate error rate controlled. However, for another common purpose of training a prognostic algorithm for predicting risk, such as gene signature as predictor of survival in cancer (Golub et al., 1999, van de Vijver et al., 2002, van’t Veer et al., 2002, Pepe et al., 2003), the genes of interest should be those with large (standardized) mean distance in gene expression between the two groups. In other words, testing the null state of equal expression seems less compelling than does ranking the extent of differential expression amongst the genes since very often many genes will be differentially expressed in these studies (Pepe et al., 2003, Bickel, 2004). Sometimes limited resources require investigators to pre-specify a number of genes for further study, regardless of the number of differentially expressed genes (Smyth et al., 2002, Matsui et al., 2008). Therefore, selecting genes based on ranking the extent of differential expression amongst genes is a more directly relevant approach. Moreover, the degree of confidence in the ranking of genes is of more concern than the power of detecting "differentially expressed" genes. Pepe et al. (2003) suggested ranking genes by using receiver operating characteristic (ROC) and partial area under curve (pAUC) as statistical measures of discrimination between distributions of gene expression. ROC and pAUC focuses on the separation of distributions rather than the difference of distributions. They also proposed selection probability P_g(k) = P[Rank(g) ≤ k] to quantify the degree of confidence in choosing the gth gene among the top k. Lu et al. (2009) extended the selection probability for each selected gene to the ReDiscovery Rate (RDR) for the k genes selected, which is defined as the expected number of top k genes being within the top k again, divided by k. The nonparametric bootstrap method was used to estimate the selection probability and RDR. Zhang et al. (2006) proposed to calculate the probability that an uninformative (not differentially expressed) gene ranks at or above the observed ranking position by a given ranking method. Saeys et al. (2007) reviewed selection techniques. Matsui et al. (2008) was the first group to advocate the application of statistical ranking and selection framework for gene selection and studied the sample size calculation for achieving minimum probability of correct selection. Cui and Wilson (2008a, 2009) showed how to calculate the probability that a given selection is correct for the d–best and g-best selections which are two extensions of the traditional definition of correct selection.

In this article, we propose two new methods for selecting the best t of k populations when the goal is to select either the most significant ones or all the significant ones. The key element of both methods lies in the utilization of the probability of c-best correct selection in the framework of ranking and selection. The c-best correct selection corresponds to the “goal I” selection criterion in the fixed subset-size approach by Mahamunulu (1967). Besides the proposed methods directly address a specific selection goal, they also have a unique optimization feature so that the optimal number of most significant genes or all significant genes are selected. Therefore, our methods ensure that the truly top-ranking genes are selected with high probability, which increases the likelihood of biologically relevant results. In what follows, the background of c-best correct selection and the corresponding probability as well as the two new selection methods are given in Section 2. In Section 3, two simulation studies are conducted to compare the proposed selection methods to the TPPFP based multiple testing procedure (MTP) under different selection goals and different levels of noise in data. We selected TPPFP for two reasons: (i) it is representative of a family of increasingly popular multiple testing procedures (Farcomeni, 2008), and (ii) it has the highest amount of mathematical and conceptual similarity and is therefore most suitable for comparison. It will be shown in simulation studies that one of the proposed methods has an advantage over TPPFP when the goal is to select the most significant genes (not all the significant genes), and it performs equally well as TPPFP when the goal is to select all significant genes. Application to two well-studied microarray datasets is given in Section 4 and the paper closes with a discussion in Section 5.

2 c-Best Selection

Consider i.i.d. observations X_ij, j = 1, …, n, arising from population π_i with location parameter family G (x − θ_i), i = 1, …, k, denoted by $X_{ij}\stackrel{iid}{\sim }G(x-{\theta }_i)$. Let θ = (θ₁, …, θ_k) and θ₍₁₎ ≤ … ≤ θ_(k) denote the ordered parameters in θ. Assuming θ and the ordering of its elements are unknown, the goal is to select the best t populations. Under the traditional ranking and selection framework (Bechhofer,1954), best t is defined as those populations with the largest parameters, θ_(k−t+1), …, θ_(k). Alternatively, best t can be defined as those populations with the smallest parameters, θ₍₁₎, …, θ_(t). Suppose the sample statistic Y_i = Y (X_i1, X_i2, …, X_in) is an estimate of θ_i with continuous cumulative distribution function F (y − θ_i), i = 1, …, k. Let Y_[1] ≤ … ≤ Y_[k] denote the ordered statistics of Y₁, …, Y_k and Y_(i) denote the sample statistic Y that corresponds to the ith ordered population with parameter θ_(i). Accordingly, the selection procedure, or "rule" R, is

$$R:\text{select}{Y}_{[k-t+1],\dots ,}{Y}_{[k]}\text{as}\text{the}\mathit{\text{best t}}\text{populations}.$$

Rule R results in a correct selection when the event CS_t occurs, where

$$C{S}_t=\{\{Y_{[k-t+1],}{Y}_{[k-t+2],\dots ,}{Y}_{[k]}\}=\{Y_{(k-t+1),}{Y}_{(k-t+2),\dots ,}{Y}_{(k)}\}\}.$$

The probability of correctly selecting the best t populations is defined as follows

$$P(C{S}_t)=P\left(\underset{1\le i\le k-t}{\text{max}}{Y}_{(i)}\le \underset{k-t+1\le j\le k}{\text{min}}{Y}_{(j)}\right)$$ (1)

$$={\displaystyle {\int }_{-\infty }^{\infty }{\displaystyle \prod _{j=k-t+1}^k\overline{F}(y-{\theta }_{(j)})d\left\{{\displaystyle \prod _{i=1}^{k-t}F}(y-{\theta }_{(i)})\right\},}}$$ (2)

where F̄ = 1 − F (see Bechhofer (1954) Equation 8). Alternatively, correct selection, CS_t, can be defined in terms of index sets (Cui and Wilson, 2008a).

Definition 1 Let the indices corresponding to the top t statistics be denoted by set s (for "sample"). Let the indices corresponding to the top t parameters, {θ_(k−t+1), θ_(k−t+2), …, θ_(k)} be denoted by the set A_t. Then the correct selection of t out of k parameters is defined by

$$C{S}_t=\{s=A_t\},\mathit{\text{for}}t=1,2,\dots ,k.$$

Notice that when the elements of θ are strictly unequal there is only one s satisfying CS_t. In the case of ties, more than one s may satisfy CS_t, thereby increasing P (CS_t). However, P (CS_t) often becomes too small to be useful when k is extremely large and the parameters (and statistics) follow a continuous bounded distribution and are therefore relatively dense (Cui and Wilson, 2008a). Under such a parameter configuration, which is very common in microarray gene expression data, it is very difficult to obtain the unique best t populations. Therefore, we extend the definition of correct selection and the corresponding PCS as follows.

Definition 2 Let the indices corresponding to the top t statistics be denoted by the set s. Let the indices corresponding to the top t parameters be denoted by the set A_t. Then

$$C{S}_{c,t}=\{s:|s\cap A_t|\ge c\},$$

where |·| denotes cardinality. A set s that satisfies CS_c,t is defined to be c-best. The probability that a c-best selection occurs P (CS_c,t) is given as follows:

$$P(C{S}_{c,t})={\displaystyle \sum _{g=1}^{|S|}{\displaystyle {\int }_{-\infty }^{\infty }{\displaystyle \prod _{j=k-t+1}^k\overline{F}(y-{\theta }_{s_{g,i}})d}}}\left\{{\displaystyle \prod _{j=1}^{k-t}F(y-{\theta }_{{\overline{s}}_{g,i}})}\right\},$$ (3)

where S is the set of all c-best sets; $({\theta }_{{\overline{s}}_{g,1}},\dots ,{\theta }_{{\overline{s}}_{g,(k-t)}},{\theta }_{s_{g,(k-t+1)}},\dots ,{\theta }_{s_{g,k}})$ is the g^th combination of parameters θ_(i), i = 1,2, …, k which satisfies CS_c,t (Cui and Wilson, 2009).

Remark 1 A c-best set is a set of size t that contains any c or more of the top t populations. c-best corresponds to "goal I" of Mahamunulu (1967).

The probability of making a c-best selection, P (CS_c,t), may be estimated by using a parametric bootstrap method. Suppose $X_{ij}\stackrel{iid}{\sim }G(x-{\theta }_i),j=1,\dots ,n$, and θ_i is estimated by Y_i = Y (X_i1, X_i2, …, X_in) ~ F (y − θ_i), i = 1, …, k. For example, X_ij ~ N (θ_i, σ²) and θ̂_i = Y_i = avg (X_i1, X_i2, …, X_in) ~ N(θ_i, σ²/n), where σ² is known. The estimate of P (CS_c,t) can be calculated as follows.

For the bth bootstrap, b = 1, … B,

1. Obtain a bootstrap sample $Y_i^{*}$ of statistic Y_i by sampling from F(y − θ̂_i), i = 1, …, k.

2. Let $Y_{[1]}^{*}\le \dots \le Y_{[k]}^{*}$ denote the ordered statistics of $Y_1^{*},\dots ,Y_k^{*}$ and θ̂_[1] ≤ … ≤ θ̂_[k] denote the ordered statistic of θ̂₁, …, θ̂_k. Denote the set of indices of populations with $Y_{[k-t+1]}^{*},\dots ,Y_{[k]}^{*}$ by s and indices of populations with θ̂_[k−t+1], …, θ̂_[k] by Â_t. Calculate m_b = |s ∩ Â_t|, where |·| denotes cardinality.

The estimated P (CS_c,t) is

The can also be estimated by using a non-parametric bootstrap method if sampling from F(y−θ̂_i) is replaced by sampling from data ${\{X_{ij}\}}_{j=1}^n.$ For further details, such as convergence speed and accuracy, see Cui and Wilson (2008b, 2009). The algorithm is implemented in the PCS package on R (www.r-project.org. Note: the parameter "c" in this paper is "L" in the package).

In microarray studies, the goal is often to examine many genes, and select some of them for additional investigation. Although how many genes should be selected is usually unknown, the minimum and maximum number of genes to be selected, as denoted by T_min and T_max, can possibly be determined based on the practitioners’ research goal, prior knowledge, and cost constraints. Denote t as the number of populations to be selected within a predefined range, T_min ≤ t ≤ T_max. Let c be the minimum number of correctly selected populations, 1 ≤ c ≤ t, so that r = c /t (correct selection ratio, CSR). Below we introduce two algorithms that can be used to determine the optimal value of t as well as the t most significant genes. The first algorithm is based on the optimization of P (CS_c,t) and the second algorithm is based on the optimization of the correct selection ratio r.

Algorithm 1 r* -selection

Fix CSR r*, 0 < r* < 1. Calculate for T_min ≤ t ≤ T_max, where ⌈r*t⌉ denotes the next integer not smaller than r*t (ceiling function). Select the t with the largest P_t.

Algorithm 2 P*-selection

Fix PCS P*, 0 < P* < 1. Calculate for T_min ≤ t ≤ T_max. Select the t with the largest r_t.

Remark 2 Note that in P*-selection, c is a unique number determined by the value of P*. In r*-selection, if for some r* there are ties of P_t, we choose the largest value of t in order to select more populations while maintaining the fixed CSR r*.

3 Simulation Study

In microarray studies, the problem of selecting "interesting" genes for additional investigation by examining tens of thousands of genes has traditionally been formulated as a hypothesis testing problem. In this article, we propose two algorithms based on maximizing the probability of correct selection (PCS) or the correct selection ratio for the gene selection problem. The fundamental difference between the proposed framework and the MTP lay in the fact that the former directly addresses the selection problem while the later indirectly solves the selection problem when "interesting" is equivalent to "significant". However, for some microarray studies, more than 50% genes are "significantly" differentially expressed, but not all of them are "interesting" (Pepe et al., 2003) and the application of MTP might lead to biased selection. The following two simulations provide some insight into this difference.

The type I error rates controlled by the MTP include family-wise error rate (FWER) and generalized family-wise error rate (gFWER) which measure the tail probability of the number of false positive and false discovery rate (FDR) which measure the expectation of the proportion of false positives among the rejected hypotheses. The tail probability of the proportion of false positives among the rejected hypotheses (TPPFP) has recently been introduced as another class of type I error rate and defined as follows (Van der Laan et al., 2004b):

$$TPPFP(q)=\text{Pr}\{V/t>q\}$$

where q is a user-supplied constant q ∈ (0, 1), t is the number of rejected hypotheses, V is the number of false positive among the rejected hypotheses. By definition, if the ‘best’ is defined as ‘significant’, it is easy to see that the CSR r = 1 − V/t and P (CS_c,t) = Pr{(t−V) > c}. Note that both TPPFP(q) and P (CS_c,t) involve the joint distribution of random variables V and t. Also conditional on t, TPPFP(1 − c/t) = 1 − P (CS_c,t). Therefore, we consider the TPPFP based MTP to be most comparable to the proposed methods.

In the first simulation, we consider the following configuration of parameters: $\mathit{\theta }={\{{\theta }_i\}}_{i=1}^{5000}$, where θ_i = 3, i = 1, …, 4950, θ_i = 7, i = 4951, …, 4975 and θ_i = 10, i = 4976, …, 5000. Genes with θ = 7 and θ = 10 are considered as differentially expressed genes. For illustration purposes, we also assume that the estimator Y_i of θ_i follow a normal distribution with mean θ_i and common known standard deviation σ = 0.5 for i = 1, …, 5000. Therefore, the results are based on a parametric bootstrapping method. However, when there is no distribution assumption for statistics Y_i, a resampling scheme with replacement can easily be implemented to generate bootstrap samples from the empirical distribution of data and similar results can be obtained if the sample size is large (Cui and Wilson, 2009).

To calculate the TPPFP adjusted p-values, the follow steps are taken:

1. Calculate z—scores using z_i = (Y_i − 3)/0.5 and the associated raw p-values using the standard normal distribution.

2. Each raw p-value is then adjusted by Sidak step-down procedure for strong control of FWER.

3. Each Sidak adjusted p-value is then adjusted again by "fwer2tppfp" procedure from the multtest package with q = 1 − r for each specified r* in r*-selection and calculated r in P*-selection.

Two significance cutoff values of α for TPPFP adjusted p-value were chosen that correspond to the largest first gap and largest second gap in the p-values. In practice, the value of α is usually specified in advance and it is obvious to see the advantage of r*-selection and P*-selection in that the selection is optimized based on data. In order to calibrate the r*-selection (or P*-selection) and the α in TPPFP so the methods are on equal footing, α is set according to the gap between adjusted p-values so that the selection using TPPFP adjusted p-value is also optimized.

The simulation was repeated 1,000 times and 100 bootstraps were performed each time for r*-selection and P*-selection. Figure 1a is the histogram of sample statistics Y where the populations corresponding to θ = 7 and θ = 10 are clearly separated from the populations corresponding to θ = 3. Figure 1b is the scatter plot of – log₁₀(p-value) versus the rank of p-values for one simulation with q = 0.05, i.e., r = r* = 0.95 (all p-values in Figure 1b are TPPFP adjusted). As expected, the plot shows two gaps in the p-values. The first 30 p-values were near zero, the negative logarithm of which is very large and hence do not appear on the plot. This is the first gap and we therefore set α = p₍₃₀₎ (any value between p₍₃₀₎ and p₍₃₁₎ will lead to the same selection) and declaring the most significant genes to be those corresponding to p_(i), 1 ≤ i ≤ 30, which is larger than the target of 25. For 1,000 simulations, recorded in Table 1, the average first gap was 35.7 with standard deviation 2.6. When r* was set to 0.60 or 0.75, TPPFP was considered to be unable to detect the first gap since it occurred at 57.2 for r* = 0.60, and 45.7 for r* = 0.75. The second gap is illustrated in Figure 1b around t = 53 using r* = 0.95. Table 1 reports that the largest gap between p-values occurred at t = 52.6 with standard deviation 1.0 over 1.000 simulations. For r*-selection, we calculated P̂_t with 1 ≤ t ≤ 100. The values of P̂_t were plotted against t in Figure 1c. When r* = 0.90,0.95, two peaks appear at t = 25,52. For 1,000 simulations, the average first peaks are 26.9 with standard deviation 2.1 and 25.1 with standard deviation 0.5 respectively and the average second peaks are 53.9 with standard deviation 0.5 and 50.1 with standard deviation 0.6 respectively. For P*-selection, we calculated r̂_t with 1 ≤ t ≤ 100. The values of r̂_t are plotted against t in Figure 1d. Two peaks appear at t = 25, 50 for different values of P*. For 1,000 simulations, the average first peaks and second peaks appear close to 25 and 50 respectively, for different values of P*.

Figure 1.

Graphs of three-point distribution. All graphs are of a single representative simulation (not mean simulation results). a) Histogram of sample statistics Y. b) Plot of – log₁₀ of the TPPFP p-values versus t to investigate gaps. c) Plot of maximum PCS P̂ versus t at four levels of r*. d) Plot of maximum CSR r̂ versus t at four levels of P*.

Table 1.

Gap locations selected by different methods. The means and standard deviations (in parentheses) of 1000 simulations are shown for given values of r* and P*. TPPFP was controlled at the level of r* and the selection for the first gap was the largest p-value near zero while the selection for the second peak was the largest p-value gap. The selection range for the first peak of the three point distribution with r*/P*-selection was 10 ≤ t ≤ 40, for the second peak it was 10 ≤ t ≤ 100. A ** entry indicates the first gap went undetected and selection signified the second gap.

Gap	r* (P*)	TPPFP	r*-sel	P*-sel
first	0.6 (0.5)	**	**	25.3 (0.8)
	0.75	**	**	25.7 (0.8)
	0.90	37.0 (2.8)	26.9 (2.1)	25.9 (0.5)
	0.95	35.7 (2.6)	25.1 (0.5)	25.7 (0.6)
second	0.6 (0.5)	84.0 (1.4)	92.4 (3.2)	50.2 (0.5)
	0.75	66.9 (1.3)	66.9 (1.2)	50.5 (0.5)
	0.90	55.6 (1.0)	53.9 (0.5)	50.0 (0.4)
	0.95	52.6 (1.0)	50.1 (0.6)	50.2 (0.5)

Table 1 summarizes the mean and standard deviation (sd) of the locations of the two gaps/peaks found by the different methods. If the goal is to select all the significant genes as "interesting" genes for further study, as suggested by Table 1, then both TPPFP based MTP and r*-selection method perform equally and the performance of P*-selection method appear to be best and robust. If the goal was to select the most significant genes as "interesting" genes for further study, both r*-selection method and P*-selection method perform equally well and outperform TPPFP based MTP.

In the second simulation study, we assessed how r* and P* selection methods perform differently from TPPFP in a control-treatment comparison when no genes are differentially expressed and the goal is to select all the significant genes. In particular, we considered the case where a tiny difference observed between the control and treatment groups was not due to a biological reason, but to system bias such as different batches of chips and/or different experiment dates for the control and treatment groups. Although various normalization methods have been proposed to remove such bias, it cannot be completely eliminated due to the complexity of system bias. The simulation study was designed as follows.

The control group had k = 5000 means, ${\{\theta \}}_{i=1}^{500}=3,{\{\theta \}}_{i=501}^{1000}=4,{\{\theta \}}_{i=1001}^{1500}=5,\cdots ,{\{\theta \}}_{i=4501}^{5000}=12.$ The standard deviations σ_i were chosen so that the coefficient of variation, CV_i = σ_i/θ_i, was fixed at CV_i = 1/12 for each gene. The ${\sigma }_i^{\prime }s$ were therefore θ_i/12, with values $\frac{3}{12},\frac{4}{12},\dots ,\frac{12}{12}.$ The treatment group had the same standard deviations as the control group, but the means were shifted by 0.4 : η = θ + 0.4. Samples of size n = 20 were simulated from normal distributions and Welch t-statistics, Y_i, were calculated for each gene (Figure 2a).

Figure 2.

Graphs of the noise distribution. All graphs are of a single representative simulation (not mean simulation results). a) Histogram of sample statistics Y for the first group b) Plot of – log₁₀ of the TPPFP p-values versus t to investigate gaps. c) Plot of maximum PCS P̂ versus t at four levels of r*. d) Plot of maximum CSR r̂ versus t at four levels of P*.

Using r* = 0.90, TPPFP selected an average of 61.8 genes (sd = 7.8, determined from 1000 simulations) at the 0.01 level of significance, despite the fact that none should be selected. Using more stringent controls of r* = 0.95 at α = 0.001, TPPFP still selected an average of 9.9 genes (sd = 3.1). By contrast, r* and P*-selection always selected the bound T_max. This selection held for all T_max studied, T_max = 100, 101, …, 300. In other words, the selection decision would change as the prespecified maximum value of t changes. Therefore, r* and P*-selection did not consider any particular selection as optimal. A single simulation for r*-selection is shown in Figure 2c where the graph also indicates no selection should be made. The reason is that for r* = 0.75, 0.90 and 0.95 there is no peak and PCS quickly moves to zero. For r* = 0.60, although PCS is non-zero, there is no peak. It monotonically increases, which indicates that the more genes selected, the more likely the CSR of 0.60 will be achieved. Since PCS monotonically increases with t before it reaches 1, arbitrary T_max would always be selected. For P*-selection, Figure 2d shows that the data is noisy in the same way as just explained for r* = 0.60. In short, r*/P*-selection correctly indicate the user should not make a selection due to too much noise (or no signal) in the data, whereas TPPFP is unable to discriminate the noise and incorrectly selects a relatively large number of erroneous genes. Such phenomena can occur in practice as shown in the Leukemia data example in the next section.

4 Applications

4.1 Apo AI Knock Out Data

Callow et al. (2000) studied the effect of "knocking out" the Apo AI gene in the livers of mice. There were 8 control and 8 treatment cDNA microarrays, from which Welch two-sample t-statistics, Y_i, i = 1, 2, …, 6384 were calculated (Dudoit et al., 2003). The processed data was taken from Bioconductor’s multtest library (www.bioconductor.org). The ApoAI data set was selected because it has biologically verified results reported, with extensive statistical multiple comparison work, in Dudoit et al. (2003). The top t = 8 genes were verified for biological significance (Dudoit et al., 2003), which implies that this dataset has good signal near the top.

Figures 3a and 3b were obtained after applying r*-selection and P*-selection to the data. Instead of assuming the statistics Y_i were Normal with common known variance, we used a shifted Student’s-t distribution with the degrees of freedom estimated using the method of moments for each gene (see Cui and Wilson, 2008b). For r*-selection we used r* = 0.90 with 1 < t < 50, which selected 6 genes. P*-selection selected 6 genes at P* = 0.90 and pre-specified 1 < t < 50 (Figure 3ab). Dudoit et al. (2003) reported 8 genes statistically significant using the maxT procedure (Westfall and Young, 1993) at α = 0.05. According to our calculations using maxT, however, only 6 had p-values below α = 0.05, while 8 had p-values below α = 0.10. Using the same maxT statistics in TPPFP with α = 0.10, and r* = 0.90, 8 genes were chosen. Since the top 8 genes are known to be significant all of the above choices are without error. All three of the methods, therefore, perform similarly in this example.

Figure 3.

Graphs of r*-selection and P*-selection results from the ApoAI and leukemia data.

4.2 Leukemia Data

Golub et al. (1999) sought to create a classifier for leukemia tissue into acute lymphoblastic (ALL) or acute myeloid (AML). They obtained a training sample of 27 Affymetrix microarrays from ALL patients and 11 arrays from AML patients. This leukemia dataset was selected because it has come to be used as a benchmark dataset in microarray studies. To apply r*-selection and P*-selection to it, we obtained the processed data on the k = 3051 genes from Bioconductor’s multtest library (www.bioconductor.org) (see Dudoit et al., 2003, p. 90). Instead of assuming the statistics Y_i were normal with common known variance, we used a shifted student’s-t distribution with the degrees of freedom estimated using the method of moments for each gene.

In the original paper, Golub et al. (1999) found their classifier to work about the same whether they selected the top 10, 50 or 200 genes. More recently, Park et al. (2007) applied a leave-one-out cross-validation approach with three different selection methods to it (Golub-correlation, Support Vector Machines, and Fisher’s Discriminant Analysis). They concluded that different splittings of the training and test sets gave significantly different classification performance results, and that the different selection methods were highly inconsistent with one another. This implies that few genes stand out above the others, which means that the dataset is noisy. On the other hand, Ge et al. (2003) used Welch t-statistics with maxT multiple testing adjusted p-values (Westfall and Young, 1993) and concluded 92 genes were significantly differentially expressed at α = 0.05. TPPFP found 109, 115, or 138 significant for r* = 0.95,0.90, or 0.75. Both r*-selection and P*-selection exhibit non-informative behavior. That is, unless one limits the initial selection to small t, e.g.1 ≤ t ≤ 5, then the methods indicate the largest pre-specified value of t should be selected, as discussed in the second simulation of the previous section. For example, if T_max = 300, then r*-selection and P*-selection would select t = 300 (see Figure 3cd). This conclusion would not necessarily be incorrect, accounting for both CSR and PCS; it is just indicative of noisy data. Thus, while there is some tension in the literature as to how many genes really stand out above the others, it is safe to conclude the data is highly noisy. This is in agreement with the original conclusions of Golub et al. (1999), Park et al. (2007), and Dudoit et al. (2003). This does not imply that the t = 92 genes declared significant by the maxT test are not statistically significant. It does, however, suggest that they may not be truly the best (Cui and Wilson, 2008a) or they may not have biological significance (Bickel, 2004).

5 Discussion

The motivation for this study was the prevalence of multiple testing methods for extremely large k applications where the true goal is not testing, but selection. Although the applications of the present paper focused on microarrays, the theory and methods are fully general. We are currently investigating its application in the identification of microorganisms that are associated with a human disease process. We have investigated two flexible selection methods, r*-selection and P*-selection, based upon the PCS statistic, and compared them to the TPPFP based MTP. The comparisons between r*-selection, P*-selection, and TPPFP based MTP revealed that, (1) P*-selection is the robust and most accurate method if the goal is to select all significant genes, while r*-selection and TPPFP based MTP perform similarly. If the goal is to select most significant genes, then r*-selection and P*-selection perform equally well while selection due to TPPFP based MTP is largely biased, (2) When the data is noisy, r*-selection and P*-selection are able to send a clear message that selection should not be made whereas TPPFP based MTP failed to do so since any MTP allows for a given level of false positives and it is expected to see some rejections even when the groups do not differ.

In light of the results presented, we recommend r*-selection or P*-selection for large k population problems where the user wants to optimize selection according to either r* (for fixed CSR find maximum PCS) or P* (for fixed PCS find maximum CSR), but not both. We also recommend r*-selection or P*-selection when the user is uncertain about the noise level of the data. We recommend TPPFP based MTP when the primary goal is to choose the largest number of populations. Relaxing r* or P* will generally increase the number selected by TPPFP based MTP, but not for r*-selection and P*-selection because of their optimization nature. As a reviewer pointed out, it is possible that TPPFP could be calculated for different levels of P* and trends of r* could be checked in an optimization manner analogous to P*–selection. However, the TPPFP statistic was designed to select as many genes as possible, within constraints, whereas the PCS statistic is designed to select the best set of genes. Hence, it is not surprising that PCS offers an advantage in an optimization context. Lastly, P*-selection gives a more consistent selection size for different values of P* and we recommend it over r*-selection in general. The gain in optimization quality with r*/P*-selection comes at the cost of higher computation time.

In order to apply r*-selection or P*-selection, some guidelines for parameter selection can be given. In general, it is best to select T_min ≥ 5, when possible, because the PCS statistic can be biased upwards for very small values of t. For a discussion of the unbiasedness of the statistic for moderate t, see Wilson (2008). T_max is determined by practical considerations, e.g. "How many genes do we have funds / lab time to investigate when selected?" Consideration should be given regarding what is expected from theory, e.g. small numbers (Apo AI data) or large numbers (Leukemia data)? It has been shown through the examples that since the purpose of r*-selection and P*-selection is optimization, a range of r*/P* values indicate the same selection decisions, which is a positive feature of these methods.

In this study, we attempted as much as possible to make a fair comparison between the methods. Ultimately, however, this broke down. For example, for TPPFP we used the null hypothesis θ_i = 3, i = 1, …, 5000, which subtly offers an advantage due to the p-values being calculated from the truth. Although r*-selection performed sensibly, P̂ shrunk rapidly for large r* under the configurations of θ attempted. In practice, improving estimation of θ could help. Both TPPFP and r*/P*-selection are sensitive to different configurations of θ. One approach to improving estimation of the θ employed in this study would be to cluster the data into the number of groups the populations lie in, and let θ̂ be the group means. Improving estimation of θ in general is an area of our current investigation.

Another issue for further research is that studies based on microarray technology have shown inconsistent results when they are executed by different labs or different platforms. This has raised concern about the reproducibility of the modern high-throughput technology (Tan et al., 2003, Marshall 2004, Shi et al., 2006). To evaluate the likelihood that a set of selected populations is reproducible across labs or studies, Lin et al. (2006) proposed the Reproducibility Probability Score method and Zhang et al. (2006) proposed the Probability of Gene’s Ranking Position. The c-best selection approach of this paper could also be applied to address reproducibility. When extremely k populations are involved in the ranking and selection procedure, it is virtually impossible to have a unique set of t populations stand out Wilson (2008). However, the chance of observing any c (< t) or more of the top t populations can be high. For example, suppose two different labs select different sets of t = 100 of genes, each including 50 of the 100 truly best genes (CS_50,100). Now, while the percentage of genes overlapping between the two labs may be anywhere from 0 to 100%, the results may still be considered reproducible in the sense that the two labs obtained the same rate of 50% of the true top genes. This suggests the gene selection is reproducible as long as the c or more out of t populations selected are among the truly top t populations.