High Dimensional Classification Using Features Annealed Independence Rules

Abstract

Classification using high-dimensional features arises frequently in many contemporary statistical studies such as tumor classification using microarray or other high-throughput data. The impact of dimensionality on classifications is largely poorly understood. In a seminal paper, Bickel and Levina (2004) show that the Fisher discriminant performs poorly due to diverging spectra and they propose to use the independence rule to overcome the problem. We first demonstrate that even for the independence classification rule, classification using all the features can be as bad as the random guessing due to noise accumulation in estimating population centroids in high-dimensional feature space. In fact, we demonstrate further that almost all linear discriminants can perform as bad as the random guessing. Thus, it is paramountly important to select a subset of important features for high-dimensional classification, resulting in Features Annealed Independence Rules (FAIR). The conditions under which all the important features can be selected by the two-sample t-statistic are established. The choice of the optimal number of features, or equivalently, the threshold value of the test statistics are proposed based on an upper bound of the classification error. Simulation studies and real data analysis support our theoretical results and demonstrate convincingly the advantage of our new classification procedure.

1 Introduction

With rapid advance of imaging technology, high-throughput data such as microarray and proteomics data are frequently seen in many contemporary statistical studies. For instance, in the analysis of Microarray data, the dimensionality is frequently thousands or more, while the sample size is typically in the order of tens (West et al., 2001; Dudoit et al., 2002). See Fan and Ren (2006) for an overview. The large number of features presents an intrinsic challenge to classification problems. For an overview of statistical challenges associated with high dimensionality, see Fan and Li (2006).

Classical methods of classification break down when the dimensionality is extremely large. For example, even when the covariance matrix is known, Bickel and Levina (2004) demonstrate convincingly that the Fisher discriminant analysis performs poorly in a minimax sense due to the diverging spectra (e.g., the condition number goes to infinity as dimensionality diverges) frequently encountered in the high-dimensional covariance matrices. Even if the true covariance matrix is not ill conditioned, the singularity of the sample covariance matrix will make the Fisher discrimination rule inapplicable when the dimensionality is larger than sample size. Bickel and Levina (2004) show that the independence rule overcomes the above two problems. However, in tumor classification using microarray data, we hope to find tens of genes that have high discriminative power. The independence rule, studied by Bickel and Levina (2004), does not possess this kind of properties.

The diffculty of high-dimensional classification is intrinsically caused by the existence of many noise features that do not contribute to the reduction of misclassification rate. Though the importance of dimension reduction and feature selection has been stressed and many methods have been proposed in the literature, very little research has been done on theoretical analysis of the impacts of high dimensionality on classification. For example, using most discrimination rules such as the linear discriminants, we need to estimate the population mean vectors from the sample. When the dimensionality is high, even though each component of the population mean vectors can be estimated with accuracy, the aggregated estimation error can be very large and this has adverse effects on the misclassification rate. Therefore, when there is only a fraction of features that account for most of the variation in the data such as tumor classification using gene expression data, using all features will increase the misclassification rate.

To illustrate the idea, we study independence classification rule. Specifically, we give an explicit formula on how the signal and noise affect the misclassification rates. We show formally how large the signal to noise ratio can be such that the effect of noise accumulation can be ignored, and how small this ratio can be before the independence classifier performs as bad as the random guessing. Indeed, as demonstrated in Section 2, the impact of the dimensionality can be very drastic. For the independence rule, the misclassification rate can be as high as the random guessing even when the problem is perfectly classifiable. In fact, we demonstrate that almost all linear discriminants can not perform any better than random guessing, due to the noise accumulation in the estimation of the population mean vectors, unless the signals are very strong, namely the population mean vectors are very far apart.

The above discussion reveals that feature selection is necessary for high-dimensional classification problems. When the independence rule is applied to selected features, the resulting Feature Annealed Independent Rules (FAIR) overcome both the issues of interpretability and the noise accumulation. One can extract the important features via variable selection techniques such as the penalized quasi-likelihood function. See Fan and Li (2006) for an overview. One can also employ a simple two-sample t-test as in Tibshirani et al.(2002) to identify important genes for the tumor classification, resulting in the nearest shrunken centroids method. Such a simple method corresponds to a componentwise regression method or a ridge regression method with ridge parameters tending to ∞ (Fan and Lv, 2007). Hence, it is a specific and useful example of the penalized quasi-likelihood method for feature selection. It is surprising that such a simple proposal can indeed extract all important features. Indeed, we demonstrate that under suitable conditions, the two sample t-statistic can identify all the features that efficiently characterize both classes.

Another popular class of the dimension reduction methods is projection. They have been widely applied to the classification based on the gene expression data. See, for example, principal component analysis in Ghosh (2002), Zou et al. (2004), and Bair et al.(2004); partial least squares in Nguyen and Rocke (2002), Huang and Pan (2003), and Boulesteix (2004); and sliced inverse regression in Chiaromonte and Martinelli (2002), Antoniadis et al.(2003), and Bura and Pfeiffer (2003). These projection methods attempt to find directions that can result in small classification errors. In fact, the directions found by these methods usually put much more weights on features that have large classification power. In general, however, linear projection methods are likely to perform poorly unless the projection vector is sparse, namely, the effective number of selected features is small. This is due to the aforementioned noise accumulation prominently featured in high-dimensional problems, recalling discrimination based on linear projections onto almost all directions can perform as bad as the random guessing.

As direct application of the independence rule is not efficient, we propose a specific form of FAIR. Our FAIR selects the statistically most significant m features according to the componentwise two-sample t-statistics between two classes, and applies the independence classifiers to these m features. Interesting questions include how to choose the optimal m, or equivalently, the threshold value of t-statistic, such that the classification error is minimized, and how this classifier performs compared with the independence rule without feature selection and the oracle-assisted FAIR. All these questions will be formally answered in this paper. Surprisingly, these results are similar to those for the adaptive Neyman test in Fan (1996). The theoretical results also indicate that FAIR without oracle information performs worse than the one with oracle information, and the difference of classification error depends on the threshold value, which is consistent with the common sense.

There is a huge literature on classification. To name a few in addition to those mention before, Bai and Saranadasa (1996) dealt with the effect of high dimensionality in a two-sample problem from a hypothesis testing viewpoint; Friedman (1989) proposed a regularized discriminant analysis to deal with the problems associated with high dimension while performing computations in the regular way; Dettling and Bühlmann (2003) and Bühlmann and Yu (2003) study boosting with logit loss and L₂ loss, respectively, and demonstrate the good performances of these methods in high-dimensional setting; Greenshtein and Ritov (2004), Greenshtein (2006) and Meinshausen (2005) introduced and studied the concept of persistence, which places more emphasis on misclassification rates or expected loss rather than the accuracy of estimated parameters.

This article is organized as follows. In Section 2, we demonstrate the impact of dimensionality on the independence classification rule, and show that discrimination based on projecting observations onto almost all linear directions is nearly the same as random guessing. We establish, in Section 3, the conditions under which two sample t-test can identify all the important features with probability tending to 1. In Section 4, we propose FAIR and give an upper bound of its classification error. Simulation studies and real data analyses are conducted in Section 5. The conclusion of our study is summarized in Section 6. All proofs are given in the Appendix.

2 Impact of High Dimensionality

Consider the p-dimensional classification problem between two classes C₁ and C₂. Suppose that from class C_k, we have n_k observations Y _k1, ⋯, Y _knk in ${\mathbb{R}}^p$. The j-th feature of the i-th sample from class C_k satisfies the model

$$Y_{kij}={\mu }_{kj}+{ϵ}_{kij},k=1,2,i=1,\cdots ,n_k,j=1,\cdots ,p,$$ (2.1)

where μ_kj is the mean effect of the j-th feature in class C_k and ϵ_kij is the corresponding Gaussian random noise for i-th observation. In matrix notation, the above model can be written as

$${\mathit{Y}}_{ki}={\mu }_k+{ϵ}_{ki},k=1,2,i=1,\cdots ,n_k,$$

where μ_k = (μ_k1, ⋯, μ_kp)′ is the mean vector of class C_k and ϵ_ki = (ϵ_ki1, ⋯, ϵ_kip)′ has the distribution N (0, Σ_k). We assume that all observations are independent across samples and in addition, within class C_k, observations Y_k1, ⋯, Y_knk are also identically distributed. Throughout this paper, we make the assumption that the two classes have compatible sample sizes, i.e., c₁ ≤ n₁/n₂ ≤ c₂ with c₁ and c₂ some positive constants.

We first investigate the impact of high dimensionality on classification. For simplicity, we temporarily assume that the two classes C₁ and C₂ have the same covariance matrix Σ. To illustrate our idea, we consider the independence classification rule, which classifies the new feature vector x into class C₁ if

$$\delta \left(\mathit{x}\right)=(\mathit{x}-\mu )\prime {\mathbf{D}}^{-1}({\mu }_1-{\mu }_2)>0,$$

where μ = (μ₁ + μ₂)/2 and D = diag(Σ). This classifier has been thoroughly studied in Bickel and Levina (2004). They showed that in the classification of two normal populations, this independence rule greatly outperforms the Fisher linear discriminant rule under broad conditions when the number of variables is large.

The independence rule depends on the marginal parameters μ₁, μ₂ and $\mathbf{D}=\mathrm{diag}\{{\sigma }_1^2,\cdots ,{\sigma }_p^2\}$. They can easily be estimated from the samples:

$${\hat{\mu }}_k=\sum _{i=1}^{n_k}{\mathit{Y}}_{ki}∕n_k,k=1,2,\hat{\mu }=({\hat{\mu }}_1+{\hat{\mu }}_2)∕2$$

and

$$\hat{\mathbf{D}}=\mathrm{diag}\{(S_{1j}^2+S_{2j}^2)∕2,j=1,\cdots ,p\},$$

where $S_{kj}^2={\sum }_{i=1}^{n_k}{(Y_{kij}-{\stackrel{‒}{Y}}_{kj})}^2∕(n_k-1)$ is the sample variance of the j-th feature in class k and ${\stackrel{‒}{Y}}_{kj}={\sum }_{i=1}^{n_k}{Y}_{ki}∕n_k$. Hence, the plug-in discrimination function is

$$\hat{\delta }\left(\mathit{x}\right)=(\mathit{x}-\hat{\mu })\prime {\hat{\mathbf{D}}}^{-1}({\hat{\mu }}_1-{\hat{\mu }}_2).$$

Denote the parameter by θ = (μ₁, μ₂, Σ). If we have a new observation X from class C₁, then the misclassification rate of $\hat{\delta }$ is

$$W(\hat{\delta },\theta )=P(\hat{\delta }\left(\mathit{X}\right)\le 0\mid {\mathit{Y}}_{ki},i=1,\cdots ,n_k,k=1,2)=1-\Phi \left(\Psi \right),$$ (2.2)

where

$$\Psi =\frac{({\mu }_1-\hat{\mu })\prime {\hat{\mathbf{D}}}^{-1}({\hat{\mu }}_1-{\hat{\mu }}_2)}{\sqrt{({\hat{\mu }}_1-{\hat{\mu }}_2)\prime {\hat{\mathbf{D}}}^{-1}\Sigma {\hat{\mathbf{D}}}^{-1}({\hat{\mu }}_1-{\hat{\mu }}_2)}},$$

and Φ(·) is the standard Gaussian distribution function. The worst case classification error is

$$W\left(\hat{\delta }\right)=\underset{\theta \in \Gamma }{\max }W(\hat{\delta },\theta ),$$

where Γ is some parameter space to be defined. Let n = n₁ + n₂. In our asymptotic analysis, we always consider the misclassification rate of observations from C₁, since the misclassification rate of observations from C₂ can be easily obtained by interchanging n₁ with n₂ and μ₁ with μ₂. The high dimensionality is modeled through its dependence on n, namely p_n → ∞. However, we will suppress its dependence on n whenever there is no confusion.

Let R = D^-1/2 ΣD^-1/2 be the correlation matrix, and λ_max(R) be its largest eigenvalue, and α ≡ (α₁, ⋯, α_p)′ = μ₁ - μ₂. Consider the parameter space

$$\Gamma =\{(\alpha ,\Sigma ):\alpha \prime {\mathbf{D}}^{-1}\alpha \ge C_p,{\lambda }_{\max }\left(\mathbf{R}\right)\le b_0,\underset{1\le j\le p,k=1,2}{\min }{\sigma }_{kj}^2>0\}$$

where C_p is a deterministic positive sequence that depends only on the dimensionality p, and b₀ is a positive constant. Note that α′D^-1α corresponds to the overall strength of signals, and the first condition α′D^-1α ≥ C_p imposes a lower bound on the strength of signals. The second condition λ_max(R) ≤ b₀ requires that the maximum eigenvalue of R should not exceed a positive constant. But since there are no restrictions on the smallest eigenvalue of R, the condition number can still diverge. The third condition $\underset{1\le j\le p,k=1,2}{\min }{\sigma }_{kj}^2>0$ ensures that there are no deterministic features that make classification trivial and the diagonal matrix D is always invertible. We will consider the asymptotic behavior of $W(\hat{\delta },\theta )$ and $W\left(\hat{\delta }\right)$.

Theorem 1

Suppose that log p = o(n), n = o(p) and nC_p → ∞. Then (i) The classification error W (δ, θ) with θ ϵ Γ is bounded from above as

$$W(\hat{\delta },\theta )\le 1-\Phi \left(\frac{{[n_1{n}_2∕\left(pn\right)]}^{1∕2}\alpha \prime {\mathbf{D}}^{-1}\alpha (1+o_P\left(1\right))+\sqrt{p∕\left(n{n}_1{n}_2\right)}(n_1-n_2)}{2\sqrt{{\lambda }_{\max }\left(\mathbf{R}\right)}{\{1+n_1{n}_2∕\left(pn\right)\alpha \prime {\mathbf{D}}^{-1}\alpha (1+o_P\left(1\right))\}}^{1∕2}}\right).$$

(ii) Suppose p/(nC_p) → 0. For the worst case classification error W (δ), we have

$$W\left(\hat{\delta }\right)=1-\Phi \left(\frac{1}{2}{[n_1{n}_2∕\left(pn{b}_0\right)]}^{1∕2}{C}_p\{1+o_P\left(1\right)\}\right).$$

Specifically, when ${\left\{\frac{n_1{n}_2}{pn}\right\}}^{1∕2}{C}_p\to C_0$ with C₀ a nonnegative constant, then

$$W\left(\hat{\delta }\right)\stackrel{P}{\to }1-\Phi (C_0∕\left(2\sqrt{b_0}\right)).$$

In particular, if C₀ = 0, then $W\left(\hat{\delta }\right)\stackrel{P}{\to }\frac{1}{2}$.

Theorem 1 reveals the trade-off between the signal strength C_p and the dimensionality, reflected in the term $C_p∕\sqrt{p}$ when all features are used for classification. It states that the independence rule $\hat{\delta }$ would be no better than the random guessing due to noise accumulation, unless the signal levels are extremely high, say, ${\left\{\frac{n}{p}\right\}}^{1∕2}{C}_p\ge B$ for some B > 0. Indeed, discrimination based on linear projections to almost all directions performs nearly the same as random guessing, as shown in the theorem below. The poor performance is caused by noise accumulation in the estimation of μ₁ and μ₂.

Theorem 2

Suppose that a is a p-dimensional uniformly distributed unit random vector on a (p - 1)-dimensional sphere. Let λ₁, ⋯, λ_p be the eigenvalues of the covariance matrix Σ. Suppose ${\lim }_p\frac{1}{p^2}{\sum }_{j=1}^p{\lambda }_j^2<\infty$ and ${\lim }_p\frac{1}{p}{\sum }_{j=1}^p{\lambda }_j=\tau$ with τ a positive constant. Moveover, assume that p^-1α′α → 0. Then if we project all the observations onto the vector a and use the classifier

$${\hat{\delta }}_{\mathbf{a}}\left(\mathit{x}\right)=(\mathbf{a}\prime \mathit{x}-\mathbf{a}\prime \hat{\mu })(\mathbf{a}\prime {\hat{\mu }}_1-\mathbf{a}\prime {\hat{\mu }}_2),$$ (2.3)

the misclassification rate of $\hat{\delta }\mathbf{a}$ satisfies

$$P({\hat{\delta }}_{\mathbf{a}}\left(\mathit{X}\right)\le 0\mid {\mathit{Y}}_{ki},i=1,\cdots ,n_k,k=1,2)\stackrel{P}{\to }\frac{1}{2},$$

where the probability is taken with respect to a and X ϵ C₁.

3 Feature Selection by Two-Sample t-Test

To extract salient features, we appeal to the two sample t-test statistics. Other componentwise tests such as the rank sum test can also be used, but we do not pursue those in detail. The two-sample t-statistic for feature j is defined as

$$T_j=\frac{{\stackrel{‒}{Y}}_{1j}-{\stackrel{‒}{Y}}_{2j}}{\sqrt{S_{1j}^2∕n_1+S_{2j}^2∕n_2}},j=1,\cdots ,p,$$ (3.1)

where ${\stackrel{‒}{Y}}_{kj}$ and $S_{kj}^2$ are the same as those defined in Section 1. We work under more relaxed technical conditions: the normality assumption is not needed. Instead, we assume merely that the noise vectors ϵ_ki,i = 1, ⋯, n_k are i.i.d. within class C_k with mean 0 and covariance matrix Σ_k, and are independent between classes. The covariance matrix Σ₁ can also differ from Σ₂.

To show that the t-statistic can select all the important features with probability 1, we need the following condition.

Condition 1

1. Assume that the vector α = μ₁ - μ₂ is sparse and without loss of generality, only the first s entries are nonzero.

2. Suppose that ϵ_kij and ${ϵ}_{kij}^2-1$ satisfy the Cramér's condition, i.e., there exist constants ν₁, ν₂, M₁ and M₂, such that $E{\mid {ϵ}_{kij}\mid }^m\le m!M_1^{m-2}{\nu }_1∕2$ and $E{\mid {ϵ}_{kij}^2-{\sigma }_{kj}^2\mid }^m\le m!M_2^{m-2}{\nu }_2∕2$ for all m = 1, 2, ⋯.

3. Assume that the diagonal elements of both Σ₁ and Σ₂ are bounded away from 0.

The following theorem describes the situation under which the two sample t-test can pick up all important features by choosing an appropriate critical value. Recall that c₁ ≤ n₁/n₂ ≤ c₂ and n = n₁ + n₂.

Theorem 3

Let s be a sequence such that log(p - s) = o(n^γ) and $\log s=o\left(n^{\frac{1}{2}-\gamma }{\beta }_n\right)$ for someβ_n → ∞ and $0<\gamma <\frac{1}{3}$. Suppose that $\underset{1\le j\le s}{\min }\frac{\mid {\alpha }_j\mid }{\sqrt{{\sigma }_{1j}^2+{\sigma }_{2j}^2}}=n^{-\gamma }{\beta }_n$. Then under Condition 1, for x ~ cn^γ/2 with c some positive constant, we have

$$P(\underset{j\le s}{\min }\mid T_j\mid \ge xand\underset{j>s}{\max }\mid T_j\mid <x)\to 1.$$

In the proof of Theorem 3, we used the moderate deviation results of the two-sample t-statistic (see Cao, 2007 or Shao, 2005). Theorem 3 allows the lowest signal level to decay with sample size n. As long as the rate of decay is not too fast and the sample size is not too small, the two sample t-test can pick up all the important features with probability tending to 1.

4 Features Annealed Independence Rules

We apply the independence classifier to the selected features, resulting in a Features Annealed Independence Rule (FAIR). In many applications such as tumor classification using gene expression data, we would expect that elements in the population mean difference vector α are sparse: most entries are small. Thus, even if we could use t-test to correctly extract out all these features, the resulting choice is not necessarily optimal, since the noise accumulation can even exceed the signal accumulation for faint features. This can be seen from Theorem 1. Therefore, it is necessary to further single out the most important features that help reduce misclassification rate.

To help us select the number of features, or the critical value of the test statistic, we first consider the ideal situation that the important features are located at the first m coordinates and our task is to merely select m to minimize the misclassification rate. This is the case when we have the ideal information about the relative importance of features, as measured by |α_j|/σ_j, say. When such an oracle information is unavailable, we will learn it from the data. In the situation that we have vague knowledge about the importance of features such as tumor classification using gene expression data, we can give high ranks to features with large |α_j|/σ_j.

In the presentation below, unless otherwise specified, we assume that the two classes C₁ and C₂ are both from Gaussian distributions and the common covariance matrix is the identity, i.e., Σ₁ = Σ₂ = I. If this common covariance matrix is known, the independence classifier $\hat{\delta }$ becomes the nearest centroids classifier

$${\hat{\delta }}_{\mathrm{NC}}\left(\mathit{x}\right)=(\mathit{x}-\hat{\mu })\prime ({\hat{\mu }}_1-{\hat{\mu }}_2).$$

If only the first m dimensions are used in the classification, the corresponding features annealed independence classifier becomes

$${\hat{\delta }}_{\mathrm{NC}}^m\left(\mathit{x}\right)=({\mathit{x}}^m-{\hat{\mu }}^m)\prime ({\hat{\mu }}_1^m-{\hat{\mu }}_2^m),$$

where the superscript m means that the vector is truncated after the first m entries. This is indeed the same as the nearest shrunken centroids method of Tibshirani et al. (2002).

Theorem 4

Consider the truncated classifier ${\hat{\delta }}_{NC}^{m_n}$ for a given sequence m_n. Suppose that $\frac{n}{\sqrt{m_n}}{\sum }_{j=1}^{m_n}{\alpha }_j^2\to \infty$ as m_n → ∞. Then the classification error of ${\hat{\delta }}_{NC}^{m_n}$ is

$$W({\hat{\delta }}_{NC}^{m_n},\theta )=1-\Phi \left(\frac{(1+o_P\left(1\right))\sum _{j=1}^{m_n}{\alpha }_j^2+m_n(n_1-n_2)∕\left(n_1{n}_2\right)}{2{\{(1+o_P\left(1\right))\sum _{j=1}^{m_n}{\alpha }_j^2+n{m}_n∕n_1{n}_2\}}^{1∕2}}\right),$$

where n = n₁ + n₂ as defined in Section 2.

In the following, we suppress the dependence of m on n when there is no confusion. The above theorem reveals that the ideal choice on the number of features is

$$m_0={\mathrm{argmax}}_{1\le m\le p}\frac{{[\sum _{j=1}^m{\alpha }_j^2+m(n_1-n_2)∕\left(n_1{n}_2\right)]}^2}{nm∕\left(n_1{n}_2\right)+\sum _{j=1}^m{\alpha }_j^2}.$$

It can be estimated as

$${\hat{m}}_0={\mathrm{argmax}}_{1\le m\le p}\frac{{[\sum _{j=1}^m{\hat{\alpha }}_j^2+m(n_1-n_2)∕\left(n_1{n}_2\right)]}^2}{nm∕\left(n_1{n}_2\right)+\sum _{j=1}^m{\hat{\alpha }}_j^2},$$

where ${\hat{\alpha }}_j={\hat{\mu }}_{1j}-{\hat{\mu }}_{2j}$. The expression for m₀ quantifies how the signal and the noise affect the misclassification rates as the dimensionality m increases. In particular, when n₁ = n₂, the express reduces to $m_0={\mathrm{argmax}}_{1\le m\le p}\frac{{\left[m^{-1∕2}{\sum }_{j=1}^m{\alpha }_j^2\right]}^2}{2∕n+{\sum }_{j=1}^m{\alpha }_j^2∕m}$. The term $m^{-1∕2}{\sum }_{j=1}^m{\alpha }_j^2$ reflects the trade-off between the signal and noise as dimensionality m increases.

The good performance of the classifier ${\hat{\delta }}_{\mathrm{NC}}^m$ depends on the assumption that the largest entries of α cluster at the first m dimensions. An ideal version of the classifier $\hat{\delta }\mathrm{NC}$ is to select a subset $\mathcal{A}=\{j:\mid {\alpha }_j\mid >a\}$ and use this subset to construct independence classifier. Let m be the number of elements in $\mathcal{A}$. The oracle classifier can be written as

$${\hat{\delta }}_{\mathrm{orc}}\left(\mathit{x}\right)=\sum _{j=1}^p{\hat{\alpha }}_j(x_j-{\hat{\mu }}_j)1_{\{\mid {\alpha }_j\mid >a\}}.$$

The misclassification rate is approximately

$$1-\Phi \left(\frac{\sum _{j\in \mathcal{A}}{\alpha }_j^2+m(n_1-n_2)∕\left(n_1{n}_2\right)}{2{\{nm∕\left(n_1{n}_2\right)+\sum _{j\in \mathcal{A}}{\alpha }_j^2\}}^{1∕2}}\right),$$ (4.1)

when $\frac{n}{\sqrt{m}}{\sum }_{j\in \mathcal{A}}{\alpha }_j^2\to \infty$ and m → ∞. This is straightforward from Theorem 4. In practice, we do not have such an oracle, and selecting the subset $\mathcal{A}$ is difficult. A simple procedure is to use the feature annealed independence rule based on the hard thresholding:

$${\hat{\delta }}_{\mathrm{FAIR}}^b\left(\mathit{x}\right)=\sum _{j=1}^p{\hat{\alpha }}_j(x_j-{\hat{\mu }}_j)1_{\{\mid {\hat{\alpha }}_j\mid >b\}}.$$

We study the classification error of FAIR and the impact of the threshold b on the classification result in the following theorem.

Theorem 5

Suppose that ${\max }_{j\in {\mathcal{A}}^c}\mid {\alpha }_j\mid <b_n$ and $\log (p-m)∕\left[n{(b_n-{\max }_{j\in {\mathcal{A}}^c}\mid {\alpha }_j\mid )}^2\right]\to 0$ with $m=\mid \mathcal{A}\mid$. Moreover, assume that $\frac{n}{\sqrt{m}}{\sum }_{j\in \mathcal{A}}{\alpha }_j^2\to \infty$ and ${\sum }_{j\in \mathcal{A}}\mid {\alpha }_j\mid ∕\left[\sqrt{n}{\sum }_{j\in \mathcal{A}}{\alpha }_j^2\right]\to 0$. Then

$$W({\hat{\delta }}_{FAIR}^{b_n},\theta )\le 1-\Phi \left(\frac{(1+o_P\left(1\right))\sum _{j\in \mathcal{A}}{\alpha }_j^2+nm{\left(n_1{n}_2\right)}^{-1}-m{b}_n^2}{2{\{(1+o_P\left(1\right))\sum _{j\in \mathcal{A}}{\alpha }_j^2+nm{\left(n_1{n}_2\right)}^{-1}\}}^{1∕2}}\right).$$

Notice that the upper bound of $W({\hat{\delta }}_{\mathrm{FAIR}}^{b_n},\theta )$ in Theorem 5 is greater than the classification error in Theorem 4, and the magnitude of difference depends on $m{b}_n^2$. This is expected as estimating the set $\mathcal{A}$ increases the classification error. These results are similar to those in Fan (1996) for high-dimensional hypothesis testing.

When the common covariance matrix is different from the identity, FAIR takes a slightly different form to adapt to the unknown componentwise variance:

$${\hat{\delta }}_{\mathrm{FAIR}}\left(\mathit{x}\right)=\sum _{j=1}^p{\hat{\alpha }}_j(x_j-{\hat{\mu }}_j)∕{\hat{\sigma }}_j^2{1}_{\{\sqrt{n∕\left(n_1{n}_2\right)}\mid T_j\mid >b\}},$$ (4.2)

where T_j is the two sample t-statistic. It is clear from (4.2) that FAIR works the same way as that we first sort the features by the absolute values of their t-statistics in the descending order, and then take out the first m features to classify the data. The number of features can be selected by minimizing the upper bound of the classification error given in Theorem 1. The optimal m in this sense is

$$m_1={\mathrm{argmax}}_{1\le m\le p}\frac{1}{{\lambda }_{\max }^m}\frac{{[\sum _{j=1}^m{\alpha }_j^2∕{\sigma }_j^2+m(1∕n_2-1∕n_1)]}^2}{nm∕\left(n_1{n}_2\right)+\sum _{j=1}^m{\alpha }_j^2∕{\sigma }_j^2},$$

where ${\lambda }_{\max }^m$ is the largest eigenvalue of the correlation matrix R^m of the truncated observations. It can be estimated from the samples:

$$\begin{array}{cc}\hfill {\hat{m}}_1& ={\mathrm{argmax}}_{1\le m\le p}\frac{1}{{\hat{\lambda }}_{\max }^m}\frac{{[\sum _{j=1}^m{\hat{\alpha }}_j^2∕{\hat{\sigma }}_j^2+m(1∕n_2-1∕n_1)]}^2}{nm∕\left(n_1{n}_2\right)+\sum _{j=1}^m{\hat{\alpha }}_j^2∕{\hat{\sigma }}_j^2}\hfill \\ \hfill & ={\mathrm{argmax}}_{1\le m\le p}\frac{1}{{\hat{\lambda }}_{\max }^m}\frac{n{[\sum _{j=1}^m{T}_j^2+m(n_1-n_2)∕n]}^2}{m{n}_1{n}_2+n_1{n}_2\sum _{j=1}^m{T}_j^2}.\hfill \end{array}$$ (4.3)

Note that the factor ${\lambda }_{\max }^m$ in (4.3) increases with m, which makes ${\hat{m}}_1$ usually smaller than ${\hat{m}}_0$.

5 Numerical Studies

In this section we use a simulation study and three real data analyses to illustrate our theoretical results and to verify the performance of our newly proposed classifier FAIR.

5.1 Simulation Study

We first introduce the model. The covariance matrices Σ₁ and Σ₂ for the two classes are chosen to be the same. For the distribution of the error ϵ_ij in (2.1), we use the same model as that in Fan, Hall and Yao (2006). Specifically, features are divided into three groups. Within each group, features share one unobservable common factor with different factor loadings. In addition, there is an unobservable common factor among all the features across three groups. For simplicity, we assume that the number of features p is a multiple of 3. Let Z_ij be a sequence of independent N(0, 1) random variables, and ${\chi }_{ij}^2$ be a sequence of independent random variables of the same distribution as $({\chi }_d^2-d)∕\sqrt{2d}$ with ${\chi }_d^2$ the Chi-square distribution with degrees of freedom d. In the simulation we set d = 6.

Let {a_j} and {b_j} be factor loading coefficients. Then the error in (2.1) is defined as

$${ϵ}_{ij}=\frac{Z_{ij}+a_{1j}{\chi }_{1i}+a_{2j}{\chi }_{2i}+a_{3j}{\chi }_{3i}+b_j{\chi }_{4i}}{{(1+a_{1j}^2+a_{2j}^2+a_{3j}^2+b_j^2)}^{1∕2}},i=1,\cdots ,n_k,j=1,\cdots ,p,$$

where a_ij = 0 except that a_1j = a_j for j = 1, ⋯, p/3, a_2j = a_j for j = (p/3) + 1, ⋯, 2p/3, and a_3j = a_j for j = (2p/3)+1, ⋯, p. Therefore, ϵ_ij = 0 and var(ϵ_ij) = 1, and in general, within group correlation is greater than the between group correlation. The factor loadings a_j and b_j are independently generated from uniform distributions U(0, 0.4) and U(0, 0.2). The mean vector μ₁ for class C₁ is taken from a realization of the mixture of a point mass at 0 and a double-exponential distribution:

$$(1-c){\delta }_0+\frac{1}{2}c\exp (-2\mid x\mid ),$$

where c ϵ (0, 1) is a constant. In the simulation, we set p = 4, 500 and c = 0.02. In other words, there are around 90 signal features on an average, many of which are weak signals. Without loss of generality, μ₂ is set to be 0. Figure 1 shows the true mean difference vector α, which is fixed across all simulations. It is clear that there are only very few features with signal levels exceeding 1 standard deviation of the noise.

Figure 1.

True mean difference vector α. x-axis represents the dimensionality, and y-axis shows the values of corresponding entries of α.

With the parameters and model above, for each simulation, we generate n₁ = 30 training data from class C₁ and n₂ = 30 training data from C₂. In addition, separate 200 samples are generated from each of the two classes in each simulation, and these 400 vectors are used as test samples. We apply our newly proposed classifier FAIR to the simulated data. Specifically, for each feature, the t-test statistic in (3.1) is calculated using the training sample. Then the features are sorted in the decreasing order of the absolute values of their t-statistics. We then examine the impact of the number of features m on the misclassification rate. In each simulation, with m ranging from 1 to 4500, we construct the feature annealed independence classifiers using the training samples, and then apply these classifiers to the 400 test samples. The classification errors are compared to those of the independence rule with the oracle ordering information, which is constructed by repeating the above procedure except that in the first step the features are ordered by their true signal levels, |α|, instead of by their t-statistics

The above procedure is repeated 100 times, and averages and standard errors of the misclassification rates (based on 400 test samples in each simulation) are calculated across the 100 simulations. Note that the average of the 100 misclassification rates is indeed computed based on 100 × 400 testing samples.

Figure 2 depicts the misclassification rate as a function of the number of features m. The solid curves represent the average of classification rates across the 100 simulations, and the corresponding dashed curves are 2 standard errors (i.e. the standard deviation of 100 misclassification rates divided by 10) away from the solid one. The misclassification rates using the first 80 features in Figure 2(a) are zoomed in Figure 2(b). Figures 2(c) and 2(d) are the same as 2(a) and 2(b) except that the features are arranged in the decreasing order of |α|, i.e., the results are based on the oracle-assisted feature annealed independence classifier. From these plots we see that the classification results of FAIR are close to those of the oracle-assisted independence classifier. Moreover, as the dimensionality m grows, the misclassification rate increases steadily due to the noise accumulation. When all the features are included, i.e. m = 4500, the misclassification rate is 0.2522, whereas the minimum classification errors are 0.0128 in plot 2(b) and 0.0020 in plot 2(d). These results are consistent with Theorem 1. We also tried to decrease the signal levels, i.e., the mean of the double exponential distribution, or to increase the dimensionality p, and found that the classification error tend to 0.5 when all the dimensions are included. Comparing Figures 2(a) and 2(b) to Figures 2(c) and 2(d), we see that the features ordered by t-statistics has higher misclassification rates than those ordered by the oracle. Also, using t-statistics results in larger minimum classification errors (see plots 2(b) and 2(d)), but the differences are not very large.

Figure 2.

Number of features versus misclassification rates. The solid curves represent the averages of classification errors across 100 simulations. The dashed curves are 2 standard errors away from the solid curves. The x-axis represents the number of features used in the classification, and the y-axis shows the misclassification rates. (a) The features are ordered in a way such that the corresponding t-statistics are decreasing in absolute values. (b) The amplified plot of the first 80 values of x-axis in plot (a). (c) The same as (a) except that the features are arranged in a way such that the corresponding true mean differences are decreasing in absolute values. (d) The amplified plot of the first 80 values of x-axis in plot (c).

Figure 3 shows the classification errors of the independence rule based on projected samples onto randomly chosen directions across 100 simulations. Specifically, in each of the simulations in Figure 2, we generate a direction vector a randomly from the (p - 1)-dimensional unit sphere, then project all the data in that simulation onto the direction a, and finally apply the Fisher discriminant to the projected data (see (2.3)). The average of these misclassification rates is 0.4986 and the corresponding standard deviation is 0.0318. These results are consistent with our Theorem 2.

Figure 3.

Classification errors of the independence rule based on projected samples onto randomly chosen directions over 100 simulations.

Finally, we examine the effectiveness of our proposed method (4.3) for selecting features in FAIR. In each of the 100 simulations, we apply (4.3) to choose the number of features and compute the resulting misclassification rate based on 400 test samples. We also use the nearest shrunken centroids of Tibshirani et al. (2003) to select the important features. Figure 4 summarizes these results. The thin curves correspond to the nearest shrunken centroids method, and the thick curves correspond to FAIR. Figure 4(a) presents the number of features calculated from these two methods, and Figure 4(b) shows the corresponding misclassification rates. For our newly proposed classifier FAIR, the average of the optimal number of features over 100 simulations is 29.71, which is very close to the smallest number of features with the minimum misclassification rate in Figure 2(d). The misclassification rates of FAIR in Figure 4(b) have average 0.0154 and standard deviation 0.0085, indicating the outstanding performance of FAIR. Nearest shrunken centroids method is unstable in selecting features. Over the 100 simulations, there are several realizations in which it chooses plenty of features. We truncated Figure 4 to make it easier to view. The average number of features chosen by the nearest shrunken centroids is 28.43, and the average classification error is 0.0216, with corresponding standard deviation 0.0179. It is clear that nearest shrunken centroids method tends to choose less features than FAIR, but the misclassification rates are larger.

Figure 4.

The thick curves correspond to FAIR, while the thin curves correspond to the nearest shrunken centroids method. (a) The numbers of features chosen by (4.3) and by the nearest shrunken centroids method over 100 simulations. (b) Corresponding classification errors based on the optimal number of features chosen in (a) over 100 simulations.

5.2 Real Data Analysis

5.2.1 Leukemia Data

Leukemia data from high-density Affymetrix oligonucleotide arrays were previously analyzed in Golub et al.(1999), and are available at http://www.broad.mit.edu/cgi-bin/cancer/datasets.cgi. There are 7129 genes and 72 samples coming from two classes: 47 in class ALL (acute lymphocytic leukemia) and 25 in class AML (acute mylogenous leukemia). Among these 72 samples, 38 (27 in class ALL and 11 in class AML) are set to be training samples and 34 (20 in class ALL and 14 in class AML) are set as test samples.

Before classification, we standardize each sample to zero mean and unit variance as done by Dudoit et al. (2002). The classification results from the nearest shrunken centroids (NSC hereafter) method and FAIR are shown in Table 1. The nearest shrunken centroids method picks up 21 genes and makes 1 training error and 3 test errors, while our method chooses 11 genes and makes 1 training error and 1 test error. Tibshirani et al.(2002) proposed and applied the nearest shrunken centroids method to the unstandardized Leukemia dataset. They chose 21 genes and made 1 training error and 2 test errors. Our results are still superior to theirs.

Table 1.

Classification errors of Leukemia data set

Method	Training error	Test error	No. of selected genes
Nearest shrunken centroids	1/38	3/34	21
FAIR	1/38	1/34	11

To further evaluate the performance of the two classifiers, we randomly split the 72 samples into training and test sets. Specifically, we set approximately 100γ% of the observations from class ALL and 100γ% of the observations from class AML as training samples, and the rest as test samples. FAIR and NSC are applied to the training data, and their performances are evaluated by the test samples. The above procedure is repeated 100 times for γ = 0.4, 0.5 and 0.6, respectively, and the distributions of test errors of FAIR, NSC and the independence rule without feature selection are summarized in Figure 5. In each of the splits, we also calculated the difference of test errors between NSC and FAIR, i.e., the test error of FAIR minus that of NSC, and the distribution is summarized in Figure 5. The top panel of Figure 6 shows the number of features selected by FAIR and NSC for γ = 0.4. The results for the other two values of γ are similar so we do not present here to save the space. From these figures we can see that the performance of independence rule improves significantly after feature selection. The classification errors of NSC and FAIR are approximately the same. As we have already noticed in the simulation study, NSC is not good with feature selection, that is, the number of features selected by NSC is very large and unstable, while the number of features selected by FAIR is quite reasonable and stable over different random splits. Clearly, the independent rule without feature selection performs poorly.

Figure 5.

Leukemia data. Boxplots of test errors of FAIR, NSC and the independence rule without feature selection over 100 random splits of 72 samples, where 100γ% of the samples from both classes are set as training samples. The three plots from left to right correspond to γ = 0.4, 0.5 and 0.6, respectively. In each boxplot above, “FAIR” refers to the test errors of the feature annealed independent rule; “NSC” corresponds to the test errors of nearest shrunken centroids method; “diff.” means the difference of the test errors of FAIR and those of NSC; and “IR” corresponds the test errors of independence rule without feature selection.

Figure 6.

Leukemia, Lung Cancer, and Prostate data sets. The number of features selected by FAIR and NSC over 100 random splits of the total samples. In each split, 100γ% of the samples from both class are set as training samples, and the rest are used as test samples. The three plots from top to bottom correspond to the Leukemia data with γ = 0.4, the Lung Cancer data with γ = 0.5 and the Prostate cancer data with γ = 0.6, respectively. The thin curves show the results from NSC, and the thick curves correspond to FAIR. The plots are truncated to make them easy to view.

5.2.2 Lung Cancer Data

We evaluate our method by classifying between malignant pleural mesothelioma (MPM) and adenocarcinoma (ADCA) of the lung. Lung cancer data were analyzed by Gordon et al.(2002) and are available at http://www.chestsurg.org. There are 181 tissue samples (31 MPM and 150 ADCA). The training set contains 32 of them, with 16 from MPM and 16 from ADCA. The rest 149 samples are used for testing (15 from MPM and 134 from ADCA). Each sample is described by 12533 genes.

As in the Leukemia data set, we first standardize the data to zero mean and unit variance, and then apply the two classification methods to the standardized data set. Classification results are summarized in Table 2. Although FAIR uses 5 more genes than the nearest shrunken centroids method, it has better classification results: both methods perfectly classify the training samples, while our classification procedure has smaller test error.

Table 2.

Classification errors of Lung Cancer data

Method	Training error	Test error	No. of selected genes
Nearest shrunken centroids	0/32	11/149	26
FAIR	0/32	7/149	31

We follow the same procedure as that in Leukemia example to randomly split the 181 samples into training and test sets. FAIR and NSC are applied to the training data, and the test errors are calculated using the test data. The procedure is repeated 100 times with γ = 0.4, 0.5 and 0.6, respectively, and the test error distributions of FAIR, NSC and the independence rule without feature selection can be found in Figure 7. We also present the difference of the test errors between FAIR and NSC in Figure 7. The numbers of features used by FAIR and NSC with γ = 0.5 are shown in the middle panel of Figure 6. Figure 7 shows again that feature selection is very important in high dimensional classification. The performance of FAIR is close to NSC in terms of classification error (Figure 7), but FAIR is stable in feature selection, as shown in the middle panel of Figure 6. One possible reason of Figure 7 might be that the signal strength in this Lung Cancer dataset is relatively weak, and more features are needed to obtain the optimal performance. However, the estimate of the largest eigenvalue is not accurate anymore when the number of features is large, which results in inaccurate estimates of m₁ in (4.3).

Figure 7.

Lung cancer data. The same as Figure 5 except that the data set is different.

5.2.3 Prostate Cancer Data

The last example uses the prostate cancer data studied in Singh et al.(2002). The data set is available at http://www.broad.mit.edu/cgi-bin/cancer/datasets.cgi. The training data set contains 102 patient samples, 52 of which (labeled as “tumor”) are prostate tumor samples and 50 of which (labeled as “Normal”) are prostate samples. There are around 12600 genes. An independent set of test samples is from a different experiment and has 25 tumor and 9 normal samples.

We preprocess the data by standardizing the gene expression data as before. The classification results are summarized in Table 3. We make the same test error as and a bit larger training error than the nearest shrunken centroids method, but the number of selected genes we use is much less.

Table 3.

Classification errors of Prostate Cancer data set

Method	Training error	Test error	No. of selected genes
Nearest shrunken centroids	8/102	9/34	6
FAIR	10/102	9/34	2

The samples are randomly split into training and test sets in the same way as before, the test errors are calculated, and the number of features used by these two methods are recorded. Figure 8 shows the test errors of FAIR, NSC and the independence rule without feature selection, and the difference of the test errors of FAIR and NSC. The bottom panel of Figure 6 presents the numbers of features used by FAIR and NSC in each random split for γ = 0.6. As we mentioned before, the plots for γ = 0.4 and 0.5 are similar so we omit them in the paper. The performance of FAIR is better than that of NSC both in terms of classification error and in terms of the selection of features. The good performance of FAIR might be caused by the strong signal level of few features in this data set. Due to the strong signal level, FAIR can attain the optimal performance with small number of features. Thus, the estimate of m₁ in (4.3) is accurate and hence the actual performance of FAIR is good.

Figure 8.

Prostate cancer data. The same as Figure 5 except that the data set is different.

6 Conclusion

This paper studies the impact of high dimensionality on classifications. To illustrate the idea, we have considered the independence classification rule, which avoids the difficulty of estimating large covariance matrix and the diverging condition number frequently associated with the large covariance matrix. When only a subset of the features capture the characteristics of two groups, classification using all dimensions would intrinsically classify the noises. We prove that classification based on linear projections onto almost all directions performs nearly the same as random guessing. Hence, it is necessary to choose direction vectors which put more weights on important features.

The two-sample t-test can be used to choose the important features. We have shown that under mild conditions, the two sample t-test can select all the important features with probability one. The features annealed independence rule using hard thresholding, FAIR, is proposed, with the number of features selected by a data-driven rule. An upper bound of the classification error of FAIR is explicitly given. We also give suggestions on the optimal number of features used in classification. Simulation studies and real data analysis support our theoretical results convincingly.

7 Appendix

Proof of Theorem 1

For θ ∈ Γ, Ψ defined in (2.2) can be bounded as

$$\Psi \ge \frac{({\mu }_1-\hat{\mu })\prime {\hat{\mathbf{D}}}^{-1}({\hat{\mu }}_1-{\hat{\mu }}_2)}{\sqrt{b_0({\hat{\mu }}_1-{\hat{\mu }}_2)\prime {\hat{\mathbf{D}}}^{-1}\mathbf{D}{\hat{\mathbf{D}}}^{-1}({\hat{\mu }}_1-{\hat{\mu }}_2)}},$$ (7.1)

where we have used the assumption that λ_max(R) ≤ b₀. Denote by

$$\tilde{\Psi }=\frac{({\mu }_1-\hat{\mu })\prime {\hat{\mathbf{D}}}^{-1}({\hat{\mu }}_1-{\hat{\mu }}_2)}{\sqrt{({\hat{\mu }}_1-{\hat{\mu }}_2)\prime {\hat{\mathbf{D}}}^{-1}{D\hat{D}}^{-1}({\hat{\mu }}_1-{\hat{\mu }}_2)}}.$$

We next study the asymptotic behavior of $\tilde{\Psi }$.

Since Condition 1(b) in Section 3 is satisfied automatically for normal distribution, by Lemma 2 below we have $\hat{D}=\mathbf{D}(1+o_P\left(1\right))$, where o_P(1) holds uniformly across all diagonal elements. Thus, the right hand side of (7.1) can be written as

$$\frac{1}{\sqrt{b_0}}\frac{({\mu }_1-\hat{\mu })\prime {\hat{\mathbf{D}}}^{-1}({\hat{\mu }}_1-{\hat{\mu }}_2)}{\sqrt{({\hat{\mu }}_1-{\hat{\mu }}_2)\prime {\mathbf{D}}^{-1}({\hat{\mu }}_1-{\hat{\mu }}_2)}}(1+o_P\left(1\right)).$$

We first consider the denominator. Notice that it can be decomposed as

$$\begin{array}{cc}\hfill ({\hat{\mu }}_1-{\hat{\mu }}_2)\prime {\mathbf{D}}^{-1}({\hat{\mu }}_1-{\hat{\mu }}_2)& =\alpha \prime {\mathbf{D}}^{-1}\alpha +2\sum {\alpha }_j\frac{{\widehat{ϵ}}_{1j}-{\widehat{ϵ}}_{2j}}{{\sigma }_j^2}+\sum \frac{{({\widehat{ϵ}}_{1j}-{\widehat{ϵ}}_{2j})}^2}{{\sigma }_j^2}\hfill \\ \hfill & =\alpha \prime {\mathbf{D}}^{-1}\alpha +2\sum \frac{{\alpha }_j}{{\sigma }_j^2}({\widehat{ϵ}}_{1j}-{\widehat{ϵ}}_{2j})+\sum \frac{{({\widehat{ϵ}}_{1j}-{\widehat{ϵ}}_{2j})}^2}{{\sigma }_j^2}\hfill \\ \hfill & \equiv \alpha \prime {\mathbf{D}}^{-1}\alpha +2(1+o_P\left(1\right))I_1+I_2,\hfill \end{array}$$ (7.2)

where ${\sigma }_j^2$ is the j-th diagonal entry of D, ${\hat{\sigma }}_j^2$, is the j-th diagonal entry of $\hat{D}$, and ${\widehat{ϵ}}_{kj}={\sum }_{i=1}^{n_k}{ϵ}_{kij}∕n_k,k=1,2$. Notice that ${\widehat{ϵ}}_1-{\widehat{ϵ}}_2~N(0,\frac{n}{n_1{n}_2}\Sigma )$. By singular value decomposition we have

$$\mathbf{R}={\mathbf{Q}}_R{\mathbf{V}}_R{\mathbf{Q}}_R^{\prime },$$

where Q_R is orthogonal matrix and ${\mathbf{V}}_R=\mathrm{diag}\{{\lambda }_{R,1},\cdots ,{\lambda }_{R,p}\}$ be the eigenvalues of the correlation matrix R. Define $\widetilde{ϵ}=\sqrt{n_1{n}_2∕n}{\mathbf{V}}_R^{-1∕2}{\mathbf{Q}}_R^{\prime }{\mathbf{D}}^{-1∕2}({\widehat{ϵ}}_1-{\widehat{ϵ}}_2)$, then $\widetilde{ϵ}~N(0,\mathbf{I})$. Hence,

$$I_2=({\widehat{ϵ}}_1-{\widehat{ϵ}}_2)\prime {\mathbf{D}}^{-1}({\widehat{ϵ}}_1-{\widehat{ϵ}}_2)=\frac{n}{n_1{n}_2}\widetilde{ϵ}\prime {\mathbf{V}}_R\widetilde{ϵ}.$$

Since ${\sum }_{i=1}^p{\lambda }_{R,i}=p$ and ${\lambda }_{R,i}\ge 0$ for all i = 1, ⋯, p, we have $\frac{1}{p^2}{\sum }_{i=1}^p{\lambda }_{R,i}^2<\infty$. By weak law of large number we have

$$n_1{n}_2{I}_2∕\left[pn\right]\stackrel{\mathrm{P}}{\to }1\mathrm{as}n\to \infty ,p\to \infty .$$ (7.3)

Next, we consider I₁. Note that I₁ has the distribution $I_1\sim N(0,\frac{n}{n_1{n}_2}{\alpha \prime D}^{-1}{\Sigma D}^{-1}\alpha )$. Since ${\lambda }_{\max }\le b_0,n{\alpha \prime D}^{-1}\alpha \ge n{C}_p\to \infty$ and

$$\alpha \prime {\mathbf{D}}^{-1}{\Sigma D}^{-1}\alpha =\alpha \prime {\mathbf{D}}^{-1∕2}{\mathbf{RD}}^{-1∕2}\alpha \le {\lambda }_{\max }\left(R\right)\alpha \prime {\mathbf{D}}^{-1}\alpha ,$$

we have $I_1={\alpha \prime D}^{-1}\alpha o_P\left(1\right)$. This together with (7.2) and (7.3) yields

$$\frac{n_1{n}_2}{pn}({\hat{\mu }}_1-{\hat{\mu }}_2)\prime {\hat{\mathbf{D}}}^1({\hat{\mu }}_1-{\hat{\mu }}_2)=1+\frac{n_1{n}_2}{pn}\sum _{j=1}^p\frac{{\alpha }_j^2}{{\sigma }_j^2}(1+o_P\left(1\right)).$$ (7.4)

Now, we consider the numerator. It can be decomposed as

$$\begin{array}{cc}\hfill ({\mu }_1-\hat{\mu })\prime {\hat{\mathbf{D}}}^{-1}({\hat{\mu }}_1-{\hat{\mu }}_2)& =\frac{1}{2}\alpha \prime {\hat{\mathbf{D}}}^{-1}\alpha -\sum \frac{{\alpha }_j}{{\hat{\sigma }}_j^2}\left({\widehat{ϵ}}_{2j}\right)-\frac{1}{2}(1+o_P\left(1\right))\sum {\widehat{ϵ}}_{1j}^2∕{\sigma }_j^2+\frac{1}{2}(1+o_P\left(1\right))\sum {\widehat{ϵ}}_{2j}^2)∕{\sigma }_j^2\hfill \\ \hfill & \equiv \frac{1}{2}\alpha \prime {\mathbf{D}}^{-1}\alpha (1+o_P\left(1\right))-I_3-\frac{1}{2}(1+o_P\left(1\right))I_4+\frac{1}{2}(1+o_P\left(1\right))I_5.\hfill \end{array}$$

Denote by ${\tilde{I}}_3=\sum \frac{{\alpha }_j}{{\sigma }_j^2}\left({\widehat{ϵ}}_{2j}\right)$. Note that

$$\max \mid \frac{{\alpha }_j}{{\sigma }_j^2}\left({\widehat{ϵ}}_{2j}\right)-\frac{{\alpha }_j}{{\hat{\sigma }}_j^2}\left({\widehat{ϵ}}_{2j}\right)\mid \le \max \mid \frac{{\sigma }_j^2}{{\hat{\sigma }}_j^2}-1\mid \max \mid \frac{{\alpha }_j}{{\sigma }_j^2}{\widehat{ϵ}}_{2j}\mid =o_P\left(1\right)\max \mid \frac{{\alpha }_j}{{\sigma }_j^2}{\widehat{ϵ}}_{2j}\mid .$$ (7.5)

Define $F_j=\sqrt{n_2}\frac{{\alpha }_j}{{\sigma }_j^2}{\widehat{ϵ}}_{2j}∕{\alpha \prime D}^{-1}\alpha$, then ${\sigma }_{F_j}^2\equiv \mathrm{var}\left(F_j\right)\le 1$ for all j. For the normal distribution, we have the following tail probability inequality

$$1-\Phi \left(x\right)\le \frac{1}{\sqrt{2\pi }}\frac{1}{x}{e}^{-x^2∕2}.$$

Since $F_j\sim N(0,{\sigma }_{F_j}^2)$, by the above inequality we have

$$P(\mid F_j\mid \ge x)\le 2\exp \{-\frac{x^2}{2C}\}.$$

with C some positive constant, for all x > 0 and j = 1,⋯, p. By Lemma 2.2.10 of van de Vaart and Wellner (1996, P102), we have

$${(\alpha \prime {\mathbf{D}}^{-1}\alpha )}^{-1}E\max \mid \frac{{\alpha }_j}{{\sigma }_j^2}{\widehat{ϵ}}_{2j}\mid =n_2^{-1}E\underset{j\le p}{\max }\mid F_j\mid \le K\sqrt{C\log (p+1)∕n_2}\stackrel{\mathrm{P}}{\to }0,$$

where K is some universal constant. This together with (7.5) ensures that

$${(\alpha \prime {\mathbf{D}}^{-1}\alpha )}^{-1}\max \mid \frac{{\alpha }_j}{{\sigma }_j^2}\left({\widehat{ϵ}}_{2j}\right)-\frac{{\alpha }_j}{{\hat{\sigma }}_j^2}\left({\widehat{ϵ}}_{2j}\right)\mid =o_P\left(1\right)$$

Hence,

$$I_3={\tilde{I}}_3+\alpha \prime {\mathbf{D}}^{-1}\alpha o_P\left(1\right).$$ (7.6)

Now we only need to consider ${\tilde{I}}_3$. Note that ${\tilde{I}}_3=\sum \frac{{\alpha }_j}{{\sigma }_j^2}{\widehat{ϵ}}_{2j}\sim N(0,\frac{1}{n_2}{\alpha \prime D}^{-1}{\Sigma D}^{-1}\alpha )$. Since the variance term can be bounded as

$$\alpha \prime {\mathbf{D}}^{-1}{\Sigma D}^{-1}\alpha \le {\lambda }_{\max }\left(\mathbf{R}\right)\alpha \prime {\mathbf{D}}^{-1}\alpha ,$$

By the assumption that $n{\alpha \prime D}^{-1}\alpha \to \infty$ and λ_max(R) is bounded, we have ${\tilde{I}}_3=\frac{1}{2}{\alpha \prime D}^{-1}\alpha o_P\left(1\right)$. Combining this with (7.6) leads to

$$I_3=\frac{1}{2}\alpha \prime {\mathbf{D}}^{-1}\alpha o_P\left(1\right)$$

We now examine I₄ and I₅. By the similar proof to (7.3) above we have

$$I_4=p∕n_1+o_P\left(\sqrt{np∕\left(n_1{n}_2\right)}\right)\text{and}{I}_5=p∕n_2+o_P\left(\sqrt{np∕\left(n_1{n}_2\right)}\right).$$

Thus the numerator can be written as

$$({\mu }_1-\hat{\mu })\prime {\hat{\mathbf{D}}}^{-1}({\hat{\mu }}_1-{\hat{\mu }}_2)=(1+o_P\left(1\right))\frac{1}{2}\sum {\alpha }_j^2∕{\sigma }_j^2-(p∕n_1-p∕n_2)∕2+o_P\left(\sqrt{np∕\left(n_1{n}_2\right)}\right).$$

and by (7.4)

$$\tilde{\Psi }=\frac{\sqrt{\frac{n_1{n}_2}{pn}}\sum {\alpha }_j^2∕{\sigma }_j^2(1+o_P\left(1\right))+\sqrt{p∕\left(n{n}_1{n}_2\right)}(n_1-n_2)}{2{\left\{1+\frac{n_1{n}_2}{pn}\sum {\alpha }_j^2∕{\sigma }_j^2(1+o_P\left(1\right))\right\}}^{1∕2}}.$$

Since $\frac{ax}{\sqrt{1+a^{2}x}}$ is an increasing function of x and $\sum \frac{{\alpha }_j^2}{{\sigma }_j^2}\ge C_p$, in view of (7.1) and the definition of the parameter space Γ, we have

$$W\left(\hat{\delta }\right)=1-\Phi \left(\frac{{[n_1{n}_2∕\left(pn\right)]}^{1∕2}{C}_p\{1+o_P\left(1\right)+\frac{p(n_1-n_2)}{n_1{n}_2{C}_p}\}}{2\sqrt{b_0}{\{1+n_1{n}_2∕\left(pn\right)C_p(1+o_P\left(1\right))\}}^{1∕2}}\right).$$

If p/(nC_p) → 0, then $W\left(\hat{\delta }\right)=1-\Phi \left(\frac{1}{2}{[n_1{n}_2∕\left(pn{b}_0\right)]}^{1∕2}{C}_p\{1+o_P\left(1\right)\}\right)$. Furthermore, if ${\left\{\frac{n_1{n}_2}{pn}\right\}}^{1∕2}{C}_p\to C_0$ with C₀ some constant, then

$$W\left(\hat{\delta }\right)\stackrel{\mathrm{P}}{\to }1-\Phi \left(\frac{C_0}{2\sqrt{b_0}}\right);$$

This completes the proof.

Proof of Theorem 2

suppose we have a new observation X from class C₁. Then the posterior classification error of using ${\hat{\delta }}_{\mathbf{a}}(\cdot )$ is

$$\begin{array}{cc}\hfill W({\delta }_{\mathbf{a}},\theta )& =E^{\mathbf{a}}\left[P({\delta }_{\mathbf{a}}\left(\mathit{X}\right)<0\mid {\mathit{Y}}_{ki},i=1,\cdots ,n_k,k=1,2,\mathbf{a})\right]\hfill \\ \hfill & =1-E^{\mathbf{a}}\Phi \left({\Psi }_{\mathbf{a}}\mathrm{sign}(\mathbf{a}\prime {\hat{\mu }}_1-\mathbf{a}\prime {\hat{\mu }}_2)\right),\hfill \end{array}$$

Where ${\Psi }_{\mathbf{a}}=\frac{\mathbf{a}\prime {\mu }_1-\mathbf{a}\prime \hat{\mu }}{\sqrt{a\prime \Sigma a}},\Phi (\cdot )$ is the standard Gaussian distribution function, and E^a means expectation taken with respect to a. We are going to show that

$${\Psi }_{\mathbf{a}}\stackrel{\mathrm{P}}{\to }0,$$ (7.7)

which together with the continuity of ·(.) and the dominated convergence theorem gives

$$\underset{p}{\lim }{E}^{\mathbf{a}}\Phi \left({\Psi }_{\mathbf{a}}\mathrm{sign}(\mathbf{a}\prime {\hat{\mu }}_1-\mathbf{a}\prime {\hat{\mu }}_2)\right)=1∕2.$$

Therefore, the posterior error $W({\hat{\delta }}_{\mathbf{a}},\theta )$ is no better than the random guessing.

Now, let us prove (7.7). Note that the random vector a can be written as

$$\mathbf{a}=\mathbf{Z}∕\parallel \mathbf{Z}\parallel ,$$

where Z is a p-dimensional standard Gaussian distributed random vector, independent of all the observations Y _ki and X. Therefore,

$${\Psi }_{\mathbf{a}}=\frac{\mathbf{a}\prime {\mu }_1-\mathbf{a}\prime \hat{\mu }}{\sqrt{\mathbf{a}\prime \Sigma a}}=\frac{\mathbf{Z}\prime \alpha ∕\sqrt{p}-\sqrt{\frac{n}{n_1{n}_{2}p}}\mathbf{Z}\prime \left[\sqrt{\frac{n_1{n}_2}{n}}({\widehat{ϵ}}_1+{\widehat{ϵ}}_2)\right]}{2\sqrt{Z\prime \Sigma Z∕p}},$$ (7.8)

where $\alpha ={\mu }_1-{\mu }_2$ and ${\widehat{ϵ}}_k=\frac{1}{n_k}{\sum }_{i=1}^{n_k}{ϵ}_{ki},k=1,2$. By the singular value decomposition we have

$$\Sigma =Q\prime \mathrm{VQ},$$

where Q is an orthogonal matrix and $\mathbf{V}=\mathrm{diag}\{{\lambda }_1,\cdots ,{\lambda }_p\}$ is a diagonal matrix. Let $\tilde{Z}=\mathbf{QZ}$, then $\tilde{Z}$ is also a p-dimensional standard Gaussian random vector. Hence the denominator of Ψ_a can be written as

$$2\sqrt{Z\prime \Sigma Z∕p}=2{\left(\frac{1}{p}\sum _{j=1}^p{\lambda }_j{\tilde{Z}}_j^2\right)}^{1∕2},$$

where ${\tilde{Z}}_j$ is the j-th entry of $\tilde{Z}$. Since it is assumed that ${\lim }_p\frac{1}{p^2}{\sum }_{j=1}^p{\lambda }_j^2<\infty$ and ${\lim }_p\frac{1}{p}{\sum }_{j=1}^p{\lambda }_j=\tau$ for some positive constant τ, by the weak law of large numbers, we have

$$\frac{1}{p}\sum _{j=1}^G{\lambda }_j{\tilde{Z}}_j^2\stackrel{\mathrm{P}}{\to }\tau .$$ (7.9)

Next, we study the numerator of Ψ_a in (7.8). Since $\frac{1}{p}{\sum }_{j=1}^p{\alpha }_j^2\to 0$, the first term of the numerator converges to 0 in probability, i.e.,

$$Z\prime \alpha ∕\sqrt{p}\stackrel{\mathrm{P}}{\to }0.$$ (7.10)

Let $\epsilon =\sqrt{\frac{n_1{n}_2}{n}}({\widehat{ϵ}}_1+{\widehat{ϵ}}_2)$ and $\tilde{\epsilon }={\mathbf{V}}^{-1∕2}\mathbf{Q}\epsilon$, then $\tilde{\epsilon }$ has distribution N(0, I) and is independent of $\tilde{Z}$. The second term of the numerator can be written as

$$Z\prime \left[\sqrt{n_1{n}_2∕n}({\widehat{ϵ}}_1+{\widehat{ϵ}}_2)\right]={\tilde{Z}\prime V}^{1∕2}\tilde{\epsilon }=\sum _{j=1}^p\sqrt{{\lambda }_j}{\tilde{Z}}_j{\tilde{\epsilon }}_j.$$

Since $\frac{n}{n_1{n}_{2}p}{\sum }_{j=1}^p{\lambda }_j\to 0<\infty$, it follows from the weak law of large number that

$$\sqrt{\frac{n}{n_1{n}_{2}p}}\sum _{j=1}^p\sqrt{{\lambda }_j}{\tilde{Z}}_j{\tilde{\epsilon }}_j\stackrel{\mathrm{P}}{\to }0.$$

This together with (7.8), (7.9), and (7.10) completes the proof.

We need the the following two lemmas to prove Theorem 3.

Lemma 1

[Cao (2005)] Let n = n₁ + n₂. Assume that there exist 0 < c₁ ≤ c₂ < 1 such that c₁ ≤ n₁/n₂ ≤ c₂. Let ${\tilde{T}}_j=T_j-\frac{{\mu }_{j1}-{\mu }_{j2}}{\sqrt{S_{1j}^2∕n_1+S_{1j}^2∕n_2}}$. Then for any $x\equiv x(n_1,n_2)$ satisfying x → ∞ and x = o(n^1/2),

$$\log P({\tilde{T}}_j\ge x)\sim -x^2∕2,as{n}_1,n_2\to \infty .$$

If in addition, if we have only $E{\mid Y_{1ij}\mid }^3<\infty$ and $E{\mid Y_{2ij}\mid }^3<\infty$, then

$$\frac{P({\tilde{T}}_j\ge x)}{1-\Phi \left(x\right)}=1+O{\left(1\right)(1+x)}^3{n}^{-1∕2}{d}^3,for0\le x\le n^{1∕6}∕d,$$

where $d=(E{\mid Y_{1ij}\mid }^3+E{\mid Y_{2ij}\mid }^3)∕{(\mathrm{var}\left(Y_{1ij}\right)+\mathrm{var}\left(Y_{2ij}\right))}^{3∕2}$ and O(1) is a finite constant depending only on c₁ and c₂. In particular,

$$\frac{P({\tilde{T}}_j\ge x)}{1-\Phi \left(x\right)}\to 1$$

uniformly in $x\in (0,o\left(n^{1∕6}\right))$.

Lemma 2

Suppose Condition 1(b) holds and log p = o(n). Let $S_{kj}^2$ be the sample variance defined in Section 1, and ${\sigma }_{kj}^2$ be the variance of the j-th feature in class C_k. Suppose min ${\sigma }_{kj}^2$ is bounded away from 0. Then we have the following uniform convergence result

$$\underset{k=1,2,j=1,\cdots ,p}{\max }\mid S_{kj}^2-{\sigma }_{kj}^2\mid \stackrel{P}{\to }0.$$

Proof of Lemma 2

For any $\epsilon >0$, we know when n_k is very large,

$$P(\underset{k=1,2,j=1,\cdots ,p}{\max }\mid S_{kj}^2-{\sigma }_{kj}^2\mid >\epsilon )\le \sum _{k=1,2}\sum _{j=1}^{p}P(\mid S_{kj}^2-{\sigma }_{kj}^2\mid >\epsilon )\le \sum _{k=1,2}\sum _{j=1}^{s}P\left(\mid \sum _{i=1}^{n_k}({ϵ}_{kij}^2-{\sigma }_{kj}^2)\mid >n_k\epsilon ∕3\right)+\sum _{k=1,2}\sum _{j=1}^{s}P\left(\mid \sum _{i=1}^{n_k}{ϵ}_{kij}\mid >n_k\sqrt{\epsilon }∕2\right)\equiv I_1+I_2.$$ (7.11)

It follows from Bernstein's inequality that

$$P\left(\mid \sum _{i=1}^{n_k}({ϵ}_{kij}^2-{\sigma }_{kj}^2)\mid >n_k\epsilon ∕3\right)\le 2\exp \{-\frac{1}{2}\frac{n_k^2{\epsilon }^2}{9\nu 1+3M_1{n}_k\epsilon }\},$$

and

$$P\left(\mid \sum _{i=1}^{n_k}{ϵ}_{kij}\mid >n_k\sqrt{\epsilon }∕2\right)\le 2\exp \{-\frac{1}{2}\frac{n_k^2\epsilon }{4{\nu }_2+2M_2{n}_k\sqrt{\epsilon }}\},$$

Since log p = o(n), we have I₁ = o(1) and I₂ = o(1). These together with (7.11) completes the proof of Lemma 2.

Proof of Theorem 3

We devidte the proof into two parts. (a) Let us first look at the probability $P({\max }_{j>s}\mid T_j\mid >x)$. Clearly,

$$P(\underset{j>s}{\max }\mid T_j\mid >x)\le \sum _{j=s+1}^{p}P(\mid T_j\mid \ge x).$$ (7.12)

Note that for all j > s, α_j = μ_j1 - μ_j2 = 0. By Condition 1(b) and Lemma 1, the following inequality holds for 0 ≤ x ≤ n^1/6/d,

$$P(T_j\ge x)=(1-\Phi \left(x\right))(1+C{(1+x)}^3{n}^{-1∕2}{d}^3),$$

where C is a constant that only depends on c₁ and c₂, and

$$d=(E{\mid Y_{1ij}\mid }^3+E{\mid Y_{2ij}\mid }^3)∕{({\sigma }_{1j}^2+{\sigma }_{2j}^2)}^{3∕2}$$

with ${\sigma }_{kj}^2$ the j-th diagonal element of ${\Sigma }_k$. For the normal distribution, we have the following tail probability inequality

$$1-\Phi \left(x\right)\le \frac{1}{\sqrt{2\pi }}\frac{1}{x}{e}^{-x^2∕2}.$$

This together with the symmetry of T_j gives

$$P(\mid T_j\mid \ge x)\le 2\frac{1}{\sqrt{2\pi }}\frac{1}{x}{e}^{-x^2∕2}(1+C{(1+x)}^3{n}^{-1∕2}{d}^3).$$

Combining the above inequality with (7.12), we have

$$\sum _{j>s}P(\mid T_j\mid \ge x)\le (p-s)\frac{2}{\sqrt{2\pi }}\frac{1}{x}{e}^{-x^2∕2}(1+C{(1+x)}^3{n}^{-1∕2}{d}^3).$$

Since log(p - s) = o(n^γ) with $0<\gamma <\frac{1}{3}$, if we let $x\sim c{n}^{\gamma ∕2}$, then

$$\sum _{j>s}P(\mid T_j\mid \ge x)\to 0,$$

which along with (7.12) yields

$$P(\underset{j>s}{\max }\mid T_j\mid >x)\to 0.$$

(b) Next, we consider P (min_j≤s |T_j| ≤ x). Notice that for j ≤ s, α_j = μ_1j - μ_2j ≠ 0. Let ${\eta }_j=\frac{{\alpha }_j}{\sqrt{S_{1j}^2∕n_1+S_{1j}^2∕n_2}}$ and define

$${\tilde{T}}_j=T_j-{\eta }_j.$$

Then following the same lines as those in (a), we have

$$\sum _{j\le s}P(\mid {\tilde{T}}_j\mid \ge x)\le s\frac{2}{\sqrt{2\pi }}\frac{1}{x}{e}^{-x^2∕2}(1+C{(1+x)}^3{n}^{-1∕2}{d}^3)\to 0.$$

It follows from Lemma 2 that,

$$\underset{j\le s}{\max }\mid S_{kj}^2-{\sigma }_{kj}^2\mid \stackrel{\mathrm{P}}{\to }0,k=1,2.$$

Hence, uniformly over j = 1, ⋯, s, we have

$${\eta }_j=\frac{\mid {\alpha }_j\mid }{\sqrt{{\sigma }_{1j}^2∕n_1+{\sigma }_{2j}^2∕n_2}}(1+o_P\left(1\right)).$$

Therefore,

$$\underset{j\le s}{\min }\mid {\eta }_j\mid =\underset{j\le s}{\min }\frac{\sqrt{n_1}\mid {\alpha }_j\mid }{\sqrt{{\sigma }_{1j}^2+{\sigma }_{2j}^2{n}_1∕n_2}}(1+o_P\left(1\right))\ge \underset{j\le s}{\min }\frac{\sqrt{n_1}\mid {\alpha }_j\mid }{\sqrt{{\sigma }_{1j}^2+c_2{\sigma }_{2j}^2}}(1+o_P\left(1\right))$$

with c₂ defined in Theorem 3. Let ${\alpha }_0={\min }_{j\le s}\mid {\mu }_{j1}-{\mu }_{j2}\mid ∕\sqrt{{\sigma }_{1j}^2+c_2{\sigma }_{2j}^2}$. Then it follows that

$$P(\underset{j\le s}{\min }\mid T_j\mid \le x)\le P(\underset{j\le s}{\max }\mid {\tilde{T}}_j\mid \ge \underset{j\le s}{\min }\mid {\eta }_j\mid -x)\le P(\underset{j\le s}{\max }\mid {\tilde{T}}_j\mid \ge \sqrt{n_1}{\alpha }_0(1+o_P\left(1\right))-x).$$

By part (a), we know that x ~ cn^γ/2 and log(p-s) = o(n^γ). Thus if ${\alpha }_0\sim \underset{j\le s}{\min }\frac{\mid {\mu }_{j1}-{\mu }_{j2}\mid }{\sqrt{{\sigma }_{j1}^2+{\sigma }_{j2}^2}}=n^{-\gamma }{\beta }_n$ for some ${\beta }_n\to \infty$, then similarly to part (a), we have

$$P(\underset{j\le s}{\min }\mid T_j\mid \le x)\to 0.$$

Combination of part (a) and part (b) completes the proof.

Proof of Theorem 4

The classification error of the truncated classifier ${\hat{\delta }}_{\mathrm{NC}}^m$ is

$$W({\hat{\delta }}_{\mathrm{NC}}^m,\theta )=1-\Phi \left(\frac{\sum _{j=1}^m{\hat{\alpha }}_j({\mu }_{1j}-{\hat{\mu }}_j)}{\sum _{j=1}^m{\hat{\alpha }}_j^2}\right).$$

We first consider the denominator. Note that ${\hat{\alpha }}_j\sim N({\alpha }_j,\frac{n}{n_1{n}_2})$. It can be shown that

$${\left(\frac{4n}{n_1{n}_2}\sum _{j=1}^m{\alpha }_j^2+\frac{2m{n}^2}{n_1^2{n}_2^2}\right)}^{-1∕2}\sum _{j=1}^m({\hat{\alpha }}_j^2-{\alpha }_j^2-\frac{n}{n_1{n}_2})\stackrel{\mathrm{D}}{\to }N(0,1),$$

which together with the assumptions $\frac{n}{\sqrt{m}}{\sum }_{j=1}^m{\alpha }_j^2\to \infty$ gives

$$\begin{array}{cc}\hfill \sum _{j=1}^m{\hat{\alpha }}_j^2& =\sum _{j=1}^m{\alpha }_j^2+\frac{mn}{n_1{n}_2}+{\{\frac{4n}{n_1{n}_2}\sum _{j=1}^m{\alpha }_j^2+\frac{2m{n}^2}{n_1^2{n}_2^2}\}}^{1∕2}{O}_p\left(1\right)\hfill \\ \hfill & =(1+o_P\left(1\right))\sum _{j=1}^m{\alpha }_j^2+\frac{mn}{n_1{n}_2}.\hfill \end{array}$$

Next, let us look at the numerator. We decompose it as

$$\sum _{j=1}^m{\hat{\alpha }}_j({\mu }_{1j}-{\hat{\mu }}_j)=\frac{1}{2}\sum _{j=1}^m{\alpha }_j^2-\sum _{j=1}^m{\alpha }_j{\widehat{ϵ}}_{2j}-\frac{1}{2}\sum _{j=1}^m({\widehat{ϵ}}_{1j}^2-{\widehat{ϵ}}_{2j}^2).$$ (7.13)

Since the second term above has the distribution $N(0,{\sum }_{j=1}^m{\alpha }_j^2∕n_2)$, it follows from the assumption $n{\sum }_{j=1}^m{\alpha }_j^2\to \infty$ that

$$\sum _{j=1}^m{\alpha }_j{\widehat{ϵ}}_{2_j}=o_P\left(1\right)\sum _{j=1}^m{\alpha }_j^2.$$

The third term in (7.13) can be written as

$$\sum _{j=1}^m({{\widehat{ϵ}}_1^2}_j-{{\widehat{ϵ}}_2^2}_j)=\frac{m}{n_1}-\frac{m}{n_2}+O_p\left(\frac{nm}{n_1{n}_2}\right)=\frac{m(n_2-n_1)}{n_1{n}_2}=o_P\left(1\right)\sum _{j=1}^m{\alpha }_j^2.$$

Hence the numerator is

$$\sum _{j=1}^m{\hat{\alpha }}_j({\mu }_{1j}-{\hat{\mu }}_j)=\frac{m(n_2-n_1)}{n_1{n}_2}+(1+o_P\left(1\right))\sum _{j=1}^m{\alpha }_j^2.$$

Therefore, the classification error is

$$W({\hat{\delta }}_{\text{NC}}^m,\theta )=1-\Phi \left(\frac{(1+o_P\left(1\right))\sum _{j=1}^m{\alpha }_j^2+m(n_1-n_2)∕\left(n_1{n}_2\right)}{2{\{(1+o_P\left(1\right))\sum _{j=1}^m{\alpha }_j^2+mn∕\left(n_1{n}_2\right)\}}^{1∕2}}\right).$$

This concludes the proof.

Proof of Theorem 5

Note that the classification error of ${\hat{\delta }}_{\mathrm{FAIR}}^{b_n}$ is

$$W({\hat{\delta }}_{\text{FAIR}}^{b_n}\left(x\right),\theta )=1-\Phi \left(\frac{\sum _j({\mu }_{1j}-{\hat{\mu }}_j){\hat{\alpha }}_{j}1\{\mid {\hat{\alpha }}_j\mid \ge b_n\}}{\sum _j{\hat{\alpha }}_j^{2}1\{\mid {\hat{\alpha }}_j\mid \ge b_n\}}\right)\equiv 1-\Phi \left({\Psi }^H\right).$$

We divide the proof into two parts: the numerator andthe denominator.

(a) First, we study the numerator of Ψ^H. It can be decomposed as

$$\sum _j({\mu }_{1j}-{\hat{\mu }}_j){\hat{\alpha }}_{j}1\{\mid {\hat{\alpha }}_j\mid \ge b_n\}=I_1+I_2,$$

where $I_1={\sum }_{j\in \mathcal{A}}({\mu }_{1j}-{\hat{\mu }}_j){\hat{\alpha }}_{j}1\{\mid {\hat{\alpha }}_j\mid \ge b_n\}$ and $I_2={\sum }_{j\in {\mathcal{A}}^c}({\mu }_{1j}-{\hat{\mu }}_j){\hat{\alpha }}_{j}1\{\mid {\hat{\alpha }}_j\mid \ge b_n\}$ with ${\mathcal{A}}^c$ the complementary of the set $\mathcal{A}$. Note that

$$\begin{array}{cc}\hfill I_2& =\frac{1}{2}\sum _{j\in {\mathcal{A}}^c}{\alpha }_j^{2}1\{\mid {\hat{\alpha }}_j\mid \ge b_n\}-\sum _{j\in A^c}{\alpha }_j{\widehat{ϵ}}_{2j}1\{\mid {\hat{\alpha }}_j\mid \ge b_n\}-\frac{1}{2}\sum _{j\in A^c}({\widehat{ϵ}}_{1j}^2-{\widehat{ϵ}}_{2j}^2)1\{\mid {\hat{\alpha }}_j\mid \ge b_n\}\hfill \\ \hfill & \equiv \frac{1}{2}{I}_{2,1}-I_{2,2}-\frac{1}{2}{I}_{2,3}.\hfill \end{array}$$

Since ${\hat{\alpha }}_j\sim N({\alpha }_j,\frac{n}{n_1{n}_2})$, it follows from the normal tail probability inequality that for every $j\in {\mathcal{A}}^c$ and $b_n>{\max }_{j\in {\mathcal{A}}^c}\mid {\alpha }_j\mid$,

$$\begin{array}{cc}\hfill P(\mid {\hat{\alpha }}_j\mid \ge b_n)& \le P(\mid {\hat{\alpha }}_j-{\alpha }_j\mid \ge b_n-\underset{j\in {\mathcal{A}}^c}{\max }\mid {\alpha }_j\mid )\hfill \\ \hfill & \le M\frac{\exp \{-n_1{n}_2{(b_n-{\max }_{j\in A^c}\mid {\alpha }_j\mid )}^2∕\left(2n\right)\}}{\sqrt{n_1{n}_2{n}^{-1}}(b_n-{\max }_{j\in {\mathcal{A}}^c}\mid {\alpha }_j\mid )},\hfill \end{array}$$ (7.14)

where M is a generic constant. Thus for every ε > 0, if $\log (p-m)∕\left[n{(b_n-{\max }_{j\in {\mathcal{A}}^c}\mid {\alpha }_j\mid )}^2\right]\to 0$ and ${\max }_{j\in {\mathcal{A}}^c}\mid {\alpha }_j\mid <b_n$, we have

$$\begin{array}{cc}\hfill P(\mid I_{2,1}\mid \ge \epsilon )& \le {\epsilon }^{-1}\sum _{j\in A^c}{\alpha }_j^{2}P(\mid {\hat{\alpha }}_j\mid \ge b_n)\hfill \\ \hfill & \le M\underset{j\in {\mathcal{A}}^c}{\max }{\alpha }_j^2\frac{(p-m)}{\epsilon }\frac{\exp \{-n_1{n}_2{(b_n-{\max }_{j\in A^c}\mid {\alpha }_j\mid )}^2∕\left(2n\right)\}}{\sqrt{n_1{n}_2{n}^{-1}}(b_n-{\max }_{j\in {\mathcal{A}}^c}\mid {\alpha }_j\mid )},\hfill \end{array}$$

which tends to zero. Hence,

$$I_{2,1}\stackrel{P}{\to }0$$ (7.15)

We next consider I_2,2. Since $E{\left({\widehat{ϵ}}_{2j}\right)}^2=\frac{1}{n_2},\log (p-m)∕\left[n{(b_n-{\max }_{j\in {\mathcal{A}}^c}\mid {\alpha }_j\mid )}^2\right]\to 0$, and ${\max }_{j\in {\mathcal{A}}^c}\mid {\alpha }_j\mid <b_n$, we have

$$\begin{array}{cc}\hfill P(\mid I_{2,2}\mid \ge \epsilon )& \le {\epsilon }^{-1}\sum _{j\in A^c}E\mid {\widehat{ϵ}}_{2j}{\alpha }_{j}1\{\mid {\hat{\alpha }}_j\mid \ge b_n\}\mid \hfill \\ \hfill & \le {\epsilon }^{-1}\sum _{j\in A^c}{\left\{E{\left({\widehat{ϵ}}_{2j}\right)}^2\right\}}^{1∕2}{\{E{\alpha }_j^{2}1\{\mid {\hat{\alpha }}_j\mid \ge b_n\}\mid \}}^{1∕2}\hfill \\ \hfill & \le M\frac{(p-m){\max }_{j\in {\mathcal{A}}^c}\mid {\alpha }_j\mid }{\epsilon \sqrt{n_2}}\frac{\exp \{-n_1{n}_2{(b_n-{\max }_{j\in A^c}\mid {\alpha }_j\mid )}^2∕\left(4n\right)\}}{\sqrt{n_1{n}_2{n}^{-1}}(b_n-{\max }_{j\in {\mathcal{A}}^c}\mid {\alpha }_j\mid )},\hfill \end{array}$$

which converges to 0. Therefore,

$$I_{2,2}\stackrel{P}{\to }0.$$ (7.16)

Then, we consider I_2,3. Since c₁ ≤ n₁/n₂ ≤ c₂ and $E{({\widehat{ϵ}}_{1j}^2-{\widehat{ϵ}}_{2j}^2)}^2=\frac{3n_1^2+3n_2^2-2n_1{n}_2}{n_1^2{n}_2^2}\le \frac{3c_2+3-2c_1}{c_1{n}_2^2}$, by (7.14) we have for every ε > 0,

$$\begin{array}{cc}\hfill P(\mid I_{2,3}\mid \ge \epsilon )& \le {\epsilon }^{-1}E\mid \sum _{j\in A^c}({\widehat{ϵ}}_{1j}^2-{\widehat{ϵ}}_{2j}^2)1\{\mid {\hat{\alpha }}_j\mid \ge b_n\}\mid \hfill \\ \hfill & \le {\epsilon }^{-1}\sum _{j\in A^c}{\left\{E{({\widehat{ϵ}}_{1j}^2-{\widehat{ϵ}}_{2j}^2)}^2\right\}}^{1∕2}P{(\mid {\hat{\alpha }}_j\mid \ge b_n)}^{1∕2}\hfill \\ \hfill & \le M\frac{\sum _{j\in {\mathcal{A}}^c}P(\mid {\hat{\alpha }}_j\mid \ge b_n)}{n_2\epsilon }\to 0,\hfill \end{array}$$

where M is some generic constant. Thus, $I_{2,3}\stackrel{\mathrm{P}}{\to }0$. Combination of this with (7.15) and (7.16) entails

$$I_2=o_P\left(1\right).$$

We now deal with I₁. Decompose I₁ similarly as

$$\begin{array}{cc}\hfill I_1& =\sum _{j\in \mathcal{A}}({\mu }_{1j}-{\hat{\mu }}_{1j}){\hat{\alpha }}_{j}1\{\mid {\hat{\alpha }}_j\mid \ge b_n\}+\frac{1}{2}\sum _{j\in \mathcal{A}}{\hat{\alpha }}_j^{2}1\{\mid {\hat{\alpha }}_j\mid \ge b_n\}\hfill \\ \hfill & \ge \sum _{j\in \mathcal{A}}({\mu }_{1j}-{\hat{\mu }}_{1j}){\hat{\alpha }}_{j}1\{\mid {\hat{\alpha }}_j\mid \ge b_n\}+\frac{1}{2}\sum _{j\in \mathcal{A}}({\hat{\alpha }}_j^2-b_n^2)\hfill \\ \hfill & \equiv I_{1,1}+\frac{1}{2}{I}_{1,2}.\hfill \end{array}$$

We first study I_1,2. By using ${\hat{\alpha }}_j\sim N({\alpha }_j,\frac{n}{n_1{n}_2})$, it can be shown that

$${\left(\frac{4n}{n_1{n}_2}\sum _{j\in \mathcal{A}}{\alpha }_j^2+\frac{2m{n}^2}{n_1^2{n}_2^2}\right)}^{-1∕2}\sum _{j\in \mathcal{A}}({\hat{\alpha }}_j^2-{\alpha }_j^2-\frac{n}{n_1{n}_2})\stackrel{\mathrm{D}}{\to }N(0,1).$$ (7.17)

since $\frac{n}{\sqrt{m}}{\sum }_{j=1}^m{\alpha }_j^2\to \infty$, we have ${(\frac{4n}{n_1{n}_2}{\sum }_{j\in \mathcal{A}}{\alpha }_j^2+\frac{2m{n}^2}{n_1^2{n}_2^2})}^{1∕2}∕{\sum }_{j\in \mathcal{A}}{\alpha }_j^2\to 0$. Therefore,

$$\begin{array}{cc}\hfill I_{1,2}& =\sum _{j\in \mathcal{A}}({\alpha }_j^2-b_n^2)+\frac{nm}{n_1{n}_2}+{\left(\frac{4n}{n_1{n}_2}\sum _{j\in \mathcal{A}}{\alpha }_j^2+\frac{2m{n}^2}{n_1^2{n}_2^2}\right)}^{1∕2}{O}_p\left(1\right)\hfill \\ \hfill & =(1+o_P\left(1\right))\sum _{j\in \mathcal{A}}{\alpha }_j^2+\frac{nm}{n_1{n}_2}-m{b}^2.\hfill \end{array}$$

Next, we look at I_1,1. For any ε > 0,

$$P(\mid I_{1,1}\mid \ge \epsilon )\le \frac{1}{\epsilon }E\mid I_{1,1}\mid \le \frac{1}{\epsilon }\sum _{j\in \mathcal{A}}{\left\{E{\mid {\mu }_{1j}-{\hat{\mu }}_{1j}\mid }^{2}E{\mid {\hat{\alpha }}_j\mid }^2\right\}}^{1∕2}=\frac{1}{\sqrt{n_1}\epsilon }\sum _{j\in \mathcal{A}}\sqrt{{\alpha }_j^2+n∕\left(n_1{n}_2\right)}.$$

When n is large enough, the above probability can be bounded by

$$P(\mid I_{1,1}\mid \ge \epsilon )\le \sqrt{2∕\left(n_1{\epsilon }^2\right)}\sum _{j\in \mathcal{A}}\mid {\alpha }_j\mid ,$$

which along with the assumption ${\sum }_{j\in \mathcal{A}}\mid {\alpha }_j\mid ∕\left[\sqrt{n}{\sum }_{j\in \mathcal{A}}{\alpha }_j^2\right]\to 0$ gives

$$I_{1,1}=o_P\left(1\right)\sum _{j\in \mathcal{A}}{\alpha }_j^2.$$

It follows that the numerator is bounded from below by

$$(1+o_P\left(1\right))\frac{1}{2}\sum _{j\in \mathcal{A}}{\alpha }_j^2+\frac{mn}{2n_1{n}_2}-\frac{1}{2}m{b}^2.$$

(b) Now, we study the denominator of Ψ. Let

$$\sum _j{\hat{\alpha }}_j^{2}1\{\mid {\hat{\alpha }}_j\mid \ge b_n\}=\sum _{j\in \mathcal{A}}{\hat{\alpha }}_j^{2}1\{\mid {\hat{\alpha }}_j\mid \ge b_n\}+\sum _{j\in {\mathcal{A}}^c}{\hat{\alpha }}_j^{2}1\{\mid {\hat{\alpha }}_j\mid \ge b_n\}\equiv J_1+J_2.$$

We first show that $J_2\stackrel{\mathrm{P}}{\to }0$. Note that $E{\hat{\alpha }}_j^4={\alpha }_j^4+6n{\left(n_1{n}_2\right)}^{-1}{\alpha }_j^2+3n^2{\left(n_1{n}_2\right)}^{-2}$. Thus,

$$\begin{array}{cc}\hfill P(\mid J_2\mid \ge \epsilon )& \le \frac{1}{\epsilon }E\mid J_2\mid =\sum _{j\in A^c}E{\hat{\alpha }}_j^{2}1\{\mid {\hat{\alpha }}_j\mid \ge b_n\}∕\epsilon \le \frac{1}{\epsilon }\sum _{j\in A^c}{\left\{E{\hat{\alpha }}_j^{4}P(\mid {\hat{\alpha }}_j\mid \ge b_n)\right\}}^{1∕2}\hfill \\ \hfill & \le \frac{1}{\epsilon }\sum _{j\in A^c}{\left\{({\alpha }_j^4+6n{\left(n_1{n}_2\right)}^{-1}{\alpha }_j^2+3n^2{\left(n_1{n}_2\right)}^{-2})P(\mid {\hat{\alpha }}_j\mid \ge b_n)\right\}}^{1∕2}.\hfill \end{array}$$

This together with (7.14) and the assumption that $\log (p-m)∕\left[n{(b_n-{\max }_{j\in {\mathcal{A}}^c}\mid {\alpha }_j\mid )}^2\right]\to 0$ yields $J_2\stackrel{\mathrm{P}}{\to }0$ as $n\to \infty ,p\to \infty$. Now we study term J₁. By (7.17), we have

$$J_1\le \sum _{j\in \mathcal{A}}{\hat{\alpha }}_j^2=(1+o_P\left(1\right))\sum _{j\in \mathcal{A}}{\alpha }_j^2+\frac{mn}{n_1{n}_2}.$$

Hence the denominator is bounded from a above by $(1+o_P\left(1\right)){\sum }_{j\in \mathcal{A}}{\alpha }_j^2+\frac{mn}{n_1{n}_2}$. Therefore,

$${\Psi }^H\ge \frac{(1+o_P\left(1\right))\sum _{j\in \mathcal{A}}{\alpha }_j^2+\frac{mn}{n_1{n}_2}-m{b}^2}{2\sqrt{(1+o_P\left(1\right))\sum _{j\in \mathcal{A}}{\alpha }_j^2+\frac{mn}{n_1{n}_2}}}.$$

It follows that the classification error is bounded from above by

$$1-\Phi \left(\frac{(1+o_P\left(1\right))\sum _{j\in \mathcal{A}}{\alpha }_j^2+\frac{mn}{n_1{n}_2}-m{b}^2}{2\sqrt{(1+o_P\left(1\right))\sum _{j\in \mathcal{A}}{\alpha }_j^2+\frac{mn}{n_1{n}_2}}}\right).$$

This completes the proof.