QUADRO: A SUPERVISED DIMENSION REDUCTION METHOD VIA RAYLEIGH QUOTIENT OPTIMIZATION

Abstract

We propose a novel Rayleigh quotient based sparse quadratic dimension reduction method—named QUADRO (Quadratic Dimension Reduction via Rayleigh Optimization)—for analyzing high-dimensional data. Unlike in the linear setting where Rayleigh quotient optimization coincides with classification, these two problems are very different under nonlinear settings. In this paper, we clarify this difference and show that Rayleigh quotient optimization may be of independent scientific interests. One major challenge of Rayleigh quotient optimization is that the variance of quadratic statistics involves all fourth cross-moments of predictors, which are infeasible to compute for high-dimensional applications and may accumulate too many stochastic errors. This issue is resolved by considering a family of elliptical models. Moreover, for heavy-tail distributions, robust estimates of mean vectors and covariance matrices are employed to guarantee uniform convergence in estimating non-polynomially many parameters, even though only the fourth moments are assumed. Methodologically, QUADRO is based on elliptical models which allow us to formulate the Rayleigh quotient maximization as a convex optimization problem. Computationally, we propose an efficient linearized augmented Lagrangian method to solve the constrained optimization problem. Theoretically, we provide explicit rates of convergence in terms of Rayleigh quotient under both Gaussian and general elliptical models. Thorough numerical results on both synthetic and real datasets are also provided to back up our theoretical results.

1. Introduction

Rapid developments of imaging technology, microarray data studies and many other applications call for the analysis of high-dimensional binary-labeled data. We consider the problem of finding a “nice” projection f : ℝ^d → ℝ that embeds all data into the real line. A projection such as f has applications in many statistical problems for analyzing high-dimensional binary-labeled data, including:

• Dimension reduction: f provides a data reduction tool for people to visualize the high-dimensional data in a one-dimensional space.

• Classification: f can be used to construct classification rules. With a carefully chosen set A ⊂ ℝ, we can classify a new data point x ∈ ℝ^d by checking whether or not f(x) ∈ A.

• Feature selection: when f(x) only depends on a small number of coordinates of x, this projection selects just a few features from numerous observed ones.

A natural question is what kind of f is a “nice” projection? It depends on the goal of statistical analysis. For classification, a good f should yield to a small classification error. In feature selection, different criteria select distinct features, and they may suit different real problems. In this paper, we propose using the following criterion for finding f:

Under the mapping f, the data are as “separable” as possible between two classes, and as “coherent” as possible within each class.

It can be formulated as to maximize the Rayleigh quotient of f. Suppose all data are drawn independently from a joint distribution of (X, Y), where X ∈ ℝ^d, and Y ∈ {0, 1} is the label. The Rayleigh quotient of f is defined as

$$\text{Rq}(f)\equiv \frac{var\{\mathbb{E}[f(\mathbf{X})\mid Y]\}}{var\{f(\mathbf{X})-\mathbb{E}[f(\mathbf{X})\mid Y]\}}.$$ (1)

Here, the numerator is the variance of X explained by the class label, and the denominator is the remaining variance of X. Simple calculation shows that Rq(f) = π (1 − π)R(f), where π ≡ ℙ(Y = 0) and

$$R(f)\equiv \frac{{\{\mathbb{E}[f(\mathbf{X})\mid Y=0]-\mathbb{E}[f(\mathbf{X})\mid Y=1]\}}^2}{\pi var[f(\mathbf{X})\mid Y=0\text{]}+(1-\pi )var[f(\mathbf{X})\mid Y=1]}.$$ (2)

Our goal is to develop a data-driven procedure to find f̂ such that Rq(f̂) is large, and f̂ is sparse in the sense that it depends on few coordinates of X.

The Rayleigh quotient, as a criterion for finding a projection f, serves different purposes. First, for dimension reduction, it takes care of both variance explanation and label explanation. In contrast, methods such as principal component analysis (PCA) only consider variance explanation. Second, when the data are normally distributed, a monotone transform of the Rayleigh quotient approximates the classification error; see Section 6. Therefore, an f with a large Rayleigh quotient enables us to construct nice classification rules. In addition, it is a convex optimization to maximize the Rayleigh quotient among linear and quadratic f (see Section 3), while minimizing the classification error is not. Third, with appropriate regularization, this criterion provides a new feature selection tool for data analysis.

The criterion (1), initially introduced by Fisher (1936) for classification, is known as Fisher’s linear discriminant analysis (LDA). In the literature of sufficient dimension reduction, the sliced inverse regression (SIR) proposed by Li (1991) can also be formulated as maximizing (1), where Y can be any variable not necessarily binary. In both LDA and SIR, f is restricted to be a linear function, and the dimension d cannot be larger than n. In this sense, our work compares directly to various versions of LDA and SIR generalized to nonlinear, high-dimensional settings. We provide a more detailed comparison to the literature in Section 8, but preview here the uniqueness of our work. First, we consider a setting where X|Y has an elliptical distribution and f is a quadratic function, which allows us to derive a simplified version of (1) and gain extra statistical efficiency; see Section 2 for details. This simplified version of (1) was never considered before. Furthermore, the assumption of conditional elliptical distribution does not satisfy the requirement of SIR and many other dimension reduction methods [Cook and Weisberg (1991), Li (1991)]. In Section 1.2, we explain the motivation of the current setting. Second, we utilize robust estimators of mean and covariance matrix, while many generalizations of LDA and SIR are based on sample mean and sample covariance matrix. As shown in Section 4, the robust estimators adapt better to heavy tails on the data. It is worth noting that QUADRO only considers the projection to a one-dimensional subspace. In contrast, more sophisticated dimension reduction methods (e.g., the kernel SIR) are able to find multiple projections f₁, …, f_m for m > 1. This reflects a tradeoff between modeling tractability and flexibility. More specifically, QUADRO achieves better computational and theoretical properties at the cost of sacrificing some flexibility.

1.1. Rayleigh quotient and classification error

Many popular statistical methods for analyzing high-dimensional binary-labeled data are based on classification error minimization, which is closely related to the Rayleigh quotient maximization. We summarize their connections and differences as follows:

1. In an “ideal” setting where two classes follow multivariate normal distributions with a common covariance matrix and the class of linear functions f is considered, the two criteria are exactly the same, with one being a monotone transform of the other.

2. In a “relaxed” setting where two classes follow multivariate normal distributions but with nonequal covariance matrices and the class of quadratic functions f (including linear functions as special cases) is considered, the two criteria are closely related in the sense that a monotone transform of the Rayleigh quotient is an approximation of the classification error.

3. In other settings, the two criteria can be very different.

We now show (a) and (c), and will discuss (b) in Section 6.

For each f, we define a family of classifiers h_c(x) = I {f (x) < c} indexed by c, where I (·) is the indicator function. For each given c, we define the classification error of h_c to be err(h_c) ≡ ℙ(h_c(X) ≠ Y). The classification error of f is then defined by

$$\text{Err}(f)\equiv \underset{c\in ℝ}{min}\{\text{err}(h_c)\}.$$

Most existing classification procedures aim at finding a data-driven projection f̂ such that Err(f̂) is small (the threshold c is usually easy to choose). Examples include linear discriminant analysis (LDA) and its variations in high dimensions [e.g., Cai and Liu (2011), Fan and Fan (2008), Fan, Feng and Tong (2012), Guo, Hastie and Tibshirani (2005), Han, Zhao and Liu (2013), Shao et al. (2011), Witten and Tibshirani (2011)], quadratic discriminant analysis (QDA), support vector machine (SVM), logistic regression, boosting, etc.

We now compare Rq(f) and Err(f). Let π = ℙ(Y = 0), μ₁ = 𝔼(X|Y = 0), Σ₁ = cov(X|Y = 0), μ₂ = 𝔼(X|Y = 1) and Σ₂ = cov(X|Y = 1). We consider linear functions {f(x) = a^⊤x + b : a ∈ ℝ^d, b ∈ ℝ}, and write Rq(a) = Rq(a^⊤x), Err(a) = Err(a^⊤x) for short. By direct calculation, when the two classes have a common covariance matrix Σ,

$$\text{Rq}(\mathbf{a})=\pi (1-\pi )\frac{{[{\mathbf{a}}^{\top }({\mathit{\mu }}_1-{\mathit{\mu }}_2)]}^2}{{\mathbf{a}}^{\top }\mathbf{\sum }\mathbf{a}}.$$

Hence, the optimal a_R = Σ⁻¹(μ₁ − μ₂). On the other hand, when data follow multivariate normal distributions, the optimal classifier is $h^{\ast }(\mathbf{x})=I\{{\mathbf{a}}_E^{\top }\mathbf{x}<c\}$, where a_E = Σ⁻¹(μ₁ − μ₂) and $c=\frac{1}{2}{\mathit{\mu }}_1^{\top }{\mathbf{\sum }}^{-1}{\mathit{\mu }}_1-\frac{1}{2}{\mathit{\mu }}_2^{\top }{\mathbf{\sum }}^{-1}{\mathit{\mu }}_2+log(\frac{1-\pi }{\pi })$. It is observed that a_R = a_E and the two criteria are the same. In fact, for all vectors a such that a^⊤(μ₁ − μ₂) > 0,

$$\text{Err}(\mathbf{a})=1-\mathrm{\Phi }\left(\frac{1}{2}{\left[\frac{\text{Rq}(\mathbf{a})}{\pi (1-\pi )}\right]}^{1/2}\right),$$

where Φ is the distribution function of a standard normal random variable, and we fix c = a^⊤(μ₁ + μ₂)/2. Therefore, the classification error is a monotone transform of the Rayleigh quotient.

When we move away from these ideal assumptions, the above two criteria can be very different. We illustrate this point using a bivariate distribution, that is, d = 2, with different covariance matrices. Specifically, π = 0.55, μ₁ = (0, 0)^⊤, μ₂ = (1.28, 0.8)^⊤, Σ₁ = diag(1, 1) and Σ₂ = diag(3, 1/3). We still consider linear functions f(x) = a^⊤x but select only one out of the two features, X₁ or X₂. Then the maximum Rayleigh quotients, by using each of the two features alone, are 0.853 and 0.923, respectively, whereas the minimum classification errors are 0.284 and 0.295, respectively. As a result, under the criterion of maximizing Rayleigh quotient, Feature 2 is selected, whereas under the criterion of minimizing classification error, Feature 1 is selected. Figure 1 displays the distributions of data after being projected to each of the two features. It shows that since data from the second class has a much larger variability at Feature 1 than at Feature 2, the Rayleigh quotient maximization favors Feature 2, although Feature 1 yields a smaller classification error.

Fig. 1.

An example in ℝ². The green and purple represent class 1 and class 2, respectively. The ellipses are contours of distributions. Probability densities after being projected to X₁ and X₂ are also displayed. The dotted lines correspond to optimal thresholds for classification using each feature.

1.2. Objective of the paper

In this paper, we consider the Rayleigh quotient maximization problem in the following setting:

• We consider sparse quadratic functions, that is, f(x) = x^⊤Ωx − 2δ^⊤x, where Ω is a sparse d × d symmetric matrix, and δ is a sparse d-dimensional vector.

• The two classes can have different covariance matrices.

• Data from these two classes follow elliptical distributions.

• The dimension is large (it is possible that d ≫ n).

Compared to Fisher’s LDA, our setting has several new ingredients. First, we go beyond linear classifiers to enhance flexibility. It is well known that the linear classifiers are inefficient. For example, when two classes have the same mean, linear classifiers perform no better than random guesses. Instead of exploring arbitrary nonlinear functions, we consider the class of quadratic functions so that the Rayleigh quotient still has a nice parametric formulation, and at the same time it helps identify interaction effects between features. Second, we drop the requirement that the two classes share a common covariance matrix, which is a critical condition for Fisher’s rule and many other high-dimensional classification methods [e.g., Cai and Liu (2011), Fan and Fan (2008), Fan, Feng and Tong (2012)]. In fact, by using quadratic discriminant functions, we take advantage of the difference of covariance matrices between the two classes to enhance classification power. Third, we generalize multivariate normal distributions to the elliptical family, which includes many heavy-tailed distributions, such as multivariate t-distributions, Laplace distributions, and Cauchy distributions. This family of distributions allows us to avoid estimating all O(d⁴) fourth cross-moments of d predictors in computing the variance of quadratic statistics and hence overcomes the computation and noise accumulation issues.

In our setting, Fisher’s rule, that is, a_R = Σ⁻¹(μ₁ − μ₂), no longer maximizes the Rayleigh quotient. We propose a new method, called quadratic dimension reduction via Rayleigh optimization (QUADRO). It is a Rayleigh-quotient-oriented procedure and is a statistical tool for simultaneous dimension reduction and feature selection. QUADRO has several properties. First, it is a statistically efficient generalization of Fisher’s linear discriminant analysis to the quadratic setting. A naive generalization involves estimation of all fourth cross-moments of the two underlying distributions. In contrast, QUADRO only requires estimating a one-dimensional kurtosis parameter. Second, QUADRO adopts rank-based estimators and robust M-estimators of the covariance matrices and the means. Therefore, it is robust to possibly heavy-tail distributions. Third, QUADRO can be formulated as a convex programming and is computationally efficient.

Theoretically, we prove that under elliptical models, the Rayleigh quotient of the estimated quadratic function f̂ converges to population maximum Rayleigh quotient at rate $O_p(s\sqrt{log(d)/n})$, where s is the number of important features (counting both single terms and interaction terms). In addition, we establish a connection between our method and quadratic discriminant analysis (QDA) under elliptical models.

The rest of this paper is organized as follows. Section 2 formulates Rayleigh quotient maximization as a convex optimization problem. Section 3 describes QUADRO. Section 4 discusses rank-based estimators and robust M-estimators used in QUADRO. Section 5 presents theoretical analysis. Section 6 discusses the application of QUADRO in elliptically distributed classification problems. Section 7 contains numerical studies. Section 8 concludes the paper. All proofs are collected in Section 9.

Notation

For 0 ≤ q ≤ ∞, |v|_q denotes the L_q-norm of a vector v, |A|_q denotes the elementwise L_q-norm of a matrix A and ||A||_q denotes the matrix L_q-norm of A. When q = 2, we omit the subscript q. λ_min(A) and λ_max(A) denote the minimum and maximum eigenvalues of A. det(A) denotes the determinant of A. Let I (·) be the indicator function: for any event B, I (B) = 1 if B happens and I (B) = 0 otherwise. Let sign(·) be the sign function, where sign(u) = 1 when u ≥ 0 and sign(u) = −1 when u < 0.

2. Rayleigh quotient for quadratic functions

We first study the population form of Rayleigh quotient for an arbitrary quadratic function. We show that it has a simplified form under the elliptical family.

For a quadratic function

$$Q(\mathbf{X})={\mathbf{X}}^{\top }\mathbf{\Omega }\mathbf{X}-2{\mathit{\delta }}^{\top }\mathbf{X},$$

using (2), its Rayleigh quotient is

$$R(\mathbf{\Omega },\mathit{\delta })=\frac{{\{\mathbb{E}[Q(\mathbf{X})\mid Y=0]-\mathbb{E}[Q(\mathbf{X})\mid Y=1]\}}^2}{\pi var[Q(\mathbf{X})\mid Y=0]+(1-\pi )var[Q(\mathbf{X})\mid Y=1]}$$ (3)

up to a constant multiplier. The Rayleigh quotient maximization can be expressed as

$$\underset{(\mathbf{\Omega },\mathit{\delta }):\mathbf{\Omega }={\mathbf{\Omega }}^{\top }}{max}R(\mathbf{\Omega },\mathit{\delta }).$$

2.1. General setting

Suppose 𝔼(Z) = μ and cov(Z) = Σ. By direct calculation,

$$\begin{array}{ll}\hfill \mathbb{E}[Q(\mathbf{Z})]& =\text{tr}(\mathbf{\Omega }\mathbf{\sum })+{\mathit{\mu }}^{\top }\mathbf{\Omega }\mathit{\mu }-2{\mathit{\delta }}^{\top }\mathit{\mu },\\ \hfill var[Q(\mathbf{Z})]& =\mathbb{E}[\text{tr}(\mathbf{\Omega }{\mathbf{ZZ}}^{\top }\mathbf{\Omega }{\mathbf{ZZ}}^{\top })]-4\mathbb{E}[{\mathit{\delta }}^{\top }\mathbf{Z}{\mathbf{Z}}^{\top }\mathbf{\Omega }\mathbf{Z}]+4{\mathit{\delta }}^{\top }\mathbf{\sum }\mathit{\delta }+4{({\mathit{\delta }}^{\top }\mathit{\mu })}^2-{\{\mathbb{E}[Q(\mathbf{Z})]\}}^2.\end{array}$$

So 𝔼[Q(Z)] is a linear combination of the elements in {Ω(i, j), 1 ≤ i ≤ j ≤ d; δ(i), 1 ≤ i ≤ d}, and var[Q(Z)] is a quadratic form of these elements. The coefficients in 𝔼[Q(Z)] are functions of μ and Σ only. However, the coefficients in var[Q(Z)] also depend on all the fourth cross-moments of Z, and there are O(d⁴) of them.

Let us define M₁(Ω, δ) = 𝔼[Q(X)|Y = 0], L₁(Ω, δ) = var[Q(X)|Y = 0] and M₂(Ω, δ), L₂(Ω, δ) similarly. Also, let κ = (1 − π)/π. We have

$$R(\mathbf{\Omega },\mathit{\delta })=\frac{{[M_1(\mathbf{\Omega },\mathit{\delta })-M_2(\mathbf{\Omega },\mathit{\delta })]}^2}{L_1(\mathbf{\Omega },\mathit{\delta })+\kappa L_2(\mathbf{\Omega },\mathit{\delta })}.$$

Therefore, both the numerator and denominator are quadratic combinations of the elements in Ω and δ. We can stack the d(d + 1)/2 elements in Ω (assuming it is symmetric) and the d elements in δ into a long vector v. Then R(Ω, δ) can be written as

$$R(\mathbf{v})=\frac{{({\mathbf{a}}^{\top }\mathbf{v})}^2}{{\mathbf{v}}^{\top }\mathbf{Av}},$$

where a is a d′ × 1 vector, A is a d′ × d′ positive semi-definite matrix and d′ = d(d + 1)/2 + d. A and a are determined by the coefficients in the denominator and numerator of R(Ω, δ), respectively. Now, max_{(Ω, δ)} R(Ω, δ) is equivalent to max_v R(v). It has explicit solutions. For example, when A is positive definite, the function R(v) is maximized at v^* = A⁻¹a. We can then reshape v^* to get the desired (Ω^*, δ^*).

Practical implementation of the above idea is infeasible in high dimensions as it involves O(d⁴) cross moments of Z. This not only poses computational challenges, but also accumulates noise in the estimation. Furthermore, good estimates of fourth moments usually require the existence of eighth moments, which is not realistic for many heavy tailed distributions. These problems can be avoided under the elliptical family, as we now illustrate in the next subsection.

2.2. Elliptical distributions

The elliptical family contains multivariate distributions whose densities have elliptical contours. It generalizes multivariate normal distributions and inherits many of their nice properties.

Given a d × 1 vector μ and a d × d positive definite matrix Σ, a random vector Z that follows an elliptical distribution admits

$$\mathbf{Z}=\mathit{\mu }+\xi {\mathbf{\sum }}^{1/2}\mathbf{U},$$ (4)

where U is a random vector which follows the uniform distribution on unit sphere S^d−1, and ξ is a nonnegative random variable independent of U. Denote the elliptical distribution by ℰ(μ, Σ, g), where g is the density of ξ. In this paper, we always assume that ℰξ⁴ < ∞ and require that ℰ (ξ²) = d for the model identifiability. Then Σ is the covariance matrix of Z.

Proposition 2.1

Suppose Z follows an elliptical distribution as in (4). Then

$$\begin{array}{ll}\hfill \mathbb{E}[Q(\mathbf{Z})]& =\text{tr}(\mathbf{\Omega }\mathbf{\sum })+{\mathit{\mu }}^{\top }\mathbf{\Omega }\mathit{\mu }-2{\mathit{\mu }}^{\top }\mathit{\delta },\\ \hfill var[Q(\mathbf{Z})]& =2(1+\gamma )\text{tr}(\mathbf{\Omega }\mathbf{\sum }\mathbf{\Omega }\mathbf{\sum })+\gamma {[\text{tr}(\mathbf{\Omega }\mathbf{\sum })]}^2+4{(\mathbf{\Omega }\mathit{\mu }-\mathit{\delta })}^{\top }\mathbf{\sum }(\mathbf{\Omega }\mathit{\mu }-\mathit{\delta }),\end{array}$$

where $\gamma =\frac{E({\xi }^4)}{d(d+2)}-1$ is the kurtosis parameter.

The proof is given in the online supplementary material [Fan et al. (2014)]. The variance of Q(Z) does not involve any fourth cross-moments, but only the kurtosis parameter γ. For multivariate normal distributions, ξ² follows a χ²-distribution with d degrees of freedom, and γ = 0. For multivariate t-distribution with degrees of freedom ν > 4, we have γ = 2/(ν − 4).

2.3. Rayleigh optimization

We assume that the two classes both follow elliptical distributions: X|(Y = 0) ~ ℰ(μ₁, Σ₁, g₁) and X|(Y = 1) ~ ℰ (μ₂, Σ₂, g₂). To facilitate the presentation, we assume the quantity γ is the same for both classes of conditional distributions. Let

$$\begin{array}{ll}\hfill M(\mathbf{\Omega },\mathit{\delta })& =-{\mathit{\mu }}_1^{\top }\mathbf{\Omega }{\mathit{\mu }}_1+{\mathit{\mu }}_2^{\top }\mathbf{\Omega }{\mathit{\mu }}_2+2{({\mathit{\mu }}_1-{\mathit{\mu }}_2)}^{\top }\mathit{\delta }-\text{tr}(\mathbf{\Omega }({\mathbf{\sum }}_1-{\mathbf{\sum }}_2)),\\ \hfill L_k(\mathbf{\Omega },\mathit{\delta })& =2(1+\gamma )\text{tr}(\mathbf{\Omega }{\mathbf{\sum }}_k\mathbf{\Omega }{\mathbf{\sum }}_k)+\gamma {[\text{tr}(\mathbf{\Omega }{\mathbf{\sum }}_k)]}^2+4{(\mathbf{\Omega }{\mathit{\mu }}_k-\mathit{\delta })}^{\top }{\mathbf{\sum }}_k(\mathbf{\Omega }{\mathit{\mu }}_k-\mathit{\delta }),\end{array}$$ (5)

for k = 1 and 2. Combining (3) with Proposition 2.1, we have

$$R(\mathbf{\Omega },\mathit{\delta })=\frac{{[M(\mathbf{\Omega },\mathit{\delta })]}^2}{L_1(\mathbf{\Omega },\mathit{\delta })+\kappa L_2(\mathbf{\Omega },\mathit{\delta })},$$ (6)

where κ = (1 − π)/π.

Note that if we multiply both Ω and δ by a common constant, R(Ω, δ) remains unchanged. Therefore, maximizing R(Ω, δ) is equivalent to solving the following constrained minimization problem:

$$\underset{(\mathbf{\Omega },\mathit{\delta }):M(\mathbf{\Omega },\mathit{\delta })=1,\mathbf{\Omega }={\mathbf{\Omega }}^{\top }}{min}\{L_1(\mathbf{\Omega },\mathit{\delta })+\kappa L_2(\mathbf{\Omega },\mathit{\delta })\}.$$ (7)

We call problem (7) the Rayleigh optimization. It is a convex problem whenever Σ₁ and Σ₂ are both positive semi-definite.

The formulation of the Rayleigh optimization only involves the means and covariance matrices, and the kurtosis parameter γ. Therefore, if we know γ (e.g., when we know which subfamily the distributions belong to) and have good estimates (μ̂₁, μ̂₂, Σ̂₁, Σ̂₂), we can solve the empirical version of (7) to obtain (Ω̂, δ̂), which is the main idea of QUADRO. In addition, (7) is a convex problem, with a quadratic objective and equality constraints. Hence it can be solved efficiently by many optimization algorithms.

3. Quadratic dimension reduction via Rayleigh optimization

Now, we formally introduce the QUADRO procedure. We fix a model parameter γ ≥ 0. Let M̂, L̂₁ and L̂₂ be the sample versions of M, L₁, L₂ in (5) by replacing (μ₁, μ₂, Σ₁, Σ₂) with their estimates. Details of these estimates will be given in Section 4. Let π̂ = n₁/(n₁ + n₂) and κ = π̂/(1 − π̂). Given tuning parameters λ₁ > 0 and λ₂ > 0, we solve

$$\underset{(\mathbf{\Omega },\mathit{\delta }):\widehat{M}(\mathbf{\Omega },\mathit{\delta })=1,\mathbf{\Omega }={\mathbf{\Omega }}^{\top }}{min}\{{\widehat{L}}_1(\mathbf{\Omega },\mathit{\delta })+\kappa {\widehat{L}}_2(\mathbf{\Omega },\mathit{\delta })+{\lambda }_1{\mid \mathbf{\Omega }\mid }_1+{\lambda }_2{\mid \mathit{\delta }\mid }_1\}.$$ (8)

We propose a linearized augmented Lagrangian method to solve (8). To simplify the notation, we write L̂ = L̂₁ + κL̂₂, and omit the hat symbol on M and L when there is no confusion. The optimization problem is then

$$\underset{(\mathbf{\Omega },\mathit{\delta }):M(\mathbf{\Omega },\mathit{\delta })=1,\mathbf{\Omega }={\mathbf{\Omega }}^{\top }}{min}\{L(\mathbf{\Omega },\mathit{\delta })+{\lambda }_1{\mid \mathbf{\Omega }\mid }_1+{\lambda }_2{\mid \mathit{\delta }\mid }_1\}.$$

For an algorithm parameter ρ > 0, and a dual variable ν, we define the augmented Lagrangian as

$$F_{\rho }(\mathbf{\Omega },\mathit{\delta },\nu )=L(\mathbf{\Omega },\mathit{\delta })+\nu [M(\mathbf{\Omega },\mathit{\delta })-1]+(\rho /2){[M(\mathbf{\Omega },\mathit{\delta })-1]}^2.$$

Using zero as the initial value, we iteratively update:

• δ^(k) = argmin_δ{F_ρ(Ω^(k−1), δ, ν^(k−1)) + λ₂|δ|₁},

• Ω^(k) = argmin_{Ω: Ω= Ω^⊤}{F_ρ (Ω, δ^(k), ν^(k−1)) + λ₁|Ω|₁},

• ν^(k) = ν^(k−1) + ρ[M(Ω^(k), δ^(k)) − 1].

Here, the first two steps are primal updates, and the third step is a dual update.

First, we consider the update of δ. When Ω and ν are fixed, we can write

$$F_{\rho }(\mathbf{\Omega },\mathit{\delta },\nu )={\mathit{\delta }}^{\top }\mathbf{A}\mathit{\delta }-2{\mathit{\delta }}^{\top }\mathbf{b}+c_{\rho }(\mathbf{\Omega },\nu ),$$

where

$$\begin{array}{ll}\hfill \mathbf{A}& =4({\mathbf{\sum }}_1+\kappa {\mathbf{\sum }}_2)+2\rho ({\mathit{\mu }}_1-{\mathit{\mu }}_2){({\mathit{\mu }}_1-{\mathit{\mu }}_2)}^{\top },\\ \hfill \mathbf{b}& =4({\mathbf{\sum }}_1\mathbf{\Omega }{\mathit{\mu }}_1+\kappa {\mathbf{\sum }}_2\mathbf{\Omega }{\mathit{\mu }}_2)+[\rho \text{tr}(\mathbf{\Omega }({\mathbf{\sum }}_1-{\mathbf{\sum }}_2))+\rho {\mathit{\mu }}_1^{\top }\mathbf{\Omega }{\mathit{\mu }}_1-\rho {\mathit{\mu }}_2^{\top }\mathbf{\Omega }{\mathit{\mu }}_2+(\rho -\nu )]({\mathit{\mu }}_1-{\mathit{\mu }}_2),\end{array}$$ (9)

and c_ρ (Ω, ν) does not depend on δ. Note that A is a positive semi-definite matrix. The update of δ is indeed a Lasso problem.

Next, we consider the update of Ω. When δ and ν are fixed, F_ρ(Ω, δ, ν) is a convex function of Ω. We propose an approximate update step: we first “linearize” F_ρ at Ω = Ω ^(k−1) to construct an upper envelope F̄_ρ, and then minimize this upper envelope. In detail, at any Ω = Ω₀, we consider the following upper bound of F_ρ (Ω, δ, ν):

$${\overline{F}}_{\rho }(\mathbf{\Omega },\mathit{\delta },\nu )\equiv F_{\rho }({\mathbf{\Omega }}_0,\mathit{\delta },\nu )+\sum _{1\le i\le j\le d}[\mathrm{\Omega }(i,j)-{\mathrm{\Omega }}_0(i,j)]\frac{\partial F_{\rho }({\mathbf{\Omega }}_0,\mathit{\delta },\nu )}{\partial \mathrm{\Omega }(i,j)}+\frac{\tau }{2}\sum _{1\le i\le j\le d}{[\mathrm{\Omega }(i,j)-{\mathrm{\Omega }}_0(i,j)]}^2,$$

where τ is a large enough constant [e.g., we can take $\tau ={\sum }_{1\le i\le j\le d}\frac{{\partial }^2{F}_{\rho }({\mathbf{\Omega }}_0,\mathit{\delta },\nu )}{\partial \mathrm{\Omega }{(i,j)}^2}$. We then minimize F̄_ρ(Ω, δ, ν) + λ₁|Ω|₁ to update Ω. This modified update step has an explicit solution,

$${\mathrm{\Omega }}^{\ast }(i,j)=\mathcal{S}\left({\mathrm{\Omega }}_0(i,j)-\frac{1}{\tau }\frac{\partial F_{\rho }({\mathbf{\Omega }}_0,\mathit{\delta },\nu )}{\partial \mathrm{\Omega }(i,j)},\frac{{\lambda }_1}{\tau }\right),$$

where 𝒮 (x, a) ≡ (|x| − a)₊ sign(x) is the soft-thresholding function. We can write Ω^* in a matrix form. Let

$$\mathbf{D}=4(1+\gamma )({\mathbf{\sum }}_1\mathbf{\Omega }{\mathbf{\sum }}_1+\kappa {\mathbf{\sum }}_2\mathbf{\Omega }{\mathbf{\sum }}_2)+2\gamma [\text{tr}(\mathbf{\Omega }{\mathbf{\sum }}_1){\mathbf{\sum }}_1+\kappa \text{tr}(\mathbf{\Omega }{\mathbf{\sum }}_2){\mathbf{\sum }}_2]+4\text{sym}({\mathbf{\sum }}_1(\mathbf{\Omega }{\mathit{\mu }}_1-\mathit{\delta }){\mathit{\mu }}_1^{\top }+\kappa {\mathbf{\sum }}_2(\mathbf{\Omega }{\mathit{\mu }}_2-\mathit{\delta }){\mathit{\mu }}_2^{\top }),$$ (10)

where sym(B) = (B + B^⊤)/2 for any square matrix B. By direct calculation,

$${\mathbf{\Omega }}^{\ast }=\mathcal{S}\left({\mathbf{\Omega }}_0-\frac{1}{\tau }\mathbf{D},\frac{{\lambda }_1}{\tau }\right).$$

We now describe our algorithm. Let us initialize Ω⁽⁰⁾ = 0_d×d, δ⁽⁰⁾ = 0 and ν⁽⁰⁾ = 0. At iteration k, the algorithm updates as follows:

• Compute A = A(Ω^(k−1), δ^(k−1), ν^(k−1)) and b = b(Ω^(k−1), δ^(k−1), ν^(k−1)) using (9). Update δ^(k) = argmin_δ{δ^⊤Aδ − 2δ^⊤b + λ₂|δ|₁}.

• Compute D = D(Ω^(k−1), δ^(k−1), ν^(k−1)) using (10). Update ${\mathbf{\Omega }}^{(k)}=\mathcal{S}({\mathbf{\Omega }}^{(k-1)}-\frac{1}{\tau }\mathbf{D},\frac{{\lambda }_1}{\tau })$.

• Update ν^(k) = ν^(k−1) + ρ[M(Ω^(k), δ^(k)) − 1].

Stop until max{ρ|Ω^(k) − Ω^(k−1)|, ρ|δ^(k) − δ^(k−1)|, |ν^(k) − ν^(k−1)|/ρ} ≤ ε for some pre-specified precision ε.

This is a modified version of the augmented Lagrangian method, where in the step of updating Ω, we minimize an upper envelope, which is obtained by locally linearizing the augmented Lagrangian.

Remark

QUADRO can be extended to folded concave penalties, for example, to SCAD [Fan and Li (2001)] or to adaptive Lasso [Zou (2006)]. Using the Local Linear Approximation algorithm [Fan, Xue and Zou (2014), Zou and Li (2008)], we can solve the SCAD-penalized QUADRO and the adaptive-Lasso-penalized QUADRO by solving L₁-penalized QUADRO with multiple-step and one-step iterations, respectively.

4. Estimation of mean and covariance matrix

QUADRO requires estimates of the mean vector and covariance matrix for each class as inputs. We will show in Section 5 that the performance of QUADRO is closely related to the maxnorm estimation error on mean vectors and covariance matrices. Sample mean and sample covariance matrix work well for Gaussian data. However, when data are from elliptical distributions, they may have inferior performance as we estimate nonpolynomially many of means and variances. In Sections 4.1–4.2, we suggest a robust M-estimator to estimate the mean and a rank-based estimator to estimate the covariance matrix, which are more appropriate for non-Gaussian data. Moreover, in Section 4.3 we discuss how to estimate the model parameter γ when it is unknown.

4.1. Estimation of the mean

Suppose x₁,…, x_n are i.i.d. samples of a random vector X = (X₁,…, X_d)^⊤ from an elliptical distribution ℰ(μ,Σ, g). Let us denote μ = (μ₁,…, μ_d)^⊤ and x_i = (x_i1,…, x_id) ^⊤ for i = 1,…, n. We estimate each μ_j marginally using the data {x_1j,…, x_nj}.

One possible estimator is the sample median

$${\widehat{\mu }}_{Mj}=\text{median}(\{x_{1j},\dots ,x_{nj}\}).$$

It can be shown that even under heavy-tailed distributions, $P(\mid {\widehat{\mu }}_{Mj}-{\mu }_j\mid >A\sqrt{log({\delta }^{-1})/n})\le \delta$ for small δ ∈ (0, 1), where A is a constant determined by the probability density at μ_j, for each fixed j. This combined with the union bound gives that ${\mid {\widehat{\mathit{\mu }}}_M-\mathit{\mu }\mid }_{\infty }=O_p(\sqrt{log(d)/n})$.

Catoni (2012) proposed another M-estimator for the mean of heavy-tailed distributions. It works for distributions where mean is not necessarily equal to median, which is essential for estimating covariance of random variables. We denote the diagonal elements of the covariance matrix Σ as ${\sigma }_1^2,{\sigma }_2^2,\dots ,{\sigma }_d^2$, and the off-diagonal elements as σ_kj for k ≠ j. The estimator μ̂_C = (μ̂_C,1,…, μ̂_C,d) ^⊤ is obtained as follows. For a strictly increasing function h:ℝ → ℝ such that −log(1 − y + y²/2) ≤ h(y) ≤ log(1 + y + y²/2), and a value δ ∈ (0, 1) such that n>2 log(1/δ), we let

$${\alpha }_{\delta }={\left\{\frac{2log({\delta }^{-1})}{n[v+(2vlog({\delta }^{-1}))/(n-2log({\delta }^{-1}))]}\right\}}^{1/2},$$

where v is an upper bound of $max\{{\sigma }_1^2,\dots ,{\sigma }_d^2\}$. For each j, we define μ̂_Cj as the unique value that satisfies ${\sum }_{i=1}^{n}h({\alpha }_{\delta }(x_{ij}-{\widehat{\mu }}_{Cj}))=0$. It was shown in Catoni (2012) that $P(\mid {\widehat{\mu }}_{Cj}-{\mu }_j\mid >\sqrt{\frac{2vlog({\delta }^{-1})}{n(1-2log({\delta }^{-1})/n)}})\le \delta$ when the variance of X_j exists. Therefore, by taking δ = 1/(n ∨ d)², ${\mid {\widehat{\mathit{\mu }}}_M-\mathit{\mu }\mid }_{\infty }\le C\sqrt{log(d)/n}$ with probability at least 1 − (n ∨ d)⁻¹, which gives the desired convergence rate.

To implement this estimator, we take h(y) = sgn(y) log(1+|y|+y²/2). For the choice of v, any value larger than $max\{{\sigma }_1^2,\dots ,{\sigma }_d^2\}$ would work in theory. Catoni (2012) introduced a Lepski’s adaptation method to choose v. For simplicity, we take $v=3max\{{\stackrel{\sim }{\sigma }}_1^2,\dots ,{\stackrel{\sim }{\sigma }}_d^2\}$, where ${\stackrel{\sim }{\sigma }}_j^2$ is the sample covariance of X_j.

The two estimators, the median and the M-estimator, both have a convergence rate of $O_p(\sqrt{log(d)/n})$ in terms of the max-norm error. In our numerical experiments, the M-estimator has a better numerical performance, and we stick to this estimator.

4.2. Estimation of the covariance matrix

To estimate the covariance matrix Σ, we estimate the marginal covariances { ${\sigma }_j^2$, 1 ≤ j ≤ d} and the correlation matrix C separately. Again, we need robust estimates even though the data have fourth moments, as we simultaneously estimate nonpolynomial number of covariance parameters.

First, we consider estimating ${\sigma }_j^2$. Note that ${\sigma }_j^2=\mathbb{E}(X_j^2)-{\mathbb{E}}^2(X_j)$. We estimate $\mathbb{E}(X_j^2)$ and 𝔼(X_j) separately. To estimate $\mathbb{E}(X_j^2)$, we use the M-estimator described above on the squared data { $x_{1j}^2,\dots ,x_{nj}^2$} and denote the estimator by η̂;_Cj. This works as $\mathbb{E}(X_j^4)$ is finite for each j in our setting; in addition, the M-estimator applies to asymmetric distributions. We then define

$${\widehat{\sigma }}_{Cj}^2=max\{{\widehat{\eta }}_{Cj}-{\widehat{\mu }}_{Cj}^2,{\delta }_0\},$$

where μ̂_Cj is the M-estimator of 𝔼(X_j) and δ₀ > 0 is a small constant ( ${\delta }_0<min\{{\sigma }_1^2,\dots ,{\sigma }_d^2\}$). It is easy to see that when the fourth moments of X_j are uniformly upper bounded by a constant and n ≥ 4 log(d²), $max\{\mid {\widehat{\sigma }}_{Cj}-{\sigma }_j\mid ,1\le j\le d\}=O_p(\sqrt{log(d)/n})$.

Next, we consider estimating the correlation matrix C. For this, we use Kendall’s tau correlation matrix proposed by Han and Liu (2012). Kendall’s tau correlation coefficients [Kendall (1938)] are defined as

$${\tau }_{jk}=\mathbb{P}((X_j-{\stackrel{\sim }{X}}_j)(X_k-{\stackrel{\sim }{X}}_k)>0)-\mathbb{P}((X_j-{\stackrel{\sim }{X}}_j)(X_k-{\stackrel{\sim }{X}}_k)<0),$$

where X̃ is an independent copy of X. They have the following relationship to the true coefficients: $C_{jk}=sin(\frac{\pi }{2}{\tau }_{jk})$ for the elliptical family. Based on this equality, we first estimate Kendall’s tau correlation coefficients using rank-based estimators

$${\widehat{\tau }}_{jk}=\{\begin{array}{ll}\frac{2}{n(n-1)}\sum _{1\le i<i^{\prime }\le n}\text{sign}\left((x_{ij}-x_{i^{\prime }j})(x_{ik}-x_{i^{\prime }k})\right)\hfill & j\ne k,\hfill \\ 1,\hfill & j=k,\hfill \end{array}$$

and then estimate the correlation matrix by Ĉ = (Ĉ_jk) with

$${\widehat{C}}_{jk}=sin\left(\frac{\pi }{2}{\widehat{\tau }}_{jk}\right).$$

It is shown in Han and Liu (2012) that ${\mid \widehat{\mathbf{C}}-\mathbf{C}\mid }_{\infty }=O_p(\sqrt{log(d)/n})$.

Finally, we combine { ${\widehat{\sigma }}_j^2$, 1 ≤ j ≤ d} and Ĉ to get Σ̂. Let

$${\mathrm{\sum }^{\sim }}_{jk}={\widehat{\sigma }}_j{\widehat{\sigma }}_k{\widehat{C}}_{jk},1\le j,k\le d$$

It follows immediately that ${\mid \mathbf{\sum }^{\sim }-\mathbf{\sum }\mid }_{\infty }=O_p(\sqrt{log(d)/n})$. However, this estimator is not necessarily positive semi-definite. To implement QUADRO, we need Σ̂ to be positive semi-definite so that the optimization in (8) is a convex problem. We obtain Σ̂ by projecting Σ̃ onto the cone of positive semi-definite matrices through the convex optimization

$$\widehat{\mathbf{\sum }}=\underset{\mathbf{A}:\mathbf{A}\text{is}\text{positive}\text{semidefinite}}{\text{argmin}}\{{\mid \mathbf{A}-\mathbf{\sum }^{\sim }\mid }_{\infty }\}.$$ (11)

Note that |Σ̂ − Σ̃|_∞ ≤ |Σ − Σ̃|_∞ by definition. Therefore, ${\mid \widehat{\mathbf{\sum }}-\mathbf{\sum }\mid }_{\infty }\le {\mid \widehat{\mathbf{\sum }}-\mathbf{\sum }^{\sim }\mid }_{\infty }+{\mid \mathbf{\sum }^{\sim }-\mathbf{\sum }\mid }_{\infty }\le 2{\mid \mathbf{\sum }^{\sim }-\mathbf{\sum }\mid }_{\infty }=O_p(\sqrt{log(d)/n})$. To compute Σ̂, we note that the optimization problem in (11) can be formulated as the dual of a graphical lasso problem corresponding to the smallest possible tuning parameter that still guarantees a feasible solution [Liu et al. (2012)]. Zhao, Roeder and Liu (2013) provide more algorithmic details.

4.3. Estimation of kurtosis parameter

When the kurtosis parameter γ is unknown, we can estimate it from data. Recall that $\gamma =\frac{1}{d(d+2)}\mathbb{E}({\xi }^4)-1$. Using decomposition (4) and the properties of U, we have

$$\mathbb{E}({\xi }^4)=\mathbb{E}\{{[{(\mathbf{X}-\mathit{\mu })}^{\top }{\mathbf{\sum }}^{-1}(\mathbf{X}-\mathit{\mu })]}^2\}.$$

Motivated by this equality, we propose the estimator

$$\widehat{\gamma }=max\left\{\frac{1}{d(d+2)}\frac{1}{n}\sum _{i=1}^n{[{({\mathbf{x}}_i-\stackrel{\sim }{\mathit{\mu }})}^{\top }\stackrel{\sim }{\mathbf{\Omega }}({\mathbf{x}}_i-\stackrel{\sim }{\mathit{\mu }})]}^2-1,0\right\},$$

where μ̃ and Ω̃ are estimators of μ and Σ⁻¹, respectively. Maruyama and Seo (2003) considered a similar estimator in low-dimensional settings, where they used the sample mean and sample covariance matrix. In high dimensions, we a robust estimate to guarantee uniform convergence. In particular, we take μ̃ = μ̂_C and Ω̃ = Ω̂_clime where Ω̂_clime is the CLIME estimator proposed in Cai, Liu and Luo (2011). We can also take the covariance estimator in Section 4.2, but we will then need to establish its sampling property as a precision matrix estimator. We decide to use the CLIME estimator since such a property has already been established by Cai, Liu and Luo (2011). Denote by Σ⁻¹ = (Ω_jk)_d×d. From simple algebra,

$$\begin{array}{l}\mid \widehat{\gamma }-\gamma \mid \le \underset{1\le j,k\le d}{max}\mid {\stackrel{\sim }{\mu }}_j{\stackrel{\sim }{\mathrm{\Omega }}}_{jk}{\stackrel{\sim }{\mu }}_k-{\mu }_j{\mathrm{\Omega }}_{jk}{\mu }_k\mid \\ \le Cmax\{{\mid \stackrel{\sim }{\mathit{\mu }}-\mathit{\mu }\mid }_{\infty },{\mid \stackrel{\sim }{\mathbf{\Omega }}-{\mathbf{\sum }}^{-1}\mid }_{\infty }\}.\end{array}$$

In Section 4.1, we have seen that ${\Vert {\widehat{\mathit{\mu }}}_C-\mathit{\mu }\Vert }_{\infty }=O_p(\sqrt{log(d)/n})$. Moreover, Cai, Liu and Luo (2011) showed that ${\mid \stackrel{\sim }{\mathbf{\Omega }}-{\mathbf{\sum }}^{-1}\mid }_{\infty }={\Vert {\mathbf{\sum }}^{-1}\Vert }_1·O_p(\sqrt{log(d)/n})$ under mild conditions, where ||·||₁ is the matrix L₁-norm. Therefore, provided that ||Σ⁻¹||₁ ≤ C, we immediately have $\mid \widehat{\gamma }-\gamma \mid =O_p(\sqrt{log(d)/n})$.

5. Theoretical properties

In this section, we establish an oracle inequality for the Rayleigh quotient of the QUADRO estimates (Ω̂,δ̂). We assume that π and γ are known. For notational simplicity, we set λ₁ = λ₂ = λ. The results can be easily generalized to the case λ₁ ≠ λ₂. Moreover, we drop the symmetry constraint Ω = Ω^⊤ in all optimization problems involved. This simplifies the expression of the regularity conditions. The analysis with the symmetry constraint is a trivial extension of current analysis.

Recall the definition of M, L₁ and L₂ in (5) and κ = (1 − π)/π and L = L₁ + κL₂, the Rayleigh quotient of (Ω, δ) is equal to (up to a multiplicative constant)

$$R(\mathbf{\Omega },\mathit{\delta })=\frac{{[M(\mathbf{\Omega },\mathit{\delta })]}^2}{L(\mathbf{\Omega },\mathit{\delta })}.$$

The QUADRO estimates are

$$(\widehat{\mathbf{\Omega }},\widehat{\mathit{\delta }})=\underset{(\mathbf{\Omega },\mathit{\delta }):\widehat{M}(\mathbf{\Omega },\mathit{\delta })=1}{\text{argmin}}\{\widehat{L}(\mathbf{\Omega },\mathit{\delta })+\lambda {\mid \mathbf{\Omega }\mid }_1+\lambda {\mid \mathit{\delta }\mid }_1\}.$$

We shall compare the Rayleigh quotient of (Ω̂,δ̂) with the Rayleigh quotients of a class of “oracle solutions.” This class includes the one that maximizes the true Rayleigh quotient, which we denote by ( ${\mathbf{\Omega }}_0^{\ast },{\mathit{\delta }}_0^{\ast }$). Here we adopt a class of solutions as the “oracle” instead of only ( ${\mathbf{\Omega }}_0^{\ast },{\mathit{\delta }}_0^{\ast }$), because we want the results not tied to the sparsity assumption on ( ${\mathbf{\Omega }}_0^{\ast },{\mathit{\delta }}_0^{\ast }$) but a weaker assumption: at least one solution in this class is sparse.

Our theoretical development is technically nontrivial. Conventional oracle inequalities are derived in a setting of minimizing a data-dependent loss without constraint, and the risk function is the expectation of the loss. Here we minimize a data-dependent loss with a data-dependent equality constraint, and the risk function—the Rayleigh quotient—is not equal to the expectation of the loss. A similar setting was considered in Fan, Feng and Tong (2012), where they introduced a data-dependent intermediate solution to deal with such equality constraint. However, the rate they obtained depends on this intermediate solution, which is very hard to quantify. In contrast, the rate in our results purely depends on the oracle solution. To get rid of the intermediate solution in the rate, we need to carefully quantify its difference from both the QUADRO solution and the oracle solution. The technique is new, and potentially useful for other problems.

5.1. Oracle solutions, the restricted eigenvalue condition

For any λ₀ ≥ 0, we define the oracle solution associated with λ₀ to be

$$({\mathbf{\Omega }}_{{\lambda }_0}^{\ast },{\mathit{\delta }}_{{\lambda }_0}^{\ast })=\underset{(\mathbf{\Omega },\mathit{\delta }):M(\mathbf{\Omega },\mathit{\delta })=1}{\text{argmin}}\{L(\mathbf{\Omega },\mathit{\delta })+{\lambda }_0{\mid \mathbf{\Omega }\mid }_1+{\lambda }_0{\mid \mathit{\delta }\mid }_1\}.$$ (12)

We shall compare the Rayleigh quotient of (Ω̂,δ̂) to that of ( ${\mathbf{\Omega }}_{{\lambda }_0}^{\ast },{\mathit{\delta }}_{{\lambda }_0}^{\ast }$), for an arbitrary λ₀. In particular, when λ₀ = 0, the associated oracle solution (may not be unique) becomes

$$({\mathbf{\Omega }}_0^{\ast },{\mathit{\delta }}_0^{\ast })=\underset{(\mathbf{\Omega },\mathit{\delta }):M(\mathbf{\Omega },\mathit{\delta })=1}{\text{argmin}}\{L(\mathbf{\Omega },\mathit{\delta })\}.$$

It maximizes the true Rayleigh quotient.

Next, we introduce a restricted eigenvalue (RE) condition jointly on Σ₁, Σ₂, μ₁ and μ₂. For any matrices A and B, let vec(A) be the vectorization of A by stacking all the elements of A column by column, and A ⊗ B be the Kronecker product of A and B. We define the matrices

$${\mathbf{Q}}_k=\left[\begin{array}{cc}(2(1+\gamma ){\mathbf{\sum }}_k+4{\mathit{\mu }}_k{\mathit{\mu }}_k^{\top })\otimes {\mathbf{\sum }}_k+\gamma \text{vec}({\mathbf{\sum }}_k)\text{vec}{({\mathbf{\sum }}_k)}^{\top }& -4{\mathit{\mu }}_k\otimes {\mathbf{\sum }}_k\\ -4{\mathit{\mu }}_k^{\top }\otimes {\mathbf{\sum }}_k& 4{\mathbf{\sum }}_k\end{array}\right],$$

for k = 1, 2. We note that there are (d² + d) coefficients to decide when maximizing R(Ω, δ): d² elements of Ω and d elements of δ. We can stack all these coefficients into a long vector x = x(Ω, δ) in ℝ^d2+d defined as

$$\mathbf{x}(\mathbf{\Omega },\mathit{\delta })\equiv {[\text{vec}{(\mathbf{\Omega })}^{\top },{\mathit{\delta }}^{\top }]}^{\top }.$$ (13)

It can be shown that L_k(Ω, δ) = x^⊤Q_kx, for k = 1, 2; see Lemma 9.1. Therefore, L(Ω, δ) = x^⊤Qx, where Q = Q₁ +κQ₂. Our RE condition is then imposed on the (d² + d)×(d² + d) matrix Q, and hence implicitly on (Σ₁, Σ₂,μ₁,μ₂).

We now formally introduce the RE condition. For a set S ⊂ {1, 2,…, d² + d} and a nonnegative value c̄, we define the restricted eigenvalue in the following way:

$$\mathrm{\Theta }(S;\overline{c})=\underset{\mathbf{v}:{\mid {\mathbf{v}}_{S^c}\mid }_1\le \overline{c}{\mid {\mathbf{v}}_S\mid }_1}{min}\frac{{\mathbf{v}}^{\top }\mathbf{Qv}}{{\mid {\mathbf{v}}_S\mid }^2}.$$

Generally speaking, Θ(S; c̄) depends on (Σ₁, Σ₂,μ₁,μ₂) in a complicated way. For c̄ = 0, the following proposition builds a connection between Θ(S;0) and (Σ₁, Σ₂,μ₁,μ₂). For each S ⊂ {1, 2,…, d² + d}, there exist sets U ⊂ {1,…, d} × {1,…, d} and V ⊂ {1,…, d} such that the support of x(Ω, δ) is S if and only if the support of Ω is U and the support of δ is V. Let

$$U^{\prime }=\underset{(i,j)\in U}{\cup }\{i,j\}.$$

Then U ⊂ U′ ×U′. The following result is proved in Fan et al. (2014).

Proposition 5.1

For any set S ⊂ {1,…, d² + d}, suppose U′ and V are defined as above. Let Σ̃_k be the submatrix of Σ_k by restricting rows and columns to U′ ∪V, μ̃_k be the subvector of μ_k by constraining elements to U′∪V, for k = 1, 2. If there exist constants v₁, v₂ > 0 such that ${\lambda }_{min}({\mathbf{\sum }^{\sim }}_k-v_1{\stackrel{\sim }{\mathit{\mu }}}_k{\stackrel{\sim }{\mathit{\mu }}}_k^{\top })\ge \frac{1}{2}{\lambda }_{min}({\mathbf{\sum }^{\sim }}_k)\ge \frac{v_2}{2}$ for k = 1, 2, then

$$\mathrm{\Theta }(S,0)\ge (1+\gamma )(1+\kappa )v_{2}min\left\{v_2,\frac{4v_1}{2+v_1(1+\gamma )}\right\}>0.$$

5.2. Oracle inequality on Rayleigh’s quotient

Suppose max{|Σ_k|_∞, |μ_k|_∞, k = 1, 2} ≤ 1 and |Σ̂_k − Σ_k|_∞ ≤ |Σ_k|_∞, |μ̂_k − μ_k|_∞ ≤ |μ_k|_∞ for k = 1, 2, without loss of generality. For any λ₀ ≥ 0, let ( ${\mathbf{\Omega }}_{{\lambda }_0}^{\ast },{\mathit{\delta }}_{{\lambda }_0}^{\ast }$) be the associated oracle solution and S be the support of ${\mathbf{x}}_{{\lambda }_0}^{\ast }={[\text{vec}{({\mathbf{\Omega }}_{{\lambda }_0}^{\ast })}^{\top },{({\mathit{\delta }}_{{\lambda }_0}^{\ast })}^{\top }]}^{\top }$. Let Δ_n = max{|Σ̂_k − Σ_k|_∞, |μ̂_k − μ_k|_∞, k = 1, 2}. We have the following result for any given estimators, the proof of which we postpone to Section 9.

Theorem 5.1

Given λ₀ ≥ 0, let S be the support of ${\mathbf{x}}_{{\lambda }_0}^{\ast }$, s₀ = |S| and $k_0=max\{s_0,R({\mathbf{\Omega }}_{{\lambda }_0}^{\ast },{\mathit{\delta }}_{{\lambda }_0}^{\ast })\}$. Suppose that Θ(S, 0) ≥ c₀, Θ(S, 3) ≥ a₀ and $R({\mathbf{\Omega }}_{{\lambda }_0}^{\ast },{\mathit{\delta }}_{{\lambda }_0}^{\ast })\ge u_0$, for some positive constants a₀, c₀ and u₀. We assume $4s_0{\mathrm{\Delta }}_n^2\le a_0{c}_0$ and $max\{s_0{\mathrm{\Delta }}_n,s_0^{1/2}{k}_0^{1/2}{\lambda }_0\}<1$ without loss of generality. Then there exist positive constants C = C(a₀, c₀,u₀) and A = A(a₀, c₀,u₀) such that for any η >1,

$$\frac{R(\widehat{\mathbf{\Omega }},\widehat{\mathit{\delta }})}{R({\mathbf{\Omega }}_{{\lambda }_0}^{\ast },{\mathit{\delta }}_{{\lambda }_0}^{\ast })}\ge 1-A{\eta }^{2}max\{s_0{\mathrm{\Delta }}_n,s_0^{1/2}{k}_0^{1/2}{\lambda }_0\},$$

by taking $\lambda =C\eta max\{s_0^{1/2}{\mathrm{\Delta }}_n,k_0^{1/2}{\lambda }_0\}{[R({\mathbf{\Omega }}_{{\lambda }_0}^{\ast },{\mathit{\delta }}_{{\lambda }_0}^{\ast })]}^{-1/2}$.

In Theorem 5.1, the rate of convergence has two parts. The term s₀Δ_n reflects how the stochastic errors of estimating (Σ ₁, Σ ₂,μ₁,μ₂) affect the Rayleigh quotient. The term $s_0^{1/2}{k}_0^{1/2}{\lambda }_0$ is an extra term that depends on the oracle solution we aim to use for comparison. In particular, if we compare R(Ω̂,δ̂) with $R_{max}\equiv R({\mathbf{\Omega }}_0^{\ast },{\mathit{\delta }}_0^{\ast })$, the population maximum Rayleigh quotient with λ₀ = 0, this extra term disappears. If we further use the estimators in Section 4, ${\mathrm{\Delta }}_n=O_p(\sqrt{log(d)/n})$. We summarize the result as follows.

Corollary 5.1

Suppose that the condition of Theorem 5.1 holds with λ₀ = 0. Then for some positive constants A and C, when $\lambda >C{s}_0^{1/2}{R}_{max}^{-1/2}{\mathrm{\Delta }}_n$, we have

$$R(\widehat{\mathbf{\Omega }},\widehat{\mathit{\delta }})\ge (1-A{s}_0{\mathrm{\Delta }}_n)R_{max}.$$

Furthermore, if the mean vectors and covariance matrices are estimated by using the robust methods in Section 4, then when $\lambda >C{s}_0^{1/2}{R}_{max}^{-1/2}\sqrt{log(d)/n}$,

$$R(\widehat{\mathbf{\Omega }},\widehat{\mathit{\delta }})\ge (1-A{s}_0\sqrt{log(d)/n})R_{max},$$

with probability at least 1− (n ∨ d)⁻¹.

From Corollary 5.1, when ( ${\mathbf{\Omega }}_0^{\ast },{\mathit{\delta }}_0^{\ast }$) is truly sparse, R(Ω̂,δ̂) is close to the population maximum Rayleigh quotient R_max. However, we note that Theorem 5.1 considers more general situations, including cases where ( ${\mathbf{\Omega }}_0^{\ast },{\mathit{\delta }}_0^{\ast }$) is not sparse. As long as there exists an “approximately optimal” and sparse solution, that is, for a small λ₀ the associated oracle solution ( ${\mathbf{\Omega }}_{{\lambda }_0}^{\ast },{\mathit{\delta }}_{{\lambda }_0}^{\ast }$) is sparse, Theorem 5.1 guarantees that R(Ω̂,δ̂) is close to $R({\mathbf{\Omega }}_{{\lambda }_0}^{\ast },{\mathit{\delta }}_0^{\ast })$ and hence close to R_max.

Remark

Our results are analogous to oracle inequalities for prediction error in linear regressions; therefore, the condition Θ (S, c̄) is similar to the RE condition in linear regressions [Bickel, Ritov and Tsybakov (2009)]. To recover the support of ( ${\mathbf{\Omega }}_0^{\ast },{\mathit{\delta }}_0^{\ast }$), conditions similar to the “irrepresentable condition” for Lasso [Zhao and Yu (2006)] are needed.

6. Application to classification

One important application of QUADRO is high-dimensional classification for elliptically-distributed data. Suppose (Ω̂,δ̂) are the QUADRO estimates. This yields the classification rule

$$\widehat{h}(\mathbf{x})=I\{{\mathbf{x}}^{\top }\widehat{\mathbf{\Omega }}\mathbf{x}-2{\widehat{\mathit{\delta }}}^{\top }\mathbf{x}<c\}.$$

In this section, we first show that for normally distributed data, the Rayleigh quotient is a proxy of the classification error, and then derive an analytic choice of c. Comparing with many other high-dimensional classification methods, QUADRO produces quadratic boundaries and can handle both non-Gaussian distributions and nonequal covariance matrices.

6.1. Approximation of classification errors

Given (Ω, δ) and a threshold c, a general quadratic rule h(x) = h(x; Ω, δ, c) is defined as

$$h(\mathbf{x};\mathbf{\Omega },\mathit{\delta },c)=I\{{\mathbf{x}}^{\top }\mathbf{\Omega }\mathbf{x}-2{\mathbf{x}}^{\top }\mathit{\delta }<c\}.$$ (14)

We reparametrize c as

$$c=t{M}_1(\mathbf{\Omega },\mathit{\delta })+(1-t)M_2(\mathbf{\Omega },\mathit{\delta }).$$ (15)

Here $M_k(\mathbf{\Omega },\mathit{\delta })={\mathit{\mu }}_k^{\top }\mathbf{\Omega }{\mathit{\mu }}_k-2{\mathit{\mu }}_k^{\top }\mathit{\delta }+\text{tr}(\mathbf{\Omega }{\mathbf{\sum }}_k)$ s the mean of Q(X) in class k, for k = 1, 2. After the reparametrization, t is scale-free. As we will see below, in most cases, given Ω and δ, the optimal t that minimizes the classification error takes values on (0, 1).

From now on, we write h(x; Ω, δ, c) = h(x; Ω, δ, t). Let Err(Ω, δ, t) be the classification error of h(·;Ω, δ, t). Due to technical difficulties, we only give results for Gaussian distributions. Suppose X|(Y = 0) ~ 𝒩(μ₁, Σ₁) and X|(Y = 1) ~ 𝒩(μ₂, Σ₂). For k = 1, 2, we write

$${\mathbf{\sum }}_k^{1/2}\mathbf{\Omega }{\mathbf{\sum }}_k^{1/2}={\mathbf{K}}_k{\mathbf{S}}_k{\mathbf{K}}_k^T,$$

where S_k is a diagonal matrix containing the nonzero eigenvalues, and the columns of K_k are corresponding eigenvectors. Let ${\mathit{\beta }}_k={\mathbf{K}}_k^T{\mathbf{\sum }}_k(\mathbf{\Omega }{\mathit{\mu }}_k-\mathbf{\delta })$. When max{|S_k|_∞, |β_k|_∞, k = 1, 2} is bounded, the following proposition shows that an approximation of Err(Ω, δ, t) is

$$\overline{\text{Err}}(\mathbf{\Omega },\mathit{\delta },t)\equiv \pi \overline{\mathrm{\Phi }}\left(\frac{(1-t)M(\mathbf{\Omega },\mathit{\delta })}{\sqrt{L_1(\mathbf{\Omega },\mathit{\delta })}}\right)+(1-\pi )\overline{\mathrm{\Phi }}\left(\frac{tM(\mathbf{\Omega },\mathit{\delta })}{\sqrt{L_2(\mathbf{\Omega },\mathit{\delta })}}\right),$$

where M, L₁ and L₂ are defined in (5), Φ is the distribution function of a standard normal variable and Φ̄ = 1 − Φ. Its proof is contained in Section 9.

Proposition 6.1

Suppose that max{|S_k|_∞, |β_k|_∞, k = 1, 2} ≤ C₀ for some constant C₀ > 0, and let q be the rank of Ω. Then as d goes to infinity,

$$\mid \text{ERR}(\mathbf{\Omega },\mathit{\delta },t)-\overline{\text{Err}}(\mathbf{\Omega },\mathit{\delta },t)\mid =\frac{O(q)+o(d)}{{[min\{L_1(\mathbf{\Omega },\mathit{\delta }),L_2(\mathbf{\Omega },\mathit{\delta })\}]}^{3/2}}.$$

In particular, if we consider all such (Ω, δ) that the variance of Q(X; Ω, δ) under both classes are lower bounded by c₀d^θ for some constants θ > 2/3 and c₀ > 0, then we have $\mid \text{Err}-\overline{\text{Err}}\mid =o(1)$.

We now take a closer look at $\overline{\text{Err}}$. Let $H(x)=\overline{\mathrm{\Phi }}(1/\sqrt{x})$, which is monotone increasing on (0,∞). Writing for short M =M₁ − M₂, M_k =M_k(Ω, δ) and L_k = L_k(Ω, δ) for k = 1, 2, we have

$$\overline{\text{Err}}(\mathbf{\Omega },\mathit{\delta },t)=\pi H\left(\frac{L_1}{{(1-t)}^2{M}^2}\right)+(1-\pi )H\left(\frac{L_2}{t^2{M}^2}\right).$$

Figure 2 shows that H(·) is nearly linear on an important range. This suggests the following approximation:

$$\overline{\text{Err}}(\mathbf{\Omega },\mathit{\delta },t)\approx H\left(\pi \frac{L_1}{{(1-t)}^2{M}^2}+(1-\pi )\frac{L_2}{t^2{M}^2}\right)=H\left(\frac{\pi }{{(1-t)}^2}\frac{1}{R^{(t)}}\right),$$ (16)

where R^(t) = R^(t)(Ω, δ) is the R(Ω, δ) in (6) corresponding to the κ value

Fig. 2.

Function $H(x)=\overline{\mathrm{\Phi }}(1/\sqrt{x})$.

$$\kappa (t)\equiv \frac{1-\pi }{\pi }\frac{{(1-t)}^2}{t^2}.$$

The approximation in (16) is quantified in the following proposition, which is proved in Fan et al. (2014).

Proposition 6.2

Given (Ω, δ, t), we write for short R_k = R_k(Ω, δ) = [M(Ω, δ)]²/L_k(Ω, δ), for k = 1, 2, and define

$$\begin{array}{ll}\hfill V_1& =V_1(\mathbf{\Omega },\mathit{\delta },t)=min\left\{{(1-t)}^2{R}_1,\frac{1}{{(1-t)}^2{R}_1}\right\},\\ \hfill V_2& =V_2(\mathbf{\Omega },\mathit{\delta },t)=min\left\{t^2{R}_2,\frac{1}{t^2{R}_2}\right\},\\ \hfill V& =V(\mathbf{\Omega },\mathit{\delta },t)=max\{V_1/V_2,V_2/V_1\}.\end{array}$$

Then there exists a constant C > 0 such that

$$\left|\overline{\text{Err}}(\mathbf{\Omega },\mathit{\delta },t)-H\left(\frac{\pi }{{(1-t)}^2{R}^{(t)}(\mathbf{\Omega },\mathit{\delta })}\right)\right|\le C{[max\{V_1,V_2\}]}^{1/2}\cdot {\mid V-1\mid }^2.$$

In particular, when t = 1/2,

$$\left|\overline{\text{Err}}(\mathbf{\Omega },\mathit{\delta },t)-H\left(\frac{\pi }{{(1-t)}^2{R}^{(t)}(\mathbf{\Omega },\mathit{\delta })}\right)\right|\le C{R}_0^{1/2}.{\left(\frac{\mathrm{\Delta }R}{R_0}\right)}^2,$$

where R₀ = max{min{R₁, 1/R₁}, min{R₂, 1/R₂}} and ΔR = |R₁ − R₂|.

Note that L₁ and L₂ are the variances of Q(X) = X^⊤ΩX − 2X^⊤δ for two classes, respectively. In cases where |L₁ − L₂| ≪ min{L₁, L₂}, ΔR ≪ R₀. Also, R₀ is always bounded by 1, and it tends to 0 in many situations, for example, when R₁, R₂ → ∞, or R₁, R₂ → 0, or R₁ → 0, R₂ → ∞. Proposition 6.2 then implies that the approximation in (16) when t = 1/2 is good.

Combining Propositions 6.1 and 6.2, the classification error of a general quadratic rule h(·; Ω, δ, t) is approximately a monotone decreasing transform of the Rayleigh quotient R^(t)(Ω, δ), corresponding to κ = κ(t). In particular, when t = 1/2 [i.e., c = (M₁ + M₂)/2], R^(1/2)(Ω, δ) is exactly the one used in QUADRO. Consequently, if we fix the threshold to be c = (M₁ + M₂)/2, then the Rayleigh quotient (upon with a monotone transform) is a good proxy for classification error. This explains why Rayleigh-quotient based procedures can be used for classification.

Remark

Even in the region that H(·) is far from being linear such that the upper bound in Proposition 6.2 is not o(1), we can still find a monotone transform of the Rayleigh quotient as an upper bound of the classification error. To see this, note that for x ∈ [1/3, ∞), H(x) is a concave function. Therefore, the approximation in (16) becomes an inequality, that is, $\overline{\text{Err}}(\mathbf{\Omega },\mathit{\delta },t)\le H(\frac{\pi R^{(t)}}{{(1-t)}^2})$. For x ∈ (0, 1/3), H(x) ≤ 0.1248x. It follows that $\overline{\text{Err}}(\mathbf{\Omega },\mathit{\delta },t)\le 0.1248\cdot \frac{\pi R^{(t)}}{{(1-t)}^2}$.

Remark

In the current setting, the Bayes classifier is a quadratic rule h(x; Ω_B, δ_B, c_B) with ${\mathbf{\Omega }}_B={\mathbf{\sum }}_1^{-1}-{\mathbf{\sum }}_2^{-1},{\mathit{\delta }}_B={\mathbf{\sum }}_1^{-1}{\mathit{\mu }}_1-{\mathbf{\sum }}_2^{-1}{\mathit{\mu }}_2$ and $c_B={\mathit{\mu }}_2^{\top }{\mathbf{\sum }}_2^{-1}{\mathit{\mu }}_2-{\mathit{\mu }}_1^{\top }{\mathbf{\sum }}_1^{-1}{\mathit{\mu }}_1$. Let ( ${\mathbf{\Omega }}_0^{\ast },{\mathit{\delta }}_0^{\ast }$) be the population solution of QUADRO when λ = 0. We note that (Ω_B, δ_B) and ( ${\mathbf{\Omega }}_0^{\ast },{\mathit{\delta }}_0^{\ast }$) are different: the former minimizes inf_t Err(Ω, δ, t), while the latter minimizes $\overline{\text{Err}}(\mathbf{\Omega },\mathit{\delta },1/2)$.

6.2. QUADRO as a classification method

Results in Section 6.1 suggest an analytic method to choose the threshold c, or equivalently t, with given (Ω, δ). Let

$$\widehat{t}\in \underset{t}{min}\left\{\pi \overline{\mathrm{\Phi }}\left(\frac{(1-t)\widehat{M}(\mathbf{\Omega },\mathit{\delta })}{\sqrt{{\widehat{L}}_1(\mathbf{\Omega },\mathit{\delta })}}\right)+(1-\pi )\overline{\mathrm{\Phi }}\left(\frac{t\widehat{M}(\mathbf{\Omega },\mathit{\delta })}{\sqrt{{\widehat{L}}_2(\mathbf{\Omega },\mathit{\delta })}}\right)\right\},$$ (17)

and set

$$\widehat{c}=(1-\widehat{t}){\widehat{M}}_1(\mathbf{\Omega },\mathit{\delta })+\widehat{t}{\widehat{M}}_2(\mathbf{\Omega },\mathit{\delta }).$$ (18)

Here (17) is a one-dimensional optimization problem and can be solved easily. The resulting QUADRO classification rule is

$${\widehat{h}}^{\text{Quad}}(\mathbf{x})=I\{{\mathbf{x}}^{\top }\widehat{\mathbf{\Omega }}\mathbf{x}-2{\mathbf{x}}^{\top }\widehat{\mathit{\delta }}-\widehat{c}<0\}.$$

As a by-product, the method to decide c, described in (17) and (18), can be used in other classification procedures on Gaussian data, such as logistic regression, quadratic discriminant analysis (QDA) and kernel support vector machine, once (Ω̂, δ̂) are given. It provides a fast and purely data-driven way to decide the threshold value in quadratic classification rules. In our numerical experiments, it performs well.

7. Numerical studies

In this section, we investigate the performance of QUADRO in several simulation examples and a real data example. The simulation studies contain both Gaussian models and general elliptical models. We compare QUADRO with several classification-oriented procedures. Performances are evaluated in terms of classification errors.

7.1. Simulations under Gaussian models

Let n₁ = n₂ = 50 and d = 40. For each given μ₁, μ₂, Σ₁ and Σ₂, we generate 100 training datasets independently, each with n₁ data from 𝒩(μ₁, Σ₁) and n₂ data from 𝒩(μ₂, Σ₂). In QUADRO, we input the sample means and sample covariance matrices. We set λ₂ = rλ₁ and work with λ₁ and r from now on. The two tuning parameters λ₁ ≥ 0 and r > 0 are selected in the following way. For various pairs of (λ₁, r), we apply QUADRO for each pair and evaluate the classification error via 4000 newly generated testing data; we then choose the (λ₁, r) that minimize the classification error.

We compare QUADRO with five classification-oriented procedures:

• Sparse logistic regression (SLR): We apply the sparse logistic regression to the augmented feature space {X_i, 1 ≤ i ≤ d; X_iX_j, 1 ≤ i ≤ j ≤ d}. The resulting estimator then gives a quadratic projection with (Ω, δ, c) decided from the fitted regression coefficients. We implement the sparse logistic regression using the R package glmnet.

• Linear sparse logistic regression (L-SLR): We apply the sparse logistic regression directly to the original feature space {X_i, 1 ≤ i ≤ d}.

• ROAD [Fan, Feng and Tong (2012)]: This is a linear classification method, which can be formulated equivalently as a modified version of QUADRO by enforcing Ω̂ as the zero matrix and plugging in the pooled sample covariance matrix.

• Penalized-LDA (P-LDA) [Witten and Tibshirani (2011)]: This is a variant of LDA, which solves an optimization problem with a nonconvex objective and L₁ penalties. Also, P-LDA only uses diagonals of the sample covariance matrices.

• FAIR [Fan and Fan (2008)]: This is a variant of LDA for high-dimensional settings, where screening is adopted to pre-select features and only the diagonals of the sample covariance matrices are used.

To make a fair comparison, the tuning parameters in SLR and L-SLR are selected in the same way as in QUADRO based on 4000 testing data. ROAD and P-LDA are self-tuned by its package. The number of features chosen in FAIR is calculated in the way suggested in [Fan and Fan (2008)].

We consider four models:

• Model 1: Σ₁ is the identity matrix. Σ₂ is a diagonal matrix in which the first 10 elements are equal to 1.3 and the rest are equal to 1. μ₁ = 0, and μ₂ = (0.7, …, 0.7, 0, …, 0)^⊤ with the first 10 elements of μ₂ being nonzero.

• Model 1L: μ₁, μ₂ are the same as in model 1, and both Σ₁ and Σ₂ are the identity matrix.

• Model 2: Σ₁ is a block-diagonal matrix. Its upper left 20×20 block is an equal correlation matrix with ρ = 0.4, and its lower right 20 × 20 block is an identity matrix. ${\mathbf{\sum }}_2={({\mathbf{\sum }}_1^{-1}+\mathbf{I})}^{-1}$. We also set μ₁ = μ₂ = 0. In this model, neither ${\mathbf{\sum }}_1^{-1}$ nor ${\mathbf{\sum }}_2^{-1}$ is sparse, but ${\mathbf{\sum }}_1^{-1}-{\mathbf{\sum }}_2^{-1}$ is.

• Model 3: Σ₁, Σ₂ and μ₁ are the same as in model 2, and μ₂ is taken from model 1.

Figure 3 contains the boxplots for the classification errors of all methods. In all four models, QUADRO outperforms other methods in terms of classification error. In model 1L, Σ₁ = Σ₂, so the Bayes classifier is linear. In this case which favors linear methods, QUADRO is still competitive with the best of all linear classifiers. In model 2, μ₁ = μ₂, so linear methods can do no better than random guessing. Therefore, ROAD, L-SLR, P-LDA and FAIR all have very poor performances. For the two quadratic methods, QUADRO is significantly better than SLR. In models 1 and 3, μ₁ ≠ μ₂ and Σ₁ ≠ Σ₂, so in the Bayes classifier, both “linear” parts and “quadratic” parts play important roles. In model 1, both Σ₁ and Σ₂ are diagonal, and the setting favors methods using only diagonals of sample covariance matrices. As a result, P-LDA and FAIR perform quite well. In model 3, Σ₁ and Σ₂ are both nondiagonal and nonsparse (but Σ₁ − Σ₂ is sparse). We see that the performances of P-LDA and FAIR are unsatisfactory. QUADRO outperforms other methods in both models 1 and 3.

Fig. 3.

Distributions of minimum classification error based on 100 replications for four different normal models. The tuning parameters for QUADRO, SLR and L-SLR are chosen to minimize the classification errors of 4000 testing samples. See Fan et al. (2014) for detailed numerical tables.

Comparing SLR and L-SLR, we see the former considers a broader class, while the latter is more robust, but neither of them perform uniformly better. However, QUADRO performs well in all cases. In terms of Rayleigh quotients, QUADRO also outperforms other methods in most cases.

7.2. Simulations under elliptical models

Let n₁ = n₂ = 50 and d = 40. For each given μ₁, μ₂, Σ₁ and Σ₂, data are generated from multivariate t distribution with degrees of freedom 5. In QUADRO, we input the robust M-estimators for means and the rank-based estimators for covariance matrices as described in Section 4. We compare the performance of QUADRO with the five methods compared under Gaussian settings. We also implement QUADRO with inputs of sample means and sample covariance matrices. We name this method QUADRO-0 to differentiate it from QUADRO.

We consider three models:

• Model 4: Here we use same parameters as those in model 1.

• Model 5: Σ₁, μ₁ and μ₂ are the same as in model 1. Σ₂ is the covariance matrix of a fractional white noise process, where the difference parameter l = 0.2. In other words, Σ₂ has the polynomial off-diagonal decay |Σ₂(i, j)| = O(|i − j|^1−2l).

• Model 6: Σ₁, μ₁ and μ₂ are the same as in model 1. Σ₂ is a matrix such that Σ₂(i, j) = 0.6^|i−j|; that is, Σ₂ has an exponential off-diagonal decay.

Figure 4 contains the boxplots of average classification error over 100 replications. QUADRO outperforms the other methods in all settings. Also, QUADRO is better than QUADRO-0 (e.g., 0.161 versus 0.173, of the average classification error in model 5), which illustrates the advantage of using the robust estimators for means and covariance matrices.

Fig. 4.

Distributions of minimum classification error based on 100 replications across different elliptical distribution models. The tuning parameters for QUADRO, SLR and L-SLR are chosen to minimize the classification errors. See Fan et al. (2014) for detailed numerical tables.

7.3. Real data analysis

We apply QUADRO to a large-scale genomic dataset, GPL96, and compare the performance of QUADRO with SLR, L-SLR, ROAD, P-LDA and FAIR. The GPL96 data set contains 20,263 probes and 8124 samples from 309 tissues. Among the tissues, breast tumor has 1142 samples, which is the largest set. We merge the probes from the same gene by averaging them, and finally get 12,679 genes and 8124 samples. We divide all samples into two groups: breast tumor or nonbreast tumor.

First, we look at the classification errors. We replicate our experiment 100 times. Each time, we proceed with the following steps:

• Randomly choose a training set of 400 samples, 200 from breast tumor and 200 from nonbreast tumor.

• For each training set, we use half of the samples to compute (Ω̂, δ̂) and the other half to select the tuning parameters by minimizing the classification error.

• Use the remaining 942 samples from breast tumor and another randomly chosen 942 samples from nonbreast tumor as testing set, and calculate the testing error.

FAIR does not have any tuning parameters, so we use the whole training set to calculate classification frontier, and the rest to calculate testing error. The results are summarized in Table 1. We see that QUADRO outperforms all other methods.

Table 1.

Classification errors on GPL96 dataset, across methods QUADRO, SLR and L-SLR. Means and standard deviations (in the parenthesis) of 100 replications are reported

QUADRO	SLR	L-SLR	ROAD	Penalized-LDA	FAIR
0.014 (0.007)	0.025 (0.007)	0.025 (0.009)	0.016 (0.007)	0.060 (0.011)	0.046 (0.009)

Next, we look at gene selection and focus on the two quadratic methods, QUADRO and SLR. We apply two-fold cross-validation to both QUADRO and SLR. In the results, QUADRO selects 139 genes and SLR selects 128 genes. According to KEGG database, genes selected by QUADRO belong to 5 of the pathways that contain more than two genes; correspondingly, genes selected by SLR belong to 7 pathways. Using the ClueGo tool [Bindea et al. (2009)], we display the overall KEGG enrichment chart in Figure 5. We see from Figure 5 that both QUADRO and SLR have focal adhesion as its most important functional group. Nevertheless, QUADRO finds ECM-receptor interaction as another important functional group. ECM-receptor interaction is a class consisting of a mixture of structural and functional macromolecules, and it plays an important role in maintaining cell and tissue structures and functions. Massive studies [Luparello (2013), Wei and Li (2007)] have found evidence that this class is closely related to breast cancer.

Fig. 5.

Overall KEGG enrichment chart, using (a) QUADRO; (b) SLR.

Besides the pathway analysis, we also perform the Gene Ontology (GO) enrichment analysis on genes selected by QUADRO. This analysis was completed by DAVID Bioinformatics Resources, and the results are shown in Table 2. We present the biological processes with p-values smaller than 10⁻³. According to the table, we see that many biological processes are significantly enriched, and they are related to previously selected pathways. For instance, the biological process cell adhesion is known to be highly related to cell communication pathways, including focal adhesion and ECM-receptor interaction.

Table 2.

Enrichment analysis results according to Gene Ontology for genes selected by QUADRO. The four columns represent GO ID, GO attribute, number of selected genes having the attribute and their corresponding p-values. We rank them according to p-values in increasing order

GO ID	GO attribute	No. of genes	p-value
0048856	Anatomical structure development	58	3.7E–12
0032502	Developmental process	62	2.9E–10
0048731	System development	52	3.1E–10
0007275	Multicellular organismal development	55	1.8E–8
0001501	Skeletal system development	15	1.3E–6
0032501	Multicellular organismal process	66	1.4E–6
0048513	Organ development	37	1.4E–6
0009653	Anatomical structure morphogenesis	28	8.7E–6
0048869	Cellular developmental process	34	1.9E–5
0030154	Cell differentiation	33	2.1E–5
0007155	Cell adhesion	18	2.4E–4
0022610	Biological adhesion	18	2.2E–4
0042127	Regulation of cell proliferation	19	2.9E–4
0009888	Tissue development	17	3.7E–4
0007398	Ectoderm development	9	4.8E–4
0048518	Positive regulation of biological process	34	5.6E–4
0009605	Response to external stimulus	20	6.3E–4
0043062	Extracellular structure organization	8	7.4E–4
0007399	Nervous system development	22	8.4E–4

8. Conclusions and extensions

QUADRO is a robust sparse high-dimensional classifier, which allows us to use differences in covariance matrices to enhance discriminability. It is based on Rayleigh quotient optimization. The variance of quadratic statistics involves all fourth cross moments, and this can create both computational and statistical problems. These problems are avoided by limiting our applications to the elliptical class of distributions. Robust M-estimator and rank-based estimation of correlations allow us to obtain the uniform convergence for nonpolynomially many parameters, even when the underlying distributions have the finite fourth moments. This allows us to establish oracle inequalities under relatively weaker conditions.

Existing methods in the literature about constructing high-dimensional quadratic classifiers can be divided into two types. One is the regularized QDA, where regularized estimates of ${\mathbf{\sum }}_1^{-1}$ and ${\mathbf{\sum }}_2^{-1}$ are plugged into the Bayes classifier; see, for example, Friedman (1989). QUADRO avoids directly estimating inverse covariance matrices, which requires strong assumptions in high dimensions. The other is to combine linear classifiers with the inner-product kernel. The main difference between QUADRO and this approach is the simplification in Proposition 2.1. Due to this simplification, QUADRO avoids incorporating all fourth cross moments from the data and gains extra statistical efficiency.

QUADRO also has deep connections with the literature of sufficient dimension reduction. Dimension reduction methods, such as SIR [Li (1991)], SAVE [Cook and Weisberg (1991)] and Directional Regression [Li and Wang (2007)], can be equivalently formulated as maximizing some “quotients.” The population objective of SIR is to maximize var{𝔼[f(X|Y)]} subject to var[f(X)] = 1. Using the same constraint, SAVE and directional regression combine var{𝔼[f(X|Y)]} and 𝔼[var(f(X|Y))] in the objective. An interesting observation is that the Rayleigh quotient maximization is equivalent to the population objective of SIR, by noting that the denominator of (1) is equal to 𝔼[var(f(X|Y))] and var[f(X)] = 𝔼[var(f(X|Y))] + var{𝔼[f(X|Y)]}. This is not a coincidence, but due instead to the known equivalence between SIR and LDA in classification [Kent (1991), Li (2000)].

Despite similar population objectives, QUADRO and the aforementioned dimension reduction methods are different in important ways. First, we clarify that even when λ₁, λ₂ are 0, QUADRO is not the same procedure as SIR combined with the inner-product kernel [Wu (2008)], although they share the same population objective. The difference is that QUADRO utilizes a simplification of the Rayleigh quotient for quadratic f, relying on the assumption that X|Y is always elliptically distributed; moreover, it adopts robust estimators of the mean vectors and covariance matrices. Second, QUADRO is designed for high-dimensional settings, in which neither SIR, SAVE nor Directional Regression can be directly implemented. These methods need to either standardize the original data X ↦ Σ̂⁻¹(X − X̄) or solve a generalized eigen-decomposition problem Av = λΣ̂v for some matrix A. Both methods require that the sample covariance matrix is well conditioned, which is often not the case in high dimensions. Possible solutions include Regularized SIR [Li and Yin (2008), Zhong et al. (2005)], solving generalized eigen-decomposition for an undetermined system [Coudret, Liquet and Saracco (2014)] and variable selection approaches [Chen, Zou and Cook (2010), Jiang and Liu (2013)]. However, these methods are not designed for Rayleigh quotient maximization. Third, our assumption on the model is different from that in dimension reduction. We require X|Y to be elliptically distributed, while many dimension reduction methods “implicitly” require X to be marginally elliptically distributed. Neither method is stronger than the other. Assuming conditional elliptical distribution is more natural in classification. In addition, our assumption is used only to simplify the variances of quadratic statistics, whereas the elliptical assumption is critical to SIR.

The Rayleigh optimization framework developed in this paper can be extended to the multi-class case. Suppose the data are drawn independently from a joint distribution of (X, Y), where X ∈ ℝ^d and Y takes values in {0, 1, …, K − 1}. Definition (1) for the Rayleigh quotient of a projection f : ℝ^d → ℝ is still well defined. Let π_k = ℙ(Y = k), for k = 0, 1, …, K − 1. In this K-class situation,

$$\text{Rq}(f)=\frac{{\sum }_{0\le k<l\le K-1}{\pi }_k{\pi }_l{\{\mathbb{E}[f(\mathbf{X})\mid Y=k]-\mathbb{E}[f(\mathbf{X})\mid Y=l]\}}^2}{{\sum }_{0\le k\le K-1}{\pi }_{k}var[f(\mathbf{X})\mid Y=k]}.$$ (19)

Let M_k(f) = 𝔼[f(X)|Y = k] and L_k(f) = var[f(X)|Y = k]. Similar to the two-class case, maximizing Rq(f) is equivalent to solving the following optimization problem:

$$\underset{f}{min}\sum _{k=0}^{K-1}{\pi }_k{L}_k(f)\text{s.t.}\sum _{0\le k<l\le K-1}{\pi }_k{\pi }_l{\mid M_k(f)-M_l(f)\mid }^2=1.$$

However, this is not a convex problem. We consider an approximate Rayleigh-quotient-maximization problem as follows:

$$\underset{f}{min}\sum _{k=0}^{K-1}{\pi }_k{L}_k(f)\text{s.t.}\sqrt{{\pi }_k{\pi }_l}\mid M_k(f)-M_l(f)\mid \ge 1,0\le k<l\le K-1.$$

To solve this problem, we first pick an order of M₁(f), …, M_K(f) to remove the absolute values in the constraints. Then it becomes a convex problem. Therefore, the whole optimization can be carried out by simultaneously solving K! convex problems. When K is small, the computational cost is reasonable. In practice, we can apply more efficient algorithms to speed up the computation.

9. Proofs

9.1. Proof of Theorem 5.1

We prove the claim by first rewriting optimization problem (8) into a vector form. For any (Ω, δ), write x = [vec(Ω)^⊤, δ^⊤]^⊤. Let Q be as defined in Section 5, and

$$\mathbf{q}={[\text{vec}{({\mathbf{\sum }}_2+{\mathit{\mu }}_2{\mathit{\mu }}_2^{\top }-{\mathbf{\sum }}_1-{\mathit{\mu }}_1{\mathit{\mu }}_1^{\top })}^{\top },2{({\mathit{\mu }}_1-{\mathit{\mu }}_2)}^{\top }]}^{\top }.$$

We introduce the following lemma which is proved in the supplementary material [Fan et al. (2014)].

Lemma 9.1

M(Ω, δ) = q^⊤x and L(Ω, δ) = x^⊤Qx.

Let ${\mathbf{x}}_{{\lambda }_0}^{\ast }={[\text{vec}{({\mathbf{\Omega }}_{{\lambda }_0}^{\ast })}^{\top },{({\mathit{\delta }}_{{\lambda }_0}^{\ast })}^{\top }]}^{\top }$ and x̂ = [vec(Ω̂)^⊤, δ̂^⊤]^⊤. Using Lemma 9.1,

$$\begin{array}{ll}\hfill {\mathbf{x}}_{{\lambda }_0}^{\ast }& =\underset{\mathbf{x}\in {ℝ}^d:{\mathbf{q}}^{\top }\mathbf{x}=1}{min}\{{\mathbf{x}}^{\top }\mathbf{Qx}+{\lambda }_0{\mid \mathbf{x}\mid }_1\},\hfill \\ \hfill \widehat{\mathbf{x}}& =\underset{\mathbf{x}\in {ℝ}^d:{\widehat{\mathbf{q}}}^{\top }\mathbf{x}=1}{\text{argmin}}\{{\mathbf{x}}^{\top }\widehat{\mathbf{Q}}\mathbf{x}+\lambda {\mid \mathbf{x}\mid }_1\},\hfill \end{array}$$

where Q̂ and q̂ are counterparts of Q and q, respectively, by replacing μ₁, μ₂, Σ₁ and Σ₂ with their estimates. Moreover, we have the Rayleigh quotient

$$R(\mathbf{\Omega },\mathit{\delta })=R(\mathbf{x})\equiv \frac{{({\mathbf{q}}^{\top }\mathbf{x})}^2}{{\mathbf{x}}^{\top }\mathbf{Qx}}.$$

In addition, we have the following lemma, which is proved in the supplementary material [Fan et al. (2014)].

Lemma 9.2

max{|Q̂ − Q|_∞, |q̂ − q|_∞} ≤ C₀ max{|Σ̂_k − Σ_k|_∞, |μ̂_k − μ_k|_∞, k = 1, 2} for some constant C₀ > 0.

Combining the above results, the claim follows immediately from the following theorem:

Theorem 9.1

For any λ₀ ≥ 0, let S be the support of ${\mathbf{x}}_{{\lambda }_0}^{\ast }$. Suppose Θ(S, 0) ≥ c₀, Θ (S, 3) ≥ a₀ and $R({\mathbf{x}}_{{\lambda }_0}^{\ast })\ge u_0$, for positive constants a₀, c₀ and u₀. Let Δ_n = max{|Q̂ − Q|_∞, |q̂ − q|_∞}, s₀ = |S| and $k_0=max\{s_0,R({\mathbf{x}}_{{\lambda }_0}^{\ast })\}$. Suppose $4s_0{\mathrm{\Delta }}_n^2<c_0{u}_0$ and $max\{s_0{\mathrm{\Delta }}_n,s_0^{1/2}{k}_0^{1/2}{\lambda }_0\}<1$. Then there exist positive constants C = C(a₀, c₀, u₀) and A = A(a₀, c₀, u₀), such that for any η > 1, by taking $\lambda =C\eta max\{s_0^{1/2}{\mathrm{\Delta }}_n,k_0^{1/2}{\lambda }_0\}{[R({\mathbf{x}}_{{\lambda }_0}^{\ast })]}^{-1/2}$,

$$\frac{R(\widehat{\mathbf{x}})}{R({\mathbf{x}}_{{\lambda }_0}^{\ast })}\ge 1-A{\eta }^{2}max\{s_0{\mathrm{\Delta }}_n,s_0^{1/2}{k}_0^{1/2}{\lambda }_0\}.$$

The main part of the proof is to show Theorem 9.1. Write for short ${\mathbf{x}}^{\ast }={\mathbf{x}}_{{\lambda }_0}^{\ast }$, R^* = R(x^*), V^* = (R^*)⁻¹= (x^*)^⊤Qx^*, V̄ ^* = (V^*)^1/2. Let ${\alpha }_n={\mathrm{\Delta }}_n{\mid {\mathbf{x}}^{\ast }\mid }_0^{1/2}$, β_n = Δ_n|x^*|₀ and $T_n({\mathbf{x}}^{\ast })=max\{s_0{\mathrm{\Delta }}_n,s_0^{1/2}{k}_0^{1/2}{\lambda }_0\}$. We define the quantity

$$\mathrm{\Gamma }(\mathbf{x})=\frac{{\mid \mathbf{Qx}-({\mathbf{x}}^{\top }\mathbf{Qx})\mathbf{q}\mid }_{\infty }}{{({\mathbf{x}}^{\top }\mathbf{Qx})}^{1/2}}\text{for}\text{any}\mathbf{x}.$$

Step 1

We introduce ${\mathbf{x}}_1^{\ast }$, a multiple of x, and use it to bound |x̂|₁.

Let Q_SS be the submatrix of Q formed by rows and columns corresponding to S. Since λ_min(Q_SS)= Θ(S, 0) ≥ c₀, we have (x^*)^⊤Qx^* ≥ c₀|x^*|². Using this fact and by the Cauchy–Schwarz inequality,

$${\mid {\mathbf{x}}^{\ast }\mid }_1\le \sqrt{{\mid {\mathbf{x}}^{\ast }\mid }_0}\mid {\mathbf{x}}^{\ast }\mid \le c_0^{-1/2}\sqrt{{\mid {\mathbf{x}}^{\ast }\mid }_0}{\overline{V}}^{\ast }.$$ (20)

It follows that

$$\mid {\widehat{\mathbf{q}}}^{\top }{\mathbf{x}}^{\ast }-{\mathbf{q}}^{\top }{\mathbf{x}}^{\ast }\mid \le {\mid \widehat{\mathbf{q}}-\mathbf{q}\mid }_{\infty }{\mid {\mathbf{x}}^{\ast }\mid }_1\le c_0^{-1/2}{\mathrm{\Delta }}_n\sqrt{{\mid {\mathbf{x}}^{\ast }\mid }_0}{\overline{V}}^{\ast }=c_0^{-1/2}{\alpha }_n{\overline{V}}^{\ast }.$$ (21)

Let t_n = q̂^⊤x^*. Then (21) says that $\mid t_n-1\mid \le c_0^{-1/2}{\alpha }_n{\overline{V}}^{\ast }$. Noting that ${\overline{V}}^{\ast }={(R^{\ast })}^{1/2}\le u_0^{-1/2}$, we have $\mid t_n-1\mid \le {(c_0{u}_0)}^{-1/2}{s}_0^{1/2}{\mathrm{\Delta }}_n<1/2$ by assumption. In particular, t_n > 0. Let

$${\mathbf{x}}_1^{\ast }=t_n^{-1}{\mathbf{x}}^{\ast }.$$

Then ${\widehat{\mathbf{q}}}^{\top }{\mathbf{x}}_1^{\ast }=1$. From the definition of x̂,

$${\widehat{\mathbf{x}}}^{\top }\widehat{\mathbf{Q}}\widehat{\mathbf{x}}+\lambda {\mid \widehat{\mathbf{x}}\mid }_1\le {({\mathbf{x}}_1^{\ast })}^{\top }\widehat{\mathbf{Q}}{\mathbf{x}}_1^{\ast }+\lambda {\mid {\mathbf{x}}_1^{\ast }\mid }_1.$$ (22)

By direct calculation,

$$\begin{array}{l}{\widehat{\mathbf{x}}}^{\top }\widehat{\mathbf{Q}}\widehat{\mathbf{x}}-{({\mathbf{x}}_1^{\ast })}^{\top }\widehat{\mathbf{Q}}{\mathbf{x}}_1^{\ast }={(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })}^{\top }\widehat{\mathbf{Q}}(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })+2{(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })}^{\top }\widehat{\mathbf{Q}}{\mathbf{x}}_1^{\ast }\\ ={(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })}^{\top }\widehat{\mathbf{Q}}(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })+2{(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })}^{\top }(\widehat{\mathbf{Q}}{\mathbf{x}}_1^{\ast }-V^{\ast }\widehat{\mathbf{q}})\\ \ge 2{(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })}^{\top }(\widehat{\mathbf{Q}}{\mathbf{x}}_1^{\ast }-V^{\ast }\widehat{\mathbf{q}}),\end{array}$$ (23)

where the second equality is due to ${\widehat{\mathbf{q}}}^{\top }\widehat{\mathbf{x}}={\widehat{\mathbf{q}}}^{\top }{\mathbf{x}}_1^{\ast }=1$. We aim to bound ${\mid \widehat{\mathbf{Q}}{\mathbf{x}}_1^{\ast }-V^{\ast }\widehat{\mathbf{q}}\mid }_{\infty }$. The following lemma is proved in the supplementary material [Fan et al. (2014)].

Lemma 9.3

When Θ(S, 0) ≥ c₀, there exists a positive constant C₁ = C₁(c₀) such that $\mathrm{\Gamma }({\mathbf{x}}_{{\lambda }_0}^{\ast })\le C_1{\lambda }_0{[max\{s_0,R({\mathbf{x}}_{{\lambda }_0}^{\ast })\}]}^{1/2}$ for any λ₀ ≥ 0.

Since ${\mathbf{x}}_1^{\ast }=t_n^{-1}{\mathbf{x}}^{\ast }$ and $t_n^{-1}<2$,

$$\begin{array}{l}{\mid \widehat{\mathbf{Q}}{\mathbf{x}}_1^{\ast }-V^{\ast }\widehat{\mathbf{q}}\mid }_{\infty }\le t_n^{-1}{\mid \widehat{\mathbf{Q}}{\mathbf{x}}^{\ast }-V^{\ast }\widehat{\mathbf{q}}\mid }_{\infty }+V^{\ast }\mid t_n^{-1}-1\mid {\mid \widehat{\mathbf{q}}\mid }_{\infty }\\ \le 2({\mid \mathbf{Q}{\mathbf{x}}^{\ast }-V^{\ast }\mathbf{q}\mid }_{\infty }+{\mid \widehat{\mathbf{Q}}-\mathbf{Q}\mid }_{\infty }{\mid {\mathbf{x}}^{\ast }\mid }_1+V^{\ast }{\mid \widehat{\mathbf{q}}-\mathbf{q}\mid }_{\infty }+V^{\ast }\mid t_n-1\mid {\mid \widehat{\mathbf{q}}\mid }_{\infty })\\ \le 2[\mathrm{\Gamma }({\mathbf{x}}^{\ast }){\overline{V}}^{\ast }+c_0^{-1/2}{\alpha }_n{\overline{V}}^{\ast }+u_0^{-1/2}{\mathrm{\Delta }}_n{\overline{V}}^{\ast }+{\mid \widehat{\mathbf{q}}\mid }_{\infty }{c}_0^{-1/2}{u}_0^{-1}{\alpha }_n{\overline{V}}^{\ast }]\\ \le C_2({\lambda }_0{k}_0^{1/2}+s_0^{1/2}{\mathrm{\Delta }}_n){\overline{V}}^{\ast }.\end{array}$$

Here the third inequality follows from (20)–(21) and $V^{\ast }={\overline{V}}^{\ast }{(R^{\ast })}^{-1/2}\le u_0^{-1/2}{\overline{V}}^{\ast }$. The last inequality is obtained as follows: from Lemma 9.2, we know that |q̂|_∞ ≤ |q|_∞ + |q̂ − q|_∞ ≤ 2C₀ (see also the assumptions in the beginning of Section 5.2); we also use Lemma 9.3 and ${\alpha }_n{\overline{V}}^{\ast }\le u_0^{-1/2}{s}_0^{1/2}{\mathrm{\Delta }}_n$. By letting C = 8C₂, the choice of $\lambda =C\eta max\{s_0^{1/2}{\mathrm{\Delta }}_n,k_0^{1/2}{\lambda }_0\}{\overline{V}}^{\ast }$ for η > 1 ensures that

$${\mid \widehat{\mathbf{Q}}{\mathbf{x}}_1^{\ast }-\widehat{\mathbf{q}}\mid }_{\infty }\le \lambda /4.$$

Plugging this result into (23) gives

$${\widehat{\mathbf{x}}}^{\top }\widehat{\mathbf{Q}}\widehat{\mathbf{x}}-{({\mathbf{x}}_1^{\ast })}^{\top }\widehat{\mathbf{Q}}{\mathbf{x}}_1^{\ast }\ge -\frac{\lambda }{2}{\mid \widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast }\mid }_1.$$ (24)

Combining (22) and (24) gives

$$\lambda {\mid \widehat{\mathbf{x}}\mid }_1-\frac{\lambda }{2}{\mid \widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast }\mid }_1\le \lambda {\mid {\mathbf{x}}_1^{\ast }\mid }_1.$$ (25)

First, since ${\mid \widehat{\mathbf{x}}\mid }_1={\mid {\widehat{\mathbf{x}}}_S\mid }_1+{\mid {\widehat{\mathbf{x}}}_{S^c}\mid }_1\ge {\mid {\mathbf{x}}_{1S}^{\ast }\mid }_1-{\mid {\widehat{\mathbf{x}}}_S-{\mathbf{x}}_{1S}^{\ast }\mid }_1+{\mid {\widehat{\mathbf{x}}}_{S^c}\mid }_1$ and ${\mid \widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast }\mid }_1={\mid {\widehat{\mathbf{x}}}_S-{\mathbf{x}}_{1S}^{\ast }\mid }_1+{\mid {\widehat{\mathbf{x}}}_{S^c}\mid }_1$, we immediately see from (25) that

$${\mid {(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })}_{S^c}\mid }_1\le 3{\mid {(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })}_S\mid }_1.$$ (26)

Second, note that ${\mid \widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast }\mid }_1\le {\mid \widehat{\mathbf{x}}\mid }_1+{\mid {\mathbf{x}}_1^{\ast }\mid }_1$. Plugging this into (25) gives

$${\mid \widehat{\mathbf{x}}\mid }_1\le 3{\mid {\mathbf{x}}_1^{\ast }\mid }_1=3t_n^{-1}{\mid {\mathbf{x}}^{\ast }\mid }_1\le 6c_0^{-1/2}\sqrt{{\mid {\mathbf{x}}^{\ast }\mid }_0}{\overline{V}}^{\ast }.$$ (27)

Step 2

We use (26)–(27) to derive an upper bound for ${(\widehat{\mathbf{x}})}^{\top }\mathbf{Q}\widehat{\mathbf{x}}-{({\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}{\mathbf{x}}_1^{\ast }$.

Note that

$$\begin{array}{l}{\widehat{\mathbf{x}}}^{\top }\widehat{\mathbf{Q}}\widehat{\mathbf{x}}-{({\mathbf{x}}_1^{\ast })}^{\top }\widehat{\mathbf{Q}}{\mathbf{x}}_1^{\ast }\ge {\widehat{\mathbf{x}}}^{\top }\mathbf{Q}\widehat{\mathbf{x}}-{({\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}{\mathbf{x}}_1^{\ast }-(\mid {\widehat{\mathbf{x}}}^{\top }\widehat{\mathbf{Q}}\widehat{\mathbf{x}}-{\widehat{\mathbf{x}}}^{\top }\mathbf{Q}\widehat{\mathbf{x}}\mid +\mid {({\mathbf{x}}_1^{\ast })}^{\top }\widehat{\mathbf{Q}}{\mathbf{x}}_1^{\ast }-{({\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}{\mathbf{x}}_1^{\ast }\mid )\\ \ge {\widehat{\mathbf{x}}}^{\top }\mathbf{Q}\widehat{\mathbf{x}}-{({\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}{\mathbf{x}}_1^{\ast }-({\mid \widehat{\mathbf{Q}}-\mathbf{Q}\mid }_{\infty }{\mid \widehat{\mathbf{x}}\mid }_1^2+{\mid \widehat{\mathbf{Q}}-\mathbf{Q}\mid }_{\infty }{\mid {\mathbf{x}}_1^{\ast }\mid }_1^2)\\ \ge {\widehat{\mathbf{x}}}^{\top }\mathbf{Q}\widehat{\mathbf{x}}-{({\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}{\mathbf{x}}_1^{\ast }-10t_n^{-2}{\mid \widehat{\mathbf{Q}}-\mathbf{Q}\mid }_{\infty }{\mid {\mathbf{x}}^{\ast }\mid }_1^2\\ \ge {\widehat{\mathbf{x}}}^{\top }\mathbf{Q}\widehat{\mathbf{x}}-{({\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}{\mathbf{x}}_1^{\ast }-C_3{\beta }_n{V}^{\ast },\end{array}$$ (28)

where the last two inequalities are direct results of (27). Combining (22) and (28),

$${\widehat{\mathbf{x}}}^{\top }\mathbf{Q}\widehat{\mathbf{x}}+\lambda {\mid \widehat{\mathbf{x}}\mid }_1\le {({\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}{\mathbf{x}}_1^{\ast }+\lambda {\mid {\mathbf{x}}_1^{\ast }\mid }_1+C_3{\beta }_n{V}^{\ast }.$$ (29)

Similar to (23), we have

$${\widehat{\mathbf{x}}}^{\top }\mathbf{Q}\widehat{\mathbf{x}}-{({\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}{\mathbf{x}}_1^{\ast }={(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })+2{(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })}^{\top }(\mathbf{Q}{\mathbf{x}}_1^{\ast }-V^{\ast }\widehat{\mathbf{q}}),$$ (30)

where

$$\begin{array}{l}{\mid \mathbf{Q}{\mathbf{x}}_1^{\ast }-V^{\ast }\widehat{\mathbf{q}}\mid }_{\infty }\le t_n^{-1}({\mid \mathbf{Q}{\mathbf{x}}^{\ast }-V^{\ast }\mathbf{q}\mid }_{\infty }+V^{\ast }{\mid \widehat{\mathbf{q}}-\mathbf{q}\mid }_{\infty })+V^{\ast }{\mid t_n^{-1}-1\mid \mid \widehat{\mathbf{q}}\mid }_{\infty }\\ \le 2[\mathrm{\Gamma }({\mathbf{x}}^{\ast }){\overline{V}}^{\ast }+u_0^{-1/2}{\mathrm{\Delta }}_n{\overline{V}}^{\ast }+{\mid \widehat{\mathbf{q}}\mid }_{\infty }{c}_0^{-1/2}{u}_0^{-1}{\alpha }_n{\overline{V}}^{\ast }]\\ \le \lambda /4.\end{array}$$

It follows that

$${\widehat{\mathbf{x}}}^{\top }\mathbf{Q}\widehat{\mathbf{x}}-{({\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}{\mathbf{x}}_1^{\ast }\ge {(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })-\frac{\lambda }{2}{\mid \widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast }\mid }_1.$$

Plugging this into (29), we obtain

$${(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })+\lambda {\mid \widehat{\mathbf{x}}\mid }_1-\frac{\lambda }{2}{\mid \widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast }\mid }_1\le \lambda {\mid {\mathbf{x}}_1^{\ast }\mid }_1+C_3{\beta }_n{V}^{\ast }.$$ (31)

We can rewrite the second and third terms on the left-hand side of (31) as

$$\lambda {\mid {\widehat{\mathbf{x}}}_S\mid }_1-\frac{\lambda }{2}{\mid {\widehat{\mathbf{x}}}_S-{\mathbf{x}}_{1S}^{\ast }\mid }_1+\frac{\lambda }{2}{\mid {\widehat{\mathbf{x}}}_{S^c}\mid }_1.$$

Plugging this into (31) and by the triangular inequality ${\mid {\mathbf{x}}_{1S}^{\ast }\mid }_1-{\mid {\widehat{\mathbf{x}}}_S\mid }_1\le {\mid {\widehat{\mathbf{x}}}_S-{\mathbf{x}}_{1S}^{\ast }\mid }_1$, we find that

$${(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })+\frac{\lambda }{2}{\mid {\widehat{\mathbf{x}}}_{S^c}\mid }_1\le \frac{3\lambda }{2}{\mid {\widehat{\mathbf{x}}}_S-{\mathbf{x}}_{1S}^{\ast }\mid }_1+C_3{\beta }_n{V}^{\ast }.$$

We drop the term $\frac{\lambda }{2}{\mid {\widehat{\mathbf{x}}}_{S^c}\mid }_1$ on the left-hand side and apply the Cauchy–Schwarz inequality to the term ${\mid {\widehat{\mathbf{x}}}_S-{\mathbf{x}}_{1S}^{\ast }\mid }_1$. This gives

$${(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })\le \frac{3\lambda }{2}\sqrt{{\mid {\mathbf{x}}_1^{\ast }\mid }_0}\mid {\widehat{\mathbf{x}}}_{1S}-{\mathbf{x}}_{1S}^{\ast }\mid +C_3{\beta }_n{V}^{\ast }.$$ (32)

Since (26) holds, by the definition of Θ(S, 3),

$${(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })\ge a_0{\mid {\widehat{\mathbf{x}}}_S-{\mathbf{x}}_{1S}^{\ast }\mid }^2.$$

We write temporarily $Y={(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })$ and b = C₃β_nV^*. Combining these with (32),

$$Y\le \frac{3\lambda }{2\sqrt{a_0}}\sqrt{{\mid {\mathbf{x}}_1^{\ast }\mid }_{0}Y}+b.$$

Note that when u ≤ au + b, we have ${(u-\frac{a}{2})}^2\le b+\frac{a^2}{4}$, and hence $u^2\le 2[\frac{a^2}{4}+{(u-\frac{a}{2})}^2]\le a^2+2b$. As a result, the above inequality implies

$${(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })\le \frac{9{\lambda }^2}{4a_0}{\mid {\mathbf{x}}^{\ast }\mid }_0+2C_3{\beta }_n{V}^{\ast },$$ (33)

where we have used ${\mid {\mathbf{x}}_1^{\ast }\mid }_0={\mid {\mathbf{x}}^{\ast }\mid }_0$. Furthermore, (30) yields that

$$\begin{array}{l}{\widehat{\mathbf{x}}}^{\top }\mathbf{Q}\widehat{\mathbf{x}}-{({\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}{\mathbf{x}}_1^{\ast }\le {(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })+\frac{\lambda }{2}{\mid \widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast }\mid }_1\\ \le {(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })+2\lambda {\mid {\mathbf{x}}_1^{\ast }\mid }_1\\ \le {(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}(\widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast })+4c_0^{-1/2}{\overline{V}}^{\ast }\lambda \sqrt{{\mid {\mathbf{x}}^{\ast }\mid }_0},\end{array}$$ (34)

where the second inequality is due to ${\mid \widehat{\mathbf{x}}-{\mathbf{x}}_1^{\ast }\mid }_1\le {\mid \widehat{\mathbf{x}}\mid }_1+\mid {\mathbf{x}}_1^{\ast }\mid \le 4{\mid {\mathbf{x}}_1^{\ast }\mid }_1$, and the last inequality is from (27). Recall that $\lambda =C\eta max\{k_0^{1/2}{\lambda }_0,s_0^{1/2}{\mathrm{\Delta }}_n\}{\overline{V}}^{\ast }$. As a result,

$$\lambda \sqrt{{\mid {\mathbf{x}}^{\ast }\mid }_0}=C\eta max\{k_0^{1/2}{s}_0^{1/2}{\lambda }_0,s_0{\mathrm{\Delta }}_n\}{\overline{V}}^{\ast }=C\eta T_n({\mathbf{x}}^{\ast }){\overline{V}}^{\ast }.$$ (35)

Combining (33), (34) and (35) gives

$$\begin{array}{l}{\widehat{\mathbf{x}}}^{\top }\mathbf{Q}\widehat{\mathbf{x}}-{({\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}{\mathbf{x}}_1^{\ast }\le \frac{9C^2}{4a_0}{\eta }^2{[T_n({\mathbf{x}}^{\ast })]}^2{V}^{\ast }+4C{c}_0^{-1/2}\eta T_n({\mathbf{x}}^{\ast })V^{\ast }+2C_3{\beta }_n{V}^{\ast }\\ \le C_4{\eta }^2{T}_n({\mathbf{x}}^{\ast })V^{\ast }.\end{array}$$ (36)

Step 3

We use (36) to give a lower bound of R(x̂).

Note that R(x̂) = (q^⊤x̂)²/(x̂^⊤Qx̂). First, we look at the denominator x̂^⊤Qx̂. From (21) and that t_n > 1/2,

$$\mid t_n^{-2}-1\mid =t_n^{-1}(1+t_n^{-1})\mid t_n-1\mid \le 6c_0^{-1/2}{\alpha }_n{\overline{V}}^{\ast }.$$

Combining with (36) and noting that ${({\mathbf{x}}_1^{\ast })}^{\top }\mathbf{Q}{\mathbf{x}}_1^{\ast }=t_n^{-2}{({\mathbf{x}}^{\ast })}^{\top }\mathbf{Q}{\mathbf{x}}^{\ast }=t_n^{-2}{V}^{\ast }$, we have

$$\begin{array}{l}{\widehat{\mathbf{x}}}^{\top }\mathbf{Q}\widehat{\mathbf{x}}\le [t_n^{-2}+C_4{\eta }^2{T}_n({\mathbf{x}}^{\ast })]{({\mathbf{x}}^{\ast })}^{\top }\mathbf{Q}{\mathbf{x}}^{\ast }\\ \le [1+6c_0^{-1/2}{\alpha }_n{\overline{V}}^{\ast }+C_4{\eta }^2{T}_n({\mathbf{x}}^{\ast })]{({\mathbf{x}}^{\ast })}^{\top }\mathbf{Q}{\mathbf{x}}^{\ast }\\ \le [1+C_5{\eta }^2{T}_n({\mathbf{x}}^{\ast })]{({\mathbf{x}}^{\ast })}^{\top }\mathbf{Q}{\mathbf{x}}^{\ast }.\end{array}$$ (37)

Second, we look at the numerator q^⊤x̂. Since q̂^⊤x̂ = 1, by (27),

$$\mid {\mathbf{q}}^{\top }\widehat{\mathbf{x}}-1\mid \le {\mid \widehat{\mathbf{q}}-\mathbf{q}\mid }_{\infty }{\mid \widehat{\mathbf{x}}\mid }_1\le 6c_0^{-1/2}{\alpha }_n{\overline{V}}^{\ast }\le C_6{T}_n({\mathbf{x}}^{\ast }).$$ (38)

Combining (37) and (38) gives

$$\begin{array}{l}R(\widehat{\mathbf{x}})=\frac{{({\mathbf{q}}^{\top }\widehat{\mathbf{x}})}^2}{{\widehat{\mathbf{x}}}^{\top }\mathbf{Q}\widehat{\mathbf{x}}}\ge \frac{{[1-C_6{T}_n({\mathbf{x}}^{\ast })]}^2}{1+C_5{\eta }^2{T}_n({\mathbf{x}}^{\ast })}\frac{1}{{({\mathbf{x}}^{\ast })}^{\top }\mathbf{Q}{\mathbf{x}}^{\ast }}\\ \ge [1-A{\eta }^2{T}_n({\mathbf{x}}^{\ast })]\frac{{({\mathbf{q}}^{\top }{\mathbf{x}}^{\ast })}^2}{{({\mathbf{x}}^{\ast })}^{\top }\mathbf{Q}{\mathbf{x}}^{\ast }}\\ =[1-A{\eta }^2{T}_n({\mathbf{x}}^{\ast })]R({\mathbf{x}}^{\ast }),\end{array}$$ (39)

where A = A(a₀, c₀, u₀) is a positive constant.

9.2. Proof of Proposition 6.1

Denote by ℙ(i|j) the probability that a new sample from class j is misclassified to class i, for i, j ∈ {1, 2} and i ≠ j. The classification error of h is

$$\text{err}(h)=\pi \mathbb{P}(2\mid 1)+(1-\pi )\mathbb{P}(1\mid 2).$$

Write M_k = M_k(Ω, δ) and L_k = L_k(Ω, δ) for short. It suffices to show that

$$\begin{array}{l}\mathbb{P}(2\mid 1)=\overline{\mathrm{\Phi }}\left(\frac{(1-t)M}{\sqrt{L_1}}\right)+\frac{O(q)+o(d)}{L_1^{3/2}},\\ \mathbb{P}(1\mid 2)=\overline{\mathrm{\Phi }}\left(\frac{tM}{\sqrt{L_2}}\right)+\frac{O(q)+o(d)}{L_2^{3/2}}.\end{array}$$

We only consider ℙ(2|1). The analysis of ℙ(1|2) is similar. Suppose $\mathbf{X}\mid \text{class}1\stackrel{(d)}{=}\mathbf{Z}~\mathcal{N}({\mathit{\mu }}_1,{\mathbf{\sum }}_1)$. Define

$$\mathbf{Y}={\mathbf{\sum }}_1^{-1/2}(\mathbf{Z}-{\mathit{\mu }}_1),$$

so that Y ~ 𝒩(0, I_d) and $\mathbf{Z}={\mathbf{\sum }}_1^{1/2}\mathbf{Y}+{\mathit{\mu }}_1$. Note that

$$\begin{array}{l}Q(\mathbf{Z})={({\mathbf{\sum }}_1^{1/2}\mathbf{Y}+{\mathit{\mu }}_1)}^{\top }\mathbf{\Omega }({\mathbf{\sum }}_1^{1/2}\mathbf{Y}+{\mathit{\mu }}_1)-2{({\mathbf{\sum }}_1^{1/2}\mathbf{Y}+{\mathit{\mu }}_1)}^{\top }\mathit{\delta }\\ ={\mathbf{Y}}^{\top }{\mathbf{\sum }}_1^{1/2}\mathbf{\Omega }{\mathbf{\sum }}_1^{1/2}\mathbf{Y}+2{\mathbf{Y}}^{\top }{\mathbf{\sum }}_1^{1/2}(\mathbf{\Omega }{\mathit{\mu }}_1-\mathit{\delta })+{\mathit{\mu }}_1^{\top }\mathbf{\Omega }{\mathit{\mu }}_1-2{\mathit{\mu }}_1^{\top }\mathit{\delta }.\end{array}$$ (40)

Recall that ${\mathbf{\sum }}_1^{1/2}\mathbf{\Omega }{\mathbf{\sum }}_1^{1/2}={\mathbf{K}}_1{\mathbf{S}}_1{\mathbf{K}}_1^{\top }$ is the eigen-decomposition by excluding the 0 eigenvalues. Since Σ₁ has full rank and the rank of Ω is q, the rank of ${\mathbf{\sum }}_1^{1/2}\mathbf{\Omega }{\mathbf{\sum }}_1^{1/2}$ is q. Therefore, S₁ is a q × q diagonal matrix, and K₁ is a d × q matrix satisfying ${\mathbf{K}}_1^{\top }{\mathbf{K}}_1={\mathbf{I}}_q$. Let K̃₁ be any d × (d − q) matrix such that K = [K₁, K̃₁] is a d × d orthogonal matrix. Since ${\mathbf{I}}_d=\mathbf{K}{\mathbf{K}}^{\top }={\mathbf{K}}_1{\mathbf{K}}_1^{\top }+{\stackrel{\sim }{\mathbf{K}}}_1{\stackrel{\sim }{\mathbf{K}}}_1^{\top }$, we have

$${\mathbf{Y}}^{\top }{\mathbf{\sum }}_1^{1/2}(\mathbf{\Omega }{\mathit{\mu }}_1-\mathit{\delta })={\mathbf{Y}}^{\top }{\mathbf{K}}_1{\mathbf{K}}_1^{\top }{\mathbf{\sum }}_1^{1/2}(\mathbf{\Omega }{\mathit{\mu }}_1-\mathit{\delta })+{\mathbf{Y}}^{\top }{\stackrel{\sim }{\mathbf{K}}}_1{\stackrel{\sim }{\mathbf{K}}}_1^{\top }{\mathbf{\sum }}_1^{1/2}(\mathbf{\Omega }{\mathit{\mu }}_1-\mathit{\delta }).$$

We recall that ${\mathit{\beta }}_1={\mathbf{K}}_1^{\top }{\mathbf{\sum }}_1^{1/2}(\mathbf{\Omega }{\mathit{\mu }}_1-\mathit{\beta })$. Let ${\stackrel{\sim }{\mathit{\beta }}}_1={\stackrel{\sim }{\mathbf{K}}}_1^{\top }{\mathbf{\sum }}_1^{1/2}(\mathbf{\Omega }{\mathit{\mu }}_1-\mathit{\delta }),\mathbf{W}={\mathbf{K}}_1^{\top }\mathbf{Y},\stackrel{\sim }{\mathbf{W}}={\stackrel{\sim }{\mathbf{K}}}_1^{\top }\mathbf{Y}$ and $c_1={\mathit{\mu }}_1^{\top }\mathbf{\Omega }{\mathit{\mu }}_1-2{\mathit{\mu }}_1^{\top }\mathit{\delta }$. It follows from (40) that

$$\begin{array}{l}Q(\mathbf{Z})={\mathbf{Y}}^{\top }{\mathbf{K}}_1{\mathbf{S}}_1{\mathbf{K}}_1^{\top }\mathbf{Y}+2{\mathbf{Y}}^{\top }{\mathbf{K}}_1{\mathit{\beta }}_1+2{\mathbf{Y}}^{\top }{\stackrel{\sim }{\mathbf{K}}}_1{\stackrel{\sim }{\mathit{\beta }}}_1+c_1\\ ={\mathbf{W}}^{\top }{\mathbf{S}}_1\mathbf{W}+2{\mathbf{W}}^{\top }{\mathit{\beta }}_1+2{\stackrel{\sim }{\mathbf{W}}}^{\top }{\stackrel{\sim }{\mathit{\beta }}}_1+c_1\\ \equiv {\overline{Q}}_1(\mathbf{W})+{\overline{F}}_1(\stackrel{\sim }{\mathbf{W}})+c_1,\end{array}$$

where Q̄₁(w) = w^⊤S₁w +2w^⊤β₁ and F̄₁(w) = 2w^⊤β̃₁. Therefore,

$$\mathbb{P}(2\mid 1)=\mathbb{P}(Q(\mathbf{Z})>c)=\mathbb{P}({\overline{Q}}_1(\mathbf{W})+{\overline{F}}_1(\stackrel{\sim }{\mathbf{W}})>c-c_1).$$

We write for convenience W = (W₁, …, W_q)^⊤, W̃= (W_q+1, …, W_d)^⊤, β₁ = (β₁₁, …, β_1q) and β̃₁ = (β_1(q+1), …, β_1d)^⊤, and notice that $W_i\stackrel{\text{i.i.d.}}{\sim }N(0,1)$ for 1 ≤ i≤ d. Moreover,

$${\overline{Q}}_1(\mathbf{W})+{\overline{F}}_1(\stackrel{\sim }{\mathbf{W}})=\sum _{i=1}^q(s_i{W}_i^2+2W_i{\beta }_{1i})+\sum _{i=q+1}^{d}2W_i{\beta }_{1i}\equiv \sum _{i=1}^d{\xi }_i,$$ (41)

where ${\xi }_i=s_i{W}_i^{2}I\{1\le i\le q\}+2W_i{\beta }_{1i}$, for 1 ≤ i ≤ d. The right-hand side of (41) is a sum of independent variables, so we can apply the Edgeworth expansion to its distribution function, as described in detail below.

Note that $\mathbb{E}(W_i^2)=1,\mathbb{E}(W_i^4)=3,E(W_i^6)=15$ and $\mathbb{E}(W_i^{2j+1})=0$ for nonnegative integers j. By direct calculation,

$$\begin{array}{l}{\eta }_1\equiv \sum _{i=1}^d\mathbb{E}({\xi }_i)=\sum _{i=1}^q{s}_i=\text{tr}({\mathbf{S}}_1)=\text{tr}(\mathbf{\Omega }{\mathbf{\sum }}_1),\\ {\eta }_2\equiv \sum _{i=1}^{d}var({\xi }_i)=\sum _{i=1}^q(2s_i^2+4{\beta }_{1i}^2)+\sum _{i=q+1}^{d}4{\beta }_{1i}^2=2\text{tr}({\mathbf{S}}_1^2)+4{\mid {\mathit{\beta }}_1\mid }^2+4{\mid {\stackrel{\sim }{\mathit{\beta }}}_1\mid }^2\\ =2\text{tr}(\mathbf{\Omega }{\mathbf{\sum }}_1\mathbf{\Omega }{\mathbf{\sum }}_1)+4{(\mathbf{\Omega }{\mathit{\mu }}_1-\mathit{\delta })}^{\top }{\mathbf{\sum }}_1(\mathbf{\Omega }{\mathit{\mu }}_1-\mathit{\delta }),\\ {\eta }_3=\sum _{i=1}^d\mathbb{E}{[{\xi }_i-\mathbb{E}({\xi }_i)]}^3=\sum _{i=1}^d(8s_i^3+24{\beta }_{1i}^2{s}_i)\\ =8\text{tr}({\mathbf{S}}_1^3)+24{\mathit{\beta }}_1^{\top }{\mathbf{S}}_1{\mathit{\beta }}_1=8\text{tr}[{(\mathbf{\Omega }{\mathbf{\sum }}_1)}^3]+24{(\mathbf{\Omega }{\mathit{\mu }}_1-\mathit{\delta })}^{\top }{\mathbf{\sum }}_1\mathbf{\Omega }{\mathbf{\sum }}_1(\mathbf{\Omega }{\mathit{\mu }}_1-\mathit{\delta }).\end{array}$$

Notice that 𝔼(|ξ_i − 𝔼(ξ_i)|³) < ∞, as max{|s_i |, |β_1i|, 1 ≤ i ≤ d}≤ C₀ by assumption. Using results from Chapter XVI of Feller (1966), we know

$$\begin{array}{l}\mathbb{P}(2\mid 1)=\mathbb{P}\left(\sum _{i=1}^d{\xi }_i>c-c_1\right)\\ =\mathbb{P}\left(\frac{{\sum }_{i=1}^d{\xi }_i-\mathbb{E}({\sum }_{i=1}^d{\xi }_i)}{\sqrt{{\sum }_{i=1}^{d}var({\xi }_i)}}>\frac{c-c_1-\mathbb{E}({\sum }_{i=1}^d{\xi }_i)}{\sqrt{{\sum }_{i=1}^{d}var({\xi }_i)}}\right)\\ =\overline{\mathrm{\Phi }}\left(\frac{c-c_1-{\eta }_1}{\sqrt{{\eta }_2}}\right)+\frac{{\eta }_3(1-({(c_1-c+{\eta }_1)}^2/{\eta }_2))}{6{\eta }_2^{3/2}}\phi \left(\frac{c_1-c+{\eta }_1}{\sqrt{{\eta }_2}}\right)+o\left(\frac{d}{{\eta }_2^{3/2}}\right),\end{array}$$

where φ is the probability density function of the standard normal distribution. It is observed that η₂ = L₁(Ω, δ) and c₁ + η₁ = M₁(Ω, δ). Also, c = tM₁(Ω, δ) + (1 − t)M₂(Ω, δ). As a result,

$$\begin{array}{l}\frac{c-c_1-{\eta }_1}{\sqrt{{\eta }_2}}=\frac{[t{M}_1+(1-t)M_2]-M_1}{\sqrt{L_1}}=\frac{(1-t)(M_2-M_1)}{\sqrt{L_1}}\\ =(1-t)\frac{M}{\sqrt{L_1}}.\end{array}$$

Plugging this into the expression of ℙ(2|1), the first term is $\overline{\mathrm{\Phi }}((1-t)\frac{M}{\sqrt{L_1}})$. Moreover, since the function (1 − u₂)φ(u) is uniformly bounded, the second term is $O(\frac{{\eta }_3}{{\eta }_2^{3/2}})$. Here η₂ = L₁, and η₃ = O(q) as s_i’s and β_1i’s are abounded in magnitude. Combining the above gives

$$\mathbb{P}(2\mid 1)=\overline{\mathrm{\Phi }}\left(\frac{(1-t)M}{\sqrt{L_1}}\right)+\frac{O(q)+o(d)}{L_1^{3/2}}.$$

The proof is now complete.