Moments and Root-Mean-Square Error of the Bayesian MMSE Estimator of Classification Error in the Gaussian Model

Abstract

The most important aspect of any classifier is its error rate, because this quantifies its predictive capacity. Thus, the accuracy of error estimation is critical. Error estimation is problematic in small-sample classifier design because the error must be estimated using the same data from which the classifier has been designed. Use of prior knowledge, in the form of a prior distribution on an uncertainty class of feature-label distributions to which the true, but unknown, feature-distribution belongs, can facilitate accurate error estimation (in the mean-square sense) in circumstances where accurate completely model-free error estimation is impossible. This paper provides analytic asymptotically exact finite-sample approximations for various performance metrics of the resulting Bayesian Minimum Mean-Square-Error (MMSE) error estimator in the case of linear discriminant analysis (LDA) in the multivariate Gaussian model. These performance metrics include the first, second, and cross moments of the Bayesian MMSE error estimator with the true error of LDA, and therefore, the Root-Mean-Square (RMS) error of the estimator. We lay down the theoretical groundwork for Kolmogorov double-asymptotics in a Bayesian setting, which enables us to derive asymptotic expressions of the desired performance metrics. From these we produce analytic finite-sample approximations and demonstrate their accuracy via numerical examples. Various examples illustrate the behavior of these approximations and their use in determining the necessary sample size to achieve a desired RMS. The Supplementary Material contains derivations for some equations and added figures.

1. Introduction

The most important aspect of any classifier is its error, ε, defined as the probability of misclassification, since ε quantifies the predictive capacity of the classifier. Relative to a classification rule and a given feature-label distribution, the error is a function of the sampling distribution and as such possesses its own distribution, which characterizes the true performance of the classification rule. In practice, the error must be estimated from data by some error estimation rule yielding an estimate, ε̂. If samples are large, then part of the data can be held out for error estimation; otherwise, the classification and error estimation rules are applied on the same set of training data, which is the situation that concerns us here. Like the true error, the estimated error is also a function of the sampling distribution. The performance of the error estimation rule is completely described by its joint distribution, (ε, ε̂).

Three widely-used metrics for performance of an error estimator are the bias, deviation variance, and root-mean-square (RMS), given by

$$\begin{array}{l}\text{Bias}[\widehat{\epsilon }]=E[\widehat{\epsilon }]-E[\epsilon ],\\ {\text{Var}}^d[\widehat{\epsilon }]=\text{Var}(\widehat{\epsilon }-\epsilon )=\text{Var}(\epsilon )+\text{Var}(\widehat{\epsilon })-2\text{Cov}(\epsilon ,\widehat{\epsilon }),\\ \text{RMS}[\widehat{\epsilon }]=\sqrt{E[{(\epsilon -\widehat{\epsilon })}^2]}=\sqrt{E[{\epsilon }^2]+E[{\widehat{\epsilon }}^2]-2E[\epsilon \widehat{\epsilon }]}=\sqrt{\text{Bias}{[\widehat{\epsilon }]}^2+{\text{Var}}^d[\widehat{\epsilon }]},\end{array}$$ (1)

respectively. The RMS (square root of mean square error, MSE) is the most important because it quantifies estimation accuracy. Bias requires only the first-order moments, whereas the deviation variance and RMS require also the second-order moments.

Historically, analytic study has mainly focused on the first marginal moment of the estimated error for linear discriminant analysis (LDA) in the Gaussian model or for multinomial discrimination [1]–[12]; however, marginal knowledge does not provide the joint probabilistic knowledge required for assessing estimation accuracy, in particular, the mixed second moment. Recent work has aimed at characterizing joint behavior. For multinomial discrimination, exact representations of the second-order moments, both marginal and mixed, for the true error and the resubstitution and leave-one-out estimators have been obtained [13]. For LDA, the exact joint distributions for both resubstitution and leave-one-out have been found in the univariate Gaussian model and approximations have been found in the multivariate model with a common known covariance matrix [14, 15]. Whereas one could utilize the approximate representations to find approximate moments via integration in the multivariate model with a common known covariance matrix, more accurate approximations, including the second-order mixed moment and the RMS, can be achieved via asymptotically exact analytic expressions using a double asymptotic approach, where both sample size (n) and dimensionality (p) approach infinity at a fixed rate between the two [16]. Finite-sample approximations from the double asymptotic method have shown to be quite accurate [16, 17, 18]. There is quite a body of work on the use of double asymptotics for the analysis of LDA and its related statistics [16, 19, 20, 21, 22, 23]. Raudys and Young provide a good review of the literature on the subject [24].

Although the theoretical underpinning of both [16] and the present paper relies on double asymptotic expansions, in which n, p → ∞ at a proportional rate, our practical interest is in the finite-sample approximations corresponding to the asymptotic expansions. In [17], the accuracy of such finite-sample approximations was investigated relative to asymptotic expansions for the expected error of LDA in a Gaussian model. Several single-asymptotic expansions (n → ∞) were considered, along with double-asymptotic expansions (n, p → ∞) [19, 20]. The results of [17] show that the double-asymptotic approximations are significantly more accurate than the single-asymptotic approximations. In particular, even with n/p < 3, the double-asymptotic expansions yield “excellent approximations” while the others “falter.”

The aforementioned work is based on the assumption that a sample is drawn from a fixed feature-label distribution F, a classifier and error estimate are derived from the sample without using any knowledge concerning F, and performance is relative to F. Research dating to 1978, shows that small-sample error estimation under this paradigm tends to be inaccurate. Re-sampling methods such as cross-validation possess large deviation variance and, therefore, large RMS [9, 25]. Scientific content in the context of small-sample classification can be facilitated by prior knowledge [26, 27, 28]. There are three possibilities regarding the feature-label distribution: (1) F is known, in which case no data are needed and there is no error estimation issue; (2) nothing is known about F, there are no known RMS bounds, or those that are known are useless for small samples; and (3) F is known to belong to an uncertainty class of distributions and this knowledge can be used to either bound the RMS [16] or be used in conjunction with the training data to estimate the error of the designed classifier. If there exists a prior distribution governing the uncertainty class, then in essence we have a distributional model. Since virtually nothing can be said about the error estimate in the first two cases, for a classifier to possess scientific content we must begin with a distributional model.

Given the need for a distributional model, a natural approach is to find an optimal minimum mean-square-error (MMSE) error estimator relative to an uncertainty class Θ [27]. This results in a Bayesian approach with Θ being given a prior distribution, π(θ), θ ∈ Θ, and the sample S_n being used to construct a posterior distribution, π^*(θ), from which an optimal MMSE error estimator, ε̂^B, can be derived. π(θ) provides a mathematical framework for both the analysis of any error estimator and the design of estimators with desirable properties or optimal performance. π^*(θ) provides a sample-conditioned distribution on the true classifier error, where randomness in the true error comes from uncertainty in the underlying feature-label distribution (given S_n). Finding the sample-conditioned MSE, MSE_θ[ε̂^B|S_n], of an MMSE error estimator amounts to evaluating the variance of the true error conditioned on the observed sample [29]. MSE_θ[ε̂^B|S_n] → 0 as n → ∞ almost surely in both the discrete and Gaussian models provided in [29, 30], where closed form expressions for the sample-conditioned MSE are available.

The sample-conditioned MSE provides a measure of performance across the uncertainty class Θ for a given sample S_n. As such, it involves various sample-conditioned moments for the error estimator: E_θ[ε̂^B|S_n], E_θ[(ε̂^B)²|S_n], and E_θ[εε̂^B|S_n]. One could, on the other hand, consider the MSE relative to a fixed feature-label distribution in the uncertainty class and randomness relative to the sampling distribution. This would yield the feature-label-distribution-conditioned MSE, MSE _Sn[ε̂^B|θ], and the corresponding moments: E_Sn[ε̂^B|θ], E_Sn[(ε̂^B)²|θ], and E_Sn[εε̂^B|θ]. From a classical point of view, the moments given θ are of interest as they help shed light on the performance of an estimator relative to fixed parameters of class conditional densities. Using this set of moments (i.e. given θ) we are able to compare the performance of the Bayesian MMSE error estimator to classical estimators of true error such as resubstitution and leave-one-out.

From a global perspective, to evaluate performance across both the uncertainty class and the sampling distribution requires the unconditioned MSE, MSE_θSn[ε̂^B], and corresponding moments E_θSn[ε̂^B], E_θSn[(ε̂^B)²], and E_θSn[εε̂^B]. While both MSE_Sn[ε̂^B|θ] and MSE_θSn[ε̂^B] have been examined via simulation studies in [27, 28, 30] for discrete and Gaussian models, our intention in the present paper is to derive double-asymptotic representations of the feature-labeled conditioned (given θ) and unconditioned MSE, along with the corresponding moments of the Bayesian MMSE error estimator for linear discriminant analysis (LDA) in the Gaussian model.

We make three modeling assumptions. As in many analytic error analysis studies, we employ stratified sampling: n = n₀ + n₁ sample points are selected to constitute the sample S_n in R^p, where given n, n₀ and n₁ are determined, and where x₁, x₂, …, x_n0 and x_n0+1, x_n0+2, …, x_n0+n1 are randomly selected from distributions Π₀ and Π₁, respectively. Π_i possesses a multivariate Gaussian distribution N(μ_i, Σ), for i = 0, 1. This means that the prior probabilities α₀ and α₁ = 1 − α₀ for classes 0 and 1, respectively, cannot be estimated from the sample (see [31] for a discussion of issues surrounding lack of an estimator for α₀). However, our second assumption is that α₀ and α₁ are known. This is a natural assumption for many medical classification problems. If we desire early or mid-term detection, then we are typically constrained to a small sample for which n₀ and n₁ are not random but for which α₀ and α₁ are known (estimated with extreme accuracy) on account of a large population of post-mortem examinations. The third assumption is that there is a known common covariance matrix for the classes, a common assumption in error analysis [32, 3, 5, 16]. The common covariance assumption is typical for small samples because it is well-known that LDA commonly performs better that quadratic discriminant analysis (QDA) for small samples, even if the actual covariances are different, owing to the estimation advantage of using the pooled sample covariance matrix. As for the assumption of known covariance, this assumption is typical simply owing to the mathematical difficulties of obtaining error representations for unknown covariance (we know of no unknown-covariance result for second-order representations). Indeed, the natural next step following this paper and [16] is to address the unknown covariance problem (although with it being outstanding for almost half a century, it may prove difficult).

Under our assumptions, the Anderson W statistic is defined by

$$W({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{x})={\left(\mathbf{x}-\frac{{\overline{\mathbf{x}}}_0+{\overline{\mathbf{x}}}_1}{2}\right)}^T{\mathbf{\sum }}^{-1}({\overline{\mathbf{x}}}_0-{\overline{\mathbf{x}}}_1),$$ (2)

where ${\overline{\mathbf{x}}}_0=\frac{1}{n_0}{\sum }_{i=1}^{n_0}{\mathbf{x}}_i$ and ${\overline{\mathbf{x}}}_1=\frac{1}{n_1}{\sum }_{i=n_0+1}^{n_0+n_1}{\mathbf{x}}_i$. The corresponding linear discriminant is defined by ψ_n(x) = 1 if W(x̄₀, x̄₁, x) ≤ c and ψ_n(x) = 0 if W(x̄₀, x̄₁, x) > c, where $c=log\frac{1-{\alpha }_0}{{\alpha }_0}$. Given sample S_n (and thus x̄₀ and x̄₁), for i = 0, 1, the error for ψ_n is given by ε = α₀ε₀ + α₁ε₁, where

$${\epsilon }_i=\mathrm{\Phi }\left(\frac{{(-1)}^{i+1}{({\mathit{\mu }}_i-\frac{{\overline{\mathbf{x}}}_0+{\overline{\mathbf{x}}}_1}{2})}^T{\mathbf{\sum }}^{-1}({\overline{\mathbf{x}}}_0-{\overline{\mathbf{x}}}_1)+{(-1)}^{i}c}{\sqrt{{({\overline{\mathbf{x}}}_0-{\overline{\mathbf{x}}}_1)}^T{\mathbf{\sum }}^{-1}({\overline{\mathbf{x}}}_0-{\overline{\mathbf{x}}}_1)}}\right)$$ (3)

and Φ(.) denotes the cumulative distribution function of a standard normal random variable.

Raudys proposed the following approximation to the expected LDA classification error [19, 24]:

$$E_{S_n}[{\epsilon }_0]=P(W({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{x})\le c\mid \mathbf{x}\in {\mathrm{\Pi }}_0)\eqsim \mathrm{\Phi }\left(\frac{-E_{S_n}[W({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{x})\mid \mathbf{x}\in {\mathrm{\Pi }}_0]+c}{\sqrt{{\text{Var}}_{S_n}[W({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{x})\mid \mathbf{x}\in {\mathrm{\Pi }}_0]}}\right)$$ (4)

We provide similar approximations for error-estimation moments and prove asymptotic exactness.

2. Bayesian MMSE Error Estimator

In the Bayesian classification framework [27, 28], it is assumed that the class-0 an class-1 conditional distributions are parameterized by θ₀ and θ₁, respectively. Therefore, assuming known α_i, the actual feature-label distribution belongs to an uncertainty class parameterized by θ = (θ₀, θ₁) according to a prior distribution, π(θ). Given a sample S_n, the Bayesian MMSE error estimator minimizes the MSE between the true error of a designed classifier, ψ_n, and an error estimate (a function of S_n and ψ_n). The expectation in the MSE is taken over the uncertainty class conditioned on S_n. Specifically, the MMSE error estimator is the expected true error, ε̂^B(S_n) = E_θ[ε(θ)|S_n]. The expectation given the sample is over the posterior density, π*(θ). Thus, we write the Bayesian MMSE error estimator as ε̂^B = E_π*[ε]. To facilitate analytic representations, θ₀ and θ₁ are assumed to be independent prior to observing the data. Denote the marginal priors of θ₀ and θ₁ by π(θ₀) and π(θ₁), respectively, and the corresponding posteriors by π*(θ₀) and π*(θ₁), respectively. Independence is preserved, i.e., π*(θ₀, θ₁) = π*(θ₀)π*(θ₁) for i = 0, 1 [27].

Owing to the posterior independence between θ₀ and θ₁ and the fact that α_i is known, the Bayesian MMSE error estimator can be expressed by [27]

$${\widehat{\epsilon }}^B={\alpha }_0{\mathrm{E}}_{{\pi }^{\ast }}[{\epsilon }_0]+{\alpha }_1{\mathrm{E}}_{{\pi }^{\ast }}[{\epsilon }_1]={\alpha }_0{\widehat{\epsilon }}_0^B+{\alpha }_1{\widehat{\epsilon }}_1^B,$$ (5)

where, letting Θ_i be the parameter space of θ_i,

$${\widehat{\epsilon }}_i^B={\mathrm{E}}_{{\pi }^{\ast }}[{\epsilon }_i]={\int }_{{\mathbf{\Theta }}_i}{\epsilon }_i({\theta }_i){\pi }^{\ast }({\theta }_i)d{\theta }_i.$$ (6)

For known Σ and the prior distribution on μ_i assumed to be Gaussian with mean m_i and covariance matrix Σ/ν_I, ${\widehat{\epsilon }}_i^B$ is given by equation (10) in [28]:

$${\widehat{\epsilon }}_i^B=\mathrm{\Phi }\left({(-1)}^i\frac{-{\left({\mathbf{m}}_i^{\ast }-\frac{{\overline{\mathbf{x}}}_0+{\overline{\mathbf{x}}}_1}{2}\right)}^T{\mathbf{\sum }}^{-1}({\overline{\mathbf{x}}}_0-{\overline{\mathbf{x}}}_1)+c}{\sqrt{{({\overline{\mathbf{x}}}_0-{\overline{\mathbf{x}}}_1)}^T{\mathbf{\sum }}^{-1}({\overline{\mathbf{x}}}_0-{\overline{\mathbf{x}}}_1)}}\sqrt{\frac{{\nu }_i^{\ast }}{{\nu }_i^{\ast }+1}}\right),$$ (7)

where

$${\mathbf{m}}_i^{\ast }=\frac{n_i{\overline{\mathbf{x}}}_i+{\nu }_i{\mathbf{m}}_i}{n_i+{\nu }_i},{\nu }_i^{\ast }=n_i+{\nu }_i.$$ (8)

and ν_i > 0 is a measure of our certainty concerning the prior knowedge – the larger ν_i is the more localized the prior distribution is about m_i. Letting $\mathit{\mu }={[{\mathit{\mu }}_0^T,{\mathit{\mu }}_1^T]}^T$, the moments that interest us are of the form E_Sn[ε̂^B|μ], E_Sn[(ε̂^B)²|μ], and E_Sn[εε̂^B|μ], which are used to obtain MSE_Sn[ε̂^B|μ], and E_μ,Sn[ε̂^B], E_μ,Sn[(ε̂^B)²], and E_μ,Sn[εε̂^B], which are needed to characterize MSE_μ,Sn[ε̂^B].

3. Bayesian-Kolmogorov Asymptotic Conditions

The Raudys-Kolmogorov asymptotic conditions [16] are defined on a sequence of Gaussian discrimination problems with a sequence of parameters and sample sizes: (μ_p,0, μ_p,1, Σ_p, n_p,0, n_p,1), p = 1, 2, …, where the means and the covariance matrix are arbitrary. The common assumptions for Raudys-Kolmogorov asymptotics are n₀ → ∞, n₁ → ∞, p → ∞, $\frac{p}{n_0}\to J_0<\infty ,\frac{p}{n_1}\to J<\infty$. For notational simplicity, we denote the limit under these conditions by $\underset{p\to \infty }{lim}$. In the analysis of classical statistics related to LDA it is commonly assumed that the Mahalanobis distance, ${\delta }_{\mathit{\mu },p}=\sqrt{{({\mathit{\mu }}_{p,0}-{\mathit{\mu }}_{p,1})}^T{\mathbf{\sum }}_p^{-1}({\mathit{\mu }}_{p,0}-{\mathit{\mu }}_{p,1})}$, is finite and $\underset{p\to \infty }{lim}{\delta }_{\mathit{\mu },p}={\overline{\delta }}_{\mathit{\mu }}$ (see [22], p. 4). This condition assures existence of limits of performance metrics of the relevant statistics [16, 22].

To analyze the Bayesian MMSE error estimator, ${\widehat{\epsilon }}_i^B$, we modify the sequence of Gaussian discrimination problems to:

$$({\mathit{\mu }}_{p,0},{\mathit{\mu }}_{p,1},{\mathbf{\sum }}_p,n_{p,0},n_{p,1},{\mathbf{m}}_{p,0},{\mathbf{m}}_{p,1},{\nu }_{p,0},{\nu }_{p,1}),p=1,2,\dots$$ (9)

In addition to the previous conditions, we assume that the following limits exist for i, j = 0, 1: $\underset{p\to \infty }{lim}{\mathbf{m}}_{p,i}^T{\mathbf{\sum }}_p^{-1}{\mathit{\mu }}_{p,j}=\overline{{\mathbf{m}}_i^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_j},\underset{p\to \infty }{lim}{\mathbf{m}}_{p,i}^T{\mathbf{\sum }}_p^{-1}{\mathbf{m}}_{p,j}=\overline{{\mathbf{m}}_i^T{\mathbf{\sum }}^{-1}{\mathbf{m}}_j}$, and $\underset{p\to \infty }{lim}{\mathit{\mu }}_{p,i}^T{\mathbf{\sum }}_p^{-1}{\mathit{\mu }}_{p,j}=\overline{{\mathit{\mu }}_i^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_j}$, where $\overline{{\mathbf{m}}_i^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_j},\overline{{\mathbf{m}}_i^T{\mathbf{\sum }}^{-1}{\mathbf{m}}_j}$, and $\overline{{\mathit{\mu }}_i^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_j}$ are some constants to which the limits converge. In [22], fairly mild sufficient conditions are given for the existence of these limits.

We refer to all of the aforementioned conditions, along with ν_i → ∞, $\frac{{\nu }_i}{n_i}\to {\gamma }_i<\infty$, as the Bayesian-Kolmogorov asymptotic conditions (b.k.a.c). We denote the limit under these conditions by lim_b.k.a.c., which means that, for i, j = 0, 1,

$$\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}(·)=\underset{\begin{array}{c}p\to \infty ,n_i\to \infty ,{\nu }_i\to \infty \\ \frac{p}{n_0}\to J_0,\frac{p}{n_1}\to J_1,\frac{{\nu }_0}{n_0}\to {\gamma }_0,\frac{{\nu }_1}{n_1}\to {\gamma }_1\\ {\gamma }_i<\infty ,J_i<\infty \\ {\mathbf{m}}_{p,i}^T{\mathbf{\sum }}_p^{-1}{\mathit{\mu }}_{p,j}=O(1),{\mathbf{m}}_{p,i}^T{\mathbf{\sum }}_p^{-1}{\mathit{\mu }}_{p,j}\to \overline{{\mathbf{m}}_i^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_j}\\ {\mathbf{m}}_{p,i}^T{\mathbf{\sum }}_p^{-1}{\mathbf{m}}_{p,j}=O(1),{\mathbf{m}}_{p,i}^T{\mathbf{\sum }}_p^{-1}{\mathbf{m}}_{p,j}\to \overline{{\mathbf{m}}_i^T{\mathbf{\sum }}^{-1}{\mathbf{m}}_j}\\ {\mathit{\mu }}_{p,i}^T{\mathbf{\sum }}_p^{-1}{\mathit{\mu }}_{p,j}=O(1),{\mathit{\mu }}_{p,i}^T{\mathbf{\sum }}_p^{-1}{\mathit{\mu }}_{p,j}\to \overline{{\mathit{\mu }}_i^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_j}\end{array}}{lim}(·)$$ (10)

This limit is defined for the case where there is conditioning on a specific value of μ_p,i. Therefore, in this case μ_p,i is not a random variable, and for each p, it is a vector of constants. Absent such conditioning, the sequence of discrimination problems and the above limit reduce to

$$({\mathbf{\sum }}_p,n_{p,0},n_{p,1},{\mathbf{m}}_{p,0},{\mathbf{m}}_{p,1},{\nu }_{p,0},{\nu }_{p,1}),p=1,2,\dots$$ (11)

and

$$\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}(·)=\underset{\begin{array}{c}p\to \infty ,n_i\to \infty ,{\nu }_i\to \infty \\ \frac{p}{n_0}\to J_0,\frac{p}{n_1}\to J_1,\frac{{\nu }_0}{n_0}\to {\gamma }_0,\frac{{\nu }_1}{n_1}\to {\gamma }_1\\ {\gamma }_i<\infty ,J_i<\infty \\ {\mathbf{m}}_{p,i}^T{\mathbf{\sum }}_p^{-1}{\mathbf{m}}_{p,j}=O(1),{\mathbf{m}}_{p,i}^T{\mathbf{\sum }}_p^{-1}{\mathbf{m}}_{p,j}\to \overline{{\mathbf{m}}_i^T{\mathbf{\sum }}^{-1}{\mathbf{m}}_j}\end{array}}{lim}(·)$$ (12)

respectively. For notational simplicity we assume clarity from the context and do not explicitly differentiate between these conditions. We denote convergence in probability under Bayesian-Kolmogorov asymptotic conditions by “ $\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{\text{plim}}$”.“ $\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}$” and “ $\stackrel{K}{\to }$” denote ordinary convergence under Bayesian-Kolmogorov asymptotic conditions. At no risk of ambiguity, we henceforth omit the subscript “p” from the parameters and sample sizes in (9) or (11).

We define η_a1,a2,a3,a4 = (a₁ − a₂)^T Σ⁻¹(a₃ − a₄) and, for ease of notation write η_a1,a2,a1,a2 as η_a1,a2. There are two special cases: (1) the square of the Mahalanobis distance in the space of the parameters of the unknown class conditional densities, ${\delta }_{\mathit{\mu }}^2={\eta }_{{\mathit{\mu }}_0,{\mathit{\mu }}_1}>0$; and (2) the square of the Mahalanobis distance in the space of prior knowledge, ${\mathrm{\Delta }}_{\mathbf{m}}^2={\eta }_{{\mathbf{m}}_0,{\mathbf{m}}_1}>0$, where $\mathbf{m}={[{\mathbf{m}}_0^T,{\mathbf{m}}_1^T]}^T$. The conditions in (10) assure existence of lim_b.k.a.c η_a1,a2,a3,a4, where the a_j’s can be any combination of m_i and μ_i, i = 0, 1. Consistent with our notations, we use η̄_a1,a2,a3,a4, ${\overline{\delta }}_{\mathit{\mu }}^2$, and ${\overline{\mathrm{\Delta }}}_{\mathbf{m}}^2$ to denote the lim_b.k.a.c of η_a1,a2,a3,a4, ${\delta }_{\mathit{\mu }}^2$, and ${\mathrm{\Delta }}_{\mathbf{m}}^2$, respectively. Thus,

$${\overline{\eta }}_{{\mathbf{a}}_1,{\mathbf{a}}_2,{\mathbf{a}}_3,{\mathbf{a}}_4}=\overline{{({\mathbf{a}}_1-{\mathbf{a}}_2)}^T{\mathbf{\sum }}^{-1}({\mathbf{a}}_3-{\mathbf{a}}_4)}=\overline{{\mathbf{a}}_1^T{\mathbf{\sum }}^{-1}{\mathbf{a}}_3}-\overline{{\mathbf{a}}_1^T{\mathbf{\sum }}^{-1}{\mathbf{a}}_4}-\overline{{\mathbf{a}}_2^T{\mathbf{\sum }}^{-1}{\mathbf{a}}_3}+\overline{{\mathbf{a}}_2^T{\mathbf{\sum }}^{-1}{\mathbf{a}}_4}.$$ (13)

The ratio p/n_i is an indicator of complexity for LDA (in fact, any linear classification rule): the VC dimension in this case is p + 1 [33]. Therefore, the conditions (10) assure the existence of the asymptotic complexity of the problem. The ratio ν_i/n_i is an indicator of relative certainty of prior knowledge to the data: the smaller ν_i/n_i, the more we rely on the data and less on our prior knowledge. Therefore, the conditions (10) state asymptotic existence of relative certainty. In the following, we let ${\beta }_i=\frac{v_i}{n_i}$, so that ${\beta }_i=\frac{{\nu }_i}{n_i}\to {\gamma }_i$.

4. First Moment of ${\widehat{\mathit{\epsilon }}}_{\mathbf{i}}^{\mathbf{B}}$

In this section we use the Bayesian-Kolmogorov asymptotic conditions to characterize the conditional and unconditional first moment of the Bayesian MMSE error estimator.

4.1. Conditional Expectation of ${\widehat{\epsilon }}_i^B:E_{S_n}[{\widehat{\epsilon }}_i^B\mid \mathit{\mu }]$

The asymptotic (in a Bayesian-Kolmogorov sense) conditional expectation of the Bayesian MMSE error estimator is characterized in the following theorem, with the proof presented in the Appendix. Note that $G_0^B,G_1^B$, and D depend on μ, but to ease the notation we leave this implicit.

Theorem 1

Consider the sequence of Gaussian discrimination problems defined by (9). Then

$$\underset{b.k.a.c.}{lim}{E}_{S_n}[{\widehat{\epsilon }}_i^B\mid \mathit{\mu }]=\mathrm{\Phi }\left({(-1)}^i\frac{-G_i^B+c}{\sqrt{D}}\right),$$ (14)

so that

$$\underset{b.k.a.c.}{lim}{E}_{S_n}[{\widehat{\epsilon }}^B\mid \mathit{\mu }]={\alpha }_0\mathrm{\Phi }\left(\frac{-G_0^B+c}{\sqrt{D}}\right)+{\alpha }_1\mathrm{\Phi }\left(\frac{G_1^B-c}{\sqrt{D}}\right),$$ (15)

where

$$\begin{array}{l}{G}_0^B=\frac{1}{2(1+{\gamma }_0)}\left({\gamma }_0({\overline{\eta }}_{{\mathbf{m}}_0,{\mathit{\mu }}_1}-{\overline{\eta }}_{{\mathbf{m}}_0,{\mathit{\mu }}_0})+{\overline{\delta }}_{\mathit{\mu }}^2+(1-{\gamma }_0)J_0+(1+{\gamma }_0)J_1\right),\\ G_1^B=\frac{-1}{2(1+{\gamma }_1)}\left({\gamma }_1({\overline{\eta }}_{{\mathbf{m}}_1,{\mathit{\mu }}_0}-{\overline{\eta }}_{{\mathbf{m}}_1,{\mathit{\mu }}_1})+{\overline{\delta }}_{\mathit{\mu }}^2+(1-{\gamma }_1)J_1+(1+{\gamma }_1)J_0\right)\\ D={\overline{\delta }}_{\mu }^2+J_0+J_1.\end{array}$$ (16)

Theorem 1 suggests a finite-sample approximation:

$$E_{S_n}[{\widehat{\epsilon }}_0^B\mid \mathit{\mu }]\eqsim \mathrm{\Phi }\left(\frac{-G_0^{B,f}+c}{\sqrt{{\delta }_{\mathit{\mu }}^2+\frac{p}{n_0}+\frac{p}{n_1}}}\right),$$ (17)

where $G_0^{B,f}$ is obtained by using the finite-sample parameters of the problem in (16), namely,

$$G_0^{B,f}=\frac{1}{2(1+{\beta }_0)}\left({\beta }_0({\eta }_{{\mathbf{m}}_0,{\mathit{\mu }}_1}-{\eta }_{{\mathbf{m}}_0,{\mathit{\mu }}_0})+{\delta }_{\mathit{\mu }}^2+(1-{\beta }_0)\frac{p}{n_0}+(1+{\beta }_0)\frac{p}{n_1}\right).$$ (18)

To obtain the corresponding approximation for $E_{S_n}[{\widehat{\epsilon }}_1^B\mid \mathit{\mu }]$, it suffices to use (17) by exchanging n₀ and n₁, ν₀ and ν₁, m₀ and m₁, and μ₀ and μ₁ in $-G_0^{B,f}$.

To obtain a Raudys-type of finite-sample approximation for the expectation of ${\widehat{\epsilon }}_0^B$, first note that the Gaussian distribution in (7) can be rewritten as

$${\widehat{\epsilon }}_0^B=P(U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\le c\mid {\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z}\in {\mathrm{\Psi }}_0,\mathit{\mu }),$$ (19)

where z is independent of S_n, Ψ_i is a multivariate Gaussian $N({\mathbf{m}}_i,\frac{(n_i+{\nu }_i+1)(n_i+{\nu }_i)}{{\nu }_i^2}\mathbf{\sum })$, and

$$U_i({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})={\left(\frac{{\nu }_i}{n_i+{\nu }_i}\mathbf{z}+\frac{n_i{\overline{\mathbf{x}}}_i}{n_i+{\nu }_i}-\frac{{\overline{\mathbf{x}}}_0+{\overline{\mathbf{x}}}_1}{2}\right)}^T{\mathbf{\sum }}^{-1}({\overline{\mathbf{x}}}_0-{\overline{\mathbf{x}}}_1).$$ (20)

Taking the expectation of ${\widehat{\epsilon }}_0^B$ relative to the sampling distribution and then applying the standard normal approximation yields the Raudys-type of approximation:

$$E_{S_n}[{\widehat{\epsilon }}_0^B\mid \mathit{\mu }]=P(U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\le c\mid \mathbf{z}\in {\mathrm{\Psi }}_0,\mathit{\mu })\eqsim \mathrm{\Phi }\left(\frac{-E_{S_n,\mathbf{z}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\mid \mathbf{z}\in {\mathrm{\Psi }}_0,\mathit{\mu }]+c}{\sqrt{{\text{Var}}_{S_n,\mathbf{z}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\mid \mathbf{z}\in {\mathrm{\Psi }}_0,\mathit{\mu }]}}\right).$$ (21)

Algebraic manipulation yields (Suppl. Section A)

$$E_{S_n}[{\widehat{\epsilon }}_0^B\mid \mathit{\mu }]\eqsim \mathrm{\Phi }\left(\frac{-G_0^{B,R}+c}{\sqrt{D_0^{B,R}}}\right),$$ (22)

where

$$G_0^{B,R}=G_0^{B,f},$$ (23)

with $G_0^{B,f}$ being presented in (18) and

$$\begin{array}{l}{D}_0^{B,R}={\delta }_{\mathit{\mu }}^2+\frac{{\delta }_{\mathit{\mu }}^2}{n_0(1+{\beta }_0)}+\frac{{\delta }_{\mathit{\mu }}^2}{n_1(1+{\beta }_0)}+\frac{{\delta }_{\mathit{\mu }}^2}{n_0{(1+{\beta }_0)}^2}\\ +\frac{{\beta }_0}{{(1+{\beta }_0)}^2}\left[\frac{n_{{\mathbf{m}}_0,{\mathit{\mu }}_1}-(1-{\beta }_0){\eta }_{{\mathbf{m}}_0,{\mathit{\mu }}_0}-{\delta }_{\mathit{\mu }}^2}{n_0}+\frac{(1+{\beta }_0){\eta }_{{\mathbf{m}}_0,{\mathit{\mu }}_1}-{\eta }_{{\mathbf{m}}_0,{\mathit{\mu }}_0}}{n_1}\right]+\frac{p}{n_0}+\frac{p}{n_1}+\frac{p}{n_0^2(1+{\beta }_0)}\\ +\frac{p}{n_0{n}_1(1+{\beta }_0)}+\frac{{(1-{\beta }_0)}^{2}p}{2n_0^2{(1+{\beta }_0)}^2}+\frac{p}{n_0{n}_1{(1+{\beta }_0)}^2}+\frac{p}{2n_1^2}.\end{array}$$ (24)

The corresponding approximation for $E_{S_n}[{\widehat{\epsilon }}_1^B\mid \mathit{\mu }]$ is

$$E_{S_n}[{\widehat{\epsilon }}_1^B\mid \mathit{\mu }]\eqsim \mathrm{\Phi }\left(\frac{G_1^{B,R}-c}{\sqrt{D_1^{B,R}}}\right),$$ (25)

where $D_1^{B,R}$ and $G_1^{B,R}$ are obtained by exchanging n₀ and n₁, ν₀ and ν₁, m₀ and m₁, and μ₀ and μ₁ in $D_0^{B,R}$ and $-G_0^{B,R}$, respectively. It is straightforward to see that

$$\begin{array}{l}{G}_0^{B,R}\stackrel{K}{\to }{G}_0^B,\\ D_0^{B,R}\stackrel{K}{\to }{\overline{\delta }}_{\mathit{\mu }}^2+J_0+J_1,\end{array}$$ (26)

with $G_0^B$ being defined in Theorem 1. Therefore, the approximation obtained in (22) is asymptotically exact and (17) and (22) are asymptotically equivalent.

4.2. Unconditional Expectation of ${\widehat{\epsilon }}_i^B:E_{\mathit{\mu },S_n}\left[{\widehat{\epsilon }}_i^B\right]$

We consider the unconditional expectation of ${\widehat{\epsilon }}_i^B$ under Bayesian-Kolmogorov asymptotics. The proof of the following theorem is presented in the Appendix.

Theorem 2

Consider the sequence of Gaussian discrimination problems defined by (11). Then

$$\underset{b.k.a.c.}{lim}{E}_{\mathit{\mu },S_n}\left[{\widehat{\epsilon }}_i^B\right]=\mathrm{\Phi }\left({(-1)}^i\frac{-H_i+c}{\sqrt{F}}\right),$$ (27)

so that

$$\underset{b.k.a.c.}{lim}{E}_{\mathit{\mu },S_n}\left[{\widehat{\epsilon }}_i^B\right]={\alpha }_0\mathrm{\Phi }\left(\frac{-H_0+c}{\sqrt{F}}\right)+{\alpha }_1\mathrm{\Phi }\left(\frac{H_1-c}{\sqrt{F}}\right),$$ (28)

where

$$\begin{array}{l}{H}_0=\frac{1}{2}\left({\overline{\mathrm{\Delta }}}_{\mathbf{m}}^2+J_1-J_0+\frac{J_0}{{\gamma }_0}+\frac{J_1}{{\gamma }_1}\right),\\ H_1=-\frac{1}{2}\left({\overline{\mathrm{\Delta }}}_{\mathbf{m}}^2+J_0-J_1+\frac{J_0}{{\gamma }_0}+\frac{J_1}{{\gamma }_1}\right),\\ F={\overline{\mathrm{\Delta }}}_{\mathbf{m}}^2+J_0+J_1+\frac{J_0}{{\gamma }_0}+\frac{J_1}{{\gamma }_1}.\end{array}$$ (29)

Theorem 2 suggests the finite-sample approximation

$$E_{\mathit{\mu },S_n}\left[{\widehat{\epsilon }}_0^B\right]\eqsim \mathrm{\Phi }\left(\frac{-H_0^R+c}{\sqrt{{\mathrm{\Delta }}_{\mathbf{m}}^2+\frac{p}{n_0}+\frac{p}{n_1}+\frac{p}{{\nu }_0}+\frac{p}{{\nu }_1}}}\right),$$ (30)

where

$$H_0^R=\frac{1}{2}\left({\mathrm{\Delta }}_{\mathbf{m}}^2+\frac{p}{n_1}-\frac{p}{n_0}+\frac{p}{{\nu }_0}+\frac{p}{{\nu }_1}\right).$$ (31)

From (19) we can get the Raudys-type approximation:

$$E_{\mathit{\mu },S_n}[{\widehat{\epsilon }}_0^B]=E_{\mathit{\mu }}[P(U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\le c\mid \mathbf{z}\in {\mathrm{\Psi }}_0,\mathit{\mu })]\eqsim \mathrm{\Phi }\left(\frac{-E_{\mathit{\mu },S_n,\mathbf{z}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\mid z\in {\mathrm{\Psi }}_0]+c}{\sqrt{{\text{Var}}_{\mathit{\mu },S_n,\mathbf{z}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\mid \mathbf{z}\in {\mathrm{\Psi }}_0]}}\right).$$ (32)

Some algebraic manipulations yield (Suppl. Section B)

$$E_{\mathit{\mu },S_n}[{\widehat{\epsilon }}_0^B]\eqsim \mathrm{\Phi }\left(\frac{-H_0^R+c}{\sqrt{F_0^R}}\right),$$ (33)

where

$$F_0^R=\left(1+\frac{1}{{\nu }_0}+\frac{1}{{\nu }_1}+\frac{1}{n_1}\right){\mathrm{\Delta }}_{\mathbf{m}}^2+p\left(\frac{1}{n_0}+\frac{1}{n_1}+\frac{1}{{\nu }_0}+\frac{1}{{\nu }_1}\right)+p\left(\frac{1}{2n_0^2}+\frac{1}{2n_1^2}+\frac{1}{2{\nu }_0^2}+\frac{1}{2{\nu }_1^2}\right)+p\left(\frac{1}{n_1{\nu }_0}+\frac{1}{{\eta }_1{\nu }_1}+\frac{1}{{\nu }_0{\nu }_1}\right).$$ (34)

It is straightforward to see that

$$\begin{array}{l}{H}_0^R\stackrel{K}{\to }{H}_0,\\ F_0^R\stackrel{K}{\to }{\overline{\mathrm{\Delta }}}_{\mathbf{m}}^2+J_0+J_1+\frac{J_0}{{\gamma }_0}+\frac{J_1}{{\gamma }_1},\end{array}$$ (35)

with H₀ defined in Theorem 2. Hence, the approximation obtained in (33) is asymptotically exact and both (30) and (33) are asymptotically equivalent.

5. Second Moments of ${\widehat{\mathit{\epsilon }}}_{\mathbf{i}}^{\mathbf{B}}$

Here we employ the Bayesian-Kolmogorov asymptotic analysis to characterize the second and cross moments with the actual error, and therefore the MSE of error estimation.

5.1. Conditional Second and Cross Moments of ${\widehat{\epsilon }}_i^B$

Defining two i.i.d. random vectors, z and z′, yields the second moment representation

$$\begin{array}{l}{E}_{S_n}[{({\widehat{\epsilon }}_0^B)}^2\mid \mathit{\mu }]=E_{S_n}[P{(U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\le c\mid {\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z}\in {\mathrm{\Psi }}_0,\mathit{\mu })}^2]\\ =E_{S_n}\left[P(U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\le c\mid {\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z}\in {\mathrm{\Psi }}_0,\mathit{\mu })P(U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,{\mathbf{z}}^{\prime })\le c\mid {\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,{\mathbf{z}}^{\prime }\in {\mathrm{\Psi }}_0,\mathit{\mu })\right]\\ =E_{S_n}\left[P\left(U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\le c,U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,{\mathbf{z}}^{\prime })\le c\mid {\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z}\in {\mathrm{\Psi }}_0,{\mathbf{z}}^{\prime }\in {\mathrm{\Psi }}_0,\mathit{\mu }\right)\right]\\ =P\left(U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\le c,U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,{\mathbf{z}}^{\prime })\le c\mid \mathbf{z}\in {\mathrm{\Psi }}_0,{\mathbf{z}}^{\prime }\in {\mathrm{\Psi }}_0,\mathit{\mu }\right),\end{array}$$ (36)

where z and z′ are independent of S_n, and Ψ_i is a multivariate Gaussian, $N({\mathbf{m}}_i,\frac{(n_i+{\nu }_i+1)(n_i+{\nu }_i)}{{\nu }_i^2}\mathbf{\sum })$, and U_i(x̄₀, x̄₁, z) being defined in (20).

We have the following theorem, with the proof presented in the Appendix.

Theorem 3

For the sequence of Gaussian discrimination problems in (9) and for i, j = 0, 1,

$$\underset{b.k.a.c.}{lim}{E}_{S_n}[{\widehat{\epsilon }}_i^B{\widehat{\epsilon }}_j^B\mid \mathit{\mu }]=\mathrm{\Phi }\left({(-1)}^i\frac{-G_i^B+c}{\sqrt{D}}\right)\mathrm{\Phi }\left({(-1)}^j\frac{-G_j^B+c}{\sqrt{D}}\right),$$ (37)

so that

$$\underset{b.k.a.c.}{lim}{E}_{S_n}[{({\widehat{\epsilon }}^B)}^2\mid \mathit{\mu }]={\left[{\alpha }_0\mathrm{\Phi }\left(\frac{-G_0^B+c}{\sqrt{D}}\right)+{\alpha }_1\mathrm{\Phi }\left(\frac{G_1^B-c}{\sqrt{D}}\right)\right]}^2,$$ (38)

where $G_0^B,G_1^B$, and D are defined in (16).

This theorem suggests the finite-sample approximation

$$E_{S_n}[{({\widehat{\epsilon }}_0^B)}^2\mid \mathit{\mu }]\eqsim {\left[\mathrm{\Phi }\left(\frac{-G_0^{B,f}+c}{\sqrt{{\delta }_{\mathit{\mu }}^2+\frac{p}{n_0}+\frac{p}{n_1}}}\right)\right]}^2,$$ (39)

which is the square of the approximation (17). Corresponding approximations for $E[{\widehat{\epsilon }}_0^B{\widehat{\epsilon }}_1^B]$ and $E[{({\widehat{\epsilon }}_1^B)}^2]$ are obtained similarly.

Similar to the proof of Theorem 3, we obtain the conditional cross moment of

ε̂^B.

Theorem 4

Consider the sequence of Gaussian discrimination problems in (9). Then for i, j = 0, 1,

$$\underset{b.k.a.c.}{lim}{E}_{S_n}[{\widehat{\epsilon }}_i^B{\epsilon }_j\mid \mathit{\mu }]=\mathrm{\Phi }\left({(-1)}^i\frac{-G_i^B+c}{\sqrt{D}}\right)\mathrm{\Phi }\left({(-1)}^j\frac{-G_j+c}{\sqrt{D}}\right),$$ (40)

so that

$$\underset{b.k.a.c.}{lim}{E}_{S_n}[{\widehat{\epsilon }}^B\epsilon \mid \mathit{\mu }]=\sum _{i=0}^1\sum _{j=0}^1\left[{\alpha }_i{\alpha }_j\mathrm{\Phi }\left({(-1)}^i\frac{-G_i^B+c}{\sqrt{D}}\right)\mathrm{\Phi }\left({(-1)}^j\frac{-G_j+c}{\sqrt{D}}\right)\right],$$ (41)

where $G_i^B$ and D are defined in (16) and G_i is defined in (47).

This theorem suggests the finite-sample approximation

$$E_{S_n}[{\widehat{\epsilon }}_0^B{\epsilon }_0\mid \mathit{\mu }]\eqsim \mathrm{\Phi }\left(\frac{-G_0^{B,f}+c}{\sqrt{{\delta }_{\mathit{\mu }}^2+\frac{p}{n_0}+\frac{p}{n_1}}}\right)\mathrm{\Phi }\left(-\frac{1}{2}\frac{{\delta }_{\mathit{\mu }}^2+\frac{p}{n_1}-\frac{p}{n_0}-c}{\sqrt{{\delta }_{\mathit{\mu }}^2+\frac{p}{n_0}+\frac{p}{n_1}}}\right).$$ (42)

This is a product of (17) and the finite-sample approximation for E_Sn[ε₀|μ] in [16].

A consequence of Theorems 1, 3, and 4 is that all the conditional variances and covariances are asymptotically zero:

$$\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{\text{Var}}_{S_n}({\widehat{\epsilon }}^B\mid \mathit{\mu })=\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{\text{Var}}_{S_n}(\epsilon \mid \mathit{\mu })=\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{\text{Cov}}_{S_n}(\epsilon ,{\widehat{\epsilon }}^B\mid \mathit{\mu })=0.$$ (43)

Hence, the deviation variance is also asymptotically zero, lim_b.k.a.c. ${\text{Var}}_{S_n}^d[{\widehat{\epsilon }}^B\mid \mathit{\mu }]=0$. Hence, defining the conditional bias as

$${\text{Bias}}_{C,n}[{\widehat{\epsilon }}^B]=E_{S_n}[{\widehat{\epsilon }}^B-\epsilon \mid \mathit{\mu }],$$ (44)

the asymptotic RMS reduces to

$$\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{\text{RMS}}_{S_n}[{\widehat{\epsilon }}^B\mid \mathit{\mu }]=\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}\mid {\text{Bias}}_{C,n}[{\widehat{\epsilon }}^B]\mid .$$ (45)

To express the conditional bias, as proven in [16],

$$\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{E}_{S_n}[\epsilon \mid \mathit{\mu }]={\alpha }_0\mathrm{\Phi }\left(\frac{-G_0+c}{\sqrt{D}}\right)+{\alpha }_1\mathrm{\Phi }\left(\frac{G_1-c}{\sqrt{D}}\right),$$ (46)

where

$$\begin{array}{l}{G}_0=\frac{1}{2}({\delta }_{\mathit{\mu }}^2+J_1-J_0),\\ G_1=-\frac{1}{2}({\delta }_{\mathit{\mu }}^2+J_0-J_1),\\ D={\delta }_{\mathit{\mu }}^2+J_0+J_1.\end{array}$$ (47)

It follows from Theorem 1 and (46) that

$$\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{\text{Bias}}_{C,n}[{\widehat{\epsilon }}^B]={\alpha }_0\left[\mathrm{\Phi }\left(\frac{-G_0^B+c}{\sqrt{D}}\right)-\mathrm{\Phi }\left(\frac{-G_0+c}{\sqrt{D}}\right)\right]+{\alpha }_1\left[\mathrm{\Phi }\left(\frac{G_1^B+c}{\sqrt{D}}\right)-\mathrm{\Phi }\left(\frac{G_1-c}{\sqrt{D}}\right)\right].$$ (48)

Recall that the MMSE error estimator is unconditionally unbiased: Bias_U,n[ε̂^B] = E_μ,_Sn[ε̂^B − ε] = 0.

We next obtain Raudys-type approximations corresponding to Theorems 3 and 4 by utilizing the joint distribution of U₀(x̄₀, x̄₁, z) and U₀(x̄₀, x̄₁, z′), defined in (20), with z′ and z′ being independently selected from populations Ψ₀ or Ψ₁. We employ the function

$$\mathrm{\Phi }(a,b;\rho )=\underset{-\infty }{\overset{a}{\int }}\underset{-\infty }{\overset{b}{\int }}\frac{1}{2\pi \sqrt{1-{\rho }^2}}exp\left\{\frac{-(x^2+y^2-2\rho xy)}{2{(1-\rho )}^2}\right\}dxdy,$$ (49)

which is the distribution function of a joint bivariate Gaussian vector with zero means, unit variances, and correlation coefficient ρ. Note that Φ(a, ∞; ρ) = Φ(a) and Φ(a, b; 0) = Φ(a) Φ(b). For simplicity of notation, we write Φ(a, a; ρ) as Φ(a; ρ). The rectangular-area probabilities involving any jointly Gaussian pair of variables (x, y) can be expressed as

$$P(x\le c,y\le d)=\mathrm{\Phi }\left(\frac{c-{\mu }_x}{{\sigma }_x},\frac{d-{\mu }_y}{{\sigma }_y};{\rho }_{xy}\right),$$ (50)

with μ_x = E[x], μ_y = E[y], ${\sigma }_x=\sqrt{\text{Var}(x)},{\sigma }_Y=\sqrt{\text{Var}(y)}$, and correlation coefficient ρ_xy.

Using (36), we obtain the second-order extension of (21) by

$$\begin{array}{l}{E}_{S_n}[{({\widehat{\epsilon }}_0^B)}^2\mid \mathit{\mu }]=P\left(U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\le c,U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,{\mathbf{z}}^{\prime })\le c\mid \mathbf{z}\in {\mathrm{\Psi }}_0,{\mathbf{z}}^{\prime }\in {\mathrm{\Psi }}_0,\mathit{\mu }\right)\\ \eqsim \mathrm{\Phi }\left(\frac{-E_{S_n,\mathbf{z}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\mid \mathbf{z}\in {\mathrm{\Psi }}_0,\mathit{\mu }]+c}{\sqrt{{\text{Var}}_{S_n,\mathbf{z}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\mid \mathbf{z}\in {\mathrm{\Psi }}_0,\mathit{\mu }]}};\frac{{\text{Cov}}_{S_n,\mathbf{z}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z}),U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,{\mathbf{z}}^{\prime })\mid \mathbf{z}\in {\mathrm{\Psi }}_0,{\mathbf{z}}^{\prime }\in {\mathrm{\Psi }}_0,\mathit{\mu }]}{{\text{Var}}_{S_n,\mathbf{z}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\mid \mathbf{z}\in {\mathrm{\Psi }}_0,\mathit{\mu }]}\right).\end{array}$$ (51)

Using (51), some algebraic manipulations yield

$$E_{S_n}[{({\widehat{\epsilon }}_0^B)}^2\mid \mathit{\mu }]\eqsim \mathrm{\Phi }\left(\frac{-G_0^{B,R}+c}{\sqrt{D_0^{B,R}}};\frac{C_0^{B,R}}{D_0^{B,R}}\right),$$ (52)

with $G_0^{B,R}$ and $D_0^{B,R}$ being presented in (23) and (24), respectively, and

$$\begin{array}{l}{C}_0^{B,R}={\text{Cov}}_{S_n,\mathbf{z}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z}),U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,{\mathbf{z}}^{\prime })\mid \mathbf{z}\in {\mathrm{\Psi }}_0,{\mathbf{z}}^{\prime }\in {\mathrm{\Psi }}_0,\mathit{\mu }]\\ =\frac{{\beta }_0}{{(1+{\beta }_0)}^2}\left[\frac{{\eta }_{{\mathbf{m}}_0,{\mathit{\mu }}_1}-(1-{\beta }_0){\eta }_{{\mathbf{m}}_0,{\mathit{\mu }}_0}-{\delta }_{\mathit{\mu }}^2}{n_0}+\frac{(1+{\beta }_0){\eta }_{{\mathbf{m}}_0,{\mathit{\mu }}_1}-{\eta }_{{\mathbf{m}}_0,{\mathit{\mu }}_0}}{n_1}\right]+\frac{{(1-{\beta }_0)}^{2}p}{2n_0^2{(1+{\beta }_0)}^2}\\ +\frac{p}{n_0{n}_1{(1+{\beta }_0)}^2}+\frac{p}{2n_1^2}+\frac{{\delta }_{\mathit{\mu }}^2}{n_1(1+{\beta }_0)}+\frac{{\delta }_{\mathit{\mu }}^2}{n_0{(1+{\beta }_0)}^2},\end{array}$$ (53)

The proof of (53) follows by expanding U₀(x̄₀, x̄₁, z) and U₀(x̄₀, x̄₁, z′) from (20) and then using the set of identities in the proof of (33), i.e. equation (S.1) from Suppl. Section B. Similarly,

$$E_{S_n}[{({\widehat{\epsilon }}_1^B)}^2\mid \mathit{\mu }]=P(U_1({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})>c,U_1({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,{\mathbf{z}}^{\prime })>c\mid \mathbf{z}\in {\mathrm{\Psi }}_1,{\mathbf{z}}^{\prime }\in {\mathrm{\Psi }}_1,\mathit{\mu })\eqsim \mathrm{\Phi }\left(\frac{G_1^{B,R}-c}{\sqrt{D_1^{B,R}}};\frac{C_1^{B,R}}{D_1^{B,R}}\right),$$ (54)

where $D_1^{B,R},G_1^{B,R}$, and $C_1^{B,R}$ are obtained by exchanging n₀ and n₁, ν₀ and ν₁, m₀ and m₁, and μ₀ and μ₁, in (24), in $-G_0^{B,f}$ obtained from (18), and in (53), respectively.

Having $C_0^{B,R}\stackrel{K}{\to }0$ together with (26) shows that (52) is asymptotically exact, that is, asymptotically equivalent to $E_{S_n}[{({\widehat{\epsilon }}_0^B)}^2\mid \mathit{\mu }]$ obtained in Theorem 3. Similarly, it can be shown that

$$\begin{array}{l}{E}_{S_n}[{\widehat{\epsilon }}_0^B{\widehat{\epsilon }}_1^B\mid \mathit{\mu }]=P(U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\le c,-U_1({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,{\mathbf{z}}^{\prime })<-c\mid \mathbf{z}\in {\mathrm{\Psi }}_0,{\mathbf{z}}^{\prime }\in {\mathrm{\Psi }}_1,\mathit{\mu })\\ \eqsim \mathrm{\Phi }\left(\frac{-G_0^{B,R}+c}{\sqrt{D_0^{B,R}}},\frac{G_1^{B,R}-c}{\sqrt{D_1^{B,R}}};\frac{C_{01}^{B,R}}{\sqrt{D_0^{B,R}{D}_1^{B,R}}}\right),\end{array}$$ (55)

where, after some algebraic manipulations we obtain

$$\begin{array}{l}{C}_{01}^{B,R}=\frac{1}{n_0(1+{\beta }_0)(1+{\beta }_1)}\left[{\beta }_0{\eta }_{{\mathbf{m}}_0,{\mathit{\mu }}_0,{\mathit{\mu }}_0,{\mathit{\mu }}_1}-{\beta }_0{\beta }_1{\eta }_{{\mathbf{m}}_0,{\mathit{\mu }}_0,{\mathbf{m}}_1,{\mathit{\mu }}_0}+{\beta }_1{\eta }_{{\mathbf{m}}_1,{\mathit{\mu }}_1,{\mathit{\mu }}_1,{\mathit{\mu }}_0}+{\beta }_1{\delta }_{\mathit{\mu }}^2+{\delta }_{\mathit{\mu }}^2\right]\\ +\frac{1}{n_1(1+{\beta }_0)(1+{\beta }_1)}\left[{\beta }_1{\eta }_{{\mathbf{m}}_1,{\mathit{\mu }}_1,{\mathit{\mu }}_1,{\mathit{\mu }}_0}-{\beta }_0{\beta }_1{\eta }_{{\mathbf{m}}_1,{\mathit{\mu }}_1,{\mathbf{m}}_0,{\mathit{\mu }}_1}+{\beta }_0{\eta }_{{\mathbf{m}}_0,{\mathit{\mu }}_0,{\mathit{\mu }}_0,{\mathit{\mu }}_1}+{\beta }_0{\delta }_{\mathit{\mu }}^2+{\delta }_{\mathit{\mu }}^2\right]\\ +\frac{p}{n_0{n}_1(1+{\beta }_0)(1+{\beta }_1)}+\frac{(1-{\beta }_0)p}{2n_0^2(1+{\beta }_0)}+\frac{(1-{\beta }_1)p}{2n_1^2(1+{\beta }_1)}.\end{array}$$ (56)

Suppl. Section C gives the proof of (56). Since $C_{01}^{B,R}\stackrel{K}{\to }0$, (55) is asymptotically exact, i.e. (55) becomes equivalent to the result of Theorem 3. We obtain the conditional cross moment similarly:

$$\begin{array}{l}{E}_{S_n}[{\widehat{\epsilon }}_0^B{\epsilon }_0\mid \mathit{\mu }]=P(U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\le c,W({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{x})\le c\mid \mathbf{z}\in {\mathrm{\Psi }}_0,\mathbf{x}\in {\mathrm{\Pi }}_0,\mathit{\mu })\\ \eqsim \mathrm{\Phi }(\frac{-E_{S_n,\mathbf{z}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\mid \mathbf{z}\in {\mathrm{\Psi }}_0,\mathit{\mu }]+c}{\sqrt{{\text{Var}}_{S_n,\mathbf{z}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\mid \mathbf{z}\in {\mathrm{\Psi }}_0,\mathit{\mu }]}},\frac{-E_{S_n,\mathbf{x}}[W({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{x})\mid \mathbf{x}\in {\mathrm{\Pi }}_0,\mathit{\mu }]+c}{\sqrt{{\text{Var}}_{S_n,\mathbf{x}}[W({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{x})\mid \mathbf{x}\in {\mathrm{\Pi }}_0,\mathit{\mu }]}};\\ \frac{{\text{Cov}}_{S_n,\mathbf{z},\mathbf{x}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z}),W({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{x})\mid \mathbf{z}\in {\mathrm{\Psi }}_0,\mathbf{x}\in {\mathrm{\Pi }}_0,\mathit{\mu }]}{\sqrt{V_{U_0}^C{V}_W^C}}),\end{array}$$ (57)

where

$$\begin{array}{l}{V}_{U_0}^{\mathrm{C}}={\text{Var}}_{S_n,\mathbf{z}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\mid \mathbf{z}\in {\mathrm{\Psi }}_0,\mathit{\mu }],\\ V_W^{\mathrm{C}}={\text{Var}}_{S_n,\mathbf{x}}[W({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{x})\mid \mathbf{x}\in {\mathrm{\Pi }}_0,\mathit{\mu }],\end{array}$$ (58)

where superscript “C” denotes conditional variance. Algebraic manipulations like those leading to (53) yield

$$E_{S_n}[{\widehat{\epsilon }}_0^B{\epsilon }_0\mid \mathit{\mu }]\eqsim \mathrm{\Phi }\left(\frac{-G_0^{B,R}+c}{\sqrt{D_0^{B,R}}}+\frac{-G_0^R+c}{\sqrt{D_0^R}};\frac{C_0^{BT,R}}{\sqrt{D_0^{B,R}{D}_0^R}}\right),$$ (59)

where

$$C_0^{BT,R}=\frac{1}{n_1(1+{\beta }_0)}\left[{\delta }_{\mathit{\mu }}^2+{\beta }_0{\delta }_{\mathit{\mu }}^2+{\beta }_0{\eta }_{{\mathbf{m}}_0,{\mathit{\mu }}_0,{\mathit{\mu }}_0,{\mathit{\mu }}_1}\right]-\frac{(1-{\beta }_0)p}{2n_0^2(1+{\beta }_0)}+\frac{p}{2n_1^2},$$ (60)

and $G_0^R$ and $D_0^R$ having been obtained previously in equations (49) and (50) of [16], namely,

$$\begin{array}{l}{G}_0^R=E_{S_n,\mathbf{x}}[W({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{x})\mid \mathbf{x}\in {\mathrm{\Pi }}_0,\mathit{\mu }]=\frac{1}{2}\left({\delta }_{\mathit{\mu }}^2+\frac{p}{n_1}-\frac{p}{n_0}\right),\\ D_0^R={\text{Var}}_{S_n,\mathbf{z}}[W({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{x})\mid \mathbf{x}\in {\mathrm{\Pi }}_0,\mathit{\mu }]={\delta }_{\mathit{\mu }}^2+\frac{{\delta }_{\mathit{\mu }}^2}{n_1}+p\left(\frac{1}{n_0}+\frac{1}{n_1}+\frac{1}{2n_0^2}+\frac{1}{2n_1^2}\right).\end{array}$$ (61)

Similarly, we can show that

$$E_{S_n}[{\widehat{\epsilon }}_1^B{\epsilon }_1\mid \mathit{\mu }]\eqsim \mathrm{\Phi }\left(\frac{G_1^{B,R}-c}{\sqrt{D_1^{B,R}}},\frac{G_1^R-c}{\sqrt{D_1^R}};\frac{C_1^{BT,R}}{\sqrt{D_1^{B,R}{D}_1^R}}\right),$$ (62)

where $D_1^{B,R}$ and $G_1^{B,R}$ are obtained as in (54), and $D_1^R,G_1^R$, and $C_1^{BT,R}$ are obtained by exchanging n₀ and n₁ in $D_0^R,-G_0^R$, and $C_0^{BT,R}$, respectively. Similarly,

$$E_{S_n}[{\widehat{\epsilon }}_0^B{\epsilon }_1\mid \mathit{\mu }]\eqsim \mathrm{\Phi }\left(\frac{-G_0^{B,R}+c}{\sqrt{D_0^{B,R}}},\frac{G_1^R-c}{\sqrt{D_1^R}};\frac{C_{01}^{BT,R}}{\sqrt{D_0^{B,R}{D}_1^R}}\right),$$ (63)

where

$$C_{01}^{BT,R}=\frac{1}{n_0(1+{\beta }_0)}\left[{\delta }_{\mathit{\mu }}^2+{\beta }_0{\eta }_{{\mathbf{m}}_0,{\mathit{\mu }}_0,{\mathit{\mu }}_0,{\mathit{\mu }}_1}\right]+\frac{(1-{\beta }_0)p}{2n_0^2(1+{\beta }_0)}-\frac{p}{2n_1^2},$$ (64)

and

$$E_{S_n}[{\widehat{\epsilon }}_1^B{\epsilon }_0\mid \mathit{\mu }]\eqsim \mathrm{\Phi }\left(\frac{G_1^{B,R}-c}{\sqrt{D_1^{B,R}}},\frac{-G_0^R+c}{\sqrt{D_0^R}};\frac{C_{10}^{BT,R}}{\sqrt{D_1^{B,R}{D}_0^R}}\right),$$ (65)

where $C_{10}^{BT,R}$ is obtained by exchanging n₀ and n₁, ν₀ and ν₁, m₀ and m₁, and μ₀ and μ₁ in $C_{01}^{BT,R}$.

We see that $C_0^{BT,R}\stackrel{K}{\to }0,C_1^{BT,R}\stackrel{K}{\to }0$, and $C_{01}^{BT,R}\stackrel{K}{\to }0$. Therefore, from (26) and the fact that $G_0^R\stackrel{K}{\to }{\overline{\delta }}_{\mathit{\mu }}^2+J_1-J_0$ and $D_0^R\stackrel{K}{\to }{\overline{\delta }}_{\mathit{\mu }}^2+J_0+J_1$, we see that expressions (59), (62), and (63), are all asymptotically exact (compare to Theorem 4).

5.2. Unconditional Second and Cross Moments of ${\widehat{\epsilon }}_i^B$

Similarly to the way (36) was obtained, we can show that

$$\begin{array}{l}{E}_{\mathit{\mu },S_n}[{({\widehat{\epsilon }}_0^B)}^2]=E_{\mathit{\mu },S_n}[P{(U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\le c\mid {\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z}\in {\mathrm{\Psi }}_0,\mathit{\mu })}^2]\\ =E_{\mathit{\mu },S_n}\left[P\left(U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\le c,U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,{\mathbf{z}}^{\prime })\le c\mid {\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z}\in {\mathrm{\Psi }}_0,{\mathbf{z}}^{\prime }\in {\mathrm{\Psi }}_0,\mathit{\mu }\right)\right]\\ =P\left(U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\le c,U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,{\mathbf{z}}^{\prime })\le c\mid \mathbf{z}\in {\mathrm{\Psi }}_0,{\mathbf{z}}^{\prime }\in {\mathrm{\Psi }}_0\right).\end{array}$$ (66)

Similarly to the proofs of Theorem 3 and 4, we get the following theorems.

Theorem 5

Consider the sequence of Gaussian discrimination problems in (11). For i, j = 0, 1,

$$\underset{b.k.a.c.}{lim}{E}_{\mathit{\mu },S_n}[{\widehat{\epsilon }}_i^B{\widehat{\epsilon }}_j^B]=\mathrm{\Phi }\left({(-1)}^i\frac{-H_i+c}{\sqrt{F}}\right)\mathrm{\Phi }\left({(-1)}^j\frac{-H_j+c}{\sqrt{F}}\right),$$ (67)

so that

$$\underset{b.k.a.c.}{lim}{E}_{\mathit{\mu },S_n}[{({\widehat{\epsilon }}^B)}^2]={\left[{\alpha }_0\mathrm{\Phi }\left(\frac{-H_0+c}{\sqrt{F}}\right)+{\alpha }_1\mathrm{\Phi }\left(\frac{H_1-c}{\sqrt{F}}\right)\right]}^2,$$ (68)

where H₀, H₁, and F are defined in (29).

Theorem 6

Consider the sequence of Gaussian discrimination problems in (11). For i, j = 0, 1,

$$\underset{b.k.a.c.}{lim}{E}_{\mathit{\mu },S_n}[{\widehat{\epsilon }}_i^B{\epsilon }_j]=\underset{b.k.a.c.}{lim}{E}_{\mathit{\mu },S_n}[{\widehat{\epsilon }}_i^B{\widehat{\epsilon }}_j^B]=\underset{b.k.a.c.}{lim}{E}_{\mathit{\mu },S_n}[{\epsilon }_i{\epsilon }_j],$$ (69)

so that

$$\underset{b.k.a.c.}{lim}{E}_{\mathit{\mu },S_n}[{\widehat{\epsilon }}^B\epsilon ]=\sum _{i=0}\sum _{j=0}\left[{\alpha }_i{\alpha }_j\mathrm{\Phi }\left({(-1)}^i\frac{H_i+c}{\sqrt{F}}\right)\mathrm{\Phi }\left({(-1)}^j\frac{H_j+c}{\sqrt{F}}\right)\right],$$ (70)

where H₀, H₁, and F are defined in (29).

Theorems 5 and 6 suggest the finite-sample approximation:

$$E_{\mathit{\mu },S_n}[{\widehat{\epsilon }}_0^B{\widehat{\epsilon }}_0^B]\eqsim E_{\mathit{\mu },S_n}[{\widehat{\epsilon }}_0^B{\epsilon }_0]\eqsim E_{\mathit{\mu },S_n}[{\epsilon }_0{\epsilon }_0]\eqsim {\left[\mathrm{\Phi }\left(-\frac{1}{2}\frac{{\mathrm{\Delta }}_{\mathbf{m}}^2+\frac{p}{n_1}-\frac{p}{n_0}+\frac{p}{{\nu }_0}+\frac{p}{{\nu }_1}-c}{\sqrt{{\mathrm{\Delta }}_{\mathbf{m}}^2+\frac{p}{n_0}+\frac{p}{n_1}+\frac{p}{{\nu }_0}+\frac{p}{{\nu }_1}}}\right)\right]}^2.$$ (71)

A consequence of Theorems 2, 5, and 6 is that

$$\begin{array}{l}\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{\text{Var}}_{\mathit{\mu },S_n}^d[{\widehat{\epsilon }}^B]=\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}\mid {\text{Bias}}_{U,n}[{\widehat{\epsilon }}^B]\mid =\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{\text{Var}}_{\mathit{\mu },S_n}({\widehat{\epsilon }}^B)=\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{\text{Var}}_{\mathit{\mu },S_n}(\epsilon )\\ =\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{\text{Cov}}_{\mu ,S_n}(\epsilon ,{\widehat{\epsilon }}^B)=\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{\text{RMS}}_{\mathit{\mu },S_n}[{\widehat{\epsilon }}^B]=0.\end{array}$$ (72)

In [30], it was shown that ε̂^B is strongly consistent, meaning that ε̂^B(S_n) − ε(S_n) → 0 almost surely as n → ∞ under rather general conditions, in particular, for the Gaussian and discrete models considered in that paper. It was also shown that MSE_μ[ε̂^B|S_n] → 0 almost surely as n → ∞ under similar conditions. Here, we have shown that ${\text{MSE}}_{\mathit{\mu },S_n}[{\widehat{\epsilon }}^B]\stackrel{K}{\to }0$ under conditions stated in (12). Some researchers refer to conditions of double asymptoticity as “comparable” dimensionality and sample size [20, 22]. Therefore, one may think of ${\text{MSE}}_{\mathit{\mu },S_n}[{\widehat{\epsilon }}^B]\stackrel{K}{\to }0$ meaning that MSE_μ,Sn [ε̂^B] is close to zero for asymptotic and comparable dimensionality, sample size, and certainty parameter.

We now consider Raudys-type approximations. Analogous to the approximation used in (51), we obtain the unconditional second moment of ${\widehat{\epsilon }}_0^B$:

$$E_{\mathit{\mu },S_n}[{({\widehat{\epsilon }}_0^B)}^2]\eqsim \mathrm{\Phi }(\frac{-E_{\mathit{\mu },S_n,\mathbf{z}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\mid \mathbf{z}\in {\mathrm{\Psi }}_0]+c}{\sqrt{{\text{Var}}_{\mathit{\mu },S_n,\mathbf{z}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\mid \mathbf{z}\in {\mathrm{\Psi }}_0]}};\frac{{\text{Cov}}_{\mathit{\mu },S_n,\mathbf{z}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z}),U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,{\mathbf{z}}^{\prime })\mid \mathbf{z}\in {\mathrm{\Psi }}_0,{\mathbf{z}}^{\prime }\in {\mathrm{\Psi }}_0]}{{\text{Var}}_{\mathit{\mu },S_n,\mathbf{z}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\mid \mathbf{z}\in {\mathrm{\Psi }}_0]}).$$ (73)

Using (73) we get

$$E_{\mathit{\mu },S_n}[{({\widehat{\epsilon }}_0^B)}^2]=\mathrm{\Phi }\left(\frac{-H_0^R+c}{\sqrt{F_0^R}};\frac{K_0^{B,R}}{F_0^R}\right)$$ (74)

with $H_0^R$ and $F_0^R$ given in (31) and (34), respectively, and

$$\begin{array}{l}{K}_0^{B,R}={\text{Cov}}_{\mathit{\mu },S_n,\mathbf{z}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z}),U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,{\mathbf{z}}^{\prime })\mid \mathbf{z}\in {\mathrm{\Psi }}_0,{\mathbf{z}}^{\prime }\in {\mathrm{\Psi }}_0]\\ =\left(\frac{1}{n_0{(1+{\beta }_0)}^2}+\frac{1}{n_1}+\frac{1}{{\nu }_0{(1+{\beta }_0)}^2}+\frac{1}{{\nu }_1}\right){\mathrm{\Delta }}_{\mathbf{m}}^2+\frac{p}{2n_0^2}\\ +\frac{p}{2{\nu }_0^2}-\frac{p}{n_0{\nu }_0}+\frac{p}{n_1{\nu }_1}+\frac{p}{2n_1^2}+\frac{p}{2{\nu }_1^2}+\frac{p}{n_0{n}_1{(1+{\beta }_0)}^2}+\frac{p}{n_0{\nu }_1{(1+{\beta }_0)}^2}+\frac{p}{n_1{\nu }_0{(1+{\beta }_0)}^2},\end{array}$$ (75)

Suppl. Section D presents the proof of (75). In a similar way,

$$E_{\mathit{\mu },S_n}[{({\widehat{\epsilon }}_1^B)}^2]=\mathrm{\Phi }\left(\frac{H_1^R-c}{\sqrt{F_1^R}};\frac{K_1^{B,R}}{F_1^R}\right),$$ (76)

where $F_1^R,H_1^R$, and $K_1^{B,R}$ are obtained by exchanging n₀ and n₁, ν₀ and ν₁, m₀ and m₁, and μ₀ and μ₁, in (34), in $-H_0^{B,f}$ obtained from (31), and (75), respectively.

Having $K_0^{B,R}\stackrel{K}{\to }0$ together with (35) makes (74) asymptotically exact. We similarly obtain

$$E_{\mathit{\mu },S_n}[{\widehat{\epsilon }}_0^B{\widehat{\epsilon }}_1^B]=\mathrm{\Phi }\left(\frac{-H_0^R+c}{\sqrt{F_0^R}},\frac{H_1^R-c}{\sqrt{F_1^R}};\frac{K_{01}^{B,R}}{\sqrt{F_0^R{F}_1^R}}\right),$$ (77)

where

$$\begin{array}{l}{K}_{01}^{B,R}=\frac{p}{(n_0+{\nu }_0)(n_1+{\nu }_1)}+\frac{(n_0-{\nu }_0)p}{2n_0^2(n_0+{\nu }_0)}+\frac{(n_1-{\nu }_1)p}{2n_1^2(n_1+{\nu }_1)}\\ +\frac{n_0{n}_{1}p}{{\nu }_0{\nu }_1(n_0+{\nu }_0)(n_1+{\nu }_1)}+\frac{(n_0-{\nu }_0)p}{2{\nu }_0^2(n_0+{\nu }_0)}+\frac{(n_1-{\nu }_1)p}{2{\nu }_1^2(n_1+{\nu }_1)}\\ +\frac{1}{n_0+{\nu }_0}\left(1+\frac{n_0}{n_1+{\nu }_1}-\frac{{\nu }_0}{n_0}\right)\frac{p}{{\nu }_0}+\frac{1}{n_1+{\nu }_1}\left(1+\frac{n_1}{n_0+{\nu }_0}-\frac{{\nu }_1}{n_1}\right)\frac{p}{{\nu }_1}+\left(\frac{1}{{\nu }_0}+\frac{1}{{\nu }_1}\right){\mathrm{\Delta }}_{\mathbf{m}}^2.\end{array}$$ (78)

Suppl. Section E presents the proof of (78). Since $K_{01}^{B,R}\stackrel{K}{\to }0$, (77) is asymptotically exact (compare to Theorem 5). Similar to (57) and (59), where we characterized conditional cross moments, we can get the unconditional cross moments as follows:

$$\begin{array}{l}{E}_{\mathit{\mu },S_n}[{\widehat{\epsilon }}_0^B{\epsilon }_0]=P(U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\le c,W({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{x})\le c\mid \mathbf{z}\in {\mathrm{\Psi }}_0,\mathbf{x}\in {\mathrm{\Pi }}_0)\\ \eqsim \mathrm{\Phi }(\frac{-E_{\mathit{\mu },S_n,\mathbf{z}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\mid \mathbf{z}\in {\mathrm{\Psi }}_0]+c}{\sqrt{{\text{Var}}_{\mathit{\mu },S_n,\mathbf{z}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\mid \mathbf{z}\in {\mathrm{\Psi }}_0]}},\frac{-E_{\mathit{\mu },S_n,\mathbf{x}}[W({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{x})\mid \mathbf{x}\in {\mathrm{\Pi }}_0]+c}{\sqrt{{\text{Var}}_{\mathit{\mu },S_n,\mathbf{x}}[W({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{x})\mid \mathbf{x}\in {\mathrm{\Pi }}_0]}};\\ \frac{{\text{Cov}}_{\mathit{\mu },S_n,\mathbf{z},\mathbf{x}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z}),W({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{x})\mid \mathbf{z}\in {\mathrm{\Psi }}_0,\mathbf{x}\in {\mathrm{\Pi }}_0]}{\sqrt{V_{U_0}^U{V}_W^U}})=\mathrm{\Phi }\left(\frac{-H_0^R+c}{\sqrt{F_0^R}};\frac{K_0^{BT,R}}{F_0^R}\right),\end{array}$$ (79)

where

$$\begin{array}{l}{V}_{U_0}^{\mathrm{U}}={\text{Var}}_{\mathit{\mu },S_n,\mathbf{z}}[U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\mid \mathbf{z}\in {\mathrm{\Psi }}_0],\\ V_W^{\mathrm{U}}={\text{Var}}_{\mathit{\mu },S_n,\mathbf{x}}[W({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{x})\mid \mathbf{x}\in {\mathrm{\Pi }}_0],\end{array}$$ (80)

the superscript “U” representing the unconditional variance, $H_0^R$ and $F_0^R$ being presented in (31) and (34), respectively, and

$$K_0^{BT,R}=\left(\frac{n_0}{{\nu }_0(n_0+{\nu }_0)}+\frac{1}{n_1}+\frac{1}{{\nu }_1}\right){\mathrm{\Delta }}_{\mathbf{m}}^2+\frac{p}{2n_1^2}+\frac{p}{2{\nu }_1^2}+\frac{p}{n_1{\nu }_1}+\frac{n_{0}p}{n_1{\nu }_0(n_0+{\nu }_0)}-\frac{(n_0-{\nu }_0)p}{2n_0^2(n_0+{\nu }_0)}+\frac{(n_0-{\nu }_0)p}{2{\nu }_0^2(n_0+{\nu }_0)}+\frac{n_{0}p}{{\nu }_0{\nu }_1(n_0+{\nu }_0)}.$$ (81)

The proof of (81) is presented in Suppl. Section F. Similarly,

$$E_{\mathit{\mu },S_n}[{\widehat{\epsilon }}_0^B{\epsilon }_1]\eqsim \mathrm{\Phi }\left(\frac{-H_0^R+c}{\sqrt{F_0^R}},\frac{H_1^R-c}{\sqrt{F_1^R}};\frac{K_{01}^{BT,R}}{\sqrt{F_0^R{F}_1^R}}\right)$$ (82)

where,

$$K_{01}^{BT,R}=\left(\frac{1}{{\nu }_0}+\frac{1}{{\nu }_1}\right){\mathrm{\Delta }}_{\mathbf{m}}^2+\frac{p}{2{\nu }_0^2}+\frac{p}{2{\nu }_1^2}+\frac{p}{{\nu }_0{\nu }_1}-\frac{p}{2n_0^2}-\frac{p}{2n_1^2}.$$ (83)

See Suppl. Section G for the proof of (83). Having $K_0^{BT,R}\stackrel{K}{\to }0$ and $K_{01}^{BT,R}\stackrel{K}{\to }0$ along with (35) makes (79) and (82) asymptotically exact (compare to Theorem 6).

5.3. Conditional and Unconditional Second Moment of ε_i

To complete the derivations and obtain the unconditional RMS of estimation, we need the conditional and unconditional second moment of the true error. The conditional second moment of the true error can be found from results in [16], which for completeness are represented here:

$$E_{S_n}[{\epsilon }_0^2\mid \mathit{\mu }]\eqsim \mathrm{\Phi }\left(\frac{-G_0^R+c}{\sqrt{D_0^R}};\frac{C_0^{T,R}}{D_0^R}\right),$$ (84)

with $G_0^R$ and $D_0^R$ defined in (61),

$$E_{S_n}[{\epsilon }_1^2\mid \mathit{\mu }]\eqsim \mathrm{\Phi }\left(\frac{G_1^R-c}{\sqrt{D_1^R}};\frac{C_1^{T,R}}{D_1^R}\right),$$ (85)

and

$$E_{S_n}[{\epsilon }_0{\epsilon }_1\mid \mathit{\mu }]\eqsim \mathrm{\Phi }\left(\frac{-G_0^R+c}{\sqrt{D_0^R}},\frac{G_1^R-c}{\sqrt{D_1^R}};\frac{C_{01}^{T,R}}{\sqrt{D_0^R{D}_1^R}}\right),$$ (86)

where

$$C_{01}^{T,R}=-\frac{p}{2n_0^2}-\frac{p}{2n_1^2}.$$ (87)

Similar to obtaining (79), we can show that

$$E_{\mathit{\mu },S_n}[{\epsilon }_0^2]\eqsim \mathrm{\Phi }\left(\frac{-H_0^R+c}{\sqrt{F_0^R}};\frac{K_0^{T,R}}{F_0^R}\right),$$ (88)

with $H_0^R$ and $F_0^R$ given in (31) and (34), respectively, and

$$K_0^{T,R}=\left(\frac{1}{{\nu }_0}+\frac{1}{{\nu }_1}+\frac{1}{n_1}\right){\mathrm{\Delta }}_{\mathbf{m}}^2+\frac{p}{2{\nu }_0^2}+\frac{p}{2{\nu }_1^2}+\frac{p}{{\nu }_0{\nu }_1}+\frac{p}{2n_0^2}+\frac{p}{2n_1^2}+\frac{p}{n_1{\nu }_0}+\frac{p}{n_1{\nu }_1}.$$ (89)

Similarly,

$$E_{\mathit{\mu },S_n}[{\epsilon }_1^2]=\mathrm{\Phi }\left(\frac{H_1^R-c}{\sqrt{F_1^R}};\frac{K_1^{T,R}}{F_1^R}\right),$$ (90)

with $K_1^{T,R}$ obtained from $K_0^{T,R}$ by exchanging n₀ and n₁, and ν₀ and ν₁. Similarly,

$$E_{\mathit{\mu },S_n}[{\epsilon }_0^2]\eqsim \mathrm{\Phi }\left(\frac{-H_0^R+c}{\sqrt{F_0^R}},\frac{H_1^R-c}{\sqrt{F_1^R}};\frac{K_{01}^{T,R}}{\sqrt{F_0^R{F}_1^R}}\right),$$ (91)

with $H_0^R$ and $F_0^R$ given in (31) and (34), respectively, and

$$K_{01}^{T,R}=\left(\frac{1}{{\nu }_0}+\frac{1}{{\nu }_1}\right){\mathrm{\Delta }}_{\mathbf{m}}^2+\frac{p}{2{\nu }_0^2}+\frac{p}{2{\nu }_1^2}+\frac{p}{{\nu }_0{\nu }_1}-\frac{p}{2n_0^2}-\frac{p}{2n_1^2}.$$ (92)

6. Monte Carlo Comparisons

In this section we compare the asymptotically exact finite-sample approximations of the first, second and mixed moments to Monte Carlo estimations in conditional and unconditional scenarios. The following steps are used to compute the Monte Carlo estimation:

1. Define a set of hyper-parameters for the Gaussian model: m₀, m₁, ν₀, ,ν₁, and Σ. We let Σ have diagonal elements 1 and off-diagonal elements 0.1. m₀ and m₁ are chosen by fixing ${\delta }_{\mathit{\mu }}^2$ ( ${\delta }_{\mathit{\mu }}^2=4$, which corresponds to Bayes error 0.1586). Setting ${\delta }_{\mathit{\mu }}^2$ and Σ fixes the means μ₀ and μ₁ of the class-conditional densities. The priors, π₀ and π₁, are defined by choosing a small deviation from μ₀ and μ₁, that is, by setting m_i = μ_i + aμ_i, where a = 0.01.

2. (unconditional case): Using π₀ and π₁, generate random realizations of μ₀ and μ₁.

3. (conditional case): Use the values of μ₀ and μ₁ obtained from Step 1.

4. For fixed Π₀ and Π₁, generate a set of training data of size n_i for class i = 0, 1.

5. Using the training sample, design the LDA classifier, ψ_n, using (2).

6. Compute the Bayesian MMSE error estimator, ε̂^B, using (5) and (7).

7. Knowing μ₀ and μ₁, find the true error of ψ_n using (3).

8. Repeat Steps 3 through 6, T₁ times.

9. Repeat Steps 2 through 7, T₂ times.

In the unconditional case, we set T₁ = T₂ = 300 and generate 90, 000 samples. For the conditional case, we set T₁ = 10, 000 and T₂ = 1, the latter because μ₀ and μ₁ are set in Step 2.

Figure 1 treats Raudys-type finite-sample approximations, including the RMS. Figure 1(a) compares the first moments obtained from equations (22) and (33). It presents E_Sn[ε̂^B|μ] and E_μ,Sn[ε̂^B] computed by Monte Carlo estimation and the analytical expressions. The label “FSA BE Uncond” identifies the curve of E_μ,Sn[ε̂^B], the unconditional expected estimated error obtained from the finite-sample approximation, which according to the basic theory is equal to E_μ,Sn[ε]. The labels “FSA BE Cond” and “FSA TE Cond” show the curves of E_Sn[ε̂^B|μ], the conditional expected estimated error, and E_Sn[ε|μ], the conditional expected true error, respectively, both obtained using the analytic approximations. The curves obtained from Monte Carlo estimation are identified by “MC” labels. The analytic curves in Figure 1(a) show substantial agreement with the Monte Carlo approximation.

Figure 1.

Comparison of conditional and unconditional performance metrics of ε̂^B using asymptotically exact finite setting approximations, with Monte Carlo estimates as a function of sample size. (a) Expectations. The case of asymptotic unconditional expectation of ε is not plotted as ε̂^B is unconditionally unbiased; (b) Second and mixed moments; (c) Conditional variance of deviation from true error, i.e. ${\text{Var}}_{S_n}^d[{\widehat{\epsilon }}^B\mid \mathit{\mu }]$ and, unconditional variance of deviation, i.e. ${\text{Var}}_{\mathit{\mu },S_n}^d[{\widehat{\epsilon }}^B]$; (d) Conditional RMS of estimation, i.e. RMS_Sn[ε̂^B|μ] and, unconditional RMS of estimation, i.e. RMS_μ,Sn[ε̂^B]; (a)–(d) correspond to the same scenario in which dimension, p, is 15 and 100, ν₀ = ν₁ = 50, m_i = μ_i + 0.01μ_i with μ₀ = −μ₁, and Bayes error = 0.1586.

To obtain the second moments, Var^d[ε̂] and RMS[ε̂^B] as defined in (1), we use equations (52), (54), (55), (59), (63), (84), (85), (86) for the conditional case and (74), (76), (77), (79), (82), (88), (90), (91) for the unconditional case. Figures 1(b), 1(c), and 1(d) compare the Monte Carlo estimation to the finite-sample approximations obtained for second/mixed moments, Var^d[ε̂], and RMS[ε̂^B], respectively. The labels are interpreted similarly to those in Figure 1(a), but for the second/mixed moments instead. For example, “MC BE×TE Uncond” identifies the MC curve of E_μ,Sn[ε̂^Bε]. The Figures 1(b), 1(c), and 1(d) show that the finite-sample approximations for the conditional and unconditional second/mixed moments, variance of deviation, and RMS are quite accurate (close to the MC value).

While Figure 1 shows the accuracy of Raudys-type of finite-sample approximations, figures in the Supplementary Materials show the the comparison between the finite-sample approximations obtained directly from Theorem 1–6, i.e. equations (29), (57), (70), (73), (76), (102), and (103), to Monte Carlo estimation.

7. Examination of the Raudys-type RMS Approximation

Equations (18), (24), (53), (56), and (63) show that RMS_Sn[ε̂^B|μ] is a function of 14 variables: p, n₀, n₁, β₀, β₁, ${\delta }_{\mathit{\mu }}^2$, η_m0,μ1, η_m0,μ0, η_m1,μ0, η_m1,μ1, η_{m0,μ0,μ0,μ1}, η_{m0,μ0,m1,μ0}, η_{m1,μ1,m0,μ1}, η_{m1,μ1,μ1,μ0}. Studying a function of this number of variables is complicated, especially because restricting some variables can constrain others. We make several simplifying assumptions to reduce the complexity. We let $n_0=n_1=\frac{n}{2}$, β₀ = β₁ = β and assume very informative priors in which m₀ = μ₀ and m₁ = μ₁. Using these assumptions, RMS_Sn[ε̂^B|μ] is only a function of p, n, β, and ${\delta }_{\mathit{\mu }}^2$. We let p ∈ [4, 200], n ∈ [40, 200], β ∈ {0.5, 1, 2}, ${\delta }_{\mathit{\mu }}^2\in \{4,16\}$, which means that the Bayes error is 0.158 or 0.022. Figure 2(a) shows plots of RMS_Sn[ε̂^B|μ] as a function of p, n, β, and ${\delta }_{\mathit{\mu }}^2$. These show that for smaller distance between classes, that is, for smaller ${\delta }_{\mathit{\mu }}^2$ (larger Bayes error), the RMS is larger, and as the distance between classes increases, the RMS decreases. Furthermore, we see that in situations where very informative priors are available, i.e. m₀ = μ₀ and m₁ = μ₁, relying more on data can have a detrimental effect on RMS. Indeed, the plots in the top row (for β = 0.5) have larger RMS than the plots in the bottom row of the figure (for β = 2).

Figure 2.

(a) The conditional RMS of estimation, i.e. RMS_Sn[ε̂^B|μ], as a function of p < 200 and n < 200. From top to bottom, the rows correspond to β = 0.5, 1, 2, respectively. From left to right, the columns correspond to ${\delta }_{\mathit{\mu }}^2=4,16$, respectively. (b) The unconditional RMS of estimation, i.e. RMS_μ,Sn[ε̂^B], as a function of p < 1000 and n < 2000. From top to bottom, the rows correspond to β = 0.5, 1, 2, respectively. From left to right, the columns correspond to ${\mathrm{\Delta }}_{\mathbf{m}}^2=4,16$, respectively.

Using the RMS expressions enables finding the necessary sample size to insure a given RMS_Sn[ε̂^B|μ] by using the same methodology as developed for the resubstitution and leave-one-out error estimators in [16, 26]. The plots in Figure 2(a) (as well as other unshown plots) show that, with m₀ = μ₀ and m₁ = μ₁, the RMS is a decreasing function of ${\delta }_{\mathit{\mu }}^2$ . Therefore, the number of sample points that guarantees ${max}_{{\delta }_{\mathit{\mu }}^2>0}{\text{RMS}}_{S_n}[{\widehat{\epsilon }}^B\mid \mathit{\mu }]={lim}_{{\delta }_{\mathit{\mu }}^2\to 0}{\text{RMS}}_{S_n}[{\widehat{\epsilon }}^B\mid \mathit{\mu }]$ being less than a predetermined value τ insures that RMS_Sn[ε̂^B|μ] < τ, for any ${\delta }_{\mathit{\mu }}^2$. Let the desired bound be ${\kappa }_{\widehat{\epsilon }}(n,p,\beta )={lim}_{{\delta }_{\mathit{\mu }}^2\to 0}{\text{RMS}}_{S_n}[{\widehat{\epsilon }}^B\mid \mathit{\mu }]$. From equations (52), (54), (55), (59), (63), (84), (85), and (86), we can find κ_ε̂(n, p, β) and increase n until κ_ε̂(n, p, β) < τ. Table 1 (β = 1: Conditional) shows the minimum number of sample points needed to guarantee having a predetermined conditional RMS for the whole range of ${\delta }_{\mathit{\mu }}^2$ (other β shown in the Supplementary Material). A larger dimensionality, a smaller τ, and a smaller β result in a larger necessary sample size needed for having κ_ε̂(n, p, β) < τ.

Table 1.

Minimum sample size, n, ( $n_0=n_1=\frac{n}{2}$) to satisfy κ_ε̂ (n, p, β) < τ.

τ	p = 2	p = 4	p = 8	p = 16	p = 32	p = 64	p = 128
β = 1: Conditional
0.1	14	22	38	70	132	256	506
0.09	18	28	48	86	164	318	626
0.08	24	36	60	110	208	404	796
0.07	32	48	80	144	272	530	1044
0.06	44	64	108	196	372	722	1424
0.05	62	94	158	284	538	1044	2056
β = 1: Unconditional
0.025	108	108	106	102	92	72	2
0.02	172	170	168	164	156	138	78
0.015	308	306	304	300	292	274	236
0.01	694	694	690	686	678	662	628
0.005	2790	2786	2782	2776	2768	2752	2720

Turning to the unconditional RMS, equations (34), (75), (78), (83), (89), and (92) show that RMS_μ,Sn[ε̂^B] is a function of 6 variables: p, n₀, n₁, ν₀,ν₁, ${\mathrm{\Delta }}_{\mathbf{m}}^2$. Figure 2(b) shows plots of RMS_μ,Sn[ε̂^B] as a function of p, n, β, and ${\mathrm{\Delta }}_{\mathbf{m}}^2$, assuming $n_0=n_1=\frac{n}{2}$, β₀ = β₁ = β. Note that setting the values of n and β fixes the value of ν₀ = ν₁ = ν in the corresponding expressions for RMS_μ,Sn[ε̂^B]. Due to the complex shape of RMS_μ,Sn[ε̂^B], we consider a large range of n and p. The plots show that a smaller distance between prior distributions (smaller ${\mathrm{\Delta }}_{\mathbf{m}}^2$) corresponds to a larger unconditional RMS of estimation. In addition, as the distance between classes increases, the RMS decreases. The plots in Figure 2(b) show that, as ${\mathrm{\Delta }}_{\mathbf{m}}^2$ increases, RMS decreases. Furthermore, Figure 2(b) (and other unshown plots) demonstrate an interesting phenomenon in the shape of the RMS. In regions defined by pairs of (p, n), for each p, RMS first increases as a function of sample size and then decreases. We further observe that with fixed p, for smaller β, this “peaking phenomenon” happens for larger n. On the other hand, with fixed β, for larger p, peaking happens for larger n. These observations are presented in Figure 3, which shows curves obtained by cutting the 3D plots in the left column of Fig. 2(b) at a few dimensions. This figure shows that, for p = 900 and β = 2, adding more sample points increases RMS abruptly at first to reach a maximum value of RMS at n = 140, the point after which the RMS starts to decrease.

Figure 3.

RMS_μ,Sn[ε̂^B]-peaking phenomenon as a function of sample size. These plots are obtained by cutting the 3D plots in the left column of Fig. 2(b) at few dimensionality (i.e. ${\mathrm{\Delta }}_{\mathbf{m}}^2=4$). From top to bottom the rows correspond to β = 0.5, 1, 2, respectively. The solid-black curves indicate RMS_μ,Sn[ε̂^B] computed from the analytical results and the red-dashed curves show the same results computed by means of Monte Carlo simulations. Due to computational burden of estimating the curves by means of Monte Carlo studies, the simulations are limited to n < 500 and p = 10, 70.

One may use the unconditional scenario to determine the the minimum necessary sample size for a desired RMS_μ,Sn[ε̂^B]. In fact, this is the more practical way to go because in practice one does not know μ. Since the unconditional RMS shows a decreasing trend in terms of ${\mathrm{\Delta }}_{\mathbf{m}}^2$, we use the previous technique to find the minimum necessary sample size to guarantee a desired unconditional RMS. Table 1 (β = 1: Unconditional) shows the minimum sample size that guarantees ${max}_{{\mathrm{\Delta }}_{\mathbf{m}}^2>0}{\text{RMS}}_{\mathit{\mu },S_n}[{\widehat{\epsilon }}^B]={lim}_{{\mathrm{\Delta }}_{\mathbf{m}}^2\to 0}{\text{RMS}}_{\mathit{\mu },S_n}[{\widehat{\epsilon }}^B]$ being less than a predetermined value τ, i.e. insures that RMS_μ,Sn[ε̂^B] < τ for any ${\mathrm{\Delta }}_{\mathbf{m}}^2$ (other β shown in the Supplementary Material).

To examine the accuracy of the required sample size that satisfies κ_ε̂(n, p, β) < τ for both conditional and unconditional settings, we have performed a set of experiments (see Supplementary Material). The results of these experiments confirm the efficacy of Table 1 in determining the minimum sample size required to insure the RMS is less than a predetermined value τ.

8. Conclusion

Using realistic assumptions about sample size and dimensionality, standard statistical techniques are generally incapable of estimating the error of a classifier in small-sample classification. Bayesian MMSE error estimation facilitates more accurate estimation by incorporating prior knowledge. In this paper, we have characterized two sets of performance metrics for Bayesian MMSE error estimation in the case of LDA in a Gaussian model: (1) the first, second, and cross moments of the estimated and actual errors conditioned on a fixed feature-label distribution, which in turn gives us knowledge of the conditional RMS_Sn[ε̂^B|θ]; and (2) the unconditional moments and, therefore, the unconditional RMS, RMS_θ,Sn[ε̂^B]. We set up a series of conditions, called the Bayesian-Kolmogorov asymptotic conditions, that allow us to characterize the performance metrics of Bayesian MMSE error estimation in an asymptotic sense. The Bayesian-Kolmogorov asymptotic conditions are set up based on the assumption of increasing n, p, and certainty parameter ν, with an arbitrary constant limiting ratio between n and p, and n and ν. To our knowledge, these conditions permit, for the first time, application of Kolmogorov-type of asymptotics in a Bayesian setting. The asymptotic expressions proposed in this paper result directly in finite-sample approximations of the performance metrics. Improved finite-sample accuracy is achieved via newly proposed Raudys-type approximations. The asymptotic theory is used to prove that these approximations are, in fact, asymptotically exact under the Bayesian-Kolmogorov asymptotic conditions. Using the derived analytical expressions, we have examined performance of the Bayesian MMSE error estimator in relation to feature-label distributions, prior knowledge, sample size, and dimensionality. We have used the results to determine the minimum sample size guaranteeing a desired level of error estimation accuracy.

As noted in the Introduction, a natural next step in error estimation theory is to remove the known-covariance condition, but as also noted, this may prove to be difficult.

Appendix

Proof of Theorem 1

We explain this proof in detail as some steps will be used in later proofs. Let

$${\widehat{G}}_0^B={\left({\mathbf{m}}_0^{\ast }-\frac{{\overline{\mathbf{x}}}_0+{\overline{\mathbf{x}}}_1}{2}\right)}^T{\mathbf{\sum }}^{-1}({\overline{\mathbf{x}}}_0-{\overline{\mathbf{x}}}_1),$$ (93)

where ${\mathbf{m}}_0^{\ast }$ is defined in (8). Then

$${\widehat{G}}_0^B=\frac{{\nu }_0{\mathbf{m}}_0^T}{n_0+{\nu }_0}{\mathbf{\sum }}^{-1}({\overline{\mathbf{x}}}_0-{\overline{\mathbf{x}}}_1)+\frac{n_0-{\nu }_0}{2(n_0+{\nu }_0)}({\overline{\mathbf{x}}}_0^T{\mathbf{\sum }}^{-1}{\overline{\mathbf{x}}}_0-{\overline{\mathbf{x}}}_0^T{\mathbf{\sum }}^{-1}{\overline{\mathbf{x}}}_1)+\frac{1}{2}({\overline{\mathbf{x}}}_1^T{\mathbf{\sum }}^{-1}{\overline{\mathbf{x}}}_1-{\overline{\mathbf{x}}}_0^T{\mathbf{\sum }}^{-1}{\overline{\mathbf{x}}}_1).$$ (94)

For i, j = 0, 1 and i ≠ j, define the following random variables:

$$y_i={\mathbf{m}}_i^T{\mathbf{\sum }}^{-1}({\overline{\mathbf{x}}}_0-{\overline{\mathbf{x}}}_1),z_i={\overline{\mathbf{x}}}_i^T{\mathbf{\sum }}^{-1}{\overline{\mathbf{x}}}_i,z_{ij}={\overline{\mathbf{x}}}_i^T{\mathbf{\sum }}^{-1}{\overline{\mathbf{x}}}_j.$$ (95)

The variance of y_i given μ does not depend on μ. Therefore, under the Bayesian-Kolmogorov conditions stated in (10), $\overline{{\mathbf{m}}_i^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_j}$ and $\overline{{\mathit{\mu }}_i^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_j}$ do not appear in the limit. Only $\overline{{\mathbf{m}}_i^T{\mathbf{\sum }}^{-1}{\mathbf{m}}_i}$ matters, which vanishes in the limit as follows:

$${\text{Var}}_{S_n}[y_i\mid \mathit{\mu }]={\mathbf{m}}_i^T(\frac{{\mathbf{\sum }}^{-1}}{n_0}+\frac{{\mathbf{\sum }}^{-1}}{n_1}){\mathbf{m}}_i\stackrel{K}{\to }\underset{n_0\to \infty }{lim}\frac{\overline{{\mathbf{m}}_i^T{\mathbf{\sum }}^{-1}{\mathbf{m}}_i}}{n_0}+\underset{n_1\to \infty }{lim}\frac{\overline{{\mathbf{m}}_i^T{\mathbf{\sum }}^{-1}{\mathbf{m}}_i}}{n_1}=0.$$ (96)

To find the variance of z_i and z_ij we can first transform z_i and z_ij to quadratic forms and then use the results of [34] to find the variance of quadratic functions of Gaussian random variables. Specifically, from [34], for y ~ N(μ, Σ) and A being a symmetric positive definite matrix, Var[y^T Ay] = 2tr(AΣ)² + 4μ^T AΣAμ′, with tr being the trace operator. Therefore, after some algebraic manipulations, we obtain

$$\begin{array}{l}{\text{Var}}_{S_n}[z_i\mid \mathit{\mu }]=2\frac{p}{n_i^2}+4\frac{{\mathit{\mu }}_i^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_i}{n_i}\stackrel{K}{\to }2\underset{n_i\to \infty }{lim}\frac{J_i}{n_i}+4\underset{n_i\to \infty }{lim}\frac{\overline{{\mathit{\mu }}_i^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_i}}{n_i}=0,\\ {\text{Var}}_{S_n}[z_{ij}\mid \mathit{\mu }]=\frac{p}{n_i{n}_j}+\frac{{\mathit{\mu }}_i^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_i}{n_j}+\frac{{\mathit{\mu }}_i^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_j}{n_i}\stackrel{K}{\to }\underset{n_j\to \infty }{lim}\frac{J_i}{n_j}+\underset{n_j\to \infty }{lim}\frac{\overline{{\mathit{\mu }}_i^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_i}}{n_j}+\underset{n_i\to \infty }{lim}\frac{\overline{{\mathit{\mu }}_j^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_j}}{n_i}=0.\end{array}$$ (97)

From the Cauchy-Schwarz inequality $(\text{Cov}[x,y]\le \sqrt{\text{Var}[x]\text{Var}[y]}),{\text{Cov}}_{S_n}[y_i,z_k\mid \mathit{\mu }]\stackrel{K}{\to }0,{\text{Cov}}_{S_n}[y_i,z_{ij}\mid \mathit{\mu }]\stackrel{K}{\to }0$, and ${\text{Cov}}_{S_n}[z_i,z_{ij}\mid \mathit{\mu }]\stackrel{K}{\to }0$ for i, j, k = 0, 1, i ≠ j, Furthermore, $\frac{n_i-{\nu }_i}{2(n_i+{\nu }_i)}\stackrel{K}{\to }\frac{1-{\gamma }_i}{2(1+{\gamma }_i)}$ and $\frac{{\nu }_i}{n_i+{\nu }_i}\stackrel{K}{\to }\frac{{\gamma }_i}{1+{\gamma }_i}$. Putting this together and following the same approach for ${\widehat{G}}_1^B$ yields ${\text{Var}}_{S_n}[{\widehat{G}}_1^B\mid \mathit{\mu }]\stackrel{K}{\to }0$. In general (via Chebyshev’s inequality), lim_n→∞Var[X_n] = 0 implies convergence in probability of X_n to lim_n→∞ E[X_n]. Hence, since ${\text{Var}}_{S_n}[{\widehat{G}}_1^B\mid \mathit{\mu }]\stackrel{K}{\to }0$, for i, j = 0, 1 and i ≠ j,

$$\begin{array}{l}\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{\text{plim}}{\widehat{G}}_i^B\mid \mathit{\mu }=\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{E}_{S_n}[{\widehat{G}}_i^B\mid \mathit{\mu }]={(-1)}^i[\frac{1}{2}\left(\overline{{\mathit{\mu }}_j^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_j}+J_j\right)+\\ \frac{{\gamma }_i(\overline{{\mathbf{m}}_i^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_i}-\overline{{\mathbf{m}}_i^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_j})}{1+{\gamma }_i}+\frac{1-{\gamma }_i}{2(1+{\gamma }_i)}\left(\overline{{\mathit{\mu }}_i^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_i}+J_i\right)-\overline{{\mathit{\mu }}_i^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_j}\left(\frac{1-{\gamma }_i}{2(1+{\gamma }_i)}+\frac{1}{2}\right)]=G_i^B.\end{array}$$ (98)

Now let

$${\widehat{D}}_i=\frac{{\nu }_i^{\ast }+1}{{\nu }_i^{\ast }}{({\overline{\mathbf{x}}}_0-{\overline{\mathbf{x}}}_1)}^T{\mathbf{\sum }}^{-1}({\overline{\mathbf{x}}}_0-{\overline{\mathbf{x}}}_1)=\frac{{\nu }_i^{\ast }+1}{{\nu }_i^{\ast }}{\widehat{\delta }}^2,$$ (99)

where δ̂² = (x̄₀ − x̄₁)^T Σ⁻¹(x̄₀ − x̄₁). Similar to deriving (97) via the variance of quadratic forms of Gaussian variables, we can show

$${\text{Var}}_{S_n}[{\widehat{\delta }}^2\mid \mathit{\mu }]=4{\delta }_{\mu }^2\left(\frac{1}{n_0}+\frac{1}{n_1}\right)+2p{\left(\frac{1}{n_0}+\frac{1}{n_1}\right)}^2.$$ (100)

Thus,

$${\text{Var}}_{S_n}[{\widehat{D}}_i\mid \mathit{\mu }]={\left(\frac{{\nu }_i^{\ast }+1}{{\nu }_i^{\ast }}\right)}^2{\text{Var}}_{S_n}[{\widehat{\delta }}^2\mid \mathit{\mu }]\stackrel{K}{\to }0.$$ (101)

As before, from Chebyshev’s inequality it follows that

$$\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{\text{plim}}{\widehat{D}}_i\mid \mathit{\mu }=\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{E}_{S_n}[{\widehat{D}}_i\mid \mathit{\mu }]=D.$$ (102)

By the Continuous Mapping Theorem (continuous functions preserve convergence in probability),

$$\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{\text{plim}}{\widehat{\epsilon }}_i^B\mid \mathit{\mu }=\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{\text{plim}}\mathrm{\Phi }\left({(-1)}^i\frac{-{\widehat{G}}_i^B+c}{\sqrt{{\widehat{D}}_i}}\right)\mid \mathit{\mu }=\mathrm{\Phi }\left(\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{\text{plim}}{(-1)}^i\frac{-{\widehat{G}}_i^B+c}{\sqrt{{\widehat{D}}_i}}\mid \mathit{\mu }\right)=\mathrm{\Phi }\left({(-1)}^i\frac{-G_i^B+c}{\sqrt{D}}\right).$$ (103)

The Dominated Convergence Theorem (|X_n| ≤ B, for some B > 0 and X_n → X in probability implies E[X_n] → E[X])), via the boundedness of ϕ(.) leads to completion of the proof:

$$\begin{array}{l}\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{E}_{S_n}[{\widehat{\epsilon }}_i^B\mid \mathit{\mu }]=\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{E}_{S_n}\left[\mathrm{\Phi }\left({(-1)}^i\frac{-{\widehat{G}}_i^B+c}{\sqrt{{\widehat{D}}_i}}\right)\mid \mu \right]\\ =E_{S_n}\left[\mathrm{\Phi }\left(\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{\text{plim}}{(-1)}^i\frac{-{\widehat{G}}_i^B+c}{\sqrt{{\widehat{D}}_i}}\mid \mathit{\mu }\right)\right]=\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{\text{plim}}{\widehat{\epsilon }}_i^B\mid \mathit{\mu }.\end{array}$$ (104)

Proof of Theorem 2

We first prove that ${\text{Var}}_{S_n}({\widehat{G}}_0^B)\stackrel{K}{\to }0$ with ${\widehat{G}}_0^B$ defined in (94). To do so we use

$${\text{Var}}_{\mathit{\mu },S_n}[{\widehat{G}}_0^B]={\text{Var}}_{\mathit{\mu }}\left[E_{S_n}[{\widehat{G}}_0^B\mid \mathit{\mu }]\right]+E_{\mathit{\mu }}\left[{\text{Var}}_{S_n}[{\widehat{G}}_0^B\mid \mathit{\mu }]\right].$$ (105)

To compute the first term on the right hand side, we have

$$E_{S_n}[{\widehat{G}}_0^B\mid \mathit{\mu }]=\frac{{\nu }_0{\mathbf{m}}_0^T}{n_0+{\nu }_0}{\mathbf{\sum }}^{-1}({\mathit{\mu }}_0-{\mathit{\mu }}_1)+\frac{n_0-{\nu }_0}{2(n_0+{\nu }_0)}\left({\mathit{\mu }}_0^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_0+\frac{p}{n_0}\right)+\frac{1}{2}\left({\mathit{\mu }}_1^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_1+\frac{p}{n_1}\right)-{\mathit{\mu }}_0^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_1\left(\frac{n_0}{n_0+{\nu }_0}\right).$$ (106)

For i, j = 0, 1 and i ≠ j define the following random variables:

$$y_i^{\prime }={\mathbf{m}}_i^T{\mathbf{\sum }}^{-1}({\mathit{\mu }}_0-{\mathit{\mu }}_1),z_i^{\prime }={\mathit{\mu }}_i^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_i,z_{ij}^{\prime }={\mathit{\mu }}_i^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_j.$$ (107)

The variables defined in (107) can be obtained by replacing x̄_i’s with μ_i’s in (95) and x̄_i ~ N(μ_i, Σ/n_i) and μ_i ~ N(m_i, Σ/ν_i). Replacing μ_i with m_i and n_i with ν_i in (96) and (97) yields

$$\begin{array}{l}{\text{Var}}_{\mathit{\mu }}(y_i^{\prime })={\mathbf{m}}_i^T(\frac{{\mathbf{\sum }}^{-1}}{{\nu }_0}+\frac{{\mathbf{\sum }}^{-1}}{{\nu }_1}){\mathbf{m}}_i\stackrel{K}{\to }0,{\text{Var}}_{\mathit{\mu }}(z_i^{\prime })=2\frac{p}{{\nu }_i^2}+4\frac{{\mathbf{m}}_i^T{\mathbf{\sum }}^{-1}{\mathbf{m}}_i}{{\nu }_i}\stackrel{K}{\to }0,\\ {\text{Var}}_{\mathit{\mu }}(z_{ij}^{\prime })=\frac{p}{{\nu }_i{\nu }_j}+\frac{{\mathbf{m}}_i^T{\mathbf{\sum }}^{-1}{\mathbf{m}}_i}{{\nu }_j}+\frac{{\mathbf{m}}_j^T{\mathbf{\sum }}^{-1}{\mathbf{m}}_j}{{\nu }_i}\stackrel{K}{\to }0.\end{array}$$ (108)

By Cauchy-Schwarz, ${\text{Cov}}_{\mathit{\mu }}(y_i^{\prime },z_k^{\prime })\stackrel{K}{\to }0,{\text{Cov}}_{\mathit{\mu }}(y_i^{\prime },z_{ij}^{\prime })\stackrel{K}{\to }0$, and ${\text{Cov}}_{\mathit{\mu }}(z_i^{\prime },z_{ij}^{\prime })\stackrel{K}{\to }0$. Hence, ${\text{Var}}_{\mathit{\mu }}\left[E_{S_n}[{\widehat{G}}_0^B\mid \mathit{\mu }]\right]\stackrel{K}{\to }0$

Now consider the second term on the right hand side of (105). The covariance of a function of Gaussian random variables can be computed from results of [35]. For instance,

$${\text{Cov}}_{S_n}[{\mathbf{a}}^T{\overline{\mathbf{x}}}_i,{\overline{\mathbf{x}}}_i^T{\mathbf{\sum }}^{-1}{\overline{\mathbf{x}}}_i\mid \mathit{\mu }]=\frac{2}{n_i}{\mathbf{a}}^T{\mathit{\mu }}_i.$$ (109)

From (109) and the independence of x̄₀ and x̄₁,

$${\text{Cov}}_{S_n}[{\overline{\mathbf{x}}}_i^T{\mathbf{\sum }}^{-1}{\overline{\mathbf{x}}}_i,{\overline{\mathbf{x}}}_i^T{\mathbf{\sum }}^{-1}{\overline{\mathbf{x}}}_i\mid \mathit{\mu }]=\frac{2}{n_i}{\mu }_j^T{\mathrm{\sum }}^{-1}{\mathit{\mu }}_i,i\ne j$$ (110)

Via (108), (109), and (110), the inner variance in the second term on the right hand side of (105) is

$$\begin{array}{l}{\text{Var}}_{S_n}[{\widehat{G}}_0^B\mid \mathit{\mu }]=\frac{{(n_0-{\nu }_0)}^{2}p}{2n_0^2{(n_0+{\nu }_0)}^2}=\frac{n_{0}p}{n_1{(n_0+{\nu }_0)}^2}+\frac{p}{2n_1^2}\\ +\left(\frac{{(n_0-{\nu }_0)}^2}{n_0{(n_0+{\nu }_0)}^2}+\frac{n_0^2}{n_1{(n_0+{\nu }_0)}^2}\right){\mathit{\mu }}_0^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_0-2\left(\frac{n_0-{\nu }_0}{{(n_0+{\nu }_0)}^2}+\frac{n_0}{n_1(n_0+{\nu }_0)}\right){\mathit{\mu }}_0^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_1\\ +2\left(\frac{{\nu }_0(n_0-{\nu }_0)}{n_0{(n_0+{\nu }_0)}^2}+\frac{{\nu }_0{n}_0}{n_1{(n_0+{\nu }_0)}^2}\right){\mathbf{m}}_0^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_0-2\left(\frac{{\nu }_0}{{(n_0+{\nu }_0)}^2}+\frac{{\nu }_0}{n_1(n_0+{\nu }_0)}\right){\mathbf{m}}_0^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_1+\\ \left(\frac{n_0}{{(n_0+{\nu }_0)}^2}+\frac{1}{n_1}\right){\mathit{\mu }}_1^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_1+\frac{{\nu }_0^2}{{(n_0+{\nu }_0)}^2}\left(\frac{1}{n_0}+\frac{1}{n_1}\right){\mathbf{m}}_0^T{\mathbf{\sum }}^{-1}{\mathbf{m}}_0.\end{array}$$ (111)

Now, again from the results of [35],

$$\begin{array}{l}{E}_{\mathit{\mu }}[{\mathit{\mu }}_i^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_i]={\mathbf{m}}_i^T{\mathbf{\sum }}^{-1}{\mathbf{m}}_i+\frac{p}{{\nu }_i},\\ E_{\mathit{\mu }}[{\mathit{\mu }}_i^T{\mathbf{\sum }}^{-1}{\mathit{\mu }}_j]={\mathbf{m}}_i^T{\mathbf{\sum }}^{-1}{\mathbf{m}}_j,i\ne j.\end{array}$$ (112)

From (111) and (112), some algebraic manipulations yield

$$\begin{array}{l}{E}_{\mathit{\mu }}\left[{\text{Var}}_{S_n}[{\widehat{G}}_0^B\mid \mathit{\mu }]\right]=\frac{{(n_0-{\nu }_0)}^{2}p}{2n_0^2{(n_0+{\nu }_0)}^2}+\frac{n_{0}p}{n_1{(n_0+{\nu }_0)}^2}\\ +\frac{p}{2n_1^2}+\left(\frac{{(n_0-{\nu }_0)}^2}{n_0{(n_0+{\nu }_0)}^2}+\frac{n_0^2}{n_1{(n_0+{\nu }_0)}^2}\right)\frac{p}{{\nu }_0}+\left(\frac{n_0}{{(n_0+{\nu }_0)}^2}+\frac{1}{n_1}\right)\frac{p}{{\nu }_1}+\left(\frac{n_0}{{(n_0+{\nu }_0)}^2}+\frac{1}{n_1}\right){\mathrm{\Delta }}_{\mathbf{m}}^2.\end{array}$$ (113)

From (113) we see that $E_{\mathit{\mu }}[{\text{Var}}_{S_n}[{\widehat{G}}_0^B\mid \mathit{\mu }]]\stackrel{K}{\to }0$. In sum, $Va{r}_{\mathit{\mu },S_n}[{\widehat{G}}_0^B]\stackrel{K}{\to }0$ and similar to the use Chebyshev’s inequality in the proof of Theorem 1, we get

$$\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{\text{plim}}{\widehat{G}}_i^B=\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{E}_{\mathit{\mu },S_n}[{\widehat{G}}_i^B]\triangleq H_i,$$ (114)

with H_i defined in (29).

On the other hand, for D̂_i defined in (99) we can write

$${\text{Var}}_{\mathit{\mu },S_n}[{\widehat{D}}_i]={\text{Var}}_{\mathit{\mu }}\left[E_{S_n}[{\widehat{D}}_i\mid \mathit{\mu }]\right]+E_{\mathit{\mu }}\left[{\text{Var}}_{S_n}[{\widehat{D}}_i\mid \mathit{\mu }]\right].$$ (115)

From similar expressions as in (112) for ${\overline{\mathbf{x}}}_i^T{\mathbf{\sum }}^{-1}{\overline{\mathbf{x}}}_j$, we get $E_{S_n}[{\widehat{\delta }}^2]={\delta }_{\mathit{\mu }}^2+\frac{p}{n_0}+\frac{p}{n_1}$. Moreover, ${\text{Var}}_{\mathit{\mu }}[{\delta }_{\mathit{\mu }}^2]$ is obtained from (100) by replacing n_i with ν_i, and ${\delta }_{\mathit{\mu }}^2$ with ${\mathrm{\Delta }}_{\mathbf{m}}^2$. Thus, from (99),

$$Va{r}_{\mathit{\mu }}\left[E_{S_n}[{\widehat{D}}_i\mid \mathit{\mu }]\right]={\left(\frac{{\nu }_i^{\ast }+1}{{\nu }_i^{\ast }}\right)}^2\left[4{\mathrm{\Delta }}_{\mathbf{m}}^2\left(\frac{1}{{\nu }_0}+\frac{1}{{\nu }_1}\right)+2p{\left(\frac{1}{{\nu }_0}+\frac{1}{{\nu }_0}\right)}^2\right]\stackrel{K}{\to }0.$$ (116)

Furthermore, since $E_{\mathit{\mu }}[{\delta }_{\mathit{\mu }}^2]={\mathrm{\Delta }}_{\mathbf{m}}^2+\frac{p}{{\nu }_0}+\frac{p}{{\nu }_1}$, from (101),

$$E_{\mathit{\mu }}\left[{\text{Var}}_{S_n}[{\widehat{D}}_i\mid \mathit{\mu }]\right]={\left(\frac{{\nu }_i^{\ast }+1}{{\nu }_i^{\ast }}\right)}^2\left[4({\mathrm{\Delta }}_{\mathbf{m}}^2+\frac{p}{{\nu }_0}+\frac{p}{{\nu }_1})\left(\frac{1}{n_0}+\frac{1}{n_1}\right)+2p{\left(\frac{1}{n_0}+\frac{1}{n_1}\right)}^2\right]\stackrel{K}{\to }0.$$ (117)

Hence, $Va{r}_{\mathit{\mu },S_n}[{\widehat{D}}_i]\stackrel{K}{\to }0$ and, similar to (114), we obtain

$$\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{\text{plim}}{\widehat{D}}_i=\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{E}_{\mathit{\mu },S_n}[{\widehat{D}}_0]=\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{E}_{\mathit{\mu },S_n}[{\widehat{D}}_1]\triangleq F,$$ (118)

with F defined in (29). Similar to the proof of Theorem 1, by using the Continuous Mapping Theorem and the Dominated Convergence Theorem, we can show that

$$\begin{array}{l}\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{E}_{\mathit{\mu },S_n}\left[{\widehat{\epsilon }}_i^B\right]=\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{E}_{\mathit{\mu },S_n}\left[\mathrm{\Phi }\left({(-1)}^i\frac{-{\widehat{G}}_i^B+c}{\sqrt{\widehat{D}}}\right)\right]\\ =E_{\mathit{\mu },S_n}\left[\mathrm{\Phi }\left(\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{\text{plim}}{(-1)}^{i+1}\frac{-{\widehat{G}}_i^B+c}{\sqrt{\widehat{D}}}\right)\right]=\mathrm{\Phi }\left({(-1)}^i\frac{-H_i+c}{\sqrt{F}}\right),\end{array}$$ (119)

and the result follows.

Proof of Theorem 3

We start from

$$E_{S_n}[{({\widehat{\epsilon }}_0^B)}^2\mid \mathit{\mu }]=E_{S_n}\left[P\left(U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\le c,U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,{\mathbf{z}}^{\prime })\le c\mid {\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z}\in {\mathrm{\Psi }}_0,{\mathbf{z}}^{\prime }\in {\mathrm{\Psi }}_0,\mathit{\mu }\mid \right)\right],$$ (120)

which was shown in (36). Here we characterize the conditional probability inside E_Sn [.]. From the independence of z, z′, x̄₀, and x̄₁,

$$\left[\begin{array}{c}{U}_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z})\\ U_0({\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,{\mathbf{z}}^{\prime })\end{array}\right]\mid {\overline{\mathbf{x}}}_0,{\overline{\mathbf{x}}}_1,\mathbf{z}\in {\mathrm{\Psi }}_0,{\mathbf{z}}^{\prime }\in {\mathrm{\Psi }}_0,\mathit{\mu }~N\left(\left[\begin{array}{c}{\widehat{G}}_0^B\\ {\widehat{G}}_0^B\end{array}\right],\left[\begin{array}{cc}\widehat{D}& 0\\ 0& \widehat{D}\end{array}\right]\right),$$ (121)

where here N (. , . ) denotes the bivariate Gaussian density function and ${\widehat{G}}_0^B$ and D̂ are defined in (94) and (99). Thus,

$$E_{S_n}[{({\widehat{\epsilon }}_0^B)}^2\mid \mathit{\mu }]={\left[\mathrm{\Phi }\left(\frac{-{\widehat{G}}_0^B+c}{\sqrt{\widehat{D}}}\right)\right]}^2\mid \mathit{\mu }.$$ (122)

Similar to the proof of Theorem 1, we get

$$\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{E}_{S_n}[{({\widehat{\epsilon }}_i^B)}^2\mid \mathit{\mu }]=\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{\text{plim}}{({\widehat{\epsilon }}_i^B)}^2\mid \mathit{\mu }={\left(\underset{\mathrm{b}.\mathrm{k}.\mathrm{a}.\mathrm{c}.}{lim}{E}_{S_n}[{\widehat{\epsilon }}_i^B\mid \mathit{\mu }]\right)}^2={\left[\mathrm{\Phi }\left({(-1)}^i\frac{-G_i^B+c}{\sqrt{D}}\right)\right]}^2.$$ (123)

Similarly, we obtain ${lim}_{b.k.a.c.}E[{\widehat{\epsilon }}_0^B{\widehat{\epsilon }}_1^B]={lim}_{b.k.a.c.}{\widehat{\epsilon }}_0^B{\epsilon }_1^B$, and the results follow.