Analytical Study of Performance of Linear Discriminant Analysis in Stochastic Settings

Abstract

This paper provides exact analytical expressions for the first and second moments of the true error for linear discriminant analysis (LDA) when the data are univariate and taken from two stochastic Gaussian processes. The key point is that we assume a general setting in which the sample data from each class do not need to be identically distributed or independent within or between classes. We compare the true errors of designed classifiers under the typical i.i.d. model and when the data are correlated, providing exact expressions and demonstrating that, depending on the covariance structure, correlated data can result in classifiers with either greater error or less error than when training with uncorrelated data. The general theory is applied to autoregressive and moving-average models of the first order, and it is demonstrated using real genomic data.

1. Introduction

It is common in practice to assume that the training data used to construct a classifier are independent and identically distributed (i.i.d). Should the data be dependent or not identically distributed, the classifier performance is affected. This paper presents a mathematical framework for analytically studying classifiers in such situations in general, and the univariate LDA (linear discriminant analysis) classifier in particular. We pay particular attention to the univariate LDA model because it is possible to obtain closed-form (not asymptotic) results for moments of the error – in analogy to moments for the error [1, 2] and error estimates [1, 3] for univariate LDA with i.i.d. sampling. The desired framework is achieved by placing classifier performance in a stochastic setting where the training data are univariate dependent and not necessarily identically distributed.

Motivation for this line of research goes back to the early 1970’s when Basu and Odell observed in remote sensing applications that the conditional expected true error of LDA is commonly higher than what is expected from a theoretical analysis [4]. They associated this observation with violation of the independence assumption on the training data.

To study the effect of correlated training data on the performance of LDA, Basu and Odell [4] used numerical examples under an equicorrellated structure of samples (see Appendix for definition of various correlation structures). They showed that misclassification probabilities change under such structures. Afterwards, McLachlan [5] used asymptotic analysis to show that even under a simple-equicorrelated structure the probability of misclassification changes. Later, Tubbs [6] used a similar asymptotic analysis but with a serially correlated structure among training data. He considered further simplifying assumptions to show that the asymptotic error rate changes with serially correlated data having positive correlations. Lawoko and McLachlan [7] used the same serially correlated structure and obtained a different asymptotic expansion of LDA true error from the one that Tubbs previously achieved in [6]. This type of asymptotic analysis was later used in [7, 8] to characterize the asymptotic expected true error of univariate LDA and Z-statistics assuming an autoregressive process of order p.

Typically, large-sample asymptotic results are not helpful in small-sample situations. Going back to 1925, R. A. Fisher wrote, “Only by systematically tackling small sample problems on their merits does it seem possible to apply accurate tests to practical data” [9, 10]. This understanding led us to study the distribution and exact moments of LDA true error and comnon estimators [11, 3, 12, 13].

Having laid the groundwork for analyzing LDA related statistics in small-sample situations, in this work we establish a framework for studying LDA in stochastic settings, thereby allowing us to obtain the exact first and second moments of univariate LDA true error in a general stochastic setting. We neither impose a specific correlation structure on the training data, nor do we assume the training data have necessarily the same mean or variance. For example the basic assumption in [4, 5, 6, 7, 8] is that the training data of the two classes are taken separately from two class conditional densities Π₀, for class 0, and Π₁, for class 1. This assumption immediately imposes several restrictions on the problem: the training data from each class have the same mean and variance (because they are coming from the same distribution) and, furthermore, only intraclass correlations exist. The stochastic setting permits us to generalize such assumptions to training data being correlated across classes or the samples from each class being differently distributed. To model such data we employ Gaussian processes and we assume the samples are taken from class conditional processes rather than class conditional densities.

Another related line of research is the work on classification of stationary time series data [14, 15, 16]. The main focus in this work is to construct linear discriminant rules with the knowledge of having stationary data. In this framework the discriminant function is commonly the one which maximizes some measure of disparity between two multivariate densities, e.g. the Kullback-Leibler information measure. This means that the linear discriminant rules constructed here are no longer what is commonly known as LDA. Therefore, the main difference between the aforementioned results on studying the performance of LDA under correlated training data and the body of work on classification of stationary times series, is that the former focuses on the analysis of the effect of correlated training data (which may have a stationary structure) on the performance of LDA, and the latter focuses on the synthesis of new classification rules with the knowledge of having stationary time series. Our work is of the first type. We study the effect of training data that can be dependent and not necessarily identically distributed or stationary on the performance of LDA.

As an application of these results, we consider two commonly used models, first-order autoregressive and moving averages. We further study the exact effect of autoregressive or moving-average model coefficients on changing the expected true error of LDA. Finally, we present numerical experiments to study several specific settings using the theory.

Before proceeding we note that univariate classification has played a major role in the history of pattern recogntion, in part, because of the ability to obtain closed-form solutions for error moments [1, 2, 3]; however, we should not overlook practical application. Indeed, most common tests for diagnosis and prognosis of cancer are univariate: PSA for prostate cancer [21], AFP for liver cancer [22], CA 125 for ovarian cancer [23], and CA 19.9 for colorectal cancer [24] are major protein markers. In addition to these protein biomarkers, there are genomic markers such as BRCA1 for breast cancer [25], BRCA2 [26] for male breast cancer, and APC for pancreatic cancer [27] that are major genomic markers.

2. Linear Discriminant Analysis and Error Estimation: Independent Sampling

In this section, we present the traditional sampling scenario in which LDA is employed in a univariate setting. Consider a set of n = n₀ + n₁ independent sample points in ℝ, where X₁, X₂, …, X_n0 come from population Π₀ and X_n0+1, X_n0+2, …, X_n0+n1 come from population Π₁. Population Π_i is assumed to follow a univariate Gaussian distribution $N({\mu }_i,{\sigma }_i^2)$, for i = 0, 1. Linear Discriminant Analysis (LDA) utilizes the Anderson W statistic, which in the univariate case is presented as

$$W({\overline{X}}^0,{\overline{X}}^1,X)=\frac{1}{{\widehat{\sigma }}^2}{\left(X-\frac{{\overline{X}}^0+{\overline{X}}^1}{2}\right)}^T({\overline{X}}^0-{\overline{X}}^1),$$ (1)

where ${\overline{X}}^0=\frac{1}{n_0}{\sum }_{i=1}^{n_0}{X}_i$ and ${\overline{X}}^1=\frac{1}{n_1}{\sum }_{i=n_0+1}^{n_0+n_1}{X}_i$ are the sample means for each class and σ̂² is the pooled estimate of the variance of classes, which is assumed to be common in the LDA discriminant. Given X̄⁰ and X̄¹, the designed LDA classifier is given by

$$\psi (X)=\{\begin{array}{cc}1,& \text{if}W({\overline{X}}^0,{\overline{X}}^1,X)\le c\\ 0,& \text{if}W({\overline{X}}^0,{\overline{X}}^1,X)>c\end{array},$$ (2)

with c being a constant. It is commonly assumed that c is zero [17], which is the assumption we also make throughout this paper. Therefore, the sign of W determines the classification of the sample point X and since σ̂² > 0, (1) reduces to

$$W({\overline{X}}^0,{\overline{X}}^1,X)=(X-\overline{X})\left({\overline{X}}^0-{\overline{X}}^1\right)$$ (3)

where $\overline{X}=\frac{{\overline{X}}^0+{\overline{X}}^1}{2}$. Given the training data S_n (and thus X̄₀ and X̄₁), the classification error, also known as true error, is given by

$$\epsilon =P(W({\overline{X}}^0,{\overline{X}}^1,X)\le 0,X\in {\mathrm{\Pi }}_0\mid {\overline{X}}^0,{\overline{X}}^1)+P(W({\overline{X}}^0,{\overline{X}}^1,X)>0,X\in {\mathrm{\Pi }}_1\mid {\overline{X}}^0,{\overline{X}}^1)={\alpha }_0{\epsilon }^0+{\alpha }_1{\epsilon }^1,$$ (4)

where α_i = P (X ∈ Π_i) is the a priori mixing probability for population Π_i and εⁱ is the error rate specific to population Π_i, with

$${\epsilon }^i=P({(-1)}^{i}W({\overline{X}}^0,{\overline{X}}^1,X)\le 0\mid X\in {\mathrm{\Pi }}_i,{\overline{X}}^0,{\overline{X}}^1).$$ (5)

The first and second moments of the classification error are given by

$$E[\epsilon ]=\sum _{i=0}^1{\alpha }_{i}P({(-1)}^{i}W({\overline{X}}^0,{\overline{X}}^1,X)\le 0\mid X\in {\mathrm{\Pi }}_i),$$ (6)

and

$$E[{\epsilon }^2]=E[{({\alpha }_0{\epsilon }^0+{\alpha }_1{\epsilon }^1)}^2]=2{\alpha }_0{\alpha }_{1}E[{\epsilon }^0{\epsilon }^1]+\sum _{i=0}^1{\alpha }_i^{2}E[{\epsilon }^i{\epsilon }^i].$$ (7)

3. Performance of LDA classifier in Univariate Gaussian Dependent Sampling (UGDS) Model of Binary Classification

We now provide the mathematical framework to study LDA performance in a stochastic setting.

Definition 1

A process X_t = {X_t : t ∈ T} with T being an ordered set, is called a Gaussian process if any finite-dimensional vector [X_t1, X_t2, …, X_tn]^T has the multivariate normal distribution N(μ_T, Σ_T), where

$${\mathit{\mu }}_T={[E(X_{t_1}),E(X_{t_2}),\dots ,E(X_{t_n})]}^T={[{\mu }_1,{\mu }_2,\dots ,{\mu }_n]}^T$$

and Σ_T is the covariance matrix dependent on T = [t₁, t₂, …, t_n].

Definition 2

We refer to the following sampling procedure as the Univariate Gaussian Dependent Sampling (UGDS) Model of Binary Classification: ${\mathbf{X}}_t^i=\{X_{t^i}^i:t^i\in {\mathbf{T}}^i\}$, with Tⁱ being two ordered sets for i = 0, 1, are two Gaussian processes such that any finite-dimensional vector constructed by stacking the random variables of ${\mathbf{X}}_{t^0}^0$ and ${\mathbf{X}}_{t^1}^1$ as ${[X_{t_1^0}^0,X_{t_2^0}^0,\dots ,X_{t_{n_0}^0}^0,X_{t_1^1}^1,X_{t_2^1}^1,\dots ,X_{t_{n_1}^1}^1]}^T$ possesses a multivariate normal distribution N(μ_T, Σ_T), where ${\mathit{\mu }}_T={[{\mu }_1^0,{\mu }_2^0,\dots ,{\mu }_{n_0}^0,{\mu }_1^1,{\mu }_2^1,\dots ,{\mu }_{n_1}^1]}^T$, and

$${\mathbf{\sum }}_T=\left[\begin{array}{cc}{\mathbf{\sum }}_{n_0\times n_0}^{00}& {\mathbf{\sum }}_{n_0\times n_1}^{01}\\ {\mathbf{\sum }}_{n_1\times n_0}^{10}& {\mathbf{\sum }}_{n_1\times n_1}^{11}\end{array}\right]$$ (8)

is a positive definite covariance matrix.

This model is univariate because both processes, ${\mathbf{X}}_{t^0}^0$ and ${\mathbf{X}}_{t^1}^1$, are collections of univariate random variables, not necessarily with the same means or variances. ${\mathbf{X}}_{t^0}^0$ and ${\mathbf{X}}_{t^1}^1$ are called class conditional processes. For ease of notations and without loss of mathematical generality, we assume that T⁰ and T¹ are the same set and, therefore, we omit the superscript i from tⁱ. Thus, henceforth we denote ${\mathbf{X}}_{t^i}^i$ by ${\mathbf{X}}_t^i$ and the stacked vector ${[X_{t_1^0}^0,X_{t_2^0}^0,\dots ,X_{t_{n_0}^0}^0,X_{t_1^1}^1,X_{t_2^1}^1,\dots ,X_{t_{n_1}^1}^1]}^T$ by ${[X_{t_1}^0,X_{t_2}^0,\dots ,X_{t_{n_0}}^0,X_{t_1}^1,X_{t_2}^1,\dots ,X_{t_{n_1}}^1]}^T$.

Remark 1

If we assume μ_T = [μ^0T, μ^1T]T, with ${\mathit{\mu }}^i={[{\mu }^i,{\mu }^i,\dots .,{\mu }^i]}_{1\times n_i}^T,{\mathbf{\sum }}_{jj}^{ii}={({\sigma }^i)}^2$, i = 0, 1, j = 1, 2, …, n_i, where (σⁱ)² is the variance of class conditional distributions and ${\mathbf{\sum }}_{jj}^{ii}$ indicates the diagonal elements of matrix Σⁱⁱ, ${\mathbf{\sum }}_{jk}^{ii}=0$, i = 0, 1, j, k = 1, …, n_i, j ≠ k, Σ⁰¹ = 0_n0×n1 = Σ^10T, and any future sample is independent from the training data and distributed either as N(μ⁰, (σ⁰)²) or N(μ¹, (σ¹)²), depending on its class, then the UGDS model reduces to the traditional i.i.d. sampling scenario defined in section 2. Because we will want to compare classifier errors in the dependent and independent scenarios, we will sometimes use ε^D and ε^I to denote errors in the respective settings.

Similar to (3), employing LDA with the UGDS model instead of traditional independent sampling in order to classify a sample point taken at t, denoted by X_t, results in the following W statistic for the univariate case

$$W({\overline{X}}_T^0,{\overline{X}}_T^1,X_t)=(X_t-{\overline{X}}_T)({\overline{X}}_T^0-{\overline{X}}_T^1),$$ (9)

where ${\overline{X}}_T^0=\frac{1}{n_0}{\sum }_{i=1}^{n_0}{X}_{t_i}^0$ and ${\overline{X}}_T^1=\frac{1}{n_1}{\sum }_{i=1}^{n_1}{X}_{t_i}^1$ are the sample means for each class and ${\overline{X}}_T=\frac{{\overline{X}}_T^0+{\overline{X}}_T^1}{2}$. The designed LDA classifier is given by

$$\psi (X_t)=\{\begin{array}{cc}1,& \text{if}W({\overline{X}}_T^0,{\overline{X}}_T^1,X_t)\le 0\\ 0,& \text{if}W({\overline{X}}_T^0,{\overline{X}}_T^1,X_t)>0\end{array}.$$ (10)

For the ease of notation, hereafter, we omit the subscript T from μ_T and Σ_T.

3.1. Stochastic true error and its moments

Let $X_{t_s}^i$ denote a test sample point, where i indicates the class conditional process in which the sample is coming from, i.e. either ${\mathbf{X}}_t^0$ or ${\mathbf{X}}_t^1$. The auto-covariance sequence of $X_{t_s}^i$ with the training data is defined as

$${\rho }_s^{ik}(j)=E[(X_{t_s}^i-{\mu }_s^i)(X_{t_j}^k-{\mu }_j^k)],i,k=0,1,j=1,2,\dots ,n_k,$$ (11)

where ${\rho }_s^{ik}(j)$ is the j^th element of the sequence ${\rho }_s^{ik}$. Since $X_{t_s}^i$ is a future sample point, we assume 2 ≤ max{n₀, n₁} < s, unless otherwise stated. Throughout the paper, we use S_A to denote the sum of all elements of a matrix or vector A. For instance, $S_{{\rho }_s^{ik}}={\sum }_{j=1}^{n_i}{\rho }_s^{ik}(j)$.

The true classifier error under the UGDS model is a function of t_s. Sample points at t_s can come from either processes and the classifier may misclassify any of these. Hence,

$${\epsilon }_{t_s}={\alpha }_{t_s}^0{\epsilon }_{t_s}^0+{\alpha }_{t_s}^1{\epsilon }_{t_s}^1,$$ (12)

where ${\alpha }_{t_s}^i=P(X_{t_s}\in {\mathbf{X}}_t^i)$, i = 0, 1, is the a priori mixing probability of the two processes ${\mathbf{X}}_t^0$ and ${\mathbf{X}}_t^1$ at t_s and ${\epsilon }_{t_s}^i$ is the error rate specific to each process, with

$${\epsilon }_{t_s}^i=P({(-1)}^{i}W({\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s})\le 0\mid {\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s}\in {\mathbf{X}}_t^i).$$ (13)

By replacing $W({\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s})$ with any proper statistic used in other classifiers, this stochastic definition of true error applies to other rules. The expected performance of true error is also specific to t_s:

$$E[{\epsilon }_{t_s}]=\sum _{i=0}^1{\alpha }_{t_s}^{i}P({(-1)}^{i}W({\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s})\le 0\mid X_{t_s}\in {\mathbf{X}}_t^i).$$ (14)

In (12), the true error is indexed. One could, if desired, define the true error of a classifier to be the average error the classifier induces over an index set of interest, namely, ${\epsilon }_{t_{s_1-s_2}}=\frac{1}{s_2-s_1}{\sum }_{s=s_1}^{s_2}{\epsilon }_{t_s}$. Since characterizing ε_ts yields a characterization of ε_ts1−s2, no generality is gained by averaging and we restrict our attention to ε_ts. The second moment is also a function of t_s and from (12) we get

$$E[{\epsilon }_{t_s}^2]=2{\alpha }_{t_s}^0{\alpha }_{t_s}^{1}E[{\epsilon }_{t_s}^0{\epsilon }_{t_s}^1]+\sum _{i=0}^1{({\alpha }_{t_s}^i)}^{2}E[{({\epsilon }_{t_s}^i)}^2].$$ (15)

First focusing on $E[{({\epsilon }_{t_s}^0)}^2]$, the square of the probability defining ${({\epsilon }_{t_s}^0)}^2$ can be factored by introducing the random variable $X_{t_s}^{\prime }\in {\mathbf{X}}_t^0$. Writing the probabilities as integrals of indicator functions allows us to apply Fubini’s theorem, which shows X_ts and $X_{t_s}^{\prime }$ to be independent (denoted $X_{t_s}\perp X_{t_s}^{\prime }$). The expectation can then be applied. Altogether,

$$\begin{array}{l}E[{({\epsilon }_{t_s}^0)}^2]=E[P{(W({\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s})\le 0\mid {\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s}\in {\mathbf{X}}_t^0)}^2]\\ =E\left[P(W({\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s})\le 0\mid {\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s}\in {\mathbf{X}}_t^0)\times P(W({\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s}^{\prime })\le 0\mid {\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s}^{\prime }\in {\mathbf{X}}_t^0)\right]\\ =E\left[P\left(W({\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s})\le 0,W({\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s}^{\prime })\le 0\mid {\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s},X_{t_s}^{\prime }\in {\mathbf{X}}_t^0,X_{t_s}\perp X_{t_s}^{\prime }\right)\right]\\ =P\left(W({\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s})\le 0,W({\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s}^{\prime })\le 0\mid X_{t_s},X_{t_s}^{\prime }\in {\mathbf{X}}_t^0,X_{t_s}\perp X_{t_s}^{\prime }\right).\end{array}$$ (16)

$E[{({\epsilon }_{t_s}^1)}^2]$ and $E[{\epsilon }_{t_s}^0{\epsilon }_{t_s}^1]$

can be expressed via similar factorizations. Hence,

$$\begin{array}{l}E[{\epsilon }_{t_s}^2]={({\alpha }_{t_s}^0)}^{2}P\left(W({\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s})\le 0,W({\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s}^{\prime })\le 0\mid X_{t_s},X_{t_s}^{\prime }\in {\mathbf{X}}_t^0,X_{t_s}\perp X_{t_s}^{\prime }\right)\\ +2{\alpha }_{t_s}^0{\alpha }_{t_s}^{1}P\left(W({\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s})\le 0,W({\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s}^{\prime })>0\mid X_{t_s}\in {\mathbf{X}}_t^0,X_{t_s}^{\prime }\in {\mathbf{X}}_t^1,X_{t_s}\perp X_{t_s}^{\prime }\right)\\ +{({\alpha }_{t_s}^1)}^{2}P\left(W({\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s})>0,W({\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s}^{\prime })>0\mid X_{t_s},X_{t_s}^{\prime }\in {\mathbf{X}}_t^1,X_{t_s}\perp X_{t_s}^{\prime }\right).\end{array}$$ (17)

To facilitate the subsequent discussion, we will explicitly denote the dependency of true error on the number of samples. Therefore, hereafter we use ε_ts,n0+n1 and ε_ts, or ${\epsilon }_{t_s,n_0+n_1}^2$ and ${\epsilon }_{t_s}^2$

, interchangeably.

Throughout the paper, we use the notations Z < 0 or Z ≥ 0 to indicate componentwise inequalities, e.g. Z = (z₁, z₂)^T < 0 means z₁ < 0, z₂ < 0.

3.2. Expected performance of LDA in the UGDS model

The first moment of the classification error for LDA under the UGDS model is expressed exactly according to the following theorem.

Theorem 1

Under the UGDS model, the expected true error of LDA at t_s is

$$E[{\epsilon }_{t_s,n_0+n_1}^D]={\alpha }_{t_s}^0\left[P(Z_s^1<\mathbf{0})+P(Z_s^1\ge \mathbf{0})\right]+{\alpha }_{t_s}^1\left[P(Z_s^{II}<\mathbf{0})+P(Z_s^{II}\ge \mathbf{0})\right],$$ (18)

where $Z_{t_s}^I$ and $Z_{t_s}^{II}$ are Gaussian bivariate vectors with

$$\begin{array}{l}{\mu }_{Z_s^I}={[\begin{array}{cc}{\mu }_s^0-\frac{\overline{\mu }}{2}& -{\mu }^{\prime }\end{array}]}^T,{\mu }_{Z_s^{II}}={[\begin{array}{cc}{\mu }_s^1-\frac{\overline{\mu }}{2}& {\mu }^{\prime }\end{array}]}^T,\\ {\mathbf{\sum }}_{Z_s^I}=\left[\begin{array}{cc}{({\sigma }_s^0)}^2-\frac{S_{{\rho }_s^{00}}}{n_0}-\frac{S_{{\rho }_s^{01}}}{n_1}+\frac{S_{{\mathbf{\sum }}^{00}}}{4n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}}{4n_1^2}+\frac{S_{{\mathbf{\sum }}^{01}}}{2n_0{n}_1}& \frac{-S_{{\rho }_s^{00}}}{n_0}+\frac{S_{{\rho }_s^{01}}}{n_1}+\frac{S_{{\mathbf{\sum }}^{00}}}{2n_0^2}-\frac{S_{{\mathbf{\sum }}^{11}}}{2n_1^2}\\ ·& \frac{S_{{\mathbf{\sum }}^{00}}}{n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}}{n_1^2}-\frac{2S_{{\mathbf{\sum }}^{01}}}{n_0{n}_1}\end{array}\right],\\ {\mathbf{\sum }}_{Z_s^{II}}=\left[\begin{array}{cc}{({\sigma }_s^1)}^2-\frac{S_{{\rho }_s^{11}}}{n_1}-\frac{S_{{\rho }_s^{10}}}{n_0}+\frac{S_{{\mathbf{\sum }}^{00}}}{4n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}}{4n_1^2}+\frac{S_{{\mathbf{\sum }}^{01}}}{2n_0{n}_1}& \frac{-S_{{\rho }_s^{11}}}{n_1}+\frac{S_{{\rho }_s^{10}}}{n_0}-\frac{S_{{\mathbf{\sum }}^{00}}}{2n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}}{2n_1^2}\\ ·& \frac{S_{{\mathbf{\sum }}^{00}}}{n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}}{n_1^2}-\frac{2S_{{\mathbf{\sum }}^{01}}}{n_0{n}_1}\end{array}\right],\end{array}$$ (19)

where $\overline{\mu }=\frac{{\sum }_{i=1}^{n_0}{\mu }_i^0}{n_0}+\frac{{\sum }_{i=1}^{n_1}{\mu }_i^1}{n_1},{\mu }^{\prime }=\frac{{\sum }_{i=1}^{n_0}{\mu }_i^0}{n_0}-\frac{{\sum }_{i=1}^{n_1}{\mu }_i^1}{n_1}$, and ${\mu }_s^i$ and ${({\sigma }_s^i)}^2$ are the mean and variance of random variables at t_s from class i, i = 0, 1, with the auto-covariance ${\rho }_s^{ik}$ defined as in (11).

Proof

See the Appendix.

We note that under conditions stated in Remark 1, Theorem 1 reduces to Theorem 1 in [3].

Let Φ(x, y; ρ) be the cumulative bivariate normal distribution defined as:

$$\begin{array}{l}\mathrm{\Phi }(x,y;\rho )=\underset{-\infty }{\overset{x}{\int }}\underset{-\infty }{\overset{y}{\int }}\psi (u,v;\rho )dudv,\\ \psi (u,v;\rho )=\frac{1}{2\pi \sqrt{1-{\rho }^2}}exp\left\{\frac{-(u^2+v^2-2\rho uv)}{2(1-{\rho }^2)}\right\}.\end{array}$$ (20)

We have the following Corollary.

Corollary 2

In the model considered in Theorem 1, let the training samples from each class have the same mean, that is μ = [μ^0T, μ^1T]^T in which ${\mathit{\mu }}^i={[{\mu }^i,{\mu }^i,\dots ,{\mu }^i]}_{1\times n_i}^T,{\mu }_s^i={\mu }^i,{\sigma }_s^i=\sigma$, i = 0, 1, meaning the test data at t_s has equal variances across classes, and ${\alpha }_{t_s}^0={\alpha }_{t_s}^1=0.5$. Furthermore, let $S_{{\rho }_s^{ik}}=0$ , i, k = 0, 1 Then

$$E[{\epsilon }_{t_s,n_0+n_1}^D]=\frac{1}{2}-\frac{L(h,k;\rho )}{2},$$ (21)

where

$$L(x,y;\rho )=\underset{-x}{\overset{x}{\int }}\underset{-y}{\overset{y}{\int }}\psi (u,v;\rho )dudv,$$ (22)

$$\begin{array}{l}h=\frac{\mu }{2\sqrt{a}},k=\frac{\mu }{\sqrt{b}},\mu ={\mu }^0-{\mu }^1,\rho =\frac{\frac{S_{{\mathbf{\sum }}^{00}}}{2n_0^2}-\frac{S_{{\mathbf{\sum }}^{11}}}{2n_1^2}}{\sqrt{a}\sqrt{b}},\\ a={\sigma }^2+\frac{S_{{\mathbf{\sum }}^{00}}}{4n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}}{4n_1^2}+\frac{S_{{\mathbf{\sum }}^{01}}}{2n_0{n}_1},b=\frac{S_{{\mathbf{\sum }}^{00}}}{n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}}{n_1^2}-\frac{2S_{{\mathbf{\sum }}^{01}}}{n_0{n}_1}.\end{array}$$ (23)

Proof

See the Appendix.

To further proceed we present the following lemma, in which ∧ denotes conjunction.

Lemma 3

Let Φ(x, y; ρ) be the cumulative bivariate normal distribution defined in (20) and defne F(x, y; ρ) and G(x, y; ρ) as follows:

$$\begin{array}{l}F(x,y;\rho )=\mathrm{\Phi }(x,y;\rho )+\mathrm{\Phi }(-x,-y;\rho ),\\ G(x,y;\rho )=F(x,y;\rho )+F(x,y;-\rho ),\end{array}$$ (24)

where x and y are two constants such that xy < 0. Then, for 0 ≤ λ_x < 1, 0 ≤ λ_y < 1,

$$(\mid {\rho }_1\mid \le \mid {\rho }_0\mid )\wedge (x_1={\lambda }_x{x}_0)\wedge (y_1={\lambda }_y{y}_0)\Rightarrow G(x_1,y_1;{\rho }_1)>G(x_0,y_0;{\rho }_0).$$ (25)

Proof

See the Appendix.

Using this Lemma, we compare the expected true error of the UGDS model with the independent sampling model.

Corollary 4

In the model considered in Corollary 2, let ${({\sigma }_j^i)}^2\triangleq {\mathbf{\sum }}_{jj}^{ii}$, i = 0, 1, j = 1, 2, …, n_i, and let

$$\begin{array}{l}{\rho }_D=\frac{\frac{S_{{\mathbf{\sum }}^{00}}}{n_0^2}-\frac{S_{{\mathbf{\sum }}^{11}}}{n_1^2}}{\sqrt{({\sigma }^2+\frac{S_{{\mathbf{\sum }}^{00}}}{4n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}}{4n_1^2}+\frac{S_{{\mathbf{\sum }}^{01}}}{2n_0{n}_1})(\frac{S_{{\mathbf{\sum }}^{00}}}{n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}}{n_1^2}-\frac{2S_{{\mathbf{\sum }}^{01}}}{n_0{n}_1})}},\\ {\rho }_I=\frac{\frac{{\sum }_{j=1}^{n_0}{({\sigma }_j^0)}^2}{n_0^2}-\frac{{\sum }_{j=1}^{n_1}{({\sigma }_j^1)}^2}{n_1^2}}{\sqrt{({\sigma }^2+\frac{{\sum }_{j=1}^{n_0}{({\sigma }_j^0)}^2}{4n_0^2}+\frac{{\sum }_{j=1}^{n_1}{({\sigma }_j^1)}^2}{4n_1^2})(\frac{{\sum }_{j=1}^{n_0}{({\sigma }_j^0)}^2}{n_0^2}+\frac{{\sum }_{j=1}^{n_1}{({\sigma }_j^1)}^2}{n_1^2})}}.\end{array}$$ (26)

Let $E[{\epsilon }_{t_s,n_0+n_1}^I]$ be the expectaton of the true error of the classifier in (10) specific to t_s and constructed as if all n₀ + n₁ training samples are i.i.d. (same mean and variance). Then

$$(\mid {\rho }_D\mid \le \mid {\rho }_I\mid )\wedge \left(\frac{S_{{\mathbf{\sum }}^{00}}^{\prime }}{n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}^{\prime }}{n_1^2}\ge max\{\frac{2S_{{\mathbf{\sum }}^{01}}}{n_0{n}_1},\frac{-2S_{{\mathbf{\sum }}^{01}}}{n_0{n}_1}\}\right)\Rightarrow E[{\epsilon }_{t_s,n_0+n_1}^D]\ge E[{\epsilon }_{t_s,n_0+n_1}^I],$$ (27)

$$(\mid {\rho }_D\mid \ge \mid {\rho }_I\mid )\wedge \left(\frac{S_{{\mathbf{\sum }}^{00}}^{\prime }}{n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}^{\prime }}{n_1^2}\le min\{\frac{2S_{{\mathbf{\sum }}^{01}}}{n_0{n}_1},\frac{-2S_{{\mathbf{\sum }}^{01}}}{n_0{n}_1}\}\right)\Rightarrow E[{\epsilon }_{t_s,n_0+n_1}^D]\le E[{\epsilon }_{t_s,n_0+n_1}^I],$$ (28)

where $S_A^{\prime }$ is the sum of the off diagonal elements of matrix A, defined as $S_A^{\prime }={\sum }_{i,j,i\ne j}{a}_{ij}$

Proof

Find the expected true error in Theorem 1 using the conditions in the corollary and compare it to the expected true error determined by setting all off diagonal elements of Σⁱⁱ to zero, i = 0, 1 and Σ⁰¹ = 0_n0×n1. The proof follows by using the results of Lemma 3 in Theorem 1.

A more restricted set of sufficient conditions than those presented in Corollary 4 follows.

Corollary 5

In the model considered in Corollary 2, let ${({\sigma }_j^i)}^2\triangleq {\mathbf{\sum }}_{jj}^{ii}$, i = 0, 1, j = 1, 2, …, n_i, and

$$\frac{S_{{\mathbf{\sum }}^{00}}}{n_0^2}-\frac{S_{{\mathbf{\sum }}^{11}}}{n_1^2}=\frac{{\sum }_{j=1}^{n_0}{({\sigma }_j^0)}^2}{n_0^2}-\frac{{\sum }_{j=1}^{n_1}{({\sigma }_j^1)}^2}{n_1^2}.$$ (29)

Then

$$\frac{S_{{\mathbf{\sum }}^{00}}^{\prime }}{n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}^{\prime }}{n_1^2}\ge max\{\frac{2S_{{\mathbf{\sum }}^{01}}}{n_0{n}_1},\frac{-2S_{{\mathbf{\sum }}^{01}}}{n_0{n}_1}\}\Rightarrow E[{\epsilon }_{t_s,n_0+n_1}^D]\ge E[{\epsilon }_{t_s,n_0+n_1}^I],$$ (30)

$$\frac{S_{{\mathbf{\sum }}^{00}}^{\prime }}{n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}^{\prime }}{n_1^2}\ge min\{\frac{2S_{{\mathbf{\sum }}^{01}}}{n_0{n}_1},\frac{-2S_{{\mathbf{\sum }}^{01}}}{n_0{n}_1}\}\Rightarrow E[{\epsilon }_{t_s,n_0+n_1}^D]\ge E[{\epsilon }_{t_s,n_0+n_1}^I],$$ (31)

where $S_A^{\prime }$ is the sum of off diagonal elements of matrix A, defined as $S_A^{\prime }={\sum }_{i,j,i\ne j}{a}_{ij}$

Proof

The proof is similar to Corollary 4.

To have a sense of the conditions stated in Corollary 5, consider a scenario in which n₀ = n₁, the sample points in each class are equi-correlated with correlation ρ, and there is independent sampling across classes. This satisfies (29). If ρ > 0, then (30) holds and $E[{\epsilon }_{t_s,n_0+n_1}^D]\ge E[{\epsilon }_{t_s,n_0+n_1}^I]$. If ρ < 0 and the class covariance matrices are positive definite, then (31) holds and $E[{\epsilon }_{t_s,n_0+n_1}^D]<E[{\epsilon }_{t_s,n_0+n_1}^I]$.

Let us reflect on Corollaries 4 and 5. A correlated set of n sample points can be considered as a set in which the points convey some information about each other. Therefore, they are often considered to be as informative as n′ independent samples with n′ < n, thereby producing a poorer classifier. This intuition aligns with the simple situation in which the sample points in each class are equi-correlated with ρ > 0 and the sample points across the two classes are uncorrelated. This scenario is a special case of (30) and $E[{\epsilon }_{t_s,n_0+n_1}^D]\ge E[{\epsilon }_{t_s,n_0+n_1}^I]$. However, (31) shows that there are correlation patterns that result in an expected true error smaller than it would be were there independent sampling, which means that sampling satisfying (31) is like having a larger sample size than if sampling were independent.

To illustrate, in the UGDS model suppose the training sample points are identically distributed as two Gaussian distribution, N(−1, 1) for class 0 and N(1, 1) for class 1. Let n₀ = n₁ = 3 and assume that any future test point is also distributed identically to the training data of its class. Furthermore, assume the data are generated via two different scenarios, a and b, such that Σ⁰¹ = 0_3×3 and, for i = 0, 1,

$${\mathbf{\sum }}_a^{ii}=\left[\begin{array}{ccc}1& -1/4& -1/4\\ -1/4& 1& \rho \\ -1/4& \rho & 1\end{array}\right],{\mathbf{\sum }}_b^{ii}=\left[\begin{array}{ccc}1& 1/4& 1/4\\ 1/4& 1& \rho \\ 1/4& \rho & 1\end{array}\right]$$ (32)

Figure 1(a) shows the expected true error of the classifier designed in scenario a as a function of ρ. It demonstrates that for some dependency patterns, as defined by the covariance matrix, the classifier has better performance than if the sampling were independent. Note that in Fig. 1(a) the curves meet at ρ = 0.5, the point of equality for the inequalities (30) and (31). Note also that for ρ = −0.499, $E[{\epsilon }_6^D]=E[{\epsilon }_{18}^I]=0.165$. Hence, for the sampling covariance matrix (32), 3 points have the effect of 9 independent points. In general, better classification accuracy may be achieved if the sample points are collected according to specific schemes. Equations (28) and (31) provide sufficient sets of conditions that result in such schemes.

Figure 1.

(a) Expected true error of constructed classifiers in scenario a as a function of ρ, (b) Expected true error of constructed classifiers in scenario b as a function of ρ. The horizontal line shows the performance of the constructed classifier as if the samples were independent. Solid lines: dependent samples; Dashed lines: independent samples.

Figure 1(b) shows the expected true error of a classifier constructed in scenario b by varying ρ in the same range as in scenario a. The only difference between scenarios a and b is changing the covariances between the first sample point and other sample points to positive values. It results in the curve for dependent sampling in Figure 1(b) being substantially above the curve for independent sampling.

3.3. Second moment of LDA true error in the UGDS model

Next we obtain the second moment of true error of LDA at t_s as defined in (17).

Theorem 6

Under the UGDS model, the second moment of LDA at t_s is

$$E[{({\epsilon }_{t_s,n_0+n_1}^D)}^2]={({\alpha }_{t_s}^0)}^2\left[P(Z_s^I<\mathbf{0})+P(Z_s^I\ge \mathbf{0})\right]+2{\alpha }_{t_s}^0{\alpha }_{t_s}^I\left[P(Z_s^{II}<\mathbf{0})+P(Z_s^{II}\ge \mathbf{0})\right]+{({\alpha }_{t_s}^1)}^2\left[P(Z_s^{\mathit{III}}<\mathbf{0})+P(Z_s^{\mathit{III}}\ge \mathbf{0})\right],$$ (33)

where $Z_s^j$ is a 3-variate Gaussian random vector with mean and covarianc matrices as follows:

$${\mu }_{Z_s^I}={[\begin{array}{ccc}{\mu }_s^0-\frac{\overline{\mu }}{2}& -{\mu }^{\prime }& {\mu }_s^0-\frac{\overline{\mu }}{2}\end{array}]}^T,{\mu }_{Z_s^{II}}={[\begin{array}{ccc}{\mu }_s^0-\frac{\overline{\mu }}{2}& -{\mu }^{\prime }& -{\mu }_s^1+\frac{\overline{\mu }}{2}\end{array}]}^T$$ (34)

and, for i, j = 0, 1, i ≠ j, letting

$$z_s^i={({\sigma }_s^i)}^2-\frac{S_{{\rho }_s^{ii}}}{n_i}-\frac{S_{{\rho }_s^{ij}}}{n_j}+\frac{S_{{\mathbf{\sum }}^{00}}}{4n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}}{4n_1^2}+\frac{S_{{\mathbf{\sum }}^{01}}}{2n_0{n}_1},$$ (35)

we have

$$\begin{array}{l}{\mathbf{\sum }}_{Z_s^I}=\left[\begin{array}{ccc}{z}_s^0& \frac{-S_{{\rho }_s^{00}}}{n_0}+\frac{S_{{\rho }_s^{01}}}{n_1}+\frac{S_{{\mathbf{\sum }}^{00}}}{2n_0^2}-\frac{S_{{\mathbf{\sum }}^{11}}}{2n_1^2}& z_s^0-{({\sigma }_s^0)}^2\\ ·& \frac{S_{{\mathbf{\sum }}^{00}}}{n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}}{n_1^2}-\frac{2S_{{\mathbf{\sum }}^{01}}}{n_0{n}_1}& \frac{-S_{{\rho }_s^{00}}}{n_0}+\frac{S_{{\rho }_s^{01}}}{n_1}+\frac{S_{{\mathbf{\sum }}^{00}}}{2n_0^2}-\frac{S_{{\mathbf{\sum }}^{11}}}{2n_1^2}\\ ·& ·& z_s^0\end{array}\right],\\ {\mathbf{\sum }}_{Z_s^{II}}=\left[\begin{array}{ccc}{z}_s^0& \frac{-S_{{\rho }_s^{00}}}{n_0}+\frac{S_{{\rho }_s^{01}}}{n_1}+\frac{S_{{\mathbf{\sum }}^{00}}}{2n_0^2}-\frac{S_{{\mathbf{\sum }}^{11}}}{2n_1^2}& -z_s^0+{({\sigma }_s^0)}^2+\frac{S_{{\rho }_s^{11}}-S_{{\rho }_s^{01}}}{2n_1}+\frac{S_{{\rho }_s^{10}}-S_{{\rho }_s^{00}}}{2n_0}\\ ·& \frac{S_{{\mathbf{\sum }}^{00}}}{n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}}{n_1^2}-\frac{2S_{{\mathbf{\sum }}^{01}}}{n_0{n}_1}& \frac{-S_{{\rho }_s^{11}}}{n_1}+\frac{S_{{\rho }_s^{10}}}{n_0}-\frac{S_{{\mathbf{\sum }}^{00}}}{2n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}}{2n_1^2}\\ ·& ·& z_s^1\end{array}\right],\end{array}$$ (36)

with $\overline{\mu }=\frac{{\sum }_{i=1}^{n_0}{\mu }_i^0}{n_0}+\frac{{\sum }_{i=1}^{n_1}{\mu }_i^1}{n_1},{\mu }^{\prime }=\frac{{\sum }_{i=1}^{n_0}{\mu }_i^0}{n_0}-\frac{{\sum }_{i=1}^{n_1}{\mu }_i^1}{n_1}$, S_A is the sum of all elements of matrix A, defined as S_A = Σ_i,j a_ij, and ${\mu }_s^i$ and ${({\sigma }_s^i)}^2$ are the mean and variance of random variables at t_s from class i, i = 0, 1. Furthermore, μ_Z^III is obtained from μ_Z^I by replacing −μ′ with μ′ and ${\mu }_s^0$ with ${\mu }_s^1$, and ${\mathbf{\sum }}_{Z_s^{\mathit{III}}}$ is obtained from ${\mathbf{\sum }}_{Z_s^I}$ by exchanging n₀ and n₁, ${({\sigma }_s^0)}^2$ and ${({\sigma }_s^1)}^2$, S_Σ⁰⁰ and S_Σ¹¹, $S_{{\rho }_s^{00}}$ and $S_{{\rho }_s^{11}}$, and $S_{{\rho }_s^{01}}$ and $S_{{\rho }_s^{10}}$.

Proof

See the Appendix.

Let $E[{\epsilon }_{t_s,n_0+n_1}^I]$ and $E[{({\epsilon }_{t_s,n_0+n_1}^I)}^2]$ be the first and second moments of true error of the classifier in (10) specific to t_s and constructed by n₀ + n₁ independent training sample points distributed according to the same mean and variance. Then we have the following corollary.

Corollary 7

In the model considered in corollary 5, further assume n₀ = n₁ = n, Σ⁰¹ = 0_n×n, and, for k, j = 1, 2, …, n,

$${\mathbf{\sum }}_{kj}^{00}={\mathbf{\sum }}_{kj}^{11}=\{\begin{array}{ll}{\sigma }^2\hfill & k=j\hfill \\ \rho >0\hfill & \mathit{otherwise}\hfill \end{array},$$ (37)

where σ is the common variance of test sample points across classes at t_s. Let m be the number of additional dependent training points in each class with the same class conditional means and dependency structure, meaning ${\mathbf{\sum }}_{kj}^{ii}$ as in (37) for k, j = 1, 2, …, n + m and Σ⁰¹ =0_(n+m)×(n+m), that are required to make $E[{\epsilon }_{t_s,2n+2m}^D]=E[{\epsilon }_{t_s,2n}^I]$. This number also makes $E[{({\epsilon }_{t_s,2n+2m}^D)}^2]=E[{({\epsilon }_{t_s,2n}^I)}^2]$ and is given by

$$m=\frac{n-1}{\frac{{\sigma }^2}{n\rho }-1}$$ (38)

Proof

The proof of $E[{\epsilon }_{t_s,2n+2m}^D]=E[{\epsilon }_{t_s,2n}^I]$ follows by equating elements of covariance matrices obtained for the dependent model in (19) with the covariance matrices for the independent sampling model. Under the conditions of the corollary, these matrices in the independent sampling scenario (given by Theorem 1 in [3]) are

$${\mathbf{\sum }}_{Z_s^{\text{I}}}={\mathbf{\sum }}_{Z_s^{\text{II}}}={\mathbf{\sum }}_{Z_s^{\text{III}}}=\left(\begin{array}{cc}{\sigma }^2+\frac{{\sigma }^2}{2n}& 0\\ 0& \frac{2{\sigma }^2}{n}\end{array}\right)$$ (39)

Furthermore, we note that the conditions stated in this corollary satisfy the condition stated in (30), and hence $E[{\epsilon }_{t_s,2n}^D]\ge E[{\epsilon }_{t_s,2n}^I]$. The proof of $E[{({\epsilon }_{t_s,2n+2m}^D)}^2]=E[{({\epsilon }_{t_s,2n}^I)}^2]$ follows similarly by equating covariance matrices presented in (36) with those presented in Theorem 2 in [3].

In (38), if $\rho >\frac{{\sigma }^2}{n}$, then m < 0, meaning that adding any additional points under the dependency model in the corollary does not lower $E[{\epsilon }_{t_s,2n+2m}^D]$ and $E[{({\epsilon }_{t_s,2n+2m}^D)}^2]$ to the level of the first and second moments of true error of the constructed LDA classifier as if the original 2n training samples were independent.

4. Applications

In this section we study applications to common models used in signal processing, the first-order autoregressive model, AR(1), and the first-order moving average model, MA(1), by assuming the training data are generated by the output processes of two models. ${\mathbf{Z}}_t^0$ and ${\mathbf{Z}}_t^1$ are two independent white noise processes and ${\mathbf{X}}_t^0$ and ${\mathbf{X}}_t^1$ are the processes producing the system output. The goal is to characterize the performance of the LDA classifier as a function of sample size, the parameters of the white noise processes, and the autoregressive coefficients.

4.1. First-order autoregressive model AR(1)

We consider two AR(1) models:

$$X_t^i=c_i+{\psi }_i{X}_{t-1}^i+Z_t^i,i=0,1,$$ (40)

where ψ_i is a constant such that 0 < |ψ_i| < 1, i = 0, 1, and $Z_t^0~N(0,{\sigma }_0^2),Z_t^1~N(0,{\sigma }_1^2)$, for all t, are independent from each other. Then ${\mathbf{X}}_t^0=\{X_t^0:0<t<\infty \}$ and ${\mathbf{X}}_t^1=\{X_t^1:0<t<\infty \}$ are two independent covariance-stationary processes and we have the following theorem.

Theorem 8

Let ${\mathbf{X}}_t^0,{\mathbf{X}}_t^1$ in the UGDS model be defined by the two independent covariance-stationary AR(1) processes as defined in (40). Then, at t_s, where max{n₀, n₁} < s, the expected true error of LDA constructed using the training samples $[X_{t_1}^0,X_{t_2}^0,\dots ,X_{t_{n_0}}^0]$ and $[X_{t_1}^1,X_{t_2}^1,\dots ,X_{t_{n_1}}^1]$ is

$$E[{\epsilon }_{t_s,n_0+n_1}^{AR(1)}]={\alpha }_{t_s}^0\left[P(Z_s^I<\mathbf{0})+P(Z_s^I\ge \mathbf{0})\right]+{\alpha }_{t_s}^I\left[P(Z_s^{II}<\mathbf{0})+P(Z_s^{II}\ge \mathbf{0})\right],$$ (41)

where $Z_{t_s}^I$ and $Z_{t_s}^{II}$ are Gaussian bivariate vectors with

$$\begin{array}{l}{\mu }_{Z_s^I}={[\begin{array}{cc}\frac{\mu }{2}& -\mu \end{array}]}^T,{\mu }_{Z_s^{II}}={[\begin{array}{cc}\frac{-\mu }{2}& \mu \end{array}]}^T,\\ {\mathbf{\sum }}_{Z_s^I}=\left[\begin{array}{cc}\frac{{\sigma }_0^2}{1-{\psi }_0^2}-\frac{S_{{\rho }_s^{00}}}{n_0}+\frac{S_{{\mathbf{\sum }}^{00}}}{4n_0^2}+\frac{S_{{\mathbf{\sum }}_{11}}}{4n_1^2}& \frac{-S_{{\rho }_s^{00}}}{n_0}+\frac{S_{{\mathbf{\sum }}^{00}}}{2n_0^2}-\frac{S_{{\mathbf{\sum }}^{11}}}{2n_1^2}\\ ·& \frac{S_{{\mathbf{\sum }}^{00}}}{n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}}{n_1^2}\end{array}\right],\\ {\mathbf{\sum }}_{Z_s^{II}}=\left[\begin{array}{cc}\frac{{\sigma }_1^2}{1-{\psi }_1^2}-\frac{S_{{\rho }_s^{11}}}{n_1}+\frac{S_{{\mathbf{\sum }}^{00}}}{4n_0^2}+\frac{S_{{\mathbf{\sum }}_{11}}}{4n_1^2}& \frac{-S_{{\rho }_s^{11}}}{n_1}-\frac{S_{{\mathbf{\sum }}^{00}}}{2n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}}{2n_1^2}\\ ·& \frac{S_{{\mathbf{\sum }}^{00}}}{n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}}{n_1^2}\end{array}\right],\end{array}$$ (42)

where

$$\begin{array}{l}\mu =\frac{c_0}{1-{\psi }_0}-\frac{c_1}{1-{\psi }_1},S_{{\rho }_s^{ii}}=\frac{{\psi }_i^{(s-n_i)}{\sigma }_i^2}{1-{\psi }_i^2}\left(\frac{1-{\psi }_i^{n_i}}{1-{\psi }_i}\right),\\ S_{{\mathbf{\sum }}^{ii}}=\frac{{\sigma }_i^2}{(1-{\psi }_i^2)(1-{\psi }_i)}\left[n_i(1+{\psi }_i)-2{\psi }_i\left(\frac{1-{\psi }_i^{n_i}}{1-{\psi }_i}\right)\right].\end{array}$$ (43)

Proof

See the Appendix.

Corollary 9

In the model considered in Theorem 8, let ψ₀ = ψ₁ = ψ, σ₀ = σ₁, ${\alpha }_{t_s}^0={\alpha }_{t_s}^1$. Let $E[{\epsilon }_{t_s,n_0+n_1}^{AR{(1)}_{\psi }}]$ denote the expected true error of an AR(1) model with AR coefficient ψ specific to t_s. Then

$$\underset{s\to \infty }{lim}E[{\epsilon }_{t_s,n_0+n_1}^{AR{(1)}_{\psi }}]=\frac{1}{2}-\frac{L(h,k;\rho )}{2},$$ (44)

where L(h, k; ρ) is defined in (22) and

$$\begin{array}{l}h=\frac{\mu }{2\sqrt{a}},k=\frac{\mu }{\sqrt{b}},\mu =\frac{c_0-c_1}{1-\psi },a=\frac{{\sigma }^2}{1-{\psi }^2}+\frac{b}{4},\\ b=\frac{{\sigma }^2\left[(1+\psi )(\frac{1}{n_0}+\frac{1}{n_1})-2\frac{\psi }{1-\psi }\left(\frac{1-{\psi }^{n_0}}{n_0^2}+\frac{1-{\psi }^{n_1}}{n_1^2}\right)\right]}{(1-{\psi }^2)(1-\psi )},\\ \rho =\frac{{\sigma }^2\left[(1+\psi )(\frac{1}{n_0}-\frac{1}{n_1})-\frac{2\psi }{1-\psi }\left(\frac{1-{\psi }^{n_0}}{n_0^2}-\frac{1-{\psi }^{n_1}}{n_1^2}\right)\right]}{2(1-{\psi }^2)(1-\psi )\sqrt{a}\sqrt{b}}.\end{array}$$ (45)

Proof

See the Appendix.

We consider $E[{\epsilon }_{t_s,2n}^{AR(1)}]$ as a function of ψ and compare it to the case where ψ= 0, which corresponds to the stochastic i.i.d setting.

Corollary 10

In the model considered in Corollary 9, let n₀ = n₁ = n. Furthermore, let $E[{\epsilon }_{t_s,2n}^I]$ be the expected true error of the LDA classifier with ψ = 0 in (40). Let ψ′ and ψ″ be two arbitrary values of the AR coefficient ψ. Then

$${\psi }^{″}>{\psi }^{\prime }\Rightarrow \underset{s\to \infty }{lim}E[{\epsilon }_{t_s,2n}^{AR{(1)}_{{\psi }^{″}}}]<\underset{s\to \infty }{lim}E[{\epsilon }_{t_s,2n}^{AR{(1)}_{{\psi }^{\prime }}}].$$ (46)

Hence,

$$\begin{array}{l}0<\psi <1\Rightarrow \underset{s\to \infty }{lim}E[{\epsilon }_{t_s,2n}^{AR{(1)}_{\psi }}]<\underset{s\to \infty }{lim}E[{\epsilon }_{t_s,2n}^I],\\ -1<\psi <0\Rightarrow \underset{s\to \infty }{lim}E[{\epsilon }_{t_s,2n}^{AR{(1)}_{\psi }}]>\underset{s\to \infty }{lim}E[{\epsilon }_{t_s,2n}^I].\end{array}$$ (47)

Proof

See the Appendix.

Corollary 10 shows that, if ψ ∈ (0, 1), then under the conditions of the Corollary, constructing an LDA classifier to differentiate between AR processes is beneficial in terms of the expected true error tested on sufficiently lagged data; however, for ψ ∈ (−1, 0), we expect larger expected true error.

4.2. First-order moving-average model MA(1)

We consider the MA(1) models

$$X_t^i=c_i+Z_t^i+{\theta }_i{Z}_{t-1}^i,i=0,1,$$ (48)

where θ_i ∈ ℝ and $Z_t^0~N(0,{\sigma }_0^2),Z_t^1~N(0,{\sigma }_1^2)$, for all t, are independent from each other. Then ${\mathbf{X}}_t^0=\{X_t^0:0<t<\infty \}$ and ${\mathbf{X}}_t^1=\{X_t^1:0<t<\infty \}$

are two independent and covariance-stationary processes regardless of the values of θ_i [18].

Theorem 11

Let ${\mathbf{X}}_t^0,{\mathbf{X}}_t^1$ in the UGDS model be defined by the two independent covariance-stationary MA(1) processes defined in (48). Then, at t_s, where max{n₀, n₁} + 1 < s, the expected true error of an LDA classifier constructed using the training samples $[X_{t_1}^0,X_{t_2}^0,\dots ,X_{t_{n_0}}^0]$ and $[X_{t_1}^1,X_{t_2}^1,\dots ,X_{t_{n_1}}^1]$ is

$$E[{\epsilon }_{t_s,n_0+n_1}^{MA(1)}]={\alpha }_{t_s}^0\left[P(Z_s^I<\mathbf{0})+P(Z_s^I\ge \mathbf{0})\right]+{\alpha }_{t_s}^1\left[P(Z_s^{II}<\mathbf{0})+P(Z_s^{II}\ge \mathbf{0})\right],$$ (49)

where $Z_{t_s}^I$ and $Z_{t_s}^{II}$ are Gaussian bivariate vectors with:

$$\begin{array}{l}{\mu }_{Z_s^I}={[\begin{array}{cc}\frac{\mu }{2}& -\mu \end{array}]}^T,{\mu }_{Z_s^{II}}={[\begin{array}{cc}\frac{-\mu }{2}& \mu \end{array}]}^T,\\ {\mathbf{\sum }}_{Z_s^I}=\left[\begin{array}{cc}{\sigma }_0^2(1+{\theta }_0^2)+\frac{S_{{\mathbf{\sum }}^{00}}}{4n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}}{4n_1^2}& \frac{S_{{\mathbf{\sum }}^{00}}}{2n_0^2}-\frac{S_{{\mathbf{\sum }}^{11}}}{2n_1^2}\\ ·& \frac{S_{{\mathbf{\sum }}^{00}}}{n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}}{n_1^2}\end{array}\right],\\ {\mathbf{\sum }}_{Z_s^{II}}=\left[\begin{array}{cc}{\sigma }_1^2(1+{\theta }_1^2)+\frac{S_{{\mathbf{\sum }}^{00}}}{4n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}}{4n_1^2}& \frac{-S_{{\mathbf{\sum }}^{00}}}{2n_0^2}-\frac{S_{{\mathbf{\sum }}^{11}}}{2n_1^2}\\ ·& \frac{S_{{\mathbf{\sum }}^{00}}}{n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}}{n_1^2}\end{array}\right],\end{array}$$ (50)

where i = 0, 1, and

$$\mu =c_0-c_1,S_{{\mathbf{\sum }}^{ii}}={\sigma }_i^2\left[n_i(1+{\theta }_i^2)+2(n_i-1){\theta }_i\right].$$ (51)

Proof

See the Appendix.

Corollary 12

For the model in Theorem 11, let θ₀ = θ₁, σ₀ = σ₁, ${\alpha }_{t_s}^0={\alpha }_{t_s}^1$. Let $E[{\epsilon }_{t_s,n_0+n_1}^{MA{(1)}_{\theta }}]$ denote the expected true error of an MA(1) model with MA coefficient θ specific to t_s. Then

$$E[{\epsilon }_{t_s,n_0+n_1}^{MA{(1)}_{\theta }}]=\frac{1}{2}-\frac{L(h,k;\rho )}{2},$$ (52)

where L(h, k; ρ) is defined in (22) and

$$\begin{array}{l}h=\frac{\mu }{2\sqrt{a}},k=\frac{\mu }{\sqrt{b}},\mu =c_0-c_1,a={\sigma }^2(1+{\theta }^2)+\frac{b}{4},\\ b={\sigma }^2\left[{(1+\theta )}^2(\frac{1}{n_0}+\frac{1}{n_1})-2\theta (\frac{1}{n_0^2}+\frac{1}{n_1^2})\right],\\ \rho =\frac{{\sigma }^2\left[{(1+\theta )}^2(\frac{1}{n_0}-\frac{1}{n_1})-2\theta (\frac{1}{n_0^2}-\frac{1}{n_1^2})\right]}{2\sqrt{a}\sqrt{b}}.\end{array}$$ (53)

Proof

The result follows by considering the assumption of the corollary in Theorem 11 and then following the same steps similar to Corollary 2.

Corollary 13

For the model in Corollary 12, let n₀ = n₁ = n. Furthermore, let $E[{\epsilon }_{t_s,2n}^I]$ be the expected true error of the LDA classifier specific to t_s with θ = 0 in (48). Let θ′ and θ″ be two arbitrary values of the MA coefficient θ. Then

$$\begin{array}{l}({\theta }^{″}>{\theta }^{\prime })\wedge \left({\theta }^{\prime },{\theta }^{″}\in (-\infty ,\frac{1}{n}-1)\right)\Rightarrow E[{\epsilon }_{t_s,2n}^{MA{(1)}_{{\theta }^{″}}}]<E[{\epsilon }_{t_s,2n}^{MA{(1)}_{{\theta }^{\prime }}}],\\ ({\theta }^{″}>{\theta }^{\prime })\wedge \left({\theta }^{\prime },{\theta }^{″}\in [\frac{\frac{1}{n}-1}{2n+1},\infty )\right)\Rightarrow E[{\epsilon }_{t_s,2n}^{MA{(1)}_{{\theta }^{″}}}]>E[{\epsilon }_{t_s,2n}^{MA{(1)}_{{\theta }^{\prime }}}],\end{array}$$ (54)

and, therefore,

$$\begin{array}{l}\theta \in (0,\infty )\Rightarrow E[{\epsilon }_{t_s,2n}^{MA{(1)}_{\theta }}]>E[{\epsilon }_{t_s,2n}^I],\\ \theta \in [\frac{\frac{1}{n}-1}{2n+1},0)\Rightarrow E[{\epsilon }_{t_s,2n}^{MA{(1)}_{\theta }}]<E[{\epsilon }_{t_s,2n}^I].\end{array}$$ (55)

Proof

See the Appendix.

Corollary 13 shows that there exists a range of moving-average coefficients, i.e. [ $\frac{\frac{1}{n}-1}{2n+1}$, 0), that is beneficial in terms of expected classification error, i.e. has a smaller expected true error than the stochastic i.i.d model. For positive values of the coefficient, the expected true error of LDA increases.

5. Numerical Examples

We now illustrate the results obtained in previous sections under several specific settings.

Experiment 1

First, we consider scenarios in which the sample points taken from each class conditional process are identically distributed. They have the same mean, μ₀ for class 0 and μ₁ for class 1, and we set μ₀ = −μ₁ and μ₀ = 0.5, 0.75, 1, 1.5. We assume that the observations have variance 1 and are equally correlated with ρ_with ∈ [ρ_l, 0.95]. The value of ρ_l is determined so that the covariance matrix defined in (8) is positive definite. In each case we consider three settings for the correlation, ρ_bet, across classes: (1) independent, ρ_bet = 0, (2) ρ_bet = ρ_with, and (3) ρ_bet = −ρ_with. For each setting we consider two sample sizes, n₀ = n₁ = n = 5 and n₀ = n₁ = n = 25. We assume any future observation from each class conditional process has a distribution similar to those of the training data from that class and ${\alpha }_{t_s}^0={\alpha }_{t_s}^1$.

Figure 2(a)–2(d) show the exact expectation and standard deviation (SD) of the LDA true error for this experiment as a function of ρ_with. The results are calculated from Theorems 1 and 6. Parts a and b of the figure show that increasing ρ_with has an incremental effect on $E[{\epsilon }_{t_s,2n}^D]$. Since future observations are identically distributed, $E[{\epsilon }_{t_s,2n}^D]$ is the same for all values of t_s. Theoretically, for ρ_bet = 0, we can easily verify the graphical behavior by using Lemma 3 in Theorem 1. To analytically see the effect of ρ_with on $E[{\epsilon }_{t_s,2n}^D]$ once ρ_bet = 0, let ρ₁, ρ₂ be two arbitrary values of ρ_with such that ρ₁ < ρ₂ and denote all distributional parameters used in Corollary 2 corresponding to ρ_k, k = 1, 2, with a super script ρ_k. With the aforementioned conditions of the experiment, we have $\frac{S_{{\mathbf{\sum }}^{00}}^{{\rho }_1}}{n_0^2}-\frac{S_{{\mathbf{\sum }}^{11}}^{{\rho }_1}}{n_1^2}=\frac{S_{{\mathbf{\sum }}^{00}}^{{\rho }_2}}{n_0^2}-\frac{S_{{\mathbf{\sum }}^{11}}^{{\rho }_2}}{n_1^2}=0$ and $\frac{S_{{\mathbf{\sum }}^{00}}^{{\rho }_k}}{n_0^2}+\frac{S_{{\mathbf{\sum }}^{11}}^{{\rho }_k}}{n_1^2}=\frac{1+2(n-1){\rho }_k}{2n}$, k = 1, 2. Therefore, a^ρ₁ < a^ρ₂ and b^ρ₁ < b^ρ₂. The results then follow from Lemma 3. For other cases where ρ_bet ≠ 0 one may analytically study the effect of changing ρ_bet on $E[{\epsilon }_{t_s,2n}^D]$ using results Theorem 1 and studying the change similar to the proof of Lemma 3.

Figure 2.

Figures (a)–(d) show the exact expectation and standard deviation of LDA true error in Experiment 1 as a function of ρ_with. (a) Expectation for n₀ = n₁ = 5; (b) Expectation for n₀ = n₁ = 25; (c) Standard deviation for n₀ = n₁ = 5; (d) Standard deviation for n₀ = n₁ = 25; (a)–(d) plot keys: ○ := ρ_bet = 0; ×:= ρ_bet = ρ_with; △ = ρ_bet = −ρ_with; solid := μ₀ = 1.5; dash := μ₀ = 1; dot := μ₀ = 0.75; dash-dot := μ₀ = 0.5. The cross section of each curve with the vertical solid line in (a)–(d) plots shows the magnitude of the expectation/variance for i.i.d sampling situation for the corresponding scenario. The small horizontal solid lines in Figures (b) and (d) show the magnitude of expectation/variance of i.i.d situation in Figures (a) and (c), respectively. Figures (e)–(f) show the exact expectation of LDA true error of the first-order autoregressive model in Experiment 2 as a function of ψ:= ψ₀ = ψ₁. (a) Case of n₀ = n₁ = 5; (b) Case of n₀ = n₁ = 25; (e)–(f) plot keys: ○ := s − n₀ = 2; × : s − n₀ = 10; solid := c₀ = 1.5; dash := c₀ = 1; dot := c₀ = 0.75; dash-dot := c₀ = 0.5. The cross section of each curve with the vertical solid line in (e)–(f) plots shows the magnitude of the expectation for i.i.d sampling situation for the corresponding scenario. The small horizontal solid lines in Figure (f) show the magnitude of expectation of i.i.d situation in Figure (e).

Figures 2(a) and 2(b) show that increasing d = |μ₀ − μ₁| has an incremental effect on $E[{\epsilon }_{t_s,2n}^D]$. This effect can also be seen from Lemma 3 and Corollary 2. Therefore, we call classification scenarios with a larger d, “easier” scenarios, and those with smaller d, “harder” scenarios. In this sense, d is an indicator of classification difficulty in our experiment. The figures suggest that having a between-class correlation of ρ_bet = ρ_with > 0 helps in classification performance in “harder” classification situations (i.e., compared with ρ_bet = 0) and has a detrimental effect on classification performance in “easier” settings. However, having ρ_bet = ρ_with < 0, helps to have a better classification performance in “easier” settings and results in a worse performance in “harder” settings. This is observed by the fact that curves for ρ_bet = ρ_with are above (below) the curves for ρ_bet = 0 for d = 3 (d = 1).

The standard deviation is more complicated to interpret. The trends seen in Figures 2(c) and 2(d) suggest that increasing ρ_with generally increases the standard deviation of the LDA true error in cases where ρ_bet = 0. Furthermore, it suggests that once ρ_bet = −ρ_with, the standard deviation generally increases as ρ_with grows, but once ρ_bet = ρ_with, increasing ρ_with in small sample sizes may increase or decrease the standard deviation depending on classification difficulty, and as the sample size gets larger, increasing ρ_with generally increases the standard deviation. Furthermore, the figures suggest that increasing the classification difficulty may first increase the standard deviation and then decreases it.

Comparing Figure 2(a) with 2(b) and Figure 2(c) with 2(d) shows that increasing sample sizes lower the magnitude of the expectation and standard deviation regardless of classification difficulty or magnitude of ρ_with.

Experiment 2

In this experiment, we use the first order autoregressive model defined in (40). We assume ${\alpha }_{t_s}^0={\alpha }_{t_s}^1$, n₀ = n₁ = n, σ₀ = σ₁ = 1, and ψ₀ = ψ₁ = ψ ∈ [−0.95, 0.95]. We consider various cases where c₀ = 0.5, 0.75, 1, 1.5 with c₀ = −c₁. Figure 2(e) and 2(f) show the exact expectation of LDA true error for this experiment. These results are exact and are calculated from Theorem 8. These figures suggest that increasing ψ decreases $E[{\epsilon }_{t_s,2n}^{AR{(1)}_{\psi }}]$. According to Corollary 10, for a sufficiently lagged t_s, $E[{\epsilon }_{t_s,2n}^{AR{(1)}_{\psi }}]$ is a decreasing function of ψ and, furthermore, $E[{\epsilon }_{t_s,2n}^{AR{(1)}_{\psi }}]<E[{\epsilon }_{t_s,2n}^I]$ for 0 < ψ < 1 and $E[{\epsilon }_{t_s,2n}^{AR{(1)}_{\psi }}]>E[{\epsilon }_{t_s,2n}^I]$ for 0 < ψ < 1. Here the same behavior is observed even for small lags of 2 and 10. Furthermore, decreasing the sample size and increasing the classification difficulty have an incremental effect on the expected true error.

Experiment 3

In this experiment, we use the AR(1) model in (48). We assume ${\alpha }_{t_s}^0={\alpha }_{t_s}^1$, n₀ = n₁ = n, σ₀ = σ₁ = 1, and θ₀ = θ₁ = θ ∈ [−10, 10]. We consider c₀ = 0.5, 0.75, 1, 1.5 with c₀ = −c₁. Figure 3 shows the exact expectation of the LDA true error for this experiment. These results are exact and are calculated from Theorem 11.

Figure 3.

Exact expectation of LDA true error of the first-order moving average model in Experiment 3 as a function of θ := θ₀ = θ₁. (a) Expectation for n = 5; (b) Expectation for n = 25; (c) Magnification of region [−1, 0.1] in Figure (a); (d) Magnification of region [−1, 0.1] in Figure (b). Plot keys: ○:= c₀ = 1.5; △ : c₀ = 1; ▽ := c₀ = 0.75; × := c₀ = 0.5; The cross section of each curve with the vertical solid line in each plot shows the magnitude of the expectation for i.i.d sampling situation for the corresponding scenario. The small horizontal solid lines in Figure (b) are drawn to facilitate comparing this figure with Figure (a) at these cross sections. The left side of blue dotted line is (−∞, $\frac{1}{n}-1$] region, which in (54) we proved that the expectation of true error is a decreasing function of θ. The right hand side of the red dashed line is [ $\frac{\frac{1}{n}-1}{2n+1}$, ∞) region, which the expectation is an increasing function as seen from (54).

Figure 3 shows that the expected true error of LDA under the MA(1) model has an inverted bell shape with a negatively biased center, and the bias decreases as the sample size increases. The results of Corollary 13 are clear in this figure: for $\theta \in (-\infty ,\frac{1}{n}-1],E[{\epsilon }_{t_s,2n}^{MA{(1)}_{\theta }}]$ is a decreasing function of θ. This region is on the left-hand side of the vertical blue dotted lines in Figure 3. For $\theta \in [\frac{\frac{1}{n}-1}{2n+1}\infty ),E[{\epsilon }_{t_s,2n}^{MA{(1)}_{\theta }}]$ is an increasing function of θ. This region is on the right-hand side of the vertical red dashed line in the figure. As proved in Corollary 13, we observe in Figure 3(c) and 3(d) that, for $\theta \in [\frac{\frac{1}{n}-1}{2n+1},0),E[{\epsilon }_{t_s,2n}^{MA{(1)}_{\theta }}]<E[{\epsilon }_{t_s,2n}^I]$. This is the region between red dashed line and the vertical black line of each plot. For θ ∈ (0, ∞), $E[{\epsilon }_{t_s,2n}^{MA{(1)}_{\theta }}]>E[{\epsilon }_{t_s,2n}^I]$. This is the region on the right-hand side of the vertical black solid lines.

Experiment 4

This experiment is an example derived from gene-expression data used in studying the prognosis of breast-cancer using 70 genes with high prognostic ability [19]. Following [20], we divide the 307 individuals used in this study into 64 “poor” prognosis (class 0) versus 243 “good” prognosis (class 1) patients. A poor prognosis is defined to be a distant metastasis within 5 years of initial diagnosis. The gene expression data used in this study have been collected by triplicating each gene on each microarray and then duplicating each measurement by dye-swaping. Therefore, for each patient, each gene, we have six measurements, three of which are positively correlated with themselves and negatively correlated with others. Using this dataset we consider a scenario in which the experimenter is only given six measurements taken from one patient from class 0 and six measurements from another patient from class 1, and a univariate LDA classifier is desired to differentiate the two groups. We assume the single variate used in this classifier is the ALDH4 gene, which has the highest correlation with prognosis of breast cancer in [20]. Therefore, in this scenario, the experimenter is given 12 “technical” replicates in total, which are now treated as our “sample points”. This is an example of the UGDS model in genomic applications in which our classification is defined by two Gaussian processes, ${\mathbf{X}}_t^0$ and ${\mathbf{X}}_t^1$, which are assumed to be independent processes. We note that the expected performance of a classifier depends on t_s, i.e. $E[{\epsilon }_{t_s,12}^D]$ in Theorem 1, which is a function of the distribution of the future data as well as the distribution of the training data and their correlation structure. We verify the Gaussianity of each of the 12 random variables, $X_{t_1}^0,X_{t_2}^0,\dots ,X_{t_6}^0,X_{t_1}^1,X_{t_2}^1,\dots ,X_{t_6}^1$, used for characterizing the two Gaussian processes of this example via a Shapiro-Wilk test (using the R statistical software) on the full dataset corresponding to each random variable. This test does not reject Gaussianity of the random variables over either of the classes at a 95% significance level after employing the Bonferroni correction of multi-hypothesis tests.

Unfortunately, taken together, the 12 random variables do not pass the Shapiro-Wilk test for multivariate Gaussianity. Nonetheless, we will proceed and demonstrate that, even with this lack of multivariate Gaussianity, Theorem 1 is much more accurate than its counterpart in [3], which assumes i.i.d. data from each distribution.

Sample means, variances, and correlation, computed on the full dataset, were used as estimates of the unknown true means, variances, and the correlation structure between samples needed in Theorem 1. Using Theorem 1, the expected performance of a classifier, $E[{\epsilon }_{t_s,12}^D]$, to differentiate samples distributed as $X_{t_5}^0$ from samples distributed as $X_{t_1}^1$ is 0.475. To further verify this expected performance we construct a classifier on each possible combination among 243 × 64 = 15552 combinations of 6 samples from either classes and each time we test the accuracy of the designed classifier on the 64 − 1 = 63 remaining realizations of $X_{t_5}^0$ and 243 − 1 = 242 realizations of $X_{t_1}^1$. The accuracy computed in this way is 0.479, which is almost the same as what is computed from Theorem 1. It is interesting to compare this accuracy to the case in which one designs a classifier without paying attention to the correlation structure between samples and various distributions governing the data (considering the data being i.i.d.). In this scenario one (incorrectly) considers the data from each class coming from a single distribution and the expected performance of a classifier can be therefore evaluated from Theorem 1 that we presented in [3]. Again we use the sample means and variances, computed on the full dataset, as estimates of the unknown true means and variances. In this case the expected performance of LDA is estimated to be 0.374, which is very far from 0.479.

6. Conclusion

In many applications, the assumption of having i.i.d. training samples is violated. This paper characterizes the performance of univariate LDA classification in stochastic settings by assuming the samples are taken from two class conditional Gaussian processes, which are not necessarily independent. Linear classification has been considered owing to its long history in pattern recognition and its suitability for small-sample classification. We do not impose a specific correlation structure on the training data. We have presented conditions in which the correlation structure can be either beneficial or detrimental in terms of classification performance. As an application we have obtained exact expressions for the performance of LDA in situations that the data are produced through auto-regressive (AR) or moving-average (MA) models of the first order. We have found ranges of AR or MA multiplicative coefficients having incremental or decremental effect on classification performance. Having characterized univariate LDA performance in closed form, we aim to follow our work in [3] and characterize the effect of non-i.i.d. samples on training-data-based error estimators.

Appendix. Various Correlation Structures

Let p-dimenional sample points of each class, X₁, X₂, …, X_ni, with X_j being a column vector, be separately taken from two p-variate normal distributions, Π₁ and Π₂, with the distribution N(μ_i, Σ). Furthermore, let V_i be the dispersion matrix of the n_ip×1 vector $X={(X_1^T,X_2^T,\dots ,X_{n_i}^T)}^T$, i = 0, 1, defined as V_i = E[(X − E(X))(X − E(X))^T]. We define three correlation structures in regard to the data: (1) equicorrelated if V_i = I_ni ⊗ (Σ − R)+ E_ni ⊗ R, with R being a symmetric matrix, I_n the n × n identity matrix, and E_n the n × n matrix with all elements being 1; (2) simply equicorrelated if V_i = I_ni ⊗ (1 − ρ)Σ + E_ni ⊗ ρΣ, where ρ is a nonzero scalar constant where |ρ| < 1; and (3) serially correlated if V_i = I_ni ⊗ Σ + E_ni ⊗ ρ_τ Σ, where τ = |k − l|, k, l = 1, 2, …, n_i, |ρ_τ| < 1, τ = 1, 2, …, n_i, ρ₀ = 0. Note that univariate sample points, equicorrelation and simple-equicorrelation structures are essentially the same.

Proof of Theorem 1

From (9), it follows that

$$\begin{array}{l}E[{\epsilon }_{t_s}^0]=P(W({\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s})\le 0\mid X_{t_s}\in {\mathbf{X}}_t^0)=\\ P(X_{t_s}-{\overline{X}}_T<0,{\overline{X}}_T^0-{\overline{X}}_T^1>0)+P(X_{t_s}-{\overline{X}}_T\ge 0,{\overline{X}}_T^0-{\overline{X}}_T^1<0),\end{array}$$

where ${\overline{X}}_T=\frac{{\overline{X}}_T^0+{\overline{X}}_T^1}{2}$. Expanding ${\overline{X}}_T^0$ and ${\overline{X}}_T^1$ as $\frac{1}{n_0}{\sum }_{i=1}^{n_0}{X}_{t_i}^0$ and $\frac{1}{n_1}{\sum }_{i=1}^{n_1}{X}_{t_i}^1$ results in

$$E[{\epsilon }_{t_s}^0]=P(Z_s^{\text{I}}<\mathbf{0})+P(Z_s^{\text{I}}\ge \mathbf{0}),$$ (56)

where $Z_s^{\text{I}}={AY}_s^0$ and $Y_s^0={[X_{t_s}^0,\dots ,X_{t_{n_0}}^0,X_{t_1}^1,\dots ,X_{t_{n_1}}^1]}^T$, where the super index 0 in $X_{t_s}^0$ is to denote explicitly $X_{t_s}\in {\mathbf{X}}_t^0$, and

$$A=\left[\begin{array}{cccccccc}1& -\frac{1}{2n_0}& \frac{-1}{2n_0}& \cdots & \frac{-1}{2n_0}& \frac{-1}{2n_1}& \cdots & \frac{-1}{2n_1}\\ 0& \frac{-1}{n_0}& \frac{-1}{n_0}& \cdots & \frac{-1}{n_0}& \frac{1}{n_1}& \cdots & \frac{1}{n_1}\end{array}\right].$$ (57)

Therefore, $Z_s^{\text{I}}$ is a Gaussian random vector with mean $A{\mu }_{Y_s^0}$ and covariance matrix $A{\mathbf{\sum }}_{Y_s^0}{A}^T$. Plugging in the values of ${\mu }_{Y_s^0}=[{\mu }_s^0,{\mu }_1^0,{\mu }_2^0,\dots ,{\mu }_{n_0}^0,{\mu }_1^1,\dots ,{\mu }_{n_1}^1]$ and noting the fact that the j^th element of vector ${\rho }_s^{ik}$ is defined as ${\rho }_s^{ik}(j)=E[(X_{t_s}^i-{\mu }_s^i)(X_{t_j}^k-{\mu }_j^k)]$, i, k = 0, 1, j = 1, 2, …, n_k, we have

$${\mathbf{\sum }}_{Y_s^0}=\left[\begin{array}{ccc}{({\sigma }_s^0)}^2& {\rho }_s^{00}& {\rho }_s^{01}\\ {({\rho }_s^{00})}^T& {\mathbf{\sum }}^{00}& {\mathbf{\sum }}^{01}\\ {({\rho }_s^{01})}^T& {({\mathbf{\sum }}^{01})}^T& {\mathbf{\sum }}^{11}\end{array}\right],$$ (58)

which leads to the expression stated in Theorem 1. Evaluating the mean and covariance matrix of vector $Z_s^{\text{II}}$, which is the counterpart for $E[{\epsilon }_{t_s}^1]$, is entirely similar, by considering $P(W({\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s})>0\mid {\overline{X}}_T^0,{\overline{X}}_T^1,X_{t_s}\in {\mathbf{X}}_t^1)$.

Proof of Corollary 2

Note that for Φ(x, y; ρ) defined in (20),

$$\mathrm{\Phi }(-x,-y;\rho )=\underset{x}{\overset{\infty }{\int }}\underset{y}{\overset{\infty }{\int }}\frac{1}{2\pi \sqrt{1-{\rho }^2}}exp\left\{\frac{-(u^2+v^2-2\rho uv)}{2(1-{\rho }^2)}\right\}\mathit{dudv}.$$ (59)

By considering the assumption of the corollary for Theorem 1, and using (20) and (59) in (18), we get

$$E[{\epsilon }_{t_s,n_0+n_1}^D]=\frac{1}{2}\left(\mathrm{\Phi }(\frac{\mu }{2\sqrt{a}},\frac{-\mu }{\sqrt{b}},\rho )+\mathrm{\Phi }(\frac{-\mu }{2\sqrt{a}},\frac{\mu }{\sqrt{b}},\rho )\right)+\frac{1}{2}\left(\mathrm{\Phi }(\frac{-\mu }{2\sqrt{a}},\frac{\mu }{\sqrt{b}},-\rho )+\mathrm{\Phi }(\frac{\mu }{2\sqrt{a}},\frac{-\mu }{\sqrt{b}},-\rho )\right),$$ (60)

with a, b, and ρ defined in the corollary. Using the identity [28]

$$2[\mathrm{\Phi }(x,y;\rho )+\mathrm{\Phi }(x,y;-\rho )-\mathrm{\Phi }(x)-\mathrm{\Phi }(y)]+1=L(x,y;\rho ),$$ (61)

where Φ(.) is the standard normal cumulative function, completes the proof.

Proof of Lemma 3

Here, we first provide a way to intuitively understand the Lemma and then we provide a rigorous proof. We have

$$G(x,y;\rho )=F(x,y;\rho )+F(x,y;-\rho )=1+L(x,y,\rho )=1-L(\mid x\mid ,\mid y\mid ,\rho ),$$ (62)

where the last equality is due to xy < 0 stated as an assumption to the lemma. Intuitively, the lemma makes sense because smaller values of |x|, |y|, and |ρ| imply not only a smaller integration region in (22), but also less mass in that region. Next we provide a rigorous proof. It is straightforward to show

$$\begin{array}{l}\frac{\partial G(x,y;\rho )}{\partial x}=\frac{2e^{-\frac{x^2}{2}}}{\sqrt{2\pi }}\left[\mathrm{\Phi }(\frac{y-\rho x}{\sqrt{1-{\rho }^2}})+\mathrm{\Phi }(\frac{y+\rho x}{\sqrt{1-{\rho }^2}})-1\right],\\ \frac{\partial G(x,y;\rho )}{\partial y}=\frac{2e^{-\frac{y^2}{2}}}{\sqrt{2\pi }}\left[\mathrm{\Phi }(\frac{x-\rho y}{\sqrt{1-{\rho }^2}})+\mathrm{\Phi }(\frac{x+\rho y}{\sqrt{1-{\rho }^2}})-1\right],\\ \frac{\partial G(x,y;\rho )}{\partial \rho }=2\psi (x,y;\rho )-2\psi (x,y;-\rho ),\end{array}$$ (63)

where the last equality comes from well known results of Gaussian distribution, where $\frac{\partial \mathrm{\Phi }(x,y;\rho )}{\partial \rho }=\frac{\mathrm{\Phi }(x,y;\rho )}{\partial x\partial y}=\psi (x,y;\rho )$. Without loss of generality, we assume x ≥ 0 and y ≤ 0. The results for x ≤ 0 and y ≥ 0 are entirely similar after exchanging x and y in the following proof. We have

$$\begin{array}{l}\rho \ge 0\Rightarrow y-\rho x\le 0,y-\rho x\le y+\rho x\le -y+\rho x\\ \Rightarrow \mathrm{\Phi }(\frac{y-\rho x}{\sqrt{1-{\rho }^2}})+\mathrm{\Phi }(\frac{y+\rho x}{\sqrt{1-{\rho }^2}})\le 1\Rightarrow \frac{\partial G}{\partial x}\le 0,\\ \rho <0\Rightarrow y+\rho x\le 0,y+\rho x\le y-\rho x\le -y-\rho x\\ \Rightarrow \mathrm{\Phi }(\frac{y-\rho x}{\sqrt{1-{\rho }^2}})+\mathrm{\Phi }(\frac{y+\rho x}{\sqrt{1-{\rho }^2}})\le 1\Rightarrow \frac{\partial G}{\partial x}\le 0.\end{array}$$ (64)

Hence, $\frac{\partial G}{\partial x}\le 0$. Similarly, $\frac{\partial G}{\partial y}\ge 0$. Furthermore,

$$\rho \ge 0\Rightarrow \frac{\partial G(x,y;\rho )}{\partial \rho }\le 0,\rho <0\Rightarrow \frac{\partial G(x,y;\rho )}{\partial \rho }>0.$$

For 0 ≤ λ ≤ 1, we set

$${\gamma }_x=\lambda x_1+(1-\lambda )x_0,{\gamma }_y=\lambda y_1+(1-\lambda )y_0,{\gamma }_{\rho }=\lambda {\rho }_1+(1-\lambda ){\rho }_0.$$ (65)

Then λ, γ_x ≥ 0, γ_y ≤ 0, ρ_i ≥ 0 ⇒ γ_ρ ≥ 0 (i = 0, 1), and ρ_i < 0 ⇒ γ_ρ < 0 (i = 0, 1). Thus, $\frac{\partial G}{\partial {\gamma }_x}\le 0,\frac{\partial G}{\partial {\gamma }_y}\ge 0$ and

$${\gamma }_{\rho }\ge 0\Rightarrow \frac{\partial G(x,y;\rho )}{\partial {\gamma }_{\rho }}\le 0,{\gamma }_{\rho }<0\Rightarrow \frac{\partial G(x,y;\rho )}{\partial {\gamma }_{\rho }}>0.$$ (66)

Then

$$\begin{array}{l}\frac{\text{d}G}{\text{d}\lambda }=\frac{\partial G}{\partial {\gamma }_x}\frac{\text{d}{\gamma }_x}{\text{d}\lambda }+\frac{\partial G}{\partial {\gamma }_y}\frac{\text{d}{\gamma }_y}{\text{d}\lambda }+\frac{\partial G}{\partial {\gamma }_{\rho }}\frac{\text{d}{\gamma }_{\rho }}{\text{d}\lambda }\\ =\frac{\partial G}{\partial {\gamma }_x}(x_1-x_0)+\frac{\partial G}{\partial {\gamma }_y}(y_1-y_0)+\frac{\partial G}{\partial {\gamma }_{\rho }}({\rho }_1-{\rho }_0).\end{array}$$ (67)

First assume ρ_i ≥ 0, i = 0, 1, so that γ_ρ ≥ 0, $\frac{\partial G(x,y;\rho )}{\partial {\gamma }_{\rho }}\le 0$. Since $\frac{\partial G}{\partial {\gamma }_x}\le 0$, x₁ ≤ x₀, $\frac{\partial G}{\partial {\gamma }_y}\ge 0$, y₁ ≥ y₀, and ρ₁ ≤ ρ₀, we have $\frac{\text{d}G}{\text{d}\lambda }\ge 0$. Therefore,

$${\int }_0^1\frac{\text{d}G}{\text{d}\lambda }\text{d}\lambda =G(1)-G(0)=G(x_1,y_1,{\rho }_1)-G(x_0,y_0,{\rho }_0)\ge 0.$$ (68)

Next assume ρ_i ≤ 0, i = 0, 1, so that γ_ρ ≤ 0, $\frac{\partial G(x,y;\rho )}{\partial {\gamma }_{\rho }}\ge 0$. Since $\frac{\partial G}{\partial {\gamma }_x}\le 0$, x₁ ≤ x₀, $\frac{\partial G}{\partial {\gamma }_y}\ge 0$, y₁ ≥ y₀, and ρ₁ ≥ ρ₀, we have $\frac{\text{d}G}{\text{d}\lambda }\ge 0$. Therefore,

$${\int }_0^1\frac{\text{d}G}{\text{d}\lambda }\text{d}\lambda =G(1)-G(0)=G(x_1,y_1,{\rho }_1)-G(x_0,y_0,{\rho }_0)\ge 0.$$

Lastly, assume the ρ_i’s have opposite signs. Without loss of generality, assume ρ₀ < 0, ρ₁ ≥ 0, and |ρ₁| ≤ |ρ₀|. Then

$$\begin{array}{l}{\int }_0^1\frac{\text{d}G}{\text{d}\lambda }\text{d}\lambda ={\int }_0^{\frac{\mid {\rho }_0\mid -{\rho }_1}{{\rho }_1-{\rho }_0}}\text{d}G+{\int }_{\frac{\mid {\rho }_0\mid -{\rho }_1}{{\rho }_1-{\rho }_0}}^1\text{d}G\\ ={\int }_0^{\frac{\mid {\rho }_0\mid -{\rho }_1}{{\rho }_1-{\rho }_0}}\text{d}G+G(x_m,y_m,-{\rho }_1)-G(x_1,y_1,{\rho }_1),\end{array}$$ (69)

where $x_m=\frac{\mid {\rho }_0\mid -{\rho }_1}{{\rho }_1-{\rho }_0}{x}_1+(1-\frac{\mid {\rho }_0\mid -{\rho }_1}{{\rho }_1-{\rho }_0})x_0,y_m=\frac{\mid {\rho }_0\mid -{\rho }_1}{{\rho }_1-{\rho }_0}{y}_1+(1-\frac{\mid {\rho }_0\mid -{\rho }_1}{{\rho }_1-{\rho }_0})y_0$, x₁ < x_m < x₀, and y₀ < y_m < y₁. From the definition of G(x, y, ρ) it is easy to see that G(x, y, ρ) = G(x, y, −ρ) and then, from the conditions that result in (68), we have G(x_m, y_m, −ρ₁) − G(x₁, y₁, ρ₁) = G(x_m, y_m, ρ₁) − G(x₁, y₁, ρ₁) ≥ 0. Hence, in order to show G(x₁, y₁, ρ₁) − G(x₀, y₀, ρ₀) ≥ 0 it is sufficient to show that ${\int }_0^{\frac{\mid {\rho }_0\mid -{\rho }_1}{{\rho }_1-{\rho }_0}}\frac{\text{d}G}{\text{d}\lambda }\text{d}\lambda \ge 0$. For $\lambda \in [0,\frac{\mid {\rho }_0\mid -{\rho }_1}{{\rho }_1-{\rho }_0}]$, we have γ_ρ ≤ 0, $\frac{\partial G(x,y;\rho )}{\partial {\gamma }_{\rho }}\ge 0$. Therefore, from (67), $\frac{\partial G(x,y;\rho )}{\partial {\gamma }_{\rho }}\ge 0$ and, furthermore, $\frac{\text{d}G}{\text{d}\lambda }\ge 0$. Thus, ${\int }_0^{\frac{\mid {\rho }_0\mid -{\rho }_1}{{\rho }_1-{\rho }_0}}\frac{\text{d}G}{\text{d}\lambda }\text{d}\lambda \ge 0$ and the result follows.

Proof of Theorem 6

From (16) and (9), it follows that

$$E[{({\epsilon }_{t_s}^0)}^2]=P(X_{t_s}-{\overline{X}}_T<0,{\overline{X}}_T^0-{\overline{X}}_T^1>0,X_{t_s}^{\prime }-{\overline{X}}_T<0)+P(X_{t_s}-{\overline{X}}_T\ge 0,{\overline{X}}_T^0-{\overline{X}}_T^1<0,X_{t_s}^{\prime }-{\overline{X}}_T\ge 0),$$ (70)

where ${\overline{X}}_T=\frac{{\overline{X}}_T^0+{\overline{X}}_T^1}{2}$. Expanding ${\overline{X}}_T^0$ and ${\overline{X}}_T^1$ as $\frac{1}{n_0}{\sum }_{i=1}^{n_0}{X}_{t_i}^0$ and $\frac{1}{n_1}{\sum }_{i=1}^{n_1}{X}_{t_i}^1$ results in

$$E[{({\epsilon }_{t_s}^0)}^2]=P(Z_s^{\text{I}}<\mathbf{0})+P(Z_s^{\text{I}}\ge \mathbf{0}),$$ (71)

where $Z_s^{\text{I}}={AY}_s^0$, in which $Y_s^0={[X_{t_s}^0,X_{t_s}^{\prime 0},X_{t_1}^0,\dots ,X_{t_{n_0}}^0,X_{t_1}^1,\dots ,X_{t_{n_1}}^1]}^T$, and the super index 0 in $X_{t_s}^0$ and $X_{t_s}^{\prime 0}$ is to denote explicitly X_ts, $X_{t_s}^{\prime }\in {\mathbf{X}}_t^0$, and

$$A=\left[\begin{array}{ccccccccc}1& 0& \frac{-1}{2n_0}& \frac{-1}{2n_0}& \cdots & \frac{-1}{2n_0}& \frac{-1}{2n_1}& \cdots & \frac{-1}{2n_1}\\ 0& 0& \frac{-1}{n_0}& \frac{-1}{n_0}& \cdots & \frac{-1}{n_0}& \frac{1}{n_1}& \cdots & \frac{1}{n_1}\\ 0& 1& \frac{-1}{2n_0}& \frac{-1}{2n_0}& \cdots & \frac{-1}{2n_0}& \frac{-1}{2n_1}& \cdots & \frac{-1}{2n_1}\end{array}\right].$$

Therefore, $Z_s^{\text{I}}$ is a Gaussian random vector with mean $A{\mu }_{Y_s^0}$ and covariance matrix $A{\mathbf{\sum }}_{Y_s^0}{A}^T$. Plugging in the values of ${\mu }_{Y_s^0}=[{\mu }_s^0,{\mu }_s^0,{\mu }_1^0,{\mu }_2^0,\dots ,{\mu }_{n_0}^0,{\mu }_1^1,{\mu }_2^1,\dots ,{\mu }_{n_1}^1]$ and noting the fact that the j^th element of vector ${\rho }_s^{ik}(j)$ is defined as ${\rho }_s^{ik}(j)=E[(X_{t_s}^i-{\mu }_s^i)(X_{t_j}^k-{\mu }_j^k)]$, i, k = 0, 1, j = 1, 2, …, n_k, and from the definition of $X_{t_s}^{\prime 0}$ it holds that $E[(X_{t_s}^0-{\mu }_s^i)(X_{t_j}^k-{\mu }_j^k)]=E[(X_{t_s}^{\prime 0}-{\mu }_s^i)(X_{t_j}^k-{\mu }_j^k)]$, we have:

$${\mathbf{\sum }}_{Y_s^0}=\left[\begin{array}{cccc}{({\sigma }_s^0)}^2& 0& {\rho }_s^{00}& {\rho }_s^{01}\\ 0& {({\sigma }_s^0)}^2& {\rho }_s^{00}& {\rho }_s^{01}\\ {({\rho }_s^{00})}^T& {({\rho }_s^{00})}^T& {\mathbf{\sum }}^{00}& {\mathbf{\sum }}^{01}\\ {({\rho }_s^{01})}^T& {({\rho }_s^{01})}^T& {({\mathbf{\sum }}^{01})}^T& {\mathbf{\sum }}^{11}\end{array}\right],$$ (72)

which leads to the expression stated in Theorem 6. Evaluating the mean and covariance matrices of vector $Z_s^{\text{II}}$ and $Z_s^{\text{III}}$ is entirely similar.

Proof of Theorem 8

Since the $Z_t^i$’s are Gaussian, $X_t^0$ and $X_t^1$ are covariance-stationary [18] and the vectors $X_{n_0}^0={[X_{t_1}^0,X_{t_2}^0,\dots ,X_{t_{n_0}}]}^T$ and $X_{n_0}^0={[X_{t_1}^1,X_{t_2}^1,\dots ,X_{t_{n_1}}^1]}^T$ are distributed normally as $X_{n_i}^i~N({\mathit{\mu }}^i,{\mathbf{\sum }}^i)$, i = 0, 1, where

$$\begin{array}{l}{\mathit{\mu }}^i={[{\mu }^i,{\mu }^i,\dots ,{\mu }^i]}_{1\times n_i}^T,{\mathbf{\sum }}^i(k,l)=\frac{{\psi }_i^{\mid k-l\mid }}{1-{\psi }_i^2}{\sigma }_i^2,k,l=1,2,\dots ,n_i,\\ {\rho }_s^{ii}(k)=\frac{{\psi }_i^{s-k}}{1-{\psi }_i^2}{\sigma }_i^2,k=1,2,\dots ,n_i,{\rho }_s^{01}={\mathbf{0}}_{1\times n_1},{\rho }_s^{10}={\mathbf{0}}_{1\times n_0},\end{array}$$ (73)

${\mu }_i=\frac{c_i}{1-{\psi }_i}$, and Σⁱ(k, l) denotes the entry in the k^th row and l^th column of matrix Σⁱ. The result follows by replacing (73) in Theorem 1.

Proof of Corollary 9

Using the corollary assumptions in Theorem 8, we get

$$E[{\epsilon }_{t_s,n_0+n_1}^{AR{(1)}_{\psi }}]=\frac{1}{2}\sum _{i=0}^1\left(\mathrm{\Phi }(h_s^i,-k;{\rho }_s^i)+\mathrm{\Phi }(-h_s^i,k;{\rho }_s^i)\right),$$ (74)

$$\begin{array}{l}{h}_s^i=\frac{\mu }{2\sqrt{a_s^i}},{\rho }_s^i=-\frac{{\psi }^{(s-n_i)}{\sigma }^2}{n_i(1-{\psi }^2)}\left(\frac{1-{\psi }^{n_i}}{1-\psi }\right)+\rho ,\\ a_s^i=a-\frac{{\psi }^{(s-n_i)}{\sigma }^2}{n_i(1-{\psi }^2)}\left(\frac{1-{\psi }^{n_i}}{1-\psi }\right),\end{array}$$ (75)

with k, a, b, ρ, and μ defined in (45). Let F(x, y; ρ) = Φ(x, y; ρ) + Φ(−x, −y; ρ), with Φ(x, y; ρ) defined in (20). Then using Scheffe’s Lemma [29] we have

$$2\underset{s\to \infty }{lim}E[{\epsilon }_{t_s,n_0+n_1}^{AR{(1)}_{\psi }}]=F(\underset{s\to \infty }{lim}{h}_s^0,-k;\underset{s\to \infty }{lim}{\rho }_s^0)+F(\underset{s\to \infty }{lim}{h}_s^1,-k;\underset{s\to \infty }{lim}{\rho }_s^1).$$ (76)

Note that by taking the limit, the term $\frac{{\psi }^{(s-n_i)}{\sigma }^2}{n_i(1-{\psi }^2)}\left(\frac{1-{\psi }^{n_i}}{1-\psi }\right)$ in $h_s^i$ and $a_s^i$ converges exponentially to 0 and we have $a_s^0=a_s^1=a,h_s^0=h_s^1=h,{\rho }_s^0={\rho }_s^1=\rho$. The result follows similarly to proof of Corollary 2.

Proof of Corollary 10

With n₀ = n₁ = n, we have ρ = 0 in (45). From (76) we get $2{lim}_{s\to \infty }E[{\epsilon }_{t_s,2n}^{AR{(1)}_{\psi }}]=G(h_{\psi },l_{\psi };0)$, with G(h_ψ, −k_ψ; 0) defined as in (24) and l_ψ= −k_ψ, where we use a subscript ψ to explicitly denote dependence of l and h on ψ. Since h_ψl_ψ< 0, we can use a proof similar to that of Lemma 3 to compare different AR models. Specifically, suppose we prove that

$${\psi }^{″}>{\psi }^{\prime }\Rightarrow \exists {\lambda }_h\in [0,1)\wedge \exists {\lambda }_l\in [0,1):h_{{\psi }^{″}}={\lambda }_h{h}_{{\psi }^{\prime }}\wedge l_{{\psi }^{″}}={\lambda }_l{l}_{{\psi }^{\prime }}$$ (77)

Then, similar to proof of Lemma 3, we can prove G(h_ψ″, l_ψ″; 0) < G(h_ψ′, l_ψ′; 0), so that

$$2\underset{s\to \infty }{lim}E[{\epsilon }_{t_s,2n}^{AR{(1)}_{{\psi }^{″}}}]=G(h_{{\psi }^{″}},l_{{\psi }^{″}};0)<\underset{s\to \infty }{lim}E[{\epsilon }_{t_s,2n}^{AR{(1)}_{{\psi }^{\prime }}}]=G(h_{{\psi }^{\prime }},l_{{\psi }^{\prime }};0),$$ (78)

thereby proving the basic inequality in the corrollary. We first demonstrate (77). Assume c₀ > c₁. We first prove that for ψ ∈ (−1, 1), we have $\frac{\text{d}{l}_{\psi }}{\text{d}\psi }<0$ and $\frac{\text{d}{h}_{\psi }}{\text{d}\psi }>0$. It is easy to see that:

$$\frac{\text{d}{l}_{\psi }}{\text{d}\psi }=-\sqrt{\frac{n}{2}}\frac{(c_0-c_1)}{\sigma }\frac{\text{d}(\sqrt{f_{\psi }})}{\text{d}\psi }=-\sqrt{\frac{n}{2f_{\psi }}}\frac{(c_0-c_1)g_{\psi }}{\sigma {(d_{\psi })}^2},$$

where

$$\begin{array}{l}{g}_{\psi }=2n(1+{\psi }^2-(n+1){\psi }^n+(n-1){\psi }^{(n+2)}),\\ d_{\psi }=n-2\psi -n{\psi }^2+2{\psi }^{n+1},f_{\psi }=\frac{n-n{\psi }^2}{d_{\psi }}.\end{array}$$ (79)

From Descartes’ Rule of Signs [30], g_ψ has either zero or two positive roots. For n ≥ 2,

$$g_{\psi }=4n{(\psi -1)}^2\left(\frac{(n-1)}{2}{\psi }^n+\sum _{j=1}^{n-1}j{\psi }^j+\frac{1}{2}\right).$$ (80)

Therefore, for all n, g_ψ has two roots at 1 and these are the only positive roots. Similarly we observe that if n is even, then g_ψ has only two negative roots at −1. If n is odd, again from Descartes’ Rule of Signs [30], g_ψ has only one negative root, denoted by ψ_−. We show that ψ₋ ∈ (−∞, −1). Let ψ₋ = −ψ₊, ψ₊ > 0. Since n is odd, we need to have

$$1+{\psi }_{+}^2+(n+1){\psi }_{+}^n-(n-1){\psi }_{+}^{(n+2)}=0.$$ (81)

Were ψ₊ ∈ (0, 1), this would imply $(n+1){\psi }_{+}^n>(n-1){\psi }_{+}^{(n+2)}$. Hence, (81) is not possible and ψ₋ ∈ (−∞, −1). Summarizing this result, we see that ψ ∈ (−1, 1) ⇒ g_ψ > 0 and therefore, $\frac{\text{d}{l}_{\psi }}{\text{d}\psi }<0$. It is straightforward to show

$$\frac{\text{d}{h}_{\psi }}{\text{d}\psi }=\sqrt{\frac{n}{2}}\frac{(c_0-c_1)}{\sigma }\frac{\text{d}(\sqrt{r_{\psi }})}{\text{d}\psi }=\sqrt{\frac{n}{2r_{\psi }}}\frac{(c_0-c_1)(4n^3{(1-\psi )}^2+g_{\psi })}{\sigma {(2n^2{(1-\psi )}^2+d_{\psi })}^2},$$

where $r_{\psi }=\frac{n-n{\psi }^2}{2n^2{(1-\psi )}^2+d_{\psi }}$. Since for ψ ∈ (−1, 1) we have g_ψ > 0, then $\frac{\text{d}{h}_{\psi }}{\text{d}\psi }>0$. We set γ_ψ = λψ₁ + (1 − λ)ψ₀, where 0 ≤ λ ≤ 1. Now we check that (78) holds. Denoting G(h_γψ,l_γψ; 0) by G, we have

$$\begin{array}{l}\frac{\text{d}G}{\text{d}\lambda }=\frac{\partial G}{\partial h_{{\gamma }_{\psi }}}\frac{\text{d}{h}_{{\gamma }_{\psi }}}{\text{d}{\gamma }_{\psi }}\frac{\text{d}{\gamma }_{\psi }}{\text{d}\lambda }+\frac{\partial G}{\partial l_{{\gamma }_{\psi }}}\frac{\text{d}{l}_{{\gamma }_{\psi }}}{\text{d}{\gamma }_{\psi }}\frac{\text{d}{\gamma }_{\psi }}{\text{d}\lambda }\\ =\frac{\partial G}{\partial h_{{\gamma }_{\psi }}}\frac{\text{d}{h}_{{\gamma }_{\psi }}}{\text{d}{\gamma }_{\psi }}({\psi }^{″}-{\psi }^{\prime })+\frac{\partial G}{\partial l_{{\gamma }_{\psi }}}\frac{\text{d}{l}_{{\gamma }_{\psi }}}{\text{d}{\gamma }_{\psi }}({\psi }^{″}-{\psi }^{\prime }).\end{array}$$ (82)

Since ψ ∈ (−1, 1), 0 ≤ λ ≤ 1, and h_ψl_ψ< 0, we can see that γ_ψ ∈ (−1, 1), h_γψl_γψ< 0, $\frac{\text{d}{h}_{{\gamma }_{\psi }}}{\text{d}{\gamma }_{\psi }}>0,\frac{\text{d}{l}_{{\gamma }_{\psi }}}{\text{d}{\gamma }_{\psi }}<0$, and from Proof of Lemma 3 in the appendix, $\frac{\partial G}{\partial h_{{\gamma }_{\psi }}}\le 0$ and $\frac{\partial G}{\partial l_{{\gamma }_{\psi }}}\ge 0$. Since ψ″ > ψ′, we see that $\frac{\text{d}G}{\text{d}\lambda }<0$. Similar to the proof of Lemma 3, integrating over λ results in G(h_ψ″, l_ψ″; 0) < G(h_ψ′, l_ψ′; 0). The same basic argument goes through for c₀ < c₁ and we have $\frac{\text{d}{h}_{{\gamma }_{\psi }}}{\text{d}{\gamma }_{\psi }}<0,\frac{\text{d}{l}_{{\gamma }_{\psi }}}{\text{d}{\gamma }_{\psi }}>0$. The remaining results follow from the definition of $E[{\epsilon }_{t_s,2n}^I]$, where we have $E[{\epsilon }_{t_s,2n}^{AR{(1)}_{\psi =0}}]=E[{\epsilon }_{t_s,2n}^I]$.

Proof of Theorem 11

Since the $Z_t^i$’s are Gaussian, $X_t^0$ and $X_t^1$ are covariance-stationary and the vectors $X_{n_0}^0={[X_{t_1}^0,X_{t_2}^0,\dots ,X_{t_{n_0}}]}^T$ and $X_{n_0}^0={[X_{t_1}^1,X_{t_2}^1,\dots ,X_{t_{n_1}}^1]}^T$ are distributed normally as $X_{n_i}^i~N({\mathit{\mu }}^i,{\mathbf{\sum }}^i)$, i = 0, 1, [18], where for k = 1, 2, …, n_i,

$$\begin{array}{l}{\mathit{\mu }}^i={[{\mu }^i,{\mu }^i,\dots ,{\mu }^i]}_{1\times n_i}^T,{\rho }_s^{ii}(k)=0,{\rho }_s^{01}={\mathbf{0}}_{1\times n_1},\\ {\rho }_s^{10}={\mathbf{0}}_{1\times n_0},{\mathbf{\sum }}^i(k,l)=\{\begin{array}{cc}{\sigma }_i^2(1+{\theta }_i^2),& k=l\\ {\sigma }_i^2{\theta }_i,& \mid k-l\mid =1\\ 0& \mathit{othewise}\end{array},\end{array}$$ (83)

where μ_i = c_i and Σⁱ(k, l) denotes the entry in the k^th row and l^th column of the matrix Σⁱ. The result follows by replacing (83) in Theorem 1.

Proof of Corollary 13

From Theorem 11, since ${\alpha }_{t_s}^0={\alpha }_{t_s}^1$ and max{n₀, n₁} + 1 < s, we have $2E[{\epsilon }_{t_s,2n}^{MA{(1)}_{\theta }}]=G(h_{\theta },l_{\theta };0)$, for any s, with l_θ = −k_θ, h_θ and k_θ defined in Corollary 12, and G(h_θ, −k_θ; 0) defined as in (24). Similar to proof of Corollary 10, the present corollary follows by setting n₀ = n₁ = n and using

$$\begin{array}{l}\frac{\text{d}{l}_{\theta }}{\text{d}\theta }=b^{-\frac{3}{2}}\frac{(c_0-c_1){\sigma }^2}{n}(2\theta +2-\frac{2}{n}),\\ \frac{\text{d}{h}_{\theta }}{\text{d}\theta }=-a^{-\frac{3}{2}}\frac{(c_0-c_1){\sigma }^2}{4n}(2n\theta +\theta +1-\frac{1}{n}),\end{array}$$ (84)

where a and b are obtained from (53).