Sequential linear regression with online standardized data

Abstract

The present study addresses the problem of sequential least square multidimensional linear regression, particularly in the case of a data stream, using a stochastic approximation process. To avoid the phenomenon of numerical explosion which can be encountered and to reduce the computing time in order to take into account a maximum of arriving data, we propose using a process with online standardized data instead of raw data and the use of several observations per step or all observations until the current step. Herein, we define and study the almost sure convergence of three processes with online standardized data: a classical process with a variable step-size and use of a varying number of observations per step, an averaged process with a constant step-size and use of a varying number of observations per step, and a process with a variable or constant step-size and use of all observations until the current step. Their convergence is obtained under more general assumptions than classical ones. These processes are compared to classical processes on 11 datasets for a fixed total number of observations used and thereafter for a fixed processing time. Analyses indicate that the third-defined process typically yields the best results.

1 Introduction

In the present analysis, A′ denotes the transposed matrix of A while the abbreviation “a.s.” signifies almost surely.

Let R = (R¹,…,R^p) and S = (S¹,…,S^q) be random vectors in ${\mathbb{R}}^p$ and ${\mathbb{R}}^q$ respectively. Considering the least square multidimensional linear regression of S with respect to R: the (p, q) matrix θ and the (q, 1) matrix η are estimated such that E[‖S − θ′ R − η‖²] is minimal.

Denote the covariance matrices

$$\begin{array}{ccc}\hfill B& =& Covar\left[R\right]=E\left[\right(R-E\left[R\right]\left){(R-E[R\left]\right)}^{\prime }\right],\hfill \\ \hfill F& =& Covar[R,S]=E\left[\right(R-E\left[R\right]\left){(S-E[S\left]\right)}^{\prime }\right].\hfill \end{array}$$

If we assume B is positive definite, i.e. there is no affine relation between the components of R, then

$$\begin{array}{c}\hfill \theta =B^{-1}F,\eta =E\left[S\right]-{\theta }^{\prime }E\left[R\right].\end{array}$$

Note that, R₁ denoting the random vector in ${\mathbb{R}}^{p+1}$ such that $R_1^{\prime }=\left(\begin{array}{cc}{R}^{\prime }& 1\end{array}\right)$, θ₁ the (p + 1, q) matrix such that ${\theta }_1^{\prime }=\left(\begin{array}{cc}{\theta }^{\prime }& \eta \end{array}\right)$, $B_1=E\left[R_1{R}_1^{\prime }\right]$ and F₁ = E[R₁ S′], we obtain ${\theta }_1=B_1^{-1}{F}_1$.

In order to estimate θ (or θ₁), a stochastic approximation process (X_n) in ${\mathbb{R}}^{p\times q}$ (or ${\mathbb{R}}^{(p+1)\times q}$) is recursively defined such that

$$\begin{array}{c}\hfill X_{n+1}=X_n-a_n(B_n{X}_n-F_n),\end{array}$$

where (a_n) is a sequence of positive real numbers, eventually constant, called step-sizes (or gains). Matrices B_n and F_n have the same dimensions as B and F, respectively. The convergence of (X_n) towards θ is studied under appropriate definitions and assumptions on B_n and F_n.

Suppose that ((R_1n, S_n), n ≥ 1) is an i.i.d. sample of (R₁, S). In the case where q = 1, $B_n=R_{1n}{R}_{1n}^{\prime }$ and $F_n=R_{1n}{S}_n^{\prime }$, several studies have been devoted to this stochastic gradient process (see for example Monnez [1], Ljung [2] and references hereafter). In order to accelerate general stochastic approximation procedures, Polyak [3] and Polyak and Juditsky [4] introduced the averaging technique. In the case of linear regression, Györfi and Walk [5] studied an averaged stochastic approximation process with a constant step-size. With the same type of process, Bach and Moulines [6] proved that the optimal convergence rate is achieved without strong convexity assumption on the loss function.

However, this type of process may be subject to the risk of numerical explosion when components of R or S exhibit great variances and may have very high values. For datasets used as test sets by Bach and Moulines [6], all sample points whose norm of R is fivefold greater than the average norm are removed. Moreover, generally only one observation of (R, S) is introduced at each step of the process. This may be not convenient for a large amount of data generated by a data stream for example.

Two modifications of this type of process are thus proposed in this article.

The first change in order to avoid numerical explosion is the use of standardized, i.e. of zero mean and unit variance, components of R and S. In fact, the expectation and the variance of the components are usually unknown and will be estimated online.

The parameter θ can be computed from the standardized components as follows. Let σ^j the standard deviation of R^j for j = 1,…,p and ${\sigma }_1^k$ the standard deviation of S^k for k = 1,…,q. Define the following matrices

$$\begin{array}{c}\hfill \Gamma =\left(\begin{array}{ccc}\frac{1}{{\sigma }^1}& \cdots & 0\\ ⋮& \ddots & ⋮\\ 0& \cdots & \frac{1}{{\sigma }^p}\end{array}\right),{\Gamma }^1=\left(\begin{array}{ccc}\frac{1}{{\sigma }_1^1}& \cdots & 0\\ ⋮& \ddots & ⋮\\ 0& \cdots & \frac{1}{{\sigma }_1^q}\end{array}\right).\end{array}$$

Let S_c = Γ¹(S − E[S]) and R_c = Γ(R − E[R]). The least square linear regression of S_c with respect to R_c is achieved by estimating the (p, q) matrix θ_c such that $E\left[{||S_c-{\theta }_c^{{}^{\prime }}{R}_c||}^2\right]$ is minimal. Then θ_c = Γ⁻¹(B⁻¹ F)Γ¹ ⇔ θ = B⁻¹ F = Γθ_c(Γ¹)⁻¹.

The second change is to use, at each step of the process, several observations of (R, S) or an estimation of B and F computed recursively from all observations until the current step without storing them.

More precisely, the convergence of three processes with online standardized data is studied in sections 2, 3, 4 respectively.

First, in section 2, a process with a variable step-size a_n and use of several online standardized observations at each step is studied; note that the number of observations at each step may vary with n.

Secondly, in section 3, an averaged process with a constant step-size and use of a varying number of online standardized observations at each step is studied.

Thirdly, in section 4, a process with a constant or variable step-size and use of all online standardized observations until the current step to estimate B and F is studied.

These three processes are tested on several datasets when q = 1, S being a continuous or binary variable, and compared to existing processes in section 5. Note that when S is a binary variable, linear regression is equivalent to a linear discriminant analysis. It appears that the third-defined process most often yields the best results for the same number of observations used or for the same duration of computing time used.

These processes belong to the family of stochastic gradient processes and are adapted to data streams. Batch gradient and stochastic gradient methods are presented and compared in [7] and reviewed in [8], including noise reduction methods, like dynamic sample sizes methods, stochastic variance reduced gradient (also studied in [9]), second-order methods, ADAGRAD [10] and other methods. This work makes the following contributions to the variance reduction methods:

• In [9], the authors proposed a modification of the classical stochastic gradient algorithm to reduce directly the gradient of the function to be optimized in order to obtain a faster convergence. It is proposed in this article to reduce this gradient by an online standardization of the data.

• Gradient clipping [11] is another method to avoid a numerical explosion. The idea is to limit the norm of the gradient to a maximum number called threshold. This number must be chosen, a bad choice of threshold can affect the computing speed. Moreover it is then necessary to compare the norm of the gradient to this threshold at each step. In our approach the limitation of the gradient is implicitly obtained by online standardization of the data.

• If the expectation and the variance of the components of R and S were known, standardization of these variables could be made directly and convergence of the processes obtained using existing theorems. But these moments are unknown in the case of a data stream and are estimated online in this study. Thus the assumptions of the theorems of almost sure (a.s.) convergence of the processes studied in sections 2 and 3 and the corresponding proofs are more general than the classical ones in the linear regression case [1–5].

• The process defined in section 4 is not a classical batch method. Indeed in this type of method (gradient descent), the whole set of data is known a priori and is used at each step of the process. In the present study, new data are supposed to arrive at each step, as in a data stream, and are added to the preceding set of data, thus reducing by averaging the variance. This process can be considered as a dynamic batch method.

• A suitable choice of step-size is often crucial for obtaining good performance of a stochastic gradient process. If the step-size is too small, the convergence will be slower. Conversely, if the step-size is too large, a numerical explosion may occur during the first iterations. Following [6], a very simple choice of the step-size is proposed for the methods with a constant step-size.

• Another objective is to reduce computing time in order to take into account a maximum of data in the case of a data stream. It appears in the experiments that the use of all observations until the current step without storing them, several observations being introduced at each step, increases at best in general the convergence speed of the process. Moreover this can reduce the influence of outliers.

As a whole the major contributions of this work are to reduce gradient variance by online standardization of the data or use of a “dynamic” batch process, to avoid numerical explosions, to reduce computing time and consequently to better adapt the stochastic approximation processes used to the case of a data stream.

2 Convergence of a process with a variable step-size

Let (B_n, n ≥ 1) and (F_n, n ≥ 1) be two sequences of random matrices in ${\mathbb{R}}^{p\times p}$ and ${\mathbb{R}}^{p\times q}$ respectively. In this section, the convergence of the process (X_n, n ≥ 1) in ${\mathbb{R}}^{p\times q}$ recursively defined by

$$\begin{array}{c}\hfill X_{n+1}=X_n-a_n(B_n{X}_n-F_n)\end{array}$$

and its application to sequential linear regression are studied.

2.1 Theorem

Let X₁ be a random variable in ${\mathbb{R}}^{p\times q}$ independent from the sequence of random variables ((B_n, F_n), n ≥ 1) in ${\mathbb{R}}^{p\times p}\times {\mathbb{R}}^{p\times q}$.

Denote T_n the σ-field generated by X₁ and (B₁, F₁),…,(B_n−1, F_n−1). X₁, X₂,…,X_n are T_n-measurable.

Let (a_n) be a sequence of positive numbers.

Make the following assumptions:

(H1a) There exists a positive definite symmetrical matrix B such that a.s.

1) ${\displaystyle \sum _{n=1}^{\infty }{a}_n||E\left[B_n|T_n\right]-B||<\infty }$

2) ${\displaystyle \sum _{n=1}^{\infty }{a}_n^{2}E\left[{||B_n-B||}^2|T_n\right]<\infty }$.

(H2a) There exists a matrix F such that a.s.

1)${\displaystyle \sum _{n=1}^{\infty }{a}_n||E\left[F_n|T_n\right]-F||<\infty }$

2) ${\displaystyle \sum _{n=1}^{\infty }{a}_n^{2}E\left[{||F_n-F||}^2|T_n\right]<\infty }$.

(H3a) ${\displaystyle \sum _{n=1}^{\infty }{a}_n=\infty ,\sum _{n=1}^{\infty }{a}_n^2<\infty }$.

Theorem 1 Suppose H1a, H2a and H3a hold. Then X_n converges to θ = B⁻¹ F a.s.

State the Robbins-Siegmund lemma [12] used in the proof.

Lemma 2 Let (Ω, A, P) be a probability space and (T_n) a non-decreasing sequence of sub-σ-fields of A. Suppose for all n, z_n, α_n, β_n and γ_n are four integrable non-negative T_n-measurable random variables defined on (Ω, A, P) such that:

$$\begin{array}{ccc}\hfill E\left[z_{n+1}|T_n\right]& \le z_n(1+{\alpha }_n)+{\beta }_n-{\gamma }_n& a.s.\hfill \end{array}$$

Then, in the set $\{{\displaystyle \sum _{n=1}^{\infty }{\alpha }_n<\infty },{\displaystyle \sum _{n=1}^{\infty }{\beta }_n<\infty }\}$, (z_n) converges to a finite random variable and ${\displaystyle \sum _{n=1}^{\infty }{\gamma }_n<\infty }$ a.s.

Proof of Theorem 1. The Frobenius norm ‖A‖ for a matrix A is used. Recall that, if ‖A‖₂ denotes the spectral norm of A, ‖AB‖ ≤ ‖A‖₂‖B‖.

$$\begin{array}{ccc}\hfill X_{n+1}-\theta & =& X_n-\theta -a_n(B_n{X}_n-F_n)\hfill \\ & =& (I-a_{n}B)(X_n-\theta )-a_n\left(\right(B_n-B)X_n-(F_n-F\left)\right)\hfill \end{array}$$

Denote Z_n = (B_n − B)X_n − (F_n − F) = (B_n − B)(X_n − θ) + (B_n − B)θ − (F_n − F) and $X_n^1=X_n-\theta$. Then:

$$\begin{array}{ccc}\hfill X_{n+1}^1& =& (I-a_{n}B)X_n^1-a_n{Z}_n\hfill \\ \hfill {||X_{n+1}^1||}^2& =& {||(I-a_{n}B)X_n^1||}^2-2a_n⟨(I-a_{n}B)X_n^1,Z_n⟩+a_n^2{||Z_n||}^2.\hfill \end{array}$$

Denote λ the smallest eigenvalue of B. As a_n → 0, we have for n sufficiently large

$$\begin{array}{c}\hfill ||I-a_n{B||}_2=1-a_n\lambda <1.\end{array}$$

Then, taking the conditional expectation with respect to T_n yields almost surely:

$$\begin{array}{ccc}\hfill E\left[{||X_{n+1}^1||}^2|T_n\right]& \le & {(1-a_n\lambda )}^2{||X_n^1||}^2+2a_n\left|⟨(I-a_{n}B)X_n^1,E\left[Z_n|T_n\right]⟩\right|+\hfill \\ & & a_n^{2}E\left[{||Z_n||}^2|T_n\right],\hfill \\ \hfill E\left[Z_n|T_n\right]& =& \left(E\right[B_n|T_n]-B)X_n^1+\left(E\right[B_n|T_n]-B)\theta -\left(E\right[F_n|T_n]-F).\hfill \end{array}$$

Denoting

$$\begin{array}{ccc}\hfill {\beta }_n& =& ||E\left[B_n|T_n\right]-B||,{\delta }_n=||E\left[F_n|T_n\right]-F||,\hfill \\ \hfill b_n& =& E\left[{||B_n-B||}^2|T_n\right],d_n=E\left[{||F_n-F||}^2|T_n\right],\hfill \end{array}$$

we obtain, as $||X_n^1||\le 1+{||X_n^1||}^2$:

$$\begin{array}{ccc}\hfill |⟨(I-a_{n}B)X_n^1,E[Z_n|T_n]⟩|& \le & ||X_n^1||||E\left[Z_n|T_n\right]||\hfill \\ & \le & {||X_n^1||}^2\left({\beta }_n\right(1+||\theta ||)+{\delta }_n)+{\beta }_n||\theta ||+{\delta }_n,\hfill \\ \hfill E\left[{||Z_n||}^2|T_n\right]& \le & 3b_n{||X_n^1||}^2+3b_n{||\theta ||}^2+3d_n,\hfill \\ \hfill E\left[{||X_{n+1}^1||}^2|T_n\right]& \le & (1+a_n^2{\lambda }^2+2(1+||\theta ||)a_n{\beta }_n+2a_n{\delta }_n+3a_n^2{b}_n){||X_n^1||}^2+\hfill \\ & & 2||\theta ||a_n{\beta }_n+2a_n{\delta }_n+3{||\theta ||}^2{a}_n^2{b}_n+3a_n^2{d}_n-2a_n\lambda {||X_n^1||}^2.\hfill \end{array}$$

Applying Robbins-Siegmund lemma under assumptions H1a, H2a and H3a implies that there exists a non-negative random variable T such that a.s.

$$\begin{array}{c}\hfill ||X_n^1||⟶T,\sum _{n=1}^{\infty }{a}_n{||X_n^1||}^2<\infty .\end{array}$$

As ${\displaystyle \sum _{n=1}^{\infty }{a}_n=\infty }$, T = 0 a.s. ∎

A particular case with the following assumptions is now studied.

(H1a’) There exist a positive definite symmetrical matrix B and a positive real number b such that a.s.

1) for all n, E[B_n|T_n] = B

2) $\begin{array}{c}\sup \\ n\end{array}$ E[‖B_n − B‖²|T_n] < b.

(H2a’) There exist a matrix F and a positive real number d such that a.s.

1) for all n, E[F_n|T_n] = F

2) $\begin{array}{c}\sup \\ n\end{array}$ E[‖F_n − F‖²|T_n] < d.

(H3a’) Denoting λ the smallest eigenvalue of B,

${\displaystyle (a_n=\frac{a}{n^{\alpha }},a>0,\frac{1}{2}<\alpha <1)}$ or ${\displaystyle (a_n=\frac{a}{n},a>\frac{1}{2\lambda })}$.

Theorem 3 Suppose H1a’, H2a’ and H3a’ hold. Then X_n converges to θ almost surely and in quadratic mean. Moreover $\overline{lim}\frac{1}{a_n}E\left[{||X_n-\theta ||}^2\right]<\infty$.

Proof of Theorem 3. In the proof of theorem 1, take β_n = 0, δ_n = 0, b_n < b, d_n < d; then a.s.:

$$\begin{array}{c}\hfill E\left[{||X_{n+1}^1||}^2|T_n\right]\le (1+{\lambda }^2{a}_n^2+3b{a}_n^2){||X_n^1||}^2+3(b{||\theta ||}^2+d)a_n^2-2a_n\lambda {||X_n^1||}^2.\end{array}$$

Taking the mathematical expectation yields:

$$\begin{array}{c}\hfill E\left[{||X_{n+1}^1||}^2\right]\le (1+({\lambda }^2+3b\left)a_n^2\right)E\left[{||X_n^1||}^2\right]+3(b{||\theta ||}^2+d)a_n^2-2a_n\lambda E\left[{||X_n^1||}^2\right].\end{array}$$

By Robbins-Siegmund lemma:

$$\begin{array}{c}\hfill \exists t\ge 0:E\left[{||X_n^1||}^2\right]⟶t;\sum _{n=1}^{\infty }{a}_{n}E\left[{||X_n^1||}^2\right]<\infty .\end{array}$$

As ${\displaystyle \sum _{n=1}^{\infty }{a}_n=\infty }$, t = 0. Therefore, there exist $N\in \mathbb{N}$ and f > 0 such that for n > N:

$$\begin{array}{c}\hfill E\left[{||X_{n+1}^1||}^2\right]\le (1-2a_n\lambda )E\left[{||X_n^1||}^2\right]+f{a}_n^2.\end{array}$$

Applying a lemma of Schmetterer [13] for ${\displaystyle a_n=\frac{a}{n^{\alpha }}}$ with ${\displaystyle \frac{1}{2}<\alpha <1}$ yields:

$$\begin{array}{c}\hfill \overline{lim}{n}^{\alpha }E\left[{||X_n^1||}^2\right]<\infty .\end{array}$$

Applying a lemma of Venter [14] for ${\displaystyle a_n=\frac{a}{n}}$ with ${\displaystyle a>\frac{1}{2\lambda }}$ yields:

$$\begin{array}{c}\hfill \overline{lim}nE\left[{||X_n^1||}^2\right]<\infty ∎\end{array}$$

2.2 Application to linear regression with online standardized data

Let (R₁, S₁),…,(R_n, S_n),… be an i.i.d. sample of a random vector (R, S) in ${\mathbb{R}}^p\times {\mathbb{R}}^q$. Let Γ (respectively Γ¹) be the diagonal matrix of order p (respectively q) of the inverses of the standard deviations of the components of R (respectively S).

Define the correlation matrices

$$\begin{array}{c}B=\Gamma E\left[\right(R-E\left[R\right]\left){(R-E[R\left]\right)}^{\prime }\right]\Gamma ,\hfill \\ F=\Gamma E\left[\right(R-E\left[R\right]\left){(S-E[S\left]\right)}^{\prime }\right]{\Gamma }^1.\hfill \end{array}$$

Suppose that B⁻¹ exists. Let θ = B⁻¹ F.

Denote ${\overline{R}}_n$ (respectively ${\overline{S}}_n$) the mean of the n-sample (R₁, R₂,…,R_n) of R (respectively (S₁, S₂,…,S_n) of S).

Denote ${\left(V_n^j\right)}^2$ the variance of the n-sample $(R_1^j,R_2^j,...,R_n^j)$ of the j^th component R^j of R, and ${\left(V_n^{1k}\right)}^2$ the variance of the n-sample $(S_1^k,S_2^k,...,S_n^k)$ of the k^th component S^k of S.

Denote Γ_n (respectively ${\Gamma }_n^1$) the diagonal matrix of order p (respectively q) whose element (j, j) (respectively (k, k)) is the inverse of ${\displaystyle \sqrt{\frac{n}{n-1}}{V}_n^j}$ (respectively ${\displaystyle \sqrt{\frac{n}{n-1}}{V}_n^{1k}}$).

Let (m_n, n ≥ 1) be a sequence of integers. Denote ${\displaystyle M_n=\sum _{k=1}^n{m}_k}$ for n ≥ 1, M₀ = 0 and I_n = {M_n−1+1,…,M_n}.

Define

$$\begin{array}{cc}\hfill B_n& ={\Gamma }_{M_{n-1}}\frac{1}{m_n}\sum _{j\in I_n}(R_j-{\overline{R}}_{M_{n-1}}){(R_j-{\overline{R}}_{M_{n-1}})}^{\prime }{\Gamma }_{M_{n-1}},\\ \hfill F_n& ={\Gamma }_{M_{n-1}}\frac{1}{m_n}\sum _{j\in I_n}(R_j-{\overline{R}}_{M_{n-1}}){(S_j-{\overline{S}}_{M_{n-1}})}^{\prime }{\Gamma }_{M_{n-1}}^1.\end{array}$$

Define recursively the process (X_n, n ≥ 1) in ${\mathbb{R}}^{p\times q}$ by

$$\begin{array}{c}\hfill X_{n+1}=X_n-a_n(B_n{X}_n-F_n).\end{array}$$

Corollary 4 Suppose there is no affine relation between the components of R and the moments of order 4 of (R, S) exist. Suppose moreover that assumption H3a” holds:

(H3a”) ${\displaystyle a_n>0,\sum _{n=1}^{\infty }\frac{a_n}{\sqrt{n}}<\infty ,\sum _{n=1}^{\infty }{a}_n^2<\infty .}$

Then X_n converges to θ a.s.

This process was tested on several datasets and some results are given in section 5 (process S11 for m_n = 1 and S12 for m_n = 10).

The following lemma is first proved.

Lemma 5 Suppose the moments of order 4 of R exist and a_n > 0, ${\displaystyle \sum _{n=1}^{\infty }\frac{a_n}{\sqrt{n}}<\infty }$. Then ${\displaystyle \sum _{n=1}^{\infty }{a}_n||{\overline{R}}_{M_{n-1}}-E\left[R\right]||<\infty }$ and ${\displaystyle \sum _{n=1}^{\infty }{a}_n||{\Gamma }_{M_{n-1}}-\Gamma ||<\infty }$ a.s.

Proof of Lemma 5. The usual Euclidean norm for vectors and the spectral norm for matrices are used in the proof.

Step 1:

Denote ${\displaystyle Var\left[R\right]=E\left[{||R-E\left[R\right]||}^2\right]=\sum _{j=1}^{p}Var\left[R^j\right]}$.

$$\begin{array}{ccc}\hfill E\left[{||{\overline{R}}_{M_{n-1}}-E\left[R\right]||}^2\right]& =& \sum _{j=1}^{p}Var\left[{\overline{R}}_{M_{n-1}}^j\right]=\sum _{j=1}^p\frac{Var\left[R^j\right]}{M_{n-1}}\le \frac{Var\left[R\right]}{n-1}.\hfill \end{array}$$

Then:

$$\begin{array}{ccc}\hfill \sum _{n=1}^{\infty }{a}_{n}E[||{\overline{R}}_{M_{n-1}}-E[R]||]& \le & \sqrt{Var\left[R\right]}\sum _{n=1}^{\infty }\frac{a_n}{\sqrt{n-1}}<\infty \text{by}\text{H}3\text{a}”.\hfill \end{array}$$

It follows that ${\displaystyle \sum _{n=1}^{\infty }{a}_n||{\overline{R}}_{M_{n-1}}-E\left[R\right]||<\infty }$ a.s.

Likewise ${\displaystyle \sum _{n=1}^{\infty }{a}_n||{\overline{S}}_{M_{n-1}}-E\left[S\right]||<\infty }$ a.s.

Step 2:

$$\begin{array}{ll}\left|\right|{\Gamma }_{M_{n-1}}-\Gamma \left|\right|\hfill & =\underset{j=1,\dots ,p}{\text{max}}\left|\frac{1}{\sqrt{\frac{M_{n-1}}{M_{n-1}-1}}{V}_{M_{n-1}}^j}-\frac{1}{\sqrt{Var\left[R^j\right]}}\right|\hfill \\ \hfill & \le {\displaystyle \sum _{j=1}^p\frac{\left|\sqrt{\frac{M_{n-1}}{M_{n-1}-1}}{V}_{M_{n-1}}^j-\sqrt{Var\left[R^j\right]}\right|}{\sqrt{\frac{M_{n-1}}{M_{n-1}-1}}{V}_{M_{n-1}}^j\sqrt{Var\left[R^j\right]}}}\hfill \\ \hfill & ={\displaystyle \sum _{j=1}^p\frac{\left|\frac{M_{n-1}}{M_{n-1}-1}{\left(V_{M_{n-1}}^j\right)}^2-Var\left[R^j\right]\right|}{\sqrt{\frac{M_{n-1}}{M_{n-1}-1}}{V}_{M_{n-1}}^j\sqrt{Var\left[R^j\right]}\left(\sqrt{\frac{M_{n-1}}{M_{n-1}-1}}{V}_{M_{n-1}}^j+\sqrt{Var\left[R^j\right]}\right)}}.\hfill \end{array}$$

Denote ${\mu }_4^j$ the centered moment of order 4 of R^j. We have:

$$\begin{array}{ccc}\hfill E\left[\right|\frac{M_{n-1}}{M_{n-1}-1}{\left(V_{M_{n-1}}^j\right)}^2-Var\left[R^j\right]\left|\right]& \le & \sqrt{Var\left[\frac{M_{n-1}}{M_{n-1}-1}{\left(V_{M_{n-1}}^j\right)}^2\right]}\hfill \\ & =& O\left(\sqrt{\frac{{\mu }_4^j-{\left(Var\right[R^j\left]\right)}^2}{M_{n-1}}}\right).\hfill \end{array}$$

Then by H3a”, as M_n−1 ≥ n−1:

$$\begin{array}{c}\hfill \sum _{n=1}^{\infty }{a}_n\sum _{j=1}^{p}E\left[\right|\frac{M_{n-1}}{M_{n-1}-1}{\left(V_{M_{n-1}}^j\right)}^2-Var\left[R^j\right]\left|\right]<\infty \\ \hfill \Rightarrow \sum _{n=1}^{\infty }{a}_n\sum _{j=1}^p|\frac{M_{n-1}}{M_{n-1}-1}{\left(V_{M_{n-1}}^j\right)}^2-Var\left[R^j\right]|<\infty \text{a}.\text{s}.\end{array}$$

As ${\displaystyle {\left(V_{M_{n-1}}^j\right)}^2\to Var\left[R^j\right]}$ a.s., j = 1,…,p, this implies:

$$\begin{array}{c}\hfill \sum _{n=1}^{\infty }{a}_n||{\Gamma }_{M_{n-1}}-\Gamma ||<\infty \text{a}.\text{s}.∎\end{array}$$

Proof of Corollary 4.

Step 1: prove that assumption H1a1 of theorem 1 is verified.

Denote R^c = R − E[R], $R_j^c=R_j-E\left[R\right]$, ${\overline{R}}_j^c={\overline{R}}_j-E\left[R\right]$.

$$\begin{array}{ccc}\hfill B_n& =& {\Gamma }_{M_{n-1}}\frac{1}{m_n}\sum _{j\in I_n}(R_j^c-{\overline{R}}_{M_{n-1}}^c){(R_j^c-{\overline{R}}_{M_{n-1}}^c)}^{\prime }{\Gamma }_{M_{n-1}}\hfill \\ & =& {\Gamma }_{M_{n-1}}\frac{1}{m_n}\sum _{j\in I_n}(R_j^c{R_j^c}^{\prime }-{\overline{R}}_{M_{n-1}}^c{R_j^c}^{\prime }-R_j^c{\left({\overline{R}}_{M_{n-1}}^c\right)}^{\prime }+{\overline{R}}_{M_{n-1}}^c{\left({\overline{R}}_{M_{n-1}}^c\right)}^{\prime }){\Gamma }_{M_{n-1}}.\hfill \\ \hfill B& =& \Gamma E\left[R^c{R^c}^{\prime }\right]\Gamma .\hfill \end{array}$$

As Γ_Mn−1 and ${\overline{R}}_{M_{n-1}}$ are T_n-measurable and $R_j^c$, j ∈ I_n, is independent of T_n, with $E\left[R_j^c\right]=0$:

$$\begin{array}{ccc}\hfill E\left[B_n|T_n\right]-B& =& {\Gamma }_{M_{n-1}}\left(E\right[R^c{R^c}^{\prime }]+{\overline{R}}_{M_{n-1}}^c{\left({\overline{R}}_{M_{n-1}}^c\right)}^{\prime }){\Gamma }_{M_{n-1}}-\Gamma E\left[R^c{R^c}^{\prime }\right]\Gamma \hfill \\ & =& ({\Gamma }_{M_{n-1}}-\Gamma )E\left[R^c{R^c}^{\prime }\right]{\Gamma }_{M_{n-1}}+\Gamma E\left[R^c{R^c}^{\prime }\right]({\Gamma }_{M_{n-1}}-\Gamma )\hfill \\ & & +{\Gamma }_{M_{n-1}}{\overline{R}}_{M_{n-1}}^c{\left({\overline{R}}_{M_{n-1}}^c\right)}^{\prime }{\Gamma }_{M_{n-1}}\text{a}.\text{s}.\hfill \end{array}$$

As Γ_Mn−1 and ${\overline{R}}_{M_{n-1}}^c$ converge respectively to Γ and 0 a.s. and by lemma 5, ${\displaystyle \sum _{n=1}^{\infty }{a}_n||{\Gamma }_{M_{n-1}}-\Gamma ||<\infty }$ and ${\displaystyle \sum _{n=1}^{\infty }{a}_n||{\overline{R}}_{M_{n-1}}^c||<\infty }$ a.s., it follows that ${\displaystyle \sum _{n=1}^{\infty }{a}_n||E\left[B_n|T_n\right]-B||<\infty }$ a.s.

Step 2: prove that assumption H1a2 of theorem 1 is verified.

$$\begin{array}{ll}{\Vert B_n-B\Vert }^2\hfill & \le 2{\Vert {\Gamma }_{M_{n-1}}\frac{1}{m_n}{\displaystyle \sum _{j\in I_n}\left(R_j^c-{\overline{R}}_{M_{n-1}}^c\right)}{\left(R_j^c-{\overline{R}}_{M_{n-1}}^c\right)}^{\prime }{\Gamma }_{M_{n-1}}\Vert }^2\hfill \\ \hfill & +2{\Vert \Gamma E\left[R^c{R}^c{}^{\prime }\right]\Gamma \Vert }^2\hfill \\ \hfill & \le 2{\Vert {\Gamma }_{M_{n-1}}\Vert }^4\frac{1}{m_n}{\displaystyle \sum _{j\in I_n}{\Vert R_j^c-{\overline{R}}_{M_{n-1}}^c\Vert }^4}+2{\Vert \Gamma E\left[R^c{R}^c{}^{\prime }\right]\Gamma \Vert }^2\hfill \\ \hfill & \le 2{\Vert {\Gamma }_{M_{n-1}}\Vert }^4\frac{1}{m_n}{\displaystyle \sum _{j\in I_n}{2}^3}\left({\Vert R_j^c\Vert }^4+{\Vert {\overline{R}}_{M_{n-1}}^c\Vert }^4\right)+2{\Vert \Gamma E\left[R^c{R}^c{}^{\prime }\right]\Gamma \Vert }^2.\hfill \end{array}$$

$$\begin{array}{ccc}\hfill E\left[{||B_n-B||}^2|T_n\right]& \le & 2^4{||{\Gamma }_{M_{n-1}}||}^4\left(E\right[{||R^c||}^4]+{||{\overline{R}}_{M_{n-1}}^c||}^4)+2{||\Gamma E\left[R^c{R^c}^{\prime }\right]\Gamma ||}^2\text{a}.\text{s}.\hfill \end{array}$$

As Γ_Mn−1 and ${\displaystyle {\overline{R}}_{M_{n-1}}^c}$ converge respectively to Γ and 0 a.s., and ${\displaystyle \sum _{n=1}^{\infty }{a}_n^2<\infty }$, it follows that ${\displaystyle \sum _{n=1}^{\infty }{a}_n^{2}E\left[{||B_n-B||}^2|T_n\right]<\infty }$ a.s.

Step 3: the proofs of the verification of assumptions H2a1 and H2a2 of theorem 1 are similar to the previous ones, B_n and B being respectively replaced by

$$\begin{array}{ccc}\hfill F_n& =& {\Gamma }_{M_{n-1}}\frac{1}{m_n}\sum _{j\in I_n}(R_j^c-{\overline{R}}_{M_{n-1}}^c){(S_j^c-{\overline{S}}_{M_{n-1}}^c)}^{\prime }{\Gamma }_{M_{n-1}}^1,\hfill \\ \hfill F& =& \Gamma E\left[R^c{S^c}^{\prime }\right]{\Gamma }^1∎\hfill \end{array}$$

3 Convergence of an averaged process with a constant step-size

In this section, the process (X_n, n ≥ 1) with a constant step-size a and the averaged process (Y_n, n ≥ 1) in ${\mathbb{R}}^{p\times q}$ are recursively defined by

$$\begin{array}{ccc}\hfill X_{n+1}& =& X_n-a(B_n{X}_n-F_n)\hfill \\ \hfill Y_{n+1}& =& \frac{1}{n+1}\sum _{j=1}^{n+1}{X}_j=Y_n-\frac{1}{n+1}(Y_n-X_{n+1}).\hfill \end{array}$$

The a.s. convergence of (Y_n, n ≥ 1) and its application to sequential linear regression are studied.

3.1 Lemma

Lemma 6 Let three real sequences (u_n), (v_n) and (a_n), with u_n > 0 and a_n > 0 for all n, and a real positive number λ such that, for n ≥ 1,

$$\begin{array}{ccc}\hfill u_{n+1}& \le & (1-a_n\lambda )u_n+a_n{v}_n.\hfill \end{array}$$

Suppose:

• 1) v_n → 0

• 2) ${\displaystyle (a_n=a<\frac{1}{\lambda })}$ or ${\displaystyle (a_n\to 0\text{,}\sum _{n=1}^{\infty }{a}_n=\infty )}$.

• Under assumptions 1 and 2, u_n → 0.

Proof of Lemma 6. In the case a_n depending on n, as a_n → 0, we can suppose without loss of generality that 1 − a_n λ > 0 for n ≥ 1. We have:

$$\begin{array}{ccc}\hfill u_{n+1}& \le & \prod _{i=1}^n(1-a_i\lambda )u_1+\sum _{i=1}^n{a}_i\prod _{l=i+1}^n(1-a_l\lambda )v_i\text{,}\text{with}\prod _{n+1}^n=1.\hfill \end{array}$$

Now, for n₁ ≤ n₂ ≤ n and 0 < c_i < 1 with c_i = a_i λ for all i, we have:

$$\begin{array}{ccc}\hfill \sum _{i=n_1}^{n_2}{c}_i\prod _{l=i+1}^n(1-c_l)& =& \sum _{i=n_1}^{n_2}(1-(1-c_i\left)\right)\prod _{l=i+1}^n(1-c_l)\hfill \\ & =& \sum _{i=n_1}^{n_2}\left(\prod _{l=i+1}^n\right(1-c_l)-\prod _{l=i}^n(1-c_l\left)\right)\hfill \\ & =& \prod _{l=n_2+1}^n(1-c_l)-\prod _{l=n_1}^n(1-c_l)\le \prod _{l=n_2+1}^n(1-c_l)\le 1.\hfill \end{array}$$

Let ϵ > 0. There exists N such that for i > N, ${\displaystyle \left|v_i\right|<\frac{ϵ}{3}\lambda }$. Then for n ≥ N, applying the previous inequality with c_i = a_i λ, n₁ = 1, n₂ = N, yields:

$$\begin{array}{ccc}\hfill u_{n+1}& \le & \prod _{i=1}^n(1-a_i\lambda )u_1+\sum _{i=1}^N{a}_i\lambda \prod _{l=i+1}^n(1-a_l\lambda )\frac{\left|v_i\right|}{\lambda }+\frac{ϵ}{3}\sum _{i=N+1}^n{a}_i\lambda \prod _{l=i+1}^n(1-a_l\lambda )\hfill \\ & \le & \prod _{i=1}^n(1-a_i\lambda )u_1+\frac{1}{\lambda }\underset{1\le i\le N}{max}\left|v_i\right|\prod _{l=N+1}^n(1-a_l\lambda )+\frac{ϵ}{3}.\hfill \end{array}$$

In the case a_n depending on n, ln(1 − a_i λ) ∼ −a_i λ as a_i → 0(i → ∞); then, as ${\displaystyle \sum _{n=1}^{\infty }{a}_n=\infty }$, ${\displaystyle \prod _{l=N+1}^n(1-a_l\lambda )\to 0(n\to \infty )}$.

In the case a_n = a, ${\displaystyle \prod _{l=N+1}^n(1-a\lambda )={(1-a\lambda )}^{n-N}\to 0(n\to \infty )}$ as 0 < 1 − aλ < 1.

Thus there exists N₁ such that u_n+1 < ϵ for n > N₁ ∎

3.2 Theorem

Make the following assumptions

(H1b) There exist a positive definite symmetrical matrix B in ${\mathbb{R}}^{p\times p}$ and a positive real number b such that a.s.

1) lim_{n → ∞}(E[B_n|T_n] − B) = 0

2) ${\displaystyle \sum _{n=1}^{\infty }\frac{1}{n}{\left(E\right[{||E\left[B_n|T_n\right]-B||}^2\left]\right)}^{\frac{1}{2}}<\infty }$

3) sup_n E[‖B_n−B‖²|T_n] ≤ b.

(H2b) There exist a matrix F in ${\mathbb{R}}^{p\times q}$ and a positive real number d such that a.s.

1) lim_n→∞(E[F_n|T_n] − F) = 0

2) sup_n E [‖F_n − F‖²|T_n] ≤ d.

(H3b) λ and λ_max being respectively the smallest and the largest eigenvalue of B, ${\displaystyle 0<a<min(\frac{1}{{\lambda }_{max}},\frac{2\lambda }{{\lambda }^2+b})}$.

Theorem 7 Suppose H1b, H2b and H3b hold. Then Y_n converges to θ = B⁻¹ F a.s.

Remark 1 Györfi and Walk [5] proved that Y_n converges to θ a.s. and in quadratic mean under the assumptions E[B_n|T_n] = B, E[F_n|T_n] = F, H1b2 and H2b2. Theorem 7 is an extension of their a.s. convergence result when E[B_n|T_n] → B and E[F_n|T_n] → F a.s.

Remark 2 Define $R_1=\left(\begin{array}{c}R\\ 1\end{array}\right)$, $B=E\left[R_1{R}_1^{\prime }\right]$, F = E[R₁ S′]. If ((R_1n, S_n), n ≥ 1) is an i.i.d. sample of (R₁, S) whose moments of order 4 exist, assumptions H1b and H2b are verified for $B_n=R_{1n}{R}_{1n}^{\prime }$ and $F_n=R_{1n}{S}_n^{\prime }$ as $E\left[R_{1n}{R}_{1n}^{\prime }|T_n\right]=E\left[R_1{R}_1^{\prime }\right]=B$ and $E\left[R_{1n}{S}_n^{\prime }|T_n\right]=F$.

Proof of Theorem 7. Denote

$$\begin{array}{ccc}\hfill Z_n& =& (B_n-B)(X_n-\theta )+(B_n-B)\theta -(F_n-F),\hfill \\ \hfill X_n^1& =& X_n-\theta ,\hfill \\ \hfill Y_n^1& =& Y_n-\theta =\frac{1}{n}\sum _{j=1}^n{X}_j^1.\hfill \end{array}$$

Step 1: give a sufficient condition to have $Y_n^1\to 0$ a.s.

We have (cf. proof of theorem 1):

$$\begin{array}{ccc}\hfill X_{n+1}^1& =& (I-aB)X_n^1-a{Z}_n,\hfill \\ \hfill Y_{n+1}^1& =& \frac{1}{n+1}{X}_1^1+\frac{1}{n+1}\sum _{j=2}^{n+1}{X}_j^1\hfill \\ & =& \frac{1}{n+1}{X}_1^1+\frac{1}{n+1}\sum _{j=2}^{n+1}(I-aB)X_{j-1}^1-a\frac{1}{n+1}\sum _{j=2}^{n+1}{Z}_{j-1}\hfill \\ & =& \frac{1}{n+1}{X}_1^1+\frac{n}{n+1}(I-aB)Y_n^1-a\frac{1}{n+1}\sum _{j=1}^n{Z}_j.\hfill \end{array}$$

Take now the Frobenius norm of $Y_{n+1}^1$:

$$\begin{array}{ccc}\Vert Y_{n+1}^1\Vert & \le & \Vert \left(I-aB\right)Y_n^1\Vert +a\Vert \frac{1}{n+1}{\displaystyle \sum _{j=1}^n{Z}_j}-\frac{1}{n+1}\frac{1}{a}{X}_1^1\Vert .\end{array}$$

Under H3b, all the eigenvalues of I − aB are positive and the spectral norm of I − aB is equal to 1 − aλ. Then:

$$\begin{array}{ccc}\left|\right|Y_{n+1}^1\left|\right|& \le & \left(1-a\lambda \right)\Vert Y_n^1\Vert +a\Vert \frac{1}{n+1}{\displaystyle \sum _{j=1}^n{Z}_j}-\frac{1}{n+1}\frac{1}{a}{X}_1^1\Vert .\end{array}$$

By lemma 6, it suffices to prove ${\displaystyle \frac{1}{n}\sum _{j=1}^n{Z}_j\to 0}$ a.s. to conclude $Y_n^1\to 0$ a.s.

Step 2: prove that assumptions H1b and H2b imply respectively ${\displaystyle \frac{1}{n}\sum _{j=1}^n{B}_j\to B}$ and ${\displaystyle \frac{1}{n}\sum _{j=1}^n{F}_j\to F}$ a.s.

The proof is only given for (B_n), the other one being similar.

Assumption H1b3 implies sup_n E[‖B_n − B‖²] < ∞. It follows that, for each element $B_n^{kl}$ and B^kl of B_n and B respectively, ${\displaystyle \sum _{n=1}^{\infty }\frac{Var[B_n^{kl}-B^{kl}]}{n^2}<\infty }$. Therefore:

$$\begin{array}{c}\hfill \frac{1}{n}\sum _{j=1}^n(B_j^{kl}-B^{kl}-E[B_j^{kl}-B^{kl}|T_j\left]\right)\to 0\text{a}.\text{s}.\end{array}$$

As $E[B_j^{kl}-B^{kl}|T_j]\to 0$ a.s. by H1b1, we have for each (k, l)

$$\begin{array}{c}\hfill \frac{1}{n}\sum _{j=1}^n(B_j^{kl}-B^{kl})⟶0\text{a}.\text{s}.\end{array}$$

Then ${\displaystyle \frac{1}{n}\sum _{j=1}^n(B_j-B)\to 0}$ a.s.

Step 3: prove now that ${\displaystyle \frac{1}{n}\sum _{j=1}^n(B_j-B)X_j^1\to 0}$ a.s.

Denote β_n = ‖E[B_n|T_n] − B‖ and γ_n = ‖E[F_n|T_n] − F‖. β_n → 0 and γ_n → 0 a.s. under H1b1 and H2b1. Then: ∀δ > 0, ∀ε > 0, ∃N(δ, ε): ∀n ≥ N(δ, ε),

$$\begin{array}{c}\hfill P\left(\right\{su{p}_{j>n}\left({\beta }_j\right)\le \delta \}\bigcap \{su{p}_{j>n}\left({\gamma }_j\right)\le \delta \left\}\right)>1-\epsilon .\end{array}$$

As ${\displaystyle a<\frac{2\lambda }{{\lambda }^2+b}}$, choose η such that:

$$\begin{array}{c}\hfill 0<\eta <\frac{1}{b}\left(\frac{2\lambda }{a}-({\lambda }^2+b)\right)\iff \lambda >\frac{a}{2}({\lambda }^2+b+\eta b).\end{array}$$

Choose δ such that

$$\begin{array}{c}\hfill 0<\delta <\frac{1}{(1-a\lambda )(||\theta ||+2)}(\lambda -\frac{a}{2}({\lambda }^2+b+\eta b\left)\right).\end{array}$$

Let ε be fixed. Denote N₀ = N(δ, ε) and, for n > N₀,

$$\begin{array}{ccc}\hfill G_n& =& \left(\right\{\underset{N_0<j\le n}{sup}\left({\beta }_j\right)\le \delta \}\bigcap \{\underset{N_0<j\le n}{sup}\left({\gamma }_j\right)\le \delta \left\}\right),\hfill \\ \hfill G& =& \left(\right\{\underset{j>N_0}{sup}\left({\beta }_j\right)\le \delta \}\bigcap \{\underset{j>N_0}{sup}\left({\gamma }_j\right)\le \delta \left\}\right)=\bigcap _{n>N_0}{G}_n.\hfill \end{array}$$

Remark that G_n is T_n-measurable and, I_G denoting the indicator of G,

$$\begin{array}{c}\hfill G\subset G_{n+1}\subset G_n\iff I_G\le I_{G_{n+1}}\le I_{G_n}.\end{array}$$

Step 3a: prove that ${sup}_{n}E\left[{||X_n^1||}^2{I}_{G_n}\right]<\infty$.

$$\begin{array}{ccc}\hfill {||X_{n+1}^1||}^2{I}_{G_{n+1}}& \le & {||X_{n+1}^1||}^2{I}_{G_n}={||(I-aB)X_n^1{I}_{G_n}-a{Z}_n{I}_{G_n}||}^2\hfill \\ & \le & {||(I-aB)X_n^1{I}_{G_n}||}^2-2a⟨(I-aB)X_n^1{I}_{G_n},Z_n{I}_{G_n}⟩+a^2{||Z_n{I}_{G_n}||}^2.\hfill \end{array}$$

As the spectral norm ‖I − aB‖ = 1 − aλ, taking the conditional expectation with respect to T_n yields a.s.

$$\begin{array}{ccc}\hfill E\left[{||X_{n+1}^1||}^2{I}_{G_{n+1}}|T_n\right]& \le & {(1-a\lambda )}^2{||X_n^1{I}_{G_n}||}^2-2a⟨(I-aB)X_n^1{I}_{G_n},E\left[Z_n|T_n\right]I_{G_n}⟩\hfill \\ & & +a^{2}E\left[{||Z_n{I}_{G_n}||}^2|T_n\right].\hfill \end{array}$$

Now:

$$\begin{array}{ccc}\hfill ||E\left[Z_n|T_n\right]I_{G_n}||& =& ||\left(E\right[B_n|T_n]-B)X_n^1{I}_{G_n}+\left(E\right[B_n|T_n]-B)\theta I_{G_n}\hfill \\ & & -\left(E\right[F_n|T_n]-F)I_{G_n}||\hfill \\ & \le & \delta ||X_n^1{I}_{G_n}||+\delta (||\theta ||+1)\hfill \\ \hfill E\left[{||Z_n{I}_{G_n}||}^2|T_n\right]& \le & (1+\eta )E\left[{||(B_n-B)X_n^1{I}_{G_n}||}^2|T_n\right]\hfill \\ & & +(1+\frac{1}{\eta })E\left[{||(B_n-B)\theta I_{G_n}-(F_n-F)I_{G_n}||}^2|T_n\right]\hfill \\ & \le & (1+\eta )b{||X_n^1{I}_{G_n}||}^2+2(1+\frac{1}{\eta })(b{||\theta ||}^2+d).\hfill \end{array}$$

Therefore:

$$\begin{array}{ccc}\hfill E\left[{||X_{n+1}^1||}^2{I}_{G_{n+1}}|T_n\right]& \le & ({(1-a\lambda )}^2+2a(1-a\lambda )\delta +a^2(1+\eta \left)b\right){||X_n^1{I}_{G_n}||}^2\hfill \\ & +& 2a(1-a\lambda )\delta (||\theta ||+1)||X_n^1{I}_{G_n}||\hfill \\ & +& 2a^2(1+\frac{1}{\eta })(b{||\theta ||}^2+d).\hfill \end{array}$$

As ${\displaystyle ||X_n^1{I}_{G_n}||\le 1+{||X_n^1{I}_{G_n}||}^2}$, taking mathematical expectation yields:

$$\begin{array}{ccc}\hfill E\left[{||X_{n+1}^1||}^2{I}_{G_{n+1}}\right]& \le & \rho E\left[{||X_n^1{I}_{G_n}||}^2\right]+e,\hfill \\ \hfill \rho & =& {(1-a\lambda )}^2+2a(1-a\lambda )\delta (||\theta ||+2)+a^2(1+\eta )b,\hfill \\ \hfill e& =& 2a(1-a\lambda )\delta (||\theta ||+1)+2a^2(1+\frac{1}{\eta })(b{||\theta ||}^2+d).\hfill \end{array}$$

As ${\displaystyle \rho =1+2a\left(\right(1-a\lambda \left)\right(||\theta ||+2)\delta -\lambda +\frac{a}{2}({\lambda }^2+b+\eta b\left)\right)<1}$ by the choice of δ, this implies $g={sup}_{n}E\left[{||X_n^1||}^2{I}_{G_n}\right]<\infty$.

Step 3b: conclusion.

$$\begin{array}{ccc}\hfill E\left[{||(B_n-B)X_n^1{I}_{G_n}||}^2\right]& =& E\left[E\right[{||(B_n-B)X_n^1{I}_{G_n}||}^2|T_n\left]\right]\hfill \\ & \le & E\left[E\right[{||B_n-B||}^2|T_n\left]{||X_n^1{I}_{G_n}||}^2\right]\hfill \\ & \le & bg.\hfill \end{array}$$

Then: ${\displaystyle \sum _{n=1}^{\infty }\frac{E\left[{||(B_n-B)X_n^1{I}_{G_n}||}^2\right]}{n^2}<\infty }$. Therefore a.s.:

$$\begin{array}{c}\hfill \frac{1}{n}\sum _{j=1}^n\left(\right(B_j-B)X_j^1{I}_{G_j}-E[(B_j-B)X_j^1{I}_{G_j}|T_j\left]\right)⟶0.\end{array}$$

Now:

$$\begin{array}{c}\hfill \sum _{n=1}^{\infty }\frac{1}{n}E[||(E\left[B_n|T_n\right]-B)X_n^1{I}_{G_n}||]\le \sum _{n=1}^{\infty }\frac{1}{n}E[||E[B_n|T_n]-B||||X_n^1{I}_{G_n}||]\\ \hfill \le \sum _{n=1}^{\infty }\frac{1}{n}{\left(E\right[{||E\left[B_n|T_n\right]-B||}^2\left]\right)}^{\frac{1}{2}}{\left(E\right[{||X_n^1{I}_{G_n}||}^2\left]\right)}^{\frac{1}{2}}\\ \hfill \le g^{\frac{1}{2}}\sum _{n=1}^{\infty }\frac{1}{n}{\left(E\right[{||E\left[B_n|T_n\right]-B||}^2\left]\right)}^{\frac{1}{2}}<\infty \text{by}\text{H1b2}.\end{array}$$

Then:

$$\begin{array}{c}\hfill \sum _{n=1}^{\infty }\frac{1}{n}||\left(E\right[B_n|T_n]-B)X_n^1{I}_{G_n}||<\infty \text{a}.\text{s}.\end{array}$$

This implies by the Kronecker lemma:

$$\begin{array}{c}\hfill \frac{1}{n}\sum _{j=1}^n\left(E\right[B_j|T_j]-B)X_j^1{I}_{G_j}⟶0\text{a}.\text{s}.\end{array}$$

Therefore:

$$\begin{array}{c}\hfill \frac{1}{n}\sum _{j=1}^n(B_j-B)X_j^1{I}_{G_j}⟶0\text{a}.\text{s}.\end{array}$$

In G, I_Gj = 1 for all j, therefore ${\displaystyle \frac{1}{n}\sum _{j=1}^n(B_j-B)X_j^1⟶0}$ a.s. Then: ${\displaystyle P\left(\frac{1}{n}\sum _{j=1}^n\right(B_j-B)X_j^1⟶0)\ge P\left(G\right)>1-\epsilon }$. This is true for every ε > 0. Thus:

$$\begin{array}{c}\hfill \frac{1}{n}\sum _{j=1}^n(B_j-B)X_j^1⟶0\text{a}.\text{s}.\end{array}$$

Therefore by step 2 and step 1, we conclude that ${\displaystyle \frac{1}{n}\sum _{j=1}^n{Z}_j⟶0}$ and ${\displaystyle Y_n^1⟶0}$ a.s. ∎

3.3 Application to linear regression with online standardized data

Define as in section 2:

$$\begin{array}{ccc}\hfill B_n& =& {\Gamma }_{M_{n-1}}\frac{1}{m_n}\sum _{j\in I_n}(R_j-{\overline{R}}_{M_{n-1}}){(R_j-{\overline{R}}_{M_{n-1}})}^{\prime }{\Gamma }_{M_{n-1}},\hfill \\ \hfill F_n& =& {\Gamma }_{M_{n-1}}\frac{1}{m_n}\sum _{j\in I_n}(R_j-{\overline{R}}_{M_{n-1}}){(S_j-{\overline{S}}_{M_{n-1}})}^{\prime }{\Gamma }_{M_{n-1}}^1.\hfill \end{array}$$

Denote U = (R − E[R])(R − E[R])′, B = ΓE[U]Γ the correlation matrix of R, λ and λ_max respectively the smallest and the largest eigenvalue of B, b₁ = E[‖ΓUΓ − B‖²], F = ΓE[(R − E[R])(S − E[S])′]Γ¹.

Corollary 8 Suppose there is no affine relation between the components of R and the moments of order 4 of (R,S) exist. Suppose H3b1 holds:

(H3b1) ${\displaystyle 0<a<min(\frac{1}{{\lambda }_{max}},\frac{2\lambda }{{\lambda }^2+b_1})}$.

Then Y_n converges to θ = B⁻¹F a.s.

This process was tested on several datasets and some results are given in section 5 (process S21 for m_n = 1 and S22 for m_n = 10).

Proof of Corollary 8.

Step 1: introduction.

Using the decomposition of E[B_n|T_n] − B established in the proof of corollary 4, as ${\overline{R}}_{M_{n-1}}⟶E\left[R\right]$ and Γ_{Mn − 1} ⟶ Γ a.s., it is obvious that E[B_n|T_n] − B ⟶ 0 a.s. Likewise E[F_n|T_n] − F ⟶ 0 a.s. Thus assumptions H1b1 and H2b1 are verified.

Suppose that Y_n does not converge to θ almost surely.

Then there exists a set of probability ε₁ > 0 in which Y_n does not converge to θ.

Denote ${\sigma }^j=\sqrt{Var\left[R^j\right]}$, j = 1,…,p.

As ${\overline{R}}_{M_{n-1}}-E\left[R\right]⟶0$, $\sqrt{\frac{M_{n-1}}{M_{n-1}-1}}{V}_{M_{n-1}}^j-{\sigma }^j⟶0$, j = 1,…,p and Γ_{Mn − 1} − Γ ⟶ 0 almost surely, there exists a set G of probability greater than $1-\frac{{\epsilon }_1}{2}$ in which these sequences of random variables converge uniformly to θ.

Step 2: prove that ${\displaystyle \sum _{n=1}^{\infty }\frac{1}{n}{\left(E\right[||{\Gamma }_{M_{n-1}}-\Gamma ||I_G\left]\right)}^{\frac{1}{2}}<\infty }$.

By step 2 of the proof of lemma 5, we have for n > N:

$$\begin{array}{ccc}\left|\right|{\Gamma }_{M_{n-1}}-\Gamma \left|\right|I_G& \le & {\displaystyle \sum _{j=1}^p\frac{\left|\frac{M_{n-1}}{M_{n-1}-1}{\left(V_{M_{n-1}}^j\right)}^2-{\left({\sigma }^j\right)}^2\right|}{\sqrt{\frac{M_{n-1}}{M_{n-1}-1}}{V}_{M_{n-1}}^j{\sigma }^j\left(\sqrt{\frac{M_{n-1}}{M_{n-1}-1}}{V}_{M_{n-1}}^j+{\sigma }^j\right)}{I}_G.}\end{array}$$

As in G, ${\displaystyle \sqrt{\frac{M_{n-1}}{M_{n-1}-1}}{V}_{M_{n-1}}^j}$ converges uniformly to σ^j for j = 1,…,p, there exists c > 0 such that

$$\begin{array}{ccc}\hfill ||{\Gamma }_{M_{n-1}}-\Gamma ||I_G& \le & c\sum _{j=1}^p|\frac{M_{n-1}}{M_{n-1}-1}{\left(V_{M_{n-1}}^j\right)}^2-{\left({\sigma }^j\right)}^2|.\hfill \end{array}$$

Then there exists d > 0 such that

$$\begin{array}{ccc}\hfill E[||{\Gamma }_{M_{n-1}}-\Gamma ||I_G]& \le & \frac{d}{\sqrt{M_{n-1}}}\le \frac{d}{\sqrt{n-1}}.\hfill \end{array}$$

Therefore ${\displaystyle \sum _{n=1}^{\infty }\frac{1}{n}{\left(E\right[||{\Gamma }_{M_{n-1}}-\Gamma ||I_G\left]\right)}^{\frac{1}{2}}<\infty }$.

Step 3: prove that assumption H1b2 is verified in G.

Using the decomposition of E[B_n|T_n] − B given in step 1 of the proof of corollary 4, with R^c = R − E[R] and ${\displaystyle {\overline{R}}_{M_{n-1}}^c={\overline{R}}_{M_{n-1}}-E\left[R\right]}$ yields a.s.:

$$\begin{array}{ccc}\hfill \left(E\right[B_n|T_n]-B)I_G& =& \left(\right({\Gamma }_{M_{n-1}}-\Gamma \left)E\right[R^c{R^c}^{\prime }]{\Gamma }_{M_{n-1}}+\Gamma E[R^c{R^c}^{\prime }\left]\right({\Gamma }_{M_{n-1}}-\Gamma )\hfill \\ & & +{\Gamma }_{M_{n-1}}{\overline{R}}_{M_{n-1}}^c{\left({\overline{R}}_{M_{n-1}}^c\right)}^{\prime }{\Gamma }_{M_{n-1}})I_G.\hfill \end{array}$$

As in G, Γ_Mn−1 − Γ and ${\displaystyle {\overline{R}}_{M_{n-1}}^c}$ converge uniformly to 0, E[B_n|T_n] − B converges uniformly to 0. Moreover there exists c₁ > 0 such that

$$\begin{array}{ccc}\hfill ||E\left[B_n|T_n\right]-B||I_G& \le & c_1(||{\Gamma }_{M_{n-1}}-\Gamma ||I_G+||{\overline{R}}_{M_{n-1}}^c||)\text{a}.\text{s}.\hfill \end{array}$$

By the proof of lemma 5: ${\displaystyle E[||{\overline{R}}_{M_{n-1}}^c||]\le {\left(\frac{Var\left[R\right]}{n-1}\right)}^{\frac{1}{2}}}$; then ${\displaystyle \sum _{n=1}^{\infty }\frac{1}{n}{\left(E\right[||{\overline{R}}_{M_{n-1}}^c||\left]\right)}^{\frac{1}{2}}<\infty }$.

By step 2: ${\displaystyle \sum _{n=1}^{\infty }\frac{1}{n}{\left(E\right[||{\Gamma }_{M_{n-1}}-\Gamma ||I_G\left]\right)}^{\frac{1}{2}}<\infty }$.

Then: ${\displaystyle \sum _{n=1}^{\infty }\frac{1}{n}{\left(E\right[||E\left[B_n|T_n\right]-B||I_G\left]\right)}^{\frac{1}{2}}<\infty }$.

As E[B_n|T_n] − B converges uniformly to 0 on G, we obtain:

$$\begin{array}{c}\hfill \sum _{n=1}^{\infty }\frac{1}{n}{\left(E\right[{||E\left[B_n|T_n\right]-B||}^2{I}_G\left]\right)}^{\frac{1}{2}}<\infty .\end{array}$$

Thus assumption H1b2 is verified in G.

Step 4: prove that assumption H1b3 is verified in G.

Denote R^c = R − E[R], $R_j^c=R_j-E\left[R\right]$, ${\overline{R}}_j^c={\overline{R}}_j-E\left[R\right]$. Consider the decomposition:

$$\begin{array}{ccc}\hfill B_n-B& =& {\Gamma }_{M_{n-1}}\frac{1}{m_n}\sum _{j\in I_n}(R_j^c-{\overline{R}}_{M_{n-1}}^c){(R_j^c-{\overline{R}}_{M_{n-1}}^c)}^{\prime }{\Gamma }_{M_{n-1}}\hfill \\ & & -\Gamma E\left[R^c{R^c}^{\prime }\right]\Gamma \hfill \\ & =& {\alpha }_n+{\beta }_n\hfill \end{array}$$

$$\begin{array}{ccc}\hfill \text{with}{\alpha }_n& =& {\Gamma }_{M_{n-1}}\frac{1}{m_n}\sum _{j\in I_n}(R_j^c{R_j^c}^{\prime }-{\overline{R}}_{M_{n-1}}^c{R_j^c}^{\prime }-R_j^c{\left({\overline{R}}_{M_{n-1}}^c\right)}^{\prime }+{\overline{R}}_{M_{n-1}}^c{\left({\overline{R}}_{M_{n-1}}^c\right)}^{\prime }){\Gamma }_{M_{n-1}}\hfill \\ & & -\Gamma \frac{1}{m_n}\sum _{j\in I_n}{R}_j^c{R_j^c}^{\prime }\Gamma \hfill \\ & =& ({\Gamma }_{M_{n-1}}-\Gamma )\left(\frac{1}{m_n}\sum _{j\in I_n}{R}_j^c{R_j^c}^{\prime }\right){\Gamma }_{M_{n-1}}+\Gamma \left(\frac{1}{m_n}\sum _{j\in I_n}{R}_j^c{R_j^c}^{\prime }\right)({\Gamma }_{M_{n-1}}-\Gamma )\hfill \\ & & -{\Gamma }_{M_{n-1}}{\overline{R}}_{M_{n-1}}^c\frac{1}{m_n}\sum _{j\in I_n}{R_j^c}^{\prime }{\Gamma }_{M_{n-1}}-{\Gamma }_{M_{n-1}}\frac{1}{m_n}\sum _{j\in I_n}{R}_j^c{\left({\overline{R}}_{M_{n-1}}^c\right)}^{\prime }{\Gamma }_{M_{n-1}}\hfill \\ & & +{\Gamma }_{M_{n-1}}{\overline{R}}_{M_{n-1}}^c{\left({\overline{R}}_{M_{n-1}}^c\right)}^{\prime }{\Gamma }_{M_{n-1}},\hfill \\ \hfill {\beta }_n& =& \Gamma (\frac{1}{m_n}\sum _{j\in I_n}{R}_j^c{R_j^c}^{\prime }-E[R^c{R^c}^{\prime }\left]\right)\Gamma .\hfill \end{array}$$

Let η > 0.

$$\begin{array}{ccc}\hfill E\left[{||B_n-B||}^2{I}_G|T_n\right]& =& E\left[{||{\alpha }_n+{\beta }_n||}^2{I}_G|T_n\right]\hfill \\ & \le & (1+\frac{1}{\eta })E\left[{||{\alpha }_n||}^2{I}_G|T_n\right]\hfill \\ & & +(1+\eta )E\left[{||{\beta }_n||}^2{I}_G|T_n\right]\text{a}.\text{s}.\hfill \end{array}$$

As random variables $R_j^c$, j ∈ I_n, are independent of T_n, as Γ_Mn−1 and ${\overline{R}}_{M_{n-1}}^c$ are T_n-measurable and converge uniformly respectively to Γ and 0 on G, E[‖α_n‖² I_G|T_n] converges uniformly to 0. Then, for δ > 0, there exists N₁ such that for n > N₁, E[‖α_n‖² I_G|T_n] ≤ δ a.s.

Moreover, denoting U = R^cR^c′ and $U_j=R_j^c{R_j^c}^{\prime }$, we have, as the random variables U_j form an i.i.d. sample of U:

$$\begin{array}{ccc}\hfill E\left[{||{\beta }_n||}^2|T_n\right]& =& E\left[{||\frac{1}{m_n}\sum _{j\in I_n}\Gamma (U_j-E[U\left]\right)\Gamma ||}^2|T_n\right]\hfill \\ & \le & E\left[{||\Gamma (U-E[U\left]\right)\Gamma ||}^2\right]=E\left[{||\Gamma U\Gamma -E[\Gamma U\Gamma ]||}^2\right]=b_1\text{a}.\text{s}.\hfill \end{array}$$

Then:

$$\begin{array}{ccc}\hfill E\left[{||B_n-B||}^2{I}_G|T_n\right]& \le & (1+\frac{1}{\eta })\delta +(1+\eta )b_1=b\text{a}.\text{s}.\hfill \end{array}$$

Thus assumption H1b3 is verified in G.

As ${\overline{S}}_{M_{n-1}}-E\left[S\right]⟶0$ and ${\Gamma }_{M_{n-1}}^1-{\Gamma }^1⟶0$ almost surely, it can be proved likewise that there exist a set H of probability greater than ${\displaystyle 1-\frac{{\epsilon }_1}{2}}$ and d > 0 such that E[‖F_n − F‖² I_H|T_n] ≤ d a.s. Thus assumption H2b2 is verified in H.

Step 5: conclusion.

As ${\displaystyle a<min(\frac{1}{{\lambda }_{max}},\frac{2\lambda }{{\lambda }^2+b_1})}$, ${\displaystyle b_1<\frac{2\lambda }{a}-{\lambda }^2}$.

Choose ${\displaystyle 0<\eta <\frac{\frac{2\lambda }{a}-{\lambda }^2}{b_1}-1}$ and ${\displaystyle 0<\delta <\frac{\frac{2\lambda }{a}-{\lambda }^2-(1+\eta )b_1}{1+\frac{1}{\eta }}}$ such that

$$\begin{array}{c}\hfill b=(1+\frac{1}{\eta })\delta +(1+\eta )b_1<\frac{2\lambda }{a}-{\lambda }^2⟺a<\frac{2\lambda }{{\lambda }^2+b}.\end{array}$$

Thus assumption H3b is verified.

Applying theorem 7 implies that Y_n converges to θ almost surely in H ∩ G.

Therefore P(Y_n ⟶ θ) ≥ P(H ∩ G) > 1 − ε₁.

This is in contradiction with $P(Y_n\nrightarrow \theta )={\epsilon }_1$. Thus Y_n converges to θ a.s. ∎

4 Convergence of a process with a variable or constant step-size and use of all observations until the current step

In this section, the convergence of the process (X_n, n ≥ 1) in ${\mathbb{R}}^{p\times q}$ recursively defined by

$$\begin{array}{c}\hfill X_{n+1}=X_n-a_n(B_n{X}_n-F_n)\end{array}$$

and its application to sequential linear regression are studied.

4.1 Theorem

Make the following assumptions

(H1c) There exists a positive definite symmetrical matrix B such that B_n ⟶ B a.s.

(H2c) There exists a matrix F such that F_n ⟶ F a.s.

(H3c) λ_max denoting the largest eigenvalue of B,

${\displaystyle (a_n=a<\frac{1}{{\lambda }_{max}})}$ or ${\displaystyle (a_n⟶0,\sum _{n=1}^{\infty }{a}_n=\infty )}$.

Theorem 9 Suppose H1c, H2c and H3c hold. Then X_n converges to B⁻¹F a.s.

Proof of Theorem 9.

Denote θ = B⁻¹F, $X_n^1=X_n-\theta$, Z_n = (B_n − B)θ − (F_n − F). Then:

$$\begin{array}{c}\hfill X_{n+1}^1=(I-a_n{B}_n)X_n^1-a_n{Z}_n.\end{array}$$

Let ω be fixed belonging to the intersection of the convergence sets {B_n ⟶ B} and {F_n ⟶ F}. The writing of ω is omitted in the following.

Denote ‖A‖ the spectral norm of a matrix A and λ the smallest eigenvalue of B.

In the case a_n depending on n, as a_n ⟶ 0, we can suppose without loss of generality ${\displaystyle a_n<\frac{1}{{\lambda }_{max}}}$ for all n. Then all the eigenvalues of I − a_nB are positive and ‖I − a_nB‖ = 1 − a_nλ.

Let 0 < ε < λ. As B_n − B ⟶ 0, we obtain for n sufficiently large:

$$\begin{array}{ccc}\hfill ||I-a_n{B}_n||& \le & ||I-a_{n}B||+a_n||B_n-B||\hfill \\ & \le & 1-a_n\lambda +a_n\epsilon \text{,}\text{with}{a}_n<\frac{1}{\lambda -\epsilon }\hfill \\ \hfill ||X_{n+1}^1||& \le & (1-a_n(\lambda -\epsilon \left)\right)||X_n^1||+a_n||Z_n||.\hfill \end{array}$$

As Z_n ⟶ 0, applying lemma 6 yields $||X_n^1||⟶0$.

Therefore X_n ⟶ B⁻¹F a.s. ∎

4.2 Application to linear regression with online standardized data

Let (m_n, n ≥ 1) be a sequence of integers. Denote ${\displaystyle M_n=\sum _{k=1}^n{m}_k}$ for n ≥ 1, M₀ = 0 and I_n = {M_{n − 1} + 1,…,M_n}.

Define

$$\begin{array}{ccc}\hfill B_n& =& {\Gamma }_{M_n}(\frac{1}{M_n}\sum _{i=1}^n\sum _{j\in I_i}{R}_j{R}_j^{\prime }-{\overline{R}}_{M_n}{\overline{R}}_{M_n}^{\prime }){\Gamma }_{M_n},\hfill \\ \hfill F_n& =& {\Gamma }_{M_n}(\frac{1}{M_n}\sum _{i=1}^n\sum _{j\in I_i}{R}_j{S}_j^{\prime }-{\overline{R}}_{M_n}{\overline{S}}_{M_n}^{\prime }){\Gamma }_{M_n}^1.\hfill \end{array}$$

As ((R_n, S_n), n ≥ 1) is an i.i.d. sample of (R, S), assumptions H1c and H2c are obviously verified with B = ΓE[(R − E[R])(R − E[R])′]Γ and F = ΓE[(R − E[R])(S − E[S])′]Γ¹. Then:

Corollary 10 Suppose there is no affine relation between the components of R and the moments of order 4 of (R, S) exist. Suppose H3c holds. Then X_n converges to B⁻¹F a.s.

Remark 3 B is the correlation matrix of R of dimension p. Then λ_max < Trace(B) = p. In the case of a constant step-size a, it suffices to take ${\displaystyle a\le \frac{1}{p}}$ to verify H3c.

Remark 4 In the definition of B_n and F_n, the R_j and the S_j are not directly pseudo-centered with respect to ${\overline{R}}_{M_n}$ and ${\overline{S}}_{M_n}$ respectively. Another equivalent definition of B_n and F_n can be used. It consists of replacing R_j by R_j − m, ${\overline{R}}_{M_n}$ by ${\overline{R}}_{M_n}-m$, S_j by S_j − m, ${\overline{S}}_{M_n}$ by ${\overline{S}}_{M_n}-m_1$, m and m₁ being respectively an estimation of E[R] and E[S] computed in a preliminary phase with a small number of observations. For example, at step n, ${\displaystyle \sum _{j\in I_n}{\Gamma }_{M_n}}\left(R_j-m\right){\left({\Gamma }_{M_n}\left(R_j-m\right)\right)}^{\prime }$ is computed instead of ${\displaystyle \sum _{j\in I_n}{\Gamma }_{M_n}}{R}_j{\left({\Gamma }_{M_n}{R}_j\right)}^{\prime }$. This limits the risk of numerical explosion.

This process was tested on several datasets and some results are given in section 5 (with a variable step-size: process S13 for m_n = 1 and S14 for m_n = 10; with a constant step-size: process S31 for m_n = 1 and S32 for m_n = 10).

5 Experiments

The three previously-defined processes of stochastic approximation with online standardized data were compared with the classical stochastic approximation and averaged stochastic approximation (or averaged stochastic gradient descent) processes with constant step-size (denoted ASGD) studied in [5] and [6]. A description of the methods along with abbreviations and parameters used is given in Table 1.

Table 1. Description of the methods.

Method type	Abbreviation	Type of data	Number of observations used at each step of the process	Use of all the observations until the current step	Step-size	Use of the averaged process
Classic	C1	Raw data	1	No	variable	No
	C2		10
	C3		1	Yes
	C4		10
ASGD	A1		1	No	constant	Yes
	A2		1
Standardization 1	S11	Online standardized data	1	No	variable	No
	S12		10
	S13		1	Yes
	S14		10
Standardization 2	S21		1	No	constant	Yes
	S22		10
Standardization 3	S31		1	Yes		No
	S32		10

With the variable S set at dimension 1, 11 datasets were considered, some of which are available in free access on the Internet, while others were derived from the EPHESUS study [15]: 6 in regression (continuous dependent variable) and 5 in linear discriminant analysis (binary dependent variable). All datasets used in our experiments are presented in detail in Table 2, along with their download links. An a priori selection of variables was performed on each dataset using a stepwise procedure based on Fisher’s test with p-to-enter and p-to-remove fixed at 5 percent.

Table 2. Datasets used in our experiments.

Dataset name	N	pa	p	Type of dependent variable	T2	Number of outliers
CADATA	20640	8	8	Continuous	1.6x106	122	www.dcc.fc.up.pt/∼ltorgo/Regression/DataSets.html
AILERONS	7154	40	9	Continuous	247.1	0	www.dcc.fc.up.pt/∼ltorgo/Regression/DataSets.html
ELEVATORS	8752	18	10	Continuous	7.7x104	0	www.dcc.fc.up.pt/∼ltorgo/Regression/DataSets.html
POLY	5000	48	12	Continuous	4.1x104	0	www.dcc.fc.up.pt/∼ltorgo/Regression/DataSets.html
eGFR	21382	31	15	Continuous	2.9x104	0	derived from EPHESUS study [15]
HEMG	21382	31	17	Continuous	6.0x104	0	derived from EPHESUS study [15]
QUANTUM	50000	78	14	Binary	22.5	1068	www.osmot.cs.cornell.edu/kddcup
ADULT	45222	97	95	Binary	4.7x1010	20	www.cs.toronto.edu/∼delve/data/datasets.html
RINGNORM	7400	20	20	Binary	52.8	0	www.cs.toronto.edu/∼delve/data/datasets.html
TWONORM	7400	20	20	Binary	24.9	0	www.cs.toronto.edu/∼delve/data/datasets.html
HOSPHF30D	21382	32	15	Binary	8.1x105	0	derived from EPHESUS study [15]

N denotes the size of global sample, p_a the number of parameters available, p the number of parameters selected and T² the trace of E[RR′]. Outlier is defined as an observation whose the L2 norm is greater than five times the average norm.

Let D = {(r_i, s_i), i = 1, 2,…,N} be the set of data in ${\mathbb{R}}^p\times \mathbb{R}$ and assuming that it represents the set of realizations of a random vector (R, S) uniformly distributed in D, then minimizing E[(S − θ′ R − η)²] is equivalent to minimizing ${\displaystyle \frac{1}{N}\sum _{i=1}^N{(s_i-{\theta }^{\prime }{r}_i-\eta )}^2}$. One element of D (or several according to the process) is randomly drawn at each step to iterate the process.

To compare the methods, two different studies were performed: one by setting the total number of observations used, the other by setting the computing time.

The choice of step-size, the initialization for each method and the convergence criterion used are respectively presented and commented below.

Choice of step-size

In all methods of stochastic approximation, a suitable choice of step-size is often crucial for obtaining good performance of the process. If the step-size is too small, the convergence rate will be slower. Conversely, if the step-size is too large, a numerical explosion phenomenon may occur during the first iterations.

For the processes with a variable step-size (processes C1 to C4 and S11 to S14), we chose to use a_n of the following type:

$$\begin{array}{ccc}\hfill a_n& =& \frac{c_{\gamma }}{{(b+n)}^{\alpha }}.\hfill \end{array}$$

The constant ${\displaystyle \alpha =\frac{2}{3}}$ was fixed, as suggested by Xu [16] in the case of stochastic approximation in linear regression, and b = 1. The results obtained for the choice ${\displaystyle c_{\gamma }=\frac{1}{p}}$ are presented although the latter does not correspond to the best choice for a classical method.

For the ASGD method (A1, A2), two different constant step-sizes a as used in [6] were tested: ${\displaystyle a=\frac{1}{T^2}}$ and ${\displaystyle a=\frac{1}{2T^2}}$, T² denoting the trace of E[RR′]. Note that this choice of constant step-size assumes knowing a priori the dataset and is not suitable for a data stream.

For the methods with standardization and a constant step-size a (S21, S22, S31, S32), ${\displaystyle a=\frac{1}{p}}$ was chosen since the matrix E[RR′] is thus the correlation matrix of R, whose trace is equal to p, such that this choice corresponds to that of [6].

Initialization of processes

All processes (X_n) were initialized by $X_1=\underset{-}{0}$, the null vector. For the processes with standardization, a small number of observations (n = 1000) were taken into account in order to calculate an initial estimate of the means and standard deviations.

Convergence criterion

The “theoretical vector” θ¹ is assigned as that obtained by the least square method in D such that ${\theta }^{1^{\prime }}=\left(\begin{array}{cc}{\theta }^{\prime }& \eta \end{array}\right)$. Let ${\Theta }_{n+1}^1$ be the estimator of θ¹ obtained by stochastic approximation after n iterations.

In the case of a process (X_n) with standardized data, which yields an estimation of the vector denoted θ_c in section 1 as θ = Γθ_c(Γ¹)⁻¹ and η = E[S] − θ′ E[R], we can define:

$$\begin{array}{ccc}\hfill {\Theta }_{n+1}^1{}^{\prime }& =& \left(\begin{array}{cc}{\Theta }_{n+1}^{\prime }& H_{n+1}\end{array}\right)\hfill \\ \hfill \text{with}{\Theta }_{n+1}& =& {\Gamma }_{M_n}{X}_{n+1}{\left({\Gamma }_{M_n}^1\right)}^{-1}\hfill \\ \hfill H_{n+1}& =& {\overline{S}}_{M_n}-{\Theta }_{n+1}^{\prime }{\overline{R}}_{M_n}.\hfill \end{array}$$

To judge the convergence of the method, the cosine of the angle formed by exact θ¹ and its estimation ${\theta }_{n+1}^1$ was used as criterion,

$$\begin{array}{c}\hfill \text{cos}({\theta }^1,{\theta }_{n+1}^1)=\frac{{{\theta }^1}^{\prime }{\theta }_{n+1}^1}{{||{\theta }^1||}_2{||{\theta }_{n+1}^1||}_2}.\end{array}$$

Other criteria, such as ${\displaystyle \frac{{||{\theta }^1-{\theta }_{n+1}^1||}_2}{{||{\theta }^1||}_2}}$ or ${\displaystyle \frac{f\left({\theta }_{n+1}^1\right)-f\left({\theta }^1\right)}{f\left({\theta }^1\right)}}$, f being the loss function, were also tested, although the results are not presented in this article.

5.1 Study for a fixed total number of observations used

For all N observations used by the algorithm (N being the size of D) up to a maximum of 100N observations, the criterion value associated with each method and for each dataset was recorded. The results obtained after using 10N observations are provided in Table 3.

Table 3. Results after using 10N observations.

	CADATA	AILERONS	ELEVATORS	POLY	EGFR	HEMG	QUANTUM	ADULT	RINGNORM	TWONORM	HOSPHF30D	Mean rank
C1	Expl.	-0.0385	Expl.	Expl.	Expl.	Expl.	0.9252	Expl.	0.9998	1.0000	Expl.	11.6
C2	Expl.	0.0680	Expl.	Expl.	Expl.	Expl.	0.8551	Expl.	0.9976	0.9996	Expl.	12.2
C3	Expl.	0.0223	Expl.	Expl.	Expl.	Expl.	0.9262	Expl.	0.9999	1.0000	Expl.	9.9
C4	Expl.	-0.0100	Expl.	Expl.	Expl.	Expl.	0.8575	Expl.	0.9981	0.9996	Expl.	12.3
A1	-0.0013	0.4174	0.0005	0.3361	0.2786	0.2005	Expl.	0.0027	0.9998	1.0000	0.0264	9.2
A2	0.0039	0.2526	0.0004	0.1875	0.2375	0.1846	0.0000	0.0022	0.9999	1.0000	0.2047	8.8
S11	1.0000	0.9516	0.9298	1.0000	1.0000	0.9996	0.9999	0.7599	0.9999	1.0000	0.7723	5.2
S12	0.9999	0.9579	0.9311	1.0000	0.9999	0.9994	0.9991	0.6842	0.9999	1.0000	0.4566	6.1
S13	1.0000	0.9802	0.9306	1.0000	1.0000	0.9998	1.0000	0.7142	0.9999	1.0000	0.7754	3.7
S14	0.9999	0.9732	0.9303	1.0000	0.9999	0.9994	0.9991	0.6225	0.9998	1.0000	0.4551	6.9
S21	0.9993	0.6261	0.9935	Expl.	Expl.	Expl.	Expl.	Expl.	0.9998	1.0000	Expl.	10.5
S22	1.0000	0.9977	0.9900	1.0000	1.0000	0.9989	0.9999	-0.0094	0.9999	1.0000	0.9454	4.1
S31	1.0000	0.9988	0.9999	1.0000	1.0000	0.9992	0.9999	0.9907	0.9999	1.0000	0.9788	2.3
S32	1.0000	0.9991	0.9998	1.0000	1.0000	0.9992	0.9999	0.9867	0.9999	1.0000	0.9806	2.2

Expl. means numerical explosion.

As can be seen in Table 3, a numerical explosion occured in most datasets using the classical methods with raw data and a variable step-size (C1 to C4). As noted in Table 2, these datasets had a high T² = Tr(E[RR′]). Corresponding methods S11 to S14 using the same variable step-size but with online standardized data quickly converged in most cases. However classical methods with raw data can yield good results for a suitable choice of step-size, as demonstrated by the results obtained for POLY dataset in Fig 1. The numerical explosion can arise from a too high step-size when n is small. This phenomenon can be avoided if the step-size is reduced, although if the latter is too small, the convergence rate will be slowed. Hence, the right balance must be found between step-size and convergence rate. Furthermore, the choice of this step-size generally depends on the dataset which is not known a priori in the case of a data stream. In conclusion, methods with standardized data appear to be more robust to the choice of step-size.

Fig 1. Results obtained for dataset POLY using 10N and 100N observations: A/ process C1 with variable step-size ${\displaystyle a_n=\frac{1}{{(b+n)}^{\frac{2}{3}}}}$ by varying b, B/ process C1 with variable step-size ${\displaystyle a_n=\frac{\frac{1}{p}}{{(b+n)}^{\frac{2}{3}}}}$ by varying b, C/ process S21 by varying constant step-size a.

The ASGD method (A1 with constant step-size ${\displaystyle a=\frac{1}{T^2}}$ and A2 with ${\displaystyle a=\frac{1}{2T^2}}$) did not yield good results except for the RINGNORM and TWONORM datasets which were obtained by simulation (note that all methods functioned very well for these two datasets). Of note, A1 exploded for the QUANTUM dataset containing 1068 observations (2.1%) whose L2 norm was fivefold greater than the average norm (Table 2). The corresponding method S21 with online standardized data yielded several numerical explosions with the ${\displaystyle a=\frac{1}{p}}$ step-size, however these explosions disappeared when using a smaller step-size (see Fig 1). Of note, it is assumed in corollary 8 that ${\displaystyle 0<a<min(\frac{1}{{\lambda }_{max}},\frac{2\lambda }{{\lambda }^2+b_1})}$; in the case of ${\displaystyle a=\frac{1}{p}}$, only ${\displaystyle a<\frac{1}{{\lambda }_{max}}}$ is certain.

Finally, for methods S31 and S32 with standardized data, the use of all observations until the current step and the very simple choice of the constant step-size ${\displaystyle a=\frac{1}{p}}$ uniformly yielded good results.

Thereafter, for each fixed number of observations used and for each dataset, the 14 methods ranging from the best (the highest cosine) to the worst (the lowest cosine) were ranked by assigning each of the latter a rank from 1 to 14 respectively, after which the mean rank in all 11 datasets was calculated for each method. A total of 100 mean rank values were calculated for a number of observations used varying from N to 100N. The graph depicting the change in mean rank based on the number of observations used and the boxplot of the mean rank are shown in Fig 2.

Fig 2. Results for a fixed total number of observations used: A/ change in the mean rank based on the number of observations used, B/ boxplot of the mean rank by method.

Overall, for these 11 datasets, a method with standardized data, a constant step-size and use of all observations until the current step (S31, S32) represented the best method when the total number of observations used was fixed.

5.2 Study for a fixed processing time

For every second up to a maximum of 2 minutes, the criterion value associated to each dataset was recorded. The results obtained after a processing time of 1 minute are provided in Table 4.

Table 4. Results obtained after a fixed time of 1 minute.

	CADATA	AILERONS	ELEVATORS	POLY	EGFR	HEMG	QUANTUM	ADULT	RINGNORM	TWONORM	HOSPHF30D	Mean rank
C1	Expl.	-0.2486	Expl.	Expl.	Expl.	Expl.	0.9561	Expl.	1.0000	1.0000	Expl.	12.2
C2	Expl.	0.7719	Expl.	Expl.	Expl.	Expl.	0.9519	Expl.	1.0000	1.0000	Expl.	9.9
C3	Expl.	0.4206	Expl.	Expl.	Expl.	Expl.	0.9547	Expl.	1.0000	1.0000	Expl.	10.6
C4	Expl.	0.0504	Expl.	Expl.	Expl.	Expl.	0.9439	Expl.	1.0000	1.0000	Expl.	10.1
A1	-0.0067	0.8323	0.0022	0.9974	0.7049	0.2964	Expl.	0.0036	1.0000	1.0000	Expl.	9.0
A2	0.0131	0.8269	0.0015	0.9893	0.5100	0.2648	Expl.	0.0027	1.0000	1.0000	0.2521	8.6
S11	1.0000	0.9858	0.9305	1.0000	1.0000	1.0000	1.0000	0.6786	1.0000	1.0000	0.9686	5.8
S12	1.0000	0.9767	0.9276	1.0000	1.0000	0.9999	1.0000	0.6644	1.0000	1.0000	0.9112	5.8
S13	1.0000	0.9814	0.9299	1.0000	1.0000	0.9999	1.0000	0.4538	1.0000	1.0000	0.9329	6.1
S14	1.0000	0.9760	0.9274	1.0000	1.0000	1.0000	0.9999	0.5932	1.0000	1.0000	0.8801	6.1
S21	-0.9998	0.2424	0.6665	Expl.	Expl.	Expl.	Expl.	0.0000	1.0000	1.0000	Expl.	11.5
S22	1.0000	0.9999	1.0000	1.0000	1.0000	1.0000	1.0000	-0.0159	1.0000	1.0000	0.9995	3.1
S31	1.0000	0.9995	1.0000	1.0000	1.0000	0.9999	1.0000	0.9533	1.0000	1.0000	0.9997	4.5
S32	1.0000	0.9999	1.0000	1.0000	1.0000	1.0000	1.0000	0.9820	1.0000	1.0000	0.9999	1.5

Expl. means numerical explosion.

The same conclusions can be drawn as those described in section 5.1 for the classical methods and the ASGD method. The methods with online standardized data typically faired better.

As in the previous study in section 5.1, the 14 methods were ranked from the best to the worst on the basis of the mean rank for a fixed processing time. The graph depicting the change in mean rank based on the processing time varying from 1 second to 2 minutes as well as the boxplot of the mean rank are shown in Fig 3.

Fig 3. Results for a fixed processing time: A/ change in the mean rank based on the processing time, B/ boxplot of the mean rank by method.

As can be seen, these methods with online standardized data using more than one observation per step yielded the best results (S32, S22). One explanation may be that the total number of observations used in a fixed processing time is higher when several observations are used per step rather than one observation per step. This can be verified in Table 5 in which the total number of observations used per second for each method and for each dataset during a processing time of 2 minutes is given. Of note, the number of observations used per second in a process with standardized data and one observation per step (S11, S13, S21, S31) was found to be generally lower than in a process with raw data and one observation per step (C1, C3, A1, A2), since a method with standardization requires the recursive estimation of means and variances at each step.

Table 5. Number of observations used after 2 minutes (expressed in number of observations per second).

	CADATA	AILERONS	ELEVATORS	POLY	EGFR	HEMG	QUANTUM	ADULT	RINGNORM	TWONORM	HOSPHF30D
C1	19843	33170	17133	14300	10979	9243	33021	476	31843	31677	10922
C2	166473	291558	159134	134249	104152	89485	281384	4565	262847	261881	102563
C3	17206	28985	16036	13449	10383	8878	28707	462	28123	28472	10404
C4	132088	194031	125880	106259	87844	76128	184386	4252	171711	166878	86895
A1	33622	35388	36540	35800	35280	34494	11815	15390	34898	34216	14049
A2	33317	32807	36271	35628	35314	34454	15439	16349	34401	34205	34890
S11	17174	17133	17166	16783	15648	14764	16296	1122	14067	13836	14334
S12	45717	47209	45893	43470	39937	37376	40943	4554	34799	34507	36389
S13	12062	12731	11888	12057	11211	10369	11466	620	9687	9526	10137
S14	43674	46080	43068	42123	38350	35338	39170	4512	33594	31333	32701
S21	15396	17997	16772	10265	8404	7238	9166	996	13942	13274	7672
S22	47156	47865	46318	43899	40325	37467	41320	4577	34478	31758	37418
S31	12495	12859	12775	12350	11495	10619	11608	621	9890	9694	10863
S32	44827	47035	45123	42398	38932	36288	39362	4532	33435	33385	35556

Of note, for the ADULT dataset with a large number of parameters selected (95), the only method yielding sufficiently adequate results after a processing time of one minute was S32, and methods S31 and S32 when 10N observations were used.

6 Conclusion

In the present study, three processes with online standardized data were defined and for which their a.s. convergence was proven.

A stochastic approximation method with standardized data appears to be advantageous compared to a method with raw data. First, it is easier to choose the step-size. For processes S31 and S32 for example, the definition of a constant step-size only requires knowing the number of parameters p. Secondly, the standardization usually allows avoiding the phenomenon of numerical explosion often obtained in the examples given with a classical method.

The use of all observations until the current step can reduce the influence of outliers and increase the convergence rate of a process. Moreover, this approach is particularly adapted to the case of a data stream.

Finally, among all processes tested on 11 different datasets (linear regression or linear discriminant analysis), the best was a method using standardization, a constant step-size equal to ${\displaystyle \frac{1}{p}}$ and all observations until the current step, and the use of several new observations at each step improved the convergence rate.

Data Availability

All datasets used in our experiments except those derived from EPHESUS study are available online and links to download these data appear in Table 2 of our article. Due to legal restrictions, data from EPHESUS study are only available upon request. Interested researchers may request access to data upon approval from the EPHESUS Executive Steering Committee of the study. This committee can be reached through Pr Faiez Zannad (f.zannad@chu-nancy.fr) who is member of this board.

Funding Statement

This work is supported by a public grant overseen by the French National Research Agency (ANR) as part of the second “Investissements d’Avenir” programme (reference: ANR-15-RHU-0004). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.