Estimating beta-mixing coefficients

Abstract

The literature on statistical learning for time series assumes the asymptotic independence or “mixing” of the data-generating process. These mixing assumptions are never tested, and there are no methods for estimating mixing rates from data. We give an estimator for the beta-mixing rate based on a single stationary sample path and show it is L1-risk consistent.

1 Introduction

Relaxing the assumption of independence is an active area of research in the statistics and machine learning literature. For time series, independence is replaced by the asymptotic independence of events far apart in time, or “mixing”. Mixing conditions make the dependence of the future on the past explicit by quantifying the decay in dependence as the future moves farther from the past. There are many definitions of mixing of varying strength with matching dependence coefficients (see [9, 7, 4] for reviews), but most of the results in the learning literature focus on β-mixing or absolute regularity. Roughly speaking (see Definition 2.1 below for a precise statement), the β-mixing coefficient at lag a is the total variation distance between the actual joint distribution of events separated by a time steps and the product of their marginal distributions, i.e., the L¹ distance from independence.

Numerous results in the statistical machine learning literature rely on knowledge of the β-mixing coefficient. As Vidyasagar [25, p. 41] notes, β-mixing is “just right” for the extension of IID results to dependent data, and so recent work has consistently focused on it. Meir [15] derives generalization error bounds for nonparametric methods based on model selection via structural risk minimization. Baraud et al. [1] study the finite sample risk performance of penalized least squares regression estimators under β-mixing. Lozano et al. [13] examine regularized boosting algorithms under absolute regularity and prove consistency. Karandikar and Vidyasagar [12] consider “probably approximately correct” learning algorithms, proving that PAC algorithms for IID inputs remain PAC with β-mixing inputs under some mild conditions. Ralaivola et al. [20] derive PAC bounds for ranking statistics and classifiers using a decomposition of the dependency graph. Finally, Mohri and Rostamizadeh [16] derive stability bounds for β-mixing inputs, generalizing existing stability results for IID data.

All these results assume not just β-mixing, but known mixing coefficients. In particular, the risk bounds in [15, 16] and [20] are incalculable without knowledge of the rates. This knowledge is never available. Unless researchers are willing to assume specific values for a sequence of β-mixing coefficients, the results mentioned in the previous paragraph are generally useless when confronted with data. To illustrate this deficiency, consider Theorem 18 of [16]:

Theorem 1.1

(Briey). Assume a learning algorithm is λ-stable.¹ Then, for any sample of size n drawn from a stationary β-mixing distribution, and ε > 0

$$\mathbb{P}(|R-\hat{R}|>\epsilon )\le \mathrm{\Gamma }(n,\lambda ,\epsilon ,a,b)+\beta (a)({\mu }_n-1)$$

where n = (a + b)μ_n, Γ has a particular functional form, and R − R̂ is the difference between the true risk and the empirical risk.

Ideally, one could use this result for model selection or to control the size of the generalization error of competing prediction algorithms (support vector machines, support vector regression, and kernel ridge regression are a few of the many algorithms known to satisfy λ-stability). However the bound depends explicitly on the mixing coefficient β(a). To make matters worse, there are no methods for estimating the β-mixing coefficients. According to Meir [15, p. 7], “there is no efficient practical approach known at this stage for estimation of mixing parameters.” We begin to rectify this problem by deriving the first method for estimating these coefficients. We prove that our estimator is consistent for arbitrary β-mixing processes. In addition, we derive rates of convergence for Markov approximations to these processes.

Application of statistical learning results to β-mixing data is highly desirable in applied work. Many common time series models are known to be β-mixing, and the rates of decay are known given the true parameters of the process. Among the processes for which such knowledge is available are ARMA models [17], GARCH models [5], and certain Markov processes — see [9] for an overview of such results. To our knowledge, only Nobel [18] approaches a solution to the problem of estimating mixing rates by giving a method to distinguish between different polynomial mixing rate regimes through hypothesis testing.

We present the first method for estimating the β-mixing coefficients for stationary time series data. Section 2 defines the β-mixing coefficient and states our main results on convergence rates and consistency for our estimator. Section 3 gives an intermediate result on the L¹ convergence of the histogram estimator with β-mixing inputs. Section 4 proves the main results from §2. Section 5 concludes and lays out some avenues for future research.

2 Estimation of β-mixing

In this section, we present one of many equivalent definitions of absolute regularity and state our main results, deferring proof to §4.

To fix notation, let $\mathbf{X}={\{X_t\}}_{t=-\infty }^{\infty }$ be a sequence of random variables where each X_t is a measurable function from a probability space (Ω, ℱ, ℙ) into a measurable space 𝒳. A block of this random sequence will be given by ${\mathbf{X}}_i^j\equiv {\{X_t\}}_{t=i}^j$ where i and j are integers, and may be infinite. We use similar notation for the sigma fields generated by these blocks and their joint distributions. In particular, ${\sigma }_i^j$ will denote the sigma field generated by ${\mathbf{X}}_i^j$, and the joint distribution of ${\mathbf{X}}_i^j$ will be denoted ${\mathbb{P}}_i^j$.

2.1 Definitions

There are many equivalent definitions of β-mixing (see for instance [9], or [4] as well as Meir [15] or Yu [28]), however the most intuitive is that given in Doukhan [9].

Definition 2.1

(β-mixing). For each positive integer a, the coefficient of absolute regularity, or β-mixing coefficient, β(a), is

$$\beta (a)\equiv \underset{t}{\text{sup}}{\Vert {\mathbb{P}}_{-\infty }^t\otimes {\mathbb{P}}_{t+a}^{\infty }-{\mathbb{P}}_{t,a}\Vert }_{TV}$$ (1)

where ‖·‖_TV is the total variation norm, and ℙ_t,a is the joint distribution of (${\mathbf{X}}_{-\infty }^t,{\mathbf{X}}_{t+a}^{\infty }$). A stochastic process is said to be absolutely regular, or β-mixing, if β(a) → 0 as a → ∞.

Loosely speaking, Definition 2.1 says that the coefficient β(a) measures the total variation distance between the joint distribution of random variables separated by a time units and a distribution under which random variables separated by a time units are independent. The supremum over t is unnecessary for stationary random processes X which is the only case we consider here.

Definition 2.2

(Stationarity). A sequence of random variables X is stationary when all its finite-dimensional distributions are invariant over time: for all t and all non-negative integers i and j, the random vectors ${\mathbf{X}}_t^{t+i}$ and ${\mathbf{X}}_{t+j}^{t+i+j}$ have the same distribution.

Our main result requires the method of blocking used by Yu [27, 28]. The purpose is to transform a sequence of dependent variables into subsequence of nearly IID ones. Consider a sample ${\mathbf{X}}_1^n$ from a stationary β-mixing sequence with density f. Let m_n and μ_n be non-negative integers such that 2m_nμ_n = n. Now divide ${\mathbf{X}}_1^n$ into 2μ_n blocks, each of length m_n. Identify the blocks as follows:

$$U_j=\{X_i:2(j-1)m_n+1\le i\le (2j-1)m_n\},$$

$$V_j=\{X_i:(2j-1)m_n+1\le i\le 2j{m}_n\}.$$

Let U be the entire sequence of odd blocks U_j, and let V be the sequence of even blocks V_j. Finally, let U′ be a sequence of blocks which are independent of ${\mathbf{X}}_1^n$ but such that each block has the same distribution as a block from the original sequence:

$$U_j^{\prime }\stackrel{D}{=}{U}_j\stackrel{D}{=}{U}_1.$$ (2)

The blocks U′ are now an IID block sequence, so standard results apply. (See [28] for a more rigorous analysis of blocking.) With this structure, we can state our main result.

2.2 Results

Our main result emerges in two stages. First, we recognize that the distribution of a finite sample depends only on finite-dimensional distributions. This leads to an estimator of a finite-dimensional version of β(a). Next, we let the finite-dimension increase to infinity with the size of the observed sample.

For positive integers t, d, and a, define

$${\beta }^d(a)\equiv {\Vert {\mathbb{P}}_{t-d+1}^t\otimes {\mathbb{P}}_{t+a}^{t+a+d-1}-{\mathbb{P}}_{t,a,d}\Vert }_{TV},$$ (3)

where ℙt,a,d is the joint distribution of (${\mathbf{X}}_{t-d+1}^t,{\mathbf{X}}_{t+a}^{t+a+d-1}$). Also, let f̂^d be the d-dimensional histogram estimator of the joint density of d consecutive observations, and let ${\hat{f}}_a^{2d}$ be the 2d-dimensional histogram estimator of the joint density of two sets of d consecutive observations separated by a time points.

We construct an estimator of β^d(a) based on these two histograms.² Define

$${\hat{\beta }}^d(a)\equiv \frac{1}{2}\int |{\hat{f}}_a^{2d}-{\hat{f}}^d\otimes {\hat{f}}^d|$$ (4)

We show that, by allowing d = d_n to grow with n, this estimator will converge on β(a). This can be seen most clearly by bounding the ℓ¹-risk of the estimator with its estimation and approximation errors:

$$|{\hat{\beta }}^d(a)-\beta (a)|\le |{\hat{\beta }}^d(a)-{\beta }^d(a)|+|{\beta }^d(a)-\beta (a)|.$$

The first term is the error of estimating β^d(a) with a random sample of data. The second term is the non-stochastic error induced by approximating the infinite dimensional coefficient, β(a), with its d-dimensional counterpart, β^d(a).

Our first theorem in this section establishes consistency of β̂^d_n (a) as an estimator of β(a) for all β-mixing processes provided d_n increases at an appropriate rate. Theorem 2.4 gives finite sample bounds on the estimation error while some measure theoretic arguments contained in §4 show that the approximation error must go to zero as d_n → ∞.

Theorem 2.3

Let ${\mathbf{X}}_1^n$ be a sample from an arbitrary β-mixing process. Let d_n = O(exp{W(log n)}) where W is the Lambert W function.³ Then ${\hat{\beta }}^{d_n}(a)\stackrel{P}{\to }\beta (a)$ as n → ∞.

A finite sample bound for the approximation error is the first step to establishing consistency for β̂^d(a). This result gives convergence rates for estimation of the finite dimensional mixing coefficient β^d(a) and also for Markov processes of known order d, since in this case, β^d(a) = β(a).

Theorem 2.4

Consider a sample ${\mathbf{X}}_1^n$ from a stationary β-mixing process. Let μ_n and m_n be positive integers such that 2μ_nm_n = n and μ_n ≥ d > 0. Then

$$\mathbb{P}(|{\hat{\beta }}^d(a)-{\beta }^d(a)|>\epsilon )\le 2\text{exp}\{-\frac{{\mu }_n{\epsilon }_1^2}{2}\}+2\text{exp}\{-\frac{{\mu }_n{\epsilon }_2^2}{2}\}+4({\mu }_n-1)\beta (m_n),$$

where ε₁ = ε/2 − 𝔼 [∫|f̂^d − f^d|] and ${\epsilon }_2=\epsilon -𝔼[\int |{\hat{f}}_a^{2d}-f_a^{2d}|]$.

Consistency of the estimator β̂^d(a) is guaranteed only for certain choices of m_n and μ_n. Clearly μ_n → ∞ and μ_nβ(m_n) → 0 as n → ∞ are necessary conditions. Consistency also requires convergence of the histogram estimators to the target densities. We leave the proof of this theorem for section 4. As an example to show that this bound can go to zero with proper choices of m_n and μ_n, the following corollary proves consistency for first order Markov processes. Consistency of the estimator for higher order Markov processes can be proven similarly. These processes are geometrically β-mixing as shown in e.g. Nummelin and Tuominen [19].

Corollary 2.5

Let ${\mathbf{X}}_1^n$ be a sample from a first order Markov process with β(a) = β¹(a) = O(r^a) for some 0 ≤ r < 1. Then under the conditions of Theorem 2.4, ${\hat{\beta }}^1(a)\stackrel{P}{\to }\beta (a)$ at a rate of $o(\sqrt{n})$ up to a logarithmic factor.

Proof

Recall that n = 2μ_nm_n. Then,

$$4({\mu }_n-1)\beta (m_n)=4{\mu }_n\beta (m_n)+4\beta (m_n)=K_1\frac{n}{m_n}{r}^{m_n}+K_2{r}^{m_n}\to 0$$

if m_n = Ω(log n) for constants K₁ and K₂. But the exponential terms are

$$\text{exp}\{-K_3\frac{n{\epsilon }_j^2}{m_n}\}$$

for j = 1, 2 and a constant K₃. Therefore, both exponential terms go to 0 as n → ∞ for m_n = o(n). Balancing the rates gives the optimal choice of $m_n=o(\sqrt{n})$ with corresponding rate of convergence (up to a logarithmic factor) of $o(\sqrt{n})$.

Proving Theorem 2.4 requires showing the L¹ convergence of the histogram density estimator with β-mixing data. We do this in the next section.

3 L¹ convergence of histograms

Convergence of density estimators is thoroughly studied in the statistics and machine learning literature. Early papers on the L^∞ convergence of kernel density estimators (KDEs) include [26, 2, 22]; Freedman and Diaconis [10] look specifically at histogram estimators, and Yu [27] considered the L^∞ convergence of KDEs for β-mixing data and shows that the optimal IID rates can be attained. Devroye and Györfi [8] argue that L¹ is a more appropriate metric for studying density estimation, and Tran [23] proves L¹ consistency of KDEs under α- and β-mixing. As far as we are aware, ours is the first proof of L¹ convergence for histograms under β-mixing.

Additionally, the dimensionality of the target density is analogous to the order of the Markov approximation. Therefore, the convergence rates we give are asymptotic in the bandwidth h_n which shrinks as n increases, but also in the dimension d which increases with n. Even under these asymptotics, histogram estimation in this sense is not a high dimensional problem. The dimension of the target density considered here is on the order of exp{W(log n)}, a rate somewhere between log n and log log n.

Theorem 3.1

If f̂ is the histogram estimator based on a (possibly vector valued) sample ${\mathbf{X}}_1^n$ from a β-mixing sequence with stationary density f, then for all ε > 𝔼 [∫|f̂ − f|],

$$\mathbb{P}(\int |\hat{f}-f|>\epsilon )\le 2\text{exp}\{-\frac{{\mu }_n{\epsilon }_1^2}{2}\}+2({\mu }_n-1)\beta (m_n)$$ (5)

where ε₁ = ε − 𝔼 [∫|f̂ − f|].

To prove this result, we use the blocking method of Yu [28] to transform the dependent β-mixing into a sequence of nearly independent blocks. We then apply McDiarmid’s inequality to the blocks to derive asymptotics in the bandwidth of the histogram as well as the dimension of the target density. For completeness, we state Yu’s blocking result and McDiarmid’s inequality before proving the doubly asymptotic histogram convergence for IID data. Combining these lemmas allows us to derive rates of convergence for histograms based on β-mixing inputs.

Lemma 3.2

(Lemma 4.1 in Yu [28]). Let ϕ be a measurable function with respect to the block sequence U uniformly bounded by M. Then,

$$|𝔼[\varphi ]-\widetilde{𝔼}[\varphi ]|\le M\beta (m_n)({\mu }_n-1),$$ (6)

where the first expectation is with respect to the dependent block sequence, U, and 𝔼̃ is with respect to the independent sequence, U′.

This lemma essentially gives a method of applying IID results to β-mixing data. Because the dependence decays as we increase the separation between blocks, widely spaced blocks are nearly independent of each other. In particular, the difference between expectations over these nearly independent blocks and expectations over blocks which are actually independent can be controlled by the β-mixing coefficient.

Lemma 3.3

(McDiarmid Inequality [14]). Let X₁,…, X_n be independent random variables, with X_i taking values in a set A_i for each i. Suppose that the measurable function f : ∏ A_i → ℝ satisfies

$$|f(\mathbf{x})-f(\mathbf{x}\prime )|\le c_i$$

whenever the vectors x and x′ differ only in the i^th coordinate. Then for any ε > 0,

$$\mathbb{P}(f-𝔼f>\epsilon )\le \text{exp}\{-\frac{2{\epsilon }^2}{\sum c_i^2}\}.$$

Lemma 3.4

For an IID sample X₁,…, X_n from some density f on ℝ^d

$$𝔼\int |\hat{f}-𝔼\hat{f}|dx=O(1/\sqrt{n{h}_n^d})$$ (7)

$$\int |𝔼\hat{f}-f|dx=O(d{h}_n)+O(d^2{h}_n^2),$$ (8)

where f̂ is the histogram estimate using a grid with sides of length h_n.

Proof of Lemma 3.4

Let p_j be the probability of falling into the j^th bin B_j. Then,

$$𝔼\int |\hat{f}-𝔼\hat{f}|=h_n^d\sum _{j=1}^J𝔼|\frac{1}{n{h}_n^d}\sum _{i=1}^{n}I(X_i\in B_j)-\frac{p_j}{h^d}|\le h_n^d\sum _{j=1}^J\frac{1}{n{h}_n^d}\sqrt{𝕍[\sum _{i=1}^{n}I(X_i\in B_j)]}=h_n^d\sum _{j=1}^J\frac{1}{n{h}_n^d}\sqrt{n{p}_j(1-p_j)}=\frac{1}{\sqrt{n}}\sum _{j=1}^J\sqrt{p_j(1-p_j)}=O(n^{-1/2})O(h_n^{-d/2})=O(1/\sqrt{n{h}_n^d}).$$

For the second claim, consider the bin B_j centered at c. Let I be the union of all bins B_j. Assume the following:

1. f ∈ L₂ and f is absolutely continuous on I, with a.e. partial derivatives $f_i=\frac{\partial }{\partial y_i}f(\mathbf{y})$

2. f_i ∈ L₂ and f_i is absolutely continuous on I, with a.e. partial derivatives $f_{ik}=\frac{\partial }{\partial y_k}{f}_i(\mathbf{y})$

3. f_ik ∈ L₂ for all i, k.

Using a Taylor expansion

$$f(\mathbf{x})=f(\mathbf{c})+\sum _{i=1}^d(x_i-c_i)f_i(\mathbf{c})+O(d^2{h}_n^2),$$

where $f_i(\mathbf{y})=\frac{\partial }{\partial y_i}f(\mathbf{y})$. Therefore, p_j is given by

$$p_j={\int }_{B_j}f(x)dx=h_n^{d}f(c)+O(d^2{h}_n^{d+2})$$

since the integral of the second term over the bin is zero. This means that for the j^th bin,

$$𝔼\widehat{f_n}(x)-f(x)=\frac{p_j}{h_n^d}-f(x)=-\sum _{i=1}^d(x_i-c_i)f_i(\mathbf{c})+O(d^2{h}_n^2).$$

Therefore,

$${\int }_{B_j}|𝔼\widehat{f_n}(x)-f(x)|={\int }_{B_j}|-\sum _{i=1}^d(x_i-c_i)f_i(\mathbf{c})+O(d^2{h}_n^2)|\le {\int }_{B_j}|-\sum _{i=1}^d(x_i-c_i)f_i(\mathbf{c})|+{\int }_{B_j}O(d^2{h}^2)={\int }_{B_j}|\sum _{i=1}^d(x_i-c_i)f_i(\mathbf{c})|+O(d^2{h}_n^{2+d})=O(d{h}_n^{d+1})+O(d^2{h}_n^{2+d})$$

Since each bin is bounded, we can sum over all J bins. The number of bins is $J=h_n^{-d}$ by definition, so

$$\int |𝔼\widehat{f_n}(x)-f(x)|dx=O(h_n^{-d})(O(d{h}_n^{d+1})+O(d^2{h}_n^{2+d}))=O(d{h}_n)+O(d^2{h}_n^2).$$

We can now prove the main result of this section.

Proof of Theorem 3.1

Let g be the L¹ loss of the histogram estimator, $g=\int |f-\widehat{f_n}|$. Here $\widehat{f_n}(x)=\frac{1}{n{h}_n^d}{\sum }_{i=1}^{n}I(X_i\in B_j(x))$ where B_j(x) is the bin containing x. Let $\widehat{f_{\mathbf{U}}}$, $\widehat{f_{\mathbf{V}}}$, and $\widehat{f_{\mathbf{U}}}$ be histograms based on the block sequences U, V, and U′ respectively. Clearly $\widehat{f_n}=\frac{1}{2}(\widehat{f_{\mathbf{U}}}+\widehat{f_{\mathbf{V}}})$ Now,

$$\mathbb{P}(g>\epsilon )=\mathbb{P}(\int |f-\widehat{f_n}|>\epsilon )=\mathbb{P}(\int |\frac{f-\widehat{f_{\mathbf{U}}}}{2}+\frac{f-\widehat{f_{\mathbf{V}}}}{2}|>\epsilon )\le \mathbb{P}(\frac{1}{2}\int |f-\widehat{f_{\mathbf{U}}}|+\frac{1}{2}\int |f-\widehat{f_{\mathbf{V}}}|>\epsilon )=\mathbb{P}(g_{\mathbf{U}}+g_{\mathbf{V}}>2\epsilon )\le \mathbb{P}(g_{\mathbf{U}}>\epsilon )+\mathbb{P}(g_{\mathbf{V}}>\epsilon )=2\mathbb{P}(g_{\mathbf{U}}-𝔼[g_{\mathbf{U}}]>\epsilon -𝔼[g_{\mathbf{U}}])=2\mathbb{P}(g_{\mathbf{U}}-𝔼[g_{\mathbf{U}\prime }]>\epsilon -𝔼[g_{\mathbf{U}\prime }])=2\mathbb{P}(g_{\mathbf{U}}-𝔼[g_{\mathbf{U}\prime }]>{\epsilon }_1),$$

where ε₁ = ε − 𝔼[g_U′]. Here,

$$𝔼[g_{\mathbf{U}\prime }]\le \widetilde{𝔼}\int |\widehat{f_{\mathbf{U}\prime }}-\widetilde{𝔼}\widehat{f_{\mathbf{U}\prime }}|dx+\int |\widetilde{𝔼}\widehat{f_{\mathbf{U}\prime }}-f|dx,$$

so by Lemma 3.4, as long as for μ_n → ∞, h_n ↓ 0 and ${\mu }_n{h}_n^d\to \infty$, then for all ε there exists n₀(ε) such that for all n > n₀(ε), ε > 𝔼[g] = 𝔼[g_U′]. Now applying Lemma 3.2 to the expectation of the indicator of the event {g_U − 𝔼[g_U′] > ε₁} gives

$$2\mathbb{P}(g_{\mathbf{U}}-𝔼[g_{\mathbf{U}\prime }]>{\epsilon }_1)\le 2\mathbb{P}(g_{\mathbf{U}\prime }-𝔼[g_{\mathbf{U}\prime }]>{\epsilon }_1)+2({\mu }_n-1)\beta (m_n)$$

where the probability on the right is for the σ-field generated by the independent block sequence U′. Since these blocks are independent, showing that g_U′ satisfies the bounded differences requirement allows for the application of McDiarmid’s inequality 3.3 to the blocks. For any two block sequences $u_1^{\prime },\dots ,u_{{\mu }_n}^{\prime }$ and ${ū}_1^{\prime },\dots ,{ū}_{{\mu }_n}^{\prime }$ with $u_{\ell }^{\prime }={ū}_{\ell }^{\prime }$ for all ℓ ≠ j, then

$$|g_{\mathbf{U}\prime }(u_1^{\prime },\dots ,u_{{\mu }_n}^{\prime })-g_{\mathbf{U}\prime }({ū}_1^{\prime },\dots ,{ū}_{{\mu }_n}^{\prime })|=|\int |\hat{f}(y;u_1^{\prime },\dots ,u_{{\mu }_n}^{\prime })-f(y)|dy-\int |\hat{f}(y;{ū}_1^{\prime },\dots ,{ū}_{{\mu }_n}^{\prime })-f(y)|dy|\le \int |\hat{f}(y;u_1^{\prime },\dots ,u_{{\mu }_n}^{\prime })-\hat{f}(y;{ū}_1^{\prime },\dots ,{ū}_{{\mu }_n}^{\prime })|dy=\frac{2}{{\mu }_n{h}_n^d}{h}_n^d=\frac{2}{{\mu }_n}.$$

Therefore,

$$\mathbb{P}(g>\epsilon )\le 2\mathbb{P}(g_{\mathbf{U}\prime }-𝔼[g_{\mathbf{U}\prime }]>{\epsilon }_1)+2({\mu }_n-1)\beta (m_n)\le 2\text{exp}\{-\frac{{\mu }_n{\epsilon }_1^2}{2}\}+2({\mu }_n-1)\beta (m_n).$$

4 Proofs

The proof of Theorem 2.4 relies on the triangle inequality and the relationship between total variation distance and the L¹ distance between densities.

Proof of Theorem 2.4

For any probability measures ν and λ defined on the same probability space with associated densities f_ν and f_λ with respect to some dominating measure π,

$${\Vert \nu -\lambda \Vert }_{TV}=\frac{1}{2}\int |f_{\nu }-f_{\lambda }|d(\pi ).$$

Let P be the d-dimensional stationary distribution of the d^th order Markov process, i.e. $P={\mathbb{P}}_{t-d+1}^t={\mathbb{P}}_{t+a}^{t+a+d-1}$ in the notation of equation 3. Let ℙ_a,d be the joint distribution of the bivariate random process created by the initial process and itself separated by a time steps. By the triangle inequality, we can upper bound β^d(a) for any d = d_n. Let P̂ and ℙ̂_a,d be the distributions associated with histogram estimators f̂^d and ${\hat{f}}_a^{2d}$ respectively. Then,

$${\beta }^d(a)={\Vert P\otimes P-{\mathbb{P}}_{a,d}\Vert }_{TV}={\Vert P\otimes P-\hat{P}\otimes \hat{P}+\hat{P}\otimes \hat{P}-{\hat{\mathbb{P}}}_{a,d}+{\hat{\mathbb{P}}}_{a,d}-{\mathbb{P}}_{a,d}\Vert }_{TV}\le {\Vert P\otimes P-\hat{P}\otimes \hat{P}\Vert }_{TV}+{\Vert \hat{P}\otimes \hat{P}-{\hat{\mathbb{P}}}_{a,d}\Vert }_{TV}+{\Vert {\hat{\mathbb{P}}}_{a,d}-{\mathbb{P}}_{a,d}\Vert }_{TV}\le 2{\Vert P-\hat{P}\Vert }_{TV}+{\Vert \hat{P}\otimes \hat{P}-{\hat{\mathbb{P}}}_{a,d}\Vert }_{TV}+{\Vert {\hat{\mathbb{P}}}_{a,d}-{\mathbb{P}}_{a,d}\Vert }_{TV}=\int |f^d-{\hat{f}}^d|+\frac{1}{2}\int |{\hat{f}}^d\otimes {\hat{f}}^d-{\hat{f}}_a^{2d}|+\frac{1}{2}\int |f_a^{2d}-{\hat{f}}_a^{2d}|$$

where $\frac{1}{2}\int |{\hat{f}}^d\otimes {\hat{f}}^d-{\hat{f}}_a^{2d}|$ is our estimator β̂^d(a) and the remaining terms are the L¹ distance between a density estimator and the target density. Thus,

$${\beta }^d(a)-{\hat{\beta }}^d(a)\le \int |f^d-{\hat{f}}^d|+\frac{1}{2}\int |f_a^{2d}-{\hat{f}}_a^{2d}|.$$

A similar argument starting from β^d(a) = ‖P ⊗ P − ℙ_a,d‖_TV shows that

$${\beta }^d(a)-{\hat{\beta }}^d(a)\ge -\int |f^d-{\hat{f}}^d|-\frac{1}{2}\int |f_a^{2d}-{\hat{f}}_a^{2d}|,$$

so we have that

$$|{\beta }^d(a)-{\hat{\beta }}^d(a)|\le \int |f^d-{\hat{f}}^d|+\frac{1}{2}\int |f_a^{2d}-{\hat{f}}_a^{2d}|.$$

Therefore,

$$\mathbb{P}\left(\right|{\beta }^d(a)-{\hat{\beta }}^d(a)|>\epsilon )\le \mathbb{P}(\int |f^d-{\hat{f}}^d|+\frac{1}{2}\int |f_a^{2d}-{\hat{f}}_a^{2d}|>\epsilon )\le \mathbb{P}(\int |f^d-{\hat{f}}^d|>\frac{\epsilon }{2})+\mathbb{P}(\frac{1}{2}\int |f_a^{2d}-{\hat{f}}_a^{2d}|>\frac{\epsilon }{2})\le 2\text{exp}\{-\frac{{\mu }_n{\epsilon }_1^2}{2}\}+2\text{exp}\{-\frac{{\mu }_n{\epsilon }_2^2}{2}\}+4({\mu }_n-1)\beta (m_n),$$

where ε₁ = ε/2 − 𝔼 [∫|f̂^d − f^d|] and ${\epsilon }_2=\epsilon -𝔼[\int |{\hat{f}}_a^{2d}-f_a^{2d}|]$.

The proof of Theorem 2.3 requires two steps which are given in the following Lemmas. The first specifies the histogram bandwidth h_n and the rate at which d_n (the dimensionality of the target density) goes to infinity. If the dimensionality of the target density were fixed, we could achieve rates of convergence similar to those for histograms based on IID inputs. However, we wish to allow the dimensionality to grow with n, so the rates are much slower as shown in the following lemma.

Lemma 4.1

For the histogram estimator in Lemma 3.4, let

$$d_n~\text{exp}\{W(\text{log}n)\},$$

$$h_n~n^{-k_n},$$

with

$$k_n=\frac{W(\text{log}n)+\frac{1}{2}\text{log}n}{\text{log}n(\frac{1}{2}\text{exp}\{W(\text{log}n)\}+1)}.$$

These choices lead to the optimal rate of convergence.

Proof

Let h_n = n^−k_n for some k_n to be determined. Then we want $n^{-1/2}{h}_n^{-d_n/2}=n^{(k_n{d}_n-1)/2}\to 0$, d_nh_n = d_nn^−k → 0, and $d_n^2{h}_n^2=d_n^2{n}^{-2k}\to 0$ all as n → ∞. Call these A, B, and C. Taking A and B first gives

$$n^{(k_n{d}_n-1)/2}~d_n{n}^{-k_n}\Rightarrow \frac{1}{2}(k_n{d}_n-1)\text{log}n~\text{log}{d}_n-k_n\text{log}n\Rightarrow k_n\text{log}n(\frac{1}{2}{d}_n+1)~\text{log}{d}_n+\frac{1}{2}\text{log}n\Rightarrow k_n~\frac{\text{log}{d}_n+\frac{1}{2}\text{log}n}{\text{log}n(\frac{1}{2}{d}_n+1)}.$$ (9)

Similarly, combining A and C gives

$$k_n~\frac{2\text{log}{d}_n+\frac{1}{2}\text{log}n}{\text{log}n(\frac{1}{2}{d}_n+2)}.$$ (10)

Equating (9) and (10) and solving for d_n gives

$$\Rightarrow d_n~\text{exp}\{W(\text{log}n)\}$$

where W(·) is the Lambert W function. Plugging back into (9) gives that

$$h_n=n^{-k_n}$$

where

$$k_n=\frac{W(\text{log}n)+\frac{1}{2}\text{log}n}{\text{log}n(\frac{1}{2}\text{exp}\{W(\text{log}n)\}+1)}.$$

It is also necessary to show that as d grows, β^d(a) → β(a). We now prove this result.

Lemma 4.2

β^d(a) converges to β(a) as d → ∞.

Proof

By stationarity, the supremum over t is unnecessary in Definition 2.1, so without loss of generality, let t = 0. Let ${\mathbb{P}}_{-\infty }^0$ be the distribution on ${\sigma }_{-\infty }^0=\sigma (\dots ,X_{-1},X_0)$, and let ${\mathbb{P}}_a^{\infty }$ be the distribution on ${\sigma }_{a+1}^{\infty }=\sigma (X_{a+1},X_{a+2},\dots )$. Let ℙ_a be the distribution on $\sigma ={\sigma }_{-\infty }^0\otimes {\sigma }_{a+1}^{\infty }$ (the product sigma-field). Then we can rewrite Definition 2.1 using this notation as

$$\beta (a)=\underset{C\in \sigma }{\text{sup}}|{\mathbb{P}}_a(C)-[{\mathbb{P}}_{-\infty }^0\otimes {\mathbb{P}}_{a+1}^{\infty }](C)|.$$

Let ${\sigma }_{-d+1}^0$ and ${\sigma }_{a+1}^{a+d}$ be the sub-σ-fields of ${\sigma }_{-\infty }^0$ and ${\sigma }_{a+1}^{\infty }$ consisting of the d-dimensional cylinder sets for the d dimensions closest together. Let σ^d be the product σ-field of these two. Then we can rewrite β^d(a) as

$${\beta }^d(a)=\underset{C\in {\sigma }^d}{\text{sup}}\Vert {\mathbb{P}}_a(C)-[{\mathbb{P}}_{-\infty }^0\otimes {\mathbb{P}}_{a+1}^{\infty }](C)|.$$ (11)

As such β^d(a) ≤ β(a) for all a and d. We can rewrite (11) in terms of finite-dimensional marginals:

$${\beta }^d(a)=\underset{C\in {\sigma }^d}{\text{sup}}|{\mathbb{P}}_{a,d}(C)-[{\mathbb{P}}_{-d+1}^0\otimes {\mathbb{P}}_{a+1}^{a+d}](C)|,$$

where ℙ_a,d is the restriction of ℙ to σ(X_−d+1,…, X₀, X_a+1,…, X_a+d). Because of the nested nature of these sigma-fields, we have

$${\beta }^{d_1}(a)\le {\beta }^{d_2}(a)\le \beta (a)$$

for all finite d₁ ≤ d₂. Therefore, for fixed ${\{{\beta }^d(a)\}}_{d=1}^{\infty }$ is a monotone increasing sequence which is bounded above, and it converges to some limit L ≤ β(a). To show that L = β(a) requires some additional steps.

Let $R={\mathbb{P}}_a-[{\mathbb{P}}_{-\infty }^0\otimes {\mathbb{P}}_a^{\infty }]$, which is a signed measure on σ. Let $R^d={\mathbb{P}}_{a,d}-[{\mathbb{P}}_{-d}^0\otimes {\mathbb{P}}_a^{a+d}]$, which is a signed measure on σ^d. Decompose R into positive and negative parts as R = Q⁺ − Q⁻ and similarly for R^d = Q^+d − Q^−d. Notice that since R^d is constructed using the marginals of ℙ, then R(E) = R^d(E) for all E ∈ σ^d. Now since R is the difference of probability measures, we must have that

$$0=R(\mathrm{\Omega })=Q^{+}(\mathrm{\Omega })-Q^{-}(\mathrm{\Omega })=Q^{+}(D)+Q^{+}(D^c)-Q^{-}(D)-Q^{-}(D^c)$$ (12)

for all D ∈ σ.

Define Q = Q⁺ + Q⁻. Let ε > 0. Let C ∈ σ be such that

$$Q(C)=\beta (a)=Q^{+}(C)=Q^{-}(C^c).$$ (13)

Such a set C is guaranteed by the Hahn decomposition theorem (letting C* be a set which attains the supremum in (11), we can throw away any subsets with negative R measure) and (12) assuming without loss of generality that ${\mathbb{P}}_a(C)>[{\mathbb{P}}_{-\infty }^0\otimes {\mathbb{P}}_a^{\infty }](C)$. We can use the field σ_f = ∪_d σ^d to approximate σ in the sense that, for all ε, we can find A ∈ σ_f such that Q(AΔC) < ε/2 (see Theorem D in Halmos [11, §13] or Lemma A.24 in Schervish [21]). Now,

$$Q(A\mathrm{\Delta }C)=Q(A\cap C^c)+Q(C\cap A^c)=Q^{-}(A\cap C^c)+Q^{+}(C\cap A^c)$$

by (13) since A∩C^c ⊆ C^c and C∩A^c ⊆ C. Therefore, since Q(AΔC) < ε/2, we have

$$Q^{-}(A\cap C^c)\le \epsilon /2$$ (14)

$$Q^{+}(A^c\cap C)\le \epsilon /2.$$

Also,

$$Q(C)=Q(A\cap C)+Q(A^c\cap C)=Q^{+}(A\cap C)+Q^{+}(A^c\cap C)\le Q^{+}(A)+\epsilon /2$$

since A∩C and A^c∩C are contained in C and A∩C ⊆ A. Therefore

$$Q^{+}(A)\ge Q(C)-\epsilon /2.$$

Similarly,

$$Q^{-}(A)=Q^{-}(A\cap C)+Q^{-}(A\cap C^c)\le 0+\epsilon /2=\epsilon /2$$

since A ∩ C ⊆ C and Q⁻(C) = 0 by (14). Finally,

$$Q^{+d}(A)\ge Q^{+d}(A)-Q^{-d}(A)=R^d(A)=R(A)=Q^{+}(A)-Q^{-}(A)\ge Q(C)-\epsilon /2-\epsilon /2=Q(C)-\epsilon =\beta (a)-\epsilon .$$

And since β^d(a) ≥ Q^+d(A), we have that for all ε > 0 there exists d such that for all d₁ > d,

$${\beta }^{d_1}(a)\ge {\beta }^d(a)\ge Q^{+d}(A)\ge \beta (a)-\epsilon .$$

Thus, we must have that L = β(a), so that β^d(a) → β(a) as desired.

Proof of Theorem 2.3

By the triangle inequality,

$$|{\hat{\beta }}^{d_n}(a)-\beta (a)|\le |{\hat{\beta }}^{d_n}(a)-{\beta }^{d_n}(a)|+|{\beta }^{d_n}(a)-\beta (a)|.$$

The first term on the right is bounded by the result in Theorem 2.4, where we have shown that d_n = O(exp{W(log n)}) is slow enough for the histogram estimator to remain consistent. That ${\beta }^{d_n}(a)\stackrel{d_n\to \infty }{\to }\beta (a)$ follows from Lemma 4.2.

5 Discussion

We have shown that our estimator of the β-mixing coefficients is consistent for the true coefficients β(a) under some conditions on the data generating process. There are numerous results in the statistics and machine learning literatures which assume knowledge of the β-mixing coefficients, yet as far as we know, this is the first estimator for them. An ability to estimate these coefficients will allow researchers to apply existing results to dependent data without the need to arbitrarily assume their values. Despite the obvious utility of this estimator, as a consequence of its novelty, it comes with a number of potential extensions which warrant careful exploration as well as some drawbacks.

The reader will note that Theorem 2.3 does not provide a convergence rate. The rate in Theorem 2.4 applies only to the difference between β̂^d(a) and β^d(a). In order to provide a rate in Theorem 2.3, we would need a better understanding of the non-stochastic convergence of β^d(a) to β(a). It is not immediately clear that this quantity can converge at any well-defined rate. In particular, it seems likely that the rate of convergence depends on the tail of the sequence ${\{\beta (a)\}}_{a=1}^{\infty }$.

Several other mixing and weak-dependence coefficients also have a total-variation flavor, perhaps most notably α-mixing [9, 7, 4]. None of them have estimators, and the same trick might well work for them, too.

The use of histograms rather than kernel density estimators for the joint and marginal densities is surprising and perhaps not ultimately necessary. As mentioned above, Tran [23] proved that KDEs are consistent for estimating the stationary density of a time series with β-mixing inputs, so perhaps one could replace the histograms in our estimator with KDEs. However, this would need an analogue of the double asymptotic results proven for histograms in Lemma 3.4. In particular, we need to estimate increasingly higher dimensional densities as n → ∞. This does not cause a problem of small-n-large-d since d is chosen as a function of n, however it will lead to increasingly higher dimensional integration. For histograms, the integral is always trivial, but in the case of KDEs, the numerical accuracy of the integration algorithm becomes increasingly important. This issue could swamp any efficiency gains obtained through the use of kernels. However, this question certainly warrants further investigation.

The main drawback of an estimator based on a density estimate is its complexity. The mixing coefficients are functionals of the joint and marginal distributions derived from the stochastic process X, however, it is unsatisfying to estimate densities and solve integrals in order to estimate a single number. Vapnik’s main principle for solving problems using a restricted amount of information is

When solving a given problem, try to avoid solving a more general problem as an intermediate step [24, p. 30].

This principle is clearly violated here, but perhaps our seed will precipitate a more aesthetically pleasing solution.