Normal Laws for Two Entropy Estimators on Infinite Alphabets

Abstract

This paper offers sufficient conditions for the Miller–Madow estimator and the jackknife estimator of entropy to have respective asymptotic normalities on countably infinite alphabets.

1. Introduction

Let $\mathcal{X}=\{{\ell }_k;k\ge 1\}$ be a finite or countably infinite alphabet, let $\mathbf{p}=\{p_k;k\ge 1\}$ be a probability distribution on $\mathcal{X}$, and define $K={\sum }_{k\ge 1}1[p_k>0]$, where $1[·]$ is the indicator function, to be the effective cardinality of $\mathcal{X}$ under $\mathbf{p}$. An important quantity associated with $\mathbf{p}$ is entropy, which is defined by [1] as

$$H=-\sum _{k\ge 1}{p}_{k}ln{p}_k.$$ (1)

Here and throughout, we adopt the convention that $0ln0=0$.

Many properties of entropy and related quantities are discussed in [2]. The problem of statistical estimation of entropy has a long history (see the survey paper [3] or the recent book [4]). It is well-known that no unbiased estimators of entropy exist, and, for this reason, much energy has been focused on deriving estimators with relatively little bias (see [5] and the references therein for a discussion of some (but far from all) of these). Perhaps the most commonly used estimator is the plug-in. Its theoretical properties have been studied going back, at least, to [6], where conditions for consistency and asymptotic normality, in the case of finite alphabets, were derived. It would be almost fifty years before corresponding conditions for the countabe case would appear in the literature. Specifically, consistency, both in terms of almost sure and $L^2$ convergence, was verified in [7]. Later, sufficient conditions for asymptotic normality were derived in two steps in [3,8].

Despite a simple form and nice theoretical properties, the plug-in suffers from large finite sample bias, which has led to the development of modifications that aim to reduce this bias. Two of the most popular are the Miller–Madow estimator of [6] and the jackknife estimator of [9]. Theoretical properties of these have not been studied, as extensively, in the literature. In this paper, we give sufficient conditions for the asymptotic normality of these two estimators. This is important for deriving confidence intervals and hypothesis tests, and it immediately implies consistency (see e.g., [4]).

We begin by introducing some notation. We say that a distribution $\mathbf{p}=\{p_k;k\ge 1\}$ is uniform if and only if its effective cardinality $K<\infty$ and for each $k=1,2,\cdots$ either $p_k=1/K$ or $p_k=0$. We write $f\sim g$ to denote ${lim}_{n\to \infty }f\left(n\right)/g\left(n\right)=1$ and we write $f=\mathcal{O}\left(g\right(n\left)\right)$ to denote ${lim\; sup}_{n\to \infty }\left|f\left(n\right)/g\left(n\right)\right|<\infty$. Furthermore, we write $\stackrel{L}{\to }$ to denote convergence in law and $\stackrel{p}{\to }$ to denote convergence in probability. If a and b are real number, we write $a\vee b$ to denote the maximum of a and b. When it is not specified, all limits are assumed to be taken as $n\to \infty$.

Let $X_1,\cdots ,X_n$ be independent and identically distributed ($iid$) random variables on $\mathcal{X}$ under $\mathbf{p}$. Let $\{Y_k;k\ge 1\}$ be the observed letter counts in the sample, i.e., $Y_k={\sum }_{i=1}^{n}1[X_i={\ell }_k]$, and let $\widehat{\mathbf{p}}=\{{\widehat{p}}_k;k\ge 1\}$, where ${\widehat{p}}_k=Y_k/n$, be the corresponding relative frequencies. Perhaps the most intuitive estimator of H is the plug-in, which is given by

$$\widehat{H}=-\sum _{k\ge 1}{\widehat{p}}_{k}ln{\widehat{p}}_k.$$ (2)

When the effective cardinality, K, is finite, [10] showed that the bias of $\widehat{H}$ is

$$\mathrm{E}\left(\widehat{H}\right)-H=-\frac{K-1}{2n}+\frac{1}{12n^2}\left(1-\sum _{k=1}^K\frac{1}{p_k}\right)+\mathcal{O}\left(n^{-3}\right).$$ (3)

One of the simplest and earliest approaches aiming to reduce the bias of $\widehat{H}$ is to estimate the first order term. Specifically, let $\widehat{m}={\sum }_{k\ge 1}1[Y_k>0]$ be the number of letters observed in the sample and consider an estimator of the form,

$${\widehat{H}}_{MM}=\widehat{H}+\frac{\widehat{m}-1}{2n}.$$ (4)

This estimator is often attributed to [6] and is known as the Miller–Madow estimator. Note that, for finite K,

$$\mathrm{E}\left(\frac{\widehat{m}-1}{2n}\right)=\frac{K-1}{2n}-\frac{{\sum }_k{(1-p_k)}^n}{2n}.$$

Since ${\sum }_k{(1-p_k)}^n\le K{(1-p_{\wedge })}^n$, where $p_{\wedge }=min\{p_k:p_k>0\}$, decays exponentially fast, it follows that, for finite K, the bias of ${\widehat{H}}_{MM}$ is

$$\mathrm{E}\left({\widehat{H}}_{MM}\right)-H=\frac{1}{12n^2}\left(1-\sum _{k=1}^K\frac{1}{p_k}\right)+\mathcal{O}\left(n^{-3}\right).$$

Among the many estimators in the literature aimed at reducing bias in entropy estimation, the Miller–Madow estimator is one of the most commonly used. Its popularity is due to its simplicity, its intuitive appeal, and, more importantly, its good performance across a wide range of different distributions including those on countably infinite alphabets. See, for instance, the simulation study in [5].

The jackknife entropy estimator is another commonly used estimator designed to reduce the bias of the plug-in. It is calculated in three steps:

1. for each $i\in \{1,2,\cdots ,n\}$ construct ${\widehat{H}}^{\left(i\right)}$, which is a plug-in estimator based on a sub-sample of size $n-1$ obtained by leaving the ith observation out;

2. obtain ${\widehat{H}}_{\left(i\right)}=n\widehat{H}-(n-1){\widehat{H}}^{\left(i\right)}$ for $i=1,\cdots ,n$; and then

3. compute the jackknife estimator

   $${\widehat{H}}_{JK}=\frac{{\sum }_{i=1}^n{\widehat{H}}_{\left(i\right)}}{n}.$$ (5)

Equivalently, (5) can be written as

$${\widehat{H}}_{JK}=n\widehat{H}-(n-1)\frac{{\sum }_{i=1}^n{\widehat{H}}^{\left(i\right)}}{n}.$$ (6)

The jackknife estimator of entropy was first described by [9]. From (2), it may be verified that, when $K<\infty$, the bias of ${\widehat{H}}_{JK}$ is

$$E\left({\widehat{H}}_{JK}\right)-H=\mathcal{O}\left(n^{-2}\right).$$ (7)

Both the Miller–Madow and the jackknife estimators are adjusted versions of the plug-in. When the effective cardinality is finite, i.e., $K<\infty$, the asymptotic normalities of both can be easily verified. A question of theoretical interest is whether these normalities still hold when the effective cardinality is countably infinite. In this paper, we give sufficient conditions for $\sqrt{n}({\widehat{H}}_{MM}-H)$ and $\sqrt{n}({\widehat{H}}_{JK}-H)$ to have asymptotic normalities on countably infinite alphabets and provide several illustrative examples. The rest of paper is organized as follows. Our main results for both the Miller–Madow and the jackknife estimators are given in Section 2. A small simulation study is given in Section 3. This is followed by a brief discussion in Section 4. Proofs are postponed to Section 5.

2. Main Results

We begin by recalling a sufficient condition due to [8] for the asymptotic normality of the plug-in estimator.

Condition 1.

The distribution, $\mathbf{p}=\{p_k;k\ge 1\}$, satisfies

$$\begin{array}{c}\hfill \sum _{k\ge 1}{p}_k{ln}^2{p}_k<\infty ,\end{array}$$ (8)

and there exists an integer-valued function $K\left(n\right)$ such that, as $n\to \infty$,

• 1.  $K\left(n\right)\to \infty$,
• 2.  $K\left(n\right)/\sqrt{n}\to 0$, and
• 3.  $\sqrt{n}{\sum }_{k\ge K\left(n\right)}{p}_{k}ln{p}_k\to 0$.

Note that, by Jensen’s inequality (see e.g., [2]), (8) implies that

$$H^2={\left(-\sum _{k\ge 1}{p}_{k}ln{p}_k\right)}^2\le \sum _{k\ge 1}{p}_k{ln}^2{p}_k<\infty ,$$

where equality holds, i.e., $H^2={\sum }_{k\ge 1}{p}_k{ln}^2{p}_k$, if and only if $\mathbf{p}$ is a uniform distribution. Thus, when (8) holds, we have $H<\infty$. The following result is given in [8].

Lemma 1.

Let $\mathbf{p}=\{p_k;k\ge 1\}$ be a distribution, which is not uniform, and set

$${\sigma }^2=\sum _{k\ge 1}{p}_k{ln}^2{p}_k-H^{2}and{\widehat{\sigma }}^2=\sum _{k\ge 1}{\widehat{p}}_k{ln}^2{\widehat{p}}_k-{\widehat{H}}^2.$$ (9)

If $\mathbf{p}$ satisfies Condition 1, then $\widehat{\sigma }\stackrel{p}{\to }\sigma$,

$$\frac{\sqrt{n}(\widehat{H}-H)}{\sigma }\stackrel{L}{⟶}N(0,1),$$

and

$$\frac{\sqrt{n}(\widehat{H}-H)}{\widehat{\sigma }}\stackrel{L}{⟶}N(0,1).$$

The following is useful for checking when Condition 1 holds.

Lemma 2.

Let $\mathbf{p}=\{p_k;k\ge 1\}$ and ${\mathbf{p}}^{\prime }=\{p_k^{\prime };k\ge 1\}$ be two distributions and assume that ${\mathbf{p}}^{\prime }$ satisfies Condition 1. If there exists a $C>0$ such that, for large enough k,

$$p_k\le C{p}_k^{\prime },$$

then $\mathbf{p}$ satisfies Condition 1 as well.

In [8], it is shown that Condition 1 holds for $\mathbf{p}=\{p_k;k\ge 1\}$ with

$$p_k=\frac{C}{k^2{ln}^{2}k},k=1,2,\cdots ,$$

where $C>0$ is a normalizing constant. It follows from Lemma 2 that any distribution with tails lighter than this satisfies Condition 1 as well.

We are interested in finding conditions under which the result of Lemma 1 can be extended to bias adjusted modifications of $\widehat{H}$. Let ${\widehat{H}}_{*}$ be any bias-adjusted estimator of the form

$${\widehat{H}}_{*}=\widehat{H}+{\widehat{B}}_{*},$$ (10)

where ${\widehat{B}}_{*}$ is an estimate of the bias. Combining Lemma 1 with Slutsky’s theorem immediately gives the following.

Theorem 1.

Let $\mathbf{p}=\{p_k;k\ge 1\}$ be a distribution, which is not uniform, and let ${\sigma }^2$ and ${\widehat{\sigma }}^2$ be as in (9). If Condition 1 holds and $\sqrt{n}{\widehat{B}}_{*}\stackrel{p}{⟶}0$, then $\widehat{\sigma }\stackrel{p}{\to }\sigma$,

$$\frac{\sqrt{n}({\widehat{H}}_{*}-H)}{\sigma }\stackrel{L}{⟶}N(0,1),$$

and

$$\frac{\sqrt{n}({\widehat{H}}_{*}-H)}{\widehat{\sigma }}\stackrel{L}{⟶}N(0,1).$$

For the Miller–Madow estimator and the jackknife estimator, respectively, the bias correction term, ${\widehat{B}}_{*}$, in (10) takes the form

$$\begin{array}{ccc}\hfill Miller-Madow:& \hfill {\widehat{B}}_{MM}& =\frac{\widehat{m}-1}{2n},\hfill \\ \hfill Jackknife:& \hfill {\widehat{B}}_{JK}& =\frac{n-1}{n}\sum _{i=1}^n\left(\widehat{H}-{\widehat{H}}^{\left(i\right)}\right).\hfill \end{array}$$

Below, we give sufficient conditions for when $\sqrt{n}{\widehat{B}}_{MM}\stackrel{p}{⟶}0$ and when $\sqrt{n}{\widehat{B}}_{JK}\stackrel{p}{⟶}0$.

2.1. Results for the Miller–Madow Estimator

Condition 2.

The distribution, $\mathbf{p}=\{p_k;k\ge 1\}$, satisfies that, for sufficiently large k,

$$\begin{array}{c}\hfill p_k\le \frac{1}{a\left(k\right)b\left(k\right)k^3},\end{array}$$ (11)

where $a\left(k\right)>0$ and $b\left(k\right)>0$ are two sequences such that

• 1. 
  $a\left(k\right)\to \infty$ as $k\to \infty$, and, furthermore,

  • (a) 
    the function $a\left(k\right)$ is eventually nondecreasing, and
  • (b) 
    there exists an $\epsilon >0$ such that

    $$\underset{k\to \infty }{lim\; sup}\frac{{\left(a\left(k\right)\right)}^{2\epsilon }}{a\left(\frac{\sqrt{k}}{{\left(a\left(k\right)\right)}^{\epsilon }}\right)}<\infty ;$$ (12)
• 2. 
  $$\sum _{k\ge 1}\frac{1}{kb\left(k\right)}<\infty .$$ (13)

Since this condition only requires that $p_k$, for sufficiently large k, is upper bounded in the appropriate way, we immediately get the following.

Lemma 3.

Let $\mathbf{p}=\{p_k;k\ge 1\}$ and ${\mathbf{p}}^{\prime }=\{p_k^{\prime };k\ge 1\}$ be two distributions and assume that ${\mathbf{p}}^{\prime }$ satisfies Condition 2. If there exists a $C>0$ such that, for large enough k,

$$p_k\le C{p}_k^{\prime },$$

then $\mathbf{p}$ satisfies Condition 2 as well.

We now give our main results for the Miller–Madow Estimator.

Theorem 2.

Let $\mathbf{p}=\{p_k;k\ge 1\}$ be a distribution, which is not uniform, and let ${\sigma }^2$ and ${\widehat{\sigma }}^2$ be as in (9). If Condition 2 holds, then $\widehat{\sigma }\stackrel{p}{\to }\sigma$,

$$\frac{\sqrt{n}({\widehat{H}}_{MM}-H)}{\sigma }\stackrel{L}{⟶}N(0,1)$$

and

$$\frac{\sqrt{n}({\widehat{H}}_{MM}-H)}{\widehat{\sigma }}\stackrel{L}{⟶}N(0,1).$$

In the proof of the theorem, we will show that Condition 2 implies that Condition 1 holds. Condition 2 requires $p_k$ to decay slightly faster than $k^{-3}$ by two factors $1/a\left(k\right)$ and $1/b\left(k\right)$, where $a\left(k\right)$ and $b\left(k\right)$ satisfy (12) and (13) respectively. While (13) is clear in its implication on $b\left(k\right)$, (12) is much less so on $a\left(k\right)$. To have a better understanding of (12), we give an important situation where (12) holds. Consider the case $a\left(n\right)=lnn$. In this case, for any $\epsilon \in (0,0.5)$

$$\frac{{\left(a\left(n\right)\right)}^{2\epsilon }}{a\left(\frac{\sqrt{n}}{{\left(a\left(n\right)\right)}^{\epsilon }}\right)}=\frac{{(lnn)}^{2\epsilon }}{0.5lnn-\epsilon lnlnn}\sim \frac{{(lnn)}^{2\epsilon }}{0.5lnn}⟶0.$$

We now give a more general situation, which shows just how slow $a\left(k\right)$ can be. First, we recall the iterated logarithm function. Define ${ln}^{\left(r\right)}\left(x\right)$, recursively for sufficiently large $x>0$, by ${ln}^{\left(0\right)}\left(x\right)=x$ and ${ln}^{\left(r\right)}\left(x\right)=ln\left({ln}^{(r-1)}x\right)$ for $r\ge 1$. By induction, it can be shown that $\frac{\mathrm{d}}{\mathrm{d}x}{ln}^{\left(r\right)}\left(x\right)={\left({\prod }_{i=0}^{r-1}{ln}^{\left(i\right)}\left(x\right)\right)}^{-1}$ for $r\ge 1$.

Lemma 4.

The function $a\left(n\right)={ln}^{\left(r\right)}\left(n\right)$ satisfies (12) with $\epsilon =0.5$ for any $r\ge 2$.

We now give three examples.

Example 1.

Let $\mathbf{p}=\{p_k;k\ge 1\}$ be such that for sufficiently large k,

$$p_k\le \frac{C}{k^3(lnk){(lnlnk)}^{2+\epsilon }},$$

where $\epsilon >0$ and $C>0$ are fixed constants. In this case, Condition 2 holds with $a\left(k\right)=lnlnk$ and $b\left(k\right)=(lnk){(lnlnk)}^{1+\epsilon }/C$ in (11).

We can consider a more general form, which allows for even heavier tails.

Example 2.

Let r be an integer with $r\ge 2$ and let $\mathbf{p}=\{p_k;k\ge 1\}$ be such that, for sufficiently large k,

$$p_k\le \frac{C}{k^3\left({\prod }_{i=1}^{r-1}{ln}^{\left(i\right)}k\right){\left({ln}^{\left(r\right)}k\right)}^{2+\epsilon }}$$

where $\epsilon >0$ and $C>0$ are fixed constants. In this case, Condition 2 holds with $a\left(k\right)={ln}^{\left(r\right)}k$ and $b\left(k\right)=\left({\prod }_{i=1}^{r-1}{ln}^{\left(i\right)}k\right){\left({ln}^{\left(r\right)}k\right)}^{1+\epsilon }/C$ in (11). The fact that $b\left(k\right)$ satisfies (13) follows by the integral test for convergence.

It follows from Lemma 3 that any distribution with tails lighter than those in this example must satisfy Condition 2. On the other hand, the tails cannot get too much heavier.

Example 3.

Let $\mathbf{p}=\{p_k;k\ge 1\}$ be such that $p_k=C{k}^{-3}$, where $C>0$ is a normalizing constant. In this case, Condition 2 does not hold. However, Condition 1 does hold.

2.2. Results for the Jackknife Estimator

For any distribution $\mathbf{p}$, let $B_n=E\left(\widehat{H}\right)-H$ be the bias of the plug-in based on a sample of size n.

Condition 3.

The distribution, $\mathbf{p}=\{p_k;k\ge 1\}$, satisfies

$$\underset{n\to \infty }{lim}{n}^{3/2}\left(B_n-B_{n-1}\right)=0.$$

Theorem 3.

Let $\mathbf{p}=\{p_k;k\ge 1\}$ be a distribution, which is not uniform, and let ${\sigma }^2$ and ${\widehat{\sigma }}^2$ be as in (9). If Conditions 1 and 3 hold, then $\widehat{\sigma }\stackrel{p}{\to }\sigma$,

$$\frac{\sqrt{n}({\widehat{H}}_{JK}-H)}{\sigma }\stackrel{L}{⟶}N(0,1)$$

and

$$\frac{\sqrt{n}({\widehat{H}}_{JK}-H)}{\widehat{\sigma }}\stackrel{L}{⟶}N(0,1).$$

It is not clear to us whether Conditions 1 and 3 are equivalent or, if not, which is more stringent. For that reason, in the statement of Theorem 3, both conditions are imposed. The proof of the theorem uses the following lemma, which gives some insight into ${\widehat{B}}_{JK}$ and Condition 3.

Lemma 5.

For any probability distribution $\mathbf{p}=\{p_k;k\ge 1\}$, we have

$${\widehat{B}}_{JK}=\frac{n-1}{n}\sum _{i=1}^n\left(\widehat{H}-{\widehat{H}}^{\left(i\right)}\right)\ge 0$$

and

$$\mathrm{E}\left[{\widehat{B}}_{JK}\right]=(n-1)\left(B_n-B_{n-1}\right)\ge 0.$$

We now give a condition, which implies Condition 3 and tends to be easier to check.

Proposition 1.

If the distribution $\mathbf{p}=\{p_k;k\ge 1\}$ is such that there exists an $\epsilon \in (1/2,1)$ with ${\sum }_{k\ge 1}{p}_k^{1-\epsilon }<\infty$, then Condition 3 is satisfied.

We now give an example where this holds.

Example 4.

Let $\mathbf{p}=\{C{k}^{-(2+\delta )};k\ge 1\},$ where $\delta >0$ is fixed and $C>0$ is a normalizing constant. In this case, the assumption of Proposition 1 holds and thus Condition 3 is satisfied.

To see that the assumption of Proposition 1 holds in this case, fix $\epsilon \in (1/2,(1+\delta )/(2+\delta \left)\right)$. Note that $-(1+\delta /2)<-(2+\delta )(1-\epsilon )<-1$, and thus

$$\sum _{k\ge 1}{p}_k^{1-\epsilon }=C^{1-\epsilon }\sum _{k\ge 1}{k}^{-(2+\delta )(1-\epsilon )}<\infty .$$

3. Simulations

The main application of the asymptotic normality results given in this paper is the construction of asymptotic confidence intervals and hypothesis tests. For instance, if $\mathbf{p}$ satisfies the assumptions of Theorem 2, then an asymptotic $(1-\alpha )100\%$ confidence interval for H is given by

$$\begin{array}{c}\hfill \left({\widehat{H}}_{MM}-z_{\alpha /2}\frac{\widehat{\sigma }}{\sqrt{n}},{\widehat{H}}_{MM}+z_{\alpha /2}\frac{\widehat{\sigma }}{\sqrt{n}}\right),\end{array}$$

where $z_{\alpha /2}$ is a number such that $P(Z>z_{\alpha /2})=\alpha /2$ and Z is a standard normal random variable. Similarly, if the assumptions of Theorem 3 are satisfied, then we can replace ${\widehat{H}}_{MM}$ with ${\widehat{H}}_{JK}$, and if the assumptions of Lemma 1 are satisfied, then we can replace ${\widehat{H}}_{MM}$ with $\widehat{H}$. In this section, we give a small-scale simulation study to evaluate the finite sample performance of these confidence intervals.

For concreteness, we focus on the geometric distribution, which corresponds to

$$p_k=p{(1-p)}^{k-1},k=1,2,\cdots ,$$

where $p\in (0,1)$ is a parameter. The true entropy of this distribution is given by $H=-p^{-1}\left(plnp+(1-p)ln(1-p)\right)$. In this case, Conditions 1, 2, and 3 all hold. For our simulations, we took $p=0.5$. The simulations were performed as follows. We began by simulating a random sample of size n and used it to evaluated a $95\%$ confidence interval for the given estimator. We then checked to see if the true value of H was in the interval or not. This was repeated 5000 times and the proportion of times when the true value was in the interval was calculated. This proportion should be close to $0.95$ when the confidence interval works well. We repeated this for sample sizes ranging from 20 to 1000 in increments of 10. The results are given in Figure 1. We can see that the Miller–Madow and jackknife estimators consistently outperform the plug-in. It may be interesting to note that, although the proofs of Theorems 1–3 are based on showing that the bias correction term approaches zero, it does not mean that the bias correction term is not useful. On the contrary, bias correction improves the finite sample performance of the asymptotic confidence intervals.

Figure 1.

Effectiveness of the $95\%$ confidence intervals as a function of sample size. The plot on the top left gives the proportions for the Miller–Madow and the plug-in estimators, while the one on the top right gives the proportions for the jackknife and the plug-in estimators. The horizontal line is at $0.95$. The closer the proportion is to this line, the better the performance. The plot on the bottom left gives the proportion for Miller–Madow minus the proportion for the plug-in, while the one of the bottom right gives the proportion for the jackknife minus the proportion for the plug-in. The larger the value, the greater the improvement due to bias correction. Here, the horizontal line is at 0.

4. Discussion

In this paper, we gave sufficient conditions for the asymptotic normality of the Miller–Madow and the Jackknife estimators of entropy. While our focus is on the case of countably infinite alphabets, our results are formulated and proved in the case where the effective cardinality K may be finite or countably infinite. As such, they hold in the case of finite alphabets as well. In fact, for finite alphabets, Conditions 1–3 always hold and we have asymptotic normality so long as the underlying distribution is not uniform. The difficulty with the uniform distribution is that it is the unique distribution for which ${\sigma }^2$, as given by (9), is zero (see the discussion just below Condition 1). When the distribution is uniform, the asymptotic distribution is chi-squared with $(K-1)$ degrees of freedom (see [6]).

In general, we do not know if our conditions are necessary. However, they cover most distributions of interest. The only distributions, which they preclude, are ones with extremely heavy tails. However, in complete generality, Conditions 1–3 may look complicated, and they are easily checked in many situations. For instance, Condition 2 always holds when, for large enough k, $p_k\le C{k}^{-3-\delta }$ for some $C,\delta >0$, i.e., when

$$\sum _{k=1}^{\infty }{k}^2{p}_k<\infty .$$

If the alphabet $\mathcal{X}=\mathbb{N}$ is the set of natural numbers, then this is equivalent to the distribution $\mathbf{p}$ having a finite variance. Similarly, Conditions 1 and 3 both holding is the case when, for large enough k, $p_k\le C{k}^{-2-\delta }$ for some $C,\delta >0$, i.e., when

$$\sum _{k=1}^{\infty }k{p}_k<\infty .$$

If the alphabet $\mathcal{X}=\mathbb{N}$ is the set of natural numbers, then this is equivalent to the distribution $\mathbf{p}$ having a finite mean.

5. Proofs

Proof of Lemma 2.

Without loss of generality, assume that $C>1$ and thus that $lnC>0$. Let $f\left(x\right)=xlnx$ for $x\in (0,1)$. It is readily checked that f is negative and decreasing for $x\in (0,e^{-1})$. Since $C{p}_k^{\prime }\to 0$ as $k\to \infty$, it follows that $C{p}_k^{\prime }<e^{-1}$ for large enough k. Now, let $K\left(n\right)$ be the sequence that works for ${\mathbf{p}}^{\prime }$ in Condition 1. For large enough n,

$$\begin{array}{ccc}\hfill 0& \ge & \sqrt{n}\sum _{k\ge K\left(n\right)}{p}_{k}ln{p}_k\ge C\sqrt{n}\sum _{k\ge K\left(n\right)}{p}_k^{\prime }ln\left(C{p}_k^{\prime }\right)\hfill \\ & \ge & Cln\left(C\right)\sqrt{n}\sum _{k\ge K\left(n\right)}{p}_k^{\prime }+C\sqrt{n}\sum _{k\ge K\left(n\right)}{p}_k^{\prime }ln\left(p_k^{\prime }\right)\hfill \\ & \ge & C\left(ln\left(C\right)+1\right)\sqrt{n}\sum _{k\ge K\left(n\right)}{p}_k^{\prime }ln\left(p_k^{\prime }\right)⟶0.\hfill \end{array}$$

Similarly, the function $g\left(x\right)=x{ln}^{2}x$ for $x\in (0,1)$ is positive and increasing for $x\in (0,e^{-2})$. Thus, there is an integer $M>0$ such that if $k\ge M$, then $C{p}_k^{\prime }<e^{-2}$ and

$$\begin{array}{ccc}\hfill 0\le \sum _{k\ge 1}{p}_k{ln}^2{p}_k& \le & \sum _{k=1}^{M-1}{p}_k{ln}^2{p}_k+C\sum _{k=M}^{\infty }{p}_k^{\prime }{ln}^2\left(C{p}_k^{\prime }\right)\hfill \\ & =& \sum _{k=1}^{M-1}{p}_k{ln}^2{p}_k+C\sum _{k=M}^{\infty }{p}_k^{\prime }{ln}^2\left(p_k^{\prime }\right)\hfill \\ & & +C{ln}^2\left(C\right)\sum _{k=M}^{\infty }{p}_k^{\prime }+2Cln\left(C\right)\sum _{k=M}^{\infty }{p}_k^{\prime }ln\left(p_k^{\prime }\right)<\infty ,\hfill \end{array}$$

as required. ☐

To prove Theorem 2, the following Lemma is needed.

Lemma 6.

If Condition 2 holds, then there exists a $K_1>0$ such that for all $k\ge K_1$

$$p_k\le \frac{1}{a\left(k\right)b\left(k\right)k^3}\le 1-{\left(1-\frac{2}{kb\left(k\right)}\right)}^{\frac{1}{a\left(k\right)k^2}}.$$ (14)

Proof. 

Observing that $e^{-x}\ge 1-x$ holds for all real x and that ${lim}_{x\to 0}(1-e^{-x})/x=1$, we have $e^{-2/\left(kb\right(k\left)\right)}\ge 1-2/\left(kb\left(k\right)\right)$, and hence

$$\begin{array}{c}\hfill 1-{\left(1-\frac{2}{kb\left(k\right)}\right)}^{\frac{1}{a\left(k\right)k^2}}\ge 1-e^{-\frac{2}{a\left(k\right)b\left(k\right)k^3}}\sim \frac{2}{a\left(k\right)b\left(k\right)k^3}.\end{array}$$

This implies that there is a $K_1>0$ such that for all $k\ge K_1$ (11) holds and

$$\frac{\frac{2}{a\left(k\right)b\left(k\right)k^3}}{1-{\left(1-\frac{2}{kb\left(k\right)}\right)}^{\frac{1}{a\left(k\right)k^2}}}\le 2.$$

It follows that, for such k,

$$p_k\le \frac{1}{a\left(k\right)b\left(k\right)k^3}=.5\frac{\frac{2}{a\left(k\right)b\left(k\right)k^3}}{1-{\left(1-\frac{2}{kb\left(k\right)}\right)}^{\frac{1}{a\left(k\right)k^2}}}\left[1-{\left(1-\frac{2}{kb\left(k\right)}\right)}^{\frac{1}{a\left(k\right)k^2}}\right]\le 1-{\left(1-\frac{2}{kb\left(k\right)}\right)}^{\frac{1}{a\left(k\right)k^2}}$$

as required. ☐

Proof of Theorem 2.

By Theorem 1, it suffices to show that Condition 2 implies that both Condition 1 and $\sqrt{n}{\widehat{B}}_{MM}\stackrel{p}{\to }0$ hold. The fact that Condition 2 implies Condition 1 follows by Example 3, Lemmas 2 and 3. We now show that $\sqrt{n}{\widehat{B}}_{MM}\stackrel{p}{\to }0$.

Fix ${\epsilon }_0\in (0,\epsilon )$. From (12) and the facts that $a\left(k\right)$ is positive, eventually nondecreasing, and approaches infinity, it follows that

$$\begin{array}{ccc}\hfill \underset{k\to \infty }{lim\; sup}\frac{{\left(a\left(k\right)\right)}^{2{\epsilon }_0}}{a\left(\frac{\sqrt{k}}{{\left(a\left(k\right)\right)}^{{\epsilon }_0}}\right)}& =& \underset{k\to \infty }{lim\; sup}{\left(a\left(k\right)\right)}^{-2(\epsilon -{\epsilon }_0)}\frac{{\left(a\left(k\right)\right)}^{2\epsilon }}{a\left(\frac{\sqrt{k}}{{\left(a\left(k\right)\right)}^{{\epsilon }_0}}\right)}\hfill \\ & \le & \underset{k\to \infty }{lim\; sup}{\left(a\left(k\right)\right)}^{-2(\epsilon -{\epsilon }_0)}\frac{{\left(a\left(k\right)\right)}^{2\epsilon }}{a\left(\frac{\sqrt{k}}{{\left(a\left(k\right)\right)}^{\epsilon }}\right)}=0.\hfill \end{array}$$ (15)

Let $K_2$ be a positive integer such that, for all $n\ge K_2$, $a\left(n\right)$ is nondecreasing, and let $r_n=\left(\sqrt{n}/{\left(a\left(n\right)\right)}^{{\epsilon }_0}\right)\vee K_3$, where $K_3=K_1\vee K_2$ and $K_1$ is as in Lemma 6. It follows that

$$\begin{array}{cc}\hfill E\left(\sqrt{n}{\widehat{B}}_{MM}\right)& =\sqrt{n}E\left(\frac{\widehat{m}-1}{2n}\right)\le \frac{1}{\sqrt{n}}E\left(\widehat{m}\right)=\frac{1}{\sqrt{n}}\sum _{k\ge 1}\left[1-{\left(1-p_k\right)}^n\right]\hfill \\ & \le \frac{1}{\sqrt{n}}\sum _{k\le r_n}1+\frac{1}{\sqrt{n}}\sum _{k>r_n}\left[1-{\left(1-\frac{2}{kb\left(k\right)}\right)}^{\frac{n}{a\left(k\right)k^2}}\right]=:S_1+S_2.\hfill \end{array}$$

We have

$$\begin{array}{cc}\hfill S_1& \le \frac{r_n}{\sqrt{n}}=\left({\left(a\left(n\right)\right)}^{-{\epsilon }_0}\right)\vee \frac{K_3}{\sqrt{n}}\to 0;and\hfill \\ \hfill S_2& \le \frac{1}{\sqrt{n}}\sum _{k>r_n}\left[1-{\left(1-\frac{2}{kb\left(k\right)}\right)}^{\frac{n}{a\left(r_n\right)r_n^2}}\right]\le \frac{1}{\sqrt{n}}\sum _{k>r_n}\left[1-{\left(1-\frac{2}{kb\left(k\right)}\right)}^{\frac{{\left(a\left(n\right)\right)}^{2{\epsilon }_0}}{a\left(\frac{\sqrt{n}}{{\left(a\left(n\right)\right)}^{{\epsilon }_0}}\right)}}\right].\hfill \end{array}$$

By (15), it follows that, for large enough $n,$

$$S_2\le \frac{1}{\sqrt{n}}\sum _{k>r_n}\left[1-\left(1-\frac{2}{kb\left(k\right)}\right)\right]=\frac{1}{\sqrt{n}}\sum _{k>r_n}\frac{2}{kb\left(k\right)}\le \left(\sum _{k\ge 1}\frac{1}{kb\left(k\right)}\right)\frac{2}{\sqrt{n}}\to 0.$$

From here, Markov’s inequality implies that $\sqrt{n}{\widehat{B}}_{MM}\stackrel{p}{⟶}0$. ☐

Proof of Lemma 4.

First note that ${ln}^{(r-1)}\left(0.5lnn-0.5{ln}^{\left(v\right)}n\right)\sim {ln}^{(r-1)}\left(0.5lnn\right)$ for any $v\ge 2$ and $r\ge 1$. This can be shown by induction on r. Specifically, the result is immediate for $r=1$. If the result is true for $r=m$, then for $r=m+1$

$$\begin{array}{ccc}\hfill {ln}^{\left(m\right)}\left(0.5lnn-0.5{ln}^{\left(v\right)}n\right)& =& ln\left({ln}^{(m-1)}\left(0.5lnn-0.5{ln}^{\left(v\right)}n\right)\right)\hfill \\ & =& ln\left(\frac{{ln}^{(m-1)}\left(0.5lnn-0.5{ln}^{\left(v\right)}n\right)}{{ln}^{(m-1)}\left(0.5lnn\right)}\right)+{ln}^{\left(m\right)}\left(0.5lnn\right)\hfill \\ & \sim & {ln}^{\left(m\right)}\left(0.5lnn\right).\hfill \end{array}$$

It follows that for $r\ge 2$

$$\begin{array}{ccc}\hfill \underset{n\to \infty }{lim}\frac{{ln}^{\left(r\right)}n}{{ln}^{\left(r\right)}\left(\sqrt{n/\left({ln}^{\left(r\right)}n\right)}\right)}& =& \underset{n\to \infty }{lim}\frac{{ln}^{\left(r\right)}n}{{ln}^{(r-1)}\left(0.5lnn-0.5{ln}^{(r+1)}n\right)}\hfill \\ & =& \underset{n\to \infty }{lim}\frac{{ln}^{(r-1)}\left(lnn\right)}{{ln}^{(r-1)}\left(0.5lnn\right)}=1,\hfill \end{array}$$

where the final equality follows by the fact that ${ln}^{(r-1)}\left(x\right)$ is a slowly varying function. Recall that a positive-valued function ℓ is called slowly varying if for any $t>0$

$$\underset{x\to \infty }{lim}\frac{\ell \left(xt\right)}{\ell \left(x\right)}=1$$

(see [11] for a standard reference). To see that ${ln}^{(r-1)}\left(x\right)$ is slowly varying, note that ln is slowly varying and that compositions of slowly varying functions are slowly varying by Proposition 1.3.6 in [11]. ☐

Proof of Lemma 5.

Observing the convention that $0ln0=0$,

$$\begin{array}{cc}\hfill \sum _{i=1}^n{\widehat{H}}^{\left(i\right)}& =\sum _{k\ge 1}\sum _{i:X_i={\ell }_k}{\widehat{H}}^{\left(i\right)}\hfill \\ & =\sum _{k\ge 1}{Y}_k\left(-\frac{Y_k-1}{n-1}ln\frac{Y_k-1}{n-1}-\sum _{j:j\ge 1,j\ne k}\frac{Y_j}{n-1}ln\frac{Y_j}{n-1}\right)\hfill \\ & =\sum _{k\ge 1}{Y}_k\left[-\frac{Y_k-1}{n-1}\left(ln\frac{Y_k-1}{Y_k}+ln\frac{Y_k}{n-1}\right)-\sum _{j:j\ge 1,j\ne k}\frac{Y_j}{n-1}ln\frac{Y_j}{n-1}\right]\hfill \\ & =\sum _{k\ge 1}{Y}_k\left[-\frac{Y_k-1}{n-1}ln\frac{Y_k-1}{Y_k}+\frac{1}{n-1}ln\frac{Y_k}{n-1}-\sum _{j\ge 1}\frac{Y_j}{n-1}ln\frac{Y_j}{n-1}\right]\hfill \\ & =-\frac{1}{n-1}\sum _{k\ge 1}{Y}_k(Y_k-1)ln\frac{Y_k-1}{Y_k}+\sum _{k\ge 1}\frac{Y_k}{n-1}ln\frac{Y_k}{n-1}-\sum _{k\ge 1}{Y}_k\sum _{j\ge 1}\frac{Y_j}{n-1}ln\frac{Y_j}{n-1}\hfill \\ & =-\frac{1}{n-1}\sum _{k\ge 1}{Y}_k(Y_k-1)ln\frac{Y_k-1}{Y_k}-(n-1)\sum _{k\ge 1}\frac{Y_k}{n-1}ln\frac{Y_k}{n-1}\hfill \\ & =-\frac{1}{n-1}\sum _{k\ge 1}{Y}_k(Y_k-1)ln\frac{Y_k-1}{Y_k}-\sum _{k\ge 1}{Y}_k\left(ln\frac{Y_k}{n}+ln\frac{n}{n-1}\right)\hfill \\ & =-\frac{1}{n-1}\sum _{k\ge 1}{Y}_k(Y_k-1)ln\frac{Y_k-1}{Y_k}-nln\frac{n}{n-1}+n\widehat{H}.\hfill \end{array}$$

Therefore,

$$\begin{array}{cc}\hfill \sum _{i=1}^n\left(\widehat{H}-{\widehat{H}}^{\left(i\right)}\right)=& \frac{1}{n-1}\sum _{k\ge 1}{Y}_k(Y_k-1)ln\frac{Y_k-1}{Y_k}+nln\frac{n}{n-1}\hfill \\ \hfill =& \frac{1}{n-1}\sum _{k\ge 1}{Y}_k\left[(Y_k-1)ln\frac{Y_k-1}{Y_k}+(n-1)ln\frac{n}{n-1}\right]\hfill \\ \hfill =& \frac{1}{n-1}\sum _{k\ge 1}{Y}_k\left[(n-1)ln\frac{n}{n-1}-(Y_k-1)ln\frac{Y_k}{Y_k-1}\right].\hfill \end{array}$$

It suffices to show that for any $y\in \{1,2,\cdots ,n\}$

$$(y-1)lny-(y-1)ln(y-1)\le (n-1)lnn-(n-1)ln(n-1).$$ (16)

Towards that end, first note that the inequality of (16) holds for $y=1$. Now, let

$$f\left(y\right)=(y-1)lny-(y-1)ln(y-1)$$

and, therefore, letting $s=1-1/y$,

$$f^{\prime }\left(y\right)=ln\frac{y}{y-1}-\frac{1}{y}=\left(1-\frac{1}{y}-1\right)-ln\left(1-\frac{1}{y}\right)=(s-1)-lns.$$

Since $s-1\ge lns$ for all $s>0$ (see e.g., 4.1.36 in [12]), $f\left(y\right)\ge 0$ for all y, $1<y\le n$, which implies (16).

For the second part, we use the first part to get

$$0\le E\left[\sum _{i=1}^n\left(\widehat{H}-{\widehat{H}}^{\left(i\right)}\right)\right]=nE[\widehat{H}-H]-\sum _{i=1}^{n}E[{\widehat{H}}^{\left(i\right)}-H]=n(B_n-B_{n-1}),$$

where the last equality follows from the facts that for each i, ${\widehat{H}}^{\left(i\right)}$ is a plug-in estimator of H based on a sample of size $(n-1)$ and that $E\left[{\widehat{H}}^{\left(i\right)}\right]$ does not depend on i due to symmetry. From here, the result follows. ☐

Proof of Theorem 3.

By Theorem 1, it suffices to show $\sqrt{n}{\widehat{B}}_{JK}\stackrel{p}{⟶}0$. Note that, by Lemma 5,

$$0\le E\left[\sqrt{n}{\widehat{B}}_{JK}\right]=\sqrt{n}(n-1)(B_n-B_{n-1})\sim n^{3/2}(B_n-B_{n-1})\to 0,$$

where the convergence follows by Condition 2. From here, the result follows by Markov’s inequality. ☐

To prove Proposition 1, we need several lemmas, which may be of independent interest.

Lemma 7.

Let $S_n$ and $S_{n-1}$ be binomial random variables with parameters $(n,p)$ and $(n-1,p)$, respectively. If $n\ge 2$ and $p\in (0,1)$, then

$$E(S_{n}ln{S}_n)=E\left(npln(S_{n-1}+1)\right).$$ (17)

The proof is given on page 178 in [7].

Lemma 8.

Let $X_1,\cdots ,X_n$ be $iid$ Bernoulli random variables with parameter $p\in (0,1)$. For $m=1,\cdots ,n$ let $S_m={\sum }_{i=1}^m{X}_i$, ${\widehat{p}}_m=S_m/m$, ${\widehat{h}}_m=-{\widehat{p}}_{m}ln{\widehat{p}}_m$, and ${\Delta }_m=E[{\widehat{h}}_m-{\widehat{h}}_{m-1}]$. Then,

$${\Delta }_n=E[{\widehat{h}}_n-{\widehat{h}}_{n-1}]\le \frac{p(2-p)}{(n-1)\left[\right(n-2)p+2]}\le \frac{2p}{(n-1)\left[\right(n-2)p+2]}.$$ (18)

Proof. 

Applying Lemma 7 to ${\Delta }_n$ gives

$$\begin{array}{ccc}\hfill {\Delta }_n& =& pln\left(\frac{n}{n-1}\right)+pE\left[ln\left(\frac{S_{n-2}+1}{S_{n-1}+1}\right)\right]\hfill \\ & =& pln\left(\frac{n}{n-1}\right)+pE\left[ln\left(\frac{S_{n-2}+1}{S_{n-2}+X_{n-1}+1}\right)\right].\hfill \end{array}$$

Conditioning on $X_{n-1}$ gives

$${\Delta }_n=pln\left(\frac{n}{n-1}\right)+p^{2}E\left[ln\left(\frac{S_{n-2}+1}{S_{n-2}+2}\right)\right].$$

Noting that $f\left(x\right)=ln(x/(x+1\left)\right)$ is a concave function for $x>0$, by Jensen’s inequality,

$${\Delta }_n\le pln\left(\frac{n}{n-1}\right)+p^{2}ln\left(\frac{(n-2)p+1}{(n-2)p+2}\right).$$

Applying the following inequalities (both of which follow from 4.1.36 in [12]) to the terms of the above expression,

$$ln\left(\frac{x}{x-1}\right)<\frac{1}{x-1} \mathrm{for} x>1\mathrm{and}ln\left(\frac{x}{x+1}\right)<-\frac{1}{x+1} \mathrm{for} x>0,$$

it follows that

$${\Delta }_n\le \frac{p}{n-1}-\frac{p^2}{(n-2)p+2}=\frac{p(2-p)}{(n-1)\left[\right(n-2)p+2]},$$

which completes the proof. ☐

For fixed $\epsilon >0$, rewriting the upper bound of (18) gives

$$\frac{2p}{(n-1)\left[\right(n-2)p+2]}=\frac{2p^{1-\epsilon }}{n-1}\left\{\frac{p^{\epsilon }}{(n-2)p+2}\right\}=:\frac{2p^{1-\epsilon }}{n-1}\left\{g(n,p,\epsilon )\right\}.$$ (19)

Lemma 9.

For any $\epsilon \in (0,1)$ and $n\ge 3$, there exists a $p_0\in (0,1)$ such that $g(n,p,\epsilon )$ defined in (19) is maximized at $p_0$ and

$$0\le g(n,p_0,\epsilon )=\mathcal{O}\left(n^{-\epsilon }\right).$$ (20)

Proof. 

Taking the derivative of $lng(n,p,\epsilon )$ with respect to p gives

$$\frac{\partial }{\partial p}lng(n,p,\epsilon )=\frac{\epsilon }{p}-\frac{n-2}{(n-2)p+2}.$$

It is readily checked that this equals zero only at

$$p_0=\frac{2\epsilon }{(1-\epsilon )(n-2)}$$

and is positive for $0<p<p_0$ and negative for $p_0<p<1$. Thus, $p_0$ is the global maximum. For a fixed $\epsilon$, we have

$$\begin{array}{c}\hfill g(n,p_0,\epsilon )=\frac{{\left(2\epsilon \right)}^{\epsilon }}{{(1-\epsilon )}^{\epsilon }{(n-2)}^{\epsilon }[\frac{2\epsilon }{(1-\epsilon )}+2]}={\left(\frac{\epsilon }{n-2}\right)}^{\epsilon }{\left(\frac{1-\epsilon }{2}\right)}^{1-\epsilon }=\mathcal{O}\left(n^{-\epsilon }\right),\end{array}$$

as required. ☐

Proof of Proposition 1.

For every k and every $m\le n$, let

$$S_{m,k}=\sum _{i=1}^{m}1[X_i={\ell }_k]\mathrm{and}{\widehat{H}}_m=-\sum _{k\ge 1}\frac{S_{m,k}}{m}ln\left(\frac{S_{m,k}}{m}\right)$$

be the observed letter counts and the plug-in estimator of entropy based on the first m observations. Thus, $S_{m,k}=Y_k$ and ${\widehat{H}}_n=\widehat{H}$. We are interested in evaluating

$$\begin{array}{cc}\hfill B_n-B_{n-1}& =E\left({\widehat{H}}_n-{\widehat{H}}_{n-1}\right)\hfill \\ & =\sum _{k\ge 1}E\left[\frac{S_{n-1,k}}{n-1}ln\left(\frac{S_{n-1,k}}{n-1}\right)-\frac{S_{n,k}}{n}ln\left(\frac{S_{n,k}}{n}\right)\right]\hfill \\ & \le \sum _{k\ge 1}\frac{2p_k}{(n-1)[(n-2)p_k+2]}\hfill \\ & =2\sum _{k\ge 1}{p}_k^{1-\epsilon }\frac{g(n,p_k,\epsilon )}{(n-1)},\hfill \end{array}$$

where the third line follows by Lemma 8. Now, applying Lemmas 5 and 9 gives

$$\begin{array}{cc}\hfill 0\le n^{3/2}(B_n-B_{n-1})& \le 2n^{3/2}\sum _{k\ge 1}{p}_k^{1-\epsilon }\frac{g(n,p_k,\epsilon )}{(n-1)}\hfill \\ & \le n^{3/2}\frac{g(n,p_0,\epsilon )}{n-1}\sum _{k\ge 1}{p}_k^{1-\epsilon }=\mathcal{O}\left(n^{1/2-\epsilon }\right),\hfill \end{array}$$

which converges to zero when $\epsilon \in (1/2,1)$. ☐

Author Contributions

C.C., M.G., A.S., J.Z. and Z.Z. contributed to the proofs; C.C., M.G. and J.Z. contributed editorial input; Z.Z. wrote the paper.

Conflicts of Interest

The authors declare no conflict of interest.