Minimax Rate-optimal Estimation of KL Divergence between Discrete Distributions

Abstract

We refine the general methodology in [1] for the construction and analysis of essentially minimax estimators for a wide class of functionals of finite dimensional parameters, and elaborate on the case of discrete distributions with support size S comparable with the number of observations n. Specifically, we determine the “smooth” and “non-smooth” regimes based on the confidence set and the smoothness of the functional. In the “non-smooth” regime, we apply an unbiased estimator for a “suitable” polynomial approximation of the functional. In the “smooth” regime, we construct a bias corrected version of the Maximum Likelihood Estimator (MLE) based on Taylor expansion.

We apply the general methodology to the problem of estimating the KL divergence between two discrete distributions from empirical data. We construct a minimax rate-optimal estimator which is adaptive in the sense that it does not require the knowledge of the support size nor the upper bound on the likelihood ratio. Moreover, the performance of the optimal estimator with n samples is essentially that of the MLE with n ln n samples, i.e., the effective sample size enlargement phenomenon holds.

I. Introduction and Main Results

The Kullback-Leibler (KL) divergence defined by [2]

$$D(P\Vert Q)=\{\begin{array}{cc}{\sum }_{i=1}^S{p}_i\text{ ln }\frac{p_i}{q_i}\hfill & \text{if }P\ll Q,\hfill \\ +\infty \hfill & \text{otherwise}.\hfill \end{array}$$ (1)

emerges as a crucial measure of the discrepancy between two discrete distributions P = (p₁, ⋯, p_S) and Q = (q₁, ⋯, q_S). Like the entropy and mutual information [3], the KL divergence is another key information theoretic measure widely used in data compression, communications, machine learning, and many other disciplines.

Given jointly independent m samples from P = (p₁, ⋯, p_S) and n samples from Q = (q₁, ⋯, q_S) over some common alphabet of size S, we would like to estimate the KL divergence D(P‖Q). Throughout we use the squared error loss, i.e., the risk function for any estimator D̂ is defined as

$$L(\hat{D};P,Q)\triangleq {𝔼}_{(P,Q)}|\hat{D}-D(P\Vert Q){|}^2.$$ (2)

The maximum risk of an estimator D̂, and the minimax risk in estimating D(P‖Q) are respectively defined as

$$R_{\text{maximum}}(\hat{D};𝒰)\triangleq \underset{(P,Q)\in 𝒰}{\text{sup}}L(\hat{D};P,Q),$$ (3)

$$R_{\text{minimax}}(𝒰)\triangleq \underset{\hat{D}}{\text{inf}}\underset{(P,Q)\in 𝒰}{\text{sup}}L(\hat{D};P,Q)$$ (4)

where 𝒰 is a given family of probability measures (P, Q), and the infimum is taken over all possible estimators D̂.

There has been several attempts to estimate the KL divergence for the continuous case, see [4]–[7] and references therein. Note that these approaches usually do not operate in the minimax framework, and only argue about consistency but not rates of convergence, unless strong smoothness conditions on the densities are imposed to achieve the parametric rate (i.e., $\frac{1}{n}$ in mean squared error). In the discrete setting, [8] and [9] proved consistency of some specific estimators without arguing minimax optimality. We note that in the discrete case, if the alphabet size S is fixed and the number of samples m, n go to infinity, the standard Hájek–Le Cam theory of classical asymptotics shows that the plug-in approach is asymptotically efficient [10, Thm. 8.11, Lemma 8.14]. The key challenge we face in the discrete setting is the regime where the support size S can be comparable to or even larger than the number of observations m, n, which classical asymptotics fails to address.

Now we consider the estimation problem of KL divergence between discrete distributions in a large-alphabet size setting. To avoid D(P‖Q) being infinity, we consider the following parameter set with bounded likelihood ratio:

$${𝒰}_{S,u(S)}=\{(P,Q):P,Q\in {ℳ}_S,\frac{p_i}{q_i}\le u(S),\forall i\}$$ (5)

where ℳ_S denotes the set of all probability measures with support size at most S, u(S) ≥ c is an upper bound on the likelihood ratio, and c > 1 is some universal constant.

The main result of this paper is as follows.

Theorem 1

For sample sizes m ≳ S/ ln S, n ≳ Su(S)/ ln S, and u(S) ≳ (ln S)², ln S ≳ max{ln m, ln n}, we have

$$\underset{\hat{D}}{\text{inf}}\underset{(P,Q)\in {𝒰}_{S,u(S)}}{\text{sup}}{𝔼}_{(P,Q)}{(\hat{D}-D(P\Vert Q))}^2\asymp {(\frac{S}{m\text{ ln }m}+\frac{\mathit{\text{Su}}(S)}{n\text{ ln }n})}^2+\frac{{(\text{ln }u(S))}^2}{m}+\frac{u(S)}{n}$$ (6)

and our estimator in Section III-B achieves this bound, and it is adaptive in the sense that it does not require the knowledge of S nor u(S).

The following corollary is a direct result of Theorem 1. Note that ln S ≳ ln n and $n\gtrsim \frac{\mathit{\text{Su}}(S)}{\text{ln }S}$ imply that ln u(S) ≲ ln S.

Corollary 1

The worst-case mean squared error of the optimal estimator for the KL divergence vanishes if and only if m ≫ S/ ln S and n ≫ Su(S)/ ln S.

Next we consider the performance of the plug-in approach and check if it is minimax rate-optimal. Denote by P_m = (p̂₁, ⋯, p̂_S), Q_n = (q̂₁, ⋯, q̂_S) the empirical distributions of P and Q, respectively. To avoid the direct plug-in estimator D(P_m‖Q_n) being infinity, we adopt the modification $Q_n^{\prime }=(\text{max}\{\frac{1}{n},{\hat{q}}_1\},\cdots ,\text{max}\{\frac{1}{n},{\hat{q}}_S\})$ and use the estimator $D(P_m\Vert Q_n^{\prime })$ to estimate D(P‖Q). The performance of the modified plug-in approach is shown in the following theorem.

Theorem 2

For m ≳ S, n ≳ Su(S), the modified plug-in estimator $D(P_m\Vert Q_n^{\prime })$ satisfies

$$\underset{(P,Q)\in {𝒰}_{S,u(S)}}{\text{sup}}{𝔼}_{(P,Q)}{(D(P_n\Vert Q_n^{\prime })-D(P\Vert Q))}^2\asymp {(\frac{S}{m}+\frac{\mathit{\text{Su}}(S)}{n})}^2+\frac{{(\text{ln }u(S))}^2}{m}+\frac{u(S)}{n}.$$ (7)

The following corollary on the minimum sample complexity is immediate.

Corollary 2

The worst-case mean squared error of the modified plug-in estimator $D(P_n\Vert Q_n^{\prime })$ vanishes if and only if m ≫ max{S, (ln u(S))²} and n ≫ Su(S).

Hence, compared with the mean squared error or the minimum sample complexity of the modified plug-in approach, the optimal estimator enjoys a logarithmic improvement. Note that (ln u(S))² ≲ (ln S)² ≪ S is negligible under the condition in Theorem 1. Specifically, the optimal estimator with (m, n) samples achieves the performance of the plug-in approach with (m ln m, n ln n) samples, which is another instantiation of the effective sample size enlargement phenomenon termed in [1].

Notations: a_n ≲ b_n means ${\text{sup}}_n\frac{a_n}{b_n}<\infty$, a_n ≳ b_n means b_n ≲ a_n, and a_n ≍ b_n means both a_n ≲ b_n and b_n ≲ a_n.

II. Approximation: the General Recipe

The estimation of KL divergence belongs to a large family of functional estimation problems: consider estimating of G(θ) of a parameter θ ∈ Θ ⊂ ℝ^p for an experiment {P_θ : θ ∈ Θ}. There has been a recent wave of study on functional estimation of high dimensional parameters, e.g., the scaled ℓ₁ norm $\frac{1}{n}{\sum }_{i=1}^n|{\theta }_i|$ in the Gaussian model [11], the Shannon entropy ${\sum }_{i=1}^S-p_i$ ln p_i [1], [12]–[14], the mutual information [1], the power sum function ${\sum }_{i=1}^S{p}_i^{\alpha }$ [1], the Rényi entropy $\frac{\text{ln }{\sum }_{i=1}^S{p}_i^{\alpha }}{1-\alpha }$ [15] and the L₁ distance ${\sum }_{i=1}^S|p_i-q_i|$ [16] in Multinomial and Poisson models. Moreover, the effective sample size enlargement phenomenon holds in all these examples: the performance of the minimax estimators with n samples is essentially that of the plug-in approach with n ln n samples.

The optimal estimators in the previous examples all essentially follow the general methodology of Approximation proposed in [1]: suppose θ̂_n is a consistent estimator for θ, where n is the number of observations. Suppose the functional G(θ) is analytic everywhere except at θ ∈ Θ₀. The following two-step procedure is conducted in estimating G(θ).

1. Classify the Regime: Compute θ̂_n, and declare that we are in the “non-smooth” regime if θ̂_n is “close” enough to Θ₀. Otherwise declare we are in the “smooth” regime;

2. Estimate:

   • If θ̂_n falls in the “smooth” regime, use an estimator “similar” to G(θ̂_n) to estimate G(θ);
   • If θ̂_n falls in the “non-smooth” regime, replace the functional G(θ) in the “non-smooth” regime by an approximation G_appr(θ) (another functional) which can be estimated without bias, then apply an unbiased estimator for the functional G_appr(θ).

Clear and simple as it may sound, this methodology still has a few drawbacks and ambiguities:

• Question 1 How to determine the “non-smooth” regime? What is the size of it?

• Question 2 If θ̂_n falls in the “non-smooth” regime, in which region should G_appr(θ) be a good approximation of G(θ) (e.g., the whole domain Θ, or a proper neighborhood of θ̂_n)?

• Question 3 If θ̂_n falls in the “smooth” regime, how to construct an estimator “similar” to G(θ̂_n)?

Other questions, such as what type/degree of approximation G_appr(θ) should be used, were answered in more detail in [1]. All these questions were partially answered in [1] and [16], but the answer to Question 1 changes in general when the domain of θ̂_n differs from that of θ, and further elaborations are also necessary for answering Question 2. As for Question 3, the previous approach can only handle order-one bias correction, while for some problems bias correction with an arbitrary order is proved to be necessary [17]. Before answering these questions, we begin with a formal definition of confidence set in statistical experiments, which is motivated by [16].

Definition 1 (Confidence Set)

Consider a statistical model (P_θ)_θ∈Θ and an estimator θ̂ ∈ Θ̂ ⊃ Θ of θ. A (1 − r)-confidence set is a collection of sets {U(x) ⊂ Θ}_x∈Θ̂ with

$$\underset{\theta \in \mathrm{\Theta }}{\text{sup}}{\mathbb{P}}_{\theta }(\theta \notin U(\hat{\theta }))\le r.$$ (8)

By definition, the true parameter θ is “localized” at U(θ̂) with probability at least 1 − r, which always exists (e.g., U(θ̂) ≡ Θ). In practice, we seek for confidence sets which are as small as possible. For example, in the Binomial model np̂ ~ 𝖡(n, p) with Θ̂ = [0, 1] and any Θ ⊂ Θ̂, for r_n ≍ n^−A, the collection {U(x)}_x∈[0,1] with $U(x)=\mathrm{\Theta }\cap W(\frac{c_1\text{ ln }n}{n},x)$ is a (1 − r_n)-confidence set given that the universal constant c₁ > 0 is large enough, where

$$W(t,z)\triangleq \{\begin{array}{cc}[0,t]\hfill & \text{if }z\le t,\hfill \\ [z-\sqrt{\mathit{\text{tz}}},z+\sqrt{\mathit{\text{tz}}}]\hfill & \text{if }t<z\le 1.\hfill \end{array}$$ (9)

A. Answer to Question 1

First we remark that we should distinguish the “smooth” (resp. “non-smooth”) regime of θ and that of θ̂_n: we determine the corresponding regimes of θ first, and then localize θ using θ̂_n since θ cannot be observed. Hence, in the first step, to make the plug-in approach G(θ̂_n) work for the estimation of G(θ), it must be ensured that with high probability θ̂_n does not fall into the non-analytic region of G(·), which is denoted by Θ̂₀ ⊂ Θ̂. Then we choose some $r_n^{(1)}\ge r_n^{(2)}$ of negligible value compared with the minimax risk, and find $(1-r_n^{(1)}),(1-r_n^{(2)})$-confidence sets {U₁(x)}_x∈Θ̂ and {U₂(x)}_x∈Θ̂. Then define the “smooth” and “non-smooth” regime of θ as

$${\mathrm{\Theta }}_{\mathrm{s}}\triangleq \mathrm{\Theta }-{\cup }_{x\in {\hat{\mathrm{\Theta }}}_0}{U}_1(x)$$ (10)

$${\mathrm{\Theta }}_{\text{ns}}\triangleq {\cup }_{x\in {\hat{\mathrm{\Theta }}}_0}{U}_2(x)$$ (11)

respectively. Note that Θ_s and Θ_ns may overlap. The key insight is that, if θ ∈ Θ_s and θ̂_n ∈ Θ̂₀, we have U₁(θ̂_n) ⊂ ∪_x∈Θ̂0 U₁(x) = Θ − Θ_s and thus θ ∉ U₁(θ̂_n). As a result,

$$\underset{\theta \in \mathrm{\Theta }\mathrm{s}}{\text{sup}}{\mathbb{P}}_{\theta }({\hat{\theta }}_n\in {\hat{\mathrm{\Theta }}}_0)\le \underset{\theta \in {\mathrm{\Theta }}_{\mathrm{s}}}{\text{sup}}{\mathbb{P}}_{\theta }(\theta \notin U_1({\hat{\theta }}_n))\le r_n^{(1)}$$

i.e., when θ ∈ Θ_s, with high probability θ̂_n does not fall into the non-analytic region of G(·), i.e., plug-in approach works.

Next define the “smooth” and “non-smooth” regime of θ̂_n:

$${\hat{\mathrm{\Theta }}}_{\mathrm{s}}\triangleq \{{\hat{\theta }}_n\in \hat{\mathrm{\Theta }}:{\mathrm{\Theta }}_{\mathrm{s}}\supset U_1({\hat{\theta }}_n)\}$$ (12)

$${\hat{\mathrm{\Theta }}}_{\text{ns}}\triangleq \{{\hat{\theta }}_n\in \hat{\mathrm{\Theta }}:{\mathrm{\Theta }}_{\text{ns}}\supset U_2({\hat{\theta }}_n)\}.$$ (13)

We assume that Θ̂_s ∪ Θ̂_ns = Θ̂ (by passing through subsets, it does not matter if Θ̂_s ∩ Θ̂_ns ≠ ∅). In many statistical models with satisfactory measure concentration properties (e.g., Multinomial, Poisson and Gaussian models), this assumption can be fulfilled by a proper choice of $r_n^{(1)}$ and $r_n^{(2)}$.

The interpretation of this approach is as follows. If θ ∈ Θ − Θ_s and θ̂_n ∉ Θ̂_ns, we have θ̂_n ∈ Θ̂_s, and then U₁(θ̂_n) ⊂ Θ_s by definition, which implies θ ∉ U₁ (θ̂_n). As a result,

$$\underset{\theta \in \mathrm{\Theta }-{\mathrm{\Theta }}_{\mathrm{s}}}{\text{sup}}{\mathbb{P}}_{\theta }({\hat{\theta }}_n\notin {\hat{\mathrm{\Theta }}}_{\text{ns}})\le \underset{\theta \in \mathrm{\Theta }-{\mathrm{\Theta }}_{\mathrm{s}}}{\text{sup}}{\mathbb{P}}_{\theta }(\theta \notin U_1({\hat{\theta }}_n))\le r_n^{(1)}$$

which means that if the true parameter θ does not fall in the “non-smooth” regime, then with high probability we will also declare based on θ̂_n that we are not in the “non-smooth” regime. And similarly for the case where θ ∈ Θ − Θ_ns.

B. Answer to Question 2

The approximation region in Question 2 can be always set to be an arbitrary (1 − r_n)-confidence set U(θ̂_n). In fact, by definition we have sup_θ∈Θ ℙ_θ(θ ∈ U(θ̂_n)) ≤ r_n, hence with probability at least 1 − r_n, the approximation region U(θ̂_n) based on θ̂_n covers θ, which allows us to operate as if θ is conditioned to be inside U(θ̂_n). Moreover, U(θ̂_n) ⊂ Θ_ns, and it can be considerably smaller than Θ_ns, which makes it a desirable regime to approximate over rather than Θ_ns [16].

C. Answer to Question 3

By the definition of Θ_s, G^(r)(θ̂_n) remains bounded with high probability for any order r > 0 and θ ∈ Θ_s. Hence, it motivates us to employ Taylor expansion to correct the bias of G(θ̂_n): based on $G(\theta )\approx {\sum }_{k=0}^r\frac{G^{(k)}({\hat{\theta }}_n)}{k!}{(\theta -{\hat{\theta }}_n)}^k$, we split samples to obtain independent ${\hat{\theta }}_n^{(1)}$ and ${\hat{\theta }}_n^{(2)}$, and then use

$${\hat{G}}_{\mathrm{s}}({\hat{\theta }}_n)=\sum _{k=0}^r\frac{G^{(k)}({\hat{\theta }}_n^{(1)})}{k!}\sum _{j=0}^k\left(\begin{array}{c}\hfill k\hfill \\ \hfill j\hfill \end{array}\right)S_j({\hat{\theta }}_n^{(2)}){(-{\hat{\theta }}_n^{(1)})}^{k-j}$$ (14)

where $S_j({\hat{\theta }}_n^{(2)})$ is an unbiased estimator of θ^j (usually exists). Then the bias will be of the order 𝔼_θ|G^(r+1)(θ̂_n)(θ − θ̂_n)^r+1|.

Based on previous answers, we propose a refined approach as follows. Fix a satisfactory confidence set {U(x)}_x∈Θ̂.

1. 1) Classify the Regime:

   • For the true parameter θ, declare that θ is in the “non-smooth” regime if θ is “close” enough to Θ̂₀ in terms of confidence set (cf. (11)). Otherwise declare θ is in the “smooth” regime (cf. (10));
   • Compute θ̂_n, and declare that we are in the “non-smooth” regime if θ̂_n ∈ Θ̂_ns (cf. (13)). Otherwise declare we are in the “smooth” regime (cf. (12));
2. 2) Estimate:

   • If θ̂_n falls in the “smooth” regime, use an estimator “similar” to G(θ̂_n) to estimate G(θ) (cf. (14));
   • If θ̂_n falls in the “non-smooth” regime, replace the functional G(θ) in the “non-smooth” regime by an approximation G_appr(θ) (another functional which well approximates G(θ) on U(θ̂_n)) which can be estimated without bias, then apply an unbiased estimator for the functional G_appr(θ).

Due to space limitations, we will only focus on the construction of the optimal estimator, and omit the minimax lower bound, analysis of the plug-in approach and all proofs here, and refer interested readers to the full version [18].

III. Estimator Construction for D(P‖Q)

In this section we apply our refined methodology to construct the optimal estimator. First note that $D(P\Vert Q)=-H(P)-{\sum }_{i=1}^S{p}_i\text{ ln }{q}_i$ where $H(P)={\sum }_{i=1}^S-p_i\text{ ln }{p}_i$ is the entropy function. Hence, the optimal estimator Ĥ for entropy [1], [12]–[14] can be used here and it remains to estimate ${\sum }_{i=1}^S{p}_i\text{ ln }{q}_i$, i.e., our target is the bivariate function f(p, q) = p ln q.

A. Estimator obtained from the general recipe

We first classify the regime. For f(p, q) = p ln q, the parameter set is Θ = {(p, q) ∈ [0, 1]² : p ≤ u(S)q}, and the function is analytic on ${\mathrm{\Theta }}_0^c=\mathrm{\Theta }-\{(0,0)\}$. For all possible values of the estimator (p̂, q̂), we have Θ̂ = [0, 1]², and non-analytic points are Θ̂₀ = [0, 1] × {0}. For the confidence set in this two-dimensional Binomial model (mp̂, nq̂) ~ 𝖡(m, p) × 𝖡(n, q), we can set r_n ≍ n^−A and use

$$U(x,y)=\mathrm{\Theta }\cap (W(\frac{c\text{ ln }m}{m},p)\times W(\frac{c\text{ ln }n}{n},q))$$ (15)

for some constants A, c > 0, where W(t, z) is given by (9), and A × B ≜ {(p, q) : p ∈ A, q ∈ B}. Hence, by choosing c = c₁/2 and c = 2c₁ respectively in (10) and (11) for some universal constant c₁ > 0 to be specified later, the “smooth” and “non-smooth” regimes for (p, q) are

$${\mathrm{\Theta }}_{\mathrm{s}}=\{(p,q)\in {[0,1]}^2:\frac{c_1\text{ ln }n}{2n}\le q\le 1,p\le u(S)q\}$$

$${\mathrm{\Theta }}_{\text{ns}}=\{(p,q)\in {[0,1]}^2:0\le q\le \frac{2c_1\text{ ln }n}{n},p\le u(S)q\}.$$

Further by (12) and (13), the ultimate “smooth” and “non-smooth” regimes are given by

$${\hat{\mathrm{\Theta }}}_{\mathrm{s}}=[0,1]\times [\frac{c_1\text{ ln }n}{n},1],{\hat{\mathrm{\Theta }}}_{\text{ns}}=[0,1]\times [0,\frac{c_1\text{ ln }n}{n}]$$

i.e., $\hat{q}\gtrless \frac{c_1\text{ ln }n}{n}$ serves as the boundary for these two regimes.

Secondly, we construct the estimator in each regime. First, if we are in the “smooth” regime, (14) suggests to use the order-one bias-corrected plug-in estimator (i.e., r = 2)

$$T_{\mathrm{s}}={\hat{p}}_1(\text{ln }{\hat{q}}_1+\frac{{\hat{q}}_2-{\hat{q}}_1}{{\hat{q}}_1}-\frac{{({\hat{q}}_2-{\hat{q}}_1)}^2}{2{\hat{q}}_1^2}+\frac{{\hat{q}}_2(1-{\hat{q}}_2)}{2n{\hat{q}}_1^2})$$ (16)

to estimate f(p, q) = p ln q. Note that sample splitting is employed here, and to ensure that T_s is well-defined, it suffices to set a value of T_s (e.g., to zero) when q̂₁ = 0.

Next consider the case where we are in the “non-smooth” regime, i.e., $\hat{q}\le \frac{c_1\text{ ln }n}{n}$. By our general recipe and (15), we should approximate f(p, q) = p ln q in the confidence set

$$U(\hat{p},\hat{q})=\mathrm{\Theta }\cap (W(\frac{c_1\text{ ln }m}{4m},\hat{p})\times [0,\frac{4c_1\text{ ln }n}{n}]).$$ (17)

As a result, due to the behavior of W(t, z) in (9), we further distinguish the “non-smooth” regime into two sub-regimes. In the first sub-regime $\hat{p}>\frac{c_1\text{ ln }m}{m}$, the approximation region U(p̂, q̂) is contained in

$$W(\frac{c_1\text{ ln }m}{4m},\hat{p})\times \left[\frac{1}{u(S)}\right(\hat{p}-\frac{1}{2}\sqrt{\frac{c_1\hat{p}\text{ ln }m}{m}}),\frac{4c_1\text{ ln }n}{n}]$$

which is a rectangle denoted by I₁ × I₂. Since q cannot hit zero in this approximation regime, and f(p, q) = p ln q is a product of p and ln q, we can consider the best polynomial approximation of ln q in this regime. As a result, in this regime, we use the approximation-based estimator T_ns,I(p̂₁, q̂₁; p̂₂, q̂₂):

$$T_{\text{ns},\mathrm{I}}=\mathrm{𝟙}({\hat{p}}_2\ge \frac{c_1\text{ ln }m}{3m})\sum _{k=0}^K{g}_{K,k}({\hat{p}}_2)·{\hat{p}}_1\prod _{j=0}^{k-1}\frac{n{\hat{q}}_1-j}{n-j}$$ (18)

where ${\sum }_{k=0}^K{g}_{K,k}(\hat{p})z^k$ is the best 1D order-K polynomial approximation of ln z on I₂. Note that ${\prod }_{j=0}^{k-1}\frac{n{\hat{q}}_1-j}{n-j}$ is an unbiased estimator of q^k for nq̂₁ ~ 𝖡(n, q), and ${\hat{p}}_2\ge \frac{c_1\text{ ln }m}{3m}$ ensures that the 1D approximation interval does not contain zero. We call this regime as “non-smooth” regime I.

In the second sub-regime $\hat{p}\le \frac{c_1\text{ ln }m}{m}$, the approximation region (“non-smooth” regime II) is given by

$$R=([0,\frac{2c_1\text{ ln }m}{m}]\times [0,\frac{2c_1\text{ ln }n}{n}])\cap \{(p,q):p\le u(S)q\}.$$

Since q may be zero in R, the usual best 1D polynomial approximation of g(q) = ln q over this region does not work, and the best 2D polynomial approximation of f(p, q) = p ln q should be employed here. Hence, in this regime the approximation-based estimator is T_ns,II(p̂₁, q̂₁) with

$$T_{\text{ns},\text{II}}=\sum _{0<k+l\le K}{h}_{K,k,l}\prod _{i=0}^{k-1}\frac{m{\hat{p}}_1-i}{m-i}\prod _{j=0}^{l-1}\frac{n{\hat{q}}_1-j}{n-j}$$ (19)

where Σ_k+l≤K h_K,k,lw^kz^l is the best 2D order-K polynomial approximation of f(w, z) = w ln z in R. Note that ${\prod }_{i=0}^{k-1}\frac{m{\hat{p}}_1-i}{m-i}{\prod }_{j=0}^{l-1}\frac{n{\hat{q}}_1-j}{n-j}$ is an unbiased estimator of p^kq^l, and the condition k + l > 0 in the summation ensures that the estimator is zero for unseen symbols. We also remark that R is a polytope, and bounding the best polynomial approximation error in general polytopes was solved very recently in [19].

In summary, our estimator D̂′ for ${\sum }_{i=1}^S{p}_i\text{ ln }{q}_i$ is constructed as follows.

Estimator Construction 1

Conduct three-fold sample splitting to obtain i.i.d samples ${\{{\hat{p}}_{i,j},{\hat{q}}_{i,j}\}}_{i=1}^S$, j = 1, 2, 3, and

$$\hat{D}\prime =\sum _{i=1}^S\left[\right({\stackrel{\sim }{T}}_{\text{ns},\mathrm{I}}({\hat{p}}_{i,1},{\hat{q}}_{i,1};{\hat{p}}_{i,2},{\hat{q}}_{i,2})\mathrm{𝟙}({\hat{p}}_{i,3}>\frac{c_1\text{ ln }m}{m})+{\stackrel{\sim }{T}}_{\text{ns},\text{II}}({\hat{p}}_{i,1},{\hat{q}}_{i,1})\mathrm{𝟙}({\hat{p}}_{i,3}\le \frac{c_1\text{ ln }m}{m}))\mathrm{𝟙}({\hat{q}}_{i,3}\le \frac{c_1\text{ ln }n}{n})+{\stackrel{\sim }{T}}_{\mathrm{s}}({\hat{p}}_{i,1},{\hat{q}}_{i,1};{\hat{p}}_{i,2},{\hat{q}}_{i,2})\mathrm{𝟙}({\hat{q}}_{i,3}>\frac{c_1\text{ ln }n}{n})],$$ (20)

where

$${\stackrel{\sim }{T}}_{\text{ns},\mathrm{I}}(x,y;x\prime ,y\prime )\triangleq \text{max}\{\text{min}\{T_{\text{ns},\mathrm{I}}(x,y;x\prime ,y\prime ),1\},-1\}$$

$${\stackrel{\sim }{T}}_{\text{ns},\text{II}}(x,y)\triangleq \text{max}\{\text{min}\{T_{\text{ns},\text{II}}(x,y),1\},-1\}$$

$${\stackrel{\sim }{T}}_{\mathrm{s}}(x,y;x\prime ,y\prime )\triangleq T_{\mathrm{s}}(x,y;x\prime ,y\prime )·\mathrm{𝟙}(y\ne 0)$$ (21)

and T_ns,I, T_ns,II, T_s are given by (18), (19) and (16), respectively. Moreover, we choose K = c₂ ln n with suitable constants c₁, c₂ > 0 to obtain the optimal bias-variance tradeoff.

A pictorial explanation is displayed in Fig. 1. The estimator D̂ for the KL divergence D(P‖Q) is D̂ = − Ĥ − D̂′, where Ĥ is the rate-optimal estimator of entropy in [1]. The following theorem shows that the estimator D̂ is essentially minimax optimal, provided that an additional condition holds.

Fig. 1.

Pictorial explanation of three regimes and our estimator D̂′ for ${\sum }_{i=1}^S{p}_i\text{ ln }{q}_i$. The point (q̂₁, p̂₁) falls in the “smooth” regime, (q̂₂, p̂₂) falls in the “non-smooth” regime I, and (q̂₃, p̂₃) falls in the “non-smooth” regime II.

Theorem 3

Assume the conditions in Theorem 1, and $\frac{n}{\text{ln }n}\lesssim \frac{\mathit{\text{mu}}(S)}{\text{ln }m}$. Then D̂ achieves the upper bound in Theorem 1 and does not require the knowledge of the support size S.

B. An adaptive estimator

So far we have obtained an essentially minimax estimator via our general recipe, which has some disadvantages. Firstly, in the estimator we do not specify the explicit form of the best 2D polynomial approximation in the “non-smooth” regime II. It has been shown in [16] that the non-uniqueness of the best 2D polynomial approximation may cause serious trouble, and in our problem it forces us to add an unnecessary condition in Theorem 3. Secondly, the estimator requires the knowledge of u(S), which may be impractical in certain situations. Thirdly, our estimator in the “non-smooth” regime I requires several polynomial approximations which complicate implementation.

To resolve these issues, we wonder whether we could find one single polynomial P(x, y) which well approximates x ln y over the entire “non-smooth” regime Θ_s. We further wonder whether P(x, y) can take the form of xQ(y). We remark that either attempt may not be doable in general; see [16] for counterexamples. However, in our problem this target can be achieved, and we can construct an explicit estimator for ${\sum }_{i=1}^S{p}_i\text{ ln }{q}_i$ as follows.

Estimator Construction 2

The estimator is constructed as

$${\hat{D}}_{\mathrm{A}}^{\prime }=\sum _{i=1}^S[{\stackrel{\sim }{T}}_{\text{ns}}({\hat{p}}_{i,1},{\hat{q}}_{i,1})\mathrm{𝟙}({\hat{q}}_{i,3}\le \frac{c_1\text{ ln }n}{n})+{\stackrel{\sim }{T}}_s({\hat{p}}_{i,1},{\hat{q}}_{i,1};{\hat{p}}_{i,2},{\hat{q}}_{i,2})\mathrm{𝟙}({\hat{q}}_{i,3}>\frac{c_1\text{ln }n}{n})]$$ (22)

where T̃_s is given by (21), and

$${\stackrel{\sim }{T}}_{\text{ns}}(x,y)=\text{max}\{\text{min}\{T_{\text{ns}}(x,y),1\},-1\}$$

$$T_{\text{ns}}(x,y)=\sum _{k=0}^K{g}_{K,k+1}{\left(\frac{2c_1\text{ ln }n}{n}\right)}^{-k}·x\prod _{l=0}^{k-1}\frac{\mathit{\text{ny}}-l}{n-l}$$

where ${\sum }_{k=0}^{K+1}{g}_{K,k}{z}^k$ is the best 1D order-(K + 1) polynomial approximation of z ln z on $[0,\frac{c_1\text{ ln }n}{n}]$. Moreover, K = c₂ ln n and c₁, c₂ are some suitably chosen constants.

Note that ${\hat{D}}_{\mathrm{A}}^{\prime }$ always sets zero to unseen symbols and does not depend on u(S), and so is the entropy estimator Ĥ in [1], the overall estimator ${\hat{D}}_{\mathrm{A}}=-{\hat{D}}_{\mathrm{A}}^{\prime }-\hat{H}$ is agnostic to both S and u(S), and is thus adaptive. Moreover, the estimator D̂_A is easy to implement in practice, since only one 1D polynomial approximation of z ln z on [0, 1] need to be implemented. Finally, D̂_A achieves the upper bound in Theorem 1.