Finite sampling inequalities: an application to two-sample Kolmogorov-Smirnov statistics

Abstract

We review a finite-sampling exponential bound due to Serfling and discuss related exponential bounds for the hypergeometric distribution. We then discuss how such bounds motivate some new results for two-sample empirical processes. Our development complements recent results by Wei and Dudley (2012) concerning exponential bounds for two-sided Kolmogorov - Smirnov statistics by giving corresponding results for one-sided statistics with emphasis on “adjusted” inequalities of the type proved originally by Dvoretzky et al. (1956) and by Massart (1990) for one-sample versions of these statistics.

1. Introduction: Serfling’s finite sampling exponential bound

Suppose that {c₁,…, c_N} is a finite population with each c_i ∈ ℝ. For n ≤ N, let Y₁,…, Y_n be a sample drawn from {c₁,…, c_N} without replacement; we can regard the finite population {c₁,…, c_N} as an urn containing N balls labeled with the numbers c₁,…, c_N. Some notation: we let

$$\begin{array}{l}{\mu }_N=N^{-1}\sum _{i=1}^N{c}_i\equiv {\overline{c}}_N,{\sigma }_N^2=N^{-1}\sum _{i=1}^N{(c_i-{\overline{c}}_N)}^2,\\ a_N\equiv \underset{1\le i\le N}{min}{c}_i,b_N\equiv \underset{1\le i\le N}{max}{c}_i,\\ f_n\equiv \frac{n-1}{N-1},\text{and}{f}_n^{\ast }\equiv \frac{n-1}{N}.\end{array}$$

It is well-known (see e.g. Rice (2007), Theorem B, page 208) that ${\overline{Y}}_n=n^{-1}{\sum }_{i=1}^n{Y}_i$ satisfies E(Ȳ_n) = μ_N and

$$\mathit{Var}({\overline{Y}}_n)=\frac{{\sigma }_N^2}{n}\left(1-\frac{n-1}{N-1}\right)=\frac{{\sigma }_N^2}{n}(1-f_n).$$ (1)

Serfling (1974), Corollary 1.1, shows that for all λ > 0

$$P(\sqrt{n}({\overline{Y}}_n-{\mu }_N)\ge \lambda )\le exp\left(-\frac{2{\lambda }^2}{(1-f_n^{\ast }){(b_N-a_N)}^2}\right).$$ (2)

This inequality is an inequality of the type proved by Hoeffding (1963) for sampling with replacement and more generally for sums of independent bounded random variables. Comparing (1) and (2), it seems reasonable to ask whether the factor $f_n^{\ast }$ in (2) can be improved to f_n ≡ (n − 1)/(N − 1)? Indeed Serfling ends his paper (on page 47) with the remark: “(it is) also of interest to obtain (2) with the usual sampling fraction instead of $f_n^{\ast }$”. Note that when n = N, Ȳ_n = μ_N, and hence the probability in (2) is 0 for all λ > 0, and the conjectured improvement of Serfling’s bound agrees with this while Serfling’s bound itself is positive when n = N.

Despite related results due to Kemperman (1973a,b,c), it seems that a definitive answer to this question is not yet known.

A special case of considerable importance is the case when the numbers on the balls in the urn are all 1’s and 0’s: suppose that c₁ = ··· = c_D = 1, while c_D+1,…, c_N = 0. Then $X\equiv n{\overline{Y}}_n={\sum }_{i=1}^n{Y}_i$ is well-known to have a Hypergeometric(n, D, N) distribution given by

$$P\left(\sum _{i=1}^n{Y}_i=k\right)=\frac{\left(\begin{array}{c}D\\ k\end{array}\right)\left(\begin{array}{c}N-D\\ n-k\end{array}\right)}{\left(\begin{array}{c}N\\ n\end{array}\right)},max\{0,D+n-N\}\le k\le min\{n,D\}.$$

In this special case μ_N = D/N, ${\sigma }_N^2={\mu }_N(1-{\mu }_N)$, while b_N = 1 and a_N = 0. Thus Serfling’s inequality (2) becomes

$$P(\sqrt{n}({\overline{Y}}_n-{\mu }_N)\ge \lambda )\le exp\left(-\frac{2{\lambda }^2}{1-f_n^{\ast }}\right)\text{for}\text{all}\lambda >0,$$

and the conjectured improvement is

$$P(\sqrt{n}({\overline{Y}}_n-{\mu }_N)\ge \lambda )\le exp\left(-\frac{2{\lambda }^2}{1-f_n}\right)\text{for}\text{all}\lambda >0.$$

Despite related results due to Chvátal (1979) and Hush and Scovel (2005) it seems that a bound of the form in the last display remains unknown.

We should note that an exponential bound of the Bennett type for the tails of the hypergeometric distribution does follow from results of Vatutin and Mikha ilov (1982) and Ehm (1991); see also Pitman (1997).

Theorem 1

(Ehm, 1991) If 1 ≤ n ≤ D ∧ (N − D), then ${\sum }_{i=1}^n{Y}_i\stackrel{d}{=}{\sum }_{i=1}^n{X}_i$ where X_i ~ Bernoulli(π_i), with π_i ∈ (0, 1), are independent.

It follows from Theorem 1 that

$$\begin{array}{l}n(D/N)=E\left(\sum _1^n{Y}_i\right)=E\left(\sum _1^n{X}_i\right)=\sum _{i=1}^n{\pi }_i,\\ n\frac{D}{N}\left(1-\frac{D}{N}\right)(1-f_n)=\mathit{Var}\left(\sum _1^n{Y}_i\right)=\mathit{Var}\left(\sum _1^n{X}_i\right)=\sum _{i=1}^n{\pi }_i(1-{\pi }_i).\end{array}$$

Furthermore, by applying Theorem 1 together with Bennett’s inequality (Bennett (1962); see also Shorack and Wellner (1986), page 851), we obtain the following exponential bound for the tail of the hypergeometric distribution:

Corollary 1

If 1 ≤ n ≤ D ∧ (N − D), then for all λ > 0

$$P(\sqrt{n}({\overline{Y}}_n-{\mu }_N)\ge \lambda )\le exp\left(-\frac{{\lambda }^2}{2{\sigma }_N^2(1-f_n)}\psi \left(\frac{\lambda }{\sqrt{n}{\sigma }_N^2(1-f_n)}\right)\right)$$

where μ_N ≡ D/N, ${\sigma }_N^2\equiv {\mu }_N(1-{\mu }_N)$, 1 − f_n ≡ 1 − (n − 1)/(N − 1) is the finite-sampling correction factor, and ψ (y) ≡ 2y⁻²h(1 + y) where h(y) ≡ y(log y − 1) + 1.

Since ${\sigma }_N^2={\mu }_N(1-{\mu }_N)\le 1/4$, the inequality of the corollary yields a further bound which is quite close to the conjectured Hoeffding type improvement of Serfling’s bound, and which now has the desired finite-sampling correction factor 1 − f_n:

Corollary 2

$$\begin{array}{l}P(\sqrt{n}({\overline{Y}}_n-{\mu }_N)\ge \lambda )\le exp\left(-\frac{2{\lambda }^2}{(1-f_n)}\psi \left(\frac{\lambda }{\sqrt{n}{\sigma }_N^2(1-f_n)}\right)\right)\\ \le exp\left(-\frac{2{\lambda }^2}{(1-f_n)}\psi \left(\frac{1}{{\sigma }_N^2(1-f_n)}\right)\right).\end{array}$$

By considerations related to the work of Talagrand (1994) and León and Perron (2003), the authors of the present paper have succeeded in proving the following exponential bound.

Theorem 2

(Greene and Wellner (2015); Greene (2016)) Suppose that ${\sum }_{i=1}^n{Y}_i~\mathit{Hypergeometric}(n,D,N)$. Define μ_N = D/N and suppose N > 4 and 2 ≤ n < D ≤ N/2. Then for all $0<\lambda <\sqrt{n}/2$ we have

$$P(\sqrt{n}({\overline{Y}}_n-{\mu }_N)\ge \lambda )\le \sqrt{\frac{1}{2\pi {\lambda }^2}}\left(\frac{1}{2}\right)\sqrt{\left(\frac{N-n}{N}\right)\left(\frac{\sqrt{n}+2\lambda }{\sqrt{n}-2\lambda }\right)\left(\frac{N-n+2\sqrt{n}\lambda }{N-n-2\sqrt{n}\lambda }\right)}·exp\left(-\frac{2}{1-\frac{n}{N}}{\lambda }^2\right)exp\left(-\frac{1}{3}\left(1+\frac{n^3}{{(N-n)}^3}\right)\frac{{\lambda }^4}{n}\right).$$

The proof of this bound, along with a complete analogue for the hypergeometric distribution of a bound of Talagrand (1994) for the binomial distribution, appears in Greene and Wellner (2015) and in the forthcoming Ph.D. thesis of the first author, Greene (2016).

The bound given in Theorem 2 involves a still better finite-sampling correction factor, namely 1 − f̄_n = 1 − n/N, which has also appeared in Lo (1986) in the context of a Bayesian analysis of finite sampling. Note that as N → ∞, the above bound yields

$$\underset{N\to \infty }{limsup}P(\sqrt{n}({\overline{Y}}_n-{\mu }_N)\ge \lambda )\le \sqrt{\frac{1}{2\pi {\lambda }^2}}\left(\frac{1}{2}\right)\sqrt{\left(\frac{\sqrt{n}+2\lambda }{\sqrt{n}-2\lambda }\right)}·exp\left(-2{\lambda }^2-\frac{{\lambda }^4}{3n}\right),$$

a bound which improves slightly on the bound given by León and Perron (2003) in the case of sums of i.i.d. Bernoulli random variables.

Before leaving this section we begin to make a connection to finite-sampling empirical distributions: Now let ${\mathbb{F}}_n(t)=n^{-1}{\sum }_{i=1}^n{1}_{(-\infty ,t]}(Y_i)$ and $F_N(t)=N^{-1}{\sum }_{i=1}^N{1}_{(-\infty ,t]}(c_i)$. Then it is easily seen that Serfling’s bound yields

$$P(\sqrt{n}({\mathbb{F}}_n(t)-F_N(t))\ge \lambda )\le exp\left(-\frac{2{\lambda }^2}{(1-(n-1)/N)}\right)$$

for each fixed λ > 0 and t ∈ ℝ. Note that since 𝔽_n(t) is equal in distribution to the sample mean of n draws without replacement from an urn containing N F_N(t) 1’s and N (1 − F_N(t)) 0’s, the bound in the last display only involves the hypergeometric special case of Serfling’s inequality. This leads to the following conjecture concerning bounds for the finite sampling empirical process { $\sqrt{n}({\mathbb{F}}_n(t)-F_N(t)):t\in ℝ$}:

Conjecture

There exist constants C, D > 0 (possibly C = 1 and D = 2?) such that

$$P\left(\sqrt{n}\underset{t}{sup}({\mathbb{F}}_n(t)-F_N(t))\ge \lambda \right)\le Cexp\left(-\frac{2{\lambda }^2}{(1-f_n)}\right),$$ (3)

$$P\left(\sqrt{n}\underset{t}{sup}\mid {\mathbb{F}}_n(t)-F_N(t)\mid \ge \lambda \right)\le Dexp\left(-\frac{2{\lambda }^2}{(1-f_n)}\right)$$ (4)

for all λ > 0. The possibility that D = 2 is suggested by the corresponding inequality established by Massart (1990) in the case of sampling with replacement.

With these strong indications of the plausibility of an improvement of Serfling’s bound and corresponding improvements in exponential bounds for the uniform-norm deviations of the finite-sampling empirical process, we can now turn to an application of the basic idea in the context of two-sample Kolmogorov-Smirnov statistics.

2. Two-sample tests and finite-sampling connections

To connect this with the two-sample Kolmogorov-Smirnov statistics, suppose that X₁, …, X_m are i.i.d. F and Y₁, …, Y_n are i.i.d. G. Let N = m+n. Then for testing H_c : F = G with F continuous versus K⁺ : F ≥ G (F ≺_s G), K⁻ : G ≥ F, (G ≺_s F), or K : F ≠ G, the classical K-S test statistics are

$$\begin{array}{l}{D}_{m,n}^{+}\equiv \sqrt{\frac{mn}{N}}\underset{x}{sup}({\mathbb{F}}_m(x)-{\mathbb{G}}_n(x)),\\ D_{m,n}^{-}\equiv \sqrt{\frac{mn}{N}}\underset{x}{sup}({\mathbb{G}}_n(x)-{\mathbb{F}}_m(x)),\text{and}\\ D_{m,n}\equiv \sqrt{\frac{mn}{N}}\underset{x}{sup}\mid {\mathbb{F}}_m(x)-{\mathbb{G}}_n(x)\mid ,\end{array}$$

respectively. It is well-known that under H_c we have

$$D_{m,n}^{\pm }{\to }_d\underset{0\le t\le 1}{sup}\mathbb{U}(t),D_{m,n}{\to }_d\underset{0\le t\le 1}{sup}\mid \mathbb{U}(t)\mid$$

if m ∧ n → ∞ where 𝕌 is a standard Brownian bridge process on [0, 1]; see e.g. Hájek and Šidák (1967), pages 189–190, Hodges (1958), and van der Vaart and Wellner (1996), pages 360–366.

Note that with λ_N ≡ m/N and

$${ℍ}_N\equiv {\lambda }_N{\mathbb{F}}_m+(1-{\lambda }_N){\mathbb{G}}_n=N^{-1}\sum _{i=1}^N{1}_{(-\infty ,·]}(Z_{(i)})$$

where Z₍₁₎ ≤ ··· ≤ Z_(N) are the order statistics of the pooled sample, we have

$$\begin{array}{l}{\mathbb{F}}_m-{ℍ}_N={\mathbb{F}}_m-{\lambda }_N{\mathbb{F}}_m-(1-{\lambda }_N){\mathbb{G}}_n=(1-{\lambda }_N)({\mathbb{F}}_m-{\mathbb{G}}_n),\text{and}\\ {\mathbb{G}}_n-{ℍ}_N={\mathbb{G}}_n-{\lambda }_N{\mathbb{F}}_m-(1-{\lambda }_N){\mathbb{G}}_n={\lambda }_N({\mathbb{G}}_n-{\mathbb{F}}_m),\end{array}$$

and hence, with λ̄_N = 1 − λ_N,

$$\begin{array}{l}\sqrt{\frac{mn}{N}}({\mathbb{F}}_m-{\mathbb{G}}_n)=\sqrt{N}\sqrt{{\lambda }_N{\overline{\lambda }}_N}\frac{1}{{\overline{\lambda }}_N}({\mathbb{F}}_m-{ℍ}_N)=\frac{1}{\sqrt{{\overline{\lambda }}_N}}\sqrt{m}({\mathbb{F}}_m-{ℍ}_N),\\ \sqrt{\frac{mn}{N}}({\mathbb{G}}_n-{\mathbb{F}}_m)=\sqrt{N}\sqrt{{\lambda }_N{\overline{\lambda }}_N}\frac{1}{{\lambda }_N}({\mathbb{G}}_n-{ℍ}_N)=\frac{1}{\sqrt{{\lambda }_N}}\sqrt{n}({\mathbb{G}}_n-{ℍ}_N).\end{array}$$

Thus, using the independence of the ranks R and the order statistics Z (both based on the pooled sample),

$$P(D_{m,n}^{+}\ge t)=E_Z{P}_R\left(\sqrt{m}{\Vert {({\mathbb{F}}_m-{ℍ}_N)}^{+}\Vert }_{\infty }>t\sqrt{1-{\lambda }_N}\right)$$

and it would follow from (3) that

$$\begin{array}{l}P(D_{m,n}^{+}\ge t)\le Cexp(-2{\overline{\lambda }}_N{t}^2/(1-f_m))\\ \le Cexp(-2(n/N)t^2/(n/(N-1)))\\ =Cexp\left(-2\frac{N-1}{N}{t}^2\right)\end{array}$$ (5)

for all t > 0. Similarly it would also follow from (3) that

$$\begin{array}{l}P(D_{m,n}^{-}\ge t)\le Cexp(-2{\lambda }_N{t}^2/(1-f_n))\\ \le Cexp(-2(m/N)t^2/(m/(N-1)))=Cexp\left(-2\frac{N-1}{N}{t}^2\right)\end{array}$$

for all t > 0. Combining the two one-sided inequalities yields a (conjectured) two-sided inequality:

$$\begin{array}{l}P(D_{m,n}\ge t)\equiv P(\sqrt{mn/N}{\Vert {\mathbb{F}}_m-{\mathbb{G}}_n\Vert }_{\infty }>t)\\ \le P(D_{m,n}^{+}>t)+P(D_{m,n}^{-}>t)\\ \le 2Cexp\left(-2\frac{N-1}{N}{t}^2\right).\end{array}$$

In the next section we will prove that bounds of this type with C = 1 and D = 2 hold in the special case m = n. For some results for the two-side two-sample Kolmogorov-Smirnov statistic in the case m = n and computational results for m ≠ n, see Wei and Dudley (2012). These authors were aiming for a bound of the form C exp(−2t²) both for m = n and m ≠ n. The above heuristics seem to suggest that a bound of the form C exp(−2((N −1)/N)t²) might be a natural goal.

3. An exponential bound for $D_{m,n}^{+}$ when m = n

Throughout this section we suppose that the null hypothesis H_c holds: G = F is a continuous distribution function.

From Hodges (1958), (2.3) on page 473 (together with $t=\sqrt{mn/Nd}$ and d = a/n from page 473, line 4), when m = n (so N = 2n),

$$\begin{array}{l}P(D_{n,n}^{+}\ge t)=P\left(\sqrt{\frac{mn}{N}}\underset{x}{sup}({\mathbb{F}}_m(x)-{\mathbb{G}}_n(x))\ge \sqrt{\frac{mn}{N}}\frac{a}{n}\right)\\ =P\left(\sqrt{\frac{n^2}{2n}}\underset{x}{sup}({\mathbb{F}}_n(x)-{\mathbb{G}}_n(x))\ge \sqrt{\frac{n^2}{2n}}\frac{a}{n}\right)\\ =\frac{\left(\begin{array}{c}2n\\ n-a\end{array}\right)}{\left(\begin{array}{c}2n\\ n\end{array}\right)}\text{for}a=1,2,\dots ,n.\end{array}$$

We first compare the exact probability from the last display with the possible upper bounds

$$\begin{array}{l}P{B}_2(n)=exp\left(-2\frac{2n-1}{2n}\frac{a^2}{2n}\right);\\ P{B}_3(n)=exp\left(-2\frac{a^2}{2n}\right).\end{array}$$

For n = 3 we find that

a	0	1	2	3
E(xact)	1	.75	0.3	0.05
PB2	1	0.7574	0.3291	0.0821
PB2 - E	0	0.0074	0.0291	0.0321
PB3	1	0.7165	0.2636	0.0498
PB3 - E	0	−0.0335	−0.0364	−0.0002

Further comparisons for m = n = 10, 12, 13, 14, 15, 25 support the validity of the bound involving the finite sampling fraction f_n. These comparisons agree with the following theorem:

Theorem 3

A. When m = n (so that N = 2n) the second bound in (5) holds for all n ≥ 1 with C = 1:

$$P(D_{n,n}^{+}\ge t)=P\left(\sqrt{\frac{mn}{N}}\underset{x}{sup}({\mathbb{F}}_m(x)-{\mathbb{G}}_n(x))\ge t\right)$$ (6)

$$\le exp\left(-2\frac{N-1}{N}{t}^2\right)\mathit{for}\mathit{all}t>0.$$ (7)

Equivalently, when m = n,

$$P\left(\sqrt{\frac{mn}{N}}\sqrt{\frac{N-1}{N}}\underset{x}{sup}({\mathbb{F}}_m(x)-{\mathbb{G}}_n(x))\ge t\right)\le exp(-2t^2)$$ (8)

for all t > 0.

B. On the other hand, when m = n (so that N = 2n), for all n ≥ 1 we have

$$P(D_{n,n}^{+}\ge t)>exp(-2t^2)\mathit{for}\mathit{all}0<t<1.$$

Proof

A. Since the inequality holds trivially for a = 0, and can be shown easily by numerical computation for a ∈ {1, 2, 3} (see the Table above), it suffices to show that

$$\frac{\left(\begin{array}{c}2n\\ n-a\end{array}\right)}{\left(\begin{array}{c}2n\\ n\end{array}\right)}\le exp\left(-2\frac{2n-1}{2n}\frac{a^2}{2n}\right)$$

for a ∈ {1,…, n} and n ≥ 4. Furthermore, we will show that it holds for a = n in a separate argument, and thus it suffices to show that it holds for a ∈ {1,…, n − 1} and n ≥ 4. By rewriting the numerator and denominator on the left side of the last display, the desired inequality can be rewritten as

$$\frac{n!n!}{(n-a)!(n+a)!}\le exp\left(-\frac{2n-1}{2n}·\frac{a^2}{n}\right).$$

By taking logarithms we can rewrite this as

$$log\left(\frac{n!n!}{(n-a)!(n+a)!}\right)+\frac{2n-1}{2n}\frac{a^2}{n}\le 0.$$ (9)

Now by Stirling’s formula with bounds (see e.g. Nanjundiah (1959)) we have

$$\sqrt{2\pi k}{\left(\frac{k}{e}\right)}^{k}exp\left(\frac{1}{12k}-\frac{1}{360k^3}\right)\le k!\le \sqrt{2\pi k}{\left(\frac{k}{e}\right)}^{k}exp\left(\frac{1}{12k}\right).$$ (10)

Using these bounds in (9) we find that the left side is bounded above by

$$\begin{array}{l}-n\left\{(1-\frac{a}{n})log(1-\frac{a}{n})+(1+\frac{a}{n})log(1+\frac{a}{n})\right\}\\ -\frac{1}{2}\left(log\left(1-\frac{a}{n}\right)+log\left(1+\frac{a}{n}\right)\right)\\ +\left\{\frac{1}{6n}-\frac{1}{12(n-a)}-\frac{1}{12(n+a)}+\frac{1}{360}\left(\frac{1}{{(n-a)}^3}+\frac{1}{{(n+a)}^3}\right)\right\}\\ +\frac{a^2}{n}-\frac{a^2}{2n^2}\\ \equiv I_1+I_2+I_3+I_4.\end{array}$$

Note that I₁ and I₂ are as defined in Wei and Dudley (2012) page 640, while I₃ and I₄ differ. From Wei and Dudley (2012) page 640,

$$I_1\le -\frac{a^2}{n}-\frac{a^4}{6n^3}-\frac{a^6}{15n^5}-\frac{a^8}{28n^7},$$ (11)

(which is proved by Taylor expansion of (1 + x) log(1 + x) + (1 − x) log(1 − x) about x = 0), and

$$I_2\le \frac{a^2}{2n^2}+\frac{a^4}{4n^4}+\frac{a^6}{6n^6(1-a^2/n^2)}.$$ (12)

Note that the lead term in the bound (11) for I₁ and lead term of I₄ cancel each other, while the first term of the bound (12) for I₂ cancels the second term of I₄. Adding the bounds yields

$$\begin{array}{l}{I}_1+I_2+I_3+I_4\le -\frac{a^4}{12n^3}-\frac{a^4}{12n^3}-\frac{a^6}{15n^5}-\frac{a^8}{28n^7}+\frac{a^4}{4n^4}+\frac{a^6}{6n^6(1-a^2/n^2)}+I_3\\ =-\frac{a^4}{n^3}\left(\frac{1}{12}-\frac{1}{4n}\right)-\frac{a^4}{12n^3}-\frac{a^6}{n^5}\left(\frac{1}{15}-\frac{1}{6n(1-a^2/n^2)}\right)-\frac{a^8}{28n^7}+I_3\\ \le -\frac{a^4}{n^3}\left(\frac{1}{12}-\frac{1}{4n}\right)-\frac{a^4}{12n^3}-\frac{a^6}{n^5}\left(\frac{1}{15}-\frac{1}{6(2-1/n)}\right)-\frac{a^8}{28n^7}+I_3\\ \le -\frac{a^4}{n^3}\left(\frac{1}{12}-\frac{1}{4n}\right)-\frac{a^4}{12n^3}+\frac{3a^6}{105n^5}-\frac{a^8}{28n^7}+I_3\\ =-\frac{a^4}{n^3}\left(\frac{1}{12}-\frac{1}{4n}\right)-\frac{a^4}{12n^3}\left(1-\frac{36a^2}{105n^2}\right)-\frac{a^8}{28n^7}+I_3\\ \le -\frac{a^4}{n^3}\left(\frac{1}{12}-\frac{1}{4n}\right)-\frac{a^4}{21n^3}-\frac{a^8}{28n^7}+I_3\\ \equiv R_{12}+I_3.\end{array}$$

Now R₁₂ ≤ 0 for n ≥ 4 and I₃ ≤ 0 for all n ≥ 2 and a ∈ {1, …, n − 1} by the following argument:

$$\begin{array}{l}{I}_3=\frac{1}{6n}-\frac{1}{12(n-a)}-\frac{1}{12(n+a)}+\frac{1}{360}\left(\frac{1}{{(n+a)}^3}+\frac{1}{{(n-a)}^3}\right)\\ =-\frac{1}{6}\frac{a^2}{n(n^2-a^2)}+\frac{2}{360}\frac{n(n^2+3a^2)}{{(n^2-a^2)}^3}\\ =-\frac{1}{6n(n^2-a^2)}\left\{a^2-\frac{2}{60}\frac{n^2(n^2+3a^2)}{{(n^2-a^2)}^2}\right\}\\ =-\frac{1}{6n(n^2-a^2)}\left\{a^2-\frac{1}{30}\frac{n^2(n^2-a^2+4a^2)}{{(n^2-a^2)}^2}\right\}\\ =-\frac{1}{6n(n^2-a^2)}\left\{a^2\left(1-\frac{2}{15}\frac{n^2}{{(n^2-a^2)}^2}\right)-\frac{n^2(n^2-a^2)}{30{(n^2-a^2)}^2}\right\}\\ \le -\frac{1}{6n(n^2-a^2)}\left\{a^2\left(1-\frac{2}{15}\frac{1}{3}\right)-\frac{n^2}{30(n^2-a^2)}\right\}\\ \text{by}\text{using}a\le n-1,\text{so}{n}^2-a^2\ge n^2-{(n-1)}^2=(2n-1),\\ \text{and}{n}^2/{(2n-1)}^2\le 1/3\text{for}n\ge 4,\\ =-\frac{1}{6n(n^2-a^2)}\left\{a^2\left(1-\frac{2}{3·15}\right)-\frac{n^2-a^2+a^2}{30(n^2-a^2)}\right\}\\ =-\frac{1}{6n(n^2-a^2)}\left\{a^2\left(1-\frac{2}{3·15}-\frac{1}{30(n^2-a^2)}\right)-\frac{1}{30}\right\}\\ \le -\frac{1}{6n(n^2-a^2)}\left\{a^2\left(1-\frac{2}{3·15}-\frac{1}{30(2n-1)}\right)-\frac{1}{30}\right\}\\ \le -\frac{1}{6n(n^2-a^2)}\left\{a^2\left(1-\frac{31}{630}\right)-\frac{1}{30}\right\}\end{array}$$

for n ≥ 4. This is a decreasing function of a for fixed n, and hence to show that it is < 0 it suffices to check it for a = 1. But when a = 1 the right side above equals

$$-\frac{1}{6n(n^2-1^2)}\left\{1-\frac{31}{630}-\frac{1}{30}\right\}=-\frac{1}{n(n^2-1)}\left\{\frac{289}{6·315}\right\}<-\frac{1}{n(n^2-1)}\left\{\frac{280}{6·315}\right\}=-\frac{4}{27n(n^2-1)}<0,$$

so we conclude that I₃ < 0 for a ∈ {1, …, n − 1} and n ≥ 4. It remains only to show that the desired bound holds for a = n; that is we have

$$\frac{1}{\left(\begin{array}{c}2n\\ n\end{array}\right)}\le exp(-(n-1/2)).$$

But this can easily be shown via the Stirling formula bounds (10).

Thus

$$exp(I_1+I_2+I_3)\le exp(-I_4)=exp\left(-\frac{2n-1}{2n}\frac{a^2}{n}\right),$$

and the claimed inequality holds for all n ≥ 4. Since the bounds hold for n = 1, 2, 3 by direct numerical computation, the claim follows.

B. We first define

$$\begin{array}{l}{r}_n(a)\equiv log\left\{\frac{\left(\begin{array}{c}2n\\ n-a\end{array}\right)/\left(\begin{array}{c}2n\\ n\end{array}\right)}{exp(-2a^2/(2n))}\right\}\\ =log\left(\begin{array}{c}2n\\ n-a\end{array}\right)-log\left(\begin{array}{c}2n\\ n\end{array}\right)+\frac{a^2}{n}.\end{array}$$

Since we can take $t=a/\sqrt{2n}$, it suffices to show that r_n(a) > 0 for $1\le a\le \lfloor \sqrt{2n}\rfloor$. We will first show this for n ≥ 31. Then the proof will be completed by checking the inequality numerically for $1\le a\le \lfloor \sqrt{2n}\rfloor$ and n ∈ {1, …, 30}.

By using the Stirling formula bounds of (10) as in the proof of A, but now with upper bounds replaced by lower bounds, we find that

$$\begin{array}{l}{r}_n(a)=2log(n!)-log(n-a)!-log(n+a)!+\frac{a^2}{n}\\ \ge -n\left\{\left(1-\frac{a}{n}\right)log\left(1-\frac{a}{n}\right)+\left(1+\frac{a}{n}\right)log\left(1+\frac{a}{n}\right)\right\}-\frac{1}{2}\left\{log\left(1-\frac{a}{n}\right)+log\left(1+\frac{a}{n}\right)\right\}+\frac{1}{6n}-\frac{1}{180n^3}-\frac{1}{12(n-a)}-\frac{1}{12(n+a)}+\frac{a^2}{n}\\ \equiv L_1+L_2+L_3+L_4.\end{array}$$

As in (11) and (12) and the displays following them, we find that

$$\begin{array}{lll}{L}_1\hfill & \ge \hfill & -n\left\{\frac{a^2}{n^2}+\frac{a^4}{6n^4}+\frac{a^6}{15n^6}+\frac{a^8}{28n^8}\left(\frac{n^2}{n^2-a^2}\right)\right\},\hfill \\ L_2\hfill & \ge \hfill & \frac{a^2}{2n^2}+\frac{a^4}{4n^4}+\frac{a^6}{6n^6},\hfill \\ L_3\hfill & =\hfill & -\frac{a^2}{6n(n^2-a^2)}-\frac{1}{180n^3},\hfill \\ L_4\hfill & =\hfill & \frac{a^2}{n}.\hfill \end{array}$$

Putting these pieces together and rearranging we find that

$$r_n(a)\ge \left[\frac{31a^2}{64n^2}+\frac{a^4}{4n^4}+\frac{a^6}{6n^6}-\frac{a^4}{6n^3}-\frac{a^6}{15n^5}-\frac{a^8}{28n^5}\left(\frac{1}{n^2-a^2}\right)\right]+\left[\frac{a^2}{64n^2}+\frac{1}{6n}-\frac{1}{180n^3}-\frac{1}{12(n+a)}-\frac{1}{12(n-a)}\right]$$ (13)

$$=:K_1+K_2>0$$ (14)

will prove the claim. Note in (13) that the a²/n term cancelled by virtue of the lower bound estimate based on the Taylor expansion of (1 + x) log(1 + x) + (1 − x) log(1 − x). First note that

$$\begin{array}{l}{K}_2=\frac{a^2}{64n^2}+\frac{1}{6n}-\frac{1}{180n^3}-\frac{1}{12(n+a)}-\frac{1}{12(n-a)}\\ =\frac{a^2[28n^3-45a^{2}n]+a^2[16n^3-480n^2]+[a^2{n}^3+16a^2-16n^2]}{2880n^3(n-a)(n+a)}\end{array}$$

The denominator of the right-hand-side is clearly positive for $a\in \{1,2,\dots ,\lfloor \sqrt{2n}\rfloor \}$. By inspection, we can see the term a²n³ + 16a² − 16n² in the numerator is increasing in a. Picking a = 1, we then see n³ + 16 − 16n² > 0 for n ≥ 31, and thus a²n³ + 16a² − 16n² > 0 for all admissible a. Next, the polynomial 28n³ − 45a²n is decreasing in the admissible a. For any fixed n, the minimum value it can attain is then larger than 28n³ − 90n². For n ≥ 31, this quantity is positive. Therefore, 28n³ − 45a²n > 0 for all admissible a when n ≥ 31. Finally, note that 16n³ − 480n² = 16n²(n − 30) > 0 for n ≥ 31. Hence we have shown K₂ > 0.

We next have

$$\begin{array}{l}{K}_1=\left[\frac{31a^2}{64n^2}-\frac{a^4}{6n^3}\right]+\left[\frac{a^4}{4n^4}-\frac{a^6}{15n^5}\right]+\left[\frac{a^6}{6n^6}-\frac{a^8}{28n^5}\left(\frac{1}{n^2-a^2}\right)\right]\\ =\left[\left(\frac{a^2}{192n^3}\right)(93n-32a^2)\right]+\left[\left(\frac{a^4}{60n^5}\right)(15n-4a^2)\right]+\left[\left(\frac{a^6}{84n^6(n^2-a^2)}\right)(14n^2-3a^{2}n-14a^2)\right]\\ \equiv [(\alpha )(93n-32a^2)]+[(\beta )(15n-4a^2)]+[(\gamma )(14n^2-3a^{2}n-14a^2)].\end{array}$$ (15)

Again since $a\in \{1,\dots ,\lfloor \sqrt{2n}\rfloor \}$, it is clear that α, β, and γ in (15) are positive for all admissible choices of a. Hence, the sign of each bracketed term will be dictated by the remaining polynomial in a. It is also clear from their form that each polynomial is decreasing in a; hence we need only evaluate at the endpoints to determine positivity. But $93n-32{(\sqrt{2n})}^2=29n>0,15n-4{(\sqrt{2n})}^2=15n-8n=7n>0$, and $14n^2-3{(\sqrt{2n})}^{2}n-14{(\sqrt{2n})}^2=14n^2-6n^2-28n=4n(2n-7)>0$ with the final inequality following as n ≥ 31. Hence all terms in (15) are positive and so K₁ > 0. Together with K₂ > 0 as proved above, the claim is proved for n ≥ 31.

Since the bound holds for $a\in \{1,\dots ,\lfloor \sqrt{2n}\rfloor \}$ and n ∈ {1, …, 30} by direct numerical computation, the claim follows.

4. Some comparisons and connections

4.1. Comparisons: two-sided tail bounds

Here we compare and contrast our results with those of Wei and Dudley (2012). As in Wei and Dudley (2012) (see also Wei and Dudley (2011)), we say that the DKW inequality holds for given m, n and C if

$$P(D_{m,n}\ge t)\le Cexp(-2t^2)\text{for}\text{all}t>0,$$

and we say that the DKWM inequality holds for given m, n if the inequality in the last display holds with C = 2. Wei and Dudley (2012) prove the following theorem:

Theorem 4

(Wei and Dudley, 2012) For m = n in the two sample case:

1. The DKW inequality always holds with C = e≐2.71828.

2. For m = n ≥ 4, the smallest n such that H_c can be rejected at level 0.05, the DKW inequality holds with C = 2.16863.

3. The DKWM inequality holds for all m = n ≥ 458.

4. For each m = n < 458, the DKWM inequality fails for some t of the form $t=k/\sqrt{2n}$.

5. For each m = n < 458, the DKW inequality holds for C = 2(1 + δ_n) for some δ_n > 0 where, for 12 ≤ n ≤ 457,

   $${\delta }_n<-\frac{0.07}{n}+\frac{40}{n^2}-\frac{400}{n^3}.$$

For comparison, the following theorem follows from Theorem 3. We say that the modified DKWM inequality holds for given m, n if

$$P(D_{m,n}\ge t)\le 2exp\left(-2\left(\frac{N-1}{N}\right)t^2\right)\text{for}\text{all}t>0,$$

Theorem 5

For m = n in the two sample case:

1. For all n ≥ 1 the modified DKWM inequality holds.

2. Alternatively, for the modified Kolmogorov statistic given by

   $$D_{m,n}^{\mathit{mod}}\equiv \sqrt{\frac{N-1}{N}}\sqrt{\frac{mn}{N}}{\Vert {\mathbb{F}}_m-{\mathbb{G}}_n\Vert }_{\infty },$$

   the DKWM inequality holds for all n ≥ 1.

We are not claiming that our “modified” version of the DKWM inequality improves on the results of Wei and Dudley (2012): it is clearly worse for m = n > 458. On the other hand, it may provide a useful clue to the formulation of DKWM type exponential bounds for two-sample Kolmogorov statistics when m ≠ n. In this direction we have the following conjecture:

Conjecture

For any m ≠ n,

$$P(D_{m,n}^{+}>t)\le exp\left(-2\left(\frac{N-1}{N}\right)t^2\right)\text{for}\text{all}t>0$$ (16)

$$P(D_{m,n}>t)\le 2exp\left(-2\left(\frac{N-1}{N}\right)t^2\right)\text{for}\text{all}t>0.$$ (17)

That is, we conjecture that the modified DKWM inequality holds for all m, n ≥ 1. This is supported by all the numerical experiments we have conducted so far.

4.2. Comparisons: one-sided tail bounds

Wei and Dudley (2012) do not treat bounds for the one-sided statistics. Here we summarize our results with a theorem which parallels their Theorem 4 above. In analogy with their terminology, we say that the one-sided DKW inequality holds for given m, n and C if

$$P(D_{m,n}^{+}\ge t)\le Cexp(-2t^2)\text{for}\text{all}t>0,$$

and we say that the one-sided DKWM inequality holds for given m, n if the inequality in the last display holds with C = 1. Moreover, we say that the modified one-sided DKWM inequality holds for given m, n if

$$P(D_{m,n}^{+}\ge t)\le exp\left(-2\left(\frac{N-1}{N}\right)t^2\right)\text{for}\text{all}t>0.$$

Theorem 6

For m = n in the two sample case:

1. The one-sided DKW inequality holds for all n ≥ 1 with C = e/2≐2.71828/2 = 1.35914. For this range of n, C = e/2 is sharp since equality occurs for n = 1 and $t=1/\sqrt{2}$ (or $a=t\sqrt{2n}=1$).

2. For m = n ≥ 5, the one-sided DKW inequality holds with C = 2.16863/2 = 1.084315.

3. The one-sided DKWM inequality fails for all m = n ≥ 1.

4. The modified one-sided DKWM inequality holds for all m = n ≥ 1.

Proof

(c) follows from Theorem 3-B. (d) follows from Theorem 3-A. It remains only to prove (a) and (b).

To prove (a), we first note that Wei and Dudley (2012) showed that for n ≥ 108 we have

$$\begin{array}{l}\frac{\left(\begin{array}{c}2n\\ n+a\end{array}\right)}{\left(\begin{array}{c}2n\\ n\end{array}\right)}<exp(-a^2/n)\text{for}\sqrt{3n}\le a\le n\\ <(e/2)exp(-a^2/n).\end{array}$$

Thus to prove that the claimed inequality holds for n ≥ 108, it suffices to show that it holds for $t_0\sqrt{n}\le a\le \sqrt{3}\sqrt{n}$ where $t_0\equiv \sqrt{(1/2)log(e/2)}$ is the smallest value of t for which the bound is less than or equal to 1.

Proceeding as in the proof of Theorem 3-A, we find that we want to show that

$$log\frac{n!n!}{(n+a)!(n-a)!}+\frac{a^2}{n}-log(e/2)<0\text{for}{t}_0\sqrt{n}\le a\le \sqrt{3}\sqrt{n}.$$

By the same arguments used in the proof of Theorem 3-A, we find that the left side in the last display is bounded above by

$$-\frac{a^4}{6n^3}-\frac{a^6}{15n^5}-\frac{a^8}{28n^7}+\frac{a^4}{4n^4}+\frac{a^6}{6n^6(1-a^2/n^2)}+I_3+\frac{a^2}{2n^2}-log(e/2)\equiv K_1+K_2.$$

Now K₁ ≤ 0 for n ≥ 4 and a ∈ {1, …, n − 1} by the previous proof, and

$$K_2\equiv \frac{a^2}{2n^2}-log(e/2)<0\text{for}\text{all}a\le \sqrt{3}\sqrt{n}$$

if

$$\frac{3}{2n}<log(e/2),\text{or}n>\frac{3}{2log(e/2)}\doteq 4.888\dots .$$

This completes the proof for n ≥ 108. Numerical computation easily shows that the claim holds for all n ∈ {1, …, 107}.

The proof of (b) is similar upon replacing e/2 by 1.084315, and again computing numerically for n ∈ {1, …, 107}.

Corollary 3

For n ≥ 5 and C = 1.084315,

$$\begin{array}{l}P(D_{n,n}^{+}\ge t)\le min\{exp(-2(1-1/N)t^2),Cexp(-2t^2)\}\\ =\{\begin{array}{ll}Cexp(-2t^2),\hfill & t\ge t_0\equiv \sqrt{\mathit{n}logC}\doteq .285\sqrt{n},\hfill \\ exp(-2(1-1/N)t^2),\hfill & t\le t_0\equiv \sqrt{\mathit{n}logC}.\hfill \end{array}\end{array}$$

Figures 1 and 2 illustrate Theorem 6.

Figure 1.

Difference between approximations and exact one-sided probabilities $P(D_{n,n}^{+}>t)$ for n = 128 and a ∈ {1, 2, …, 128}. Negative values indicate the exact probability exceeds the approximation. Serfling DKWM is the bound obtained via the heuristic of section 2, using the sampling fraction $1-f_n^{\ast }=(N-n+1)/N$. Modified DKWM uses the sampling fraction 1 − f_n = (N − n)/(N − 1). DKWM uses the fraction from Wei and Dudley.

Figure 2.

Difference between approximations and exact one-sided probabilities $P(D_{n,n}^{+}>t)$ for n = 23 and a ∈ {1, 2, …, 23}. Negative values indicate the exact probability exceeds the approximation. DKWM6a corresponds to the DKWM bound with the constant e/2, discussed in Theorem 6(a). DKWM6b corresponds to the DKWM bound with the constant 2.16863/2, discussed in Theorem 6(b).