U-statistic with side information

Abstract

In this paper we study U-statistics with side information incorporated using the method of empirical likelihood. Some basic properties of the proposed statistics are investigated. We find that by implementing the side information properly, the proposed U-statistics can have smaller asymptotic variance than the existing U-statistics in the literature. The proposed U-statistics can achieve asymptotic efficiency in a formal sense and their weak limits admit a convolution result. We also find that the corresponding U-likelihood ratio procedure, as well as the U-empirical likelihood based confidence interval construction, do not benefit from incorporating side information, a result that is consistent with the result under the standard empirical likelihood ratio procedure. The impact of incorrect side information implementation in the proposed U-statistics is also explored. Simulation studies are conducted to assess the finite sample performance of the proposed method. The numerical results show that with side information implemented, the deduction of asymptotic variance can be substantial in some cases, and the coverage probability of the confidence interval using the U-empirical likelihood ratio based method outperforms that of the normal approximation based method, in particular in the cases when the underlying distribution is skewed.

1. Introduction

Since the pioneering work of Hoeffding [15], the U-statistics have been an active research field in statistics due to their wide range of applications. Hoeffding [16] established some fundamental properties of U-statistics which had close relationship with the V-statistic proposed by von Mises [37]. Berk [5] discovered the reverse martingale structure for U-statistic. Sen (e.g. [33]) made a number of contributions in this topic. Parallel to the result for V-statistics, Gregory [13] obtained the asymptotic distribution for degenerate U-statistics with rank two. The asymptotic distribution of U-statistics with arbitrary rank was developed by Janson [17] and Rubin and Vitale [32], etc. Borovskich [7] extended the results to Hilbert space. A detailed review and major historical developments in this field can be found in the book by Koroljuk and Borovskich [21], hereafter denoted as KB.

The empirical likelihood (EL) is one of the recent major developments in statistics. The original idea can be traced back to Thomas and Grunkemeier [35]. The work of Owen [25–27] formally established the advantages and application scopes of this method, and paved the road of increasing popularity of EL due to the wide range of applications, the theoretical advantages, the simplicity of usage and the flexibility to incorporate auxiliary (or side) information in various forms. EL has been applied in various problems, for example, nonparametric confidence regions [9], the generalized linear model [20], survival analysis [1], density and quantile estimations [8,39], goodness-of-fit measure [3], nonparametric regression [10,29], marginal and conditional likelihood [30], ROC curve [31], econometrics [19], etc. It is well known that incorporating side information via empirical likelihood can reduce asymptotic variance of the estimators [28]. Motivated by this fact, we explore to incorporate side information into the U-statistic using the EL method, and expect that the new procedure can improve the performance of U-statistic under appropriate conditions.

It is also known that constructing confidence regions using EL ratio has various advantages than using normal approximation based method or bootstrap. For example, Wood et al. [38] and Jing et al. [18] have considered the EL method to U-statistics to construct confidence intervals without side information incorporated. We investigate the U-statistic to construct confidence intervals using the empirical likelihood by incorporating side information, and the resulting confidence intervals are compared with those based on normal approximation. Our method of formulating the weights of U-statistics is parallel to those in the EL, and is different from that in [38,18]. We find that by incorporating the side information properly, the proposed U-statistics will have smaller asymptotic variance than the existing U-statistics methods without side information. The proposed U-statistics can achieve asymptotic efficiency in a formal sense, and their weak limits admit a convolution result. We also find that the U-statistic EL based likelihood ratio procedure do not benefit from incorporating the side information asymptotically, a result that is consistent with the result under the standard empirical likelihood ratio procedure. The resulting coverage probability based on finite sample still outperforms that of the normal based approximation. The impact of incorrect side information incorporation is also explored.

In Section 2 we introduce the framework of the proposed U-statistics with side information incorporated, and investigate the basic asymptotic properties of the proposed U-statistics in Section 3. The U-empirical likelihood ratio with side information is formulated in Section 4. Examples and simulations results are given in Section 5 to illustrate the proposed method. All the relevant proofs are left in the Appendix.

2. Incorporating side information in U-Statistics

Let X₁, …, X_n be independent and identically distributed (i.i.d.) random variables with unknown distribution function F (x) = P(X_i ≤ x). In this paper, we assume X_i being random variable for simplicity, although there is no essential difference to extend it to the case of random vectors. Denote X = (X₁, …, X_m)′, (m ≥ 2). Let i = (i₁, …, i_m)′, X_i = (X_i1, …, X_im)′. D_n,m = {i : 1 ≤ i₁ < ⋯ < i_m ≤ n} denotes the collection of indices for the U-statistic of degree m. Let $C_n^m$ be the combination number of m elements out of n, x = (x₁, …, x_m)′, $F_m(\mathbf{x})={\prod }_{j=1}^{m}F(x_j)$ and F_n,m(x) be the empirical distribution function of F_m based on the sample 𝒳_n ≔ {X_i : i ∈ D_n,m}, with mass $1/C_n^m$ at each point in 𝒳_n. Given an m-variate symmetric kernel h, the U-statistic is defined as

$$U_n={(C_n^m)}^{-1}\sum _{\mathbf{i}\in D_{n,m}}h({\mathbf{X}}_{\mathbf{i}})=E_{F_{n,m}}h(\mathbf{X}).$$

The goal is to estimate θ = E_Fmh(X), where E_Fm denotes the expectation with respect to F_m. It is known that the U-statistic U_n is the minimal variance unbiased estimator of θ [34, p. 176].

Since the work of Owen [25], the empirical likelihood (EL) has gained increasing popularity due to its wide range of applications, simplicity to use and flexibility to incorporate auxiliary (or side) information. We examine here to combine both the EL method to flexibly incorporate side information and theU-statistics to achieve a better variance for the estimator.

We consider the set-up for EL as in [28]. Suppose the side information can be incorporated into the EL through a d-dimensional known function g(x) = (g₁(x), …, g_d(x))′ via the relationship

$$E[g(X_1)]=\mathbf{0},$$

where E[·] denotes the expectation with respect to F. The EL is defined as

$$L(F)=\prod _{i=1}^n{w}_i,$$

where the w_is are the nonparametric maximum likelihood estimated empirical masses assigned to the observation X_is. With the side information constraints, the EL is

$$\underset{w}{\text{max}}\prod _{i=1}^n{w}_i\text{subject to}\sum _{i=1}^n{w}_i=1\text{and}\sum _{i=1}^n{w}_{i}g(X_i)=\mathbf{0}.$$

Let t = (t₁, …, t_d)′ be the Lagrange multipliers corresponding to the constraint of g(·), and as in [26], we get

$$w_i=\frac{1}{n}\frac{1}{1+t\prime g(X_i)},$$

where t_j = t_j(X₁, …, X_n)(j = 1, …, d) are determined by

$$\sum _{i=1}^n\frac{g(X_i)}{1+t\prime g(X_i)}=\mathbf{0}.$$

To combine the EL method and U-statistics, Wood et al. [38] considered a weighted U-statistic

$${(C_n^m)}^{-1}\sum _{1\le i_1<\cdots <i_m\le n}{n}^m{w}_{i_1}\cdots w_{i_m}h(X_{i_1},\dots ,X_{i_m})$$

with weight w(i₁, …, i_m) = w_i1 ⋯ w_im being estimated using EL procedure. Jing et al. [18] proposed a Jackknife EL for the U-statistic without side information considered. They first merge the $C_n^m$ observed h(X_i)’s into a Jackknife sample, then treat this Jackknife pseudo sample as a sample of n i.i.d observations and apply the standard EL method for the mean to obtain the EL estimate for U-statistic.

In this paper, our goal is to estimate θ = E_Fmh(X) under the information constraints to incorporate side information in the form

$$E_{F_m}g(\mathbf{X})=\mathbf{0}.$$ (1)

Without loss of generality g(·) is assumed symmetric with respect to its arguments (otherwise we can set g(x₁, …, x_m) = 1/m! ∑_(p) g(x_i1, …, x_im) to make it symmetric, where the notation ∑_(p) denote summation over the indices (i₁, …, i_m) of all the permutations of (1, …, m)). This function g includes constraints E_F (g(X₁)) = 0 as a special case by setting a componentwise product $g(\mathbf{X})={\prod }_{j=1}^{m}g(X_j)$. Some examples of g(·) will be given in Section 5 for illustration.

To formulate the proposed U-statistic we consider a different but a direct way to define the weights w(i₁, …, i_m)’s. Let w_i = F_m({X_i}) and w = (w_i : i ∈ D_n,m). Since the w_i’s are unknown (as is F_m), we maximizes the product of the w_i’s subject to appropriate constraints (they may not be independent of each other). Re-write the EL subject to the side information constraints as

$$\underset{w}{\text{max}}\prod _{\mathbf{i}\in D_{n,m}}{w}_{\mathbf{i}}\text{subject to}\sum _{\mathbf{i}\in D_{n,m}}{w}_{\mathbf{i}}=1\text{and}\sum _{\mathbf{i}\in D_{n,m}}{w}_{\mathbf{i}}g({\mathbf{X}}_{\mathbf{i}})=\mathbf{0}.$$

We get, as in [26], that

$$w_{\mathbf{i}}={(C_n^m)}^{-1}\frac{1}{1+t\prime g({\mathbf{X}}_{\mathbf{i}})},$$ (2)

and t = t_n = (t_n1, …, t_nd)′ with t_nj = t_nj(X₁, …, X_n)(j = 1, …, d) being determined by

$$\sum _{\mathbf{i}\in D_{n,m}}\frac{g({\mathbf{X}}_{\mathbf{i}})}{1+t\prime g({\mathbf{X}}_{\mathbf{i}})}=\mathbf{0}.$$ (3)

For details regarding the existence of t as the solution of (3) see, for example, the papers by Owen and others. The proposed weights for U-statistics are parallel to those in the EL, and simpler than some existing method in that there is no need to form a product of m elements from w₁, …, w_n as in [38], nor to merge the data as in [18].

Similar to Hoeffding [15], for any kernel h(·) with E_Fm(h(X)) < ∞, let h_c (x₁, …, x_c) = E(h(X₁, …, X_m)|X₁ = x₁, …, X_c = x_c), $h_c^o=h_c-\theta$ be its centered version (c = 1, …, m), ${\mathrm{h̃}}_1(x_1)=h_1^o(x_1),{\mathrm{h̃}}_2(x_1,x_2)=h_2^o(x_1,x_2)-{\mathrm{h̃}}_1(x_1)-{\mathrm{h̃}}_1(x_2),{\mathrm{h̃}}_3(x_1,x_2,x_3)=h_3^o(x_1,x_2,x_3)-{\sum }_{i=1}^3{\mathrm{h̃}}_1(x_i)-{\sum }_{1\le i\le j\le 3}{\mathrm{h̃}}_2(x_i,x_j)$, and in general,

$${\mathrm{h̃}}_c(x_1,\dots ,x_c)=h^o(x_1,\dots ,x_c)-\sum _{i=1}^c{\mathrm{h̃}}_1(x_i)-\sum _{1\le i<j\le c}{\mathrm{h̃}}_2(x_i,x_j)-\cdots -\sum _{1\le i_1<\cdots <i_{c-1}\le c}{\mathrm{h̃}}_{c-1}(x_{i_1},\dots ,x_{i_{c-1}})=\int \cdots \int h_c(y_1,\dots ,y_c)\prod _{s=1}^{c}d({\delta }_{x_s}(y_s)-F(y_s)),(c=1,\dots ,m),$$

where δ_xs (y_s) is the Dirac function, taking value 1 if y_s = x_s and 0 otherwise. The integration representation above can be found in KB. The h̃_c’s are called canonical forms of h. If h̃₁ = ⋯ = h̃_k−1 = 0 and h̃_k ≠ 0 (or equivalently Var(h̃₁) = ⋯ = Var(h̃_k−1) = 0 and Var(h̃_k) ≠ 0), the U-statistic U_n with kernel h is said of rank k(1 ≤ k ≤ m). When k > 1U_n is called degenerate; when k = m it is called complete degenerate. U_n has the following Hoeffding [16] representation

$$U_n-\theta =\sum _{c=k}^m{C}_m^c{U}_{nc},U_{nc}={(C_n^c)}^{-1}\sum _{1\le i_1<\cdots <i_c\le n}{\mathrm{h̃}}_c(X_{i_1},\dots ,X_{i_c}).$$

Let ${\eta }_c^2=E[{\mathrm{h̃}}_c^2](c=1,\dots ,m)$, U_n has the following variance formula [16]

$$\text{Var}(U_n)={(C_n^m)}^{-1}\sum _{c=1}^m{C}_m^c{C}_{n-m}^{m-c}{\eta }_c^2.$$

Define g_c = (g_c,1, …, g_c,d)′ with

$$g_{c,j}(x_1,\dots ,x_c)=E_{F_m}(g_j(X_1,\dots ,X_m)|X_1=x_1,\dots ,X_c=x_c),(j=1,\dots ,d;c=1,\dots ,m)$$

and the canonical forms g̃_c = (g̃_c,1, …, g̃_c,d)′ for g as,

$${\mathrm{g̃}}_c(x_1,\dots ,x_c)=\int \cdots \int g_c(y_1,\dots ,y_c)\prod _{s=1}^{c}d({\delta }_{x_s}(y_s)-F(y_s)),(j=1,\dots ,d;c=1,\dots ,m).$$

Similarly, let q_c (c = 1, …, m) be the canonical forms of g(·)h(·) = (g₁(·)h(·), …, g_d(·)h(·))′. The canonical forms h̃_c and q̃_c (c = 1, …, m) exist theoretically, but are unknown in practice since F is unknown. Let r_o = min{rank(g₁), …, rank(g_d)}, r = rank(h), r₁ = min{rank(g₁h), …, rank(g_dh)}, and F̃_nm be the empirical distribution with mass w_i at the observation x_i. Using the weights w_i’s given in (2) and (3), we define the U-statistic with side information given by the constraints g as

$${Ũ}_n=\sum _{\mathbf{i}\in D_{n,m}}{w}_{\mathbf{i}}h({\mathbf{X}}_{\mathbf{i}})=E_{{\mathrm{F̃}}_{n,m}}h(\mathbf{X}).$$ (4)

In comparison, the commonly used U-statistic U_n has weight ${(C_n^m)}^{-1}$ at each observation h(X_i), while with the EL formulation, the weights are replaced by w_i. In the following we investigate the basic asymptotic properties of Ũ_n.

3. The asymptotic properties of Ũ_n

In this section we study some basic asymptotic behavior of the proposed U-statistic, including its convergence, asymptotic distribution, uniform convergence, and asymptotic efficiency. The following conditions will be used in this section:

• (C1). Ω ≔ E[g(X)g′(X)] is positive definite.

• (C2). E‖g(X)‖^α < ∞ for some α > 0 to be specified.

• (C3). E_Fm|h(X)| < ∞.

• (C4). E_Fmh²(X) < ∞.

• (C5). E_Fm[‖g(X)‖²|h(X)|] < ∞.

where ‖·‖ denotes the Euclidean norm. We note that (C2) with α ≥ 4 plus (C4) implies (C5).

3.1. Convergence rate of Ũ_n

We first give a lemma to characterize the asymptotic form of the weight w_i’s, which will be used repeatedly in the asymptotic study.

Lemma. Assume (C1) and (C2) for α > 2m/r_o, we have

1. $$w_{\mathbf{i}}\stackrel{\mathrm{a}.\mathrm{s}.}{=}\frac{1}{C_n^m}(1-g\prime ({\mathbf{X}}_{\mathbf{i}}){\Omega }^{-1}\frac{1}{C_n^m}\sum _{\mathbf{j}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{j}})+1_d^{\prime }g({\mathbf{X}}_{\mathbf{i}})O({\rho }_n{n}^{-1/2}{(\text{log log}n)}^{1/2})+[1_d^{\prime }g({\mathbf{X}}_{\mathbf{i}})+{\Vert g({\mathbf{X}}_{\mathbf{i}})\Vert }^2]O({\rho }_n^2)),$$

   where, 1_d = (1, …, 1)′ is the d-dimensional vector of 1’s, the O(·) terms are uniformly for all X_i’s and i’s, with

   $${\rho }_n=\{\begin{array}{cc}O(n^{-1/2}{(\text{log log}n)}^{1/2}),\hfill & r_o=1;\hfill \\ o(n^{-r_o/2}\text{log}n),\hfill & 1<r_o\le m.\hfill \end{array}$$
2. $$w_{\mathbf{i}}=\frac{1}{C_n^m}(1-g\prime ({\mathbf{X}}_{\mathbf{i}}){\Omega }^{-1}\frac{1}{C_n^m}\sum _{\mathbf{j}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{j}})+1_d^{\prime }g({\mathbf{X}}_{\mathbf{i}})O_p(n^{-(r_o+1)/2})+[1_d^{\prime }g({\mathbf{X}}_{\mathbf{i}})+{\Vert g({\mathbf{X}}_{\mathbf{i}})\Vert }^2]O_p(n^{-r_o})).$$

The O_p(·) terms above are uniformly for all the x_i’s and i’s.

Theorem 1. (i). Assume the conditions in the lemma plus (C3) and (C5), if r = 1, then

$$n^q({Ũ}_n-\theta )\to 0,a.s.\mathit{\text{for all}}q<1/2.$$

(ii). Assume conditions in the lemma plus (C4) and (C5), if r > 1, then

$$a_n({Ũ}_n-\theta )\to 0,\mathit{(}a.s.\mathit{)},\mathit{\text{where}}{a}_n=\{\begin{array}{cc}{n}^q\mathit{\text{for all}}q<1/2,\hfill & \text{if}{r}_1=r_o=1;\hfill \\ n^{\text{min}\{r/2,1\}}/\text{log}n,\hfill & \mathit{\text{if}}{r}_1>r_o=1;\hfill \\ n^{\text{min}\{r_o,r\}/2}/\text{log}n,\hfill & \mathit{\text{if}}1=r_1<r_o;\hfill \\ \text{min}\{\frac{n^{r/2}}{\text{log}n},\frac{n^{\text{min}\{(r_0+r_1)/2,r_0\}}}{{(\text{log}n)}^2}\},\hfill & \mathit{\text{if}}{r}_o,r_1>1.\hfill \end{array}$$

(iii). Assume (C4) and conditions of Lemma (i), if r = 1, then with σ² given in Theorem 2 (i),

$$\underset{n}{\text{lim}}\text{sup}{\left(2{\sigma }^2\frac{\text{log log}n}{n}\right)}^{-1/2}|{Ũ}_n-\theta |=1,\mathit{(}a.s.\mathit{)}.$$

3.2. Asymptotic distribution of Ũ_n

Let J₁(h) be the Gaussian process indexed by h ∈ L²(R, ℬ, F) with mean E(J₁(h)) = 0 and covariance Cov(J₁(h), J₁(g)) = ∫ h(x)g(x)F(dx) for all h, g ∈ L²(R, ℬ, F). Let W(·) be the Gaussian random measure on L²(R, ℬ, P) defined by W(A) = J₁(I_A), A ∈ ℬ. J₁(h) = ∫ h(x)W(dx) is called the Wiener–Itô integral of order 1. Generally, for h ∈ L²(R^r, ℬ^r, F^r), the Wiener–Itô integral of order r is defined as

$$J_r(h)=\int \cdots \int h(x_1,\dots ,x_r)W(d{x}_1)\cdots W(d{x}_r),\forall h\in L^2(R^r,{ℬ}^r,F),$$

and its covariance is given by

$$\text{Cov}(J_r(h),J_r(g))=r!\int \cdots \int h(x_1,\dots ,x_r)g(x_1,\dots ,x_r)F(d{x}_1)\cdots F(d{x}_r),\forall h,g\in L^2(R^r,{ℬ}^r,F).$$

For a vector function h = (h₁, …, h_d)′ with h_j ∈ L²(R^r, ℬ^r, F)(j = 1, …, d), define J_r (h) componentwisely as a d-dimensional random process. Denote $\stackrel{D}{\to }$ for convergence in distribution.

Theorem 2. (i) Assume (C4) and conditions of the lemma, if r = 1, then

$$\sqrt{n}({Ũ}_n-\theta )\stackrel{D}{\to }N(0,{\sigma }^2),{\sigma }^2=\{\begin{array}{cc}{m}^2({\eta }_1^2-2A\prime {\Omega }^{-1}{A}_1+A\prime {\Omega }^{-1}{\Omega }_1{\Omega }^{-1}A),\hfill & r_o=1;\hfill \\ m^2{\eta }_1^2,\hfill & r_o>1;\hfill \end{array}$$

where ${\eta }_1^2=E_F{\mathrm{h̃}}_1^2(X_1),{\Omega }_1=E_F({\mathrm{g̃}}_1(X_1){\mathrm{g̃}}_1^{\prime }(X_1))$, A = E_Fm[g(X)h(X)] and A₁ = E_F [g̃₁(X₁)h̃₁(X₁)].

(ii) Assume (C4), conditions of Lemma (ii) and r > 1, then

$$n^{b/2}({Ũ}_n-\theta )\stackrel{D}{\to }Z,\mathit{\text{where}}$$

when A ≠ 0,

$$\{\begin{array}{ccc}b=r_o,\hfill & Z=-C_m^{r_o}A\prime {\Omega }^{-1}{J}_{r_o}({\mathrm{g̃}}_{r_o}),\hfill & \mathit{\text{if}}{r}_o<r;\hfill \\ b=r,\hfill & Z=C_m^r{J}_r({\mathrm{h̃}}_r-A\prime {\Omega }^{-1}{\mathrm{g̃}}_r),\hfill & \mathit{\text{if}}{r}_o=r;\hfill \\ b=r,\hfill & Z=C_m^r{J}_r({\mathrm{h̃}}_r),\hfill & \mathit{\text{if}}{r}_o>r;\hfill \end{array}$$

when A = 0,

$$\{\begin{array}{ccc}b=r_o,\hfill & Z=O_P(1),\hfill & \mathit{\text{if}}{r}_o\le \text{min}\{r_1,r/2\};\hfill \\ b=r,\hfill & Z=C_m^r{J}_r({\mathrm{h̃}}_r),\hfill & \mathit{\text{if}}r<\text{min}\{r_1+r_o,2r_o\};\hfill \\ b=r,\hfill & Z=C_m^r{J}_r({\mathrm{h̃}}_r)-C_m^{r_1}{C}_m^{r_o}{J}_{r_1}({\mathrm{q̃}}_{r_1}){\Omega }^{-1}{J}_{r_o}({\mathrm{g̃}}_{r_o}),\hfill & \mathit{\text{if}}{r}_1+r_o=2r;\hfill \\ b=r_o+r_1,\hfill & Z=-C_m^{r_1}{C}_m^{r_o}{J}_{r_1}({\mathrm{q̃}}_{r_1}){\Omega }^{-1}{J}_{r_o}({\mathrm{g̃}}_{r_o}),\hfill & \mathit{\text{if}}{r}_1+r_o<r\mathit{\text{or}}{r}_1<r_o.\hfill \end{array}$$

From Theorem 2 we see that the most interesting case is r = r_o = r₁ = 1, in which $\sqrt{n}({Ũ}_n-\theta )$ is asymptotic non-degenerate normal, with asymptotic variance being smaller than that of $\sqrt{n}(U_n-\theta )$. σ² is the same as that of U_n either when r₁ > 1, A = 0, or when r_o > 1, A₁ = 0 and Ω₁ = 0. Thus, for the side information to be of practical meaning, we need r = r_o = r₁ = 1.

It is interesting to note that if we have full information about the parameter to be estimated, then we can “estimate” the parameter with perfection, i.e., its asymptotic variance being reduced to zero. As an artificial example, let a and b are nonzero known constants, μ = E(X₁), $h(X_1,\dots ,X_m)=a{\sum }_{k=1}^m{X}_k$ and $g(X_1,\dots ,X_m)=b{\sum }_{k=1}^m(X_k-\mu )$. Since g is a known function, μ must be known. We are to estimate θ = Eh(X₁, …, X_m) = amμ using the U-statistic (4), with the w_i’s given by (2) and (3). In this case, θ is already known as is μ, and we can “estimate” θ with zero asymptotic variance, as in the following

Corollary 1. Assume τ² = Var(X₁) < ∞, and h and g are as given in the above. Then Theorem 2 (i) holds with σ² = 0.

3.3. The optimality property of Ũ_n

To study the asymptotic efficiency of the estimators of θ, let 𝕀(θ|g) be the information bound [6] for estimating θ given the side information in g, to be given in Theorem 3(i) below. When the asymptotic variance of an estimator achieves this bound or equals to this bound up to a known multiplicative positive constant, the estimator is called asymptotically efficient. It is the limit version of the Cramer–Rao lower bound for variances of unbiased estimators. For Euclidean parameters without g, the information (lower) bound is the inverse of Fisher information.

Suppose f (·|θ) is the density function of X given θ, θ_n = θ + n^−1/2b for some b ∈ C, the complex plane. An estimator T_n = T_n(X₁, …, X_n) is said to be regular, if under f (·|θ_n), $W_n≔\sqrt{n}(T_n-{\theta }_n)\stackrel{D}{\to }W$ for some random variable W, and the result does not depend on the sequence {θ_n}. Let Z ⊕ V denote the summation of two independent random variables Z and V, I(θ) be the Fisher information at θ, and Z ~ N(0, I⁻¹(θ)). The convolution Theorem [14] states that for any regular estimator T_n with weak limit W, there is a V such that

$$W=Z\oplus V.$$

This result further characterizes the weak limit of an asymptotic efficient estimator without side information: it is a normal random variable with mean zero and variance I⁻¹(θ). Below we obtain the information bound and convolution result for the proposed estimators with the presence of side information.

Theorem 3. Assume r = r_o = 1, (C4) and conditions in the lemma, we have

1. $$𝕀(\theta |g)={\eta }_1^2-A_1^{\prime }{\Omega }_1^{-1}{A}_1.$$

   Thus, if we set g(x) = (g(x₁) + ⋯ + g(x_m))/m, then rank(g) = 1, A = mA₁, Ω = mΩ₁, σ² = m²𝕀(θ|g) and Ũ_n is efficient.
2. Assume further that the density f (·|θ) of X has the second order continuous partial derivative with respect to θ, then for any regular estimator T_n with weak limit W of $W_n≔\sqrt{n}(T_n-\theta )$, W can be decomposed as, for some V,

   $$W=Z\oplus V,\mathit{\text{with}}Z~N(0,𝕀(\theta |g)).$$

It is easy to see that any U-statistic with side information of the form Ũ_n is regular, thus is optimal in the sense of convolution under the conditions of Theorem 3. Without side information, the asymptotic variance of $\sqrt{n}(U_n-\theta )$ is ${\eta }_1^2$; when side information presents, the asymptotic variance of $\sqrt{n}({Ũ}_n-\theta )$ is ${\eta }_1^2-A_1^{\prime }{\Omega }_1^{-1}{A}_1$, with a reduction of $A_1^{\prime }{\Omega }_1^{-1}{A}_1$. From the proof of Theorem 3(i) we see that 𝕀(θ|g) is the length of the projection of h̃₁(X) onto [g̃₁(X)^⊥], the linear span of the orthogonal complements of g̃₁(X). Increasing the components in g (and thus in g̃₁) shrinks the space [g̃₁(X)^⊥], and shortens the length of the projection or increases the efficiency of Ũ_n, or increasing the number of information constraints reduces the asymptotic variance of the U-statistic.

Remark. By Theorem 2(i) or Theorem 3(i), given a nominal level α, a level (1 − α) confidence interval of θ can be obtained as [Ũ_n ± n^−1/2σΦ⁻¹(1 − α/2)], without using likelihood ratio, where Φ⁻¹(·) is the standard normal quantile function. Here σ is smaller with the presence of side information than no side information involved, hence the inference becomes more accurate.

3.4. The uniform SLLN and CLT of Ũ_n-processes

Let P̃_n,m, P_n,m, P_m and P be the (random) probability measures induced by F̃_n,m, F_n,m, F_m and F respectively. For a function h, denote P̃_n,mh = ∑_i∈Dn,m w_ih(X_i), P_mh = E_Pmh(X), ${\mathrm{𝔾̃}}_{n,m}h=\sqrt{n}({\mathrm{P̃}}_{n,m}h-P_{m}h)$ and ${𝔾}_{n,m}h=\sqrt{n}(P_{n,m}h-P_{m}h)$. For fixed h and g, we have shown that, under appropriate conditions,

$${\mathrm{P̃}}_{n,m}h\to P_{m}h=P{\mathrm{h̃}}_1(\mathrm{a}.\mathrm{s}.)\text{and}{\mathrm{𝔾̃}}_{n,m}h\stackrel{D}{\to }N(0,{\sigma }^2)$$

with ${\sigma }^2={\sigma }^2(h)=P{\mathrm{h̃}}_1^2-P({\mathrm{g̃}}_1^{\prime }{\mathrm{h̃}}_1){\Omega }_1^{-1}P({\mathrm{g̃}}_1{\mathrm{h̃}}_1)$. In contrast, ${𝔾}_{n,m}h\stackrel{D}{\to }N(0,{\eta }_1^2)$ with ${\eta }_1^2=P{\mathrm{\eta ̃}}_1^2$. Thus incorporating the side information g reduces the asymptotic variance by the amount of $P({\mathrm{g̃}}_1^{\prime }{\mathrm{h̃}}_1){\Omega }_1^{-1}P({\mathrm{g̃}}_1\mathrm{h̃})$.

It is of interest to have a uniform version of the above SLLN and CLT over a class of functions ℋ. The uniformity means supremum over ℋ, which may or may not be measurable, thus the almost sure and weak convergence results here will be in the sense of outer measure P* of P (cf. [36]; hereafter VW). When the corresponding quantity is measurable, the convergence is automatically in the sense of the measure P itself. Nolan and Pollard [23,24] study the uniform SLLN and the CLT for U-process of order two. Giné and Zinn [12], Arcones and Giné [2] and Giné [11], among others, study other types of uniform problems in general situations. Here we explore the uniform laws for U-statistics under different conditions.

Let ℋ be a class of functions satisfying (C4), and for any probability measure Q, denote ‖Qh‖_ℱ = sup{|Qh| : h ∈ ℋ}. Let L^∞(ℋ) be the space of functionals z : ℋ ↦ R with norm ‖z‖_ℋ = sup_h∈ℋ |z(h)| and the metric on L^∞(ℋ) is given by d(z₁, z₂) = ‖z₁ − z₂‖_ℋ for z₁, z₂ ∈ L^∞(ℋ). For an integer m-vector k = (k₁, …, k_m), a subset 𝒳 of R^m and a function h : 𝒳 ↦ R, denote |k| = k₁ + ⋯ + k_m, $D^{k}h(\mathbf{x})={\partial }^{|k|}h(\mathbf{x})/(\partial x_1^{k_1}\cdots \partial x_m^{k_m})$. ‖x‖ is the Euclidean norm for x ∈ 𝒳,

$${\Vert h\Vert }_s=\underset{|k|\le s}{\text{max}}\underset{x\in 𝒳}{\text{sup}}|D^{k}h(\mathbf{x})|+\underset{|k|=s}{\text{max}}\underset{\mathbf{x},\mathbf{y}\in 𝒳}{\text{sup}}|D^{k}h(\mathbf{x})-D^{k}h(\mathbf{y})|,$$

and C_M(𝒳) is the set of functions h : 𝒳 ↦ R with ‖h‖_s ≤ M. Let $R^m={\cup }_{j=1}^{\infty }{I}_j$ be a partition of R^m into bounded convex sets with non-empty interior, ℋ₁ be the class of functions h such that the restrictions ℋ_1|Ij belong to C_Mj (I_j) for every j and M = max_j M_j < ∞. Let ℋ₂ be the class of convex functions h : C ↦ R for some convex compact C ⊂ R^m such that |h(x) − h(y)| ≤ L‖x − y‖ for some 0 < L < ∞, all x, y ∈ C and h ∈ ℋ₂, and ‖ ∑_i∈Dn,m e_ih(x_i)‖_ℋ2 is measurable for each n and each e_i ∈ {−1, 1} (ℋ₂ is then called P-measurable in VW). An envelope function G of ℋ is a function such that |h(x)| ≤ G(x) for all x, and h ∈ ℋ. Let ℋ = ℋ₁ ∪ ℋ₂ with (C4) satisfied on ℋ and λ(·) be the Lebesgue measure on R^m. Let $\stackrel{D}{\Rightarrow }$ denote weak convergence in L^∞(ℋ). We have

Theorem 4. (i) Under the conditions of Theorem 1(i), for ℋ defined above, assume that ∀h ∈ ℋ, gh ∈ ℋ in the componentwise sense, ℋ₁ has a square integrable envelope function H, max_j λ(I_j) < ∞, ${\sum }_{j=1}^{\infty }{M}_j^{1/2}{P}_m^{1/2}(I_j)<\infty$, and ℋ₂ is bounded. Then we have

$$\underset{h\in \mathscr{H}}{\text{sup}}|{\mathrm{P̃}}_{n,m}h-P_{m}h|=0,(a.s{.}^{*}).$$

(ii) Under the conditions of Theorem 3(ii), assume ℋ has a square integrable envelope function H, max_j λ(I_j) < ∞, max_j λ(I_j) < ∞, m < 4, s > m/2 for ℋ₁, and ${\sum }_{j=1}^{\infty }{M}_j^{\frac{2\upsilon }{\upsilon +2}}{P}^{\frac{\upsilon }{\upsilon +2}}(I_j)<\infty$ with υ = m/s. Then

$${\mathrm{𝔾̃}}_{n,m}\stackrel{D}{\Rightarrow }𝔾\mathit{\text{in}}{L}^{\infty }(\mathscr{H}),$$

where 𝔾 is a Gaussian process indexed by ℋ, with E_P (𝔾h) = 0 and ${\text{Cov}}_P(𝔾h,𝔾q)=P({\mathrm{h̃}}_1{\mathrm{q̃}}_1)-P({\mathrm{g̃}}_1^{\prime }{\mathrm{h̃}}_1){\Omega }_1^{-1}P({\mathrm{g̃}}_1{\mathrm{q̃}}_1)$ for all h, q ∈ ℋ.

Using results in [2,11], we can get many more results, below we only mention one.

Corollary 2. For a class of functions ℋ, assume that ∀h ∈ ℋ, gh ∈ ℋ; that ℋ is a measurable VC-subgraph class of functions with envelope H and P_mH < ∞, or ∀ε > 0, $N_{[]}^{(1)}(\epsilon ,\mathscr{H},P_m)<\infty$ (for definition, see p. 1512, [2]). Then

$$\underset{h\in \mathscr{H}}{\text{sup}}|{\mathrm{P̃}}_{n,m}h-P_{m}h|\to 0,a.s.$$

4. The empirical likelihood ratio for U-statistics with side information

Next we define the empirical likelihood ratio for θ, and construct the confidence interval for θ in the presence of side information. Let G(x|θ) = (g′(x), h(x) − θ)′, we have E_Fm G(X|θ) = 0. Without side information, the weights that maximize ∏_i∈Dn,m w_i subject to ∑_i∈Dn,m w_i = 1 are $w_{\mathbf{i}}={(C_n^m)}^{-1}$ for all i ∈ D_n,m; while the weights that maximize ∏_i∈Dn,m w_i subject to ∑_i∈Dn,m w_i = 1 and ∑_i∈Dn,m w_iG(X_i|θ) = 0 are $w_{\mathbf{i}}={(C_n^m)}^{-1}1/(1+t\prime G({\mathbf{X}}_{\mathbf{i}}|\theta ))$ and t is determined by (3) with g(·) replaced by G(·|θ). Therefore we define the empirical log likelihood ratio of θ with the presence of side information by

$$R_G(\theta )=L_n(\theta )/{(C_n^m)}^{-C_n^m}=\prod _{\mathbf{i}\in D_{n,m}}(C_n^m{w}_{\mathbf{i}}),$$

where

$$L_n(\theta )=\underset{{\sum }_{\mathbf{i}\in D_{n,m}}{w}_{\mathbf{i}}=1,{\sum }_{\mathbf{i}\in D_{n,m}}{w}_{\mathbf{i}}G({\mathbf{X}}_{\mathbf{i}}|\theta )=\mathbf{0}}{\text{max}}\prod _{\mathbf{i}\in D_{n,m}}{w}_{\mathbf{i}}$$

and denote

$$l(\theta )=-\text{log}{R}_G(\theta )=\sum _{\mathbf{i}\in D_{n,m}}\text{log}[1+t\prime G({\mathbf{X}}_{\mathbf{i}}|\theta )].$$

Let $\Lambda =E_{F_m}(G(\mathbf{x}|\theta )G\prime (\mathbf{X}|\theta ))=\left(\begin{array}{cc}\hfill \Omega \hfill & \hfill A\hfill \\ \hfill A\prime \hfill & \hfill {\eta }^2\hfill \end{array}\right)$, η² = Var(h(X)); and Λ₁ = Cov(G̃₁), G̃₁ is the first canonical form (vector) of G.

Note that when there is no side information, G(·|θ) reduces to h(·) − θ, and t is a scalar determined by ∑_i∈Dn,m(h(X_i) − θ)/[1 + t(h(X_i) − θ)] = 0. The corresponding log-likelihood ratio is

$$l_h(\theta )=\sum _{\mathbf{i}\in D_{n,m}}\text{log}[1+t(h({\mathbf{X}}_{\mathbf{i}})-\theta )].$$

Theorem 5. (i) Under conditions of Theorem 2 (i) or Theorem 3 (i), assume r_o = 1 and Λ to be positive definite, then

$$\frac{2n}{m^2{C}_n^m}l(\theta )\stackrel{D}{\to }{Z}_{d+1}^{\prime }{\Lambda }_1^{1/2}{\Lambda }^{-1}{\Lambda }_1^{1/2}{Z}_{d+1},Z_{d+1}~N(0,I_{d+1}).$$

(ii) Assume (C4), then

$$\frac{2n{\eta }^2}{m^2{C}_n^m{\eta }_1^2}{l}_h(\theta )\stackrel{D}{\to }{\chi }_1^2.$$

When m = 1, ${\Lambda }_1^{1/2}={\Lambda }^{1/2}$ and the above result for U-statistic automatically reduces to that for the common EL ratio, and the right hand side in Theorem 5(i) is ${\chi }_{d+1}^2$ (see the corresponding result in Theorem 2 of Qin and Lawless [28]), therefore with side information incorporated in the likelihood ratio, the length of confidence region for θ cannot be reduced, this is an interesting contrast to the estimation with side information, in which the asymptotic variance is reduced. However, using the EL ratio, the shape of the confidence region is more natural than many other commonly used methods, such as the normal approximation, which are forced to be symmetric. The latter method may have poorer coverage probability because of the shorter interval length and its subjective shape.

Although side information is widely applied in practice to improve performance of estimators via the EL, the following Corollary 3 describes the effects when incorrect side information is used, thus side information should be used with care, and be justified properly before its use.

Corollary 3. If E_Fmg(X) = δ ≠ 0, then

1. Under conditions of Theorem 1 (i),

   $${Ũ}_n-\theta \stackrel{\mathrm{a}.\mathrm{s}.}{\to }A\prime {\Omega }^{-1}\delta .$$
2. Under conditions of Theorem 2 (i),

   $$\sqrt{n}({Ũ}_n-\theta -A\prime {\Omega }^{-1}\delta )\stackrel{D}{\approx }N(0,{\sigma }^2).$$
3. If E_FmG(X) = δ ≠ 0, then under conditions of Theorem 5 (i),

   $$-\frac{2n}{C_n^m}{R}_G(\theta )\stackrel{D}{\approx }{Z}_{d+1}^{\prime }{\Lambda }_1^{1/2}{\Lambda }^{-1}{\Lambda }_1^{1/2}{Z}_{d+1},Z_{d+1}\stackrel{D}{~}N(\sqrt{n}{\Lambda }_1^{-1/2}\delta ,I_{d+1}),$$

   when Λ = Λ₁, $Z_{d+1}^{\prime }{\Lambda }_1^{1/2}{\Lambda }^{-1}{\Lambda }_1^{1/2}{Z}_{d+1}=Z_{d+1}^2(n\delta \prime {\Lambda }^{-1}\delta )$, the chi-squared distribution of degree d + 1 with noncentrality parameter nδ′Λ⁻¹δ.

5. Examples and simulation studies

5.1. Examples

In this section we give some examples for illustration.

Example 1. For a given distribution F, let θ(F) = ∫ (x − μ)²dF(x) be the variance, where μ is the mean. Let μ_k, k ≥ 2 be the k-th moment of F. For the kernel h(x₁, x₂) = (x₁ − x₂)²/2, we have h̃₁(x₁) = [(x₁ − μ)² − θ]/2, η² = E(h²) − θ² = (μ₄ + θ²)/2, ${\eta }_1^2=E({\mathrm{h̃}}_1^2)=({\mu }_4-{\theta }^2)/4$. Without side information, the asymptotic variance of U_n based on kernel h(x₁, x₂) is ${\sigma }_0^2=4{\eta }_1^2={\mu }_4-{\theta }^2$, which is the same as that for the sample variance estimator ${\theta }_n≔{(n-1)}^{-1}{\sum }_{i=1}^n{(X_i-\mathrm{X̅})}^2$.

If we know that F has median at 0: F (0) = 1/2, we take g(x₁, x₂) = [I(x₁ ≤ 0) + I(x₂ ≤ 0)]/2 − 1/2. Then g̃₁(x₁) = [I(x₁ ≤ 0) − 1/2]/2, $A_1=E({\mathrm{g̃}}_1{\mathrm{h̃}}_1)=[{\int }_{-\infty }^0{(x-\mu )}^{2}dF(x)-\theta /2]/4$, and ${\Omega }_1=E({\mathrm{g̃}}_1^2)=1/16$. So by Theorem 3(i), the asymptotic variance of Ũ_n is now ${\sigma }^2={\sigma }_0^2-A_1^2{\Omega }_1^{-1}=4{\eta }_1^2-{[{\int }_{-\infty }^0{(x-\mu )}^{2}dF(x)-{\sigma }^2/2]}^2$, a deduction of ${[{\int }_{-\infty }^0{(x-\mu )}^{2}dF(x)-{\sigma }^2/2]}^2$ from ${\sigma }_0^2$.

Example 2. For the Wilcoxon one-sample statistic, θ(F) = P_F (x₁ + x₂ ≤ 0), the kernel for the corresponding U-statistic is h(x₁, x₂) = I(x₁ + x₂ ≤ 0), h̃₁(x₁) = F (−x₁) − θ, ${\eta }_1^2=E_F({\mathrm{h̃}}_1(x_1))=\int F^2(-x)dF(x)-{\theta }^2$. Without side information, the asymptotic variance of U_n based on kernel h(x₁, x₂) is ${\sigma }_0^2=4{\eta }_1^2$.

Suppose we know the distribution is symmetric about a > 0: F (x − a) = 1 − F (a − x) for all x. Take g(x₁, x₂) = [I(x₁ ≤ 0) + I(x₁ ≤ 2a) + I(x₂ ≤ 0) + I(x₂ ≤ 2a)]/2 − 1, then g̃₁(x₁) = [I(x₁ ≤ 0) + I(x₁ ≤ 2a)]/2 − 1/2, Ω₁ = F (−a)/2, $A_1=[{\int }_{-\infty }^{a}F(-x)dF(x)+{\int }_{-\infty }^{-a}F(-x)dF(x)]/2-\int F(-x)dF(x)/2$. The deduction of asymptotic variance is $A_1^2{\Omega }^{-1}$.

Example 3. For the Gini difference, θ(F) = E_F |x₁ − x₂|, the corresponding kernel for U-statistic is h(x₁, x₂) = |x₁ − x₂|. We have ${\mathrm{h̃}}_1(x_1)={\int }_{x_1}^{\infty }xdF(x)-{\int }_{-\infty }^{x_1}xdF(x)-\theta ,{\eta }_1^2=\int {({\int }_{x_1}^{\infty }xdF(x)-{\int }_{-\infty }^{x_1})}^{2}dF(x_1)-{\theta }^2$. Without side information, the asymptotic variance of U_n based on kernel h(x₁, x₂) is ${\sigma }_0^2=4{\eta }_1^2$.

If we know the distribution mean μ, and take g(x₁, x₂) = (x₁ + x₂)/2 − μ, then g̃₁(x₁) = (x₁ − μ)/2, Ω₁ = ∫(x − μ)²dF(x), $A_1=\{\int x_1[{\int }_{x_1}^{\infty }xdF(x)-{\int }_{-\infty }^{x_1}xdF(x)]dF(x_1)-\theta \}/2$. The deduction of asymptotic variance is $A_1^2{\Omega }^{-1}$.

5.2. Simulation studies

Simulation studies are conducted to assess the finite-sample performance of the proposed methods in this section. These studies are based on Examples 1 and 2 in Section 5.1. We compare variance estimates of U-statistics with and without side information, and calculate the variance reduction under different sample sizes. We also compare various U-EL based and normal approximation-based confidence intervals for θ in terms of coverage probability. Although side information does not have effect asymptotically, as indicated by Theorem 5, the finite sample property of constructing confidence intervals using the U-EL ratio is still of interest, and will be compared with those obtained through normal approximation based method.

Based on the U-EL theory developed in Section 4, we can construct three U-EL based intervals for θ as follows:

The first one, called EL1 interval, is defined as

$$\{\theta :\frac{2n}{m^2{C}_n^m}l(\theta )\le q_{1-\alpha }\}$$

where q_1−α is the (1 − α)-th quantile of the distribution of $Z_{d+1}^{\prime }{\Lambda }_1^{1/2}{\Lambda }^{-1}{\Lambda }_1^{1/2}{Z}_{d+1}$. q_1−α can be estimated by using the sample estimates of Λ₁ and Λ and Monte Carlo method.

One can also approximate the quantile of the distribution of l(θ) by using bootstrap method. Let $\{l_b^{*}(\mathrm{\theta ̂}):b=1,\dots ,B\}$ (B ≥ 200 is recommended) are B bootstrap replicates of l(θ). Then, the second EL-based interval for θ, called EL2, is given by

$$\{\theta :l(\theta )\le l_{([B(1-\alpha )])}^{*}(\mathrm{\theta ̂})\},$$

where $l_{(b)}^{*}(\theta )$ is the b-th ordered value of $l_b^{*}(\theta )$’s, and [x] represents the integer part of x.

The third one, called EL3 interval, is constructed as follows:

$$\{\theta :c^{*}l(\theta )\le {\chi }_{d+1,1-\alpha }^2\},$$

where $c^{*}=\frac{d+1}{\frac{1}{B}{\sum }_{b=1}^B{l}_b^{*}(\mathrm{\theta ̂})}$. This interval is motivated by the fact that the distribution of $Z_{d+1}^{\prime }{\Lambda }_1^{1/2}{\Lambda }^{-1}{\Lambda }_1^{1/2}{Z}_{d+1}$ can be approximated by a scaled chi-squares distribution. i.e.,

$$c·l(\theta )\stackrel{D}{\to }{\chi }_{d+1}^2,$$

where c is an unknown constant, and $c\approx \frac{E({\chi }_{d+1}^2)}{E(l(\theta ))}=\frac{d+1}{E(l(\theta ))}$.

The asymptotic normal distribution obtained in Theorem 2 can be used to construct two additional confidence intervals for θ, called AN1 and AN2 intervals, as follows:

$$\{{Ũ}_n-z_{1-\alpha /2}\mathrm{\sigma ̂}/\sqrt{n},{Ũ}_n+z_{1-\alpha /2}\mathrm{\sigma ̂}/\sqrt{n}\},\{{Ũ}_n-z_{1-\alpha /2}{\mathrm{\sigma ̂}}^{*}/\sqrt{n},{Ũ}_n+z_{1-\alpha /2}{\mathrm{\sigma ̂}}^{*}/\sqrt{n}\},$$

where σ̂ is the estimate of σ by plugging the sample estimates of all population quantities in Theorem 2. σ̂* is the bootstrap estimate of σ based on B bootstrap samples. For computation consideration, we take B = 200 in the simulation studies.

Examples 1 and 2 in Section 5.1 are considered in the simulation study. In the first example, the underlying distribution is chosen to be a skewed distribution with median 0. Here we take X ~ exp(1) − ln(2), the standard exponential distribution with a shifted center. Then EX = 1 − ln 2, Median(X) = 0, and θ = Var(X) = 1. In the second example, we consider a symmetric distribution with mean a. We choose X ~ 𝒩(1, 4), then $\theta =\Phi (-\frac{1}{4})$, where Φ(x) is the cdf of the standard normal distribution. The simulation results are presented in Tables 1–4.

Table 1.

The asymptotic variance estimation of U-statistics. X ~ exp(1) − ln(2).

Method	n = 50	n = 100	n = 150	n = 200
Without side information	8.5239	7.8569	7.3839	7.1557
With side information	8.4572	7.5524	7.2673	7.0791
Variance reduction	0.0667	0.3045	0.1165	0.0766

Table 4.

Coverage probabilities of various 95% confidence intervals for θ with side information. X ~ 𝒩(1, 4).

Sample size	EL1	EL2	EL3	AN1	AN2
n = 50	0.876	0.983	0.918	0.956	0.945
n = 100	0.882	0.981	0.931	0.961	0.947
n = 150	0.926	0.978	0.968	0.978	0.968
n = 200	0.942	0.956	0.984	0.970	0.954

Tables 1 and 2 show the estimated asymptotic variances of U-statistics with or without side information respectively under sample size n = 50, 100, 150, 200. The reduction of variance is also calculated. The results are based on 1000 repetitions.

Table 2.

The asymptotic variance estimation of U-statistics. X ~ 𝒩(1, 4).

Method	n = 50	n = 100	n = 150	n = 200
Without side information	0.2413	0.2208	0.2199	0.2203
With side information	0.0548	0.0526	0.0527	0.0572
Variance reduction	0.1865	0.1682	0.1673	0.1631

Tables 3 and 4 show the coverage probabilities of the EL-based intervals (EL1, EL2 and EL3) and the normal approximation-based intervals (AN1 and AN2) with side information.

Table 3.

Coverage probabilities of various 95% confidence intervals for θ with side information. X ~ exp(1) − ln(2).

Sample size	EL1	EL2	EL3	AN1	AN2
n = 50	0.942	0.949	0.994	0.783	0.784
n = 100	0.929	0.950	0.991	0.858	0.878
n = 150	0.934	0.954	0.990	0.872	0.880
n = 200	0.950	0.949	0.989	0.898	0.904

The simulation results show that the proposed U-statistic Ũ_n performs well in finite sample cases. From Tables 1 and 2 we can clearly see a reduction of the variance of estimating θ. The variance reduction can be significant, as in Example 2, which shows that the proposed method could offer a more accurate estimation.

From the coverage probabilities in Tables 3 and 4, we see that the U-EL based confidence intervals work significantly better than the normal approximation-based confidence intervals when the underlying distribution is a skewed distribution (Example 1). When the underlying distribution is a symmetric distribution (Example 2), the performances of these methods of methods are comparable. Furthermore, in most cases, bootstrap-based methods work better than plugin methods.

Concluding remarks

We studied a method to implement side information into the U-statistic, via the empirical likelihood approach, and investigated some asymptotic behavior of the proposed method. We show, for parameter estimation, the proposed U-statistic with side information has advantages, such as smaller asymptotic variance, over that without side information incorporated. We also explored the construction of confidence intervals using U-statistic based empirical likelihood ratio. Although such U-EL ratio does not benefit from side information asymptotically, our simulation studies show that the corresponding confidence intervals still out perform those based on normal approximation in finite sample cases. We also note that, if incorrect side information is incorporated, the resulting estimates can be seriously biased. Thus in practice the incorporation of side information should be justified properly.

Appendix

Proof of the Lemma. (i) As in [26], write t = t_n = ρ_ne with ρ_n ≥ 0 and e = e(X₁, …, X_n) a d-vector with ‖e‖ = 1. We first find the asymptotic order of t_n. Denote $b(t)={(C_n^m)}^{-1}{\sum }_{\mathbf{i}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{i}})/(1+t\prime g({\mathbf{X}}_{\mathbf{i}}))$, Z_n = max{|e′g(X_i)| : i ∈ D_n,m}, and Z̃_n = max{|e′g(X̃_i)| : i ∈ D̃_n,m} with $|{\mathrm{D̃}}_{n,m}|=C_n^m$, ∀i and j ∈ D̃_n,m, i and j have no common entry, and X̃_i = (X̃_i1, …, X̃_im), X̃_i1, …, X̃_im i.i.d. with X₁ for all i₁, …, i_m (i = 1, …, n). Since Z̃_n is a maximum over $C_n^m$ i.i.d. samples, while Z_n is that over $C_n^m$ dependent samples come from the same distribution, for large n, we have Z_n ≤ Z̃_n (a.s.). Since E‖e′g(X)‖^α < ∞, ${\mathrm{Z̃}}_n=o({(C_n^m)}^{1/\alpha })$ (a.s.) as in [26], and so $Z_n=o({(C_n^m)}^{1/\alpha })$ (a.s.). We have

$$0=\Vert b({\rho }_{n}e)\Vert \ge |e\prime b({\rho }_{n}e)|=\frac{1}{C_n^m}|e\prime (\sum _{\mathbf{i}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{i}})-{\rho }_n\sum _{\mathbf{i}\in D_{n,m}}\frac{g({\mathbf{X}}_{\mathbf{i}})e\prime g({\mathbf{X}}_{\mathbf{i}})}{1+{\rho }_{n}e\prime g({\mathbf{X}}_{\mathbf{i}})}\left)\right|\ge \frac{{\rho }_n}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}\frac{{[e\prime g({\mathbf{X}}_{\mathbf{i}})]}^2}{1+{\rho }_{n}e\prime g({\mathbf{X}}_{\mathbf{i}})}-|\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}e\prime g({\mathbf{X}}_{\mathbf{i}})|\ge \frac{{\rho }_n}{1+{\rho }_n{Z}_n}\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}{[e\prime g({\mathbf{X}}_{\mathbf{i}})]}^2-|\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}e\prime g({\mathbf{X}}_{\mathbf{i}})|=\frac{{\rho }_n}{1+{\rho }_n{Z}_n}e\prime (\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{i}})g\prime ({\mathbf{X}}_{\mathbf{i}}))e-|e\prime (\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{i}})\left)\right|≔\frac{{\rho }_n}{1+{\rho }_n{Z}_n}e\prime R_{1,n}e-|e\prime R_{2,n}|.$$

Below we will show, for some 0 < c < C < ∞, for all large n, uniformly for all the X_i’s and i’s,

$$c<e\prime R_{1,n}e=C\mathrm{a}.\mathrm{s}.;\text{and}e\prime R_{2,n}=\{\begin{array}{c}O(n^{-1/2}{(\text{log log}n)}^{1/2})\mathrm{a}.\mathrm{s}.\text{if}{r}_o=1,\hfill \\ o(n^{-r_o/2}\text{log}n)\mathrm{a}.\mathrm{s}.\text{if}{r}_o>1.\hfill \end{array}$$ (A.1)

In fact, R_1,n is a (matrix valued) U-statistic with a.s. limit 0 < E[g(X)g′(X)] = Ω < ∞, where the “0 <” is in the matrix positive definite sense and the “<∞” is in the componentwise sense. Let 0 < λ₁ ≤ ⋯ ≤ λ_d < ∞ be all the eigenvalues of Ω, we have Ω = Q′diag(λ₁, …, λ_d)Q with Q being orthonormal. Denote η = Qe, then η′η = 1. Then for large n, R_1,n > Ω/2 (a.s.) thus e′R_1,ne > e′Ωe/2 = η′diag(λ₁, …, λ_d)η/2 ≥ λ₁/2 ≔ c > 0 (a.s.). Similarly, for large n, R_1,n < 2Ω (a.s.) and e′R_1,ne < 2λ_d ≔ C (a.s.).

Since ‖e‖ = 1, we only need to prove the second assertion in (A.1) for R_2,n. Note R_2,n is a (vector) U-statistic with kernel g(x) satisfying E(g(X)) = 0. Recall g_c is the canonical forms of g, let R_2,nc be the corresponding Hoeffding forms of R_2,n(c = 1, …, m). By the given condition we have E(‖g_c (X)‖²) ≤ E‖g(X)‖² < ∞, (c = r_o, …, m), E‖g(X)‖^4/3 < ∞, and $R_{2,n}={\sum }_{c=r_o}^m{C}_m^c{R}_{2,nc}$ in componentwise sense (component j in R_2,nc is zero for c = r_o, …, r_j −1 if r_j ≔ rank(g_j) > r_o). If r_o = 1, let ${\eta }_{2,1}^2≔E({\Vert g_1(\mathbf{X})\Vert }^2)$, by Theorem 9.1.1 in KB, we get

$$\underset{n}{\text{lim}}\text{sup}{\left(2m^2{\eta }_{2,1}^2\frac{\text{log log}n}{n}\right)}^{-1/2}|R_{2,n}|=1(\mathrm{a}.\mathrm{s}.).\text{or}{R}_{2,n}=O(n^{-1/2}{(\text{log log}n)}^{1/2})(\mathrm{a}.\mathrm{s}.).$$

If r_o > 1, by Lemma 9.2.1 in KB,

$$\frac{n^{c/2}}{\text{log}n}{R}_{2,nc}\to 0(\mathrm{a}.\mathrm{s}.),(c=r_o,\dots ,m);\text{and so}{R}_{2,n}=o(n^{-r_o/2}\text{log}n)(\mathrm{a}.\mathrm{s}.).$$

Now, since R_1,n = O(1) (a.s.) with 0 < O(1) < ∞ and $Z_n=O({(C_n^m)}^{1/\alpha })=O(C^{m/\alpha })$, we have, for r_o = 1, ρ_n/(1 + ρ_nZ_n) = O(|e′R_2,n|) = O(n^−1/2(log log n)^1/2) (a.s.), or ρ_n(1 − o(n^{−(1/2−m/α)}(log log n)^1/2)) = O(n^−1/2(log log n)^1/2) (a.s.). For r_o > 1, ρ_n(1 − o(n^m/α−r_o/2 log n)) = o(n^−r_o/2 log n) (a.s.). Thus we have

$$\Vert t_n\Vert ={\rho }_n=\{\begin{array}{cc}O(n^{-1/2}{(\text{log log}n)}^{1/2}),\hfill & r_o=1;\hfill \\ o(n^{-r_o/2}\text{log}n),\hfill & 1<r_o\le m\hfill \end{array}(\mathrm{a}.\mathrm{s}.).$$

Since for all i, $|t_n^{\prime }g({\mathbf{X}}_{\mathbf{i}})|\le \Vert t_n\Vert Z_n=o({\rho }_n{n}^{m/\alpha })\to 0$ (a.s.), thus for large n, max_i∈Dn,m |t′g(X_i)| < 1 (a.s.), so we have

$$\mathbf{0}=\sum _{\mathbf{i}\in D_{n,m}}\frac{g({\mathbf{X}}_{\mathbf{i}})}{1+t\prime g({\mathbf{X}}_{\mathbf{i}})}=\sum _{\mathbf{i}\in D_{n,m}}(g({\mathbf{X}}_{\mathbf{i}})\sum _{j=0}^{\infty }{(-1)}^j{[t\prime g({\mathbf{X}}_{\mathbf{i}})]}^j)=\sum _{\mathbf{i}\in D_{n,m}}(g({\mathbf{X}}_{\mathbf{i}})[1-t\prime g({\mathbf{X}}_{\mathbf{i}})+O{[t\prime g({\mathbf{X}}_{\mathbf{i}})]}^2])=\sum _{\mathbf{i}\in D_{n,m}}(g({\mathbf{X}}_{\mathbf{i}})[1-t\prime g({\mathbf{X}}_{\mathbf{i}})+{\Vert g({\mathbf{X}}_{\mathbf{i}})\Vert }^{2}O({\rho }_n^2)]),(\mathrm{a}.\mathrm{s}.)$$

or

$$\sum _{\mathbf{i}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{i}})g\prime ({\mathbf{X}}_{\mathbf{i}})t=\sum _{\mathbf{i}\in D_{n,m}}[g({\mathbf{X}}_{\mathbf{i}})+{\Vert g({\mathbf{X}}_{\mathbf{i}})\Vert }^{2}g({\mathbf{X}}_{\mathbf{i}})O({\rho }_n^2)],(\mathrm{a}.\mathrm{s}.)$$

thus

$$t={(\sum _{\mathbf{i}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{i}})g\prime ({\mathbf{X}}_{\mathbf{i}}))}^{-1}\sum _{\mathbf{i}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{i}})+O({\rho }_n^2){(\sum _{\mathbf{i}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{i}})g\prime ({\mathbf{X}}_{\mathbf{i}}))}^{-1}\sum _{\mathbf{i}\in D_{n,m}}{\Vert g({\mathbf{X}}_{\mathbf{i}})\Vert }^{2}g({\mathbf{X}}_{\mathbf{i}})≔B_n+O({\rho }_n^2){(\sum _{\mathbf{i}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{i}})g\prime ({\mathbf{X}}_{\mathbf{i}}))}^{-1}\sum _{\mathbf{i}\in D_{n,m}}{\Vert g({\mathbf{X}}_{\mathbf{i}})\Vert }^{2}g({\mathbf{X}}_{\mathbf{i}})(\mathrm{a}.\mathrm{s}.).$$

We have already shown $R_{2,n}={(C_n^m)}^{-1}{\sum }_{\mathbf{i}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{i}})=O({\rho }_n)$ (a.s.). Also g(·)g′(·) is non-degenerate by (C1), also since m ≥ 2, E‖g(X)‖⁴ < ∞ by (C2), thus by the law of iterated logarithm (LIL) for U-statistics, ${(C_n^m)}^{-1}{\sum }_{\mathbf{i}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{i}})g\prime ({\mathbf{X}}_{\mathbf{i}})=\Omega +O(n^{-1/2}{(\text{log log}n)}^{1/2})$ (a.s.), hence

$$B_n=[{\Omega }^{-1}+O(n^{-1/2}{(\text{log log}n)}^{1/2})]\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{i}})={\Omega }^{-1}\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{i}})+O({\rho }_n{n}^{-1/2}{(\text{log log}n)}^{1/2})=O({\rho }_n)(\mathrm{a}.\mathrm{s}.).$$

Similarly,

$$O({\rho }_n^2){(\sum _{\mathbf{i}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{i}})g\prime ({\mathbf{X}}_{\mathbf{i}}))}^{-1}\sum _{\mathbf{i}\in D_{n,m}}{\Vert g({\mathbf{X}}_{\mathbf{i}})\Vert }^{2}g({\mathbf{X}}_{\mathbf{i}})=O({\rho }_n^2){\Omega }^{-1}E[{\Vert g(\mathbf{X})\Vert }^{2}g(\mathbf{X})]=o(B_n),(\mathrm{a}.\mathrm{s}.)$$

so,

$$t=t_n=B_n+O({\rho }_n^2)={\Omega }^{-1}\frac{1}{C_n^m}\sum _{\mathbf{j}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{j}})+O({\rho }_n{n}^{-1/2}{(\text{log log}n)}^{1/2})+O({\rho }_n^2)(\mathrm{a}.\mathrm{s}.).$$

From this we get, (a.s.),

$$w_{\mathbf{i}}=\frac{1}{C_n^m}\frac{1}{1+t\prime g({\mathbf{X}}_{\mathbf{i}})}=\frac{1}{C_n^m}[1-t\prime g({\mathbf{X}}_{\mathbf{i}})+{\Vert g({\mathbf{X}}_{\mathbf{i}})\Vert }^{2}O({\rho }_n^2)]=\frac{1}{C_n^m}(1-g\prime ({\mathbf{X}}_{\mathbf{i}}){\Omega }^{-1}\frac{1}{C_n^m}\sum _{\mathbf{j}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{j}})+1_d^{\prime }g({\mathbf{X}}_{\mathbf{i}})O({\rho }_n{n}^{-1/2}{(\text{log log}n)}^{1/2})+(1_d^{\prime }g({\mathbf{X}}_{\mathbf{i}})+{\Vert g({\mathbf{X}}_{\mathbf{i}})\Vert }^2)O({\rho }_n^2)).$$

(ii) As in the proof of (i), we only need to show the results regardless of e. Treating gg′ and R_1,n as vectors of length d². Under the given conditions, note gg′ is non-degenerate, by the central limit theorem (CLT) for U-statistics,

$$\sqrt{n}(R_{1,n}-\mathit{\text{Egg}}\prime (\mathbf{X}))\stackrel{D}{\to }N(0,\Xi ),\text{or}{R}_{1,n}=\mathit{\text{Egg}}\prime (\mathbf{X})+O_p(n^{-1/2}),$$

where Ξ is determined by gg′. Similarly, for R_2,n, since Eg(X) = 0, if r_o = 1, by standard U-statistics theory, $\sqrt{n}{R}_{2,n}$ is asymptotical normal, or R_2,n = O_p(n^−1/2). If r_o > 1, note r_o ≤ m, the given conditions implies E|g_c (x₁, …, x_c)|^{2c/(2c−r_o)} < ∞ for c = r_o, …, m. So, by Theorem 4.4.1 in KB, $n^{r_o/2}{R}_{2,n}\stackrel{D}{\to }{R}_2$ for some non-degenerate random variable R₂, thus R_2,n = O_p(n^−r_o/2). So, as in the proof of (i), we get ρ_n(1 − o_P (n^m/α−r_o/2)) = O_p(n^−r_o/2). Since o_P (n^m/α−r_o/2) = o_p(1), we have ‖t_n‖ = ρ_n = O_p(n^−r_o/2).

Going through the proofs in (i), with O(n^−1/2(log log n)^1/2) replaced by O_p(n^−1/2), we get $t=t_n={\Omega }^{-1}\frac{1}{C_n^m}{\sum }_{\mathbf{j}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{j}})+O_p({\rho }_n{n}^{-1/2})$, and, with ρ_n = n^−r_o/2,

$$w_{\mathbf{i}}=\frac{1}{C_n^m}(1-g\prime ({\mathbf{X}}_{\mathbf{i}}){\Omega }^{-1}\frac{1}{C_n^m}\sum _{\mathbf{j}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{j}})+1_d^{\prime }g({\mathbf{X}}_{\mathbf{i}})O_p({\rho }_n{n}^{-1/2})+(1_d^{\prime }g({\mathbf{X}}_{\mathbf{i}})+{\Vert g({\mathbf{X}}_{\mathbf{i}})\Vert }^2)O_p({\rho }_n^2)).$$

Proof of Theorem 1. (i) By Lemma (i), we have

$${Ũ}_n=U_n-(\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}g\prime ({\mathbf{X}}_{\mathbf{i}})h({\mathbf{X}}_{\mathbf{i}})){\Omega }^{-1}(\frac{1}{C_n^m}\sum _{\mathbf{j}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{j}}))+O({\rho }_n{n}^{-1/2}{(\text{log log}n)}^{1/2})\times \frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}{1}_d^{\prime }g({\mathbf{X}}_{\mathbf{i}})h({\mathbf{X}}_{\mathbf{i}})+O({\rho }_n^2)\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}(1_d^{\prime }g({\mathbf{X}}_{\mathbf{i}})+{\Vert g({\mathbf{X}}_{\mathbf{i}})\Vert }^2)h({\mathbf{X}}_{\mathbf{i}})(\mathrm{a}.\mathrm{s}.).$$

By the given conditions and the SLLN of U-statistics, $U_n\stackrel{\mathrm{a}.\mathrm{s}.}{\to }\theta$,

$$U_{0,n}≔\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{i}})\stackrel{\mathrm{a}.\mathrm{s}.}{\to }{E}_{F_m}g(\mathbf{X})=0,U_{1,n}≔\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}g({\mathbf{X}}_{\mathbf{i}})h({\mathbf{X}}_{\mathbf{i}})\stackrel{\mathrm{a}.\mathrm{s}.}{\to }A<\infty ,$$

where A = E_Fm[g(X)h(X)], and

$$U_{2,n}≔\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}(1_d^{\prime }g({\mathbf{X}}_{\mathbf{i}})+{\Vert g({\mathbf{X}}_{\mathbf{i}})\Vert }^2)h({\mathbf{X}}_{\mathbf{i}})\stackrel{\mathrm{a}.\mathrm{s}.}{\to }{E}_{F_m}[(1_d^{\prime }g(\mathbf{X})+{\Vert g(\mathbf{X})\Vert }^2)h(\mathbf{X})]<\infty .$$

Since by the given conditions, E_Fm|h̃_c (X)|^γ < ∞ and E_Fm |g̃_c (X)| < ∞ for γ = cp/(p(c − 1) + 1), c = 1, …, m and 1 < p < 2, so by Corollary 3.4.1 in KB, n^−1/p+1(U_n − θ) → 0 (a.s.), or n^q(U_n − θ) → 0 (a.s.) and n^qU_0,n → 0 (a.s.) for all q < 1/2, and consequently, for all q < 1/2,

$$n^q({Ũ}_n-\theta )=n^q(U_n-\theta )-O(n^q{U}_{0,n})+O({\rho }_n{n}^{-(1/2-q)}{(\text{log log}n)}^{1/2})+O(n^q{\rho }_n^2)\to 0(\mathrm{a}.\mathrm{s}.).$$

(ii) Use notations in (i), we have

$${Ũ}_n=U_n-U_{1,n}^{\prime }{\Omega }^{-1}{U}_{0,n}+O({\rho }_n{n}^{-1/2}{(\text{log log}n)}^{1/2})U_{1,n}+O({\rho }_n^2)U_{2,n},(\mathrm{a}.\mathrm{s}.).$$

Recall for any U-statistic U_n with rank r and canonical forms h̃_c (c = r, …, m) the following decomposition holds

$$U_n-\theta =\sum _{c=r}^m{C}_m^c{U}_{n,c},U_{n,c}={(C_n^c)}^{-1}\sum _{1\le i_1<\cdots <i_c\le n}{\mathrm{h̃}}_c(x_{i_1},\dots ,x_{i_c}).$$

Since $E{h}_c^2<\infty$ for c = r, …, m, by Lemma 9.2.1 in KB, U_n,c = o(n^−c/2 log n) (a.s.), so U_n = o(n^−r/2 log n) (a.s.). Thus, when r₁ = r₀ = 1, U_1,n = O(1) (a.s.), U_0,n = o(n^−q) (a.s.) for any q < 1/2, and U_2,n = O(1) (a.s.) as its kernel is always non-degenerate, so by Lemma (i) we have

$${Ũ}_n-\theta =o(n^{-r/2}\text{log}n)+o(n^{-q})+O(n^{-1}\text{log log}n)+O(n^{-1}\text{log log}n)=o(n^{-q}),q<1/2.$$

When r₁ > r₀ = 1, by Lemma 9.2.1 in KB, U_1,n = o(n^−r₁/2 log n) (a.s.), note O(n⁻¹ log log n) = o(n⁻¹ log n), and so

$${Ũ}_n-\theta =o(n^{-r/2}\text{log}n)+o(n^{-(r_1/2+q)}\text{log}n)+o(n^{-(1+r_1/2)}\text{log}n{(\text{log log}n)}^{1/2})+O(n^{-1}\text{log log}n)=o(n^{-\text{min}\{r/2,(r_1+1)/2,1\}}\text{log}n)=o(n^{-\text{min}\{r/2,1\}}\text{log}n).$$

When 1 = r₁ < r₀,

$${Ũ}_n-\theta =o(n^{-r/2}\text{log}n)+o(n^{-(r_0/2)}\text{log}n)+o(n^{-(1+r_o)/2}\text{log}n{(\text{log log}n)}^{1/2})+o(n^{-r_o}{(\text{log}n)}^2)=o(n^{-\text{min}\{r_o,r\}/2}\text{log}n);$$

and when r₁, r₀ > 1,

$${Ũ}_n-\theta =o(n^{-r/2}\text{log}n)+o(n^{-(r_0+r_1)/2}{(\text{log}n)}^2)+o(n^{-(1+r_o+r_1)/2}{(\text{log}n)}^2{(\text{log log}n)}^{1/2})+o(n^{-r_o}{(\text{log}n)}^2)=o(\text{max}\{n^{-r/2}\text{log}n,n^{-\text{min}\{(r_0+r_1)/2,r_0\}}{(\text{log}n)}^2\}).$$

(iii) Using Lemma (i) and notations in the proof of (i), we have, a.s.,

$${Ũ}_n-\theta =\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}(h({\mathbf{X}}_{\mathbf{i}})-\theta -A\prime {\Omega }^{-1}g({\mathbf{X}}_{\mathbf{i}}))-(U_{1,n}^{\prime }-A\prime ){\Omega }^{-1}{U}_{0,n}+O({\rho }_n{n}^{-1/2}{(\text{log log}n)}^{1/2})U_{1,n}+O({\rho }_n^2)U_{2,n}.$$

Recall that U_0,n = o(n^−q), U_1,n −A = o(n^−q) (a.s.) for all 0 < q < 1/2, and U_2,n → C₂ (a.s.) for some C₂ < ∞. We have, a.s., for all 0 < q < 1/2,

$${Ũ}_n-\theta =\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}(h({\mathbf{X}}_{\mathbf{i}})-\theta -A\prime {\Omega }^{-1}g({\mathbf{X}}_{\mathbf{i}}))+o(n^{-2q})+O({\rho }_n{n}^{-1/2}{(\text{log log}n)}^{1/2})+O({\rho }_n^2).$$

By Theorem 9.1.1 of KB, the LIL holds for the first term above, and the above equation gives the desired result.

Proof of Theorem 2. (i) Using the fact that U_1,n → A (a.s.) and U_2,n → C₂ (a.s.) for some C₂ < ∞ as proved in Theorem 1(i), thus by Lemma (ii),

$$\sqrt{n}({Ũ}_n-\theta )=\sqrt{n}\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}(h({\mathbf{X}}_{\mathbf{i}})-\theta -A\prime {\Omega }^{-1}g({\mathbf{X}}_{\mathbf{i}}))-\sqrt{n}(U_{1,n}^{\prime }-A\prime ){\Omega }^{-1}{U}_{0,n}+O_P(n^{-r_o/2})U_{1,n}+O_P(n^{-(r_o-1/2)})U_{2,n}.$$

The second term above is, for all 0 < q < 1/2, n^1/2O(n^−2q) = o_P (1); the third term above is O_P (n^−r_o/2) as U_1,n → A(a.s.) < ∞; and the last term above is O_P (n^{−(r_o−1/2)}) as U_2,n → C (a.s.) for some C < ∞; Thus we only need to deal with the first term above.

Let

$$H(\mathbf{x})=h(\mathbf{x})-{\mathrm{h̃}}_1(x_1)-\cdots -{\mathrm{h̃}}_1(x_m)-\theta ,G(\mathbf{x})=g(\mathbf{x})-{\mathrm{g̃}}_1(x_1)-\cdots -{\mathrm{g̃}}_1(x_m),$$

then H₁(x₁) ≔ E[H(X)|X₁ = x₁] = 0, i.e. H(x) is a degenerate kernel. Similarly, G(x) is degenerate, so is K(x) = H(x) − A′Ω⁻¹G(x), with E_FmK(X) = 0 and r_k ≔ rank(K) ≥ 2. Now we have

$$\sqrt{n}({Ũ}_n-\theta )=\sqrt{n}\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}({\mathrm{h̃}}_1(X_{i_1})+\cdots +{\mathrm{h̃}}_1(X_{i_m})-A\prime {\Omega }^{-1}[{\mathrm{g̃}}_1(X_{i_1})+\cdots +{\mathrm{g̃}}_1(X_{i_m})])+\sqrt{n}\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}K({\mathbf{X}}_{\mathbf{i}})+O_P(n^{-1/2})+o_P(1)=\sqrt{n}\frac{m}{n}\sum _{i=1}^n({\mathrm{h̃}}_1(X_i)-A\prime {\Omega }^{-1}{\mathrm{g̃}}_1(X_i))+\sqrt{n}\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}K({\mathbf{X}}_{\mathbf{i}})+O_P(n^{-1/2})+o_P(1).$$

Let K̃_c be the canonical forms of K, and ${\xi }_c^2=E_{F_m}{K}_c^2(\mathbf{X})<\infty$ by the given conditions, (c = r_k, …, m). So by Hoeffding’s formula,

$$\text{Var}(\sqrt{n}\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}K({\mathbf{X}}_{\mathbf{i}}))=n\sum _{c=r_k}^m{(C_m^c)}^2{(C_n^c)}^{-1}{\xi }_c^2=O(n^{-(r_k-1)})\to 0,$$

and so $\sqrt{n}{(C_n^m)}^{-1}{\sum }_{\mathbf{i}\in D_{n,m}}K({\mathbf{X}}_{\mathbf{i}})\stackrel{P}{\to }0$. We get

$$\sqrt{n}({Ũ}_n-\theta )=\sqrt{n}\frac{m}{n}\sum _{i=1}^n({\mathrm{h̃}}_1(X_i)-A\prime {\Omega }^{-1}{\mathrm{g̃}}_1(X_i))+o_P(1)+O_P(n^{-1/2}).$$

Since Var[m(h̃₁(X₁) − A′Ω⁻¹ g̃₁(X₁))] = σ² if $r_o=1,\text{and}=m^2{\eta }_1^2$ when r_o > 1 (in this case g̃₁(·) ≡ 0). Now the result follows from the standard CLT and Slutsky’s theorem.

(ii) We have, since U_2,n = O_P (1),

$${Ũ}_n-\theta =U_n-\theta -U_{1,n}^{\prime }{\Omega }^{-1}{U}_{0,n}+O_P(n^{-(r_o+1)/2})U_{1,n}+O_P(n^{-r_0}).$$

By Theorem 4.4.2 in KB, $n^{r/2}(U_n-\theta )\stackrel{D}{\to }{C}_m^r{J}_r({\mathrm{h̃}}_r)$, or U_n − θ = O_P (n^−r/2). Similarly in summary we have

$$U_n-\theta =O_p(n^{-r/2}),U_{0,n}=O_P(n^{-r_0/2}),U_{1,n}-A=O_P(n^{-r_1/2}).$$

Also, U_1,n → A (a.s.), and when r₀ = 1, $\sqrt{n}{U}_{0,n}\stackrel{D}{\to }N(0,m^2{\Omega }_1)$.

First we consider the case A ≠ 0. In this case, U_n − θ = O_P (n^−r/2), $U_{1,n}^{\prime }{\Omega }^{-1}{U}_{0,n}=O_P(n^{-r_o/2})$ and O_P (n^−(r_o+1)/2)U_1,n + O_P (n^−r_o) = o_P (n^−r_o/2).

Thus when r_o < r, we have

$$r_o^{1/2}({Ũ}_n-\theta )=-U_{1,n}^{\prime }{\Omega }^{-1}{n}^{-r_o/2}{U}_{0,n}+o_P(1)\stackrel{D}{\to }-C_m^{r_o}A\prime {\Omega }^{-1}{J}_{r_o}({\mathrm{g̃}}_{r_o}).$$

When r_o = r,

$$n^{r/2}({Ũ}_n-\theta )=n^{r/2}(U_n-\theta )-U_{1,n}^{\prime }{\Omega }^{-1}{n}^{r/2}{U}_{0,n}+o_P(1)\stackrel{D}{\to }{C}_m^r{J}_r({\mathrm{h̃}}_r-A\prime {\Omega }^{-1}{\mathrm{g̃}}_r).$$

When r_o > r,

$$n^{r/2}({Ũ}_n-\theta )=n^{r/2}(U_n-\theta )+o_P(1)\stackrel{D}{\to }{C}_m^r{J}_r({\mathrm{h̃}}_r).$$

Now we consider the case A = 0, then $U_{1,n}^{\prime }{\Omega }^{-1}{U}_{0,n}=O_P(n^{-(r_1+r_o/2})$, and

$${Ũ}_n-\theta =U_n-\theta -U_{1,n}^{\prime }{\Omega }^{-1}{U}_{0,n}+O_P(n^{-(r_1+r_o+1)/2})+O_P(n^{-r_0}).$$

When r_o ≤ min{r₁, r/2}, Ũ_n − θ = O_P (n^−r_o), and its distribution needs more accurate expansion to evaluate. When r < min{2r_o, r₁ + r_o},

$$n^{-r/2}({Ũ}_n-\theta )=n^{-r/2}(U_n-\theta )+o_P(1)\stackrel{D}{\to }{C}_m^r{J}_r({\mathrm{h̃}}_r).$$

When r₁ + r_o < r or r₁ < r_o,

$$n^{-(r_o+1)/2}({Ũ}_n-\theta )=-n^{-r_1/2}{U}_{1,n}^{\prime }{\Omega }^{-1}{n}^{-r_o/2}{U}_{0,n}+o_P(1)\stackrel{D}{\to }{C}_m^{r_1}{C}_m^{r_o}{J}_{r_1}({\mathrm{q̃}}_{r_1}){\Omega }^{-1}{J}_{r_o}({\mathrm{g̃}}_{r_o}).$$

When r₁ + r_o = r,

$$n^{-r/2}({Ũ}_n-\theta )=n^{-r/2}(U_n-\theta )-n^{-r_1/2}{U}_{1,n}^{\prime }{\Omega }^{-1}{n}^{-r_o/2}{U}_{0,n}+o_P(1)\stackrel{D}{\to }{C}_m^r{J}_r({\mathrm{h̃}}_r)-C_m^{r_1}{C}_m^{r_o}{J}_{r_1}({\mathrm{q̃}}_{r_1}){\Omega }^{-1}{J}_{r_o}({\mathrm{g̃}}_{r_o}).$$

Proof of Corollary 1. In this case we have h₁(X₁) = a[X₁ + (m − 1)μ], h̃₁(X₁) = a(X₁ − ν), ${\eta }_1^2=a^2{\tau }^2$. Also, g₁(X₁) = b(X₁ − μ) = g̃₁(X₁), A₁ = abτ², $A=abE[{\sum }_{k=1}^m{X}_k{\sum }_{k=1}^m(X_k-\mu )]=mab{\tau }^2$, Ω₁ = b²τ², $\Omega =b^{2}E[{\sum }_{k=1}^m(X_k-\mu ){\sum }_{k-1}^m(X_k-\mu )]=m{b}^2{\tau }^2$ and r_o = 1. So by Theorem 2(i) we have σ² = m²(a²τ² − 2a²τ² + a²τ²) = 0.

Proof of Theorem 3. (i) Note θ = E_Fmh(X). The information bound is for parameter of the form E_F (s(X₁)) for some s(·). Recall h₁(x₁) = E[h(X₁, …, X_m)|X₁ = x₁] and E_F (h₁(X₁)) = θ, thus we take s(·) = h₁(·). Similarly, the constraint for computing the information bound should be a uni-variate function, we take it to be g₁(x₁).

Let f (x) be the density/mass function of F (x) with respect to some dominating measure μ(x), denote γ (f) = ∫h₁(x)f (x)dμ(x) = θ as a functional of f, γ̇ (f)(x) be the adjoint (evaluated at 1) of its pathwise derivative with respect to log f (for definition, see, for example, [6]), γ₁(f) = E_f [g₁(X)] for the side information constraint, γ̇₁(f)(x) the adjoint (evaluated at 1) of its pathwise derivative, L_2,d,r (f) = {s(x) : s : R^d → R^r, E_f [s(X)s′(X)] < ∞}, for s₁ ∈ L_2,d,k(f) and s₂ ∈ L_2,d,r (f), define the inner product (matrix) $\langle s_1,s_2\rangle =E_f[s_1(X)s_2^{\prime }(X)]=\int s_1(x)s_2^{\prime }(x)f(x)d\mu (x)$, the norm (matrix) ‖s₁‖² = 〈s₁, s₁〉 and ‖s₁‖⁻² ≔ (‖s₁‖²)⁻¹ when ‖s₁‖² is non-degenerate.

By Proposition A.5.2 in [6], we have γ̇ (f) = h₁(X) − θ = h̃₁(X) and γ̇₁(f) = g₁(X) = g̃₁(X). Let ∏(υ|υ₁) be the projection of υ onto [υ₁], the linear span of υ₁ with respect to f and μ, and ${\upsilon }_1^{\perp }$ the orthogonal complement of [υ₁] with respect to f and μ. Without side information, the efficient influence function ℐ(X, γ (f)) for estimating γ (f) is ℐ(X, γ (f)) = γ̇ (f) and the information bound is ‖ℐ(X, γ (f))‖². In the presence of side information γ₁(f), by Example 3.2.3 in [6], the efficient influence function ℐ(X, γ (f)|γ₁(f)) for estimating γ (f) is

$$ℐ(X,\gamma (f)|{\gamma }_1(f))=\Pi (\mathrm{\gamma ̇}(f)|{\mathrm{\gamma ̇}}_1{(f)}^{\perp })=\mathrm{\gamma ̇}(f)-\Pi (\mathrm{\gamma ̇}(f)|{\mathrm{\gamma ̇}}_1(f))=\mathrm{\gamma ̇}(f)-\langle \mathrm{\gamma ̇}(f),{\mathrm{\gamma ̇}}_1(f)\rangle {\Vert {\mathrm{\gamma ̇}}_1(f)\Vert }^{-2}{\mathrm{\gamma ̇}}_1(f)={\mathrm{h̃}}_1(X)-A_1^{\prime }{\Omega }_1^{-1}{\mathrm{g̃}}_1(X)$$

and the information (lower) bound for estimating θ, with side information g, is $𝕀(\theta |g)={\Vert ℐ(X,\gamma (f)|{\gamma }_1(f))\Vert }^2={\eta }_1^2-A_1^{\prime }{\Omega }_1^{-1}{A}_1$.

When g(x) = (g(x₁) + ⋯ + g(x_m))/m, we have g̃₁(x₁) = g₁(x₁) = E[g(x₁, …, x_m)|x₁] = g(x₁)/m, A = E[g(X)h(X)] = E[g(X₁)h₁(X₁)] = mE[g₁(X₁)h₁(X₁)] = mE[ g̃₁(X₁)h̃₁(X₁)] = mA₁, $\Omega =E[g(\mathbf{X})g\prime (\mathbf{X})]=E[g(X_1)g\prime (X_1)]/m=mE[{\mathrm{g̃}}_1(X_1){\mathrm{g̃}}_1^{\prime }(X_1)]=m{\Omega }_1$, thus

$${\sigma }^2=m^2({\eta }_1^2-2A\prime {\Omega }^{-1}{A}_1+A\prime {\Omega }^{-1}{\Omega }_1{\Omega }^{-1}A)=m^2𝕀(\theta |g).$$

Since m² is a known positive constant, we can just divide Ũ_n by m so that its asymptotic variance is 𝕀(θ|g), and thus it is efficient.

Since σ² = ‖∏(γ̇ (f)| γ̇₁(f)^⊥)‖² ≥ 0, with “=” iff γ̇ (f) = h̃₁(X) ∈ [γ̇₁(f)], the linear span of γ̇₁(f) = g̃₁(X), or θ is completely determined by g̃₁(X), which is impossible. Also ${\Vert \prod (\mathrm{\gamma ̇}(f)|{\mathrm{\gamma ̇}}_1{(f)}^{\perp })\Vert }^2\le {\Vert \mathrm{\gamma ̇}(f)\Vert }^2={\eta }_1^2$, with “=” iff γ̇ (f) ∈ [γ̇₁(f)^⊥], or 0 = 〈γ̇ (f), γ̇₁(f)〉 = A.

(ii) Let f (x|θ, g) be the density function given the parameter θ and the information constraint g, S(x|θ, g) = ∂ log f (x|θ, g)/∂θ be the corresponding score function. The corresponding Fisher information is I(θ|g) = ‖S(X|θ, g)‖². Although S(x|θ, g), hence I(θ|g), is not directly available, the corresponding efficient influence function ℐ(X, γ (f)|γ₁(f)) is given in (i), and we have the following relationship between the information bound 𝕀(θ|g) and the Fisher information I(θ|g)

$$𝕀(\theta |g)={\Vert ℐ(X,\gamma (f)|{\gamma }_1(f))\Vert }^2={\eta }_1^2-A\prime {\Omega }^{-1}A=I^{-1}(\theta |g).$$

Let $L(X^n|\theta ,g)={\sum }_{i=1}^n\text{log}f(X_i|\theta ,g)$ be the log-likelihood, we have the following locally asymptotic normality [22] of the likelihood ratio

$${\lambda }_n≔L(X^n|{\theta }_n)-L(X^n|\theta )=b{V}_n-b^{2}I(\theta |g)/2+o_P(1),$$

where $V_n=n^{-1/2}{\sum }_{i=1}^{n}S(X_i|\theta ,g)\stackrel{D}{\to }V~N(0,I(\theta |g))$.

Let ϕ_Y (t) = E[exp{itY}] be the characteristic function of a random variable Y. We are to show lim_n ϕ_Wn (t) = ϕ_U (t)ϕ_Z (t). In fact, by assumption of regularity,

$${\varphi }_{W_n}(t)=E_{f(·|\theta )}[\text{exp}\{it{W}_n\}]=E_{f(·|{\theta }_n)}[\text{exp}\{it(W_n-b)\}]=E_{f(·|\theta )}[\text{exp}\{it(W_n-b)+{\lambda }_n\}]\to E[\text{exp}\{it(W-b)+bV-b^{2}I(\theta |g)/2\}],$$

where the last step above is by the same argument as in [4]. Since b ∈ C is arbitrary, take b = −itI⁻¹(θ|g), we get

$$it(W-b)+bV-b^{2}I(\theta |g)/2=it(W-I^{-1}(\theta |g)V)-I^{-1}(\theta |g)t^2/2=it(W-𝕀(\theta |g)V)-𝕀(\theta |g)t^2/2,$$

thus

$$\underset{n}{\text{lim}}{\varphi }_{W_n}(t)=E[\text{exp}\{it(W-𝕀(\theta |g)V)\}]\text{exp}\{-𝕀(\theta |g)t^2/2\}={\varphi }_{W-𝕀(\theta |g)V}(t){\varphi }_Z(t).$$

Now take U = W − 𝕀(θ|g)V, the proof is complete.

Proof of Theorem 4. (i) Denote the related U-statistics as functions of h, and note the O(·) terms in the lemma are independent of h. Note U_1,n is a functional of gh, U_2,n is a functional of $(1_d^{\prime }g+{\Vert g\Vert }^2)h$, θ and U_n are functionals of h, and U_0,n is a functional of g. As in the proof of Theorem 1(ii), we have

$${\mathrm{P̃}}_{n,m}h-P_{m}h={Ũ}_n(h)-\theta (h)=U_n(h)-\theta (h)-U_{1,n}^{\prime }(gh){\Omega }^{-1}{U}_{0,n}(g)+O({\rho }_n{n}^{-1/2}{(\text{log log}n)}^{1/2})U_{1,n}(gh)+O({\rho }_n^2)U_{2,n}((1_{d}g+{\Vert g\Vert }^2)h),(\mathrm{a}.\mathrm{s}.).$$

Since U_0,n(g) → 0 (a.s.) and is independent of h, we only need to show, a.s.,

$$\underset{h\in \mathscr{H}}{\text{sup}}|U_n(h)-\theta (h)|\to 0;\underset{h\in \mathscr{H}}{\text{sup}}|U_{1,n}(gh)|<\infty ,;\text{and}\underset{h\in \mathscr{H}}{\text{sup}}|U_{2,n}((1_d^{\prime }g+{\Vert g\Vert }^2)h)|<\infty .$$

In fact, since gh ∈ ℋ for all h ∈ ℋ, and U_1,n(h) = U_n(h), we have sup_h∈ℋ |U_1,n(gh)| ≤ sup_h∈ℋ |U_1,n(gh) − P_m(gh)| + sup_h∈ℋ |P_m(gh)| ≤ sup_h∈ℋ |U_1,n(h) − P_m(h)| + sup_h∈ℋ |P_m(h)| = sup_h∈ℋ |U_n(h) − θ(h)| + sup_h∈ℋ |P_m(h)|. Since ℋ has an integrable envelope H, sup_h∈ℋ |P_m(h)| ≤ P_m(sup_h∈ℋ |h|) ≤ P_m(H) < ∞. Thus, sup_h∈ℋ |U_1,n(gh)| < ∞ (a.s.), if sup_h∈ℋ |U_n(h) − θ(h)| → 0 (a.s.).

Similarly, since ‖g‖²h ∈ ℋ for all h ∈ ℋ, and U_2,n(h) = U_n(h), we have sup_h∈ℋ |U_2,n((1_dg + ‖g‖²)h)| < ∞ (a.s.), if sup_h∈ℋ |U_n(h) − θ(h)| → 0 (a.s.).

Now we only need to prove sup_h∈ℋ |U_n(h) − θ(h)| → 0 (a.s.), (the class ℋ is then called P-Glivenko–Cantelli). Since the property of P-Glivenko–Cantelli is permanent for finite union of classes, we only need to prove this on ℋ₁ and ℋ₂ separately.

We first prove ℋ₁ is P-Glivenko–Cantelli. For ε > 0, let N_{[ ]}(ε, ℋ₁, L₁(P_m)) be the bracketing entropy of the class ℋ₁ with L₁(P) norm: ∀h ∈ ℋ₁, ‖h‖_Pm = E_Pm|h|. We first prove that if N_{[ ]}(ε, ℋ₁, L₁(P_m)) < ∞ for all ε > 0, then the conclusion is true. In fact, given ε > 0, since N_{[ ]}(ε, ℋ₁, L₁(P_m)) < ∞, there are finite many ε-brackets [l_i, u_i] whose union covers ℋ₁ and such that P_m(u_i − l_i) < ε for all i. Then for any h ∈ ℋ₁, there is an upper bracket u_i such that

$$U_n(h)-P_m(h)=(P_{n,m}-P_m)h=(P_{n,m}-P_m)u_i+(P_m-P_{n,m})(u_i-h)\le (P_{n,m}-P_m)u_i+P_m(u_i-h)\le (P_{n,m}-P_m)u_i+\epsilon .$$

Consequently,

$$\underset{h\in {\mathscr{H}}_1}{\text{sup}}(U_n(h)-P_m(h))\le \underset{i}{\text{max}}(P_{n,m}-P_m)u_i+\epsilon .$$

Since by SLLN for U-statistics, (P_n,m − P_m)u_i → 0 (a.s.), thus lim_n sup_h∈ℋ1 (U_n(h) − P_m(h)) ≤ ε (a.s.). Similarly, lim_n inf_h∈ℋ1 (U_n(h) − P_m(h)) ≥ −ε (a.s.). Since ε > 0 is arbitrary, we get sup_h∈ℋ1 |U_n(h) − P_m(h)| → 0 (a.s.).

Now we show N_{[ ]}(ε, ℋ₁, L₁(P_m)) < ∞ for all ε > 0. Let $I_j^1=\{x\in R^m:\Vert x-I_j\Vert <1\}$. By Corollary 2.7.4 in VW, for some constant K depending only on υ and m,

$$\text{log}{N}_{[]}(\epsilon ,{\mathscr{H}}_1,L_1(P_m))\le K{\epsilon }^{-\upsilon }{\left(\sum _{j=1}^{\infty }\lambda {(I_j^1)}^{\frac{1}{\upsilon +1}}{M}_j^{\frac{\upsilon }{\upsilon +1}}{P}_m{(I_j)}^{\frac{\upsilon }{\upsilon +1}}\right)}^{\upsilon +1},$$

for every ε > 0, υ ≥ 1 and probability P. The given condition implies that max_j $\lambda (I_j^1)<\infty$. Take υ = 1, the right hand side above is finite by the given condition.

Now we show sup_h∈ℋ2 |U_n(h) − θ(h)| → 0 (a.s.). Let, for ε > 0, N(ε, ℋ₂, L₁(Q)) be the entropy of ℋ₂ without bracketing under norm L₁(Q) for some probability measure Q and N(ε, ℋ₂, ‖·‖_∞) be that under norm ‖·‖_∞. Recall H is also an envelope function on ℋ₂. Since L₁(Q) ≤ ‖·‖_∞, N(ε ‖H‖_Q, ℋ₂, L₁(Q)) ≤ N(ε ‖H‖_Q, ℋ₂, ‖·‖_∞), with ‖H‖_Q = (∫ H²dQ)^1/2. Let M be the bound on ℋ₂, and ℋ̃₂ = {(h − inf_x∈C h(x))/M : h ∈ ℋ₂}, then ℋ̃₂ is the class of convex functions h : C ↦ [0, 1] with Lipschitz constant L/M, and N(ε, ℋ₂, ‖·‖_∞) = N(εM, ℋ̃₂, ‖·‖_∞). By Corollary 2.7.10 in VW, for any ε > 0,

$$\text{log}N(\epsilon M,{\mathrm{\mathscr{H}̃}}_2,{\Vert ·\Vert }_{\infty })\le K{(1+L/M)}^{m/2}{M}^{-m/2}{\epsilon }^{-m/2},$$

for any probability measure Q, and K only depends on m and C. Also, since H is an envelope function over ℋ₂ ≠ {0}, thus inf_Q ‖H‖_Q ≥ δ for some δ > 0, and the infimum is over 𝒬 of all probability measures Q on C, with ‖H‖_Q < ∞. Thus we have, for any ε > 0,

$$\text{log}N(\epsilon {\Vert H\Vert }_{P_m},{\mathscr{H}}_2,L_1(P_m))\le \underset{Q\in 𝒬}{\text{sup}}\text{log}N(\epsilon {\Vert H\Vert }_Q,{\mathscr{H}}_2,L_1(Q))\le \text{log}N(\epsilon \delta ,{\mathscr{H}}_2,{\Vert ·\Vert }_{\infty })=\text{log}N(\epsilon \delta M,{\mathrm{\mathscr{H}̃}}_2,{\Vert \cdot \Vert }_{\infty })\le K{(1+L/M)}^{m/2}{M}^{-m/2}{(\epsilon \delta )}^{-m/2}<\infty ,$$

also, ℋ₂ is P-measurable by its definition, thus ℋ₂ is P-Glivenko–Cantelli (cf. the statement in lines −5 to −3, p. 84 of VW).

(ii) The class ℋ with the stated property is called P-Donsker. First, it is apparent that for any k and h₁, …, h_k ∈ ℋ, $({\mathrm{𝔾̃}}_{n,m}{h}_1,\dots ,{\mathrm{𝔾̃}}_{n,m}{h}_k)\prime \stackrel{D}{\to }(𝔾h_1,\dots ,𝔾h_k)\prime$ for the Gaussian process 𝔾 on ℋ as stated. So by Theorem 1.5.4 in VW, we only need to show that {𝔾̃_n,m} is asymptotically tight on ℋ. Using Lemma (ii) and similar argument as in the proof of (i), since $\sqrt{n}{\rho }_n^2=o(1)$ and $\sqrt{n}{\rho }_n{n}^{-1/2}{(\text{log log}n)}^{1/2}=o(1)$, we only need to show this for {𝔾_n,m}, and by Theorem 1.5.7 in VW, we only need to show that {𝔾_n,m} is asymptotically equicontinuous and totally bounded on ℋ. Below we will show that if

$${\int }_0^{\infty }\underset{Q\in 𝒬}{\text{sup}}\sqrt{\text{log}N(\epsilon {\Vert H\Vert }_{Q,2},\mathscr{H},L_2(Q))}d\epsilon <\infty$$ (A.2)

then {𝔾_n,m} is asymptotically equicontinuous and totally bounded on ℋ, where 𝒬 is the collection of all measures Q with ‖H‖_Q < ∞.

With (A.2), Theorem 2.5.2 in VW asserted the corresponding conclusion for empirical measures. Now we extend the result to U-statistics. For this, we point out that the symmetrization Lemma 2.3.1 in VW still holds for U-statistics, also Hoeffding’s inequality holds for U-statistics (Arcones and Giné [2, Proposition 2.3, p. 1501]), thus the proofs there are valid in our situation.

To check (A.2) on ℋ, and we only need to check it for ℋ₁ and ℋ₂ separately. Using Corollary 2.7.4 in VW, we have

$$\text{log}{N}_{[]}(\epsilon ,{\mathscr{H}}_1,L_2(P))\le K{\epsilon }^{-\upsilon }{(\sum _{j=1}^{\infty }\lambda {(I_j^1)}^{\frac{2}{\upsilon +2}}{M}_j^{\frac{2\upsilon }{\upsilon +2}}{P}^{\frac{\upsilon }{\upsilon +2}}(I_j))}^{\frac{\upsilon +2}{2}},$$

for all ε > 0, υ ≥ m/s. Since the given condition implies max_j $\lambda (I_j^1)<\infty$, in the above inequality choose υ = m/s, then ${\sum }_{j=1}^{\infty }\lambda {\left(I_j^1\right)}^{\frac{2}{\upsilon +2}}{M}_j^{\frac{2\upsilon }{\upsilon +2}}{P}^{\frac{\upsilon }{\upsilon +2}}(I_j)<\infty$ by the given condition, and since υ < 2, we have

$${\int }_0^1\sqrt{\text{log}{N}_{[]}(\epsilon ,{\mathscr{H}}_1,L_2(P))}d\epsilon <\infty ,$$

hence by the statement in p. 85 in VW, ℋ₁ satisfies (A.2). The original statement in VW is for the integral ${\int }_0^{\infty }$. Since ℋ₁ has a square integrable envelope function H, so ∀h₁, h₂ ∈ ℋ, ‖h₁ − h₂‖_L2(P) ≤ ‖h₁‖_L2(P) + ‖h₂‖_L2(P) ≤ 2‖H‖_L2(P) < ∞, i.e., ℋ₁ itself is a ball with radius no greater than 2‖H‖_L2(P), or N_{[ ]}(ε, ℋ₁, L₂(P)) = 1 for ε ≥ 2‖H‖_L2(P), thus its entropy is zero for ε ≥ 2‖H‖_L2(P), so the integration ${\int }_0^{\infty }$ is finite iff ${\int }_0^1$ is finite.

For ℋ₂, similarly as in the proof of (i), for some η > 0,

$$\underset{Q\in 𝒬}{\text{sup}}\text{log}N(\epsilon {\Vert H\Vert }_Q,{\mathscr{H}}_2,L_2(Q))\le K{(1+L/M)}^{m/2}{M}^{-m/2}{(\epsilon \eta )}^{-m/2}.$$

Since m < 4, so

$${\int }_0^1\underset{Q\in 𝒬}{\text{sup}}\sqrt{\text{log}N(\epsilon {\Vert H\Vert }_Q,{\mathscr{H}}_2,L_2(Q))}d\epsilon \le K^{1/2}{(1+L/M)}^{m/4}{M}^{-m/4}{\eta }^{-m/4}{\int }_0^1{\epsilon }^{-m/4}d\epsilon <\infty ,$$

thus by (2.1.7) in VW, ℋ₂ is P-Donsker.

Proof of Corollary 2. From proof of Theorem 4(i), we only need to show sup_h∈ℋ |P_n,m − P_mh| → 0 (a.s.), which is true by Corollary 3.3, or 3.5 respectively in [2].

Proof of Theorem 5. (i) As in the proofs of the previous theorems, with g replaced by G, since r_o ≔ min{rank(g₁), …, rank(g_d), rank(h)} = 1, by Lemma (ii) we have

$$w_{\mathbf{i}}=\frac{1}{C_n^m}(1-G\prime ({\mathbf{X}}_{\mathbf{i}}|\theta ){\Lambda }^{-1}\frac{1}{C_n^m}\sum _{\mathbf{j}\in D_{n,m}}G({\mathbf{X}}_{\mathbf{j}}|\theta )+[1_{d+1}^{\prime }G({\mathbf{X}}_{\mathbf{i}}|\theta )+{\Vert G({\mathbf{X}}_{\mathbf{i}}|\theta )\Vert }^2]O_P(n^{-1})),$$

and by standard U-statistics theory,

$$\sqrt{n}\frac{1}{C_n^m}\sum _{\mathbf{j}\in D_{n,m}}G({\mathbf{X}}_{\mathbf{j}}|\theta )\stackrel{D}{\to }N(0,m^2{\Lambda }_1),\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}G({\mathbf{X}}_{\mathbf{i}}|\theta )G\prime ({\mathbf{X}}_{\mathbf{i}}|\theta )=\Lambda +O_P(n^{-1/2}).$$

Also, $t≔{\Lambda }^{-1}\frac{1}{C_n^m}{\sum }_{\mathbf{j}\in D_{n,m}}G({\mathbf{X}}_{\mathbf{j}}|\theta )+O_P(n^{-1})=O_P(n^{-1/2}),\frac{1}{C_n^m}{\sum }_{\mathbf{i}\in D_{n,m}}{\Vert G({\mathbf{X}}_{\mathbf{i}}|\theta )\Vert }^2\stackrel{\mathrm{a}.\mathrm{s}.}{\to }E{\Vert G(\mathbf{X}|\theta )\Vert }^2<\infty$, and max_i $|t\prime G({\mathbf{X}}_{\mathbf{i}}|\theta )|=O_P(n^{-1/2}{n}^{m/\alpha })\stackrel{P}{\to }0$ since m/α < r_o/2 = 1/2, so

$$-\frac{2n}{C_n^m}{R}_G(\theta )=\frac{2n}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}\text{log}[1+t\prime G({\mathbf{X}}_{\mathbf{i}}|\theta )]=\frac{2n}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}(t\prime G({\mathbf{X}}_{\mathbf{i}}|\theta )-\frac{1}{2}t\prime G({\mathbf{X}}_{\mathbf{i}}|\theta )G({\mathbf{X}}_{\mathbf{i}}|\theta )t+o_P(n^{-1}){\Vert G({\mathbf{X}}_{\mathbf{i}}|\theta )\Vert }^2)=2n\frac{1}{C_n^m}\sum _{\mathbf{j}\in D_{n,m}}G\prime ({\mathbf{X}}_{\mathbf{j}}|\theta ){\Lambda }^{-1}\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}G({\mathbf{X}}_{\mathbf{i}}|\theta )-\frac{n}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}(\frac{1}{C_n^m}\sum _{\mathbf{j}\in D_{n,m}}G\prime ({\mathbf{X}}_{\mathbf{j}}|\theta ){\Lambda }^{-1})\times G({\mathbf{X}}_{\mathbf{i}}|\theta )G\prime ({\mathbf{X}}_{\mathbf{i}}|\theta )({\Lambda }^{-1}\frac{1}{C_n^m}\sum _{\mathbf{j}\in D_{n,m}}G({\mathbf{X}}_{\mathbf{j}}|\theta ))+o_P(n^{-1})\frac{n}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}{\Vert G({\mathbf{X}}_{\mathbf{i}}|\theta )\Vert }^2=2n\frac{1}{C_n^m}\sum _{\mathbf{j}\in D_{n,m}}G\prime ({\mathbf{X}}_{\mathbf{j}}|\theta ){\Lambda }^{-1}\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}G({\mathbf{X}}_{\mathbf{i}}|\theta )-n(\frac{1}{C_n^m}\sum _{\mathbf{j}\in D_{n,m}}G\prime ({\mathbf{X}}_{\mathbf{j}}|\theta ){\Lambda }^{-1})\times \frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}G({\mathbf{X}}_{\mathbf{i}}|\theta )G\prime ({\mathbf{X}}_{\mathbf{i}}|\theta )({\Lambda }^{-1}\frac{1}{C_n^m}\sum _{\mathbf{j}\in D_{n,m}}G({\mathbf{X}}_{\mathbf{j}}|\theta ))+o_P(1)=2n\frac{1}{C_n^m}\sum _{\mathbf{j}\in D_{n,m}}G\prime ({\mathbf{X}}_{\mathbf{j}}|\theta ){\Lambda }^{-1}\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}G({\mathbf{X}}_{\mathbf{i}}|\theta )-n\frac{1}{C_n^m}\sum _{\mathbf{j}\in D_{n,m}}G({\mathbf{X}}_{\mathbf{j}}|\theta ){\Lambda }^{-1}\times (\Lambda +O_P(n^{-1/2})){\Lambda }^{-1}\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}G\prime ({\mathbf{X}}_{\mathbf{i}}|\theta )+o_P(1)=n\frac{1}{C_n^m}\sum _{\mathbf{j}\in D_{n,m}}G\prime ({\mathbf{X}}_{\mathbf{j}}|\theta ){\Lambda }^{-1}\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}G({\mathbf{X}}_{\mathbf{i}}|\theta )-O_P(n^{-1/2})+o_P(1).$$

This completes the proof since

$$\sqrt{n}{m}^{-1}{\Lambda }_1^{-1/2}\frac{1}{C_n^m}\sum _{\mathbf{i}\in D_{n,m}}G({\mathbf{X}}_{\mathbf{i}}|\theta )\stackrel{D}{\to }N(0,I_{r+1}).$$

(ii) This is a special case of (i).