HIGHER ORDER ESTIMATING EQUATIONS FOR HIGH-DIMENSIONAL MODELS

Abstract

We introduce a new method of estimation of parameters in semi-parametric and nonparametric models. The method is based on estimating equations that are U-statistics in the observations. The U-statistics are based on higher order influence functions that extend ordinary linear influence functions of the parameter of interest, and represent higher derivatives of this parameter. For parameters for which the representation cannot be perfect the method leads to a bias-variance trade-off, and results in estimators that converge at a slower than $\sqrt{n}\text{-rate}$. In a number of examples the resulting rate can be shown to be optimal. We are particularly interested in estimating parameters in models with a nuisance parameter of high dimension or low regularity, where the parameter of interest cannot be estimated at $\sqrt{n}\text{-rate}$, but we also consider efficient $\sqrt{n}\text{-estimation}$ using novel nonlinear estimators. The general approach is applied in detail to the example of estimating a mean response when the response is not always observed.

1. Introduction

Let X₁, X₂, …, X_n be a random sample from a density p relative to a measure μ on a sample space $(\mathcal{X},\mathcal{A})$. It is known that p belongs to a collection $\mathcal{P}$ of densities, and the problem is to estimate the value χ(p) of a functional $\chi :\mathcal{P}\to ℝ$. Our main interest is in the situation of a semiparametric or nonparametric model, where $\mathcal{P}$ is infinite dimensional.

Estimating equations have been found a good strategy for constructing estimators in semiparametric models [2, 34, 40]. Because the model is of (much) higher dimension than the parameter of interest, setting up a good estimating equation often requires an initial estimator ${\widehat{\eta }}_n$ of a “nuisance parameter”, and an estimating equation for θ = χ(p) may take the form

$${\mathbb{P}}_n{\psi }_{\theta },{\widehat{\eta }}_n=0.$$

Here ${\mathbb{P}}_{n}f$ is short for $n^{-1}{\displaystyle {\sum }_{i=1}^{n}f(X_i)}$, and x ↦ ψ_θ,η(x) is a given measurable map, for each (θ, η). In the present paper it will be more convenient to work with a one-step version of this estimator, defined by the method of Newton-Rhapson from a linearization of the map $\theta \mapsto {\mathbb{P}}_n{\psi }_{\theta ,{\widehat{\eta }}_n}$ around an initial estimator ${\widehat{\theta }}_n$, leading to an estimator of the form ${\widehat{\theta }}_n+{\widehat{V}}_n^{-1}{\mathbb{P}}_n{\psi }_{{\widehat{\theta }}_n,{\widehat{\eta }}_n}$, for ${\widehat{V}}_n$ an estimate of the derivative of the estimating equation. In more general notation such an estimator can be written as

$${\widehat{\chi }}_n=\chi ({\widehat{p}}_n)+{\mathbb{P}}_n{\chi }_{{\widehat{p}}_n},$$ (1.1)

for ${\widehat{p}}_n$ an initial estimator for p, and x ↦ χ_p(x) a given measurable function, for each $p\in \mathcal{P}$.

One possible choice in (1.1) is χ_p = 0, leading to the plug-in estimator $\chi ({\widehat{p}}_n)$. However, unless the initial estimator ${\widehat{p}}_n$possesses special properties, this choice is typically suboptimal. Better functions χ_p can be constructed by consideration of the tangent space of the model. To see this, we write (with $P{\chi }_{\widehat{p}}$ shorthand for ${\displaystyle \int {\chi }_{\widehat{p}}(x)dP(x)}$)

$${\widehat{\chi }}_n-\chi (p)=\left[\chi ({\widehat{p}}_n)-\chi (p)+P{\chi }_{{\widehat{p}}_n}\right]+({\mathbb{P}}_n-P){\chi }_{{\widehat{p}}_n}.$$ (1.2)

Because it is properly centered, we may expect the sequence $\sqrt{n}({\mathbb{P}}_n-P){\chi }_{{\widehat{p}}_n}$ to tend in distribution to a mean-zero normal distribution. The term between square brackets on the right of (1.2), which we shall refer to as the bias term, depends on the initial estimator ${\widehat{p}}_n$, and it would be natural to construct the function χ_p such that this term does not contribute to the limit distribution, or at least is not dominating the expression. Thus we would like to choose this function such that the “bias term” is at least of the order O_P (n^−1/2). A good choice is to ensure that the term $P{\chi }_{{\widehat{p}}_n}$ acts as minus the first derivative of the functional χ in the “direction” ${\widehat{p}}_n-p$. Functions x ↦ χ_p(x) with this property are known as influence functions in semiparametric theory [16, 22, 35, 5, 2], go back to the von Mises calculus due to [33], and play an important role in robust statistics [14, 11], or [40], Chapter 20.

For an influence function we may expect that the “bias term” is quadratic in the error $d({\widehat{p}}_n,p)$, for an appropriate distance d. In that case it is certainly negligible as soon as this error is of order o_P (n^−1/4). Such a “no-bias” condition is well known in semiparametric theory (e.g. condition (25.52) in [40] or (11) in [20]). However, typically it requires that the model $\mathcal{P}$ be “not too big”. For instance, a regression or density function on d-dimensional space can be estimated at rate n^−1/4 if it is a-priori known to have at least d/2 derivatives (indeed α/(2α + d) ≥ 1/4 if α ≥ d/2). The purpose of this paper is to develop estimation procedures for the case that no estimators exist that attain a O_P (n^−1/4) rate of convergence. The estimator (1.1) is then suboptimal, because it fails to make a proper trade-off between “bias” and “variance”: the two terms in (1.2) have different magnitudes. Our strategy is to replace the linear term ${\mathbb{P}}_n{\chi }_p$ by a general U-statistic ${\mathbb{U}}_n{\chi }_p$, for an appropriate m-dimensional influence function (x₁, …, x_m) ↦ χ_p(x₁, …, x_m), chosen using a von Mises expansion of p ↦χ(p). Here the order m is adapted to the size of the model $\mathcal{P}$ and the type of functional to be estimated.

Unfortunately, “exact” higher-order influence functions turn out to exist only for special functionals χ. To treat general functionals χ we approximate these by simpler functionals, or use approximate influence functions. The rate of the resulting estimator is then determined by a trade-off between bias and variance terms. It may still be of order $1/\sqrt{n}$, but it is often slower. In the former case, surprisingly, one may obtain semiparametric efficiency by estimators whose variance is determined by the linear term, but whose bias is corrected using higher order influence functions.

The conclusion that the “bias term” in (1.2) is quadratic in the estimation error $d({\widehat{p}}_n,p)$ is based on a worst case analysis. First, there exist a large number of models and functionals of interest that permit a first order influence function that is unbiased in the nuisance parameter. (E.g. adaptive models as considered in [1], models allowing a sufficient statistic for the nuisance parameter as in [38, 39], mixture models as considered in [19, 24, 36], and convex-linear models in survival analysis.) In such models there is no need for higher-order estimating equations. Second, the analysis does not take special, structural properties of the initial estimators ${\widehat{p}}_n$ into account. An alternative approach would be to study the bias of a particular estimator in detail, and adapt the estimating equation to this special estimator. The strategy in this paper is not to use such special properties and focus on estimating equations that work with general initial estimators ${\widehat{p}}_n$.

The motivation for our new estimators stems from studies in epidemiology and econometrics that include covariates whose influence on an outcome of interest cannot be reliably modelled by a simple model. These covariates may themselves not be of interest, but are included in the analysis to adjust the analysis for possible bias. For instance, the mechanism that describes why certain data is missing is in terms of conditional probabilities given several covariates, but the functional form of this dependence is unknown. Or, to permit a causal interpretation in an observational study one conditions on a set of covariates to control for confounding, but the form of the dependence on the confounding variables is unknown. One may hypothesize in such situations that the functional dependence on a set of (continuous) covariates is smooth (e.g. d/2 times differentiable in the case of d covariates), or even linear. Then the usual estimators will be accurate (at order O_P(n^−1/2)) if the hypothesis is true, but they will be badly biased in the other case. In particular, the usual normal-theory based confidence intervals may be totally misleading: they will be both too narrow and wrongly located. The methods in this paper yield estimators with (typically) wider corresponding confidence intervals, but they are correct under weaker assumptions.

The mathematical contributions of the paper are to provide a heuristic for constructing minimax estimators in semiparametric models, and to apply this to a concrete model, which is a template for a number of other models (see [27, 37]). The methods connect to earlier work [13, 21] on the estimation of functionals on nonparametric models, but differs by our focus on functionals that are defined in terms of the structure of a semiparametric model. This requires an analysis of the inverse map from the density of the observations to the parameters, in terms of the semiparametric tangent spaces of the models. Our second order estimators are related to work on quadratic functionals, or functionals that are well approximated by quadratic functionals, as in [10, 15, 3, 4, 17, 18, 6, 7]. While we place the construction of minimax estimators for these special functionals in a wider framework, our focus differs by going beyond quadratic estimators and to consider semiparametric models.

Our mathematical results are in part conditional on a scale of regularity parameters, through a dimension (8.9) and partition of this dimension that depends on two of these parameters. We hope to discuss adaptation to these parameters in future work.

General heuristics of our construction are given in Section 3. Sections 4–8 are devoted to constructing new estimators for the mean response effect in missing data problems. In Section 9 we briefly discuss some other problems, including the problem of estimating a density at a point, where already first order influence functions do not exist and our heuristics naturally lead to projection estimators. Section 10 collects technical proofs. Sections 11, 12 and 13 (in the supplement [26]) discuss three key concepts of the paper: influence functions, projections and U-statistics.

2. Notation

Let ${\mathbb{U}}_n$ denote the empirical U-statistic measure, viewed as an operator on functions. For given k ≤ n and a function $f:{\mathcal{X}}^k\to ℝ$ on the sample space this is defined by

$${\mathbb{U}}_{n}f=\frac{1}{n(n-1)\cdots (n-k+1)}{\displaystyle \sum _{1\le i_1}{\displaystyle \sum _{\ne i_2\ne \cdots }\cdots {\displaystyle \sum _{\ne i_k\le n}f(X_{i_1},X_{i_2},\cdots ,X_{i_k})}}}.$$

We do not let the order k show up in the notation ${\mathbb{U}}_{n}f$. This is unnecessary, as the notation is consistent in the following sense: if a function $f:{\mathcal{X}}^l\to ℝ$ of l < k arguments is considered a function of k arguments that is constant in its last k −l arguments, then the right side of the preceding display is well defined and is exactly the corresponding U-statistic of order l. In particular, ${\mathbb{U}}_{n}f$ is the empirical distribution ${\mathbb{P}}_n$ applied to f if $f:\mathcal{X}\to ℝ$ depends on only one argument.

We write $P^n{\mathbb{U}}_{n}f=P^{k}f$ for the expectation of ${\mathbb{U}}_{n}f$ if X₁, …, X_n are distributed according to the probability measure P. We also use this operator notation for the expectations of statistics in general.

We call f degenerate relative to P if ∫ f(x₁, …, x_k) dP(x_i) = 0 for every i and every (x_j: j ≠ i), and we call f symmetric if f(x₁, …, x_k) is invariant under permutation of the arguments x₁, …, x_k. Given an arbitrary measurable function $f:{\mathcal{X}}^k\to ℝ$ we can form a function that is degenerate relative to P by subtracting the orthogonal projection in L₂ (P^k) onto the functions of at most k − 1 variables. This degenerate function can be written in the form (e.g. [40], Lemma 11.11)

$$(D_{P}f)(X_1,\dots ,X_k)={\displaystyle \sum _{A\subset \{1,\dots ,k\}}{(-1)}^{k-\left|A\right|}{\mathrm{E}}_P\left[f(X_1,\dots ,X_k)|X_i:i\in A\right]},$$ (2.1)

where the sum if over all subsets A of {1, …, k}, including the empty set, for which the conditional expectation is understood to be P^kf. If the function f is symmetric, then so is the function D_Pf.

Given two functions $g,h:\mathcal{X}\to ℝ$ we write g × h for the function (x, y) ↦ g(x)h(y). More generally, given k functions g₁, …,g_k we write g₁ ×⋯× g_k for the tensor product of these functions. Such product functions are·degenerate iff all functions in the product have mean zero.

A kernel operator $K:L_r(\mathcal{X},\mathcal{A},\mu )\to L_r(\mathcal{X},\mathcal{A},\mu )$ takes the form $(Kf)(x)={\displaystyle \int \overline{K}(x,y)f(y)d\mu (y)}$ for some measurable function $\overline{K}:{\mathcal{X}}^2\to ℝ$. We shall abuse notation in denoting the operator K and the kernel $\overline{K}$ with the same symbol: $K=\overline{K}$. A (weighted) projection on a finite-dimensional space is a kernel operator. We discuss such projections in Section 12.

The set of measurable functions whose rth absolute power is μ-integrable is denoted L_r(μ), with norm ‖·‖_r,μ, or ‖·‖_r if the measure is clear; or also as L_r(w) with norm ‖·‖_r,w if w is a density relative to a given dominating measure. For r = ∞ the notation ‖·‖_∞ refers to the uniform norm.

3. Heuristics

Our basic estimator has the form (1.1) except that we replace the linear term by a general U-statistic. Given measurable functions ${\chi }_p:{\mathcal{X}}^m\to ℝ$, for a fixed order m, we consider estimators ${\widehat{\chi }}_n$ of χ(p) of the type

$${\widehat{\chi }}_n=\chi ({\widehat{p}}_n)+{\mathbb{U}}_n{\chi }_{{\widehat{p}}_n}.$$ (3.1)

The initial estimators ${\widehat{p}}_n$ are thought to have a certain (optimal) convergence rate $d({\widehat{p}}_n,p)\to 0$, but need not possess (further) special properties. Throughout we shall treat these estimators as being based on an independent sample of observations, so that the stochasticities in (3.1) present in ${\widehat{p}}_n$ and ${\mathbb{U}}_n$ are independent. This takes away technical complications, and allows us to focus on rates of estimation in full generality. (A simple way to avoid the resulting asymmetry would be to swap the two samples, calculate the estimator a second time and take the average.)

3.1. Influence functions

The key is to find suitable “influence functions” χ_p. A decomposition of type (1.2) for the estimator (3.1) yields

$${\widehat{\chi }}_n-\chi (p)=\left[\chi ({\widehat{p}}_n)-\chi (p)+P^m{\chi }_{{\widehat{p}}_n}\right]+({\mathbb{U}}_n-P^n){\chi }_{{\widehat{p}}_n}.$$ (3.2)

This suggests to construct the influence functions such that $-P^m{\chi }_{{\widehat{p}}_n}$ represents the first m terms of the Taylor expansion of $\chi ({\widehat{p}}_n)-\chi (p)$. First this implies that the influence function used in (3.1) must be unbiased:

$$P^m{\chi }_p=0.$$

Next, to operationalize a “Taylor expansion” on the (infinite-dimensional) “manifold” $\mathcal{P}$ we employ “smooth” submodels t ↦ p_t mapping a neighbourhood of 0 ∈ ℝ to $\mathcal{P}$ and passing through p at t= 0 (i.e. p₀ = p). We determine χ_p such that for each chosen submodel

$${\frac{d^j}{d{t}^j}}_{|t=0}\chi (p_t)=-{\frac{d^j}{d{t}^j}}_{|t=0}{P}^m{\chi }_{p_t},j=1,\dots ,m.$$

A slight strengthening is to impose this condition “everywhere” on the path, i.e. the jth derivative of t ↦ χ(p_t) at t is the jth derivative of $h\mapsto -P_t^m{\chi }_{p_{t+h}}$ at h = 0, for every t. If the map $(s,t)\mapsto P_s^m{\chi }_{p_t}$ is smooth, then the latter implies (cf. Lemma 11.1 applied with χ = f and $g(s,t)=-P_t^m{\chi }_{p_s}$)

$${\frac{d^j}{d{t}^j}}_{|t=0}\chi (p_t)={\frac{d^j}{d{t}^j}}_{|t=0}{P}_t^m{\chi }_p,j=1,\dots ,m.$$ (3.3)

Relative to the previous formula the subscript t on the right hand side has changed places, and the negative sign has disappeared. Under regularity conditions equation (3.3) for m = 1 can be written in the form

$${\frac{d}{dt}}_{|t=0}\chi (p_t)={\frac{d}{dt}}_{|t=0}{P}_t{\chi }_p=P{\chi }_{p}g,$$ (3.4)

where g = (d/dt)_t=0p_t/p is the score function of the model t ↦ p_t at t = 0.

A function χ_p satisfying (3.3) is exactly what is called an influence function in semiparametric theory: a function in L₂(p) whose inner products with the elements of the tangent space (the set of score functions g of appropriate submodels t ↦ p_g,t) represent the derivative of the functional ([40], page 363, or [2, 22, 39, 35]). For m > 1 equation (3.3) can be expanded similarly in terms of inner products of the influence function with score functions, but higher-order score functions arise next to ordinary score functions. Suitable higher-order tangent spaces are discussed in [27] (also see [37]), using score functions as defined in [43]. A discussion of second order scores and tangent spaces can be found in [28]. Second order tangent spaces are also discussed in [23], from a different point of view and with a different purpose.

Here we take a different route, defining higher order influence functions as influence functions of lower order ones. Any mth order, zero-mean U-statistic can be decomposed as the sum of m degenerate U-statistics of orders 1, 2, …, m, by way of its Hoeffding decomposition. In the present situation we can write

$${\mathbb{U}}_n{\chi }_p={\mathbb{U}}_n{\chi }_p^{(1)}+\frac{1}{2}{\mathbb{U}}_n{\chi }_p^{(2)}+\cdots +\frac{1}{m!}{\mathbb{U}}_n{\chi }_p^{(m)},$$

where ${\chi }_p^{(j)}:{\mathcal{X}}^j\to ℝ$ is a degenerate kernel of j arguments, defined uniquely as a projection of χ_p (cf. [42] and (2.1)). (At m = n the left side evaluates to the symmetrized function χ_p, which is expressed in the functions ${\chi }_p^{(j)}$.) Suitable functions ${\chi }_p^{(j)}$ in this decomposition can be found by the following algorithm:

• Let $x_1\mapsto {\overline{\chi }}_p^{(1)}(x_1)$ be a first order influence function of the functional p ↦ χ(p).

• Let $x_j\mapsto {\overline{\chi }}_p^{(j)}(x_1,\dots ,x_j)$ be a first order influence function of the functional $p\mapsto {\overline{\chi }}_p^{(j-1)}(x_1,\dots ,x_{j-1})$, for each x₁, …, x_j−1, and j = 2, …, m.

• Let ${\chi }_p^{(j)}=D_P{\overline{\chi }}_p^{(j)}$ be the degenerate part of ${\overline{\chi }}_p^{(j)}$ relative to P, as defined in (2.1).

See Lemma 11.2 for a proof. Thus higher order influence functions are constructed as first order influence functions of influence functions. Somewhat abusing language we shall refer to the function ${\chi }_p^{(j)}$ also as a “jth order influence function”. The overall order m will be fixed at a suitable value; for simplicity we do not let this show up in the notation χ_p.

Because only inner products with scores matter, an “influence function” is unique only up to projections onto the tangent space and (averaging over) permutations of its arguments. With any choice of influence functions the algorithm produces some influence function. In particular, the starting influence function ${\overline{\chi }}_p^{(1)}$ in step [1] may be any function χ_p that satisfies (3.4) for every score g; it does not have to possess mean zero, or be an element of the tangent space. A similar remark applies to the (first order) influence functions found in step [2]. It is only in step [3] that we make the influence functions degenerate.

3.2. Bias-variance trade-off

Because it is centered, the “variance part” in (3.2), the variable $({\mathbb{U}}_n-P^n){\chi }_{{\widehat{p}}_n}$, should not change noticeably if we replace ${\widehat{p}}_n$ by p, and be of the same order as $({\mathbb{U}}_n-P^n){\chi }_p$. For a fixed square-integrable function χ_p the latter centered U-statistic is well known to be of order O_P(n^−1/2), and asymptotically normal if suitably scaled. A completely successful representation of the “bias” $R_n=\chi ({\widehat{p}}_n)-\chi (p)+P^m{\chi }_{{\widehat{p}}_n}$ in (3.2) would lead to an error $R_n=O_P(d{({\widehat{p}}_n,p)}^{m+1})$, which becomes smaller with increasing order m. Were this achievable for any m, then a $\sqrt{n}\text{-estimator}$ would exist no matter how slow the convergence rate $d({\widehat{p}}_n,p)$ of the initial estimator. Not surprisingly, in many cases of interest this ideal situation is not real. This is due to the non-existence of influence functions that can exactly represent the Taylor expansion of $\chi ({\widehat{p}}_n)-\chi (p)$. The technical reason is that multi-linear maps (even smooth ones) may not be representable through kernels.

In general, we have to content ourselves with a partial representation. Next to a remainder term of order $O_P(d{({\widehat{p}}_n,p)}^{m+1})$, we then incur a “representation bias”. The latter bias can be made arbitrarily small by choice of the influence function, but only at the cost of increasing its variance. We thus obtain a trade-off between a variance and two biases. This typically results in a variance that is larger than 1/n, and a rate of convergence that is slower than $1/\sqrt{n}$, although sometimes a nontrivial bias correction is possible without increasing the variance.

3.3. Approximate functionals

An attractive method to find approximating influence functions is to compute exact influence functions for an approximate functional. Because smooth functionals on finite-dimensional models typically possess influence functions to any order, projections on finite-dimensional models may deliver such approximations.

A simple approximation would be $\chi (\stackrel{\sim }{p})$ for a given map $p\mapsto \stackrel{\sim }{p}$ mapping the model $\mathcal{P}$ onto a suitable “smaller” model $\stackrel{\sim }{\mathcal{P}}$ (typically a submodel $\stackrel{\sim }{\mathcal{P}}\subset \mathcal{P}$). A closer approximation can be obtained by also including a derivative term. Consider the functional $\stackrel{\sim }{\chi }:\mathcal{P}\to ℝ$ defined by, for a given map $p\mapsto \stackrel{\sim }{p}$,

$$\stackrel{\sim }{\chi }(p)=\chi (\stackrel{\sim }{p})+P{\chi }_{\stackrel{\sim }{p}}^{(1)}.$$ (3.5)

(A more complete notation would be $\stackrel{\sim }{p}(p)$; the right hand side depends on p in three ways.) By the definition of an influence function the term $-P{\chi }_{\stackrel{\sim }{p}}^{(1)}$ acts as the first order Taylor expansion of $\chi (\stackrel{\sim }{p})-\chi (p)$ Consequently, we may expect that

$$\left|\stackrel{\sim }{\chi }(p)-\chi (p)\right|=O(d{(\stackrel{\sim }{p},p)}^2).$$ (3.6)

This ought to be true for any “projection” $p\mapsto \stackrel{\sim }{p}$. If we choose the projection such that, for any path $t\mapsto p_t$,

$${\frac{d}{dt}}_{|t=0}\left(\chi ({\stackrel{\sim }{p}}_t)+P_0{\chi }_{{\stackrel{\sim }{p}}_t}^{(1)}\right)=0,$$ (3.7)

then the functional $p\mapsto \stackrel{\sim }{\chi }(p)$ will be locally (around p₀) equivalent to the functional $p\mapsto \chi ({\stackrel{\sim }{p}}_0)+P{\chi }_{{\stackrel{\sim }{p}}_0}^{(1)}$ (which depends on p in only one place, p₀ being fixed) in the sense that the first order influence functions are the same. The first order influence function of the second, linear functional at p₀ is equal to ${\chi }_{{\stackrel{\sim }{p}}_0}^{(1)}$, and hence for a projection satisfying (3.7) the first order influence function of the functional $p\mapsto \stackrel{\sim }{\chi }(p)$ will be

$${\stackrel{\sim }{\chi }}_p^{(1)}={\chi }_{\stackrel{\sim }{p}}^{(1)}.$$ (3.8)

In words, this means that the influence function of the approximating functional $\stackrel{\sim }{\chi }$ satisfying (3.5) and (3.7) at p is obtained by substituting $\stackrel{\sim }{p}$ for p in the influence function of the original functional.

This is relevant when obtaining higher order influence functions. As these are recursive derivatives of the first order influence function (see [1]–[3] in Section 3.1), the preceding display shows that we must compute influence functions of

$$p\mapsto {\chi }_{\stackrel{\sim }{p}}^{(1)}(x),$$

i.e. we “differentiate on the model $\stackrel{\sim }{\mathcal{P}}$”. If the latter model is sufficiently simple, for instance finite-dimensional, then exact higher order influence functions of the functional $p\mapsto \stackrel{\sim }{\chi }(p)$ ought to exist. We can use these as approximate influence functions of $p\mapsto \chi (p)$.

4. Estimating the mean response in missing data models

Suppose that a typical observation is distributed as X = (Y A, A, Z), for Y and A taking values in the two-point set {0, 1} and conditionally independent given Z.

This model is standard in biostatistical applications, with Y an “outcome” or “response variable”, which is observed only if the indicator A takes the value 1. The covariate Z is chosen such that it contains all information on the dependence between the response and the missingness indicator A, thus making the response missing at random. Alternatively, we think of Y as a “counterfactual” outcome if a treatment were given (A = 1) and estimate (half) the treatment effect under the assumption of no unmeasured confounders.

The model can be parameterized by the marginal density f of Z (relative to some dominating measure ν) and the probabilities b(z) = P(Y = 1|Z = z) and a(z)⁻¹ = P(A = 1| Z = z). (Using a for the inverse probability simplifies later formulas.) Alternatively, the model can be parameterized by the pair (a, b) and the function g = f/a, which is the conditional density of Z given A = 1, up to the norming factor P(A = 1). Thus the density p of an observation X is described by the triplet (a, b, f), or equivalently the triplet (a, b, g). For simplicity of notation we write p instead of p_a,b,f or p_a,b,g, with the implicit understanding that a generic p corresponds one-to-one to a generic (a, b, f) or (a, b, g).

We wish to estimate the mean response EY = Eb(Z), i.e. the functional

$$\chi (p)={\displaystyle \int \mathit{bf}\mathit{d\nu }=}{\displaystyle \int \mathit{abg}\mathit{d\nu }}.$$

Estimators that are $\sqrt{n}\text{-consistent}$ and asymptotically efficient in the semi-parametric sense have been constructed using a variety of methods (e.g. [30, 31]), but only if a or b, or both, parameters are restricted to sufficiently small regularity classes. For instance, if the covariate Z ranges over a compact, convex subset $\mathcal{Z}$ of ℝ^d, then the mentioned papers provide $\sqrt{n}\text{-consistent}$ estimators under the assumption that a and b belong to Hölder classes $C^{\alpha }(\mathcal{Z})$ and $C^{\beta }(\mathcal{Z})$ with α and β large enough that

$$\frac{\alpha }{2\alpha +d}+\frac{\beta }{2\beta +d}\ge \frac{1}{2}.$$ (4.1)

(See e.g. Section 2.7.1 in [41] for the definition of Hölder classes). For moderate to large dimensions d this is a restrictive requirement. In the sequel we consider estimation for arbitrarily small α and β.

Throughout we assume that the parameters a, b and g are contained in Hölder spaces $C^{\alpha }(\mathcal{Z})$, $C^{\beta }(\mathcal{Z})$ and $C^{\gamma }(\mathcal{Z})$ of functions on a compact, convex domain in ℝ^d. We derive two types of results:

1. In Section 7 we show that a $\sqrt{n}\text{-rate}$ is attainable by using a higher order estimating equation (of order determined by γ) as long as

   $$\frac{\alpha +\beta }{2}\ge \frac{d}{4}.$$ (4.2)

   This condition is strictly weaker than the condition (4.1) under which the linear estimator attains a $\sqrt{n}\text{-rate}$. Thus even in the $\sqrt{n}\text{-situation}$ higher order estimating equations may yield estimators that are applicable in a wider range of models. For instance, in the case that α = β the cut-off (4.1) arises for α = β ≥ d/4, whereas (4.2) reduces to α = β ≥ d/2.

2. We consider minimax estimation in the case (α + β)/2 < d/4, when the rate becomes slower than $1/\sqrt{n}$. It is shown in [29] that even if g = f/a were known, then the minimax rate for a and b ranging over balls in the Hölder classes $C^{\alpha }(\mathcal{Z})$ and $C^{\beta }(\mathcal{Z})$ cannot be faster than n^{−(2α+2β)/(2α+2β+d)}. In Section 8 we show that this rate is attainable if g is known, and also if g is unknown, but is a-priori known to belong to a Hölder class $C^{\gamma }(\mathcal{Z})$ for sufficiently large γ, as given by (8.11). (Heuristic arguments, not discussed in this paper, appear to indicate that for smaller γ the minimax rate is slower than n^{−(2α+2β)/(2α+2β+d)}.)

After reviewing the tangent space and first order theory in Section 4.1, we discuss the second order estimator separately in Section 5. The preceding results are next obtained in Sections 7 ( $\sqrt{n}\text{-rate}$ if (α + β)/2 ≥ d/4) and 8 (slower rate if (α + β)/2 < d/4), using the higher-order influence functions of an approximate functional, which is defined in the intermediate Section 6.

Assumption 4.1

We assume throughout that the functions 1/a, b, g and their preliminary estimators $1/\widehat{a},\widehat{b},\widehat{g}$ are bounded away from their extremes: 0 and 1 for the first two, and 0 and ∞ for the third.

4.1. Tangent space and first order influence function

The one-dimensional submodels t ↦ p_t induced by paths of the form a_t = a + tα, b_t = b + tβ, and f_t = f(1 + tϕ) for given directions α, β and ϕ (where ∫ ϕ f dν = 0) yield score functions

$$B_p^a\alpha (X)=-\frac{Aa(Z)-1}{a(Z)(a-1)(Z)}\alpha (Z),a-\text{score,}$$

$$B_p^b\beta (X)=\frac{A(Y-b(Z))}{b(Z)(1-b)(Z)}\beta (Z),b-\text{score,}$$

$$B_p^f\varphi (X)=\varphi (Z),f-\text{score}.$$

Here $B_p^a$, $B_p^b$, $B_p^f$ are the score operators for the three parameters, whose direct sum is the overall score operator, which we write as B_p: B_p(α, β, ϕ)(X) is the sum of the three left sides of the preceding equation. The first-order influence function is well known to take the form

$${\chi }_p^{(1)}(X)=Aa(Z)(Y-b(Z))+b(Z)-\chi (p).$$ (4.3)

Indeed, it is straightforward to verify that this function satisfies, for every path t ↦ p_t as described previously,

$${\frac{d}{dt}}_{|t=0}\chi (p_t)={\mathrm{E}}_p{\chi }_p^{(1)}(X)B_p(\alpha ,\beta ,\varphi )(X).$$

The advantage of choosing a an inverse probability is clear from the form of the (random part of the) influence function, which is bilinear in (a, b). The corresponding “first order bias” can be computed to be

$$\chi (\widehat{p})-\chi (p)+P{\chi }_{\widehat{p}}^{(1)}=-{\displaystyle \int (\widehat{a}-a)(\widehat{b}-b)g\mathit{d\nu }}.$$ (4.4)

In agreement with the heuristics given in Sections 1 and 3 this bias is quadratic in the errors of the initial estimator.

Actually, the form of the bias term is special in that square estimation errors ${(\widehat{a}-a)}^2$ and ${(\widehat{b}-b)}^2$ of the two initial estimators $\widehat{a}$ and $\widehat{b}$ do no arise, but only the product $(\widehat{a}-a)(\widehat{b}-b)$ of their errors. This property, termed “double robustness” in [34], makes that for first order inference it suffices that one of the two parameters be estimated well. A prior assumption that the parameters a and b are α and β regular, respectively, would allow estimation errors of the orders n^{−α/(2α+d)} and n^{−β/(2β+d)}. If the product of these rates is O(n^−1/2), then the bias term matches the variance. This leads to the (unnecessarily restrictive) condition (4.1).

If the preliminary estimators $\widehat{a}$ and $\widehat{b}$ are solely selected for having small errors $\Vert \widehat{a}-a\Vert$ and $\Vert \widehat{b}-b\Vert$ (e.g. minimax in the L₂-norm), then it is hard see why (4.4) would be small unless the product $\Vert \widehat{a}-a\Vert \Vert \widehat{b}-b\Vert$ of the errors is small. Special estimators might exploit that the bias is an integral, in which cancellation of errors could occur. As we do not wish to use special estimators, our approach will be to replace the linear estimating equation by a higher order one, leading to an analogue of (4.4) that is a cubic or higher order polynomial of the estimation errors.

It may be noted that the marginal density f (or g) does not enter into the first order influence function (4.3). Even though the functional depends on f (or g), a rate on the initial estimator of this function is not needed for the construction of the first order estimator. This will be different at higher orders.

5. Second order estimator

In this section we derive a second order influence function for the missing data problem, and analyze the risk of the corresponding estimator. This estimator is minimax if (α + β)/2 ≥ d/4 and

$$\frac{\gamma }{2\gamma +d}\ge \frac{1}{2}^\frac{2\alpha +2\beta }{d+2\alpha +2\beta }-\frac{\alpha }{2\alpha +d}-\frac{\beta }{2\beta +d}.$$ (5.1)

In the other case, higher order estimators have smaller risk, as shown in Sections 7-8. However, it is worth while to treat the second order estimator separately, as its construction exemplifies essential elements, without involving technicalities attached to the higher order estimators.

To find a second order influence function, we follow the strategy [1]–[3] of Section 3.1, and try and find a function ${\chi }_p^{(2)}:{\mathcal{X}}^2\to ℝ$ such that, for every x₁ = (y₁a₁, a₁, z₁), and all directions α, β, ϕ,

$${\frac{d}{dt}}_{|t=0}\left[{\chi }_{p_t}^{(1)}(x_1)+\chi (p_t)\right]={\mathrm{E}}_p{\chi }_p^{(2)}(x_1,X_2)B_p(\alpha ,\beta ,\varphi )(X_2).$$

Here the expectation E_p on the right side is relative to the variable X₂ only, with x₁ fixed. This equation expresses that $x_2\mapsto {\chi }_p^{(2)}(x_1,x_2)$ is a first order influence function of $p\mapsto {\chi }_p^{(1)}(x_1)+\chi (p)$, for fixed x₁. On the left side we added the “constant” χ(p_t) to the first order influence function (giving another first order influence function) to facilitate the computations. This is justified as the strategy [1]–[3] works with any influence function. In view of (4.3) and the definitions of the paths t ↦ a+tα, t ↦ b+tβ and t ↦ f(1+tϕ), this leads to the equation

$$a_1\left(y_1-b(z_1)\right)\alpha (z_1)-\left(a_{1}a(z_1)-1\right)\beta (z_1)={\mathrm{E}}_p{\chi }_p^{(2)}(x_1,X_2)B_p(\alpha ,\beta ,\varphi )(X_2).$$ (5.2)

Unfortunately, no function ${\chi }_p^{(2)}$ that solves this equation for every (α, β, ϕ) exists. To see this note that for the special triplets with β = ϕ = 0 the requirement can be written in the form

$$\alpha (z_1)={\mathrm{E}}_p\left[\frac{{\chi }_p^{(2)}(x_1,X_2)}{a_1\left(y_1-b(z_1)\right)}\frac{1-A_{2}a(Z_2)}{a(Z_2)(a-1)(Z_2)}\right]\alpha (Z_2).$$

The right side of the equation can be written as ∫ K(z₁, z₂)α(z₂) dF (z₂), for K(z₁, Z₂) the conditional expectation of the function in square brackets given Z₂. Thus it is the image of α under the kernel operator with kernel K. If the equation were true for any α, then this kernel operator would work as the identity operator. However, on infinite-dimensional domains the identity operator is not given by a kernel. (Its kernel would be a “Dirac function on the diagonal”.)

Therefore, we have to be satisfied with an influence function that gives a partial representation only. In particular, a projection onto a finite-dimensional linear space possesses a kernel, and acts as the identity on this linear space. A “large” linear space gives representation in “many” directions. By reducing the expectation in (5.2) to an integral relative to the marginal distribution of Z₂, we can use an orthogonal projection ∏_p:L₂(g) → L₂(g) onto a subspace L of L₂(g). Writing also ∏_p for its kernel, and letting S₂h denote the symmetrization (h(X₁, X₂) + h(X₂, X₁))/2 of a function $h:{\mathcal{X}}^2\to ℝ$, we define

$${\chi }_p^{(2)}(X_1,X_2)=-2S_2\left[A_1\left(Y_1-b(Z_1)\right){\mathrm{\Pi }}_p(Z_1,Z_2)(A_{2}a(Z_2)-1)\right].$$ (5.3)

Lemma 5.1

For ${\chi }_p^{(2)}$ defined by (5.3) with Π_p the kernel of an orthogonal projection Π_p: L₂(g) → L₂(g) onto a subspace L ⊂ L₂(g), equation (5.2) is satisfied for every path t ↦ p_t corresponding to directions (α, β, ϕ) such that α ∈ L and β ∈ L.

Proof

By definition E(A|Z) = (1/a)(Z) and E(Y|Z) = b(Z). Also var (Aa(Z)| Z) = a(Z) − 1 and var(Y|Z) = b(Z)(1 − b)(Z). By direct computation using these identities, we find that for the influence function (5.3) the right side of (5.2) reduces to

$$a_1(y_1-b(z_1)){\mathrm{\Pi }}_p\alpha (z_1)-(a_{1}a(z_1)-1){\mathrm{\Pi }}_p\beta (z_1).$$

Thus (5.2) holds for every (α, β, ϕ) such that Π_pα = α and Π_pβ = β.

Together with the first order influence function (4.3) the influence function (5.3) defines the (approximate) influence function ${\chi }_p={\chi }_p^{(1)}+\frac{1}{2}{\chi }_p^{(2)}$. For an initial estimator $\widehat{p}$ based on independent observations we now construct the estimator (3.1), i.e.

$${\widehat{\chi }}_n=\chi (\widehat{p})+{\mathbb{P}}_n{\chi }_{\widehat{p}}^{(1)}+\frac{1}{2}{\mathbb{U}}_n{\chi }_{\widehat{p}}^{(2)}.$$ (5.4)

Unlike the first order influence function, the second order influence function does depend on the density f of the covariates, or rather the function g = f/a (through the kernel ∏_p, which is defined relative to L₂(g)), and hence the estimator (5.4) involves a preliminary estimator of g. As a consequence, the quality of the estimator of the functional χ depends on the precision by which g (as part of the plug-in $\widehat{p}=(\widehat{a},\widehat{b},\widehat{g})$) can be estimated.

Let ${\widehat{\mathrm{E}}}_p$ and ${\widehat{\text{var}}}_{p0}$ denote conditional expectations given the observations used to construct $\widehat{p}$, let ‖·‖_r be the norm of L_r(g), and let ‖∏‖_r denote the norm of an operator ∏:L_r(g) → L_r(g).

Theorem 5.1

The estimator ${\widehat{\chi }}_n$ given in (5.4) with influence functions ${\chi }_p^{(1)}$ and ${\chi }_p^{(2)}$ defined by (4.3) and (5.3), for Π_p the kernel of an orthogonal projection in L₂(g) onto a k-dimensional linear subspace, satisfies, for r ≥ 2 (with r/(r − 2) = ∞ if r = 2),

$${\widehat{\mathrm{E}}}_p{\widehat{\chi }}_n-\chi (p)=O_P\left({\Vert {\mathrm{\Pi }}_p\Vert }_r{\Vert {\mathrm{\Pi }}_{\widehat{p}}\Vert }_r{\Vert \widehat{a}-a\Vert }_r{\Vert \widehat{b}-b\Vert }_r{\Vert \widehat{g}-g\Vert }_{r/(r-2)}\right)+O_P\left({\Vert (I-{\mathrm{\Pi }}_p)(a-\widehat{a})\Vert }_2{\Vert (I-{\mathrm{\Pi }}_p)(b-\widehat{b})\Vert }_2\right),$$

$${\widehat{\text{var}}}_p{\widehat{\chi }}_n=O_P\left(\frac{1}{n}+\frac{k}{n^2}\right).$$

The two terms in the bias result from having to estimate p in the second order influence function (giving “third order bias”) and using an approximate influence function (leaving the remainders I − Π_p after projection), respectively. The terms 1/n and k/n² in the variance appear as the variances of ${\mathbb{U}}_n{\chi }_p^{(1)}$ and ${\mathbb{U}}_n{\chi }_p^{(2)}$, the second being a degenerate second order U-statistic (giving 1/n², see (13.1)) with a kernel of variance k.

The proof of the theorem is deferred to Section 10.1.

Assume now that the range space of the projections Π_p can be chosen such that, for some constant C,

$${\Vert a-{\mathrm{\Pi }}_{p}a\Vert }_2\le C{\left(\frac{1}{k}\right)}^{\alpha /d},{\Vert b-{\mathrm{\Pi }}_{p}b\Vert }_2\le C{\left(\frac{1}{k}\right)}^{\beta /d}.$$ (5.5)

Furthermore, assume that there exist estimators $\widehat{a}$ and $\widehat{b}$ and $\widehat{g}$ that achieve convergence rates n^{−α/(2α+d)}, n^{−β/(2β+d)} and n^{−γ/(2γ+d)}, respectively, in L_r(g) and L_r/(r−2)(g), uniformly over these a-priori models and a model for g (e.g. for r = 3), and that the preceding displays also hold for $\widehat{a}$ and $\widehat{b}$. These assumptions are satisfied if the unknown functions a and b are “regular” of orders α and β on a compact subset of ℝ^d (see e.g. [32]). Then the estimator ${\widehat{\chi }}_n$ of Theorem 5.1 attains the square rate of convergence

$${\left(\frac{1}{n}\right)}^{2\alpha /(2\alpha +d)+2\beta /(2\beta +d)+2\gamma /(2\gamma +d)}\vee {\left(\frac{1}{k}\right)}^{(2\alpha +2\beta )/d}\vee \frac{1}{n}\vee \frac{k}{n^2}.$$ (5.6)

We shall see in the next section that the first of the four terms in this maximum can be made smaller by choosing an estimating equation of order higher than 2, while the other three terms arise at any order. This motivates to determine a “second order ‘optimal” value of k by balancing the second, third and fourth terms. We next would use the second order estimator if γ is large enough so that the first term is negligible relative to the other terms.

For (α+β)/2 ≥ d/4 we can choose k = n and the resulting rate (the square root of (5.6)) is n^−1/2 provided that (5.1) holds. The latter condition is certainly satisfied under the sufficient condition (4.1) for the linear estimator to yield rate n^−1/2.

More interestingly, for (α + β)/2 < d/4 we choose k ~ n^{2d/(d+2α+2β)} and obtain the rate, provided that (5.1) holds,

$$n^{-(2\alpha +2\beta )/(d+2\alpha +2\beta )}.$$

This rate is slower than n^−1/2, but better than the rate n^{−α/(2α+d)−β/(2β+d)} obtained by the linear estimator. In [29] this rate is shown to be the fastest possible in the minimax sense, for the model in which a and b range over balls in $C^{\alpha }(\mathcal{Z})$ and $C^{\beta }(\mathcal{Z})$, and g being known.

In both cases the second order estimator is better than the linear estimator, but minimax only for sufficiently large γ. This motivates to consider higher order estimators.

6. Approximate functional

Even though the functional of interest does not possess an exact second-order influence function, we might proceed to higher orders by differentiating the approximate second-order influence function ${\chi }_p^{(2)}$ given in (5.3), and balancing the various terms obtained. However, the formulas are much more transparent if we compute exact higer-order influence functions of an approximating functional instead. In this section we first define a suitable functional and next compute its influence functions.

Following the heuristics of Section 3.3, we define an approximate functional by equation (3.5), using a particular projection $p\mapsto \stackrel{\sim }{p}$ of the parameters. We choose this projection to map the parameters a and b onto finite-dimensional models and leave the parameter g unaltered: p is mapped into an element $\stackrel{\sim }{p}$ of the approximating model, or equivalently a triplet (a, b, g) into a triplet $(\stackrel{\sim }{a},\stackrel{\sim }{b},g)$ in the approximating model for the three parameters (where g is unaltered). (Even though this is not evident in the notation, the projection is joint in the three parameters: the induced maps $(a,b,g)\mapsto \stackrel{\sim }{a}$ and $(a,b,g)\mapsto \stackrel{\sim }{b}$ do not reduce to maps $a\mapsto \stackrel{\sim }{a}$ and $b\mapsto \stackrel{\sim }{b}$, but $\stackrel{\sim }{a}$ and $\stackrel{\sim }{b}$ depend on the full triplet (a, b, g).)

As “model” for (a, b) we consider the product of two affine linear spaces

$$\left(\widehat{a}+\underset{\_}{a}L\right)\times \left(\widehat{b}+\underset{\_}{b}L\right),$$ (6.1)

for a given finite-dimensional subspace L of L₂(ν) and fixed functions $\widehat{a},\underset{\_}{a},\widehat{b},\underset{\_}{b}:\mathcal{Z}\to ℝ$ that are bounded away from zero and infinity. (Later the functions $\widehat{a}$ and $\widehat{b}$ are taken equal to the preliminary estimators; one choice for the other functions is $\underset{\_}{a}=\underset{\_}{b}=1$.) The pair $(\stackrel{\sim }{a},\stackrel{\sim }{b})$ of projections are defined as elements of the model (6.1) satisfying equation (3.7). In view of (4.4), for any path ${\stackrel{\sim }{p}}_t\leftrightarrow ({\stackrel{\sim }{a}}_t,{\stackrel{\sim }{b}}_t,g)=(\stackrel{\sim }{a}+t\underset{\_}{a}l,\stackrel{\sim }{b}+t\underset{\_}{b}{l}^{\prime },g)$, for given l, l′ ∈ L,

$$\chi ({\stackrel{\sim }{p}}_t)+P{\chi }_{{\stackrel{\sim }{p}}_t}^{(1)}=\chi (p)-{\displaystyle \int (\stackrel{\sim }{a}+t\underset{\_}{a}l-a)(\stackrel{\sim }{b}+t\underset{\_}{b}{l}^{\prime }-b)g\mathit{d\nu }}.$$ (6.2)

Equation (3.7) requires that the derivative of this expression with respect to t at t = 0 vanishes. Thus the functions $\stackrel{\sim }{a}$ and $\stackrel{\sim }{b}$ must be chosen to satisfy the set of stationary equations, for every l; l′ ∈ L,

$$0={\displaystyle \int (\stackrel{\sim }{a}-a)\underset{\_}{b}{l}^{\prime }g\mathit{d\nu }}={\displaystyle \int \left(\frac{\stackrel{\sim }{a}-\widehat{a}}{\underset{\_}{a}}-\frac{a-\widehat{a}}{\underset{\_}{a}}\right)l^{\prime }\underset{\_}{ab}g\mathit{d\nu }},l^{\prime }\in L,$$ (6.3)

$$0={\displaystyle \int \underset{\_}{a}l(\stackrel{\sim }{b}-b)g\mathit{d\nu }}={\displaystyle \int \left(\frac{\stackrel{\sim }{b}-\widehat{b}}{\underset{\_}{b}}-\frac{b-\widehat{b}}{\underset{\_}{b}}\right)l\underset{\_}{ab}g\mathit{d\nu }},l^{\prime }\in L.$$ (6.4)

Because the functions $(\stackrel{\sim }{a}-\widehat{a})/\underset{\_}{a}$ and $(\stackrel{\sim }{b}-\widehat{b})/\underset{\_}{b}$ are required to be in L, the second way of writing these equations shows that the latter two functions are the orthogonal projections of the functions $(a-\widehat{a})/\underset{\_}{a}$ and $(b-\widehat{b})/\underset{\_}{b}$ onto L in $L_2(\underset{\_}{ab}g)$.

As explained in Section 3.3, as it satisfies (3.7) the projection $(a,b,g)\mapsto (\stackrel{\sim }{a},\stackrel{\sim }{b},g)$ renders the first order influence function of the approximate functional $\stackrel{\sim }{\chi }$ equal to the first order influence function of χ evaluated at the projection. Furthermore, the difference between χ and $\stackrel{\sim }{\chi }$ is quadratic in the distance between $\stackrel{\sim }{p}$ and p (see (3.6)). The following theorem summarizes the preceding and verifies these properties in the present concrete situation.

Theorem 6.1

For given measurable functions $\widehat{a},\underset{\_}{a},\widehat{b},\underset{\_}{b}:\mathcal{Z}\to ℝ$ with $\underset{\_}{a}$ and $\underset{\_}{b}$ bounded away from zero and infinity, define a map $(a,b,g)\mapsto (\stackrel{\sim }{a},\stackrel{\sim }{b},g)$ by letting $(\stackrel{\sim }{a}-\widehat{a})/\underset{\_}{a}$ and $(\stackrel{\sim }{b}-\widehat{b})/\underset{\_}{b}$ be the orthogonal projections of $(a-\widehat{a})/\underset{\_}{a}$ and $(b-\widehat{b})/\underset{\_}{b}$ in $L_2(\underset{\_}{ab}g)$ onto a closed subspace L. Let $\stackrel{\sim }{p}$ correspond to $(\stackrel{\sim }{a},\stackrel{\sim }{b},g)$ and define $\stackrel{\sim }{\chi }(p)=\chi (\stackrel{\sim }{p})+P{\chi }_{\stackrel{\sim }{p}}^{(1)}$. Then $\stackrel{\sim }{\chi }$ has influence function

$${\stackrel{\sim }{\chi }}_p^{(1)}(X)=A\stackrel{\sim }{a}(Z)\left(Y-\stackrel{\sim }{b}(Z)\right)+\stackrel{\sim }{b}(Z)-\chi (\stackrel{\sim }{p}).$$ (6.5)

Furthermore, for $\underset{\_}{g}=\underset{\_}{ab}g$,

$$\left|\stackrel{\sim }{\chi }(p)-\chi (p)\right|\le {\Vert (I-{\mathrm{\Pi }}_p)\frac{\widehat{a}-a}{\underset{\_}{a}}\Vert }_{2,\underset{\_}{g}}{\Vert (I-{\mathrm{\Pi }}_p)\frac{\widehat{b}-b}{\underset{\_}{b}}\Vert }_{2,\underset{\_}{g}}.$$

Proof

The formula for the influence function agrees with the combination of equations (3.8) and (4.3), and can also be verified directly. In view of (3.5) and (4.4),

$$\stackrel{\sim }{\chi }(p)-\chi (p)=-{\displaystyle \int (\stackrel{\sim }{a}-a)(\stackrel{\sim }{b}-b)g\mathit{d\nu }}.$$

We rewrite the right side as an integral relative to $\underset{\_}{g}$ dν, and next apply the Cauchy-Schwarz inequality. Finally we note that $(\stackrel{\sim }{a}-a)/\underset{\_}{a}=(\stackrel{\sim }{a}-\widehat{a})/\underset{\_}{a}-(a-\widehat{a})/\underset{\_}{a}=(I-{\mathrm{\Pi }}_p)\left((\widehat{a}-a)/\underset{\_}{a}\right)$, and similarly for b.

The approximation error $\stackrel{\sim }{\chi }(p)-\chi (p)$ can be rendered arbitrarily small by choosing the space L large enough. Of course, we choose L to be appropriate relative to a-priori assumptions on the functions a and b. If these functions are known to belong to Hölder classes, then L can for instance be chosen as the linear span of the first k basis elements of a suitable orthonormal wavelet basis of L₂(ν).

To compute higher order influence functions of $\stackrel{\sim }{\chi }$ we recursively determine influence functions of influence functions, according to the algorithm [1]–[3] in Section 3.1, starting with the influence function of $p\mapsto {\stackrel{\sim }{\chi }}_p^{(1)}(x_1)+\chi (\stackrel{\sim }{p})$, for a fixed x₁. We defer the details of this derivation to Section 10.2, and summarize the result in the following theorem.

To simplify notation, define

$$\begin{array}{l}\stackrel{\sim }{Y}=A\left(Y-\stackrel{\sim }{b}(Z)\right)\underset{\_}{a}(Z),\\ \stackrel{\sim }{A}=\left(A\stackrel{\sim }{a}(Z)-1\right)\underset{\_}{b}(Z),\\ \underset{\_}{A}=A\underset{\_}{a}(Z)\underset{\_}{b}(Z).\end{array}$$ (6.6)

These are the generic variables; indexed versions ${\stackrel{\sim }{Y}}_i,{\stackrel{\sim }{A}}_i,{\underset{\_}{A}}_i,\dots$ are defined by adding an index to every variable in the equalities. With this notation and with $\underset{\_}{a}=\underset{\_}{b}=1$ the second order influence function (5.3) at $p=\stackrel{\sim }{p}$ can be written as the symmetrization of $-2{\stackrel{\sim }{Y}}_1{\mathrm{\Pi }}_p(Z_1,Z_2){\stackrel{\sim }{A}}_2$. This function was derived in an ad-hoc manner as an approximate or partial influence function of χ, but it is also the exact influence function of $\stackrel{\sim }{\chi }$. The higher order influence functions of $\stackrel{\sim }{\chi }$ possess an equally attractive form.

Theorem 6.2

An mth order influence function ${\stackrel{\sim }{\chi }}_p^{(m)}$ evaluated at (X₁,…, X_m) of the functional $\stackrel{\sim }{\chi }$ defined in Theorem 6.1 is the degenerate (in L₂(p)) part of the variable

$${(-1)}^{j-1}j!{\stackrel{\sim }{A}}_1{\mathrm{\Pi }}_{1,2}{\underset{\_}{A}}_2{\mathrm{\Pi }}_{2,3}{\underset{\_}{A}}_3{\mathrm{\Pi }}_{3,4}{\underset{\_}{A}}_4\times \cdots \times {\underset{\_}{A}}_{m-1}{\mathrm{\Pi }}_{m-1,m}{\stackrel{\sim }{Y}}_m.$$

Here ∏_i,j is the kernel of the orthogonal projection in $L_2(\underset{\_}{ab}g)$ onto L, evaluated at (Z_i, Z_j).

To obtain the degenerate part of the variable in the preceding lemma, we apply the general formula (2.1) together with Lemma 10.2. Assertions (i) and (ii) of the latter lemma show that the variable is already degenerate relative to X₁ and X_m, while assertion (iii) shows that integrating out the variable X_i for 1 < i < m simply collapses ${\mathrm{\Pi }}_{i-1,i}{\underset{\_}{A}}_i{\mathrm{\Pi }}_{i,i+1}$ into Π_i−1,i+1. For instance, with S_m denoting symmetrization of a function of m variables,

$${\stackrel{\sim }{\chi }}_p^{(2)}(X_1,X_2)=-2S_2[{\stackrel{\sim }{A}}_1{\mathrm{\Pi }}_{1,2}{\stackrel{\sim }{Y}}_2],{\stackrel{\sim }{\chi }}_p^{(3)}(X_1,X_2,X_3)=6S_3\left[{\stackrel{\sim }{A}}_1{\mathrm{\Pi }}_{1,2}{\underset{\_}{A}}_2{\mathrm{\Pi }}_{2,3}{\stackrel{\sim }{Y}}_3-{\stackrel{\sim }{A}}_1{\mathrm{\Pi }}_{1,3}{\stackrel{\sim }{Y}}_3\right],{\stackrel{\sim }{\chi }}_p^{(4)}(X_1,X_2,X_3,X_4)=-24S_4\left[{\stackrel{\sim }{A}}_1{\mathrm{\Pi }}_{1,2}{\underset{\_}{A}}_2{\mathrm{\Pi }}_{2,3}{\underset{\_}{A}}_3{\mathrm{\Pi }}_{3,4}{\stackrel{\sim }{Y}}_4-{\stackrel{\sim }{A}}_1{\mathrm{\Pi }}_{1,3}{\underset{\_}{A}}_3{\mathrm{\Pi }}_{3,4}{\stackrel{\sim }{Y}}_4-{\stackrel{\sim }{A}}_1{\mathrm{\Pi }}_{1,2}{\underset{\_}{A}}_2{\mathrm{\Pi }}_{2,4}{\stackrel{\sim }{Y}}_4+{\stackrel{\sim }{A}}_1{\mathrm{\Pi }}_{1,4}\stackrel{\sim }{Y}\right].$$ (6.7)

As shown on the left, but not on the right of the equations, these quantities depend on the unknown parameter p = (a, b, g). In the right sides, the variables ${\stackrel{\sim }{Y}}_i$ and ${\stackrel{\sim }{A}}_i$ depend on p through $\stackrel{\sim }{b}$ and $\stackrel{\sim }{a}$, and hence are not observables. Furthermore, the kernels Π_i,j depend on g as they are orthogonal projections in $L_2(\underset{\_}{ab}g)$.

7. Parametric rate ((α + β)/2 ≥ d/4)

In this section we show that the parameter χ(p) is estimable at $1/\sqrt{n}\text{-rate}$ provided the average smoothness (α + β)/2 is at least d/4. We achieve this using the estimator

$${\widehat{\chi }}_n=\chi (\widehat{p})+{\mathbb{U}}_n\left({\stackrel{\sim }{\chi }}_{\widehat{p}}^{(1)}+\frac{1}{2}{\stackrel{\sim }{\chi }}_{\widehat{p}}^{(2)}+\cdots +\frac{1}{m!}{\chi }_{\widehat{p}}^{(m)}\right),$$ (7.1)

with the influence functions ${\stackrel{\sim }{\chi }}_p^{(j)}$ those of the approximate functional $\stackrel{\sim }{\chi }$ in Section 6: they are given in Theorems 6.1 and 6.2 for j = 1, and j = 2, …, m, respectively. (Because the map $p\mapsto \stackrel{\sim }{p}$ maps $\widehat{p}$ into itself, the influence function for j = 1 in the display is also the first order influence function (6.5) of of χ, when evaluated at $p=\widehat{p}$.)

We assume that the projections Π_p and ${\mathrm{\Pi }}_{\widehat{p}}$ map $L_s(\underset{\_}{ab}g)$ to $L_s(\underset{\_}{ab}g)$, for every s ∈ =[r/(r − 1), r], with uniformly bounded norms. (For r = 2 this entails only s = 2; in this case we define r/(r − 2) = ∞.)

Theorem 7.1

The estimator (7.1), with Π_p a kernel of an orthogonal projection in $L_2(\underset{\_}{ab}g)$ satisfying (12.1) with sup_x Π_p(x, x) ≲ k, satisfies, for a constant c that depends on ${\Vert p/\widehat{p}\Vert }_{\infty }$ only, and r ≥ 2,

$${\widehat{\mathrm{E}}}_p{\widehat{\chi }}_n-\chi (p)=O\left({\Vert \widehat{a}-a\Vert }_r{\Vert \widehat{b}-b\Vert }_r{\Vert \widehat{g}-g\Vert }_{(m-1)r/(r-2)}^{m-1}\right)+O\left({\Vert (I-{\mathrm{\Pi }}_p)\frac{\widehat{a}-a}{\underset{\_}{a}}\Vert }_2{\Vert (I-{\mathrm{\Pi }}_p)\frac{\widehat{b}-b}{\underset{\_}{b}}\Vert }_2\right),$$

$${\widehat{\text{var}}}_p{\widehat{\chi }}_n\le {\displaystyle \sum _{j=1}^m\frac{1}{\left(\frac{n}{j}\right)}{c}^j{k}^{j-1}}.$$

The first term in the bias is of the order 1 + 1 + (m − 1) = m + 1, as to be expected for an estimator based on an mth order influence function; the second term is due to estimating $\stackrel{\sim }{\chi }$ rather than χ; it is independent of m, and the same as in Theorem 5.1 if $\underset{\_}{a}=\underset{\_}{b}=1$. The bound on the variance can roughly be understood in that each of the degenerate U-statistics ${\mathbb{U}}_n{\stackrel{\sim }{\chi }}_{\widehat{p}}^{(j)}$ in (7.1) contributes a term of order k^j−1/n^j.

For α-, β- and γ-regular parameters a, b, g on a d-dimensional domain the range space of the projections Π_p can be chosen so that (5.5) holds and such that there exist estimators $\widehat{a},\widehat{b},\widehat{g}$ of a, b, g, with the first two taking values in this range space, with convergence rates n^{−α/(2α+d)}, n^{−β/(2β+d)} and n^{−γ/(2γ+d)}. Then the second term in the bias (with $\underset{\_}{a}=\underset{\_}{b}=1$) is of order (1/k)^α/d+β/d. If (α + β)/2 ≥ d/4 and we choose k = n, then this is of order $1/\sqrt{n}$. For k = n the standard deviation of the resulting estimator is also of the order $1/\sqrt{n}$, while the first term in the bias can be made arbitrarily small by choosing a sufficiently large order m. Specifically, the estimator ${\widehat{\chi }}_n$ attains a $\sqrt{n}\text{-rate}$ of convergence as soon as

$$m-1\ge \left(\frac{1}{2}-\frac{\alpha }{2\alpha +d}-\frac{\beta }{2\beta +d}\right)\left(\frac{2\gamma +d}{\gamma }\right).$$ (7.2)

For any γ > 0 there exists an order m that satisfies this, and hence the parameter is $\sqrt{n}\text{-estimable}$ as soon as (α + β)/2 ≥ d/4.

More ambitiously, we may aim at attaining the parametric rate for every γ > 0, without a-priori knowledge of γ. This can be achieved if (α + β)/2 > d/4 by using orders m = m_n that increase to infinity with the sample size. In this case the estimator can also be shown to be asymptotically efficient in the semiparametric sense.

Theorem 7.2

If (α + β)/2 > d/4, then the estimator (7.1), with m = log n and Π_p a kernel of an orthogonal projection in $L_2(\underset{\_}{ab}g)$ on a k = n/(log n)²-dimensional space satisfying (5.5) and (12.1) with sup_x Π_p(x, x) ≲ k, based on preliminary estimators $\widehat{a},\widehat{b},\widehat{g}$ that attain rates (log n/n)^{−δ/(2δ+d)} relative to the uniform norm, satisfies

$$\sqrt{n}\left({\widehat{\chi }}_n-\chi (p)-{\mathbb{P}}_n{\stackrel{\sim }{\chi }}_p^{(1)}\right)\stackrel{P}{\to }0.$$

An estimator that is asymptotically linear in the first order efficient influence function, as in the theorem, is asymptotically optimal in terms of the local asymptotic minimax and convolution theorems (see e.g. [40], Chapter 25). The present estimator ${\widehat{\chi }}_n$ actually looses its efficiency by splitting the sample in a part used to construct the preliminary estimators and a part to form ${\mathbb{P}}_n$. This can be easily remedied by crossing over the two parts of the split, and taking the average of the two estimators so obtained. By the theorem these are both asymptotically linear in their sample, and hence their average is asymptotically linear in the full sample and asymptotically efficient.

The proofs of the theorems are deferred to Section 10.3.

8. Minimax rate at lower smoothness ((α + β)/2 < d/4)

If the average a-priori smoothness (α + β)/2 of the functions a and b falls below d/4, then the functional χ cannot be estimated any more at the parametric rate ([29]). The estimator (7.1) of Theorem 7.1 can still be used and, with its bias and variance as given in the theorem properly balanced, attains a certain rate of convergence, faster than the current state-of-the-art linear estimators. However, in this section we present an estimator that is always better, and attains the minimax rate of convergence n^{−(2α+2β)/(2α+2β+d)} provided that the parameter g is sufficiently regular.

This estimator takes the same general form

$${\widehat{\chi }}_n=\chi (\widehat{p})+{\mathbb{U}}_n\left({\stackrel{\sim }{\chi }}_{\widehat{p}}^{(1)}+\frac{1}{2}{\stackrel{\sim }{\chi }}_{\widehat{p}}^{(2)}+\cdots +\frac{1}{m!}{\stackrel{\sim }{\chi }}_{\widehat{p}}^{(m)}\right),$$ (8.1)

as the estimator (7.1), but the influence functions ${\chi }_p^{(j)}$ for j ≥ 3 will be different. The idea is to “cut out” certain terms from the influence functions in (7.1) in order to decrease the variance, but without increasing the bias. For clarity we first consider the third order estimator, and next extend to the general mth order. To attain the minimax rate the order m must be fixed to a large enough value so that the first term in the bias given in Theorem 7.1 is no larger than n^{−(2α+2β)/(2α+2β+d)}. (Apart from added complexity there is no loss in choosing m larger than needed.)

The third order kernel ${\stackrel{\sim }{\chi }}_p^{(3)}$ in (6.7) is the symmetrization of the variable

$$6{\stackrel{\sim }{A}}_1\left({\mathrm{\Pi }}_p(Z_1,Z_2){\underset{\_}{A}}_2{\mathrm{\Pi }}_p(Z_2,Z_3)-{\mathrm{\Pi }}_p(Z_1,Z_3)\right){\stackrel{\sim }{Y}}_3.$$

Here Π_p is the kernel of an orthogonal projection in $L_2(\underset{\_}{ab}g)$ onto a k-dimensional linear space, which we may view as the sum of k projections on one-dimensional spaces. The quantity k² in the order O(k²/n³) of the variance in Theorem 7.1 for m = 3 arises as the number of terms in the product ${\mathrm{\Pi }}_p(Z_1,Z_2){\underset{\_}{A}}_2{\mathrm{\Pi }}_p(Z_2,Z_3)$ of the two k-dimensional projection kernels. It turns out that this order can be reduced without increasing the bias by cutting out “products of projections on higher base elements”.

To make this precise, we partition the projection space in blocks, and decompose the two projections in the influence function over the blocks:

$${\mathrm{\Pi }}_p={\displaystyle \sum _{r=0}^R{\mathrm{\Pi }}_p^{\left(k_{r-1},k_r\right]}},{\mathrm{\Pi }}_p={\displaystyle \sum _{s=0}^S{\mathrm{\Pi }}_p^{\left(l_{s-1},l_s\right]}}.$$ (8.2)

Here ${\mathrm{\Pi }}_p^{\left(m,n\right]}$ is the projection on the subspace spanned by base elements with index in intervals (m, n], and 1 = k₋₁ < k₀ < k₁ < ⋯ < k_R = k and 1 = l₋₁ < l₀ < l₁ < ⋯ < l_S = k are suitable partitions of the set {1, …, k}. (“Full” partitions in singleton sets would make the construction conceptual simpler, but a small number of blocks will be needed in our proofs.) The product of the two kernels now becomes a double sum, from which we retain only terms with small values of (r, s). The improved third order influence function is, with as before S₃ denoting symmetrization,

$${\chi }_p^{(3)}(X_1,X_2,X_3)=6S_3\left[{\displaystyle \sum _{\begin{array}{l}(r,s):r+s\le D\\ {\vee }_r=0{\vee }_s=0\end{array}}{\displaystyle \sum {\stackrel{\sim }{A}}_1\left({\mathrm{\Pi }}_p^{\left(k_{r-1},k_r\right]}(Z_1,Z_2){\underset{\_}{A}}_2{\mathrm{\Pi }}_p^{\left(l_{s-1},l_s\right]}(Z_2,Z_3)-{\mathrm{\Pi }}_p^{\left(k_{r-1}\vee l_{s-1},k_r\wedge l_s\right]}(Z_1,Z_3)\right)}{\stackrel{\sim }{Y}}_3}\right].$$ (8.3)

The negative term in the display is the conditional expectation given Z₁, Z₃ of the leading term, and maintains the degeneracy of the kernel. w For the decomposition (8.2) to be valid, the subspaces corresponding to the blocks must be orthogonal in $L_2(\underset{\_}{ab}g)$. We may achieve this by starting with a standard basis e₁, ε₂, …, with good approximation properties for a target model, and next replacing this by an orthonormal basis in $L_2(\underset{\_}{ab}g)$ by the Gram-Schmidt procedure. For a bounded g the approximation properties will be preserved.

The grids are defined by

$$k_{-1}=1,k_r~n{2}^{r/\alpha },r=0,\dots ,R,$$ (8.4)

$$l_{-1}=1,l_s~n{2}^{s/\beta },s=0,\dots ,S,$$ (8.5)

where R and S are chosen such that k_R ∼ l_S ∼ k (note that k₀ = l₀ = n). In these definitions the notation ~ means “equal up to a fixed multiple” (needed to allow that k_r and l_s are (dyadic) integers). For ease of notation let l_s = l₋₁ for s ≤ −1, and l_s = l_S for s ≥ S.

The grids k₀ < k₁ < ⋯ < k_R and l₀ < l₁ < ⋯ < l_S partition the integers n, n + 1, …, k in R and S groups. As $k_r^{\alpha }{l}_s^{\beta }=2^{r+s}{n}^{\alpha +\beta }$, for every r, s ≥ 0, the cut-off r+s ≤ D in (8.3) is delimited by the “hyperbola” i^αj^β ~ 2^Dn^α+β in the space of indices (i, j) ∈ {1, …, k}² of base elements used in the two kernels, with only the pairs below the hyperbola retained (see Figure 1). The intuition behind this hyperbolic cut-off is the product form of the bias (4.4): a higher order correction on the estimator of a may combine with a lower order correction on b, and vice versa, to give an overall correction of the desired order. The overall bias is smaller if the cut-off D is chosen larger, but then more terms are included in the estimator and the variance will be bigger.

Fig 1.

Both axis carry the indices of the basis functions spanning the projection space L, and point in the plane refers to a product of two projections. Products of projections on pairs of basis functions in the shaded area are included in the third order influence function. The step function refers to the partitions of the indices as in (8.2).

Before deriving an optimal value of D, we introduce the mth order estimator for general m ≥ 3. Again we take the estimator of Theorem 7.1 as starting point, but modify the higher order influence functions ${\stackrel{\sim }{\chi }}_p^{(j)}$, for j = 4, …, m, similar and in addition to the modification of the third order influence function. For given j the former influence function is given in Theorem 6.2 (with m of the theorem taken equal to j), and is based on a product of j − 1 projection kernels. We modify this in two steps. For each of the j − 2 contiguous pairs of kernels ((1st, 2nd), (2nd, 3rd), …, ((j − 2)th, (j − 1)th)) we form a new kernel by truncating the pair at the hyperbola as described previously for the third order kernel, and truncating all other kernels at n. Next the modified jth order kernel is the sum of the resulting j − 2 kernels. More formally, the modified jth order kernel is equal to

$${\chi }_p^{(j)}(X_1,\dots ,X_j)={\displaystyle \sum _{i=1}^{j-2}{\chi }_p^{(j,i)}}(X_1,\dots ,X_j),$$ (8.6)

where ${\chi }_p^{(j,i)}(X_1,\dots ,X_j)$ is the symmetrized, degenerate (relative to L₂(p)) part of the variable, for i = 1, …, j − 2, written in the notation of Theorem 7.1,

$$j!{(-1)}^{j-1}{\stackrel{\sim }{Y}}_1{\mathrm{\Pi }}_{1,2}^{\left(0,n\right]}{\underset{\_}{A}}_2\times \cdots \times {\underset{\_}{A}}_{i-1}{\mathrm{\Pi }}_{i-1,i}^{\left(0,n\right]}{\underset{\_}{A}}_i\times \times \left[{\displaystyle \sum _{\begin{array}{l}(r,s):r+s\le D\\ {\vee }_r=0{\vee }_s=0\end{array}}{\displaystyle \sum {\mathrm{\Pi }}_{i,i+1}^{\left(k_{r-1},k_r\right]}{\underset{\_}{A}}_{i+1}{\mathrm{\Pi }}_{i+1,i+2}^{\left(l_{s-1},l_s\right]}}}\right]{\underset{\_}{A}}_{i+2}{\mathrm{\Pi }}_{i+2,i+3}^{\left(0,n\right]}\times \cdots \times {\underset{\_}{A}}_{j-1}{\mathrm{\Pi }}_{j-1,j}^{\left(0,n\right]}{\stackrel{\sim }{A}}_j.$$

For j = 3 there is only one pair of kernels, and the construction reduces to the modification (8.3) as discussed previously.

We assume that the projections ${\mathrm{\Pi }}_p^{\left(0,l\right]}$ and ${\mathrm{\Pi }}_{\widehat{p}}^{\left(0,l\right]}$ map $L_s(\underset{\_}{ab}g)$ to $L_s(\underset{\_}{ab}g)$, for every s ∈ [r/(r − 1), r], with uniformly bounded norms.

Theorem 8.1

The estimator (8.1) for m ≥ 3 with the influence functions ${\stackrel{\sim }{\chi }}_p^{(j)}$ and ${\chi }_p^{(j)}$ given in (6.5) and (6.7) for j = 1, 2, respectively, and in (8.6) for j ≥ 3, and with ${\mathrm{\Pi }}_p^{\left(0,l\right]}$ kernels of orthogonal projections in $L_2(\underset{\_}{ab}g)$ satisfying (12.1) with ${\sup }_x{\mathrm{\Pi }}_{\widehat{p}}^{\left(0,l\right]}(x,x)\lesssim l$, satisfies, for r ≥ 2 (and r/(r − 2) = ∞ if r = 2),

$${\widehat{\mathrm{E}}}_p{\widehat{\chi }}_n-\chi (p)=O\left({\Vert \widehat{a}-a\Vert }_r{\Vert \widehat{b}-b\Vert }_r{\Vert \widehat{g}-g\Vert }_{\frac{mr}{r-2}}^{m-1}\right)+O\left({\Vert (I-{\mathrm{\Pi }}_p^{\left(0,k\right]})\frac{\widehat{a}-a}{\underset{\_}{a}}\Vert }_2{\Vert (I-{\mathrm{\Pi }}_p^{\left(0,k\right]})\frac{\widehat{b}-b}{\underset{\_}{b}}\Vert }_2\right),+O\left({\displaystyle \sum _{r=1}^R{\Vert (I-{\mathrm{\Pi }}_{\widehat{p}}^{\left(0,k_{r-1}\right]})\left(\frac{\widehat{a}-a}{\underset{\_}{a}}\right)\Vert }_r{\Vert (I-{\mathrm{\Pi }}_{\widehat{p}}^{\left(0,l_{D-1}\right]})\left(\frac{\widehat{b}-b}{\underset{\_}{b}}\right)\Vert }_r{\Vert \widehat{g}-g\Vert }_{\frac{r}{r-2}}}\right)+O\left(R{\Vert (I-{\mathrm{\Pi }}_{\widehat{p}}^{\left(0,n\right]})\frac{\widehat{a}-a}{\underset{\_}{a}}\Vert }_r{\Vert (I-{\mathrm{\Pi }}_{\widehat{p}}^{\left(0,n\right]})\frac{\widehat{b}-b}{\underset{\_}{b}}\Vert }_r{\Vert \widehat{g}-g\Vert }_{\frac{mr}{r-2}}^2\right),$$

$${\widehat{\text{var}}}_p{\widehat{\chi }}_n\lesssim \frac{1}{n}+\frac{k}{n^2}+\frac{D{2}^{(\frac{1}{\alpha }\vee \frac{1}{\beta })D}}{n}.$$

A proof of the theorem is presented in Sections 10.4 and 10.5.

The first two terms in the bias are the same as in Theorem 7.1; the third and fourth terms are the price paid for cutting out terms from the influence function. The benefit is a reduced variance. We shall show that the boundary parameter D can be chosen such that the third term in the variance (resulting from the third and higher order parts of the influence function) is not bigger than the second term, while the increase in bias is negligible.

Assume that the functions a and b and their estimates are known to belong to models that are well approximated by the base functions e₁, e₂, … in the sense that, for $p\in \{p,\widehat{p}\}$, and every value l in one of the two grids (8.4)–(8.5),

$${\Vert (I-{\mathrm{\Pi }}_p^{\left(0,l\right]})\left(\frac{\widehat{a}-a}{\underset{\_}{a}}\right)\Vert }_r\lesssim {\left(\frac{1}{l}\right)}^{\alpha /d},$$ (8.7)

$${\Vert (I-{\mathrm{\Pi }}_p^{\left(0,l\right]})\left(\frac{\widehat{b}-b}{\underset{\_}{b}}\right)\Vert }_r\lesssim {\left(\frac{1}{l}\right)}^{\beta /d}.$$ (8.8)

Then the second term in the bias is of the order (1/k)^α/d+β/d, as in Theorem 7.1, which is smaller than the minimax rate n^{−(2α+2β)/(2α+2β+d)} for

$$k~n^{2d/(2\alpha +2\beta +d)}.$$ (8.9)

With this choice of k, the upper bound on the variance is of the square minimax rate n^{−(4α+4β)/(2α+2β+d)} if D is chosen to satisfy

$$2^{(\frac{1}{\alpha }\vee \frac{1}{\beta })D}~\frac{1}{\log n}{n}^{(d-2\alpha -2\beta )/(d+2\alpha +2\beta )}.$$ (8.10)

Furthermore, under (8.9) the numbers R, S of grid points are of the order log n.

In the third term of the bias we apply assumptions (8.7)–(8.8) and the identity $k_{r-1}^{\alpha }{l}_{D-r}^{\beta }~n^{\alpha +\beta }{2}^D$, which results from (8.4)–(8.5), to see that the third term of the bias is of order

$${\displaystyle \sum _{r=1}^R{\left(\frac{1}{k_{r-1}}\right)}^{\alpha /d}{\left(\frac{1}{l_{D-r}}\right)}^{\beta /d}{\Vert \widehat{g}-g\Vert }_{r/(r-2)}\le R{\left(\frac{1}{n^{\alpha +\beta }{2}^D}\right)}^{1/d}{\Vert \widehat{g}-g\Vert }_{r/(r-2)}}.$$

If the convergence rate of $\widehat{g}$ is n^{−γ/(2γ+d)}, then, for the choice of D given in (8.10), this can (by a calculation) seen to be of smaller order than the minimax rate n^{−(2α+2β)/(2α+2β+d)} if γ is large enough that

$$\frac{\gamma }{2\gamma +d}>\left(\frac{\alpha \vee \beta }{d}\right)\left(\frac{d-2\alpha -2\beta }{d+2\alpha +2\beta }\right).$$ (8.11)

The fourth term in the bias can by a similar analysis be seen to be of the order

$$R{\left(\frac{1}{n}\right)}^{\alpha /d}{\left(\frac{1}{n}\right)}^{\beta /d}{\Vert \widehat{g}-g\Vert }_{(m-2)r/(r-2)}^2.$$

Again this is smaller than the minimax rate if γ satisfies assumption (8.11).

Finally, if the convergence rates of $\widehat{a}$ and $\widehat{b}$ are n^{−α/(2α+d)} and n^{−β/(2β+d)}, then the first term in the upper bound of the bias is of the order

$${\left(\frac{1}{n}\right)}^{\alpha /(2\alpha +d)+\beta /(2\beta +d)+(m-1)\gamma /(2\gamma +d)}.$$

We choose m large enough so that this is of smaller order than the preceding terms. In particular, we can choose it so that this is smaller than the minimax rate.

We summarize this in the following corollary, which is the most advanced result of the paper.

Corollary 8.1

If (8.7)–(8.11) hold, and ${\mathrm{\Pi }}_p^{\left(0,l\right]}$ are kernels of orthogonal projections in $L_2(\underset{\_}{ab}g)$ satisfying (12.1) with ${\sup }_x{\mathrm{\Pi }}_{\widehat{p}}^{\left(0,l\right]}(x,x)\le l$, then the mth order estimator with the kernels (8.6) for j ≥ 3 and sufficiently large m and suitable initial estimators, attains the rate n^{−(2α+2β)/(2α+2β+d)} for estimating χ(p).

9. Other examples

In this section we briefly indicate a number of other examples for which our general heuristics have been worked out, leading to well known or novel estimators.

9.1. Density estimation

Consider estimating a density χ(p) = p(a) at the fixed point a based on a random sample from p. A first order influence function of this functional would satisfy, for every smooth path t ↦ p_t with score function g at t = 0,

$${\displaystyle \int {\chi }_p^{(1)}\mathit{gpd}\mu ={\frac{d}{dt}}_{|t=0}\chi (p_t)=g(a)p(a)}.$$

In a nonparametric situation every zero-mean function g arises as a score function, and hence ${\chi }_p^{(1)}$ would have to be a “Dirac function at a”. Because this does not exist (except for very special p), in this example already a first order influence function fails to exist.

We may approximate the Dirac function by the function x ↦ Π(a, x) for Π the kernel of an orthogonal projection onto a given (large) subspace L of L₂(μ). Because ∫ Π(a, x)g(x)p(x) dμ(x) = g(a)p(a) for every function g such that gp ∈ L, the function x ↦ Π(a, x) achieves representation for a large set of scores. The corresponding degenerate version is x ↦ Π(a, x) − Πp(a), for Πp = ∫ Π(·, x)p(x) dμ(x) the projection of p. The corresponding first order estimator (3.1) is

$${\widehat{\chi }}_n=\chi ({\widehat{p}}_n)+{\mathbb{P}}_n(\mathrm{\Pi }(a,\cdot )-\mathrm{\Pi }{\widehat{p}}_n(a))={\mathbb{P}}_n\mathrm{\Pi }(a,\cdot )+\left((I-\mathrm{\Pi }){\widehat{p}}_n\right)(a).$$

If ${\widehat{p}}_n\in L$, then the second term vanishes and the estimator reduces to ${\mathbb{P}}_n{\displaystyle \mathrm{\Pi }\left(a,\cdot \right)}$. This is the usual projection estimator (cf. [25, 32]): if L is spanned by the orthonormal set e₁, e₂, …, e_k, then $\mathrm{\Pi }(x_1,x_2)={\displaystyle {\sum }_{i=1}^k{e}_i(x_1)e_i(x_2)}$ and ${\widehat{\chi }}_n={\displaystyle {\sum }_{i=1}^k({\mathbb{P}}_n{e}_i)e_i(a)}$.

Alternative to viewing x ↦ Π(a, x) as an approximation to the “ideal” influence function, we can derive it as the exact influence function of the approximate functional $\stackrel{\sim }{\chi }(p)=\chi (\mathrm{\Pi }p)$.

9.2. Quadratic functionals

Consider estimating the functional χ(p) = ∫ p² dμ based on a random sample of size n from the density p.

The first order influence function of this functional exists on the full non-parametric model, and can be seen to take the form

$${\chi }_p^{(1)}(x)=2\left(p(x)-\chi (p)\right).$$

By the algorithm [1]–[3] of Section 3.1 a second order influence function can be computed as the degenerate part of an influence function of the functional $p\mapsto {\overline{\chi }}_p^{(1)}(x_1)=2p(x_1)$, for fixed x₁. As seen in Section 9.1, point evaluation is not a differentiable functional, but has the kernel Π of an orthogonal projection in L₂(μ) as an approximate influence function. Thus an approximate second order influence function of the present functional, minus its projection onto the degenerate functions, is given by

$${\stackrel{\sim }{\chi }}_p^{(2)}(x_1,x_2)=2\mathrm{\Pi }(x_1,x_2)-2\mathrm{\Pi }p(x_1)-2\mathrm{\Pi }p(x_2)+2{\displaystyle \int {(\mathrm{\Pi }p)}^{2}d\mu }.$$

This may also be derived as an exact influence function of the approximate functional $\stackrel{\sim }{\chi }(p)=\chi (\mathrm{\Pi }p)$.

It can be checked that the estimator (7.1) for m = 2, given an initial estimator ${\widehat{p}}_n$ that is contained in the range of Π, reduces to ${\widehat{\chi }}_n={\mathbb{U}}_n\mathrm{\Pi }$, which is a well known estimator ([17]).

9.3. Doubly robust models

The heuristics described in Section 3 ought to be applicable in a wide range of estimation problems, but the detailed treatment of the missing data problem in Sections 4–8 shows that their implementation can be involved. Inspection of the proofs reveals that the particular implementation in the latter sections is based on the structure (4.3) of the first order influence function in the missing data problem. The argument extends to semiparametric models with first order influence function of the form

$${\chi }_p^{(1)}(x)=a(z)b(z)S_1(x)+a(z)S_2(x)+b(z)S_3(x)+S_4(x)-\chi (p),$$ (9.1)

for known functions S_i(x) of the data (i.e. S = (S₁, S₂, S₃, S₄) is a given statistic). The full parameter may be a quadruplet p ↔ (a, b, c, f), in which f is the marginal density of an observable covariate Z, and c does not appear in (9.1). Other examples of this structure are described in [27, 37].

10. Proofs

10.1. Proof of Theorem 5.1

Write $\widehat{\mathrm{\Pi }}$ and Π for ${\mathrm{\Pi }}_{\widehat{p}}$ and Π_p, respectively, for both the kernels and the corresponding projection operators, and drop p also in ${\widehat{\mathrm{E}}}_p$ and ${\widehat{\text{var}}}_p$. From (4.4) and (5.3) we have

$$\widehat{\mathrm{E}}{\widehat{\chi }}_n-\chi (p)=-{\displaystyle \int (\widehat{a}-a)(\widehat{b}-b)g\mathit{d\nu }-\widehat{\mathrm{E}}{A}_1(Y_1-\widehat{b}(Z_1))(A_2\widehat{a}(Z_2)-1)\widehat{\mathrm{\Pi }}(Z_1,Z_2)}=-{\displaystyle \int (\widehat{a}-a)(\widehat{b}-b)g\mathit{d\nu }+{\displaystyle \int {\displaystyle \int [(\widehat{a}-a)\times (\widehat{b}-b)](g\times g)\widehat{\mathrm{\Pi }}\mathit{d\nu }\times \nu }}.}$$

The double integral on the far right with $\widehat{\mathrm{\Pi }}$ replaced by Π can be written as the single integral ${\displaystyle \int (\widehat{a}-a)\mathrm{\Pi }(\widehat{b}-b)g\mathit{d\nu }}$, for $\mathrm{\Pi }(\widehat{b}-b)$ the image of $\widehat{b}-b$ under the projection Π. Added to the first integral on the right this gives $-{\displaystyle \int (\widehat{a}-a)(I-\mathrm{\Pi })(\widehat{b}-b)g\mathit{d\nu }}$, which is bounded in absolute value by the second term in the upper bound for the bias.

Replacement of $\widehat{\mathrm{\Pi }}$ by Π in the double integral gives a difference

$${\displaystyle \int {\displaystyle \int [(\widehat{a}-a)\times (\widehat{b}-b)]g\times g(\widehat{\mathrm{\Pi }}-\mathrm{\Pi })\mathit{d\nu }\times \nu }={\displaystyle \int (\widehat{a}-a)\left(\widehat{\mathrm{\Pi }}\left((\widehat{b}-b)\frac{g}{\widehat{g}}\right)-\mathrm{\Pi }(\widehat{b}-b)\right)g\mathit{d\nu }}\le {\Vert \widehat{a}-a\Vert }_s{\Vert \widehat{\mathrm{\Pi }}\left((\widehat{b}-b)\frac{g}{\widehat{g}}\right)-\mathrm{\Pi }(\widehat{b}-b)\Vert }_{r,\widehat{g}}{\Vert g/\widehat{g}\Vert }_{\infty }^{1/r}},$$

by Hölder’s inequality, for a conjugate pair (r, s). Considering $\widehat{\mathrm{\Pi }}$ as the projection in $L_2(\widehat{g})$ with weight 1, and Π as the weighted projection in $L_2(\widehat{g})$ with weight function $\widehat{w}=g/\widehat{g}$, we can apply Lemma 12.7(i) (with q = s/r and rp = s/(s − 2)) to see that this is bounded in absolute value by

$${\Vert \widehat{a}-a\Vert }_s{\Vert \widehat{\mathrm{\Pi }}\Vert }_{s,\widehat{g}}{\Vert \mathrm{\Pi }\Vert }_{s,\widehat{g}}{\Vert \widehat{b}-b\Vert }_{s,\widehat{g}}{\Vert \widehat{w}-1\Vert }_{s/(s-2),\widehat{g}}{\Vert w\Vert }_{\infty }^{1/r}.$$

Because $\widehat{w}$ is assumed bounded away from 0 and infinity, this is of the same order as the first term in the upper bound on the bias (if r replaces s).

Because the function ${\chi }_{\widehat{p}}^{(1)}$ is uniformly bounded, the (conditional) variance of ${\mathbb{U}}_n{\chi }_{\widehat{p}}^{(1)}$ is of the order O(1/n). Thus for the variance bound it suffices to consider the (conditional) variance of ${\mathbb{U}}_n{\chi }_{\widehat{p}}^{(2)}$. In view of Lemma 13.1 and (13.1) this is bounded above by a multiple of

$${\left(1+{\Vert \frac{p}{\widehat{p}}\Vert }_{\infty }\right)}^2{\widehat{P}}^n{({\mathbb{U}}_n{\chi }_{\widehat{p}}^{(2)})}^2={\left(1+{\Vert \frac{p}{\widehat{p}}\Vert }_{\infty }\right)}^2{\left(\frac{n}{2}\right)}^{-2}{\widehat{P}}^2{({\chi }_{\widehat{p}}^{(2)})}^2.$$

The variables $A(Y-\widehat{b}(Z))$ and $(A\widehat{a}(Z)-1)$ are uniformly bounded. Hence the last term on the right is bounded above by a multiple of $n^{-2}{\displaystyle \int {\widehat{\mathrm{\Pi }}}^2(\widehat{g}\times \widehat{g})\mathit{d\nu }\times \nu }$, which is equal to k/n², by Lemma 12.3.

10.2. Proof of Theorem 6.2

We compute the higher order influence functionals of the approximate functional $\stackrel{\sim }{\chi }$ using algorithm [1]–[3] in Section 3. This starts by computing the second order influence function as the derivative of $p\mapsto {\stackrel{\sim }{\chi }}_p^{(1)}(x_1)+\chi (\stackrel{\sim }{p})$, for fixed x₁. Because the latter functional (given in (6.5)) depends on the parameters only through $\stackrel{\sim }{a}(z_1)$ and $\stackrel{\sim }{b}(z_1)$, the following lemma does the main part of the work.

Lemma 10.1

For fixed z₁ influence functions of $p\mapsto \stackrel{\sim }{a}(z_1)$ and $p\mapsto \stackrel{\sim }{b}(z_1)$ are given by

$$x_2\mapsto -\underset{\_}{a}(z_1){\mathrm{\Pi }}_p(z_1,z_2)\left(a_2\stackrel{\sim }{a}(z_2)-1\right)\underset{\_}{b}(z_2),x_2\mapsto \underset{\_}{b}(z_1){\mathrm{\Pi }}_p(z_1,z_2)a_2\left(y_2-\stackrel{\sim }{b}(z_2)\right)\underset{\_}{a}(z_2),$$

where Π_p is the kernel of the orthogonal projection in $L_2(\underset{\_}{ab}g)$ onto L.

Proof

We can write the equation (6.3) determining $\stackrel{\sim }{a}$ as $\mathrm{E}\left(A\stackrel{\sim }{a}(Z)-1\right)\underset{\_}{b}(Z)l(Z)=0$, for every l ∈ L. Insert a sufficiently regular path p_t, given by parameters (a_t, b_t, f_t), and differentiate the equality relative to t at t = 0 to find, with γ a score function of the path

$$\mathrm{E}{\frac{d}{dt}}_{|t=0}{\stackrel{\sim }{a}}_t(Z)A\underset{\_}{b}(Z)l(Z)=-\mathrm{E}\left(A\stackrel{\sim }{a}(Z)-1\right)\underset{\_}{b}(Z)l(Z)\gamma (X).$$

Using the fact that E(A|Z) = 1/a(Z), where a is bounded away from zero, we can also write this as

$$\mathrm{E}\frac{{\frac{d}{dt}}_{|t=0}{\stackrel{\sim }{a}}_t(Z)}{\underset{\_}{a}(Z)}\frac{(\underset{\_}{ab})}{a}(Z)l(Z)=-\mathrm{E}\frac{\left(A\stackrel{\sim }{a}(Z)-1\right)a(Z)\gamma (X)}{\underset{\_}{a}(Z)}\frac{(\underset{\_}{ab})(Z)}{a(Z)}l(Z).$$

Because the function $({\stackrel{\sim }{a}}_t-\widehat{a})/\underset{\_}{a}$ is contained in L for every t by construction, the function ${(d/dt)}_{|t=0}{\stackrel{\sim }{a}}_t/\underset{\_}{a}$ is also contained in L. Combined with the validity of the preceding display for every l ∈ L, we conclude that ${(d/dt)}_{|t=0}{\stackrel{\sim }{a}}_t(Z)/\underset{\_}{a}(Z)$ is the weighted projection of $-\left(A\stackrel{\sim }{a}(Z)-1\right)a(Z)\gamma (X)/\underset{\_}{a}(Z)$ in L₂(P) onto the space {l(Z): l ∈ L} relative to the weight $(\underset{\_}{ab}/a)(Z)$. The projection can be represented in terms of a kernel operator (cf. Lemma 12.1). If ${\mathrm{\Pi }}_p(z_1,z_2)(\underset{\_}{ab})(z_2)/a(z_2)$ denotes the kernel, then

$$\begin{gathered} \frac{{\frac{d}{dt}}_{|t=0}{\stackrel{\sim }{a}}_t(z_1)}{\underset{\_}{a}(z_1)}=\mathrm{E}{\mathrm{\Pi }}_p(z_1,Z_2)\frac{\left(A_2\stackrel{\sim }{a}(Z_2)-1\right)a(Z_2)\gamma (X_2)}{\underset{\_}{a}(Z_2)}\left(\frac{\underset{\_}{ab}}{a}\right)(Z_2) \\ =-\mathrm{E}{\mathrm{\Pi }}_p(z_1,Z_2)\left(A_2\stackrel{\sim }{a}(Z_2)-1\right)\underset{\_}{b}(Z_2)\gamma (X_2). \end{gathered}$$

This represents the derivative on the left as an inner product of the score function γ with the function on the right of the first equation of the lemma (evaluated at X₂). Thus the first assertion of the lemma is proved.

The second assertion is proved similarly. Using that E(Y|Z) = b(Z) and E(A|Z) = 1/a(Z), we start by writing the equation (6.4) defining $\stackrel{\sim }{b}$ as $\mathrm{E}(\stackrel{\sim }{b}(Z)-Y)/\underset{\_}{b}(Z)(\underset{\_}{ab}/a)(Z)l(Z)=0$, for every l ∈ L. By the same arguments as before we conclude that ${(d/dt)}_{|t=0}{\stackrel{\sim }{b}}_t(Z)$ is the weighted projection of $(Y-\stackrel{\sim }{b}(Z))\gamma (X)/\underset{\_}{b}(Z)$ in L₂(P) onto the space {l(Z): l ∈ L}, relative to the weight (ab/a)(Z). ■

The first order influence function (6.5) depends on p only through $\stackrel{\sim }{a}$ and $\stackrel{\sim }{b}$ and hence the chain rule and the preceding lemma imply that a second order influence function of $\stackrel{\sim }{\chi }$ is given by the degenerate part of

$${\stackrel{\sim }{\chi }}_p^{(2)}(X_1,X_2)=-{\displaystyle {\mathrm{\Pi }}_p(Z_1,Z_2)\left[A_1(Y_1-\stackrel{\sim }{b}(Z_1))\underset{\_}{a}(Z_1)(A_2\stackrel{\sim }{a}(Z_2)-1)\underset{\_}{b}(Z_2)+(A_1\stackrel{\sim }{a}(Z_1)-1)\underset{\_}{b}(Z_1)A_2(Y_2-\stackrel{\sim }{b}(Z_2))\underset{\_}{a}(Z_2)\right]}.$$ (10.1)

(Note that this function is symmetric in (X₁, X₂); Π_p is symmetric, because it is an orthogonal projection kernel.) Actually, this function is already degenerate and hence is the second order influence function of $\stackrel{\sim }{\chi }$.

Lemma 10.2

For any fixed z₁ and z₃,

1. ${\mathrm{E}}_p{\mathrm{\Pi }}_p(z_1,Z_2)(A_2\stackrel{\sim }{a}(Z_2)-1)\underset{\_}{b}(Z_2)=0$.

2. ${\mathrm{E}}_p{\mathrm{\Pi }}_p(z_1,Z_2)(A_2(Y_2-\stackrel{\sim }{b}(Z_2))\underset{\_}{a}(Z_2)=0$.

3. ${\mathrm{E}}_p{\mathrm{\Pi }}_p(z_1,Z_2)A_2(\underset{\_}{ab})(Z_2){\mathrm{\Pi }}_p(Z_2,z_3)={\mathrm{\Pi }}_p(z_1,z_3)$.

Proof

Because $(\stackrel{\sim }{a}-\widehat{a})(Z)/\underset{\_}{a}(Z)$ and $(\stackrel{\sim }{b}-\widehat{b})(Z)/\underset{\_}{a}(Z)$ are the weighted projections in L₂(P) of $(a-\widehat{a})(Z)/\underset{\_}{a}(Z)$ and $(Y-\widehat{b}(Z))/\underset{\_}{b}(Z)$, respectively, onto {l(Z):l ∈ L} relative to the weights (ab/a)(Z),

$${\mathrm{E}}_{X_2}{\mathrm{\Pi }}_p(Z_1,Z_2)\left[\frac{\stackrel{\sim }{a}(Z_2)-\widehat{a}(Z_2)}{\underset{\_}{a}(Z_2)}-\frac{a(Z_2)-\widehat{a}(Z_2)}{\underset{\_}{a}(Z_2)}\right]\frac{\underset{\_}{ab}}{a}(Z_2)=0,$$ (10.2)

$${\mathrm{E}}_{X_2}{\mathrm{\Pi }}_p(Z_1,Z_2)\left[\frac{\stackrel{\sim }{b}(Z_2)-\widehat{b}(Z_2)}{\underset{\_}{b}(Z_2)}-\frac{Y_2-\widehat{b}(Z_2)}{\underset{\_}{b}(Z_2)}\right](\underset{\_}{ab})(Z_2)A=0.$$ (10.3)

These two assertions imply (i) and (ii). The third assertion follows from the fact that Π_p is the kernel of the weighted projection in L₂(P) onto L relative to the weight (ab/a)(Z). ■

The second order influence function (10.1) depends on p through $\stackrel{\sim }{a}$ and $\stackrel{\sim }{b}$ and through the kernel Π_p. We proceed to higher orders by differentiating the influence function relative to these components, and applying the chain rule, where we use the influence functions of $p\mapsto \stackrel{\sim }{a}(x)$ and $p\mapsto \stackrel{\sim }{b}(x)$ as given previously in Lemma 10.1, and the influence function of $p\mapsto {\mathrm{\Pi }}_p(z_1,z_2)$ as given in Lemma 12.8.

Proof of Theorem 6.2

Denote the symmetrization of the variable in the theorem by ${\stackrel{\sim }{\chi }}_p^{(m)}(X_1,\dots ,X_m)$. Then ${\stackrel{\sim }{\chi }}_p^{(2)}$ is the function ${\stackrel{\sim }{\chi }}_p^{(2)}$ given by (10.1), which was seen to be a second order influence function in the preceding discussion. We show by induction on m that $x_{m+1}\mapsto {\stackrel{\sim }{\chi }}_p^{(m+1)}(x_1,\dots ,x_m,x_{m+1})$ is an influence function of $p\mapsto {\stackrel{\sim }{\chi }}_p^{(m)}(x_1,\dots ,x_m)$. The theorem is then a corollary of Lemma 11.2.

By Lemmas 10.1 and 12.8,

1. The influence function of $p\mapsto {\stackrel{\sim }{Y}}_1$ is $x_{m+1}\mapsto -{\mathrm{\Pi }}_p(Z_1,z_{m+1}){\underset{\_}{A}}_1{\stackrel{\sim }{y}}_{m+1}$

2. The influence function of $p\mapsto {\stackrel{\sim }{A}}_1$ is $x_{m+1}\mapsto -{\mathrm{\Pi }}_p(Z_1,z_{m+1}){\underset{\_}{A}}_1{\stackrel{\sim }{a}}_{m+1}$.

3. The influence function of $p\mapsto {\underset{\_}{A}}_1$ is zero.

4. The influence function of $p\mapsto {\mathrm{\Pi }}_p(Z_1,Z_2)$ is $x_{m+1}\mapsto -{\mathrm{\Pi }}_p(Z_1,z_{m+1}){\underset{\_}{A}}_{m+1}{\mathrm{\Pi }}_p(z_{m+1},Z_2)$.

Applying this repeatedly readily gives an expression for the influence function of $p\mapsto {\stackrel{\sim }{A}}_1{\mathrm{\Pi }}_{1,2}{\underset{\_}{A}}_2{\mathrm{\Pi }}_{2,3}{\underset{\_}{A}}_3{\mathrm{\Pi }}_{3,4}{\underset{\_}{A}}_4\times \cdots \times {\underset{\_}{A}}_{m-1}{\mathrm{\Pi }}_{m-1,m}{\stackrel{\sim }{Y}}_m$

. The symmetrization of this expression is the same expression, but then with m replaced by m + 1 and an added minus sign. ■

10.3. Proof of Theorems 7.1 and 7.2

Let  and Ŷ be $\stackrel{\sim }{A}$ and $\stackrel{\sim }{Y}$ as in (6.6) with a and b in their definitions replaced by â and $\widehat{b}$. Because â and $\widehat{b}$ are projected onto themselves under the map $(a,b,g)\mapsto (\stackrel{\sim }{a},\stackrel{\sim }{b},g)$ (see Theorem 6.1), we actually obtain the same variables by replacing $\stackrel{\sim }{a}$ and $\stackrel{\sim }{b}$ by â and $\widehat{b}$, respectively: $\widehat{A}=(A\widehat{a}(Z)-1)\underset{\_}{b}(Z)$ and $\widehat{Y}=A(Y-\widehat{b}(Z))\underset{\_}{a}(Z)$. Furthermore, let Π and $\widehat{\mathrm{\Pi }}$ denote the operators Π_p and ${\mathrm{\Pi }}_{\widehat{p}}$, respectively, and Π_i,j and ${\widehat{\mathrm{\Pi }}}_{i,j}$ their kernels evaluated at (Z_i, Z_j).

By explicit calculations,

$$\chi (\widehat{p})+{\widehat{\mathrm{E}}}_p{\stackrel{\sim }{\chi }}_{\widehat{p}}^{(1)}-\chi (p)=-{\displaystyle \int (\widehat{a}-a)(\widehat{b}-b)gdv=\widehat{\mathrm{E}}}{\widehat{A}}_1{\mathrm{\Pi }}_{1,2}{\widehat{Y}}_2-\widehat{R},$$ (10.4)

for $\widehat{R}$ defined by

$$\widehat{R}={\displaystyle \int \left(\frac{\widehat{a}-a}{\underset{\_}{a}}\right)}(I-\mathrm{\Pi })\left(\frac{\widehat{b}-b}{\underset{\_}{b}}\right)\underset{\_}{ab}qdv.$$

The variable $\widehat{R}$ is bounded by the second term in the expression for ${\widehat{\mathrm{E}}}_p{\widehat{\chi }}_n-\chi (p)$ in the statement of the theorem. We next show by induction on m that

$$\widehat{R}+\chi (\widehat{p})+\widehat{\mathrm{E}}{\stackrel{\sim }{\chi }}_{\widehat{p}}^{(1)}+\cdots +\frac{1}{m!}\widehat{\mathrm{E}}{\stackrel{\sim }{\chi }}_{\widehat{p}}^{(m)}-\chi (p)={(-1)}^{m-1}\widehat{\mathrm{E}}{\widehat{A}}_1{(\widehat{\mathrm{\Pi }}-\mathrm{\Pi })}_{1,2}{\underset{\_}{A}}_2{(\widehat{\mathrm{\Pi }}-\mathrm{\Pi })}_{2,3}\times \cdots \times {\underset{\_}{A}}_{m-1}{(\widehat{\mathrm{\Pi }}-\mathrm{\Pi })}_{m-1,m}{\widehat{Y}}_m.$$ (10.5)

The analysis of the bias can then be concluded by showing that the right side of (10.5) is of the order as the first term given in the theorem.

Equation (10.4) and the definition of ${\stackrel{\sim }{\chi }}_p^{(2)}$ readily show that identity (10.5) is true for m = 2. We proceed to general m by induction. Relative to its value for m the left side receives for (m+ 1) the extra term $\widehat{\mathrm{E}}{\chi }_{\widehat{p}}^{(m+1)}/(m+1)!$ which is equal to (−1)^m times $\widehat{\mathrm{E}}{\widehat{A}}_1{\widehat{\mathrm{\Pi }}}_{1,2}{\underset{\_}{A}}_2{\widehat{\mathrm{\Pi }}}_{2.3}\times \cdots \times {\underset{\_}{A}}_m{\widehat{\mathrm{\Pi }}}_{m,m+1}{\widehat{Y}}_{m+1}$ minus a sum of terms resulting from projections of this leading term. This extra term without the factor (−1)^m (but including the projections) can be written (cf. (6.7) and (2.1))

$${\displaystyle \sum _{i=0}^{m-1}\left(\frac{m-1}{i}\right)}\widehat{\mathrm{E}}{\widehat{A}}_1{\widehat{\mathrm{\Pi }}}_{1,2}{\underset{\_}{A}}_2{\widehat{\mathrm{\Pi }}}_{2,3}\times \cdots \times {\underset{\_}{A}}_{m-i}{\widehat{\mathrm{\Pi }}}_{m-i,m-i+1}{\widehat{Y}}_{m-i+1}{(-1)}^i.$$ (10.6)

To prove the induction hypothesis for m + 1 it suffices to show that this is equal to

$$\widehat{\mathrm{E}}{\widehat{A}}_1{(\widehat{\mathrm{\Pi }}-\mathrm{\Pi })}_{1,2}{\underset{\_}{A}}_2{(\widehat{\mathrm{\Pi }}-\mathrm{\Pi })}_{2,3}\times \cdots \times {\underset{\_}{A}}_{m-1}{(\widehat{\mathrm{\Pi }}-\mathrm{\Pi })}_{m-1,m}{\widehat{Y}}_m+\widehat{\mathrm{E}}{\widehat{A}}_1{(\widehat{\mathrm{\Pi }}-\mathrm{\Pi })}_{1,2}{\underset{\_}{A}}_2{(\widehat{\mathrm{\Pi }}-\mathrm{\Pi })}_{2,3}\times \cdots \times {\underset{\_}{A}}_m{(\widehat{\mathrm{\Pi }}-\mathrm{\Pi })}_{m,m+1}{\widehat{Y}}_{m+1}.$$ (10.7)

To achieve this we expand the two terms of the preceding display into sums of expressions of the form, with each $K_{j,j+1}^{(j)}$ equal to ${\widehat{\mathrm{\Pi }}}_{j,j+1}$ or Π_j,j+1 and l the number of j for which the first alternative is true,

$$B_l:={(-1)}^{m-1-l}\widehat{\mathrm{E}}{\widehat{A}}_1{K}_{1,2}^{(1)}{\underset{\_}{A}}_2{K}_{2,3}^{(2)}\times \cdots \times {\underset{\_}{A}}_{m-1}{K}_{m-1,m}^{(m-1)}{\widehat{Y}}_m,$$ (10.8)

and of the same form with m+1 replacing m for the second term of (10.7). As the notation suggests the expression in (10.8) depends on l (and m, but this is fixed), but not on which K are equal to $\widehat{\mathrm{\Pi }}$ or Π. To see this we use that Π is a projection onto L in L₂(abg), so that ∫ Π_1,2γ(z₂)(abg)(z₂)dv(z₂) = γ(z₁) for every γ ∈ L; and $\widehat{\mathrm{\Pi }}$ is also a projection onto L, so that as a function of one argument ${\widehat{\mathrm{\Pi }}}_{1,2}$ is contained in L. This observation yields the identities, for K equal to $\widehat{\mathrm{\Pi }}$ or Π,

$${\widehat{\mathrm{E}}}_{Z_j}{\mathrm{\Pi }}_{j-1}{\underset{\_}{A}}_j{K}_{j,j+1}=K_{j-1,j+1}={\widehat{\mathrm{E}}}_{Z_j}{K}_{j-1,j}{\underset{\_}{A}}_j{\mathrm{\Pi }}_{j,j+1}.$$

This allows to reduce (10.8) to

$$B_l={(-1)}^{m-1-l}\widehat{\mathrm{E}}{\widehat{A}}_1{\widehat{\mathrm{\Pi }}}_{1,2}{\underset{\_}{A}}_2{\widehat{\mathrm{\Pi }}}_{2,3}\times \cdots \times {\underset{\_}{A}}_1{\widehat{\mathrm{\Pi }}}_{l,l+1}{\widehat{Y}}_{l+1},l\ge 1,$$

$$B_0={(-1)}^{m-1}\widehat{\mathrm{E}}{\widehat{A}}_1{\mathrm{\Pi }}_{1,2}{\widehat{Y}}_2.$$

Thus after expanding the two terms of (10.7) in the quantities B_l, and simplifying these quantities, we can write their sum (10.7)

$$(B_0-B_0)+{\displaystyle \sum _{l=1}^{m-1}\left(\left(\genfrac{}{}{0ex}{}{m}{l}\right)-\left(\genfrac{}{}{0ex}{}{m-1}{l}\right)\right)B_l{(-1)}^{m-l}+B_m}.$$

The difference of the binomial coefficients is $\left(\frac{m-1}{l-1}\right)$. The expression is equal to (10.6), as claimed. This completes the proof of (10.5).

Next we bound the right side of (10.5), by taking the expectation in turn with respect to X_m, X_m−1, …, X₁. For M_ŵ multiplication by the function ŵ = g/ĝ,

$${\widehat{\mathrm{E}}}_{X_m}{(\widehat{\mathrm{\Pi }}-\mathrm{\Pi })}_{m-1,m}{\widehat{Y}}_m=(\widehat{\mathrm{\Pi }}{M}_{\widehat{w}}-\mathrm{\Pi })\left(\frac{\widehat{b}-b}{\underset{\_}{b}}\right)(Z_m-1).$$

Next, for any function h and i = m − 1, m − 2, …, 2,

$${\widehat{\mathrm{E}}}_{X_i}{(\widehat{\mathrm{\Pi }}-\mathrm{\Pi })}_{i-1,i}{\underset{\_}{A}}_{i}h(Z_i)=(\widehat{\mathrm{\Pi }}{M}_{\widehat{w}}-\mathrm{\Pi })h(Z_{i-1}).$$

Combining these equations, we can write the right side of (10.5) in the form

$${(-1)}^{m-1}{\displaystyle \int \left(\frac{a-\widehat{a}}{\underset{\_}{a}}\right)}\left[{(\widehat{\mathrm{\Pi }}{M}_{\widehat{w}}-\mathrm{\Pi })}^{m-1}\left(\frac{\widehat{b}-b}{\underset{\_}{b}}\right)\right]\underset{\_}{\mathit{abg}}dv.$$

We bound this by first applying Hölder’s inequality, with conjugate pair (τ, t) with τ equal to r as in the statement of the theorem, and next Lemma 12.7(iii), with $\widehat{\mathrm{\Pi }}$ and Π viewed as weighted orthogonal projections in L₂(abĝ) with weights 1 and ŵ, respectively, and r = τ(m−1)/(m+τ−3), p = (m+τ−3)/(τ−2) and q = (m+τ−3)/(m−1), so that rp = (m−1)τ/(τ−2) and rq = τ (and m of the lemma taken equal to the present m minus 1).

To bound the (conditional) variance of ${\widehat{\chi }}_n$ we use Lemma 13.1 to see that

$$P^n{({\mathbb{U}}_n{\stackrel{\sim }{\chi }}_{\widehat{p}}^{(j)})}^2\le 2j{\left[1+{\Vert \frac{p}{\widehat{p}}\Vert }_{\infty }\right]}^{2j}{\widehat{P}}^n{({\mathbb{U}}_n{\stackrel{\sim }{\chi }}_{\widehat{p}}^{(j)})}^2\lesssim {\left[1+{\Vert \frac{p}{\widehat{p}}\Vert }_{\infty }\right]}^{2j}\frac{2j}{\left(\frac{n}{j}\right)}{\widehat{P}}^j{({\stackrel{\sim }{\chi }}_p^{(j)})}^2,$$

because ${\stackrel{\sim }{\chi }}_{\widehat{p}}^{(j)}$ is degenerate under $\widehat{P}$. The variable ${\stackrel{\sim }{\chi }}_{\widehat{p}}^{(j)}(X_1,\dots ,X_j)/j!$ is the symmetrization of the projection of ${\widehat{A}}_1{\widehat{\mathrm{\Pi }}}_{1,2}{\underset{\_}{A}}_2\dots {\widehat{\mathrm{\Pi }}}_{j-1,j}{\widehat{Y}}_j$ onto the degenerate variables. Because the second moment of a mean of (arbitrary) random variables is bounded above by the maximum of the second moments of the terms, we can ignore the symmetrization, while the projection decreases the second moment. This shows that

$$\frac{1}{{(j!)}^2}{\widehat{P}}^j{({\stackrel{\sim }{\chi }}_p^{(j)})}^2\le {\widehat{P}}^j{({\widehat{A}}_1{\widehat{\mathrm{\Pi }}}_{1,2}{\underset{\_}{A}}_2\dots {\widehat{\mathrm{\Pi }}}_{j-1,j}{\widehat{Y}}_j)}^2\lesssim k^{j-1},$$

by Lemma 12.4 and the assumption that the kernels are bounded by k on the diagonal.

We complete the proof of Theorem 7.1 by bounding the square of ${\widehat{\chi }}_n-\chi (\widehat{p})$ by ${\displaystyle {\sum }_{j=1}^m{2}^j{({\mathbb{U}}_n{\stackrel{\sim }{\chi }}_{\widehat{p}}^{(j)}/j!)}^2}{\displaystyle {\sum }_j{2}^{-j}}$. The extra factor 2^j can be incorporated in the constant c in the theorem.

For the proof of Theorem 7.2 it clearly suffices to show that

$${\widehat{\mathrm{E}}}_p\sqrt{n}({\widehat{\chi }}_n-\chi (p)-{\mathbb{P}}_n{\stackrel{\sim }{\chi }}_{(p)}^{(1)})\stackrel{p}{\to }0,$$

$${\text{var}}_p\sqrt{n}({\widehat{\chi }}_n-\chi (p)-{\mathbb{P}}_n{\stackrel{\sim }{\chi }}_{(p)}^{(1)})\stackrel{p}{\to }0.$$

Because an influence function is centered at mean zero, the first is simply $\sqrt{n}$ times the bias of ${\widehat{\chi }}_n$. By Theorem 7.1 the bias is of the order

$${\left(\frac{\log }{n}\right)}^{\alpha /2(\alpha +d)+\beta /(2\beta +d)+\gamma (m-1)/(2\gamma +d)}+{\left(\frac{1}{k}\right)}^{(\alpha +\beta )/d}.$$

The first term is trivially o(n^−1/2), as m_n → ∞. In the second we write (α + β)/d = r/2, where r > 1 by assumption, and see that it is o(n^−1/2), since kn^−1/r → ∞.

To handle the variance we split the estimator ${\widehat{\chi }}_n$ in its linear and higher order terms. The sum of the variances of the U-statistics of orders 2 to m in ${\widehat{\chi }}_n$ is bounded by the sum of the terms j ≥ 2 in Theorem 7.1, i.e.

$${\displaystyle \sum _{j=2}^m\frac{c^j{k}^{j-1}}{\left(\frac{n}{j}\right)}\le }\frac{1}{n}{{\displaystyle \sum _{j=2}^m\left(\frac{2\mathit{ckj}}{n}\right)}}^{j-1}\frac{c2j\sqrt{j}}{e^j},$$

by the inequalities $\left(\frac{n}{j}\right)\ge {(n/2)}^j/j!$, for j < n/2, and $j!\lesssim {(j/e)}^j\sqrt{j}$, by Stirling’s approximation with bound. The expression in brackets is bounded by 2ckm/n ≲ 1/log n, for m ~ log n and k ~ n/(log n)². Thus the sum tends to zero by dominated convergence. Finally the linear term in ${\widehat{\chi }}_n$ gives the contribution

$${\widehat{\text{var}}}_p\sqrt{n}({\mathbb{P}}_n{\stackrel{\sim }{\chi }}_{\widehat{p}}^{(1)}-\chi (p)-{\mathbb{P}}_n{\stackrel{\sim }{\chi }}_p^{(1)})=\mathrm{v}\widehat{\mathrm{a}}\mathrm{r}({\stackrel{\sim }{\chi }}_{\widehat{p}}^{(1)}-{\stackrel{\sim }{\chi }}_p^{(1)}).$$

From the explicit expression (4.3) for the first order influence function (or (6.5) in the case of $\widehat{p}$, which gives an identical function), this is seen to tend to zero by the dominated convergence theorem.

10.4. Proof of Theorem 8.1 for m = 3

The theorem asserts that the bias of the estimator ${\widehat{\chi }}_n$ is equal to the sum of four terms, the first two of which also arise in the bias of the estimator considered in Theorem 7.1. Therefore, we can prove the assertion on the bias by showing that the expected values of the current estimator ${\widehat{\chi }}_n$ (for m = 3) and the estimator in Theorem 7.1 differ by less than the additional bias terms in Theorem 8.1.

The two estimators differ only in their third order influence functions, where the present estimator retains only the terms in the double sum (8.3) with r = 0, s = 0, or r + s ≤ D. Thus the difference of the expectations of the two estimators is equal to

$${\displaystyle \sum _{\begin{array}{l}r+s>D\\ r,s\ge 1\end{array}}{\displaystyle \sum {\widehat{\mathrm{E}}}_p{\widehat{A}}_1\left[{\widehat{\mathrm{\Pi }}}^{(k_{r-1},k_r]}(Z_1,Z_2){\underset{\_}{A}}_2{\widehat{\mathrm{\Pi }}}^{(l_{s-1},l_s]}(Z_2,Z_3)-{\widehat{\mathrm{\Pi }}}^{(k_{r-1}\vee l_{s-1},k_r\wedge l_s]}(Z_1,Z_3)\right]}}{\widehat{Y}}_3.$$

The expectation Ê_p refers to the variable (X₁, X₂, X₃) for fixed values of the preliminary samples, which are indicated in the “hat” symbols on Â₁, Ŷ₃ and the kernels, and hence is an integral relative to the density (x₁, x₂, x₃) ↦ p(x₁)p(x₂)p(x₃). If we replace p(x₂) in this density by $\widehat{p}(x_2)$, then the integral will be zero, as the kernel is degenerate under $\widehat{P}$. Thus we may integrate against $(x_1,x_2,x_3)\mapsto p(x_1)(p-\widehat{p})(x_2)p(x_3)$. In that case the projection term ${\widehat{A}}_1{\widehat{\mathrm{\Pi }}}^{(k_{r-1}\vee l_{s-1},k_r\wedge l_s]}(Z_1,Z_3){\widehat{Y}}_3$ integrates to zero, as it does not depend on X₂ and ${\displaystyle \int (p-\widehat{p})}(x_2)d\mu (x_2)=0$, and hence can be dropped. Next we condition ${\widehat{A}}_1$ and ${\widehat{Y}}_3$ on Z₁, Z₂, Z₃ and write the preceding display in the form

$${\displaystyle \sum _{\begin{array}{l}r+s>D\\ r,s\ge 1\end{array}}{\displaystyle \sum {\displaystyle \int {\displaystyle \int {\displaystyle \int \frac{\widehat{a}-a}{\underset{\_}{a}}}}}}}(z_1){\widehat{\mathrm{\Pi }}}^{(k_{r-1},k_r]}(z_1,z_2){\widehat{\mathrm{\Pi }}}^{(l_{s-1},l_s]}(z_2,z_3)\frac{\widehat{b}-b}{\underset{\_}{b}}(z_3)\times d\rho (z_1)d(\rho -\widehat{\rho })(z_2)d\rho (z_3).$$

for ρ and $\widehat{\rho }$ the measures defined by dρ = abg dv and $d\widehat{\rho }=\underset{\_}{ab\widehat{g}}dv$. The double sum can be rewritten as the sum over r running from 1 to R and over s from D − r + 1 to S, which gives the equivalent representation, with the × referring to “tensor products” as explained in Section 2,

$${\displaystyle \sum _{r=1}^R{\displaystyle \int \left(\frac{\widehat{a}-a}{\underset{\_}{a}}\times 1\times \frac{\widehat{b}-b}{\underset{\_}{b}}\right)}\left({\widehat{\mathrm{\Pi }}}^{(k_{r-1},k_r]}\times {\widehat{\mathrm{\Pi }}}^{(l_{D-r},k]}\right)d(\rho \times (\rho -\widehat{\rho })\times \rho )}.$$

We write ${\widehat{\mathrm{\Pi }}}^{(k_{r-1},k_r]}={\widehat{\mathrm{\Pi }}}^{(k_{r-1},k]}-{\widehat{\mathrm{\Pi }}}^{(k_r,k]}$, and next arrive at the difference of two expressions of the type, with $k_r^{\prime }=k_{r-1}$ and $k_r^{\prime }=k_r$, respectively,

$${\displaystyle \sum _{r=1}^R{\displaystyle \int \left(\frac{\widehat{a}-a}{\underset{\_}{a}}\times 1\times \frac{\widehat{b}-b}{\underset{\_}{b}}\right)\left({\widehat{\mathrm{\Pi }}}^{\left(k_r^{\prime },k\right]}\times {\widehat{\mathrm{\Pi }}}^{\left(l_{D-r},k\right]}\right)d(\rho \times (\rho -\widehat{\rho })\times \rho )}}.$$

If the measure of integration were $\widehat{\rho }\times (\rho -\widehat{\rho })\times \rho$ (with $\widehat{\rho }$ instead of ρ), then we could perform the integrals on z₁ and z₃ and next apply Hölder’s inequality to bound the resulting expression in absolute value by

$${\displaystyle \sum _{r=1}^R{\Vert {\widehat{\mathrm{\Pi }}}^{\left(k_r^{\prime },k\right]}\left(\frac{\widehat{a}-a}{\underset{\_}{a}}\right)\Vert }_r}{\Vert {\widehat{\mathrm{\Pi }}}^{\left(l_{D-r},k\right]}\left(\frac{\widehat{b}-b}{\underset{\_}{b}}\right)\Vert }_r{\Vert \frac{g}{\widehat{g}}-1\Vert }_{r/(r-2)},$$

where the norms are those of $L_2(\underset{\_}{ab}\widehat{g})$, which are equivalent to those of L₂(ν), by assumption. We can write ${\widehat{\mathrm{\Pi }}}^{\left(l,k\right]}={\widehat{\mathrm{\Pi }}}^{\left(0,k\right]}(I-{\widehat{\mathrm{\Pi }}}^{\left(0,l\right]})$ and use the assumed boundedness of ${\widehat{\mathrm{\Pi }}}^{\left(0,l\right]}$ as an operator on $L_r(\underset{\_}{ab}g)$ to bound this by the third term in the bias.

Replacing $\rho \times (\rho -\widehat{\rho })\times \rho$ by $\widehat{\rho }\times (\rho -\widehat{\rho })\times \widehat{\rho }$ can be achieved by writing the first and last occurrence of ρ as $\rho =\widehat{\rho }+(\rho -\widehat{\rho })$ and expanding the resulting expression on the + signs into four terms. One of these has the measure $\widehat{\rho }\times (\rho -\widehat{\rho })\times \widehat{\rho }$. The other three terms have two or three occurrences of $\rho -\widehat{\rho }$, and can be bounded by the first term in the bias (with m = 3). This is argued precisely under (10.12) below.

Because the first and second order influence functions are equal to those of the estimator considered in Theorem 7.1, the (conditional) variances of ${\mathbb{U}}_n{\stackrel{\sim }{\chi }}_{\widehat{p}}^{(j)}$ for j = 1, 2 can be seen to be of the orders O(1/n) and O(k/n²), respectively, by the same proof. By Lemma 13.1 the variance for j = 3 is bounded by (see (13.1))

$$6{\left(1+{\Vert \frac{p}{\widehat{p}}\Vert }_{\infty }\right)}^6{\widehat{P}}^n{({\mathbb{U}}_n{\chi }_{\widehat{p}}^{(3)})}^2\lesssim \frac{1}{\left(\frac{n}{3}\right)}{\widehat{P}}^3{\left({\displaystyle \sum _{r=0}^R{\widehat{A}}_1{\widehat{\mathrm{\Pi }}}^{\left(k_{r-1},k_r\right]}(Z_1,Z_2){\underset{\_}{A}}_2{\widehat{\mathrm{\Pi }}}^{\left(0,l_{D-r}^{\prime }\right]}(Z_2,Z_3){\widehat{Y}}_3}\right)}^2,$$

where $l_{D-r}^{\prime }=l_{D-r}\vee n$. After bounding out ${\widehat{A}}_1^2$ and ${\widehat{Y}}_3^2$, we write the squared sum as a double sum. From the fact that the projections ${\widehat{\mathrm{\Pi }}}^{\left(k_{r-1},k_r\right]}$ are orthogonal for different r, it follows that the off-diagonal terms of the double sum vanish (the expectation with respect to X₁ is zero). Thus the preceding display is bounded above by a multiple of

$$\frac{1}{n^3}{\displaystyle \sum _{r=0}^R{\widehat{P}}^3{({\widehat{\mathrm{\Pi }}}^{\left(k_{r-1},k_r\right]}(Z_1,Z_2){\underset{\_}{A}}_2{\widehat{\mathrm{\Pi }}}^{\left(0,l_{D-r}^{\prime }\right]}(Z_2,Z_3))}^2}.$$

By Lemmas 12.4 and 12.3 and the assumption that ${\sup }_z{\widehat{\mathrm{\Pi }}}^{\left(0,l\right]}(z,z)\lesssim l$ this is bounded by a multiple of

$$\frac{1}{n^3}{\displaystyle \sum _{r=0}^R(k_r-k_{r-1})l_{D-r}^{\prime }\le \frac{1}{n^3}\left(nk+{\displaystyle \sum _{r=1}^R(k_r-k_{r-1})(l_{D-r}+n)}\right)}.$$

By (8.4) $k_r-k_{r-1}=(1-2^{-\alpha })k_r\lesssim k_r=n{2}^{r/\alpha }$ for r ≥ 1. On substituting this in the display, and noting that l_D−r = 0 if r > D, we see that this is bounded k/n² + 2^D/α˅D/β/n if α ≠ β and bounded by k/n² + D2^D/α/n if α = β.