Quadratic semiparametric Von Mises calculus

Abstract

We discuss a new method of estimation of parameters in semiparametric and nonparametric models. The method is based on U-statistics constructed from quadratic influence functions. The latter extend ordinary linear influence functions of the parameter of interest as defined in semiparametric theory, and represent second order derivatives of this parameter. For parameters for which the matching cannot be perfect the method leads to a bias-variance trade-off, and results in estimators that converge at a slower than n^–1/2-rate. In a number of examples the resulting rate can be shown to be optimal. We are particularly interested in estimating parameters in models with a nuisance parameter of high dimension or low regularity, where the parameter of interest cannot be estimated at n^–1/2-rate.

1 Introduction

Let X₁, X₂, . . . , X_n be a random sample from a distribution P_η with density p_η relative to a measure μ on a sample space $(\mathcal{X},\mathcal{A})$, where the parameter η is known to belong to a subset H of a normed space. We wish to estimate the value χ(η) of a functional $\chi :H\to \mathbb{R}$ with the help of the observations X₁, . . . , X_n. Our main interest is in the situation of a semiparametric or nonparametric model, where H is an infinite-dimensional set, and the dependence $\eta \mapsto p_{\eta }$ is smooth.

This problem has been studied under the heading “semiparametric statistics” in the 1980s and 1990s. A theory of asymptotic lower bounds for “regular parameters” χ(η) based on Le Cam's concept of local asymptotic normality (Le Cam 1960) was developed starting with Koševnik and Levit (1976) and Pfanzagl (1982), and worked out for many examples in, among others, Begun et al. (1983), van der Vaart (1988) and Bickel et al. (1993). There are many examples of ad-hoc estimators that attain these bounds, and the behaviour of principled methods such as maximum likelihood (including its sieved and penalized variants) or estimating equations is understood to a certain extent (e.g., van der Vaart 1994; Murphy and van der Vaart 2000; Bolthausen et al. 2002; Wellner et al. 1993; van der Laan and Robins 2003).

Certain combinations of models (P_η: η ∈ H) and parameter χ(η) possess structural properties that allow to estimate the parameter at n^–1/2-rate, no matter the size of the parameter set H. In this paper we are interested in the other situations, where the rate of estimation drops when the complexity of the model exceeds a certain limit. Such examples arise for instance when many covariates must be included in a model to correct for possible confounding in a causal study, or for modelling the probability that an individual is included in a sample in a study with missing observations. If simple (e.g., linear) models for the dependence on these covariates are not plausible, which is typical in epidemiological studies, then the resulting model must be taken so large that the usual methods fail. These methods typically focus on variance only, because the bias is negligible due to the structure of the model, or by explicitly assuming a “no-bias condition” (see Klaassen 1987; Murphy and van der Vaart 2000). In this paper we develop new methods that make a bias-variance trade-off when necessary.

These methods are based on quadratic estimating equations rather than the usual linear estimating equations.

Quadratic expansions for semiparametric models were previously investigated by Pfanzagl and Wefelmeyer (Pfanzagl 1985), but from the very different perspective of second order efficiency, i.e., the refinement of first order bounds by adding a lower order term. Our aim is to show that second order influence functions can be used for first order inference, because they permit balancing of bias and variance.

Following linear and quadratic is cubic, and so on. Extension of our approach to still higher orders is possible, but comes with many new complications. We shall pursue this elsewhere.

The paper is organized as follows. In Sect. 2 we review linear estimators from our current perspective. Next in Sect. 3 we introduce our new method of constructing quadratic estimators. This section has mostly a heuristic nature. In Sects. 4 and 5 we give rigorous constructions and results for two examples. The first is a classical theoretical example. The second is more extensive and concerns estimating a mean response when the response is not always observed.

Notation Let ${\mathbb{P}}_n$ and ${\mathbb{U}}_n$ denote the empirical measure and empirical U-statistic measure, viewed as an operators on functions: for given functions $f:\mathcal{X}\to \mathbb{R}$ and $g:{\mathcal{X}}^2\to \mathbb{R}$ these are given by

$${\mathbb{P}}_{n}f=\frac{1}{n}\sum _{i=1}^{n}f\left(X_i\right),{\mathbb{U}}_{n}g=\frac{1}{n(n-1)}\underset{1\le i\ne j\le n}{\sum \sum }g(X_i,X_j).$$

We use the notation ${\mathbb{U}}_{n}f$ also for $f:\mathcal{X}\to \mathbb{R}$ a function of one argument, with the interpretation ${\mathbb{U}}_{n}f={\mathbb{P}}_{n}f$. This is consistent with the given formulas if a function of one argument is considered as a function of 2 arguments that is constant in its second argument.

We write $P^n{\mathbb{U}}_{n}g=P^{2}g$ for the expectation of ${\mathbb{U}}_{n}g$ if X₁, . . . , X_n are distributed according to the probability measure P. We also use the operator notation for the expectations of statistics in general.

We call a measurable function $g:{\mathcal{X}}^2\to \mathbb{R}$ degenerate relative to P if $\int g(x_1,x_2)dP\left(x_i\right)=0$ for i = 1, 2, and we call it symmetric if g(x₁, x₂) = g(x₂, x₁) for every $x_1,x_2\in \mathcal{X}$.

Given two functions $g,h:\mathcal{X}\to \mathbb{R}$ we write g × h the function $(x,y)\mapsto g\left(x\right)h\left(y\right)$. Such tensor products functions are degenerate if both functions f and g have mean zero. The corresponding notation P × Q of two measures P and Q gives the product measure.

2 Linear estimator

Given an initial estimator $\widehat{\eta }$ of η, the plug-in estimator $\chi \left(\widehat{\eta }\right)$ is typically a consistent estimator of the parameter of interest χ(η), but it may not be a good estimator. In particular, if $\widehat{\eta }$ is a general purpose estimator, not specially constructed to yield a good plug-in, then $\chi \left(\widehat{\eta }\right)$ will often have a suboptimal precision. To gain insight in this situation we assume that the parameter permits a Taylor expansion of the form

$$\chi \left(\eta \right)=\chi \left(\widehat{\eta }\right)+{\chi }_{\widehat{\eta }}^{\prime }(\eta -\widehat{\eta })+O\left({\Vert \eta -\widehat{\eta }\Vert }^2\right).$$ (1)

Such an expansion suggests that the plug-in estimator will have an error of the order $O_P\left(\Vert \eta -\widehat{\eta }\Vert \right)$, unless the linear term in the expansion vanishes.

The expansion (1) also suggests that better estimators can be obtained by “estimating” the linear term in the expansion. To achieve this we assume a “generalized von-Mises representation” of the derivative of the form

$${\chi }_{\widehat{\eta }}^{\prime }(\eta -\widehat{\eta })=\int {\stackrel{.}{\chi }}_{\widehat{\eta }}d(P_{\eta }-P_{\widehat{\eta }})=\int {\stackrel{.}{\chi }}_{\widehat{\eta }}d{P}_{\eta }+O\left({\Vert \eta -\widehat{\eta }\Vert }^2\right),$$ (2)

for some measurable function ${\stackrel{.}{\chi }}_{\widehat{\eta }}:\mathcal{X}\to \mathbb{R}$, referred to as an influence function. The second equality is valid if ${\stackrel{.}{\chi }}_{\eta }$ is degenerate relative to $P_{\eta }(\mathrm{i}.\mathrm{e}.,P_{\eta }{\stackrel{.}{\chi }}_{\eta }=0)$ for every η, which can always be arranged by a recentering, as $\int 1d(P_{\eta }-P_{\widehat{\eta }})=0$. The von-Mises representation and Eq. (1) suggest the “corrected plug-in estimator”

$$\chi \left(\widehat{\eta }\right)+{\mathbb{P}}_n{\stackrel{.}{\chi }}_{\widehat{\eta }}.$$ (3)

This estimator should have an error of the order $O_P\left(n^{-1∕2}\right)+O_P\left({\Vert \eta -\widehat{\eta }\Vert }^2\right)$, as the difference $({\mathbb{P}}_n-P_{\eta }){\stackrel{.}{\chi }}_{\widehat{\eta }}$ is “centered” and ought to have “variance” of the order O(1/n).

We put “centered” and “variance” in quotes, because the randomness in the initial estimator $\widehat{\eta }$ prevents a simple calculation of mean and variance. Empirical process theory can be used to show that the effect of replacing ${\stackrel{.}{\chi }}_{\eta }$ by ${\stackrel{.}{\chi }}_{\widehat{\eta }}$ is negligible, if the class of functions ${\stackrel{.}{\chi }}_{\eta }$ is not too rich. In the present paper we are interested in orders of magnitude only, and then a simpler approach is to split the sample and use separate observations to construct $\widehat{\eta }$ and to construct ${\mathbb{P}}_n$. Then the orders can be justified by reasoning conditionally on the first sample, and it suffices that $\int {\stackrel{.}{\chi }}_{\widehat{\eta }}^{2}d{P}_{\eta }$ remains bounded in probability.

Von Mises (1947) originally introduced the expansions that are named after him in order to investigate functionals of empirical distributions. The idea to use expansions (1) for estimation in nonparametric models occurs in Emery et al. (2000). Our situation is more involved, because we are interested in models (P_η: η ∈ H) that are structured through a map $\eta \mapsto p_{\eta }$, and we are interested in a functional χ(η) of the parameter. In this situation a von-Mises type expansion can fail for two reasons. First a derivative ${\chi }_{\eta }^{\prime }$ is by definition a continuous, linear map on the underlying normed space, and such maps may or may not be representable as an integral, depending on the normed space. Second, our von Mises expansion (2) represents this derivative as an integral relative to the distribution P_η and hence also involves the inverse map P_η → η from the distribution of the data to the parameter. We require representation through P_η, because this allows us to construct the estimator (3) by replacing P_η by the empirical distribution.

These issues are related to investigations in the theory of semiparametric models (see Koševnik and Levit 1976; Pfanzagl 1982; van der Vaart 1988; Bickel et al. 1993). These papers define a tangent set of a semiparametric model (P_η: η ∈ H) as the set of functions ${\stackrel{.}{g}}_{\eta }:\mathcal{X}\to \mathbb{R}$ obtainable as

$$\frac{1}{2}{\stackrel{.}{g}}_{\eta }\sqrt{p_{\eta }}=\underset{t\downarrow 0}{\text{lim}}\frac{\sqrt{p_{{\eta }_t}}-\sqrt{p_{\eta }}}{t},$$

where the limit is taken in the L₂-sense, and $t\mapsto {\eta }_t$ ranges over a collection of maps from $[0,1]\subset \mathbb{R}$ to H for which the limit exists. Informally, a “tangent vector” ${\stackrel{.}{g}}_{\eta }$ is just a score function

$${\stackrel{.}{g}}_{\eta }={\frac{\partial }{\partial t}}_{\mid t=0}\text{log}{p}_{{\eta }_t}=\frac{{\frac{\partial }{\partial t}}_{\mid t=0}{p}_{{\eta }_t}}{p_{\eta }},$$ (4)

of a one-dimensional submodel (P_ηt : t ≥ 0) at t = 0, where η₀ = η. (Taking the derivative in the L₂-sense is appropriate for asymptotic information theory, but not necessarily so for the present heuristic discussion.) An influence function is defined as a measurable map ${\stackrel{.}{\chi }}_{\eta }:\mathcal{X}\to \mathbb{R}$ such that, for all paths $t\mapsto {\eta }_t$ considered,

$${\frac{d}{dt}}_{\mid t=0}\chi \left({\eta }_t\right)=P_{\eta }{\stackrel{.}{\chi }}_{\eta }{\stackrel{.}{g}}_{\eta }.$$ (5)

It is not difficult to see that the latter influence function is the same as the influence function needed in the von-Mises expansion (2), if the various types of derivatives match up. (Note that the middle expression in (2) with η replaced by η_t and $\widehat{\eta }$ by η expands to $P_{\eta }{\stackrel{.}{\chi }}_{\eta }{\stackrel{.}{g}}_{\eta }+o\left(t\right)$, as $p_{{\eta }_t}-p_{\eta }=t{\stackrel{.}{g}}_{\eta }{p}_{\eta }+o\left(t\right)$.) Necessary and sufficient conditions for existence of an influence function in terms of the derivatives of the maps $\eta \mapsto \chi \left(\eta \right)$ and $\eta \mapsto p_{\eta }$ were investigated in van der Vaart (1991).

An influence function is not necessarily unique, as only its inner products with elements ${\stackrel{.}{g}}_{\eta }$ of the tangent set matter. The projection of any influence function that is contained in the closed linear span of the tangent set is called the efficient influence function or canonical gradient, as it is the influence function of asymptotically efficient estimators. It minimizes the variance ${\text{var}}_{\eta }{\mathbb{P}}_n{\stackrel{.}{\chi }}_{\eta }$ over all influence functions.

3 Quadratic estimator

If the preliminary estimator $\widehat{\eta }$ attains a rate of convergence $\Vert \widehat{\eta }-\eta \Vert =o_P\left(n^{-1∕4}\right)$, then the plug-in estimator (3) attains a n^–1/2-rate of convergence. Typically this will require that the parameter set H is not too big. If the preliminary estimator is less precise, then the remainder term of the expansion (1) will dominate. This suggests to take the expansion further to

$$\chi \left(\eta \right)=\chi \left(\widehat{\eta }\right)+{\chi }_{\widehat{\eta }}^{\prime }(\eta -\widehat{\eta })+\frac{1}{2}{\chi }_{\widehat{\eta }}^{\prime \prime }(\eta -\widehat{\eta },\eta -\widehat{\eta })+O\left({\Vert \eta -\widehat{\eta }\Vert }^3\right).$$ (6)

The generalization of the first order construction now requires a von Mises type representation of the form, for measurable functions ${\stackrel{.}{\chi }}_{\eta }:\mathcal{X}\to \mathbb{R}$ and ${\ddot{\chi }}_{\eta }:{\mathcal{X}}^2\to \mathbb{R}$,

$$\begin{array}{cc}\hfill {\chi }_{\widehat{\eta }}^{\prime }(\eta -\widehat{\eta })+\frac{1}{2}{\chi }_{\widehat{\eta }}^{\prime \prime }(\eta -\widehat{\eta },\eta -\widehat{\eta })=& \int {\stackrel{.}{\chi }}_{\widehat{\eta }}d(P_{\eta }-P_{\widehat{\eta }})+\frac{1}{2}\iint {\ddot{\chi }}_{\widehat{\eta }}d(P_{\eta }-P_{\widehat{\eta }})\hfill \\ \hfill & \times (P_{\eta }-P_{\widehat{\eta }})+O\left({\Vert \eta -\widehat{\eta }\Vert }^3\right).\hfill \end{array}$$ (7)

We assume without loss of generality that the functions ${\stackrel{.}{\chi }}_{\eta }$ and ${\ddot{\chi }}_{\eta }$ are degenerate relative to P_η. The von-Mises representation then suggests the “corrected plug-in estimator”

$$\chi \left(\widehat{\eta }\right)+{\mathbb{P}}_n{\stackrel{.}{\chi }}_{\widehat{\eta }}+\frac{1}{2}{\mathbb{U}}_n{\ddot{\chi }}_{\widehat{\eta }}.$$ (8)

The empirical measure and two-sample U-statistic serve as unbiased estimators of the expectations of their kernels. For simplicity we may again base the initial estimator $\widehat{\eta }$ and these two U-statistics on independent samples of observations. Because the variance of a U-statistic is of order O(1/n), this estimator ought to have an error of the order $O_P\left(n^{-1∕2}\right)+O_P\left({\Vert \eta -\widehat{\eta }\Vert }^3\right)$. We shall discuss the validity of this later.

To characterize the first and second order influence functions we can again employ smooth one-dimensional submodels (P_ηt : t ≥ 0). With the first and second order derivatives of these models denoted by

$${\stackrel{.}{g}}_{\eta }=\frac{{\frac{\partial }{\partial t}}_{\mid t=0}{p}_{{\eta }_t}}{p_{\eta }},{\ddot{g}}_{\eta }=\frac{{\frac{{\partial }^2}{\partial t^2}}_{\mid t=0}{p}_{{\eta }_t}}{p_{\eta }},$$ (9)

the von Mises expansion (7) can informally be seen to imply

$${\frac{d}{dt}}_{\mid t=0}\chi \left({\eta }_t\right)=P_{\eta }{\stackrel{.}{\chi }}_{\eta }{\stackrel{.}{g}}_{\eta },$$ (10)

$${\frac{d^2}{d{t}^2}}_{\mid t=0}\chi \left({\eta }_t\right)=P_{\eta }{\stackrel{.}{\chi }}_{\eta }{\ddot{g}}_{\eta }+P_{\eta }^2{\ddot{\chi }}_{\eta }({\stackrel{.}{g}}_{\eta }\times {\stackrel{.}{g}}_{\eta }).$$ (11)

The Eq. (10) is identical to Eq. (5), and hence a first order influence function ${\stackrel{.}{\chi }}_{\eta }$ can be taken as before. Following Pfanzagl (1985) we define a second order influence function ${\ddot{\chi }}_{\eta }$ as a measurable function ${\ddot{\chi }}_{\eta }:{\mathcal{X}}^2\to \mathbb{R}$ that satisfies (11) for every path $t\mapsto {\eta }_t$ under consideration. From Eq. (11) we see that ${\ddot{\chi }}_{\eta }$ is unique only up to functions that are orthogonal to functions of the form ${\stackrel{.}{g}}_{\eta }\times {\stackrel{.}{g}}_{\eta }$, for ${\stackrel{.}{g}}_{\eta }$ belonging to the tangent set. In particular, a second order influence function ${\ddot{\chi }}_{\eta }$ can always be taken to be symmetric and degenerate relative to P_η. It must be taken so in the construction of the estimator (8).

The two influence functions occur together in Eq. (11), and hence should be considered a pair $({\stackrel{.}{\chi }}_{\eta },{\ddot{\chi }}_{\eta })$ of functions rather than as two separate functions. This is particularly important if the tangent set is not “full”, i.e., smaller than the set of all mean-zero functions in L₂(P_η), the tangent set of a nonparametric model. Both first and second order influence functions are then non-unique, but their different versions cannot be freely combined into valid pairs $({\stackrel{.}{\chi }}_{\eta },{\ddot{\chi }}_{\eta })$. This is connected to the fact that first and second order derivatives ${\stackrel{.}{g}}_{\eta }$ and ${\ddot{g}}_{\eta }$ are also not clearly separated. A simple change of speed $t\mapsto \varphi \left(t\right)$ of a path through a second order diffeomorphism ϕ: [0, 1] → [0, 1] leads to the submodel (P_ηϕ(t): t ≥ 0) with first and second order derivatives, by the chain rule,

$${\varphi }^{\prime }\left(0\right){\stackrel{.}{g}}_{\eta },{\varphi }^{\prime }{\left(0\right)}^2{\ddot{g}}_{\eta }+{\varphi }^{\prime \prime }\left(0\right){\stackrel{.}{g}}_{\eta }.$$

Thus the first order derivative becomes part of the second order derivative after reparameterization. Pfanzagl (Pfanzagl 1985, 2.4.4) has shown, under assumptions of smoothness of the tangent set as a function of the parameter, that the sum of every first order derivative and every second order derivative occurs as the second order derivative of some path. Thus the set of second order derivatives ${\ddot{g}}_{\eta }$ is only defined up to equivalence modulo the tangent set.

From Eq. (11) it is also clear that second order influence functions involve the joint distribution of two observations. Correspondingly, we prefer to define a second order tangent space of the model not through the second order derivatives ${\ddot{g}}_{\eta }$ along paths $t\mapsto p_{{\eta }_t}$, but through the functions of two arguments

$${\ddot{s}}_{\eta }:=\frac{{\frac{{\partial }^2}{\partial t^2}}_{\mid t=0}(p_{{\eta }_t}\times p_{{\eta }_t})}{p_{\eta }\times p_{\eta }}={\ddot{g}}_{\eta }\times 1+2{\stackrel{.}{g}}_{\eta }\times {\stackrel{.}{g}}_{\eta }+1\times {\ddot{g}}_{\eta }.$$ (12)

The function ${\ddot{s}}_{\eta }$ is a second order score for the model (P_η × P_η: η ∈ H) for two observations. The corresponding first order scores are

$${\stackrel{.}{s}}_{\eta }:=\frac{{\frac{\partial }{\partial t}}_{\mid t=0}(p_{{\eta }_t}\times p_{{\eta }_t})}{p_{\eta }\times p_{\eta }}={\stackrel{.}{g}}_{\eta }\times 1+1\times {\stackrel{.}{g}}_{\eta }.$$ (13)

With these notations the Eqs. (10), (11) defining the influence functions can also be written as, if ${\ddot{\chi }}_{\eta }$ is chosen degenerate,

$${\frac{d}{dt}}_{\mid t=0}\chi \left({\eta }_t\right)=P_{\eta }^2({\stackrel{.}{\chi }}_{\eta }+\frac{1}{2}{\ddot{\chi }}_{\eta }){\stackrel{.}{s}}_{\eta }={\frac{d}{dt}}_{\mid t=0}{P}_{{\eta }_t}^2({\stackrel{.}{\chi }}_{\eta }+\frac{1}{2}{\ddot{\chi }}_{\eta }),$$ (14)

$${\frac{d^2}{d{t}^2}}_{\mid t=0}\chi \left({\eta }_t\right)=P_{\eta }^2({\stackrel{.}{\chi }}_{\eta }+\frac{1}{2}{\ddot{\chi }}_{\eta }){\ddot{s}}_{\eta }={\frac{d^2}{d{t}^2}}_{\mid t=0}{P}_{{\eta }_t}^2({\stackrel{.}{\chi }}_{\eta }+\frac{1}{2}{\ddot{\chi }}_{\eta }).$$ (15)

Here we interprete the function ${\stackrel{.}{\chi }}_{\eta }:\mathcal{X}\to \mathbb{R}$ as a function ${\stackrel{.}{\chi }}_{\eta }:{\mathcal{X}}^2\to \mathbb{R}$ that depends on the first argument only (and is constant in the second), or (better) replace it by its symmetrization $(x_1,x_2)\mapsto \frac{1}{2}({\stackrel{.}{\chi }}_{\eta }\left(x_1\right)+{\stackrel{.}{\chi }}_{\eta }\left(x_2\right))$. The equations show that the overall influence function ${\stackrel{.}{\chi }}_{\eta }+\frac{1}{2}{\ddot{\chi }}_{\eta }$ is characterized by having “correct” inner products with the overall scores ${\stackrel{.}{s}}_{\eta }$ and ${\ddot{s}}_{\eta }$. This overall influence function uniquely defines its constituents ${\stackrel{.}{\chi }}_{\eta }$ and $\frac{1}{2}{\ddot{\chi }}_{\eta }$ provided ${\ddot{\chi }}_{\eta }$ is restricted to be degenerate. The overall influence function is itself unique only up to projection onto the closed linear span in L₂(P_η × P_η) of all functions ${\stackrel{.}{s}}_{\eta }$ and ${\ddot{s}}_{\eta }$.

The equality of the far left and right sides of Eqs. (14), (15) gives an alternative characterization of the overall influence function (at η₀) as a function such that the maps $\eta \mapsto \chi \left(\eta \right)$ and $\eta \mapsto P_{\eta }^2({\stackrel{.}{\chi }}_{{\eta }_0}+\frac{1}{2}{\ddot{\chi }}_{{\eta }_0})$ possess the same first and second order derivatives at η₀. Because the derivatives of a map ϕ on an open subset H of a normed space are completely characterized by the derivatives of the maps $t\mapsto \varphi ({\eta }_0+th)$, for h ranging over the space (“Gateaux derivatives”), we conclude that in the case of such parameters sets H it suffices to consider linear paths $t\mapsto {\eta }_t={\eta }_0+th$. (The mixed second derivative ${\varphi }_{{\eta }_0}^{\prime \prime }(g,h)$ can be recovered from ${\varphi }_{{\eta }_0}^{\prime \prime }(g+h,g+h)$ and ${\varphi }_{{\eta }_0}^{\prime \prime }(g-h,g-h)$ by “polarization”.) This is true more generally for parameter spaces H defined by a linear constraint, but in the case of nonlinear constraints the use of curved paths is necessary.

The plug-in estimator (8) can be written $\chi \left(\widehat{\eta }\right)+{\mathbb{U}}_n({\stackrel{.}{\chi }}_{\widehat{\eta }}+\frac{1}{2}{\ddot{\chi }}_{\widehat{\eta }})$. A definition of an efficient or canonical second order influence function, should therefore refer to the variance of the U-statistic ${\mathbb{U}}_n({\stackrel{.}{\chi }}_{\eta }+\frac{1}{2}{\ddot{\chi }}_{\eta })$. Unlike in the linear case this does not translate in the variance of the influence function ${\stackrel{.}{\chi }}_{\eta }+\frac{1}{2}{\ddot{\chi }}_{\eta }$ itself (except for n = 2 if ${\stackrel{.}{\chi }}_{\eta }$ is interpreted as the symmetric function $(x_1,x_2)\mapsto \frac{1}{2}({\stackrel{.}{\chi }}_{\eta }\left(x_1\right)+{\stackrel{.}{\chi }}_{\eta }\left(x_2\right))$). By Eq. (5), if ${\ddot{\chi }}_{\eta }$ is chosen degenerate and symmetric,

$$n{\text{var}}_{\eta }{\mathbb{U}}_n({\stackrel{.}{\chi }}_{\eta }+\frac{1}{2}{\ddot{\chi }}_{\eta })=P_{\eta }{\stackrel{.}{\chi }}_{\eta }^2+\frac{1}{2(n-1)}{P}_{\eta }^2{\ddot{\chi }}_{\eta }^2.$$

Thus the second order part adds a term of order O(1/n) relative to the first order contribution. The norm of the function ${\stackrel{.}{\chi }}_{\eta }+\frac{1}{2}{\ddot{\chi }}_{\eta }$ in L₂(P_η × P_η) is irrelevant, even though the inner product of this space determines the influence functions.

It is possible to resolve this discrepancy by working in the model with n observations. From the expansion

$$\begin{array}{cc}\hfill \prod _{i=1}^n\frac{p_{{\eta }_t}}{p_{\eta }}\left(x_i\right)& =\prod _{i=1}^n\left(1+t{\stackrel{.}{g}}_{\eta }\left(x_i\right)+\frac{1}{2}{t}^2{\ddot{g}}_{\eta }\left(x_i\right)+\cdots \right)\hfill \\ \hfill & =1+t\sum _{i=1}^n{\stackrel{.}{g}}_{\eta }\left(x_i\right)+t^2\left(\frac{1}{2}\sum _{i=1}^n{\ddot{g}}_{\eta }\left(x_i\right)+\underset{1\le i<j\le n}{\sum \sum }{\stackrel{.}{g}}_{\eta }\left(x_i\right){\stackrel{.}{g}}_{\eta }\left(x_j\right)\right)+\cdots ,\hfill \end{array}$$

we see that first and second order scores for the model $(P_{\eta }^n:\eta \in H)$ take the forms

$${\stackrel{.}{s}}_{\eta }^{\left(n\right)}=\frac{{\frac{\partial }{\partial t}}_{\mid t=0}(p_{{\eta }_t}\times \cdots \times p_{{\eta }_t})}{p_{\eta }\times \cdots \times p_{\eta }}=n{\mathbb{P}}_n{\stackrel{.}{g}}_{\eta },$$ (16)

$${\ddot{s}}_{\eta }^{\left(n\right)}=\frac{{\frac{{\partial }^2}{\partial t^2}}_{\mid t=0}(p_{{\eta }_t}\times \cdots \times p_{{\eta }_t})}{p_{\eta }\times \cdots \times p_{\eta }}=n{\mathbb{P}}_n{\ddot{g}}_{\eta }+n(n-1){\mathbb{U}}_n({\stackrel{.}{g}}_{\eta }\times {\stackrel{.}{g}}_{\eta }).$$ (17)

Rather than in the form Eqs. (14), (15), the Eqs. (10), (11) that define the influence functions can then be written in the form

$${\frac{d}{dt}}_{\mid t=0}\chi \left({\eta }_t\right)=P_{\eta }^n\left({\mathbb{U}}_n({\stackrel{.}{\chi }}_{\eta }+\frac{1}{2}{\ddot{\chi }}_{\eta })\right){\stackrel{.}{s}}_{\eta }^{\left(n\right)},$$ (18)

$${\frac{d^2}{d{t}^2}}_{\mid t=0}\chi \left({\eta }_t\right)=P_{\eta }^n\left({\mathbb{U}}_n({\stackrel{.}{\chi }}_{\eta }+\frac{1}{2}{\ddot{\chi }}_{\eta })\right){\ddot{s}}_{\eta }^{\left(n\right)}.$$ (19)

We conclude that the influence functions are determined by the inner products of the U-statistic ${\mathbb{U}}_n({\stackrel{.}{\chi }}_{\eta }+\frac{1}{2}{\ddot{\chi }}_{\eta })$ in $L_2\left(P_{\eta }^n\right)$ with the score functions ${\stackrel{.}{s}}_{\eta }^{\left(n\right)}$ and ${\ddot{s}}_{\eta }^{\left(n\right)}$. The influence functions that yield a minimal variance are found by projecting this U-statistic onto the closed linear span of these score functions. Thus it is natural to define the latter span as the second order tangent space of the model.

For computation in examples the defining Eq. (11) or (15) of a second order influence function can be tedious. It is usually easier to apply the rule that a second derivative is the derivative of the first derivative. In the present situation this takes the following form (Pfanzagl 1985, 4.3.11): if ${\ddot{\chi }}_{\eta }:{\mathcal{X}}^2\to \mathbb{R}$ is a function such that $x_2\mapsto {\ddot{\chi }}_{\eta }(x_1,x_2)$ is a first order influence function of the parameter $\eta \mapsto {\stackrel{.}{\chi }}_{\eta }\left(x_1\right)$, for every fixed x₁ and a first order influence function ${\stackrel{.}{\chi }}_{\eta }$ (not necessarily degenerate), then ${\ddot{\chi }}_{\eta }$ is a second order influence function.

Lemma 1 Suppose that (P_ηt : t ≥ 0) is a sufficiently smooth submodel and ${\stackrel{.}{\chi }}_{{\eta }_t}:\mathcal{X}\to \mathbb{R}$ and ${\ddot{\chi }}_{{\eta }_t}:{\mathcal{X}}^2\to \mathbb{R}$ are measurable functions that satisfy

$$\begin{array}{cc}\hfill \frac{d}{dt}\chi \left({\eta }_t\right)& =\int {\stackrel{.}{\chi }}_{{\eta }_t}\frac{d}{dt}{p}_{{\eta }_t}d\mu ,(t\ge 0),\hfill \\ \hfill {\frac{d}{dt}}_{\mid t=0}{\stackrel{.}{\chi }}_{{\eta }_t}\left(x_1\right)& =\int {\ddot{\chi }}_{\eta }(x_1,x_2){\stackrel{.}{g}}_{\eta }\left(x_2\right)d{P}_{\eta }\left(x_2\right),(x_1\in \chi ).\hfill \end{array}$$

Then the function ${\ddot{\chi }}_{\eta }$ is a second order influence function, and so is the symmetrization of its orthogonal projection onto the degenerate functions in L₂(P_η × P_η).

Proof By differentiation of the first identity (under the integral) we see that

$$\frac{d^2}{d{t}^2}\chi \left({\eta }_t\right)=\int \frac{d}{dt}{\stackrel{.}{\chi }}_{{\eta }_t}\frac{d}{dt}{p}_{{\eta }_t}d\mu +\int {\stackrel{.}{\chi }}_{{\eta }_t}\frac{d^2}{d{t}^2}{p}_{{\eta }_t}d\mu .$$

We evaluate this at t = 0 and substitute the second identity in the first term on the right to arrive at Eq. (11). The equation remains valid if ${\ddot{\chi }}_{\eta }$ is replaced by its projection and symmetrization.

Just as for first order influence functions there is no guarantee that a second order influence function exists. The difference is that, for the examples we are interested in, nonexistence of a second order influence function is typical. A first indication that this might happen is that the informal conclusion reached in the preceding that the quadratic estimator (8) will have an error of the order $O_P\left(n^{-1∕2}\right)+O_P\left({\Vert \eta -\widehat{\eta }\Vert }^3\right)$ is overly optimistic. In comparison to the linear estimator (3), this estimator would have reduced the dependence on the preliminary estimator $\widehat{\eta }$ from $O_P\left({\Vert \eta -\widehat{\eta }\Vert }^2\right)$ to $O_P\left({\Vert \eta -\widehat{\eta }\Vert }^3\right)$, apparently without a serious penalty on the variance of the estimator. In our examples this does not occur, simply because a second order influence function does not exist.

As for the first order influence function, the nonexistence of the second order influence function may be caused by a lack of invertibility of the map η → p_η or by failure of a von Mises type representation. The invertibility is again necessary, because we need representation of the derivatives of $\eta \mapsto \chi \left(\eta \right)$ in terms of the distribution P_η of the observation. This is similar as in the linear situation. The second cause for failure of representation also arose in the linear situation, but appears to arise in a much more serious way at the second order. Whereas a continuous, linear map $B:L_2\left(P_{\eta }\right)\to \mathbb{R}$ is always representable as an inner product $B\left(g\right)=P_{h}g{\stackrel{.}{\chi }}_{\eta }$ for some function ${\stackrel{.}{\chi }}_{\eta }$, a continuous, bilinear map $B:L_2\left(P_{\eta }\right)\times L_2\left(P_{\eta }\right)\to \mathbb{R}$ is not necessarily representable through a measurable function ${\ddot{\chi }}_{\eta }:{\mathcal{X}}^2\to \mathbb{R}$, in the form

$$B(g,h)=\iint g\left(x_1\right){\ddot{\chi }}_{\eta }(x_1,x_2)h\left(x_2\right)d{P}_{\eta }\left(x_1\right)d{P}_{\eta }\left(x_2\right).$$ (20)

It can be shown that a continuous, bilinear map can always be written in the form B(g, h) = ∫ g(Ah) dP_η for a continuous, linear operator A: L₂(P_η) → L₂(P_η), but the latter operator is not necessarily a kernel operator in that $Ah\left(x_1\right)=\int {\ddot{\chi }}_{\eta }(x_1,x_2)h\left(x_2\right)d{P}_{\eta }\left(x_2\right)$ for some kernel ${\ddot{\chi }}_{\eta }$. The latter representation is necessary for the von-Mises representation (7) of the second derivative.

Failure of existence of ${\ddot{\chi }}_{\eta }$ does not mean that the idea to use a quadratic expansion for improved estimation is not fruitful. Failure does mean that we cannot construct the estimator (8) and the estimation rate $O_P\left(n^{-1∕2}\right)+O_P\left({\Vert \widehat{\eta }-\eta \Vert }^3\right)$ may not be attainable. However, we may return to Eq. (6) and try and estimate the quadratic term as well as possible, and still improve on the linear estimator. A key observation is that a bilinear map on a finite-dimensional subspace L × L ⊂ L₂(P_η) × L₂(P_η) is always representable by a kernel.

Lemma 2 If L ⊂ L₂(P_η) is a finite-dimensional subspace and $B:L\times L\to \mathbb{R}$ is continuous and bilinear, then there exists a function ${\ddot{\chi }}_{\eta }\in L_2(P_{\eta }\times P_{\eta })$ such that (20) holds for every g, h ∈ L.

Proof For an arbitrary orthonormal basis e₁, . . . , e_k of L we can express an element g ∈ L as $g={\sum }_{i=1}^k{\langle g,e_i\rangle }_{\eta }{e}_i$, for 〈·, ·〉_η the inner product of L₂(P_η). By bilinearity

$$\begin{array}{cc}\hfill B(g,h)& =\sum _{i=1}^k\sum _{j=1}^k{\langle g,e_i\rangle }_{\eta }{\langle h,e_j\rangle }_{\eta }B(e_i,e_j)\hfill \\ \hfill & =\iint g\left(x_1\right)h\left(x_2\right)\sum _{i=1}^k\sum _{j=1}^{k}B(e_i,e_j)e_i\left(x_1\right)e_j\left(x_2\right)d{P}_{\eta }\left(x_1\right)d{P}_{\eta }\left(x_2\right).\hfill \end{array}$$

Thus the function $(x_1,x_2)\mapsto {\sum }_{i=1}^k{\sum }_{j=1}^{k}B(e_i,e_j)e_i\left(x_1\right)e_j\left(x_2\right)$ is a kernel for the map B.

If the invertibility $\eta \mapsto p_{\eta }$ can be resolved, we can therefore always represent the second derivative in Eq. (6) at differences $\eta -\widehat{\eta }$ within a given finite-dimensional linear space. The estimator (8) based on the resulting “partial second order influence function” then will add a representation error to the remainder $O_P\left({\Vert \widehat{\eta }-\eta \Vert }^3\right)$. This representation error can be made arbitrarily small by choosing the finite-dimensional linear space sufficiently large. However, the corresponding partial influence functions depend on the approximating linear spaces, the estimator now having the form

$$\chi \left(\widehat{\eta }\right)+{\mathbb{P}}_n{\stackrel{.}{\chi }}_{\widehat{\eta }}+\frac{1}{2}{\mathbb{U}}_n{\ddot{\chi }}_{L,\widehat{\eta }},$$ (21)

where ${\ddot{\chi }}_{L,\eta }$ is a partial second order influence function based on an approximating space L. To obtain a good estimator we must balance the representation error, remainder $O\left({\Vert \widehat{\eta }-\eta \Vert }^3\right)$, and the variance of the estimator. In an asymptotic framework we let the approximating space L increase to the full space when n → ∞. We shall see that this may cause the variance of ${\mathbb{U}}_n{\ddot{\chi }}_{L,\widehat{\eta }}$ to dominate the variance of the linear term ${\mathbb{P}}_n{\stackrel{.}{\chi }}_{\widehat{\eta }}$ and the overall variance may be bigger than O(1/n). However, by proper balancing of the three terms we do never worse than the linear estimator (3), and we gain over it if the parameter set H is large.

4 Estimating the square of a density

Consider the problem of estimating the functional χ(p) = ∫ p² dμ based on a random sample of size n from the density p. This problem was discussed among others in Bickel and Ritov (1988) and Laurent (1996), Laurent (1997). We shall rederive the estimator by Laurent (1996) through our general approach.

As the underlying model $\mathcal{P}$ we use a set of densities that is restricted only qualitatively, for instance a Hölder space of functions on the unit square in ${\mathbb{R}}^d$. We parameterize this model by the density itself, which we denote by p (hence p_η = η = p). The tangent space of the model can then be taken equal to the set of all mean zero functions ${\stackrel{.}{g}}_p:\mathcal{X}\to \mathbb{R}$ in L₂(P), and the first order influence function takes the form

$${\stackrel{.}{\chi }}_p\left(x\right)=2(p\left(x\right)-\chi \left(p\right)).$$ (22)

To see this, it suffices to note that this function is mean-zero (i.e., degenerate) and satisfies

$${\frac{d}{dt}}_{\mid t=0}\chi \left(p_t\right)=\int 2p_t{\stackrel{.}{p}}_{t}d{\mu }_{\mid t=0}=P2p{\stackrel{.}{g}}_p=P{\stackrel{.}{\chi }}_p{\stackrel{.}{g}}_p,$$

for any sufficiently regular path $t\mapsto p_t$ with p₀ = p and score function ${\stackrel{.}{g}}_p={\stackrel{.}{p}}_0∕p_0$ at t = 0. This first order influence function exists without making assumptions on p or $\mathcal{P}$.

We compute a second order influence function as the influence function of the functional $p\mapsto {\stackrel{‒}{\chi }}_p\left(x_1\right)=2p\left(x_1\right)$, which is the first order influence function up to centering. This entails point evaluation at a fixed point x₁, which, unfortunately, is not a differentiable functional in the sense of possessing an influence function. For any sufficiently regular path $t\mapsto p_t$ with score function ${\stackrel{.}{g}}_p$,

$${\frac{d}{dt}}_{\mid t=0}{p}_t\left(x_1\right)={\stackrel{.}{g}}_p\left(x_1\right)p\left(x_1\right).$$

Existence of an influence function of the functional $p\mapsto p\left(x_1\right)$ would require the map $g\mapsto g\left(x_1\right)p\left(x_1\right)$ to be representable as an inner product in L₂(P) on the tangent space. Such a representation is not possible (unless p has finite support), because the map is not continuous relative to the L₂(P)-norm.

Thus we content ourselves with partial representation of the second derivative. To this aim it is useful to think of the point evaluation map as integrating versus the Dirac measure (at x₁). Full representation of the functional $g\mapsto g\left(x_1\right)p\left(x_1\right)$ would be possible if there existed a function $\Pi :\mathcal{X}\times \mathcal{X}\to \mathbb{R}$ such that,

$$g\left(x_1\right)p\left(x_1\right)=\int \Pi (x_1,x_2)g\left(x_2\right)d\mu \left(x_2\right).$$ (23)

If this were true for every function g, then the measure $B\mapsto {\int }_B\Pi (x_1,x_2)d\mu \left(x_2\right)$ would, for each fixed x₁, act as a Dirac measure at x₁. In other words, the desired, but not existing, function Π would be a “Dirac measure” on the diagonal of $\mathcal{X}\times \mathcal{X}$. Our second best is a function for which Eq. (23) is true, if not for all, then for a large collection of g. The kernel Π of a projection operator Π: L₂(μ) → L₂(μ) onto a (large) subspace is a candidate, because it satisfies the display whenever gp is in the subspace: if Πf (x₁) = ∫ Π(x₁, x₂) f (x₂) dμ(x₂), then the equation gp = Π(gp), which is valid for every gp in the range of the projection, gives the preceding display.

Lemma 3 An orthogonal projection Π: L₂(μ) → L ⊂ L₂(μ) onto a finite-dimensional subspace L can be represented as Πf (x₁) = ∫ Π(x₁, x₂) f (x₂) dμ(x₂) for the kernel function $\Pi (x_1,x_2)={\sum }_{i=1}^k{e}_i\left(x_1\right)e_i\left(x_2\right)$ and e₁, . . . , e_k an orthonormal basis of L. This kernel satisfies ∫ Π² dμ × μ = k.

Proof We have $\Pi f\left(x_1\right)={\sum }_{i=1}^k{\langle f,e_i\rangle }_{\mu }{e}_i\left(x_1\right)$ for ${\langle f,e_i\rangle }_{\mu }=\int f{e}_{i}d\mu$. The representation follows by exchanging the order of summation and integration.

The square kernel is ${\sum }_{i=1}^k{\sum }_{j=1}^k{e}_i\left(x_1\right)e_j\left(x_1\right)e_i\left(x_2\right)e_j\left(x_2\right)$. By the orthonormality of the basis (e_i) the (double) integral of the off-diagonal terms (i ≠ j) vanishes and the double integral of the diagonal terms is equal to 1. Thus the double integral is k.

We also arrive at a projection operator from the formula ${\chi }_p^{\prime \prime }(g,h)=2\int gh{p}^{2}d\mu$ for the second derivative of χ. We can write this in the form ${\chi }_p^{\prime \prime }(g,h)=2\int g\left(A_{p}h\right)dP$ for the operator A_p: L₂(P) → L₂(P) given by A_ph = hp. The operator A_p is not of kernel form, but we can approximate it by ΠA_p, leading to the approximation 2 ∫ g(ΠA_ph) dP = 2 ∫ gp (Π(hp)) dμ for ${\chi }_p^{\prime \prime }(g,h)$.

For a given orthonormal basis e₁, e₂, . . . of L₂(μ) we take the kernel Π(x₁, x₂) of the projection onto the span of the first k elements, given by Lemma 3, as a “partial” influence function of the functional $p\mapsto p\left(x_1\right)$, and $x_2\mapsto 2\Pi (x_1,x_2)$ as a “partial” influence function of the functional $p\mapsto {\stackrel{‒}{\chi }}_p\left(x_1\right)$. The projection of this function onto the degenerate functions is

$${\ddot{\chi }}_p(x_1,x_2)=2\Pi (x_1,x_2)-2\Pi p\left(x_1\right)-2\Pi p\left(x_2\right)+2\int {\left(\Pi p\right)}^{2}d\mu .$$ (24)

The quadratic estimator (8), given the initial estimator $\widehat{p}$, takes the form

$$\begin{array}{cc}\hfill \chi \left(\widehat{p}\right)+{\mathbb{P}}_n{\stackrel{.}{\chi }}_{\widehat{p}}+{\mathbb{U}}_n{\ddot{\chi }}_{\widehat{p}}& ={\mathbb{U}}_n\Pi +{\mathbb{U}}_n\left((I-\Pi )\widehat{p}\right)\hfill \\ \hfill & ={\mathbb{U}}_n\left(\sum _{i=1}^k{e}_i\times e_i\right)+{\mathbb{U}}_n\left(\sum _{i=k+1}^{\infty }{\widehat{\theta }}_i{e}_i\right),\hfill \end{array}$$

for ${\widehat{\theta }}_i=\int \widehat{p}{e}_{i}d\mu$ the Fourier coefficients of $\widehat{p}$. If we choose the initial estimator to take values in the range of Π, then ${\widehat{\theta }}_i=0$ for i > k and the second term vanishes. The resulting estimator reduces to the estimator considered by Laurent (1996, 1997), who showed that the estimator is minimax if p is a-priori known to belong to a multiple of the unit ball in the Hölder space C^β[0, 1] of regularity β and (e_i) is a basis suited to this a-priori model. In fact, mean and variance of the estimator satisfy, with θ_i the Fourier coefficients of p,

$$\begin{array}{cc}\hfill {\mathrm{E}}_p{\mathbb{U}}_n\left(\sum _{i=1}^k{e}_i\times e_i\right)& ={\mathrm{E}}_p\sum _{i=1}^k{e}_i\times e_i=\sum _{i=1}^k{\theta }_i^2,\hfill \\ \hfill {\text{var}}_p{\mathbb{U}}_n\left(\sum _{i=1}^k{e}_i\times e_i\right)& \le \frac{4}{n}P{\left(\Pi p\right)}^2+\frac{2k}{n(n-1)}.\hfill \end{array}$$

The bound on the variance follows from Lemma 6. If it is a-priori known that ${\sum }_{i=1}^{\infty }{\theta }_i^2{i}^{2\beta }<\infty$, then the bias is bounded by ${\sum }_{i>k}{\theta }_i^2\le k^{-2\beta }$. The square bias is balanced against the variance if k is chosen of the order k_n = n^2/(4β+1) if β ≤ 1/4 and k_n = n if β ≥ 1/4. The resulting rate of convergence is n^{–2β/(4β+1} if β ≤ 1/4 and n^–1/2 if β ≥ 1/4. In Robins et al. (2007) it is shown that it is also asymptotically normal.

5 Estimating the mean response in missing data models

Suppose that a typical observation is distributed as X = (Y A, A, Z) for Y and A taking values in the two-point set {0, 1} and conditionally independent given Z. We think of Y as a response variable, which is observed only if the indicator A takes the value 1. The covariate Z is chosen such that it contains all information on the dependence between response and missingness indicator (missing at random). Alternatively, we think of Y as a counterfactual outcome if a treatment were given (A = 1) and estimate (half) the treatment effect under the assumption of “no unmeasured confounders”. Both applications may require that Z is high-dimensional (e.g., of dimension 10), and there is typically insufficient a-priori information to model the dependence of A and Y on Z.

The model can be parameterized by the marginal density f of Z (relative to some dominating measure ν) and the probabilities b(z) = P(Y = 1|Z = z) and a(z)^–1 = P A = 1|Z = z). (We use a for the inverse probability, because this simplifies later formulas.) Thus the density p_η of an observation X is described by the triple η = (a, b, f).

We wish to estimate the mean response EY, i.e., the parameter

$$\chi \left(\eta \right)=\int bfd\nu .$$

Estimators that are n^–1/2-consistent and asymptotically efficient in the semiparametric sense have been constructed using a variety of methods (e.g., Robins and Rotnitzky 1992; van der Laan and Robins 2003; van der Vaart 1998), but only if a or b, or both, parameters are restricted to sufficiently small regularity classes. For instance, if the covariate ranges over a compact, convex subset $\mathcal{Z}$ of ${\mathbb{R}}^d$, then the mentioned papers provide n^–1/2-consistent estimators under the assumption that a and b belong to Hölder classes C^α(Z) and C^β(Z) with α and β large enough that

$$\frac{\alpha }{2\alpha +d}+\frac{\beta }{2\beta +d}\ge \frac{1}{2}.$$ (25)

For moderate to large dimensions d this is a restrictive requirement. We shall show that a quadratic estimator of the type (8) can attain a n^–1/2-rate in a bigger model and obtains a strictly better rate than the usual estimators if the n^–1/2-rate is not obtainable.

Prelimary estimators

The parameter 1/a(z) = E(A|Z = z) is the regression of A on Z and hence can be estimated by any nonparametric regression estimator, such as a kernel or a truncated series estimator. Similarly, the function b(z) = P(Y = 1|Z = z, A = 1) is the regression of the observed Y on Z and can be estimated by nonparametric regression based on the subsample (Y_i : A_i = 1) on the corresponding Z_i. We shall see below that the parameter f/a is more fundamental than the parameter f. By Bayes’ rule (f/a)(z) = P(A = 1|Z = z) f (z) is P(A = 1) times the conditional density of Z given A = 1. Therefore, we may estimate f/a by a nonparametric density estimator based on the subsample (Z_i : A_i = 1) times $n^{-1}{\sum }_{i=1}^n{A}_i$.

Tangent space and first order influence function

The one-dimensional submodels $t\mapsto p_{{\eta }_t}$ induced by paths of the form a_t = a + tα, b_t = b + tβ, and f_t = f (1 + tϕ) for given directions α, β and ϕ yield scores $B_{\eta }(\alpha ,\beta ,\varphi )=B_{\eta }^a\alpha +B_{\eta }^b\beta +B_{\eta }^f\varphi$, for $B_{\eta }^a$, $B_{\eta }^b$, $B_{\eta }^f$ the score operators for the three parameters, given by

$$\begin{array}{cc}\hfill B_{\eta }^a\alpha \left(X\right)& =-\frac{Aa\left(Z\right)-1}{a\left(Z\right)(a-1)\left(Z\right)}\alpha \left(Z\right),a-\text{score},\hfill \\ \hfill B_{\eta }^b\beta \left(X\right)& =\frac{A(Y-b\left(Z\right))}{b\left(Z\right)(1-b)\left(Z\right)}\beta \left(Z\right),b-\text{score},\hfill \\ \hfill B_{\eta }^f\varphi \left(X\right)& =\varphi \left(Z\right),f-\text{score}.\hfill \end{array}$$

The first-order influence function is well known to take the form

$${\stackrel{.}{\chi }}_{\eta }\left(X\right)=Aa\left(Z\right)(Y-b\left(Z\right))+b\left(Z\right)-\chi \left(\eta \right).$$ (26)

To see this it must be verified that this function satisfies, for every path $t\mapsto p_{{\eta }_t}$ as described previously,

$${\frac{d}{dt}}_{\mid t=0}\chi \left({\eta }_t\right)={\mathrm{E}}_{\eta }{\stackrel{.}{\chi }}_{\eta }\left(X\right)B_{\eta }(\alpha ,\beta ,\varphi )\left(X\right).$$

For the paths a_t = a + tα, b_t = b + tβ and f_t = f(1 + tϕ) the left side of this equation is ∫ (β + bϕ) f dν. The right side can easily be evaluated to be the same, where it may be noted that conditional expectations of functions of Y and A given Z factorize, with E(Y – b(Z)|Z) = E(Aa(Z) – 1|Z) = 0 and E _(Y – b(Z)²|Z = b(1 – b)(Z).

The advantage of choosing a an inverse probability is clear from the form of the (random part of the) influence function, which is a bilinear function in (a, b). The error of the corresponding von-Mises representation can be computed to be, for a given initial estimator $\widehat{\eta }=(\widehat{a},\widehat{b},\widehat{f})$,

$$\chi \left(\widehat{\eta }\right)-\chi \left(\eta \right)+P_{\eta }{\stackrel{.}{\chi }}_{\widehat{\eta }}=-\int (\widehat{a}-a)(\widehat{b}-b)\frac{f}{a}d\nu .$$ (27)

This is quadratic in the errors of the initial estimators. Actually, the form of the bias term is special in that square estimation errors of the two initial estimators $\widehat{a}$ and $\widehat{b}$ do no arise, but only the product of their errors. This property, termed “double robustness” in Rotnitzky and Robins (1995), Robins and Rotnitzky (2001), van der Laan and Robins (2003), makes that it suffices that one of the two parameters is estimated well. A prior assumption that the parameters a and b are α and β regular, respectively, would allow estimation errors with rates n^{–α/(2α+d)} and n^{–β/(2β+d)}. If the product of these rates is o(n^–1/2), then the bias term is negligible, and the linear estimator (3) attains a rate n^–1/2. This leads to the condition (25). If this condition fails, then the “bias” (27) is greater than O(n^–1/2). The linear estimator then does not balance bias and variance and is suboptimal.

It may be noted that the marginal density f does not enter the first order influence function. Even though the functional depends on f, a rate on the initial estimator of this function is not needed for the construction of the first order estimator. This will be different at second order.

Quadratic estimator

We proceed to the computation of a second order influence function using Lemma 1, by searching a function ${\ddot{\chi }}_{\eta }:{\mathcal{X}}^2\to \mathbb{R}$ such that, for every x₁ = (y₁a₁, a₁, z₁), and all directions α, β, ϕ,

$$\begin{array}{cc}\hfill a_1(y_1-b\left(z_1\right))\alpha \left(z_1\right)-(a_{1}a\left(z_1\right)-1)\beta \left(z_1\right)& ={\frac{d}{dt}}_{\mid t=0}[{\stackrel{.}{\chi }}_{{\eta }_t}\left(x_1\right)+\chi \left({\eta }_t\right)]\hfill \\ \hfill & ={\mathrm{E}}_{\eta }{\ddot{\chi }}_{\eta }(x_1,X_2)B_{\eta }(\alpha ,\beta ,\varphi )\left(X_2\right).\hfill \end{array}$$ (28)

Here the expectation is relative to the variable X₂ only. Let $K_{\eta }:{\mathcal{Z}}^2\to \mathbb{R}$ be the kernel of an operator K_η: L₂(f) → L₂(f) (i.e., K_ηg(x₁) = ∫ K(x₁, x₂)g(x₂) f (x₂) dμ(x₂)), and define

$$\begin{array}{cc}\hfill {\ddot{\chi }}_{\eta }(X_1,X_2)=& -A_1(Y_1-b\left(Z_1\right))a\left(Z_2\right)(A_{2}a\left(Z_2\right)-1)K_{\eta }(Z_1,Z_2)\hfill \\ \hfill & -(A_{1}a\left(Z_1\right)-1)a\left(Z_2\right)A_2(Y_2-b\left(Z_2\right))K_{\eta }(Z_1,Z_2).\hfill \end{array}$$ (29)

For this choice the right side of Eq. (28) can be seen to reduce to

$$a_1(y_1-b\left(z_1\right))K_{\eta }\alpha \left(z_1\right)-(a_{1}a\left(z_1\right)-1)K_{\eta }\beta \left(z_1\right).$$

(Note that var (Aa(Z)|Z) = a(Z) – 1.) Thus the choice Eq. (29) of ${\ddot{\chi }}_{\eta }$ satisfies Eq. (28) for every (α,β,ϕ) such that K_ηα = α and K_ηβ = β. Were K_η equal to the identity operator, then Eq. (28) would be satisfied for every (α, β, ϕ), and an exact second order influence function would exist. Unfortunately, the identity operator is not given by a kernel. As in Sect. 4 we have to be satisfied with an influence function that gives partial representation.

To ensure that ${\ddot{\chi }}_{\eta }$ is symmetric we choose K_η(z₁, z₂) = Π_η(z₁, z₂)/a(z₂) for Π_η a symmetric function. Specifically, we choose Π_η the kernel of an orthogonal projection Π_η: L₂(f/a) → L₂(f/a) onto a space L. The corresponding operators then (trivially) satisfy K_ηg = Π_ηg for every g ∈ L₂(f/a), and hence K_η will approximate the identity if L is large. The function (29) that results from this choice can be seen to be both symmetric and degenerate, and hence is a candidate “approximate” influence function. If S₂ symmetrizes a function of two variables (i.e., 2 S₂ g(X₁, X₂) = g(X₁, X₂) + g(X₂, X₁)), then this influence function can be written as

$${\ddot{\chi }}_{\eta }(X_1,X_2)=-{\mathrm{S}}_2\left[A_1(Y_1-b\left(Z_1\right)){\Pi }_{\eta }(Z_1,Z_2)(A_{2}a\left(Z_2\right)-1)\right].$$ (30)

For an initial estimator $\widehat{\eta }$ based on independent observations we now construct the estimator (8).

Let $\widehat{\mathrm{E}}$ and $\mathrm{v}\widehat{\mathrm{a}}\mathrm{r}$ denote conditional expectations given the observations used to construct $\widehat{\eta }$, and let ∥ · ∥₂ be the norm of L₂(f/a). Assume that the true functions a, f and the estimators $\widehat{a}$, $\widehat{f}$ are bounded away from 0 and ∞.

Theorem 1 The estimator ${\widehat{\chi }}_n=\chi \left(\widehat{\eta }\right)+{\mathbb{P}}_n{\stackrel{.}{\chi }}_{\widehat{\eta }}+\frac{1}{2}{\mathbb{U}}_n{\ddot{\chi }}_{\widehat{\eta }}$ with (approximate) influence functions ${\stackrel{.}{\chi }}_{\eta }$ and ${\ddot{\chi }}_{\eta }$ defined by (26) and (30) with Πη the kernel of an orthogonal projection in L₂(f/a) onto a k-dimensional linear subspace satisfies

$$\begin{array}{cc}\hfill {\widehat{\mathrm{E}}}_{\eta }{\widehat{\chi }}_n-\chi \left(\eta \right)=& O_P\left({\Vert \widehat{a}-a\Vert }_2{\Vert \widehat{b}-b\Vert }_2{\Vert \frac{\widehat{f}}{\widehat{a}}-\frac{f}{a}\Vert }_2\right)\hfill \\ \hfill & +O_P\left({\Vert a-{\Pi }_{\eta }a\Vert }_2{\Vert b-{\Pi }_{\eta }b\Vert }_2\right),\hfill \\ \hfill \mathrm{v}\widehat{\mathrm{a}}{\mathrm{r}}_{\eta }{\widehat{\chi }}_n=& O_P\left(\frac{k}{n^2}\vee \frac{1}{n}\right).\hfill \end{array}$$

Proof From Eqs. (27) and (30) we have

$$\begin{array}{cc}\hfill \widehat{\mathrm{E}}{\widehat{\chi }}_n-\chi \left(\eta \right)=& -\int (\widehat{a}-a)(\widehat{b}-b)\frac{f}{a}d\nu \hfill \\ \hfill & -\widehat{\mathrm{E}}{A}_1\left(Y_1-\widehat{b}\left(Z_1\right)\right){\Pi }_{\widehat{\eta }}(Z_1,Z_2)(A_2\widehat{a}\left(Z_2\right)-1){\Pi }_{\widehat{\eta }}(Z_1,Z_2)\hfill \\ \hfill =& -\int (\widehat{a}-a)(\widehat{b}-b)\frac{f}{a}d\nu +\iint \left((\widehat{a}-a)\times (\widehat{b}-b)\right)\hfill \\ \hfill & \times {\Pi }_{\widehat{\eta }}\left(\frac{f}{a}\times \frac{f}{a}\right)d\nu \times \nu .\hfill \end{array}$$

The double integral on the far right with ${\Pi }_{\widehat{\eta }}$ replaced by Π_η can be written as the single integral $\int (\widehat{a}-a){\Pi }_{\eta }(\widehat{b}-b)(f∕a)d\nu$. Added to the first integral on the right this gives

$$-\int (\widehat{a}-a)(I-{\Pi }_{\eta })(\widehat{b}-b)(f∕a)d\nu .$$

By the Cuachy-Schwarz inequality this is bounded in absolute value by the second term in the upper bound for the bias.

Replacement of ${\Pi }_{\widehat{\eta }}$ by Π_η in the double integral gives a difference

$$\begin{array}{cc}\hfill & \iint \left((\widehat{a}-a)\times (\widehat{b}-b)\right)({\Pi }_{\widehat{\eta }}-{\Pi }_{\eta })\left(\frac{f}{a}\times \frac{f}{a}\right)d\nu \times \nu \hfill \\ \hfill & =\int (\widehat{a}-a)\left({\Pi }_{\widehat{\eta }}\left((\widehat{b}-b)\frac{f∕a}{\widehat{f}∕\widehat{a}}\right)-{\Pi }_{\eta }(\widehat{b}-b)\right)\frac{f}{a}d\nu .\hfill \end{array}$$

By the Cauchy-Schwarz inequality the absolute value of this is bounded above by

$${\Vert \widehat{a}-a\Vert }_2{\Vert ({\Pi }_{\widehat{\eta }}\circ M_{\widehat{w}}-{\Pi }_{\eta })(\widehat{b}-b)\Vert }_{2,\widehat{\nu }}{\Vert \widehat{w}\Vert }_{\infty }.$$

Here $M_{\widehat{w}}$ is multiplication by the function $\widehat{w}=(f∕a)∕(\widehat{f}∕\widehat{a})(\text{defined by}{M}_{\widehat{w}}g=g\widehat{w})$, and ${\Vert \cdot \Vert }_{2,\widehat{\nu }}$ is the $L_2\left(\widehat{\nu }\right)\text{-norm}$ for the measure $\widehat{\nu }$ defined by $d\widehat{\nu }=(\widehat{f}∕\widehat{a})d\nu$. Considering ${\Pi }_{\widehat{\eta }}$ as the projection in $L_2\left(\widehat{\nu }\right)$ with weight 1, and Π_η as the weighted projection in $L_2\left(\widehat{\nu }\right)$ with weight function $\widehat{w}$, we can apply Lemma 4 to the middle term and conclude that this is bounded in absolute value by ${\Vert {\Pi }_{\eta }\Vert }_{2,\widehat{\nu }}\mid {\Vert \widehat{w}-1\Vert }_{\infty }{\Vert \widehat{b}-b\Vert }_{2,\widehat{\nu }}$. Because we assume that the functions f/a and $\widehat{f}∕\widehat{a}$ are bounded away from zero and infinity, this can be seen to yield the first term in the upper bound on the bias.

The function ${\stackrel{.}{\chi }}_{\widehat{\eta }}$ is uniformly bounded and hence the (conditional) variance of ${\mathbb{P}}_n{\stackrel{.}{\chi }}_{\widehat{\eta }}$ is of the order O_P(1/n). Thus for the variance bound it suffices to consider the (conditional) variance of ${\mathbb{U}}_n{\ddot{\chi }}_{\widehat{\eta }}$. In view of Lemma 6 this is bounded above by

$$\frac{4}{n}{\mathrm{E}}_{\eta }{\mathrm{E}}_{\eta }{({\ddot{\chi }}_{\widehat{\eta }}(X_1,X_2)\mid X_1)}^2+\frac{2}{n(n-1)}{\mathrm{E}}_{\eta }{\ddot{\chi }}_{\widehat{\eta }}^2(X_1,X_2).$$

The variables $A\left(Y-\widehat{b}\left(Z\right)\right)$ and $(A\widehat{a}\left(Z\right)-1)$ are uniformly bounded. Hence the last term on the right is bounded above by a multiple of $n^{-2}\int \int {\Pi }_{\widehat{\eta }}^2(f∕a\times f∕a)d\nu \times \nu$, which is bounded by ${\Vert \widehat{w}\Vert }_{\infty }^{2}k∕n^2$, by Lemma 3. The first order term is of the order O(1/n). To see this we first note that

$$\begin{array}{cc}\hfill {\mathrm{E}}_{\eta }({\ddot{\chi }}_{\widehat{\eta }}(X_1,X_2)\mid X_1)=& -A_1\left(Y_1-\widehat{b}\left(Z_1\right)\right){\Pi }_{\widehat{\eta }}\left((\widehat{a}-a)\widehat{w}\right)\left(X_1\right)\hfill \\ \hfill & -(A_1\widehat{a}\left(Z_1\right)-1){\Pi }_{\widehat{\eta }}\left((\widehat{b}-b)\widehat{w}\right)\left(X_1\right).\hfill \end{array}$$

Here the variables $A_1\left(Y_1-\widehat{b}\left(Z_1\right)\right)$ and $(A_1\widehat{a}\left(Z_1\right)-1)$ are uniformly bounded, and the second moment of ${\Pi }_{\widehat{\eta }}g$ is bounded by ${\Vert \widehat{w}\Vert }_{\infty }$ times the second moment of g in $L_2\left(\widehat{\nu }\right)$, for every g.

Conclusion

Assume that the parameters a, b and f/a are known to be “regular” of degrees α, β and ϕ, respectively, in the sense that there exists a sequence of k-dimensional linear spaces L_k such that, for some constant C,

$${\Vert a-L_k\Vert }_2\le C{\left(\frac{1}{k}\right)}^{\alpha ∕d},{\Vert b-L_k\Vert }_2\le C{\left(\frac{1}{k}\right)}^{\beta ∕d},{\Vert \frac{f}{a}-L_k\Vert }_2\le C{\left(\frac{1}{k}\right)}^{\beta ∕d}.$$

This is true, for instance, if the functions a, b and f/a are defined on a compact, convex domain in ${\mathbb{R}}^d$ and are known to belong to Hölder (or Besov) spaces of functions of smoothness α, β and ϕ. The approximation is then valid even with the uniform norm on the left side, where the spaces L_k can be taken to be generated by polynomials, splines or wavelets.

In this case there also exist estimators $\widehat{a}$ and $\widehat{b}$ and $\widehat{f}∕\widehat{a}$ that achieve convergence rates n^{–α/(2α+d)}, n^{–β/(2β+d)} and n^{–ϕ/(2ϕ+d)}, respectively, uniformly over these a-priori models. Then the estimator ${\widehat{\chi }}_n$ of Theorem 1 attains the square rate of convergence

$$\frac{k}{n^2}\vee \frac{1}{n}\vee {\left(\frac{1}{n}\right)}^{2\alpha ∕(2\alpha +d)+2\beta ∕(2\beta +d)+2\varphi ∕(2\varphi +d)}\vee {\left(\frac{1}{k}\right)}^{(2\alpha +2\beta )∕d}.$$

The optimal value of k balances the first and fourth terms and is of the order k ~ n^{2d/(d+2α+2β)}. The resulting rate is n^–γ for

$$\gamma =\left(\frac{1}{2}\right)\wedge \left(\frac{\alpha }{2\alpha +d}+\frac{\beta }{2\beta +d}+\frac{\varphi }{2\varphi +d}\right)\wedge \left(\frac{2\alpha +2\beta }{d+2\alpha +2\beta }\right).$$

This reduces to the rate n^–1/2 under condition (25), but also if (α + β)/2 ≥ d/4 and ϕ is sufficiently large:

$$\frac{\varphi }{2\varphi +d}\ge \frac{1}{2}-\frac{\alpha }{2\alpha +d}-\frac{\beta }{2\beta +d}.$$

(In this case we can also choose k = n independent of α and β.) In case the rate n^–γ is slower than n^–1/2, then it is still better than the rate n^{–α/(2α+d)–β/(2β+d)} obtained by the linear estimator (3).

Thus the quadratic estimator outperforms the linear estimator.

6 Technical results

Let L be a given closed subspace of $L_2(\mathcal{X},\mathcal{A},\mu )$ and $w:\mathcal{X}\to \mathbb{R}$ a bounded, measurable function. Define operators $\Pi ,{\Pi }_w:L_2\left(\mu \right)\to L_2\left(\mu \right)$ by

$$\begin{array}{cc}\hfill {\Pi }_g& =\underset{l\in L}{\text{argmin}}\int {(g-l)}^{2}d\mu ,\hfill \\ \hfill {\Pi }_{w}g& =\underset{l\in L}{\text{argmin}}\int {(g-l)}^{2}wd\mu .\hfill \end{array}$$

Thus Π is the ordinary orthogonal projection on the space L, and Π_w is a weighted projection. The projections can be characterized by the orthogonality relationships ∫ (g – Πg)l dμ = 0 and ∫ (g – Πg)l w dμ = 0, for every l ∈ L.

Lemma 4 Let Π_w and Π be theweighted projections onto a fixed subspace L of L₂(μ) relative to the weight functions w and 1, respectively, and let M_w be multiplication by the function w. Then ${\Vert {\Pi }_w-\Pi M_w\Vert }_2\le {\Vert {\Pi }_w\Vert }_2{\Vert w-1\Vert }_{\infty }$.

Proof The orthogonality relationships for the projections Π and Π_w imply that, for every l ∈ L and g,

$$\int \Pi \left(wg\right)ld\mu =\int wgld\mu =\int w\left({\Pi }_{w}g\right)ld\mu .$$

Because Π_wg – (wg) is contained in L,

$$\begin{array}{cc}\hfill {\Vert {\Pi }_{w}g-\Pi \left(wg\right)\Vert }_2^2& =\int ({\Pi }_{w}g-\Pi \left(wg\right))({\Pi }_{w}g-\Pi \left(wg\right))d\mu ,\hfill \\ \hfill & =\int ({\Pi }_{w}g-\Pi \left(wg\right))({\Pi }_{w}g-\left({\Pi }_{w}g\right)w)d\mu .\hfill \end{array}$$

An application of the Cauchy-Schwarz inequality and next cancellation of one factor ${\Vert {\Pi }_{w}g-\Pi \left(wg\right)\Vert }_2$ gives that ${\Vert {\Pi }_{w}g-\Pi \left(wg\right)\Vert }_2\le {\Vert \left({\Pi }_{w}g\right)(1-w)\Vert }_2$. The right side is bounded above by ${\Vert {\Pi }_w\Vert }_2{\Vert g\Vert }_2{\Vert 1-w\Vert }_{\infty }$.

Lemma 5 For degenerate, symmetric functions $f,g:{\mathcal{X}}^2\to \mathbb{R}$ we have $P^n{\mathbb{U}}_{n}f=0$ and

$$P^n\left({\mathbb{U}}_{n}f\right)\left({\mathbb{U}}_{n}g\right)=\frac{1}{\left(\begin{array}{c}\hfill n\hfill \\ \hfill 2\hfill \end{array}\right)}{P}^{2}fg.$$

Lemma 6 For any measurable function $f:{\mathcal{X}}^2\to \mathbb{R}$, and f₁(x₁) = ∫ f(x₁, x₂) dP(x₂),

$$\text{var}{\mathbb{U}}_{n}f\le \frac{4}{n}P{f}_1^2+\frac{2}{n(n-1)}P{f}^2.$$

Proof The first lemma follows by writing the square sum ${\left({\mathbb{U}}_{n}f\right)}^2$ as a double sum (over ordered pairs i < j). The expected values of the off-diagonal terms vanish by degeneracy.

For a general measurable function $f:{\mathcal{X}}^2\to \mathbb{R}$ the mean Pf² is the projection onto the constant functions, and the function ${\stackrel{‒}{f}}_1$ defined by ${\stackrel{‒}{f}}_1\left(x_1\right)=\int f(x_1,x_2)dP\left(x_2\right)-P^{2}f$ is the projection of f in L₂(P²) onto the mean zero functions of one variable. The decomposition

$$f(x_1,x_2)=P^{2}f+{\stackrel{‒}{f}}_1\left(x_1\right)+{\stackrel{‒}{f}}_1\left(x_2\right)+f_{12}(x_1,x_2),$$

where f₁₂ is defined by the equation yields the Hoeffding decomposition ${\mathbb{U}}_{n}f=P^{2}f+2{\mathbb{P}}_n{\stackrel{‒}{f}}_1+{\mathbb{U}}_n{f}_{12}$ of the U-statistic in orthogonal parts, with ${\mathbb{U}}_n{f}_{12}$ degenerate. Using Lemma 5 we see that the variance of ${\mathbb{U}}_{n}f$ is equal to $(4∕n)P{\stackrel{‒}{f}}_1^2+2∕\left(n(n-1)\right)P^2{f}_{12}^2$. The norm of ${\stackrel{‒}{f}}_1$ is smaller than the norm of f₁. Because f₁₂ is projection of f, its norm is bounded by the norm of f.