UNIFORMLY VALID POST-REGULARIZATION CONFIDENCE REGIONS FOR MANY FUNCTIONAL PARAMETERS IN Z-ESTIMATION FRAMEWORK

Abstract

In this paper, we develop procedures to construct simultaneous confidence bands for $\tilde{p}$ potentially infinite-dimensional parameters after model selection for general moment condition models where $\tilde{p}$ is potentially much larger than the sample size of available data, n. This allows us to cover settings with functional response data where each of the $\tilde{p}$ parameters is a function. The procedure is based on the construction of score functions that satisfy Neyman orthogonality condition approximately. The proposed simultaneous confidence bands rely on uniform central limit theorems for high-dimensional vectors (and not on Donsker arguments as we allow for $\tilde{p}\gg n$). To construct the bands, we employ a multiplier bootstrap procedure which is computationally efficient as it only involves resampling the estimated score functions (and does not require resolving the high-dimensional optimization problems). We formally apply the general theory to inference on regression coefficient process in the distribution regression model with a logistic link, where two implementations are analyzed in detail. Simulations and an application to real data are provided to help illustrate the applicability of the results.

1. Introduction.

High-dimensional models have become increasingly popular in the last two decades. Much research has been conducted on estimation of these models. However, inference about parameters in these models is much less understood, although the literature on inference is growing quickly; see the list of references below. In this paper, we construct simultaneous confidence bands for many functional parameters parameters in a very general framework of moment condition models, where each parameter itself can be an infinite-dimensional object, and the number of such parameters can be much larger than the sample size of available data. Our paper builds upon [11], where simultaneous confidence bands have been constructed for many scalar parameters in a high-dimensional sparse z-estimation framework.

As a substantive application, we apply our general results to provide simultaneous confidence bands for parameters in a logistic regression model with functional response data

$${\text{E}}_P\left[Y_u|D,X\right]=\mathrm{\Lambda }\left(D^{\prime }{\theta }_u+X^{\prime }{\beta }_u\right),u\in \mathcal{U},$$ (1.1)

where $D={\left(D_1,\dots ,D_{\tilde{p}}\right)}^{\prime }$ is a $\tilde{p}$-vector of covariates whose effects are of interest, X = (X₁,…,X_p)′ is a p-vector of controls, $\mathrm{\Lambda }:ℝ\to ℝ$ is the logistic link function, $\mathcal{U}=[0,1]$ is a set of indices and for each $u\in \mathcal{U}$, $Y_u=1\left\{Y\le (1-u)\underset{\_}{y}+u\overline{y}\right\}$ for some constants $\underset{\_}{y}\le \overline{y}$ and the response variable Y, ${\theta }_{u1}={\left({\theta }_{u1},\dots ,{\theta }_{u\tilde{p}}\right)}^{\prime }$ is a vector of target parameters and β_u = (β_u1,…,β_up)′ is a vector of nuisance parameters. Here, both $\tilde{p}$ and p are allowed to be potentially much larger than the sample size n, and we have $\tilde{p}$ functional target parameters ${\left({\theta }_{uj}\right)}_{u\in \mathcal{U}}$ and p functional nuisance parameters ${\left({\beta }_{uj}\right)}_{u\in \mathcal{U}}$. This example is important because it demonstrates that our methods can be used for inference about the whole distribution of the response variable Y given D and X in a high-dimensional setting, and not only about some particular features of it such as mean or median. This model is called a distribution regression model in [22] and a conditional transformation model in [26], who argue that the model provides a rich class of models for conditional distributions, and offers a useful generalization of traditional proportional hazard models as well as a useful alternative to quantile regression. We develop inference methods to construct simultaneous confidence bands for many functional parameters of this model in Section 3.

Toward this goal, our contributions include to effectively estimate a continuum of high-dimensional nuisance parameters, allow for approximately sparse models, control sparse eigenvalues of a continuum of random matrices, establish an approximate linearization for a collection of “orthogonalized” (or “de-biased”) estimators and establish the validity of a multiplier bootstrap for the construction of confidence bands for the many functional parameters of interest based on these estimators. In particular, these contributions build upon but go much beyond [11] (Corollary 4), which considers the special case of many scalar parameters in a z-estimation framework, and beyond [19] (Theorem 5.1), where simultaneous confidence bands are constructed via multiplier bootstrap for any large collection of approximately linear scalar estimator $\sqrt{n}\left({\widehat{\theta }}_j-{\theta }_j\right)=n^{-1}\sum _{j=1}^p{\psi }_j+r_{nj}$, j = 1,…,p, where the ℓ_∞-norm of r_n is $o_P(1/\sqrt{\text{log\hspace{0.17em}}p})$.

Our general results refer to the problem of estimating the set of parameters ${\left({\theta }_{uj}\right)}_{u\in \mathcal{U},j\in [\tilde{p}]}$ in the moment condition model,

$$Ep\left[{\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}\right)\right]=0,u\in \mathcal{U},j\in [\tilde{p}],$$ (1.2)

where W is a random element that takes values in a measurable space $\left(\mathcal{W},{\mathcal{A}}_{\mathcal{W}}\right)$ according to a probability measure P, $\mathcal{U}\subset {ℝ}^{d_u}$ and $[\tilde{p}]:=\left\{1,\dots ,\tilde{p}\right\}$ are sets of indices, and for each $u\in \mathcal{U}$ and $j\in [\tilde{p}]$, ψ_uj is a known score function, θ_uj is a scalar parameter of interest and η_uj is a potentially high-dimensional (or infinite-dimensional) nuisance parameter. Assuming that a random sample of size n, ${\left(W_i\right)}_{i=1}^n$, from the distribution of W is available together with suitable estimators ${\widehat{\eta }}_{uj}$ of η_uj, we aim to construct simultaneous confidence bands for ${\left({\theta }_{uj}\right)}_{u\in \mathcal{U},j\in [\tilde{p}]}$ that are valid uniformly over a large class of probability measures P, say ${\mathcal{P}}_n$. Specifically, for each $u\in \mathcal{U}$ and $j\in [\tilde{p}]$, we construct an appropriate estimator ${\check{\theta }}_{uj}$ of θ_uj along with an estimator of the standard deviation of $\sqrt{n}\left({\check{\theta }}_{uj}-{\theta }_{uj}\right),{\widehat{\sigma }}_{uj}$, such that

$$P_P\left({\check{\theta }}_{uj}-\frac{c_{\alpha }{\widehat{\sigma }}_{uj}}{\sqrt{n}}\le {\theta }_{uj}\le {\check{\theta }}_{uj}+\frac{c_{\alpha }{\widehat{\sigma }}_{uj}}{\sqrt{n}},\forall u\in \mathcal{U},j\in [\tilde{p}]\right)\to 1-\alpha ,$$ (1.3)

uniformly over $P\in {\mathcal{P}}_n$, where α ∈ (0, 1) and c_α is an appropriate critical value, which we choose to construct using a multiplier bootstrap method. The left- and the right-hand sides of the inequalities inside the probability statement (1.3) then can be used as bounds in simultaneous confidence bands for θ_uj’s. In this paper, we are particularly interested in the case when $\tilde{p}$ is potentially much larger than n and $\mathcal{U}$ is an uncountable subset of ${ℝ}^{d_u}$, so that for each $j\in [\tilde{p}]$, ${\left({\theta }_{uj}\right)}_{u\in \mathcal{U}}$ is an infinite-dimensional (i.e., functional) parameter.

In the presence of high-dimensional nuisance parameters, construction of valid confidence bands is delicate. Dealing with high-dimensional parameters requires relying upon regularization that leads to lack of asymptotic linearization of the estimators of target parameters since regularized estimators of nuisance parameters suffer from a substantial bias and this bias spreads into the estimators of the target parameters. This lack of asymptotic linearization in turn typically translates into severe distortions in coverage probability of the confidence bands constructed by traditional techniques that are based on perfect model selection; see [30–32, 40]. To deal with this problem, we assume that the score functions ψ_uj are constructed to satisfy a near-orthogonality condition that makes them immune to first-order changes in the value of the nuisance parameter, namely

$${\partial }_r\left\{{\text{E}}_P\left[{\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}+r\tilde{\eta }\right)\right]\right\}{|}_{r=0}\approx 0,u\in \mathcal{U},j\in [\tilde{p}],$$ (1.4)

for all $\tilde{\eta }$ in an appropriate set where ∂_r denotes the derivative with respect to r. We shall often refer to this condition as Neyman orthogonality, since in low­dimensional parametric settings the orthogonality property originates in the work of Neyman on the C(α) test in the 50s. In Section 2 below, we describe a few general methods for constructing the score functions ψ_uj obeying the Neyman orthogonality condition.

The Neyman orthogonality condition (1.4) is important because it helps to make sure that the bias from the estimators of the high-dimensional nuisance parameters does not spread into the estimators of the target parameters. In particular, under (1.4), it follows that

$${\text{E}}_{P,W}\left[{\psi }_{uj}\left(W,{\theta }_{uj},{\widehat{\eta }}_{uj}\right)\right]\approx 0,u\in \mathcal{U},j\in [\tilde{p}],$$

at least up to the first order, where the index W in E_P,W [·] means that the expectation is taken over W only. This makes the estimators of the target parameters θ_uj immune to the bias in the estimators ${\widehat{\eta }}_{uj}$, which in turn improves their statistical properties and opens up the possibilities for valid inference.

As the framework (1.2) covers a broad variety of applications, it is instructive to revisit the logistic regression model with functional response data (1.1). To construct score functions ψ_uj that satisfy both the moment conditions (1.2) and the Neyman orthogonality condition (1.4) in this example, for $u\in \mathcal{U}$ and $j\in [\tilde{p}]$, define a $(\tilde{p}+p-1)$-vector of additional nuisance parameters

$${\gamma }_u^j=\text{arg}\underset{\gamma \in {ℝ}^{\tilde{p}+p-1}}{\text{min}}{E}_P\left[f_u^2{\left\{D_j-X^j\gamma \right\}}^2\right],$$ (1.5)

where $X^j=\left(D_{[\tilde{p}]\backslash j}^{\prime },X^{\prime }\right),D_{[\tilde{p}]\backslash j}={\left(D_1,\dots ,D_{j-1},D_{j+1},\dots ,D_{\tilde{p}}\right)}^{\prime }$, and

$$f_u^2=f_u^2(D,X)={\text{Var}}_P\left(Y_u|D,X\right).$$ (1.6)

Then, denoting W = (Y,D,X) and splitting θ_u into θ_uj and ${\theta }_{u[\tilde{p}]\backslash j}={\left({\theta }_{u1},\dots ,{\theta }_{uj-1},{\theta }_{uj+1},\dots ,{\theta }_{u[\tilde{p}]}^{\prime }\right)}^{\prime }$, we set

$${\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}\right)=\left\{Y_u-\mathrm{\Lambda }\left(D_j{\theta }_{uj}+X^j{\beta }_u^j\right)\right\}\left(D_j-X^j{\gamma }_u^j\right),$$

where ${\eta }_{uj}=\left({\beta }_u^j,{\gamma }_u^j\right)$ and ${\beta }_u^j={\left({\theta }_{u[\tilde{p}]\backslash j}^{\prime },{\beta }_u^{\prime }\right)}^{\prime }$. It is straightforward to see that these score functions ψ_uj satisfy the moment conditions (1.2) and to see that they also satisfy the Neyman orthogonality condition (1.4), observe that

$$\begin{array}{l}{\partial }_{\beta }\left\{{\text{E}}_P\left[{\psi }_{uj}\left(W,{\theta }_{uj},\beta ,{\gamma }_u^j\right)\right]\right\}{|}_{\beta ={\beta }_u^j}=-{\text{E}}_P\left[f_u^2\left\{D_j-X^j{\gamma }_u^j\right\}{\left(X^j\right)}^{\prime }\right]=0,\hfill \\ {\partial }_{\gamma }\left\{{\text{E}}_P\left[{\psi }_{uj}\left(W,{\theta }_{uj},{\beta }_u^j,\gamma \right)\right]\right\}{|}_{\gamma =y_{ii}^j}=-{\text{E}}_P\left[\left\{Y_u-\mathrm{\Lambda }\left(D^{\prime }{\theta }_u+X^{\prime }{\beta }_u\right)\right\}{\left(X^j\right)}^{\prime }\right]=0,\hfill \end{array}$$

where the first line by definition of $f_u^2$ and ${\gamma }_u^j$ since Var_P (Y_u | D, X) = Λ′ (D′ θ_u + X′ β_u), and the second by (1.1). Because of this orthogonality condition, we can exploit the moment conditions (1.2) to construct regular, $\sqrt{n}$-consistent, estimators of θ_uj even if nonregular, regularized or post-regularized, estimators of ${\eta }_{uj}=\left({\beta }_u^j,{\gamma }_u^j\right)$ are used to cope with high-dimensionality. Using these regular estimators of θ_uj, we then can construct valid confidence bands (1.3).

Our general approach to construct simultaneous confidence bands, which is developed in Section 2, can be described as follows. First, we construct the moment conditions (1.2) that satisfy the Neyman orthogonality condition (1.4), and use these moment conditions to construct estimators ${\check{\theta }}_{uj}$ of θ_uj for all $u\in \mathcal{U}$ and $j\in [\tilde{p}]$. Second, under appropriate regularity conditions, we establish a Bahadur representation for ${\check{\theta }}_{uj}$’s. Third, employing the Bahadur representation, we are able to derive a suitable Gaussian approximation for the distribution of ${\check{\theta }}_{uj}$’s. Importantly, the Gaussian approximation is possible even if both $\tilde{p}$ and the dimension of the index set $\mathcal{U}$, d_u, are allowed to grow with n, and $\tilde{p}$ asymptotically remains much larger than n. Finally, from the Gaussian approximation, we construct simultaneous confidence bands using a multiplier bootstrap method. Here, the Gaussian and bootstrap approximations are constructed by applying the results on high­dimensional central limit and bootstrap theorems established in [16–21] by verifying the conditions there.

Although regularity conditions underlying our approach can be verified for many models defined by moment conditions, for illustration purposes, we explicitly verify these conditions for the logistic regression model with functional response data (1.1) in Section 3. We also note that the regularity conditions, in particular those related to the entropy of the nuisance parameter estimators, can be substantially relaxed if we use sample splitting, so that the nuisance parameters and parameters of interest are estimated on separate samples; see [15]. In addition, we examine the performance of the proposed procedures in a Monte Carlo simulation study and provide an example based on real data in Section 5. Moreover, in the Supplementary Material [5], we discuss the construction of simultaneous confidence bands based on a double-selection estimator. This estimator does not require to explicitly construct the score functions satisfying the Neyman orthogonality condition but nonetheless is first-order equivalent to the estimator based on such functions.

We also develop new results for ℓ₁-penalized M-estimators in Section 4 to handle functional data and criterion functions that depend on nuisance functions for which only estimates are available building on ideas in [3,4,12] (for brevity of the paper, generic results are deferred to Supplementary Material, and Section 4 only contains results that are relevant for the logistic regression model studied in Section 3). Specifically, we develop a method to select penalty parameters for these estimators and extend the existing theory to cover functional data to achieve rates of convergence and sparsity guarantees that hold uniformly over $u\in \mathcal{U}$. The ability to allow both for functional data and for nuisance functions is crucial in the implementation and in theoretical analysis of the methods proposed in this paper.

Orthogonality conditions like that in (1.4) have played an important role in statistics and econometrics. In low-dimensional settings, a similar condition was used by Neyman in [37] and [38] while in semiparametric models the orthogonality conditions were used in [1, 35, 36, 41] and [33]. In high-dimensional settings, [7] and [2] were the first to use the orthogonality condition (1.4) in a linear instrumental variables model with many instruments. Related ideas have also been used in the literature to construct confidence bands in high-dimensional linear models, generalized linear models and other nonlinear models; see [6, 8–11, 13, 27,28,43, 46, 47] and [39], where we can interpret each procedure as implicitly or explicitly constructing and solving an approximately Neyman-orthogonal estimating equation. We contribute to this quickly growing literature by providing procedures to construct simultaneous confidence bands for many infinite-dimensional parameters identified by moment conditions.

Throughout the paper, we use the standard notation from the empirical process theory. In particular, we use ${\mathbb{E}}_n$ to denote the expectation with respect to the empirical measure associated with the data ${\left(W_i\right)}_{i=1}^n$, and we use ${\mathbb{G}}_n$ to denote the empirical process $\sqrt{n}\left({\mathbb{E}}_n-{\text{E}}_P\right)$. More details about the notation are given in the Supplementary Material.

2. Confidence regions for function-valued parameters based on moment conditions.

2.1. Generic construction of confidence regions.

In this section, we state our results under high-level conditions. In the next section, we will apply these results to construct simultaneous confidence bands for many infinite-dimensional parameters in the logistic regression model with functional response data.

Recall that we are interested in constructing simultaneous confidence bands for a set of target parameters ${\left({\theta }_{uj}\right)}_{u\in \mathcal{U},j\in [\tilde{p}]}$ where for each $u\in \mathcal{U}\subset {ℝ}^{d_u}$ and $j\in [\tilde{p}]=\left\{1,\dots ,\tilde{p}\right\}$, the parameter θ_uj satisfies the moment condition (1.2) with η_uj being a potentially high-dimensional (or infinite-dimensional) nuisance parameter. Assume that θ_uj ∈ Θ_uj, a finite or infinite interval in $ℝ$, and that η_uj ∈ T_uj, a convex set in a normed space equipped with a norm ∥ · ∥_e. We allow $\mathcal{U}$ to be a possibly uncountable set of indices, and $\tilde{p}$ to be potentially large.

We assume that a random sample ${\left(W_i\right)}_{i=1}^n$ from the distribution of W is available for constructing the confidence bands. We also assume that for each $u\in \mathcal{U}$ and $j\in [\tilde{p}]$, the nuisance parameter η_uj can be estimated by ${\widehat{\eta }}_{uj}$ using the same data ${\left(W_i\right)}_{i=1}^n$. In the next section, we discuss examples where ${\widehat{\eta }}_{uj}$’s are based on Lasso or Post-Lasso methods (although other modern regularization and post­regularization methods can be applied). Our confidence bands will be based on the estimators ${\check{\theta }}_{uj}$ of θ_uj that are for each $u\in \mathcal{U}$ and $j\in [\tilde{p}]$ defined as approximate ϵ_n-solutions in Θ_uj to sample analogs of the moment conditions (1.2), that is,

$$\underset{u\in \mathcal{U},j\in [\tilde{p}]}{\text{sup}}\left\{\left|{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,{\check{\theta }}_{uj},{\widehat{\eta }}_{uj}\right)\right]\right|-\underset{\theta \in {\Theta }_{uj}}{\text{inf}}\left|{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,\theta ,{\widehat{\eta }}_{uj}\right)\right]\right|\right\}\le {ϵ}_n,$$ (2.1)

where ϵ_n = o(δ_nn^−1/2) for all n ≥ 1 and some sequence (δ_n)_n≥1 of positive constants converging to zero.

To motivate the construction of the confidence bands based on the estimators ${\check{\theta }}_{uj}$, we first study distributional properties of these estimators. To do that, we will employ the following regularity conditions. Let C₀ be a strictly positive (and finite) constant, and for each $u\in \mathcal{U}$ and $j\in [\tilde{p}]$, let ${\mathcal{T}}_{uj}$ be some subset of T_uj, whose properties are specified below in assumptions. In particular, we will choose the sets ${\mathcal{T}}_{uj}$ so that, on the one hand, their complexity does not grow too fast with n but, on the other hand, for each $u\in \mathcal{U}$ and $j\in [\tilde{p}]$, the estimator ${\widehat{\eta }}_{uj}$ takes values in ${\mathcal{T}}_{uj}$ with high probability. As discussed before, we rely on the following near­orthogonality condition.

DEFINITION 2.1 (Near-orthogonality condition). For each $u\in \mathcal{U}$ and $j\in [\tilde{p}]$, we say that ψ_uj obeys the near-orthogonality condition with respect to ${\mathcal{T}}_{uj}\subset T_{uj}$ if the following conditions hold: The Gateaux derivative map

$${\text{D}}_{u,j,\overline{r}}\left[\eta -{\eta }_{uj}\right]:={\partial }_r\left\{{\text{E}}_P\left[{\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}+r\left(\eta -{\eta }_{uj}\right)\right)\right]\right\}{|}_{r=\overline{r}}$$

exists for all $\overline{r}\in [0,1)$ and $\eta \in {\mathcal{T}}_{uj}$ and (nearly) vanishes at $\overline{r}=0$, namely,

$$\left|{\text{D}}_{u,j,0}\left[\eta -{\eta }_{uj}\right]\right|\le C_0{\delta }_n{n}^{-1/2},$$ (2.2)

for all $\eta \in {\mathcal{T}}_{uj}$.

At the end of this section, we describe several methods to obtain score functions ψ_uj that obey the near-orthogonality condition. Together these methods cover a wide variety of applications.

Let ω and c₀ be some strictly positive (and finite) constants, and let n₀ ≥ 3 be some positive integer. Also, let (B_1n)_n≥1 and (B_2n)_n≥1 be some sequences of positive constants, possibly growing to infinity, where B_1n ≥ 1 for all n ≥ 1. In addition, denote

$$\begin{array}{l}{\mathcal{S}}_n:={\text{E}}_P\left[\underset{u\in \mathcal{U},j\in [\tilde{p}]}{\text{sup}}|\sqrt{n}{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}\right)\right]\right],\hfill \\ J_{uj}:={\partial }_{\theta }\left\{{\text{E}}_P\left[{\psi }_{uj}\left(W,\theta ,{\eta }_{uj}\right)\right]\right\}{|}_{\theta ={\theta }_{uj}}.\hfill \end{array}$$ (2.3)

The quantity ${\mathcal{S}}_n$ measures how rich the process $\left\{{\psi }_{uj}\left(\cdot ,{\theta }_{uj},{\eta }_{uj}\right):u\in \mathcal{U},j\in [\tilde{p}]\right\}$ is. The quantity J_uj measures the degree of identifiability of θ_uj by the moment condition (1.2). In many applications, it is bounded in absolute value from above and away from zero. Finally, let ${\mathcal{P}}_n$ be a set of probability measures P of possible distributions of W on the measurable space $\left(\mathcal{W},{\mathcal{A}}_{\mathcal{W}}\right)$.

We collect our main conditions on the score functions ψ_uj and the true values of the target parameters θ_uj in the following assumption.

Assumption 2.1 (Moment condition problem). For all n ≥ n₀, $P\in {\mathcal{P}}_n$, $u\in \mathcal{U}$, and $j\in [\tilde{p}]$, the following conditions hold: (i) The true parameter value θ_uj obeys (1.2), and Θ_uj contains a ball of radius $C_0{n}^{-1/2}{\mathcal{S}}_n\text{\hspace{0.17em}log\hspace{0.17em}}n$ centered at θ_uj. (ii) The map (θ,η) ↦ E_P[ψ_uj(W,θ,η)] is twice continuously Gateaux-differentiable on ${\Theta }_{uj}\times {\mathcal{T}}_{uj}$. (iii) The score function ψ_uj obeys the near­orthogonality condition given in Definition 2.1 for the set ${\mathcal{T}}_{uj}\subset T_{uj}$. (iv) For all θ ∈ Θ_uj, |E_P[ψ_uj(W,θ,η_uj)]| ≥ 2⁻¹|J_uj(θ - θ_uj)| ∧ c₀, where J_uj satisfies c₀ ≤ |J_uj | ≤ C₀. (v) For all r ∈ [0, 1), θ ∈ Θ_uj, and η ∈ T_uj:

1. ${\text{E}}_P\left[{\left({\psi }_{uj}(W,\theta ,\eta )-{\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}\right)\right)}^2\right]\le C_0{\left(\left|\theta -{\theta }_{uj}\right|\vee \Vert \eta -{\eta }_{uj}{\Vert }_e\right)}^{\omega }$,

2. $\left|{\partial }_r{\text{E}}_P\left[{\psi }_{uj}\left(W,\theta ,{\eta }_{uj}+r\left(\eta -{\eta }_{uj}\right)\right)\right]\right|\le B_{1n}\Vert \eta -{\eta }_{uj}{\Vert }_e$,

3. $\left|{\partial }_r^2{\text{E}}_P\left[{\psi }_{uj}\left(W,{\theta }_{uj}+r\left(\theta -{\theta }_{uj}\right),{\eta }_{uj}+r\left(\eta -{\eta }_{uj}\right)\right)\right]\right|\le B_{2n}\left({\left|\theta -{\theta }_{uj}\right|}^2\vee \Vert \eta -{\eta }_{uj}{\Vert }_e^2\right)$.

Assumption 2.1 is mild and standard in moment condition problems. Assumption 2.1 (i) requires θ_uj to be sufficiently separated from the boundary of Θ_uj. Assumption 2.1(ii) requires that the functions (θ,η) ↦ E_P[ψ_uj(W,θ,η)] are smooth. It is a mild condition because it does not require smoothness of the functions (θ, η) ↦ ψ_uj(W,θ,η). Assumption 2.1(iii) is our key condition and is discussed above. Assumption 2.1(iv) implies sufficient identifiability of θ_uj. In particular, it implies that the equation E_P[ψ_uj(W,θ,η_uj)] = 0 has only one solution θ = θ_uj. If this equation has multiple solutions, Assumption 2.1(iv) implies that the set Θ_uj is restricted enough so that there is only one solution in Θ_uj. Assumption 2.1(v-a) means that the functions (θ, η) ↦ ψ_uj(W,θ,η) mapping Θ_uj × T_uj into L²(P) are Lipschitz-continuous at (θ, η) = (θ_uj,η_uj) with Lipschitz order ω/2. In most applications, we can set ω = 2. Assumptions 2.1(v-b,v-c) impose smoothness bounds on the functions (θ, η) ↦ E_P[ψ_uj(W,θ,η)].

Next, we state our conditions related to the estimators ${\widehat{\eta }}_{uj}$. Let (Δ_n)_n≥1 and (τ_n)_n≥1 be some sequences of positive constants converging to zero. Also, let (a_n)_n≥1, (v_n)_n≥1, and (K_n)_n≥1 be some sequences of positive constants, possibly growing to infinity, where a_n ≥ n ∨ K_n and v_n ≥ 1 for all n ≥ 1. Finally, let q ≥ 2 be some constant.

Assumption 2.2 (Estimation of nuisance parameters). For all n ≥ n₀ and $P\in {\mathcal{P}}_n$, the following conditions hold: (i) With probability at least 1 − Δ_n, we have ${\widehat{\eta }}_{uj}\in {\mathcal{T}}_{uj}$ for all $u\in \mathcal{U}$ and $j\in [\tilde{p}]$. (ii) For all $u\in \mathcal{U}$, $j\in [\tilde{p}]$, and $\eta \in {\mathcal{T}}_{uj}$, ∥η − η_uj ∥_e ≤ τ_n. (iii) For all $u\in \mathcal{U}$ and $j\in [\tilde{p}]$, we have ${\eta }_{uj}\in {\mathcal{T}}_{uj}$. (iv) The function class ${\mathcal{F}}_1=\left\{{\psi }_{uj}(\cdot ,\theta ,\eta ):u\in \mathcal{U},j\in [\tilde{p}],\theta \in {\Theta }_{uj},\eta \in {\mathcal{T}}_{uj}\right\}$ is suitably measurable and its uniform entropy numbers obey

$$\underset{Q}{\text{sup}}\text{\hspace{0.17em}log\hspace{0.17em}}N\left(ϵ\Vert F_1{\Vert }_{Q,2},{\mathcal{F}}_1,\Vert \cdot {\Vert }_{Q,2}\right)\le v_n\text{\hspace{0.17em}log}\left(a_n/ϵ\right)\text{\hspace{1em} for all }0<ϵ\le 1,$$ (2.4)

where F₁ is a measurable envelope for ${\mathcal{F}}_1$ that satisfies ∥F₁∥_P,_q ≤ K_n. (v) For all $f\in {\mathcal{F}}_1$, we have c₀ ≤ ∥ f ∥_P,2 ≤ C₀. (vi) The complexity characteristics a_n and v_n satisfy:

1. ${\left(v_n\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n/n\right)}^{1/2}\le C_0{\tau }_n$,

2. ${\left(B_{1n}{\tau }_n+{\mathcal{S}}_n\text{\hspace{0.17em}log\hspace{0.17em}}n/\sqrt{n}\right)}^{\omega /2}{\left(v_n\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n\right)}^{1/2}+n^{-1/2+1/q}{v}_n{K}_n\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n\le C_0{\delta }_n$,

3. $n^{1/2}{B}_{1n}^2{B}_{2n}{\tau }_n^2\le C_0{\delta }_n$.

Assumption 2.2 provides sufficient conditions for the estimation of the nuisance parameters ${\left({\eta }_{uj}\right)}_{u\in \mathcal{U},j\in [\tilde{p}]}$. Assumption 2.2 (i) requires that the set ${\mathcal{T}}_{uj}$ is large enough so that ${\widehat{\eta }}_{uj}\in {\mathcal{T}}_{uj}$ with high probability. Assumptions 2.2 (i,ii) together require that the estimator ${\widehat{\eta }}_{uj}$ converges to η_uj with the rate τ_n. This rate should be fast enough so that Assumptions 2.2(vi-b,vi-c) are satisfied. Assumption 2.2(iv) gives a bound on the complexity of the set ${\mathcal{T}}_{uj}$ expressed via uniform entropy numbers, and Assumptions 2.2(vi-a,vi-b) require that the set ${\mathcal{T}}_{uj}$ is small enough so that its complexity does not grow too fast. Assumption 2.2(v) requires that the functions (θ, η) ↦ ψ_uj(W,θ,η) are scaled properly. Suitable measurability of ${\mathcal{F}}_1$, required in Assumption 2.2(iv), is a mild condition that is satisfied in most practical cases; see the Supplementary Material and [25] for clarifications. Overall, Assumption 2.2 shows the trade-off in the choice of the sets ${\mathcal{T}}_{uj}$: setting ${\mathcal{T}}_{uj}$ large, on the one hand, makes it easy to satisfy Assumption 2.2(i) but, on the other hand, yields large values of a_n and v_n in Assumption 2.2(iv) making it difficult to satisfy Assumption 2.2(vi).

We stress that the class ${\mathcal{F}}_1$ does not need to be Donsker because its uniform entropy numbers are allowed to increase with n. This is important because allowing for non-Donsker classes is necessary to deal with high-dimensional nuisance parameters. Note also that our conditions are very different from the conditions imposed in various settings with nonparametrically estimated nuisance functions; see, for example, [44, 45] and [29].

In addition, we emphasize that the conditions stated in Assumption 2.2 are sufficient for our results for the general model (1.2) but can often be relaxed if the structure of the functions ψ_uj(W,θ,η) is known. For example, it is possible to relax Assumption 2.2(vi) if the functions ψ_uj(W,θ,η) are linear in θ, which happens in the linear regression model with θ being the coefficient on the covariate of interest; see [9]. Moreover, it is possible to relax the entropy condition (2.4) of Assumption 2.2 by relying upon sample splitting, where part of the data is used to estimate η_uj, and the other part is used to estimate θ_uj given an estimate ${\widehat{\eta }}_{uj}$ of η_uj; see [2] and [15]. By swapping the role of two parts, and averaging the resulting two estimators, we do not incur any efficiency losses.

The following theorem is our first main result in this paper.

Theorem 2.1 (Uniform Bahadur representation). Under Assumptions 2.1 and 2.2, for an estimator ${\left({\check{\theta }}_{uj}\right)}_{u\in \mathcal{U},j\in [\tilde{p}]}$ that obeys (2.1), we have

$$\sqrt{n}{\sigma }_{uj}^{-1}\left({\check{\theta }}_{uj}-{\theta }_{uj}\right)={\mathbb{G}}_n{\overline{\psi }}_{uj}+O_P\left({\delta }_n\right)$$ (2.5)

in ${\ell }^{\infty }(\mathcal{U}\times [\tilde{p}])$ uniformly over $P\in {\mathcal{P}}_n$, where ${\overline{\psi }}_{uj}(\cdot ):=-{\sigma }_{uj}^{-1}{J}_{uj}^{-1}{\psi }_{uj}\left(\cdot ,{\theta }_{uj},{\eta }_{uj}\right)$ and ${\sigma }_{uj}^2:=J_{uj}^{-2}{\text{E}}_P\left[{\psi }_{uj}^2\left(W,{\theta }_{uj},{\eta }_{uj}\right)\right]$.

Comment 2.1 (On the proof ofTheorem 2.1). To prove this theorem, we use the following identity:

$$\sqrt{n}{\text{E}}_{P,W}\left[{\psi }_{uj}\left(W,{\check{\theta }}_{uj},{\widehat{\eta }}_{uj}\right)-{\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}\right)\right]=-{\sqrt{n\mathbb{E}}}_n\left[{\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}\right)\right]$$ (2.6)

$$+\sqrt{n}{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,{\check{\theta }}_{uj},{\widehat{\eta }}_{uj}\right)\right]+{\mathbb{G}}_n{\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}\right)-{\mathbb{G}}_n{\psi }_{uj}\left(W,{\check{\theta }}_{uj},{\widehat{\eta }}_{uj}\right).$$ (2.7)

Here, the term on the right-hand side of (2.6) is the main term on the right-hand side of (2.5), up to a normalization (σ_uj J_uj)^−l. Also, we show that the first term in (2.7) is O_P (δ_n) since ${\check{\theta }}_{uj}$ satisfies (2.1). Moreover, using arather standard theory of Z-estimators, we show that ${\check{\theta }}_{uj}-{\theta }_{uj}=O_P\left(B_{1n}{\tau }_n\right)$. This in turn allows us to show with the help of empirical process arguments that the difference of the last two terms in (2.7) is O_P (δ_n) as well. (In [15], we also point out that this difference is O_P (δ_n) under much weaker entropy conditions than those in Assumption 2.2 if ${\widehat{\eta }}_{uj}$ and ${\check{\theta }}_{uj}$ are obtained using separate samples.) Thus, it remains to show that the left-hand side of (2.6) is equal to the left-hand side of (2.5) up to an approximation error O_P (δ_n) and up to a normalization (σ_ui j_uj)⁻¹. To do so, we use second-order Taylor’s expansion of the function

$$f(r)=\sqrt{n}{\text{E}}_{P,W}\left[{\psi }_{uj}\left(W,{\theta }_{uj}+r\left({\check{\theta }}_{uj}-{\theta }_{uj}\right),{\eta }_{uj}+r\left({\widehat{\eta }}_{uj}-{\eta }_{uj}\right)\right)\right]$$

at r = 1 around r = 0. This gives

$$\begin{array}{l}\sqrt{n}{\text{E}}_{P,W}\left[{\psi }_{uj}\left(W,{\check{\theta }}_{uj},{\widehat{\eta }}_{uj}\right)-{\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}\right)\right]\hfill \\ =f(1)-f(0)\hfill \\ =\sqrt{n}{J}_{uj}\left({\check{\theta }}_{uj}-{\theta }_{uj}\right)+\sqrt{n}{D}_{u,j,0}\left[{\widehat{\eta }}_{uj}-{\eta }_{uj}\right]+\sqrt{n}{f}^{″}(\overline{r})/2\hfill \end{array}$$

for some $\overline{r}\in (0,1)$. Here, $\sqrt{n}{f}^{″}(\overline{r})=O_P\left({\delta }_n\right)$ follows from Assumptions 2.1 and 2.2 and the key near-orthogonality condition also allows us to show that $\sqrt{n}{D}_{u,j,0}\left[{\widehat{\eta }}_{uj}-{\eta }_{uj}\right]=O_P\left({\delta }_n\right)$. Without this condition, the term $\sqrt{n}{D}_{u,j,0}\left[{\widehat{\eta }}_{uj}-{\eta }_{uj}\right]$ would give first-order bias and lead to slower-than-$\sqrt{n}$ rate of convergence of the estimator ${\check{\theta }}_{uj}$. Finally, again using the empirical process arguments, we can show that all the bounds including the term O_P (δ_n) hold uniformly over $u\in \mathcal{U}$ and $j\in [\tilde{p}]$.

Comment 2.2 (On uniformity in u in Theorem 2.1). When the functions $u\mapsto \sqrt{n}{\sigma }_{uj}^{-1}\left({\check{\theta }}_{uj}-{\theta }_{uj}\right)-{\mathbb{G}}_n{\overline{\psi }}_{uj}$ are Lipschitz-continuous, one can use a simple discretization argument to conclude that the approximation in (2.5) holds uniformly over $(u,j)\in \mathcal{U}\times [\tilde{p}]$ as long as we can show that it holds for each $(u,j)\in \mathcal{U}\times [\tilde{p}]$. However, in many applications, including the distribution regression model discussed in Section 3, this function is actually not continuous, and the location of jumps depends on the data. Therefore, we have to rely on a more complicated argument to establish uniformity in u in the bound (2.5).

The uniform Bahadur representation derived in Theorem 2.1 is useful for the construction of simultaneous confidence bands for ${\left({\theta }_{uj}\right)}_{u\in \mathcal{U},j\in [\tilde{p}]}$ as in (1.3). For this purpose, we apply new high-dimensional central limit and bootstrap theorems that have been recently developed in a sequence of papers [16, 18–20] and [21]. To apply these theorems, we make use of the following regularity condition.

Let ${\left({\overline{\delta }}_n\right)}_{n\ge 1}$ be a sequence of positive constants converging to zero. Also, let (ϱ_n)_n≥1, ${\left({\overline{\varrho }}_n\right)}_{n\ge 1}$, (A_n)_n≥1, ${\left({\overline{A}}_n\right)}_{n\ge 1}$, and (L_n)_n≥1 be some sequences of positive constants, possibly growing to infinity, where ϱ_n ≥ 1, A_n ≥ n, and ${\overline{A}}_n\ge n$ for all n ≥ 1. In addition, from now on, we assume that q > 4. Denote by ${\widehat{\psi }}_{uj}(\cdot ):=-{\widehat{\sigma }}_{uj}^{-1}{\widehat{J}}_{uj}^{-1}{\psi }_{uj}\left(\cdot ,{\check{\theta }}_{uj},{\widehat{\eta }}_{uj}\right)$ an estimator of ${\widehat{\psi }}_{uj}(\cdot )$, with ${\widehat{J}}_{uj}$ and ${\widehat{\sigma }}_{uj}$ being suitable estimators of J_uj and σ_uj.

Assumption 2.3 (Additional score regularity). For all n ≥ n₀ and $P\in {\mathcal{P}}_n$, the following conditions hold: (i) The function class ${\mathcal{F}}_0=\left\{{\overline{\psi }}_{uj}(\cdot ):u\in \mathcal{U},j\in [\tilde{p}]\right\}$ is suitably measurable and its uniform entropy numbers obey

$$\underset{Q}{\text{sup}}\text{\hspace{0.17em}log\hspace{0.17em}}N\left(ϵ\Vert F_0{\Vert }_{Q,2},{\mathcal{F}}_0,\Vert \cdot {\Vert }_{Q,2}\right)\le {\varrho }_n\text{log}\left(A_n/ϵ\right)\text{\hspace{1em} for all }0<ϵ\le 1,$$

where F₀ is a measurable envelope for ${\mathcal{F}}_0$ that satisfies ∥F₀∥_P,q ≤ L_n. (ii) For all $f\in {\mathcal{F}}_0$ and k = 3,4, we have ${\text{E}}_P\left[|f(W){|}^k\right]\le C_0{L}_n^{k-2}$. (iii) The function class ${\widehat{\mathcal{F}}}_0=\left\{{\overline{\psi }}_{uj}(\cdot )-{\widehat{\psi }}_{uj}(\cdot ):u\in \mathcal{U},j\in [\tilde{p}]\right\}$ satisfies with probability $1-{\mathrm{\Delta }}_n:\text{log\hspace{0.17em}}N\left(ϵ,{\widehat{\mathcal{F}}}_0,\Vert \cdot \Vert {\mathbb{P}}_{n,2}\right)\le {\overline{\varrho }}_n\text{log\hspace{0.17em}}\left({\overline{A}}_n/ϵ\right)$ for all 0 < ϵ ≤ 1 and $\Vert f{\Vert }_{{\mathbb{P}}_n,2}\le {\overline{\delta }}_n$ for all $f\in {\widehat{\mathcal{F}}}_0$.

This assumption is technical, and its verification in applications is rather standard. For the Gaussian approximation result below, we actually only need the first and the second part of this assumption. The third part will be needed for establishing validity of the simultaneous confidence bands based on the multiplier bootstrap procedure. As a side note, observe that Assumption 2.3 allows to bound ${\mathcal{S}}_n$ defined in (2.3) and used in Assumptions 2.1 and 2.2; see Appendix G of the Supplementary Material.

Next, let ${\left({\mathcal{N}}_{uj}\right)}_{u\in \mathcal{U},j\in [\tilde{p}]}$ denote a tight zero-mean Gaussian process indexed by $\mathcal{U}\times [\tilde{p}]$ with covariance operator given by ${\text{E}}_P\left[{\overline{\psi }}_{uj}(W){\overline{\psi }}_{u^{\prime }{j}^{\prime }}(W)\right]$ for $u,u^{\prime }\in \mathcal{U}$ and $\begin{array}{l}j,j^{\prime }\in [\tilde{p}]\hfill \\ \hfill \end{array}$. We have the following corollary of Theorem 2.1, which is our second main result in this paper.

Corollary 2.1 (Gaussian approximation). Suppose that Assumptions 2.1, 2.2 and 2.3(i,ii) hold. In addition, suppose that thefollowing growth conditions are satisfied: ${\delta }_n^2{\varrho }_n\text{log\hspace{0.17em}}{A}_n=o(1)$, $L_n^{2/7}{\varrho }_n\text{log\hspace{0.17em}}{A}_n=o\left(n^{1/7}\right)$ and $L_n^{2/3}{\varrho }_n\text{log\hspace{0.17em}}{A}_n=o\left(n^{1/3-2/(3q)}\right)$. Then

$$\underset{t\in ℝ}{\text{sup}}\left|{\text{P}}_P\left(\underset{u\in \mathcal{U},j\in [\tilde{p}]}{\text{sup}}\left|\sqrt{n}{\sigma }_{uj}^{-1}\left({\check{\theta }}_{uj}-{\theta }_{uj}\right)\right|\le t\right)-{\text{P}}_P\left(\underset{u\in \mathcal{U},j\in [\tilde{p}]}{\text{sup}}\left|{\mathcal{N}}_{uj}\right|\le t\right)\right|=o(1)$$

uniformly over $P\in {\mathcal{P}}_n$.

Based on Corollary 2.1, we are now able to construct simultaneous confidence bands for θ_uj’s as in (1.3). In particular, we will use the Gaussian multiplier bootstrap method employing the estimates ${\widehat{\psi }}_{uj}$ of ${\overline{\psi }}_{uj}$. To describe the method, define the process

$$\widehat{\mathcal{G}}={\left({\widehat{\mathcal{G}}}_{uj}\right)}_{u\in \mathcal{U},j\in [\tilde{p}]}={\left(\frac{1}{\sqrt{n}}\sum _{i=1}^n{\xi }_i{\widehat{\psi }}_{uj}\left(W_i\right)\right)}_{u\in \mathcal{U},j\in [\tilde{p}]},$$ (2.8)

where ${\left({\xi }_i\right)}_{i=1}^n$ are independent standard normal random variables which are independent from the data ${\left(W_i\right)}_{i=1}^n$. Then the multiplier bootstrap critical value c_α is defined as the (1 − α) quantile of the conditional distribution of ${\text{sup}}_{u\in \mathcal{U},j\in [\tilde{p}]}\left|{\widehat{\mathcal{G}}}_{uj}\right|$ given the data ${\left(W_i\right)}_{i=1}^n$. To prove validity of this critical value for the construction of simultaneous confidence bands of the form (1.3), we will impose the following additional assumption. Let (ε_n)_n≥1 be a sequence of positive constants converging to zero.

Assumption 2.4 (Variation estimation). For all n ≥ n₀ and $P\in {\mathcal{P}}_n$,

$${\text{P}}_P\left(\underset{u\in \mathcal{U},j\in [\tilde{p}]}{\text{sup}}\left|\frac{{\widehat{\sigma }}_{uj}}{{\sigma }_{uj}}-1\right|>{\epsilon }_n\right)\le {\mathrm{\Delta }}_n.$$

The following corollary establishing validity of the multiplier bootstrap critical value c_a for the simultaneous confidence bands construction is our third main result in this paper.

Corollary 2.2 (Simultaneous confidence bands). Suppose that Assumptions 2.1–2.4 hold. In addition, suppose that the growth conditions of Corollary 2.1 hold. Finally, suppose that ε_n ϱ_n log A_n = o(1), and ${\overline{\delta }}_n^2{\overline{\varrho }}_n{\varrho }_n\left(\text{log\hspace{0.17em}}{\overline{A}}_n\right)\cdot \left(\text{log\hspace{0.17em}}{A}_n\right)=o\left(1\right)$. Then (1.3) holds uniformly over $P\in {\mathcal{P}}_n$.

Comment 2.3 (Confidence bands based on other bootstrap schemes). Results in [24] suggest that the conditions of Corollary 2.2 can be somewhat relaxed if, instead of using the Gaussian weights in the multiplier bootstrap method, we use Mammen’s weights as in [34] or if we use the empirical bootstrap instead of the multiplier bootstrap. Since the results in [24] apply only to high-dimensional random vectors and do not apply to infinite-dimensional random processes, we leave a formal discussion of the results under these alternative bootstrap schemes to future work.

2.2. Construction of score functions satisfying the orthogonality condition.

Here, we discuss several methods for generating orthogonal scores in a wide variety of settings, including the classical Neyman’s construction. In what follows, since the argument applies to each u and j, it is convenient to omit the indices u and j and also to use the subscript 0 to indicate the true values of the parameters. For simplicity, we also focus the discussion on the exactly orthogonal case. With these simplifications, we can restate the orthogonality condition as follows: we say that the score ψ obeys the Neyman orthogonality condition with respect to ${\eta }_0\in \mathcal{T}$ if the following conditions hold: The Gateaux derivative map

$${\text{D}}_{\overline{r}}\left[\eta -{\eta }_0\right]:={\partial }_r\{{\text{E}}_P[\psi \left(W,{\theta }_0,{\eta }_0+r\left(\eta -{\eta }_0\right)\right]\}{|}_{r=\overline{r}}$$

exists for all $\overline{r}\in [0,1)$ and $\eta \in \mathcal{T}$ and vanishes at $\overline{r}=0$, namely,

$${\text{D}}_{u,j,0}\left[\eta -{\eta }_0\right]=0,$$ (2.9)

for all $\eta \in \mathcal{T}$.

(1) Orthogonal scores for likelihood problems with finite-dimensional nuisance parameters. In likelihood settings with finite-dimensional parameters, the construction of orthogonal equations was proposed by Neyman [37] who used them in construction of his celebrated C(α)-statistic.¹

To describe the construction, suppose that the log-likelihood function associated to observation W is (θ, β) ↦ ℓ(W, θ, β), where $\theta \in \Theta \subset {ℝ}^d$ is the target parameter and $\beta \in T\subset {ℝ}^{p_0}$ is the nuisance parameter. Under regularity conditions, the true parameter values θ₀ and β₀ obey

$${\text{E}}_P\left[{\partial }_{\theta }\ell \left(W,{\theta }_0,{\beta }_0\right)\right]=0,{\text{E}}_P\left[{\partial }_{\beta }\ell \left(W,{\theta }_0,{\beta }_0\right)\right]=0.$$ (2.10)

Now consider the new score function

$$\psi (W,\theta ,\eta )={\partial }_{\theta }\ell (W,\theta ,\beta )-\mu {\partial }_{\beta }\ell (W,\theta ,\beta ),$$ (2.11)

where the nuisance parameter is

$$\eta ={\left({\beta }^{\prime },\text{ vec }{(\mu )}^{\prime }\right)}^{\prime }\in T\times \mathcal{D}\subset {ℝ}^p,p=p_0+d{p}_0,$$

μ is the d × p₀ orthogonalization parameter matrix whose true value μ₀ solves the equation

$$J_{\theta \beta }-\mu J_{\beta \beta }=0\left(\text{ i.e., }{\mu }_0=J_{\theta \beta }{J}_{\beta \beta }^{-1}\right),$$

And

$$J=\left(\begin{array}{cc}{J}_{\theta \theta }& J_{\theta \beta }\\ J_{\beta \theta }& J_{\beta \beta }\end{array}\right)={\partial }_{\left({\theta }^{\prime },{\beta }^{\prime }\right)}{\text{E}}_P\left[{\partial }_{{\left({\theta }^{\prime },{\beta }^{\prime }\right)}^{\prime }}\ell (W,\theta ,\beta )\right]{|}_{\theta ={\theta }_0;\beta ={\beta }_0}.$$

Provided that μ₀ is well defined, we have by (2.10) that E_P[ψ(W, θ₀,η₀)] = 0, where ${\eta }_0={\left({\beta }_0^{\prime },vec{\left({\mu }_0\right)}^{\prime }\right)}^{\prime }$. Moreover, it is trivial to verify that under standard regularity conditions the score function ψ obeys the near orthogonality condition (2.2) exactly (i.e., with C₀ = 0), that is,

$${\partial }_{\eta }{\text{E}}_P\left[\psi \left(W,{\theta }_0,\eta \right)\right]{|}_{\eta ={\eta }_0}=0.$$

Note that in this example, μ₀ not only creates the necessary orthogonality but also creates the efficient score for inference on the main parameter θ, as emphasized by Neyman.

(2) Orthogonal scores for likelihood problems with infinite-dimensional nuisance parameters. The Neyman’s construction can be extended to semi-parametric models, where the nuisance parameter β is a function. In this case, the original score functions (θ, β) ↦ ∂_θℓ(W, θ, β) corresponding to the log-likelihood function (θ, β) ↦ ℓ(W, θ, β) associated to observation W can be transformed into efficient score functions ψ that obey the exact orthogonality condition (2.9) by projecting the original score functions onto the orthocomplement of the tangent space induced by the nuisance parameter β; see Chapter 25 of [44] for a detailed description of this construction. Note that the projection may create additional nuisance parameters, so that the new nuisance parameter η could be of larger dimension than β. Other relevant references include [9, 29, 45] and [11]. The approach is related to Neyman’s construction in the sense that the score ψ arising in this model is actually the Neyman’s score arising in a one-dimensional least favorable parametric subfamily, [42]; see Chapter 25 of [44] for details.

(3) Orthogonal scores for conditional moment problems with infinite-dimensional nuisance parameters. Next, consider a conditional moment restrictions framework studied by Chamberlain [14]. To define the framework, let W, D and V be random vectors in ${ℝ}^{d_W}$, ${ℝ}^{d_D}$ and ${ℝ}^{d_V}$, respectively, with D and V being sub­vectors of W, so that d_D + d_V ≤ d_W. Also, let $\theta \in \Theta \subset {ℝ}^{d_{\theta }}$ be a finite-dimensional parameter whose true value θ₀ is of interest, and let $h:{ℝ}^{d_V}\to {ℝ}^{d_h}$ be a vector­valued functional nuisance parameter, with the true value being $h_0:{ℝ}^{d_V}\to {ℝ}^{d_h}$. The conditional moment restrictions framework assumes that θ₀ and h₀ satisfy the following equation:

$${\text{E}}_P\left[m\left(W,{\theta }_0,h_0(V)\right)|D,V\right]=0,$$ (2.12)

where $m:{ℝ}^{d_W}\times {ℝ}^{d_{\theta }}\times {ℝ}^{d_h}\to {ℝ}^{d_m}$ is some known function. This framework is of interest because it covers an extremely rich variety of models, without having to explicitly rely on the likelihood formulation. For example, it covers the partial linear model

$$Y=D{\theta }_0+h_0(V)+U,{\text{ E}}_P[U|D,V]=0,$$ (2.13)

where Y is a scalar dependent random variable, D is a scalar independent treatment random variable, V is a vector of control random variables and U is a scalar unobservable noise random variable. Indeed, (2.13) implies (2.12) by setting W = (Y, D,V′)′ and m(W, θ, h) = Y − Dθ − h.

Here, we would like to build a (generalized) score function (θ, η) ↦ ψ(W, θ, η) for estimating θ₀, the true value of parameter θ, where η is a new nuisance parameter with true value η₀, that obeys the near orthogonality condition (2.2). There are many ways to do so but one particularly useful way is the following. Consider the functional parameters $\mathrm{\Sigma }:{ℝ}^{d_D+d_V}\to {ℝ}^{d_m\times d_m}$ and $\phi :{ℝ}^{d_D+d_V}\to {ℝ}^{d_{\theta }\times d_m}$ whose true values are given by

$$\begin{array}{l}{\mathrm{\Sigma }}_0(D,V)={\text{E}}_P\left[m\left(W,{\theta }_0,h_0(V)\right)m{\left(W,{\theta }_0,h_0(V)\right)}^{\prime }|D,V\right],\hfill \\ {\phi }_0(D,V)={\left(A_0(D,V)-{\Gamma }_0(D,V)G_0(V)\right)}^{\prime },\hfill \end{array}$$

where

$$\begin{array}{l}{A}_0(D,V)={\partial }_{{\theta }^{\prime }}{E}_P\left[m\left(W,\theta ,h_0(V)\right)|D,V\right]{|}_{\theta ={\theta }_0},\hfill \\ {\mathrm{\Gamma }}_0(D,V)={\partial }_{h^{\prime }}{E}_P\left[m\left(W,{\theta }_0,h\right)|D,V\right]{|}_{h=h_0(V)},\hfill \\ G_0(V)={\left({\text{E}}_P\left[{\mathrm{\Gamma }}_0{(D,V)}^{\prime }{\mathrm{\Sigma }}_0{(D,V)}^{-1}{\mathrm{\Gamma }}_0(D,V)|V\right]\right)}^{-1}\times {\text{E}}_P\left[{\mathrm{\Gamma }}_0{(D,V)}^{\prime }{\mathrm{\Sigma }}_0{(D,V)}^{-1}{A}_0(D,V)|V\right].\hfill \end{array}$$

Then set η = (h, φ, Σ) and η₀ = (h₀,φ₀,Σ₀), and define the score function:

$$\psi (W,\theta ,\eta )=\underset{\text{"instrument"}}{\underbrace{\phi (D,V)}}\underset{\text{ weight }}{\underbrace{\mathrm{\Sigma }{(D,V)}^{-1}}}\underset{\text{ residual }}{\underbrace{m(W,\theta ,h(V))}}.$$

It is rather straightforward to verify that under mild regularity conditions, the score function ψ satisfies the moment condition, E_P[ψ(W, θ₀,η₀)] = 0, and in addition, the orthogonality condition:

$${\partial }_{\eta }{\text{E}}_P\left[\psi \left(W,{\theta }_0,\eta \right)\right]{|}_{\eta ={\eta }_0}=0.$$

Note that this construction gives the efficient score function ψ that yields an estimator of θ₀ achieving the semiparametric efficiency bound, as calculated by Chamberlain [14].

3. Application to logistic regression model with functional response data.

In this section, we apply our main results to a logistic regression model with functional response data described in the Introduction.

3.1. Model.

We consider a response variable $Y\in ℝ$ that induces a functional response ${\left(Y_u\right)}_{u\in \mathcal{U}}$ by $Y_u=1\left\{Y\le (1-u)\underset{\_}{y}+u\overline{y}\right\}$ for a set of indices $\mathcal{U}=[0,1]$ and some constants $\underset{\_}{y}\le \overline{y}$. We are interested in the dependence of this functional response on a $\tilde{p}$-vector of covariates, $D={\left(D_1,\dots ,D_{\tilde{p}}\right)}^{\prime }\in {ℝ}^{\tilde{p}}$, controlling for a p-vector of additional covariates $X={\left(X_1,\dots ,X_p\right)}^{\prime }\in {ℝ}^p$. We allow both $\tilde{p}$ and p to be (much) larger than the sample size of available data, n.

For each $u\in \mathcal{U}$, we assume that Y_u satisfies the generalized linear model with the logistic link function

$${\text{E}}_P\left[Y_u|D,X\right]=\mathrm{\Lambda }\left(D^{\prime }{\theta }_u+X^{\prime }{\beta }_u\right)+r_u,$$ (3.1)

where ${\theta }_{u1}={\left({\theta }_{u1},\dots ,{\theta }_{u\tilde{p}}\right)}^{\prime }$ is a vector of parameters that are of interest, β_u = (βu₁,…,β_up)′ is a vector of nuisance parameters, r_u = r_u(D, X) is an approximation error, $\mathrm{\Lambda }:ℝ\to ℝ$ is the logistic link function defined by

$$\mathrm{\Lambda }(t)=\frac{\text{exp}(t)}{1+\text{exp}(t)},t\in ℝ,$$

and $P\in {\mathcal{P}}_n$ is the distribution of the triple W = (Y,D,X). As in the previous section, we construct simultaneous confidence bands for the parameters ${\left({\theta }_{uj}\right)}_{u\in \mathcal{U},j\in [\tilde{p}]}$ based on a random sample ${\left(W_i\right)}_{i=1}^n={\left(Y_i,D_i,X_i\right)}_{i=1}^n$ from the distribution of W = (Y, D,X).

3.2. Orthogonal score functions.

Before setting up score functions that satisfy both the moment conditions (1.2) and the orthogonality condition (1.4), observe that “naive” score functions that follow directly from the model (3.1),

$$m_{uj}\left(W,{\theta }_{uj},{\theta }_{u[\tilde{p}]\backslash j},{\beta }_u,r_u\right)=\left\{Y_u-\mathrm{\Lambda }\left(D_j\theta +X^j{\left({\theta }_{u[\tilde{p}]\backslash j}^{\prime },{\beta }_u^{\prime }\right)}^{\prime }\right)-r_u\right\}{D}_j,$$

where $X^j=\left(D_{[\tilde{p}]\backslash j}^{\prime },X^{\prime }\right)$, satisfy the moment conditions E_P[m_uj(W,θ_uj)] = 0 but violate the orthogonality condition (1.4) [with m_uj replacing ψ_uj and ${\eta }_{uj}=\left({\theta }_{u[\tilde{p}]\backslash j},{\beta }_u,r_u\right)$]. To satisfy the orthogonality condition (1.4), we proceed using an approach from Section 2.2 as in the Introduction. Specifically, for each $u\in \mathcal{U}$ and $j\in [\tilde{p}]$, define the $(\tilde{p}+p-1)$-vector of additional nuisance parameters ${\gamma }_u^j$ by (1.5) where $f_u^2=f_u^2(D,X)$ is defined in (1.6). Thus, by the first-order condition of (1.5), the nuisance parameters ${\gamma }_u^j$ satisfy

$$f_u{D}_j=f_u{X}^j{\gamma }_u^j+v_u^j,{\text{E}}_P\left[f_u{X}^j{v}_u^j\right]=0.$$ (3.2)

Also, denote ${\beta }_u^j={\left({\theta }_{u[\tilde{p}]\backslash j}^{\prime },{\beta }_u^{\prime }\right)}^{\prime }$. Then we set

$${\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}\right)=\left\{Y_u-\mathrm{\Lambda }\left(D_j{\theta }_{uj}+X^j{\beta }_u^j\right)-r_u\right\}\left(D_j-X^j{\gamma }_u^j\right),$$

where the nuisance parameter is ${\eta }_{uj}=\left(r_u,{\beta }_u^j,{\gamma }_u^j\right)$. As we formally demonstrate in the proof of Theorem 3.1 below, this function satisfies the near-orthogonality condition (1.4).

3.3. Estimation using orthogonal score functions.

Next, we discuss estimation of η_uj’s and θ_uj’s. First, we assume that the approximation error r_u = r_u (D, X) is asymptotically negligible, so that it can be estimated by $\mathcal{O}=\mathcal{O}(D,X)$, the identically zero function of D and X. Second, for ${\gamma }_u^j$, we consider an estimator ${\tilde{\gamma }}_u^j$ defined as a post-regularization weighted least squares estimator corresponding to the problem (1.5). Third, for ${\beta }_u^j$, we consider a plug-in estimator ${\widehat{\beta }}_u^j=\left({\tilde{\theta }}_{u[\tilde{p}]\backslash j}^{\prime },{\tilde{\beta }}_u^{\prime }\right)$, where ${\tilde{\theta }}_u$ and ${\tilde{\beta }}_u$ are suitable estimators of θ_u and β_u. In particular, we assume that ${\tilde{\theta }}_u$ and ${\tilde{\beta }}_u$ are post-regularization maximum likelihood estimators corresponding to the log-likelihood function (θ, β) ↦ −M_u (W, θ, β) where

$$M_u(W,\theta ,\beta )=-\left(1\left\{Y_u=1\right\}\text{\hspace{0.17em}log\hspace{0.17em}}\mathrm{\Lambda }\left(D^{\prime }\theta +X^{\prime }\beta \right)+1\left\{Y_u=0\right\}\text{\hspace{0.17em}log\hspace{0.17em}}\left(1-\mathrm{\Lambda }\left(D^{\prime }\theta +X^{\prime }\beta \right)\right)\right).$$ (3.3)

The details of the estimators ${\tilde{\theta }}_u$, ${\tilde{\beta }}_u$ and ${\tilde{\gamma }}_u^j$ are given in Algorithm 1 below. The results in this paper can also be easily extended to the case where ${\tilde{\theta }}_u$, ${\tilde{\beta }}_u$ and ${\tilde{\gamma }}_u^j$ are replaced by penalized maximum likelihood estimators ${\widehat{\theta }}_u$ and ${\widehat{\beta }}_u$ and penalized weighted least squares estimator ${\widehat{\gamma }}_u^j$, respectively.

Then our estimator of η_uj is ${\widehat{\eta }}_{uj}=\left(\mathcal{O},{\widehat{\beta }}_u^j,{\tilde{\gamma }}_u^j\right)$. Substituting this estimator into the score function ψ_uj gives

$${\psi }_{uj}\left(W,{\theta }_{uj},{\widehat{\eta }}_{uj}\right)=\left\{Y_u-\mathrm{\Lambda }\left(D_j{\theta }_{uj}+X^j{\widehat{\beta }}_u^j\right)\right\}\left(D_j-X^j{\tilde{\gamma }}_u^j\right),$$ (3.4)

which, using the sample analog (2.1) of the moment conditions (1.2), gives the following estimator of θ_uj:

$${\check{\theta }}_{uj}\in \text{arg}\underset{\theta \in {\Theta }_{uj}}{\text{inf}}\left|{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,\theta ,{\widehat{\eta }}_{uj}\right)\right]\right|.$$ (3.5)

The algorithm is summarized as follows.

ALGORITHM 1. For each $u\in \mathcal{U}$ and $j\in [\tilde{p}]$:

• Step 1. Run post-ℓ₁-penalized logistic estimator (4.2) of Y_u on D and X to compute $\left({\tilde{\theta }}_u,{\tilde{\beta }}_u\right)$.

• Step 2. Define the weights ${\widehat{f}}_u^2={\widehat{f}}_u^2(D,X)={\mathrm{\Lambda }}^{\prime }\left(D^{\prime }{\tilde{\theta }}_u+X^{\prime }{\tilde{\beta }}_u\right)$.

• Step 3. Run the Post-Lasso estimator (4.5) of $\widehat{f_u}{D}_j$ to compute ${\tilde{\gamma }}_u^j$.

• Step 4. Compute ${\widehat{\beta }}_u^j={\left({\tilde{\theta }}_{u[\tilde{p}]\backslash j}^{\prime },{\tilde{\beta }}_u^{\prime }\right)}^{\prime }$.

• Step 5. Solve (3.5) with ${\psi }_{uj}\left(W,\theta ,{\widehat{\eta }}_{uj}\right)$ defined in (3.4) to compute ${\check{\theta }}_{uj}$.

3.4. Regularity conditions.

Next, we specify our regularity conditions. For all $u\in \mathcal{U}$ and $j\in [\tilde{p}]$, denote $Z_u^j=D_j-X^j{\gamma }_u^j$. Also, denote $a_n=p\vee \tilde{p}\vee n$. Let q, c₁ and C₁ be some strictly positive (and finite) constants where q > 4. Moreover, let (δ_n)_n≥1 and (Δ_n)_n≥1 be some sequences of positive constants converging to zero. Finally, let (M_n,1)_n≥1 and (M_n,2)_n≥1 be some sequences of positive constants, possibly growing to infinity, where M_n,1 ≥ 1 and M_n,2 ≥ 1 for all n.

Assumption 3.1 (Parameters). For all $u\in \mathcal{U}$, we have $\Vert {\theta }_u\Vert +\Vert {\beta }_u\Vert +{\text{max}}_{j\in [\tilde{p}]}\Vert {\gamma }_u^j\Vert \le C_1$ and ${\text{max}}_{j\in [\tilde{p}]}{\text{\hspace{0.17em}sup}}_{\theta \in {\Theta }_{uj}}\left|\theta \right|\le C_1$. In addition, for all $u_1,u_2\in \mathcal{U}$, we have $\left(\Vert {\theta }_{u_2}-{\theta }_{u_1}\Vert +\Vert {\beta }_{u_2}-{\beta }_{u_1}\Vert \right)\le C_1\left|u_2-u_1\right|$. Finally, for all $u\in \mathcal{U}$ and $j\in [\tilde{p}]$, Θ_uj contains a ball of radius (log log n) (log a_n) ^3/2/n^1/2 centered at θ_uj.

Assumption 3.2 (Sparsity). There exist s = s_n and ${\overline{\gamma }}_u^j$, $u\in \mathcal{U}$ and $j\in [\tilde{p}]$, such that for all $u\in \mathcal{U}$, $\Vert {\beta }_u{\Vert }_0+\Vert {\theta }_u{\Vert }_0+{\text{max}}_{j\in [\tilde{p}]}\Vert {\overline{\gamma }}_u^j{\Vert }_0\le s_n$ and ${\text{max}}_{j\in [\tilde{p}]}\left(\Vert {\overline{\gamma }}_u^j-{\gamma }_u^j\Vert +s_n^{-1/2}\Vert {\overline{\gamma }}_u^j-{\gamma }_u^j{\Vert }_1\right)\le C_1{\left(s_n\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n/n\right)}^{1/2}$.

Assumption 3.3 (Distribution of Y). The conditional pdf of Y given (D, X) is bounded by C₁.

Assumptions 3.1–3.3 are mild and standard in the literature. In particular, Assumption 3.1 requires the parameter spaces Θ_uj to be bounded, and also requires that for each $u\in \mathcal{U}$ and $j\in [\tilde{p}]$, the parameter θ_uj to be sufficiently separated from the boundaries of the parameter space Θ_uj. Assumption 3.2 requires approximate sparsity of the model (3.1). Note that in Assumption 3.2, given that ${\overline{\gamma }}_u^j$’s exist, we can and will assume without loss of generality that ${\overline{\gamma }}_u^j={\gamma }_{uT}^j$ for some $T\subset \left\{1,\dots ,p+\tilde{p}-1\right\}$ with |T| ≤ s_n, where $T=T_u^j$ is allowed to depend on u and j. Here, the $(p+\tilde{p}-1)$-vector ${\gamma }_{uT}^j$ is defined from ${\gamma }_u^j$ by keeping all components of ${\gamma }_u^j$ that are in T and setting all other components to be zero. Assumption 3.3 can be relaxed at the expense of more technicalities.

Assumption 3.4 (Covariates). For all $u\in \mathcal{U}$, the following inequalities hold: (i) ${\text{inf}}_{\Vert \xi \Vert =1}{\text{E}}_P\left[f_u^2{\left\{\left(D^{\prime },X^{\prime }\right)\xi \right\}}^2\right]\ge c_1$, (ii) ${\text{min}}_{j,k}\left({\text{E}}_P\left[{\left|f_u^2{Z}_u^j{X}_k^j\right|}^2\right]\wedge {\text{E}}_P\left[{\left|f_u^2{D}_j{X}_k^j\right|}^2\right]\right)\ge c_1$, and (iii) ${\text{max}}_{j,k}{\text{E}}_P{\left[{\left|Z_u^j{X}_k^j\right|}^3\right]}^{1/3}{\text{\hspace{0.17em}log}}^{1/2}{a}_n\le {\delta }_n{n}^{1/6}$. In addition, we have that (iv) ${\text{sup}}_{\Vert \xi \Vert =1}{\text{E}}_P\left[{\left\{\left(D^{\prime },X^{\prime }\right)\xi \right\}}^4\right]\le C_1$, (v) $M_{n,1}\ge {\text{E}}_P{\left[{\text{sup}}_{u\in \mathcal{U},j\in [\tilde{p}]}{\left|Z_u^j\right|}^{2q}\right]}^{1/(2q)}$ (vi) $M_{n,1}^2{s}_n\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n\le {\delta }_n{n}^{1/2-1/q}$, (vii) $M_{n,2}\ge {\left\{{\text{E}}_P\left[{\left(\Vert D{\Vert }_{\infty }\vee \Vert X{\Vert }_{\infty }\right)}^{2q}\right]\right\}}^{1/(2q)}$, (viii) $M_{n,2}^2{s}_n{\text{\hspace{0.17em}log\hspace{0.17em}}}^{1/2}{a}_n\le {\delta }_n{n}^{1/2-1/q}$ and (ix) $M_{n,1}^2{M}_{n,2}^4{s}_n\le {\delta }_n{n}^{1-3/q}$.

This assumption requires that there is no multicollinearity between covariates in vectors D and X. In addition, it requires that the constants $\underset{\_}{y}$ and $\overline{y}$ are chosen so that the probabilities of $Y<\overline{y}$ and $Y>\underset{\_}{y}$ are both nonvanishing since otherwise we would have $\text{E}\left[f_u^2\right]=\text{E}\left[{\text{Var}}_P\left(Y_u|D,X\right)\right]$ vanishing either for u = 0 or u = 1 violating Assumption 3.4(i). Intuitively, sending $\underset{\_}{y}$ and $\overline{y}$ to the left and to the right tails of the distribution of Y, respectively, would blow up the variance of the estimators ${\check{\theta }}_{uj}$, given by ${\sigma }_{uj}^2$ in Theorem 2.1, and leading eventually to the estimators with slower-than-$\sqrt{n}$ rate of convergence. Although our results could be extended to allow for the case where $\underset{\_}{y}$ and $\overline{y}$ are sent to the tails of the distribution of Y slowly, we skip this extension for the sake of clarity. Moreover, Assumption 3.4 imposes constraints on various moments of covariates. Since these constraints might be difficult to grasp, at the end of this section, in Corollary 3.3, we provide an example for which these constraints simplify into easily interpretable conditions.

Assumption 3.5 (Approximation error). For all $u\in \mathcal{U}$, we have (i) ${\text{sup}}_{\Vert \xi \Vert =1}{\text{E}}_P\left[r_u^2{\left\{\left(D^{\prime },X^{\prime }\right)\xi \right\}}^2\right]\le C_1{\text{E}}_P\left[r_u^2\right]$, (ii) ${\text{E}}_P\left[r_u^2\right]\le C_1{s}_n\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n/n$, (iii) ${\text{max}}_{j\in [\tilde{p}]}\left|E_P\left[r_u{Z}_u^j\right]\right|\le {\delta }_n{n}^{-1/2}$, and (iv) $\left|r_u(D,X)\right|\le f_u^2(D,X)/4$ almost surely. In addition, with probability $1-{\overline{\mathrm{\Delta }}}_n$, (v) ${\text{sup}}_{u\in \mathcal{U},j\in [\tilde{p}]}\left({\mathbb{E}}_n\left[{\left(r_u{Z}_u^j/f_u\right)}^2\right]+{\mathbb{E}}_n\left[r_u^2/f_u^6\right]\right)\le C_1{s}_n\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n/n$.

This assumption requires the approximation error r_u = r_u(D, X) to be sufficiently small. Under Assumption 3.4, the first condition of Assumption 3.5 holds if the approximation error is such that $r_u^2\le C{\text{E}}_P\left[r_u^2\right]$ almost surely for some constant C.

3.5. Formal results.

Under specified assumptions, our estimators ${\check{\theta }}_{uj}$ satisfy the following uniform Bahadur representation theorem.

Theorem 3. 1 (Uniform Bahadur representation for logistic model). Suppose that Assumptions 3.1–3.5 hold for all $P\in {\mathcal{P}}_n$. In addition, suppose that the following growth condition holds: ${\delta }_n^2\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n=o(1)$. Then for the estimators ${\check{\theta }}_{uj}$ satisfying (3.5), we have

$$\sqrt{n}{\sigma }_{uj}^{-1}\left({\check{\theta }}_{uj}-{\theta }_{uj}\right)={\mathbb{G}}_n{\overline{\psi }}_{uj}+O_P\left({\delta }_n\right)$$ (3.6)

in ${\ell }^{\infty }(\mathcal{U}\times [\tilde{p}]),$ uniformly over $P\in {\mathcal{P}}_n$, where ${\overline{\psi }}_{uj}(W):=-{\sigma }_{uj}^{-1}{J}_{uj}^{-1}{\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}\right)$, ${\sigma }_{uj}^2:={\text{E}}_P\left[J_{uj}^{-2}{\psi }_{uj}^2\left(W,{\theta }_{uj},{\eta }_{uj}\right)\right]$, and J_uj is defined in (2.3).

This theorem allows us to establish a Gaussian approximation result for the supremum of the process $\left\{\sqrt{n}{\sigma }_{uj}^{-1}\left({\check{\theta }}_{uj}-{\theta }_{uj}\right):u\in \mathcal{U},j\in [\tilde{p}]\right\}$:

Corollary 3.1 (Gaussian approximation for logistic model). Suppose that Assumptions 3.1–3.5 hold for all $P\in {\mathcal{P}}_n$. In addition, suppose that the following growth conditions hold: ${\delta }_n^2\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n=o(1)$, $M_{n,1}^{2/7}\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n=o\left(n^{1/7}\right)$ and $M_{n,1}^{2/3}\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n=o\left(n^{1/3-2/(3q)}\right)$. Then

$$\underset{t\in ℝ}{\text{sup}}\left|{\text{P}}_P\left(\underset{u\in \mathcal{U},j\in [\tilde{p}]}{\text{sup}}\left|\sqrt{n}{\sigma }_{uj}^{-1}\left({\check{\theta }}_{uj}-{\theta }_{uj}\right)\right|\le t\right)-{\text{P}}_P\left(\underset{u\in \mathcal{U},j\in [\tilde{p}]}{\text{sup}}\left|{\mathcal{N}}_{uj}\right|\le t\right)\right|=o(1)$$

uniformly over $P\in {\mathcal{P}}_n$, where ${\left({\mathcal{N}}_{uj}\right)}_{u\in \mathcal{U},j\in [\tilde{p}]}$ is a tight zero-mean Gaussian process indexed by $\mathcal{U}\times [\tilde{p}]$ with the covariance given by ${\text{E}}_P\left[{\overline{\psi }}_{uj}(W){\overline{\psi }}_{u^{\prime }{j}^{\prime }}(W)\right]$ for u, $u^{\prime }\in \mathcal{U}$ and j, $j,j^{\prime }\in [\tilde{p}]$.

Based on this corollary, we are now able to construct simultaneous confidence bands for the parameters θ_uj. Observe that

$$J_{uj}=-{\text{E}}_P\left[{\mathrm{\Lambda }}^{\prime }\left(D_j{\theta }_{uj}+X^j{\beta }_u^j\right)D_j\left(D_j-X^j{\gamma }_u^j\right)\right],u\in \mathcal{U},j\in [\tilde{p}],$$

and so it can be estimated by

$${\widehat{J}}_{uj}=-{\mathbb{E}}_n\left[{\mathrm{\Lambda }}^{\prime }\left(D_j{\tilde{\theta }}_{uj}+X^j{\widehat{\beta }}_u^j\right)D_j\left(D_j-X^j{\tilde{\gamma }}_u^j\right)\right],u\in \mathcal{U},j\in [\tilde{p}].$$

In addition, ${\sigma }_{uj}^2={\text{E}}_P\left[J_{uj}^{-2}{\psi }_{uj}^2\left(W,{\theta }_{uj},{\eta }_{uj}\right)\right]$, and so it can be estimated by

$${\widehat{\sigma }}_{uj}^2={\mathbb{E}}_n\left[{\widehat{J}}_{uj}^{-2}{\psi }_{uj}^2\left(W,{\tilde{\theta }}_{uj},{\widehat{\eta }}_{uj}\right)\right],u\in \mathcal{U},j\in [\tilde{p}].$$

Moreover, as in Section 2, define ${\widehat{\psi }}_{uj}(W)=-{\widehat{\sigma }}_{uj}^{-1}{\widehat{J}}_{uj}^{-1}{\psi }_{uj}\left(W,{\check{\theta }}_{uj},{\widehat{\eta }}_{uj}\right)$, and let c_α be the (1 – α) quantile of the conditional distribution of ${\text{sup}}_{u\in \mathcal{U},j\in [\tilde{p}]}\left|{\widehat{\mathcal{G}}}_{uj}\right|$ given the data ${\left(W_i\right)}_{i=1}^n$ where the process $\widehat{\mathcal{G}}={\left({\widehat{\mathcal{G}}}_{uj}\right)}_{u\in \mathcal{U},j\in [\tilde{p}]}$ is defined in (2.8). Then we have the following.

Corollary 3.2 (Simultaneous confidence bands for logistic model). Suppose that Assumptions 3.1–3.5 hold for all $P\in {\mathcal{P}}_n$. In addition, suppose that the following growth conditions hold: ${\delta }_n^2\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n=o(1)$, $M_{n,1}^{2/7}\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n=o\left(n^{1/7}\right)$, $M_{n,1}^{2/3}\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n=o\left(n^{1/3-2/(3q)}\right)$ and s_n log³ a_n = o(n). Then (1.3) holds uniformly over $P\in {\mathcal{P}}_n$.

To conclude this section, we provide an example for which conditions of Corollary 3.2 are easy to interpret. Recall that $a_n=n\vee p\vee \tilde{p}$.

Corollary 3.3 (Uniform confidence bands for logistic regression model under simple conditions). Suppose that Assumptions 3.1–3.3, 3.4(i,ii,iv) and 3.5(i,ii,iv,v) hold for q > 4for all $P\in {\mathcal{P}}_n$. In addition, suppose that ${\left\{{\text{E}}_P\left[{\left(\Vert D{\Vert }_{\infty }\vee \Vert X{\Vert }_{\infty }\right)}^{2q}\right]\right\}}^{1/(2q)}\le C_1$ and ${\text{sup}}_{u\in \mathcal{U},j\in [\tilde{p}]}\Vert {\gamma }_u^j{\Vert }_1\le C_1$. Moreover, suppose that the following growth conditions hold: log⁷ a_n /n = o(1), $s_n^2{\text{\hspace{0.17em}log\hspace{0.17em}}}^3{a}_n/n^{1-2/q}=o(1)$ and ${\text{sup}}_{u\in \mathcal{U},j\in [\tilde{p}]}\left|{\text{E}}_P\left[r_u{Z}_u^J\right]\right|=o\left({\left(n\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n\right)}^{-1/2}\right)$. Then (1.3) holds uniformly over $P\in {\mathcal{P}}_n$.

Comment 3.1 (Estimation of variance). When constructing the confidence bands based on (1.3), we find in simulations that it is beneficial to replace the estimators ${\widehat{\sigma }}_{uj}^2$ of ${\sigma }_{uj}^2$ by max $\left\{{\widehat{\sigma }}_{uj}^2,{\widehat{\mathrm{\Sigma }}}_{uj}^2\right\}$ where ${\widehat{\mathrm{\Sigma }}}_{uj}^2={\mathbb{E}}_n\left[{\widehat{f}}_u^2{\left(D-X^j{\tilde{\gamma }}_u^j\right)}^2\right]$ is an alternative consistent estimator of ${\sigma }_{uj}^2$.

Comment 3.2 (Alternative implementations, double selection). We note that the theory developed here is applicable for different estimators that construct the new score function with the desired orthogonality condition implicitly. For example, the double selection idea yields an implementation of an estimator that is first-order equivalent to the estimator based on the score function. The algorithm yielding the double selection estimator is as follows.

ALGORITHM 2. For each $u\in \mathcal{U}$ and $j\in [\tilde{p}]$:

• Step 1′. Run post-ℓ₁-penalized logistic estimator (4.2) of Y_u on D and X to compute $\left({\tilde{\theta }}_u,{\tilde{\beta }}_u\right)$.

• Step 2′. Define the weights ${\widehat{f}}_u^2={\widehat{f}}_u^2(D,X)={\mathrm{\Lambda }}^{\prime }\left(D_i^{\prime }{\tilde{\theta }}_u+X_i^{\prime }{\tilde{\beta }}_u\right)$.

• Step 3′. Run the Lasso estimator (4.4) of $\widehat{f_u}{D}_j$ on $\widehat{f_u}X$ to compute ${\widehat{\gamma }}_u^j$.

• Step 4′. Run logistic regression of Y_u on D_j and all the selected variables in Steps 1′ and 3′ to compute ${\check{\theta }}_{uj}$.

As mentioned by a referee, it is surprising that the double selection procedure has uniform validity. The use of the additional variables selected in Step 3′, through the first-order conditions of the optimization problem, induces the necessary near­orthogonality condition. We refer to the Supplementary Material for a more detailed discussion.

Comment 3.3 (Alternative implementations, one-step correction). Another implementation for which the theory developed here applies is to replace Step 5 in Algorithm 1 with a one-step procedure. This relates to the debiasing procedure proposed in [43] to the case when the set $\mathcal{U}$ is a singleton. In this case, instead of minimizing the criterion (3.5) in Step 5, the method makes a full Newton step from the initial estimate,

• Step 5″. Compute ${\overline{\theta }}_{uj}={\widehat{\theta }}_{uj}-{\widehat{J}}_{uj}^{-1}{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,{\widehat{\theta }}_{uj},{\widehat{\eta }}_{uj}\right)\right]$.

The theory developed here directly apply to those estimators as well.

Comment 3.4 (Extension to other approximately sparse generalized linear models). Inspecting the proofs of Theorem 3.1 and Corollaries 3.1–3.3 reveal that these results can be extended with minor modifications to cover other approximately sparse generalized linear models. For example, the results can be extended to cover the model (3.1) where we use the probit link function instead of the logit link function Λ.

4. ℓ₁-Penalized M-estimators: Nuisance functions and functional data.

In this section, we define the estimators ${\tilde{\theta }}_u$, ${\tilde{\beta }}_u$ and ${\tilde{\gamma }}_u^j$, which were used in the previous section, and study their properties. We consider the same setting as that in the previous section. The results in this section rely upon a set of new results for ℓ₁-penalized M-estimators with functional data presented in Appendix M of the Supplementary Material.

4.1. ℓ₁-Penalized logistic regression for functional response data: Asymptotic properties.

Here, we consider the generalized linear model with the logistic link function and functional response data (3.1). As explained in the previous section, we assume that ${\tilde{\theta }}_u$ and ${\tilde{\beta }}_u$ are post-regularization maximum likelihood estimators of θ_u and β_u corresponding to the log-likelihood function M_u(W, θ, β) =M_u(Y_u, D, X, θ, ß) defined in (3.3). To define these estimators, let ${\widehat{\theta }}_u$ and ${\widehat{\beta }}_u$ be ℓ_l-penalized maximum likelihood (logistic regression) estimators

$$\left({\widehat{\theta }}_u,{\widehat{\beta }}_u\right)\in \text{arg}\underset{\theta ,\beta }{\text{min}}\left({\mathbb{E}}_n\left[M_u\left(Y_u,D,X,\theta ,\beta \right)\right]+\frac{\lambda }{n}\Vert {\widehat{\Psi }}_u{\left({\theta }^{\prime },{\beta }^{\prime }\right)}^{\prime }{\Vert }_1\right),$$ (4.1)

where λ is a penalty level and ${\widehat{\Psi }}_u$ a diagonal matrix of penalty loadings. We choose parameters λ and ${\widehat{\Psi }}_u$ according to Algorithm 3 described below. Using the ℓ₁ - penalized estimators ${\widehat{\theta }}_u$ and ${\widehat{\beta }}_u$, we then define post-regularization estimators ${\tilde{\theta }}_u$ and ${\tilde{\beta }}_u$ by

$$\left({\tilde{\theta }}_u,{\tilde{\beta }}_u\right)\in \text{arg}\underset{\theta }{\text{min}}{\mathbb{E}}_n\left[M_u\left(Y_u,D,X,\theta ,\beta \right)\right]:supp(\theta ,\beta )\subseteq supp\left({\widehat{\theta }}_u,{\widehat{\beta }}_u\right).$$ (4.2)

We derive the rate of convergence and sparsity properties of ${\tilde{\theta }}_u$ and ${\tilde{\beta }}_u$ as well as of ${\widehat{\theta }}_u$ and ${\widehat{\beta }}_u$ in Theorem 4.1 below. Recall that $a_n=n\vee p\vee \tilde{p}$.

Algorithm 3 (Penalty level and loadings for logistic regression). Choose γ ∈ [1/n, 1/log n] and c > 1 (in practice, we set c = 1.1 and γ = 0.1/log n). Define $\lambda =c\sqrt{n}{\Phi }^{-1}\left(1-\gamma /\left(2(p+\tilde{p})N_n\right)\right)$ with N_n = n. To select ${\widehat{\Psi }}_u$, choose a constant $\overline{m}\ge 0$ as an upper bound on the number of loops and proceed as follows: (0) Let $\tilde{X}={\left(D^{\prime },X^{\prime }\right)}^{\prime }$, m = 0 and initialize ${\widehat{l}}_{uk,0}=\frac{1}{2}{\left\{{\mathbb{E}}_n\left[{\tilde{X}}_k^2\right]\right\}}^{1/2}$ for $k\in \left[p+\tilde{p}\right]$. (1) Compute $\left({\widehat{\theta }}_u,{\widehat{\beta }}_u\right)$ and $\left({\tilde{\theta }}_u,{\tilde{\beta }}_u\right)$ based on ${\widehat{\Psi }}_u{}_u=diag\left(\left\{{\widehat{l}}_{uk,m},k\in [p+\tilde{p}]\right\}\right)$. (2) Set ${\widehat{l}}_{uk,m+1}:={\left\{{\mathbb{E}}_n\left[{\tilde{X}}_k^2{\left(Y_u-\mathrm{\Lambda }\left(D^{\prime }{\tilde{\theta }}_u+X^{\prime }{\tilde{\beta }}_u\right)\right)}^2\right]\right\}}^{1/2}$. (3) If $m\ge \overline{m}$, report the current value of ${\widehat{\Psi }}_u{}_u$ and stop; otherwise set m ← m + 1 and go to step (1).

Theorem 4.1 (Rates and sparsity for functional response under logistic link). Suppose that Assumptions 3.1–3.5 hold for all $P\in {\mathcal{P}}_n$. In addition, suppose that the penalty level λ and the matrices of penalty loadings ${\widehat{\Psi }}_u{}_u$ are chosen according to Algorithm 3. Moreover, suppose that the following growth condition holds: ${\delta }_n^2\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n=o(1)$. Then there exists a constant $\overline{C}$ such that uniformly over all $P\in {\mathcal{P}}_n$ with probability 1 − o(1),

$$\begin{array}{l}\underset{u\in \mathcal{U}}{\text{sup}}\left(\Vert {\widehat{\theta }}_u-{\theta }_u\Vert +\Vert {\widehat{\beta }}_u-{\beta }_u\Vert \right)\le \overline{C}\sqrt{\frac{s_n\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n}{n}},\hfill \\ \underset{u\in \mathcal{U}}{\text{sup}}\left(\Vert {\widehat{\theta }}_u-{\theta }_u{\Vert }_1+\Vert {\widehat{\beta }}_u-{\beta }_u{\Vert }_1\right)\le \overline{C}\sqrt{\frac{s_n^2\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n}{n}},\hfill \end{array}$$

and the estimators ${\widehat{\theta }}_u$ and ${\widehat{\beta }}_u$ are uniformly sparse: ${\text{sup}}_{u\in \mathcal{U}}\Vert {\widehat{\theta }}_u{\Vert }_0+\Vert {\widehat{\beta }}_u{\Vert }_0\le \overline{C}{s}_n$. Also, uniformly overall $P\in {\mathcal{P}}_n$, with probability 1 − o(1),

$$\begin{array}{l}\underset{u\in \mathcal{U}}{\text{sup}}\left(\Vert {\tilde{\theta }}_u-{\theta }_u\Vert +\Vert {\tilde{\beta }}_u-{\beta }_u\Vert \right)\le \overline{C}\sqrt{\frac{s_n\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n}{n}},\hfill \\ \underset{u\in \mathcal{U}}{\text{sup}}\left(\Vert {\tilde{\theta }}_u-{\theta }_u{\Vert }_1+\Vert {\tilde{\beta }}_u-{\beta }_u{\Vert }_1\right)\le \overline{C}\sqrt{\frac{s_n^2\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n}{n}}.\hfill \end{array}$$

4.2. Lasso with estimated weights: Asymptotic properties.

Here, we consider the weighted linear model (3.2) for $u\in \mathcal{U}$ and $j\in [\tilde{p}]$. Using the parameter ${\overline{\gamma }}_u^j$ appearing in Assumption 3.2, it will be convenient to rewrite this model as

$$f_u{D}_j=f_u{X}^j{\overline{\gamma }}_u^j+f_u{\overline{r}}_{uj}+v_u^j,{\text{E}}_P\left[f_u{X}^j{v}_u^j\right]=0,$$ (4.3)

where ${\overline{r}}_{uj}=X^j\left({\gamma }_u^j-{\overline{\gamma }}_u^j\right)$ is an approximation error, which is asymptotically negligible under Assumption 3.2. As explained in the previous section, we assume that ${\tilde{\gamma }}_u^j$ is a post-regularization weighted least squares estimator of ${\gamma }_u^j$ (or ${\overline{\gamma }}_u^j$). To define this estimator, let ${\widehat{\gamma }}_u^j$ be an ℓ₁-penalized (weighted Lasso) estimator

$${\widehat{\gamma }}_u^j\in \text{arg\hspace{0.17em}}\underset{\gamma }{\text{min}}\left(\frac{1}{2}{\mathbb{E}}_n\left[{\widehat{f}}_u^2{\left(D_j-X^j\gamma \right)}^2\right]+\frac{\lambda }{n}\Vert {\widehat{\Psi }}_{uj}\gamma {\Vert }_1\right),$$ (4.4)

where λ and ${\widehat{\Psi }}_u{}_{uj}$ are the associated penalty level and the diagonal matrix of penalty loadings specified below in Algorithm 4 and where ${\widehat{f}}_u^2$’s are estimated weights. As in Algorithm 1 in the previous section, we set ${\widehat{f}}_u^2={\widehat{f}}_u^2(D,X)={\mathrm{\Lambda }}^{\prime }\left(D^{\prime }{\tilde{\theta }}_u+X^{\prime }{\tilde{\beta }}_u\right)$. Using ${\widehat{\gamma }}_u^j$, we define a post-regularized weighted least squares estimator:

$${\tilde{\gamma }}_u^j\in \text{arg\hspace{0.17em}}\underset{\gamma }{\text{min}}\frac{1}{2}{\mathbb{E}}_n\left[{\widehat{f}}_u^2{\left(D_j-X^j\gamma \right)}^2\right]:\text{supp}(\gamma )\subseteq \text{supp}\left({\widehat{\gamma }}_u^j\right).$$ (4.5)

We derive the rate of convergence and sparsity properties of ${\tilde{\gamma }}_u^j$ as well as of ${\widehat{\gamma }}_u^j$ in Theorem 4.2 below.

Algorithm 4 (Penalty level and loadings for weighted Lasso). Choose γ ∈ [1/n, 1/log n] and c > 1 (in practice, we set c = 1.1 and γ = 0.1/log n). Define $\lambda =c\sqrt{n}{\Phi }^{-1}\left(1-\gamma /\left(2(p+\tilde{p})N_n\right)\right)$ with $N_n=p{\tilde{p}}^2{n}^2$. To select ${\widehat{\Psi }}_u{}_{uj}$, choose a constant $\overline{m}\ge 1$ as an upper bound on the number of loops and proceed as follows: (0) Set m = 0 and ${\widehat{l}}_{ujk,0}={\text{max}}_{1\le i\le n}\Vert {\widehat{f}}_{ui}{X}_i^j{\Vert }_{\infty }{\left\{{\mathbb{E}}_n\left[{\widehat{f}}_u^2{D}_j^2\right]\right\}}^{1/2}$. (1) Compute ${\widehat{\gamma }}_u^j$ and ${\tilde{\gamma }}_u^j$ based on ${\widehat{\Psi }}_{uj}=diag\left(\left\{{\widehat{l}}_u{}_{jk,m},k\in \left[p+\tilde{p}-1\right]\right\}\right)$. (2) Set ${\widehat{l}}_{ujk,m+1}:={\left\{{\mathbb{E}}_n\left[{\widehat{f}}_u^4{\left(D_j-X^j{\tilde{\gamma }}_u^j\right)}^2{\left(X_k^j\right)}^2\right]\right\}}^{1/2}$. (3) If $m\ge \overline{m}$, report the current value of ${\widehat{\Psi }}_u{}_{uj}$ and stop; otherwise set m ← m + 1 and go to step (1).

Theorem 4.2 (Rates and sparsity for Lasso with estimated weights). Suppose that Assumptions 3.1–3.5 hold for all $P\in {\mathcal{P}}_n$. In addition, suppose that the penalty level λ and the matrices of penalty loadings ${\widehat{\Psi }}_{uj}$ are chosen according to Algorithm 4. Moreover, suppose that the following growth condition holds: ${\delta }_n^2\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n=o(1)$. Then there exists a constant $\overline{C}$ such that uniformly over all $P\in {\mathcal{P}}_n$ with probability 1 − o(1),

$$\underset{j\in [\tilde{p}]}{\text{max\hspace{0.17em}}}\underset{u\in \mathcal{U}}{\text{sup}}\Vert {\widehat{\gamma }}_u^j-{\overline{\gamma }}_u^j\Vert \le \overline{C}\sqrt{\frac{s_n\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n}{n}},\underset{j\in [\tilde{p}]}{\text{max\hspace{0.17em}}}\underset{u\in \mathcal{U}}{\text{sup}}\Vert {\widehat{\gamma }}_u^j-{\overline{\gamma }}_u^j{\Vert }_1\le \overline{C}\sqrt{\frac{s_n^2\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n}{n}},$$

and the estimator ${\widehat{\gamma }}_u^j$ is uniformly sparse, ${\text{max\hspace{0.17em}}}_{j\in [\tilde{p}]}{\text{sup}}_{u\in \mathcal{U}}\Vert {\widehat{\gamma }}_u^j{\Vert }_0\le \overline{C}{s}_n$. Also, uniformly over all $P\in {\mathcal{P}}_n$, with probability 1 − o(1),

$$\underset{j\in [\tilde{p}]}{\text{max\hspace{0.17em}}}\underset{u\in \mathcal{U}}{\text{sup}}\Vert {\tilde{\gamma }}_u^j-{\overline{\gamma }}_u^j\Vert \le \overline{C}\sqrt{\frac{s_n\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n}{n}},\underset{j\in [\tilde{p}]}{\text{max\hspace{0.17em}}}\underset{u\in \mathcal{U}}{\text{sup}}{\Vert {\tilde{\gamma }}_u^j-{\overline{\gamma }}_u^j\Vert }_1\le \overline{C}\sqrt{\frac{s_n^2\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n}{n}}.$$

5. Monte Carlo simulations.

Here, we investigate the finite sample properties of the proposed estimators and the associated confidence regions. We report only the performance of the estimator based on the double selection procedure due to space constraints and note that it is very similar to the performance of the estimator based on score functions with near-orthogonality property. We will compare the proposed procedure with the traditional estimator that refits the model selected by the corresponding ℓ₁-penalized M-estimator (naive post-selection estimator).

We consider a logistic regression model with functional response data where the response Y_u = 1{y ≤ u} for $u\in \mathcal{U}$ a compact set. We specify two different designs: (1) a location model, y = x′ β₀ + ξ, where ξ is distributed as a logistic random variable, the first component of x is the intercept and the other p − 1 components are distributed as N (0, Σ) with Έ_k,_j = |0.5|^|k−j|; (2) a location-shift model, y = {(x′ β₀ + ξ)/x′ϑ₀}³, where ξ is distributed as alogistic random variable, x_j = |w_j| where w is a p-vector distributed as N (0, Σ) with Σ_k,j = |0.5|^|k−j|, and ϑ₀ has nonnegative components. Such specification implies that for each $u\in \mathcal{U}$:

Design1: θ_u = u(1,0,…,0)′ − β₀ and Design 2: θ_u = u^1/3ϑ₀ − β₀. In our simulations, we will consider n = 500 and p = 2000. For the location model (Design 1), we will consider two different choices for β₀: (i)${\beta }_{0j}^{(i)}=2/j^2$ for j =1,…,p, and (ii) ${\beta }_{0j}^{(ii)}=(1/2)/{\left\{j-3.5\right\}}^2$ for j > 1 with the intercept coefficient ${\beta }_{0j}^{(ii)}=-10$. [These choices ensure max_j>1 |β_0j | = 2 and that y is around zero in Design 2(ii).] We set ${\vartheta }_0=\frac{1}{8}(1,1,1,1,0,0,\dots ,0,0,1,1,1,1{)}^{\prime }$. For Design 1, we have $\mathcal{U}=\left[1,2.5\right]$ and for Design 2 we have $\mathcal{U}=\left[-0.5,0.5\right]$. The results are based on 500 replications (the bootstrap procedure is performed 5000 times for each replication).

We report the (empirical) rejection frequencies for confidence regions with 95% nominal coverage, so that 0.05 is the target rejection frequency. We report the rejection frequencies for the proposed estimator and the post-naive selection estimator. Table 1 presents the performance of the methods when applied to construct a confidence interval for a single parameter ($\tilde{p}=1$ and $\mathcal{U}$ is a singleton). Since the setting is not symmetric, we investigate the performance for different components. Specifically, we consider {u} × {j} for j = 1,…, 5. First, consider the location model (Design 1). The difference between the performance of the naive estimator for Design 1(i) and 1(ii) highlights its fragile performance which is highly dependent on the unknown parameters. We can see from Table 1 that in Design 1(i) the Naive method achieve (pointwise) rejection frequencies up to 0.162 when the nominal level is 0.05. In Design 1(ii), it can be as high as 0.886. We also note that it is important to look at the performance of each component and avoid averaging across components (large j components are essentially not in the model, indeed for j > 50 we obtain rejection frequencies very close to 0.05 regardless of the model selection procedure). In contrast, the proposed estimator exhibits a much more robust behavior. For Design 1(i), the rejection frequencies are between 0.040 and 0.062 while for Design 1(ii) the rejection frequencies of the proposed estimator are between 0.040 and 0.056.

Table 1.

We report the pointwise rejection frequencies of each method for (pointwise) confidence intervals for each j ∈ {1,…, 5}. For Design 1, we used $\mathcal{U}=\left\{1\right\}$ and for Design 2, we used $\mathcal{U}=\left\{0.5\right\}$. The results are based on 500 replications

p = 2000, n = 500		Rejection frequencies for j ∈{1,…, 5}
Design	Method	j = 1	j = 2	j = 3	j = 4	j = 5
1(i)	Proposed	0.042	0.040	0.062	0.050	0.044
	Naive	0.100	0.098	0.108	0.108	0.162
1(ii)	Proposed	0.044	0.040	0.054	0.056	0.056
	Naive	0.038	0.030	0.070	0.886	0.698
2(i)	Proposed	0.046	0.054	0.044	0.052	0.054
	Naive	0.046	0.050	0.038	0.070	0.054
2(ii)	Proposed	0.092	0.074	0.034	0.088	0.082
	Naive	0.034	0.972	0.182	0.312	0.916

Table 2 presents the performance for simultaneous confidence bands of the form $\left\{\left[{\tilde{\theta }}_{uj}-cv{\tilde{\sigma }}_{uj},{\tilde{\theta }}_{uj}+cv{\tilde{\sigma }}_{uj}\right]\text{ for }u\in \mathcal{U}\times [\tilde{p}]\right\}$ where ${\tilde{\theta }}_{uj}$ is a point estimate, ${\tilde{\sigma }}_{uj}$ is an estimate of the pointwise standard deviation and cv is a critical value that accounts for the uniform estimation. For the point estimate, we consider the proposed estimator and the post-naive selection estimator which have estimates of standard deviation. We consider two critical values: from the multiplier bootstrap (MB) procedure and the Bonferroni (BF) correction (which we expect to be conservative). For each of the four different designs [1(i), 1(ii), 2(i) and 2(ii) described above], we consider four different choices of $\mathcal{U}\times [\tilde{p}]$. Table 2 displays rejection frequencies for confidence regions with 95% nominal coverage (and again 0.05 would be the ideal performance). The simulation results confirms the differences between the performance of the methods and overall the proposed procedure is closer to the nominal value of 0.05. The proposed estimator performed within a factor of two to the nominal value in 10 out of the 16 designs considered (and 13 out 16 within a factor of three). The post-naive selection estimator performed within a factor of two only in 3 out of the 16 designs when using the multiplier bootstrap as critical value (7 out of 16 within a factor of three) and similarly with the Bonferroni correction as the critical value.

Table 2.

We report the rejection frequencies of each method for the (uniform) confidence bands for $\mathcal{U}\times \tilde{p}$. The proposed estimator computes the critical value based on the multiplier bootstrap procedure. For the naive post-selection estimator, we report the results for two choices of critical values, one choice based on the multiplier bootstrap (MB) and another based on Bonferroni (BF) correction. The results are based on 500 replications

p = 2000, n = 500		Uniform over $\mathcal{U}\times \tilde{p}$
Design	Method	[1,2.5] × {1}	{1} × [10]	[1,2.5] × [10]	{1} × [1000]
1(i)	Proposed	0.054	0.036	0.048	0.040
	Naive (MB)	0.126	0.136	0.172	0.032
	Naive (BF)	0.014	0.124	0.026	0.032
1(ii)	Propose	0.270	0.036	0.032	0.142
	Naive (MB)	0.014	0.802	0.934	0.404
	Naive (BF)	0.000	0.802	0.718	0.376
Design	Method	[−0.5, 0.5] × {1}	{0.5} × [10]	[−0.5,0.5] × [10]	{0.5} × [1000]
2(i)	Proposed	0.364	0.038	0.052	0.062
	Naive (MB)	0.116	0.040	0.022	0.048
	Naive (BF)	0.018	0.038	0.000	0.046
2(ii)	Proposed	0.140	0.090	0.408	0.084
	Naive (MB)	0.002	0.946	0.996	0.362
	Naive (BF)	0.000	0.946	0.944	0.298

APPENDIX A: PROOFS FOR SECTION 2

In this appendix, we use C to denote a strictly positive constant that is independent of n and $P\in {\mathcal{P}}_n$. The value of C may change at each appearance. Also, the notation an ≲ b_n means that an ≤ Cb_n for all n and some C. The notation a_n ≳ b_n means that b_n ≲ an. Moreover, the notation a_n = o(1) means that there exists a sequence (b_n)_{n ≥ 1} of positive numbers such that (i) |a_n| ≤ b_n for all n, (ii) b_n is independent of $P\in {\mathcal{P}}_n$ for all n and (iii) b_n → 0 as n → ∞. Finally, the notation an = O_p(b_n) means that for all ϵ > 0, there exists C such that P_p(a_n > Cb_n) ≤ 1 - ϵ for all n. Using this notation allows us to avoid repeating “uniformly over $P\in {\mathcal{P}}_n$“ many times in the proofs of Theorem 2.1 and Corollaries 2.1 and 2.2. Throughout this appendix, we assume that n ≥ n₀.

Proof of Theorem 2.1.

We split the proof into five steps.

Step 1. (Preliminary rate result). We claim that with probability 1 − o(1), ${\text{sup}}_{u\in \mathcal{U},j\in [\tilde{p}]}\left|{\check{\theta }}_{uj}-{\theta }_{uj}\right|\lesssim B_{1n}{\tau }_n$. To show that, note that by definition of ${\check{\theta }}_{uj}$, we have for each $u\in \mathcal{U}$ and $j\in [\tilde{p}]$,

$$\left|{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,{\check{\theta }}_{uj},{\widehat{\eta }}_{uj}\right)\right]\right|\le \underset{\theta \in {\Theta }_{uj}}{\text{inf}}\left|{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,\theta ,{\widehat{\eta }}_{uj}\right)\right]\right|+{ϵ}_n,$$

which implies via the triangle inequality that uniformly over $u\in \mathcal{U}$ and $j\in [\tilde{p}]$, with probability 1 − o(1),

$${\left|{\text{E}}_P\left[{\psi }_{uj}\left(W,\theta ,{\eta }_{uj}\right)\right]\right|}_{\theta ={\check{\theta }}_{uj}}|\le {ϵ}_n+2I_1+2I_2\lesssim B_{1n}{\tau }_n,$$ (A.1)

where

$$\begin{array}{l}{I}_1:=\underset{u\in \mathcal{U},j\in [\tilde{p}],\theta \in {\Theta }_{uj}}{\text{sup}}\left|{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,\theta ,{\widehat{\eta }}_{uj}\right)\right]-{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,\theta ,{\eta }_{uj}\right)\right]\right|\lesssim B_{1n}{\tau }_n,\hfill \\ I_2:=\underset{u\in \mathcal{U},j\in [\tilde{p}],\theta \in {\Theta }_{uj}}{\text{sup}}\left|{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,\theta ,{\eta }_{uj}\right)\right]-{\text{E}}_P\left[{\psi }_{uj}\left(W,\theta ,{\theta }_{uj}\right)\right]\right|\lesssim {\tau }_n,\hfill \end{array}$$

and the bounds on I₁ and I₂ are derived in Step 2 [note also that ∊_n = o(τ_n) by construction of the estimator and Assumption 2.2(vi)]. Since by Assumption 2.1(iv), $2^{-1}\left|J_{uj}\left({\check{\theta }}_{uj}-{\theta }_{uj}\right)\right|\wedge c_0$ does not exceed the left-hand side of (A.1), ${\text{inf}}_{u\in \mathcal{U},j\in [\tilde{p}]}\left|J_{uj}\right|\gtrsim 1$, and by Assumption 2.2(vi), B_1nτ_n = o(1), we conclude that

$$\underset{u\in \mathcal{U},j\in [\tilde{p}]}{\text{sup}}\left|{\check{\theta }}_{uj}-{\theta }_{uj}\right|\lesssim {\left(\underset{u\in \mathcal{U},j\in [\tilde{p}]}{\text{inf}}\left|J_{uj}\right|\right)}^{-1}{B}_{1n}{\tau }_n\lesssim B_{1n}{\tau }_n,$$ (A.2)

with probability 1 − o(1) yielding the claim of this step.

Step 2. (Bounds on I₁ and I₂). We claim that with probability 1 − o(1), I₁ ≲ B_1nτ_n and I₂ < τ_n. To show these relations, observe that with probability 1 − o(1), we have I₁ ≤ 2I_1a + I_1b and I₂ ≤ I_1a, where

$$\begin{array}{l}{I}_{1a}:=\underset{u\in \mathcal{U},j\in [\tilde{p}],\theta \in {\Theta }_{uj},\eta \in {\mathcal{T}}_{uj}}{\text{sup}}\left|{\mathbb{E}}_n\left[{\psi }_{uj}(W,\theta ,\eta )\right]-{\text{E}}_P\left[{\psi }_{uj}(W,\theta ,\eta )\right]\right|,\hfill \\ I_{1b}:=\underset{u\in \mathcal{U},j\in \left[\tilde{p}\right],\theta \in {\Theta }_{uj},\eta \in {\mathcal{T}}_{uj}}{\text{sup}}\left|{\text{E}}_P\left[{\psi }_{uj}(W,\theta ,\eta )\right]-{\text{E}}_P\left[{\psi }_{uj}\left(W,\theta ,{\eta }_{uj}\right)\right]\right|.\hfill \end{array}$$

To bound I_1b, we employ Taylor’s expansion:

$$I_{1b}\le \underset{u\in \mathcal{U},j\in [\tilde{p}],\theta \in {\Theta }_{uj},\eta \in {\mathcal{T}}_{uj},r\in [0,1)}{\text{sup}}{\partial }_r{\text{E}}_P\left[{\psi }_{uj}\left(W,\theta ,{\eta }_{uj}+r\left(\eta -{\eta }_{uj}\right)\right)\right]\le B_{1n}\underset{u\in \mathcal{U},j\in [\tilde{p}],\eta \in {\mathcal{T}}_{uj}}{\text{sup}}\Vert \eta -{\eta }_{uj}{\Vert }_e\le B_{1n}{\tau }_n,$$

by Assumptions 2.1(v) and 2.2(ii).

To bound I_1a, we apply the maximal inequality of Lemma P.2 to the class ${\mathcal{F}}_1$ defined in Assumption 2.2 to conclude that with probability 1 − o(1),

$$I_{1a}\lesssim n^{-1/2}\left(\sqrt{v_n\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n}+n^{-1/2+1/q}{v}_n{K}_n\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n\right).$$ (A.3)

Here, we used: $\text{log\hspace{0.17em}}{\text{sup}}_{Q}N\left(ϵ\Vert F_1{\Vert }_{Q,2},{\mathcal{F}}_1,\Vert \cdot {\Vert }_{Q,2}\right)\le v_n\text{\hspace{0.17em}log\hspace{0.17em}}\left(a_n/ϵ\right)$ for all 0 < ∊ ≤ 1 with ‖F₁‖_P,q ≤ K_n by Assumption 2.2(iv); ${\text{sup}}_{f\in {\mathcal{F}}_1}\Vert f{\Vert }_{P,2}^2\le C_0$ by Assumption 2.2(v); a_n ≥ n ∨ K_n and v_n ≥ 1 by the choice of a_n and v_n. In turn, the right-hand side of (A.3) is bounded from above by O(τ_n) by Assumption 2.2(vi) since ${\left(v_n\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n/n\right)}^{1/2}\lesssim {\tau }_n$ and

$$n^{-1/2}{n}^{-1/2+1/q}{v}_n{K}_n\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n\lesssim n^{-1/2}{\delta }_n\lesssim n^{-1/2}\lesssim {\tau }_n.$$

Combining presented bounds gives the claim of this step.

Step 3. (Linearization). Here, we prove the claim of the theorem. Fix $u\in \mathcal{U}$ and $j\in [\tilde{p}]$. By definition of ${\check{\theta }}_{uj}$, we have

$$\sqrt{n}\left|{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,{\check{\theta }}_{uj},{\widehat{\eta }}_{uj}\right)\right]\right|\le \underset{\theta \in {\Theta }_{uj}}{\text{inf}}\sqrt{n}\left|{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,\theta ,{\widehat{\eta }}_{uj}\right)\right]\right|+{ϵ}_n\sqrt{n}.$$ (A.4)

Also, for any θ ∊ Θ_uj and $\eta \in {\mathcal{T}}_{uj}$, we have

$$\sqrt{n}{\mathbb{E}}_n\left[{\psi }_{uj}(W,\theta ,\eta )\right]=\sqrt{n}{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}\right)\right]-{\mathbb{G}}_n{\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}\right)-\sqrt{n}\left({\text{E}}_P\left[{\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}\right)\right]-{\text{E}}_P\left[{\psi }_{uj}(W,\theta ,\eta )\right]\right)+{\mathbb{G}}_n{\psi }_{uj}(W,\theta ,\eta ).$$ (A.5)

Moreover, by Taylor’s expansion of the function r ↦ E_P [ψ_uj(W, θ_uj + r(θ − θ_uj), η_uj + r(η − η_uj))],

$${\text{E}}_P\left[{\psi }_{uj}(W,\theta ,\eta )\right]-{\text{E}}_P\left[{\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}\right)\right]=J_{uj}\left(\theta -{\theta }_{uj}\right)+{\text{D}}_{u,j,0}\left[\eta -{\eta }_{uj}\right]+2^{-1}{\partial }_r^2{\text{E}}_P\left[W,{\theta }_{uj}+r\left(\theta -{\theta }_{uj}\right),{\eta }_{uj}+r\left(\eta -{\eta }_{uj}\right)\right]{|}_{r=\overline{r}}$$ (A.6)

for some $\overline{r}\in \left(0,1\right)$. Substituting this equality into (A.5), taking $\theta ={\check{\theta }}_{uj}$ and $\eta ={\widehat{\eta }}_{uj}$, and using (A.4) gives

$$\sqrt{n}\left|{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}\right)\right]+J_{uj}\left({\check{\theta }}_{uj}-{\theta }_{uj}\right)+{\text{D}}_{u,j,0}\left[{\widehat{\eta }}_{uj}-{\eta }_{uj}\right]\right|\le {ϵ}_n\sqrt{n}+\underset{\theta \in {\Theta }_{uj}}{\text{inf}}\sqrt{n}\left|{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,\theta ,{\widehat{\eta }}_{uj}\right)\right]\right|+\left|I{I}_1(u,j)\right|+\left|I{I}_2(u,j)\right|,$$ (A.7)

where

$$\begin{array}{l}I{I}_1(u,j):=\sqrt{n}\underset{r\in [0,1)}{\text{sup}}\left|{\partial }_r^2{\text{E}}_P\left[{\psi }_{uj}\left(W,{\theta }_{uj}+r\left(\theta -{\theta }_{uj}\right),{\eta }_{uj}+r\left\{\eta -{\eta }_{uj}\right\}\right)\right]\right|,\hfill \\ I{I}_2(u,j):={\mathbb{G}}_n\left({\psi }_{uj}(W,\theta ,\eta )-{\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}\right)\right)\hfill \end{array}$$

with $\theta ={\check{\theta }}_{uj}$ and $\eta ={\widehat{\eta }}_{uj}$. It will be shown in Step 4 that

$$\underset{u\in U,j\in [\tilde{p}]}{\text{sup}}\left(\left|I{I}_1(u,j)\right|+\left|I{I}_2(u,j)\right|\right)=O_P\left({\delta }_n\right).$$ (A.8)

In addition, it will be shown in Step 5 that

$$\underset{u\in \mathcal{U},j\in [\tilde{p}]}{\text{sup}}\underset{\theta \in {\Theta }_{uj}}{\text{inf}}\sqrt{n}\left|{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,\theta ,{\widehat{\eta }}_{uj}\right)\right]\right|=O_P\left({\delta }_n\right).$$ (A.9)

Moreover, ${ϵ}_n\sqrt{n}=o\left({\delta }_n\right)$ by construction of the estimator. Therefore, the expression in (A.7) is O_P(δ_n). Also, ${\text{sup}}_{u\in \mathcal{U},j\in [\tilde{p}]}\left|{\text{D}}_{u,j,0}\left[{\widehat{\eta }}_{uj}-{\eta }_{uj}\right]\right|=O_P\left({\delta }_n{n}^{-1/2}\right)$ by the near-orthogonality condition since ${\widehat{\eta }}_{uj}\in {\mathcal{T}}_{uj}$ for all $u\in \mathcal{U}$ and $j\in [\tilde{p}]$ with probability 1 — o(1) by Assumption 2.2(i). Therefore, Assumption 2.1(iv) gives

$$\underset{u\in \mathcal{U},j\in [\tilde{p}]}{\text{sup}}\left|J_{uj}^{-1}\sqrt{n}{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}\right)\right]+\sqrt{n}\left({\check{\theta }}_{uj}-{\theta }_{uj}\right)\right|=O_P\left({\delta }_n\right).$$

The asserted claim now follows by dividing both parts of the display above by σ_uj (under the supremum on the left-hand side) and noting that σ_uj is bounded below from zero uniformly over $u\in \mathcal{U}$ and $j\in [\tilde{p}]$ by Assumptions 2.2(iii) and 2.2(v).

Step 4. [Bounds on II₁(u, j) and II₂(u, j)]. Here, we prove (A.8). First, with probability 1 − o(1),

$$\underset{u\in \mathcal{U},j\in [\tilde{p}]}{\text{sup}}\left|I{I}_1(u,j)\right|\le \sqrt{n}{B}_{2n}\underset{u\in \mathcal{U},j\in [\tilde{p}]}{\text{sup}}|{\check{\theta }}_{uj}-{\theta }_{uj}{|}^2\vee \Vert {\widehat{\eta }}_{uj}-{\eta }_{uj}{\Vert }_e^2\lesssim \sqrt{n}{B}_{1n}^2{B}_{2n}{\tau }_n^2\lesssim {\delta }_n,$$

where the first inequality follows from Assumptions 2.1(v) and 2.2(i), the second from Step 1 and Assumptions 2.2(ii) and 2.2(vi) and the third from Assumption 2.2(vi).

Second, we have with probability 1 − o(1) that ${\text{sup}}_{u\in \mathcal{U},j\in [\tilde{p}]}\left|I{I}_2(u,j)\right|\lesssim {\text{sup}}_{f\in {\mathcal{F}}_2}\left|{\mathbb{G}}_n(f)\right|$, where

$${\mathcal{F}}_2=\left\{{\psi }_{uj}(\cdot ,\theta ,\eta )-{\psi }_{uj}\left(\cdot ,{\theta }_{uj},{\eta }_{uj}\right):u\in \mathcal{U},j\in [\tilde{p}],\eta \in {\mathcal{T}}_{uj},\left|\theta -{\theta }_{uj}\right|\le C{B}_{1n}{\tau }_n\right\}$$

for sufficiently large constant C. To bound ${\text{sup}}_{f\in {\mathcal{F}}_2}\left|{\mathbb{G}}_n(f)\right|$, we applyLemmaP.2. Observe that

$$\begin{array}{l}\underset{f\in {\mathcal{F}}_2}{\text{sup}}\Vert f{\Vert }_{P,2}^2\hfill \\ \le \underset{u\in \mathcal{U},j\in [\tilde{p}],|\theta -C{B}_{1n}{\tau }_n,\eta \in {\mathcal{T}}_{uj}}{\text{sup}}{E}_P\left[{\left({\psi }_{uj}(W,\theta ,\eta )-{\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}\right)\right)}^2\right]\hfill \\ \le \underset{u\in \mathcal{U},j\in [\tilde{p}],|\theta -C{B}_{1n}{\tau }_n,\eta \in {\mathcal{T}}_{uj}}{\text{sup}}{C}_0{\left(\left|\theta -{\theta }_{uj}\right|\vee \Vert \eta -{\eta }_{uj}{\Vert }_e\right)}^{\omega }\lesssim {\left(B_{1n}{\tau }_n\right)}^{\omega },\hfill \end{array}$$

where we used Assumption 2.1(v) and Assumption 2.2(ii). Also, observe that (B_1n τ_n)^ω/2 ≥ n^−ω/4 by Assumption 2.2(vi) since B_1n ≥ 1. Therefore, an application of Lemma P.2 with an envelope F₂ = 2F₁ and σ = (CB_1n τ_n)^ω/2 for sufficiently large constant C gives with probability 1 − o(1),

$$\underset{f\in {\mathcal{F}}_2}{\text{sup}}\left|{\mathbb{G}}_n(f)\right|\lesssim {\left(B_{1n}{\tau }_n\right)}^{\omega /2}\sqrt{v_n\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n}+n^{-1/2+1/q}{v}_n{K}_n\text{\hspace{0.17em}log\hspace{0.17em}}{a}_n,$$ (A.10)

since ${\text{sup}}_{f\in {\mathcal{F}}_2}\left|f\right|\le 2{\text{sup}}_{f\in {\mathcal{F}}_1}\left|f\right|\le 2F_1$ and ‖F₁‖_P,q ≤ K_n by Assumption 2.2(iv) and

$$\text{\hspace{0.17em}log\hspace{0.17em}}\underset{Q}{\text{sup}}N\left(ϵ\Vert F_2{\Vert }_{Q,2},{\mathcal{F}}_2,\Vert \cdot {\Vert }_{Q,2}\right)\lesssim v_n\text{\hspace{0.17em}log\hspace{0.17em}}\left(a_n/ϵ\right)\text{\hspace{1em} for all }0<ϵ\le 1$$

by Lemma O.1 because ${\mathcal{F}}_2\subset {\mathcal{F}}_1-{\mathcal{F}}_1$ for ${\mathcal{F}}_1$ defined in Assumption 2.2(iv). The claim of this step now follows from an application of Assumption 2.2(vi) to bound the right-hand side of (A.10).

Step 5. Here, we prove (A.9). For all $u\in \mathcal{U}$ and $j\in [\tilde{p}]$, let ${\overline{\theta }}_{uj}={\theta }_{uj}-J_{uj}^{-1}{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}\right)\right]$. Then ${\text{sup}}_{u\in \mathcal{U},j\in [\tilde{p}]}\left|{\overline{\theta }}_{uj}-{\theta }_{uj}\right|=O_P\left({\mathcal{S}}_n/\sqrt{n}\right)$ since ${\mathcal{S}}_n={\text{E}}_P\left[{\text{sup}}_{u\in \mathcal{U},j\in [\tilde{p}]}\left|\sqrt{n}{\mathbb{E}}_n\left[{\psi }_{uj}\left(W_{uj},{\theta }_{uj},{\eta }_{uj}\right)\right]\right|\right]$ and J_uj is bounded in absolute value below from zero uniformly over $u\in \mathcal{U}$ and $j\in [\tilde{p}]$ by Assumption 2.1(iv). Therefore, ${\overline{\theta }}_{uj}\in {\Theta }_{uj}$ for all $u\in \mathcal{U}$ and $j\in [\tilde{p}]$ with probability 1 − o(1) by Assumption 2.1(i). Hence, with the same probability, for all $u\in \mathcal{U}$ and $j\in [\tilde{p}]$,

$$\underset{\theta \in {\Theta }_{uj}}{\text{inf}}\sqrt{n}\left|{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,\theta ,{\widehat{\eta }}_{uj}\right)\right]\right|\le \sqrt{n}\left|{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,{\overline{\theta }}_{uj},{\widehat{\eta }}_{uj}\right)\right]\right|,$$

and so it suffices to show that

$$\underset{u\in \mathcal{U},j\in [\tilde{p}]}{\text{sup}}\sqrt{n}\left|{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,{\overline{\theta }}_{uj},{\widehat{\eta }}_{uj}\right)\right]\right|=O_P\left({\delta }_n\right).$$ (A.11)

To prove (A.11), for given $u\in \mathcal{U}$ and $j\in [\tilde{p}]$, substitute $\theta ={\overline{\theta }}_{uj}$ and $\eta ={\widehat{\eta }}_{uj}$ into (A.5) and use Taylor’s expansion in (A.6). This gives

$$\sqrt{n}\left|{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,{\overline{\theta }}_{uj},{\widehat{\eta }}_{uj}\right)\right]\right|\le \left|{\widetilde{II}}_1(u,j)\right|+\left|{\widetilde{II}}_2(u,j)\right|+\sqrt{n}\left|{\mathbb{E}}_n\left[{\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}\right)\right]+J_{uj}\left({\overline{\theta }}_{uj}-{\theta }_{uj}\right)+{\text{D}}_{u,j,0}\left[{\widehat{\eta }}_{uj}-{\eta }_{uj}\right]\right|,$$

where ${\widetilde{II}}_1\left(u,j\right)$ and ${\widetilde{II}}_2\left(u,j\right)$ are defined as II₁ (u, j) and II₂(u, j) in Step 3 but with ${\check{\theta }}_{uj}$ replaced by ${\overline{\theta }}_{uj}$. Then, given that ${\text{sup}}_{u\in \mathcal{U},j\in [\tilde{p}]}\left|{\overline{\theta }}_{uj}-{\theta }_{uj}\right|\lesssim {\mathcal{S}}_n\text{\hspace{0.17em}log\hspace{0.17em}}n/\sqrt{n}$ with probability 1 − o(1), the argument in Step 4 shows that

$$\underset{u\in \mathcal{U},j\in [\tilde{p}]}{\text{sup}}\left(\left|{\widetilde{II}}_1(u,j)\right|+\left|{\widetilde{II}}_2(u,j)\right|\right)=O_P\left({\delta }_n\right).$$

In addition, ${\mathbb{E}}_n\left[{\psi }_{uj}\left(W,{\theta }_{uj},{\eta }_{uj}\right)\right]+J_{uj}\left({\overline{\theta }}_{uj}-{\theta }_{uj}\right)=0$ by the definition of ${\overline{\theta }}_{uj}$ and ${\text{sup}}_{u\in \mathcal{U},j\in [\tilde{p}]}\left|{\text{D}}_{u,j,0}\left[{\widehat{\eta }}_{uj}-{\eta }_{uj}\right]\right|=O_P\left({\delta }_n{n}^{-1/2}\right)$ by the near-orthogonality condition. Combining these bounds gives (A.11), so that the claim of this step follows, and completes the proof of the theorem.

APPENDIX B: REMAINING PROOFS FOR SECTION 2

See the Supplementary Material.

APPENDIX C: PROOFS FOR SECTIONS 3 AND 4

See the Supplementary Material.