Non-Asymptotic Oracle Inequalities for the High-Dimensional Cox Regression via Lasso

Abstract

We consider finite sample properties of the regularized high-dimensional Cox regression via lasso. Existing literature focuses on linear models or generalized linear models with Lipschitz loss functions, where the empirical risk functions are the summations of independent and identically distributed (iid) losses. The summands in the negative log partial likelihood function for censored survival data, however, are neither iid nor Lipschitz.We first approximate the negative log partial likelihood function by a sum of iid non-Lipschitz terms, then derive the non-asymptotic oracle inequalities for the lasso penalized Cox regression using pointwise arguments to tackle the difficulties caused by lacking iid Lipschitz losses.

1. Introduction

Since it was introduced by Tibshirani (1996), the lasso regularized method for high-dimensional regression models with sparse coefficients has received a great deal of attention in the literature. Properties of interest for such regression models include the finite sample oracle inequalities. Among the extensive literature of the lasso method, Bunea, Tsybakov, and Wegkamp (2007) and Bickel, Ritov, and Tsybakov (2009) derived the oracle inequalities for prediction risk and estimation error in a general nonparametric regression model, including the high-dimensional linear regression as a special example, and van de Geer (2008) provided oracle inequalities for the generalized linear models with Lipschitz loss functions, e.g., logistic regression and classification with hinge loss. Bunea (2008) and Bach (2010) also considered the lasso regularized logistic regression. For censored survival data, the lasso penalty has been applied to the regularized Cox regression in the literature, see e.g. Tibshirani (1997) and Gui and Li (2005), among others. Recently, Bradic, Fan, and Jiang (2011) studied the asymptotic properties of the lasso regularized Cox model. However, its finite sample non-asymptotic statistical properties have not yet been established in the literature to the best of our knowledge, largely due to lacking iid Lipschitz losses from the partial likelihood. Nonetheless, the lasso approach has been studied extensively in the literature for other models, see e.g. Martinussen and Scheike (2009) and Gaiffas and Guilloux (2012), among others, for the additive hazards model.

We consider the non-asymptotic statistical properties of the lasso regularized high-dimensional Cox regression. Let T be the survival time and C the censoring time. Suppose we observe a sequence of iid observations (X_i, Y_i, Δ_i), i = 1, …, n, where X_i = (X_i1, ⋯ X_im) are the m-dimensional covariates in 𝒳, Y_i = T_i ∧ C_i, and Δ_i = I_{Ti≤Ci}. Due to a large amount of parallel material, we follow closely the notation in van de Geer (2008). Let

$$ℱ=\{f_{\theta }(\mathbf{x})=\sum _{k=1}^m{\theta }_k{x}_k,\theta \in \mathrm{\Theta }\subset {\mathbf{R}}^{\mathbf{m}}\}.$$

Consider the Cox model (Cox (1972)):

$$\mathrm{\lambda }(t|\mathbf{X})={\mathrm{\lambda }}_0(t)e^{f_{\theta }(\mathbf{x})},$$

where θ is the parameter of interest and λ₀ is the unknown baseline hazard function. The negative log partial likelihood function for θ is

$$l_n(\theta )=-\frac{1}{n}\sum _{i=1}^n\{f_{\theta }({\mathbf{X}}_i)-\text{log}[\frac{1}{n}\sum _{j=1}^{n}1(Y_j\ge Y_i)e^{f_{\theta }({\mathbf{X}}_j)}\left]\right\}{\mathrm{\Delta }}_i.$$ (1.1)

The corresponding estimator with lasso penalty is denoted by

$${\widehat{\theta }}_n≔\underset{\theta \in \mathrm{\Theta }}{\text{arg min}}\{l_n(\theta )+{\mathrm{\lambda }}_{n}I(\theta )\},$$

where $I(\theta )≔{\sum }_{k=1}^m{\sigma }_k\left|{\theta }_k\right|$ is the weighted l₁ norm of the vector θ ∈ R^m. van de Geer (2008) considered σ_k to be the square-root of the second moment of the k-th covariate X_k, either at the population level (fixed) or at the sample level (random). For normalized X_k, σ_k = 1. We consider fixed weights σ_k, k = 1, ⋯ ,m. The results for random weights can be easily obtained from the case with fixed weights following van de Geer (2008), and we leave the detailed calculation to interested readers.

Clearly the negative log partial likelihood (1.1) is a sum of non-iid random variables. For ease of calculation, consider an intermediate function as a “replacement” of the negative log partial likelihood function

$${\tilde{l}}_n(\theta )=-\frac{1}{n}\sum _{i=1}^n\{f_{\theta }({\mathbf{X}}_i)-\text{log}\mu (Y_i;f_{\theta })\}{\mathrm{\Delta }}_i$$ (1.2)

that has the iid structure, but with an unknown population expectation

$$\mu (t;f_{\theta })=E_{\mathbf{X},Y}\{1(Y\ge t)e^{f_{\theta }(\mathbf{X})}\}.$$

The negative log partial likelihood function (1.1) can then be viewed as a “working” model for the empirical loss function (1.2). The corresponding loss function is

$${\gamma }_{f_{\theta }}=\gamma (f_{\theta }(\mathbf{X}),Y,\mathrm{\Delta })≔-\{f_{\theta }(\mathbf{X})-\text{log}\mu (Y;f_{\theta })\}\mathrm{\Delta },$$ (1.3)

with expected loss

$$l(\theta )=-E_{Y,\mathrm{\Delta },\mathbf{X}}\left[\right\{f_{\theta }(\mathbf{X})-\text{log}\mu (Y;f_{\theta })\left\}\mathrm{\Delta }\right]=P{\gamma }_{f_{\theta }},$$ (1.4)

where P denotes the distribution of (Y, Δ, X). Define the target function f̅ as

$$\overline{f}≔\underset{f\in ℱ}{\text{arg min}}P{\gamma }_f≔f_{\overline{\theta }},$$

where θ̅ = arg min_θ∈Θ Pγ_fθ. It is well-known that Pγ_fθ is convex with respect to θ for the regular Cox model, see for example, Andersen and Gill (1982), thus the above minimum is unique if the Fisher information matrix of θ at θ̅ is non-singular. Define the excess risk of f by

$$ℰ(f)≔P{\gamma }_f-P{\gamma }_{\overline{f}}.$$

It is desirable to show similar non-asymptotic oracle inequalities for the Cox regression model as in, for example, van de Geer (2008) for generalized linear models. That is, with large probability,

$$ℰ(f_{{\widehat{\theta }}_n})\le \mathit{\text{const.}}\times \underset{\theta \in \mathrm{\Theta }}{\text{min}}\{ℰ(f_{\theta })+{𝒱}_{\theta }\}.$$

Here 𝒱_θ is called the “estimation error”, which is typically proportional to ${\lambda }_n^2$ times the number of nonzero elements in θ.

Note that the summands in the negative log partial likelihood function (1.1) are not iid, and the intermediate loss function γ(·, Y, Δ) given in (1.3) is not Lipschitz. Hence the general result of van de Geer (2008) that requires iid Lipschitz loss functions does not apply to the Cox regression. We tackle the problem using pointwise arguments to obtain the oracle bounds of two types of errors: one is between empirical loss (1.2) and expected loss (1.4) without involving the Lipschitz requirement of van de Geer (2008), and one is between the negative log partial likelihood (1.1) and empirical loss (1.2) which establishes the iid approximation of non-iid losses. These steps distinguish our work from that of van de Geer (2008); we rely on the Mean Value Theorem with van de Geer’s Lipschitz condition replaced by the similar, but much less restrictive, boundedness assumption for regression parameters in Bühlmann (2006).

The article is organized as follows. In Section 2, we provide assumptions that are used throughout the paper. In Section 3, we define several useful quantities followed by the main result. We then provide a detailed proof in Section 4 by introducing a series of lemmas and corollaries useful for deriving the oracle inequalities for the Cox model. To avoid duplicate material as much as possible, we refer to the preliminaries and some results in van de Geer (2008) from place to place in the proofs without providing much detail.

2. Assumptions

We impose five basic assumptions. Let ‖·‖ be the L₂(P) norm and ‖·‖_∞ the sup norm.

Assumption A. K_m ≔ max_1≤k≤m{‖X_k‖_∞/σ_k} < ∞.

Assumption B. There exists an η > 0 and strictly convex increasing G, such that for all θ ∈ Θ with ‖f_θ − f̅‖_∞ ≤ η, one has ℰ(f_θ) ≥ G(‖f_θ − f̅‖).

In particular, G can be chosen as a quadratic function with some constant C₀, i.e., G(u) = u²/C₀, then the convex conjugate of function G, denoted by H, such that uv ≤ G(u) + H(v) is also quadratic.

Assumption C. There exists a function D(·) on the subsets of the index set {1, ⋯ m}, such that for all 𝒦 ⊂ {1, ⋯ , m}, and for all θ ∈ Θ and θ̃ ∈ Θ, we have ${\sum }_{k\in 𝒦}{\sigma }_k|{\theta }_k-{\tilde{\theta }}_k|\le \sqrt{D(𝒦)}\Vert f_{\theta }-f_{\tilde{\theta }}\Vert$. Here, D(𝒦) is chosen to be the cardinal number of 𝒦.

Assumption D. $L_m≔{\text{sup}}_{\theta \in \Theta }{\sum }_{k=1}^m\left|{\theta }_k\right|<\infty$.

Assumption E. The observation time stops at a finite time τ > 0, with ξ ≔ P(Y ≥ τ) > 0.

Assumptions A, B, and C are identical to those in van de Geer (2008) with her ψ_k the identity function. Assumptions B and C can be easily verified for the random design setting where X is random (van de Geer (2008)) together with the usual assumption of non-singular Fisher information matrix at θ̅ (and its neighborhood) for the Cox model. Assumption D has a similar flavor to the assumption (A2) in Bühlmann (2006) for the persistency property of boosting method in high-dimensional linear regression models, but is much less restrictive in the sense that L_m is allowed to depend on m in contrast with the fixed constant in Bühlmann (2006). Here it replaces the Lipschitz assumption in van de Geer (2008). Assumption E is commonly used for survival models with censored data, see for example, Andersen and Gill (1982). A straightforward extension of Assumption E is to allow τ (thus ξ) to depend on n.

From Assumptions A and D, we have, for any θ ∈ Θ,

$$e^{|f_{\theta }({\mathbf{X}}_i)|}\le e^{K_m{L}_m{\sigma }_{(m)}}≔U_m<\infty$$ (2.1)

for all i, where σ_(m) = max_1≤k≤m σ_k. Note that U_m is allowed to depend on m.

3. Main result

Let $I(\theta )≔{\sum }_{k=1}^m{\sigma }_k\left|{\theta }_k\right|$ be the l₁ norm of θ. For any θ and θ̃ in Θ, denote

$$I_1(\theta |\tilde{\theta })≔\sum _{k:{\tilde{\theta }}_k\ne 0}{\sigma }_k|{\theta }_k|,I_2(\theta |\tilde{\theta })≔I(\theta )-I_1(\theta |\tilde{\theta }).$$

Consider the estimator

$${\widehat{\theta }}_n≔\underset{\theta \in \mathrm{\Theta }}{\text{arg min}}\{l_n(\theta )+{\mathrm{\lambda }}_{n}I(\theta )\}.$$

3.1. Useful quantities

We first define a set of useful quantities that are involved in the oracle inequalities.

• a̅n = 4a_n, $a_n=\sqrt{\frac{2K_m^2\text{log}(2m)}{n}}+\frac{K_m\text{log}(2m)}{n}$.

• r₁ > 0, b > 0, d > 1, and 1 > δ > 0 are arbitrary constants.

• $d_b≔d(\frac{b+d}{(d-1)b}\vee 1)$.

• ${\overline{\lambda }}_{n,0}={\overline{\lambda }}_{n,0}^A+{\overline{\lambda }}_{n,0}^B$, where

  $${\overline{\mathrm{\lambda }}}_{n,0}^A≔{\overline{\mathrm{\lambda }}}_{n,0}^A(r_1)≔{\overline{a}}_n(1+2r_1\sqrt{2(K_m^2+{\overline{a}}_n{K}_m)}+\frac{4r_1^2{\overline{a}}_n{K}_m}{3}),{\overline{\mathrm{\lambda }}}_{n,0}^B≔{\overline{\mathrm{\lambda }}}_{n,0}^B(r_1)≔\frac{2K_m{U}_m^2}{\xi }(2{\overline{a}}_n{r}_1+\sqrt{\frac{\text{log}(2m)}{n}}).$$

• λ_n ≔ (1 + b)λ̅_n,0.

• δ₁ = (1 + b)^−N₁ and δ₂ = (1 + b)^−N₂ are arbitrary constants for some N₁ and N₂, where N₁ ∈ N ≔ {1, 2, …} and N₂ ∈ N ∪ {0}.

• $d({\delta }_1,{\delta }_2)=1+\frac{1+(d^2-1){\delta }_1}{(d-1)(1-{\delta }_1)}{\delta }_2$.

• W is a fixed constant given in Lemma 4.3 for a class of empirical processes.

• D_θ ≔ D({k : θ_k ≠ 0, k = 1, …, m}) is the number of nonzero θ_k’s, where D(·) is given in Assumption C.

• ${𝒱}_{\theta }≔2\delta H\left(\frac{2{\lambda }_n\sqrt{D_{\theta }}}{\delta }\right)$, where H is the convex conjugate of function G defined in Assumption B.

• ${\theta }_n^{*}≔{\text{arg min}}_{\theta \in \Theta }\{ℰ(f_{\theta })+{𝒱}_{\theta }\}$.

• ${\epsilon }_n^{*}≔(1+\delta )ℰ(f_{{\theta }_n^{*}})+{𝒱}_{{\theta }_n^{*}}$.

• ${\zeta }_n^{*}≔\frac{{\epsilon }_n^{*}}{{\lambda }_{n,0}}$.

• $\theta ({\epsilon }_n^{*})≔{\text{arg min}}_{\theta \in \Theta ,I(\theta -{\theta }_n^{*})\le d_b{\zeta }_n^{*}/b}\{\delta ℰ(f_{\theta })-2{\lambda }_n{I}_1(\theta -{\theta }_n^{*}|{\theta }_n^{*})\}$

In the above, the dependence of ${\theta }_n^{*}$ on the sample size n is through 𝒱_θ that involves the tuning parameter λ_n. We also impose conditions as in van de Geer (2008):

Condition I(b, δ). ${\Vert f_{{\theta }_n^{*}}-\overline{f}\Vert }_{\infty }\le \eta$.

Condition II(b, δ, d). ${\Vert f_{\theta ({\epsilon }_n^{*})}-\overline{f}\Vert }_{\infty }\le \eta$.

In both conditions, η is given in Assumption B.

3.2. Oracle inequalities

We now provide our theorem on oracle inequalities for the Cox model lasso estimator, with detailed proof given in the next section. The key idea of the proof is to find bounds of differences between empirical errors of the working model (1.2) and between approximation errors of the partial likelihood, denoted as Z_θ and R_θ in the next section.

Theorem 3.1. Suppose Assumptions A-E and Conditions I(b, δ) and II(b, δ, d) hold. With

$$\mathrm{\Delta }(b,\delta ,{\delta }_1,{\delta }_2)≔d({\delta }_1,{\delta }_2)\frac{1-{\delta }^2}{\delta b}\vee 1,$$

we have, with probability at least

$$1-\left\{{\text{log}}_{1+b}\frac{{(1+b)}^2\mathrm{\Delta }(b,\delta ,{\delta }_1,{\delta }_2)}{{\delta }_1{\delta }_2}\right\}\left\{\right(1+\frac{3}{10}{W}^2)\text{exp}(-n{\overline{a}}_n^2{r}_1^2)+2\text{exp}(-n{\xi }^2/2)\},$$

that

$$ℰ(f_{{\widehat{\theta }}_n})\le \frac{1}{1-\delta }{\epsilon }_n^{*}\mathit{\text{and}}I({\widehat{\theta }}_n-{\theta }_n^{*})\le d({\delta }_1,{\delta }_2)\frac{{\mathrm{\zeta }}_n^{*}}{b}.$$

4. Proofs

4.1. Preparations

Denote the empirical probability measure based on the sample {(X_i, Y_i, Δ_i) : i = 1, … n} by P_n. Let ε₁, ⋯ , ε_n be a Rademacher sequence, independent of the training data (X₁, Y₁, Δ₁), ⋯ , (X_n, Y_n, Δ_n). For some fixed θ* ∈ Θ and some M > 0, denote ℱ_M ≔ {f_θ : θ ∈ Θ, I(θ−θ*) ≤ M}. Later we take ${\theta }^{*}={\theta }_n^{*}$, which is the case of interest. For any θ where I(θ − θ*) ≤ M, denote

$$Z_{\theta }(M)≔|(P_n-P)[{\gamma }_{f_{\theta }}-{\gamma }_{f_{{\theta }^{*}}}]|=\left|\right[{\tilde{l}}_n(\theta )-l(\theta )]-[{\tilde{l}}_n({\theta }^{*})-l({\theta }^{*})\left]\right|.$$

Note that van de Geer (2008) sought to bound sup_f∈ℱM Z_θ(M), thus the contraction theorem of Ledoux and Talagrand (1991) (Theorem A.3 in van de Geer (2008)) was needed, which holds for Lipschitz functions. We find that the calculation in van de Geer (2008) does not apply to the Cox model due to the lack of Lipschitz property. However, the pointwise argument is adequate for our purpose because only the lasso estimator or the difference between the lasso estimator θ̂_n and the oracle ${\theta }_n^{*}$ is of interest. Note the notational difference between an arbitrary θ* in the above Z_θ(M) and the oracle ${\theta }_n^{*}$.

Lemma 4.1. Under Assumptions A, D, and E, for all θ satisfying I(θ − θ*) ≤ M, we have EZ_θ(M) ≤ a̅_nM.

Proof. By the symmetrization theorem, see e.g. van der Vaart and Wellner (1996) or Theorem A.2 in van de Geer (2008), for a class of only one function we have

$$E{Z}_{\theta }(M)\le 2E\left(\right|\frac{1}{n}\sum _{i=1}^n{\epsilon }_i\{[f_{\theta }({\mathbf{X}}_i)-\text{log}\mu (Y_i;f_{\theta })]{\mathrm{\Delta }}_i-[f_{{\theta }^{*}}({\mathbf{X}}_i)-\text{log}\mu (Y_i;f_{{\theta }^{*}})]{\mathrm{\Delta }}_i\}\left|\right)\le 2E\left(\right|\frac{1}{n}\sum _{i=1}^n{\epsilon }_i\{f_{\theta }({\mathbf{X}}_i)-f_{{\theta }^{*}}({\mathbf{X}}_i)\}{\mathrm{\Delta }}_i\left|\right)+2E\left(\right|\frac{1}{n}\sum _{i=1}^n{\epsilon }_i\{\text{log}\mu (Y_i;f_{\theta })-\text{log}\mu (Y_i;f_{{\theta }^{*}})\}{\mathrm{\Delta }}_i\left|\right)=A+B.$$

For A we have

$$A\le 2\left(\sum _{k=1}^m{\sigma }_k\right|{\theta }_k-{\theta }_k^{*}\left|\right)E\left(\underset{1\le k\le m}{\text{max}}\right|\frac{1}{n}\sum _{i=1}^n{\epsilon }_i{\mathrm{\Delta }}_i{X}_{ik}/{\sigma }_k\left|\right).$$

Applying Lemma A.1 in van de Geer (2008), we obtain

$$E\left(\underset{1\le k\le m}{\text{max}}\right|\frac{1}{n}\sum _{i=1}^n{\epsilon }_i{\mathrm{\Delta }}_i\frac{X_{ik}}{{\sigma }_k}\left|\right)\le a_n.$$

Thus we have

$$A\le 2a_{n}M.$$ (4.1)

For B, instead of using the contraction theorem that requires Lipschitz, we use the Mean Value Theorem:

$$|\frac{1}{n}\sum _{i=1}^n{\epsilon }_i\{\text{log}\mu (Y_i;f_{\theta })-\text{log}\mu (Y_i;f_{{\theta }^{*}})\}{\mathrm{\Delta }}_i|=|\frac{1}{n}\sum _{i=1}^n{\epsilon }_i{\mathrm{\Delta }}_i\sum _{k=1}^m\frac{1}{\mu (Y_i;f_{{\theta }^{**}})}{\int }_{Y_i}^{\infty }{\int }_{𝒳}({\theta }_k-{\theta }_k^{*})x_k{e}^{f_{{\theta }^{**}}(x)}d{P}_{\mathbf{X},Y}(\mathbf{x},y)|=|\sum _{k=1}^m{\sigma }_k({\theta }_k-{\theta }_k^{*})\frac{1}{n}\sum _{i=1}^n\frac{{\epsilon }_i{\mathrm{\Delta }}_i}{\mu (Y_i;f_{{\theta }^{**}}){\sigma }_k}{\int }_{Y_i}^{\infty }{\int }_{𝒳}{x}_k{e}^{f_{{\theta }^{**}}(x)}d{P}_{\mathbf{X},Y}(\mathbf{x},y)|\le |\sum _{k=1}^m{\sigma }_k({\theta }_k-{\theta }_k^{*})|\underset{1\le k\le m}{\text{max}}|\frac{1}{n}\sum _{i=1}^n{\epsilon }_i{\mathrm{\Delta }}_i{F}_{{\theta }^{**}}(k,Y_i)|\le M\underset{1\le k\le m}{\text{max}}|\frac{1}{n}\sum _{i=1}^n{\epsilon }_i{\mathrm{\Delta }}_i{F}_{{\theta }^{**}}(k,Y_i)|,$$

where θ** is between θ and θ*, and

$$F_{{\theta }^{**}}(k,t)=\frac{E[1(Y\ge t)X_k{e}^{f_{{\theta }^{**}}(\mathbf{X})}}{\mu (t;f_{{\theta }^{**}}){\sigma }_k}$$ (4.2)

satisfying

$$|F_{{\theta }^{**}}(k,t)|\le \frac{({\Vert X_k\Vert }_{\infty }/{\sigma }_k)E[1(Y\ge t)e^{f_{{\theta }^{**}}(\mathbf{X})}]}{\mu (t;f_{{\theta }^{**}})}\le K_m.$$

Since for all i,

$$E[{\epsilon }_i{\mathrm{\Delta }}_i{F}_{{\theta }^{**}}(k,Y_i)]=0,{\Vert {\epsilon }_i{\mathrm{\Delta }}_i{F}_{{\theta }^{**}}(k,Y_i)\Vert }_{\infty }\le K_m,\text{and}\frac{1}{n}\sum _{i=1}^{n}E{[{\epsilon }_i{\mathrm{\Delta }}_i{F}_{{\theta }^{**}}(k,Y_i)]}^2\le \frac{1}{n}\sum _{i=1}^{n}E{[F_{{\theta }^{**}}(k,Y_i)]}^2\le E{K}_m^2=K_m^2,$$

following Lemma A.1 in van de Geer (2008), we obtain

$$B\le 2a_{n}M.$$ (4.3)

Combining (4.1) and (4.3), the upper bound for EZ_θ(M) is achieved.

We now can bound the tail probability of Z_θ(M) using the Bousquet’s concentration theorem noted as Theorem A.1 in van de Geer (2008).

Corollary 4.1. Under Assumptions A, D, and E, for all M > 0, r₁ > 0 and all θ satisfying I(θ − θ*) ≤ M, it holds that

$$P(Z_{\theta }(M)\ge {\overline{\lambda }}_{n,0}^{A}M)\le \text{exp}(-n{\overline{a}}_n^2{r}_1^2).$$

Proof. Using the triangular inequality and the Mean Value Theorem, we obtain

$$|{\gamma }_{f_{\theta }}-{\gamma }_{f_{{\theta }^{*}}}|\le |f_{\theta }(\mathbf{X})-f_{{\theta }^{*}}(\mathbf{X})|\mathrm{\Delta }+|\text{log}\mu (Y;f_{\theta })-\text{log}\mu (Y;f_{{\theta }^{*}})|\mathrm{\Delta }\le \sum _{k=1}^m{\sigma }_k|{\theta }_k-{\theta }_k^{*}|\frac{\left|X_k\right|}{{\sigma }_k}+|\text{log}\mu (Y;f_{\theta })-\text{log}\mu (Y;f_{{\theta }^{*}})|\le M{K}_m+\sum _{k=1}^m{\sigma }_k|{\theta }_k-{\theta }_k^{*}|·\underset{1\le k\le m}{\text{max}}|F_{{\theta }^{**}}(k,Y)|\le 2M{K}_m,$$

where θ** is between θ and θ*, and F_θ**(k, Y) is defined in (4.2). So we have

$${\Vert {\gamma }_{f_{\theta }}-{\gamma }_{f_{{\theta }^{*}}}\Vert }_{\infty }\le 2M{K}_m,\text{and}P{({\gamma }_{f_{\theta }}-{\gamma }_{f_{{\theta }^{*}}})}^2\le 4M^2{K}_m^2.$$

Therefore, in view of Bousquet’s concentration theorem and Lemma 4.1, for all M > 0 and r₁ > 0,

$$P(Z_{\theta }(M)\ge {\overline{a}}_{n}M(1+2r_1\sqrt{2(K_m^2+{\overline{a}}_n{K}_m)}+\frac{4r_1^2{\overline{a}}_n{K}_m}{3}\left)\right)\le \text{exp}(-n{\overline{a}}_n^2{r}_1^2).$$

Now for any θ satisfying I(θ − θ*) ≤ M, we bound

$$R_{\theta }(M):=\left|\right[l_n(\theta )-{\tilde{l}}_n(\theta )]-[l_n({\theta }^{*})-{\tilde{l}}_n({\theta }^{*})\left]\right|=\frac{1}{n}\sum _{i=1}^n\left|\right[\text{log}\frac{1}{n}\sum _{j=1}^n\frac{1(Y_j\ge Y_i)e^{f_{\theta }({\mathbf{X}}_j)}}{\mu (Y_i;f_{\theta })}-\text{log}\frac{1}{n}\sum _{j=1}^n\frac{1(Y_j\ge Y_i)e^{f_{{\theta }^{*}}({\mathbf{X}}_j)}}{\mu (Y_i;f_{{\theta }^{*}})}\left]{\mathrm{\Delta }}_i\right|\le \underset{0\le t\le \tau }{\text{sup}}|\text{log}\frac{1}{n}\sum _{j=1}^n\frac{1(Y_j\ge t)e^{f_{\theta }({\mathbf{X}}_j)}}{\mu (t;f_{\theta })}-\text{log}\frac{1}{n}\sum _{j=1}^n\frac{1(Y_j\ge t)e^{f_{{\theta }^{*}}({\mathbf{X}}_j)}}{\mu (t;f_{\theta *})}|.$$

Here recall that τ is given in Assumption E. By the Mean Value Theorem, we have

$$R_{\theta }(M)\le \underset{0\le t\le \tau }{\text{sup}}|\sum _{k=1}^m({\theta }_k-{\theta }_k^{*}){\left\{\frac{{\sum }_{j=1}^{n}1(Y_j\ge t)e^{f_{{\theta }^{**}}({\mathbf{X}}_j)}}{\mu (t;f_{{\theta }^{**}})}\right\}}^{-1}\{\frac{{\sum }_{j=1}^{n}1(Y_j\ge t)X_{jk}{e}^{f_{{\theta }^{**}}({\mathbf{X}}_j)}}{\mu (t;f_{{\theta }^{**}})}-\frac{{\sum }_{j=1}^{n}1(Y_j\ge t)e^{f_{{\theta }^{**}}({\mathbf{X}}_j)}E[1(Y\ge t)X_k{e}^{f_{{\theta }^{**}}(\mathbf{X})}]}{\mu {(t;f_{{\theta }^{**}})}^2}\left\}\right|=\underset{0\le t\le \tau }{\text{sup}}|\sum _{k=1}^m{\sigma }_k({\theta }_k-{\theta }_k^{*})\{\frac{{\sum }_{j=1}^{n}1(Y_j\ge t)(X_{jk}/{\sigma }_k)e^{f_{{\theta }^{**}}({\mathbf{X}}_j)}}{{\sum }_{j=1}^{n}1(Y_j\ge t)e^{f_{\theta **}({\mathbf{X}}_j)}}-\frac{E[1(Y\ge t)(X_k/{\sigma }_k)e^{f_{{\theta }^{**}}(\mathbf{X})}]}{E[1(Y\ge t)e^{f_{{\theta }^{**}}(\mathbf{X})}]}\left\}\right|\le M\underset{0\le t\le \tau }{\text{sup}}{[\frac{1}{n}\sum _{i=1}^{n}1(Y_i\ge t)e^{f_{{\theta }^{**}}({\mathbf{X}}_i)}]}^{-1}\underset{0\le t\le \tau }{\text{sup}}\left\{\underset{1\le k\le m}{\text{max}}\right|\frac{1}{n}\sum _{i=1}^{n}1(Y_i\ge t)(X_{ik}/{\sigma }_k)e^{f_{{\theta }^{**}}({\mathbf{X}}_i)}-E[1(Y\ge t)(X_k/{\sigma }_k)e^{f_{{\theta }^{**}}(\mathbf{X})}]|+K_m|\frac{1}{n}\sum _{i=1}^{n}1(Y_i\ge t)e^{f_{{\theta }^{**}}({\mathbf{X}}_i)}-E[1(Y\ge t)e^{f_{{\theta }^{**}}(\mathbf{X})}]\left|\right\},$$ (4.4)

where θ** is between θ and θ* and, by (2.1), we have

$$\underset{0\le t\le \tau }{\text{sup}}{[\frac{1}{n}\sum _{i=1}^{n}1(Y_i\ge t)e^{f_{{\theta }^{**}}({\mathbf{X}}_i)}]}^{-1}\le U_m{[\frac{1}{n}\sum _{i=1}^{n}1(Y_i\ge \tau )]}^{-1}.$$ (4.5)

Lemma 4.2. Under Assumption E, we have

$$P(\frac{1}{n}\sum _{i=1}^{n}1(Y_i\ge \tau )\le \frac{\mathrm{\xi }}{2})\le 2e^{-n{\mathrm{\xi }}^2/2}.$$

Proof. This is obtained directly from Massart (1990) for the Kolmogorov statistic by taking $r=\xi \sqrt{n}/2$ in the following:

$$P(\frac{1}{n}\sum _{i=1}^{n}1(Y_i\ge \tau )\le \frac{\mathrm{\xi }}{2})\le P\left(\sqrt{n}\right|\frac{1}{n}\sum _{i=1}^{n}1(Y_i\ge \tau )-\mathrm{\xi }|\ge r)\le P\left(\underset{0\le t\le \tau }{\text{sup}}\sqrt{n}\right|\frac{1}{n}\sum _{i=1}^{n}1(Y_i\ge t)-P(Y\ge t)|\ge r)\le 2e^{-2r^2}.$$

Lemma 4.3. Under Assumptions A, D, and E, for all θ we have

$$P\left(\underset{0\le t\le \tau }{\text{sup}}\right|\frac{1}{n}\sum _{i=1}^{n}1(Y_i\ge t)e^{f_{\theta }({\mathbf{X}}_i)}-\mu (t;f_{\theta })|\ge U_m{\overline{a}}_n{r}_1)\le \frac{1}{5}{W}^2{e}^{-n{\overline{a}}_n^2{r}_1^2},$$ (4.6)

where W is a fixed constant.

Proof. For a class of functions indexed by t, ℱ = {1(y ≥ t)e^fθ(x)/U_m : t ∈ [0, τ], y ∈ R, e^fθ(x) ≤ U_m}, we calculate its bracketing number. For any nontrivial ∈ satisfying 1 > ∈ > 0, let t_i be the i-th ⌈1/ε⌉ quantile of Y, so

$$P(Y\le t_i)=i\epsilon ,i=1,\dots ,\lceil 1/\epsilon \rceil -1,$$

where ⌈x⌉ is the smallest integer that is greater than or equal to x. Furthermore, take t₀ = 0 and t_⌈1/ε⌉ = +∞. For i = 1, ⋯, ⌈1/ε⌈, define brackets [L_i, U_i] with

$$L_i(x,y)=1(y\ge t_i)e^{f_{\theta }(x)}/U_m,U_i(x,y)=1(y>t_{i-1})e^{f_{\theta }(x)}/U_m$$

such that L_i(x, y) ≤ 1(y ≥ t)e^fθ(x)/U_m ≤ U_i(x, y) when t_i−1 < t ≤ t_i. Since

$${\{E{[U_i-L_i]}^2\}}^{1/2}\le {\left\{E{[\frac{e^{f_{\theta }(\mathbf{X})}}{U_m}\{1(Y\ge t_i)-1(Y>t_{i-1})\}]}^2\right\}}^{1/2}\le {\{P(t_{i-1}<Y\le t_i)\}}^{1/2}=\sqrt{\epsilon },$$

we have $N_{\left[\right]}(\sqrt{\epsilon },ℱ,L_2)\le \lceil 1/\epsilon \rceil \le 2/\epsilon$, which yields

$$N_{[]}(\epsilon ,ℱ,L_2)\le \frac{2}{{\epsilon }^2}={\left(\frac{K}{\epsilon }\right)}^2,$$

where $K=\sqrt{2}$. Thus, from Theorem 2.14.9 in van der Vaart and Wellner (1996), we have for any r > 0,

$$P\left(\sqrt{n}\underset{0\le t\le \tau }{\text{sup}}\right|\frac{1}{n}\sum _{i=1}^n\frac{1(Y_i\ge t)e^{f_{\theta }({\mathbf{X}}_i)}}{U_m}-\frac{\mu (t;f_{\theta })}{U_m}|\ge r)\le \frac{1}{2}{W}^2{r}^2{e}^{-2r^2}\le \frac{1}{5}{W}^2{e}^{-r^2},$$

where W is a constant that only depends on K. Note that r²e^−r2 is bounded by e⁻¹. With $r=\sqrt{n}{\overline{a}}_n{r}_1$, we obtain (4.6).

Lemma 4.4. Under Assumptions A, D, and E, for all θ we have

$$P\left(\underset{0\le t\le \tau }{\text{sup}}\underset{1\le k\le m}{\text{max}}\right|\frac{1}{n}\sum _{i=1}^{n}1(Y_i\ge t)\frac{X_{ik}}{{\sigma }_k}{e}^{f_{\theta }({\mathbf{X}}_i)}-E[1(Y\ge t)\frac{X_k}{{\sigma }_k}{e}^{f_{\theta }(\mathbf{X})}]|\ge K_m{U}_m[{\overline{a}}_n{r}_1+\sqrt{\frac{\text{log}(2m)}{n}}\left]\right)\le \frac{1}{10}{W}^2{e}^{-n{\overline{a}}_n^2{r}_1^2}.$$ (4.7)

Proof. Consider the classes of functions indexed by t,

$${𝒢}^k=\{1(y\ge t)e^{f_{\theta }(x)}{x}_k/({\sigma }_k{K}_m{U}_m):t\in [0,\tau ],y\in \mathbf{R},|e^{f_{\theta }(x)}{x}_k/{\sigma }_k|\le K_m{U}_m\},k=1,\dots ,m.$$

Using the argument in the proof of Lemma 4.3, we have

$$N_{[]}(\epsilon ,{𝒢}^k,L_2)\le {\left(\frac{K}{\epsilon }\right)}^2,$$

where $K=\sqrt{2}$, and then for any r > 0,

$$P\left(\sqrt{n}\underset{0\le t\le \tau }{\text{sup}}\right|\frac{1}{n}\sum _{i=1}^n\frac{1(Y_i\ge t)e^{f_{\theta }({\mathbf{X}}_i)}{X}_{ik}}{{\sigma }_k{K}_m{U}_m}-E\left[\frac{1(Y\ge t)e^{f_{\theta }(\mathbf{X})}{X}_k}{{\sigma }_k{K}_m{U}_m}\right]|\ge r)\le \frac{1}{5}{W}^2{e}^{-r^2}.$$

Thus we have

$$P\left(\sqrt{n}\underset{0\le t\le \tau }{\text{sup}}\underset{1\le k\le m}{\text{max}}\right|\frac{1}{n}\sum _{i=1}^{n}1(Y_i\ge t)e^{f_{\theta }({\mathbf{X}}_i)}{X}_{ik}/({\sigma }_k{U}_m{K}_m)-E[1(Y\ge t)e^{f_{\theta }(\mathbf{X})}{X}_k/({\sigma }_k{U}_m{K}_m)]|\ge r)\le P\left(\underset{k=1}{\overset{m}{\cup }}\sqrt{n}\underset{0\le t\le \tau }{\text{sup}}\right|\frac{1}{n}\sum _{i=1}^{n}1(Y_i\ge t)e^{f_{\theta }({\mathbf{X}}_i)}{X}_{ik}/({\sigma }_k{U}_m{K}_m)-E[1-(Y\ge t)e^{f_{\theta }(\mathbf{X})}{X}_k/({\sigma }_k{U}_m{K}_m)]|\ge r)\le mP\left(\sqrt{n}\underset{0\le t\le \tau }{\text{sup}}\right|\frac{1}{n}\sum _{i=1}^{n}1(Y_i\ge t)e^{f_0({\mathbf{X}}_i)}{X}_{ik}/({\sigma }_k{U}_m{K}_m)-E[1(Y\ge t)e^{f_{\theta }(\mathbf{X})}{X}_k/({\sigma }_k{U}_m{K}_m)]|\ge r)\le \frac{m}{5}{W}^2{e}^{-r^2}=\frac{1}{10}{W}^2{e}^{\text{log}(2m)-r^2}.$$

Let $\text{log}(2m)-r^2=-n{\overline{a}}_n^2{r}_1^2$, so $r=\sqrt{n{\overline{a}}_n^2{r}_1^2+\text{log}(2m)}$. Since

$$\sqrt{{\overline{a}}_n^2{r}_1^2+\frac{\text{log}(2m)}{n}}\le {\overline{a}}_n{r}_1+\sqrt{\frac{\text{log}(2m)}{n}},$$

, we obtain (4.7).

Corollary 4.2. Under Assumptions A, D, and E, for all M > 0, r₁ > 0, and all θ that satisfy I(θ − θ*) ≤ M, we have

$$P(R_{\theta }(M)\ge {\overline{\mathrm{\lambda }}}_{n,0}^{B}M)\le 2\text{exp}(-n{\mathrm{\xi }}^2/2)+\frac{3}{10}{W}^2\text{exp}(-n{\overline{a}}_n^2{r}_1^2).$$ (4.8)

Proof. From (4.4) and (4.5) we have

$$P(R_{\theta }(M)\le {\overline{\mathrm{\lambda }}}_{n,0}^B·M)\ge P(E_1^c\cap E_2^c\cap E_3^c),$$

where the events E₁, E₂ and E₃ are defined as

$$E_1=\{\frac{1}{n}\sum _{i=1}^{n}1(Y_i\ge \tau )\le \mathrm{\xi }/2\},$$

$$E_2=\left\{\underset{0\le t\le \tau }{\text{sup}}\right|\frac{1}{n}\sum _{i=1}^{n}1(Y_i\ge t)e^{f_{{\theta }^{**}}({\mathbf{X}}_i)}-\mu (t;f_{{\theta }^{**}})|\ge U_m{\overline{a}}_n{r}_1\},$$

$$E_3=\left\{\underset{1\le k\le m}{\text{max}}\underset{0\le t\le \tau }{\text{sup}}\right|\frac{1}{n}\sum _{i=1}^{n}1(Y_i\ge t)\frac{X_{ik}}{{\sigma }_k}{e}^{f_{{\theta }^{**}}({\mathbf{X}}_i)}-E[1(Y\ge t)\frac{X_k}{{\sigma }_k}{e}^{f_{{\theta }^{**}}(\mathbf{X})}]|\ge K_m{U}_m({\overline{a}}_n{r}_1+\sqrt{\frac{\text{log}(2m)}{n})}\}.$$

Thus

$$P(R_{\theta }(M)\ge {\overline{\mathrm{\lambda }}}_{n,0}^B·M)\le P(E_1)+P(E_2)+P(E_3),$$

and the result follows from Lemmas 4.2, 4.3 and 4.4.

Now with ${\theta }^{*}={\theta }_n^{*}$, we have the following results.

Lemma 4.5. Suppose Conditions I(b, δ) and II(b, δ, d) are met. Under Assumptions B and C, for all θ ∈ Θ with $I(\theta -{\theta }_n^{*})\le d_b{\zeta }_n^{*}/b$, it holds that

$$2{\mathrm{\lambda }}_n{I}_1(\theta -{\theta }_n^{*})\le \delta ℰ(f_{\theta })+{\epsilon }_n^{*}-ℰ(f_{{\theta }_n^{*}}).$$

Proof. The proof is exactly the same as that of Lemma A.4 in van de Geer (2008), with the λ_n defined in Subsection 3.1.

Lemma 4.6. Suppose Conditions I(b, δ) and II(b, δ, d) are met. Consider any random θ̃ ∈ Θ with $l_n(\tilde{\theta })+{\lambda }_{n}I(\tilde{\theta })\le l_n({\theta }_n^{*})+{\lambda }_{n}I({\theta }_n^{*})$. Let 1 < d₀ ≤ d_b. It holds that

$$P(I(\tilde{\theta }-{\theta }_n^{*})\le d_0\frac{{\mathrm{\zeta }}_n^{*}}{b})\le P(I(\tilde{\theta }-{\theta }_n^{*})\le (\frac{d_0+b}{1+b}\left)\frac{{\mathrm{\zeta }}_n^{*}}{b}\right)+(1+\frac{3}{10}{W}^2)\text{exp}(-n{\overline{a}}_n^2{r}_1^2)+2\text{exp}(-n{\mathrm{\xi }}^2/2).$$

Proof. The idea is similar to the proof of Lemma A.5 in van de Geer (2008). Let ℰ̃ = ℰ(f_θ̃) and ${ℰ}^{*}=ℰ(f_{{\theta }_n^{*}})$. We will use short notation: $I_1(\theta )=I_1(\theta |{\theta }_n^{*})$ and $I_2(\theta )=I_2(\theta |{\theta }_n^{*})$. Since $l_n(\tilde{\theta })+{\lambda }_{n}I(\tilde{\theta })\le l_n({\theta }_n^{*})+{\lambda }_{n}I({\theta }_n^{*})$, on the set where $I(\tilde{\theta }-{\theta }_n^{*})\le d_0{\zeta }_n^{*}/b$ and $Z_{\tilde{\theta }}(d_0{\zeta }_n^{*}/b)\le d_0{\zeta }_n^{*}/b·{\overline{\lambda }}_{n,0}^A$, we have

$$R_{\tilde{\theta }}(d_0{\mathrm{\zeta }}_n^{*}/b)\ge [l_n({\theta }_n^{*})+{\mathrm{\lambda }}_{n}I({\theta }_n^{*})]-[l_n(\tilde{\theta })+{\mathrm{\lambda }}_{n}I(\tilde{\theta })]-{\mathrm{\lambda }}_{n}I({\theta }_n^{*})+{\mathrm{\lambda }}_{n}I(\tilde{\theta })-[{\tilde{l}}_n({\theta }_n^{*})-{\tilde{l}}_n(\tilde{\theta })]\ge -{\mathrm{\lambda }}_{n}I({\theta }_n^{*})+{\mathrm{\lambda }}_{n}I(\tilde{\theta })-[{\tilde{l}}_n({\theta }_n^{*})-{\tilde{l}}_n(\tilde{\theta })]\ge -{\mathrm{\lambda }}_{n}I({\theta }_n^{*})+{\mathrm{\lambda }}_{n}I(\tilde{\theta })-[l({\theta }_n^{*})-l(\tilde{\theta })-d_0{\mathrm{\zeta }}_n^{*}/b·{\overline{\mathrm{\lambda }}}_{n,0}^A\ge -{\mathrm{\lambda }}_{n}I({\theta }_n^{*})+{\mathrm{\lambda }}_{n}I(\tilde{\theta })-{ℰ}^{*}+\widetilde{ℰ}-d_0{\mathrm{\lambda }}_{n,0}^A{\mathrm{\zeta }}_n^{*}/b.$$ (4.9)

By (4.8) we know that $R_{\tilde{\theta }}(d_0{\zeta }_n^{*}/b)$ is bounded by $d_0{\overline{\lambda }}_{n,0}^B{\zeta }_n^{*}/b$ with probability at least $1-\frac{3}{10}{W}^2\text{exp}(-n{\overline{a}}_n^2{r}_1^2)-2\text{exp}(-n{\xi }^2/2)$, then we have

$$\widetilde{ℰ}+{\mathrm{\lambda }}_{n}I(\tilde{\theta })\le {\overline{\mathrm{\lambda }}}_{n,0}^B{d}_0{\mathrm{\zeta }}_n^{*}/b+{ℰ}^{*}+{\mathrm{\lambda }}_{n}I({\theta }_n^{*})+{\overline{\mathrm{\lambda }}}_{n,0}^A{d}_0{\mathrm{\zeta }}_n^{*}/b.$$

Since I(θ̃) = I₁(θ̃) + I₂(θ̃) and $I({\theta }_n^{*})=I_1({\theta }_n^{*})$, using the triangular inequality, we obtain

$$\widetilde{ℰ}+(1+b){\overline{\mathrm{\lambda }}}_{n,0}{I}_2(\tilde{\theta })\le {\overline{\mathrm{\lambda }}}_{n,0}{d}_0{\mathrm{\zeta }}_n^{*}/b+{ℰ}^{*}+(1+b){\overline{\mathrm{\lambda }}}_{n,0}{I}_1({\theta }_n^{*})-(1+b){\overline{\mathrm{\lambda }}}_{n,0}{I}_1(\tilde{\theta })\le {\overline{\mathrm{\lambda }}}_{n,0}{d}_0{\mathrm{\zeta }}_n^{*}/b+{ℰ}^{*}+(1+b){\overline{\mathrm{\lambda }}}_{n,0}{I}_1(\tilde{\theta }-{\theta }_n^{*}).$$ (4.10)

Adding $(1+b){\overline{\lambda }}_{n,0}{I}_1(\tilde{\theta }-{\theta }_n^{*})$ to both sides and from Lemma 4.5,

$$\widetilde{ℰ}+(1+b){\overline{\mathrm{\lambda }}}_{n,0}I(\tilde{\theta }-{\theta }_n^{*})\le {\mathrm{\lambda }}_{n,0}{d}_0\frac{{\mathrm{\zeta }}_n^{*}}{b}+{ℰ}^{*}+2(1+b){\overline{\mathrm{\lambda }}}_{n,0}{I}_1(\tilde{\theta }-{\theta }_n^{*})\le ({\overline{\mathrm{\lambda }}}_{n,0}{d}_0+b{\overline{\mathrm{\lambda }}}_{n,0})\frac{{\mathrm{\zeta }}_n^{*}}{b}+\delta \widetilde{ℰ}=(d_0+b){\overline{\mathrm{\lambda }}}_{n,0}\frac{{\mathrm{\zeta }}_n^{*}}{b}+\delta \widetilde{ℰ}.$$

Because 0 < δ < 1, it follows that

$$I(\tilde{\theta }-{\theta }_n^{*})\le \frac{d_0+b}{1+b}\frac{{\mathrm{\zeta }}_n^{*}}{b}.$$

Hence,

$$P\left(\right\{I(\tilde{\theta }-{\theta }_n^{*})\le d_0\frac{{\mathrm{\zeta }}_n^{*}}{b}\}\cap \{Z_{\tilde{\theta }}(d_0{\mathrm{\zeta }}_n^{*}/b)\le d_0{\overline{\mathrm{\lambda }}}_{n,0}^A\frac{{\mathrm{\zeta }}_n^{*}}{b}\}\cap \{R_{\tilde{\theta }}(d_0{\mathrm{\zeta }}_n^{*}/b)\le d_0{\overline{\mathrm{\lambda }}}_{n,0}^B\frac{{\mathrm{\zeta }}_n^{*}}{b}\left\}\right)\le P(I(\tilde{\theta }-{\theta }_n^{*})\le \frac{d_0+b}{1+b}\frac{{\mathrm{\zeta }}_n^{*}}{b}),$$

which yields the desired result.

Corollary 4.3. Suppose Conditions I(b, δ) and II(b, δ, d) are met. Consider any random θ̃ ∈ Θ with $l_n(\tilde{\theta })+{\lambda }_{n}I(\tilde{\theta })\le l_n({\theta }_n^{*})+{\lambda }_{n}I({\theta }_n^{*})$. Let 1 < d₀ ≤ d_b. It holds that

$$P(I(\tilde{\theta }-{\theta }_n^{*})\le d_0\frac{{\mathrm{\zeta }}_n^{*}}{b})\le P(I(\tilde{\theta }-{\theta }_n^{*})\le [1+(d_0-1){(1+b)}^{-N}]\frac{{\mathrm{\zeta }}_n^{*}}{b})+N\left\{\right(1+\frac{3}{10}{W}^2)\text{exp}(-n{\overline{a}}_n^2{r}_1^2)+2\text{exp}(-n{\mathrm{\xi }}^2/2\}.$$

Proof. Repeat Lemma 4.6 N times.

Lemma 4.7. Suppose Conditions I(b, δ) and II(b, δ, d) hold. If ${\tilde{\theta }}_s=s{\widehat{\theta }}_n+(1-s){\theta }_n^{*}$, where

$$s=\frac{d{\mathrm{\zeta }}_n^{*}}{d{\mathrm{\zeta }}_n^{*}+bI({\widehat{\theta }}_n-{\theta }_n^{*})},$$

then for any integer N, with probability at least

$$1-N\left\{\right(1+\frac{3}{10}{W}^2)\text{exp}(-n{\overline{a}}_n^2{r}_1^2)+2\text{exp}(-n{\mathrm{\xi }}^2/2\},$$

we have

$$I({\tilde{\theta }}_s-{\theta }_n^{*})\le (1+(d-1){(1+b)}^{-N}\frac{{\mathrm{\zeta }}_n^{*}}{b}.$$

Proof. Since the negative log partial likelihood l_n(θ) and the lasso penalty are both convex with respect to θ, applying Corollary 4.3, we obtain the above inequality. This proof is similar to the proof of Lemma A.6 in van de Geer (2008).

Lemma 4.8. Suppose Conditions I(b, δ) and II(b, δ, d) are met. Let N₁ ∈ N ≔ {1, 2, …} and N₂ ∈ N ∪ {0}. With δ₁ = (1+b)^−N₁ and δ₂ = (1+b)^−N₂ , for any n, with probability at least

$$1-(N_1+N_2)\left\{\right(1+\frac{3}{10}{W}^2)\text{exp}(-n{\overline{a}}_n^2{r}_1^2)+2\text{exp}(-n{\mathrm{\xi }}^2/2\},$$

we have

$$I({\widehat{\theta }}_n-{\theta }_n^{*})\le d({\delta }_1,{\delta }_2)\frac{{\mathrm{\zeta }}_n^{*}}{b},$$

where

$$d({\delta }_1,{\delta }_2)=1+\frac{1+(d^2-1){\delta }_1}{(d-1)(1-{\delta }_1)}{\delta }_2.$$

Proof. The proof is the same as that of Lemma A.7 in van de Geer (2008), with a slightly different probability bound.

4.2. Proof of Theorem 3.1

Proof. The proof follows the same ideas in the proof of Theorem A.4 in van de Geer (2008), with exceptions of pointwise arguments and slightly different probability bounds. Since this is our main result, we provide a detailed proof here despite the amount of overlaps.

Define ℰ̂ ≔ ℰ(f_θ̂n and ${ℰ}^{*}≔ℰ(f_{{\theta }_n^{*}})$; use the notation $I_1(\theta )≔I_1(\theta |{\theta }_n^{*})$ and $I_2(\theta )≔I_2(\theta |{\theta }_n^{*})$; set c ≔ δb/(1 − δ²). Consider the cases (a) c < d(δ₁, δ₂) and (b) c ≥ d(δ₁, δ₂).

(a) c < d(δ₁, δ₂). Let J be an integer satisfying (1 + b)^J−1 c ≤ d(δ₁, δ₂) and (1 + b)^J c > d(δ₁, δ₂). We consider the cases (a1) $c{\zeta }_n^{*}/b<I({\widehat{\theta }}_n-{\theta }_n^{*})\le d({\delta }_1,{\delta }_2){\zeta }_n^{*}/b$ and (a2) $I({\widehat{\theta }}_n-{\theta }_n^{*})\le c{\zeta }_n^{*}/b$.

(a1) If $c{\zeta }_n^{*}/b<I({\widehat{\theta }}_n-{\theta }_n^{*})\le d({\delta }_1,{\delta }_2){\zeta }_n^{*}/b$, then

$${(1+b)}^{j-1}c\frac{{\mathrm{\zeta }}_n^{*}}{b}<I({\widehat{\theta }}_n-{\theta }_n^{*})\le {(1+b)}^{j}c\frac{{\mathrm{\zeta }}_n^{*}}{b}$$

for some j ∈ {1, ⋯ , J}. Let d₀ = c(1 + b)^j−1 ≤ d(δ₁, δ₂) ≤ d_b. From Corollary 4.1, with probability at least $1-\text{exp}(-n{\overline{a}}_n^2{r}_1^2)$ we have $Z_{{\widehat{\theta }}_n}((1+b)d_0{\zeta }_n^{*}/b)\le (1+b)d_0{\overline{\lambda }}_{n,0}^A{\zeta }_n^{*}/b)$.

Since $l_n({\tilde{\theta }}_n)+{\lambda }_{n}I({\tilde{\theta }}_n)\le l_n({\theta }_n^{*})+{\lambda }_{n}I({\theta }_n^{*})$, from (4.9) we have

$$\widehat{ℰ}+{\mathrm{\lambda }}_{n}I({\widehat{\theta }}_n)\le R_{{\widehat{\theta }}_n}((1+b)d_0\frac{{\mathrm{\zeta }}_n^{*}}{b})+{ℰ}^{*}+{\mathrm{\lambda }}_{n}I({\theta }_n^{*})+(1+b){\overline{\mathrm{\lambda }}}_{n,0}^A{d}_0\frac{{\mathrm{\zeta }}_n^{*}}{b}.$$

By (4.8), $R_{{\widehat{\theta }}_n}((1+b)d_0{\zeta }_n^{*}/b)$ is bounded by $(1+b){\overline{\lambda }}_{n,0}^B{d}_0{\zeta }_n^{*}/b)$ with probability at least

$$1-\frac{3}{10}{W}^2\text{exp}(-n{\overline{a}}_n^2{r}_1^2)-2\text{exp}(-n{\mathrm{\xi }}^2/2).$$

Then we have

$$\widehat{ℰ}+(1+b){\overline{\mathrm{\lambda }}}_{n,0}I({\widehat{\theta }}_n)\le (1+b){\overline{\mathrm{\lambda }}}_{n,0}^B{d}_0\frac{{\mathrm{\zeta }}_n^{*}}{b}+{ℰ}^{*}+(1+b){\overline{\mathrm{\lambda }}}_{n,0}I({\theta }_n^{*})+(1+b){\overline{\mathrm{\lambda }}}_{n,0}^A{d}_0\frac{{\mathrm{\zeta }}_n^{*}}{b}\le (1+b){\overline{\mathrm{\lambda }}}_{n,0}I({\widehat{\theta }}_n-{\theta }_n^{*})+{ℰ}^{*}+(1+b){\overline{\mathrm{\lambda }}}_{n,0}I({\theta }_n^{*}).$$

Since $I({\widehat{\theta }}_n)=I_1({\widehat{\theta }}_n)+I_2({\widehat{\theta }}_n),I({\widehat{\theta }}_n-{\theta }_n^{*})=I_1({\widehat{\theta }}_n-{\theta }_n^{*})+I_2({\widehat{\theta }}_n)$, and $I({\theta }_n^{*})=I_1({\theta }_n^{*})$, by triangular inequality we obtain $\widehat{ℰ}\le 2(1+b){\overline{\lambda }}_{n,0}{I}_1({\widehat{\theta }}_n-{\theta }_n^{*})+{ℰ}^{*}$. From Lemma 4.5, $\widehat{ℰ}\le \delta \widehat{ℰ}+{\epsilon }_n^{*}-{ℰ}^{*}+{ℰ}^{*}=\delta \widehat{ℰ}+{\epsilon }_n^{*}$. Hence, $\widehat{ℰ}\le {\epsilon }_n^{*}/(1-\delta )$.

(a2) If $I({\widehat{\theta }}_n-{\theta }_n^{*})\le c{\zeta }_n^{*}/b$, from (4.10) with d₀ = c, with probability at least

$$1-\left\{\right(1+\frac{3}{10}{W}^2)\text{exp}(-n{\overline{a}}_n^2{r}_1^2)+2\text{exp}(-n{\mathrm{\xi }}^2/2)\},$$

we have

$$\widehat{ℰ}+(1+b){\overline{\mathrm{\lambda }}}_{n,0}I({\widehat{\theta }}_n)\le \frac{\delta }{1-{\delta }^2}{\overline{\mathrm{\lambda }}}_{n,0}{\mathrm{\zeta }}_n^{*}+{ℰ}^{*}+(1+b){\overline{\mathrm{\lambda }}}_{n,0}I({\theta }_n^{*}).$$

By the triangular inequality, Lemma 4.5 and (A4),

$$\widehat{ℰ}\le \frac{\delta }{1-{\delta }^2}{\overline{\mathrm{\lambda }}}_{n,0}{\mathrm{\zeta }}_n^{*}+{ℰ}^{*}+(1+b){\overline{\mathrm{\lambda }}}_{n,0}{I}_1({\widehat{\theta }}_n-{\theta }_n^{*})\le \frac{\delta }{1-{\delta }^2}{\overline{\mathrm{\lambda }}}_{n,0}\frac{{\epsilon }_n^{*}}{{\overline{\mathrm{\lambda }}}_{n,0}}+{ℰ}^{*}+\frac{\delta }{2}\widehat{ℰ}+\frac{1}{2}{\epsilon }_n^{*}-\frac{1}{2}{ℰ}^{*}=(\frac{\delta }{1-{\delta }^2}+\frac{1}{2}){\epsilon }_n^{*}+\frac{1}{2}{ℰ}^{*}+\frac{\delta }{2}\widehat{ℰ}\le (\frac{\delta }{1-{\delta }^2}+\frac{1}{2}){\epsilon }_n^{*}+\frac{1}{2(1+\delta )}{\epsilon }_n^{*}+\frac{\delta }{2}\widehat{ℰ}.$$

Hence,

$$\widehat{ℰ}\le \frac{2}{2-\delta }[\frac{\delta }{1-{\delta }^2}+\frac{1}{2}+\frac{1}{2(1+\delta )}]{\epsilon }_n^{*}=\frac{1}{1-\delta }{\epsilon }_n^{*}.$$

Furthermore, by Lemma 4.8, we have with probability at least

$$1-(N_1+N_2)\left\{\right(1+\frac{3}{10}{W}^2)\text{exp}(-n{\overline{a}}_n^2{r}_1^2)+2\text{exp}(-n{\mathrm{\xi }}^2/2)\}$$

that $I({\widehat{\theta }}_n-{\theta }_n^{*})\le d({\delta }_1,{\delta }_2)\frac{{\zeta }_n^{*}}{b}$, where

$$N_1={\text{log}}_{1+b}\left(\frac{1}{{\delta }_1}\right),N_2={\text{log}}_{1+b}\left(\frac{1}{{\delta }_2}\right).$$

(b) c ≥ d(δ₁, δ₂). On the set where $I({\widehat{\theta }}_n-{\theta }_n^{*})\le d({\delta }_1,{\delta }_2){\zeta }_n^{*}/b$, from equation (4.10) we have with probability at least

$$1-\left\{\right(1+\frac{3}{10}{W}^2)\text{exp}(-n{\overline{a}}_n^2{r}_1^2)+2\text{exp}(-n{\mathrm{\xi }}^2/2)\}$$

that

$$\widehat{ℰ}+(1+b){\overline{\mathrm{\lambda }}}_{n,0}I({\widehat{\theta }}_n)\le {\overline{\mathrm{\lambda }}}_{n,0}d({\delta }_1,{\delta }_2)\frac{{\mathrm{\zeta }}_n^{*}}{b}+{ℰ}^{*}+(1+b){\overline{\mathrm{\lambda }}}_{n,0}I({\theta }_n^{*})\le \frac{\delta }{1-{\delta }^2}{\overline{\mathrm{\lambda }}}_{n,0}{\mathrm{\zeta }}_n^{*}+{ℰ}^{*}+(1+b){\overline{\mathrm{\lambda }}}_{n,0}I({\theta }_n^{*}),$$

which is the same as (a2) and leads to the same result.

To summarize, let

$$A=\{\widehat{ℰ}\le \frac{1}{1-\delta }{\epsilon }_n^{*}\},B=\{I({\widehat{\theta }}_n-{\theta }_n^{*})\le d({\delta }_1,{\delta }_2)\frac{{\mathrm{\zeta }}_n^{*}}{b}\}.$$

Note that

$$J+1\le {\text{log}}_{1+b}\left(\frac{{(1+b)}^{2}d({\delta }_1,{\delta }_2)}{c}\right).$$

Under case (a), we have

$$P(A\cap B)=P(\mathrm{a}1)-P(A^c\cap \mathrm{a}1)+P(\mathrm{a}2)-P(A^c\cap \mathrm{a}2)\ge P(\mathrm{a}1)-J\left\{\right(1+\frac{3}{10}{W}^2)\text{exp}(-n{\overline{a}}_n^2{r}_1^2)+2\text{exp}(-n{\mathrm{\xi }}^2/2)\}+P(\mathrm{a}2)-\left\{\right(1+\frac{3}{10}{W}^2)\text{exp}(-n{\overline{a}}_n^2{r}_1^2)+2\text{exp}(-n{\mathrm{\xi }}^2/2)\}=P(B)-(J+1)\left\{\right(1+\frac{3}{10}{W}^2)\text{exp}(-n{\overline{a}}_n^2{r}_1^2)+2\text{exp}(-n{\mathrm{\xi }}^2/2)\}\ge 1-(N_1+N_2+J+1)\left\{\right(1+\frac{3}{10}{W}^2)\text{exp}(-n{\overline{a}}_n^2{r}_1^2)+2\text{exp}(-n{\mathrm{\xi }}^2/2)\}\ge 1-{\text{log}}_{1+b}\{\frac{{(1+b)}^2}{{\delta }_1{\delta }_2}·\frac{d({\delta }_1,{\delta }_2)(1-{\delta }^2)}{\delta b}\}\left\{\right(1+\frac{3}{10}{W}^2)\text{exp}(-n{\overline{a}}_n^2{r}_1^2)+2\text{exp}(-n{\mathrm{\xi }}^2/2)\}.$$

Under case (b),

$$P(A\cap B)=P(B)-P(A^c\cap B)\ge P(B)-\left\{\right(1+\frac{3}{10}{W}^2)\text{exp}(-n{\overline{a}}_n^2{r}_1^2)+2\text{exp}(-n{\mathrm{\xi }}^2/2)\}\ge 1-(N_1+N_2+2)\left\{\right(1+\frac{3}{10}{W}^2)\text{exp}(-n{\overline{a}}_n^2{r}_1^2)+2\text{exp}(-n{\mathrm{\xi }}^2/2)\}=1-{\text{log}}_{1+b}\left\{\frac{{\left(1+b\right)}^2}{{\delta }_1{\delta }_2}\right\}\left\{\right(1+\frac{3}{10}{W}^2)\text{exp}(-n{\overline{a}}_n^2{r}_1^2)+2\text{exp}(-n{\mathrm{\xi }}^2/2)\}.$$

We thus obtain the desired result.