ORACLE INEQUALITIES FOR THE LASSO IN THE COX MODEL

Abstract

We study the absolute penalized maximum partial likelihood estimator in sparse, high-dimensional Cox proportional hazards regression models where the number of time-dependent covariates can be larger than the sample size. We establish oracle inequalities based on natural extensions of the compatibility and cone invertibility factors of the Hessian matrix at the true regression coefficients. Similar results based on an extension of the restricted eigenvalue can be also proved by our method. However, the presented oracle inequalities are sharper since the compatibility and cone invertibility factors are always greater than the corresponding restricted eigenvalue. In the Cox regression model, the Hessian matrix is based on time-dependent covariates in censored risk sets, so that the compatibility and cone invertibility factors, and the restricted eigenvalue as well, are random variables even when they are evaluated for the Hessian at the true regression coefficients. Under mild conditions, we prove that these quantities are bounded from below by positive constants for time-dependent covariates, including cases where the number of covariates is of greater order than the sample size. Consequently, the compatibility and cone invertibility factors can be treated as positive constants in our oracle inequalities.

1. Introduction

The Cox (1972) proportional hazards model is widely used in the regression analysis of censored survival data, notably in identifying risk factors in epidemiological studies and treatment effects in clinical trials when the outcome variable is time to event. In a traditional biomedical study, the number of covariates p is usually relatively small as compared with the sample size n. Theoretical properties of the maximum partial likelihood estimator in the fixed p and large n setting have been well established. For example, Tsiatis (1981) proved the asymptotic normality of the maximum partial likelihood estimator. Andersen and Gill (1982) formulated the Cox model in the context of the more general counting process framework and studied the asymptotic properties of the maximum partial likelihood estimator using martingale techniques. These results provide a solid foundation for the applications of the Cox model in a diverse range of problems where time to event is the outcome of interest.

In recent years, technological advancement has resulted in the proliferation of massive high-throughput and high-dimensional genomic data in studies that attempt to find genetic risk factors for disease and clinical outcomes, such as the age of disease onset or time to death. Finding genetic risk factors for survival is fundamental to modern biostatistics, since survival is an important clinical endpoint. However, in such problems, the standard approach to the Cox model is not applicable, since the the number of potential genetic risk factors is typically much larger than the sample size. In addition, traditional variable selection methods such as subset selection are not computationally feasible when p >> n.

The ℓ₁-penalized least squares estimator, or the Lasso, was introduced by Tibshirani (1996). In the wavelet setting, the ℓ₁ penalized method was introduced by Chen, Donoho and Saunders (1998) as Basis Pursuit. Since then, the Lasso has emerged as a widely used approach to variable selection and estimation in sparse, high-dimensional statisticalproblems. It has also been extended to the Cox model (Tibshirani, 1997). Gui and Li (2005) implemented the LARS algorithm (Efron et al., 2004) to approximate the Lasso in the Cox regression model and applied their method to survival data with microarray gene expression covariates. Their work demonstrated the effectiveness of the Lasso for variable selection in the Cox model in a p >> n setting.

There exists a substantial literature on the Lasso and other penalized methods for survival models with a fixed number of covariates p. Zhang and Lu (2007) considered an adaptive Lasso for the Cox model and showed that, under certain regularity conditions and with a suitable choice of the penalty parameter, their method possesses the asymptotic oracle property when the maximum partial likelihood estimator is used as the initial estimator. Fan and Li (2002) proposed the use of the smoothly clipped absolute deviation (SCAD) penalty (Fan, 1997; Fan and Li, 2001) for variable selection and estimation in the Cox model which may include a frailty term. With a suitable choice of the penalty parameter, they showed that a local maximizer of the penalized log-partial likelihood has an asymptotic oracle property under certain regularity conditions on the Hessian of the log-partial likelihood and the censoring mechanism.

Extensive efforts have been focused upon the analysis of regularization methods for variable selection in the p >> n setting. In particular, considerable progress has been made in theoretical understanding of the Lasso. However, most results are developed in the linear regression model. Greenshtein and Ritov (2004) studied the prediction performance of the Lasso in high-dimensional linear regression. Meinshausen and Bühlmann (2006) showed that, for neighborhood selection in the Gaussian graphical model, under a neighborhood stability condition and certain additional regularity conditions, the Lasso is consistent even when p/n → ∞. Zhao and Yu (2006) formalized the neighborhood stability condition in the context of linear regression as a strong irrepresentable condition on the design matrix. Oracle inequalities for the prediction and estimation error of the Lasso was developed in Bunea, Tsybakov and Wegkamp (2007), Zhang and Huang (2008), Meinshausen and Yu (2009), Bickel, Ritov and Tsybakov (2009), Zhang (2009) and Ye and Zhang (2010), among many others.

A number of papers analyzed penalized methods beyond linear regression. Fan and Peng (2004) established oracle properties for a local solution of concave penalized estimator in a general setting with n >> p → ∞. van de Geer (2008) studied the Lasso in high-dimensional generalized linear models (GLM) and obtained prediction and ℓ₁ estimation error bounds. Negahban, Ravikumar, Wainwright and Yu (2010) studied penalized M-estimators with a general class of regularizers, including an ℓ₂ error bound for the Lasso in GLM under a restricted convexity and other regularity conditions. Bradic et al (2011) made significant progress by extending the results of Fan and Li (2001) to a more general class of penalties in the Cox regression model with large p under different sets of regularity conditions. Huang and Zhang (2012) studied weighted absolute penalty and its adaptive, multistage application in GLM.

In view of the central role of the Cox model in survival analysis, its widespread applications and the proliferation of p >> n data, it is of great interest to understand the properties of the related Lasso approach. The main goal of the present paper is to establish theoretical properties for the Lasso in the Cox model when p >> n. Specifically, we extend certain basic inequalities from linear regression to the case of the Cox regression. We generalize the compatibility and cone invertibility factors from the linear regression model and establish oracle inequalities for the Lasso in the Cox regression model in terms of these factors at the true parameter value. Moreover, we prove that the compatibility and cone invertibility factors can be treated as constants under mild regularity conditions.

A main feature of our results is that they are derived under the more general counting process formulation of the Cox model with potentially a larger number of time-dependent covariates than the sample size. This formulation is useful because it “permits a regression analysis of the intensity of a recurrent event allowing for complicated censoring patterns and time-dependent covariates” (Andersen and Gill, 1982).

A second main feature of our results is that the regularity conditions on the counting processes and time-dependent covariates are directly imposed on the compatibility and cone invertibility factors of the Hessian of the negative log-partial likelihood evaluated at the true regression coefficients. Under such regularity conditions, the Lasso estimator is proven to live in a neighborhood where the ratio between the estimated and true hazards is uniformly bounded away from zero and infinity. This allows unbounded and near zero ratios between the true and baseline hazards. Our analysis can be also used to prove oracle inequalities based on the restricted eigenvalue. However, since the compatibility and cone invertibility factors are greater than the corresponding restricted eigenvalue (van de Geer and Bühlmann, 2009; Ye and Zhang, 2010), the presented results are sharper.

A third main feature of our results is that the compatibility and cone invertibility factors used, and the smaller corresponding restricted eigenvalue, are proven to be greater than a fixed positive constant under mild conditions on the counting processes and time-dependent covariates, including cases where p >> n. In the Cox regression model, the Hessian matrix is based on weighted averages of the cross-products of time-dependent covariates in censored risk sets, so that the compatibility and cone invertibility factors and the restricted eigenvalue are random variables even when they are evaluated for the Hessian at the true regression coefficients. Under mild conditions, we prove that these quantities are bounded from below by positive constants as certain truncated population versions of them. Thus, the compatibility and cone invertibility factors can be treated as constants in our oracle inequalities.

The main results of this paper and the analytical methods used for deriving them are identical to those in its predecessor submitted in November 2011, with Section 4 as an exception. The difference in Section 4 is that the lower bound for the compatibility and cone invertibility factors and the restricted eigenvalue is improved to allow time-dependent covariates.

During the revision process of our paper, we became aware of a number of papers on hazard regression with censored data. Kong and Nan (2012) took an approach of van de Geer (2008) to derive prediction and ℓ₁ error bounds for the Lasso in the Cox proportional hazards regression under a quite different set of conditions from us. For example, they required an ℓ₁ bound on the regression coefficients to guarantee a uniformly bounded ratio between hazard functions under consideration. Lemler (2012) considered the joint estimation of the baseline hazard function and regression coefficients in the Cox model. As a result, Lemler’s (2012) error bounds for regression coefficients are of greater order than ours when the intrinsic dimension of the unknown baseline hazard function is of greater order than the number of nonzero regression coefficients. Gaïffas and Guilloux (2012) considered a quadratic loss function in place of a negative log-likelihood function in an additive hazards model. A nice feature of the additive hazards model is that the quadratic loss actually produces unbiased linear estimation equations so that the analysis of the Lasso is similar to that of linear regression. The oracle inequalities in these three papers and ours can be all viewed as non-asymptotic. Unlike our paper, none of these three papers consider time-dependent covariates or constant lower bounds of the restricted eigenvalue or related key factors for the analysis of the Lasso.

The rest of this paper is organized as follows. In Section 2 we provide basic notation and model specifications. In Section 3 we develop oracle inequalities for the Lasso in the Cox model. In Section 4 we study the compatibility and cone invertibility factors and the corresponding restricted eigenvalue of the Hessian of the log-partial likelihood in the Cox model. In Section 5 we make some additional remarks. All proofs are provided either right after the statement of the result or deferred to the Appendix.

2. Cox model with the ℓ₁ penalty

Following Andersen and Gill (1982), consider an n-dimensional counting process N⁽ⁿ⁾(t) = (N₁(t), …, N_n(t)), t ≥ 0, where N_i(t) counts the number of observed events for the ith individual in the time interval [0, t]. The sample paths of N₁, …, N_n are step functions, zero at t = 0, with jumps of size +1 only. Furthermore, no two components jump at the same time. For t = 0, let ℱ_t be the σ-filtration representing all the information available up to time t. Assume that for {ℱ_t, t ≥ 0}, N⁽ⁿ⁾ has predictable compensator Λ⁽ⁿ⁾ = (Λ₁, …, Λ_n) with

$$d{\mathrm{\Lambda }}_i(t)=Y_i(t)\text{exp}\{{\mathit{Z}}_i^{\prime }(t){\mathit{\beta }}^o\}d{\mathrm{\Lambda }}_0(t),$$ (2.1)

where β^o is a p-vector of true regression coefficients, Λ₀ is an unknown baseline cumulative hazard function and, for each i, Y_i(t) ∈ {0, 1} is a predictable at risk indicator process that can be constructed from data, and Z_i(t) = (Z_i,1(t), …, Z_i,p(t))′ is a p-dimensional vector-valued predictable covariate process. In this setting the σ-filtration can be the natural ℱ_t = σ{N_i(s), Y_i(s),Z_i(s); s ≤ t, i = 1, …, n} or a richer one. We are interested in the problem of variable selection in sparse, high-dimensional settings where p, the number of possible covariates, is large, but the number of important covariates is relatively small.

2.1. Maximum partial likelihood estimator with ℓ₁ penalty

Define logarithm of the Cox partial likelihood for survival experience at time t,

$$C(\mathit{\beta };t)=\sum _{i=1}^n{\int }_0^t{\mathit{Z}}_i^{\prime }(s)\mathit{\beta }d{N}_i(s)-{\int }_0^t\text{log}[\sum _{i=1}^n{Y}_i(s)e^{{\mathit{Z}}_i^{\prime }(s)\mathit{\beta }}]d\mathrm{N̄}(s),$$

where $\mathrm{N̄}={\sum }_{i=1}^n{N}_i$. The log-partial likelihood function is C(β, ∞) = lim_t→∞ C(β, t). Let ℓ(β) = −C(β; ∞)/n. The maximum partial likelihood estimator is the value that minimizes ℓ(β).

An approach to variable selection in sparse, high-dimensional settings for the Cox model is to minimize an ℓ₁-penalized negative log-partial likelihood criterion,

$$L(\mathit{\beta };\lambda )=\ell (\mathit{\beta })+\lambda {|\mathit{\beta }|}_1$$ (2.2)

(Tibshirani, 1997), where λ ≥ 0 is a penalty parameter. Henceforth, we use notation ${|\mathit{\beta }|}_q={\{{\sum }_{i=1}^p{|{\beta }_i|}^q\}}^{1/q}$ for 1 ≤ q < ∞, |β|_∞ = max_1≤i≤p |β_i| and |β|₀ = #{j : β_j ≠ 0}. For a given λ, the ℓ₁-penalized maximum partial likelihood estimator, or the Lasso estimator hereafter, is defined as

$$\mathit{\beta ̂}(\lambda )=\underset{\mathit{\beta }}{\text{argmin}}L(\mathit{\beta };\lambda ).$$ (2.3)

2.2. The Karush-Kuhn-Tucker conditions

The Lasso estimator can be characterized by the Karush-Kuhn-Tucker (KKT) conditions. Since the log-partial likelihood belongs to an exponential family, ℓ(β) must be convex in β and so is L(β; λ). It follows that a vector β̂ = (β̂₁, …, β̂_p)′ is a solution to (2.3) if and only if the following KKT conditions hold:

$$\{\begin{array}{c}{\mathrm{\ell ̇}}_j(\mathit{\beta ̂})=-\lambda \text{sgn}({\mathrm{\beta ̂}}_j),\text{if}{\mathrm{\beta ̂}}_j\ne 0,\hfill \\ |{\mathrm{\ell ̇}}_j(\mathit{\beta ̂})|\le \lambda ,\text{if}{\mathrm{\beta ̂}}_j=0,\hfill \end{array}$$ (2.4)

where ℓ̇(β) = (ℓ̇₁(β), …, ℓ̇_p(β))′ = ∂ℓ(β)/∂β is the gradient of ℓ(β). The necessity and sufficiency of (2.4) can be proved by subdifferentiation of the convex penalized loss (2.2). This does not require strict convexity.

The KKT conditions indicate that the Lasso in the Cox regression model may be analyzed in a similar way to the Lasso in linear regression. As can be seen in the subsequent developments, such analysis can be carried out by proving that |ℓ̇(β^o)|_∞ is sufficiently small and the Hessian of ℓ(β) does not vanish for a sparse β at the true β = β^o. The (local) martingales for the counting process will play a crucial role to ensure that these requirements are satisfied.

2.3. Additional notation

Since the Λ_i are compensators,

$$M_i(t)=N_i(t)-{\int }_0^t{Y}_i(s)\text{exp}({\mathit{Z}}_i^{\prime }(s){\mathit{\beta }}^o)d{\mathrm{\Lambda }}_0(s),1\le i\le n,t\ge 0,$$

are (local) martingales with predictable variation/covariation processes

$$\langle M_i,M_i\rangle (t)={\int }_0^t{Y}_i(s)\text{exp}({\mathit{Z}}_i^{\prime }(s){\mathit{\beta }}^o)d{\mathrm{\Lambda }}_0(s)\text{and}\langle M_i,M_j\rangle =0,i\ne j.$$

For a vector υ, let υ^⊗0 = 1 ∈ IR, υ^⊗1 = υ and υ^⊗2 = υυ′. Define

$$S^{(k)}(t,\mathit{\beta })=\frac{1}{n}\sum _{i=1}^n{\mathit{Z}}_i^{\otimes k}(t)Y_i(t)e^{{\mathit{Z}}_i^{\prime }(t)\mathit{\beta }},k=0,1,2,R_n(t,\mathit{\beta })=\frac{1}{n}\sum _{i=1}^n{Y}_i(t)e^{{\mathit{Z}}_i^{\prime }(t)\mathit{\beta }},{\mathbf{Z̄}}_n(t,\mathit{\beta })=\frac{S^{(1)}(t,\mathit{\beta })}{S^{(0)}(t,\mathit{\beta })},V_n(t,\mathit{\beta })=\sum _{i=1}^n{w}_{ni}(t,\mathit{\beta }){({\mathit{Z}}_i(t)-{\mathbf{Z̄}}_n(t,\mathit{\beta }))}^{\otimes 2}=\frac{S^{(2)}(t,\mathit{\beta })}{S^{(0)}(t,\mathit{\beta })}-{\mathbf{Z̄}}_n{(t,\mathit{\beta })}^{\otimes 2},$$

where $w_{ni}(t,\mathit{\beta })=Y_i(t)\text{exp}({\mathit{Z}}_i^{\prime }(t)\mathit{\beta })/[n{S}^{(0)}(t,\mathit{\beta })]$. By differentiation and rearrangement of terms, it can be shown as in Anderson and Gill (1982) that the gradient of ℓ(β) is

$$\mathrm{\ell ̇}(\mathit{\beta })\equiv \frac{\partial \ell (\mathit{\beta })}{\partial \mathit{\beta }}=-\frac{1}{n}\sum _{i=1}^n{\int }_0^{\infty }[{\mathit{Z}}_i(s)-{\mathbf{Z̄}}_n(s,\mathit{\beta })]d{N}_i(s),$$ (2.5)

and the Hessian matrix of ℓ(β) is

$$\mathrm{\ell ̈}(\mathit{\beta })\equiv \frac{{\partial }^2\ell (\mathit{\beta })}{\partial \mathit{\beta }\partial \mathit{\beta }\prime }=\frac{1}{n}{\int }_0^{\infty }{V}_n(s,\mathit{\beta })d\mathrm{N̄}(s).$$ (2.6)

3. Oracle inequalities

In this section, we derive oracle inequalities for the estimation error of Lasso in the Cox regression model. Let β^o be the vector of true regression coefficients and define $𝒪=\{j:{\beta }_j^o\ne 0\},{𝒪}^c=\{j:{\beta }_j^o=0\}$ and d_o = |𝒪|, where |𝒰| for a set 𝒰 denotes its cardinality.

Making use of the KKT conditions (2.4), we first develop a basic inequality involving the symmetric Bregman divergence and ℓ₁ estimation error in the support 𝒪 of β^o and its complement. The symmetric Bregman divergence, defined as

$$D^s(\mathit{\beta ̂},\mathit{\beta })=(\mathit{\beta ̂}-\mathit{\beta })\prime (\mathrm{\ell ̇}(\mathit{\beta ̂})-\mathrm{\ell ̇}(\mathit{\beta })),$$

can be viewed as symmetric partial Kullback-Leibler distance between the partial likelihood at β̂ and β. Thus, D^s(β̂, β) can be viewed as a measure of prediction performance. The basic inequality, given in Lemma 3.1 below, serves as a vehicle for establishing the desired oracle inequalities.

Lemma 3.1. Let β̂ be defined as in (2.3), θ̃ = β̂ − β^o and z* = |ℓ̇(β^o)|_∞. Then, the following inequalities hold,

$$(\lambda -z^{*}){|{\mathit{\theta ̃}}_{{𝒪}^c}|}_1\le D^s(\mathit{\beta ̂},\mathit{\beta })+(\lambda -z^{*}){|{\mathit{\theta ̃}}_{{𝒪}^c}|}_1\le (\lambda +z^{*}){|{\mathit{\theta ̃}}_{𝒪}|}_1,$$ (3.1)

where θ̃_𝒪 and θ̃_𝒪^c denote the subvectors of θ̃ of components in 𝒪 and 𝒪^c, respectively. In particular, for any ξ > 1, |θ̃_𝒪^c|₁ ≤ ξ|θ̃_𝒪|₁ in the event z* ≤ (ξ − 1)/(ξ + 1)λ.

It follows from Lemma 3.1 that in the event z* ≤ (ξ − 1)/(ξ + 1)λ, the estimation error θ̃ = β̂ − β^o belongs to the cone

$$𝒞(\xi ,𝒪)=\{\mathit{b}\in \mathrm{I}{\mathrm{R}}^p:{|{\mathit{b}}_{{𝒪}^c}|}_1\le \xi {|{\mathit{b}}_{𝒪}|}_1\}.$$ (3.2)

In linear regression, the invertibility of the Gram matrix in the same cone, expressed in terms of restricted eigenvalues and related quantities, has been used to control the estimation error of the Lasso. In what follows, we prove that a direct extension of the compatibility and cone invertibility factors can be used to control the estimation error of the Lasso in the Cox regression.

For the cone in (3.2) and a given p × p nonnegative-definite matrix Σ̅, define

$$\kappa (\xi ,𝒪;\mathbf{\Sigma ̅})=\underset{0\ne \mathit{b}\in 𝒞(\xi ,𝒪)}{\text{inf}}\frac{d_o^{1/2}{(\mathit{b}\prime \mathbf{\Sigma ̅}\mathit{b})}^{1/2}}{{|{\mathit{b}}_{𝒪}|}_1},$$ (3.3)

as the compatibility factor (van de Geer, 2007; van de Geer and Bühlmann, 2009), and

$$F_q(\xi ,𝒪;\mathbf{\Sigma ̅})=\underset{0\ne \mathit{b}\in 𝒞(\xi ,𝒪)}{\text{inf}}\frac{d_o^{1/q}\mathit{b}\prime \mathbf{\Sigma ̅}\mathit{b}}{{|{\mathit{b}}_{𝒪}|}_1{|\mathit{b}|}_q}$$ (3.4)

as the weak cone invertibility factor (Ye and Zhang, 2010). These quantities are closely related to the restricted eigenvalue (Bickel, Ritov and Tsybakov, 2009; Koltchinskii, 2009),

$$\mathrm{R}\mathrm{E}(\xi ,𝒪;\mathbf{\Sigma ̅})=\underset{0\ne \mathit{b}\in 𝒞(\xi ,𝒪)}{\text{inf}}\frac{{(\mathit{b}\prime \mathbf{\Sigma ̅}\mathit{b})}^{1/2}}{{|\mathit{b}|}_2}.$$ (3.5)

In linear regression, the Hessian of the squared loss ${|\mathit{y}-\mathit{X}\mathit{\beta }|}_2^2/(2n)$ is taken as Σ̅, and the oracle inequalities established in the papers cited in the above paragraph can be summarized as follows: in the event z* = |X′(y − Xβ^o)/n|_∞ ≤ λ(ξ − 1)/(ξ + 1),

$${|\mathit{X}(\mathit{\beta ̂}-{\mathit{\beta }}^o)|}_2^2/n\le \frac{4{(1+1/\xi )}^{-2}{\lambda }^2{d}_o}{{\kappa }^2(\xi ,𝒪;\mathit{X}\prime \mathit{X}/n)},{|\mathit{\beta ̂}-{\mathit{\beta }}^o|}_1\le \frac{2\xi d_o\lambda }{{\kappa }^2(\xi ,𝒪;\mathit{X}\prime \mathit{X}/n)},$$ (3.6)

and

$${|\mathit{\beta ̂}-{\mathit{\beta }}^o|}_2\le \frac{2{(1+1/\xi )}^{-1}{d}_o^{1/2}\lambda }{\mathrm{R}{\mathrm{E}}^2(\xi ,𝒪;\mathit{X}\prime \mathit{X}/n)},{|\mathit{\beta ̂}-{\mathit{\beta }}^o|}_q\le \frac{2{(1+1/\xi )}^{-1}{d}_o^{1/q}\lambda }{F_q(\xi ,𝒪;\mathit{X}\prime \mathit{X}/n)},q\ge 1.$$ (3.7)

In the Cox regression model, we still take the Hessian of the log-partial likelihood as Σ̅, in fact the Hessian at the true β^o, so that (3.3) and (3.4) become

$$\kappa (\xi ,𝒪)=\kappa (\xi ,𝒪;\mathrm{\ell ̈}({\mathit{\beta }}^o)),F_q(\xi ,𝒪)=F_q(\xi ,𝒪;\mathrm{\ell ̈}({\mathit{\beta }}^o)).$$ (3.8)

The reason for using these factors is that they yield somewhat sharper oracle inequalities than the restricted eigenvalue. It follows from ${|{\mathit{b}}_{𝒪}|}_1\le d_o^{1/2}{|\mathit{b}|}_2$ that F₂(ξ, 𝒪) ≥κ(ξ, 𝒪)RE(ξ, 𝒪) and κ(ξ, 𝒪) ≥ RE(ξ, 𝒪). Therefore, the first inequality of (3.7) is subsumed by the second with q = 2. Moreover, it is possible to have κ(ξ, 𝒪) >> RE(ξ, 𝒪) (van de Geer and Bühlmann, 2009), and consequently, the ℓ₂ error bound based on the cone invertibility factor may be of sharper order that the one based on the restricted eigenvalue. The following theorem extends (3.6) and (3.7) from the linear regression model to the proportional hazards regression model. Let

$$\underset{i<i\prime \le n}{\text{max}}\underset{0\le t<\infty }{\text{sup}}\underset{j\le p}{\text{max}}|Z_{i,j}(t)-Z_{i\prime ,j}(t)|\le K.$$ (3.9)

Let $𝒪=\{j:{\beta }_j^o\ne 0\}$ and F_q(ξ, 𝒪) be as in (3.8).

Theorem 3.1. Let τ = K(ξ + 1)d_oλ/{2κ²(ξ, 𝒪)} with a certain K > 0. Suppose condition (3.9) holds and τ ≤ 1/e. Then, in the event |ℓ̇(β^o)|_∞ ≤ (ξ − 1)/(ξ + 1)λ,

$$D^s(\mathit{\beta ̂},\mathit{\beta })\le \frac{4e^{\eta }{(1+1/\xi )}^{-2}{\lambda }^2{d}_o}{{\kappa }^2(\xi ,𝒪)},{|\mathit{\beta ̂}-{\mathit{\beta }}^o|}_1\le \frac{e^{\eta }(\xi +1)d_o\lambda }{2{\kappa }^2(\xi ,𝒪)},$$ (3.10)

and

$${|\mathit{\beta ̂}-{\mathit{\beta }}^o|}_q\le \frac{e^{\eta }2{(1+1/\xi )}^{-1}{d}_o^{1/q}\lambda }{F_q(\xi ,𝒪)},q\ge 1,$$ (3.11)

where η ≤ 1 is the smaller solution of η^e−η = τ.

Compared with (3.6) and (3.7), the new inequalities (3.10) and (3.11) contain an extra factor e^η ≤ e. This is due to the nonlinearity in the Cox regression score equation. Aside from this factor, the error bounds for the Cox regression have the same form as those for linear regression, except for an improvement of a factor of 4ξ/(1 + ξ) ≥ 2 for the ℓ₁ oracle inequality.

The theorem assumes condition (3.9), which asserts |Z_i(t) − Zi′(t)|_∞ ≤ K uniformly in {t, i, i′}. This condition is a consequence of the uniform boundedness of the individual covariates, and is reasonable in most practical situations (e.g. single-nucleotide polymorphism data). In the case where the covariates are normal variables with uniformly bounded variance, the condition holds with K = K_n,p of $\sqrt{\text{log}(np)}$ order.

From an analytical perspective, an important feature of (3.10) and (3.11) is that the constant factors (3.3) and (3.4) are both defined with the true β^o in (3.8). No condition is imposed on the gradient and Hessian of the log-partial likelihood for β ≠ β^o. In other words, the key condition τ < 1/e, expressed in terms of {K, d^o, λ} and the compatibility factor κ²(ξ, 𝒪) at the true β^o, is sufficient to guarantee the error bounds in Theorem 3.1. Thus, our results are much simpler to state and conditions easier to verify than existing ones requiring regularity conditions in a neighborhood of β^o in the Cox regression model. This feature of Theorem 3.1 plays a crucial role in our derivation of lower bounds for κ²(ξ, 𝒪) and F_q(ξ, 𝒪) for time-dependent covariates in Section 4. We note that the local martingale structure is valid only at the true β^o.

To prove Theorem 3.1, we develop a sharpened version of an inequality of Hjort and Pollard (1993). This inequality, given in Lemma 3.2 below, explicitly controls the symmetric Bregman-divergence and Hessian of the log-partial likelihood in a neighborhood of β. Based on this relationship, Theorem 3.1 is proved using the definition of the quantities in (3.8) and the membership of the error β̂ − β^o in the cone 𝒞 (ξ, 𝒪) (3.2). For two symmetric matrices A and B, A ≤ B means B − A is nonnegative-definite.

Lemma 3.2. Let ℓ(β) and its Hessian ℓ̈(β) be as in (2.2) and (2.6). Then,

$$e^{-{\eta }_{\mathit{b}}}\mathit{b}\prime \mathrm{\ell ̈}(\mathit{\beta })\mathit{b}\le D^s(\mathit{\beta }+\mathit{b},\mathit{\beta })=\mathit{b}\prime [\mathrm{\ell ̇}(\mathit{\beta }+\mathit{b})-\mathrm{\ell ̇}(\mathit{\beta })]\le e^{{\eta }_{\mathit{b}}}\mathit{b}\prime \mathrm{\ell ̈}(\mathit{\beta })\mathit{b},$$ (3.12)

where η_b = max_0≤s≤1 max_i,j|b′Z_i(s) − b′Z_j(s)|. Moreover,

$$e^{-2{\eta }_{\mathit{b}}}\mathrm{\ell ̈}(\mathit{\beta })\le \mathrm{\ell ̈}(\mathit{\beta }+\mathit{b})\le e^{2{\eta }_{\mathit{b}}}\mathrm{\ell ̈}(\mathit{\beta }).$$ (3.13)

Under the conditions of Theorem 3.1, the factors e^±η_b and e^±2η_b in the inequalities in Lemma 3.2 are bounded by positive constants. These factors lead to the factor e^η for η ≤ 1 (and thus e^η ≤ e) in the upper bounds in (3.10) and (3.11).

Since the oracle inequalities in Theorem 3.1 are guaranteed to hold only within the event |ℓ̇(β^o)|_∞ ≤ (ξ − 1)/(ξ +1)λ, a probabilistic upper bound is needed for |ℓ̇(β^o)|_∞. Lemma 3.3 below provides such a probability bound. Similar inequalities can be found in de la Peña (1999).

Lemma 3.3. (i) Let $f_n(t)=n^{-1}{\sum }_{i=1}^n{\int }_0^t{a}_i(s)\{d{N}_i(t)-Y_i(s)\text{exp}({\mathit{Z}}_i^{\prime }(s){\mathit{\beta }}^o)d{\mathrm{\Lambda }}_0(s)\}$ with [−1, 1]-valued predictable processes a_i(s). Then,

$$\mathrm{P}\{\underset{t>0}{\text{max}}|f_n(t)|>C_{0}x,\sum _{i=1}^n{\int }_0^{\infty }{Y}_i(t)d{N}_i(t)\le C_0^{2}n\}\le 2e^{-n{x}^2/2},\forall C_0>0.$$ (3.14)

(ii) Suppose that max_i≤n sup_t≤0 max_j≤p |Z_i,j(t) − Z̄_n,j(t, β^o)|_∞ ≤ K, where Z̄_n,j(t, β^o) are the components of Z̄_n(t, β^o). Let ℓ̇(β) be the gradient in (2.5). Then,

$$\mathrm{P}\{{|\mathrm{\ell ̇}({\mathit{\beta }}^o)|}_{\infty }>C_{0}Kx,\sum _{i=1}^n{\int }_0^{\infty }{Y}_i(t)d{N}_i(t)\le C_0^{2}n\}\le 2p{e}^{-n{x}^2/2},\forall C_0>0.$$ (3.15)

In particular, if max_i≤n N_i(1) ≤ 1, then P{|ℓ̇(β^o)|_∞ > Kx} ≤ 2pe^−nx2/2.

The following theorem states an upper bound of the estimation error, which follows directly from Theorem 3.1 and Lemma 3.3.

Theorem 3.2. Suppose (3.9) holds and N_i(∞) ≤ 1 for all i ≤ n and t ≥ 0. Let ξ > 1 and $\lambda =\{(\xi +1)/(\xi -1)\}K\sqrt{(2/n)\text{log}(2p/\epsilon )}$ with a small ε > 0 (e.g. ε = 1%). Let C_κ > 0 satisfying $\tau =K(\xi +1)d_o\lambda /(2C_{\kappa }^2)\le 1/e$. Let η ≤ 1 be the smaller solution of ηe^−η = τ. Then, for any C_F,q > 0,

$$D^s(\mathit{\beta ̂},\mathit{\beta })\le \frac{4e^{\eta }{\xi }^2{\lambda }^2{d}_o}{{(1+\xi )}^2{C}_{\kappa }^2},{|\mathit{\beta ̂}-{\mathit{\beta }}^o|}_1\le \frac{e^{\eta }(\xi +1)d_o\lambda }{2C_{\kappa }^2},{|\mathit{\beta ̂}-{\mathit{\beta }}^o|}_q\le \frac{2e^{\eta }\xi d_o^{1/q}\lambda }{(\xi +1)C_{F,q}}$$

all hold with at least probability P{κ(ξ, 𝒪) ≥ C_κ, F_q(ξ, 𝒪) ≥ C_F,q} − ε.

It is noteworthy that this theorem gives an upper bound of the estimation error for all the ℓ_q norms with q ≥ 1. From this theorem, for the ℓ_q error |β̂ − β^o|_q with q ≥ 1 to be small with high probability, we need to ensure that d_oλ → 0 as n → ∞. This requires $p=\text{exp}(o(n/d_o^2))$. If d_o is bounded, then p can be as large as e^o(n).

Bradic et al (2011) considered estimation as well as variable selection and oracle properties for general concave penalties, including the Lasso. Their broader scope seems to have led to more elaborate statements and some key conditions harder to verify than those of Theorems 3.1 and 3.2, e.g., their Condition 2 (i) on a uniformly small spectrum bound between S⁽²⁾(t, β₁) and its population version for a sparse β₁ in a neighborhood of β^o.

Proof of Theorem 3.2. Let C₀ = 1 and $x=\lambda (\xi -1)/\{K(\xi +1)\}=\sqrt{(2/n)\text{log}(2p/\epsilon )}$ in Lemma 3.3. The probability of the event |ℓ̇(β^o)|_∞ > (ξ − 1)/(ξ + 1)λ is at most ε. The desired result follows directly from Theorem 3.1.

4. Compatibility and invertibility factors and restricted eigenvalues

In Section 3, the oracle inequalities in Theorems 3.1 and 3.2 are expressed in terms of the compatibility and weak cone invertibility factors. However, as mentioned in the introduction, these quantities are still random variables. This section provides sufficient conditions under which they can be treated as constants. Since these factors appear in the denominator of error bounds, it suffices to bound them from below. We also derive a lower bound for the restricted eigenvalue to facilitate further analysis of the Cox model in high-dimension. We will prove that these quantities are bounded from below by the population version of their certain truncated versions.

Compared with linear regression, our problem poses two additional difficulties in the Cox model: (a) time dependence of covariates, and (b) stochastic integration of the Hessian over random risk sets. Fortunately, the compatibility and weak cone invertibility factors in Theorems 3.1 and 3.2 involve only the Hessian of the log-partial likelihood at the true β^o, so that a martingale argument can be used.

To simplify the statement of our results, we use ϕ(ξ, 𝒪, Σ̅) to denote any of the following quantities:

$$\varphi (\xi ,𝒪;\mathbf{\Sigma ̅})={\kappa }^2(\xi ,𝒪;\mathbf{\Sigma ̅}),F_q(\xi ,𝒪;\mathbf{\Sigma ̅})\text{or}\mathrm{R}{\mathrm{E}}^2(\xi ,𝒪;\mathbf{\Sigma ̅}),$$ (4.1)

where κ(ξ, 𝒪; Σ̅), F_q(ξ, 𝒪; Σ̅), and RE(ξ, 𝒪; Σ̅) are as in (3.3), (3.4) and (3.5) respectively. If we make a claim about ϕ(ξ, 𝒪; Σ̅), we mean that the claim holds for any quantity it represents. Let ϕ_min denote the smallest eigenvalue. The following lemma provides some key properties ϕ(ξ, 𝒪; Σ̅) used in the derivation of its lower bounds.

Lemma 4.1. Let κ(ξ, 𝒪;Σ̅), F_q(ξ, 𝒪; Σ̅), RE(ξ, 𝒪; Σ̅) and ϕ(ξ, 𝒪; Σ̅) be as in (4.1). Let Σ̅_jk be the elements of Σ̅ and Σ̅ be another nonnegative-definite matrix with elements Σ_jk.

(i) For 1 ≤ q ≤ 2,

$$\text{min}\{{\kappa }^2(\xi ,𝒪;\mathbf{\Sigma ̅}),{(1+\xi )}^{2/q-1}{F}_q(\xi ,𝒪;\mathbf{\Sigma ̅})\}\ge \mathrm{R}{\mathrm{E}}^2(\xi ,𝒪;\mathbf{\Sigma ̅})\ge {\varphi }_{\text{min}}(\mathbf{\Sigma ̅}).$$

(ii) ϕ(ξ, 𝒪; Σ̅) ≥ ϕ(ξ, 𝒪; Σ) − d^o(ξ + 1)² max_{1≤j≤k≤p} |Σ̅_jk − Σ_jk|.

(iii) If Σ̅ ≥ Σ, then ϕ(ξ, 𝒪; Σ̅) ≥ ϕ(ξ, 𝒪; Σ).

Proof of Lemma 4.1. By the Hölder inequality, ${|\mathit{b}|}_q\le {|\mathit{b}|}_1^{2/q-1}{|\mathit{b}|}_2^{2-2/q}$. Since |b|₁ ≤ (1 + ξ)|b_𝒪|₁ in the cone and ${|{\mathit{b}}_{𝒪}|}_1\le d_o^{1/2}{|\mathit{b}|}_2$ we have

$${|{\mathit{b}}_{𝒪}|}_1{|\mathit{b}|}_q/d_o^{1/q}\le {(1+\xi )}^{2/q-1}{|{\mathit{b}}_{𝒪}|}_1^{2/q}{|\mathit{b}|}_2^{2-2/q}/d_o^{1/q}\le {(1+\xi )}^{2/q-1}{|\mathit{b}|}_2^2.$$

This and ${|{\mathit{b}}_{𝒪}|}_1\le d_o^{1/2}{|\mathit{b}|}_2$ yields part (i) by the definition of the quantities involved. Part (ii) follows from $|\mathit{b}\prime \mathbf{\Sigma ̅}\mathit{b}-\mathit{b}\prime \mathbf{\Sigma }\mathit{b}|\le {|\mathit{b}|}_1^2{\text{max}}_{j,k}|{\mathrm{\Sigma ̅}}_{jk}-{\mathrm{\Sigma }}_{jk}|$ and ${|\mathit{b}|}_1\le (\xi +1){|{\mathit{b}}_{𝒪}|}_1\le (\xi +1)d_o^{1/q}{|\mathit{b}|}_q$. Part (iii) follows immediately from the definition of the quantities in (4.1).

It follows from Lemma 4.1 (ii) and (iii) that quantities of type ϕ(ξ, 𝒪; Σ̅) in (4.1) can be bounded from below in two ways. The first is to bound the matrix Σ̅ from below and the second is to approximate Σ̅ under the supreme norm for its elements. In the p >> n setting, our problem is essentially the rank deficiency of Σ̅ to begin with, so that its lower bound is still rank deficient. However, a lower bound of the random matrix Σ̅ = ℓ̈(β^o), e.g. a certain truncated version of it, may have a smaller variability to allow an approximation by its population version. This is our basic idea. In fact, our analysis takes advantage of this argument twice to remove different sources of randomness.

According to our plan described in the previous paragraph, we first choose a suitable truncation of Σ̅ = ℓ̈(β^o) as a lower bound of the matrix. This is done by truncating the maximum event time under consideration. It follows from (2.6) that for t* > 0,

$$\mathrm{\ell ̈}({\mathit{\beta }}^o)\ge \mathrm{\ell ̈}({\mathit{\beta }}^o;t^{*}),\text{where}\mathrm{\ell ̈}({\mathit{\beta }}^o;t^{*})=n^{-1}{\int }_0^{t^{*}}{V}_n(s,{\mathit{\beta }}^o)d\mathrm{N̄}(s).$$ (4.2)

This allows us to remove the randomness from the counting process by replacing the average counting measure n⁻¹ N̄ (t) by its compensator R_n(s, β^o)dΛ₀(s), where Λ₀ is the baseline cumulative hazard function. This approximation of ℓ̈(β^o; t*) can be written as

$$\mathbf{\Sigma ̅}(t^{*})={\int }_0^{t^{*}}{V}_n(s,{\mathit{\beta }}^o)R_n(s,{\mathit{\beta }}^o)d{\mathrm{\Lambda }}_0(s).$$ (4.3)

To completely remove the randomness with Σ̅(t*), we apply the method again by truncating the weights $e^{{\mathit{Z}}_i^{\prime }(t){\mathit{\beta }}^o}$ with R_n(s, β^o). For M > 0, define

$$\mathbf{\Sigma ̅}(t^{*};M)={\int }_0^{t^{*}}{\mathit{Ĝ}}_n(s;M)d{\mathrm{\Lambda }}_0(s),$$ (4.4)

where ${\mathit{Ĝ}}_n(t;M)=n^{-1}{\sum }_{i=1}^n{\{{\mathit{Z}}_i-{\mathit{Z̄}}_n(t;M)\}}^{\otimes 2}{Y}_i(t)\text{min}\{M,\text{exp}({\mathit{Z}}_i^{\prime }(t){\mathit{\beta }}^o)\}$ with

$${\mathbf{Z̄}}_n(t;M)=\frac{{\sum }_{i=1}^n{\mathit{Z}}_i(t)Y_i(t)\text{min}\{M,\text{exp}({\mathbf{Z}}_i^{\prime }(t){\mathit{\beta }}^o)\}}{{\sum }_{i=1}^n{Y}_i(t)\text{min}\{M,\text{exp}({\mathbf{Z}}_i^{\prime }(t){\mathit{\beta }}^o)\}}.$$

We will prove that the matrix (4.4) is a lower bound of (4.3). Suppose {Y_i(t), Z_i(t), t ≥ 0} are iid stochastic precesses from {Y(t), Z(t), t ≥ 0}. The population version of (4.4) is then

$$\mathbf{\Sigma }(t^{*};M)=E{\int }_0^{t^{*}}{\mathit{G}}_n(s;M)d{\mathrm{\Lambda }}_0(s),$$ (4.5)

where ${\mathit{G}}_n(t;M)=n^{-1}{\sum }_{i=1}^n{\{{\mathit{Z}}_i-\mathit{\mu }(t;M)\}}^{\otimes 2}{Y}_i(t)\text{min}\{M,\text{exp}({\mathit{Z}}_i^{\prime }(t){\mathit{\beta }}^o)\}$ with

$$\mathit{\mu }(t;M)=\frac{E[\mathit{Z}(t)Y(t)\text{min}\{M,\text{exp}(\mathit{Z}\prime {\mathit{\beta }}^o)\}]}{E[Y(t)\text{min}\{M,\text{exp}(\mathit{Z}\prime {\mathit{\beta }}^o)\}]}.$$

The analysis outlined above leads to the following main result of this section. For ξ ≥ 1 and 𝒪 ⊂ {1, …, p} with |𝒪| = d_o, let ϕ(ξ, 𝒪, Σ̅) represent all quantities of interest given in (4.1), κ(ξ, 𝒪) and F_q(ξ, 𝒪) be the compatibility and weak cone invertibility factors in (3.8) with the Hessian ℓ̈(β^o) in (2.6) at the true β, and RE(ξ, 𝒪; Σ̅) be as in (3.5). Let $L_n(t)=\sqrt{(2/n)\text{log}t}$.

Theorem 4.1. Suppose {Y_i(t), Z_i(t), t ≥ 0} are iid processes from {Y(t), Z(t), t ≥ 0} with sup_t P{|Z_i(t) − Z(t)|_∞ ≤ K} = P{max_i N_i(∞) ≤ 1} = 1. Let {t*, M} be positive constants and r_* = EY (t*) min{M, exp(Z′(t*)β^o)}. Then,

$$\varphi (\xi ,𝒪;\mathrm{\ell ̈}({\mathit{\beta }}^o))\ge \varphi (\xi ,𝒪;\mathbf{\Sigma }(t^{*};M))-d_o{(\xi +1)}^2{K}^2\{C_1{L}_n(p(p+1)/\epsilon )+C_2{t}_{n,p,\epsilon }^2\}$$ (4.6)

with at least probability 1 − 3ε, where C₁ = 1 + Λ₀(t*), C₂ = (2/r_*)Λ₀(t*), and t_n,p,ε is the solution of $p(p+1)\text{exp}\{-n{t}_{n,p,\epsilon }^2/(2+2t_{n,p,\epsilon }/3)\}=\epsilon /2.221$. Consequently, for 1 ≤ q ≤ 2,

$$\text{min}\{{\kappa }^2(\xi ,𝒪),{(1+\xi )}^{2/q-1}{F}_q(\xi ,𝒪)\}\ge \mathrm{R}{\mathrm{E}}^2(\xi ,𝒪;\mathrm{\ell ̈}({\mathit{\beta }}^o))\ge {\rho }_{*}-d_o{(\xi +1)}^2{K}^2\{C_1{L}_n(p(p+1)/\epsilon )+C_2{t}_{n,p,\epsilon }^2\}$$ (4.7)

with at least probability 1 − 3ε, where ρ_* = ϕ_min(Σ(t*; M)) with the matrix in (4.5).

Theorem 4.1 implies that the compatibility and cone invertibility factors and the restricted eigenvalue can be all treated as constants in high-dimensional Cox model with time-dependent covariates. We note that $C_2{t}_{n,p,\epsilon }^2$ is of smaller order than L_n(p(p + 1)/ε) so that the lower bounds in (4.6) and (4.7) depend on the choice of t* and M marginally through C₁ and ρ_*. If d^o $\sqrt{(\text{log}p)/n}$ is sufficiently small as assumed in Theorem 3.2, the right-hand side of (4.7) can be treated as ρ_*/2. It is reasonable to treat ρ_* as a constant since it is the smallest eigenvalue of a population integrated covariance matrix in (4.5).

In the proof of Theorem 4.1, the martingale exponential inequality in Lemma 3.3 is used to bound the difference between (4.2) and (4.3). The following Bernstein inequality for V-statistics is used to bound the difference between (4.4) and (4.5). This inequality can be viewed as an extension of the Hoeffding (1963) inequality for sums of bounded independent variables and non-degenerate U-statistics.

Lemma 4.2. Let X_i be a sequence of independent stochastic processes and f_i,j be functions of X_i and X_j with |f_i,j| ≤ 1. Suppose f_i,j are degenerate in the sense of E[f_i,j|X_i] = E[f_i,j|X_j] = 0 for all i ≠ j. Let $V_n={\sum }_{i=1}^n{\sum }_{j=1}^n{f}_{i,j}$. Then,

$$P\{\pm V_n>{(nt)}^2\}\le \frac{2{\epsilon }_n(t)(1+{\epsilon }_n(t))}{{(1+{\epsilon }_n^2(t))}^2}\le 2.221\text{exp}(-\frac{n{t}^2/2}{1+t/3}),$$

where ε_n(t) = e^{−(nt2/2)/(1+t/3)}.

Our discussion focuses on the quantities in (4.1) for the Hessian matrix Σ̅ = ℓ̈(β^o) evaluated at the true vector of coefficients. Still, through Lemma 3.2, Theorem 4.1 also provide lower bounds for these quantities at any β not far from the true β^o in terms of the ℓ₁ distance. We formally state this result in the following corollary.

Corollary 4.1. Let ϕ(ξ, 𝒪; Σ̅) represent any quantities in (4.1). Then,

$$e^{-2{\eta }_{\mathit{b}}}\varphi (\xi ,𝒪;\mathrm{\ell ̈}({\mathit{\beta }}^o))\le \varphi (\xi ,𝒪;\mathrm{\ell ̈}({\mathit{\beta }}^o+\mathit{b}))\le e^{2{\eta }_{\mathit{b}}}\varphi (\xi ,𝒪;\mathrm{\ell ̈}({\mathit{\beta }}^o)),$$

where η_b = sup_s max_i,j |b′Z_i(s) − b′Z_j(s)|. Consequently, when |Z_i(s) − Z_j(s)|_∞ ≥ K,

$$\text{inf}\{\varphi (\xi ,𝒪;\mathrm{\ell ̈}(\mathit{\beta })):{|\mathit{\beta }-{\mathit{\beta }}^o|}_1\le \eta /(2K)\}\ge e^{-\eta }\varphi (\xi ,𝒪;\mathrm{\ell ̈}({\mathit{\beta }}^o))\ge e^{-\eta }[{\rho }_{*}-d_o{(\xi +1)}^2{K}^2\{C_1{L}_n(p(p+1)/\epsilon )+C_2{t}_{n,p,\epsilon }^2\}]$$

under the conditions of Theorem 4.1.

It is worthwhile to point out that unlike typical “small ball” analysis based on Taylor expansion, Corollary 4.1 provides non-asymptotic control of the quantities in an ℓ₁ ball of constant size. Since b′Σ̅b appears in the numerator of the quantities represented by ϕ(ξ, 𝒪; Σ̅), Corollary 4.1 follows immediately from Theorem 4.1 and (3.13). It implies that the Hessian has sufficient invertibility properties in the analysis of the Lasso when the estimator is not far from the true β^o in ℓ₁ distance. On the other hand, if the Hessian has sufficient invertibility properties in a ball of fixed size, non-asymptotic error bounds for the Lasso estimator can be established. This “chicken and egg” problem is directly solved in the proof of Theorem 3.1.

5. Concluding remarks

This paper deals with the Cox proportional hazards regression model when the number of time-dependent covariates p is potentially much larger than the sample size n. The ℓ₁ penalty is used to regularize the log-partial likelihood function. Error bounds parallel to those of the Lasso in linear regression are established. In establishing these bounds, we extend the notion of the restricted eigenvalue and compatibility and cone invertibility factors to the Cox model. We show that these quantities indeed provide useful error bounds.

An important issue is the choice of the penalty level λ. Theorem 3.2 requires a λ slightly larger than $K\sqrt{(2/n)\text{log}p}$, where K is a uniform upper bound for the range of individual real covariates. This indicates that the Lasso is tuning insensitive since the theoretical choice does not depend on the unknowns. In practice, cross validation can be used to fine tune the penalty level λ. Theoretical investigation of the performance of the Lasso with cross-validated λ, an interesting and challenging problem in and of itself even in the simpler linear regression model, is beyond the scope of this paper.

General concave penalized estimators in the Cox regression model have been considered in Bradic et al (2011) where oracle inequalities and properties of certain local solutions are considered. Zhang and Zhang (2012) has provided a unified treatment of global and local solutions for concave penalized least squares estimators in linear regression. Since this unified treatment relies on an oracle inequality for the global solution based on the cone invertibility factor, the results in this paper point to a possible extension of such a unified treatment of global and local solutions of general concave regularized methods in the Cox regression model.

Appendix

Here we prove Lemmas 3.1, 3.2, 3.3 and 4.2 and Theorems 3.1 and 4.1.

Proof of Lemma 3.1. Since ℓ(β) is a convex function, D^s(β̂, β) = θ̃′ {ℓ̇(β^o + θ̃) − ℓ̇(β^o)} ≥ 0, so that the first inequality holds. Since θ̃_j = β̂_j for j ∈ 𝒪^c, (2.4) gives

$$\mathit{\theta ̃}\left\{\mathrm{\ell ̇}\right({\mathit{\beta }}^o+\mathit{\theta ̃})-\mathrm{\ell ̇}({\mathit{\beta }}^o)\}=\sum _{j\in {𝒪}^c}{\mathrm{\theta ̃}}_j{(\mathrm{\ell ̇}({\mathit{\beta }}^o+\mathit{\theta ̃}))}_j+\sum _{j\in 𝒪}{\mathrm{\theta ̃}}_j{(\mathrm{\ell ̇}({\mathit{\beta }}^o+\mathit{\theta ̃}))}_j+\mathit{\theta ̃}\prime (-\mathrm{\ell ̇}({\mathit{\beta }}^o))\le \sum _{j\in {𝒪}^c}{\mathrm{\beta ̂}}_j(-\lambda \mathrm{s}\mathrm{g}\mathrm{n}({\mathrm{\beta ̂}}_j))+\sum _{j\in 𝒪}|{\mathrm{\theta ̃}}_j|\lambda +{|\mathit{\theta ̃}|}_1{z}^{*}=\sum _{j\in {𝒪}^c}-\lambda |{\mathrm{\theta ̃}}_j|+{|{\mathit{\theta ̃}}_{𝒪}|}_1\lambda +z^{*}{|{\mathit{\theta ̃}}_{𝒪}|}_1+z^{*}{|{\mathit{\theta ̃}}_{{𝒪}^c}|}_1=(z^{*}-\lambda ){|{\mathit{\theta ̃}}_{{𝒪}^c}|}_1+(\lambda +z^{*}){|{\mathit{\theta ̃}}_{𝒪}|}_1.$$

Thus the second inequality in (3.1) holds. Note that the inequality in the third line above requires (ℓ̇(β^o + θ̃))_j = −λsgn(β̂_j) only in the set 𝒪^c ∩ {j : β̂_j ≠ 0}, since ${\mathrm{\theta ̃}}_j={\mathrm{\beta ̂}}_j-{\mathrm{\beta ̂}}_j^o=0$ when j ∈ 𝒪^c and β̂_j = 0.

Proof of Lemma 3.2. We use similar notation as in Hjort and Pollard (1993). Let a_i = a_i(s) = b′ {Z_i(s) − Z̄_n (s, β)}, w_i = w_i(s) = Y_i(s) exp[β′ Z_i(s)], and c = c(s) = (max_i a_i(s) + min_i a_i(s))/2. Clearly, max_i |a_i − c| ≤ (1/2)η_b. By the definition of Z̄_n (t, β),

$$\mathit{b}\prime \{{\mathit{Z̄}}_n(s,\mathit{\beta }+\mathit{b})-{\mathit{Z̄}}_n(s,\mathit{\beta })\}=\sum _i\mathit{b}\prime {\mathit{Z}}_i(s)w_i{e}^{\mathit{b}\prime {\mathit{Z}}_i(s)}/\sum _i{w}_i{e}^{\mathit{b}\prime {\mathit{Z}}_i(s)}-\sum _i\mathit{b}\prime {\mathit{Z}}_i(s)w_i/\sum _i{w}_i=\sum _i{a}_i{w}_i{e}^{a_i}/\sum _i{w}_i{e}^{a_i}-\sum _i{a}_i{w}_i/\sum _i{w}_i=\sum _{i,j}{w}_i{w}_j{a}_i(e^{a_i}-e^{a_j})/\sum _{i,j}{w}_i{w}_j{e}^{a_i}=\sum _{i,j}{w}_i{w}_j(a_i-a_j)(e^{a_i-c}-e^{a_j-c})/\sum _{i,j}2w_i{w}_j{e}^{a_i-c}\ge \text{exp}(-2\underset{i}{\text{max}}|a_i-c|)\sum _{i,j}{w}_i{w}_j{(a_i-a_j)}^2/\sum _{i,j}2w_i{w}_j\ge \text{exp}(-{\eta }_{\mathit{b}})\sum _i{w}_i{a}_i^2/\sum _i{w}_i,$$

where the first inequality comes from (e^y − e^x)/(y − x) ≥ e^{−(|y|∨|x|)} and, since ∑_i w_ia_i = 0, the second one from ${\sum }_{i,j}{w}_i{w}_j{(a_i-a_j)}^2=2{\sum }_i{w}_i{\sum }_i{w}_i{a}_i^2$. Thus, since $a_i^2=\mathit{b}\prime {\{{\mathit{Z}}_i(s)-{\mathit{Z̄}}_n(s,\mathit{\beta })\}}^{\otimes 2}\mathit{b}$, (2.6) and (2.5) give

$$e^{-{\eta }_{\mathit{b}}}\mathit{b}\prime \mathrm{\ell ̈}(\mathit{\beta })\mathit{b}=\frac{e^{-{\eta }_{\mathit{b}}}}{n}{\int }_0^{\infty }\sum _{i=1}^n{w}_i{a}_i^2{\left(\sum _{i=1}^n{w}_i\right)}^{-1}d\mathrm{N̄}(s)\le \mathit{b}\prime \{\mathrm{\ell ̇}(\mathit{\beta }+\mathit{b})-\mathrm{\ell ̇}(\mathit{\beta })\}.$$

This implies the lower bound in (3.12). Similarly, the lower bound in (3.13) follows from

$$\mathrm{\ell ̈}(\mathit{\beta }+\mathit{b})=\frac{1}{n}{\int }_0^{\infty }\frac{{\sum }_{i,j}{w}_i{w}_j\{{\mathit{Z}}_i(s){\mathit{Z}}_i^{\prime }(s)-{\mathit{Z}}_i(s){\mathit{Z}}_j^{\prime }(s)\}{e}^{a_i+a_j}}{{\sum }_{i,j}{w}_i{w}_j{e}^{a_i+a_j}}d\mathrm{N̄}(s)=\frac{1}{n}{\int }_0^{\infty }\frac{{\sum }_{i,j}{w}_i{w}_j{({\mathit{Z}}_i(s)-{\mathit{Z}}_j(s))}^{\otimes 2}{e}^{(a_i-c)+(a_j-c)}}{{\sum }_{i,j}2w_i{w}_j{e}^{(a_i-c)+(a_j-c)}}d\mathrm{N̄}(s)$$

and

$$\mathrm{\ell ̈}(\mathit{\beta })=\frac{1}{n}{\int }_0^{\infty }\frac{{\sum }_{i,j}{w}_i{w}_j{({\mathit{Z}}_i(s)-{\mathit{Z}}_j(s))}^{\otimes 2}}{{\sum }_{i,j}2w_i{w}_j}d\mathrm{N̄}(s).$$

The proof of the upper bounds in (3.12) and (3.13), nearly identical to the proof of the lower bounds, is omitted.

Proof of Lemma 3.3. Applying the union bound and changing the scale of the covariates if necessary, we assume without loss of generality that p = K = 1. In this case

$$\mathrm{\ell ̇}({\mathit{\beta }}^o)=\frac{1}{n}\sum _{i=1}^n{\int }_0^{\infty }{a}_i(s)d{N}_i(s)=\frac{1}{n}\sum _{i=1}^n{\int }_0^{\infty }{a}_i(s)d{M}_i(s),$$

where a_i(t) = Z_i1 (t) − Z̄_n,1 (t), i = 1, …, n, are predictable and satisfy |a_i(t)| ≤ 1. Thus, (3.15) follows from (3.14).

Let t_j be the time of the jth jump of the process ${\sum }_{i=1}^n{\int }_0^{\infty }{Y}_i(t)d{N}_i(t)$, j = 1, …, m, and t₀ = 0. Then, t_j are stopping times. For j = 0, …, m, define

$$X_j=\sum _{i=1}^n{\int }_0^{t_j}{a}_i(s)d{N}_i(s)=\sum _{i=1}^n{\int }_0^{t_j}{a}_i(s)d{M}_i(s).$$ (7.1)

Since M_i(s) are martingales and a_i(s) are predictable, {X_j, j = 0, 1, …} is a martingale with the difference |X_j − X_j−1| ≤ max_{s, i} |a_i(s)| ≤ 1. Let m be the greatest integer lower bound of $C_0^{2}n$. By the martingale version of the Hoeffding (1963) inequality (Azuma, 1967),

$$\mathrm{P}(|X_m|>n{C}_{0}x)\le 2\text{exp}(-n^2{C}_0^2{x}^2/(2m))\le e^{-n{x}^2/2}.$$

By (7.1), X_m = nℓ̇(β^o) if and only if ${\sum }_{i=1}^n{\int }_0^{\infty }{Y}_i(t)d{N}_i(t)\le m$. Thus, the left-hand side of (3.15) is no greater than P(|X_m| > nC₀x) ≤ e^−nx2/2.

Proof of Lemma 4.2. For integers j, m, i₁, …, i_m, let #(j; i₁, …, i_m) be the number of appearances of j in the sequence {i₁, …, i_m}. Since f_i,j are degenerate,

$$E{(\pm V_n)}^m=\sum _{1\le i_1,\dots ,i_{2m}\le n}{(\pm 1)}^m{f}_{i_1,i_2}\cdots f_{i_{2m-1},i_{2m}}\le \sum _{1\le i_1,\dots ,i_{2m}\le n}\prod _{j=1}^{n}I\{\#(j;i_1,\dots ,i_{2m})=1\}.$$

This is due to the fact that all terms with exactly one appearance of an index j have zero expectation and all other terms are bounded by 1. Let E₀ be the expectation under which i₁, …, i_2m are iid uniform variables in {1, …, n} and k_j = #(j; i₁, …, i_2m). Since (k₁, …, k_n) is multinomial(2m, 1/n, …, 1/n), the above inequality can be written as

$$E{(\pm V_n)}^m\le n^{2m}{E}_0\prod _{j=1}^{n}I\{k_j=1\}=(2m)!\sum _{k_1+\cdots +k_n=2m}\prod _{j=1}^n\frac{I\{k_j=1\}}{k_j!}.$$

Let $f_0(x)={\sum }_{m=0}^{\infty }{x}^m/(2m)!=\text{cos}\mathrm{h}({|x|}^{1/2})I\{x\ge 0\}+\text{cos}({|x|}^{1/2})I\{x<0\}$ and λ = t/(1 + t/3). It follows from the above moment inequality that

$$E{f}_0(\pm {\lambda }^2{V}_n)=\sum _{m=0}^{\infty }{\lambda }^{2m}E{(\pm V_n)}^m/(2m)!\le \sum _{m=0}^{\infty }{\lambda }^{2m}\sum _{k_1+\cdots +k_n=2m}\prod _{j=1}^n\frac{I\{k_j=1\}}{k_j!}\le \sum _{m=0}^{\infty }{\lambda }^m\sum _{k_1+\cdots +k_n=m}\prod _{j=1}^n\frac{I\{k_j=1\}}{k_j!}={(\sum _{k=0}^{\infty }{\lambda }^{k}I\{k\ne 1\}/k!)}^n.$$

Since ${\sum }_{k=0}^{\infty }{\lambda }^{k}I\{k\ne 1\}/k!\le 1+({\lambda }^2/2)/(1-\lambda /3)=1+\lambda t/2$, we find Ef₀(±λ²V_n) ≤ e^nλt/2. Consequently, the monotonicity of f(x) = cosh(x^1/2) for x > 0 and the lower bound f(x) ≥ −1 allow us to apply the Markov inequality as follows:

$$P\{\pm V_n>{(nt)}^2\}\le P\{1+f_0(\pm {\lambda }^2{V}_n)>1+f_0({(\lambda nt)}^2)\}\le {\{1+f_0({(n\lambda t)}^2)\}}^{-1}E\{1+f_0(\pm {\lambda }^2{V}_n)\}\le {\{1+\text{cos}\mathrm{h}(n\lambda t)\}}^{-1}(1+e^{n\lambda t/2})=2e^{-n\lambda t/2}(1+e^{-n\lambda t/2})/{(1+e^{-n\lambda t})}^2.$$

The conclusion follows from 2 max_0≤x≤1 (1 + x)/(1 + x²)² ≤ 2.221.

Proof of Theorem 3.1. Let θ̃ = β̂ − β^o ≠ 0 and b = θ̃/|θ̃|₁. It follows from the convexity of ℓ(β^o + xb), as a function of x, and Lemma 3.1 that, in the event |ℓ̇(β^o)|_∞ ≤ (ξ − 1)/(ξ + 1)λ,

$$\mathit{b}\prime \{\mathrm{\ell ̇}({\mathit{\beta }}^o+x\mathit{b})-\mathrm{\ell ̇}({\mathit{\beta }}^o)\}+\frac{2\lambda }{\xi +1}{|{\mathit{b}}_{{𝒪}^c}|}_1\le \frac{2\xi \lambda }{\xi +1}{|{\mathit{b}}_{𝒪}|}_1,$$ (7.2)

for x ∈ [0, |θ̃|₁] and b ∈ 𝒞 (ξ, 𝒪). Consider all nonnegative x satisfying (7.2). We need to establish a lower bound for

$$\mathit{b}\prime \{\mathrm{\ell ̇}({\mathit{\beta }}^o+x\mathit{b})-\mathrm{\ell ̇}({\mathit{\beta }}^o)\}=\frac{1}{n}{\int }_0^{\infty }\mathit{b}\prime \{{\mathrm{Z̄}}_n(s,{\mathit{\beta }}^o+x\mathit{b})-{\mathrm{Z̄}}_n(s,{\mathit{\beta }}^o)\}d\mathrm{N̄}(s)$$

Since η_xb = max_0≤s≤1 max_{i, j} |xb′Z_i(s) − xb′Z_j(s)| ≤ Kx|b|₁ = Kx, Lemma 3.2 yields

$$x\mathit{b}\prime \{\mathrm{\ell ̇}({\mathit{\beta }}^o+x\mathit{b})-\mathrm{\ell ̇}({\mathit{\beta }}^o)\}\ge x^2\text{exp}(-{\eta }_{x\mathit{b}})\mathit{b}\prime \mathrm{\ell ̈}({\mathit{\beta }}^o)\mathit{b}\ge x^2\text{exp}(-Kx)\mathit{b}\prime \mathrm{\ell ̈}({\mathit{\beta }}^o)\mathit{b}.$$ (7.3)

This, combined with (7.2) and the definition of κ(ξ, 𝒪), gives

$$x{e}^{-Kx}{\kappa }^2(\xi ,𝒪){|{\mathit{b}}_{𝒪}|}_1^2/d_o\le x{e}^{-Kx}\mathit{b}\prime \mathrm{\ell ̈}({\mathit{\beta }}^o)\mathit{b}\le \frac{2\xi \lambda }{\xi +1}{|{\mathit{b}}_{𝒪}|}_1-\frac{2\lambda }{\xi +1}{|{\mathit{b}}_{{𝒪}^c}|}_1=2\lambda |{\mathit{b}}_{𝒪}{|}_1-\frac{2\lambda }{\xi +1}\le \lambda (\xi +1){|{\mathit{b}}_{𝒪}|}_1^2/2.$$

In other words, any x satisfying (7.2) must satisfy

$$Kx\text{exp}(-Kx)\le \frac{K(\xi +1)\lambda d_o}{2{\kappa }^2(\xi ,𝒪)}=\tau .$$ (7.4)

Since b′{ℓ̇(β^o + xb) − ℓ̇(β^o)} is an increasing function of x due to the convexity of ℓ, the set of all nonnegative x satisfying (7.2) is a closed interval [0, x̃] for some x̃ > 0. Thus, (7.4) implies Kx̃ ≤ η, the smaller solution of ηe^−η = τ. This yields

$${|\mathit{\theta ̃}|}_1\le \mathrm{x̃}\le \frac{\eta }{K}=\frac{e^{\eta }\tau }{K}=\frac{e^{\eta }(\xi +1)\lambda d_o}{2{\kappa }^2(\xi ,𝒪)}$$

in (3.10). The first part of (3.10) follows from (3.3), (3.8), (3.12) and (3.1), due to

$$e^{-\eta }{\kappa }^2(\xi ,𝒪)|{\mathit{\theta ̃}}_{𝒪}{|}_1^2/d^o\le e^{-\eta }\mathit{\theta ̃}\prime \mathrm{\ell ̈}({\mathit{\beta }}^o)\mathit{\theta ̃}\le D^s(\mathit{\beta ̂},\mathit{\beta })\le \frac{2\xi \lambda {|{\mathit{\theta ̃}}_{𝒪}|}_1}{\xi +1}.$$

Finally, it follows from the definition of F_q(ξ, 𝒪), (7.3) and (7.2) that, for x = |θ̃|₁,

$$x{e}^{-\eta }\le \frac{x{e}^{-Kx}\mathit{b}\prime \mathrm{\ell ̈}({\mathit{\beta }}^o)\mathit{b}}{F_q(\xi ,𝒪)({|{\mathit{b}}_{𝒪}|}_1/d_o^{1/q}){|\mathit{b}|}_q}\le \frac{\mathit{b}\prime \{\mathrm{\ell ̇}({\mathit{\beta }}^o+x\mathit{b})-\mathrm{\ell ̇}({\mathit{\beta }}^o)\}}{F_q(\xi ,𝒪)({|{\mathit{b}}_{𝒪}|}_1/d_o^{1/q}){|\mathit{b}|}_q}\le \frac{2\xi \lambda d_o^{1/q}}{(\xi +1)F_q(\xi ,𝒪){|\mathit{b}|}_q}.$$

This gives the second inequality in (3.11) due to |β̂ − β^o|_q = |θ̃|₁|b|_q.

Proof of Theorem 4.1. Let

$$a(s)={(V_n(s,{\mathit{\beta }}^o))}_{jk}/K^2=\sum _{i=1}^n{w}_{ni}(t,{\mathit{\beta }}^o)\{Z_{i,j}(s)-{\mathrm{Z̄}}_{n,j}(s)\}\{Z_{i,k}(s)-{\mathrm{Z̄}}_{n,k}(s)\}/K^2.$$

It follows from Lemma 3.3 (i) with a_i(s) = a(s) and C₀ = 1 that

$$P\left\{\right|{({\int }_0^{t^{*}}{V}_n(s,{\mathit{\beta }}^o)d\mathrm{N̄}(s)-{\int }_0^{t^{*}}{V}_n(s,{\mathit{\beta }}^o)R_n(s,{\mathit{\beta }}^o)d{\mathrm{\Lambda }}_0(s))}_{jk}|>K^{2}x\}\le 2e^{-n{x}^2/2}.$$

Thus, P{max_j,k |(ℓ̈(β^o; t*) − Σ̅(t*)|_j,k ≥ K²L_n(p(p + 1)/ε)} ≤ ε by the union bound and the respective definitions of ℓ̈ (β^o; t*) and Σ̅(t*) in (4.2) and (4.3). Consequently, by (4.2) and Lemma 4.1 (iii) and (ii)

$$P\{\varphi (\xi ,𝒪;\mathrm{\ell ̈}({\mathit{\beta }}^o))\ge \varphi (\xi ,𝒪;\mathbf{\Sigma ̅}(t^{*}))-d_o{(\xi +1)}^2{K}^2{L}_n(p(p+1)/\epsilon )\}\ge 1-\epsilon .$$ (7.5)

Let us take the sample mean of i-indexed quantities with weights $Y_i(t)\text{min}\{M,\text{exp}({\mathit{Z}}_i^{\prime }(t){\mathit{\beta }}^o)\}$, so that Z̄_n(t; M) is the sample mean of Z_i(t). Since V_n(t, β^o) R_n (t, β^o) = Ĝ_n (t; ∞),

$$\mathit{u}\prime {\mathit{Ĝ}}_n(t;\infty )\mathit{u}\ge \frac{1}{n}\sum _{i=1}^n{[\mathit{u}\prime \{{\mathbf{Z}}_i-{\mathbf{Z̄}}_n(t;\infty )\}]}^2{Y}_i(t)\text{min}\{M,\text{exp}(Z_i^{\prime }(t){\mathit{\beta }}^o)\}\ge \mathit{u}\prime {\mathit{Ĝ}}_n(t;M)\mathit{u}.$$

Thus, by the definition of Σ̅(t*; M) in (4.4) and Lemma 4.1 (iii),

$$\varphi (\xi ,𝒪;\mathbf{\Sigma ̅}(t^{*}))\ge \varphi (\xi ,𝒪;\mathbf{\Sigma ̅}(t^{*};M)).$$ (7.6)

In addition, the relationship between the sample second moment and variance gives

$${\mathit{G}}_n(t;M)={\mathit{Ĝ}}_n(t;M)+{\{{\mathbf{Z̄}}_n(t;M)-\mathit{\mu }(t;M)\}}^{\otimes 2}$$

by the definition of G_n (t; M) and Ĝ_n (t; M), so that (4.4) can be written as

$$\mathbf{\Sigma ̅}(t^{*};M)={\int }_0^{t^{*}}{\mathit{G}}_n(s;M)d{\mathrm{\Lambda }}_0(s)-{\int }_0^{t^{*}}{\{{\mathit{Z̄}}_n(t;M)-\mathit{\mu }(t;M)\}}^{\otimes 2}d{\mathrm{\Lambda }}_0(s).$$ (7.7)

We first bound the second term on the right-hand side of (7.7). Define

$$R_n(t;M)=\frac{1}{n}\sum _{i=1}^n{Y}_i(t)\text{min}\{M,\text{exp}({\mathit{Z}}_i^{\prime }(t){\mathit{\beta }}^o)\},\mathbf{\Delta }(t;M)=R_n(t;M)\{{\mathbf{Z̄}}_n(t;M)-\mathit{\mu }(t;M)\}=\frac{1}{n}\sum _{i=1}^n{Y}_i(t)\text{min}\{M,\text{exp}({\mathit{Z}}_i^{\prime }(t){\mathit{\beta }}^o)\}\{{\mathit{Z}}_i(t)-\mathit{\mu }(t;M)\}.$$

Since Y_i(t) is non-increasing in t,

$$0\le {\int }_0^{t^{*}}{\{{\mathbf{Z̄}}_n(t;M)-\mathit{\mu }(t;M)\}}^{\otimes 2}d{\mathrm{\Lambda }}_0(s)\le \frac{{\int }_0^{t^{*}}{\mathbf{\Delta }}^{\otimes 2}(t;M)d{\mathrm{\Lambda }}_0(s)}{R_n^2(t^{*},M).}$$ (7.8)

Since R_n (t*, M) is the average of iid variables uniformly bounded by M and ER_n(t*, M) = r_*, the Hoeffding (1963) inequality gives

$$P\{R_n(t^{*},M)<r_{*}/2\}\le e^{-n{r}_{*}^2/(8M^2).}$$

Since Δ(t; M) is an average of iid mean zero vectors, ${(n^2{\int }_0^{t^{*}}{\mathbf{\Delta }}^{\otimes 2}(t;M)d{\mathrm{\Lambda }}_0(s))}_{jk}$ is a degenerate V -statistic for each (j, k). Moreover, since the summands of these V -statistics are all bounded by K²Λ₀(t*), Lemma 4.2 yields

$$\underset{1\le j,k\le p}{\text{max}}P\{\pm {({\int }_0^{t^{*}}{\mathbf{\Delta }}^{\otimes 2}(t;M)d{\mathrm{\Lambda }}_0(s))}_{jk}>K^2{\mathrm{\Lambda }}_0(t^{*})t^2\}\le 2.221\text{exp}\left(\frac{-n{t}^2/2}{1+t/3}\right).$$

Thus, by (7.7), (7.8), the above two probability bounds and Lemma 4.1 (ii),

$$\varphi (\xi ,𝒪;\mathbf{\Sigma ̅}(t^{*};M))\ge \varphi (\xi ,𝒪;{\int }_0^{t^{*}}{\mathit{G}}_n(s;M)d{\mathrm{\Lambda }}_0(s))-d^o{(\xi +1)}^2{K}^2{\mathrm{\Lambda }}_0(t^{*})t_{n,p,\epsilon }^2/(r_{*}/2)$$ (7.9)

with at least probability $1-e^{-n{r}_{*}^2/(8M^2)}-\epsilon$.

Finally, by (4.5), ${\int }_0^{t^{*}}{\mathit{G}}_n(s;M)d{\mathrm{\Lambda }}_0(s)$ is an average of iid matrices with mean Σ(t*; M) and the summands of ${({\int }_0^{t^{*}}{\mathit{G}}_n(s;M)d{\mathrm{\Lambda }}_0(s))}_{jk}$ are uniformly bounded by K²Λ₀(t*), so that the Hoeffding (1963) inequality gives

$$P\left\{\underset{j,k}{\text{max}}\right|{({\int }_0^{t^{*}}{\mathit{G}}_n(s;M)d{\mathrm{\Lambda }}_0(s)-\mathbf{\Sigma }(t^{*},M))}_{jk}|>K^2{\mathrm{\Lambda }}_0(t^{*})t\}\le p(p+1)e^{-n{t}^2/2}.$$

By (7.5), (7.6), (7.9), the above inequality with t = L_n (p(p + 1)/ε) and Lemma 4.1 (ii),

$$\varphi (\xi ,𝒪;\mathrm{\ell ̈}({\mathit{\beta }}^o))\ge \varphi (\xi ,𝒪;{\int }_0^{t^{*}}{\mathit{G}}_n(s;M)d{\mathrm{\Lambda }}_0(s))-d_o{(\xi +1)}^2{K}^2\{L_n(p(p+1)/\epsilon )+(2/r_{*}){\mathrm{\Lambda }}_0(t^{*})t_{n,p,\epsilon }^2\}\ge \varphi (\xi ,𝒪;\mathbf{\Sigma }(t^{*},M))-d_o{(\xi +1)}^2{K}^2\{(1+{\mathrm{\Lambda }}_0(t^{*}))L_n(p(p+1)/\epsilon )+(2/r_{*}){\mathrm{\Lambda }}_0(t^{*})t_{n,p,\epsilon }^2\}$$

with at least probability $1-e^{-n{r}_{*}^2/(8M^2)}-3\epsilon$. Since

$$\varphi (\xi ,𝒪;\mathbf{\Sigma }(t^{*},M))\ge \mathrm{R}{\mathrm{E}}^2(\xi ,𝒪;\mathbf{\Sigma }(t^{*},M))\ge {\rho }_{*}$$

by Lemma 4.1 (i) and the definition in (3.5), the conclusion follows.