BOOSTED NONPARAMETRIC HAZARDS WITH TIME-DEPENDENT COVARIATES

Abstract

Given functional data from a survival process with time-dependent covariates, we derive a smooth convex representation for its nonparametric log-likelihood functional and obtain its functional gradient. From this we devise a generic gradient boosting procedure for estimating the hazard function nonparametrically. An illustrative implementation of the procedure using regression trees is described to show how to recover the unknown hazard. The generic estimator is consistent if the model is correctly specified; alternatively an oracle inequality can be demonstrated for tree-based models. To avoid overfitting, boosting employs several regularization devices. One of them is step-size restriction, but the rationale for this is somewhat mysterious from the viewpoint of consistency. Our work brings some clarity to this issue by revealing that step-size restriction is a mechanism for preventing the curvature of the risk from derailing convergence.

1. Introduction.

Flexible hazard models involving time-dependent covariates are indispensable tools for studying systems that track covariates over time. In medicine, electronic health records systems make it possible to log patient vitals throughout the day, and these measurements can be used to build real-time warning systems for adverse outcomes such as cancer mortality [2]. In financial technology, lenders track obligors’ behaviours over time to assess and revise default rate estimates. Such models are also used in many other fields of scientific inquiry since they form the building blocks for transitions within a Markovian state model. Indeed, this work was partly motivated by our study of patient transitions in emergency department queues and in organ transplant waitlist queues [20]. For example, allocation for a donor heart in the U.S. is defined in terms of coarse tiers [23], and transplant candidates are assigned to tiers based on their health status at the time of listing. However, a patient’s condition may change rapidly while awaiting a heart, and this time-dependent information may be the most predictive of mortality and not the static covariates collected far in the past.

The main contribution of this paper is to introduce a fully nonparametric boosting procedure for hazard estimation with time-dependent covariates. We describe a generic gradient boosting procedure for boosting arbitrary weak base learners for this setting. Generally speaking, gradient boosting adopts the view of boosting as an iterative gradient descent algorithm for minimizing a loss functional over a target function space. Early work includes Breiman [6, 7, 8] and Mason et al. [21, 22]. A unified treatment was provided by Friedman [13], who coined the term “gradient boosting” which is now generally taken to be the modern interpretation of boosting.

Most of the existing boosting approaches for survival data focus on time-static covariates and involve boosting the Cox model. Examples include the popular R-packages mboost (Bühlmann and Hothorn [10]) and gbm (Ridgeway [26]) which apply gradient boosting to the Cox partial likelihood loss. Related work includes the penalized Cox partial likelihood approach of Binder and Schumacher [4]. Other important approaches, but not based on the Cox model, include L₂Boosting [11] with inverse probability of censoring weighting (IPCW) [16, 18], boosted transformation models of parametric families [15], and boosted accelerated failure time models [27].

While there are many boosting methods for dealing with time-static covariates, the literature is far more sparse for the case of time-dependent covariates. In fact, to our knowledge there is no general nonparametric approach for dealing with this setting. This is because in order to implement a fully nonparametric estimator, one has to contend with the issue of identifying the gradient, which turns out to be a non-trivial problem due to the functional nature of the data. This is unlike most standard applications of gradient boosting where the gradient can easily be identified and calculated.

1.1. Time-dependent covariate framework.

To explain why this is so challenging, we start by formally defining the survival problem with time-dependent covariates. Our description follows the framework of Aalen [1]. Let T denote the potentially unobserved failure time. Conditional on the history up to time t– the probability of failing at T ∈ [t, t+dt) equals

$$\lambda (t,X(t))Y(t)dt.$$ (1)

Here λ(t, x) denotes the unknown hazard function, $X(t)\in \mathfrak{X}\subseteq {ℝ}^p$ is a predictable covariate process, and Y (t) ∈ {0, 1} is a predictable indicator of whether the subject is at risk at time t.¹ To simplify notation, without loss of generality we normalize the units of time so that Y (t) = 0 for t > 1.² In other words, the subject is not at risk after time t = 1, so we can restrict attention to the time interval (0, 1].

If failure is observed at T ∈ (0, 1] then the indicator Δ = Y (T) equals 1, otherwise Δ = 0 and we set T = ∞. Throughout we assume we observe n independent and identically distributed functional data samples ${\left\{\left(X_i(\cdot ),Y_i(\cdot ),T_i\right)\right\}}_{i=1}^n$. The evolution of observation i’s failure status can then be thought of as a sequence of coin flips at time increments t = 0, dt, 2dt, ⋯, with the probability of “heads” at each time point given by (1). Therefore, observation i’s contribution to the likelihood is

$$\left\{1-\lambda \left(0,X_i(0)\right)Y_i(0)dt\right\}\times \left\{1-\lambda \left(dt,X_i(dt)\right)Y_i(dt)dt\right\}\times \cdots \times \lambda {\left(T_i,X_i\left(T_i\right)\right)}^{{\Delta }_i}\underset{dt\downarrow 0}{\to }{e}^{-{\int }_0^1{Y}_i(t)\lambda \left(t,X_i(t)\right)dt}\lambda {\left(T_i,X_i\left(T_i\right)\right)}^{{\Delta }_i},$$

where the limit can be understood as a product integral. Hence, if the log-hazard function is

$$F(t,x)=\text{log }\lambda (t,x),$$

then the (scaled) negative log-likelihood functional is

$$R_n(F)=\frac{1}{n}\sum _{i=1}^n{\int }_0^1{Y}_i(t)e^{F\left(t,X_i(t)\right)}dt-\frac{1}{n}\sum _{i=1}^n{\Delta }_{i}F\left(T_i,X_i\left(T_i\right)\right),$$ (2)

which we shall refer to as the likelihood risk. The goal is to estimate the hazard function λ(t, x) = e^{F(t, x)} nonparametrically by minimizing R_n(F).

1.2. The likelihood does not have a gradient in generic function spaces.

As mentioned, our approach is to boost F using functional gradient descent. However the chief difficulty is that the canonical representation of the likelihood risk functional does not have a gradient. To see this, observe that the directional derivative of (2) equals

$${\frac{d}{d\theta }{R}_n(F+\theta f)|}_{\theta =0}=\frac{1}{n}\sum _{i=1}^n{\int }_0^1{Y}_i(t)e^{F\left(t,X_i(t)\right)}f\left(t,X_i(t)\right)dt-\frac{1}{n}\sum _{i=1}^n{\Delta }_{i}f\left(T_i,X_i\left(T_i\right)\right),$$ (3)

which is the difference of two different inner products 〈e^F, f〉_† − 〈1, f〉_‡ where

$${\langle g,f\rangle }_{†}=\frac{1}{n}\sum _{i=1}^n{\int }_0^1{Y}_i(t)g\left(t,X_i(t)\right)f\left(t,X_i(t)\right)dt,$$

$${\langle g,f\rangle }_{‡}=\frac{1}{n}\sum _{i=1}^n{\Delta }_{i}g\left(T_i,X_i\left(T_i\right)\right)f\left(T_i,X_i\left(T_i\right)\right).$$

Hence, (3) cannot be expressed as a single inner product of the form 〈gF, f〉 for some function gF (t, x). Were it possible to do so, gF would then be the gradient function.

In simpler non-functional data settings like regression or classification, the loss can be written as $L(Y,\tilde{F}(x))$, where $\tilde{F}$ is the non-functional target statistical model and Y is the outcome, so the gradient is simply $\partial L(Y,\tilde{F}(x))/\partial \tilde{F}(x)$. The negative gradient is then approximated using a base learner $f\in \tilde{\mathcal{F}}$ from a predefined class of functions $\tilde{\mathcal{F}}$ (this being either parametric; for example linear learners, or nonparametric; for example tree learners). Typically, the optimal base learner $\widehat{f}$ is chosen to minimize the L²-approximation error and then scaled by a regularization parameter 0 < ν ≤ 1 to obtain the updated estimate of $\tilde{F}$:

$$\tilde{F}\leftarrow \tilde{F}-\nu \widehat{f},\widehat{f}=\underset{f\in \tilde{\mathcal{F}}}{\text{argmin}}{\Vert \frac{\partial L}{\partial \tilde{F}}-f\Vert }_2.$$

Importantly, in the simpler non-functional setting the gradient does not depend on the space that $\tilde{F}$ belongs to. By contrast, a key insight of this paper is that the gradient of R_n(F) can only be defined after carefully specifying an appropriate sample-dependent domain for R_n(F). The likelihood risk can then be re-expressed as a smooth convex functional, and an analogous representation also exists for the population risk. These representations resolve the difficulty above, allow us to describe and implement a gradient boosting procedure, and are also crucial to establishing guarantees for our estimator.

1.3. Contributions of the paper.

A key discovery that unlocks the boosted hazard estimator is Proposition 1 of Section 2. It provides an integral representation for the likelihood risk from which several results follow, including, importantly, an explicit representation for the gradient. Proposition 1 relies on defining a suitable space of log-hazard functions defined on the time-covariate domain $[0,1]\times \mathfrak{X}$. Identifying this space is the key insight that allows us to rescue the likelihood approach and to derive the gradient needed to implement gradient boosting. Arriving at this framework is not conceptually trivial, and may explain the absence of boosted nonparametric hazard estimators until now.

Algorithm 1 of Section 2 describes our estimator. The algorithm minimizes the likelihood risk (2) over the defined space of log-hazard functions. In the special case of regression tree learners, expressions for the likelihood risk and its gradient are obtained from Proposition 1, which are then used to describe a tree-based implementation of our estimator in Section 4. In Section 5 we apply it to a high-dimensional dataset generated from a naturalistic simulation of patient service times in an emergency department.

Section 3 establishes the consistency of the procedure. We show that the hazard estimator is consistent if the space is correctly specified. In particular, if the space is the span of regression trees, then the hazard estimator satisfies an oracle inequality and recovers λ up to some error tolerance (Propositions 3 and 4).

Another contribution of our work is to clarify the mechanisms used by gradient boosting to avoid overfitting. Gradient boosting typically applies two types of regularization to invoke slow learning: (i) A small step-size is used for the update; and (ii) The number of boosting iterations is capped. The number of iterations used in our algorithm is set using the framework of Zhang and Yu [31], whose work shows how stopping early ensures consistency. On the other hand, the role of step-size restriction is more mysterious. While [31] demonstrates that small step-sizes are needed to prove consistency, unrestricted greedy step-sizes are already small enough for classification problems [28] and also for commonly used regression losses (see the Appendix of [31]). We show in Section 3.3 that shrinkage acts as a counterweight to the curvature of the risk (see Lemma 2). Hence if the curvature is unbounded, as is the case for hazard regression, then the step-sizes may need to be explicitly controlled to ensure convergence. This important result adds to our understanding of statistical convergence in gradient boosting. As noted by Biau and Cadre [3] the literature for this is relatively sparse, which motivated them to propose another regularization mechanism that also prevents overfitting. Our work adds to this by considering the functional data setting.

Concluding remarks can be found in Section 6. Proofs not appearing in the body of the paper can be found in the Appendix.

2. The boosted hazard estimator.

In this section, we describe our boosted hazard estimator. To provide readers with concrete examples for the ideas introduced here, we will show how the quantities defined in this section specialize in the case of regression trees, which is one of a few possible ways to implement boosting.

We begin by defining in Section 2.1 an appropriate sample-dependent domain for the likelihood risk R_n(F). As explained, this key insight allows us to re-express the likelihood risk and its population analogue as smooth convex functionals, thereby enabling us to compute their gradients in closed form in Propositions 1 and 2 of Section 2.2. Following this, the boosting algorithm is formally stated in Section 2.3.

2.1. Specifying a domain for R_n(F).

We will make use of two identifiability conditions (A1) and (A2) to define the domain for R_n(F). Condition (A1) below is the same as Condition 1(iv) of Huang and Stone [19]. Assumption (A1). The true hazard function λ(t, x) is bounded between some interval [Λ_L, Λ_U] ⊂ (0, ∞) on the time-covariate domain $[0,1]\times \mathfrak{X}$.

Recall that we defined X(·) and Y(·) to be predictable processes, and so it can be shown that the integrals and expectations appearing in this paper are all well defined. Denoting the indicator function as I(·), define the following population and empirical sub-probability measures on $[0,1]\times \mathfrak{X}$:

$$\mu (B)=\mathbb{E}\left({\int }_0^{1}Y(t)\cdot I[\{t,X(t)\}\in B]dt\right),$$

$${\mu }_n(B)=\frac{1}{n}\sum _{i=1}^n{\int }_0^1{Y}_i(t)\cdot I\left[\left\{t,X_i(t)\right\}\in B\right]dt,$$

and note that $\mathbb{E}{\mu }_n(B)=\mu (B)$ because the data is i.i.d. by assumption. Intuitively, μ_n measures the denseness of the observed sample time-covariate paths on $[0,1]\times \mathfrak{X}$. For any integrable f,

$$\int fd\mu =\mathbb{E}\left({\int }_0^{1}Y(t)\cdot f(t,X(t))dt\right),$$ (4)

$$\int fd{\mu }_n=\frac{1}{n}\sum _{i=1}^n{\int }_0^1{Y}_i(t)\cdot f\left(t,X_i(t)\right)dt.$$ (5)

This allows us to define the following (random) norms and inner products

$$\Vert f{\Vert }_{{\mu }_n,1}=\int |f|d{\mu }_n$$

$$\Vert f{\Vert }_{{\mu }_n,2}={\left(\int f^{2}d{\mu }_n\right)}^{1/2}$$

$$\Vert f{\Vert }_{\infty }=\text{sup}\left\{\right|f(t,x)|:(t,x)\in [0,1]\times \mathfrak{X}\}$$

$${\langle f_1,f_2\rangle }_{{\mu }_n}=\int f_1{f}_{2}d{\mu }_n,$$

and note that ‖ · ‖_{μn, 1} ≤ ‖ · ‖_{μn, 2} ≤ ‖ · ‖_∞ because ${\mu }_n([0,1]\times \mathfrak{X})\le 1$.

By careful design, μ_n allows us to specify a natural domain for R_n(F). Let ${\left\{{\varphi }_j(t,x)\right\}}_{j=1}^d$ be a set of bounded functions $[0,1]\times \mathfrak{X}\mapsto [-1,1]$ that are linearly independent, in the sense that ${\int }_{[0,1]\times \mathfrak{X}}{\left({\sum }_j{c}_j{\varphi }_j\right)}^2\mathit{\text{dtdx}}=0$ if and only if c₁ = ⋯ = c_d = 0 (when some of the covariates are discrete-valued, dx should be interpreted as the product of a counting measure and the Lebesgue measure). The span of the functions is

$$\mathcal{F}=\left\{\sum _{j=1}^d{c}_j{\varphi }_j:c_j\in ℝ\right\}.$$

For example, the span of all regression tree functions that can be defined on $[0,1]\times \mathfrak{X}$, is $\mathcal{F}=\left\{{\sum }_j{c}_j{I}_{B_j}(t,x):c_j\in ℝ\right\}$,³ which are linear combinations of indicator functions over disjoint time-covariate cubes indexed⁴ by j = (j₀, j₁,·⋯, j_p):

$$B_j=\left\{\begin{array}{ll}(t,x)\in [0,1]\times \mathfrak{X}:\hfill & \begin{array}{c}{t}^{\left(j_0\right)}<t\le t^{\left(j_0+1\right)}\\ x^{\left(1,j_1\right)}<x^{(1)}\le x^{\left(1,j_1+1\right)}\\ ⋮\\ x^{\left(p,j_p\right)}<x^{(p)}\le x^{\left(p,j_p+1\right)}\end{array}\hfill \end{array}\right\}.$$ (6)

Remark 1. The regions B_j are formed using all possible split points ${\left\{x^{\left(k,j_k\right)}\right\}}_{j_k}$ for the k-th coordinate x^(k), with the spacing determined by the precision of the measurements. For example, if weight is measured to the closest kilogram, then the set of all possible split points will be {0.5, 1.5, 2.5, ⋯} kilograms. Note that these split points are the finest possible for any realization of weight that is measured to the nearest kilogram. While abstract treatments of trees assume that there is a continuum of split points, in reality they fall on a discrete (but fine) grid that is pre-determined by the precision of the data.

When $\mathcal{F}$ is equipped with 〈·, ·〉_μn, we obtain the following sample-dependent subspace of L²(μ_n), which is the appropriate domain for R_n(F):

$$\left(\mathcal{F},{\langle \cdot ,\cdot \rangle }_{{\mu }_n}\right).$$

Note that the elements in $\left(\mathcal{F},{\langle \cdot ,\cdot \rangle }_{{\mu }_n}\right)$ are equivalence classes rather than actual functions that have well defined values at each (t, x). This is a problem because the likelihood risk (2) requires evaluating F(t, x) at the points (T_i, X_i(T_i)) where Δ_i = 1. We resolve this by fixing an orthonormal basis {φ_nj(t, x)}_j for $\left(\mathcal{F},{\langle \cdot ,\cdot \rangle }_{{\mu }_n}\right)$, and represent each member of $\left(\mathcal{F},{\langle \cdot ,\cdot \rangle }_{{\mu }_n}\right)$ uniquely in the form ${\sum }_j{c}_j{\phi }_{nj}(t,x)$. For example in the case of regression trees, applying the Gram-Schmidt procedure to ${\left\{{\varphi }_j(t,x)=I_{B_j}(t,x)\right\}}_j$ gives

$${\left\{{\phi }_{nj}(t,x)\right\}}_j=\left\{\frac{I_{B_j}(t,x)}{{\mu }_n{\left(B_j\right)}^{1/2}}:{\mu }_n\left(B_j\right)>0\right\},$$

which by design have disjoint support.

The second condition we impose is for ${\left\{{\varphi }_j\right\}}_{j=1}^d$ to be linearly independent in L²(μ), that is ${\Vert {\sum }_j{c}_j{\varphi }_j\Vert }_{\mu ,2}^2={\sum }_{ij}{c}_i\left(\int {\varphi }_i{\varphi }_{j}d\mu \right)c_j=0$ if and only if c₁ = ⋯ = c_d = 0. Since by construction ${\left\{{\varphi }_j\right\}}_{j=1}^d$ are already linearly independent in $[0,1]\times \mathfrak{X}$, the condition intuitively requires the set of all possible time-covariate trajectories to be adequately dense in $[0,1]\times \mathfrak{X}$ to intersect a sufficient amount of the support of every ϕ_j. This is weaker than the identifiability conditions 1(ii)-1(iii) in [19] which require X(t) to have a positive joint probability density on $[0,1]\times \mathfrak{X}$.

Assumption (A2). The Gram matrix ${\Sigma }_{ij}=\int {\varphi }_i{\varphi }_{j}d\mu$ is positive definite.

2.2. Integral representations for the likelihood risk.

Having deduced the appropriate domain for R_n(F), we can now recast the risk as a smooth convex functional on $\left(\mathcal{F},{\langle \cdot ,\cdot \rangle }_{{\mu }_n}\right)$. Proposition 1 below provides closed form expressions for this and its gradient. We note that if the risk is actually of a certain simpler form, it might be possible to estimate its gradient empirically from our risk expression using [24].

Proposition 1. For functions F(t, x), f(t, x) of the form ${\sum }_j{c}_j{\phi }_{nj}(t,x)$, the likelihood risk (2) can be written as

$$R_n(F)=\int \left(e^F-{\lambda }_{n}F\right)d{\mu }_n,$$ (7)

where ${\lambda }_n\in \left(\mathcal{F},{\langle \cdot ,\cdot \rangle }_{{\mu }_n}\right)$ is the function

$${\lambda }_n(t,x)=\frac{1}{n}\sum _j\left\{\sum _{i=1}^n{\Delta }_i{\phi }_{nj}\left(T_i,X_i\left(T_i\right)\right)\right\}{\phi }_{nj}(t,x).$$

Thus there exists ρ ∈ (0, 1) (depending on F and f) for which the Taylor representation

$$R_n(F+f)=R_n(F)+{\langle g_F,f\rangle }_{{\mu }_n}+\frac{1}{2}\int e^{F+\rho f}{f}^{2}d{\mu }_n$$ (8)

holds, where the gradient

$$g_F(t,x)=\sum _j{\langle e^F,{\phi }_{nj}\rangle }_{{\mu }_n}{\phi }_{nj}(t,x)-{\lambda }_n(t,x)$$ (9)

of R_n(F) is the projection of e^F −λ_n onto $\left(\mathcal{F},{\langle \cdot ,\cdot \rangle }_{{\mu }_n}\right)$. Hence if gF = 0 then the infimum of R_n(F) over the span of {φ_nj(t, x)}_j is uniquely attained at F.

For regression trees the expressions (7) and (9) simplify further because $\mathcal{F}$ is closed under pointwise exponentiation, i.e. $e^F\in \mathcal{F}$ for $F\in \mathcal{F}$. This is because the B_j’s are disjoint so $F={\sum }_j{c}_j{I}_{B_j}$ and hence $e^F={\sum }_j{e}^{c_j}{I}_{B_j}$. Thus

$${\lambda }_n(t,x)=\sum _{j:{\mu }_n\left(B_j\right)>0}\frac{{\text{ Fail}}_j}{n{\mu }_n\left(B_j\right)}{I}_{B_j}(t,x),$$ (10)

$$R_n(F)=\sum _{j:{\mu }_n\left(B_j\right)>0}\left(e^{c_j}{\mu }_n\left(B_j\right)-\frac{c_j{\text{Fail}}_j}{n}\right),$$ (11)

$$g_F(t,x)=\sum _{j:{\mu }_n\left(B_j\right)>0}\left(e^{c_j}-\frac{{\text{ Fail}}_j}{n{\mu }_n\left(B_j\right)}\right)I_{B_j}(t,x),$$ (12)

where

$${\text{Fail}}_j=\sum _i{\Delta }_{i}I\left[\left\{T_i,X_i\left(T_i\right)\right\}\in B_j\right]$$

is the number of observed failures in the time-covariate region B_j.

Proof of Proposition 1. Fix a realization of ${\left\{\left(X_i(\cdot ),Y_i(\cdot ),T_i\right)\right\}}_{i=1}^n$. Using (5) we can rewrite (2) as

$$R_n(F)=\int e^{F}d{\mu }_n-\frac{1}{n}\sum _{i=1}^n{\Delta }_{i}F\left(T_i,X_i\left(T_i\right)\right).$$

We can express F in terms of the basis {φ_nk}_k as $F(t,x)={\sum }_k{c}_k{\phi }_{nk}(t,x)$. Hence

$$\begin{gathered} \int {\lambda }_{n}Fd{\mu }_n=\int \frac{1}{n}\sum _j\left\{\sum _{i=1}^n{\Delta }_i{\phi }_{nj}\left(T_i,X_i\left(T_i\right)\right)\right\}{\phi }_{nj}(t,x)F(t,x)d{\mu }_n \\ =\frac{1}{n}\sum _j\left\{\sum _{i=1}^n{\Delta }_i{\phi }_{nj}\left(T_i,X_i\left(T_i\right)\right)\right\}\int {\phi }_{nj}(t,x)F(t,x)d{\mu }_n \\ =\frac{1}{n}\sum _j\left\{\sum _{i=1}^n{\Delta }_i{\phi }_{nj}\left(T_i,X_i\left(T_i\right)\right)\right\}\int {\phi }_{nj}(t,x)\sum _k{c}_k{\phi }_{nk}(t,x)d{\mu }_n \\ =\frac{1}{n}\sum _j\left\{\sum _{i=1}^n{\Delta }_i{\phi }_{nj}\left(T_i,X_i\left(T_i\right)\right)\right\}{c}_j \\ =\frac{1}{n}\sum _{i=1}^n{\Delta }_i\sum _j{c}_j{\phi }_{nj}\left(T_i,X_i\left(T_i\right)\right) \\ =\frac{1}{n}\sum _{i=1}^n{\Delta }_{i}F\left(T_i,X_i\left(T_i\right)\right), \end{gathered}$$

where the fourth equality follows from the orthonormality of the basis. This completes the derivation of (7).

By an interchange argument we obtain

$$\frac{d}{d\theta }{R}_n(F+\theta f)=\int \left(e^{F+\theta f}-{\lambda }_n\right)fd{\mu }_n,$$

$$\frac{d^2}{d{\theta }^2}{R}_n(F+\theta f)=\int e^{F+\theta f}{f}^{2}d{\mu }_n,$$

the latter being positive whenever f ≠ 0; i.e., R_n(F) is convex. The Taylor representation (8) then follows from noting that gF is the orthogonal projection of e^F − λ_n ∈ L²(μ_n) onto $\left(\mathcal{F},{\langle \cdot ,\cdot \rangle }_{{\mu }_n}\right)$. □

The expectation of the likelihood risk also has an integral representation. A special case of the representation (13) below is proved in Proposition 3.2 of [19] for right-censored data only, under assumptions that do not allow for internal covariates. In the statement of the proposition below recall that Λ_L and Λ_U are defined in (A1). The constant ${\alpha }_{\mathcal{F}}$ is defined later in (28).

Proposition 2. For $F\in \mathcal{F}\cup \left\{\text{log }\lambda \right\}$,

$$R(F)=\mathbb{E}\left\{R_n(F)\right\}=\int \left(e^F-\lambda F\right)d\mu .$$ (13)

Furthermore the restriction of R(F) to $\mathcal{F}$ is coercive:

$$\frac{1}{2}R(F)\ge \frac{{\Lambda }_L}{{\alpha }_{\mathcal{F}}}\Vert F{\Vert }_{\infty }+{\Lambda }_U\text{min}\left\{0,1-\text{log }\left(2{\Lambda }_U\right)\right\},$$ (14)

and it attains its minimum at a unique point $F^{*}\in \left(\mathcal{F},{\langle \cdot ,\cdot \rangle }_{\mu }\right)$. If $\mathcal{F}$ contains the underlying log-hazard function then F* = log λ.

Remark 2. Coerciveness (14) implies that any F with expected risk R(F) less than R(0) ≤ 1 < 3 is uniformly bounded:

$$\Vert F{\Vert }_{\infty }<\frac{{\alpha }_{\mathcal{F}}}{{\Lambda }_L}\left[3/2+{\Lambda }_U\text{max}\left\{0,\text{log }\left(2{\Lambda }_U\right)-1\right\}\right]\le {\alpha }_{\mathcal{F}}{\beta }_{\Lambda }$$ (15)

where the constant

$${\beta }_{\Lambda }=\frac{3/2+{\Lambda }_U\text{max}\left\{0,\text{log }\left(2{\Lambda }_U\right)-1\right\}}{\text{min}\left\{1,{\Lambda }_L\right\}}$$ (16)

is by design no smaller than 1 in order to simplify subsequent analyses.

2.3. The boosting procedure.

In gradient boosting the key idea is to update an iterate in a direction that is approximately aligned to the negative gradient. To model this direction formally, we introduce the concept of an ε-gradient.

Definition 1. Suppose gF ≠ 0. We say that a unit vector $g_F^{\epsilon }\in \left(\mathcal{F},{\langle \cdot ,\cdot \rangle }_{{\mu }_n}\right)$ is an ε-gradient at F if for some 0 < ε ≤ 1,

$${\langle \frac{g_F}{{\Vert g_F\Vert }_{{\mu }_n,2}},g_F^{\epsilon }\rangle }_{{\mu }_n}\ge \epsilon .$$ (17)

Call $-g_F^{\epsilon }$ a negative ε-gradient if $g_F^{\epsilon }$ is an ε-gradient.

Our boosting procedure seeks approximations $g_F^{\epsilon }$ that satisfy (17) for some pre-specified alignment value ε. The larger ε is, the closer the alignment is between the negative gradient and the negative ε-gradient, and the greater the risk reduction. In particular, −gF is the unique negative 1-gradient with maximal risk reduction. In practice, however, we find that using a smaller value of ε leads to simpler approximations that prevent overfitting in finite samples. This is consistent with other implementations of boosting: It is well known that the statistical performance of gradient descent generally improves when simpler base learners are used.

Algorithm 1 describes the proposed boosting procedure for estimating λ. For a given level of alignment ε, Line 3 finds an ε-gradient $g_{F_m}^{\epsilon }$ at F_m satisfying (18) at the m-th iteration, and uses its negation for the boosting update in Line 4. If the ε-gradients are regression tree learners, as is the case with the implementation in Section 4, then the trees cannot be grown in the same way as the standard boosting algorithm in Friedman [13]. This is because the standard approach grows all regression trees to a fixed depth, which may or may not ensure ε-alignment at each boosting iteration.

To ensure ε-alignment, the depth of the trees are not fixed in the implementation in Section 4. Instead, at each boosting iteration a tree is grown to whatever depth is needed to satisfy (18). This can always be done because the alignment ε is non-decreasing in the number of tree splits, and with enough splits we can recover the gradient gF_m itself up to μ_n-almost everywhere.⁵ As mentioned earlier, we recommend using small values of ε, which can be determined in practice using cross-validation. This differs from the standard approach where cross-validation is used to select a common tree depth to use for all boosting iterations.

In addition to the gradient alignment ε, Algorithm 1 makes use of two other regularization parameters, Ψ_n and ν_n. The first defines the early stopping criterion (how many boosting iterations to use), while the second controls the step-sizes of the boosting updates. These are two common regularization techniques used in boosting:

1. Early stopping. The number of boosting iterations $\widehat{m}$ is controlled by stopping the algorithm before the uniform norm of the estimator ${\Vert F_{\widehat{m}}\Vert }_{\infty }$ reaches or exceeds

   $${\Psi }_n=W\left(n^{1/4}\right)\to \infty ,$$ (19)

   where W(y) is the branch of the Lambert function that returns the real root of the equation ze^z = y for y > 0.
2. Step-sizes. The step-size ν_n ≪ 1 used in gradient boosting is typically held constant across iterations. While we can also do this in our procedure,⁶ the role of step-size shrinkage becomes more salient if we use ν_n/(m+1) instead as the step-size for the m-th iteration in Algorithm 1. This step-size is controlled in two ways. First, it is made to decrease with each iteration according to the Robbins-Monro condition that the sum of the steps diverges while the sum of squared steps converges. Second, the shrinkage factor ν_n is selected to make the step-sizes decay with n at rate

   $${\nu }_n^2{e}^{{\Psi }_n}<1,{\nu }_n^2{e}^{{\Psi }_n}\to 0.$$ (20)

   This acts as a counterbalance to R_n(F)’s unbounded curvature:

   $${\frac{d^2}{d{\theta }^2}{R}_n(F+\theta f)|}_{\theta =0}=\int e^F{f}^{2}d{\mu }_n,$$ (21)

   which is upper bounded by $e^{{\Psi }_n}$ when ‖F‖_∞ < Ψ_n and $\Vert f{\Vert }_{{\mu }_n,2}=1$.

3. Consistency.

Under (A1) and (A2), guarantees for our hazard estimator ${\widehat{\lambda }}_{\text{boost}}$ in Algorithm 1 can be derived for two scenarios of interest. The guarantees rely on the regularizations described in Section 2.3 to avoid overfitting. In the following development, recall from Proposition 2 that F* is the unique minimizer of R(F), so it satisfies the first order condition

$${\langle e^{F^{*}}-\lambda ,F\rangle }_{\mu }=0$$ (22)

for all $F\in \mathcal{F}$. Recall that the span of all trees is closed under pointwise exponentiation $\left(e^F\in \mathcal{F}\right)$, in which case (22) implies that ${\lambda }^{*}=e^{F^{*}}$ is the orthogonal projection of λ onto $\left(\mathcal{F},{\langle \cdot ,\cdot \rangle }_{\mu }\right)$.

1. Consistency when $\mathcal{F}$ is correctly specified. If the true log-hazard function log λ is in $\mathcal{F}$, then Proposition 2 asserts that F* = log λ. It will be shown in this case that ${\widehat{\lambda }}_{\text{boost}}$ is consistent:

   $${\Vert {\widehat{\lambda }}_{\text{boost}}-\lambda \Vert }_{\mu ,2}^2=o_p(1).$$
2. Oracle inequality for regression trees. If $\mathcal{F}$ is closed under pointwise exponentiation, it follows from (22) that λ* is the best L²(μ)-approximation to λ among all candidate hazard estimators $\left\{e^F:F\in \mathcal{F}\right\}$. It can then be shown that ${\widehat{\lambda }}_{\text{boost}}$ converges to this best approximation:

   $${\Vert {\widehat{\lambda }}_{\text{boost}}-\lambda \Vert }_{\mu ,2}^2={\Vert {\lambda }^{*}-\lambda \Vert }_{\mu ,2}^2+o_p(1).$$

   This oracle result is in the spirit of the type of guarantees available for tree-based boosting in the non-functional data setting. For example, if tree stumps are used for L₂-regression, then the regression function estimate will converge to the best approximation to the true regression function in the span of tree stumps [9]. Similar results also exist for boosted classifiers [5].

Propositions 3 and 4 below formalize these guarantees by providing bounds on the error terms above. While sharper bounds may exist, the chief purpose of this paper is to introduce our generic estimator for the first time and to provide guarantees that apply across different implementations. More refined convergence rates may exist for a specific implementation, just like the analysis in Bühlmann and Yu [11] for L₂Boosting when componentwise spline learners are specifically used. We leave this for future research.

En route to establishing the guarantees, Lemma 2 below clarifies the role played by step-size restriction in ensuring convergence of the estimator. As explained in the Introduction, explicit shrinkage is not necessary for classification and regression problems where the risk has bounded curvature. Lemma 2 suggests that it may, however, be needed when the risk has unbounded curvature, as is the case with R_n(F). Seen in this light, shrinkage is really a mechanism for controlling the growth of the risk curvature.

3.1. Strategy for establishing guarantees.

The representations for R_n(F) and its population analogue R(F) from Section 2 are the key ingredients for formalizing the guarantees. We use them to first show that $F_{\widehat{m}}\in \left(\mathcal{F},{\langle \cdot ,\cdot \rangle }_{{\mu }_n}\right)$ converges to $F^{*}\in \left(\mathcal{F},{\langle \cdot ,\cdot \rangle }_{\mu }\right)$: Applying Taylor’s theorem to the representation for R(F) in Proposition 2 yields

$${\Vert F_{\widehat{m}}-F^{*}\Vert }_{\mu ,2}^2\le 2\frac{R\left(F_{\widehat{m}}\right)-R\left(F^{*}\right)}{{\text{min}}_{t,x}\left({\lambda }^{*}\wedge {\widehat{\lambda }}_{\text{boost}}\right)}.$$ (23)

The problem is thus transformed into one of risk minimization $R\left(F_{\widehat{m}}\right)\to R\left(F^{*}\right)$, for which [31] suggests analyzing separately the terms of the decomposition

$$\begin{array}{ll}0\le R\left(F_{\widehat{m}}\right)-R\left(F^{*}\right)\hfill & \hfill \\ \le \left|R_n\left(F_{\widehat{m}}\right)-R\left(F_{\widehat{m}}\right)\right|\hfill & \text{ (I) COMPLEXITY ARGUMENT }\hfill \\ +\left|R_n\left(F^{*}\right)-R\left(F^{*}\right)\right|\hfill & \text{ (II) STANDARD ARGUMENT }\hfill \\ +\left\{R_n\left(F_{\widehat{m}}\right)-R_n\left(F^{*}\right)\right\}.\hfill & \text{ (III) CURVATURE ARGUMENT }\hfill \end{array}$$ (24)

The authors argue that in boosting, the point of limiting the number of iterations $\widehat{m}$ (enforced by lines 5–10 in Algorithm 1) is to prevent $F_{\widehat{m}}$ from growing too fast, so that (I) converges to zero as n → ∞. At the same time, $\widehat{m}$ is allowed to grow with n in a controlled manner so that the empirical risk $R_n\left(F_{\widehat{m}}\right)$ in (III) is eventually minimized as n → ∞. Lemmas 1 and 2 below show that our procedure achieves both goals. Lemma 1 makes use of complexity theory via empirical processes, while Lemma 2 deals with the curvature of the likelihood risk. The term (II) will be bounded using standard concentration results.

3.2. Bounding (I) using complexity.

To capture the effect of using a simple negative ε-gradient (17) as the descent direction, we bound (I) in terms of the complexity of⁷

$$\begin{gathered} {\mathcal{F}}_{\epsilon }={\mathcal{F}}_{\epsilon ,\text{boost}}\cup \left\{F\in \mathcal{F}:\Vert F{\Vert }_{\infty }=1\right\}\subseteq \mathcal{F}, \\ \text{ where }{\mathcal{F}}_{\epsilon ,\text{boost}}=\left\{F_m=-\sum _{k=0}^{m-1}\frac{{\nu }_n}{k+1}{g}_{F_k}^{\epsilon }:m=0,1,\dots \right\}. \end{gathered}$$ (25)

Depending on the choice of weak learners for the ε-gradients, ${\mathcal{F}}_{\epsilon }$ may be much smaller than $\mathcal{F}$. For example, coordinate descent might only ever select a small subset of basis functions {ϕ_j}_j because of sparsity. As another example if λ(t, x) is additively separable in time and also in each covariate, then regression trees might only ever select simple tree stumps (one tree split).

The measure of complexity we use below comes from empirical process theory. Define ${\mathcal{F}}_{\epsilon }^{\Psi }=\left\{F\in {\mathcal{F}}_{\epsilon }:\Vert F{\Vert }_{\infty }<\Psi \right\}$ for Ψ > 0 and suppose that Q is a sub-probability measure on $[0,1]\times \mathfrak{X}$. Then the L²(Q)-ball of radius δ > 0 centred at some F ∈ L₂(Q) is $\left\{F^{\prime }\in {\mathcal{F}}_{\epsilon }^{\Psi }:{\Vert F^{\prime }-F\Vert }_{Q,2}<\delta \right\}$. The covering number $\mathcal{N}\left(\delta ,{\mathcal{F}}_{\epsilon }^{\Psi },Q\right)$ is the minimum number of such balls needed to cover ${\mathcal{F}}_{\epsilon }^{\Psi }$ (Definitions 2.1.5 and 2.2.3 of van der Vaart and Wellner [29]), so $\mathcal{N}\left(\delta ,{\mathcal{F}}_{\epsilon }^{\Psi },Q\right)=1$ for δ ≥ Ψ. A complexity measure for ${\mathcal{F}}_{\epsilon }$ is

$$J_{{\mathcal{F}}_{\epsilon }}=\underset{\Psi ,Q}{\text{sup}}\left\{{\int }_0^1{\left\{\text{log }\mathcal{N}\left(u\Psi ,{\mathcal{F}}_{\epsilon }^{\Psi },Q\right)\right\}}^{1/2}du\right\},$$ (26)

where the supremum is taken over Ψ > 0 and over all non-zero sub-probability measures. As discussed, $J_{\mathcal{F}}$ is never greater than, and potentially much smaller than $J_{\mathcal{F}}$, the complexity of $\mathcal{F}$, which is fixed and finite.

Before stating Lemma 1, we note that the result also shows an empirical analogue to the norm equivalences

$$\Vert F{\Vert }_{\mu ,1}\le \Vert F{\Vert }_{\mu ,2}\le \Vert F{\Vert }_{\infty }\le \frac{{\alpha }_{\mathcal{F}}}{2}\Vert F{\Vert }_{\mu ,1}\text{ for all }F\in \mathcal{F}$$ (27)

exists, where

$${\alpha }_{\mathcal{F}}=2\underset{F\in \mathcal{F}:\Vert F{\Vert }_{\infty }=1}{\text{sup}}\left(\frac{\Vert F{\Vert }_{\infty }}{\Vert F{\Vert }_{\mu ,1}}\right)=\frac{2}{\underset{F\in \mathcal{F}:\Vert F{\Vert }_{\infty }=1}{\text{inf}}\Vert F{\Vert }_{\mu ,1}}>1.$$ (28)

The factor of 2 serves to simplify the presentation, and can be replaced with anything greater than 1.

Lemma 1. There exists a universal constant κ such that for any 0 < η < 1, with probability at least

$$1-4\text{ exp }\left\{-{\left(\frac{\eta n^{1/4}}{\kappa {\alpha }_{\mathcal{F}}{J}_{{\mathcal{F}}_{\epsilon }}}\right)}^2\right\}$$

an empirical analogue to (27) holds for all $F\in \mathcal{F}$:

$$\Vert F{\Vert }_{{\mu }_n,1}\le \Vert F{\Vert }_{{\mu }_n,2}\le \Vert F{\Vert }_{\infty }\le {\alpha }_{\mathcal{F}}\Vert F{\Vert }_{{\mu }_n,1},$$ (29)

and for all $F\in {\mathcal{F}}_{\epsilon }^{{\Psi }_n}$,

$$\left|\left\{R_n(F)-R_n(0)\right\}-\{R(F)-R(0)\}\right|<\eta .$$ (30)

Remark 3. The equivalences (29) imply that $\text{dim}\left(\mathcal{F},{\langle \cdot ,\cdot \rangle }_{{\mu }_n}\right)$ equals its upper bound $\text{dim}\mathcal{F}=d$. That is, if ${\Vert {\sum }_j{c}_j{\varphi }_j\Vert }_{{\mu }_n,2}=0$, then ${\Vert {\sum }_j{c}_j{\varphi }_j\Vert }_{\infty }=0$, so c₁ = ⋯ = c_d = 0 because ${\left\{{\varphi }_j\right\}}_{j=1}^d$ are linearly independent on $[0,1]\times \mathfrak{X}$.

3.3. Bounding (III) using curvature.

We use the representation in Proposition 1 to study the minimization of the empirical risk R_n(F) by boosting. Standard results for exact gradient descent like Theorem 2.1.15 of Nesterov [25] are in terms of the norm of the minimizer, which may not exist for R_n(F).⁸ If coordinate descent is used instead, Section 4.1 of [31] can be applied to convex functions whose infimum may not be attainable, but its curvature is required to be uniformly bounded above. Since the second derivative of R_n(F) is unbounded (21), Lemma 2 below provides two remedies: (i) Use the shrinkage decay (20) of ν_n to counterbalance the curvature; (ii) Use coercivity (15) to show that with increasing probability, ${\left\{F_m\right\}}_{m=0}^{\widehat{m}}$ are uniformly bounded, so the curvatures at those points are also uniformly bounded. Lemma 2 combines both to derive a result that is simpler than what can be achieved from either one alone. In doing so, the role played by step-size restriction becomes clear. The lemma relies in part on adapting the analysis in Lemma 4.1 of [31] for coordinate descent to the case for generic ε-gradients. The conditions required below will be shown to hold with high probability.

Lemma 2. Suppose (29) holds and that

$$\left|R_n\left(F^{*}\right)-R\left(F^{*}\right)\right|<1,\underset{F\in {\mathcal{F}}_{\text{s}}^{*}n}{\text{sup}}\left|R_n(F)-R(F)\right|<1.$$

Then the largest gap between F* and ${\left\{F_m\right\}}_{m=0}^{\widehat{m}}$,

$$\widehat{\gamma }=\underset{m\le \widehat{m}}{\text{max}}{\Vert F_m-F^{*}\Vert }_{\infty }\vee 1,$$ (31)

is bounded by a constant no greater than $2{\alpha }_{\mathcal{F}}{\beta }_{\Lambda }$, and for n ≥ 55,

$$R_n\left(F_{\widehat{m}}\right)-R_n\left(F^{*}\right)<2e{\beta }_{\Lambda }{\left(\frac{\text{log }n}{4n^{1/4}}\right)}^{\epsilon /\left({\alpha }_{\mathcal{F}}\widehat{\gamma }\right)}+{\nu }_n^2{e}^{{\Psi }_n}.$$ (32)

Remark 4. The last term in (32) suggests that the role of the step-size shrinkage ν_n is to keep the curvature of the risk in check, to prevent it from derailing convergence. Recall from (21) that $e^{{\Psi }_n}$ describes the curvature of R_n(F_m). Thus our result clarifies the role of step-size restriction in boosting functional data.

Remark 5. Regardless of whether the risk curvature is bounded or not, smaller step-sizes always improve the convergence bound. This can be seen from the parsimonious relationship between ν_n and (32). Fixing n, pushing the value of ν_n down towards zero yields the lower limit

$$2e{\beta }_{\Lambda }{\left(\frac{\text{log }n}{4n^{1/4}}\right)}^{\epsilon /\left({\alpha }_{\mathcal{F}}\widehat{\gamma }\right)}.$$

However, this limit is unattainable as ν_n must be positive in order to decrease the risk. This effect has been observed in practical applications of boosting. Friedman [13] noted improved performance for gradient boosting with the use of a small shrinkage factor ν. At the same time, it was also noted there was diminishing performance gain as ν became very small, and this came at the expense of an increased number of boosting iterations. This same phenomenon has also been observed for L₂Boosting [11] with componentwise linear learners. It is known that the solution path for L₂Boosting closely matches that of lasso as ν → 0. However, the algorithm exhibits cycling behaviour for small ν, which greatly increases the number of iterations and offsets the performance gain in trying to approximate the lasso (see Ehrlinger and Ishwaran [12]).

3.4. Formal statements of guarantees.

As a reminder, we have defined the following quantities:

• ${\widehat{\lambda }}_{\text{boost}}$ = $e^{F_{\widehat{m}}}$, the boosted hazard estimator in Algorithm 1

• λ* = $e^{F^{*}}$, where F* is the unique minimizer of R(F) in Proposition 2

• Λ_L,Λ_U = lower and upper bounds on λ(t, x) as defined in (A1)

• $\widehat{\gamma }$ = maximum gap between F* and ${\left\{F_m\right\}}_{m=0}^{\widehat{m}}$ defined in (31)

• κ = a universal constant

• ${\alpha }_{\mathcal{F}}$ = constant defined in (28) β_Λ = constant defined in (16)

• $J_{{\mathcal{F}}_{\epsilon }}$ = complexity measure (26), bounded above by $J_{\mathcal{F}}$

To simplify the results, we will assume that n ≥ 55 and also set the shrinkage to satisfy ${\nu }_n^2{e}^{{\Psi }_n}=\text{log }n/\left(64n^{1/4}\right)$. Our first guarantee shows that our hazard estimator is consistent if the model is correctly specified.

Proposition 3. (Consistency under correct model specification). Suppose $\mathcal{F}$ contains the true log-hazard function log λ. Then with probability

$$1-8\text{ exp }\left\{-{\left(\frac{\text{log }n}{\kappa {\alpha }_{\mathcal{F}}\left({\Lambda }_L^{-1}\vee {\Lambda }_U\right)J_{{\mathcal{F}}_{\epsilon }}}\right)}^2\right\}$$

we have that ${\Vert {\widehat{\lambda }}_{\text{boost}}\Vert }_{\infty }$ is bounded and

$${\Vert {\widehat{\lambda }}_{\text{boost}}-\lambda \Vert }_{\mu ,2}^2<13{\beta }_{\Lambda }\frac{{\text{max}}_{t,x}{\left(\lambda \vee {\widehat{\lambda }}_{\mathit{\text{boost}}}\right)}^2}{{\text{min}}_{t,x}\left(\lambda \wedge {\widehat{\lambda }}_{\mathit{\text{boost}}}\right)}{\left(\frac{\text{log }n}{4n^{1/4}}\right)}^{\epsilon /\left({\alpha }_{\mathcal{F}}\widehat{\gamma }\right)}.$$

Thus ${\widehat{\lambda }}_{\text{boost}}$ is consistent.

Via the tension between ε and $J_{{\mathcal{F}}_{\epsilon }}$, Proposition 3 captures the trade-off in statistical performance between weak and strong learners in gradient boosting. The advantage of low complexity (weak learners) is reflected in the increased probability of the L²(μ)-bound holding, with this probability being maximized when $J_{{\mathcal{F}}_{\epsilon }}\to 0$, which generally occurs as ε → 0. However, diametrically opposed to this, we find that the L²(μ)-bound is minimized by ε → 1, which occurs with the use of stronger learners that are more aligned with the gradient. This same trade-off is also captured by our second guarantee which establishes an oracle inequality for tree learners.

Proposition 4. (Oracle inequality for tree learners). Suppose $e^F\in \mathcal{F}$ for $F\in \mathcal{F}$. Then among $\left\{e^F:F\in \mathcal{F}\right\}$, λ* is the best L²(μ)-approximation to λ, that is

$${\lambda }^{*}=\text{arg}\underset{e^F:F\in \mathcal{F}}{\text{ min}}{\Vert e^F-\lambda \Vert }_{\mu ,2}.$$

Moreover, ${\widehat{\lambda }}_{\text{boost}}$ converges to this best approximation λ*: With probability

$$1-8\text{ exp }\left\{-{\left(\frac{\text{log }n}{\kappa {\alpha }_{\mathcal{F}}\left({\Lambda }_L^{-1}\vee {\Lambda }_U\right)J_{{\mathcal{F}}_{\epsilon }}}\right)}^2\right\}$$

we have that ${\Vert {\widehat{\lambda }}_{\mathit{\text{boost}}}\Vert }_{\infty }$ is bounded and

$${\Vert {\widehat{\lambda }}_{\mathit{\text{boost}}}-\lambda \Vert }_{\mu ,2}^2<{\rho }_{\mathcal{F}}^2+13{\beta }_{\Lambda }\frac{{\text{max}}_{t,x}{\left({\Lambda }_U\vee {\widehat{\lambda }}_{\mathit{\text{boost}}}\right)}^2}{{\text{min}}_{t,x}\left({\Lambda }_L\wedge {\widehat{\lambda }}_{\text{boost}}\right)}{\left(\frac{\text{log }n}{4n^{1/4}}\right)}^{\epsilon /\left({\alpha }_{\mathcal{F}}\widehat{\gamma }\right)},$$

where ${\rho }_{\mathcal{F}}^2={\Vert {\lambda }^{*}-\lambda \Vert }_{\mu ,2}^2$ is the smallest error one can achieve from using functions in $\left\{e^F:F\in \mathcal{F}\right\}$ to approximate λ.

For tree learners, λ*(t, x) is constant over each region B_j in (6), and its value equals the local average of λ over B_j,

$${{\lambda }^{*}(t,x)|}_{B_j}=\frac{1}{\mu \left(B_j\right)}{\int }_{B_j}\lambda d\mu .$$

Hence if the B_j’s are small, λ* should closely approximate λ (recall from Remark 1 that the size of the B_j’s is fixed by the data). To estimate the approximation error ${\rho }_{\mathcal{F}}$ in terms of B_j, suppose that λ is sufficiently smooth, e.g. Hölder continuous $\left|\lambda (t,x)-\lambda \left(t^{\prime },x^{\prime }\right)\right|\precsim {\Vert \left(t-t^{\prime },x-x^{\prime }\right)\Vert }^b$ for some b > 0. Then since ${\text{inf}}_{B_j}\lambda \le {{\lambda }^{*}|}_{B_j}\le {\text{sup}}_{B_j}\lambda$,

$${\rho }_{\mathcal{F}}\le {\Vert {\lambda }^{*}-\lambda \Vert }_{\infty }\precsim \underset{j}{\text{max}}{\left(\text{diam}{B}_j\right)}^b.$$

4. A tree-based implementation.

Here we describe an implementation of Algorithm 1 using regression trees, whereby the ε-gradient $g_{F_m}^{\epsilon }$ is obtained by growing a tree to satisfy (18) for a pre-specified ε.

To explain the tree growing process, first observe that the m-th step log-hazard estimator is an additive expansion of CART basis functions. Thus it can be written as

$$\begin{gathered} F_m(t,x)=\sum _{b=0}^{m-1}\sum _{l=1}^{L_b}{\gamma }_{b,l}{I}_{A_{b,l}}(t,x) \\ =\sum _j{c}_{m,j}{I}_{B_j}(t,x), \end{gathered}$$ (33)

where A_b,l is the l-th leaf region of the b-th tree. Recall from Section 2.3 that each tree is grown until (18) is satisfied, so the number of leaf nodes L_b can vary from tree to tree. The leaf regions are typically large subsets of the time-covariate space $[0,1]\times \mathfrak{X}$ adaptively determined by the tree growing process (to be discussed shortly). Since each leaf region can be further decomposed into the finer disjoint regions B_j in (6), F_m(t, x) can be rewritten as (33). However, many of these regions will share the same coefficient value, so (33) can be written more compactly as

$$F_m(t,x)=\sum _j{c}_{m,j}{I}_{B_{m,j}^{\prime }}(t,x),$$

where $B_{m,j}^{\prime }$ is the union of contiguous regions whose coefficient equals c_m,j. This smooths the hazard estimator ${\widehat{\lambda }}_{\text{boost}}(t,x)$ over $[0,1]\times \mathfrak{X}$, thanks to the regularization imposed by limiting the number of trees (early stopping) and also by the use of weak tree learners. This is unlike the unconstrained hazard MLE λ_n(t, x) defined in (10), which can take on a different value in each region B_j, making it prone to overfit the data.

To construct an ε-gradient $g_{F_m}^{\epsilon }$ with ε-alignment to gF_m defined by (12),

$$g_{F_m}(t,x)=\sum _{j:{\mu }_n\left(B_j\right)>0}\left(e^{c_{m,j}}-\frac{{\text{ Fail }}_j}{n{\mu }_n\left(B_j\right)}\right)I_{B_j}(t,x),$$

the tree splits are adaptively chosen to reduce the L²(μ_n)-approximation error between $g_{F_m}^{\epsilon }$ and gF_m. We implement tree splits for both time and covariates. Specifically, suppose we wish to split a leaf region $A\subseteq [0,1]\times \mathfrak{X}$ into left and right daughter subregions A₁ and A₂, and assign values γ₁ and γ₂ to them. For example, a split on the k-th covariate could propose left and right daughters such as

$$A_1=\left\{(t,x)\in A:x^{(k)}\le s\right\},A_2=\left\{(t,x)\in A:x^{(k)}>s\right\},$$ (34)

or a split on time t could propose regions

$$A_1=\{(t,x)\in A:t\le s\},A_2=\{(t,x)\in A:t>s\}.$$ (35)

Now note that gF_m is constant within each region B_j. We denote its value by $g_{F_m}\left(t_{B_j},x_{B_j}\right)$ where $\left(t_{B_j},x_{B_j}\right)$ is the centre of B_j. Hence the best split of A into A₁ and A₂ is the one that minimizes

$$\begin{gathered} \underset{{\gamma }_1}{\text{min}}{\int }_{A_1}{\left\{g_{F_m}(t,x)-{\gamma }_1\right\}}^{2}d{\mu }_n+\underset{{\gamma }_2}{\text{min}}{\int }_{A_2}{\left\{g_{F_m}(t,x)-{\gamma }_2\right\}}^{2}d{\mu }_n \\ =\underset{{\gamma }_1}{\text{min}}\sum _{j:B_j\subseteq A_1}{\mu }_n\left(B_j\right)\cdot {\left\{g_{F_m}\left(t_{B_j},x_{B_j}\right)-{\gamma }_1\right\}}^2+\underset{{\gamma }_2}{\text{min}}\sum _{k:B_k\subseteq A_2}{\mu }_n\left(B_k\right)\cdot {\left\{g_{F_m}\left(t_{B_k},x_{B_k}\right)-{\gamma }_2\right\}}^2 \\ =\underset{{\gamma }_1}{\text{min}}\sum _{\begin{array}{c}j:z_j\in A_1,\\ w_j>0\end{array}}{w}_j\cdot {\left({\tilde{y}}_j-{\gamma }_1\right)}^2+\underset{{\gamma }_2}{\text{min}}\sum _{\begin{array}{c}k:z_k\in A_2,\\ w_k>0\end{array}}{w}_k\cdot {\left({\tilde{y}}_k-{\gamma }_2\right)}^2, \end{gathered}$$ (36)

where

$${\tilde{y}}_j=g_{F_m}\left(t_{B_j},x_{B_j}\right)=e^{c_{m,j}}-\frac{{\text{Fail}}_j}{n{\mu }_n\left(B_j\right)}$$

represents the j-th pseudo-response, $z_j=\left(t_{B_j},x_{B_j}\right)$ its covariate and w_j = μ_n(B_j) its weight. Thus the splits use a weighted least squares criterion, which can be efficiently computed as usual.

We split the tree until (18) is satisfied, resulting in L_m leaf nodes (L_m − 1 splits). As discussed in Section 2.3, we can always find a deep enough tree that is an ε-gradient because with enough splits we can recover the gradient gF_m itself. Recall also that a small value of ε performs best in practice, and this can be chosen by cross-validating on a set of small-sized candidates: For each one we implement Algorithm 1, and we select the one that minimizes the cross-validated risk R_n(F) defined in (11). By contrast, the standard boosting algorithm [13] uses cross-validation to select a common number of splits to use for all trees, which does not ensure that each tree is an ε-gradient.

Regarding the possible split points for the covariates (34), note that the k-th covariate x^(k) = x^(k)(t) is a time series that is sampled periodically. This yields a set of unique values equal to the union of all of the sampled values for the n observations. In direct analogy to non-functional data boosting, we place candidate split points in-between the sorted values in this set. In other words, splits for covariates only occur at values corresponding to the observed data just as in non-functional boosting.

The resolution for the grid of candidate time splits (35) is set equal to the temporal resolution. For example, the covariate trajectories in the simulation in Section 5 are piecewise constant and may change every 0.002 days. Placing the candidate split points at 0.002, 0.004, ⋯ days simplifies the exact computation of μ_n(B_j) because every covariate trajectory is constant between these points. Again, notice that the splits for time only occur at values informed by the observed data.

Putting it together, the setup above leverages our insight in (36) by transforming the survival functional data into the data values ${\left\{w_j,{\tilde{y}}_j,z_j\right\}}_{j:w_j}>0$, which enables the implementation to proceed like standard gradient boosting for non-functional data. Only the pseudo-response ${\tilde{y}}_j$ in $\left\{w_j,{\tilde{y}}_j,z_j\right\}$ needs to be updated at each boosting iteration, while the other two do not change. In terms of storage it costs $\mathcal{O}(np|\mathcal{T}|)$ to store ${\left\{w_j,{\tilde{y}}_j,z_j\right\}}_{j:w_j}>0$, where $|\mathcal{T}|$ is the cardinality of the set of candidate time splits.⁹ Computationally, choosing a new tree split requires testing $\mathcal{O}(np|\mathcal{T}|)$ candidate splits.¹⁰ The space and time complexities of the implementation are reasonable given that they are $\mathcal{O}(np)$ for non-functional data boosting: In the functional data setting, each sample can have up to $|\mathcal{T}|$ observations, so n functional data samples is akin to $\mathcal{O}(n|\mathcal{T}|)$ samples in a non-functional data setting.

5. Numerical experiment.

We now apply the boosting procedure of Section 4 to a high-dimensional dataset generated from a naturalistic simulation. This allows us to compare the performance of our estimator to existing boosting methods. The simulation is of patient service times in an emergency department (ED), and the hazard function of interest is patient service rate in the ED. The study of patient transitions in an ED queue is an important one in healthcare operations, because without a high resolution model of patient flow dynamics, the ED may be suboptimally utilized which would deny patients of timely critical care.

5.1. Service rate.

The service rate model used in the simulation is based upon a service time dataset from the ED of an academic hospital in the United States. The dataset contains information on 86,983 treatment encounters from 2014 to early 2015. Recorded for each encounter was: Age, gender, Emergency Severity Index (ESI)¹¹, time of day when treatment in the ED ward began, day of week of ED visit, and ward census. The last one represents the total number of occupied beds in the ED ward, which varies over the course of the patient’s stay. Hence it is a time-dependent variable. Lastly, we also have the duration of the patient’s stay (service time).

The service rate function is developed from the data in the following way. First, we apply our nonparametric estimator to the data to perform exploratory analysis. We find that:

1. The key variables affecting the service rate (based on relative variable importance [13]) are ESI, age, and ward census. In addition, two of the most pronounced interaction terms identified by the tree splits are (AGE ≥ 34, ESI = 5) and (AGE ≥ 34, ESI ≤ 4).

2. Holding all the variables fixed, the shapes of the estimated service rate function resemble the hazard functions of log-normal distributions. This agrees with the queuing literature that find log-normality to be a reasonable parametric fit for service durations.

Guided by these findings, we specify the service rate λ(t, X(t)) for the simulation as a log-normal accelerated failure time (AFT) model, and estimate its parameters from data. This yields the service rate

$$\lambda (t,x)=\theta (x)\cdot \frac{{\varphi }_l(\theta (x)t;m,\sigma )}{1-{\Phi }_l(\theta (x)t;m,\sigma )},$$ (37)

where ϕ_l(·;m, σ) and Φ_l(·;m, σ) are the PDF and CDF of the log-normal distribution with log-mean m = −1.8 and log-standard deviation σ = 0.74. The function θ(x) captures the dependence of the service rate on the covariates:

$$\text{log }\theta (X(t))=-0.0071\cdot \text{AGE}+0.022\cdot \text{ESI}-\text{min}\left\{a\cdot \frac{{\text{CENSUS}}_t}{70},2\right\}+0.10\cdot I(\text{AGE}\ge 34,\text{ESI}=5)-0.10\cdot I(\text{AGE}\ge 34,\text{ESI}\le 4)+0\cdot {\text{ NUISANCE }}_1+\cdots +0\cdot {\text{NUISANCE}}_{43}.$$ (38)

The specification for θ(X(t)) above is a slight modification of the original estimate, with the free parameter a allowing us to study the effect of time-dependent covariates on hazard estimation. When a = 0, the service rate does not depend on time-varying covariates, but as a increases, the dependency becomes more and more significant. In the data, the ward census never exceeds 70, so we set the capacity of the simulated ED to 70 as well. The min operator caps the impact that census can have on the simulated service rate as a grows. The irrelevant covariates NUISANCE₁, ⋯, NUISANCE₄₃ are added to the data in order to assess how boosting performs in high dimensions. We explicitly include them in (38) to remind ourselves that the simulated data is high-dimensional. Forty of the irrelevant variables are generated synthetically as described in the next subsection, while the rest are variables from the original dataset not used in the simulation.

5.2. Simulation model.

Using (37) and (38), we simulate a naturalistic dataset of 10,000 patient visit histories. The value of a will be varied from 0 to 3 in order to study the impact of time-dependent covariates on hazard estimation. Each patient is associated with a 46-dimensional covariate vector consisting of:

• The time-varying ward census. The initial value is sampled from its marginal empirical distribution in the original dataset. To simulate its trajectory over a patient’s stay, for every timestep advance of 0.002 days (≈3 minutes), a Bernoulli(0.02) random variable is generated. If it is one, then the census is incremented by a normal random variable with zero mean and standard deviation 10. The result is truncated if it lies outside the range [1, 70], the upper end being the capacity of the ED.

• The other five time-static covariates in the original dataset. These are sampled from their marginal empirical distributions in the original dataset. Two of the variables (age and ESI) influence the service rate, while the other three are irrelevant.

• An additional forty time-static covariates that do not affect the service rate (irrelevant covariates). Their values are drawn uniformly from [0, 1].

We also generate independent censoring times (rounded to the nearest 0.002 days) for each visit from an exponential distribution. For each simulation, the rate of the exponential distribution is set to achieve an approximate target of 25% censoring.

5.3. Comparison benchmarks.

When the covariates are static in time, a few software packages are available for performing hazard estimation with tree ensembles. Given that the data is simulated from a log-normal hazard, we compare our nonparametric method to two correctly specified parametric estimators:

1. The blackboost estimator in the R package mboost [10] provides a tree boosting procedure for fitting the log-normal hazard function. In order to apply this to the simulated data, we make ward census a time-static covariate by fixing it at its initial value.

2. Transformation forests [17] in the R package trtf can also fit log-normal hazards. Moreover, it allows for left-truncated and right-censored data. Since the ward census variable is simulated to be piecewise constant over time, we can treat each segment as a left-truncated and right-censored observation. Thus for this simulation, transformation forests are able to handle time-dependent covariates with time-static effects. This falls in between the static covariate/static effect blackboost estimator and our fully nonparametric one.

Since the service rate model used in the simulations is in fact log-normal, the benchmark methods above enjoy a significant advantage over our nonparametric one, which is not privy to the true distribution. In fact, when a = 0 the log-normal hazard (37) depends only on time-static covariates, so the benchmarks should outperform our nonparametric estimator. However, as a grows, we would expect a reversal in relative performance.

To compare the performances of the estimators, we use Monte Carlo integration to evaluate the relative mean squared error

$$\%\text{MSE}=\frac{{\mathbb{E}}_X\left[{\int }_0^1\{\lambda (t,X)-\widehat{\lambda }(t,X){\}}^{2}dt\right]}{{\mathbb{E}}_X\left[{\int }_0^1\lambda {(t,X)}^{2}dt\right]}.$$

The Monte Carlo integrations are conducted using an independent test set of 10,000 uncensored patient visit histories. For the test set, ward census is held fixed over time at the initial value, and we use the grid {0, 0.02, 0.04, ⋯, 1} for the time integral. The nominator above is then estimated by the average of ${\{\lambda (t,x)-\widehat{\lambda }(t,x)\}}^2$ evaluated at the 51×10,000 points of (t, x). The denominator is estimated in the same manner.

5.4. Results.

For the implementation of our estimator in Section 4, the value of ε and the number of trees $\widehat{m}$ are jointly determined using ten-fold cross validation. The candidate values we tried for ε are {0.003, 0.004, 0.005, 0.006, 0.007}, and we limit $\widehat{m}$ to no more than 1,000 trees. A wider range of values can be of course be explored for better performance (at the cost of more computations). As comparison, we run an ad-hoc version of our algorithm in which all trees use the same number of splits, as is the case in standard boosting [13]. This approach does not explicitly ensure that the trees will be ε-gradients for a pre-specified ε. The number of splits and the number of trees used in the ad-hoc method are jointly determined using ten-fold cross-validation.

In order to speed up convergence at the m-th iteration for both approaches, instead of using the step-size ν_n/(m+1) of Algorithm 1, we performed line-search within the interval (0, ν_n/(m+1)]. While Lemma 2 shows that a smaller shrinkage ν_n is always better, this comes at the expense of a larger $\widehat{m}$ and hence computation time. For simplicity we set ν_n = 1 for all the experiments here.

For fitting the blackboost estimator, we use the default setting of nu = 0.1 for the step-size taken at each iteration. The other hyperparameters, mstop (the number of trees) and maxdepth (maximum depth of trees), are chosen to directly minimize the relative MSE on the test set. This of course gives the blackboost estimator an unfair advantage over our estimator, which is on top of the fact that it is based on the same distribution as the true model. Transformation forest (using also the true distribution) is fit using code kindly provided by Professor T. Hothorn.¹²

Variable selection.

The relative importances of variables [13] for our estimator are given in Table 1 for all four cases a = 0, 1, 2, 3. The four factors that influence the service rate (38) are explicitly listed, while the irrelevant covariates are grouped together in the last column. When a = 0, the service rate does not depend on census, and we see that the importance of census and the other irrelevant covariates are at least an order of magnitude smaller than the relevant ones. As a increases, census becomes more and more important as correctly reflected in the table. Across all the cases the importance of the relevant covariates are at least an order of magnitude larger than the others, suggesting that our estimator is able to pick out the influential covariates and largely avoid the irrelevant ones.

Table 1.

Relative importances of variables in the boosted nonparametric estimator. The numbers are scaled so that the largest value in each row is 1.

a	Time	Age	ESI	Census	All other variables
0	1	0.21	0.025	0.0011	<0.0010
1	1	0.22	0.013	0.46	<0.0003
2	0.34	0.064	0.0020	1	0
3	0.11	0.011	<0.0001	1	0

Presence of time-dependent covariates.

Table 2 presents the relative MSEs for the estimators as the service rate function (38) becomes increasingly dependent on the time-varying census variable. When a = 0 the service rate depends only on time-static covariates, so as expected, the parametric log-normal benchmarks perform the best when applied to data simulated from a log-normal AFT model.

Table 2.

Comparative performances (%MSE) as the service rate (38) becomes increasingly dependent on the time-varying ward census variable (by increasing a).

a	blackboost (set to true log-normal distribution)	Transformation forest (set to true log-normal distribution)	Boosted hazards (ε fixed for all iterations)	Ad-hoc (# splits fixed for all iterations)
0	5.0%	5.0%	7.8%	7.1%
1	17%	6.1%	4.5%	8.1%
2	46%	9.7%	5.4%	7.0%
3	67%	18%	7.2%	7.4%

However, as a increases, the service rate becomes increasingly dependent on census. The corresponding performances of both benchmarks deteriorate dramatically, and is handily outperformed by the proposed estimator. We note that the inclusion of just one time-dependent covariate is enough to degrade the performances of the benchmarks, despite the fact that they have the exact same parametric form as the true model.

Finally we find comparable performance among the ad-hoc boosted estimator and our proposed one, although a slight edge goes to the latter especially in the more difficult simulations with larger a. The results here demonstrate that there is a place in the survival boosting literature for fully nonparametric methods like this one that can flexibly handle time-dependent covariates.

6. Discussion.

Our estimator can also potentially be used to evaluate the goodness-of-fit of simpler parametric hazard models. Since our approach is likelihood-based, future work might examine whether model selection frameworks like those in [30] can be extended to cover likelihood functionals. For this, [10] provides some guidance for determining the effective degrees of freedom for the boosting estimator. The ideas in [32] may also be germane.

The implementation presented in Section 4 is one of many possible ways to implement our estimator. We defer the design of a more refined implementation to future research, along with open-source code.

APPENDIX: PROOFS

Proof of Proposition 2.

Proof. Writing

$$R(F)=\mathbb{E}\left({\int }_0^{1}Y(t)\cdot e^{F(t,X(t))}dt-\Delta F(T,X(T))\right),$$

we can apply (4) to establish the first part of the integral in (13) when $F\in \mathcal{F}\cup \left\{\text{log }\lambda \right\}$. To complete the representation, it suffices to show that the point process

$$M(B)=\Delta \cdot I[\{T,X(T)\}\in B]$$

has mean ${\int }_B\lambda d\mu$, and then apply Campbell’s formula. To this end, write N(t) = I(T ≤ t) and consider the filtration σ{X(s), Y(s),N(s) : s ≤ t}. Then N(t) has the Doob-Meyer form dN(t) = λ(t, X(t))Y(t)dt + dM(t) where M(t) is a martingale. Hence

$$\begin{gathered} \mathbb{E}\{M(B)\}=\mathbb{E}\left({\int }_0^{1}I[\{t,X(t)\}\in B]dN(t)\right) \\ =\mathbb{E}\left({\int }_0^{1}Y(t)\cdot I[\{t,X(t)\}\in B]\cdot \lambda (t,X(t))dt\right)+\mathbb{E}\left({\int }_0^{1}I[\{t,X(t)\}\in B]dM(t)\right) \\ ={\int }_B\lambda d\mu +\mathbb{E}\left({\int }_0^{1}I[\{t,X(t)\}\in B]dM(t)\right), \end{gathered}$$

where the last equality follows from (4). Since I[{t, X(t)} ∈ B] is predictable because X(t) is, the desired result follows if the stochastic integral ${\int }_0^{1}I[\{t,X(t)\}\in B]dM(t)$ is a martingale. By Section 2 of Aalen [1], this is true if M(t) is square-integrable. In fact, $M(t)=N(t)-{\int }_0^t\lambda (t,X(t))dt$ is bounded because λ(t, x) is bounded above by (A1). This establishes (13).

Now note that for a positive constant Λ the function e^y − Λy is bounded below by both −Λy and Λy + 2Λ{1 – log 2Λ}, hence e^y − Λy ≥ Λ|y| + 2Λmin{0, 1 – log 2Λ}. Since Λ min{0, 1 − log2Λ} is non-increasing in Λ, (A1) implies that

$$\begin{gathered} e^{F(t,x)}-\lambda (t,x)F(t,x)\ge \text{min}\left\{e^{F(t,x)}-{\Lambda }_{L}F(t,x),e^{F(t,x)}-{\Lambda }_{U}F(t,x)\right\} \\ \ge {\Lambda }_L|F(t,x)|+2{\Lambda }_U\text{min}\left\{0,1-\text{log }\left(2{\Lambda }_U\right)\right\}. \end{gathered}$$

Integrating both sides and using the norm equivalence relation (27) shows that

$$\begin{gathered} R(F)\ge {\Lambda }_L\Vert F{\Vert }_{\mu ,1}+2{\Lambda }_U\text{min}\left\{0,1-\text{log }\left(2{\Lambda }_U\right)\right\} \\ \ge \frac{2{\Lambda }_L}{{\alpha }_{\mathcal{F}}}\Vert F{\Vert }_{\infty }+2{\Lambda }_U\text{min}\left\{0,1-\text{log }\left(2{\Lambda }_U\right)\right\} \\ \ge \frac{2{\Lambda }_L}{{\alpha }_{\mathcal{F}}}\Vert F{\Vert }_{\mu ,2}+2{\Lambda }_U\text{min}\left\{0,1-\text{log }\left(2{\Lambda }_U\right)\right\}. \end{gathered}$$

The lower bound (14) then follows from the second inequality. The last inequality shows that R(F) is coercive on $\left(\mathcal{F},{\langle \cdot ,\cdot \rangle }_{\mu }\right)$. Moreover the same argument used to derive (8) shows that R(F) is smooth and convex on $\left(\mathcal{F},{\langle \cdot ,\cdot \rangle }_{\mu }\right)$. Therefore a unique minimizer F* of R(F) exists in $\left(\mathcal{F},{\langle \cdot ,\cdot \rangle }_{\mu }\right)$. Since (A2) implies there is a bijection between the equivalent classes of $\left(\mathcal{F},{\langle \cdot ,\cdot \rangle }_{\mu }\right)$ and the functions in $\mathcal{F}$, F* is also the unique minimizer of R(F) in $\mathcal{F}$. Finally, since e^{F(t, x)} − λ(t, x)F(t, x) is pointwise bounded below by $\lambda (t,x)\{1-\text{log }\lambda (t,x)\}$, $R(F)\ge \int (\lambda -\lambda \text{log }\lambda )d\mu =R(\text{log }\lambda )$ for all $F\in \mathcal{F}$. □

Proof of Lemma 1.

Proof. By a pointwise-measurable argument (Example 2.3.4 of [29]) it can be shown that all suprema quantities appearing below are sufficiently well behaved, so outer integration is not required. Define the Orlicz norm $\Vert X{\Vert }_{\Phi }=\text{inf}\{C>0:\mathbb{E}\Phi (|X|/C)\le 1\}$ where $\Phi (x)=e^{x^2}-1$. Suppose the following holds:

$${\Vert \underset{F\in {\mathcal{F}}_{\epsilon }^{{\Psi }_n}}{\text{sup}}\left|\left\{R_n(F)-R_n(0)\right\}-\{R(F)-R(0)\}\right|\Vert }_{\Phi }\le {\kappa }^{\prime }{J}_{{\mathcal{F}}_{\epsilon }}/n^{1/4},$$ (39)

$${\Vert \underset{G\in {\mathcal{F}}_{\epsilon }:\Vert G{\Vert }_{\infty }\le 1}{\text{sup}}\left|\Vert G{\Vert }_{{\mu }_n,1}-\Vert G{\Vert }_{\mu ,1}\right|\Vert }_{\Phi }\le {\kappa }^{\prime \prime }{J}_{{\mathcal{F}}_{\epsilon }}/n^{1/2},$$ (40)

where $J_{{\mathcal{F}}_{\epsilon }}$ is the complexity measure (26), and κ′,κ″ are universal constants. Then by Markov’s inequality, (30) holds with probability at least $1-2\text{exp}\left[-{\left\{\eta n^{1/4}/\left({\kappa }^{\prime }{J}_{{\mathcal{F}}_{\epsilon }}\right)\right\}}^2\right]$, and

$$\underset{G\in {\mathcal{F}}_{\epsilon }:\Vert G{\Vert }_{\infty }\le 1}{\text{sup}}\left\{\Vert G{\Vert }_{\mu ,1}-\Vert G{\Vert }_{{\mu }_n,1}\right\}<1/{\alpha }_{\mathcal{F}}$$ (41)

holds with probability at least $1-2\text{exp}\left[-{\left\{n^{1/2}/\left({\alpha }_{\mathcal{F}}{\kappa }^{\prime \prime }{J}_{{\mathcal{F}}_{\epsilon }}\right)\right\}}^2\right]$. Since ${\alpha }_{\mathcal{F}}>1$ and η < 1, (30) and (41) jointly hold with probability at least $1-4\text{exp}\left[-{\left\{\eta n^{174}/\left(\kappa {\alpha }_{\mathcal{F}}{J}_{{\mathcal{F}}_{\epsilon }}\right)\right\}}^2\right]$. The lemma then follows if (41) implies (29). Indeed, for any non-zero $F\in \mathcal{F}$, its normalization G = F/‖F‖_∞ is in ${\mathcal{F}}_{\epsilon }$ by construction (25). Then (41) implies that

$$\frac{\Vert F{\Vert }_{\infty }}{\Vert F{\Vert }_{{\mu }_n,1}}=1/\Vert G{\Vert }_{{\mu }_n,1}\le {\alpha }_{\mathcal{F}}$$

because

$$1/{\alpha }_{\mathcal{F}}>\Vert G{\Vert }_{\mu ,1}-\Vert G{\Vert }_{{\mu }_n,1}\ge 2/{\alpha }_{\mathcal{F}}-\Vert G{\Vert }_{{\mu }_n,1},$$

where the last inequality follows from the definition of ${\alpha }_{\mathcal{F}}$ (28).

Thus it remains to establish (39) and (40), which can be done by applying the symmetrization and maximal inequality results in Sections 2.2 and 2.3.2 of [29]. Write $R_n(F)=(1/n){\sum }_{i=1}^n{l}_i(F)$ where $l_i(F)={\int }_0^1{Y}_i(t)e^{F\left(t,X_i(t)\right)}dt-{\Delta }_{i}F\left(T_i,X_i\left(T_i\right)\right)$ are independent copies of the loss

$$l(F)={\int }_0^{1}Y(t)\cdot e^{F(t,X(t))}dt-\Delta \cdot F(T,X(T)),$$ (42)

which is a stochastic process indexed by $F\in \mathcal{F}$. As was shown in Proposition 2, $\mathbb{E}\{l(F)\}=R(F)$. Let ζ₁, ⋯ , ζ_N be independent Rademacher random variables that are independent of $Z={\left\{\left(X_i(\cdot ),Y_i(\cdot ),T_i\right)\right\}}_{i=1}^n$. It follows from the symmetrization Lemma 2.3.6 of [29] for stochastic processes that the left hand side of (39) is bounded by twice the Orlicz norm of

$$\begin{gathered} \underset{F\in {\mathcal{F}}_{\epsilon }^{{\Psi }_n}}{\text{sup}}\mid \frac{1}{n}\sum _{i=1}^n{\zeta }_i\left\{l_i(F)-l_i(0)\right\} \\ \le \frac{1}{n}\underset{F\in {\mathcal{F}}_{\epsilon }^{\Psi n}}{\text{sup}}\left|\sum _{i=1}^n{\zeta }_i{\int }_0^1{Y}_i(t)\left\{e^{F\left(t,X_i(t)\right)}-1\right\}dt\right|+\frac{1}{n}\underset{F\in {\mathcal{F}}_{\text{E}}^{\Psi n}}{\text{sup}}\left|\sum _{i=1}^n{\zeta }_i{\Delta }_{i}F\left(T_i,X_i\left(T_i\right)\right)\right|. \end{gathered}$$ (43)

Now hold Z fixed so that only ζ₁, ⋯ , ζ_n are stochastic, in which case the sum in the second line of (43) becomes a separable subgaussian process. Since the Orlicz norm of ${\sum }_{i=1}^n{\zeta }_i{a}_i$ is bounded by ${\left(6{\sum }_{i=1}^n{a}_i^2\right)}^{1/2}$ for any constant a_i, we obtain the following the Lipschitz property for any F₁, $F_2\in {\mathcal{F}}_{\epsilon }^{{\Psi }_n}$:

$$\begin{gathered} {\Vert \sum _{i=1}^n{\zeta }_i{\int }_0^1{Y}_i(t)\left\{e^{F_1\left(t,X_i(t)\right)}-e^{F_2\left(t,X_i(t)\right)}\right\}dt\Vert }_{\Phi ,\zeta \mid Z}^2 \\ \le 6\sum _{i=1}^n{\left[{\int }_0^1{Y}_i(t)\left\{e^{F_1\left(t,X_i(t)\right)}-e^{F_2\left(t,X_i(t)\right)}\right\}dt\right]}^2 \\ \le 6e^{2{\Psi }_n}\sum _{i=1}^n{\left({\int }_0^1{Y}_i(t)\cdot \left|F_1\left(t,X_i(t)\right)-F_2\left(t,X_i(t)\right)\right|dt\right)}^2 \\ \le 6e^{2{\Psi }_n}\sum _{i=1}^n{\int }_0^1{Y}_i(t){\left\{F_1\left(t,X_i(t)\right)-F_2\left(t,X_i(t)\right)\right\}}^{2}dt \\ =6n{e}^{2{\Psi }_n}{\Vert F_1-F_2\Vert }_{{\mu }_n,2}^2, \end{gathered}$$

where the second inequality follows from |e^x − e^y| ≤ e^{max(x, y)}|x−y| and the last from the Cauchy-Schwarz inequality. Putting the Lipschitz constant ${(6n)}^{1/2}{e}^{{\Psi }_n}$ obtained above into Theorem 2.2.4 of [29] yields the following maximal inequality: There is a universal constant κ′ such that

$$\begin{gathered} {\Vert \underset{F\in {\mathcal{F}}_{\epsilon }^{{\Psi }_n}}{\text{sup}}\left|\sum _{i=1}^n{\zeta }_i{\int }_0^1{Y}_i(t)\left\{e^{F\left(t,X_i(t)\right)}-1\right\}dt\right|\Vert }_{\Phi ,\zeta \mid Z} \\ \le {\kappa }^{\prime }{n}^{1/2}{e}^{{\Psi }_n}{\int }_0^{{\Psi }_n}{\left\{\text{log }\mathcal{N}\left(u,{\mathcal{F}}_{\epsilon }^{{\Psi }_n},{\mu }_n\right)\right\}}^{1/2}du \\ \le {\kappa }^{\prime }{n}^{1/2}{e}^{{\Psi }_n}{\Psi }_n{J}_{{\mathcal{F}}_{\epsilon }}, \end{gathered}$$

where the last line follows from (26). Likewise the conditional Orlicz norm for the supremum of $\left|\sum _{i=1}^n{\zeta }_i{\Delta }_{i}F\left(T_i,X_i\left(T_i\right)\right)\right|$ is bounded by ${\kappa }^{\prime }{J}_{{\mathcal{F}}_{\epsilon }}{n}^{1/2}{\Psi }_n$. Since neither bounds depend on Z, plugging back into (43) establishes (39):

$$\begin{gathered} {\Vert \underset{F\in {\mathcal{F}}_{\epsilon }^{{\Psi }_n}}{\text{sup}}\left|\left\{R_n(F)-R_n(0)\right\}-\{R(F)-R(0)\}\right|\Vert }_{\Phi } \\ \le 2{\kappa }^{\prime }{J}_{{\mathcal{F}}_{\epsilon }}\frac{{\Psi }_n{e}^{{\Psi }_n}}{n^{1/2}}\left\{1+e^{-{\Psi }_n}\right\} \\ \le 4{\kappa }^{\prime }\frac{J_{{\mathcal{F}}_{\epsilon }}}{n^{1/4}}, \end{gathered}$$

where ${\Psi }_n{e}^{{\Psi }_n}=n^{1/4}$ by (19). On noting that

$$\Vert G{\Vert }_{{\mu }_n,1}=\frac{1}{n}\sum _{i=1}^n{\int }_0^1{Y}_i(t)\left|G\left(t,X_i(t)\right)\right|dt,\Vert G{\Vert }_{\mu ,1}=\mathbb{E}\left\{{\int }_0^{1}Y(t)|G(t,X(t))|dt\right\},$$

(40) can be established using the same approach. □

Proof of Lemma 2.

Proof. For $m<\widehat{m}$, applying (8) to $R_n\left(F_{m+1}\right)=R_n\left(F_m-\frac{{\nu }_n}{m+1}{g}_{F_m}^{\epsilon }\right)$ yields

$$\begin{gathered} R_n\left(F_{m+1}\right)=R_n\left(F_m\right)-\frac{{\nu }_n}{m+1}{\langle g_{F_m},g_{F_m}^{\epsilon }\rangle }_{{\mu }_n}+\frac{{\nu }_n^2}{2{(m+1)}^2}\int {\left(g_{F_m}^{\epsilon }\right)}^2\text{exp}\left\{F_m-\rho \left(F_{m+1}-F_m\right)\right\}d{\mu }_n \\ <R_n\left(F_m\right)-\frac{\epsilon {\nu }_n}{m+1}{\Vert g_{F_m}\Vert }_{{\mu }_n,2}+\frac{{\nu }_n^2{e}^{{\Psi }_n}}{2{(m+1)}^2}, \end{gathered}$$ (44)

where the bound for the second term is due to (18) and the bound for the integral follows from $\int {\left(g_{F_m}^{\epsilon }\right)}^{2}d{\mu }_n=1$ (Definition 1 of an ε-gradient) and ‖F_m‖_∞, ‖F_m+1‖_∞ < Ψ_n for $m<\widehat{m}$ (lines 5–6 of Algorithm 1). Hence for $m\le \widehat{m}$, (44) implies that

$$R_n\left(F_m\right)<R_n(0)+\sum _{m=0}^{\infty }\frac{{\nu }_n^2{e}^{{\Psi }_n}}{2{(m+1)}^2}<R_n(0)+1\le 2$$

because ${\nu }_n^2{e}^{{\Psi }_n}<1$ under (20). Since ${\text{max}}_{m\le \widehat{m}}{\Vert F_m\Vert }_{\infty }<{\Psi }_n$, and using our assumption ${\text{sup}}_{F\in {\mathcal{F}}_{\epsilon }^{{\Psi }_n}}\left|R_n\left(F\right)-R\left(F\right)\right|<1$ in the statement of the lemma, we have

$$R\left(F_m\right)\le R_n\left(F_m\right)+\left|R_n\left(F_m\right)-R\left(F_m\right)\right|<3.$$

Clearly the minimizer F* also satisfies R(F*) ≤ R(0) < 3. Thus coercivity (15) implies that

$${\Vert F_m\Vert }_{\infty },{\Vert F^{*}\Vert }_{\infty }<{\alpha }_{\mathcal{F}}{\beta }_{\Lambda },$$

so the gap $\widehat{\gamma }$ defined in (31) is bounded as claimed.

It remains to establish (32), for which we need only consider the case $R_n\left(F_{\widehat{m}}\right)-R_n\left(F^{*}\right)>0$. The termination criterion gF_m = 0 in Algorithm 1 is never triggered under this scenario, because by Proposition 1 this would imply that $F_{\widehat{m}}$ minimizes R_n(F) over the span of {φ_nj(t, x)}_j, which also contains F* (Remark 3). Thus either $\widehat{m}=\infty$, or the termination criterion ${\Vert F_{\widehat{m}}-\frac{{\nu }_n}{\widehat{m}+1}{\widehat{g}}_{F_{\widehat{m}}}^{\epsilon }\Vert }_{\infty }\ge {\Psi }_n$ in line 5 of Algorithm 1 is met. In the latter case

$$\begin{gathered} {\Psi }_n\le {\Vert F_{\widehat{m}}-\frac{{\nu }_n}{\widehat{m}+1}{g}_{F_m}^{\epsilon }\Vert }_{\infty }\le {\alpha }_{\mathcal{F}}{\Vert F_{\widehat{m}}-\frac{{\nu }_n}{\widehat{m}+1}{g}_{F_m}^{\epsilon }\Vert }_{{\mu }_n,2} \\ \le {\alpha }_{\mathcal{F}}\left(\sum _{m=0}^{\widehat{m}-1}\frac{{\nu }_n}{m+1}+1\right) \end{gathered}$$ (45)

where the inequalities follow from (29) and from ${\Vert g_{F_m}^{\epsilon }\Vert }_{{\mu }_n,2}=1$. Since the sum is diverging, the inequality also holds for $\widehat{m}$ sufficiently large (e.g. $\widehat{m}=\infty$).

Given that F* lies in the span of {φ_nj(t, x)}_j, the Taylor expansion (8) is valid for R_n(F*). Since the remainder term in the expansion is non-negative, we have

$$\begin{gathered} R_n\left(F^{*}\right)=R_n\left(F_m+F^{*}-F_m\right) \\ \ge R_n\left(F_m\right)+{\langle g_{F_m},F^{*}-F_m\rangle }_{{\mu }_n}. \end{gathered}$$

Furthermore for $m\le \widehat{m}$,

$$\begin{gathered} {\langle g_{F_m},F_m-F^{*}\rangle }_{{\mu }_n}\le {\Vert F_m-F^{*}\Vert }_{{\mu }_n,2}\cdot {\Vert g_{F_m}\Vert }_{{\mu }_n,2} \\ \le {\Vert F_m-F^{*}\Vert }_{\infty }\cdot {\Vert g_{F_m}\Vert }_{{\mu }_n,2} \\ \le \widehat{\gamma }\cdot {\Vert g_{F_m}\Vert }_{{\mu }_n,2}. \end{gathered}$$

Putting both into (44) gives

$$\begin{gathered} R_n\left(F_{m+1}\right)<R_n\left(F_m\right)+\frac{\epsilon {\nu }_n}{\widehat{\gamma }(m+1)}{\langle g_{F_m},F^{*}-F_m\rangle }_{{\mu }_n}+\frac{{\nu }_n^2{e}^{{\Psi }_n}}{2{(m+1)}^2} \\ \le R_n\left(F_m\right)+\frac{\epsilon {\nu }_n}{\widehat{\gamma }(m+1)}\left\{R_n\left(F^{*}\right)-R_n\left(F_m\right)\right\}+\frac{{\nu }_n^2{e}^{{\Psi }_n}}{2{(m+1)}^2}. \end{gathered}$$

Subtracting R_n(F*) from both sides above and denoting δ_m = R_n(F_m) − R_n(F*), we obtain

$${\delta }_{m+1}<\left(1-\frac{\epsilon {\nu }_n}{\widehat{\gamma }(m+1)}\right){\delta }_m+\frac{{\nu }_n^2{e}^{{\Psi }_n}}{2{(m+1)}^2}.$$

Since the term inside the first parenthesis is between 0 and 1, solving the recurrence yields

$$\begin{gathered} {\delta }_{\widehat{m}}<{\delta }_0\prod _{m=0}^{\widehat{m}-1}\left(1-\frac{\epsilon {\nu }_n}{\widehat{\gamma }(m+1)}\right)+{\nu }_n^2{e}^{{\Psi }_n}\sum _{m=0}^{\infty }\frac{1}{2{(m+1)}^2} \\ \le \text{max}\left\{0,{\delta }_0\right\}\text{exp}\left(-\frac{\epsilon }{\widehat{\gamma }}\sum _{m=0}^{\widehat{m}-1}\frac{{\nu }_n}{m+1}\right)+{\nu }_n^2{e}^{{\Psi }_n} \\ \le e\text{max}\left\{0,{\delta }_0\right\}\text{exp}\left(-\frac{\epsilon }{{\alpha }_{\mathcal{F}}\widehat{\gamma }}{\Psi }_n\right)+{\nu }_n^2{e}^{{\Psi }_n}, \end{gathered}$$

where in the second inequality we used the fact that 0 ≤ 1 + y ≤ e^y for |y| < 1, and the last line follows from (45).

The Lambert function (19) in Ψ_n = W(n^1/4) is asymptotically log y −log log y, and in fact by Theorem 2.1 of [14], W(y) ≥ log y −log log y for y ≥ e. Since by assumption n ≥ 55 > e⁴, the above becomes

$${\delta }_{\widehat{m}}<e\text{max}\left\{0,{\delta }_0\right\}{\left(\frac{\text{log }n}{4n^{1/4}}\right)}^{\epsilon /\left({\alpha }_{\mathcal{F}}\widehat{\gamma }\right)}+{\nu }_n^2{e}^{{\Psi }_n}.$$

The last step is to control δ₀, which is bounded by 1 − R_n(F*) because R_n(F₀) = R_n(0) ≤ 1. Then under the hypothesis |R_n(F*) − R(F*)| < 1, we have

$${\delta }_0\le 1-R\left(F^{*}\right)+1<2-R\left(F^{*}\right).$$

Since (14) implies R(F*) ≥ 2Λ_U min{0, 1 − log(2Λ_U)},

$${\delta }_0<2-R\left(F^{*}\right)\le 2+2{\Lambda }_U\text{max}\left\{0,\text{log }\left(2{\Lambda }_U\right)-1\right\}<2{\beta }_{\Lambda }.$$

□

Proof of Proposition 3.

Proof. Let δ = log n/(4n^1/4) which is less than one for n ≥ 55 > e⁴. Since ${\alpha }_{\mathcal{F}}$, $\widehat{\gamma }\ge 1$ it follows that

$$\delta <{\left(\frac{\text{log }n}{4n^{1/4}}\right)}^{\epsilon /\left({\alpha }_{\mathcal{F}}\widehat{\gamma }\right)}.$$ (46)

Now define the following probability sets

$$S_1=\left\{\underset{F\in {\mathcal{F}}_{\epsilon }^{\Psi n}}{\text{sup}}\left|\left\{R_n(F)-R_n(0)\right\}-\{R(F)-R(0)\}\right|<\delta /3\right\}$$

$$S_2=\left\{\left|R_n(0)-R(0)\right|<\delta /3\right\}$$

$$S_3=\left\{\left|R_n\left(F^{*}\right)-R\left(F^{*}\right)\right|<\delta /3\right\}$$

$$S_4=\{(29)\text{ holds }\},$$

and fix a sample realization from ${\cap }_{k=1}^4{S}_k$. Then the conditions required in Lemma 2 are satisfied with ${\text{sup}}_{F\in {\mathcal{F}}_{\epsilon }^{{\Psi }_n}}\left|R_n(F)-R(F)\right|<2\delta /3$, so $\widehat{\gamma }$ (and hence ${\Vert {\widehat{\lambda }}_{\text{boost}}\Vert }_{\infty }$) is bounded and (32) holds. Since Algorithm 1 ensures that ${\Vert F_{\widehat{m}}\Vert }_{\infty }<{\Psi }_n$, we have $F_{\widehat{m}}\in {\mathcal{F}}_{\epsilon }^{{\Psi }_n}$ and therefore it also follows that $\left|R_n\left(F_{\widehat{m}}\right)-R\left(F_{\widehat{m}}\right)\right|<2\delta /3$. Combining (23) and (24) gives

$$\begin{gathered} {\Vert F_{\widehat{m}}-F^{*}\Vert }_{\mu ,2}^2\le \frac{2}{{\text{min}}_{t,x}\left({\lambda }^{*}\wedge {\widehat{\lambda }}_{\text{boost}}\right)}\left(\frac{2\delta }{3}+\frac{\delta }{3}+\left\{R_n\left(F_{\widehat{m}}\right)-R_n\left(F^{*}\right)\right\}\right) \\ <\frac{2}{{\text{min}}_{t,x}\left({\lambda }^{*}\wedge {\widehat{\lambda }}_{\text{boost}}\right)}\left(\delta +2e{\beta }_{\Lambda }{\left(\frac{\text{log }n}{4n^{1/4}}\right)}^{\epsilon /\left({\alpha }_{\mathcal{F}}\widehat{\gamma }\right)}+\frac{1}{16}\cdot \frac{\text{log }n}{4n^{1/4}}\right) \\ <\frac{13{\beta }_{\Lambda }}{{\text{min}}_{t,x}\left({\lambda }^{*}\wedge {\widehat{\lambda }}_{\text{boost}}\right)}{\left(\frac{\text{log }n}{4n^{1/4}}\right)}^{\epsilon /\left({\alpha }_{\mathcal{F}}\widehat{\gamma }\right)}, \end{gathered}$$

where the second inequality follows from (32) and ${\nu }_n^2{e}^{{\Psi }_n}=\text{log }n/\left(64n^{1/4}\right)$, and the last from (46). Now, using the inequality |e^x − e^y| ≤ max(e^x, e^y)|x − y| yields

$${\Vert {\widehat{\lambda }}_{\text{boost}}-{\lambda }^{*}\Vert }_{\mu ,2}^2<13{\beta }_{\Lambda }\frac{{\text{max}}_{t,x}{\left({\lambda }^{*}\vee {\widehat{\lambda }}_{\text{boost}}\right)}^2}{{\text{min}}_{t,x}\left({\lambda }^{*}\wedge {\widehat{\lambda }}_{\text{boost}}\right)}{\left(\frac{\text{log }n}{4n^{1/4}}\right)}^{\epsilon /\left({\alpha }_{\mathcal{F}}\widehat{\gamma }\right)},$$

and the stated bound follows from F* = log λ since $\mathcal{F}$ is correctly specified (Proposition 2).

The next task is to lower bound $\mathbb{P}\left({\cap }_{k=1}^4{S}_k\right)$. It follows from Lemma 1 that

$$\mathbb{P}\left(S_1\cap S_4\right)\ge 1-4\text{exp}\left\{-{\left(\frac{\text{log }n}{12\kappa {\alpha }_{\mathcal{F}}{J}_{{\mathcal{F}}_{\epsilon }}}\right)}^2\right\}.$$

Bounds on $\mathbb{P}\left(S_2\right)$ and $\mathbb{P}\left(S_3\right)$ can be obtained using Hoeffding’s inequality. Note from (2) that $R_n\left(0\right)={\sum }_{i=1}^n{\int }_0^1{Y}_i(t)dt/n$ and $R_n\left(F^{*}\right)={\sum }_{i=1}^n{l}_i\left(F^{*}\right)/n$ for the loss l(·) defined in (42). Since $0\le {\int }_0^1{Y}_i(t)dt\le 1$ and $-{\Vert F^{*}\Vert }_{\infty }<l\left(F^{*}\right)\le {\Vert e^{F^{*}}\Vert }_{\infty }+{\Vert F^{*}\Vert }_{\infty }$,

$$\mathbb{P}\left(S_2\right)\ge 1-2\text{exp}\left\{-2n^{1/2}{\left(\frac{\text{log }n}{12}\right)}^2\right\},\mathbb{P}\left(S_3\right)\ge 1-2\text{exp}\left\{-2n^{1/2}{\left(\frac{\text{log }n}{36e^{\Vert F^{*}{\Vert }_{\infty }}}\right)}^2\right\}.$$

By increasing the value of κ and/or replacing $J_{{\mathcal{F}}_{\epsilon }}$ with $\text{max}\left(1,J_{{\mathcal{F}}_{\epsilon }}\right)$ if necessary, we can combine the inequalities to get a crude but compact bound:

$$\mathbb{P}\left\{{\cap }_{k=1}^4{S}_k\right\}\ge 1-8\text{exp}\left\{-{\left(\frac{\text{log }n}{\kappa {\alpha }_{\mathcal{F}}{J}_{{\mathcal{F}}_{\epsilon }}{e}^{{\Vert F^{*}\Vert }_{\infty }}}\right)}^2\right\}.$$ (47)

Finally, since ‖F*‖_∞ = ‖log λ‖_∞ < max{|log Λ_L|, |log Λ_U|}, we can replace $e^{{\Vert F^{*}\Vert }_{\infty }}$ in the probability bound above by ${\Lambda }_L^{-1}\vee {\Lambda }_U$. □

Proof of Proposition 4.

Proof. It follows from (22) that λ* is the orthogonal projection of λ onto $\left(\mathcal{F},{\langle \cdot ,\cdot \rangle }_{\mu }\right)$. Hence

$$\begin{gathered} {\Vert {\widehat{\lambda }}_{\text{boost}}-\lambda \Vert }_{\mu ,2}^2={\Vert e^{F^{*}}-\lambda \Vert }_{\mu ,2}^2+{\Vert e^{F_m}-e^{F^{*}}\Vert }_{\mu ,2}^2 \\ =\underset{F\in \mathcal{F}}{\text{min}}{\Vert e^F-\lambda \Vert }_{\mu ,2}^2+{\Vert e^{F_m}-e^{F^{*}}\Vert }_{\mu ,2}^2 \\ \le \underset{F\in \mathcal{F}}{\text{min}}{\Vert e^F-\lambda \Vert }_{\mu ,2}^2+\underset{t,x}{\text{max}}{\left({\lambda }^{*}\vee {\widehat{\lambda }}_{\text{boost}}\right)}^2{\Vert F_{\widehat{m}}-F^{*}\Vert }_{\mu ,2}^2, \end{gathered}$$

where the inequality follows from |e^x − e^y| ≤ max(e^x, e^y)|x − y|. Bounding the last term in the same way as Proposition 3 completes the proof. To replace $e^{{\Vert F^{*}\Vert }_{\infty }}$ in (47) by ${\Lambda }_L^{-1}\vee {\Lambda }_U$, it suffices to show that Λ_L ≤ λ*(t, x) ≤ Λ_U. Since the value of λ* over one of its piecewise constant regions B is ${\int }_B\lambda d\mu /\mu (B)$, the desired bound follows from (A1). We can also replace ${\text{max}}_{t,x}\left({\lambda }^{*}\vee {\widehat{\lambda }}_{\text{boost}}\right)$ and ${\text{min}}_{t,x}\left({\lambda }^{*}\wedge {\widehat{\lambda }}_{\text{boost}}\right)$ with ${\text{max}}_{t,x}\left({\Lambda }_U\vee {\widehat{\lambda }}_{\text{boost}}\right)$ and ${\text{min}}_{t,x}\left({\Lambda }_L\wedge {\widehat{\lambda }}_{\text{boost}}\right)$ respectively. □