Parameter Estimation for Semiparametric Ordinary Differential Equation Models

Abstract

We propose a new class of two-stage parameter estimation methods for semiparametric ordinary differential equation (ODE) models. In the first stage, state variables are estimated using a penalized spline approach; In the second stage, form of numerical discretization algorithms for an ODE solver is used to formulate estimating equations. Estimated state variables from the first stage are used to obtain more data points for the second stage. Asymptotic properties for the proposed estimators are established. Simulation studies show that the method performs well, especially for small sample. Real life use of the method is illustrated using Influenza specific cell-trafficking study.

1. Introduction

We consider a semiparametric coefficient ODE model, which can be written as

$$\{\begin{array}{l}\frac{d}{dt}\mathit{\text{X}}\left(t\right)=\mathit{\text{F}}\left\{t,\mathit{\text{X}}\left(t\right),\mathit{\text{Z}}\left(t\right),\beta ,\mathit{\eta }\left(t\right)\right\},\hspace{1em}\forall t\hspace{0.17em}\in \hspace{0.17em}\left[t_0,T\right],\hfill \\ \mathit{\text{X}}\left(t_0\right)={\mathit{\text{X}}}_0,\hfill \end{array}$$ (1)

where t ∈ [t₀, T ] (0 ≤ t₀ < T < ∞) is time, X(t) = {X₁(t), · · ·, X_ϑ(t)}T is a ϑ-dimensional state vector, F (.) = {F₁(.), · · ·, F_ϑ(.)}^T is a ϑ-dimensional vector of smoothing functions with known forms, Z(t) = {Z₁(t), · · ·, Z_H(t)}^T is a H-dimensional co-variable vector, β = (β₁, · · ·, β_d)^T is a d-dimensional vector of unknown constant parameters with a true value β0, η(t) = {η₁(t), · · ·, η_b(t)}^T is a b-dimensional vector of unknown time-varying parameters with a true curve η0(t), and X(t₀) = X₀ is the initial value, which may be known or unknown. The function F is assumed to fulfill the Lipschitz assumption on X ensuring existence and uniqueness of the solutions to Eq. (1) (see Hairer, Nørsett and Wanner 1993; and Mattheij and Molenaar 2002).

We assume that each X_k(t) (k = 1, · · ·, ϑ) is measured with noise and the measurement model can be written as

$$Y_{k,i}={\mathit{\text{X}}}_k\left(t_i\right)+{\in }_{k,i},$$ (2)

at random time points t₁, · · ·, t_n, where the measurement errors s₁, · · ·, s_n are assumed to be i.i.d. with mean zero and variance ${\sigma }_k^2$.

For ODE models with only constant coefficients, the existing statistical methods include the nonlinear least squares (NLS) method (Bard 1974; Li, Osborne, and Prvan 2005; Xue, Miao, and Wu 2010), the two-stage smoothing estimation method (Varah 1982; Liang and Wu 2008; Brunel 2008; Wu, Xue, and Kumar 2012; Gugushvili and Klaassen 2012), the principal differential analysis (PDA) (Ramsay 1996) and its extension, the generalized profiling approach (Ramsay et al. 2007; Qi and Zhao 2010), and the Bayesian approaches (Putter et al. 2002; Huang, Liu, and Wu 2006; Donnet and Samson 2007). For ODE models with only time-varying coefficients, Chen and Wu (2008a,b) considered the two-stage smoothing estimation method, but their models are linear and additive for time-varying coefficients as follows:

$$\frac{d}{dt}\mathit{\text{X}}\left(t\right)={\mathit{\text{Z}}}^T\left(t\right)\mathit{\eta }\left(t\right)+g\left\{\mathit{\text{X}}\left(t\right)\right\},$$

and

$$\frac{d}{dt}\mathit{\text{X}}\left(t\right)=\mathit{\eta }\left(t\right)+{\mathit{\beta }}^T\mathit{\text{X}}\left(t\right),$$

where g(.) is a known function and β is a known constant vector. However, ODE model (1) with both constant and time-varying coefficients was widely used in practical applications such as in physiolotics (Thomaseth et al. 1996), pharmacokinetic (Li et al. 2002), and HIV studies (Xue, Miao, and Wu 2010; Liang, Miao, and Wu 2010; Cao, Huang, and Wu 2012).

In this article, we propose a novel two-stage estimation method for ODE model (1). In the first stage, we estimate the curve X(t) and its derivative $X^{\prime }\left(t\right)$ by the penalized spline smoothing approach (Ruppert, Wand, and Carroll 2003; Li and Ruppert 2008; Claeskens, Krivobokova, and Opsomer 2009; Wu, Xue, and Kumar 2012). Then in the second stage, we propose a class of estimation methods which are formed using the one-step discretization structure of the numerical algorithms for solving ODEs. In the second stage, we use more sample points from the smoothed curves from the first stage than the original data points, which was first suggested by Varah (1982) for parameter estimation of ODEs, and has been applied to gene regulatory network modeling (D’Haeseleer et al. 1999; Wessels, van Someren, and Reinders 2001; Bansal, Della Gatta, and di Bernardo 2006). This novel method is based on the original two-stage smoothing estimation method for ODE (Varah 1982; Chen and Wu 2008a,b; Liang and Wu 2008; Brunel 2008) like the numerical discretization-based estimation method (Wu, Xue, and Kumar 2012). The method, however, is different from the numerical discretization-based estimation method (Wu, Xue, and Kumar 2012) because the coefficients of ODE models are semiparametric. The derivation of the asymptotic properties for this model is challenging because the proposed estimation method plugs-in the nonparametric estimator obtained from the first stage to the second stage, and therefore, we cannot directly use the common Huber-Pollard Z-theorem (see Theorem 3.3.1 in van der Vaart and Wellner 1996) to derive the asymptotic normality. The article extends the Huber-Pollard Z-theorem to the two-stage estimation case.

The rest of the article is organized as follows. We introduce our new estimation methods in Section 2. The asymptotic properties of the proposed estimators are studied in Section 3. Estimation of variance using the bootstrap method is also discussed in Section 3. A real data application is given in Section 4 to illustrate the usefulness of the proposed methods, and simulation results are presented to demonstrate the finite-sample behavior of the proposed methods in Section 5. We conclude with some remarks in Section 6. The proofs of all theoretical results are given in the Appendix.

2. Estimation Methods

In the first stage, we use penalized splines to estimate the state variable X(t) as a smooth function of t. Every state variable X_k(t) (k = 1, · · ·, ϑ) is approximated at time t as follows ${\mathit{\text{X}}}_k\left(t\right)\approx {\sum }_{j=-v}^K{\delta }_{k,j}{N}_j\left(t\right)={\mathit{\text{N}}}^T\left(t\right){\mathit{\delta }}_k$,where ${\mathit{\delta }}_k={\left({\delta }_{k,-v},\cdots ,{\delta }_{k,K}\right)}^T$ is the unknown coefficient vector to be estimated from the data, and $\mathit{\text{N}}\left(t\right)={\left\{N_{-v}\left(t\right),\cdots ,N_K\left(t\right)\right\}}^T$ is the B-spline basis function vector of degree ν (order ν + 1) at knots t₀ = τ_−ν = τ_−ν+1 = · · · = τ₋₁ = τ₀ < τ₁ < · · · < τ_K < τ_K+1 = τ_K+2 = · · · = τ_K+ν+1 = T on the interval [t₀, T ]. Define n × K + ν + 1 spline design matrix N = {N (t₁), · · ·, N (t_n)}T, Y _k = (Y_k,1, · · ·, Y_k,n)^T and let ${\mathit{V}}_{\mathit{N}}={\int }_{t_0}^T\left[{\mathit{\text{N}}}^{″}\left(t\right)\right]{\left[{\mathit{\text{N}}}^{″}\left(t\right)\right]}^{T}dt$. The estimation objective function contains a sum of squared differences with a penalty term using the integrated squared second order derivative of the spline function as

$$L_k\left({\mathit{\delta }}_k;{\lambda }_k\right)={\left({\mathit{\text{Y}}}_k-\mathit{\text{N}}{\mathit{\delta }}_k\right)}^T\left({\mathit{\text{Y}}}_k-\mathit{\text{N}}{\mathit{\delta }}_k\right)+{\lambda }_k{\mathit{\delta }}_k^T{\mathit{V}}_{\mathit{N}}{\mathit{\delta }}_k.$$ (3)

The minimizer of (3) takes the form ${\widehat{\mathit{\delta }}}_k={\left({\mathit{\text{N}}}^T\mathit{\text{N}}+{\lambda }_k{\mathit{V}}_{\mathit{N}}\right)}^{-1}{\mathit{\text{N}}}^T{\mathit{\text{Y}}}_k$. Then $\widehat{\mathit{\text{X}}}\left(t\right)={\mathit{\text{N}}}^T\left(t\right){\widehat{\mathit{\delta }}}_k$ and $\widehat{\mathit{\text{X}}}\prime \left(t\right)={\left[{\mathit{\text{N}}}^{\prime }\left(t\right)\right]}^T{\widehat{\mathit{\delta }}}_k$. The derivatives of spline functions can be expressed in terms of lower order spline functions, hence the expressions of V_N and ${\mathit{\text{N}}}^{\prime }\left(t\right)$ can be explicitly obtained. To determine the penalty parameter λ_k, we use the standard generalized cross validation (GCV) method (Craven and Wahba 1979).

In the second stage, by one-step discretization methods for ODEs (Hairer, Norsett, and Wanner 1993; Mattheij and Molenaar 2002), we have

$$\frac{\mathit{\text{X}}\left(s_{j+1}\right)-\mathit{\text{X}}\left(s_j\right)}{s_{j+1}-s_j}=\mathbf{\Phi }\left\{s_j,\mathit{\text{X}}\left(s_j\right),\mathit{\text{Z}}\left(s_j\right),\mathit{\text{X}}\left(s_{j+1}\right),\mathit{\text{Z}}\left(s_{j+1}\right),\beta ,\mathit{\eta }\left(s_j\right)\right\}+O\left(h^p\right),$$ (4)

for j = 1, · · ·, m − 1, where t₀ = s₁ < s₂ < · · · < s_m = T are sample points on the interval [t₀, T ], which may be different from the observation time points t₁, · · ·, t_n with m ≥ n, h = max_j(s_j+1 − s_j). Variable m is the sample size of the augmented data set and h = O(m⁻¹). The form of $\Phi \left\{s_j,\mathit{\text{X}}\left(s_j\right),\mathit{\text{Z}}\left(s_j\right),\mathit{\text{X}}\left(s_{j+1}\right),\mathit{\text{Z}}\left(s_{j+1}\right),\beta ,\mathit{\eta }\left(t\right)\right\}$ and the order p are determined by the discretization methods. Similar to Wu, Xue, and Kumar (2012), we consider three different discretization methods: Euler’s method, trapezoidal rule and Runge-Kutta method. For each of these methods, the form of the function Φ and order p are given as follows. For the Euler’s method, we have

$$\Phi \left\{s_j,\mathit{\text{X}}\left(s_j\right),\mathit{\text{Z}}\left(s_j\right),\mathit{\text{X}}\left(s_{j+1}\right),\mathit{\text{Z}}\left(s_{j+1}\right),\beta ,\mathit{\eta }\left(s_j\right)\right\}=\mathit{\text{F}}\left\{s_j,\mathit{\text{X}}\left(s_j\right),\mathit{\text{Z}}\left(s_j\right),\beta ,\mathit{\eta }\left(s_j\right)\right\}$$

with p = 1; the trapezoidal rule gives the form

$$\begin{array}{l}\Phi \left\{s_j,\mathit{\text{X}}\left(s_j\right),\mathit{\text{Z}}\left(s_j\right),\mathit{\text{X}}\left(s_{j+1}\right),\mathit{\text{Z}}\left(s_{j+1}\right),\beta ,\mathit{\eta }\left({\mathit{s}}_{\mathit{j}}\right)\right\}\\ =\frac{1}{2}\left[\mathit{\text{F}}\left\{s_j,\mathit{\text{X}}\left(s_j\right),\mathit{\text{Z}}\left(s_j\right),\beta ,\mathit{\eta }\left({\mathit{s}}_{\mathit{j}}\right)\right\}+\mathit{\text{F}}\left\{s_{j+1},\mathit{\text{X}}\left(s_{j+1}\right),\mathit{\text{Z}}\left(s_{j+1}\right),\beta ,\mathit{\eta }\left({\mathit{\text{s}}}_{\mathit{\text{j}}+1}\right)\right\}\right]\end{array}$$

with p = 2; for the fourth-order Runge-Kutta method, we have

$$\Phi \left\{s_j,\mathit{\text{X}}\left(s_j\right),\mathit{\text{Z}}\left(s_j\right),\mathit{\text{X}}\left(s_{j+1}\right),\mathit{\text{Z}}\left(s_{j+1}\right),\beta ,\mathit{\eta }\left({\mathit{s}}_{\mathit{j}}\right)\right\}=\frac{k_1}{6}+\frac{k_2}{3}+\frac{k_3}{3}+\frac{k_4}{6}$$

with p = 4, where

• $k_1=\mathit{F}\left\{s_j,X\left(s_j\right),Z\left(s_j\right),\beta ,\mathbf{\eta }\left({\mathbf{s}}_{\mathbf{j}}\right)\right\},$

• $k_2=\mathit{F}\left\{s_j+h_j/2,X\left(s_j\right)+h_j{k}_1/2,Z\left(s_j+h_j/2\right),\beta ,\mathbf{\eta }\left({\mathbf{s}}_{\mathbf{j}}+{\mathbf{h}}_{\mathbf{j}}/2\right)\right\},$

• $k_3=\mathit{F}\left\{s_j+h_j/2,X\left(s_j\right)+h_j{k}_2/2,Z\left(s_j+h_j/2\right),\beta ,\mathbf{\eta }\left({\mathbf{s}}_{\mathbf{j}}+{\mathbf{h}}_{\mathbf{j}}/2\right)\right\},\text{and}$

• $k_4=\mathit{F}\left\{s_j+h_j,X\left(s_j\right)+h_j{k}_3,Z\left(s_j+h_j\right),\beta ,\mathbf{\eta }\left({\mathbf{s}}_{\mathbf{j}}+{\mathbf{h}}_{\mathbf{j}}\right)\right\}.$

Further we approximate each component of η(t) by regression splines with the same B-spline basis functions. Let t₀ = u₀ < u₁ < · · · < u_q = T be a partition of the interval [t₀, T ], where $q=O\left(n^{\varpi }\right)\left(0<\varpi <0.5\right)$ is a positive integer such that ${\max }_{1\le j\le q}\left|u_j-u_{j-1}\right|=O\left(n^{-\varpi }\right)$. Then we have r = q + l normalized B-spline basis functions of order l + 1 that form a basis for the linear spline space. We denote these basis functions as a vector π(t) = {B₁(t), · · ·, B_r(t)}T. Parameter η(t) can be approximated by π^T (t)α, where $\mathit{\alpha }={\left({\alpha }_1,\cdots ,{\alpha }_r\right)}^T\in {\mathcal{R}}^r$ is the spline coefficient vector with α₀ corresponding to η₀(t).

Replacing the state variable, X(t), with its estimate, $\widehat{\mathit{\text{X}}}\left(t\right)$, the expression (4) can be re-written as

$$\frac{\widehat{\mathit{\text{X}}}\left(s_{j+1}\right)-\widehat{\mathit{\text{X}}}\left(s_j\right)}{s_{j+1}+s_j}=\Phi \left\{s_j,\widehat{\mathit{\text{X}}}\left(s_j\right),\mathit{\text{Z}}\left(s_j\right),\widehat{\mathit{\text{X}}}\left(s_{j+1}\right),\mathit{\text{Z}}\left(s_{j+1}\right),\beta ,{\mathit{\pi }}^T\left(s_j\right)\mathit{\alpha }\right\}+{\Gamma }_j,$$ (5)

where Γ_j is the sum of discretization error, approximation error by splines and estimation error of $\widehat{\mathit{\text{X}}}(.)$ from the first stage. Obviously, Γ_j are not i.i.d., but dependent. For a prescribed non-negative weight function w_k(t) on [t₀, T ] (k = 1, · · ·, G), we propose to estimate $\left({\widehat{\beta }}_n,{\widehat{\alpha }}_n\right)$ of (β, α) by minimizing the following weighted least squares criterion:

$${\sum _{j=1}^m\sum _{k1}^{\vartheta }{w}_k\left(s_j\right)\left[\frac{{\widehat{\mathit{\text{X}}}}_k\left(s_{j+1}\right)-{\widehat{\mathit{\text{X}}}}_k\left(s_j\right)}{s_{j+1}+s_j}-{\Phi }_k\left\{s_j,\widehat{\mathit{\text{X}}}\left(s_j\right),\mathit{\text{Z}}\left(s_j\right),\widehat{\mathit{\text{X}}}\left(s_{j+1}\right),\mathit{\text{Z}}\left(s_{j+1}\right),\beta ,{\mathit{\pi }}^T\left(s_j\right)\mathit{\alpha }\right\}\right]}^2.$$ (6)

Then ${\widehat{\mathit{\eta }}}_n\left(t\right)={\mathit{\pi }}^T\left(t\right){\widehat{\mathit{\alpha }}}_n$ Assumption required on w_k(.) for the asymptotic results is discussed in Section 3. From numerical studies, we found that the selection of w_k(.) have little affect on the performance of the proposed estimators.

In general, a higher order discretization method is expected to give a better estimation accuracy since its discretization error is smaller, but its computational cost is higher. The trade-off between the estimation accuracy and computational cost needs to be considered in practical applications. For convenience, we use the abbreviations EDB, TDB, and RDB for the Euler’s discretization-based estimator, the trapezoidal discretization-based estimator, and the Runge-Kutta discretization-based estimator, respectively. We study the asymptotic properties of the proposed estimators in Section 3 and evaluate the finite-sample performance of these estimators in Sections 4–5.

3. Asymptotic Properties

In this section, we establish the asymptotic properties of the proposed two-stage estimator for the semiparametric ODE models. This is challenging since we need to integrate the asymptotic results from the two-stage smoothing-based estimation method for ODE models (Varah 1982; Brunel 2008; Chen and Wu 2008a,b; Liang and Wu 2008; Wu, Xue, and Kumar 2012) and the spline approximation error for the time-varying coefficients together especially when the resulting error terms Γ_j in (5) are not i.i.d., but dependent.

Let µ be a nonnegative integer and γ ∈ (0, 1] such that ϱ = µ + γ > 0.5. Let $\mathcal{A}$ be the collection of functions η on [t₀, T ] whose µ-th derivative η^(µ) exists and satisfies the Lipschitz condition of order $\gamma :\left|{\eta }^{\left(\mu \right)}\left(s\right)-{\eta }^{\left(\mu \right)}\left(t\right)\right|\le L{\left|s-t\right|}^{\gamma }$ for s, t ∈ [t₀, T ] with a general positive constant L. Denote the parametric vector θ = {β^T, η^T (t)}T and define ${\Vert \eta \left(t\right)\Vert }_2={\left[{\int }_{t_0}^T{\eta }^2\left(t\right)\rho \left(t\right)dt\right]}^{1/2}$ for any function η(t) whenever the integral exists, where ρ(t) is defined in the Assumption A5. For any θ₁ and θ₂, we define a distance $\rho \left({\theta }_1,{\theta }_2\right)=\Vert {\beta }_1-{\beta }_2\Vert +{\sum }_{j=1}^b{\Vert {\eta }_{j,1}-{\eta }_{j,2}\Vert }_2$. For 1 ≤ k ≤ ϑ, the assumptions A-D needed for the theorems are listed below:

Assumption A:

• (A1)For $k={\max }_{0\le j\le K}\left({\tau }_{k,j+1}-{\tau }_{k,j}\right)$, there exists a constant M > 0 such that $k/\underset{0\le j\le K}{\min }\left({\tau }_{k,j+1}-{\tau }_{k,j}\right)\le M$ and $\underset{0\le j\le K}{\max }\left|\frac{{\tau }_{k,j+1}-{\tau }_{k,j}}{{\tau }_{k,j}-{\tau }_{k,j-1}}-1\right|=o\left(1\right)$.
• (A2)K ∼ cn^ψ with $1/\left(2v+3\right)\le \psi <1$ and ${\lambda }_k=O\left(n^{\pi }\right)$ with $\pi \le v/\left(2v+3\right)$.
• (A3)X_k(t) ∈ C^ν+1[t₀, T ] with ν ≥ 2.
• (A4)$K^{*}\left(K+v-1\right){\left({\lambda }_k{\tilde{c}}_1\right)}^{1/4}{n}^{-1/4}<1$ for some constant ${\tilde{c}}_1$.
• (A5)t₁, · · ·, t_n are i.i.d. with a cumulative distribution function Q(t) and a positive and continuous derivative density ρ(t). Moreover, ρ(t) is bounded away from 0 and +∞ and has a bounded and continuous first-order derivative.

Assumption B:

• (B1)F_k{t, X(t), Z(t), β, η(t)} is a continuous function for $\mathit{\beta }\in \mathcal{B}$ and ${\eta }_j\left(t\right)\in \mathcal{F}\left(1\le j\le b\right)$, where B is a compact subset of ${\mathcal{R}}^d$ with a finite diameter R_β.
• (B2)All partial Fréchet derivatives of F_k{t, X, Z(t), β, η} up to order p with respect to t, X, Z and η exist and are continuous.
• (B3)The numerical method for solving ODEs is of order p.
• (B4)The true parameter β0 is an interior point of B.

Assumption C:

• (C1)The first and the second partial Fr´echet derivatives of F_k{t, X(t), Z(t), β, η}; $\frac{\partial {\mathit{\text{F}}}_k}{\partial \mathit{\beta }},\frac{\partial {\mathit{\text{F}}}_k}{\partial \mathit{\eta }},\frac{{\partial }^2{\mathit{\text{F}}}_k}{\partial \mathit{\beta }\partial {\mathit{\beta }}^T},\frac{{\partial }^2{\mathit{\text{F}}}_k}{\partial \mathit{\beta }\partial {\mathit{\eta }}^T}$ and $\frac{{\partial }^2{\mathit{\text{F}}}_k}{\partial \mathit{\eta }\partial {\mathit{\eta }}^T}$ exist and are continuous and uniformly bounded for all t, β, X(t) and η.
• (C2)The first partial Fr´echet derivatives $\frac{\partial }{\partial t}{F}_k\left\{t,\mathit{\text{X}}\left(t\right),\mathit{\text{Z}}\left(t\right),\beta ,\mathit{\eta }\left(t\right)\right\}$ and $\frac{\partial }{\partial \mathit{\text{X}}}{F}_k\left\{t,\mathit{\text{X}},\mathit{\text{Z}}\left(t\right),\beta ,\mathit{\eta }\left(t\right)\right\}$ are continuous and uniformly bounded for all t, β, X(t), Z(t) and η(t).
• (C3)The measurement error red 𝜖_k,1, · · ·, 𝜖_k,n are i.i.d. with mean 0 and variance ${\sigma }_k^2\left(0<{\sigma }_k^2<\infty \right)$ Moreover, their density function, ρ(.), is bounded away from 0 and +∞ and has a bounded and continuous first-order derivative.
• (C4)l + 1 ≥ ϱ.
• (C5)For any $\mathit{\beta }\in \mathcal{B}$ and $\mathit{\eta }\in \mathcal{A}$, ${\sum }_{k=1}^{\vartheta }\text{E}\left(w_k\left(t\right){\left[F_k\left\{t,\mathit{\text{X}}\left(t\right),\mathit{\text{Z}}\left(t\right),\theta \right\}-F_k\left\{t,\mathit{\text{X}}\left(t\right),\mathit{\text{Z}}\left(t\right),{\theta }_0\right\}\right]}^2\right)=0$ if and only if β = β0 and P {t : η(t) = η0(t)} = 1.

Assumption D:

• (D1)The weight function w_k(.) is bounded and nonnegative on [t₀, T ] with w_k(t₀) = w_k(T) = 0. Moreover, w_k(t) has a bounded and continuous first-order derivative.
• (D2)ϖ satisfies the restrictions 0.25/ϱ < ϖ < 0.5 and ϖ(2 + ϱ) > 0.5.

Note that Assumption A is a standard condition to derive local asymptotic properties of the penalized spline estimator for the nonparametric regression function (Claeskens, Krivobokova, and Opsomer 2009; Wu, Xue, and Kumar 2012). Assumption B is required for the precision of the discretization algorithm (Wu, Xue, and Kumar 2012), where Assumption B3 from Mattheij and Molenaar (2002, p.55–56) defines the precision of the numerical algorithm. For example, the Euler backward method, the trapezoidal rule and the 4-stage Runge-Kutta are of order 1, 2 and 4, respectively. Assumptions C1-C3 are needed for the proof of consistency. Assumption C4 is a general constrained condition in spline theories. Assumption C5 is required for identifiability. Assumption D1 on the boundary constraint of weight functions is required for some technical reasons, which was also used by Wu, Xue, and kumar (2012) to ensure the convergence rate of the parametric components’ estimators to achieve the common root n. Assumption D2 is needed for the proof of asymptotic normality.

Theorem 1: Under Assumptions A, B and C, we have $\rho \left(\widehat{\mathit{\theta }},{\mathit{\theta }}_0\right)\to 0$, almost surely, under $P_{{\mathit{\theta }}_0}$.

Theorem 2: Under Assumptions A, B, C and D, we have $\rho \left({\widehat{\mathit{\theta }}}_n,{\mathit{\theta }}_0\right)=O_p\left(n^{-\varpi \varrho }+n^{-\left(1-\varpi \right)/2}\right)$. Moreover, if $\varpi =1/\left(1+2\varrho \right),\hspace{0.17em}{\Vert {\widehat{\eta }}_n\left(t\right)-{\eta }_0\left(t\right)\Vert }_2=O_p\left(n^{-\varrho /\left(1+2\varrho \right)}\right)$, which is the same as the optimal rate of the standard nonparametric function estimation.

Theorem 3: Under Assumptions A, B, C and D, if ϑ ≥ 2, then we have $n^{1/2}\left({\widehat{\beta }}_n-{\beta }_0\right)\stackrel{d}{\to }N\left(0,A^{-1}{\sum }_2{\left(A^{-1}\right)}^T\right)$ with the matrixes A and ${\Sigma }_2$ given in (11) and (13) in Section 7, respectively.

Remark 1: For ϑ = 1, Theorem 3 does not hold because of unidentifiability problem, which is similar to Theorem 4.2 in Xue, Miao, and Wu (2010).

The asymptotic variance-covariance matrix needs to be estimated in order to perform statistical inference for unknown parameters β. There are some standard methods that can be used. We recommend the nonparametric bootstrap method (Efron 1982), which has been used for statistical inference for ODE models with constant coefficients (Joshi, Seidel-Morgenstern, and Kremling 2006).

Let {(t_i∗, Y_i∗), i = 1, · · ·, n} be independent and drawn with replacement from the original sample {(t_i, Y_i), i = 1, · · ·, n}. In the first stage, based on the bootstrap sample {(t_i∗, Y_i∗), i = 1, · · ·, n}, we can obtain estimators ${\widehat{\mathit{\text{X}}}}_{*}\left(t\right)$ and ${\widehat{\mathit{\text{X}}}}_{*}\prime \left(t\right)$. Plugging them into the second stage, we can get the bootstrap M-estimator $\left({\widehat{\beta }}_{n*},{\widehat{\alpha }}_{n*}\right)$. From Theorem 1 in Cheng and Huang 2010) and Theorem 3 in this article, given $\left\{t_i,\mathit{\text{Y}}\left(t_i\right)\right\},\sqrt{n}\left({\widehat{\beta }}_{n*}-{\widehat{\beta }}_n\right)$ and $\sqrt{n}\left({\widehat{\beta }}_n-{\widehat{\beta }}_0\right)$ have the same limiting distribution, which can be used for inference on ${\widehat{\beta }}_n$ and ${\widehat{\eta }}_n\left(t\right)$.

4. Real data analysis

We used the proposed two-stage method to estimate parameters of an ODE model that describes kinetics of influenza-specific CD8⁺ T cells during the primary immune response. Goal of the study is to estimate biological parameters that determine the distribution of CD8⁺ T cells during influenza infection in mice. Since these parameters are not directly estimable from experiments, a mathematical model is developed to mimic the kinetics of CD8⁺ T cells during influenza infection as follows (Wu et al. 2011):

$$\begin{array}{l}\frac{d{T}_E^m}{dt}=\left({\rho }_m\left(t\right)-{\gamma }_{ms}\right)T_E^m,\\ \frac{d{T}_E^s}{dt}=\left({\rho }_s\left(t\right)-{\gamma }_{sl}\right)T_E^s+{\gamma }_{ms}{T}_E^m,\\ \frac{d{T}_E^l}{dt}={\gamma }_{sl}{T}_E^s-{\delta }_l{T}_E^l.\end{array}$$ (7)

The state variable $T_E^m$ denotes effector CD8⁺ T cell count in Mediastinal lymph node (MLN), $T_E^s$ denotes effector CD8⁺ T cell count in spleen, and $T_E^l$ denotes effector CD8⁺ T cell count in lung. The model has two time-varying parameters: ρ_m(t) and ρ_s(t) and three constant parameters: γ_ms, γ_sl, and δ_l. Parameter ρ_m(t) denotes the proliferation rate of effector CD8⁺ T cells in MLN and ρ_s(t) denotes the proliferation rate of effector CD8⁺ T cells in spleen. Constant parameter γ_ms denotes the migration rate of effector CD8⁺ T cells from MLN to spleen, γ_sl denotes the migration rate of effector CD8⁺ T cells from spleen to lung, and δ_l denotes the death rate of effector CD8⁺ T cells in lung. Data on $T_E^m$, $T_E^s$ and $T_E^l$ were collected through experiments on Female C57BL/6 mice (see Wu et al. 2011 for more details). Mice were anesthetized and intranasally inoculated with H3N2 A/Hong Kong/X31 influenza-A virus. From mice tissues, CD8⁺ T cell data were collected on days 0,4,5,6,7,8,9,10,11,12,14, and 28. At each time point, data were collected from 6 mice except day 12 for which data were collected from 12 mice. For estimating parameters, we only used the data from day 5 to day 14 when adaptive immune response is active. Also one observation on day 8 was identified as an outlier and hence removed from the data for model fitting.

Wu, Xue, and Kumar (2012) showed that the TDB method is the best among all the two-stage methods when both accuracy and computational cost are taken into account. Hence we used the TDB algorithm to estimate the parameters of the model (7). The time-varying parameters ρ_m(t) was approximated by a linear sum of four B-splines with order 4:${\rho }_m\left(t\right)={\sum }_{j=0}^3{\alpha }_j{B}_j\left(t\right),\hspace{0.17em}5\le t\le 14$. These B-splines were evaluated at knot points (5,5,5,8,11,14,14,14). Two inner knot points at 8 and 14 allowed us to capture the bi-modal shape of the time-varying parameters. The time-varying parameter ρ_s(t) was also approximated by a linear sum of four B-splines with order 4. We used the weight function w(t_i) = sin((t_i − 5)π/9) in the objective function. We adopted the data augmentation size m = 11 ≈ 1.25n with n = 9 in the second stage. To find confidence intervals of the parameters, we used the bootstrap method. The bootstrap method was repeated 500 times to obtain 500 bootstrap estimates of each parameter. A 95% confidence interval for a parameter was obtained by finding the 2.5th percentile and the 97.5th percentile of the bootstrap parameter estimates. Point estimates along with 95% confidence intervals for the parameters of model (7) are given in Table 1. Plots of the estimates of the time-varying parameter ρ_m and ρ_s along with their 95% confidence bands are shown in the left and right sides of Figure 1, respectively. From these estimation results, we can see that our estimates for the constant parameters (γ_ms, γ_sl, δ_l) are on the similar scale of the original estimates in Wu et al. (2011). Our time-varying parameter estimates for proliferation rates demonstrate an interesting bi-mode shape, which may indicate an earlier peak during the adaptive immune response and a later peak during the recovery period after the influenza virus is removed at Day 10. But these biological implications need to be confirmed by further experiments in the future.

Table 1:

Parameter point estimates and 95% bootstrap confidence intervals in the real data example.

Parameter	Estimate	Confidence interval
γms	1.57	(0.94,3.03)
γsl	0.92	(0.55,1.21)
δl	4.60	(2.26,6.62)

Figure 1:

Estimates of the proliferation rates of CD8⁺ T cells in MLN (left) and spleen (right) along with 95% confidence bands.

5. Simulation Studies

We conducted simulation studies to evaluate the performance of the proposed two-stage estimation method. Similar to Section 4, we only considered the TDB method. We used model (7) to generate the simulation data. Constant parameters for simulation studies were γ_ms = 1.25, γ_sl = 1.25 and δ_l = 9. The time-varying parameters ρ_m(t) and ρ_s(t) that were used to generate simulation data are shown in Figure 2. These simulation parameters were selected so that the solution to (7) is well-behaved (shown in Figures 3).

Figure 2:

ρ_m(t) (left) and ρ_s(t) (right) used to generate simulation data.

Figure 3:

Solutions of $T_E^m$ (left), $T_E^s$(middle) and $T_E^l$ (right) for the simulation parameters.

We generated two simulation data sets: one of 20 observations and the other of 40 observations for each state variable. Simulation data were generated at equally-spaced time points between 5 and 14. A fourth order Runge-Kutta method was used to solve (7) to generate the simulation data. Initial conditions $T_E^m\left(5\right)=5000$, $T_E^s\left(5\right)=45000$, and $T_E^l\left(5\right)=1300$ were used to solve (7) numerically. Noise generated from independent normal distributions with zero mean and standard deviations ${\sigma }_E^m$, ${\sigma }_E^s$, and ${\sigma }_E^l$ were added to the simulation data for $T_E^m$, $T_E^s$, and $T_E^l$ respectively. The standard deviation for the noise was calculated by taking 1% of the standard deviation of the noiseless simulation data. We tried two different data augmentation sizes: m = n and m = 2n. Similar to the real data analysis, we used the weight function w(t_i) = sin((t_i − 5)π/9) in the objective function.

We estimated ρ_m(t), γ_ms and γ_sl, and assumed that the rest of the parameters are known. This reduced the computational burden on the optimization routine avoiding local convergence problem and hence giving a fair comparison of the performance of different data augmentation sizes. The time-varying parameter ρ_m(t) was approximated by a linear sum of four B-spline basis functions of order 4. R function “genoud”, which is a global optimization routine, was used to minimize (6) to obtain the parameter estimates. The algorithm “genoud” searches for the estimate of the parameters in the range [0,10] and starts the search at 5 for each parameter. For each simulation case, 500 runs were replicated. The following average relative error (ARE) of the constant parameter estimates were used to compare for different data augmentation sizes,

$$\text{ARE}\left(\theta \right)=\frac{1}{500}\sum _{i=1}^{500}\left|\frac{{\theta }_i-{\theta }_0}{{\theta }_0}\right|,$$ (8)

where θ₀ is the true value of a parameter θ and θ_i is the i^th simulation run. For the time-varying parameter ρ_m, we calculated its average mean square error (AMSE) as follows,

$$\text{AMSE}\left({\rho }_m\right)=\frac{1}{500}\sum _{i=1}^{500}\left(\frac{1}{1000}\sum _{j=1}^{1000}{\left({\rho }_{m_i}\left(t_j\right)-{\rho }_{m_0}\left(t_j\right)\right)}^2\right),$$ (9)

where ρ_m (t_j) is the estimate of ρ_m(t_j) in the i^th simulation run and ρ_m (t_j) is the true value of ρ_m at the time point t_j. Time points t_j’s in the equation (9) are equi-spaced points between 5 and 14. In Table 2, we show ARE for the constant parameters γ_ms and γ_sl and AMSE for the time-varying parameter ρ_m at two data augmentation sizes: m = n and m = 2n. We observed the trend that both ARE and AMSE decrease as the data augmentation size increases. This simulation study clearly demonstrates the merit of data augmentation in the second stage in improving the overall fit of the model to the data.

Table 2:

ARE for γ_ms and γ_sl and AMSE for ρ_m

n	Parameter	m/n	ARE or AMSE
20	γms	1	6.08
		2	1.40
	γsl	1	0.17
		2	0.15
	ρm	1	0.0044
		2	0.0005
40	γms	1	0.91
		2	0.78
	γsl	1	0.13
		2	0.11
	ρm	1	0.0002
		2	0.0001

6. Conclusion and Discussion

In this article, we have proposed a new class of estimation methods to estimate both constant and time-varying coefficients for semiparametric ODE models, which exploits the form of numerical discretization algorithms for an ODE solver and uses more sample points than the original data points. This new class of methods are shown to have benefits in terms of computational efficiency and estimation accuracy over the original two-stage smoothing estimation methods for ODE models. Other estimation approaches for the semiparametric ODE models, such as the NLS method and the generalized profiling approach (Ramsay et al. 2007) may also be considered, but this new class of estimation methods provides an alternative estimation method with a low computation cost. In this article, we have not explored how to select the data augmentation size in the second stage. As per our simulation studies, the bigger the data augmentation size, the better the estimation. However, a bigger data augmentation size comes with a higher computational cost. The trade-off between the data augmentation size and the computational cost is an open problem.

Appendix: Technical Proofs

For presentation and notation simplicity, we mainly focus on the case without input variable Z and t in (1). And we only consider the two-stage smoothing estimation method, i.e., replacing (6) with the following objection function:

$$\sum _{j=1}^m\sum _{k=1}^{\vartheta }{w}_k\left(s_j\right){\left[{\widehat{\mathit{\text{X}}}}_k^{\prime }\left(s_j\right)-F_k\left\{\widehat{\mathit{\text{X}}}\left(s_j\right),\beta ,{\mathit{\pi }}^T\left(s_j\right)\alpha \right\}\right]}^2.$$

All of the proofs can be extended to the numerical discretization-based methods based on the objection function (6) by similar arguments in the proofs of Lemma 5, Proposition 1 and Proposition 2 in Wu, Xue, and Kumar (2012). We also only discuss the univariate case for η(t), which can be extended to the multivariate case using similar approach as in He, Xue, and Shi (2010). Denote

$${\tilde{S}}_n(\mathit{\theta })=\frac{1}{m}\sum _{j=1}^m\sum _{k=1}^{\vartheta }{w}_k\left(s_j\right){[{\widehat{\mathit{\text{X}}}}_k^{\prime }\left(s_j\right)-F_k\left\{\widehat{\mathit{\text{X}}}\left(s_j\right),\mathit{\theta }\right\}]}^2,$$

$$S_n(\mathit{\theta })=\frac{1}{m}\sum _{j=1}^m\sum _{k=1}^{\vartheta }{w}_k\left(s_j\right){\left[{\mathit{\text{X}}}_k^{\prime }\left(s_j\right)-F_k\left\{\mathit{\text{X}}\left(s_j\right),\mathit{\theta }\right\}\right]}^2,$$

$$S(\mathit{\theta })=\sum _{k=1}^{\vartheta }\text{E}\left(w_k\left(t\right){\left[{\mathit{\text{X}}}_k^{\prime }\left(t\right)-F_k\left\{\mathit{\text{X}}\left(t\right),\mathit{\theta }\right\}\right]}^2\right).$$

Proof of Theorem 1. Denote set ${\mathcal{A}}_n=\left\{\eta \left(t\right)={\sum }_{i=1}^r{B}_i\left(t\right){\alpha }_i:{\max }_{1\le i\le r}\left|{\alpha }_i\right|\le {\mathcal{l}}_n\right\}$, where ${\mathcal{l}}_n\le n^{\left(2l-1\right)/\left[2l^{\prime }\left(2l+1\right)\right]}$ with a constant $l^{\prime }$ arbitrarily close to l (see Shen 1997, p. 2560), and ${\Theta }_n=\left\{\mathit{\theta }:\beta \in \mathcal{B},\eta \in {\mathcal{A}}_n\right\}=\mathcal{B}\times {\mathcal{A}}_n$. For any $\mathit{\theta }\in \Theta \dot{=}\left\{\mathit{\theta }:\beta \in \mathcal{B},\eta \in \mathcal{A}\right\}=\mathcal{B}\times \mathcal{A}$, under Assumption C4, by Corollary 6.21 in Schumaker (1981), there exists θ_n ∈ Θ_n such that $\rho \left(\mathit{\theta },{\mathit{\theta }}_n\right)=O\left(n^{-\varpi \varrho }\right)$

First, we claim that ${\sup }_{\theta }\left|{\tilde{S}}_n(\mathit{\theta })-S_n(\mathit{\theta })\right|\to 0$, a.s., under $p_{{\theta }_0}$. In fact, by the mid-value theorem, we have that

$$\begin{array}{l}{\tilde{S}}_n(\mathit{\theta })=\frac{1}{m}\sum _{j=1}^m\sum _{k=1}^{\vartheta }{w}_k\left(s_j\right){[{\widehat{\mathit{\text{X}}}}_k^{\prime }\left(s_j\right)-F_k\left\{\widehat{\mathit{\text{X}}}\left(s_j\right),\mathit{\theta }\right\}]}^2\\ \hspace{1em}\hspace{1em}\hspace{1em}=\frac{1}{m}\sum _{j=1}^m\sum _{k=1}^{\vartheta }{w}_k\left(s_j\right)[{\mathit{\text{X}}}_k^{\prime }\left(s_j\right)-F_k\left\{\mathit{\text{X}}\left(s_j\right),\theta \right\}+{\widehat{\mathit{\text{X}}}}_k^{\prime }\left(s_j\right)-{\mathit{\text{X}}}_k^{\prime }\left(s_j\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{+F_k\left\{\mathit{\text{X}}\left(s_j\right),\mathit{\theta }\right\}-F_k\left\{\widehat{\mathit{\text{X}}}\left(s_j\right),\mathit{\theta }\right\}]}^2\\ \hspace{1em}\hspace{1em}\hspace{1em}=\frac{1}{m}\sum _{j=1}^m\sum _{k=1}^{\vartheta }{w}_k\left(s_j\right)[{\mathit{\text{X}}}_k^{\prime }\left(s_j\right)-F_k\left\{\mathit{\text{X}}\left(s_j\right),\theta \right\}+{\widehat{\mathit{\text{X}}}}_k^{\prime }\left(s_j\right)-{\mathit{\text{X}}}_k^{\prime }\left(s_j\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{-\sum _{\iota =1}^{\vartheta }\frac{\partial F_k\left\{\mathit{\text{X}},\mathit{\theta }\right\}}{\partial {\mathit{\text{X}}}_{\iota }}|{}_{{\mathit{\text{X}}}_1={\mathit{\text{X}}}_1\left(s_j\right),\cdots ,{\mathit{\text{X}}}_{\iota }={\widetilde{\mathit{\text{X}}}}_{\iota ,j},\cdots ,{\mathit{\text{X}}}_{\vartheta }={\mathit{\text{X}}}_{\vartheta }\left(s_j\right)}[{\widehat{\mathit{\text{X}}}}_{\iota }\left(s_j\right)-{\mathit{\text{X}}}_{\iota }\left(s_j\right)]]}^2\\ \hspace{1em}\hspace{1em}\hspace{1em}=\frac{1}{m}\sum _{j=1}^m\sum _{k=1}^{\vartheta }{w}_k\left(s_j\right){\left[{\mathit{\text{X}}}_k^{\prime }\left(s_j\right)-F_k\left\{\mathit{\text{X}}\left(s_j\right),\mathit{\theta }\right\}\right]}^2\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\frac{1}{m}\sum _{j=1}^m\sum _{k=1}^{\vartheta }{w}_k\left(s_j\right){\left[{\widehat{\mathit{\text{X}}}}_k^{\prime }\left(s_j\right)-{\mathit{\text{X}}}_k^{\prime }\left(s_j\right)\right]}^2\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}+\frac{1}{m}\sum _{j=1}^m\sum _{k=1}^{\vartheta }\sum _{\iota =1}^{\vartheta }{w}_k\left(s_j\right){\left(\frac{\partial F_k\left\{\mathit{\text{X}},\mathit{\theta }\right\}}{\partial {\mathit{\text{X}}}_{\iota }}\right)}^2{[{\widehat{\mathit{\text{X}}}}_{\iota }\left(s_j\right)-{\mathit{\text{X}}}_{\iota }\left(s_j\right)]}^2+R_n(\mathit{\theta }),\end{array}$$ (10)

where ${\widetilde{\mathit{\text{X}}}}_{\iota ,j}$ is some point between X_ι(s_j) and ${\widehat{\mathit{\text{X}}}}_{\iota }\left(s_j\right)$ and R_n(θ) is the left term of interaction. Under Assumptions A and C1, by the strong law of large numbers and Lemma 2 in Wu, Xue, and Kumar (2012), we have

$$\frac{1}{m}\sum _{j=1}^m\sum _{k=1}^{\vartheta }{w}_k\left(s_j\right){\left[{\widehat{\mathit{\text{X}}}}_k^{\prime }\left(s_j\right)-{\mathit{\text{X}}}_k^{\prime }\right]}^2\to \sum _{k=1}^{\vartheta }\text{E}\left(w_k\left(t\right){\left[{\widehat{\mathit{\text{X}}}}_k^{\prime }\left(t\right)-{\mathit{\text{X}}}_k^{\prime }\left(t\right)\right]}^2\right)\to 0,\hspace{1em}\text{a}.\text{s}.,$$

and

$$\begin{array}{l}\underset{\theta }{\text{sup}}\frac{1}{m}\sum _{j=1}^m\sum _{k=1}^{\vartheta }\sum _{\iota =1}^{\vartheta }{w}_k\left(s_j\right){\left(\frac{\partial F_k\left\{\mathit{\text{X}},\mathit{\theta }\right\}}{\partial X_{\iota }}\right)}^2{|}_{{\mathit{\text{X}}}_1={\mathit{\text{X}}}_1\left(s_j\right),\cdots ,{\mathit{\text{X}}}_{\iota }={\widetilde{\mathit{\text{X}}}}_{\iota ,j},\cdots ,{\mathit{\text{X}}}_{\vartheta }={\mathit{\text{X}}}_{\vartheta }\left(s_j\right)}{\left[{\widehat{\mathit{\text{X}}}}_{\iota }\left(s_j\right)-{\mathit{\text{X}}}_{\iota }\left(s_j\right)\right]}^2\\ \to \sum _{k=1}^{\vartheta }\sum _{\iota =1}^{\vartheta }\text{E}\left[w_k\left(t\right)\underset{\mathit{\theta }}{\text{sup}}{\left(\frac{\partial F_k\left\{\mathit{\text{X}},\mathit{\theta }\right\}}{\partial {\mathit{\text{X}}}_{\iota }}\right)}^2{|}_{\mathit{\text{X}}=\mathit{\text{X}}\left(t\right)}{\left[{\widehat{\mathit{\text{X}}}}_{\iota }\left(t\right)-{\mathit{\text{X}}}_{\iota }\left(t\right)\right]}^2\right]\to 0.\hspace{1em}\text{a}.\text{s}.\end{array}$$

By the Cauchy-Schwarz inequality, it follows that the left term supθ R_n(θ) is bounded by a product of a lower term than that other three terms in (10) and can be ignored. Thus the claim holds.

Next, let ${\mathcal{A}}_n^{\delta }$ be the set $\left\{\eta \in {\mathcal{A}}_n,{\Vert \eta -{\eta }_{n0}\Vert }_2\le \delta \right\}$ and $N_2\left(\in ,L_{\infty },{\mathcal{A}}_n^{\delta }\right)$ be its bracketing number with respect to L_∞ (see Definition 2.1.6, van der Vaart and Wellner 1996), where η_n0 is the map point of η₀ in the sieve A_n. By the calculation of Shen and Wong (1994, p.597), for any 𝜖 ≤ δ, we have $N_2\left(\in ,L_{\infty },{\mathcal{A}}_n^{\delta }\right)\le C{\left(\delta /\in \right)}^r$, where r = q + l is the number of B-splines basis functions. Let ${\mathcal{F}}_n$ be the set $\left\{S_n(\mathit{\theta })-S(\mathit{\theta }):\Vert \beta -{\beta }_0\Vert \le \delta ,\eta \in {\mathcal{A}}_n,{\Vert \eta -{\eta }_{n0}\Vert }_2\le \delta \right\}$. For any θ₁, θ₂ ∈ Θ_n, we can easily obtain $\left|S_n\left({\mathit{\theta }}_1\right)-S_n\left({\mathit{\theta }}_2\right)\right|\le C\left(\Vert {\beta }_1-{\beta }_2\Vert +{\Vert {\eta }_1-{\eta }_2\Vert }_{\infty }\right)$ using Taylor expansion. Hence

$$\begin{array}{l}{N}_2(\in ,L_{\infty },{\mathcal{F}}_n)\le N_1\left(\in /2,L_2,\mathcal{B}\right)\times N_2\left(\in /2,L_{\infty },{\mathcal{A}}_n^{\delta }\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\le C{\left(3R_{\beta }/\in \right)}^d\left(\delta /\in \right)\le C^{\prime }{\left(1/\in \right)}^{r+d}.\end{array}$$

Then by similar arguments of (A.2) in Xue, Lam, and Li (2004), we can obtain ${\sup }_{{\mathcal{F}}_n}\left|S_n(\mathit{\theta })-S(\mathit{\theta })\right|\to 0$ a.s., under $p_{{\theta }_0}$.

Third, under Assumption C5, it is easy to obtain that S(θ) reaches its unique minimum at θ = θ₀. Under Assumption B4, it follows that the first-order derivative $\frac{\partial S(\mathit{\theta })}{\partial \mathit{\theta }}$ of S(θ) at θ₀ equals to zero and the second-order derivative $\frac{{\partial }^{2}S(\mathit{\theta })}{\partial \mathit{\theta }\partial {\mathit{\theta }}^T}$ of S(θ) at θ₀ is positive definite. By Assumptions A5, C1 and C3, the second-order derivative of S(θ) in a small neighborhood of θ₀ is bounded away from 0 and ∞. Then the second-order Taylor expansion of S(θ) gives that there exists a constant 0 < C < ∞ such that $S\left({\widehat{\mathit{\theta }}}_n\right)-S\left({\mathit{\theta }}_0\right)\ge C_p\left({\widehat{\mathit{\theta }}}_n,{\mathit{\theta }}_0\right)$. Moreover,

$$\begin{gathered} 0\le S\left({\widehat{\theta }}_n\right)-S\left({\mathit{\theta }}_0\right)=S\left({\widehat{\theta }}_n\right)-{\tilde{S}}_n\left({\widehat{\theta }}_n\right)+{\tilde{S}}_n\left({\widehat{\theta }}_n\right)-S\left({\mathit{\theta }}_0\right) \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\le S\left({\widehat{\mathit{\theta }}}_n\right)-{\tilde{S}}_n\left({\widehat{\mathit{\theta }}}_n\right)+{\tilde{S}}_n\left({\widehat{\mathit{\theta }}}_0\right)-S\left({\mathit{\theta }}_0\right) \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\le 2\underset{\mathit{\theta }}{\sup }\left|{\tilde{S}}_n(\mathit{\theta })-S(\mathit{\theta })\right| \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\le 2\underset{\mathit{\theta }}{\sup }\left|{\tilde{S}}_n(\mathit{\theta })-S_n(\mathit{\theta })\right|+2\underset{\mathit{\theta }}{\sup }\left|S_n(\mathit{\theta })-S(\mathit{\theta })\right| \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\to 0,\hspace{0.17em}\text{a}.\text{s}. \end{gathered}$$

Thus, $\rho \left({\widehat{\theta }}_n,{\mathit{\theta }}_0\right)\to 0$ a.s., under $p_{{\mathit{\theta }}_0}$.

Proof of Theorem 2. We apply Theorem 3.4.1 in van der Vaart and Wellner (1996) to obtain the rate of convergence.

First, let θ_n0 be the map point of θ₀ in Θ_n in the proof of Theorem 1, and define ${\mathit{\theta }}_{n0}{\mapsto }_{\rho 1}\left(\mathit{\theta },{\mathit{\theta }}_{n0}\right)$ be a map from Θ_n to [0, ∞) as ${\rho }_1^2\left(\mathit{\theta },{\mathit{\theta }}_{n0}\right)=S(\mathit{\theta })-S\left({\mathit{\theta }}_{n0}\right)$ Choose δ_n = ρ(θ₀, θ_n0). For δ_n < δ < ∞, denote Ω = {θ : θ ∈ Θ_n, δ/2 < ρ(θ, θ_n0) ≤ δ}. From the definition of ρ₁, we have ${\sup }_{\Omega }S\left({\mathit{\theta }}_{n0}\right)-S(\mathit{\theta })\le -\frac{{\delta }^2}{4}$ Let ${\Xi }_n$ be the set $\left\{S_n(\mathit{\theta })-S\left({\mathit{\theta }}_{n0}\right):\mathit{\theta }\in {\Theta }_n\right\}$ and $\tilde{J}\left(\delta ,L_2\left(P\right),{\Xi }_n\right)$ be the L₂(P)-norm bracketing integral of Θ_n. Similar to the proof of Theorem 4.2 of Xue, Miao, and Wu (2010), we have $\tilde{J}\left(\delta ,L_2\left(P\right),{\Xi }_n\right)\le C{r}^{\frac{1}{2}}\delta$. Let

$${\varphi }_n\left(\delta \right)=\tilde{J}\left(\delta ,L_2\left(P\right),{\Xi }_n\right)\left(1+\tilde{J}\left(\delta ,L_2\left(P\right),{\Xi }_n\right)/\left({\delta }^2\sqrt{n}\right)\right)=r^{\frac{1}{2}}\delta +r/\sqrt{n}.$$

Obviously, ϕ_n(δ)/δ^1+ι is a decreasing function in δ for 0 < τ < 1. Then by Lemma 3.4.2 in Van der Vaart and Wellner (1996), we have $E_0\left[{\sup }_{\Omega }\left(S_n-S\right)\left(\mathit{\theta }-{\mathit{\theta }}_{n0}\right)\right]\underset{\_}{\prec }{\varphi }_n\left(\delta \right)/\sqrt{n}$.

Next, we claim that ${\sup }_{\Omega }\left[{\tilde{S}}_n(\mathit{\theta })-S_n\left(\theta \right)\right]=o_p\left(n^{-\frac{1}{2}}\right)$ From (10), it is sufficient to prove $\frac{1}{m}{{\sum }_{j=1}^m{\sum }_{k=1}^{\vartheta }{w}_k\left(s_j\right)\left[{\widehat{\mathit{\text{X}}}}_k^{\prime }\left(s_j\right)-{\mathit{\text{X}}}_k^{\prime }\left(s_j\right)\right]}^2-o_p\left(n^{-\frac{1}{2}}\right)$ and ${\sup }_{\Omega }\frac{1}{m}{\sum }_{j=1}^m{\sum }_{k=1}^{\vartheta }{\sum }_{\iota =1}^{\vartheta }{w}_k\left(s_j\right){\left(\frac{\partial F_k\left\{\mathit{\text{X}},\mathit{\theta }\right\}}{\partial {\mathit{\text{X}}}_{\iota }}\right)}^2{\left[{\widehat{\mathit{\text{X}}}}_{\iota }\left(s_j\right)-{\mathit{\text{X}}}_{\iota }\left(s_j\right)\right]}^2=o_p\left(n^{-\frac{1}{2}}\right)$. In fact, for fixed $\widehat{\mathit{\text{X}}}\left(t\right)$ and θ, from the numerical error results on the Trapezoidal Rule integration, we have that

$$\begin{gathered} \frac{1}{m}\sum _{j=1}^m\sum _{k=1}^{\vartheta }\sum _{\iota =1}^{\vartheta }{w}_k\left(s_j\right)\underset{\Omega }{\sup }\left[\frac{\partial F_k\left\{\mathit{\text{X}},\mathit{\theta }\right\}}{\partial {\mathit{\text{X}}}_{\iota }}\right][{\widehat{\mathit{\text{X}}}}_{\iota }\left(s_j\right)-{\mathit{\text{X}}}_{\iota }\left(s_j\right)] \\ =\sum _{k=1}^{\vartheta }\sum _{\iota =1}^{\vartheta }{\int }_{t_0}^T{w}_k\left(t\right)\rho \left(t\right)\underset{\Omega }{\sup }[\frac{\partial F_k\left\{\mathit{\text{X}},\mathit{\theta }\right\}}{\partial {\mathit{\text{X}}}_{\iota }}][{\widehat{\mathit{\text{X}}}}_{\iota }\left(t\right)-{\mathit{\text{X}}}_{\iota }\left(t\right)dt+O_p\left(1/m\right)]. \end{gathered}$$

Since $m\ge n,O_p\left(1/m\right)=O_p\left(n^{-\frac{1}{2}}\right)$. Denote $g\left(t\right)=w_k\left(t\right)\rho \left(t\right){\sup }_{\Omega }\left[\frac{\partial F_k\left\{\mathit{\text{X}},\mathit{\theta }\right\}}{\partial {\mathit{\text{X}}}_{\iota }}\right]$. Under Assumption A, by Lemma 6 in Wu, Xue and Kumar (2012), ${\int }_{t_0}^{T}g\left(t\right)\left[{\widehat{\mathit{\text{X}}}}_{\iota }\left(t\right)-{\mathit{\text{X}}}_{\iota }\left(t\right)\right]dt$ is asymptotically normal with mean as

$${\mu }_1={\int }_{t_0}^{T}g\left(t\right)\text{E}\left[{\widehat{\mathit{\text{X}}}}_{\iota }\left(t\right)-{\mathit{\text{X}}}_{\iota }\left(t\right)\right]dt=O_p\left(k^{\upsilon +1}\right)=o_p\left(n^{-1/2}\right),$$

and variance as

$${\Sigma }_1={\int }_{t_0}^T{\int }_{t_0}^{T}g\left(s\right)\text{Cov}[{\widehat{\mathit{\text{X}}}}_{\iota }\left(s\right)-{\mathit{\text{X}}}_{\iota }\left(s\right),{\widehat{\mathit{\text{X}}}}_{\iota }\left(t\right)-{\mathit{\text{X}}}_{\iota }\left(t\right)]g\left(t\right)dsdt=O_p\left(1/n\right).$$

Thus $\frac{\sqrt{n}}{m}{\sum }_{j=1}^m{\sum }_{k=1}^{\vartheta }{\sum }_{\iota =1}^{\vartheta }{w}_k\left(s_j\right){\sup }_{\Omega }\left[\frac{\partial F_k\left\{\mathit{\text{X}},\mathit{\theta }\right\}}{\partial {\mathit{\text{X}}}_{\iota }}\right]\left[{\widehat{\mathit{\text{X}}}}_{\iota }\left(s_j\right)-{\mathit{\text{X}}}_{\iota }\left(s_j\right)\right]$ is asymptotically normal with mean 0 and variance of order O_p(1). Using the Delta method (van der Vaart and Wellner 1996, p.377), we have that

$$\frac{\sqrt{n}}{m}\sum _{j=1}^m\sum _{k=1}^{\vartheta }{\sum _{\iota =1}^{\vartheta }{w}_k\left(s_j\right)\underset{\Omega }{\sup }\left[\frac{\partial F_k\left\{\mathit{\text{X}},\mathit{\theta }\right\}}{\partial {\mathit{\text{X}}}_{\iota }}\right]}^2{[{\widehat{\mathit{\text{X}}}}_{\iota }\left(s_j\right)-{\mathit{\text{X}}}_{\iota }\left(s_j\right)]}^2$$

is also asymptotically normal with mean 0 and variance of order O_p(1). So

$$\frac{1}{m}\sum _{j=1}^m\sum _{k=1}^{\vartheta }{\sum _{\iota =1}^{\vartheta }{w}_k\left(s_j\right)\underset{\Omega }{\sup }\left[\frac{\partial F_k\left\{\mathbf{X},\mathbf{\theta }\right\}}{\partial X_{\iota }}\right]}^2{[{\widehat{X}}_{\iota }\left(s_j\right)-X\left(s_j\right)]}^2=o_p\left(n^{-\frac{1}{2}}\right)$$

Similarly, for fixed $\widehat{X}\prime \left(t\right)$, we have

$$\frac{1}{m}\sum _{j=1}^m\sum _{k=1}^{\vartheta }{w}_k\left(s_j\right)\left[{{\widehat{X}}^{\prime }}_k\left(s_j\right)-{X^{\prime }}_k\left(s_j\right)\right]=\sum _{k=1}^{\vartheta }{\int }_{t_0}^T{w}_k\left(t\right)\rho \left(t\right)\left[{\widehat{X^{\prime }}}_k\left(t\right)-{X^{\prime }}_k\left(t\right)\right]dt+o_p\left(m^{-\frac{1}{2}}\right).$$

Further, applying integration by parts and Assumption D1 w_k(t₀) = w_k(T) = 0 for k = 1, · · ·, ϑ, we have

$$\begin{gathered} \sum _{k=1}^{\vartheta }{\int }_{t_0}^T{w}_k\left(t\right)\rho \left(t\right)\left[{\widehat{\mathit{\text{X}}}}_k^{\prime }\left(t\right)-{\mathit{\text{X}}}_k^{\prime }\left(t\right)\right]dt \\ =\sum _{k=1}^{\vartheta }\left(w_k\left(t\right)\rho \left(t\right)\left[{\widehat{\mathit{\text{X}}}}_k\left(t\right)-{\mathit{\text{X}}}_k\left(t\right)\right]\right){|}_{t_0}^T-\sum _{k=1}^{\vartheta }{\int }_{t_0}^T\frac{d}{dt}\left[w_k\left(t\right)\rho \left(t\right)\right]\left[{\widehat{\mathit{\text{X}}}}_k\left(t\right)-{\mathit{\text{X}}}_k\left(t_j\right)\right]dt \\ =-\sum _{k=1}^{\vartheta }{\int }_{t_0}^T\frac{d}{dt}\left[w_k\left(t\right)\rho \left(t\right)\right][{\widehat{\mathit{\text{X}}}}_k\left(t\right)-{\mathit{\text{X}}}_k\left(t\right)]dt. \end{gathered}$$

Then by Lemma 6 in Wu, Xue and Kumar (2012), $\frac{\sqrt{n}}{m}{\sum }_{j=1}^m{\sum }_{k=1}^{\vartheta }{w}_k\left(s_j\right)[{\widehat{\mathit{\text{X}}}}_k^{\prime }\left(s_j\right)-{\mathit{\text{X}}}_k^{\prime }\left(s_j\right)]$ is asymptotically normal with mean 0 and variance of order O_p(1). Using the Delta method again, it follows that $\frac{\sqrt{n}}{m}{\sum }_{j=1}^m{{\sum }_{k=1}^{\vartheta }{w}_k\left(s_j\right)[{\widehat{\mathit{\text{X}}}}_k^{\prime }\left(s_j\right)-{\mathit{\text{X}}}_k^{\prime }\left(s_j\right)]}^2$ is also asymptotically normal with mean 0 and variance of order O_p(1). So $\frac{1}{m}{\sum }_{j=1}^m{{\sum }_{k=1}^{\vartheta }{w}_k\left(s_j\right)[{\widehat{\mathit{\text{X}}}}_k^{\prime }\left(s_j\right)-{\mathit{\text{X}}}_k^{\prime }\left(s_j\right)]}^2=o_p\left(n^{-\frac{1}{2}}\right)$. Thus the claim holds and we have $E_0[{\sup }_{\Omega }\left({\tilde{S}}_n-S\right)\left(\mathit{\theta }-{\mathit{\theta }}_{n0}\right)]\underset{\_}{\prec }{\varphi }_n\left(\delta \right)/\sqrt{n}+o_p\left(1\right)/\sqrt{n}\simeq {\varphi }_n/\sqrt{n}$. It follows that $E_0[{\sup }_{\Omega }\sqrt{n}\left({\tilde{S}}_n-S\right)\left(\mathit{\theta }-{\mathit{\theta }}_{n0}\right)]\underset{\_}{\prec }{\varphi }_n\left(\delta \right)$

Thus the conditions of Theorem 3.4.1 in Van der Vaart and Wellner (1996) are satisfied for the δ_n, ρ₁ and ϕ_n(δ) above. Therefore we have $-r_n^2{\rho }_1\left({\widehat{\mathit{\theta }}}_n,{\mathit{\theta }}_{n0}\right)=O_p\left(1\right)$, where r_n satisfies $r_n^2{\varphi }_n\left(\frac{1}{r_n}\right)\le \sqrt{n}$. It follows that $r_n=r^{-\frac{1}{2}}{n}^{\frac{1}{2}}=n^{\left(1-\varpi \right)/2}$. Thus ${\rho }_1\left({\widehat{\mathit{\theta }}}_n,{\mathit{\theta }}_{n0}\right)=O_p\left(n^{-\left(1-\varpi \right)/2}\right)$. Further, by similar arguments in the proof of Theorem 4.2 in Xue, Miao, and Wu (2010), we have that $\rho \left({\widehat{\mathit{\theta }}}_n,{\mathit{\theta }}_{n0}\right)=O_p\left(n^{-\varpi \varrho }+n^{-\left(1-\varpi \right)/2}\right)$.

Proof of Theorem 3: To prove Theorem 3, we cannot directly use the Huber-Pollard Ztheorem (see Theorem 3.3.1 in van der Vaart and Wellner 1996), since in the second stage of our proposed estimation method, $\widehat{\mathit{\text{X}}}\left(t\right)$ and ${\widehat{\mathit{\text{X}}}}^{\prime }\left(t\right)$ are nonparametric plug-in estimators from the first stage. We extend the original Huber-Pollard Z-theorem to such two-stage case.

Some definitions are needed to prove Theorem 3. For any fixed η ∈$\mathcal{A}$, let ${\mathcal{A}}_0$= {η_h(.) : h in a neighborhood of 0 ∈ $\mathcal{R}$ } be a smooth curve in $\mathcal{A}$ running through η₀ at h = 0, that is η_h=0(t) = η₀(t). Denote $\frac{\partial }{\partial h}{\eta }_h\left(t\right){|}_{h=0}=a\left(t\right)$ and the space generated by such a(t) as ϒ. For any a ∈ ϒ we define

$$\begin{gathered} {\dot{S}}_1(\mathit{\theta })=\frac{\partial S(\mathit{\theta })}{\partial \beta }=-2\sum _{k=1}^{\vartheta }\text{E}\left(w_k\left(t\right)\left[{\mathit{\text{X}}}_k^{\prime }\left(t\right)-F_k\left\{\mathit{\text{X}}\left(t\right),\mathit{\theta }\right\}\right]\frac{\partial F_k\left\{\mathit{\text{X}}\left(t\right),\mathit{\theta }\right\}}{\partial \beta }\right), \\ \begin{array}{l}{\dot{S}}_2(\mathit{\theta })\left[a\right]=\frac{\partial S\left(\beta ,{\eta }_h\right)}{\partial h}{|}_{h=0}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=-2\sum _{k=1}^{\vartheta }\text{E}\left(w_k\left(t\right)\left[{\mathit{\text{X}}}_k^{\prime }\left(t\right)-F_k\left\{\mathit{\text{X}}\left(t\right),\mathit{\theta }\right\}\right]\frac{\partial F_k\left\{\mathit{\text{X}}\left(t\right),\mathit{\theta }\right\}}{\partial \eta }a\left(t\right)\right).\end{array} \end{gathered}$$

We set

$$\begin{array}{l}{\dot{S}}_{11}(\mathit{\theta })=\frac{{\partial }^{2}S(\mathit{\theta })}{\partial \beta \partial {\beta }^T}\\ \hspace{1em}\hspace{1em}\hspace{1em}=2\sum _{k=1}^{\vartheta }\text{E}\left(w_k\left(t\right)\frac{\partial F_k\left\{\mathit{\text{X}}\left(t\right),\mathit{\theta }\right\}}{\partial \beta }\frac{\partial F_k\left\{\mathit{\text{X}}\left(t\right),\mathit{\theta }\right\}}{\partial {\beta }^T}\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-2\sum _{k=1}^{\vartheta }\text{E}\left(w_k\left(t\right)\left[{\mathit{\text{X}}}_k^{\prime }\left(t\right)-F_k\left\{\mathit{\text{X}}\left(t\right),\mathit{\theta }\right\}\right]\frac{{\partial }^2{F}_k\left\{\mathit{\text{X}}\left(t\right),\mathit{\theta }\right\}}{\partial \beta \partial {\beta }^T}\right),\end{array}$$

$$\begin{array}{l}{\dot{S}}_{12}(\mathit{\theta })\left[a\right]={\dot{S}}_{21}^T(\mathit{\theta })\left[a\right]=\frac{{\partial }^{2}S\left(\beta ,{\eta }_h\right)}{\partial \beta \partial h}{|}_{h=0}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=2\sum _{k=1}^{\vartheta }\text{E}\left(w_k\left(t\right)\frac{\partial F_k\left\{\mathit{\text{X}}\left(t\right),\mathit{\theta }\right\}}{\partial \beta }\frac{\partial F_k\left\{\mathit{\text{X}}\left(t\right),\mathit{\theta }\right\}}{\partial \eta }a\left(t\right)\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-2\sum _{k=1}^{\vartheta }\text{E}\left(w_k\left(t\right)\left[{\mathit{\text{X}}}_k^{\prime }\left(t\right)-F_k\left\{\mathit{\text{X}}\left(t\right),\mathit{\theta }\right\}\right]\frac{{\partial }^2{F}_k\left\{\mathit{\text{X}}\left(t\right),\mathit{\theta }\right\}}{\partial \beta \partial \eta }a\left(t\right)\right),\end{array}$$

$$\begin{array}{l}{\dot{S}}_{22}(\mathit{\theta })\left[a_1,a_2\right]=\frac{{\partial }^{2}S\left(\beta ,{\eta }_{h_j}\right)}{\partial h_1\partial h_2}{|}_{h_1=0,h_2=0}\equiv \frac{\partial }{\partial h_2}{\dot{S}}_2\left(\beta ,{\eta }_{h_2}\right)\left[a_1\right]{|}_{h_2=0}\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=2\sum _{k=1}^{\vartheta }\text{E}\left(w_k\left(t\right){\left[\frac{\partial F_k\left\{\mathit{\text{X}}\left(t\right),\mathit{\theta }\right\}}{\partial \eta }\right]}^2{a}_1\left(t\right)a_2\left(t\right)\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}-2\sum _{k=1}^{\vartheta }\text{E}\left(w_k\left(t\right)\left[{\mathit{\text{X}}}_k^{\prime }\left(t\right)-F_k\left\{\mathit{\text{X}}\left(t\right),\mathit{\theta }\right\}\right]\frac{{\partial }^2{F}_k\left\{\mathit{\text{X}}\left(t\right),\mathit{\theta }\right\}}{{\partial }^2\eta }{a}_1\left(t\right)a_2\left(t\right)\right),\end{array}$$

where a₁(t),a₂(t) ∈ ϒ. We also define ${\tilde{S}}_{1n}(\mathit{\theta })=\frac{\partial {\tilde{S}}_n(\mathit{\theta })}{\partial \beta },{\tilde{S}}_{2n}(\mathit{\theta })\left[a\right]=\frac{\partial {\tilde{S}}_n\left(\beta ,{\eta }_h\right)}{\partial h}{|}_{h=0},S_{1n}(\mathit{\theta })=\frac{\partial S_n(\mathit{\theta })}{\partial \beta },$ and $S_{2n}(\mathit{\theta })\left[a\right]=\frac{\partial {\tilde{S}}_n\left(\beta ,{\eta }_h\right)}{\partial h}{|}_{h=0}$. Further, for a = (a₁,···, a_d)^T ∈ ϒ^d, where a_j ∈ ϒ for j = 1,···, d, we denote ${\dot{S}}_2(\mathit{\theta })\left[a\right]={\left({\dot{S}}_2(\mathit{\theta })\left[a_1\right],\cdots ,{\dot{S}}_2(\mathit{\theta })\left[a_d\right]\right)}^T,{\dot{S}}_{12}(\mathit{\theta })\left[a\right]={\left({\dot{S}}_{12}(\mathit{\theta })\left[a_1\right],\cdots ,{\dot{S}}_{12}(\mathit{\theta })\left[a_d\right]\right)}^T,{\dot{S}}_{21}(\mathit{\theta })\left[a\right]={\left({\dot{S}}_{21}(\mathit{\theta })\left[a_1\right],\cdots ,{\dot{S}}_{21}(\mathit{\theta })\left[a_d\right]\right)}^T,{\dot{S}}_{22}(\mathit{\theta })\left[\mathit{\text{a}},a\right]={\left({\dot{S}}_{22}(\mathit{\theta })\left[a_1,a\right],\cdots ,{\dot{S}}_{22}(\mathit{\theta })\left[a_d,a\right]\right)}^T,{\tilde{S}}_{2n}(\mathit{\theta })\left[a\right]={\left({\dot{S}}_2(\mathit{\theta })\left[a_1\right],\cdots ,{\tilde{S}}_{2n}(\mathit{\theta })\left[a_d\right]\right)}^T$ and $S_{2n}(\mathit{\theta })\left[\mathit{\text{a}}\right]={\left(S_{2n}(\mathit{\theta })\left[a_1\right],\cdots ,S_{2n}(\mathit{\theta })\left[a_d\right]\right)}^T$ Then we can obtain the following results E1–E7.

• E1.From Theorem 1 and Theorem 2, we have that $\Vert {\widehat{\beta }}_n-{\beta }_0|\hspace{0.17em}=\hspace{0.17em}{o}_p\left(1\right),\hspace{0.17em}{\Vert {\widehat{\eta }}_n-{\eta }_0\Vert }_2\hspace{0.17em}=\hspace{0.17em}{O}_p\left(n^{-\zeta }\right)$, where $O_p\left(n^{-\zeta }\right)$ is just the rate of convergence in Theorem 2. Moreover, $\zeta >\frac{1}{4}$.
• E2.Ṡ₁(θ₀) = 0 and Ṡ₂(θ₀)[a] = 0 for all a ∈ ϒ.
• E3.
  Following the arguments in Ma and Kosorok (2005) and Wellner and Zhang (2007), we need to find a least favorable ${\mathit{a}}^{*}\left(t\right)={\left\{a_1^{*}\left(t\right),\dots ,a_d^{*}\left(t\right)\right\}}^T\in {\Upsilon }^d$, where $a\in \Upsilon$. for j = 1,···, d, such that Ṡ₁₂(θ₀)[a] − Ṡ₂₂(θ₀)[a^∗,a] = 0, for all a ∈ ϒ. Moreover, the matrix A = −Ṡ₁₁(θ₀) + Ṡ₁₂(θ₀)[a^∗] is nonsingular. Some calculations yield

  $$\begin{array}{l}\hspace{1em}{\dot{S}}_{12}\left({\theta }_0\right)\left[a\right]-{\dot{S}}_{22}\left({\theta }_0\right)\left[{\mathit{a}}^{*},a\right]\\ =\hspace{0.17em}2\sum _{k=1}^{\vartheta }\text{E}\left[[w_k\left(t\right)\frac{\partial F_k\left\{\mathrm{X}\left(t\right),{\theta }_0\right\}}{\partial {\beta }_0}\frac{\partial F_k\left\{\mathrm{X}\left(t\right),{\theta }_0\right\}}{\partial {\eta }_0}a\left(t\right)\right]\\ \hspace{1em}-2\sum _{k=1}^{\vartheta }\text{E}\left[w_k\left(t\right){\left(\frac{\partial F_k\left\{\mathrm{X}\left(t\right),{\theta }_0\right\}}{\partial {\eta }_0}\right)}^2{\mathit{a}}^{*}\left(t\right)a\left(t\right)\right]\\ \hspace{1em}-2\sum _{k=1}^{\vartheta }\text{E}\left[w_k\left(t\right)\left[{\mathit{\text{X}}}_k^{\prime }\left(t\right)-F_k\left\{\mathrm{X}\left(t\right),{\theta }_0\right\}\right]\frac{{\partial }^2{F}_k\left\{\mathrm{X}\left(t\right),{\theta }_0\right\}}{\partial {\beta }_0\partial {\eta }_0}a\left(t\right)\right]\\ \hspace{1em}+2\sum _{k=1}^{\vartheta }\text{E}\left[w_k\left(t\right)\left[{\mathit{\text{X}}}_k^{\prime }\left(t\right)-F_k\left\{\mathrm{X}\left(t\right),{\theta }_0\right\}\right]\frac{{\partial }^2{F}_k\left\{\mathrm{X}\left(t\right),{\theta }_0\right\}}{{\partial }^2\eta }{\mathit{a}}^{*}\left(t\right)a\left(t\right)\right],\\ =\hspace{0.17em}2\sum _{k=1}^{\vartheta }\text{E}\left[w_k\left(t\right)\frac{\partial F_k\left\{\mathrm{X}\left(t\right),{\theta }_0\right\}}{\partial {\beta }_0}\frac{\partial F_k\left\{\mathrm{X}\left(t\right),{\theta }_0\right\}}{\partial {\eta }_0}a\left(t\right)\right]\\ \hspace{1em}-2\sum _{k=1}^{\vartheta }\text{E}\left[w_k\left(t\right){\left(\frac{\partial F_k\left\{\mathrm{X}\left(t\right),{\theta }_0\right\}}{\partial {\eta }_0}\right)}^2{\mathit{a}}^{*}\left(t\right)a\left(t\right)\right].\end{array}$$

  Therefore, an obvious choice of a^∗ is

  $${\mathit{a}}^{*}\left(t\right)=\frac{{\sum }_{k=1}^{\vartheta }{w}_k\left(t\right)\frac{\partial F_k\left\{\mathrm{X}\left(t\right),{\theta }_0\right\}}{\partial {\beta }_0}\frac{\partial F_k\left\{\mathrm{X}\left(t\right),{\theta }_0\right\}}{\partial {\eta }_0}}{{\sum }_{k=1}^{\vartheta }{w}_k\left(t\right){\left[\frac{\partial F_k\left\{\mathrm{X}\left(t\right),{\theta }_0\right\}}{\partial {\eta }_0}\right]}^2}.$$

  Moreover, for ϑ ≥ 2,

  $$\begin{gathered} \mathrm{A}\hspace{0.17em}=\hspace{0.17em}-2\sum _{k=1}^{\vartheta }\text{E}\left[w_k\left(t\right)\frac{\partial F_k\left\{\mathit{X}\left(t\right),{\theta }_0\right\}}{\partial {\beta }_0}\frac{\partial F_k\left\{\mathit{X}\left(t\right),{\theta }_0\right\}}{\partial {\beta }_0^T}\right] \\ \hspace{1em}\hspace{1em}+2\sum _{k=1}^{\vartheta }\text{E}\left[w_k\left(t\right)\frac{\partial F_k\left\{\mathit{X}\left(t\right),{\theta }_0\right\}}{\partial {\beta }_0}\frac{\partial F_k\left\{\mathit{X}\left(t\right),{\theta }_0\right\}}{\partial {\eta }_0}{\mathit{a}}^{*}\left(t\right)\right] \end{gathered}$$ (11)

  is nonsingular. Otherwise, A = 0 for ϑ = 1.
• E4.By the similar arguments of Condition (i) of the proof of Theorem 4 in Xue, Lam and Li (2004), it follows that the estimator ${\widehat{\theta }}_n$ satisfies ${\tilde{S}}_{1n}\left({\widehat{\theta }}_n\right)=0$ and ${\tilde{S}}_{2n}\left({\widehat{\theta }}_n\right)\left[{\mathit{a}}^{*}\right]=o_p\left(n^{-1/2}\right)$
• E5.
  For any δ_n ↓ 0 and C > 0, setting $\mathcal{F}\hspace{0.17em}=\hspace{0.17em}\left\{\theta \hspace{0.17em}\text{:}\hspace{0.17em}\Vert \beta -\beta \Vert \hspace{0.17em}\le \hspace{0.17em}{\delta }_n,{\Vert \eta -{\eta }_0\Vert }_2\hspace{0.17em}\le \hspace{0.17em}{C}_n{}^{-\zeta }\right\}$, we claim

  $$\begin{gathered} \underset{\mathcal{F}}{\text{sup}}\left|\sqrt{n}\left({\tilde{S}}_{1n}\hspace{0.17em}-\hspace{0.17em}{\dot{S}}_1\right)\left(\theta \right)\hspace{0.17em}-\hspace{0.17em}\sqrt{n}\left({\tilde{S}}_{1n}\hspace{0.17em}-\hspace{0.17em}{\dot{S}}_1\right)\left({\theta }_0\right)\right|\hspace{0.17em}=\hspace{0.17em}{o}_p\left(1\right), \\ \underset{\mathcal{F}}{\text{sup}}\left|\sqrt{n}\left({\tilde{S}}_{2n}\hspace{0.17em}-\hspace{0.17em}{\dot{S}}_2\right)\left(\theta \right)\left[{\mathit{\text{a}}}^{*}\right]\hspace{0.17em}-\hspace{0.17em}\sqrt{n}\left({\tilde{S}}_{2n}\hspace{0.17em}-\hspace{0.17em}{\dot{S}}_2\right)\left({\theta }_0\right)\left[{\mathit{\text{a}}}^{*}\right]\right|\hspace{0.17em}=\hspace{0.17em}{o}_p\left(1\right). \end{gathered}$$

  In fact, similar to C3 of Theorem 4 in Xue, Lam, and Li (2004), it follows that

  $$\begin{gathered} \underset{\mathcal{F}}{\text{sup}}\left|\sqrt{n}\left(S_{1n}\hspace{0.17em}-\hspace{0.17em}{\dot{S}}_1\right)\left(\theta \right)\hspace{0.17em}-\hspace{0.17em}\sqrt{n}\left(S_{1n}\hspace{0.17em}-\hspace{0.17em}{\dot{S}}_1\right)\left({\theta }_0\right)\right|\hspace{0.17em}=\hspace{0.17em}{o}_p\left(1\right), \\ \underset{\mathcal{F}}{\text{sup}}\left|\sqrt{n}\left(S_{2n}\hspace{0.17em}-\hspace{0.17em}{\dot{S}}_2\right)\left(\theta \right)\left[{\mathit{\text{a}}}^{*}\right]\hspace{0.17em}-\hspace{0.17em}\sqrt{n}\left(S_{2n}\hspace{0.17em}-\hspace{0.17em}{\dot{S}}_2\right)\left({\theta }_0\right)\left[{\mathit{\text{a}}}^{*}\right]\right|\hspace{0.17em}=\hspace{0.17em}{o}_p\left(1\right). \end{gathered}$$

  Moreover, similar to the proof of Theorem 2, we have

  $$\begin{gathered} \underset{\mathcal{F}}{\text{sup}}\left|\sqrt{n}\left({\tilde{S}}_{1n}\hspace{0.17em}-\hspace{0.17em}{S}_1\right)\left(\theta \right)\hspace{0.17em}-\hspace{0.17em}\sqrt{n}\left({\tilde{S}}_{1n}\hspace{0.17em}-\hspace{0.17em}{S}_1\right)\left({\theta }_0\right)\right|\hspace{0.17em}=\hspace{0.17em}{o}_p\left(1\right), \\ \underset{\mathcal{F}}{\text{sup}}\left|\sqrt{n}\left({\tilde{S}}_{2n}\hspace{0.17em}-\hspace{0.17em}{S}_2\right)\left(\theta \right)\left[{\mathit{\text{a}}}^{*}\right]\hspace{0.17em}-\hspace{0.17em}\sqrt{n}\left({\tilde{S}}_{2n}\hspace{0.17em}-\hspace{0.17em}{S}_2\right)\left({\theta }_0\right)\left[{\mathit{\text{a}}}^{*}\right]\right|\hspace{0.17em}=\hspace{0.17em}{o}_p\left(1\right). \end{gathered}$$

  Combining them together, it follows that the claim holds.
• E6.
  By the Taylor expansion, for θ ∈ F, it follows that

  $$\left|{\dot{S}}_1\left(\theta \right)\hspace{0.17em}-\hspace{0.17em}{\dot{S}}_1\left({\theta }_0\right)\hspace{0.17em}-\hspace{0.17em}{\dot{S}}_{11}\left({\theta }_0\right)\left(\beta \hspace{0.17em}-\hspace{0.17em}{\beta }_0\right)\hspace{0.17em}-\hspace{0.17em}{\dot{S}}_{12}\left({\theta }_0\right)\left[\eta \hspace{0.17em}-\hspace{0.17em}{\eta }_0\right]\right|\hspace{0.17em}=\hspace{0.17em}o\left(\Vert \beta \hspace{0.17em}-\hspace{0.17em}{\beta }_0\Vert \right)\hspace{0.17em}+\hspace{0.17em}O\left({\Vert \eta \hspace{0.17em}-\hspace{0.17em}{\eta }_0\Vert }_2^2\right),$$

  and

  $$\begin{array}{l}\left|{\dot{S}}_2\left(\theta \right)\left[{\mathit{\text{a}}}^{*}\right]\hspace{0.17em}-\hspace{0.17em}{\dot{S}}_2\left({\theta }_0\right)\left[{\mathit{\text{a}}}^{*}\right]\hspace{0.17em}-\hspace{0.17em}{\dot{S}}_{21}\left({\theta }_0\right)\left[{\mathit{\text{a}}}^{*}\right]\left(\beta -{\beta }_0\right)\hspace{0.17em}-\hspace{0.17em}{\dot{S}}_{22}\left({\theta }_0\right)\left[{\mathit{\text{a}}}^{*},\eta \hspace{0.17em}-\hspace{0.17em}{\eta }_0\right]\right|\hspace{0.17em}=\hspace{0.17em}o\left(\Vert \beta -{\beta }_0\Vert \right)\hspace{0.17em}+\hspace{0.17em}\\ O\left({\Vert \eta \hspace{0.17em}-\hspace{0.17em}{\eta }_0\Vert }_2^2\right).\end{array}$$
• E7.
  $\sqrt{n}\left({\tilde{S}}_{1n}\left({\theta }_0\right)-{\tilde{S}}_{2n}\left({\theta }_0\right)\left[{\mathit{\text{a}}}^{*}\right]\right)$ is asymptotically normal. In fact, by the Taylor expansion, we have

  $$\begin{array}{l}\hspace{1em}{\tilde{S}}_{1n}\left({\theta }_0\right)-{\tilde{S}}_{2n}\left({\theta }_0\right)\left[{\mathit{\text{a}}}^{*}\right]\\ =\hspace{0.17em}-\frac{2}{m}\sum _{j=1}^m\sum _{k=1}^{\vartheta }{w}_k\left(s_j\right)\left(\frac{\partial F_k\left\{\widehat{\mathit{\text{X}}}\left(s_j\right),{\theta }_0\right\}}{\partial \beta }\hspace{0.17em}-\hspace{0.17em}\frac{\partial F_k\left\{\widehat{\mathit{\text{X}}}\left(s_j\right),{\theta }_0\right\}}{\partial \eta }\left[{\mathit{\text{a}}}^{*}\right]\right)\\ \hspace{1em}\times \left[{\widehat{\mathit{\text{X}}}}_k^{\prime }\left(s_j\right)-F_k\left\{\widehat{\mathit{\text{X}}}\left(s_j\right),{\theta }_0\right\}\right]\\ =\hspace{0.17em}-\frac{2}{m}\sum _{j=1}^m\sum _{k=1}^{\vartheta }{w}_k\left(s_j\right)\left(\frac{\partial F_k\left\{\widehat{\mathit{\text{X}}}\left(s_j\right),{\theta }_0\right\}}{\partial \beta }\hspace{0.17em}-\hspace{0.17em}\frac{\partial F_k\left\{\widehat{\mathit{\text{X}}}\left(s_j\right),{\theta }_0\right\}}{\partial \eta }\left[{\mathit{\text{a}}}^{*}\right]\right)\\ \hspace{1em}\times \left[{\widehat{\mathit{\text{X}}}}_k^{\prime }\left(s_j\right)-{\widehat{\mathit{\text{X}}}}_k^{\prime }\left(s_j\right)+F_k\left\{\mathit{\text{X}}\left(s_j\right),{\theta }_0\right\}\hspace{0.17em}-\hspace{0.17em}{F}_k\left\{\widehat{\mathit{\text{X}}}\left(s_j\right),{\theta }_0\right\}\right]\\ =\hspace{0.17em}-\frac{2}{m}\sum _{j=1}^m\sum _{k=1}^{\vartheta }{w}_k\left(s_j\right)\left(\frac{\partial F_k\left\{\widehat{\mathit{\text{X}}}\left(s_j\right),{\theta }_0\right\}}{\partial \beta }\hspace{0.17em}-\hspace{0.17em}\frac{\partial F_k\left\{\widehat{\mathit{\text{X}}}\left(s_j\right),{\theta }_0\right\}}{\partial \eta }\left[{\mathit{\text{a}}}^{*}\right]\right)\\ \hspace{1em}\times (-\sum _{\iota =1}^{\vartheta }\frac{\partial F_k\left\{\mathit{\text{X}},{\theta }_0\right\}}{\partial {\mathit{\text{X}}}_{\iota }}|{}_{{\mathit{\text{X}}}_1={\mathit{\text{X}}}_1\left(s_j\right),\cdots ,{\mathit{\text{X}}}_{\iota }={\widetilde{\mathit{\text{X}}}}_{\iota ,j},\cdots ,{\mathit{\text{X}}}_{\vartheta }={\mathit{\text{X}}}_{\vartheta }\left(s_j\right)}[{\widehat{\mathit{\text{X}}}}_{\iota }\left(s_j\right)\hspace{0.17em}-\hspace{0.17em}{\mathit{\text{X}}}_{\iota }\left(s_j\right)]\\ \hspace{1em}+{\widehat{\mathit{\text{X}}}}_k^{\prime }\left(s_j\right)-{\mathit{\text{X}}}_k^{\prime }\left(s_j\right))\end{array}$$

  with ${\widetilde{\mathit{\text{X}}}}_{\iota ,j}$ being some point between X_ι(s_j) and ${\widehat{\mathit{\text{X}}}}_{\iota }\left(s_j\right)$. Further, denoting $A_k\left(t\right)\hspace{0.17em}=\hspace{0.17em}\frac{\partial F_k\left\{\mathit{\text{X}}\left(t\right),{\theta }_0\right\}}{\partial \beta }\hspace{0.17em}-\hspace{0.17em}\frac{\partial F_k\left\{\mathit{\text{X}}\left(t\right),{\theta }_0\right\}}{\partial \eta }\left[{\mathit{\text{a}}}^{*}\right]$ and applying the numerical error results on the Trapezoidal Rule integration and integration by parts, we have

  $$\begin{array}{l}\hspace{1em}{\tilde{S}}_{1n}\left({\theta }_0\right)\hspace{0.17em}-\hspace{0.17em}{\tilde{S}}_{2n}\left({\theta }_0\right)[{\mathit{\text{a}}}^{*}]\\ =\hspace{0.17em}-2\sum _{k=1}^{\vartheta }{\int }_{t_0}^T{w}_k\left(t\right)\rho \left(t\right)A_k\left(t\right)[{\widehat{\mathit{\text{X}}}}_k^{\prime }\left(t\right)\hspace{0.17em}-\hspace{0.17em}{\mathit{\text{X}}}_k^{\prime }\left(t\right)]dt\\ +\hspace{0.17em}2\sum _{k=1}^{\vartheta }\sum _{\iota =1}^{\vartheta }{\int }_{t_0}^T{w}_k\left(t\right)\rho \left(t\right)A_k\left(t\right)\frac{\partial F_k\left\{\mathit{\text{X}},{\theta }_0\right\}}{\partial {\mathit{\text{X}}}_{\iota }}|{}_{\mathit{\text{X}}\hspace{0.17em}=\hspace{0.17em}\mathit{\text{X}}\left(t\right)}[{\widehat{\mathit{\text{X}}}}_{\iota }\left(t\right)\hspace{0.17em}-\hspace{0.17em}{\mathit{\text{X}}}_{\iota }\left(t\right)]dt\hspace{0.17em}+\hspace{0.17em}{O}_p\left(m^{-1}\right)\\ =\hspace{0.17em}-2\sum _{k=1}^{\vartheta }\left\{w_k\left(t\right)\rho \left(t\right)A_k\left(t\right)\left[{\widehat{\mathit{\text{X}}}}_k\left(t\right)\hspace{0.17em}-\hspace{0.17em}{\mathit{\text{X}}}_k\left(t\right)\right]\right\}|{}_{t_0}^T\\ +\hspace{0.17em}2\sum _{k=1}^{\vartheta }{\int }_{t_0}^T\frac{d}{dt}\left[w_k\left(t\right)\rho \left(t\right)A_k\left(t\right)\right]\hspace{0.17em}\left[{\widehat{\mathit{\text{X}}}}_k\left(t\right)-{\mathit{\text{X}}}_k\left(t\right)\right]dt\\ +\hspace{0.17em}2\sum _{k=1}^{\vartheta }\sum _{\iota =1}^{\vartheta }{\int }_{t_0}^T{w}_k\left(t\right)\rho \left(t\right)A_k\left(t\right)\frac{\partial F_k\left\{\mathit{\text{X}},{\theta }_0\right\}}{\partial {\mathit{\text{X}}}_{\iota }}|{}_{\mathit{\text{X}}\hspace{0.17em}=\hspace{0.17em}\mathit{\text{X}}\left(t\right)}\left[{\widehat{\mathit{\text{X}}}}_{\iota }\left(t\right)\hspace{0.17em}-\hspace{0.17em}{\mathit{\text{X}}}_{\iota }\left(t\right)\right]dt\hspace{0.17em}+\hspace{0.17em}{o}_p\left(n^{-\frac{1}{2}}\right).\end{array}$$

Under the assumption of w_k(t₀) = w_k(T) = 0 for k = 1,···, ϑ, we have

$${\tilde{S}}_{1n}\left({\theta }_0\right)-{\tilde{S}}_{2n}\left({\theta }_0\right)\left[{\mathit{\text{a}}}^{*}\right]$$

$$=\hspace{0.17em}2\sum _{k=1}^{\vartheta }{\int }_{t_0}^T{B}_k\left(t\right)\left[{\widehat{\mathit{\text{X}}}}_k\left(t\right)\hspace{0.17em}-\hspace{0.17em}{\mathit{\text{X}}}_k\left(t\right)\right]dt\hspace{0.17em}+\hspace{0.17em}{o}_p\left(n^{-\frac{1}{2}}\right)$$ (12)

With $B_k\left(t\right)\hspace{0.17em}=\hspace{0.17em}\frac{d}{dt}\left[w_k\left(t\right)\rho \left(t\right)A_k\left(t\right)\right]\hspace{0.17em}+\hspace{0.17em}{\sum }_{\iota =1}^{\vartheta }\left[w_{\iota }\left(t\right)\rho \left(t\right)A_{\iota }\left(t\right)\frac{\partial F_{\iota }\left\{\mathit{\text{X}},{\theta }_0\right\}}{\partial {\mathit{\text{X}}}_k}|{}_{\mathit{\text{X}}=\mathit{\text{X}}\left(t\right)}\right]$. Similar to the proof of Theorem 2, by Lemma 6 in Wu, Xue and Kumar (2012), it follows that $2\sqrt{n}{\sum }_{k=1}^{\vartheta }{\int }_{t_0}^T{B}_k\left(t\right)\left[{\widehat{\mathit{\text{X}}}}_k\left(t\right)\hspace{0.17em}-\hspace{0.17em}{\mathit{\text{X}}}_k\left(t\right)\right]dt\stackrel{d}{\to }N\left(0,\hspace{0.17em}{\sum }_2\right)$ with

$$\begin{array}{l}{\Sigma }_2\hspace{0.17em}=\hspace{0.17em}4\sum _{k=1}^{\vartheta }{\sigma }_k^2[{\int }_{t_0}^T{\int }_{t_0}^T{B}_k\left(s\right){\mathit{\text{N}}}^T\left(s\right){\left(\mathit{\text{G}}+\frac{{\lambda }_k}{n}{\mathit{\text{D}}}_q\right)}^{-1}\\ \hspace{1em}\hspace{0.17em}\times \hspace{0.17em}\mathit{\text{G}}{\left(\mathit{\text{G}}+\frac{{\lambda }_k}{n}{\mathit{\text{D}}}_q\right)}^{-1}\mathit{\text{N}}\left(t\right)B_k^T\left(t\right)dsdt],\end{array}$$ (13)

and ${\Sigma }_2\hspace{0.17em}=\hspace{0.17em}{O}_p\left(1\right)$. Then from the expression (12), we have $\sqrt{n}{\tilde{S}}_{1n}\left({\theta }_0\right)\hspace{0.17em}-\hspace{0.17em}{\tilde{S}}_{2n}\left({\theta }_0\right)\left[{\mathit{\text{a}}}^{*}\right]\stackrel{d}{\to }N\left(0,\hspace{0.17em}{\Sigma }_2\right)$. Thus the claim E7 holds.

Then, by E1 and E5, we have $\sqrt{n}\left({\tilde{S}}_{1n}\hspace{0.17em}-\hspace{0.17em}{\dot{S}}_1\right)\left({\widehat{\theta }}_n\right)\hspace{0.17em}-\hspace{0.17em}\sqrt{n}\left({\tilde{S}}_{1n}\hspace{0.17em}-\hspace{0.17em}{\dot{S}}_1\right)\left({\theta }_0\right)\hspace{0.17em}=\hspace{0.17em}{o}_p\left(1\right)$. Since ${\tilde{S}}_{1n}\left({\widehat{\theta }}_n\right)=0$ by E4 and ${\dot{S}}_1\left({\theta }_0\right)=0$ by E2, it follows that $\sqrt{n}{\dot{S}}_1\left({\widehat{\theta }}_n\right)+\sqrt{n}{\tilde{S}}_{1n}\left({\theta }_0\right)\hspace{0.17em}=\hspace{0.17em}{o}_p\left(1\right)$ Similarly, $\sqrt{n}{\dot{S}}_2\left({\widehat{\theta }}_n\right)\left[{\mathit{\text{a}}}^{*}\right]+\sqrt{n}{\tilde{S}}_{2n}\left({\theta }_0\right)\left[{\mathit{\text{a}}}^{*}\right]\hspace{0.17em}=\hspace{0.17em}{o}_p\left(1\right)$ Combining these equalities and E6 yields

$$\begin{gathered} {\dot{S}}_{11}\left({\widehat{\beta }}_n\hspace{0.17em}-\hspace{0.17em}{\beta }_0\right)\hspace{0.17em}+\hspace{0.17em}{\dot{S}}_{12}\left[{\widehat{\eta }}_n\hspace{0.17em}-\hspace{0.17em}{\eta }_0\right]\hspace{0.17em}+\hspace{0.17em}{\tilde{S}}_{1n}\left({\theta }_0\right) \\ =\hspace{0.17em}{o}_p\left(n^{-\frac{1}{2}}\right)\hspace{0.17em}+\hspace{0.17em}0\left(\Vert \widehat{\beta }\hspace{0.17em}-{\beta }_0\hspace{0.17em}\Vert \right)\hspace{0.17em}+\hspace{0.17em}O\left({\Vert {\widehat{\eta }}_n\hspace{0.17em}-\hspace{0.17em}{\eta }_0\Vert }_2^2\right), \end{gathered}$$ (14)

and

$$\begin{gathered} {\dot{S}}_{21}\left[{\mathit{\text{a}}}^{*}\right]\left({\widehat{\beta }}_n\hspace{0.17em}-\hspace{0.17em}{\beta }_0\right)\hspace{0.17em}+\hspace{0.17em}{\dot{S}}_{22}\left[{\mathit{\text{a}}}^{*},\hspace{0.17em}{\widehat{\eta }}_n\hspace{0.17em}-\hspace{0.17em}{\eta }_0\right]\hspace{0.17em}+\hspace{0.17em}{\tilde{S}}_{2n}\left({\theta }_0\right)\left[{\mathit{\text{a}}}^{*}\right] \\ =\hspace{0.17em}{o}_p\left(n^{-\frac{1}{2}}\right)\hspace{0.17em}+\hspace{0.17em}0\left(\Vert \widehat{\beta }\hspace{0.17em}-{\beta }_0\hspace{0.17em}\Vert \right)\hspace{0.17em}+\hspace{0.17em}O\left({\Vert {\widehat{\eta }}_n\hspace{0.17em}-\hspace{0.17em}{\eta }_0\Vert }_2^2\right). \end{gathered}$$ (15)

E1 implies $\sqrt{n}O\left({\Vert {\widehat{\eta }}_n\hspace{0.17em}-\hspace{0.17em}{\eta }_0\Vert }_2^2\right)\hspace{0.17em}=\hspace{0.17em}{o}_p\left(1\right)$. Thus by E3 and (14) minus (15), it follows that

$$\begin{gathered} \left({\dot{S}}_{11}\left({\theta }_0\right)\hspace{0.17em}-\hspace{0.17em}{\dot{S}}_{21}\left({\theta }_0\right)\left[{\mathit{\text{a}}}^{*}\right]\right)\left({\widehat{\beta }}_n\hspace{0.17em}-\hspace{0.17em}{\beta }_0\right)\hspace{0.17em}+\hspace{0.17em}o\left(\Vert {\widehat{\beta }}_n\hspace{0.17em}-\hspace{0.17em}{\beta }_0\Vert \right) \\ =\hspace{0.17em}-\left({\tilde{S}}_{1n}\left({\theta }_0\right)\hspace{0.17em}-\hspace{0.17em}{\tilde{S}}_{2n}\left({\theta }_0\right)\left[{\text{a}}^{*}\right]\right)\hspace{0.17em}+\hspace{0.17em}{o}_p\left(n^{-\frac{1}{2}}\right). \end{gathered}$$

That is $-\left[A+o\left(1\right)\right]\left({\widehat{\beta }}_n-{\beta }_0\right)=-\left({\tilde{S}}_{1n}\left({\theta }_0\right)-{\tilde{S}}_{2n}\left({\theta }_0\right)\left[{\mathit{a}}^{*}\right]\right)+o_p\left(n^{-\frac{1}{2}}\right)$ Combining with E7, it follows that

$$\begin{gathered} \sqrt{n}\left({\widehat{\beta }}_n\hspace{0.17em}-\hspace{0.17em}\beta \right)\hspace{0.17em}=\hspace{0.17em}{\left[A\hspace{0.17em}+\hspace{0.17em}o\left(1\right)\right]}^{-1}\sqrt{n}\left({\tilde{S}}_{1n}\left({\theta }_0\right)-{\tilde{S}}_{2n}\left({\theta }_0\right)\left[{\mathit{a}}^{*}\right]\right)+o_p\left(1\right) \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\stackrel{d}{\to }\hspace{0.17em}N\left(0,\hspace{0.17em}{A}^{-1}{\Sigma }_2{\left(A^{-1}\right)}^T\right) \end{gathered}$$

with $A^{-1}{\Sigma }_2{\left(A^{-1}\right)}^T\hspace{0.17em}=\hspace{0.17em}{O}_p\left(1\right)$