Robust nonlinear regression in applications

SUMMARY

Robust statistical methods, such as M-estimators, are needed for nonlinear regression models because of the presence of outliers/influential observations and heteroscedasticity. Outliers and influential observations are commonly observed in many applications, especially in toxicology and agricultural experiments. For example, dose response studies, which are routinely conducted in toxicology and agriculture, sometimes result in potential outliers, especially in the high dose groups. This is because response to high doses often varies among experimental units (e.g., animals). Consequently, this may result in outliers (i.e., very low values) in that group. Unlike the linear models, in nonlinear models the outliers not only impact the point estimates of the model parameters but can also severely impact the estimate of the information matrix. Note that, the information matrix in a nonlinear model is a function of the model parameters. This is not the case in linear models. In addition to outliers, heteroscedasticity is a major concern when dealing with nonlinear models. Ignoring heteroscedasticity may lead to inaccurate coverage probabilities and Type I error rates. Robustness to outliers/influential observations and to heteroscedasticity is even more important when dealing with thousands of nonlinear regression models in quantitative high throughput screening assays. Recently, these issues have been studied very extensively in the literature (references are provided in this paper), where the proposed estimator is robust to outliers/influential observations as well as to heteroscedasticity. The focus of this paper is to provide the theoretical underpinnings of robust procedures developed recently.

1 Introduction

Nonlinear regression models are widely used in many applications such as in dose response studies conducted in agricultural sciences, toxicology and other biological sciences. One of the reasons to consider nonlinear regression models is that in many applications it is not reasonable to assume that the rate of change in mean response E(y), where y is the response variable of interest, is constant with respect to the explanatory variables such as dose (x). For example, in toxicology it is common to assume, due to saturation in the mean response, the rate of change in E(y) may asymptotically go to zero as x → ∞. By nonlinear regression models we mean that the parameters of the model appear nonlinearly. Alternatively, one could consider models where the parameters appear linearly but the model is nonlinear in x, such as high order polynomials in x. An advantage of such models is that the parameters arise in the model linearly and hence are easy to estimate and one can even perform exact statistical inferences (under suitable assumptions). However, disadvantages outweigh the advantages of using such polynomial regression models. Often, unless very high degree polynomial is used, it is not always feasible to capture the underlying scientific phenomena using a polynomial curve. As the degree of the polynomial increases the power of statistical inference will decrease. Lastly, in toxicology and in other sciences, researchers are often interested in estimating dose corresponding to desired response, such as ED₅₀. If one were to use standard polynomial regression models then such parameters are often nonlinear functions of the model parameters (e.g. ratio of regression parameters) which can potentially lead to Fieller’s problem which may not always have a satisfactory solution.

For the above reasons nonlinear regression models, which are not necessarily polynomials in dose, are widely used in applications [1, 7–10, 14–16]. More precisely, a model such as the following is widely used

$$y_i=f(x_i,\theta )+{\sigma }_i{\epsilon }_i,i=1,2,\dots ,k,$$ (1)

where y_i = (y_i1,…, y_i,ni)^T are n_i × 1 response vectors, x_i are m × 1 vectors of known regression constants (e.g., dose), θ = (θ₁, …, θ_p)^T is a vector of unknown parameter, f is a pre-specified nonlinear function of θ and ε_i are independent and identically distributed random vectors with N(0, I). The total sample size n is given by ${\sum }_{i=1}^k{n}_i$.

Throughout this paper we assume that σ_i ≡ σ (z_i,τ) where z_i are q × 1 vectors of known constants (e.g., dose), τ = (τ₁, …, τ_q)^T is a vector of unknown parameters and τ is a pre-specified function of τ. Thus in the case of homoscedasticity σ (z_i, τ) is constant for all z_i and is not constant when the data are heteroscedastic. In toxicological studies, the variability in response y may sometimes depend upon the mean response, which is likely to be monotonic in dose. In practice, however, one never knows a priori if the data are homoscedastic or heteroscedastic. Consequently, one never knows, a priori, whether to use ordinary least squares (OLS) based methodology or to use iterated weighted least squares (IWLS) based methodology. Adding to the complexity, it is not uncommon to find outliers and/or influential observations in these data. It is well known that the efficiency and the accuracy of OLS and IWLS depend upon the underlying variance structure [7–10] and potential outliers and/or influential observations in the data. For these reasons, [9, 10] developed a preliminary test (PT) based methodology that is robust to the underlying variance structure and For the method to be robust against potential outliers and/or influential observations, they used M-estimation based methodology. PTE chooses either the ordinary M-estimator (OME) or the weighted M-estimator (WME) based on the results of a preliminary test for heteroscedasticity.

In Section 2 we define three robust estimators (OME, WME and PTE) and their asymptotic properties. In Section 3 we illustrate the methodology described in [9, 10] using a data set obtained from NIEHS quantitative high throughput screening (qHTS) assays. The asymptotic theory developed and used in [9, 10] require the asymptotic linearity of WME, which is demonstrated in Section 4 of this paper. For clarity of exposition, we relegated all regularity conditions and lemmas to the Appendix of the paper.

2 Estimators

In this section we define various estimators for estimating parameters in the model (1) and their asymptotic covariance matrices. Note that some of the definitions of the matrices and related quantities are provided in the Appendix.

The ordinary M-estimator (OME) for θ is defined as the solution of the following minimization problem:

$${\mathrm{\theta ̃}}_n=\mathit{\text{Argmin}}[\sum _{i,j}{h}^2(y_{ij}-f(x_i,\theta )):\theta \in {\Re }^p],$$

where h is a suitable function such as Huber-score function. Asymptotic covariance matrix of OME is expressed as follows:

$$E\left[n\right({\mathrm{\theta ̃}}_n-\theta \left){({\mathrm{\theta ̃}}_n-\theta )}^{\mathrm{T}}\right]={(\frac{1}{n}{\mathrm{\Gamma }}_{4n}(\theta ))}^{-1}(\frac{1}{n}{\mathrm{\Gamma }}_{33n}(\theta )){(\frac{1}{n}{\mathrm{\Gamma }}_{4n}(\theta ))}^{-1},$$

where

$${\mathrm{\Gamma }}_{33n}(\theta )={\sigma }_{\psi 3}^2\sum _{i=1}^k{n}_i{w}_1(x_i)f_{\theta }(x_i,\theta )f_{\theta }^{\mathrm{T}}(x_i,\theta ),{\mathrm{\Gamma }}_{4n}(\theta )={\gamma }_4\sum _{i=1}^k{n}_i{f}_{\theta }(x_i,\theta )f_{\theta }^{\mathrm{T}}(x_i,\theta ),$$

${\sigma }_{\psi 3}^2{w}_1(x)=E{\psi }^2(\sigma (z,\tau )\epsilon )$, γ₄ = Eψ′ (σ(z,τ)ε), f_θ(x_i,θ) = (∂/∂θ) f (x_i,θ), ψ(u) = (∂/∂u)h²(u), and ψ′(u) = (∂/∂u)ψ (u).

In [9], we defined WME of the unknown parameter vector, (θ^T,τ^T)^T as follows:

$$\left(\begin{array}{c}{\mathrm{\theta ̂}}_n\hfill \\ {\mathrm{\tau ̂}}_n\hfill \end{array}\right)=\mathit{\text{Argmin}}\left[\sum _{i,j}\right\{h^2\left(\frac{y_{ij}-f(x_i,\theta )}{\sigma (z_i,\tau )}\right)+\text{log}\sigma (z_i,\tau )\}:\theta \in {\Re }^p,\tau \in {\Re }^q].$$ (2)

The main theory in this paper together with the methodology described in [9] legitimizes the following asymptotic distribution of WME:

$${\mathrm{\Gamma ̂}}^{-\frac{1}{2}}\surd n\left(\begin{array}{c}\hfill {\mathrm{\theta ̂}}_n-\theta \hfill \\ \hfill {\mathrm{\tau ̂}}_n-\tau -{\nu }_n(\theta ,\tau )\hfill \end{array}\right)\to N_{p+q}(0,I_{p+q}),$$

where

$${\nu }_n(\theta ,\tau )={(\frac{1}{n}{\mathrm{\Gamma }}_{2n}(\theta ,\tau ))}^{-1}\frac{{\gamma }_1-1}{n}\sum _{i=1}^{n}k(z_i,\tau ){\sigma }_{\tau }(z_i,\tau ),\mathrm{\Gamma ̂}={(\frac{1}{n}{\mathrm{\Gamma }}_{5n}({\mathrm{\theta ̂}}_n,{\mathrm{\tau ̂}}_n))}^{-1}(\frac{1}{n}{\mathrm{\Gamma }}_{3n}({\mathrm{\theta ̂}}_n,{\mathrm{\tau ̂}}_n)){(\frac{1}{n}{\mathrm{\Gamma }}_{5n}({\mathrm{\theta ̂}}_n,{\mathrm{\tau ̂}}_n))}^{-1},{\mathrm{\Gamma }}_{3n}(\theta ,\tau )=\left(\begin{array}{cc}\hfill {\mathrm{\Gamma }}_{31n}(\theta ,\tau )\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill {\mathrm{\Gamma }}_{32n}(\theta ,\tau )\hfill \end{array}\right),{\mathrm{\Gamma }}_{5n}(\theta ,\tau )=\left(\begin{array}{cc}\hfill {\mathrm{\Gamma }}_{1n}(\theta ,\tau )\hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill {\mathrm{\Gamma }}_{2n}(\theta ,\tau )\hfill \end{array}\right),{\mathrm{\Gamma }}_{31n}(\theta ,\tau )={\sigma }_{\psi 1}^2\sum _{i=1}^n{k}^2(z_i,\tau )f_{\theta }(x_i,\theta )f_{\theta }^{\mathrm{T}}(x_i,\theta ),{\mathrm{\Gamma }}_{32n}(\theta ,\tau )={\sigma }_{\psi 2}^2\sum _{i=1}^n{k}^2(z_i,\tau ){\sigma }_{\tau }(z_i,\tau ){\sigma }_{\tau }^{\mathrm{T}}(z_i,\tau ),$$

σ_τ(z_i, τ) = (∂/∂τ)σ(z_i, τ), and k(z_i, τ) = 1/σ(z_i, τ).

To keep the model parsimonious, as in [9, 10], throughout this paper we model the standard deviation using the following log-linear model:

$$\text{log}\sigma (z_i,\tau )={\tau }_0+{\tau }_1{x}_i.$$

We take z_i ≡ (1, x_i)^T where x_i is the i^th dose. We derive the preliminary test estimator (PTE) as was done in [9, 10] as follows. We first fit the nonlinear model (1) under homoscedasticity using OME. Let ε̃_i = y_i − f(x_i, θ̃). Using these residuals we fit the following log-linear model using standard OLS methodology to estimate τ₀ and τ₁:

$$\text{log}|{\mathrm{\epsilon ̃}}_i|={\tau }_0+{\tau }_1{x}_i+{\eta }_i.$$

The PTE is then defined as

$${\mathrm{\theta ̂}}_n^{\mathrm{P}\mathrm{T}}=\{\begin{array}{cc}{\mathrm{\theta ̃}}_n\hfill & \text{if}{T}_n\le t_{\alpha ,n-2}\hfill \\ {\mathrm{\theta ̂}}_n\hfill & \text{if}{T}_n>t_{\alpha ,n-2},\hfill \end{array}$$

where T_n = τ̂_1n/ √Var (τ̂_1n, τ̂_1n is the least squares estimator of τ₁, t_α,n−2 is the critical value of the t-distribution with n − 2 degrees of freedom having probability 1 − α and α is the significance level of the preliminary test.

The following asymptotic covariance matrix for PTE is derived in [9]:

$$E\left[n\right({\mathrm{\theta ̂}}_n^{\mathrm{P}\mathrm{T}}-\theta \left){\left({\mathrm{\theta ̂}}_n^{\mathrm{P}\mathrm{T}}-\theta \right)}^{\mathrm{T}}\right]=F_t(t_{\alpha ,n-2}-\frac{{\tau }_1}{\surd \text{var}({\mathrm{\tau ̂}}_{1n})})E\left[n\right({\mathrm{\theta ̃}}_n-\theta \left){\left({\mathrm{\theta ̃}}_n-\theta \right)}^{\mathrm{T}}\right]+\{1-F_t(t_{\alpha ,n-2}-\frac{{\tau }_1}{\surd \text{var}({\mathrm{\tau ̂}}_{1n})}\left)\right\}E\left[n\right({\mathrm{\theta ̂}}_n-\theta \left){\left({\mathrm{\theta ̂}}_n-\theta \right)}^{\mathrm{T}}\right]=F_t(t_{\alpha ,n-2}-\frac{{\tau }_1}{\surd \text{var}({\mathrm{\tau ̂}}_{1n})}){\left(\frac{1}{n}{\mathrm{\Gamma }}_{4n}(\theta )\right)}^{-1}\left(\frac{1}{n}{\mathrm{\Gamma }}_{33n}(\theta )\right){\left(\frac{1}{n}{\mathrm{\Gamma }}_{4n}(\theta )\right)}^{-1}+\{1-F_t(t_{\alpha ,n-2}-\frac{{\tau }_1}{\surd \text{var}({\mathrm{\tau ̂}}_{1n})}\left)\right\}{\left(\frac{1}{n}{\mathrm{\Gamma }}_{1n}(\theta )\right)}^{-1}\left(\frac{1}{n}{\mathrm{\Gamma }}_{31n}(\theta )\right){\left(\frac{1}{n}{\mathrm{\Gamma }}_{1n}(\theta )\right)}^{-1},$$

where F_t is the cdf of the t-distribution with n − 2 degrees of freedom. Note that the asymptotic covariance matrix of PTE is a weighted average of the corresponding matrices for OME and WME.

Remark 2.1. From [9, 10] we now have the following confidence intervals:

1. The 100(1 − α)% confidence interval for parameters θ_i, i = 1, …, p, of a nonlinear model (1) using OME method can be computed using the following formula [9]:

   $${\mathrm{\theta ̃}}_i\pm t_{\alpha /2,n-p}\sqrt{{\upsilon }_{ii}^o}$$

   where ${\upsilon }_{ii}^o$ is the (i, i)-element of the estimated covariance matrix of θ̃_n

   $${\left({\mathrm{\Gamma }}_{4n}({\mathrm{\theta ̃}}_n)\right)}^{-1}{\mathrm{\Gamma }}_{33n}({\mathrm{\theta ̃}}_n){\left({\mathrm{\Gamma }}_{4n}({\mathrm{\theta ̃}}_n)\right)}^{-1}.$$
2. The 100(1 − α)% confidence interval for parameters θ_i, i = 1, …, p, of a nonlinear model (1) using WME method can be computed using the following formula [9]:

   $${\mathrm{\theta ̃}}_i\pm t_{\alpha /2,n-p-q}\sqrt{{\upsilon }_{ii}^w}$$

   where ${\upsilon }_{ii}^w$ is the (i, i)-element of the estimated covariance matrix of θ̂_n

   $${\left({\mathrm{\Gamma }}_{1n}({\mathrm{\theta ̂}}_n)\right)}^{-1}{\mathrm{\Gamma }}_{31n}({\mathrm{\theta ̂}}_n){\left({\mathrm{\Gamma }}_{1n}({\mathrm{\theta ̂}}_n)\right)}^{-1}.$$
3. The 100(1 − α)% confidence interval for parameters θ_i, i = 1, …, p, of a nonlinear model (1) using PTE method can be computed using the following formula [10]:

   $${\mathrm{\theta ̂}}_i^{PT}\pm t_{\alpha /2,n-p-q}\text{max}\{\sqrt{{\upsilon }_{ii}^o},\sqrt{{\upsilon }_{ii}^w}\}.$$

   The confidence limits for PTE use the maximum of the standard errors of OME and WME in their construction rather than a standard error based on the weighted average mentioned earlier. PTE can be expressed as a weighted average of OME and WME which both are asymptotically normally distributed. Therefore, the asymptotic distribution of PTE is a mixture of normal distributions and hence the usual confidence limits using a critical value from a t-distribution and the standard error of PTE may not be appropriate. That is the reason why we used the maximum of the standard errors of OME and WME, which is derived in [10].

3 Illustration

For illustration purposes we consider dose-response data for two compounds from the US National Toxicology Program (NTP) library of 1408 compounds that were evaluated using the quantitative high throughput screening (qHTS) assay. The raw data are provided in the Appendix and they are plotted against dose in Figures 1 (a) and (b), respectively. A visual inspection of Figure 1 (a) suggests that the data are perhaps homoscedastic, whereas Figure 1 (b) suggests potential heteroscedasticity. Indeed the Bartlett’s test for homogeneity yields p-values of 0.817 and 0.066 for Examples 1 and 2, respectively. As commonly done by toxicologists, we fitted the following nonlinear function, called the Hill function, for the two data sets:

$$f(x,\theta )={\theta }_0+{\theta }_1\frac{{\theta }_3^{{\theta }_2}}{{\theta }_3^{{\theta }_2}+x^{{\theta }_2}}.$$

Figure 1.

US NTP qHTS data sets; the corresponding fitted curves using OME (solid line) and WME (dashed line) methods.

We computed OME, WME and PTE (described in [9]) using Huber function with γ = 1.5. Note that PTE chooses either OME or WME based on the result of the preliminary test [9] and hence the fitted curve using PTE is the same as one of the two fitted curves using OME and WME. We computed the 95% confidence intervals for OME, WME and PTE using the formulas provided in the previous section. Results are summarized in Table 1. As expected, the performance of OME and WME depends upon whether the data are homoscedastic or heteroscedastic. For homoscedastic data even though the point estimates of OME and WME are quite similar, the standard errors of WME are almost two times larger than those of OME. On the other hand, for heteroscedastic data OME for θ₂ and the standard errors of OME for θ₂ and θ₃ are very large. Consequently, the confidence intervals are also very highly affected by the choice of the method. The PTE provides the desired compromise between the two methods.

Table 1.

Estimate, standard error and 95% confidence interval for parameters of the models for US NTP qHTS data using OME, WME and PTE methods.

	Parameter	Method	Estimate	SE	95% Con dence Interval
Example 1	θ0	OME	−36.7	4.7	(−47.7, −25.8)
		WME	−37.7	8.0	(−56.5, −18.9)
		PTE	−36.7	6.6	(−55.6, −17.9)
	θ1	OME	34.5	4.9	(23.1, 45.9)
		WME	35.3	8.3	(16.0, 54.6)
		PTE	34.5	6.8	(15.2, 53.8)
	θ2	OME	3.1	1.3	(0.1, 6.1)
		WME	2.6	1.4	(−0.6, 5.9)
		PTE	3.1	1.3	(−0.2, 6.3)
	θ3	OME	28.9	5.7	(15.7, 42.2)
		WME	29.9	9.1	(8.5, 51.2)
		PTE	28.9	7.6	(7.5, 50.3)
Example 2	θ0	OME	−45.7	2.8	(−52.3, −39.1)
		WME	−45.1	6.1	(−59.3, −30.9)
		PTE	−45.1	6.0	(−59.3, −30.9)
	θ1	OME	46.7	3.2	(39.4, 54.1)
		WME	46.4	6.2	(31.9, 61.0)
		PTE	46.4	6.2	(31.9, 61.0)
	θ2	OME	8.4	48.1	(−103.9, 120.8)
		WME	4.2	2.2	(−1.0, 9.4)
		PTE	4.2	9.0	(−108.4, 116.8)
	θ3	OME	18.9	8.9	(−2.0, 39.8)
		WME	17.9	2.4	(12.3, 23.4)
		PTE	17.9	2.8	(−3.0, 38.8)

4 Asymptotic linearity of weighted M-estimator (WME)

In this section we investigate the asymptotic linearity of WME (2) for the heteroscedastic nonlinear regression model (1). M-estimation methods have been well studied in the literature over the several decades for linear models (cf. [3, 4, 12]). Huber [2] proposed the M-estimator of the regression parameters in the univariate linear model and showed that under certain regularity conditions the M-estimator is consistent and asymptotically normal. The asymptotic theory for the M-estimator has been extensively studied (see, e.g., [2, 6, 17, 24]); for multivariate linear models one may refer to [5, 11, 13, 22, 23] among others. In the context of nonlinear regression models, Sanhueza [18] and Sanhueza and Sen [19, 20] proposed M-estimators and studied their asymptotic properties. More recently, Sanhueza et al. [21] extended these methods to nonlinear models for repeated measures data.

Asymptotic linearity of M-estimators is the first and important step to derive the asymptotic normality of the M-estimators in nonlinear regression models ([18–21]). Because we defined an M-estimator of not only the regression parameter θ, but also the variance parameter τ for heteroscedastic nonlinear regression model, the derivation of asymptotic linearity of the M-estimator is nontrivial.

The estimating equation for the minimization in (2) is given by

$$\sum _{i,j}\lambda (x_i,y_{ij},{\mathrm{\theta ̂}}_n,{\mathrm{\tau ̂}}_n)=0,$$ (3)

where

$$\lambda (x_i,y_{ij},\theta ,\tau )=\left(\begin{array}{c}\hfill k(z_i,\tau )\psi ({\epsilon }_{ij})f_{\theta }(x_i,\theta )\hfill \\ \hfill k(z_i,\tau )\{\psi ({\epsilon }_{ij}){\epsilon }_{ij}-1\}{\sigma }_{\tau }(z_i,\tau )\hfill \end{array}\right).$$ (4)

We explain the derivation of the uniform asymptotic linearity of WME very briey as follows: First we apply the Taylor’s expansion to λ(x_i,y_ij, θ, τ) and decompose the first order term to several terms. After that we take expectation on the terms which are going to zero by the regularity conditions.

Then under the regularity conditions and lemmas provided in the Appendix, we now state and prove the uniform asymptotic linearity of WME defined in (2).

Theorem 4.1. Under the regularity conditions (A), (B) and (C) described in the Appendix of this paper, for any given C₁, C₂ > 0,

$$\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\Vert \frac{1}{\surd n}\sum _{i=1}^n\{\lambda (x_i,y_i,\theta +n^{-\frac{1}{2}}t,\tau +n^{-\frac{1}{2}}s)-\lambda (x_i,y_i,\theta ,\tau )\}+\frac{1}{n}{({\mathrm{\Gamma }}_{1n}^{\mathrm{T}}(\theta ,\tau ),0^{\mathrm{T}})}^{\mathrm{T}}t+\frac{1}{n}{(0^{\mathrm{T}},{\mathrm{\Gamma }}_{2n}^{\mathrm{T}}(\theta ,\tau ))}^{\mathrm{T}}s\Vert =o_p(1)$$ (5)

as n → ∞, where λ(x_i, y_i, θ, τ) was defined in (4).

Proof. We consider the lth element of the vector λ(x_i, y_i, θ, τ) denoted for

$${\lambda }_l(x_i,y_i,\theta ,\tau )=\{\begin{array}{cc}k(z_i,\tau )\psi ({\epsilon }_i)f_{{\theta }_l}(x_i,\theta ),\hfill & l=1,\dots ,p\hfill \\ k(z_i,\tau )\{\psi ({\epsilon }_i){\epsilon }_i-1\}{\sigma }_{{\tau }_{l-p}}(z_i,\tau ),\hfill & l=p+1,\dots ,p+q.\hfill \end{array}$$

Using the first order term in the Taylor’s expansion, we have for 0 < u, υ < 1,

$${\lambda }_l(x_i,y_i,\theta +n^{-\frac{1}{2}}t,\tau +n^{-\frac{1}{2}}s)-{\lambda }_l(x_i,y_i,\theta ,\tau )=\frac{1}{\surd n}\sum _{j=1}^p{t}_j\{(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\}+\frac{1}{\surd n}\sum _{j=1}^q{s}_j\{(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\}+\frac{1}{\surd n}\sum _{j=1}^p{t}_j\{(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\theta }_r){\lambda }_l(x_i,y_i,\theta ,\tau )\}+\frac{1}{\surd n}\sum _{j=1}^q{s}_j\{(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\},$$

where for j = 1,… p,

$$(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta ,\tau )=\{\begin{array}{cc}k(z_i,\tau )\psi ({\epsilon }_i)({\partial }^2/\partial {\theta }_l\partial {\theta }_j)f(x_i,\theta )-k^2(z_i,\tau )\psi \prime ({\epsilon }_i)f_{{\theta }_l}(x_i,\theta )f_{{\theta }_j}(x_i,\theta ),\hfill & l=1,\dots ,p\hfill \\ -k^2(z_i,\tau )\psi ({\epsilon }_i)f_{{\theta }_j}(x_i,\theta ){\sigma }_{{\tau }_{l-p}}(z_i,\tau )-k^2(z_i,\tau )\psi \prime ({\epsilon }_i){\epsilon }_i{f}_{{\theta }_j}(x_i,\theta ){\sigma }_{{\tau }_{l-p}}(z_i,\tau ),\hfill & l=p+1,\dots ,p+q,\hfill \end{array}$$

and for j = 1, …, q,

$$(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )=\{\begin{array}{cc}-k^2(z_i,\tau )\psi ({\epsilon }_i)f_{{\theta }_l}(x_i,\theta ){\sigma }_{{\tau }_j}(z_i,\tau )-k^2(z_i,\tau )\psi \prime ({\epsilon }_i){\epsilon }_i{f}_{{\theta }_l}(x_i,\theta ){\sigma }_{{\tau }_j}(z_i,\tau ),\hfill & l=1,\dots ,p\hfill \\ -k^2(z_i,\tau )\{2\psi ({\epsilon }_i){\epsilon }_i-1\}{\sigma }_{{\tau }_{l-p}}(x_i,\theta ){\sigma }_{{\tau }_j}(z_i,\tau )-k^2(z_i,\tau )\psi \prime ({\epsilon }_i){\epsilon }_i^2{\sigma }_{{\tau }_{l-p}}(x_i,\theta ){\sigma }_{{\tau }_j}(z_i,\tau )+k(z_i,\tau )\{\psi ({\epsilon }_i){\epsilon }_i-1\}({\partial }^2/\partial {\tau }_{l-p}\partial {\tau }_j)\sigma (z_i,\tau ),\hfill & l=p+1,\dots ,p+q.\hfill \end{array}$$

Then, we have

1. for l = 1,…, p,

   $$\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\left|\frac{1}{\surd n}\sum _{i=1}^n\right\{{\lambda }_l(x_i,y_i,\theta +n^{-\frac{1}{2}}t,\tau +n^{-\frac{1}{2}}s)-{\lambda }_l(x_i,y_i,\theta ,\tau )\}+\frac{{\gamma }_2}{n}\sum _{i=1}^n\sum _{j=1}^p{t}_j{k}^2(z_i,\tau )f_{{\theta }_l}(x_i,\theta )f_{{\theta }_j}(x_i,\theta )|\le \underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\left|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^p{t}_j\right\{(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\left\}\right|+\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\left|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j\right\{(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\left\}\right|+\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^p{t}_j(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta ,\tau )+\frac{{\gamma }_2}{n}\sum _{i=1}^n\sum _{j=1}^p{t}_j{k}^2(z_i,\tau )f_{{\theta }_l}(x_i,\theta )f_{{\theta }_j}(x_i,\theta )|+\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )|.$$
   Using Lemma 6.1 through 6.4 provided in the Appendix we deduce that for l = 1, …, p

   $$\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\left|\frac{1}{\surd n}\sum _{i=1}^n\right\{{\lambda }_l(x_i,y_i,\theta +n^{-\frac{1}{2}}t,\tau +n^{-\frac{1}{2}}s)-{\lambda }_l(x_i,y_i,\theta ,\tau )\}+\frac{{\gamma }_2}{n}\sum _{i=1}^n\sum _{j=1}^p{t}_j{k}^2(z_i,\tau )f_{{\theta }_l}(x_i,\theta )f_{{\theta }_j}(x_i,\theta )|=o_p(1).$$ (6)
2. for l = p + 1, …, p + q,

   $$\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\left|\frac{1}{\surd n}\sum _{i=1}^n\right\{{\lambda }_l(x_i,y_i,\theta +n^{-\frac{1}{2}}t,\tau +n^{-\frac{1}{2}}s)-{\lambda }_l(x_i,y_i,\theta ,\tau )\}+\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j\{\frac{2{\gamma }_1+{\gamma }_3-1}{{\sigma }^2(z_i,\tau )}{\sigma }_{{\tau }_{l-p}}(z_i,\tau ){\sigma }_{{\tau }_j}(z_i,\tau )+\frac{1-{\gamma }_1}{\sigma (z_i,\tau )}({\partial }^2/\partial {\tau }_{l-p}\partial {\tau }_j)\sigma (z_i,\tau )\left\}\right|\le \underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\left|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^p{t}_j\right\{(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\left\}\right|+\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\left|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j\right\{(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\left\}\right|+\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^p{t}_j(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta ,\tau )|+\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )+\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j\{\frac{2{\gamma }_1+{\gamma }_3-1}{{\sigma }^2(z_i,\tau )}{\sigma }_{{\tau }_{l-p}}(z_i,\tau ){\sigma }_{{\tau }_j}(z_i,\tau )+\frac{1-{\gamma }_1}{\sigma (z_i,\tau )}({\partial }^2/\partial {\tau }_{l-p}\partial {\tau }_j)\sigma (z_i,\tau )\left\}\right|.$$
   And, from Lemma 6.5 through 6.8 provided in the Appendix we conclude that for l = p + 1, …, p + q

   $$\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\left|\frac{1}{\surd n}\sum _{i=1}^n\right\{{\lambda }_l(x_i,y_i,\theta +n^{-\frac{1}{2}}t,\tau +n^{-\frac{1}{2}}s)-{\lambda }_l(x_i,y_i,\theta ,\tau )\}+\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j\{\frac{2{\gamma }_1+{\gamma }_3-1}{{\sigma }^2(z_i,\tau )}{\sigma }_{{\tau }_{l-p}}(z_i,\tau ){\sigma }_{{\tau }_j}(z_i,\tau )+\frac{1-{\gamma }_1}{\sigma (z_i,\tau )}({\partial }^2/\partial {\tau }_{l-p}\partial {\tau }_j)\sigma (z_i,\tau )\left\}\right|=o_p(1).$$ (7)

   Therefore, the result of (5) follows from both (6) and (7).

5 Concluding remarks

We describe in this paper three recently published robust estimators for nonlinear regression models. All three estimators are robust to outliers and influential observations. OME is developed for homoscedastic data while WME is for heteroscedastic data. PTE is robust to error variance structure by choosing either OME or WME based on the result of a preliminary test for heteroscedasticity. PTE is an important estimator because a priori one may not know whether data are homoscedastic or heteroscedastic. Especially when there are number of data sets as in the case of qHTS assay PTE may be useful because of its robustness to outliers/influential observations as well as to error variance structure.

We illustrate these estimators by considering two real data sets from qHTS assay. One data set is potentially homoscedastic and the other is heteroscedastic. We fitted the Hill function for the data sets using the three estimators and observed that the performance and confidence intervals are affected by the choice of method.

We derive the uniform asymptotic linearity of WME which is theoretically important for developing the asymptotic normality of WME and the asymptotic covariance matrix of PTE. Because WME is estimating regression parameters as well as variance parameters simultaneously the derivation is quite complicated. However, under the suitably established regularity conditions and lemmas we prove the asymptotic linearity.

Optimal designs in the context of linear and nonlinear models are well studied in the literature. Yet, as observed in [10], very little has been done in the context of qHTS assays where a researcher is not only interested in performing thousands of statistical tests but is also interested in estimating parameters of the nonlinear model corresponding to each selected compound. Often it is a challenge to estimate the parameters of the nonlinear model precisely.

To illustrate this point, we considered dose-response data for a compound (Example 3 in Appendix) from NTP’s 1408 library of compounds. We fitted the Hill model for these data (see Figure 2) using both OME and WME. Visually, both OME as well as WME seem to fit the data well (especially OME), however, as seen from Table 2, the standard errors of all estimates appear to be so large that all confidence intervals contain zero. There could be several reasons for this to happen. For example, the total sample size may not be large enough for the amount of variability present in the data (especially at the high dose). A second reason could be dose-spacing. Although the chemical is tested at a large number of low doses, the chemical was not tested at higher doses containing the point of inflection. In fact there is no dose between 45 and 90 (µM) where the curve begins to drop. As consequence, it is not surprising that OME does not provide precise estimates for θ₂, the slope parameter (standard error of 409.9), or θ₃, the ED₅₀ (standard error of 264.1). As noted in [10], with the advent of qHTS assays, optimal design for nonlinear models is an important area of future research.

Figure 2.

A US NTP qHTS data set; the corresponding fitted curves using OME (solid line) and WME (dashed line) methods.

Table 2.

Estimate, standard Error and 95% confidence interval for parameters of the models for US NTP qHTS data using OME, WME and PTE methods.

Parameter	Method	Estimate	SE	95% Con dence Interval
θ0	OME	−76.1	77.1	(−256.0, 103.7)
	WME	−78.0	83.3	(−272.8, 116.7)
	PTE	−78.0	82.0	(−272.8, 116.7)
θ1	OME	72.9	77.3	(−107.5, 253.3)
	WME	74.8	83.5	(−120.6, 270.2)
	PTE	74.8	82.3	(−120.6, 270.2)
θ2	OME	9.7	409.9	(−946.9, 966.4)
	WME	4.2	7.1	(−12.3, 20.7)
	PTE	4.2	187.3	(−954.7, 963.0)
θ3	OME	51.6	264.1	(−564.6, 667.9)
	WME	63.4	56.2	(−68.0, 194.7)
	PTE	63.4	130.5	(−554.3, 681.0)

Table 3.

Data corresponding to three examples described in this paper. In each example we provide responses corresponding to the three replicates for each dose.

Dose (µM)	Example 1			Example 2			Example 3
0.0005874	7.327	−6.043	−0.428	−1.733	−5.901	8.032	−0.428	−1.733	−5.901
0.002937	3.110	−2.340	3.103	−6.491	1.196	−1.665	3.103	−6.491	1.196
0.01468	−11.981	−4.885	−0.566	−9.442	−11.521	5.464	−0.566	−9.442	−11.521
0.03283	−4.475	−5.747	−1.660	−5.877	−0.634	9.618	−1.660	−5.877	−0.634
0.07341	−4.922	−4.183	−0.981	−1.986	−5.771	−0.509	−0.981	−1.986	−5.771
0.1642	6.090	−10.287	−1.919	−10.696	−0.011	7.133	−1.919	−10.696	−0.011
0.3670	−11.767	−5.345	−7.134	−8.302	−1.625	−12.242	−7.134	−8.302	−1.625
0.8207	−3.393	−2.099	−4.490	−4.888	0.283	−0.770	−4.490	−4.888	0.283
1.835	−6.812	−5.808	−9.666	−4.959	1.045	8.512	−9.666	−4.959	1.045
4.103	−0.600	−5.707	−4.203	−3.499	5.638	−5.215	−4.203	−3.499	5.638
9.175	4.313	−8.110	0.388	−5.977	−2.008	−9.651	0.388	−5.977	−2.008
20.52	−10.838	−22.657	3.820	−3.014	−8.214	−39.837	3.820	−3.014	−8.214
45.87	−34.218	−30.895	−23.929	−24.587	−8.603	−44.239	−23.929	−24.587	−8.603
91.74	−30.911	−44.233	−36.776	−76.046	−83.195	−23.337	−36.776	−76.046	−83.195

6 Appendix

6.1 The data

We provide the three data sets used in the Illustration.

6.2 Regularity conditions and proofs

We shall make the following sets of regularity assumptions concerning (A) the score function ψ, (B) the function f, and (C) the function σ. These are a generalization of the assumption established by [18].

[A1]:

ψ is a nonconstant, odd function which is absolutely continuous and differentiable with respect to θ.

[A2]: Let ε = {y − f(x, θ)}/σ(z, τ),

1. Eψ(ε) = 0; $E{\psi }^2(\epsilon )={\sigma }_{\psi 1}^{2}u(x)<\infty$; E{ψ(ε)ε} = γ₁(≠ 0);

   $\text{Var}\{\psi (\epsilon )\epsilon \}={\sigma }_{\psi 2}^2\nu (x)<\infty$

2. E|ψ′(ε)|^1+δ < ∞, E|ψ′(ε)ε|^1+δ < ∞, E|ψ′(ε)ε²|^1+δ < ∞ for some 0 < δ ≤ 1, and

   Eψ′(ε) = γ₂ (≠ 0); E{ψ′(ε)ε} = 0; E{ψ′(ε)ε²} = γ₃ (≠ 0);

   Eψ′(σ(z, τ)ε) = γ₄ (≠ 0)

[A3]: Let ε(θ, τ) = {y − f(x, θ)}/σ(z, τ),

1. $\underset{{\delta }_1\to 0}{\text{lim}}\underset{{\delta }_2\to 0}{\text{lim}}E\{\underset{\Vert {\mathrm{\Delta }}_1\Vert \le {\delta }_1,\Vert {\mathrm{\Delta }}_2\Vert \le {\delta }_2}{\text{sup}}|\psi (\epsilon (\theta +{\mathrm{\Delta }}_1,\tau +{\mathrm{\Delta }}_2))-\psi (\epsilon (\theta ,\tau ))|\}=0$

2. $\underset{{\delta }_1\to 0}{\text{lim}}\underset{{\delta }_2\to 0}{\text{lim}}E\{\underset{\Vert {\mathrm{\Delta }}_1\Vert \le {\delta }_1,\Vert {\mathrm{\Delta }}_2\Vert \le {\delta }_2}{\text{sup}}|\psi (\epsilon (\theta +{\mathrm{\Delta }}_1,\tau +{\mathrm{\Delta }}_2))\epsilon (\theta ,{\mathrm{\Delta }}_1,\tau +{\mathrm{\Delta }}_2)-\psi (\epsilon (\theta ,\tau ))\epsilon (\theta ,\tau )|\}=0$

3. $\underset{{\delta }_1\to 0}{\text{lim}}\underset{{\delta }_2\to 0}{\text{lim}}E\{\underset{\Vert {\mathrm{\Delta }}_1\Vert \le {\delta }_1,\Vert {\mathrm{\Delta }}_2\Vert \le {\delta }_2}{\text{sup}}|\psi \prime (\epsilon (\theta +{\mathrm{\Delta }}_1,\tau +{\mathrm{\Delta }}_2))-\psi \prime (\epsilon (\theta ,\tau ))|\}=0$

4. $\underset{{\delta }_1\to 0}{\text{lim}}\underset{{\delta }_2\to 0}{\text{lim}}E\{\underset{\Vert {\mathrm{\Delta }}_1\Vert \le {\delta }_1,\Vert {\mathrm{\Delta }}_2\Vert \le {\delta }_2}{\text{sup}}|\psi \prime (\epsilon (\theta +{\mathrm{\Delta }}_1,\tau +{\mathrm{\Delta }}_2))\epsilon (\theta +{\mathrm{\Delta }}_1,\tau +{\mathrm{\Delta }}_2)-\psi \prime (\epsilon (\theta ,\tau ))\epsilon (\theta ,\tau )|\}=0$

5. $\underset{{\delta }_1\to 0}{\text{lim}}\underset{{\delta }_2\to 0}{\text{lim}}E\{\underset{\Vert {\mathrm{\Delta }}_1\Vert \le {\delta }_1,\Vert {\mathrm{\Delta }}_2\Vert \le {\delta }_2}{\text{sup}}|\psi \prime (\epsilon (\theta +{\mathrm{\Delta }}_1,\tau +{\mathrm{\Delta }}_2)){\epsilon }^2(\theta +{\mathrm{\Delta }}_1,\tau +{\mathrm{\Delta }}_2)-\psi \prime (\epsilon (\theta ,\tau )){\epsilon }^2(\theta ,\tau )|\}=0$

[B1]:

f(x, θ) is continuous and twice differentiable with respect to θ ∈ ℜ^p.

[B2]:

lim_n→∞ n⁻¹Γ_1n(θ, τ) = Γ₁(θ, τ), where

$${\mathrm{\Gamma }}_{1n}(\theta ,\tau )={\gamma }_2\sum _{i=1}^n{k}^2(z_i,\tau )f_{\theta }(x_i,\theta )f_{\theta }^{\mathrm{T}}(x_i,\theta ).$$

[B3]: For j, l = 1, …, p

1. lim_δ→0 sup_{‖Δ‖≤δ} |(∂/∂θ_j) f (x, θ + Δ) (∂/∂θ_l) f (x, θ + Δ) − (∂/∂θ_j) f (x, θ) (∂/∂θ_l) f (x, θ)| = 0

2. lim_δ→0 sup_{‖Δ‖≤δ} |(∂²/∂θ_j∂θ_l) f (x, θ + Δ) − (∂²/∂θ_j∂θ_l) f (x, θ)| = 0

[C1]:

σ(z, τ) is continuous and twice differentiable with respect to τ ∈ ℜ^q.

[C2]:

lim_n→∞ n⁻¹Γ_2n(θ, τ) = Γ₂(θ, τ), where

$${\mathrm{\Gamma }}_{2n}(\theta ,\tau )=\sum _{i=1}^n\{\frac{2{\gamma }_1+{\gamma }_3-1}{{\sigma }^2(z_i,\tau )}{\sigma }_{\tau }(z_i,\tau ){\sigma }_{\tau }^{\mathrm{T}}(z_i,\tau )+\frac{1-{\gamma }_1}{\sigma (z_i,\tau )}{\mathrm{\Sigma }}_{\tau }(z_i,\tau )\},$$

and Σ_τ (z_i, τ) = (∂²/∂τ∂τ^T)σ(z_i, τ).

[C3]: For j, l = 1, …, q

1. lim_δ→0 sup_{‖Δ‖≤δ} |(∂/∂τ_j)σ(z, τ + Δ) (∂/∂τ_l)σ(z, τ + Δ) − (∂/∂τ_j)σ(z, τ) (∂/∂τ_l)σ(z, τ)| = 0

2. lim_δ→0 sup_{‖Δ‖≤δ} |(∂²/∂τ_j∂τ_l)σ(z, τ + Δ) − (∂²/∂τ_j∂τ_l)σ(z, τ)| = 0

In order to prove the uniform asymptotic linearity of WME (θ̂^T, τ̂^T)^T in (3), we need to consider the following series of lemmas. When proving the main theorem we decompose the first order term of the Taylor’s expansion to λ(x_i, y_i, θ, τ), into several sub-terms which are partial derivatives of λ(x_i, y_i, θ, τ) with respect to θ and τ. By these lemmas we put bounds on quantities about such partial derivatives which are used in the main theorem about the uniform asymptotic linearity of WME.

Lemma 6.1. Let the regularity conditions in Section 6.2 hold and let λ_l(x_i, y_i, θ, τ) be the lth element of the vector λ(x_i, y_i, θ, τ), defined in (4), for l = 1, …, p+q. Then for l = 1, …, p

$$\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\left|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^p{t}_j\right\{(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\left\}\right|=o_p(1).$$ (8)

where

$${\lambda }_l(x_i,y_i,\theta ,\tau )=k(z_i,\tau )\psi ({\epsilon }_i(\theta ,\tau ))f_{{\theta }_l}(x_i,\theta ),l=1,\dots ,p,$$ (9)

and f_θl(x_i, θ) = (∂/∂θ_l) f (x_i, θ).

Proof. By the definition of derivative, we may write for j, l = 1, …, p

$$(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta ,\tau )=k(z_i,\tau )\psi ({\epsilon }_i(\theta ,\tau ))({\partial }^2/\partial {\theta }_l\partial {\theta }_j)f(x_i,\theta )-k^2(z_i,\tau )\psi \prime ({\epsilon }_i(\theta ,\tau ))f_{{\theta }_l}(x_i,\theta )f_{{\theta }_j}(x_i,\theta ).$$ (10)

Then,

$$\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\left|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^p{t}_j\right\{(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\left\}\right|\le \frac{1}{n}{C}_1\sum _{i=1}^n\sum _{j=1}^p\underset{\Vert t\Vert \le C1,\Vert s\Vert \le C_2}{\text{sup}}|(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta ,\tau )|$$

and

Then by taking expectations on both sides we get

Thus, by conditions [A3] (i),(iii), and [B3] (i),(ii), we have that:

$$E\left\{\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\right|(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\left|\right\}\to 0,\forall i,$$

and

$$E\left[\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\right|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^p{t}_j\{(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\}\left|\right]\to 0.$$

Also,

$$\text{var}\left[\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\right|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^p{t}_j\{(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\}\left|\right]\le \frac{C_1^2}{n^2}\sum _{i=1}^n\text{var}\left\{\sum _{j=1}^p\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\right|(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\left|\right\}\le C_1^2{K}_1/n\to 0.$$

Therefore, we have (8).

Lemma 6.2. Let the regularity conditions in Section 6.2 hold and let λ_l(x_i, y_i, θ, τ) be the lth element of the vector λ(x_i, y_i, θ, τ), defined in (4), for l = 1, …, p+q. Then for l = 1, …, p

$$\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^p{t}_j(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta ,\tau )+\frac{{\gamma }_2}{n}\sum _{i=1}^n\sum _{j=1}^p{t}_j{k}^2(z_i,\tau )f_{{\theta }_l}(x_i,\theta )f_{{\theta }_j}(x_i,\theta )|=o_p(1),$$ (11)

where λ_l(x_i, y_i, θ, τ) is defined in (9).

Proof. From (10), we have

$$\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^p{t}_j(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta ,\tau )+\frac{{\gamma }_2}{n}\sum _{i=1}^n\sum _{j=1}^p{t}_j{k}^2(z_i,\tau )f_{{\theta }_l}(x_i,\theta )f_{{\theta }_j}(x_i,\theta )|=\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^p{t}_{j}k(z_i,\tau )\psi ({\epsilon }_i)({\partial }^2/\partial {\theta }_l\partial {\theta }_j)f(x_i,\theta )-\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^p{t}_j{k}^2(z_i,\tau )\{\psi \prime ({\epsilon }_i)-{\gamma }_2\}{f}_{{\theta }_l}(x_i,\theta )f_{{\theta }_j}(x_i,\theta )|\le C_1\sum _{j=1}^p|\frac{1}{n}\sum _{i=1}^{n}k(z_i,\tau )\psi ({\epsilon }_i)({\partial }^2/\partial {\theta }_l\partial {\theta }_j)f(x_i,\theta )|+C_1\sum _{j=1}^p|\frac{1}{n}\sum _{i=1}^n{k}^2(z_i,\tau )\{\psi \prime ({\epsilon }_i)-{\gamma }_2\}{f}_{{\theta }_l}(x_i,\theta )f_{{\theta }_j}(x_i,\theta )|$$

which by using the Markov WLLN and conditions [A2] (i)–(ii) yields

$$\frac{1}{n}\sum _{i=1}^{n}k(z_i,\tau )\psi ({\epsilon }_i)({\partial }^2/\partial {\theta }_l\partial {\theta }_j)f(x_i,\theta )=o_p(1)$$

and

$$\frac{1}{n}\sum _{i=1}^n{k}^2(z_i,\tau )\{\psi \prime ({\epsilon }_i)-{\gamma }_2\}{f}_{{\theta }_l}(x_i,\theta )f_{{\theta }_j}(x_i,\theta )=o_p(1).$$

Therefore, we have (11).

Lemma 6.3. Let the regularity conditions in Section 6.2 hold and let λ_l(x_i, y_i, θ, τ) be the lth element of the vector λ(x_i, y_i, θ, τ), defined in (4), for l = 1, …, p+q. Then for l = 1, …, p

$$\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\left|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j\right\{(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\left\}\right|=o_p(1).$$ (12)

where λ_l(x_i, y_i, θ, τ) is defined in (9).

Proof. By the definition of derivative, we may write for l = 1, …, p, j = 1, …, q,

$$(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )=-k^2(z_i,\tau )\psi ({\epsilon }_i(\theta ,\tau ))f_{{\theta }_l}(x_i,\theta ){\sigma }_{{\tau }_j}(z_i,\tau )-k^2(z_i,\tau )\psi \prime ({\epsilon }_i(\theta ,\tau )){\epsilon }_i(\theta ,\tau )f_{{\theta }_l}(x_i,\theta ){\sigma }_{{\tau }_j}(z_i,\tau ),$$ (13)

where σ_τj(z_i, τ) = (∂/∂τ_j)σ(z_i, τ). Then,

$$\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\left|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j\right\{(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\left\}\right|\le \frac{1}{n}{C}_2\sum _{i=1}^n\sum _{j=1}^q\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}|(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )|$$

and

Then by taking expectations on both sides we get

Then, by conditions [A3] (i),(iv), we have that

$$E\left\{\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\right|(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\left|\right\}\to 0,\forall i,$$

and

$$E\left[\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\right|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j\{(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\}\left|\right]\to 0.$$

Also,

$$\text{var}\left[\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\right|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j\{(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\}\left|\right]\le \frac{C_2^2}{n^2}\sum _{i=1}^n\text{var}\left\{\sum _{j=1}^q\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\right|(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )|\}\le C_2^2{K}_2/n\to 0.$$

Therefore, we have (12).

Lemma 6.4. Let the regularity conditions in Section 6.2 hold and let λ_l(x_i, y_i, θ, τ) be the lth element of the vector λ(x_i, y_i, θ, τ), defined in (4), for l = 1, …, p+q. Then for l = 1, …, p

$$\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )|=o_p(1).$$ (14)

where λ_l(x_i, y_i, θ, τ) is defined in (9).

Proof. From (13), we have

$$\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )|=\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j{k}^2(z_i,\tau )\psi ({\epsilon }_i)f_{{\theta }_l}(x_i,\theta ){\sigma }_{{\tau }_j}(z_i,\tau )+\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j{k}^2(z_i,\tau )\psi \prime ({\epsilon }_i){\epsilon }_i{f}_{{\theta }_l}(x_i,\theta ){\sigma }_{{\tau }_j}(z_i,\tau )|\le C_2\sum _{j=1}^q|\frac{1}{n}\sum _{i=1}^n{k}^2(z_i,\tau )\psi ({\epsilon }_i)f_{{\theta }_l}(x_i,\theta ){\sigma }_{{\tau }_j}(z_i,\tau )|+C_2\sum _{j=1}^q|\frac{1}{n}\sum _{i=1}^n{k}^2(z_i,\tau )\psi \prime ({\epsilon }_i){\epsilon }_i{f}_{{\theta }_l}(x_i,\theta ){\sigma }_{{\tau }_j}(z_i,\tau )|$$

which by using the Markov’s Weak Law of Large Numbers (WLLN) and conditions [A2] (i)–(ii) yields

$$\frac{1}{n}\sum _{i=1}^n{k}^2(z_i,\tau )\psi ({\epsilon }_i)f_{{\theta }_l}(x_i,\theta ){\sigma }_{{\tau }_j}(z_i,\tau )=o_p(1),$$

and

$$\frac{1}{n}\sum _{i=1}^n{k}^2(z_i,\tau )\psi \prime ({\epsilon }_i){\epsilon }_i{f}_{{\theta }_l}(x_i,\theta ){\sigma }_{{\tau }_j}(z_i,\tau )=o_p(1).$$

Therefore, we have (14).

Lemma 6.5. Let the regularity conditions in Section 6.2 hold and let λ_l(x_i, y_i, θ, τ) be the lth element of the vector λ(x_i, y_i, θ, τ), defined in (4), for l = 1, …, p+q. Then for l = p + 1, …, p + q

$$\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^p{t}_j\{(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\left\}\right|=o_p(1).$$ (15)

where

$${\lambda }_l(x_i,y_i,\theta ,\tau )=k(z_i,\tau )\{\psi ({\epsilon }_i){\epsilon }_i-1\}{\sigma }_{{\tau }_{l-p}}(z_i,\tau ),l=p+1,\dots ,p+q.$$ (16)

Proof. By the definition of derivative, we may write for j = 1, …, p, l = p+1, …, p+q,

$$(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta ,\tau )=-k^2(z_i,\tau )\psi ({\epsilon }_i)f_{{\theta }_j}(x_i,\theta ){\sigma }_{{\tau }_{l-p}}(z_i,\tau )-k^2(z_i,\tau )\psi \prime ({\epsilon }_i){\epsilon }_i{f}_{{\theta }_j}(x_i,\theta ){\sigma }_{{\tau }_{l-p}}(z_i,\tau ).$$ (17)

Then since (17) is the same as (13), we can follow the proof of Lemma 6.4 to obtain the result of (15).

Lemma 6.6. Let the regularity conditions in Section 6.2 hold and let λ_l(x_i, y_i, θ, τ) be the lth element of the vector λ(x_i, y_i, θ, τ), defined in (4), for l = 1, …, p+q. Then for l = p + 1, …, p + q

$$\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^p{t}_j(\partial /\partial {\theta }_j){\lambda }_l(x_i,y_i,\theta ,\tau )|=o_p(1).$$

where λ_l(x_i, y_i, θ, τ) is defined in (16).

Proof. We can also follow the proof of Lemma 6.4.

Lemma 6.7. Let the regularity conditions in Section 6.2 hold and let λ_l(x_i, y_i, θ, τ) be the lth element of the vector λ(x_i, y_i, θ, τ), defined in (4), for l = 1, …, p+q. Then for l = p + 1, …, p + q

$$\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\left|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j\right\{(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\left\}\right|=o_p(1).$$ (18)

where λ_l(x_i, y_i, θ, τ) is defined in (16).

Proof. By the definition of derivative, we may write for l = p + 1, …, p + q, j = 1, …, q,

$$(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )=-k^2(z_i,\tau )\{2\psi ({\epsilon }_i(\theta ,\tau )){\epsilon }_i(\theta ,\tau )-1\}{\sigma }_{{\tau }_{l-p}}(z_i,\tau ){\sigma }_{{\tau }_j}(z_i,\tau )-k^2(z_i,\tau )\psi \prime ({\epsilon }_i(\theta ,\tau )){\epsilon }_i^2(\theta ,\tau ){\sigma }_{{\tau }_{l-p}}(z_i,\tau ){\sigma }_{{\tau }_j}(z_i,\tau )+k(z_i,\tau )\{\psi ({\epsilon }_i(\theta ,\tau )){\epsilon }_i(\theta ,\tau )-1\}({\partial }^2/\partial {\tau }_{l-p}\partial {\tau }_j)\sigma (z_i,\tau ).$$ (19)

Then,

$$\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\left|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j\right\{(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\left\}\right|\le \frac{1}{n}{C}_2\sum _{i=1}^n\sum _{j=1}^q\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}|(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )|$$

and

Then by taking expectations on both sides we get

Then, by conditions [A3] (ii),(v), [C3] (i),(ii), we have that

$$E\left\{\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\right|(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\left|\right\}\to 0,{\forall }_i,$$

and

$$E\left[\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\right|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j\{(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\}\left|\right]\to 0.$$

Also,

$$\text{var}\left[\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\right|\frac{1}{2}\sum _{i=1}^n\sum _{j=1}^q{s}_j\{(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\}\left|\right]\le \frac{C_2^2}{n^2}\sum _{i=1}^n\text{var}\left\{\sum _{j=1}^q\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}\right|(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta +\frac{ut}{\surd n},\tau +\frac{\upsilon s}{\surd n})-(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )\left|\right\}\le C_2^2{K}_2/n\to 0.$$

Thus we have (18).

Lemma 6.8. Let the regularity conditions in Section 6.2 hold and let λ_l(x_i, y_i, θ, τ) be the lth element of the vector λ(x_i, y_i, θ, τ), defined in (4), for l = 1, …, p+q. Then for l = p + 1, …, p + q

$$\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )+\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j\{\frac{2{\gamma }_1+{\gamma }_3-1}{{\sigma }^2(z_i,\tau )}\times {\sigma }_{{\tau }_{l-p}}(z_i,\tau ){\sigma }_{{\tau }_j}(z_i,\tau )+\frac{1-{\gamma }_1}{\sigma (z_i,\tau )}({\partial }^2/\partial {\tau }_{l-p}\partial {\tau }_j)\sigma (z_i,\tau )\left\}\right|=o_p(1).$$ (20)

where λ_l(x_i, y_i, θ, τ) is defined in (16).

Proof. From (19), we have

$$\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}|\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j(\partial /\partial {\tau }_j){\lambda }_l(x_i,y_i,\theta ,\tau )+\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j\{\frac{2{\gamma }_1+{\gamma }_3-1}{{\sigma }^2(z_i,\tau )}\times {\sigma }_{{\tau }_{l-p}}(z_i,\tau ){\sigma }_{{\tau }_j}(z_i,\tau )+\frac{1-{\gamma }_1}{\sigma (z_i,\tau )}({\partial }^2/\partial {\tau }_{l-p}\partial {\tau }_j)\sigma (z_i,\tau )\left\}\right|=\underset{\Vert t\Vert \le C_1,\Vert s\Vert \le C_2}{\text{sup}}|-\frac{2}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j{k}^2(z_i,\theta )\{\psi ({\epsilon }_i){\epsilon }_i-{\gamma }_1\}{\sigma }_{{\tau }_{l-p}}(z_i,\tau ){\sigma }_{{\tau }_j}(z_i,\tau )-\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_j{k}^2(z_i,\tau )\{\psi \prime ({\epsilon }_i){\epsilon }_i^2-{\gamma }_3\}{\sigma }_{{\tau }_{l-p}}(z_i,\tau ){\sigma }_{{\tau }_j}(z_i,\tau )+\frac{1}{n}\sum _{i=1}^n\sum _{j=1}^q{s}_{j}k(z_i,\tau )\{\psi ({\epsilon }_i){\epsilon }_i-{\gamma }_1\}({\partial }^2/\partial {\tau }_{l-p}\partial {\tau }_j)\sigma (z_i,\tau )|\le 2C_2\sum _{j=1}^q|\frac{1}{n}\sum _{i=1}^n{k}^2(z_i,\theta )\{\psi ({\epsilon }_i){\epsilon }_i-{\gamma }_1\}{\sigma }_{{\tau }_{l-p}}(z_i,\tau ){\sigma }_{{\tau }_j}(z_i,\tau )|+C_2\sum _{j=1}^q|\frac{1}{n}\sum _{i=1}^n{k}^2(z_i,\theta )\{\psi \prime ({\epsilon }_i){\epsilon }_i^2-{\gamma }_3\}{\sigma }_{{\tau }_{l-p}}(z_i,\tau ){\sigma }_{{\tau }_j}(z_i,\tau )|+C_2\sum _{j=1}^q|\frac{1}{n}\sum _{i=1}^{n}k(z_i,\theta )\{\psi ({\epsilon }_i){\epsilon }_i-{\gamma }_1\}({\partial }^2/\partial {\tau }_{l-p}\partial {\tau }_j)\sigma (z_i,\tau )|$$

which by using the Markov WLLN and conditions [A2] (i)–(ii) yields

$$\frac{1}{n}\sum _{i=1}^n{k}^2(z_i,\theta )\{\psi ({\epsilon }_i){\epsilon }_i-{\gamma }_1\}{\sigma }_{{\tau }_{l-p}}(z_i,\tau ){\sigma }_{{\tau }_j}(z_i,\tau )=o_p(1),\frac{1}{n}\sum _{i=1}^n{k}^2(z_i,\theta )\{\psi \prime ({\epsilon }_i){\epsilon }_i^2-{\gamma }_3\}{\sigma }_{{\tau }_{l-p}}(z_i,\tau ){\sigma }_{{\tau }_j}(z_i,\tau )=o_p(1),$$

and

$$\frac{1}{n}\sum _{i=1}^{n}k(z_i,\theta )\{\psi ({\epsilon }_i){\epsilon }_i-{\gamma }_i\}({\partial }^2/\partial {\tau }_{l-p}\partial {\tau }_j)\sigma (z_i,\tau )=o_p(1).$$

Therefore, we have the result in (20).