Model estimation and selection for partial linear varying coefficient EV models with longitudinal data

Abstract

In this paper, we consider the estimation and model selection for longitudinal partial linear varying coefficient errors-in-variables (EV) models when the covariates are measured with some additive errors. Bias-corrected penalized quadratic inference functions method is proposed based on quadratic inference functions with two penalty function terms. The proposed method can not only handle the measurement errors of covariates and within-subject correlations but also estimate and select significant non-zero parametric and nonparametric components simultaneously. With some regularization conditions, the resulting estimators of parameters are asymptotically normal and the estimators of nonparametric varying coefficient achieves the optimal convergence rate. Furthermore, we present simulation studies and a real example analysis to evaluate the finite sample performance of the proposed method.

1. Introduction

Varying coefficient models [7] have better interpretability and flexibility than linear models and can avoid the curse of dimensionality. They are usually applied for analysis of longitudinal and clustered data. In varying coefficient model, the regression coefficients are unknown nonparametric functions and allowed to depend on time or some other covariates. They facilitate the study of dynamic features. A survey about studies for varying coefficient models can be seen in [15].

As we know, not all of the coefficients are varying in some special cases. Thus, we consider the partial linear varying coefficient model [12] based on longitudinal data. Suppose the longitudinal data

$$\left\{(Y_{ij},X_{ij},Z_{ij},t_{ij}):i=1,2,\dots ,n,j=1,2,\dots ,n_i\right\}$$

satisfies the following partial linear varying coefficient model as

$$Y_{ij}=X_{ij}^T\mathrm{\text{β}}+Z_{ij}^T\alpha \left(t_{ij}\right)+{ϵ}_{ij},$$ (1)

where $Y_{ij}\in \mathbb{R}$ and $(X_{ij},Z_{ij})\in {\mathbb{R}}^p\times {\mathbb{R}}^q$ are the response and covariate at $t_{ij}\in \mathbb{R}$, $X_{ij}=(X_{ij}^1,{X_{ij}}^2,\dots ,X_{ij}^p{)}^T$, $Z_{ij}=(Z_{ij}^1,Z_{ij}^2,\dots ,Z_{ij}^q{)}^T$, ${ϵ}_{ij}\in \mathbb{R}$ is a zero-mean stochastic process, ${ϵ}_{ij}$ and $(X_{ij},Z_{ij})$ are independent of each other. $\mathrm{\text{β}}=({\mathrm{\text{β}}}_1,{\mathrm{\text{β}}}_2,\dots ,{\mathrm{\text{β}}}_p{)}^T$ is a regression parameter vector. $\mathrm{\forall }t\in \mathbb{R}$, $\alpha \left(t\right)=\left({\alpha }_1\right(t),{\alpha }_2(t),\dots ,{\alpha }_q(t){)}^T$ is coefficient functions vector with ${\alpha }_l\left(t\right)(l=1,2,\dots ,q)$ being unknown smoothing coefficient functions of t, $t\in [0,1]$. We further state some assumptions on the first two moments of model (1) as $E(Y_{ij}|X_{ij},Z_{ij},t_{ij})={\mu }_{ij}$ and $\text{var}(Y_{ij}|X_{ij},Z_{ij},t_{ij})=\nu \left({\mu }_{ij}\right)$, where $E(\cdot )$, $\text{var}(\cdot )$ are the expectation and variance respectively, and $\nu (\cdot )$ is known function.

Model (1) has the advantages of linear model and varying coefficient model, can reduce the modeling bias and avoid the curse of dimensionality. Recently, it has been studied by some different statistical methods, such as local polynomial fitting method [32], profile least square method [4], empirical likelihood method [10,30], quantile regression method [23], penalized quadratic inference function (pQIF) method [20] and so on. An important assumption in these methods is that covariates can be observed.

However, as all we know, it is impossible to get accurate data in practice, especially for some important covariates. No matter how data is collected, measurement errors are unavoidable or some covariates are unobserved. Ignoring measurement errors may result in biased estimators or even uncorrected results. Therefore, it is meaningful to incorporate measurement errors into model (1). In view of these, we consider that X and Z are measured with additive errors in model (1), which is the so-called partial linear varying coefficient errors-in-variables (EV) model as

$$\begin{array}{r}\left\{\begin{array}{l}{Y}_{ij}=X_{ij}^T\mathrm{\text{β}}+Z_{ij}^T\alpha \left(t_{ij}\right)+{ϵ}_{ij}\\ W_{ij}=X_{ij}+w_{ij}\\ U_{ij}=Z_{ij}+u_{ij}\end{array}\right.,i=1,2,\dots ,n,j=1,2,\dots ,n_i,\end{array}$$ (2)

where $W_{ij}=(W_{ij}^1,W_{ij}^2,\dots ,W_{ij}^p{)}^T\in {\mathbb{R}}^p$ and $U_{ij}=(U_{ij}^1,U_{ij}^2,\dots ,U_{ij}^q{)}^T\in {\mathbb{R}}^q$ can be observed directly, $w_{ij}=(w_{ij}^1,w_{ij}^2,\dots ,w_{ij}^p{)}^T\in {\mathbb{R}}^p$ and $u_{ij}=(u_{ij}^1,u_{ij}^2,\dots ,u_{ij}^q{)}^T\in {\mathbb{R}}^q$ are zero-mean measurement errors with diagonal covariance matrix ${\mathrm{\Sigma }}_w$ and ${\mathrm{\Sigma }}_u$ respectively. In addition, we assume that $\text{cov}(w_{i{j}_1},w_{i{j}_2})=0,\text{cov}(u_{i{j}_1},u_{i{j}_2})=0$ for $j_1\ne j_2$, $w_{ij}$ and $u_{ij}$ are independent of each other and are all independent of $(X_{ij},Z_{ij},t_{ij},{ϵ}_{ij})$, where $\text{cov}(\cdot )$ denotes the covariance operator. Although these assumptions are not the weakest possible condition, to deal with measurement errors, we need extra information about ${\mathrm{\Sigma }}_w$ and ${\mathrm{\Sigma }}_u$ in practice. For example, we usually assume that ${\mathrm{\Sigma }}_w$ and ${\mathrm{\Sigma }}_u$ are known or can be estimated.

Model (2) was studied in the literatures. For the case that only $X_{ij}$ is measured with additive error, You and Chen [30] proposed two different estimators for parametric and nonparametric components with cross-sectional data. Empirical likelihood inference can be seen in Hu et al. [9], Zhao and Xue [35], Xia and Da[28], Zhou et al. [38], Fan et al. [3] and Wang et al. [25]. The cases for some linear covariates are unobserved with available ancillary variables can been in Zhou and Liang [37]. Variable selection procedure for the high-dimensional situation was studied by Wang and Xue [26]. The estimation and testing problems can be seen in Zhang et al. [33]. Wang and Zou [22] studied the model average problem of model (2). Wei [27] proposed a restricted modified profile least squares estimator for the parametric components.

For the models that only $Z_{ij}$ is unobserved and measured with additive error, empirical likelihood inference and Local bias-corrected restricted profile least squares estimators can be used for model estimation [2,6]. For generalized partial linear varying coefficient model with some linear covariates being error prone but ancillary variables being available, Zhang et al. [31] proposed a variable selection method. Zhao and Xue [36] proposed a variable selection method for the case that $X_{ij}$ and $Z_{ij}$ are measured with errors simultaneously based on the cross-sectional data.

On the other hand, model selection is an important topic for longitudinal data analysis. Some valuable research can be seen in Tian and Xue [19], Zhao et al. [34]. As far as we know, there is no study being reported on model selection for model (2) with longitudinal data when $X_{ij}$ and $Z_{ij}$ are measured with additive errors simultaneously. Taking this issue into account, inspired by [34] and [19], we mainly study the variable selection for model (2). In view of the advantages of quadratic inference functions (QIF)[16] versus generalized estimating equations (GEE) [14], a bias-corrected penalized quadratic inference functions (pQIF) method for model (2) is proposed in this paper, which can estimate and select non-zero regression parameters and coefficient functions simultaneously. Furthermore, the asymptotic properties of the proposed method and estimators are constructed.

The rest of this paper is organized as follows. In section 2, we propose the bias-corrected pQIF method. In section 3, we study the asymptotic properties of model estimation and selection results. Some issues in practical implementation are presented in section 4. Simulation studies and a real example analysis are presented in section 5. In Section 6, we present a brief conclusion and discussion of the results and methods. The proofs of some asymptotic results are provided in the Appendix.

2. Model estimation and selection method

Denote B-spline basis vector $B\left(t\right)$ with the order d as $B\left(t\right)=\left(B_1\right(t),B_2(t),\dots ,B_L(t){)}^T$, where L = K + d, $K(>0)$ is the number of interior knots. Hence, following He et al. [8], ${\alpha }_l\left(t\right)(l=1,2,\dots ,q)$ can be represented approximately as

$${\alpha }_l\left(t\right)\approx B(t{)}^T{\gamma }_l,l=1,2,\dots ,q,$$ (3)

where ${\gamma }_l\in {\mathbb{R}}^{K+d}$ is a regression coefficient vector of B-spline basis.

Replace ${\alpha }_l\left(t\right)(l=1,2,\dots ,q)$ by (3), model (2) can be represented as

$$\begin{array}{r}\left\{\begin{array}{l}{Y}_{ij}\approx X_{ij}^T\mathrm{\text{β}}+{\tilde{Z}}_{ij}^T\gamma +{ϵ}_{ij}\\ W_{ij}=X_{ij}+w_{ij}\\ {\tilde{U}}_{ij}={\tilde{Z}}_{ij}+{\tilde{u}}_{ij}\end{array}\right.,\end{array}$$ (4)

where $\gamma =({\gamma }_1^T,{\gamma }_2^T,\dots ,{\gamma }_q^T{)}^T$, $B_{ij}=I_q\otimes B\left(t_{ij}\right)$, ${\tilde{Z}}_{ij}=B_{ij}{Z}_{ij}$, ${\tilde{U}}_{ij}=B_{ij}{U}_{ij}$, ${\tilde{u}}_{ij}=B_{ij}{u}_{ij}$, $I_q$ is the $q\times q$ identity matrix. From the assumption of model (2), we can see that $w_{ij}$ and ${\tilde{u}}_{ij}$ are independent of each other, and are all independent of $(X_{ij},Z_{ij},t_{ij},{ϵ}_{ij})$, $E\left({\tilde{u}}_{ij}\right)=0$, $\text{cov}\left({\tilde{u}}_{ij}\right)={\mathrm{\Sigma }}_{\tilde{u}}$, ${\mathrm{\Sigma }}_{\tilde{u}}=B_{ij}{\mathrm{\Sigma }}_u{B}_{ij}^T$, $\text{cov}({\tilde{u}}_{i{j}_1},{\tilde{u}}_{i{j}_2})=0$ for $j_1\ne j_2$. From (4), we can get the GEE about $\theta =({\mathrm{\text{β}}}^T,{\gamma }^T{)}^T$ as

$$\sum _{i=1}^n{(W_i,{\tilde{U}}_i)}^T{V}_i^{-1}(Y_i-(W_i,{\tilde{U}}_i\left)\theta \right)=0$$ (5)

Then we can get

$$\begin{array}{rl}E& \left(\sum _{i=1}^n\left\{{(W_i,{\tilde{U}}_i)}^T{V}_i^{-1}\right(Y_i-(W_i,{\tilde{U}}_i)\theta \left)\right\}\right)\\ & =E\left(\sum _{i=1}^n\left\{{(X_i,{\tilde{Z}}_i)}^T{V}_i^{-1}\right(Y_i-(X_i,{\tilde{Z}}_i)\theta \left)\right\}\right)-nE\left((w_i,{\tilde{u}}_i{)}^T{V}_i^{-1}\right(w_i,{\tilde{u}}_i\left)\theta \right)\\ & =-nE\left((w_i,{\tilde{u}}_i{)}^T{V}_i^{-1}\right(w_i,{\tilde{u}}_i\left)\theta \right)\ne 0.\end{array}$$

This shows that Equation (5) is biased, then we can get the bias-corrected GEE about θ as

$$\sum _{i=1}^n\left\{{(W_i,{\tilde{U}}_i)}^T{V}_i^{-1}(Y_i-W_i\mathrm{\text{β}}-{\tilde{U}}_i\gamma )+D_i\theta \right\}=0,$$ (6)

where $W_i=(W_{i1},W_{i2},\dots ,W_{i{n}_i}{)}^T$, ${\tilde{U}}_i=({\tilde{U}}_{i1},{\tilde{U}}_{i2},\dots ,{\tilde{U}}_{i{n}_i}{)}^T$, $Y_i=(Y_{i1},Y_{i2},\dots ,Y_{i{n}_i}{)}^T$, $D_i=E(w_i,{\tilde{u}}_i{)}^T{V}_i^{-1}(w_i,{\tilde{u}}_i))$, $w_i=(w_{i1},w_{i2},\dots ,w_{i{n}_i}{)}^T$, ${\tilde{u}}_i=({\tilde{u}}_{i1}^T,{\tilde{u}}_{i2}^T,\dots ,{\tilde{u}}_{i{n}_i}^T{)}^T$, $V_i$ is the covariance of $Y_i$. Obviously, equation (6) is unbiased. From the GEE method, we take $V_i$ as $V_i=A_i^{1/2}{R}_i\left(\rho \right)A_i^{1/2}$, where $A_i=\text{diag}\left(\text{var}\right(Y_{i1}),\text{var}(Y_{i2}),\dots ,\text{var}(Y_{i{n}_i}\left)\right)=\text{diag}\left(\text{var}\right({ϵ}_{i1}),\text{var}({ϵ}_{i2}),\dots ,\text{var}({ϵ}_{i{n}_i}\left)\right)$, $R_i\left(\rho \right)$ is a working correlation matrix, ρ is a nuisance parameter. Liang and Zeger [14] pointed out that consistent estimator of ρ may not exist in some certain simple cases, which may invalidate the GEE method.

To overcome this drawback of the GEE, Qu and Li [17] proposed the QIF method to analyze longitudinal data by assuming that $R_i^{-1}\left(\rho \right)=\sum _{\kappa =1}^s{a}_{\kappa }{M}_{\kappa }$, where $M_{\kappa }(\kappa =1,2,\dots ,s)$ are some simple known matrices, $a_{\kappa }(\kappa =1,2,\dots ,s)$ are unknown constants. This approach treats $a_{\kappa }(\kappa =1,2,\dots ,s)$ as nuisance parameters [17]. Substituting it into (6), we get the new bias-corrected GEE as

$$\begin{array}{r}\sum _{i=1}^n\left\{{(W_i,{\tilde{U}}_i)}^T{A}_i^{-1/2}\left(\sum _{\kappa =1}^s{a}_{\kappa }{M}_{\kappa }\right)A_i^{-1/2}(Y_i-W_i\mathrm{\text{β}}-{\tilde{U}}_i\gamma )+\right.\\ \left.E\left({(w_i,{\tilde{u}}_i)}^T{A}_i^{-1/2}\left(\sum _{\kappa =1}^s{a}_{\kappa }{M}_{\kappa }\right)A_i^{-1/2}(w_i,{\tilde{u}}_i)\right)\theta \right\}=0.\end{array}$$ (7)

Unlike the GEE method, we do not need to estimate $a=(a_1,a_2,\dots ,a_s)$. Instead, define the bias-corrected extended score function ${\overline{g}}_n\left(\theta \right)$ as

$$\begin{array}{r}{\overline{g}}_n\left(\theta \right)=\frac{1}{n}\sum _{i=1}^n{g}_i\left(\theta \right)=\frac{1}{n}\sum _{i=1}^n\left(\begin{array}{c}{(W_i,{\tilde{U}}_i)}^T{A}_i^{-1/2}{M}_1{A}_i^{-1/2}(Y_i-W_i\mathrm{\text{β}}-{\tilde{U}}_i\gamma )+D_i^{\left(1\right)}\theta \\ ⋮\\ {(W_i,{\tilde{U}}_i)}^T{A}_i^{-1/2}{M}_s{A}_i^{-1/2}(Y_i-W_i\mathrm{\text{β}}-{\tilde{U}}_i\gamma )+D_i^{\left(s\right)}\theta \end{array}\right),\end{array}$$ (8)

where $D_i^{\left(\kappa \right)}=E\left((w_i,{\tilde{u}}_i{)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}\right(w_i,{\tilde{u}}_i\left)\right)$, $\kappa =1,2,\dots ,s$. Obviously, $w_{ij}$ and $u_{ij}$ are independent of each other, so we can get that, $w_i$ and ${\tilde{u}}_i$ are independent of each other and $E\left(w_i^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}{\tilde{u}}_i\right)=0$ and $E\left({\tilde{u}}_i^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}{w}_i\right)=0$. we can get

$$\begin{array}{r}{D}_i^{\left(\kappa \right)}=\left(\begin{array}{cc}{D}_{11,i}^{\left(\kappa \right)}& 0\\ 0& D_{22,i}^{\left(\kappa \right)}\end{array}\right),\end{array}$$ (9)

where $D_{11,i}^{\left(\kappa \right)}=E\left(w_i^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}{w}_i\right)$, $D_{22,i}^{\left(\kappa \right)}=E\left({\tilde{u}}_i^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}{\tilde{u}}_i\right)$, $\kappa =1,2,\dots ,s$. By some simple matrix calculations, following as Zhao et al. (2020)[34], we have

$$\begin{array}{rl}{D}_{11,i}^{\left(\kappa \right)}& =\text{tr}\left(A_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}\right)\cdot {\mathrm{\Sigma }}_w\end{array}$$ (10)

$$\begin{array}{rl}{D}_{22,i}^{\left(\kappa \right)}& ={\mathrm{\Sigma }}_u\otimes \left(B_i\text{diag}\right(A_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}\left)B_i^T\right)\end{array}$$ (11)

where $B_i=\left(B\right(t_{i1}),B(t_{i2}),\dots ,B(t_{i{n}_i}\left)\right)$, $\text{diag}(\cdot )$ denotes a diagonal matrix operator. However, the covariance matrix ${\mathrm{\Sigma }}_u$ and ${\mathrm{\Sigma }}_v$ are usually unknown in advance, so we need to estimate ${\mathrm{\Sigma }}_u$ in practice. Under some conditions, ${\mathrm{\Sigma }}_u$ and ${\mathrm{\Sigma }}_v$ can usually be estimated by partial replication similar as [1].

If the longitudinal data is balanced, that is, $n_i=n_0<\mathrm{\infty }(i=1,2,\dots ,n)$. Suppose that $W_{ij}$ and $U_{ij}$ can be observed $m_i$ times for ith subject, $W_{ij}^{\left(r\right)}=X_{ij}+w_{ij}^{\left(r\right)}$, $U_{ij}^{\left(r\right)}=Z_{ij}+u_{ij}^{\left(r\right)}$, $r=1,2,\dots ,m_i$, we can get two consistent, unbiased estimators ${\hat{\mathrm{\Sigma }}}_w$ and ${\hat{\mathrm{\Sigma }}}_u$ for ${\mathrm{\Sigma }}_w$ and ${\mathrm{\Sigma }}_u$, respectively, as

$$\begin{array}{rl}{\hat{\mathrm{\Sigma }}}_w& =\frac{1}{n{n}_0}\sum _{i=1}^n\sum _{j=1}^{n_0}\left(\frac{1}{m_i-1}\sum _{r=1}^{m_i}(W_{ij}^{\left(r\right)}-{\overline{W}}_{ij}){(W_{ij}^{\left(r\right)}-{\overline{W}}_{ij})}^T\right)\end{array}$$ (12)

$$\begin{array}{rl}{\hat{\mathrm{\Sigma }}}_u& =\frac{1}{n{n}_0}\sum _{i=1}^n\sum _{j=1}^{n_0}\left(\frac{1}{m_i-1}\sum _{r=1}^{m_i}(U_{ij}^{\left(r\right)}-{\overline{U}}_{ij}){(U_{ij}^{\left(r\right)}-{\overline{U}}_{ij})}^T\right)\end{array}$$ (13)

where ${\overline{W}}_{ij}=m_i^{-1}{\sum }_{i=1}^{m_i}{W}_{ij}^{\left(r\right)}$, ${\overline{U}}_{ij}=m_i^{-1}{\sum }_{i=1}^{m_i}{U}_{ij}^{\left(r\right)}$. Furthermore, we can get two consistent, unbiased estimators ${\hat{D}}_{11,i}^{\left(\kappa \right)}$ and ${\hat{D}}_{22,i}^{\left(\kappa \right)}$ for $D_{11,i}^{\left(\kappa \right)}$ and $D_{22,i}^{\left(\kappa \right)}$, respectively, as

$$\begin{array}{rl}{\hat{D}}_{11,i}^{\left(\kappa \right)}& =\text{tr}\left(A_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}\right)\cdot {\hat{\mathrm{\Sigma }}}_w\end{array}$$ (14)

$$\begin{array}{rl}{\hat{D}}_{22,i}^{\left(\kappa \right)}& ={\hat{\mathrm{\Sigma }}}_u\otimes \left(B_i\text{diag}\right(A_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}\left)B_i^T\right)\end{array}$$ (15)

Substituting (14) and (15) into (9), we can get a consistent, unbiased estimator ${\hat{D}}_i^{\left(\kappa \right)}$ for $D_i^{\left(\kappa \right)}$ as

$$\begin{array}{r}{\hat{D}}_i^{\left(\kappa \right)}=\left(\begin{array}{cc}{\hat{D}}_{11,i}^{\left(\kappa \right)}& 0\\ 0& {\hat{D}}_{22,i}^{\left(\kappa \right)}\end{array}\right),\kappa =1,2,\dots ,s.\end{array}$$ (16)

If the longitudinal data is unbalanced, following Xue et al. [29], it can be reformulated to balanced, the details are omitted here.

According to (16), we can get a estimator ${\hat{\overline{g}}}_n\left(\theta \right)$ for ${\overline{g}}_n\left(\theta \right)$ as

$$\begin{array}{r}{\hat{\overline{g}}}_n\left(\theta \right)=\frac{1}{n}\sum _{i=1}^n{\hat{g}}_i\left(\theta \right)=\frac{1}{n}\sum _{i=1}^n\left(\begin{array}{c}{(W_i,{\tilde{U}}_i)}^T{A}_i^{-1/2}{M}_1{A}_i^{-1/2}(Y_i-W_i\mathrm{\text{β}}-{\tilde{U}}_i\gamma )+{\hat{D}}_i^{\left(1\right)}\theta \\ ⋮\\ {(W_i,{\tilde{U}}_i)}^T{A}_i^{-1/2}{M}_s{A}_i^{-1/2}(Y_i-W_i\mathrm{\text{β}}-{\tilde{U}}_i\gamma )+{\hat{D}}_i^{\left(s\right)}\theta \end{array}\right).\end{array}$$ (17)

Obviously, ${\hat{\overline{g}}}_n\left(\theta \right)$ is a $s(p+q(K+d\left)\right)\times 1$ vector, however, θ is a $(p+q(K+d\left)\right)\times 1$ parameter vector. Equation $E\left({\hat{\overline{g}}}_n\right(\theta \left)\right)=0$ is over-identified and can not be used to solve the $\hat{\theta }$. To solve this problem, following Qu et al. [16], we construct the bias-corrected QIF about θ as

$$Q_n\left(\theta \right)=n{\hat{\overline{g}}}_n^T\left(\theta \right){\mathrm{\Omega }}_n^{-1}{\hat{\overline{g}}}_n\left(\theta \right),$$ (18)

where ${\mathrm{\Omega }}_n=\frac{1}{n}{\sum }_{i=1}^n{\hat{g}}_i\left(\theta \right){\hat{g}}_i^T\left(\theta \right)$. Furthermore, we can get $\tilde{\theta }$ as

$$\tilde{\theta }=\arg \underset{\theta }{min}{Q}_n\left(\theta \right).$$ (19)

As mentioned above, the bias-corrected QIF can correct the bias of estimating equations and handle within-subject correlations simultaneously. However, the bias-corrected QIF method for nonparametric coefficient functions is the spline regression approach and usually over-fitted. Not only that, but the true model is unknown in practice. To solve these issues, we construct the bias-corrected pQIF to estimate and select significant parameters and varying coefficients simultaneously, defined as

$$Q_p\left(\theta \right)=Q_n\left(\theta \right)+n\sum _{k=1}^p{p}_{{\lambda }_{1k}}\left(|{\mathrm{\text{β}}}_k|\right)+n\sum _{l=1}^q{p}_{{\lambda }_{2l}}\left({\parallel {\gamma }_l\parallel }_H\right),$$ (20)

where $\parallel {\gamma }_l{\parallel }_H=({\gamma }_l^{T}H{\gamma }_l{)}^{1/2}$, $H=(h_{ij}{)}_{L\times L}$, $h_{ij}={\int }_0^1{B}_i\left(t\right)B_j^T\left(t\right)dt$, $p_{\lambda }(\cdot )$ is the SCAD penalty function [5] defined as

$$p_{\lambda }^{\mathrm{\prime }}\left(w\right)=\lambda \left\{I(w\le \lambda )+\frac{{(a\lambda -w)}_{+}}{(a-1)\lambda }I(w>\lambda )\right\},$$ (21)

where a = 3.7, w>0 and $p_{\lambda }\left(0\right)=0$, λ is a tuning parameter and measures the amount of penalty. Therefore, denote ${\lambda }_{1k}(k=1,2,\dots ,p)$ and ${\lambda }_{2l}(l=1,2,\dots ,q)$ for ${\mathrm{\text{β}}}_k(k=1,2,\dots ,p)$ and ${\alpha }_l\left(t\right)(l=1,2,\dots ,q)$ respectively in (20). The bias-corrected pQIF estimator $\hat{\theta }$ is given by

$$\hat{\theta }=({\widehat{\mathrm{\text{β}}}}^T,{\hat{\gamma }}^T{)}^T=\arg \underset{\theta }{min}{Q}_p(\theta ).$$ (22)

Furthermore, the estimators of ${\mathrm{\text{β}}}_l\left(t\right)(l=1,2,\dots ,q)$ can be obtained by

$${\hat{\alpha }}_l\left(t\right)=B(t{)}^T{\hat{\gamma }}_l,l=1,2,\dots ,q.$$ (23)

3. Asymptotic properties

We now construct the asymptotic properties of ${\widehat{\mathrm{\text{β}}}}_k(k=1,2,\dots ,p)$ and ${\hat{\alpha }}_l\left(t\right)(l=1,2,\dots ,q)$. Firstly, let ${\mathrm{\text{β}}}_0=({\mathrm{\text{β}}}_{10},{\mathrm{\text{β}}}_{20},\dots ,{\mathrm{\text{β}}}_{p0}{)}^T$ and ${\alpha }_0\left(t\right)=\left({\alpha }_{10}\right(t),{\alpha }_{20}(t),\dots ,{\alpha }_{q0}(t){)}^T$ be the true regression parameters and coefficient functions, ${\gamma }_{l0}(l=1,2,\dots ,q)$ be B-spline regression coefficient vectors from the spline approximation to ${\alpha }_{l0}\left(t\right)(l=1,2,\dots ,q)$. Furthermore, we assume that

$$\begin{array}{rl}& {\mathrm{\text{β}}}_{k0}\ne 0(k=1,2,\dots ,p_1),{\mathrm{\text{β}}}_{k0}=0(k=p_1+1,p_1+2,\dots ,p),\\ & {\alpha }_{l0}\left(t\right)\ne 0(l=1,2,\dots ,q_1),{\alpha }_{l0}\left(t\right)=0(l=q_1+1,q_2+2,\dots ,q).\end{array}$$

Some necessary regularity conditions for the asymptotic properties are as follows.

• C1: $0<n_i<\mathrm{\infty }$ for $i=1,2,\dots ,n$.
• C2: ${\alpha }_l\left(t\right)(l=1,2,\dots ,q)$ are rth continuously differentiable on $(0,1)$, where $r\ge 2$.
• C3:∃ unique ${\theta }_0\in \mathrm{\Theta }$ satisfies $E\left({\hat{\overline{g}}}_n\right({\theta }_0\left)\right)=o\left(1\right)$, where Θ is the parameter space.
• C4:∃ invertible ${\mathrm{\Omega }}_0$ such that ${\mathrm{\Omega }}_n⟹a.s.{\mathrm{\Omega }}_0$.
• C5: $E\left({ϵ}_i{ϵ}_i^T\right)=V_i,\underset{i}{sup}\parallel V_i\parallel <\mathrm{\infty }$ , and ∃ $\delta >0$ such that $\underset{i}{sup}E\left\{{\parallel {ϵ}_i\parallel }^{2+\delta }\right\}<\mathrm{\infty }$, $E\parallel w_i{\parallel }^8<\mathrm{\infty }$, $E\parallel u_i{\parallel }^8<\mathrm{\infty }$, where $\parallel \cdot \parallel$ is the modulus of the largest singular values.
• C6: $A_i\ge 0$, $\underset{i}{sup}\parallel A_i\parallel <\mathrm{\infty }$.
• C7: $E\parallel X_i{\parallel }^4<\mathrm{\infty }$, $E\parallel Z_i{\parallel }^4<\mathrm{\infty }$, $i=1,2,\dots ,n$.
• C8:Denote interior knots as $\{{\tau }_i,i=1,2,\dots ,K\}$ and satisfy $\underset{1\le i\le K}{max}|\mathrm{\Delta }{\tau }_{i+1}-\mathrm{\Delta }{\tau }_i|=o\left(K^{-1}\right)$ and $\frac{\mathrm{\Delta }{\tau }_{max}}{\mathrm{\Delta }{\tau }_{min}}\le C$, where $C\ge 0$, $\mathrm{\Delta }{\tau }_{max}=\underset{1\le i\le K}{max}{\tau }_i,\mathrm{\Delta }{\tau }_{min}=\underset{1\le i\le K}{min}{\tau }_i$, $\mathrm{\Delta }{\tau }_i={\tau }_i-{\tau }_{i-1}$, ${\tau }_0=0$, ${\tau }_{K+1}=1$.
• C9:
  ${\dot{\hat{\overline{g}}}}_n\left(\theta \right)=\frac{\mathrm{\partial }{\hat{\overline{g}}}_n\left(\theta \right)}{\mathrm{\partial }\theta }$ exists and is continuous, and according to the weak law of large number, when $\hat{\theta }⟹p{\theta }_0$, ∃ $J_0$ such that

  $$\begin{array}{r}\underset{n\to \mathrm{\infty }}{lim}\frac{1}{n}\sum _{i=1}^{n}E\left(\begin{array}{c}{(W_i,{\tilde{U}}_i)}^T{A}_i^{-1/2}{M}_1{A}_i^{-1/2}(W_i,{\tilde{U}}_i)\\ ⋮\\ {(W_i,{\tilde{U}}_i)}^T{A}_i^{-1/2}{M}_s{A}_i^{-1/2}(W_i,{\tilde{U}}_i)\end{array}\right)\equiv J_0.\end{array}$$ (24)
• C10:Denote $a_n=\underset{k,l}{max}\left\{|p_{{\lambda }_{1k}}^{\mathrm{\prime }\mathrm{\prime }}\left(|{\mathrm{\text{β}}}_{k0}|\right)|,|p_{{\lambda }_{2l}}^{\mathrm{\prime }\mathrm{\prime }}(\parallel {\gamma }_{l0}{\parallel }_H)|,{\mathrm{\text{β}}}_{k0}\ne 0,{\gamma }_{l0}\ne 0\right\}$, then $a_n\to 0$ as $n\to \mathrm{\infty }$.
• C11:
  $p_{\lambda }\left(t\right)$ satisfies

  $$\begin{array}{rl}& \underset{n\to \mathrm{\infty }}{lim inf}\underset{{\mathrm{\text{β}}}_k\to 0^{+}}{lim inf}{\lambda }_{1k}^{-1}{p}_{{\lambda }_{1k}}^{\mathrm{\prime }}\left(|{\mathrm{\text{β}}}_k|\right)>0,k=p_1+1,p_1+2,\dots ,p.\end{array}$$ (25)

  $$\begin{array}{rl}& \underset{n\to \mathrm{\infty }}{lim inf}\underset{{\parallel {\gamma }_l\parallel }_H\to 0}{lim inf}{\lambda }_{2l}^{-1}{p}_{{\lambda }_{2l}}^{\mathrm{\prime }}\left({\parallel {\gamma }_l\parallel }_H\right)>0,l=q_1+1,q_1+2,\dots ,q.\end{array}$$ (26)

Remark 3.1

These conditions are often used in the literatures for nonparametric and semi-parametric statistical inference. C1 implies $N=\sum _{i=1}^n{n}_i=O\left(n\right)$. C2 is the smoothness condition about ${\alpha }_l\left(t\right)(l=1,2,\dots ,q)$ and necessary condition to study the convergence rate of B-spline estimator. C4 and C9 can be easily obtained by the weak law of large numbers when $n\to \mathrm{\infty }$. C3, C5-C7, C9 can be seen in [20]. C8 is necessary for knots of B-spline basis approximations [18]. C10 and C11 can be seen in [5,20,36].

According these conditions above, some asymptotic properties about resulting estimators are presented as follows.

Theorem 3.1

If C1-C11 hold, and $K=O\left(N^{1/(2r+1)}\right)$, we have

$$|{\hat{\alpha }}_l(\cdot )-{\alpha }_{l0}(\cdot )|=O_p\left(n^{-r/(2r+1)}\right),l=1,2,\dots ,q.$$ (27)

Theorem 3.2

If C1-C11 hold, and $K=O\left(N^{1/(2r+1)}\right)$, let ${\lambda }_{max}=\underset{k,l}{max}\{{\lambda }_{1k},{\lambda }_{2l}\}$, ${\lambda }_{min}=\underset{k,l}{min}\{{\lambda }_{1k},{\lambda }_{2l}\}$ satisfy ${\lambda }_{max}\to 0$, $n^{r/(2r+1)}{\lambda }_{min}\to +\mathrm{\infty }$, then with probability tending to 1, we have

1. ${\widehat{\mathrm{\text{β}}}}_k(\cdot )=0,k=p_1+1,\dots ,p,$

2. ${\hat{\alpha }}_l(\cdot )\equiv 0,l=q_1+1,\dots ,q.$

Theorem 3.3

Denote ${\widehat{\mathrm{\text{β}}}}^{\ast }=({\widehat{\mathrm{\text{β}}}}_1,{\widehat{\mathrm{\text{β}}}}_2,\dots ,{\widehat{\mathrm{\text{β}}}}_{p_1}{)}^T$ as the estimator of ${\mathrm{\text{β}}}^{\ast }=({\mathrm{\text{β}}}_1,{\mathrm{\text{β}}}_2,\dots ,{\mathrm{\text{β}}}_{p_1}{)}^T$. If C1-C11 hold, and $K=O\left(N^{1/(2r+1)}\right)$, we have

$$\sqrt{n}({\widehat{\mathrm{\text{β}}}}^{\ast }-{\mathrm{\text{β}}}_0^{\ast })⟹LN(0,{\mathbf{A}}_0(J_{{\theta }_0^{\ast }}{\mathrm{\Omega }}_0^{-1}{J}_{{\theta }_0^{\ast }}^T{)}^{-1}{\mathbf{A}}_0^T)$$ (28)

where ${\mathbf{A}}_0$ is denoted as Equation (A11) in Appendix, “ $⟹L$” represents the convergence in distribution.

Remark 3.2

Theorem 1 shows that the estimators of varying coefficients have the optimal convergence rate, Theorem 2 shows that the estimators of constant coefficients and varying coefficients have sparse property. From Theorem 1-3, we know that the proposed method possesses the oracle property.

4. Computational algorithm and selection of tuning parameters

4.1. Computational algorithm

It is obvious that $\hat{\theta }$ by (22) does not have closed form and $Q_n(\cdot )$ is irregular at the origin which means that we can only get numerical solution of $\hat{\theta }$. Therefore, $Q_n(\cdot )$ can be approximated around a given point ${\theta }^{\left(0\right)}$ using Taylor expansion as

$$Q_n\left(\theta \right)\approx Q_n\left({\theta }^{\left(0\right)}\right)+{\dot{Q}}_n\left({\theta }^{\left(0\right)}{)}^T\right(\theta -{\theta }^{\left(0\right)})+\frac{1}{2}(\theta -{\theta }^{\left(0\right)}{)}^T{\ddot{Q}}_n({\theta }^{\left(0\right)}\left)\right(\theta -{\theta }^{\left(0\right)}),$$

where ${\dot{Q}}_n(\cdot )=\frac{\mathrm{\partial }{Q}_n(\cdot )}{\mathrm{\partial }\theta }$ and ${\ddot{Q}}_n(\cdot )=\frac{\mathrm{\partial }\dot{\phantom{a}}{Q}_n(\cdot )}{\mathrm{\partial }\theta }$. On the other hand, $p_{\lambda }(\cdot )$ can be approximated as

$$p_{\lambda }\left(|t|\right)\approx p_{\lambda }\left(|t_0|\right)+\frac{1}{2}\frac{p_{\lambda }^{\mathrm{\prime }}\left(|t_0|\right)}{|t_0|}(t^2-t_0^2),\mathrm{f}\mathrm{o}\mathrm{r}t\approx t_0,$$

where $t_0$ is an initial value. Therefore, apart from a constant, the bias-corrected pQIF can represented as

$$\begin{array}{rl}{Q}_p\left(\theta \right)& \approx Q_n\left({\theta }^{\left(0\right)}\right)+{\dot{Q}}_n\left({\theta }^{\left(0\right)}{)}^T\right(\theta -{\theta }^{\left(0\right)})\\ & +\frac{1}{2}(\theta -{\theta }^{\left(0\right)}{)}^T{\ddot{Q}}_n\left({\theta }^{\left(0\right)}\right)(\theta -{\theta }^{\left(0\right)})+\frac{n}{2}{\theta }^T{\mathrm{\Sigma }}_{\lambda }\left({\theta }^{\left(0\right)}\right)\theta ,\end{array}$$ (29)

where

$$\begin{array}{rl}& {\mathrm{\Sigma }}_{\lambda }\left({\theta }^{\left(0\right)}\right)\\ & =\text{diag}\left\{\frac{p_{{\lambda }_{11}}^{\mathrm{\prime }}\left(|{\mathrm{\text{β}}}_1^{\left(0\right)}|\right)}{|{\mathrm{\text{β}}}_1^{\left(0\right)}|},\dots ,\frac{p_{{\lambda }_{1p}}^{\mathrm{\prime }}\left(|{\mathrm{\text{β}}}_p^{\left(0\right)}|\right)}{|{\mathrm{\text{β}}}_p^{\left(0\right)}|},\frac{p_{{\lambda }_{21}}^{\mathrm{\prime }}(\parallel \gamma {}_1^{\left(0\right)}{\parallel }_H)}{\parallel {\gamma }_1^{\left(0\right)}{\parallel }_H}H,\dots ,\frac{p_{{\lambda }_{2q}}^{\mathrm{\prime }}(\parallel {\gamma }_q^{\left(0\right)}{\parallel }_H)}{\parallel {\gamma }_q^{\left(0\right)}{\parallel }_H}H\right\}.\end{array}$$

According to (29), $\hat{\theta }$ can be solved by following calculation algorithm

$${\theta }^{\left(1\right)}\approx {\theta }^{\left(0\right)}-\left\{{\ddot{Q}}_n\right({\theta }^{\left(0\right)})+n{\mathrm{\Sigma }}_{\lambda }({\theta }^{\left(0\right)}){\}}^{-1}\left\{{\dot{Q}}_n\right({\theta }^{\left(0\right)})+n{\mathrm{\Sigma }}_{\lambda }({\theta }^{\left(0\right)}\left){\theta }^{\left(0\right)}\right\}.$$

The detailed computational algorithm iterative calculation method is shown below.

• Step 1: Take the bias-corrected QIF estimator $\tilde{\theta }$ denoted by (19) as ${\theta }^{\left(0\right)}$.

• Step 2: Update $\hat{\theta }$ at the $(k+1)$th iteration by

  $${\theta }^{(k+1)}\approx {\theta }^{\left(k\right)}-\left\{{\ddot{Q}}_n\right({\theta }^{\left(k\right)})+n{\mathrm{\Sigma }}_{\lambda }({\theta }^{\left(k\right)}){\}}^{-1}\left\{{\dot{Q}}_n\right({\theta }^{\left(k\right)})+n{\mathrm{\Sigma }}_{\lambda }({\theta }^{\left(k\right)}\left){\theta }^{\left(k\right)}\right\}.$$

• Step 3: Repeat Step 2 until certain convergence criterion is satisfied.

4.2. Selection of tuning parameters

As all we know, ${\lambda }_{1k}$ and ${\lambda }_{2l}$ control the amount of penalty and determine the values of the penalty function $p_{{\lambda }_{1k}}(\cdot )(k=1,2,\dots ,p)$ and $p_{{\lambda }_{2l}}(l=1,2,\dots ,q)$. However, they are unknown in practice. These mean that unknown ${\lambda }_{1k}$ and ${\lambda }_{2l}$ determine the results of model estimation and selection indirectly. Thus, it is important for selection of ${\lambda }_{1k}$ and ${\lambda }_{2l}$ in the implementation. As Wang et al. [21] presented, the BIC criterion for SCAD estimator can select the true model with probability tending to one. In our work, we apply the BIC criterion to select the optimal tuning parameters ${\lambda }_{1k}$ and ${\lambda }_{2l}$.

However, it is a challenge to select p + q parameters simultaneously in real applications. A wise way for selection of ${\lambda }_{1k}(k=1,2,\dots ,p)$ and ${\lambda }_{2l}(l=1,2,\dots ,q)$ is to give a larger value to a zero parameter or a zero coefficient function than to a non-zero parameter or non-zero coefficient function. This method aims to give more amount of penalty to zero parameters or zero coefficient functions than to non-zero parameters or non-zero coefficient functions, which is good for selecting significantly non-zero parameters or non-zero coefficient functions and can reduce computational complexity. This kind of tuning parameters usually are called adaptive tuning parameters. The proposed method using the adaptive tuning parameters can estimate large parameters and coefficient functions unbiasedly and shrink the small ones toward zero simultaneously. Thus, denote ${\lambda }_{1k}(k=1,2,\dots ,p)$ and ${\lambda }_{2l}(l=1,2,\dots ,q)$ as

$${\lambda }_{1k}=\frac{\lambda }{|{\widetilde{\mathrm{\text{β}}}}_k|},{\lambda }_{2l}=\frac{\lambda }{{\parallel {\tilde{\gamma }}_l\parallel }_H},$$

where ${\widetilde{\mathrm{\text{β}}}}_k(k=1,2,\dots ,p)$ and ${\tilde{\gamma }}_l(l=1,2,\dots ,q)$ are defined by (19). Consequently, the selection of ${\lambda }_{1k}(k=1,2,\dots ,p)$ and ${\lambda }_{2l}(l=1,2,\dots ,q)$ becomes a problem of selection of $\lambda$, which is an easier univariate problem and can reduce computational complexity greatly. Define BIC as

$$\mathrm{B}\mathrm{I}\mathrm{C}\left(\lambda \right)=Q_n\left({\hat{\theta }}_{\lambda }\right)+d{f}_{\lambda }\cdot \log \left(n\right),$$ (30)

where ${\hat{\theta }}_{\lambda }=({\widehat{\mathrm{\text{β}}}}_{\lambda }^T,{\hat{\gamma }}_{\lambda }^T{)}^T$ is defined by (22) for a given λ, $d{f}_{\lambda }$ is the number of non-zero parameters and coefficients of ${\widehat{\mathrm{\text{β}}}}_{1\lambda },{\widehat{\mathrm{\text{β}}}}_{2\lambda },\dots ,{\widehat{\mathrm{\text{β}}}}_{p\lambda }$ and $\parallel {\hat{\gamma }}_{1\lambda }{\parallel }_H,\parallel {\hat{\gamma }}_{2\lambda }{\parallel }_H,\dots ,\parallel {\hat{\gamma }}_{q\lambda }{\parallel }_H$, ${\widehat{\mathrm{\text{β}}}}_{\lambda }=({\widehat{\mathrm{\text{β}}}}_{1\lambda },{\widehat{\mathrm{\text{β}}}}_{2\lambda },\dots ,{\widehat{\mathrm{\text{β}}}}_{p\lambda }{)}^T$, ${\hat{\gamma }}_{\lambda }=({\hat{\gamma }}_{1\lambda }^T,{\hat{\gamma }}_{2\lambda }^T,\dots ,{\hat{\gamma }}_{q\lambda }^T{)}^T$. So we can get the optimal $\hat{\lambda }$ as

$$\hat{\lambda }=\underset{\lambda }{min}\mathrm{B}\mathrm{I}\mathrm{C}\left(\lambda \right)$$ (31)

In practice, $\hat{\lambda }$ can be obtained by the grid searching method.

5. Numerical studies

5.1. Simulations studies

We conducted some numerical simulations to asses the performance of the bias-corrected pQIF method in terms of estimation accuracy and selection performance in finite samples. Firstly, the generalized mean square error (GMSE) [20,36] is defined as

$$\mathrm{G}\mathrm{M}\mathrm{S}\mathrm{E}=(\widehat{\mathrm{\text{β}}}-\mathrm{\text{β}}{)}^{T}E(X{X}^T\left)\right(\widehat{\mathrm{\text{β}}}-\mathrm{\text{β}}).$$

Obviously, the smaller the GMSE, the better the estimation effect for β. The square root of average square (RASE) is defined as

$$\mathrm{R}\mathrm{A}\mathrm{S}\mathrm{E}={\left\{\frac{1}{m}\sum _{l=1}^q\sum _{\ell =1}^m{\left[{\hat{\alpha }}_l\left(t_{\ell }\right)-{\alpha }_l\left(t_{\ell }\right)\right]}^2\right\}}^{1/2}.$$

Smaller RASE indicates better estimation accuracy which implies $\widehat{\mathrm{\text{β}}}\left(t\right)$ is more closer to the true function $\alpha \left(t\right)$. In our work, we set m = 200, and $t_{\ell }(\ell =1,2,\dots ,m)$ are equally spaced on $[0,1]$.

‘C’ in tables below means the average number of ${\widehat{\mathrm{\text{β}}}}_k=0(k=p_1+1,\dots ,p)$ or ${\alpha }_l\left(t\right)=0(l=q_1+1,q_1+2.\cdots ,q)$, and ‘IC’ denotes the average number of ${\widehat{\mathrm{\text{β}}}}_k=0(k=1,2,\dots ,p_1)$ or ${\alpha }_l\left(t\right)=0(l=1,2,\dots ,q_1)$. Obviously, larger ‘C’ and smaller ‘IC’ imply better model selection results. The performance of the bias-corrected pQIF method is assessed by the GMSE, RASE, ‘C’ and ‘IC’ simultaneously.

In our simulation studies, for model (2), let $\mathrm{\text{β}}=({\mathrm{\text{β}}}_1,{\mathrm{\text{β}}}_2,{\mathrm{\text{β}}}_3,{\mathrm{\text{β}}}_4{)}^T$ with ${\mathrm{\text{β}}}_1=2$, ${\mathrm{\text{β}}}_2=0.7$, ${\mathrm{\text{β}}}_k=0(k=3,4)$, $\alpha \left(t\right)=\left({\alpha }_1\right(t),{\alpha }_2(t),\dots ,{\alpha }_6(t){)}^T$, ${\alpha }_l\left(t\right)\equiv 0(l=3,4,5,6)$ and

$${\alpha }_1\left(t\right)=7.5+0.1\exp (3t-1),{\alpha }_2\left(t\right)=\sin \left(2\pi t\right)$$

We took $X_{ij}\sim N(2,{\sigma }_X^2{I}_4)$, $Z_{ij}\sim N(2,{\sigma }_Z^2{I}_6)$, $w_{ij}\sim N(0,{\sigma }_w^2{I}_4)$, $u_{ij}\sim N(0,{\sigma }_u^2{I}_6)$, where $j=1,2,\dots ,10$, ${\sigma }_X={\sigma }_Z=2$, $I_4$ is $4\times 4$ identify matrix, $I_6$ is $6\times 6$ identify matrix. We set ${\sigma }_w={\sigma }_u$ as 0.2, 0.4, 0.6. $t_{ij}\sim U[0,1]$. ${ϵ}_i=({ϵ}_{i1},{ϵ}_{i2},\dots ,{ϵ}_{i{n}_i}{)}^T\sim N(0,{\sigma }^2\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}({ϵ}_{\mathrm{i}},\rho ))$, where ${\sigma }^2=1$ and $\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}({ϵ}_{\mathrm{i}},\rho ))$ is a known correlation matrix with parameter ρ. So we can get $A_i=\text{diag}(1,1,\dots ,1)$. We set $n_i=10$ and considered ${ϵ}_i$ has the first-order autoregressive (AR(1)) and exchangeable (EX) correlation structures with $\rho =0.3$ and $\rho =0.7$. We generated n = 150, 200, 300 subjects. The cubic B-spline basis was applied, the knots were equally spaced in $[0,1]$, $K=\lfloor c\times N^{1/5}\rfloor$, where $\lfloor c\rfloor$ denotes the largest integer less than c [8]. Following Tian et al. [20], we choose c = 0.6.

For each simulated longitudinal data, we compared the bias-corrected pQIF method with the LASSO and the SCAD penalty functions and the one neglecting measurement errors with SCAD penalty function (denoted as ‘nSCAD’). For the sake of simplicity, the bias-corrected pQIF with LASSO and SCAD penalty functions are denoted by ‘LASSO’ and ‘SCAD’ respectively. ${\hat{\lambda }}_{1k}(k=1,2,\dots ,p)$ and ${\hat{\lambda }}_{2l}(l=1,2,\dots ,q)$ were chosen by (31). Furthermore, we did 500 simulation runs under each simulation setup and presented the median of GMSE and RASE in the following tables.

In summary, from Tables 1 to 4, we can get some conclusions as follows:

1. The performances of the LASSO and SCAD methods are much more better than the nSCAD method in all of cases, which implies that bias-corrected method we proposed is valuable and neglecting measurement errors results in biased estimation and poor variable selection results for model (2).

2. Under the same conditions, the performance of the SCAD and LASSO methods become better as the sample size becomes larger. Furthermore, the SCAD methods is better than the LASSO method as for estimation and selection for parametric and nonparametric parts.

3. Under the same conditions, the SCAD and LASSO methods become worse when the measurement error increases. There is little difference between the performance of the SCAD and LASSO methods when the measurement error is small. However, the SCAD method is significantly better than the LASSO method when the measurement error is large, which implies that the LASSO method is less robust than that of the SCAD method.

Table 2.

Variable selections for $\alpha (\cdot )$ with the EX correlation structure.

			n = 150			n = 200			n = 300
ρ	${\sigma }_u$	Method	C	IC	RASE	C	IC	RASE	C	IC	RASE
$0.3$	0.2	LASSO	3.442	0	0.11303	3.798	0	0.09892	3.972	0	0.08785
		SCAD	3.502	0	0.11307	3.808	0	0.09879	3.974	0	0.08779
		nSCAD	3.410	0	0.13200	3.764	0	0.11801	3.948	0	0.10970
	0.4	LASSO	3.378	0	0.17904	3.696	0	0.14573	3.916	0	0.12109
		SCAD	3.414	0	0.17833	3.766	0	0.14346	3.956	0	0.11848
		nSCAD	3.134	0	0.30538	3.524	0	0.29042	3.850	0	0.27680
	0.6	LASSO	3.140	0	0.26483	3.628	0	0.20670	3.888	0	0.16369
		SCAD	3.282	0	0.25851	3.674	0	0.20247	3.922	0	0.15911
		nSCAD	2.834	0	0.60510	3.178	0	0.57348	3.588	0	0.56202
$0.7$	0.2	LASSO	3.520	0	0.10967	3.752	0	0.09818	3.970	0	0.08738
		SCAD	3.538	0	0.10946	3.776	0	0.09803	3.972	0	0.08711
		nSCAD	3.434	0	0.12746	3.804	0	0.11733	3.958	0	0.10913
	0.4	LASSO	3.334	0	0.17337	3.684	0	0.14602	3.940	0	0.12035
		SCAD	3.386	0	0.17244	3.710	0	0.14444	3.954	0	0.11865
		nSCAD	3.136	0	0.30833	3.502	0	0.29786	3.856	0	0.27834
	0.6	LASSO	3.180	0	0.26489	3.522	0	0.20862	3.900	0	0.16294
		SCAD	3.242	0	0.26246	3.662	0	0.20480	3.938	0	0.15718
		nSCAD	2.688	0	0.61163	3.132	0	0.58300	3.600	0	0.56400

Table 3.

Variable selections for β with the AR(1) correlation structure.

			n = 150			n = 200			n = 300
ρ	${\sigma }_u$	Method	C	IC	GMSE	C	IC	GMSE	C	IC	GMSE
$0.3$	0.2	LASSO	1.834	0	0.00027	1.950	0	0.00018	1.992	0	0.00010
		SCAD	1.842	0	0.00017	1.960	0	0.00011	1.992	0	7.6E-05
		nSCAD	1.796	0	0.00079	1.880	0	0.00061	1.964	0	0.00060
	0.4	LASSO	1.380	0	0.00676	1.586	0	0.00184	1.754	0	0.00601
		SCAD	1.396	0	0.00154	1.594	0	0.00113	1.758	0	0.00075
		nSCAD	0.988	0	0.03460	0.954	0	0.00621	0.892	0	0.00111
	0.6	LASSO	1.028	0	0.03337	1.150	0	0.03183	1.408	0	0.03737
		SCAD	1.060	0	0.00929	1.178	0	0.00516	1.410	0	0.00418
		nSCAD	0.448	0	0.24760	0.286	0	0.16020	0.162	0	0.09212
$0.7$	0.2	LASSO	1.884	0	0.00026	1.934	0	0.00015	1.992	0	0.00011
		SCAD	1.886	0	0.00014	1.936	0	0.00010	1.996	0	6.9E-05
		nSCAD	1.814	0	0.00092	1.904	0	0.00076	1.948	0	0.00062
	0.4	LASSO	1.420	0	0.00301	1.588	0	0.00662	1.794	0	0.00126
		SCAD	1.474	0	0.00168	1.614	0	0.00107	1.824	0	0.00084
		nSCAD	1.046	0	0.00856	0.982	0	0.00197	0.886	0	0.00628
	0.6	LASSO	0.984	0	0.03522	1.212	0	0.03437	1.428	0	0.04007
		SCAD	0.994	0	0.00885	1.236	0	0.00583	1.436	0	0.00406
		nSCAD	0.460	0	0.23031	0.308	0	0.16011	0.136	0	0.08534

Table 1.

Variable selections for β with the EX correlation structure.

			n = 150			n = 200			n = 300
ρ	${\sigma }_u$	Method	C	IC	GMSE	C	IC	GMSE	C	IC	GMSE
$0.3$	0.2	LASSO	1.834	0	0.00025	1.950	0	0.00017	1.988	0	0.00010
		SCAD	1.850	0	0.00014	1.960	0	9.7E-05	1.988	0	7.6E-05
		nSCAD	1.802	0	0.00089	1.878	0	0.00072	1.974	0	0.00068
	0.4	LASSO	1.420	0	0.00339	1.546	0	0.00200	1.784	0	0.00120
		SCAD	1.448	0	0.00151	1.580	0	0.00116	1.792	0	0.00077
		nSCAD	0.892	0	0.00762	1.000	0	0.00683	1.038	0	0.00534
	0.6	LASSO	1.046	0	0.01934	1.180	0	0.01422	1.404	0	0.00954
		SCAD	1.082	0	0.00820	1.200	0	0.00610	1.438	0	0.00435
		nSCAD	0.188	0	0.03413	0.314	0	0.03321	0.478	0	0.03836
$0.7$	0.2	LASSO	1.882	0	0.00028	1.962	0	0.00017	1.994	0	0.00012
		SCAD	1.892	0	0.00013	1.968	0	9.3E-05	1.996	0	6.1E-05
		nSCAD	1.870	0	0.00099	1.926	0	0.00085	1.984	0	0.00070
	0.4	LASSO	1.456	0	0.00340	1.626	0	0.00192	1.752	0	0.00121
		SCAD	1.466	0	0.00176	1.598	0	0.00117	1.784	0	0.00079
		nSCAD	0.974	0	0.00883	0.994	0	0.00723	1.000	0	0.00630
	0.6	LASSO	1.024	0	0.03529	1.262	0	0.01561	1.438	0	0.00424
		SCAD	1.068	0	0.00821	1.264	0	0.00594	1.448	0	0.00421
		nSCAD	0.170	0	0.03655	0.330	0	0.03483	0.496	0	0.02724

Table 4.

Variable selections for $\alpha (\cdot )$ with the AR(1) correlation structure.

			n = 150			n = 200			n = 300
ρ	${\sigma }_u$	Method	C	IC	RASE	C	IC	RASE	C	IC	RASE
$0.3$	0.2	LASSO	3.446	0	0.11341	3.790	0	0.10114	3.952	0	0.08951
		SCAD	3.454	0	0.11330	3.816	0	0.10087	3.958	0	0.08931
		nSCAD	3.488	0	0.13156	3.716	0	0.11831	3.960	0	0.10979
	0.4	LASSO	3.324	0	0.18076	3.672	0	0.14687	3.924	0	0.11895
		SCAD	3.396	0	0.17887	3.744	0	0.14504	3.954	0	0.11742
		nSCAD	3.064	0	0.30761	3.546	0	0.29106	3.836	0	0.27361
	0.6	LASSO	3.132	0	0.26516	3.628	0	0.20858	3.868	0	0.16263
		SCAD	3.276	0	0.25738	3.732	0	0.19892	3.934	0	0.15858
		nSCAD	2.694	0	0.60629	2.986	0	0.57842	3.468	0	0.55978
$0.7$	0.2	LASSO	3.416	0	0.11313	3.780	0	0.09946	3.964	0	0.08908
		SCAD	3.474	0	0.11247	3.806	0	0.09909	3.966	0	0.08878
		nSCAD	3.396	0	0.13017	3.710	0	0.11741	3.968	0	0.10843
	0.4	LASSO	3.278	0	0.17854	3.706	0	0.14404	3.924	0	0.11779
		SCAD	3.372	0	0.17591	3.782	0	0.14356	3.950	0	0.11729
		nSCAD	3.150	0	0.30647	3.496	0	0.28994	3.808	0	0.27429
	0.6	LASSO	3.088	0	0.26305	3.574	0	0.20768	3.86	0	0.16407
		SCAD	3.226	0	0.26080	3.67	0	0.20441	3.92	0	0.15910
		nSCAD	2.656	0	0.6041	3.018	0	0.58127	3.522	0	0.56007

5.2. Real example analysis

We now describe the performance of the proposed method through analysis of the AIDS dataset. This dataset contains some variables such as the mean CD4 percentage, smoking status, the pre-HIV infection CD4 percentage and age. It is unbalanced and available in the R package timereg. More details of the study design and medical implications can be found in [11]. It has been analyzed to illustrate partial linear varying coefficient models [20] and partial linear varying coefficient EV models [35]. Zhao et al. [37] and Tian et al. [20] indicated that only the baseline function varies over time and pre-CD4 has a constant effect over time. We now consider measurement errors for the covariates and analyze this dataset using the proposed method.

For simplicity, following Zhao and Xue [36], we considered the following model. Let Y be the individuals CD4 percentage, $X_1$ be the centered preCD4 percentage, $X_2=X_1^2$, $Z_1$ be the centered age at HIV infection, $Z_2=Z_1^2$.

$$Y=X_1{\mathrm{\text{β}}}_1+X_2{\mathrm{\text{β}}}_2+{\alpha }_0\left(t\right)+Z_1{\alpha }_1\left(t\right)+Z_2{\alpha }_2\left(t\right)+ϵ$$ (32)

where ${\alpha }_0\left(t\right)$ is the baseline of CD4 percentage; ${\mathrm{\text{β}}}_1$ and ${\mathrm{\text{β}}}_2$ describe the first-order and second-order effects of preCD4 percentage, ${\alpha }_1\left(t\right)$ and ${\alpha }_2\left(t\right)$ describes the first-order and second-order effects of the age at HIV infection, t is the visiting time for each patient.

For the AIDS dataset, we cannot get repeated measurements of the covariates or estimate the variance of the measurement error. Following Lin and Carroll (2000), a sensitivity analysis can be used to test the practicability of the proposed method. Similar as in Zhao and Xue (2010)[36], we considered $X_1$ and $Z_1$ have additive measurement errors as follows

$$W_1=X_1+w_1,U_1=Z_1+u_1$$

where $w_1\sim N(0,{\sigma }_w^2)$, $u_1\sim N(0,{\sigma }_u^2)$. In our work, we took ${\sigma }_w={\sigma }_u=0,0.5,1$ to represent different measurement errors. It is obvious that ${\sigma }_w={\sigma }_u=0$ implies no measurement errors.

We repeated the proposed model selection procedure and the proposed method identified two non-zero coefficients ${\mathrm{\text{β}}}_1$ and ${\alpha }_0\left(t\right)$ every time under different measurement errors, which means that the first-order or second-order effects of age at HIV infection have no significant impact on the mean CD4 percentage. The same goes for the second-order effect of the centered preCD4 percentage and the interaction effect between the preCD4 percentage and age at HIV infection. Our result is same as in Zhao et al. (2009).

Figure 1 shows the curve of ${\hat{\alpha }}_0\left(t\right)$ over time under different measurement errors. It shows that ${\alpha }_0\left(t\right)$ decreases quickly at the beginning of HIV infection, and the rate of decrease slows down, which is similar as in Zhao and Xue [36]. Furthermore, we found that the estimated functional curve ${\hat{\alpha }}_0\left(t\right)$ preserves its shape under different measurement errors, which means that our bias-corrected model selection scheme works well. This further demonstrates that the proposed model estimation and selection method has good practical value.

Figure 1.

The curve of ${\hat{\alpha }}_0\left(t\right)$ for the cases ${\sigma }_w={\sigma }_u=0$ (solid curve), ${\sigma }_w={\sigma }_u=0.5$ (dashed curve) and ${\sigma }_w={\sigma }_u=1$(dotted curve).

6. Conclusion and discussion

Longitudinal data are widely used in some scientific fields, and it is of great significance to consider measurement error in longitudinal research. Longitudinal data have unknown within-subject correlations. Thus, the processing of within-subject correlations and measurement errors is an important subject for analysis for longitudinal data with measurement errors. In our work, we consider cases where covariates of model (2) have additive measurement errors. For model (2), some scholars have done valuable researches, such as [9,33,35–37]. However, no studies have been reported on model estimation and selection simultaneously for model (2) with longitudinal data. In our work, we proposed a bias-corrected penalized quadratic inference functions method to do model estimation and selection for model (2) with longitudinal data. This method can deal with both within-subject correlation and measurement errors. Under some conditions, the proposed method can select significant non-zero parameters and varying coefficients. Furthermore, the estimators of non-zero coefficient functions achieve the optimal convergence rate, the estimators of parameters are asymptotic normal. The performance of the proposed method in the case of finite samples can be demonstrated by numerical studies. Finally, it can be concluded that the proposed method has good theoretical and practical value for model estimation and selection of model (2).

The proposed method can also be applied to other models, such as generalized partial linear additive models, generalized partial linear single index models and many others. In addition, the proposed method can also be used for other types of correlated data analysis, such as panel data, clustered data and so on. In future, we will use this method to study more complex models.

Appendix. Proof of theorems.

Lemma 1. If C1-C11 hold, and $K=O\left(N^{1/(2r+1)}\right)$, then we have

$${\dot{\hat{\overline{g}}}}_n\left(\mathrm{\text{β}}\right)⟹p-J_0,\sqrt{n}{\hat{\overline{g}}}_n\left({\theta }_0\right)⟹LN(0,{\mathrm{\Omega }}_0).$$

Proof.

According to (17), we have

$$\begin{array}{r}{\dot{\hat{\overline{g}}}}_n\left(\theta \right)=\frac{1}{n}\sum _{i=1}^n\left(\begin{array}{c}-{(W_i,{\tilde{U}}_i)}^T{A}_i^{-1/2}{M}_1{A}_i^{-1/2}(W_i,{\tilde{U}}_i)+{\hat{D}}_i^{\left(1\right)}\\ ⋮\\ -{(W_i,{\tilde{U}}_i)}^T{A}_i^{-1/2}{M}_s{A}_i^{-1/2}(W_i,{\tilde{U}}_i)+{\hat{D}}_i^{\left(s\right)}\end{array}\right).\end{array}$$

Denote the κth block matrix of ${\dot{\hat{\overline{g}}}}_n\left(\mathrm{\text{β}}\right)$ as ${\dot{\hat{\overline{g}}}}_{n\kappa }\left(\mathrm{\text{β}}\right)$, $\kappa =1,2,\dots ,s$

$$\begin{array}{rl}{\dot{\hat{\overline{g}}}}_{n\kappa }\left(\theta \right)& =-\frac{1}{n}\sum _{i=1}^n\left({(W_i,{\tilde{U}}_i)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}\right(W_i,{\tilde{U}}_i)-{\hat{D}}_i^{\left(k\right)})\\ & =-\frac{1}{n}\sum _{i=1}^n\left({({\mathit{X}}_i+w_i,{\tilde{Z}}_i+{\tilde{u}}_i)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}\right({\mathit{X}}_i+w_i,{\tilde{Z}}_i+{\tilde{u}}_i)-{\hat{D}}_i^{\left(\kappa \right)})\\ & =-\frac{1}{n}\sum _{i=1}^n\left({({\mathit{X}}_i,{\tilde{Z}}_i)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}\right({\mathit{X}}_i,{\tilde{Z}}_i)+{({\mathit{X}}_i,{\tilde{Z}}_i)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}(w_i,{\tilde{u}}_i)\\ & +(w_i,{\tilde{u}}_i{)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}({\mathit{X}}_i,{\tilde{Z}}_i)+(w_i,{\tilde{u}}_i{)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}(w_i,{\tilde{u}}_i)-{\hat{D}}_i^{\left(k\right)})\\ & =-({\mathrm{\Delta }}_1+{\mathrm{\Delta }}_2+{\mathrm{\Delta }}_3+{\mathrm{\Delta }}_4-\frac{1}{n}\sum _{i=1}^n{\hat{D}}_i^{\left(\kappa \right)})\end{array}$$

Now, we prove ${\mathrm{\Delta }}_4-\frac{1}{n}{\sum }_{i=1}^n{\hat{D}}_i^{\left(k\right)}⟹p0$ as $n\to \mathrm{\infty }$.

$$\begin{array}{rl}{\mathrm{\Delta }}_4-\frac{1}{n}\sum _{i=1}^n{\hat{D}}_i^{\left(k\right)}& =\frac{1}{n}\sum _{i=1}^n(w_i,{\tilde{u}}_i{)}^T{A}_i^{-1/2}{M}_k{A}_i^{-1/2}(w_i,{\tilde{u}}_i)-D_i^{\left(k\right)}+D_i^{\left(k\right)}-\frac{1}{n}\sum _{i=1}^n{\hat{D}}_i^{\left(k\right)}\end{array}$$

Clearly, according to the law of large numbers, we have $\frac{1}{n}{\sum }_{i=1}^n(w_i,{\tilde{u}}_i{)}^T{A}_i^{-1/2}{M}_k{A}_i^{-1/2}(w_i,{\tilde{u}}_i)-D_i^{\left(k\right)}⟹p0$ and $D_i^{\left(k\right)}-\frac{1}{n}\sum _{i=1}^n{\hat{D}}_i^{\left(k\right)}⟹p0$ as the $n\to \mathrm{\infty }$. So we get ${\mathrm{\Delta }}_4-\frac{1}{n}{\sum }_{i=1}^n{\hat{D}}_i^{\left(\kappa \right)}⟹p0$. Under C9, we can get ${\mathrm{\Delta }}_1⟹p{J}_0^{\left(\kappa \right)}$. Now, let's prove that ${\mathrm{\Delta }}_2⟹p0$ and ${\mathrm{\Delta }}_3⟹p0$.

Denote ${\mathrm{\Delta }}_2=\frac{1}{n}{\sum }_{i=1}^n{\xi }_{i\kappa }$, where ${\xi }_{i\kappa }={({\mathit{X}}_i,{\tilde{Z}}_i)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}(w_i,{\tilde{u}}_i).$ Obviously, we can get $E\left({\xi }_{i\kappa }\right)=0$ and

$$\text{cov}\left({\xi }_{i\kappa }\right)={({\mathit{X}}_i,{\tilde{Z}}_i)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}E(w_i,{\tilde{u}}_i)(w_i,{\tilde{u}}_i{)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}({\mathit{X}}_i,{\tilde{Z}}_i)$$

where $E\left(\right(w_i,{\tilde{u}}_i\left)\right(w_i,{\tilde{u}}_i{)}^T)=\text{diag}(E\left(w_i{w}_i^T\right),E\left({\tilde{u}}_i{\tilde{u}}_i^T\right))$. From C4-C7, we see that $E\left(w_i{w}_i^T\right)$ and $E\left({\tilde{u}}_i{\tilde{u}}_i^T\right))$ are bounded. By the law of large numbers, we can get ${\mathrm{\Delta }}_3^T={\mathrm{\Delta }}_2⟹p0$. Thus, we have ${\dot{\hat{\overline{g}}}}_{n\kappa }\left(\theta \right)⟹p-J_0^{\left(k\right)}$ and ${\dot{\hat{\overline{g}}}}_n\left(\theta \right)⟹p-J_0$ where $J_0=(J_0^{\left(1\right)},J_0^{\left(2\right)},\dots ,J_0^{\left(s\right)}{)}^T$.

According to the Taylor expansion to ${\hat{\overline{g}}}_n\left(\theta \right)$ at ${\theta }_0$, we have

$${\hat{\overline{g}}}_n\left(\theta \right)={\hat{\overline{g}}}_n\left({\theta }_0\right)+{\dot{\hat{\overline{g}}}}_n\left({\theta }_0\right)(\theta -{\theta }_0)+o(\theta -{\theta }_0).$$ (A1)

Denote the κth block matrix of ${\hat{\overline{g}}}_n\left({\theta }_0\right)$ as ${\hat{\overline{g}}}_{n\kappa }\left({\theta }_0\right)$, $\kappa =1,2,\dots ,s$

$$\begin{array}{rl}{\hat{\overline{g}}}_{n\kappa }\left({\theta }_0\right)& =\frac{1}{n}\sum _{i=1}^n\left({(W_i,{\tilde{U}}_i)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}\right(Y_i-W_i{\mathrm{\text{β}}}_0-{\tilde{U}}_i{\gamma }_0)+{\hat{D}}_i^{\left(\kappa \right)}{\theta }_0)\\ & =\frac{1}{n}\sum _{i=1}^n\left({\left(\right(X_i,{\tilde{Z}}_i)+(w_i,{\tilde{u}}_i\left)\right)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}\right({ϵ}_i-(w_i,{\tilde{u}}_i){\theta }_0+Z_{i}R\left(t_i\right))+{\hat{D}}_i^{\left(\kappa \right)}{\theta }_0)\\ & =\frac{1}{n}\sum _{i=1}^n{(X_i,{\tilde{Z}}_i)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}{ϵ}_i-\frac{1}{n}\sum _{i=1}^n{(X_i,{\tilde{Z}}_i)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}(w_i,{\tilde{u}}_i){\theta }_0\\ & +\frac{1}{n}\sum _{i=1}^n{(w_i,{\tilde{u}}_i)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}{Z}_{i}R\left(t_i\right)+\frac{1}{n}\sum _{i=1}^n{(w_i,{\tilde{u}}_i)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}{ϵ}_i\\ & +\frac{1}{n}\sum _{i=1}^n{(X_i,{\tilde{Z}}_i)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}{Z}_{i}R\left(t_i\right)\\ & -\frac{1}{n}\sum _{i=1}^n{(w_i,{\tilde{u}}_i)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}(w_i,{\tilde{u}}_i){\theta }_0+\frac{1}{n}\sum _{i=1}^n{D}_i^{\left(\kappa \right)}{\theta }_0\\ & =J_1-J_2+J_3+J_4+J_5-J_6+\frac{1}{n}\sum _{i=1}^n{D}_i^{\left(\kappa \right)}{\theta }_0\end{array}$$

where $R\left(t\right)=\left(R_1\right(t),R_2(t),\dots ,R_q(t){)}^T$, $R_l\left(t\right)={\alpha }_l\left(t\right)-B^T\left(t\right){\gamma }_{l0},l=1,2,\dots ,q$.

Denote $J_1=\frac{1}{n}{\sum }_{i=1}^n{\phi }_i$, where ${(X_i,{\tilde{Z}}_i)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}{ϵ}_i$. According to C5-C7 and Lemma 1, we have $E\left({\phi }_i\right)=0$ and

$$\text{cov}\left({\phi }_i\right)=(X_i,{\tilde{Z}}_i{)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}{V}_i{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}(X_i,{\tilde{Z}}_i)<\mathrm{\infty }$$

By the law of large numbers, we get $J_1⟹p0$. Similarly, we have $J_2⟹p0$ and $J_3⟹p0$.

Denote $J_4=\frac{1}{n}{\sum }_{i=1}^n{\varphi }_i$, where ${\varphi }_i=(w_i,{\tilde{u}}_i{)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}{ϵ}_i$. And since ${ϵ}_i,w_i,{\tilde{u}}_i$ are independent of each other, we have $E\left({\varphi }_i\right)=0$. According to the Cauchy-Schwarz inequality and C5-C7 we have

$$\left(\text{cov}\right({\phi }_i){)}^2=E\left((w_i,{\tilde{u}}_i{)}^T{A}_i^{-1/2}{M}_k{A}_i^{-1/2}\right(w_i,{\tilde{u}}_i\left)\right)E\left({ϵ}_i^T{A}_i^{-1/2}{M}_k{A}_i^{-1/2}{ϵ}_i\right)<\mathrm{\infty }$$

Thus, $J_4⟹p0$. By the law of large numbers, from the definition of ${\hat{D}}_i^{\left(\kappa \right)}$, we have $J_6-\frac{1}{n}{\sum }_{i=1}^n{\hat{D}}_i^{\left(k\right)}{\theta }_0⟹p0$. From C8 and Lemma 1, we have $J_5=O_p\left(n^{-1/2}{K}^{-r}\right)=o_p\left(n^{-1/2}\right)$ and $J_3=o_p\left(n^{-1/2}\right)$. So, according to (A1), we have ${\hat{\overline{g}}}_n\left(\theta \right)⟹p{J}_0({\theta }_0-\theta ),\mathrm{\forall }\theta \in \mathrm{\Theta }$.

Following [20], according to the results above, we have

$$\begin{array}{rl}{\hat{\overline{g}}}_{n\kappa }\left({\theta }_0\right)& =\frac{1}{n}\sum _{i=1}^n{\left(\right(X_i,{\tilde{Z}}_i)+(w_i,{\tilde{u}}_i\left)\right)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}({ϵ}_i-(w_i,{\tilde{u}}_i\left){\theta }_0\right)+{\hat{D}}_i^{\left(\kappa \right)}{\theta }_0)+o_p\left(n^{-1/2}\right)\\ & =\frac{1}{n}\sum _{i=1}^n{(X_i,{\tilde{Z}}_i)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}{ϵ}_i-{(X_i,{\tilde{Z}}_i)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}(w_i,{\tilde{u}}_i){\theta }_0+o_p\left(n^{-1/2}\right)\\ & +(w_i,{\tilde{u}}_i{)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}{ϵ}_i-(w_i,{\tilde{u}}_i{)}^T{A}_i^{-1/2}{M}_{\kappa }{A}_i^{-1/2}(w_i,{\tilde{u}}_i){\theta }_0-{\hat{D}}_i^{\left(\kappa \right)}{\theta }_0)+o_p(n^{-1/2})\\ & =\frac{1}{n}\sum _{i=1}^n({\psi }_{i\kappa 1}+{\psi }_{i\kappa 2}+{\psi }_{i\kappa 3}+{\psi }_{i\kappa 4})+o_p\left(n^{-1/2}\right)\\ & =\frac{1}{n}\sum _{i=1}^n{\psi }_{i\kappa }+o_p\left(n^{-1/2}\right)\end{array}$$

where ${\psi }_i=({\psi }_{i1},{\psi }_{i2},\dots ,{\psi }_{is}{)}^T$, ${\psi }_{i\kappa }={\psi }_{i\kappa 1}+{\psi }_{i\kappa 2}+{\psi }_{i\kappa 3}+{\psi }_{i\kappa 4}$. So we have

$${\hat{\overline{g}}}_n\left({\theta }_0\right)=\frac{1}{n}\sum _{i=1}^n{\psi }_i+o_p\left(n^{-1/2}\right),\mathrm{a}\mathrm{n}\mathrm{d}{\mathrm{\Omega }}_n\left({\theta }_0\right)=\frac{1}{n}\sum _{i=1}^n{\psi }_i{\psi }_i^T+o\left(1\right).$$

From C5-C7, we get $E\left({\psi }_{ikm}\right)=0,\text{cov}\left({\psi }_{ikm}\right)<\mathrm{\infty },m=1,2,3,4$. Following the properties of covariance matrix, we have

$$\begin{array}{rl}& \text{cov}\left({\psi }_{ik}\right)\le \sum _{m=1}^4\text{cov}\left({\psi }_{ikm}\right)+\sum _{m\ne l}\sqrt{\text{cov}\left({\psi }_{ikm}\right)\text{cov}\left({\psi }_{ikl}\right)}<\mathrm{\infty },\\ & \mathrm{\forall }a\in R^{s(p+q(K+d\left)\right)},a^{T}a=1,E\left(a^T{\psi }_i\right)=0,\underset{i}{sup}E\parallel a^T{\psi }_i\parallel \le \parallel a^T\parallel \underset{i}{sup}\parallel {\psi }_i{\parallel }^3.\end{array}$$

So $a^T{\psi }_i$ satisfies the Lyapunov condition for the central limit theorem. Thus

$${\left(a^T\sum _{i=1}^n\text{cov}\left({\psi }_i\right)a\right)}^{-1/2}\left(\sum _{i=1}^n{a}^T{\psi }_i\right)⟹LN(0,1).$$

According to the Slutsky Theorem, we have $\sqrt{n}{\hat{\overline{g}}}_n\left({\theta }_0\right)⟹LN(0,{\mathrm{\Omega }}_0),{\hat{\overline{g}}}_n\left({\theta }_0\right)=O_p\left(n^{-1/2}\right)$. The proof of Lemma 1 is completed.

Lemma 1

If C1-C11 hold, $K=O\left(n^{1/(2r+1)}\right)$ then

$$\begin{array}{rl}& \parallel n^{-1}{\dot{Q}}_n\left({\theta }_0\right)-2{\dot{\hat{\overline{g}}}}_n^T\left({\theta }_0\right){\mathrm{\Omega }}_n^{-1}{\hat{\overline{g}}}_n\left({\theta }_0\right)\parallel =O_p\left(n^{-1}\right),\end{array}$$ (A2)

$$\begin{array}{rl}& \parallel n^{-1}{\ddot{Q}}_n\left({\theta }_0\right)-2{\dot{\hat{\overline{g}}}}_n^T\left({\theta }_0\right){\mathrm{\Omega }}_n^{-1}{\dot{\hat{\overline{g}}}}_n\left({\theta }_0\right)\parallel =o_p\left(1\right).\end{array}$$ (A3)

Proof.

The proof of Lemma 2 is similar as Lemma 2 in Tian et al. [20] and is omitted here.

Proof of Theorem 1.

Proof.

Let $\delta =n^{-r/(2r+1)}$, $\mathrm{\text{β}}={\mathrm{\text{β}}}_0+\delta C_1$, $\gamma ={\gamma }_0+\delta C_2$ and $C=(C_1^T,C_2^T{)}^T$. To prove Theorem 1, it is sufficient to show that $\mathrm{\forall }ϵ>0$, ∃ a large constant $C_0$ satisfies

$$P\left\{{inf}_{\parallel C\parallel =C_0}{Q}_p\left(\theta \right)-Q_p\left({\theta }_0\right)\right\}\ge 1-ϵ.$$ (A4)

Obviously, when $ϵ\ge 1$, $1-ϵ<0$, (A4) is always true. Therefore, we consider the case that $ϵ\in (0,1)$. Assume ${\mathrm{\text{β}}}_k=0(k=p_1+1,\dots ,p)$, ${\theta }_l(\cdot )=0(l=q_1+1,\dots ,q)$ and $p_{\lambda }\left(0\right)=0$. Let $\mathrm{\Delta }(\mathrm{\text{β}},\gamma )=\frac{1}{K}\left[Q_p\right(\theta )-Q_p({\theta }_0\left)\right]$, ${\theta }_0=({\mathrm{\text{β}}}_0^T,{\gamma }_0^T{)}^T$, we have

$$\begin{array}{rl}\mathrm{\Delta }(\mathrm{\text{β}},\gamma )& \ge \frac{1}{K}\left[Q_n\right(\theta )-Q_n({\theta }_0\left)\right]+\frac{n}{K}\sum _{k=1}^{p_1}\left[p_{{\lambda }_{1k}}\right(|{\mathrm{\text{β}}}_k|)-p_{{\lambda }_{1k}}(|{\mathrm{\text{β}}}_{k0}|\left)\right]\\ & +\frac{n}{K}\sum _{l=1}^{q_1}\left[p_{{\lambda }_{2l}}\left({\parallel {\gamma }_l\parallel }_H\right)-p_{{\lambda }_{2l}}\left({\parallel {\gamma }_{l0}\parallel }_H\right)\right]\\ & ={\mathrm{\Delta }}_1+{\mathrm{\Delta }}_2+{\mathrm{\Delta }}_3.\end{array}$$

Apply Taylor expansion to $Q_n\left(\theta \right)$ at ${\theta }_0$, we have

$$Q_n\left(\theta \right)=Q_n({\theta }_0+\delta C)=Q_n\left({\theta }_0\right)+\delta C^T{\dot{Q}}_n\left({\theta }_0\right)+\frac{1}{n};2{\delta }^2{C}^T{\ddot{Q}}_n\left(\tilde{\theta }\right)C,$$

where $\tilde{\theta }$ lies between β and ${\mathrm{\text{β}}}_0$. According to Lemma 1 and Lemma 2, we can get

$$\delta C^T{\dot{Q}}_n\left({\theta }_0\right)=\delta C^T\left\{2n{\dot{\hat{\overline{g}}}}_n^T\right({\theta }_0\left){\mathrm{\Omega }}_n^{-1}{\hat{\overline{g}}}_n\right({\mathrm{\text{β}}}_0)+n{O}_p(n^{-1}\left)\right\}=\parallel C\parallel O_p\left(\sqrt{n}\delta \right)+∥C∥O_p\left(\delta \right),$$

and

$$\begin{array}{rl}{\delta }^2{C}^T{\ddot{Q}}_n\left({\theta }_0\right)C& ={\delta }^2{C}^T\left\{2n{\dot{\hat{\overline{g}}}}_n^T\right({\theta }_0\left){\mathrm{\Omega }}_n^{-1}{\dot{\hat{\overline{g}}}}_n\right({\theta }_0)+n{o}_p(1\left)\right\}C\\ & =n{\delta }^2{C}^T{\dot{\hat{\overline{g}}}}_n^T\left({\theta }_0\right){\mathrm{\Omega }}_n^{-1}{\dot{\hat{\overline{g}}}}_n\left({\theta }_0\right)C+n{\delta }^2\parallel C{\parallel }^2{o}_p\left(1\right).\end{array}$$

Therefore, we have

$$\begin{array}{c}{\mathrm{\Delta }}_1=\frac{1}{K}\{n{\delta }^2\parallel C{\parallel }^2{J}_0^T{\mathrm{\Omega }}_0^{-1}{J}_0+\parallel C\parallel O_p(\sqrt{n}\delta )+\parallel C\parallel O_p(\delta )+n{\delta }^2\parallel C{\parallel }^2{o}_p(1\left)\right\}.\end{array}$$

Obviously, $n{\delta }^2\parallel C{\parallel }^2{J}_0^T{\mathrm{\Omega }}_0^{-1}{J}_0\ge 0$. When C is large enough,

$$n{\delta }^2\parallel C{\parallel }^2{J}_0^T{\mathrm{\Omega }}_0^{-1}{J}_0\ge \parallel C\parallel O_p\left(\sqrt{n}\delta \right),n{\delta }^2\parallel C{\parallel }^2{J}_0^T{\mathrm{\Omega }}_0^{-1}{J}_0\ge n{\delta }^2\parallel C{\parallel }^2{o}_p\left(1\right).$$

So when C is large enough, ${\mathrm{\Delta }}_1>0$. Next, by Taylor expression, we get that

$$\begin{array}{rl}{\mathrm{\Delta }}_2& =\frac{n}{K}\sum _{k=1}^{p_1}\left[p_{{\lambda }_{1k}}\right(|{\mathrm{\text{β}}}_k|)-p_{{\lambda }_{1k}}(|{\mathrm{\text{β}}}_{k0}|\left)\right]\\ & =\frac{1}{K}\sum _{k=1}^{p_1}\left[n\delta p_{{\lambda }_{2k}}^{\mathrm{\prime }}\right(|{\mathrm{\text{β}}}_{k0}|\left)\text{sgn}\right({\mathrm{\text{β}}}_{k0})|C_1|+n{\delta }^2{p}_{{\lambda }_{2k}}^{\mathrm{\prime }\mathrm{\prime }}({\mathrm{\text{β}}}_{k0}\left)|C_1{|}^2\right(1+o\left(1\right)\left)\right]\\ & \le \frac{1}{K}\{\sqrt{p_1}n\delta a_n\parallel C\parallel +n{\delta }^2{a}_n\parallel C{\parallel }^2\}.\end{array}$$

Then, ${\mathrm{\Delta }}_2$ is dominated by ${\mathrm{\Delta }}_1$ uniformly in $\parallel C\parallel =C_0$ for a sufficiently large $C_0$.

Assume ${\lambda }_{1k}\to 0$, ${\lambda }_{2l}\to 0$ and $K=O\left(n^{1/(2r+1)}\right)$. When n is large enough, following Xue et al. [29], we have $\parallel {\gamma }_l{\parallel }_H\ge a\lambda ,\parallel {\gamma }_{l0}{\parallel }_H\ge a{\lambda }_{2l}$. According to the definition of the penalty function, we get

$$p_{{\lambda }_l}\left(\parallel {\gamma }_l{\parallel }_H\right)=p_{{\lambda }_l}\left(\parallel {\gamma }_{l0}{\parallel }_H\right)=\frac{(1+a){\lambda }_l^2}{2},\sum _{l=1}^{q_1}n\left[p_{{\lambda }_l}\right({∥{\gamma }_l∥}_H)-p_{{\lambda }_l}({\parallel {\gamma }_{l0}\parallel }_H\left)\right]=0.$$

So, $\mathrm{\forall }ϵ>0$, ∃ a large enough $C_0$ satisfies (A4), which further implies that there exists $\hat{\theta }$ satisfies $\parallel \hat{\theta }-{\theta }_0\parallel =O_p\left(\delta \right)=O_p\left(n^{-r/(2r+1)}\right)$. Note that

$$\begin{array}{rl}\parallel {\hat{\alpha }}_l\left(t\right)-{\alpha }_l\left(t\right){\parallel }^2& ={\int }_0^1{\left\{B^T\right(t){\hat{\gamma }}_l-B^T(t){\gamma }_{l0}+{\alpha }_l(t)+B^T(t\left){\gamma }_{l0}\right\}}^{2}dt\\ & \le 2{\int }_0^1{\left\{B^T\right(t){\hat{\gamma }}_l-B^T(t\left){\gamma }_{l0}\right\}}^{2}dt+2{\int }_0^1\{{\alpha }_l\left(t\right)-B^T\left(t\right){\mathrm{\text{β}}}_{l0}{\}}^{2}dt\\ & =2({\hat{\gamma }}_l-{\gamma }_{l0}{)}^T\left({\int }_0^1{B}^T\left(t\right)B\left(t\right)dt\right)({\hat{\gamma }}_l-{\gamma }_{l0})+2{\int }_0^1\left\{{\alpha }_l\right(t)-B^T(t){\mathrm{\text{β}}}_{l0}{\}}^{2}dt\\ & =2({\hat{\gamma }}_l-{\gamma }_{l0}{)}^{T}H({\hat{\gamma }}_l-{\gamma }_{l0})+2{\int }_0^1{R}_l(t{)}^{2}dt.\end{array}$$

With the same arguments above, we can get $\parallel \hat{\gamma }-\gamma \parallel =O_p\left(n^{-r/(2r+1)}\right)$. Therefore, invoking $H=O\left(1\right))$, we have $({\hat{\gamma }}_l-{\gamma }_{l0}{)}^{T}H({\hat{\gamma }}_l-{\gamma }_{l0})=O_P(n^{-2r/(2r+1)})$.

Suppose C2 and C8 hold and $K=O\left(N^{1/(2r+1)}\right)$, with the Corollary 6.21 in [18], ∃ a constant $c_0$ that satisfies

$$\underset{t\in [0,1]}{sup}|{\alpha }_l\left(t\right)-B^T\left(t\right){\gamma }_{l0}|\le c_0{K}^{-r},l=1,2,\dots ,q.$$ (A5)

So we get ${\int }_O^1{R}_l(t{)}^{2}dt=O_P(n^{-2r/(2r+1)})$. Thus, the proof of Theorem 1 is complete.

Proof of Theorem 2.

Proof.

Part (i). Denote $Q_p\left(\theta \right)=Q_p(\mathrm{\text{β}},\gamma )$. According to Theorem 1, similar as [20], it suffices to show that, $\mathrm{\forall }\gamma$ that satisfies $\parallel \gamma -{\gamma }_0\parallel =O_p\left(n^{-r/(2r+1)}\right)(k=1,2,\dots ,p_1)$, $\mathrm{\forall }\mathrm{\text{β}}$ that satisfies $\parallel {\mathrm{\text{β}}}_k-{\mathrm{\text{β}}}_0\parallel =O_p\left(n^{-r/(2r+1)}\right)(k=1,2,\dots ,p_1)$, and ∃ small $ϵ=O_p\left(n^{-1/(2r+1)}\right)$, when $n\to \mathrm{\infty }$, with probability tending to one, we have

$$\frac{\mathrm{\partial }{Q}_p(\mathrm{\text{β}},\gamma )}{\mathrm{\partial }{\mathrm{\text{β}}}_k}>0,\mathrm{f}\mathrm{o}\mathrm{r}0<{\mathrm{\text{β}}}_k<ϵ,k=p_1+1,\dots ,p$$ (A6)

and

$$\frac{\mathrm{\partial }{Q}_p(\mathrm{\text{β}},\gamma )}{\mathrm{\partial }{\mathrm{\text{β}}}_k}<0,\mathrm{f}\mathrm{o}\mathrm{r}-ϵ<{\mathrm{\text{β}}}_k<0,k=p_1+1,\dots ,p$$ (A7)

obviously, (A6) and (A7) imply that minimizer of $Q_p(\mathrm{\text{β}},\gamma )$ about β attains at ${\mathrm{\text{β}}}_k=0(k=1,2,\dots ,p)$.

According to Lemma 2, we have

$$\begin{array}{rl}\frac{\mathrm{\partial }Q(\mathrm{\text{β}},\gamma )}{\mathrm{\partial }{\mathrm{\text{β}}}_k}& =2n\frac{\mathrm{\partial }{\hat{\overline{g}}}_n^T(\mathrm{\text{β}},\gamma )}{\mathrm{\partial }{\mathrm{\text{β}}}_k}{\mathrm{\Omega }}_n^{-1}{\hat{\overline{g}}}_n(\mathrm{\text{β}},\gamma )+o_p\left(1\right)+n{p}_{{\lambda }_{1k}}^{\mathrm{\prime }}\left(|{\mathrm{\text{β}}}_k|\right)\text{sgn}\left({\mathrm{\text{β}}}_k\right)\\ & =n{\lambda }_{1k}\left\{2{\lambda }_{1k}^{-1}\frac{\mathrm{\partial }{\hat{\overline{g}}}_n^T(\mathrm{\text{β}},\gamma )}{\mathrm{\partial }{\mathrm{\text{β}}}_k}{\mathrm{\Omega }}_n^{-1}{\hat{\overline{g}}}_n\right(\mathrm{\text{β}},\gamma )+{\lambda }_{1k}^{-1}{p}_{{\lambda }_{1k}}^{\mathrm{\prime }}(|{\mathrm{\text{β}}}_k|\left)\text{sgn}\right({\mathrm{\text{β}}}_k\left)\right\}+o_p\left(1\right)\end{array}$$

Denote $\frac{\mathrm{\partial }{\hat{\overline{g}}}_n^T\left(\theta \right)}{\mathrm{\partial }\theta }=(\frac{\mathrm{\partial }{\hat{\overline{g}}}_n^T(\mathrm{\text{β}},\gamma )}{\mathrm{\partial }{\mathrm{\text{β}}}_1},\frac{\mathrm{\partial }{\hat{\overline{g}}}_n^T(\mathrm{\text{β}},\gamma )}{\mathrm{\partial }{\mathrm{\text{β}}}_2},\dots ,\frac{\mathrm{\partial }{\hat{\overline{g}}}_n^T(\mathrm{\text{β}},\gamma )}{\mathrm{\partial }{\mathrm{\text{β}}}_p},\frac{\mathrm{\partial }{\hat{\overline{g}}}_n^T(\mathrm{\text{β}},\gamma )}{\mathrm{\partial }{\gamma }_1},\frac{\mathrm{\partial }{\hat{\overline{g}}}_n^T(\mathrm{\text{β}},\gamma )}{\mathrm{\partial }{\gamma }_2},\dots ,\frac{\mathrm{\partial }{\hat{\overline{g}}}_n^T(\mathrm{\text{β}},\gamma )}{\mathrm{\partial }{\gamma }_q})$. According to Lemma 2, we have $\frac{\mathrm{\partial }{\hat{\overline{g}}}_n^T(\mathrm{\text{β}},\gamma )}{\mathrm{\partial }{\mathrm{\text{β}}}_k}⟹p-J_{{\mathrm{\text{β}}}_k}$, where $J_0=(J_{{\mathrm{\text{β}}}_1},J_{{\mathrm{\text{β}}}_2},\dots ,J_{{\mathrm{\text{β}}}_p},J_{{\gamma }_1},J_{{\gamma }_2},\dots ,J_{{\gamma }_q})$. Thus, we get

$$\frac{\mathrm{\partial }Q(\mathrm{\text{β}},\gamma )}{\mathrm{\partial }{\mathrm{\text{β}}}_k}=n{\lambda }_{1k}\left\{O_p\right({\lambda }_{1k}^{-1}{n}^{-1/2})+{\lambda }_{1k}^{-1}{p}_{{\lambda }_{1k}}^{\mathrm{\prime }}(|{\mathrm{\text{β}}}_k|\left)\text{sgn}\right({\mathrm{\text{β}}}_k\left)\right\}+o_p\left(1\right)$$

In addition, C11 implies that $\underset{n\to \mathrm{\infty }}{lim inf}\underset{{\mathrm{\text{β}}}_k\to 0^{+}}{lim inf}{\lambda }_{1k}^{-1}{p}_{{\lambda }_{1k}}^{\mathrm{\prime }}\left(|{\mathrm{\text{β}}}_k|\right)>0$, and ${\lambda }_{1k}^{-1}{n}^{-1/2}\to 0$, which means that the sign of $\frac{\mathrm{\partial }Q(\mathrm{\text{β}},\gamma )}{\mathrm{\partial }{\mathrm{\text{β}}}_k}$ is same as that of ${\mathrm{\text{β}}}_k$. So, (A6) and (A7) hold, the proof of part (i) is completed.

We then prove part (ii). Denote

$$\begin{array}{rl}{\mathrm{\Theta }}_1& =\{\theta :\theta =({\mathrm{\text{β}}}^T,{\gamma }^T{)}^T,{\gamma }_l=\mathbf{0},l=q_1+1,\dots ,q\},\\ {\mathrm{\Theta }}_l& =\{\theta :\theta =({\mathrm{\text{β}}}^T,{\mathbf{0}}^T,\dots ,{\mathbf{0}}^T,{\gamma }_l^T,{\mathbf{0}}^T,\dots ,{\mathbf{0}}^T),l=q_1+1,\dots ,q\},\end{array}$$

where $\mathbf{0}$ is a $(K+d)\times 1$ vector with all of components being zero.

To prove part (ii), it is sufficient to show that, $\mathrm{\forall }\theta \in {\mathrm{\Theta }}_1$ and ${\theta }_l^{\ast }\in {\mathrm{\Theta }}_l$, $Q_p(\theta +{\theta }_l^{\ast })\ge Q_p\left(\theta \right)$ is true with probability tending to 1.

$$\begin{array}{rl}{Q}_p(\theta +{\theta }_l^{\ast })-Q_p\left(\theta \right)& =Q_n(\theta +{\theta }_l^{\ast })-Q_n\left(\theta \right)+n{p}_{{\lambda }_{2l}}\left({\parallel {\gamma }_l^{\ast }\parallel }_H\right)\\ & =\theta {}_l^{\ast T}{\dot{Q}}_n\left(\theta \right)+\frac{1}{2}\theta {}_l^{\ast T}{\ddot{Q}}_n\left({\hat{\theta }}_l^{\ast }\right){\theta }_l^{\ast }(1+o_p(1\left)\right)+n{p}_{{\lambda }_{2l}}\left({\parallel {\gamma }_l^{\ast }\parallel }_H\right)\\ & =n{\lambda }_{2l}\parallel B^T\left(t\right){\gamma }_l^{\ast }\parallel \left\{\frac{R_l^{\ast }}{{\lambda }_{2l}}+\frac{p_{{\lambda }_{2l}}^{\mathrm{\prime }}\left(t\right)}{{\lambda }_{2l}}\right\}(1+o_p(1\left)\right),\end{array}$$

where ${\hat{\theta }}_l^{\ast }$ lies between $\theta +{\theta }_l^{\ast }$ and θ, $t\in (0,\parallel {\gamma }_l^{\ast }{\parallel }_H)$. Furthermore, we get

$$R_l^{\ast }=\frac{\theta {{}_l^{\ast }}^T{n}^{-1}{\dot{Q}}_n\left({\hat{\theta }}_l^{\ast }\right)+\frac{1}{2}\theta {{}_l^{\ast }}^T{n}^{-1}{\ddot{Q}}_n\left({\hat{\theta }}_l^{\ast }\right){\theta }_l^{\ast }}{\parallel B^T\left(t\right){\gamma }_l^{\ast }\parallel }.$$

Note that ${\alpha }_{l0}=0,l=q_1+1,\dots ,q$, from Lemma 1 and [29], we have $\parallel B^T\left(t\right){\gamma }_l^{\ast }\parallel =O\left(n^{r/(2r+1)}\right)$ and $\parallel B^T\left(t\right){\gamma }_l^{\ast }\parallel {\lambda }_{2l}=O\left(n^{r/(2r+1)}{\lambda }_l\right)\to \mathrm{\infty }$. According to Lemma 2 and Lemma 3, we have

$$\begin{array}{rl}\theta {}_l^{\ast T}{n}^{-1}{\dot{Q}}_n\left({\hat{\theta }}_l^{\ast }\right)& =O_p\left(n^{-1/2}\right)=o_p\left(1\right),\theta {}_l^{\ast T}{n}^{-1}{\ddot{Q}}_n\left({\hat{\theta }}_l^{\ast }\right){\theta }_l^{\ast }=\theta {}_l^{\ast T}{J}_0^T{\mathrm{\Omega }}_0^{-1}{J}_0{\theta }_l^{\ast }+o_p\left(1\right)<+\mathrm{\infty },\\ \frac{R_l^{\ast }}{{\lambda }_{2l}}& =\frac{\theta {{}_l^{\ast }}^T{J}_0^T{\mathrm{\Omega }}_0^{-1}{J}_0{\theta }_l^{\ast }}{\parallel B^T\left(t\right){\gamma }_l^{\ast }\parallel {\lambda }_{2l}}+o_p\left(1\right)\to 0.\end{array}$$

Form C10 and C11, for t lies between 0 and $\parallel {\gamma }_l^{\ast }{\parallel }_H$

$$\underset{n\to \mathrm{\infty }}{lim inf}\underset{{\parallel {\gamma }_l^{\ast }\parallel }_H\to 0}{lim inf}\frac{p_{{\lambda }_{2l}}^{\mathrm{\prime }}\left(t\right)}{{\lambda }_{2l}}>0,l=q_1+1,\dots ,q.$$

Thus, for any $\theta \in \mathrm{\Theta }$ and ${\theta }_l^{\ast }\in {\mathrm{\Theta }}_l$, $Q_p(\theta +{\theta }_l^{\ast })\ge Q_p\left(\theta \right)$ is true with probability tending to 1. The part (ii) is real. So, the proof of Theorem 2 is completed.

Proof of Theorem 3.

Proof.

Let ${\mathrm{\text{β}}}_0^{\ast }$ be the true value of ${\mathrm{\text{β}}}^{\ast }$. Let ${\alpha }^{\ast }\left(t\right)=\left({\alpha }_1\right(t),{\alpha }_2(t),\dots ,{\alpha }_{q_1}(t){)}^T$ and ${\alpha }_0^{\ast }\left(t\right)$ be the true value of ${\alpha }^{\ast }\left(t\right)$, ${\gamma }^{\ast }$ and ${\gamma }_0^{\ast }$ are the spline coefficients of ${\alpha }^{\ast }\left(t\right)$ and ${\alpha }_0^{\ast }\left(t\right)$ respectively. Then, Theorems 1 and 2 imply that $Q_p\left(\theta \right)$ attains the minimal value at $({\widehat{\mathrm{\text{β}}}}^{\ast T},\mathbf{0}{)}^T$ and $({\hat{\gamma }}^{\ast T},\mathbf{0}{)}^T$.

Denote ${\theta }^{\ast }=({\mathrm{\text{β}}}^{\ast T},\mathbf{0},{\gamma }^{\ast T},\mathbf{0}{)}^T$, ${\theta }_0^{\ast }=({\mathrm{\text{β}}}_0^{\ast T},\mathbf{0},{\gamma }_0^{\ast T},\mathbf{0}{)}^T$, and ${\hat{\theta }}^{\ast }=({\widehat{\mathrm{\text{β}}}}^{\ast T},\mathbf{0},{\hat{\gamma }}^{\ast T},\mathbf{0}{)}^T$, write ${\dot{\hat{\overline{g}}}}_n\left(\theta \right)=\frac{\mathrm{\partial }}{\mathrm{\partial }\theta }{\hat{\overline{g}}}_n\left(\theta \right)=\left(\frac{\mathrm{\partial }}{\mathrm{\partial }\mathrm{\text{β}}}{\hat{\overline{g}}}_n\right(\theta ),\frac{\mathrm{\partial }}{\mathrm{\partial }\gamma }{\hat{\overline{g}}}_n(\theta \left)\right)=\left({\dot{\hat{\overline{g}}}}_{\mathrm{\text{β}}}\right(\theta ),{\dot{\hat{\overline{g}}}}_{\gamma }(\theta \left)\right)$, we have

$$\begin{array}{rl}{\mathbf{S}}_{\mathbf{n}}\left(\theta \right)& =\left(\begin{array}{c}{\dot{\hat{\overline{g}}}}_{\mathrm{\text{β}}}^T\left(\theta \right){\mathrm{\Omega }}_n^{-1}{\hat{\overline{g}}}_n\left(\theta \right)\\ {\dot{\hat{\overline{g}}}}_{\gamma }^T\left(\theta \right){\mathrm{\Omega }}_n^{-1}{\hat{\overline{g}}}_n\left(\theta \right)\end{array}\right),{\mathbf{H}}_{\mathbf{n}}\mathbf{\left(}\theta \mathbf{\right)}\mathbf{=}\left(\begin{array}{cc}{\mathbf{H}}_{\mathbf{11}}& {\mathbf{H}}_{\mathbf{12}}\\ {\mathbf{H}}_{\mathbf{21}}& {\mathbf{H}}_{\mathbf{22}}\end{array}\right)\\ & =\left(\begin{array}{cc}{\dot{\hat{\overline{g}}}}_{\mathrm{\text{β}}}^T\left(\theta \right){\mathrm{\Omega }}_n^{-1}{\dot{\hat{\overline{g}}}}_{\mathrm{\text{β}}}\left(\theta \right)& {\dot{\hat{\overline{g}}}}_{\mathrm{\text{β}}}^T\left(\theta \right){\mathrm{\Omega }}_n^{-1}{\dot{\hat{\overline{g}}}}_{\gamma }\left(\theta \right)\\ {\dot{\hat{\overline{g}}}}_{\gamma }^T\left(\theta \right){\mathrm{\Omega }}_n^{-1}{\dot{\hat{\overline{g}}}}_{\mathrm{\text{β}}}\left(\theta \right)& {\dot{\hat{\overline{g}}}}_{\gamma }^T\left(\theta \right){\mathrm{\Omega }}_n^{-1}{\dot{\hat{\overline{g}}}}_{\gamma }\left(\theta \right)\end{array}\right).\end{array}$$

Denote $p_{\lambda }\left(\theta \right)=\sum _{k=1}^p{p}_{{\lambda }_{1k}}\left(|{\mathrm{\text{β}}}_k|\right)+\sum _{l=1}^q{p}_{{\lambda }_{2l}}\left({\parallel {\gamma }_l\parallel }_H\right)$. According to (22) we have

$${\dot{Q}}_p\left({\hat{\theta }}^{\ast }\right)={\dot{Q}}_n\left({\hat{\theta }}^{\ast }\right)+n{\dot{p}}_{\lambda }\left({\hat{\theta }}^{\ast }\right){\hat{\theta }}^{\ast }=0.$$ (A8)

Applying the Taylor expression to (A8), we have

$${\dot{Q}}_n\left({\hat{\theta }}^{\ast }\right)+n{\dot{p}}_{\lambda }\left({\hat{\theta }}^{\ast }\right){\hat{\theta }}^{\ast }={\dot{Q}}_n\left({\hat{\theta }}_0^{\ast }\right)+n{\dot{p}}_{\lambda }\left({\hat{\theta }}_0^{\ast }\right){\hat{\theta }}_0^{\ast }+\left\{{\ddot{Q}}_n\right({\hat{\theta }}_0^{\ast })+n{\ddot{p}}_{\lambda }({\tilde{\theta }}_0^{\ast }\left)\right\}({\hat{\theta }}^{\ast }-{\hat{\theta }}_0^{\ast })=0,$$ (A9)

where ${\tilde{\theta }}_0^{\ast }$ lies between ${\theta }_0^T$ and ${\hat{\theta }}_0^{\ast }$. Therefore, we have

$$-n^{-1}{\dot{Q}}_n\left({\hat{\theta }}_0^{\ast }\right)+{\dot{p}}_{\lambda }\left({\hat{\theta }}_0^{\ast }\right){\hat{\theta }}_0^{\ast }=n^{-1}\left\{{\ddot{Q}}_n\right({\hat{\theta }}_0^{\ast })+{\ddot{p}}_{\lambda }({\tilde{\theta }}_0^{\ast }\left)\right\}({\hat{\theta }}^{\ast }-{\hat{\theta }}_0^{\ast }).$$ (A10)

Note that ${\dot{p}}_{{\lambda }_1}\left({\mathrm{\text{β}}}^{\ast }\right)=\sum _{k=1}^{p_1}{p}_{{\lambda }_{1k}}^{\mathrm{\prime }}\left(|{\widehat{\mathrm{\text{β}}}}_k^{\ast }|\right)\mathrm{sgn}\left({\widehat{\mathrm{\text{β}}}}^{\ast }\right)$, ${\dot{p}}_{{\lambda }_2}\left({\gamma }^{\ast }\right)=\sum _{l=1}^{q_1}{p}_{{\lambda }_{2l}}^{\mathrm{\prime }}(\parallel {\hat{\gamma }}_l^{\ast }\parallel )\frac{H{\hat{\gamma }}_l^{\ast }}{\parallel {\hat{\gamma }}_l^{\ast }\parallel }$, and ${\dot{p}}_{\lambda }\left(\theta \right)={\dot{p}}_{{\lambda }_1}\left(\mathrm{\text{β}}\right)+{\dot{p}}_{{\lambda }_2}\left(\gamma \right)$. Apply Taylor expression to $p_{{\lambda }_{1k}}^{\mathrm{\prime }}\left(|{\widehat{\mathrm{\text{β}}}}_k^{\ast }|\right)$, we have

$$p_{{\lambda }_{1k}}^{\mathrm{\prime }}\left(|{\widehat{\mathrm{\text{β}}}}_k^{\ast }|\right)=p_{{\lambda }_{1k}}^{\mathrm{\prime }}\left(|{\widehat{\mathrm{\text{β}}}}_{0k}|\right)+\left\{p_{{\lambda }_{1k}}^{\mathrm{\prime }\mathrm{\prime }}\right(|{\mathrm{\text{β}}}_{0k}|)+o_p(1\left)\right\}({\widehat{\mathrm{\text{β}}}}_k^{\ast }-{\mathrm{\text{β}}}_{0k})$$

C9 implies that $p_{{\lambda }_{1k}}^{\mathrm{\prime }\mathrm{\prime }}\left(|{\mathrm{\text{β}}}_{0k}^{\ast }|\right)=o_p\left(1\right)$, and note that $p_{{\lambda }_{1k}}^{\mathrm{\prime }}\left(|{\mathrm{\text{β}}}_{0k}^{\ast }|\right)=0$ as ${\lambda }_{max}\to =o_p({\widehat{\mathrm{\text{β}}}}^{\ast }-{\mathrm{\text{β}}}_0^{\ast })$. Following [29], we know that $\parallel {\hat{\gamma }}_l^{\ast }{\parallel }_H\ge a{\lambda }_{2l}$ for n large enough. Thus, $p_{{\lambda }_{2l}}^{\mathrm{\prime }}(\parallel {\hat{\gamma }}_l^{\ast }{\parallel }_H)=0$ and $p_{{\lambda }_{2l}}^{\mathrm{\prime }\mathrm{\prime }}(\parallel {\hat{\gamma }}_l^{\ast }{\parallel }_H)=0$, which imply that ${\dot{p}}_{{\lambda }_2}\left({\gamma }^{\ast }\right)=0=o_p({\hat{\gamma }}^{\ast }-{\gamma }_0^{\ast })$ and ${\dot{p}}_{\lambda }\left({\theta }^{\ast }\right)=0$. So we have

$$\begin{array}{rl}\sqrt{n}\mathbf{H}({\hat{\theta }}^{\ast }-{\theta }_0^{\ast })& =\sqrt{n}{\mathbf{S}}_n+o_p\left(1\right)\\ \sqrt{n}({\widehat{\mathrm{\text{β}}}}^{\ast }-{\mathrm{\text{β}}}_0^{\ast })& =\left\{{\mathbf{H}}_{11}\right({\hat{\theta }}_0^{\ast })-{\mathbf{H}}_{12}({\hat{\theta }}_0^{\ast }\left){\mathbf{H}}_{22}^{-1}\left({\hat{\theta }}_0^{\ast }\right){\mathbf{H}}_{21}\right({\hat{\theta }}_0^{\ast }){\}}^{-1}\\ & \times (I,{\mathbf{H}}_{21}({\hat{\theta }}_0^{\ast }\left){\mathbf{H}}_{22}^{-1}\left({\hat{\theta }}_0^{\ast }\right)\right)\sqrt{n}{\mathbf{S}}_n\left({\hat{\theta }}_0^{\ast }\right)+o_p\left(1\right)\end{array}$$

From Lemma 1, we can get $\parallel n^{-1}{\dot{Q}}_n\left({\theta }_0^{\ast }\right)-2{\mathbf{S}}_n\left({\theta }_0^{\ast }\right)\parallel =O_p\left(n^{-1}\right)$, $\parallel n^{-1}{\ddot{Q}}_n\left({\theta }_0^{\ast }\right)-2{\mathbf{H}}_n\left({\theta }_0^{\ast }\right)\parallel =o_p\left(1\right)$. According to C9 and Lemma 2, we have

$$\frac{\mathrm{\partial }{\hat{\overline{g}}}_n\left(\theta \right)}{\mathrm{\partial }{\theta }^{\ast }}⟹P-J_{0{\theta }^{\ast }},\frac{\mathrm{\partial }{\hat{\overline{g}}}_n\left(\theta \right)}{\mathrm{\partial }{\mathrm{\text{β}}}^{\ast }}⟹P-J_{0{\mathrm{\text{β}}}^{\ast }},\frac{\mathrm{\partial }{\hat{\overline{g}}}_n\left(\theta \right)}{\mathrm{\partial }{\gamma }^{\ast }}⟹P-J_{0{\gamma }^{\ast }}$$

and

$$\begin{array}{rl}& \sqrt{n}{\hat{\overline{g}}}_n\left({\theta }_0^{\ast }\right)=O_p\left(n^{-1/2}\right),\sqrt{n}{\mathbf{S}}_n\left({\hat{\theta }}_0^{\ast }\right)=J_{0{\theta }^{\ast }}{\mathrm{\Omega }}_0^{-1}\sqrt{n}{\hat{\overline{g}}}_n\left({\theta }_0^{\ast }\right)+o_p\left(1\right)=O_p\left(n^{-1/2}\right)\\ & {\mathbf{H}}_{11}\left({\hat{\theta }}_0^{\ast }\right)\to J_{0{\mathrm{\text{β}}}^{\ast }}^T{\mathrm{\Omega }}_0^{-1}{J}_{0{\mathrm{\text{β}}}^{\ast }}={\mathbf{H}}_{11}^0\left({\hat{\theta }}_0^{\ast }\right),{\mathbf{H}}_{22}\left({\hat{\theta }}_0^{\ast }\right)\to J_{0{\gamma }^{\ast }}^T{\mathrm{\Omega }}_0^{-1}{J}_{0{\gamma }^{\ast }}={\mathbf{H}}_{22}^0\left({\hat{\theta }}_0^{\ast }\right)\\ & {\mathbf{H}}_{12}\left({\hat{\theta }}_0^{\ast }\right)\to J_{0{\mathrm{\text{β}}}^{\ast }}^T{\mathrm{\Omega }}_0^{-1}{J}_{0{\gamma }^{\ast }}={\mathbf{H}}_{12}^0\left({\hat{\theta }}_0^{\ast }\right),{\mathbf{H}}_{21}\left({\hat{\theta }}_0^{\ast }\right)\to J_{0{\gamma }^{\ast }}^T{\mathrm{\Omega }}_0^{-1}{J}_{0{\mathrm{\text{β}}}^{\ast }}={\mathbf{H}}_{21}^0\left({\hat{\theta }}_0^{\ast }\right)\end{array}$$

where $J_{0{\theta }^{\ast }}=(J_{0{\mathrm{\text{β}}}^{\ast }},J_{0{\gamma }^{\ast }})$.

Denote $\mathbf{A}=\left\{{\mathbf{H}}_{11}\right({\hat{\theta }}_0^{\ast })-{\mathbf{H}}_{12}({\hat{\theta }}_0^{\ast }\left){\mathbf{H}}_{22}^{-1}\left({\hat{\theta }}_0^{\ast }\right){\mathbf{H}}_{21}\right({\hat{\theta }}_0^{\ast }){\}}^{-1}(I,{\mathbf{H}}_{21}({\hat{\theta }}_0^{\ast }\left){\mathbf{H}}_{22}^{-1}\left({\hat{\theta }}_0^{\ast }\right)\right)$, Hence we can get

$$\mathbf{A}⟹p\left\{{\mathbf{H}}_{11}^0\right({\hat{\theta }}_0^{\ast })-{\mathbf{H}}_{12}^0({\hat{\theta }}_0^{\ast }\left){\mathbf{H}}_{22}^{-1}\left({\hat{\theta }}_0^{\ast }\right){\mathbf{H}}_{21}^0\right({\hat{\theta }}_0^{\ast }){\}}^{-1}(I,{\mathbf{H}}_{21}^0({\hat{\theta }}_0^{\ast }\left){\mathbf{H}}_{22}^{-1}\left({\hat{\theta }}_0^{\ast }\right)\right)={\mathbf{A}}_{\mathbf{0}}$$ (A11)

$$\sqrt{n}{\mathbf{S}}_n\left({\hat{\theta }}_0^{\ast }\right)⟹LN(0,(J_{{\theta }_0^{\ast }}{\mathrm{\Omega }}_0^{-1}{J}_{{\theta }_0^{\ast }}^T{)}^{-1})$$

According to the Slutsky Theorem, we can see that $\stackrel{ˆ}{{\mathrm{\text{β}}}^{\ast }}$ is consistent and asymptotic normality,

$$\sqrt{n}({\widehat{\mathrm{\text{β}}}}^{\ast }-{\mathrm{\text{β}}}_0^{\ast })⟹LN(0,{\mathbf{A}}_0(J_{{\theta }_0^{\ast }}{\mathrm{\Omega }}_0^{-1}{J}_{{\theta }_0^{\ast }}^T{)}^{-1}{\mathbf{A}}_0^T)$$ (A12)

This completes the proof of Theorem 3.

Funding Statement

This work is supported by grants from the Social Science Foundation of China [grant number 15CTJ008 to M. Z.], the Natural Science Foundation of Anhui Universities [grant number KJ2017A433 to K. Z.], the Social Science Foundation of the Ministry of Education of China [grant number 19YJCZH250 to K. Z.], the National Science Foundation of China [grant numbers 12071305, 11871390 and 11871411 to Y. Z.], the Excellent Young Talents Fund Program of Higher Education Institutions of Anhui Province [grant number gxyqZD2019031 to Y. Z.], the National Science Foundation of China [grant number 71803001 to Y. Z.]. This paper is partially supported by the National Natural Science Foundation of China [grant number 11901401].

Disclosure statement

No potential conflict of interest was reported by the author(s).