Sparse regression for low-dimensional time-dynamic varying coefficient models with application to air quality data

Abstract

Time dynamic varying coefficient models play an important role in applications of biology, medicine, environment, finance, etc. Traditional methods, such as kernel smoothing and spline smoothing, are popular. But explicit expressions are unavailable using these methods, and the convergence rate of coefficient function estimators is slow. To address these problems, we expand the varying component with appropriate basis functions. And then we solve a sparse regression problem via a sequential thresholded least-squares estimator. The “parameterization” leads to explicit expressions and fast computation speed. Convergence of the sequential thresholded least squares algorithm is guaranteed. The asymptotic distribution of the coefficient function estimator is derived under certain assumptions. Our simulation shows the proposed method has higher precision and computing speed. Finally, our proposed method is applied to the study of PM ${}_{2.5}$ concentration in Beijing. We analyze the relationship between PM ${}_{2.5}$ and other impact factors.

1. Introduction

There has been much research on varying coefficient models since they were proposed by [9]. Varying coefficient model is an extension of the classical linear model. Their objective is to examine how regression coefficients change over time or other covariates. Park et al. [16] gave an overview of methodological and theoretical developments of earlier studies. To construct a varying coefficient model, one strategy is to let all regression coefficients depend on a mutual covariate. Zhang and Fan [25] proposed a two-step local polynomial least squares estimation procedure. Chiang et al. [5] proposed a componentwise smoothing spline method. The other is to let different regression coefficients depend on different covariates. Marginal integration technique [13] and smooth backfitting technique [14] are used frequently.

The problem, how a covariate $X_j$ affects response Y over time in the dynamical longitudinal analysis, is of interest. Varying coefficient models are useful and play an important role to solve the problem. The simplest functional regression model is

$$Y(t)={\beta }_1(t)X_1(t)+\cdots +{\beta }_p(t)X_p(t)+\epsilon (t),$$ (1)

where ${\beta }_{j}s(j=1,\dots ,p)$ are unknown coefficient functions and ε is a mean zero stochastic process. Here the coefficient functions ${\beta }_j$ depend only on time t. Lee et al. [11] studied a varying coefficient model with the response variable dependent on the covariates as follows

$$Y={\beta }_1(X_1)Z_1+\cdots +{\beta }_p(X_p)Z_p+\sigma (\mathit{X},\mathit{Z})\epsilon ,$$ (2)

where $\mathit{X}=(X_1,\dots ,X_p)$, $\mathit{Z}=(Z_1,\dots ,Z_p)$, $E(\epsilon |\mathit{X},\mathit{Z})=0$, and $\mathrm{Var}(\epsilon |\mathit{X},\mathit{Z})=1.$ Zhang et al. [26] proposed the following model

$$Y(t)={\beta }_1(t,Z_1(t))+\cdots +{\beta }_p(t,Z_d(t))+\epsilon (t),$$ (3)

and a new backfitting algorithm was developed to estimate the unknown regression functions with the convergence rate $n^{1/3}$. Lee et al. [10] studied a new varying coefficient model

$$Y(t)={\beta }_1(t,Z_1(t))X_1(t)+\cdots +{\beta }_p(t,Z_p(t))X_p(t)+\epsilon (t),$$ (4)

where ε is a stochastic process such that $E(\epsilon |X_1,Z_1,\dots ,X_p,Z_p)=0$. Different from model (1), the coefficient functions change with time and a covariate. Kernel smoothing technique was developed to estimate this model. And the convergence rate of the coefficient function estimators was also $n^{1/3}$. Although similar to model (3), model (4) takes nonlinear interaction effects into account.

The estimation procedure of kernel smoothing depends on the selection of bandwidths. Spline smoothing methods depend on knots and smoothing parameters. And the backfitting algorithms are computationally cumbersome. Furthermore, varying coefficient models are semiparametric and have bad interpretability. We are likely to estimate such models by deep learning methods [1] due to their ability to mine nonlinear structures hidden in the data. However, we will not develop this point here for lack of interpretability.

Brunton et al. [3] combined sparsity-promoting and machine learning techniques to discover governing equations from noisy measurement data. Their proposed data-driven modeling framework, called sparse identification of nonlinear dynamics (short for SINDy), could identify nonlinear low-order models. Motivated by this, we propose our data-driven estimator for time-dynamic varying coefficient models. We expand functional regression coefficients globally by constructing a library of candidate basis functions. This is a basis expansion method. The basis functions are chosen more flexibly based on domain knowledge. Then, the time-dynamic varying coefficient model becomes a sparse linear regression model. One advantage is that an explicit functional expression may be found with strong interpretability. Even though the constructed functions are not in the correct functional space in some cases, the proposed procedure can still obtain an approximate estimator.

We can use regularization techniques, such as LASSO [21], to fit a sparse linear regression model. There is ample literature on high dimensional linear regression, see [17] for an overview. Besides, [3] proposed a sequentially least squares algorithm to compute the sparse regression problem. Zhang and Schaeffer [22] provided some theoretical results on the behavior and convergence of SINDy. In further study, [18] found that LASSO performed poorly with highly correlated data. And they approximated the problem to a ridge regression problem with hard thresholding, called sequential threshold ridge regression (STRidge). We use sequential thresholded least squares estimator to estimate the varying functional regression coefficients in this research.

This paper is organized as follows. Section 2 introduces time dynamic varying coefficient models and our proposed estimation procedure. $\sqrt{n}$ consistency and some other theoretical properties are shown in Section 3. Simulation studies are conducted to verify the efficiency of this approach in Section 4. We analyze the air quality data set using our method in Section 5. Finally, we draw some conclusions and directions for future research in Section 6.

2. Model

In this section, we take the model (4) for an example to introduce our estimation procedure, then extend it to model (1) and (3). The observed response and covariates in the longitudinal setting satisfy the model

$$Y(T)={\beta }_1(T,Z_1(T))X_1(T)+\cdots +{\beta }_p(T,Z_p(T))X_p(T)+\epsilon (T),$$ (5)

where T denotes the random variable that represents the independent time points allowed to be different for different subjects, the processes $\mathit{X}=(X_1,\dots ,X_p{)}^{\mathrm{\top }}$, $\mathit{Z}=(Z_1,\dots ,Z_p{)}^{\mathrm{\top }}$, and Y are observed, the coefficients ${\beta }_{j}s(j=1,\dots ,p)$ are bivariate functions and vary with time T. Some terms, such as polynomials, often appear in ${\beta }_j$ in practice. If we can construct possible basis functions, the bivariate function ${\beta }_j$ will expand in a linear combination of the basis functions.

2.1. Sparse identification of nonlinear dynamics

Sparse identification of nonlinear dynamics (SINDy) is developed for discovering ordinary differential equations and partial differential equations in engineering. Let's have a brief review of sparse identification of nonlinear dynamics. Consider the dynamical system of some time series data $u(t)$ as follows

$$\dot{u}=f(u).$$

SINDy combines sparsity regression and nonlinear function libraries to identify a nonlinear and dynamical system from time-series data. The main idea is to convert the nonlinear model identification problem to a linear system

$$A\omega =b,$$ (6)

where the column full rank matrix $A\in {\mathbb{R}}^{r\times q}(r\ge q)$ is a data-driven dictionary whose columns are candidate functions of the given data u, the unknown vector $\omega \in {\mathbb{R}}^q$ represents the coefficients of the selected terms in the governing equation f, and the vector $b\in {\mathbb{R}}^r$ is an approximation to the first-order time derivative $\dot{u}$. A sparse vector ω which approximately solves (6) is generated by the following iterative scheme:

$$\begin{array}{rl}{S}^k& =\{1\le j\le q:|{\omega }_j^k|\ge \lambda \},\end{array}$$ (7)

$$\begin{array}{rl}{\omega }^{k+1}& =\arg \underset{\omega \in {\mathbb{R}}^q:supp(x)\subset S^k}{min}\Vert A\omega -b{\Vert }_2^2;\end{array}$$ (8)

where $\lambda >0$ is a thresholding parameter, and $supp(\omega )$ is the support set of ω. The calculating process is illustrated in Algorithm 1 [3].

2.2. Data-Driven estimator for dynamic varying coefficient models

We rewrite the model (5) as follows

$$y^{ij}(t^{ij})={\beta }_1(t^{ij},z_1^{ij}(t^{ij}))x_1^{ij}(t^{ij})+\cdots +{\beta }_p(t^{ij},z_p^{ij}(t^{ij}))x_p^{ij}(t^{ij})+\epsilon (t^{ij}),$$ (9)

where $i=1,\dots ,M$, $j=1,\dots ,M_i$. Let y be the matrix

$$\left(\begin{array}{ccc}{y}^{11}& \cdots & y^{1M_1}\\ ⋮& \ddots & ⋮\\ y^{M1}& \cdots & y^{M{M}_M}\end{array}\right).$$

We vectorize the response y to be $(y^{11},\dots ,y^{1M_1},\dots ,y^{M1},\dots ,y^{M{M}_M}{)}^{\mathrm{\top }}$, let ${\mathit{M}}_l=(Z_l,T),l=1,\dots ,p$, a library of candidate nonlinear functions $\mathrm{\Theta }({\mathit{M}}_l)$ can be constructed as follows

$$\mathbf{\Theta }\mathbf{(}{\mathit{M}}_{\mathit{l}}\mathbf{)}=\left[\begin{array}{ccc}{\mathit{f}}_1({\mathit{M}}_{\mathit{l}}),& \dots & ,{\mathit{f}}_m({\mathit{M}}_{\mathit{l}})\end{array}\right],$$ (10)

where ${\mathit{f}}_1,\dots ,{\mathit{f}}_m$ are univariate functions or bivariate functions from ${\mathbb{R}}^2$ to $\mathbb{R}$. These functions can be chosen with prior knowledge. Here, prior information can be viewed as domain knowledge. We can cooperate with domain experts or search the literature to know prior knowledge. For example, the right-hand side of ordinary differential equations often appears polynomials, then we can choose polynomials as candidate functions. If we want to discover the dynamic transmission model of COVID-19 [12], SIR (susceptible, infective, and recovered), SEIR (susceptible, exposed, infective, and recovered), SEIJR (susceptible, exposed, infective, diagnosed, and recovered), and SIRD (susceptible, infective, recovered and dead) models, which are designed for infectious diseases, can be regarded as our prior knowledge. We select polynomials in this case. Even though the constructing functions in some cases are not in the correct functional space, the proposed procedure can still gain an approximate estimator.

Remark 2.1

Any continuous function can be well approximated by either polynomial base or trigonometric base. It is also available to include both in the library. We aim to get a global functional expression of the model of interest. Basis functions, such as B-spline functions, cannot be used in our research. Because B-spline methods recover data interval by interval, leading to a local solution.

We take Example 4.1 in Section 4 [10] for an example, three varying coefficients are specified as follows

$$\begin{array}{rl}{\beta }_1(t,z_1)& =t(-{(z_1-0.5)}^2+2z_1-0.5t),\\ {\beta }_2(t,z_2)& =2(z_2-0.5)(t-z_2+3),\\ {\beta }_3(t,z_3)& =3.5(-z_3+(t-0.5{)}^2+0.4).\end{array}$$ (11)

An equivalent formulation of (11) is

$$\begin{array}{rl}{\beta }_1(t,z_1)& =-z_1^{2}t+3z_{1}t-0.5t^2-0.25t,\\ {\beta }_2(t,z_2)& =2(-z_2^2+3.5z_2+z_{2}t-0.5t-1.5),\\ {\beta }_3(t,z_3)& =3.5(-z_3+t^2-t+0.65),\end{array}$$

it is interesting to find each ${\beta }_j$ is polynomial. Then

$$\begin{array}{rl}y(t)& =(-z_1^{2}t+3z_{1}t-0.5t^2-0.25t)x_1(t)+2(-z_2^2+3.5z_2+z_{2}t-0.5t-1.5)x_2(t)\\ & +(3.5(-z_3+t^2-t+0.65))x_3(t)+\epsilon (t).\end{array}$$

If each varying coefficient is expanded in three-order polynomial functions, we get

$$\mathbf{\Theta }\mathbf{(}{\mathit{M}}_{\mathit{i}}\mathbf{)}=\left[\begin{array}{cccccccccc}\mathbf{1}& T& Z_i& T^2& T{Z}_1& Z_i^2& T^3& T^2{Z}_i& T{Z}_i^2& Z_i^3\end{array}\right].$$

Thus the original varying coefficient model now becomes a 30-dimensional linear sparse regression model. The true coefficients are illustrated in Table 1.

Table 1.

Basis functions and corresponding coefficients in Example 4.1.

$\mathrm{\Theta }(M)$	${\beta }_1$	${\beta }_2$	${\beta }_3$
1	0	$-$3	2.275
t	$-$0.25	$-$1	$-$3.5
z	0	7	$-$3.5
tt	$-$0.5	0	3.5
tz	3	2	0
zz	0	$-$2	0
ttt	0	0	0
ttz	0	0	0
tzz	$-$1	0	0
zzz	0	0	0

Remark 2.2

We assume the constructed library of candidate functions contains true function ${\beta }_j(T,Z_j)$. It will be feasible as long as the set of basis functions can have more terms than ${\beta }_j(T,Z_j)$. We tend to pick a moderately large set. The computation cost will be expensive if the set is too large. Basis functions should be three-order or greater than three-order in (11). The higher the order is, the harder it is for computation.

We assume true functions are a subset of the functions in (10), then model (5) is

$$y^{ij}(t^{ij})=\mathrm{\Theta }({\mathit{M}}_1^{ij})\circ x_1^{ij}(t^{ij}){\psi }_1^{\ast }+\cdots +\mathrm{\Theta }({\mathit{M}}_p^{ij})\circ x_p^{ij}(t^{ij}){\psi }_p^{\ast }+\epsilon (t^{ij}),$$

where ${\psi }_1^{\ast },\dots ,{\psi }_p^{\ast }\in {\mathbb{R}}^m$, and ° denotes a new product

$$\mathrm{\Theta }({\mathit{M}}_l^{ij})\circ x_l^{ij}(t^{ij})=\left[\begin{array}{ccc}{\mathit{f}}_1({\mathit{M}}_l^{ij})x_l^{ij}(t^{ij}),& \cdots & ,{\mathit{f}}_m({\mathit{M}}_l^{ij})x_l^{ij}(t^{ij})\end{array}\right].$$ (12)

Let $$\begin{gathered} {\theta }_l=(\mathrm{\Theta }({\mathit{M}}_l^{11})\circ x_l^{11}(t^{11}),\dots ,\mathrm{\Theta }({\mathit{M}}_l^{1M_1})\circ x_l^{1M_1}(t^{1M_1}),\dots ,\mathrm{\Theta }({\mathit{M}}_l^{M1})\circ x_l^{M1}(t^{M1}),\dots , \\ \mathrm{\Theta }({\mathit{M}}_l^{M{M}_M})\circ x_l^{M{M}_M}(t^{M{M}_M}){)}^{\mathrm{\top }},l=1,\dots ,p \end{gathered}$$, $\mathrm{\Theta }=({\theta }_1,\dots ,{\theta }_p)$, ${\mathrm{\Psi }}^{\ast }=({{\psi }_1^{\ast }}^{\mathrm{\top }},\dots ,{{\psi }_p^{\ast }}^{\mathrm{\top }}{)}^{\mathrm{\top }}$, and $\epsilon =({\epsilon }_{11},\dots ,{\epsilon }_{1M_1},\dots ,{\epsilon }_{M1},\dots ,{\epsilon }_{M{M}_M}{)}^{\mathrm{\top }}$, we rewrite (9) as

$$y=\mathrm{\Theta }{\mathrm{\Psi }}^{\ast }+\epsilon ,$$ (13)

where y is a vector of $(M_1+\cdots +M_M)$ observed outcomes, Θ is an $(M_1+\cdots +M_M)\times mp$ matrix of predictors, ${\mathrm{\Psi }}^{\ast }=({\psi }_1^{\ast },\dots ,{\psi }_{mp}^{\ast }{)}^{\mathrm{\top }}$ is the unknown parameter of interest, and ε is a vector of $i.i.d.$ additive noise with mean 0 and variance ${\sigma }^2$.

As estimates for ${\mathrm{\Psi }}^{\ast }$, we use sequential thresholded least squares estimator

$$\begin{array}{rl}& \hat{\mathrm{\Psi }}=\underset{\mathrm{\Psi }\in {\mathbb{R}}^{mp\times 1}}{\arg min}{\Vert y-\mathrm{\Theta }\mathrm{\Psi }\Vert }_2^2,\\ & \mathrm{s}.\mathrm{t}.|{\mathrm{\Psi }}_l|>\lambda ,l=1,\dots ,mp.\end{array}$$ (14)

where λ is a tuning parameter, and the calculating process is illustrated in Algorithm 2. Refer to [3] for more details.

For the tuning parameter λ, we can use k-fold cross-validation or AIC criteria [2] to determine it. Sequential thresholded least squares estimator in (14) is equivalent to the ${\ell }_0$ penalized least squares as follows

$$\underset{\mathrm{\Psi }\in {\mathbb{R}}^{mp}}{\arg min}\Vert \mathrm{\Theta }\mathrm{\Psi }-y{\Vert }_2^2+{\lambda }^2\Vert \mathrm{\Psi }{\Vert }_0,$$

where ${\lambda }^2$ is much smaller when λ is a small value, which has little effect on the results. We fix λ to a small value, such as 0.05, in our simulation. The results do not change much when λ is a small value.

Remark 2.3

We use sequential thresholded least squares estimator for the reason it is equivalent to ${\ell }_0$ penalized regression. ${\ell }_0$ penalized least squares estimator penalizes the number of variables in a regression model. It has an advantage over other penalties for sparse regressions in nature. More sparse solutions can be obtained using ${\ell }_0$ penalty. There is an enormous amount of literature studying sparse regression in linear models. STRidge in [18] is one of them developed for sparse linear regression models. The calculation procedure is similar between sequential thresholded least squares and STRidge. But STRidge needs to specify one more hyperparameter.

2.3. Estimator of non-structured varying coefficient models

The estimation procedure of (4) is not difficult when we expand (4) in an appropriate basis. But it is hard to decide which covariate interacts with both time and another covariate arises. This problem won't occur in (1) and (3). To deal with it, we consider the following model

$$Y(T)=m(T,Z(T),X(T))+\epsilon (T),$$ (15)

where $m(\cdot )$ is an unknown function and ε is a stochastic process such that $E(\epsilon |X_1,Z_1,\dots ,X_p,Z_p)=0$. We expand the model (15) with appropriate basis functions. An explicit expression can still be obtained though there is severe sparsity of the basis functions. Furthermore, it would be helpful for us to decide which covariate interacts with time and another covariate if we just use three-order polynomials as basis functions. This is a pre-selection process. Then we update the estimation in model (4) using Algorithm 2.

3. Theoretical properties

In the work of [22], they showed the SINDy algorithm could find local minimizers of (16). Thus, it has theoretical guarantees summarized below in the setting of our dynamic varying coefficient models.

Lemma 3.1

Let $n=M_1+\cdots +M_M,$ $\mathrm{\Theta }\in {\mathbb{R}}^{n\times mp}$ with $\Vert \mathrm{\Theta }{\Vert }_2=1,\mathrm{\Psi }\in {\mathbb{R}}^{mp},$ and $\lambda \ge 0$. Assume that $n\ge mp,$ ${\hat{\mathrm{\Psi }}}^{(k)}$ is the sequence generated by Algorithm 2. Define the objective function F by

$$F(\mathrm{\Psi }):=\Vert \mathrm{\Theta }\mathrm{\Psi }-y{\Vert }_2^2+{\lambda }^2\Vert \mathrm{\Psi }{\Vert }_0,\mathrm{\Psi }\in {\mathbb{R}}^{mp}.$$ (16)

We have

1. ${\hat{\mathrm{\Psi }}}^{(k)}$ converges to a fixed point of the iterative scheme defined by (16) at most mp steps;

2. a fixed point of the scheme is a local minimizer of F;

3. a global minimizer of F is a fixed point of the scheme;

4. the inequality ${\hat{\mathrm{\Psi }}}^{(k+1)}<{\hat{\mathrm{\Psi }}}^{(k)}$ holds unless the iterations are stationary.

Proof.

For the proof, we refer the reader to [22].

In (13), let $S_0=\{j:{\mathrm{\Psi }}_j^{\ast }\ne 0,j=1,\dots ,mp\}$ be the true model with size $d_0$, $|S_0|=d_0$, where $|S_0|$ denotes the cardinality of $S_0$. For $S\subset \{1,\dots ,mp\}$, let ${\mathrm{\Psi }}_S=({\mathrm{\Psi }}_i{)}_{j\in S}\in {\mathbb{R}}^{|S|}$ be the $|S|$ dimensional sub-vector of Ψ indexed by S, and let ${\mathrm{\Theta }}_S$ be the $n\times |S|$ matrix obtained from Θ by extracting columns corresponding to S. If the estimate $\hat{S}=\{j:{\hat{\mathrm{\Psi }}}_j\ne 0,j=1,\dots ,mp\}$ of $S_0$ satisfies $P(\hat{S}\ne S_0)\to 0$ as $n,mp\to \mathrm{\infty }$, selection consistency is obtained. Then we can say SINDy consistently selects the correct model $S_0$.

By Lemma 3.1, the solution of SINDy algorithm equals to ${\ell }_0$ penalized least squares solution. The ${\ell }_0$ penalty directly penalizes the number of nonzero components in (16), which is appealing in sparse regression. Though ${\ell }_0$ penalized least squares has a good property, it is discontinuous and the computation complexity is NP-hard. To overcome such difficulty, many researchers [4,7,8] suggested to relax the ${\ell }_0$ regularization and considered the ${\ell }_1$ regularization instead, for example, the LASSO [21]. Dicker et al. [6] proposed the seamless- ${\ell }_0$ (SELO) penalty, a smooth function on $[0,1)$, very close to the ${\ell }_0$ penalty. They also showed the model selection consistency and asymptotical normality. Applying to our time dynamic varying coefficient models, Lemma 3.2 holds.

Lemma 3.2

Assume that true basis functions of each coefficient are a subset of the functions in (10) and the size of library m is fixed, then under

condition 1: $n\to \mathrm{\infty }$ and $mp{\sigma }^2/n\to 0$;

condition 2: $\rho \sqrt{n/(mp{\sigma }^2)}\to \mathrm{\infty },$ where $\rho =\underset{l\in S}{min}\Vert {\mathrm{\Psi }}_l\Vert$;

condition 3: There exist positive constants $r_0,R_0>0$ such that $r_0\le {\lambda }_{min}(n^{-1}{\mathrm{\Theta }}^{\mathrm{\top }}\mathrm{\Theta })<{\lambda }_{max}(n^{-1}{\mathrm{\Theta }}^{\mathrm{\top }}\mathrm{\Theta })<R_0,$ where ${\lambda }_{min}(n^{-1}{\mathrm{\Theta }}^{\mathrm{\top }}\mathrm{\Theta })$ and ${\lambda }_{max}(n^{-1}{\mathrm{\Theta }}^{\mathrm{\top }}\mathrm{\Theta })$ are the smallest and largest eigenvalues of $n^{-1}{\mathrm{\Theta }}^{\mathrm{\top }}\mathrm{\Theta },$ respectively;

condition 4: $\lambda =O(1),$ $\lambda \sqrt{n/(mp{\sigma }^2)}\to \mathrm{\infty };$

we have: $\underset{n\to \mathrm{\infty }}{lim}P(\{l;{\hat{\mathrm{\Psi }}}_l\ne 0\}=S_0)=1$.

Proof.

For the proof we refer the reader to [6].

According to Lemma 3.2, the SINDy algorithm in (16) approximately selects the true model with probability 1. Shen et al. [19] gave an error bound for a global minimizer of (16) using ${\ell }_0$ penalty. Lemma 3.3 can be obtained for our time dynamic varying coefficient models.

Lemma 3.3

Let ${\hat{\mathrm{\Psi }}}^{{\ell }_0}=({\hat{\mathrm{\Psi }}}_{\hat{S}}^{{\ell }_0},\mathbf{0}{)}^{\mathrm{\top }}$ and ${\hat{\mathrm{\Psi }}}^{ol}=({\hat{\mathrm{\Psi }}}_{S_0}^{ol},{\mathbf{0}}_{S_0^c}{)}^{\mathrm{\top }}$ under (16) with $\alpha >1,$ then

$$\begin{array}{rl}P({\hat{\mathrm{\Psi }}}^{{\ell }_0}\ne {\hat{\mathrm{\Psi }}}^{ol})& \le 4\exp (-(\frac{n{C}_{min}^{\ast }}{18{\sigma }^2}-(\alpha +1)\log (mp+1)-\frac{{\lambda }^2}{2{\sigma }^2}))\\ & +4\exp (-(\frac{(\alpha -1){\lambda }^2}{3\alpha {\sigma }^2}-(\frac{1}{\alpha }+1)\log (mp+1)-\frac{2}{3})),\end{array}$$

where

$$C_{min}^{\ast }\equiv \underset{\{{\mathrm{\Psi }}_S:S\ne S_0,|S|\le \alpha d_0\}}{min}\frac{1}{nmax(|S_0\setminus S|,1)}\Vert {\mathrm{\Theta }}_{S_0}{\mathrm{\Psi }}_{S_0}^{\ast }-{\mathrm{\Theta }}_S{\mathrm{\Psi }}_S\Vert .$$

Moreover, if $\underset{{\mathrm{\Psi }}^{\ast }\in B_0(u,l)}{sup}\frac{1}{n}\Vert \mathrm{\Theta }{\mathrm{\Psi }}^{\ast }{\Vert }^2\le c_1\exp (c_2{d}_0)$ for some constant $c_j;j=1,2,$ then

$$P({\hat{\mathrm{\Psi }}}^{{\ell }_0}\ne {\hat{\mathrm{\Psi }}}^{ol})\le \frac{e+1}{e-1}\exp (\frac{n}{18{\sigma }^2}(C_{min}^{\ast }-36\frac{\log mp}{n}{\sigma }^2)),$$

where $B_0(u,l)=\{\mathrm{\Psi }:d_0=\sum _{j=1}^{mp}I({\mathrm{\Psi }}_j\ne 0)\le u,C_{min}^{\ast }(\mathrm{\Psi },\mathrm{\Theta })\ge l\}$ with $l=d_1{\sigma }^2\frac{\log mp}{n}$ and constant $d_1>0,$ which is called an ${\ell }_0$-band with upper and lower radii u and l $(u>l>0)$.

Proof.

For the proof we refer the reader to [19].

According to Lemma 3.3, model selection consistency of (16) is clear. Now, we analyze the asymptotic normality of (16) in Theorem 3.4.

Theorem 3.4

If the model selection consistency of (16) mentioned in Lemmas 3.2 and 3.3 is guaranteed, then

$$\sqrt{n}({\hat{\mathrm{\Psi }}}_{S_0}-{\mathrm{\Psi }}_{S_0}^{\ast })\to N(0,G).$$

in distribution, where $G=({\mathrm{\Theta }}_{S_0}^{\mathrm{\top }}{\mathrm{\Theta }}_{S_0}{)}^{-1}{\sigma }^2$.

Proof.

Let $L=\frac{1}{2n}\Vert Y-{\mathrm{\Theta }}_{S_0}{\mathrm{\Psi }}_{S_0}\Vert +{\lambda }^2{d}_0$, the derivative of L to ${\mathrm{\Psi }}_{S_0}$ is

$$\frac{\mathrm{\partial }L}{\mathrm{\partial }{\mathrm{\Psi }}_{S_0}}=({\mathrm{\Psi }}_{S_0}{)}^{\mathrm{\top }}(Y-{\mathrm{\Theta }}_{S_0}{\mathrm{\Psi }}_{S_0}),$$

and solve $\frac{\mathrm{\partial }L}{\mathrm{\partial }{\mathrm{\Psi }}_{S_0}}=0$, it is the ordinary least squares solution

$${\hat{\mathrm{\Psi }}}_{S_0}=({\mathrm{\Theta }}_{S_0}^{\mathrm{\top }}{\mathrm{\Theta }}_{S_0}{)}^{-1}{\mathrm{\Theta }}_{S_0}^{\mathrm{\top }}Y.$$

By basic calculus,

$$\begin{array}{rl}{\hat{\mathrm{\Psi }}}_{S_0}& =({\mathrm{\Theta }}_{S_0}^{\mathrm{\top }}{\mathrm{\Theta }}_{S_0}{)}^{-1}{\mathrm{\Theta }}_{S_0}^{\mathrm{\top }}Y\\ & =({\mathrm{\Theta }}_{S_0}^{\mathrm{\top }}{\mathrm{\Theta }}_{S_0}{)}^{-1}{\mathrm{\Theta }}_{S_0}^{\mathrm{\top }}({\mathrm{\Theta }}_{S_0}{\mathrm{\Psi }}_{S_0}^{\ast }+\epsilon )\\ & ={\mathrm{\Psi }}_{S_0}^{\ast }+({\mathrm{\Theta }}_{S_0}^{\mathrm{\top }}{\mathrm{\Theta }}_{S_0}{)}^{-1}{\mathrm{\Theta }}_{S_0}^{\mathrm{\top }}\epsilon .\end{array}$$

Thus $E({\hat{\mathrm{\Psi }}}_{S_0}-{\mathrm{\Psi }}_{S_0}^{\ast })=0$, $Var({\hat{\mathrm{\Psi }}}_{S_0}-{\mathrm{\Psi }}_{S_0}^{\ast })=({\mathrm{\Theta }}_{S_0}^{\mathrm{\top }}{\mathrm{\Theta }}_{S_0}{)}^{-1}{\sigma }^2$,

$$\sqrt{n}({\hat{\mathrm{\Psi }}}_{S_0}-{\mathrm{\Psi }}_S)\to N(0,G).$$

in distribution, where $G=({\mathrm{\Theta }}_{S_0}^{\mathrm{\top }}{\mathrm{\Theta }}_{S_0}{)}^{-1}{\sigma }^2.$

Denote ${\hat{S}}_{0(i)}=\{j:{\hat{\mathrm{\Psi }}}_j^{\ast }\ne 0,j=m(i-1)+1,\dots ,mi\}$, $i=1,\dots ,p$, size of ${\hat{S}}_{0(i)}$ is $d_{0i}$, $d_{00}=0$. Let $G_i^{\ast }$ be a submatrix of G,

$$G_i^{\ast }=G_{ij},i=\left(\sum _{k=0}^{i-1}{d}_{0k}\right)+1,\dots ,\sum _{k=1}^i{d}_{0k},j=\left(\sum _{k=0}^{i-1}{d}_{0k}\right)+1,\dots ,\sum _{k=1}^i{d}_{0k}.$$

Note ${\mathit{M}}_l=(Z_l,T),l=1,\dots ,p$, and $\mathrm{\Theta }({\mathit{M}}_l)$ is a map from ${\mathbb{R}}^2$ to ${\mathbb{R}}^m$. Let

$$\begin{array}{rl}{\mathit{U}}_l& ={\left({\mathit{M}}_l\right)}_h,h=k-m(l-1),k\in {\hat{S}}_{0(i)},\\ {\nu }_l^{\ast }& ={\left({\psi }_l^{\ast }\right)}_k,k\in {\hat{S}}_{0(i)}.\end{array}$$

By Theorem 3.4 and the transformation ${\beta }_l(T,Z_l)={\mathit{M}}_l^{\mathrm{\top }}{\psi }_l^{\ast }={\mathit{U}}_l^{\mathrm{\top }}{\nu }_l^{\ast }$, it is evident that ${\hat{\beta }}_l(T,Z_l)-{\beta }_l(T,Z_l)(l=1,\dots ,p)\to N(0,{\mathit{U}}_l^{\mathrm{\top }}{G}_l^{\ast }{\mathit{U}}_l)$ in distribution. So

$$\sqrt{n}\left(\begin{array}{c}{\hat{\beta }}_l(T,Z_l)-{\beta }_0(T,Z_l)\\ ⋮\\ {\hat{\beta }}_l(T,Z_l)-{\beta }_0(T,Z_l)\end{array}\right)\to N(\mathbf{0},diag({\mathit{U}}_l^{\mathrm{\top }}{G}_l^{\ast }{\mathit{U}}_l)),$$

where $\mathbf{0}$ is a $p\times 1$ vector, and $diag({\mathit{U}}_l^{\mathrm{\top }}{G}_l^{\ast }{\mathit{U}}_l)$ is a $p\times p$ diagonal matrix with elements ${\mathit{U}}_l^{\mathrm{\top }}{G}_l^{\ast }{\mathit{U}}_l$, $l=1,\dots ,p$.

Thus the convergence rate of our proposed method is $n^{\frac{1}{2}}$, while the convergence rate of backfitting algorithms is $n^{\frac{1}{3}}$. Our proposed method has a faster convergence rate. According to asymptotic distributions, backfitting algorithms are biased, while our proposed method is unbiased.

4. Simulation

Simulation studies are conducted in this section to check the performance of the proposed estimator. We use matlab 2019a on a server with Intel(R) core(TM) i5-10210U CPU @1.60 GHz. Our proposed method aims at time dynamic varying coefficient regression models. So we compare with methods developed for time dynamic varying coefficient regression models. We focus on two types of varying coefficient models with Case 1 in [10] and Case 2 in [26].

In Examples 4.1 and 4.2, we generate 50 pseudo samples of sizes n = 200 and 400 from the model (4) with d = 3. The numbers of time points $M_i$, $1\le i\le n$, are $i.i.d.$ having a discrete uniform distribution supported on the set $\{2,\dots ,9\}$. The time points $T_{il}$, $1\le i\le n$, $1\le l\le M_i$, are sampled from $U(0,1)$, the uniform distribution on the unit interval $[0,1]$, independently across i and l. The values of the linear covariates $X_j$ at $T_{il}$ are obtained according to

$$\begin{array}{l}{X}_1^{il}=T_{il}+V_{1i}+V_{2i}(T_{il}-0.5)+{\zeta }_1^{il},\\ X_2^{il}={(T_{il}-0.5)}^3+V_{3i}+V_{4i}{T}_{il}+{\zeta }_2^{il},\\ X_3^{il}\equiv 1,\end{array}$$

where V's and ζ's are random variables to be specified below. The values of the nonlinear covariates $Z_j$ at $T_{il}$ are

$$\begin{array}{l}{Z}_1^{il}=\exp (T_{il}-5)-0.2W_{1i}+0.1W_{2i}(T_{il}-0.5)+{\eta }_1^{il},\\ Z_2^{il}=0.5{(T_{il}-0.5)}^3+0.2W_{3i}+0.1W_{4i}(T_{il}-0.5)+{\eta }_2^{il},\\ Z_3^{il}={\eta }_3^{il},\end{array}$$

where W's and η's are random variables to be specified below. The random vectors $(V_{1i},V_{2i},V_{3i},V_{4i})$ are $i.i.d.$  $N(\mathbf{0},{\mathrm{\Sigma }}_X)$ with

$${\mathrm{\Sigma }}_X=\left(\begin{array}{cccc}0.07& -0.02& 0.04& -0.02\\ -0.02& 0.03& -0.02& 0.01\\ 0.04& -0.02& 0.07& -0.02\\ -0.02& 0.01& -0.02& 0.03\end{array}\right),$$

and $(W_{1i},W_{2i},W_{3i},W_{4i})$ are $i.i.d.$ $N(\mathbf{0},{\mathrm{\Sigma }}_Z)$ with

$${\mathrm{\Sigma }}_Z=\left(\begin{array}{cccc}0.04& -0.02& 0.01& -0.02\\ -0.02& 0.04& -0.02& 0.01\\ 0.01& -0.02& 0.04& -0.02\\ -0.02& 0.01& -0.02& 0.04\end{array}\right).$$

The random variables ${\zeta }_j^{il}$ are $i.i.d.$ $N(0,{0.3}^2)$, and ${\eta }_j^{il}$ are $i.i.d.$ $U(0,1)$ independent of ${\zeta }_j^{il}$. They are sampled also independent of V's and W's. The errors ${ϵ}^{il}$ in ${\epsilon }^{il}={ϵ}^{il}+{\delta }^{il}$ are generated by ${ϵ}^{il}=R_i(\cos (2\pi T^{il})+\sin (2\pi T^{il}))$ with $R_i$ being $i.i.d.$ $N(0,{0.5}^2)$, and ${\delta }^{il}$ are $i.i.d.$ $N(0,{0.1}^2)$ independent of $R_i$.

We consider two regression functions specified below.

Example 4.1

$$\begin{array}{rl}{\beta }_1(t,z_1)& =t(-{(z_1-0.5)}^2+2z_1-0.5t),\\ {\beta }_2(t,z_2)& =2(z_2-0.5)(t-z_2+3),\\ {\beta }_3(t,z_3)& =3.5(-z_3+(t-0.5{)}^2+0.4).\end{array}$$

Example 4.2

$$\begin{array}{rl}{\beta }_1(t,z_1)& =2(-\sin (\pi t)+\cos (0.5\pi z_1))\\ & =-2\sin (\pi t)+2\cos (0.5\pi z_1)),\\ {\beta }_2(t,z_2)& =2(\sin (\pi t)\cdot \cos (\pi z_2)),\\ {\beta }_3(t,z_3)& =3.5(-z_3+(t-0.5{)}^2+0.4)\\ & =3.5(-z_3+t^2-t+0.65).\end{array}$$

We use basis functions as illustrated in Table 1 in Example 4.1. And we add sine, cosine basis functions for each ${\beta }_i(i=1,2,3)$ in Example 4.2 as follows $$\begin{gathered} (\sin (0.5\pi T),\sin (\pi T),\sin (1.5\pi T),\sin (0.5\pi Z_i),\sin (\pi Z_i),\sin (1.5\pi Z_i),\cos (0.5\pi T), \\ \cos (\pi T),\cos (1.5\pi T),\cos (0.5\pi Z_i),\cos (\pi Z_i),\cos (1.5\pi Z_i)). \end{gathered}$$ It can be seen the set of constructing candidate basis functions consists of the set of true functions in Example 4.1. But in Example 4.2, it is not the situation, so we can only give approximate results.

Figures 1 and 2 depict the true and the estimated coefficient functions for sample size n = 400 averaged over the 50 pseudo samples using our proposed method. The estimated surfaces in the figures suggest that our method estimates the true coefficients well. We compare our proposed method with that of [10], denoted by backfitting algorithm 1. We compute the mean integrated squared errors (MISE), the integrated squared biases (ISB), the integrated variances (IV), the number of iterations, and the computation time of the estimators. The formulas of $MIS{E}_j$, $IS{B}_j$, and $I{V}_j$, j = 1, 2, 3, are given by

$$\begin{array}{rl}{\mathrm{MISE}}_j& =E\int {({\hat{\beta }}_j(t,z_j)-{\beta }_j(t,z_j))}^2\mathrm{d}{z}_j\mathrm{d}t\\ & =\int {(E{\hat{\beta }}_j(t,z_j)-{\beta }_j(t,z_j))}^2\mathrm{d}{z}_j\mathrm{d}t+\int E{({\hat{\beta }}_j(t,z_j)-E{\hat{\beta }}_j(t,z_j))}^2\mathrm{d}{z}_j\mathrm{d}t\\ & \equiv IS{B}_j+I{V}_j,j=1,2,3,\end{array}$$

and $\mathrm{MISE}={\mathrm{MISE}}_1+{\mathrm{MISE}}_2+{\mathrm{MISE}}_3$. The BFA ${}^1$ depends on the selected bandwidths. We select the optimal bandwidths $h_k$, $0\le k\le 3$, that minimize the asymptotic mean integrated squared error (AMISE), where $AMISE=\sum _{k=1}^{3}AMIS{E}_k$. For a given $h_0$, we choose $h_k(h_0)$, dependent on $h_0$, which minimizes $AMIS{E}_k$. Denote such minima by $AMIS{E}_k(h_0)$. Then, we choose the optimal $h_0$ which minimizes $\sum _{k=1}^{3}AMIS{E}_k$. The optimal $h_k,1\le k\le 3$, are correspondingly obtained. We use Epanechnikov kernel, the convergence criteria for $1\le k\le d$ is

$$\int \int ({\beta }_k^{(r)}(t,z_k)-{\beta }_k^{(r-1)}(t,z_k){)}^2\mathrm{d}t\mathrm{d}{z}_k<ϵ,$$

where ${\beta }_k^{(r)}(t,z_k)$ is the tth iteration result of true ${\beta }_k(t,z_k)$, the threshold value ϵ is 0.00001. We summarize values of bandwidths in Figure 3.

Figure 1.

The lower panels are the true coefficient functions, and the upper panels are their estimates for n = 400 averaged over 50 pseudo samples in Example 4.1.

Figure 2.

The lower panels are the true coefficient functions, and the upper panels are their estimates for n = 400 averaged over 50 pseudo samples in Example 4.2.

Figure 3.

Values of bandwidth in Examples 4.1 and 4.2 in simulations.

Remark 4.1

The convergence threshold is a number close to zero in the numerical calculation, such as ${10}^{-5}$ and ${10}^{-7}$. It depends on researcher's tolerence. Sometimes the convergence threshold is also called the tolerence. Researchers give threshold values without any justification in most cases. For example, the convergence threshold in [10] is ${10}^{-5}$; the convergence threshold in [26] is ${10}^{-4}$. We set the threshold to ${10}^{-5}$ considering the calculation time and accuracy. Higher accuracy requires more computation time. As we mentioned before, the convergence rate of backfitting algorithms is $n^{1/3}$. Even though we set the threshold to ${10}^{-20}$, it is not going to make a qualitative difference.

Tables 2 and 3 summarize the results. The values of MISE, ISB, and IV decrease as the sample size n increases. Our method is higher in accuracy and more rapid in speed.

Table 2.

Comparison of the mean integrated squared errors (MISE), the integrated squared biases (ISB), the integrated variances (IV), the number of iterations, and the computing time in Example 4.1.

	n = 200		n = 400
	Our proposed method	Backfitting algorithm 1	Our proposed method	Backfitting algorithm 1
MISE ${}_1$	0.0105	0.0331	0.0033	0.0259
MISE ${}_2$	0.0209	0.0668	0.0096	0.0186
MISE ${}_3$	0.0032	0.0270	0.0024	0.0123
MISE	0.0346	0.1269	0.0153	0.0568
ISB ${}_1$	0.0019	0.0217	0.0010	0.0176
ISB ${}_2$	0.0074	0.0460	0.0038	0.0091
ISB ${}_3$	0.0013	0.0234	0.0010	0.0083
IV ${}_1$	0.0086	0.0114	0.0023	0.0083
IV ${}_2$	0.0135	0.0208	0.0058	0.0095
IV ${}_3$	0.0019	0.0036	0.0014	0.0041
Time	0.0327s	110.6247s	0.0422s	257.85s
Iterations	−	7.5200	−	7.7800

Table 3.

Comparison of the mean integrated squared errors (MISE), the integrated squared biases (ISB), the integrated variances (IV), the number of iterations, and the computing time in Example 4.2.

	n = 200		n = 400
	Our proposed method	Backfitting algorithm 1	Our proposed method	Backfitting algorithm 1
MISE ${}_1$	0.0281	0.0655	0.0102	0.0357
MISE ${}_2$	0.0781	0.1059	0.0234	0.0358
MISE ${}_3$	0.0041	0.0183	0.0009	0.0113
MISE	0.1103	0.1897	0.0345	0.0828
ISB ${}_1$	0.0106	0.0215	0.0016	0.0173
ISB ${}_2$	0.0502	0.0598	0.0105	0.0151
ISB ${}_3$	0.0007	0.0085	0.0007	0.0064
IV ${}_1$	0.0175	0.0439	0.0086	0.0184
IV ${}_2$	0.0279	0.0461	0.0129	0.0207
IV ${}_3$	0.0035	0.0098	0.0008	0.0049
Time	0.0766s	140.665s	0.1335s	330.8357s
Iterations	−	8.3800	−	8.5600

In Examples 4.3 and 4.4, two longitudinal covariates are generated by

$$\begin{array}{rl}{Z}_{ij1}& =2T_{ij}-0.5+a_{i1}+b_{i1}{T}_{ij}+c_{ij1},\\ Z_{ij2}& =0.5+4(T_{ij}-0.5{)}^3+a_{i2}+b_{i2}{T}_{ij}+c_{ij2},\end{array}$$

where $(a_{i1},b_{i1},a_{i2},b_{i2}{)}^{\mathrm{\top }}$ are $i.i.d.$ having a multivariate normal distribution with $\mathbf{0}$ mean and covariance

$$\left[\begin{array}{cccc}0.09& -0.03& 0.045& -0.03\\ -0.03& 0.04& -0.03& 0.02\\ 0.045& -0.03& 0.09& -0.03\\ -0.03& 0.02& -0.03& 0.04\end{array}\right],$$

and $c_{ij1}$ and $c_{ij2}$ are $i.i.d.$ $N(0,{0.4}^2)$ and independent of $({\alpha }_{i1},{\beta }_{i1},{\alpha }_{i2},{\beta }_{i2})$. We take d = 2 and consider the following two regression functions.

Example 4.3

$m(t,\mathit{z})=1+(t-0.5{)}^2+t{z}_1^2+(1-t)z_2^2$.

Example 4.4

$m(t,\mathit{z})=1+(t-0.5{)}^2+\sin (2\pi t)z_1^2+(1-\sin (2\pi t))z_2^2.$

The stochastic components $w_{ij}$ are generated by $w_{ij}=w_i(T_{ij})=\sum _{q=1}^4{A}_{iq}{\varphi }_q(T_{ij})$, where $A_{iq}$ are independent random variables having $N(0,{\lambda }_q)$ distribution with ${\lambda }_q=1/(q+1{)}^2$, $1\le q\le 4$, and

$$\begin{array}{rl}{\varphi }_1(t)& =\sqrt{2}\cos (2\pi t),{\varphi }_2(t)=\sqrt{2}\sin (2\pi t),\\ {\varphi }_3(t)& =\sqrt{2}\cos (4\pi t),{\varphi }_2(t)=\sqrt{2}\sin (4\pi t).\end{array}$$

Then, the response data $Y_{ij}=m(Tij,{\mathit{Z}}_{ij})+w_{ij}+e_{ij}$ are obtained, where $e_{ij}$ are i.i.d random noises having $N(0,{0.05}^2)$ distribution.

If we use polynomials as basis functions, true coefficients in Example 4.3 are shown in Table 4. In Example 4.4, time T is expanded in third-order polynomials, sine and cosine functions. Sine and cosine functions include $(\sin (2\pi T),\sin (4\pi T),\sin (6\pi T),\cos (2\pi T),\cos (4\pi T),\cos (6\pi T))$, $Z_1$ and $Z_2$ are expanded in third-order polynomials. So

$$\mathrm{\Theta }=(M(T),M(Z_1),M(Z_2),M(T)\circ M(Z_1),M(T)\circ M(Z_2)),$$

where $M(T)$ consists of third-order polynomials, sine and cosine functions, $M(Z_1)$ and $M(Z_2)$ are third-order polynomials.

Table 4.

Basis functions and corresponding coefficients in Example 4.3.

$\mathrm{\Theta }(M)$	${\mu }_0$	${\mu }_1$	${\mu }_2$
1	1.25	0	0
t	$-$1	0	0
$z_1$	0	0	0
$z_2$	0	0	0
$t^2$	1	0	0
$t{z}_1$	0	0	0
$t{z}_2$	0	0	0
$z_1^2$	0	0	0
$z_1{z}_2$	0	0	0
$z_2^2$	0	0	1
$t^3$	0	0	0
$t^2{z}_1$	0	0	0
$t^2{z}_2$	0	0	0
$t{z}_1^2$	0	1	0
$t{z}_1{z}_2$	0	0	0
$t{z}_2^2$	0	0	$-1$
$z_1^3$	0	0	0
$z_1^2{z}_2$	0	0	0
$z_1{z}_2^2$	0	0	0
$z_2^3$	0	0	0

Figures 4 and 5 depict the true and the estimated coefficient functions for sample size n = 400 averaged over the 50 pseudo samples using our proposed method. The estimated surfaces in the figures suggest that our method estimates the true coefficients well. We compare our proposed method with that of [26], denoted by backfitting algorithm 2. The bandwidths selection method and convergence criteria are the same as Examples 4.3 and 4.4. Values of bandwidths are illustrated in Figure 6.

Figure 4.

The lower panels are the true coefficient functions, and the upper panels are their estimates for n = 400 averaged over 50 pseudo samples in Example 4.3.

Figure 5.

The lower panels are the true coefficient functions, and the upper panels are their estimates for n = 400 averaged over 50 pseudo samples in Example 4.4.

Figure 6.

Values of bandwidth in Examples 4.3 and 4.4 in simulations.

Tables 5 and 6 summarize the results. The values of MISE, ISB, and IV decrease as the sample size n increases. Our method is higher in accuracy and more rapid in speed.

Table 5.

Comparison of the mean integrated squared errors (MISE), the integrated squared biases (ISB), the integrated variances (IV), the number of iterations, and the computing time in Example 4.3.

	n = 200		n = 400
	Our proposed method	Backfitting algorithm 2	Our proposed method	Backfitting algorithm 2
MISE ${}_0$	0.0296	−	0.0043	−
MISE ${}_1$	0.0088	0.0498	0.0029	0.0347
MISE ${}_2$	0.0076	0.0340	0.0008	0.0272
MISE	0.0164	0.0838	0.0037	0.0619
ISB ${}_0$	0.0026	−	0.0007	−
ISB ${}_1$	0.0032	0.0174	0.0005	0.0111
ISB ${}_2$	0.0009	0.0250	0.0005	0.0213
IV ${}_0$	0.0270	−	0.0036	−
IV ${}_1$	0.0056	0.0324	0.0024	0.0235
IV ${}_2$	0.0075	0.0090	0.0003	0.0059
Time	0.0048s	25.7134s	0.0062s	43.9962s
Iterations	−	4.5600	−	4.8800

Table 6.

Comparison of the mean integrated squared errors (MISE), the integrated squared biases (ISB), the integrated variances (IV), the number of iterations, and the computing time in Example 4.4.

	n = 200		n = 400
	Our proposed method	Backfitting algorithm 2	Our proposed method	Backfitting algorithm 2
MISE ${}_0$	0.0621	−	0.0077	−
MISE ${}_1$	0.0391	0.0895	0.0066	0.0692
MISE ${}_2$	0.0039	0.0222	0.0012	0.0160
MISE	0.0430	0.1117	0.0078	0.0852
ISB ${}_0$	0.0561	−	0.0023	−
ISB ${}_1$	0.0299	0.0518	0.0015	0.0496
ISB ${}_2$	0.0006	0.0162	0.0004	0.0148
IV ${}_0$	0.0059	−	0.0054	−
IV ${}_1$	0.0092	0.0377	0.0051	0.0196
IV ${}_2$	0.0033	0.0060	0.0008	0.0012
Time	0.0589s	48.2093s	0.0924s	127.2423s
Iterations	−	6.5600	−	7.2600

5. Real data analysis

The data set comes from the UCI machine learning repository, called Beijing Multi-Site Air-Quality Data Data Set [24]. It includes hourly air pollutants data from 12 nationally controlled air-quality monitoring sites. The air-quality data are collected by the Beijing Municipal Environmental Monitoring Center. Variables consist of $P{M}_{2.5}$ concentration $(\mathrm{\text{μ}}\mathrm{g}/{\mathrm{m}}^3)$, $P{M}_{10}$ concentration $(\mathrm{\text{μ}}\mathrm{g}/{\mathrm{m}}^3)$, ${\mathrm{SO}}_2$ concentration $(\mathrm{\text{μ}}\mathrm{g}/{\mathrm{m}}^3)$, ${\mathrm{NO}}_2$ concentration $(\mathrm{\text{μ}}\mathrm{g}/{\mathrm{m}}^3)$, CO concentration $(\mathrm{\text{μ}}\mathrm{g}/{\mathrm{m}}^3)$ and ${\mathrm{O}}_3$ concentration $(\mathrm{\text{μ}}\mathrm{g}/{\mathrm{m}}^3)$. The meteorological data in each air-quality site are matched with the nearest weather station from the China Meteorological Administration, including temperature (degree Celsius), pressure (hPa), dew point temperature (degree Celsius), precipitation, wind direction, and wind speed (m/s). The period is from March 1st, 2013 to February 28th, 2017. Missing data are denoted as NA. One of the interesting purposes is to determine the relationship between $P{M}_{2.5}$ and other impact factors.

According to domain knowledge and previous research, meteorology factors and gaseous pollutants are the main contributors to the formation process of $P{M}_{2.5}$. For gaseous pollutants, ${\mathrm{SO}}_2$, ${\mathrm{NO}}_2$, and CO are selected in our models. The reason we do not consider ${\mathrm{O}}_3$ is that [15] concluded ${\mathrm{O}}_3$ had little effect on $P{M}_{2.5}$. Besides, the correlation coefficient between $P{M}_{2.5}$ and ${\mathrm{O}}_3$ is $-0.18$, weakly correlated. For meteorology factors, temperature and wind speed are chosen. Dew point temperature and pressure are also weakly correlated to the concentration of $P{M}_{2.5}$, with correlation coefficients 0.02 and 0.01, respectively.

The original data set is from 12 nationally-controlled air-quality monitoring sites, we only analyze data from Wanshouxigong station in this study. There are 30666 hourly samples without missing values. Viewed as longitudinal data, there are n = 1461 daily observations with each day's time point $1<M_i\le 24$ during the period. We scale the interval of time point from $[0,23]$ to $[0,1]$, and the concentration of CO is divided by 100. We split the data into a training set with 25000 randomly sampled data and a test set with the remaining data. We consider different types of varying coefficient models to analyze the relationship between the concentration of $P{M}_{2.5}$ and the selected covariates. The measure of assessment is the relative mean squared error (RMSE) given by

$$\mathrm{RMSE}=\frac{\sum _{i,j}({P{M}_{2.5}}_{ij}-{\widehat{P{M}_{2.5}}}_{ij}{)}^2}{\sum _{i,j}({P{M}_{2.5}}_{ij}-{\overline{P{M}_{2.5}}}_{ij}{)}^2},$$

It can been seen the smaller the RMSE is, the better the model fits. We do 100 simulations and calculate the mean RMSE as the final RMSE in test data.

Model 1: additive model, refer to [20].

$$P{M}_{2.5}(t)=f_0(t)+f_1({\mathrm{SO}}_2)+f_2({\mathrm{NO}}_2)+f_3(\mathrm{CO})+f_4(T)+f_5(WS)+noise,$$

we use ‘gam’ in R package ‘mgcv’ to fit Model 1, the RMSE is 0.2998.

Model 2: varying coefficient model in (1)

$$\begin{array}{rl}P{M}_{2.5}(t)& ={\beta }_0(t)+{\beta }_1(t)({\mathrm{SO}}_2)+{\beta }_2(t)({\mathrm{NO}}_2)+{\beta }_3(t)(\mathrm{CO})\\ & +{\beta }_4(t)(T)+{\beta }_5(t)(WS)+noise,\end{array}$$

the RMSE is 0.2930 when basis functions of ${\beta }_j$ are three-order polynomials and sin functions.

Model 3: varying coefficient model in (3)

$$\begin{array}{rl}P{M}_{2.5}(t,\mathit{z})& ={\mu }_0(t)+{\mu }_1(t,{\mathrm{SO}}_2)+{\mu }_2(t,{\mathrm{NO}}_2)+{\mu }_3(t,\mathrm{CO})\\ & +{\mu }_4(t,T)+{\mu }_5(t,WS)+noise,\end{array}$$

the RMSE is 0.2836 when basis functions of ${\mu }_j$ are three-order polynomials and sin functions. Model 2 is a special case of Model 3. By comparing RMSE, Model 3 is better than Model 2. Figure 7 depicts the estimated surfaces of different components. The concentration of $P{M}_{2.5}$ increases as ${\mathrm{NO}}_2$, CO, and temperature increase. The result is consistent with the fact ${\mathrm{NO}}_2$ and CO are pollutants that compose $P{M}_{2.5}$. The temperature has a higher impact on the concentration of $P{M}_{2.5}$ at noon than in the morning and evening when the temperature is high. In contrast, it has a higher impact in the morning and evening than at noon when the temperature is low.

Figure 7.

Estimates of coefficient functions in Model 3.

Model 4: varying coefficient model in (4)

$$\begin{array}{rl}P{M}_{2.5}(t)& ={\beta }_0(t,WS(t))+{\beta }_1(t,WS(t)){\mathrm{SO}}_2(t)+{\beta }_2(t,WS(t)){\mathrm{NO}}_2(t)\\ & +{\beta }_3(t,WS(t))\mathrm{CO}(t)+{\beta }_4(t,WS(t))(T(t))+noise,\end{array}$$

different from the above-mentioned models, the problems which covariate should be the linear component and which covariate should be the varying coefficient component, are unclear for Model 4. We use cubic polynomials as the library of functions for full data, and find out the coefficients of terms $windspeed\times t\times \mathrm{CO}$ and $windspeed\times t\times {\mathrm{SO}}_2$ are not zero. According to the result in reference [24], air quality will be better once the wind blows away $P{M}_{2.5}$. Cooling is often accompanied by strong winds. This result is consistent with our life experience in Beijing. So Model 4 makes sense according to our data-driven pre-estimation result and research in [24].

If we only use cubic polynomials, the RMSE is 0.2935; if we use cubic polynomials and sine functions simultaneously, the RMSE is 0.2916; if we use second-order polynomials, the RMSE is 0.3020; if we use both second-order polynomial and sine functions, the RMSE is 0.3020. By comparing RMSE, we choose cubic polynomials and sine functions in this case. Figure 8 depicts the estimated surface of each component.

Figure 8.

Estimates of coefficient functions in Model 4.

By comparing RMSE of four models, Model 3 performs best.

6. Discussion

Designing efficient and accurate algorithms to estimate varying coefficient models is of significance as more and more data is available. We focus on different types of dynamic varying coefficient models and propose a new procedure to estimate the varying component efficiently in our research. The main idea is to expand the varying component with appropriate basis functions and then solve a sparse regression problem. Finally, we can get an explicit expression of the model under certain conditions. Furthermore, the proposed procedure can also find out an expression without model assumptions as shown in the real data analysis.

Constructing a library of functions, especially when the number of covariates is large is a challenge to us. To deal with this, we can incorporate domain knowledge and other related methods. Still taking the concentration of $P{M}_{2.5}$ as an example, the existing research results show meteorology factors and gaseous pollutants are the main contributors to the formation process of $P{M}_{2.5}$, so the feature extraction process will depend on these two aspects. Curse of dimensionality will occur when the number of covariates is large. To handle this, before implementing the above-mentioned procedure, we can delete some unimportant variables as in the variable selection and feature screening procedure. This is based on the sparsity assumption. Sensitivity to noise is also troublesome. One technique is to use absolute loss function or check function in quantile regression to get robust estimation. And the other is to take advantage of Bayesian inference [23]. We will study robust estimation under the frame of SINDy in our future research.

Funding Statement

The work was partially supported by the National Natural Science Foundation of China [grant number 11861042], and the China Statistical Research Project [grant number 2020LZ25].

Disclosure statement

No potential conflict of interest was reported by the author(s).