Quadratic inference with dense functional responses

Abstract

We address the challenge of estimation in the context of constant linear effect models with dense functional responses. In this framework, the conditional expectation of the response curve is represented by a linear combination of functional covariates with constant regression parameters. In this paper, we present an alternative solution by employing the quadratic inference approach, a well-established method for analyzing correlated data, to estimate the regression coefficients. Our approach leverages non-parametrically estimated basis functions, eliminating the need for choosing working correlation structures. Furthermore, we demonstrate that our method achieves a parametric $\sqrt{n}$-convergence rate, contingent on an appropriate choice of bandwidth. This convergence is observed when the number of repeated measurements per trajectory exceeds a certain threshold, specifically, when it surpasses $n^{a_0}$, with $n$ representing the number of trajectories. Additionally, we establish the asymptotic normality of the resulting estimator. The performance of the proposed method is compared with that of existing methods through extensive simulation studies, where our proposed method outperforms. Real data analysis is also conducted to demonstrate the proposed method.

1. Introduction

Longitudinal data analysis (LDA) involves tracking repeated measurements on the same individuals over time, allowing us to study changes over time and identify influencing factors. Unlike cross-sectional studies, which capture only “between-individual” responses at a single time point, LDA can capture “within-individual” changes through repeated measurements. Longitudinal data are often observed in clusters, with each cluster representing measurements from one individual. As data complexity increases with advancing technology, functional data analysis (FDA) has become a vital tool, extending our understanding from finite to infinite dimensions. In LDA, data are generally observed with noise for measurements at each time-point [6, 16]. Moreover, a few repeated measurements are required in LDA, and the data are observed sparsely with noise. On the other hand, in FDA, data are densely observed as a continuous-time stochastic process without or with noise [22,37]. Often, the sampling plan can have an effect on the performance of the estimation procedures and inference [14]. In some situations, data are typically functions by nature and are observed densely over time. Chiou et al. [4] proposed a class of semi-parametric functional regression models to describe the influence of vector-valued covariates on a sample of the response curve. When data collection leads to experimental error, smoothing is performed at closely spaced time-points in order to reduce the effect of noise. The current developments of functional regression techniques have been rigorously studied in Chen et al. [3], Fan and Zhang [8], Hall and Horowitz [13]. See a more recent and complete review in Li et al. [22].

The generalized estimating equation (GEE) technique proposed by Liang and Zeger [23] has been extensively used in LDA for estimation of parameters. Although an efficient technique, the GEE may not be efficient when the correlation matrix is not correctly specified. Hence, without requiring the estimation of the correlation parameters, the quadratic inference function (QIF) approach proposed by Qu et al. [29] is useful for parameter estimation in longitudinal studies [6] and cluster randomized trials [31]. By representing the inverse of the working correlation matrix in terms of linear combinations of the basis matrices and involving multiple sets of score functions, the QIF approach has improved efficiency over GEE when the working correlation matrix is not specified correctly. Although it maintains the same efficiency as in the situation where the working correlation matrix is specified correctly, the QIF method is not independent of the choice of the working correlation matrix. A QIF method-based approach to varying-coefficient models for longitudinal data was proposed by Qu and Li [28]. The related work of Bai et al. [1] is an extension of QIF for the partial linear model. An alternative method was presented in Yu et al. [34] where each set of score equations was solved separately and their solutions were combined afterward; thereby providing results on inference for an optimally weighted estimator and extending those insights to the general setting with over-identified estimating equations.

The fundamental limitations that all the above-mentioned powerful techniques suffer from are: (1) all the above methodologies require prior information on the working correlation structure or choose appropriate basis matrices, and (2) the performance of the classical QIF approach is unknown for dense functional data. Our study is motivated by problems from multiple real data applications that involve dense functional data when information on the working correlation structure is lacking. Let us discuss two motivating examples that we will use to illustrate the proposed method in this paper (see Section 5 for details). In the Beijing2017-data example, particulate matter (PM) with a diameter of less than 2.5 micrometers is collected over different time-points in different locations in China. Scientists are interested in knowing the linear dependence of the pollution factor PM_2.5 with other atmospheric chemicals [24]. Figure 1 (left) in Section 5 pictorially demonstrates the readings of PM_2.5 for the given locations over several hourly time-points; therefore, dense functional data analysis can be implemented. In another example from a neuroimaging study Apnea-data, scientists are interested in modeling the change of white matter structure among voxels in each region of interest (ROI) of the human brain. Xiong et al. [32] investigated white matter structural alterations using diffusion tensor imaging (DTI) in obstructive sleep apnea (OSA) patients. Here, the change of DTI parameters such as fractional anisotropy (FA) is investigated by using constant linear effect models (see subsection 2.1 for a formal definition) with the interaction of the count of lapses obtained from the Psychomotor Vigilance Task and voxel locations as predictors. We applied this model to each ROI and compared the results obtained from all the ROIs.

Fig. 1:

Beijing2017-data: (Left) Reading of hourly PM_2.5 measures for twelve different locations over 608 hourly time-points during January 2017. (Right) Scree plots of a fraction of variance explained (FVE) to determine the optimal number of components to retain in functional principal component analysis. The red line indicates the cumulative fraction of variance across the number of components.

We propose a data-driven way to select the working covariance matrix and express the inverse of the covariance function in terms of the empirical eigen-functions of the covariance operator. The covariance operator can be estimated as in Hsing and Eubank [18] and other related methods based on functional principal component analysis (FPCA) as found in Dauxois et al. [5], Hall and Horowitz [13], Hall and Hosseini-Nasab [14], Li and Hsing [21], Yao et al. [33], Zhang and Li [35]. Note that the estimation of the eigen-functions is nonparametric and it introduces some errors in the proposed estimation method. In this article, we answer the following question: while we estimate the eigen-functions nonparametrically from data, is the estimation of coefficient vectors in a semi-parametric problem $\sqrt{n}$-consistent in dense functional data, and can we achieve asymptotic normality? The advantages of our proposed method are the following: First, our method preserves the good properties of the QIF method and is easier to implement since the eigen-functions can be estimated using the existing packages in statistical software such as R. Second, under some mild conditions, our proposed estimator can obtain the optimal convergence rate and is asymptotically normally distributed with less variance as compared to the classical QIF methods. Third, asymptotic results show the estimation accuracy of the coefficients in semi-parametric functional model, therefore, making the influence of the dimension reduction step using FPCA redundant. The error in the estimation of the eigen-functions contributes to the error in the estimation of the parameters. Under some mild bandwidth conditions, the above-mentioned error contribution is of the same order of magnitude as an error in parameter estimation if eigen-functions are known in advance.

The rest of the paper is organized as follows. In Section 2, we introduce the basic concept of QIF along with our proposed method. The asymptotic results for the proposed estimator are presented in Section 3. In Section 4, we demonstrate the performance for finite samples. We also apply the proposed method to real data-sets in Section 5. We conclude with some remarks in Section 6. All technical proofs are presented in the Appendix of the article. The supplementary material contains additional tables from simulation results.

2. Functional response model and estimation procedure

2.1. Basic model

To analyze longitudinal data, a straightforward application of a generalized linear model [27] for single response variables is not applicable due to the lack of independence between repeated measures. To account for the high correlation in the longitudinal data, some special techniques are required. A seminal work by Liang and Zeger [23] proposed the use of generalized linear models for the analysis of longitudinal data. The model we consider in this article is commonly observed in spatial modeling, where associations among variables do not change over the functional domain (see Zhang and Banerjee [36] and references therein); which is termed as constant linear effects model. In neuroimaging studies, constant linear effects model is a popular choice for the region of interest analysis (see Friston et al. [9, 10], Lindquist [25], Xiong et al. [32]) due to its easy and practical interpretations of the constant coefficients. In this paper, the variable time is used as a functional domain variable.

Let $y\left(t\right)$ be the response variable at time-point $t$ and $\mathbf{x}\left(t\right)$ be $p$-dimensional covariates observed at time $t\in 𝒯$ where without loss of generality we assume $𝒯=\left[\mathrm{0,1}\right]$, is the spectrum of the time-points. The stochastic process $y\left(t\right)$ is square-integrable with conditional mean $\mathbb{E}\left\{y\right(t)\mid \mathbf{x}(t\left)\right\}$ and finite covariance function; the regression parameter $\mathit{\beta }$ is unknown and is to be efficiently estimated. Thus, the constant linear effects model with longitudinal data has the following expression,

$$y\left(t\right)=\mathbf{x}(t{)}^{\mathrm{T}}\mathit{\beta }+e(t),$$ (1)

where the stochastic process $y\left(t\right)$ is decomposed into two parts: one is the mean function $\mu \left(t\right)=\mathbf{x}(t{)}^{\mathrm{T}}\mathit{\beta }$ that depends on time-varying covariates and vector-valued coefficient vector $\mathit{\beta }$, and the other is the random error part $e\left(t\right)$ where $\mathbb{E}\left\{e\right(t\left)\right\}=0$ and has finite second-order covariance $R(s,t)$. Let $y_i\left(t\right)$ be i.i.d. copies of the stochastic process and for each individual, the measurements are taken on $m_i$ discrete time-points $T_{ij}$ for $j\in \left\{1,\dots ,m_i\right\};i\in \{1,\dots ,n\}$. Therefore, at time $T_{ij}$, we observe a $m_i\times 1$ response vector $y_i\left(T_{ij}\right)$ and corresponding covariates ${\mathbf{x}}_i\left(T_{ij}\right)$ for the $i-\text{th}$ subject. We assume that $m_i\prime \mathrm{s}$ are all of the same order as $m=n^a$ for some $a\ge 0$, thus $m_i/m$ is bounded below and above by some constants. Functional data are considered to be sparse depending on the choice of $a$ [14]. Data with bounded $m$ or $a=0$ are called sparse functional data and if $a\ge a_0$ where $a_0$ is a transition point, are called dense functional data. Moreover, the regions $\left(0,a_0\right)$ are sometimes referred to as moderately dense. Furthermore, we denote $y_{ij}$ and ${\mathbf{x}}_{ij}$ as $y_i\left(T_{ij}\right)$ and ${\mathbf{x}}_i\left(T_{ij}\right)$ respectively. Therefore, ${\left(y_{i1},\dots ,y_{i{m}_i}\right)}^{\mathrm{T}}$ and ${\left({\mu }_{i1},\dots ,{\mu }_{i{m}_i}\right)}^{\mathrm{T}}$ are $m_i$ component vectors, denoted as ${\mathbf{y}}_i$ and ${\mathit{\mu }}_i$ respectively. In addition, define residuals ${\mathbf{e}}_i={\mathbf{y}}_i-{\mathit{\mu }}_i,m_i\times p$ design matrices ${\mathbf{X}}_i={\left({\mathbf{x}}_{ij}\right)}_{i=1,j=1}^{m_{i}p}$, and the derivative of ${\mathit{\mu }}_i$ with respect to $\mathit{\beta }$, denoted as ${\dot{\mathit{\mu }}}_i$, is a $m_i\times p$ matrix. For instance, in case of the linear model discussed in (1), ${\dot{\mathit{\mu }}}_i={\mathbf{X}}_i$. We keep the notation ${\dot{\mathit{\mu }}}_i$ because it could be generalized for other types of responses.

In the classical problem of GEE, we estimate $\mathit{\beta }$ by solving the quasi-likelihood equations [23]:

$$\sum _{i=1}^n{\dot{\mathit{\mu }}}_i^{\mathrm{T}}{\mathbf{V}}_i^{-1}\left({\mathbf{y}}_i-{\mathit{\mu }}_i\right)=0.$$ (2)

We denote ${\mathbf{V}}_i=v{\mathbf{A}}_i^{1/2}{\mathbf{R}}_i\left(\rho \right){\mathbf{A}}_i^{1/2}$ where ${\mathbf{R}}_i\left(\rho \right)$ is the working correlation matrix, $v$ is an over-dispersion parameter and ${\mathbf{A}}_i$ is a diagonal matrix where entries are marginal variances $\text{Var}\left(y_{i1}\right),\dots ,\text{Var}\left(y_{i{m}_i}\right)$. In this article, we simply set $v=1$ while the extension to a general $v$ is straightforward. In practice, the prior knowledge of the working correlation matrix is unknown, and the estimation of the coefficient is influenced by its choice. Therefore, Qu et al. [29] suggested an expansion of the inverse of the working correlation matrix as $\mathbf{R}(\rho {)}^{-1}=\sum _{k=1}^{{\kappa }_0}{a}_k(\rho ){\mathbf{M}}_k$ where ${\mathbf{M}}_k$ are some basis matrices. Zhou and Qu [38] modified linear representation by grouping the basis matrices into an identity matrix and some symmetric basis matrices. For example, if the working correlation matrix is exchangeable/compound symmetric, $\mathbf{R}(\rho {)}^{-1}=c_1{\mathbf{I}}_m+c_2{\mathbf{J}}_m$ where ${\mathbf{I}}_m$ is the $m\times m$ identity matrix and ${\mathbf{J}}_m$ is the $m\times m$ matrix such that 0 is in diagonal and 1 is in off-diagonal positions. On the other hand, for first-order auto-regressive correlation matrix, $\mathbf{R}(\rho {)}^{-1}=c_1{\mathbf{I}}_m+c_2{\mathbf{J}}_m^{\left(1\right)}+c_3{\mathbf{J}}_m^{\left(2\right)}$ where ${\mathbf{J}}_m^{\left(1\right)}$ is a matrix with 1 on the two main off-diagonal positions and 0 otherwise, ${\mathbf{J}}_m^{\left(2\right)}$ is a matrix such that 1 is in the corner positions, viz. (1, 1) and $(m,m)$ and 0 elsewhere. Here, $c_k\mathrm{s}$ are real constants that depend on the nuisance parameter $\rho$. Therefore, (2) reduces to the linear combination of the score vectors:

$$\overline{\mathbf{g}}\left(\beta \right)≔n^{-1}\sum _{i=1}^n{\mathbf{g}}_i\left(\beta \right)≔\left[\begin{array}{c}{\overline{\mathbf{g}}}^{\left(1\right)}\left(\beta \right)\\ ⋮\\ {\overline{\mathbf{g}}}^{\left({\kappa }_0\right)}\left(\beta \right)\end{array}\right]=\left[\begin{array}{c}{n}^{-1}\sum _{i=1}^n{\dot{\mathit{\mu }}}_i^{\mathrm{T}}{\mathbf{A}}_i^{-1/2}{\mathbf{M}}_1{\mathbf{A}}_i^{-1/2}\left({\mathbf{y}}_i-{\mathit{\mu }}_i\right)\\ ⋮\\ n^{-1}\sum _{i=1}^n{\dot{\mathit{\mu }}}_i^{\mathrm{T}}{\mathbf{A}}_i^{-1/2}{\mathbf{M}}_{{\kappa }_0}{\mathbf{A}}_i^{-1/2}\left({\mathbf{y}}_i-{\mathit{\mu }}_i\right)\end{array}\right],$$ (3)

where ${\overline{\mathbf{g}}}^{\left(k\right)}\left(\mathit{\beta }\right)≔n^{-1}\sum _{i=1}^n{\mathbf{g}}^{\left(k\right)}\left(\mathit{\beta }\right)≔n^{-1}\sum _{i=1}^n{\dot{\mathit{\mu }}}_i^{\mathrm{T}}{\mathbf{A}}_i^{-1/2}{\mathbf{M}}_k{\mathbf{A}}_i^{-1/2}\left({\mathbf{y}}_i-{\mathit{\mu }}_i\right)$ for $k\in \left\{1,\dots ,{\kappa }_0\right\}$, each ${\overline{\mathbf{g}}}^{\left(k\right)}\left(\mathit{\beta }\right)$ is a $p\times 1$ vector and $\overline{\mathbf{g}}\left(\mathit{\beta }\right)$ is $p{\kappa }_0\times 1$ vector after stacking all ${\overline{\mathbf{g}}}^{\left(k\right)}\left(\mathit{\beta }\right)\mathrm{s}$. Due to the higher dimension of $\overline{\mathbf{g}}$, Qu et al. [29] used the generalized method of moments (GMM) [17] for which the method of estimation boils down to minimization of the quadratic inference function $Q\left(\mathit{\beta }\right)=n\overline{\mathbf{g}}\left(\mathit{\beta }{)}^{\mathrm{T}}\widehat{\mathbf{C}}\right(\mathit{\beta }{)}^{-1}\overline{\mathbf{g}}\left(\mathit{\beta }\right)$ where $\widehat{\mathbf{C}}\left(\mathit{\beta }\right)=n^{-1}\sum _{i=1}^n{\overline{\mathbf{g}}}_i\left(\mathit{\beta }\right){\overline{\mathbf{g}}}_i(\mathit{\beta }{)}^{\mathrm{T}}$ is the sample covariance matrix of (3). To obtain the solution of $\mathit{\beta }$, the Newton-Raphson method is used which iteratively updates the value of $\mathit{\beta }$.

2.2. Incorporating eigen-functions in QIF

Now, due to standard Karhunen-Loève expansions of $e_i\left(t\right)=y_i\left(t\right)-{\mu }_i\left(t\right)$ [20, 26]

$$e_i\left(t\right)=\sum _{r=1}^{\infty }{\xi }_{ir}{\varphi }_r\left(t\right),$$ (4)

with probability 1, where uncorrelated random variables ${\xi }_{ir}=<e_i,{\varphi }_r>$ have zero mean and variance ${\lambda }_r$ for ordered eigen-values ${\lambda }_r$ such that ${\lambda }_1\ge {\lambda }_2\ge \dots \ge 0$ and ${\varphi }_r\mathrm{s}$ are orthonormal eigen-functions such that $\int {\varphi }_r\left(t\right){\varphi }_l\left(t\right)dt=1(r=l)$. Due to Mercer’s theorem [19], a symmetric continuous non-negative definite kernel function $R$ has the representation $R(s,t)=\sum _{r=1}^{\infty }{\lambda }_r{\varphi }_r\left(s\right){\varphi }_r\left(t\right)$ where the sum is absolutely and uniformly continuous. We extract the main directions of the variation of the response variables using FPCA. In this situation, we take the first ${\kappa }_0$ terms which provide a good approximation of the infinite sum in (4) by considering that the majority of the variations in the data are contained in the subspace spanned by few eigen-functions [3]. For finite ${\kappa }_0\ge 1$, we therefore consider the rank-${\kappa }_0$ FPCA model,

$$\mathbb{E}\left\{y\left(t\right)∣\mathbf{x}\left(t\right)\right\}\approx \mu \left(t\right)+\sum _{r=1}^{{\kappa }_0}\mathbb{E}\left\{{\xi }_r\mid \mathbf{x}\left(t\right)\right\}{\varphi }_r\left(t\right).$$ (5)

Unlike existing models in Chiou et al. [4] where the main interest of the work is to study how the vector covariates influence the whole response curve, we study point-wise effect of covariates on functional response the influence of covariates vectors indirectly on the functional response through the regression of scores on covariates, via., $\mathbb{E}\left\{{\xi }_r\mid \mathbf{x}\left(t\right)\right\}$. An analogue of the truncated empirical version of (4) and Mercer’s representation can be provided easily, and we discuss the proposed method based on this truncated version. Moreover, we discuss how to choose ${\kappa }_0$ in our situation in Sections 3 and 4 in detail.

In this paper, we propose a data-driven way to compute the basis matrices to obtain the approximate inverse of $\mathbf{V}$ as discussed earlier. In this approach, it is enough to find the eigen-functions to construct a GEE. Let us define

$$\overline{\mathbf{g}}\left(\mathit{\beta }\right)≔\left[\begin{array}{c}{\overline{\mathbf{g}}}^{\left(1\right)}\left(\beta \right)\\ ⋮\\ {\overline{\mathbf{g}}}^{\left({\kappa }_0\right)}\left(\beta \right)\end{array}\right]≔\left[\begin{array}{c}{n}^{-1}\sum _{i=1}^n{\dot{\mathit{\mu }}}_i^{\mathrm{T}}{\widehat{\mathbf{\Phi }}}_{i1}\left({\mathbf{y}}_i-{\mathit{\mu }}_i\right)\\ ⋮\\ n^{-1}\sum _{i=1}^n{\dot{\mathit{\mu }}}_i^{\mathrm{T}}{\widehat{\mathbf{\Phi }}}_{i{k}_0}\left({\mathbf{y}}_i-{\mathit{\mu }}_i\right)\end{array}\right],$$ (6)

where, for $k=1,\dots ,{\kappa }_0,{\widehat{\mathbf{\Phi }}}_{ik}={\left(m_i^{-2}{\widehat{\varphi }}_k\left(T_{ij}\right){\widehat{\varphi }}_k\left(T_{i{j}^{\text{'}}}\right)\right)}_{j,j^{\text{'}}=1,\dots ,m_i}$ is an $m_i\times m_i$ symmetric matrix and

$${\overline{\mathbf{g}}}^{\left(k\right)}\left(\mathit{\beta }\right)=n^{-1}\sum _{i=1}^n{\mathbf{g}}_i^{\left(k\right)}\left(\mathit{\beta }\right)=n^{-1}\sum _{i=1}^n{\dot{\mu }}_i^{\mathrm{T}}{\widehat{\mathbf{\Phi }}}_{ik}\left({\mathbf{y}}_i-{\mathit{\mu }}_i\right).$$

Since the dimension of $\overline{\mathbf{g}}$ in (6) is greater than the number of parameters to estimate, instead of setting $\overline{\mathbf{g}}$ to zero, we minimize quadratic function by ${\widehat{\mathit{\beta }}=\text{arg}\text{min}}_{\beta }Q\left(\mathit{\beta }\right)$ where $Q\left(\mathit{\beta }\right)=n\overline{\mathbf{g}}\left(\mathit{\beta }{)}^{\mathrm{T}}\widehat{\mathbf{C}}\right(\mathit{\beta }{)}^{-1}\overline{\mathbf{g}}\left(\mathit{\beta }\right)$ where, $\widehat{\mathbf{C}}\left(\mathit{\beta }\right)=n^{-1}\sum _{i=1}^n{\mathbf{g}}_i\left(\mathit{\beta }\right){\mathbf{g}}_i(\mathit{\beta }{)}^{\mathrm{T}}$, where ${\mathbf{g}}_i\left(\mathit{\beta }\right)={\left({\mathbf{g}}_i^{\left(1\right)\mathrm{T}}\left(\mathit{\beta }\right),\dots ,{\mathbf{g}}_i^{\left({\kappa }_0\right)\mathrm{T}}\left(\mathit{\beta }\right)\right)}^{\mathrm{T}}$ and ${\mathbf{g}}_i^{\left(k\right)}\left(\mathit{\beta }\right)={\dot{\mathit{\mu }}}_i^{\mathrm{T}}{\widehat{\mathbf{\Phi }}}_{ik}\left({\mathbf{y}}_i-{\mathit{\mu }}_i\right)$. For the existence of ${\widehat{\mathbf{C}}}^{-1}$ we need the additional restriction: $n\ge {\kappa }_0$ where ${\kappa }_0$ is the number of eigen-functions. Under the given set-up, by Equation (8) in Qu et al. [29] the estimating equation for $\mathit{\beta }$ will be $\dot{Q}\left(\mathit{\beta }\right)\approx 2\dot{\overline{\mathbf{g}}}\left(\mathit{\beta }{)}^{\mathrm{T}}\widehat{\mathbf{C}}\right(\mathit{\beta }{)}^{-1}\overline{\mathbf{g}}\left(\mathit{\beta }\right)$. To obtain the solution of the above equation, we use a Newton-like method. In practice, the standard Newton method does not lead to a decrease in the objective function, that is, at each step of the iteration, there is no guarantee that $Q\left({\mathit{\beta }}_{s+1}\right)<Q\left({\mathit{\beta }}_s\right)$. Therefore, we use the Algorithm 1 to estimate $\mathit{\beta }$ using the Quasi-Newton method with halving [12].

Algorithm 1.

Estimation of $\mathit{\beta }$ using the Quasi-Newton method with halving.

Set ${\widehat{\mathit{\beta }}}_1\leftarrow \tilde{\mathit{\beta }}$ (initial estimates)

Set ${ϵ}_0$ (threshold, a small number)

Set max.count (maximum number of repetition)

Set $l\leftarrow 0$

while $\text{Error}>{ϵ}_0$ and $l\le \max .\text{count}$ do

Calculate $\dot{\mathit{Q}}\left({\widehat{\mathit{\beta }}}_1\right)$ and $\ddot{\mathit{Q}}\left({\widehat{\mathit{\beta }}}_1\right)$ based on ${\widehat{\mathit{\beta }}}_1$ using proposed method

Initialise $r_0\text{=1}$

${\widehat{\mathit{\beta }}}_2\leftarrow {\widehat{\mathit{\beta }}}_1-r_0\ddot{\mathit{Q}}{\left({\widehat{\mathit{\beta }}}_1\right)}^{-1}\dot{\mathit{Q}}\left({\widehat{\mathit{\beta }}}_1\right)$

Calculate $\mathit{Q}\left({\widehat{\mathit{\beta }}}_1\right)$ and $\mathit{Q}\left({\widehat{\mathit{\beta }}}_2\right)$ based on ${\widehat{\mathit{\beta }}}_1$ and ${\widehat{\mathit{\beta }}}_2$ respectively using proposed method

while $\mathit{Q}\left({\widehat{\mathit{\beta }}}_2\right)>\mathit{Q}\left({\widehat{\mathit{\beta }}}_1\right)$ do

$r_0\leftarrow r_0/2$

${\widehat{\mathit{\beta }}}_2\leftarrow {\widehat{\mathit{\beta }}}_1-r_0\ddot{\mathit{Q}}{\left({\widehat{\mathit{\beta }}}_1\right)}^{-1}\dot{\mathit{Q}}\left({\widehat{\mathit{\beta }}}_1\right)$

Calculate $\mathit{Q}\left({\widehat{\mathit{\beta }}}_1\right)$ and $\mathit{Q}\left({\widehat{\mathit{\beta }}}_2\right)$ based on ${\widehat{\mathit{\beta }}}_1$ and ${\widehat{\mathit{\beta }}}_2$ respectively using proposed method

end while

Calculate $\text{Error}={‖{\widehat{\mathit{\beta }}}_2-{\widehat{\mathit{\beta }}}_1‖}^2$

${\widehat{\mathit{\beta }}}_1\leftarrow {\widehat{\mathit{\beta }}}_2$

$l\leftarrow l+1$

end while

Output $\mathit{\beta }$

2.3. Estimation of eigen-functions

Estimation of eigen-functions is an important step in our proposed quadratic inference technique. In general, FPCA plays an important role as a dimension-reduction technique in functional data analysis. Some important theories on FPCA have been developed in recent years. In particular, Hall and Hosseini-Nasab [14] proved various asymptotic expressions for FPCA for densely observed functional data. Later, Hall and Hosseini-Nasab [15] showed more common theoretical arguments, including the effect of the gap between eigen-value (a.k.a., spacing) on the property of eigen-value estimators. In Li and Hsing [21], uniform rates of convergence of the mean and covariance functions are given, which are equipped for all possible choices/scenarios of $m_i\mathrm{s}$.

Note that the error process $e\left(t\right)$ has mean zero, defined on compact set $𝒯=\left[\mathrm{0,1}\right]$ satisfying ${\int }_{𝒯}\mathbb{E}\left\{e^2\left(t\right)\right\}dt<\infty$. The functional principal components can be constructed via the covariance function $R(s,t)$ (induces the kernel operator $ℱ$) defined as $R(s,t)=\mathbb{E}\left\{e\right(s\left)e\right(t\left)\right\}$ which is assumed to be square-integrable. An empirical analogue of the spectral decomposition of $R$ can be obtained $\widehat{R}(s,t)=\sum _{r=1}^{\infty }{\widehat{\lambda }}_r{\widehat{\varphi }}_r\left(s\right){\widehat{\varphi }}_r\left(t\right)$ where the random variables ${\widehat{\lambda }}_1\ge {\widehat{\lambda }}_2\ge \dots \ge 0$ are the eigen-values of the estimated operator $\widehat{ℱ}$ and the corresponding sequence of eigen-functions are ${\widehat{\varphi }}_1,{\widehat{\varphi }}_2,\dots$. Further, assume that ${\int }_{𝒯}{\varphi }_r{\widehat{\varphi }}_r\ge 0$ to avoid the issue regarding change of sign [14] for practical comparison of eigen-functions, otherwise there is no impact on the convergence rate of eigen-functions and hence the proposed estimators.

Suppose that $T_{ij}$ are observational points with a positive density function $f_T(\cdot )$. Assume $m_i\ge 2$ and define $N=\sum _{i=1}^n{N}_i$ where $N_i=m_i\left(m_i-1\right)$. This approach is based on local linear smoother which is popular in functional data analysis, including Fan and Gijbels [7], Li and Hsing [21], Yao et al. [33] among many others. Let $K(\cdot )$ be a symmetric probability density function on [−1, 1], which is used as kernel and $h>0$ be bandwidth, thus re-scaled kernel function is defined as $K_h(\cdot )=h^{-1}K(\cdot )$. Therefore, for given $s,t\in 𝒯$, choose (${\widehat{a}}_0,{\widehat{b}}_1,{\widehat{b}}_2$) be the minimizer of the following equation.

$$n^{-1}\sum _{i=1}^n{N}_i^{-1}\underset{j_1\ne j_2}{\sum _{j_1=1}^{m_i}\sum _{j_2=1}^{m_i}}{\left\{e_i\left(T_{i{j}_1}\right)e_i\left(T_{i{j}_2}\right)-a_0-b_1\left(T_{i{j}_1}-s\right)-b_2\left(T_{i{j}_2}-t\right)\right\}}^2{K}_h\left(T_{i{j}_1}-s\right)K_h\left(T_{i{j}_2}-t\right).$$ (7)

Thus we estimate $R(s,t)=\mathbb{E}\left\{e\right(s\left)e\right(t\left)\right\}$ using the quantity ${\widehat{a}}_0$, viz., $\widehat{R}(s,t)={\widehat{a}}_0$. The operator $\widehat{ℱ}$ is in general positive semi-definite and the estimated eigen-values ${\widehat{\lambda }}_r$ are non-negative; indeed, $\widehat{R}$ is symmetric. In practice, $e_i\left(t\right)\prime \mathrm{s}$ are not observable and this is replaced by ${\tilde{e}}_i\left(t\right)=y_i\left(t\right)-{\mathbf{x}}_i^{\mathrm{T}}\left(t\right)\tilde{\mathit{\beta }}$ where $\tilde{\mathit{\beta }}$ is an initial estimate that is consistent to $\mathit{\beta }$ but may not be efficient. For example, one can choose the initial estimator as an ordinary least squares of $\mathit{\beta }$.

3. Asymptotic properties

In this section, we study the asymptotic properties of the proposed estimator. Let us introduce some notations. Assume that $m_i\mathrm{s}$ are all of the same order, viz, $m\equiv m\left(n\right)=n^a$ for some $a>0$. Define, $d_{n1}\left(h\right)=h^2+h\overline{m}/m$ and $d_{n2}\left(h\right)=h^4+h^3\overline{m}/m+h^2\overline{\overline{m}}/m^2$ where $\bar{m}=\text{lim}{\text{sup}}_{n\to \infty }{n}^{-1}\sum _{i=1}^{n}m/m_i$ and $\overline{\overline{m}}=\text{lim}{\text{sup}}_{n\to \infty }{n}^{-1}{\sum }_{i=1}^n{\left(m/m_i\right)}^2$. Denote ${\delta }_{n1}\left(h\right)={\left\{d_{n1}\left(h\right)\text{log}n/\left(n{h}^2\right)\right\}}^{1/2}$ and ${\delta }_{n2}\left(h\right)={\left\{d_{n2}\left(h\right)\text{log}n/\left(n{h}^4\right)\right\}}^{1/2}$. Further, $v_{a,b}=\int t^a{K}^b\left(t\right)dt$. Define, $\mathbf{W}={\left(\mathit{\varphi }{\left(t_1\right)}^{\mathrm{T}},\dots ,\mathit{\varphi }{\left(t_m\right)}^{\mathrm{T}}\right)}^{\mathrm{T}}$ is a matrix of order $m\times {\kappa }_0$ obtaining after stacking all ${\mathit{\varphi }}_k\mathrm{S}$ and random components ${\xi }_i=$ ${\left({\xi }_{i1},\dots ,{\xi }_{i{k}_0}\right)}^{\mathrm{T}}$. Further, $\mathit{\xi }$ has mean zero and variance $\mathbf{\Lambda }$ which is a diagonal matrix with components ${\lambda }_1,\dots ,{\lambda }_{{\kappa }_0}$. The sign “≲” indicates that the left-hand side of the inequality is bounded by the right-hand up to a multiplicative positive constant, i.e. for two sequence of positive real numbers $b_{n1}$ and $b_{n2}$ we define for large $n,b_{n1}\lesssim b_{n2}$ as $b_{n1}\le C{b}_{n2}$ where $C$ is a positive constant not involving $n$. The following conditions are needed for further discussion of the asymptotic properties:

• (C1) Kernel function $K(\cdot )$ is a symmetric density function defined on bounded support [−1, 1].

• (C2) Density function $f_T$ of $T$ is bounded above and away from infinity. Also, the density function is bounded below away from zero. Moreover, $f$ is differentiable and the derivative is continuous.

• (C3) $R(s,t)$ is twice differentiable, and all second-order partial derivatives are bounded on [0, 1]².

• (C4) $\mathbb{E}\left\{{\text{sup}}_{t\in \left[\mathrm{0,1}\right]}\left|e\right(t){|}^{\gamma }\right\}<\infty$ and $\mathbb{E}\left\{{\text{sup}}_{t\in \left[\mathrm{0,1}\right]}{\left|{\mathbf{x}}_i\left(t\right)\right|}^{\gamma }\right\}<\infty$ for some $\gamma \in (4,\infty )$.

• (C5) $h\to 0$ as $n\to \infty$ such that $d_{n1}^{-1}(\text{log}n/n{)}^{1-2/\gamma }\to 0$ and $d_{n2}^{-1}(\text{log}n/n{)}^{1-4/\gamma }\to 0$ for $\gamma \in (4,\infty )$.

• (C6) Condition for eigen-components:

  1. for each $1\le k<r<\infty ,{\lambda }_k/\left|{\lambda }_k-{\lambda }_r\right|\le C_{0}r/|k-r|$ for non-zero finite generic constant $C_0$.
  2. For some $\alpha >0$, with the condition $V_r{\lambda }_r^{-2}{r}^{1+\alpha }\to 0$ as $r\to \infty$ where $V_r=\mathbb{E}{\left\{\int \dot{\mu }\left(t\right){\varphi }_r\left(t\right)dt\right\}}^2$.
     The above two conditions hold if ${\lambda }_r=r^{-{\tau }_1}\Lambda \left(r\right)$ and $V_r=r^{-{\tau }_2}\Gamma \left(r\right)$ for slowly varying functions $\Lambda$ and $\Gamma$ where ${\tau }_2>1+2{\tau }_1>3$ and $\tau =\alpha +{\tau }_1$.
  3. $\int {\varphi }_k^4\left(t\right)dt$ and $\int {\dot{\mu }}^2\left(t\right){\varphi }_k^2\left(t\right)dt$ are finite for all $k\ge 1$.

• (C7) $\widehat{\mathbf{C}}\left({\mathit{\beta }}_0\right)$ converges almost surely to an invertible matrix ${\mathbf{C}}_0=\mathbb{E}\left\{\mathbf{g}\left({\mathit{\beta }}_0\right)\mathbf{g}{\left({\mathit{\beta }}_0\right)}^{\mathrm{T}}\right\}$ where $\mathbf{g}\left({\mathit{\beta }}_0\right)={\left({\mathbf{g}}^{\left(1\right)\mathrm{T}}\left({\mathit{\beta }}_0\right),\dots ,{\mathbf{g}}^{\left({\kappa }_0\right)\mathrm{T}}\left({\mathit{\beta }}_0\right)\right)}^{\mathrm{T}}$, where ${\mathit{\beta }}_0$ is the true value of $\mathit{\beta }$.

• (C8) Conditions for $h$ and ${\kappa }_0$.

  1. If $a>1/4,{\kappa }_0=O\left(n^{1/(3-\tau )}\right)$ and $n^{-1/4}\lesssim h\lesssim n^{-(a+1)/5}$.
  2. If $0<a\le 1/4,{\kappa }_0=O\left(n^{4(1+a)/5(3-\tau )}\right)$ and $h\lesssim n^{-1/4}$.

Remark 1. Condition (C1) is commonly used in non-parametric regression. The bound condition for the density function of time-points has the standard Condition (C2) for random design. Similar results can be obtained for a fixed design where the grid-points are pre-fixed according to the design density ${\int }_0^{T_j}f\left(t\right)dt=j/m$ for $j\in \{1,\dots ,m\}$, for $m\ge 1$. Furthermore, it is important to note that this approach does not involve the requirement to obtain sample path differentiation when we invoke the estimation of eigenfunctions from Li and Hsing [21]. Therefore the method could be applicable for Brownian motion which has a continuously non-differentiable sample path. To expand in Taylor series, Condition (C3) is required, and is also common in non-parametric regression. Condition (C4) is required for a uniform bound for certain higher-order expectations to show uniform convergence. This is a similar condition adopted from Li and Hsing [21]. Smoothness conditions in (C5) and (C8) is common in kernel smoothing and functional data to control bias and variance. The first condition for tuning the parameters mentioned in (C5) is similar to Li and Hsing [21]. The required spacing assumptions for eigen-values in Conditions (C6)a and (C6)b are similar as in Hall and Hosseini-Nasab [15]. Condition (C6)c is a trivial assumption that frequently arises in functional data analysis literature. In most of the situations, this condition automatically holds. Using the weak law of large numbers, Condition (C7) holds for large $n$. Similar kind of conditions can be invoked such as convexity assumption, i.e., ${\lambda }_r-{\lambda }_{r+1}\le {\lambda }_{r-1}-{\lambda }_r$ for all $r\ge 2$. Condition (C8) is determined to control the rate of the number of repeated measurements.

We present the following theorem to provide the asymptotic expansion and consistency of the proposed estimator for $\widehat{\mathit{\beta }}$.

Theorem 1. Let ${\mathit{\beta }}_0$ be the true value of $\mathit{\beta }$. Under the Conditions (C1)-(C6), for $k\in \left\{1,\dots ,{\kappa }_0\right\}$, we have the asymptotic mean square error for ${\overline{\mathbf{g}}}^{\left(k\right)}\left({\mathit{\beta }}_0\right)$ (see Equation (6)) as

$$\mathit{\text{AMSE}}\left\{{\overline{\mathbf{g}}}^{\left(k\right)}\left({\mathit{\beta }}_0\right)\right\}=O\left\{n^{-1}+n^{-1}{\kappa }_0^{3-\tau }{R}_n\left(h\right)\right\},\mathit{\text{almost}}\mathit{\text{surely}},$$ (8)

where $R_n\left(h\right)=\left\{h^4+(1/n)+(1/nmh)+\left(1/n^2{m}^2{h}^2\right)+\left(1/n^2{m}^4{h}^4\right)+\left(1/n^{2}mh\right)+\left(1/n^2{m}^3{h}^3\right)\right\}$. Moreover, under Condition (C8), AMSE $\left\{{\overline{\mathbf{g}}}^{\left(k\right)}\left({\mathit{\beta }}_0\right)\right\}=O\left(n^{-1}\right)$. Therefore, if in addition, Condition (C7) holds, as $n\to \infty ,‖\widehat{\mathit{\beta }}-{\mathit{\beta }}_0‖=O\left(n^{-1/2}\right)$ in probability.

The following theorem states the results of asymptotic normality.

Theorem 2. Define $C_i=\sum _{k_1=1}^{{\kappa }_0}\sum _{k_2=1}^{{\kappa }_0}{\mathbf{\Phi }}_{k_1}{\mathbf{X}}_i{\mathbf{C}}_{i,k_1,k_2}^{-1}{\mathbf{X}}_i^T{\mathbf{\Phi }}_{k_2}$, where ${\mathbf{C}}_{i,k_1,k_2}^{-1}$ is the $\left(k_1,k_2\right)$ block of ${\mathbf{C}}_{i,0}^{-1}$ with ${\mathbf{C}}_{i,0}=\mathbb{E}\left\{{\mathbf{g}}_i\left({\mathit{\beta }}_0\right){\mathbf{g}}_i^T\left({\mathit{\beta }}_0\right)\right\}$. Assume that the conditions for Theorem 1 holds. Then $\sqrt{n}\left(\widehat{\mathit{\beta }}-{\mathit{\beta }}_0\right)\stackrel{d}{\to }N(0,\mathbf{\Sigma })$, where $\mathbf{\Sigma }={\mathbf{B}}^{-1}{\mathbf{A}\mathbf{B}}^{-1},\mathbf{A}=\underset{n\to \infty }{\text{lim}}{n}^{-1}\sum _{i=1}^n\mathbb{E}\left\{{\mathbf{X}}_i^T{𝒞}_i{\mathbf{e}}_i{\mathbf{e}}_i^T{𝒞}_i{\mathbf{X}}_i\right\}$ and $\mathbf{B}=\underset{n\to \infty }{\text{lim}}{n}^{-1}\sum _{i=1}^n\mathbb{E}\left\{{\mathbf{X}}_i^T{𝒞}_i{\mathbf{X}}_i\right\}$.

Remark 2. Here, the selection of the bandwidth only affects the second-order term of the MSE of $\widehat{\mathit{\beta }}$ and has no effect on the asymptotic result of normality as long as $h$ satisfies the Conditions (C5) and (C8) along with some restrictions on ${\kappa }_0$. Moreover, it is important to observe that ${\mathbf{\Phi }}_k$ is normalized by the number of repeated measurements.

Remark 3. The proposed method is applicable to both sparse and dense functional data. However, this article focuses on dense functional data as discussed in Section 1. The second part of Theorem 1 and Theorem 2 are derived based on the dense functional response and we assume $m_i$ is growing with sample size. This assumption enables us to simplify the asymptotic leading order term of the AMSE expression in Theorem 1.

Outline of the proofs of the proposed theorems is discussed in the Appendix with additional technical details.

4. Simulation studies

We conduct extensive numerical studies to compare the finite sample performance of our proposed method to that of Qu et al. [29] under different correlation structures.

4.1. Simulation set-up

Consider the normal response model

$$y_i\left(T_{ij}\right)={\mathbf{x}}_i{\left(T_{ij}\right)}^{\mathrm{T}}\mathit{\beta }+e_i\left(T_{ij}\right).$$ (9)

For $p=2$, we set coefficient vectors, $\mathit{\beta }={\left({\beta }_1,{\beta }_2\right)}^{\mathrm{T}}$, where ${\beta }_1=1$ and ${\beta }_2=0.5$. Consider the covariates $x_{ik}\left(t\right)={\chi }_{i1}^{\left(k\right)}+{\chi }_{i2}^{\left(k\right)}\sqrt{2}\text{sin}\left(\pi t\right)+{\chi }_{i3}^{\left(k\right)}\sqrt{2}\text{cos}\left(\pi t\right)$ where the coefficients ${\chi }_{i1}^{\left(k\right)}~N\left(0,{\left(2^{-0.5(k-1)}\right)}^2\right),{\chi }_{i2}^{\left(k\right)}~N\left(0,{\left(0.85\times 2^{-0.5(k-1)}\right)}^2\right),{\chi }_{i3}^{\left(k\right)}~N\left(0,{\left(0.7\times 2^{-0.5(k-1)}\right)}^2\right)$ and ${\chi }_{ij}\mathrm{s}$ are mutually independent for each trajectories $i$ and each $j$. In fixed design situations, associated observational times are fixed. Sample trajectories are observed at $m=100$ equidistant time-points $\left\{t_1,\dots ,t_m\right\}$ on [0, 1]. Set number of trajectories $n\in \left\{\mathrm{100,300,500}\right\}$. The residual process $e_i\left(t\right)$ is a smoothed function with mean zero and unknown covariance function, where each $e_i$ is distributed as $e_i=\sum _{k\ge 1}{\xi }_i{\varphi }_i$ and ${\xi }_k\mathrm{s}$ are independent normal random variables with mean zero and respective variances ${\lambda }_k$. For numerical computation, we truncate the finite series at $k=3$ in Karhunen-Loève expansions for Situations (a), (b), and (c) as described below. In Situations (d) and (e), the error process is generated from given covariance functions.

1. Brownian motion. The covariance function for the Brownian motion is $\text{min}(s,t),{\lambda }_k=4/{\pi }^2(2k-1{)}^2$ and ${\varphi }_k\left(t\right)=\sqrt{2}\text{sin}\left(t/\sqrt{{\lambda }_k}\right)$.

2. Linear process. Consider the eigen-values be ${\lambda }_k=k^{-2l_0}$ and ${\varphi }_k\left(t\right)=\sqrt{2}\text{cos}\left(k\pi t\right)$. We fix $l_0\in \{\mathrm{1,2},3\}$.

3. Ornstein Uhlenbeck (OU) process. For positive constants ${\mu }_0$ and ${\rho }_0$, we have a stochastic differential equation for $e\left(t\right)$ as $\partial e\left(t\right)=-{\mu }_{0}e\left(t\right)\partial t+{\rho }_0\partial w\left(t\right)$ for the Brownian motion $w\left(t\right)$. It can be shown that $\text{cov}\left\{e\left(t\right),e\left(s\right)\right\}=c\text{exp}\left\{-{\mu }_0|t-s|\right\}$ where $c={\rho }_0^2/2{\mu }_0$. Here we assume $c=1$. Thus, by solving the integral equation we have ${\varphi }_k\left(t\right)=A_k\text{cos}\left({\omega }_{k}t\right)+B_k\text{sin}\left({\omega }_{k}t\right)$ and ${\lambda }_k=2{\mu }_0/\left({\omega }_k^2+{\mu }_0^2\right)$ where $\omega$ is solution of $\text{cot}\left(\omega \right)=\left({\omega }^2-{\mu }_0^2\right)/2{\mu }_0\omega$. The constants $A_k$ and $B_k$ are defined as $B_k={\mu }_0{A}_k/{\omega }_k$ where $A_k=\sqrt{2{\omega }_k^2/\left(2{\mu }_0+{\mu }_0^2+{\omega }_k^2\right)}$. Here ${\mu }_0$ is chosen to be 1 or 3.

4. Power exponential. $R(s,t)=\text{exp}\left\{{\left(-|s-t|/a_0\right)}^{b_0}\right\}$ where scale parameter $a_0=1$ and shape parameter $b_0\in \{\mathrm{1,2},5\}$.

5. Rational quadratic. $R(s,t)={\left\{1+(s-t{)}^2/a_0^2\right\}}^{-b_0}$ where scale parameter $a_0=1$ and shape parameter $b_0\in \{\mathrm{1,2},5\}$.

4.2. Comparison and evaluation

For each of the situations, we perform 500 simulation replicates. To execute Qu et al. [29]’s approach, we construct the scores using basis matrices as described in Example 1 (approximation of the compound symmetric correlation structure, denoted as ldaCS in the tables) and Example 2 (for the first-order autoregressive correlation structure, denoted as ldaAR in the tables) in their paper. Ordinary least squares estimate (init) is taken as the initial estimate of $\mathit{\beta }$ for both ours and Qu et al. [29]’s approach. We indicate $\mathtt{\text{fda}}-{\kappa }_0$ as our proposed method with the number of basis functions ${\kappa }_0$. Additionally, we denote fda-AIC as our proposed method where the number of basis functions is determined by the AIC, as suggested by Yao et al. [33]. The estimation procedure in the iterative algorithm converges when the square difference between the estimated values of two consecutive steps is bounded by a small number 10⁻¹⁰ or the maximum number of steps crosses 500, whichever happens earlier. To make theoretical results and numerical examples consistent, we use FPCA function in R which is available in fdapace packages [11]. Bandwidths are selected using generalized cross-validation and the Epanechnikov kernel $K\left(x\right)=0.75{\left(1-x^2\right)}_{+}$ is used for estimation where $(a{)}_{+}=\text{max}(a,0)$.

The means and standard deviations (SD) of the regression coefficients based on 500 simulations are given as summary measures. We calculate the standard deviation mentioned in the tables based on 500 estimates from 500 replications that can be viewed as the true standard error. Moreover, we also compute absolute bias (defined as $\text{AB}=\sum _{b=1}^{500}{\widehat{\mathit{\beta }}}_b-\mathit{\beta }\mid /500$) and mean square error (defined as $\text{MSE}{\left.=\sum _{b=1}^{500}{\widehat{\mathit{\beta }}}_b-\mathit{\beta }\right)}^2/500$) to compare the performance of estimation, where for $b-\text{th}$ replication ${\widehat{\mathit{\beta }}}_b$ be the estimated value for $\mathit{\beta }$, MSEs are reported in the order of 10⁻². In the the last column of all the tables, we report the average fraction of variance explained (FVE) under different values of ${\kappa }_0$. Since our objective is to see the performance of the proposed method, we compare the results for different choices of ${\kappa }_0$. However, in practice, the number of selected eigen-functions plays a critical role in the proposed method. We can choose ${\kappa }_0$ based on a scree plot where we plot the number of the component against the FVE. The elbow of the graph is found and the components to the left are considered as significant. Another possibility is to use AIC to select the number of eigen-functions as we have demonstrated in the numerical results.

4.3. Simulation results

Simulation results associated with the Brownian motion are shown in Table 1. In this situation, we observe that our approach produces better results in terms of the dispersion measures. Tables S1, S2 and S3 of supplementary material show results for linear processes, our proposed method performs better in situations with a working correlation matrix as AR but is comparable for exchangeable structure for $l_0=1,2,3$. Moreover, in our proposed method, as $l$ increases all dispersion measures such as MSE decrease. Results based on the OU process are documented in Tables S4 and S5 in supplementary material. Our method outperforms the existing methods for both situations and as ${\mu }_0$ increases, the MSE decreases. For three different parameter choices of the power exponential and rational quadratic covariance structure, the numerical results are presented in Tables S6, S7, S8 and S9, S10, S11 in the supplementary material, respectively. As before, we observe that our proposed method is finer than the existing ones in all sub-cases; but interestingly, when $b_0$ increases, the MSE decreases for the power exponential, whereas it increases for the rational quadratic covariance structure as expected due to the covariance structure. Overall, we observe that for all the above situations, as sample size increases, the dispersion measures, for example, SD and MSE decrease. It establishes that as sample size increases, the parameter estimates get closer and closer to the true parameters. In each of the above situations, the SDs for the proposed methods decrease as we increase ${\kappa }_0$ and stabilize after some value of ${\kappa }_0$ where the fraction of variance (FVE) is approximately 100%. In most cases, the estimated ${\kappa }_0$ is close to 3 using AIC approach. However, in the specific scenario described in (d) with $a_0=b_0=1$, the median ${\kappa }_0$ selected by AIC is 1, 19 and 20 for $n=100,300,500$ respectively. We observe that the proposed method, which selects the number of eigenvalues using the AIC approach, performs better than the existing methods by Qu et al. [29]. Additionally, the results selected by AIC are generally consistent with those chosen by the scree plot criteria of FVE when FVE is sufficiently large.

Table 1:

Performance of the estimation procedure where the residuals are generated from Brownian motion. Mean of the estimated coefficients, standard deviation, absolute bias, mean square error (×100), and FVE in percentage are summarized. The results are based on: (1) ordinary least squares (init); (2) Qu et al. [29]’s approach with first-order autoregressive (ldaAR), and compound symmetric (ldaCS) correlation structures; (3) our proposed approach ($\mathtt{\text{fda}}-{\kappa }_0$), where ${\kappa }_0$ denotes the number of basis functions, and fda-AIC with AIC-selected ${\kappa }_0$.

	${\beta }_1$				${\beta }_2$
Method	Mean	SD	AB	MSE	Mean	SD	AB	MSE	FVE %-age
n = 100
init	0.9999	0.0373	0.0297	0.1391	0.4995	0.0486	0.0384	0.2354
ldaAR	0.9991	0.0331	0.0265	0.1095	0.5004	0.0445	0.0353	0.1972
ldaCS	0.9987	0.0316	0.0253	0.1000	0.4997	0.0411	0.0322	0.1685
fda-1	0.9998	0.0564	0.0447	0.3180	0.5006	0.0743	0.0587	0.5516	86.2672
fda-2	1.0001	0.0269	0.0213	0.0723	0.4971	0.0362	0.0290	0.1314	96.3746
fda-3	0.9998	0.0231	0.0181	0.0532	0.4978	0.0317	0.0251	0.1010	99.9220
fda-4	1.0004	0.0052	0.0014	0.0028	0.4994	0.0092	0.0022	0.0085	99.9979
fda-5	0.9999	0.0021	0.0008	0.0004	0.4999	0.0051	0.0012	0.0026	100.0000
fda-AIC	0.9998	0.0231	0.0181	0.0532	0.4978	0.0317	0.0251	0.1010	99.9220
n = 300
init	1.0002	0.0200	0.0162	0.0401	0.5003	0.0288	0.0226	0.0825
ldaAR	1.0003	0.0184	0.0147	0.0336	0.5000	0.0259	0.0203	0.0670
ldaCS	1.0007	0.0170	0.0134	0.0288	0.4995	0.0242	0.0190	0.0583
fda-1	1.0002	0.0309	0.0251	0.0955	0.5008	0.0443	0.0350	0.1962	86.7578
fda-2	1.0002	0.0142	0.0114	0.0202	0.4998	0.0213	0.0169	0.0451	96.4747
fda-3	1.0002	0.0122	0.0098	0.0150	0.4992	0.0179	0.0144	0.0321	99.9745
fda-4	1.0002	0.0021	0.0003	0.0004	0.4998	0.0032	0.0005	0.0010	99.9993
fda-5	1.0000	0.0002	0.0001	0.0000	0.5000	0.0003	0.0002	0.0000	100.0000
fda-AIC	1.0002	0.0122	0.0098	0.0150	0.4992	0.0179	0.0144	0.0321	99.9745
n = 500
init	1.0002	0.0148	0.0117	0.0219	0.5006	0.0223	0.0177	0.0497
ldaAR	1.0005	0.0138	0.0111	0.0189	0.5000	0.0206	0.0162	0.0422
ldaCS	1.0000	0.0128	0.0102	0.0164	0.4992	0.0184	0.0146	0.0340
fda-1	1.0007	0.0234	0.0185	0.0545	0.5012	0.0348	0.0277	0.1213	86.7520
fda-2	0.9996	0.0105	0.0083	0.0110	0.5002	0.0157	0.0126	0.0247	96.5174
fda-3	0.9991	0.0091	0.0074	0.0084	0.4999	0.0133	0.0107	0.0177	99.9851
fda-4	1.0000	0.0002	0.0001	0.0000	0.5000	0.0003	0.0002	0.0000	99.9996
fda-5	1.0000	0.0002	0.0001	0.0000	0.5000	0.0003	0.0002	0.0000	100.0000
fda-AIC	0.9991	0.0091	0.0074	0.0084	0.4999	0.0133	0.0107	0.0177	99.9851

5. Data analysis

In this section, we apply our proposed method to motivating examples in two different data-sets.

5.1. Beijing’s PM_2.5 pollution study

In the atmosphere, suspended microscopic particles of solid and liquid matter are commonly known as particulates or particulate matter (PM). Such particulates often have a strong noxious impact toward human health, climate, and visibility. One such common and fine type of atmospheric particle is PM_2.5 with a diameter less than 2.5 micrometers. Many developed and developing cities across the world are experiencing chronic air pollution, with major pollutants being PM_2.5; Beijing and a substantial part of China are among such places. Some studies show that there are many non-ignorable sources of variability in the distribution and transmission pattern of PM_2.5, which are confounded with secondary chemical generation. The atmospheric PM_2.5 data used in our analysis were collected from the UCI machine learning repository https://archive.ics.uci.edu/ml/datasets/Beijing+Multi-Site+Air-Quality+Data [24]. The data-set includes daily measurements of PM_2.5 and associated covariates at twelve different locations in China, namely Aotizhongxin, Changping, Dingling, Dongsi, Guanyuan, Gucheng, Huairou, Nongzhanguan, Shunyi, Tiantan, Wanliu, and Wanshouxigong during January 2017. After excluding missing data, there were 608 hourly data-points in Beijing2017-data. We assume that the atmospheric measurements are independent since they are located quite apart. The objective of our analysis is to describe the trend of functional response PM_2.5 (as shown in Fig 1 (left)) and evaluate the effect of covariates including the chemical compounds such as sulfur dioxide (SO₂), nitrogen dioxide (NO₂), carbon monoxide (CO) and ozone (O₃) over time. We smooth the covariates and responses to reduce variability and center them. Subsequently, we consider the following model

$$Y_i\left(t\right)={\beta }_0+{\text{SO}}_2\left(t\right){\beta }_1+{\text{NO}}_2\left(t\right){\beta }_2+\text{CO}\left(t\right){\beta }_3+{\mathrm{O}}_3\left(t\right){\beta }_4+e_i\left(t\right).$$ (10)

We use Algorithm 1 to estimate the coefficients of the regression model mentioned above. Through the simulation results, we observe that if the values of ${\kappa }_0$ increase, the standard deviation of the coefficients decreases. For small FVE such as 50%, the corresponding ${\kappa }_0=1$ and the estimation procedure performs poorly; whereas for large FVE percentages, the estimation procedure has adequately improved in terms of standard error. The estimated values for ${\beta }_0,{\beta }_1,{\beta }_2,{\beta }_3$, and ${\beta }_4$ produce similar results across different choices of ${\kappa }_0$. From the estimated standard error, using scree-plot of standard error, we conclude that the suitable choice of ${\kappa }_0$ is approximately 10. The estimated scaled eigen-values are provided in Figure 1 (right) which clearly shows their decay rate. The estimated coefficients with standard error are 0.0009 (1.1644), 0.0829 (0.2584), 0.9503 (0.1586), 0.0196 (0.0037) and 1.1523 (0.1198), respectively.

5.2. DTI study for sleep apnea patients

The diffusion tensor imaging data (DTI) to understand the white matter structural alteration using diffusion tensor imaging (DTI) are used to illustrate the application of the proposed method and its estimation procedure [32]. MRI is a powerful technique for investigating the structural and functional changes in the brain during pathological and neuro-psychological processes. Due to the advancement in diffusion tensor imaging (DTI), several studies on white matter alterations associated with clinical variables can be found in recent literature. For our analysis, we use Apnea-data obtained from one such study on obstructive sleep apnea (OSA) patients [32]. The data consists of 29 male patients between the ages 30–55 years who underwent a study for the diagnosis of continuous positive airway pressure (CPAP) therapy. Among those who have sleep disorders other than OSA, night-shift workers, patients with psychiatric disorders, hypertension, diabetes, and other neurological disorders were excluded. In this study, the psychomotor vigilance task (PVT) was performed in which a light was randomly switched-on on a screen for several seconds in a certain interval of time, and subjects were asked to press a button as soon as they saw the light appear on screen; such an experiment provides a numerical measure of sleepiness by counting the number of “lapses” for each individual. Psychomotor vigilance task was performed after continuous positive airway pressure treatment which counts the number of lapses in the attention tracking test. In such a test, subjects were asked to press a button as soon as the light appeared on the screen which was turned on randomly for several seconds in a certain interval of time. The interesting measure is sustained attention and provides a numerical measure of sleepiness by counting the number of “lapses” for each individual

DTI was performed at 3T MRI scanner using a commercial 32-channel head coil, followed by the analysis using tract-based spatial statistics to investigate the difference in fractional anisotropy (FA) and other DTI parameters. The image acquisition is as follows. An axial T1-weighted image of the brain (3D-BRAVO) is collected with repetition slice thicknesstime (TR) = 12ms, echo time (TE) = 5,2ms, flip angle = 13°, inversion time = 450 ms, matrix = 384 × 256, voxel size = 1.2 × 0.57 × 0.69mm and scan time = 2 min 54 sec. DTI are obtained in the axial plane using a spin-echo echo planner imaging sequence with TR = 4500ms, TE = 89.4ms, field of view = 20 × 20cm², matrix size = 160 × 132, slice thickness = 3mm, slice spacing = 1mm, b-values = 0, 1000 s/mm².

Our objective is to investigate the structural alteration of white matter using DTI in patients with OSA over each voxel at various regions of the brain (called ROIs). Thus, our response variable is one of the DTI parameters, viz., fractional anisotropy (FA) and we are interested in studying the effect of the changes of FA over continuous domains such as voxels with the interaction of the lapses and the voxel locations in each ROIs. We consider the following model for each ROI.

$${\mathrm{F}\mathrm{A}}_i\left(s\right)={\beta }_0+{\beta }_1{\text{lapses}}_i\times s+e_i\left(s\right)$$ (11)

where $s\in 𝒮$, a set of voxels in the considered ROIs. Using the Algorithm 1, we estimate the coefficients ${\beta }_1$ and ${\beta }_2$ as mentioned in the model 11 and the results are presented in Table 2. We find that the coefficient estimates are close enough to their initial estimates and the estimated standard error is smaller for the coefficients based on the proposed method. Here ${\kappa }_0$ (i.e., the number of eigen-functions) is determined for simplicity using FVE, which is fixed at 0.99, resulting in ${\kappa }_0=7$.

Table 2:

Estimated values and associated standard errors for the regression coefficients are provided upto four decimal places based on the existing and proposed methods. First line corresponding to each ROI shows results based on initial estimates and the second line corresponds to that of proposed estimates.

		${\beta }_0$		${\beta }_1$
region	# functional points	Estimate	Std. Error (×100)	Estimate (×100)	Std. Error (×100)
ROI.6	659	0.4512	0.1343	−0.0606	0.0130
		0.4512	0.0983	−0.0605	0.0028
ROI.7	1362	0.5048	0.0628	0.0309	0.0061
		0.5050	0.0681	0.0342	0.0007
ROI.8	1370	0.5256	0.0586	−0.0667	0.0057
		0.5271	0.0346	−0.0733	0.0006
ROI.9	690	0.4951	0.0910	0.2904	0.0088
		0.5443	0.0874	0.1660	0.0014
ROI.10	699	0.4951	0.0892	0.3314	0.0086
		0.5262	0.1398	0.4231	0.0014
ROI.11	968	0.4372	0.0979	0.1323	0.0095
		0.4380	0.0637	0.1311	0.0009
ROI.12	968	0.4529	0.0948	0.0965	0.0092
		0.4664	0.0750	0.0504	0.0013
ROI.13	992	0.5448	0.1060	0.3453	0.0103
		0.5449	0.0856	0.3559	0.0011
ROI.14	992	0.5435	0.1068	0.3432	0.0104
		0.5436	0.0754	0.3437	0.0003
ROI.37	1236	0.3695	0.0779	−0.1126	0.0076
		0.3713	0.0669	−0.1175	0.0017
ROI.38	1155	0.3564	0.0819	−0.1356	0.0079
		0.3578	0.0420	−0.1356	0.0009
ROI.39	1124	0.4618	0.0760	0.1972	0.0074
		0.4621	0.0615	0.1996	0.0007
ROI.40	1125	0.4786	0.0658	0.0953	0.0064
		0.4780	0.0369	0.1016	0.0005
ROI.45	380	0.4189	0.1071	0.1647	0.0104
		0.4190	0.0175	0.1648	0.0001
ROI.46	376	0.4074	0.1033	0.1988	0.0100
		0.4074	0.0159	0.1994	0.0002
ROI.47	596	0.4596	0.0932	0.1304	0.0090
		0.4594	0.0191	0.1349	0.0001
ROI.48	600	0.4045	0.0868	0.1100	0.0084
		0.4036	0.0644	0.1067	0.0006

6. Discussion

In this article, we propose an estimation procedure for the constant linear effects model, which is commonly used in statistics [36] especially in spatial modeling. One of the key factors of this estimation procedure is the fact that it is based on the quadratic inference methodology that has served a huge role in the analysis of correlated data since it was discovered by Qu et al. [29]. In contrast with the existing method, our approach allows the number of repeated measurements to grow with sample size; therefore, the trajectories of individuals can be observed on a dense grid of a continuum domain. Instead of assuming a working correlation structure, we propose a data-driven way by estimating the eigen-functions that are obtained by functional principal component analysis. Here, we achieve $\sqrt{n}$-consistency of the parametric estimates in the regression model, even though the eigen-functions are estimated non-parametrically.

Additionally, our method is easy to implement in a wide range of applications. The applicability of the proposed method is illustrated by extensive simulation studies. Moreover, two real-data applications in different scientific domains are provided which confirm the efficacy of the proposed method.

Appendix A. Some preliminary definitions and concepts of operators

Consider the standard ${ℒ}^2\left[\mathrm{0,1}\right]$ space that defines the set of square-integrable functions defined on the closed set [0, 1] that takes values on the real line. The space ${ℒ}^2\left[\mathrm{0,1}\right]$ is equipped with an inner product and is defined as $⟨f,g⟩={\int }_0^{1}f\left(t\right)g\left(t\right)dt$ for $f$ and $g$ in that space, and forms a Hilbert space. Moreover, we denote the norm $\Vert \cdot {\Vert }_2$ in ${ℒ}^2$ which is defined as $\Vert f{\Vert }_2={\left\{\int f^2\left(u\right)du\right\}}^{1/2}$. Define $ℱ$ be an operator that assigns an element $f$ in ${ℒ}^2\left[\mathrm{0,1}\right]$ to a new element $ℱf$ in ${ℒ}^2\left[\mathrm{0,1}\right]$ moreover, $ℱ$ is linear and bounded. A linear mapping $ℱf(\cdot )=\int R(\cdot ,u)f\left(u\right)du$ for any function $f\in {ℒ}^2\left[\mathrm{0,1}\right]$ and for some integrable function $R(\cdot ,\cdot )$ on [0, 1] × [0, 1]. This function is preferably known as integral operator and the bivariate function $R$ is known as a kernel in statistics and functional analysis literature. Furthermore, under the assumption that $\iint R^2(u,v)\mathit{\text{dudv}}<\infty$. It is easy to see that $ℱf(\cdot )$ is uniformly continuous and compact for a non-negative definite symmetric kernel $R$. For some $\lambda$, in Fredholm integral equation, $ℱ\varphi =\lambda \varphi$ has non-zero solution $\varphi$ then we call $\lambda$ as eigen-value of $ℱ$ and the solution of the eigen-equation is called eigen-functions, altogether, the pair of eigen-values and eigen-function, viz., $(\lambda ,\varphi )$ are called eigen-elements. Due to non-negative definiteness of $ℱ$, the eigen-values are ordered as ${\lambda }_1\ge {\lambda }_2\ge \dots \ge 0$.

Suppose for self-adjoint compact operator on Hilbert space $\mathscr{H}$ consider two operators $ℱ$ and $𝒢$, define perturbation operator $\Delta =𝒢-ℱ$ such that $𝒢=ℱ+\Delta$ where $𝒢$ is an approximation to $ℱ$ where $\Delta$ amount of error is occurred. Let $ℱ$ and $𝒢$ have kernels $F$ and $G$ respectively with eigen-elements $\left({\theta }_r,{\psi }_r\right)$ and $\left({\lambda }_r,{\varphi }_r\right)$. For simplicity, we assume that the eigen-values are distinct. Then the following Lemma provides perturbation of the eigen-functions.

Lemma 1 (Theorem 5.1.8 in Hsing and Eubank [18]). Let $(\lambda ,\varphi )$ be the eigen-components of $ℱ$ and $(\theta ,\psi )$ be that of $𝒢$ with multiplicity of all eigen-values are restricted to be 1. Define ${{\eta }_k=\text{min}}_{r\ne k}\left|{\lambda }_r-{\lambda }_k\right|$. Assume $⟨{\varphi }_r,{\psi }_r⟩\ge 0$ and ${\eta }_k>0$. Then

$${\psi }_k-{\varphi }_k=\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{\left({\theta }_k-{\lambda }_r\right)}^{-1}{𝒫}_r\Delta {\psi }_k+{𝒫}_k\left({\psi }_k-{\varphi }_k\right).$$ (A.1)

The above equation follows

$${\psi }_k-{\varphi }_k=\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{\left({\theta }_k-{\lambda }_r\right)}^{-1}{𝒫}_r\Delta {\psi }_k+O\left(\Vert \Delta {\Vert }^2\right).$$ (A.2)

Remark 4. Equation (A.2) plays an important role in finding the bound of the proposed estimator introduced in Section 2.2. Note that ${\text{sup}}_{r\ge 1}\left|{\theta }_r-{\lambda }_r\right|\le \Vert \Delta \Vert \le {\text{inf}}_{r\ne k}\left|{\lambda }_k-{\lambda }_r\right|$ (see Theorem 4.2.8 in Hsing and Eubank [18] for proof). Thus, it is easy to see, $\left|{\theta }_r-{\lambda }_r\right|\le \left|{\lambda }_k-{\lambda }_r\right|$ which implies from Equation (A.1)

$$\begin{gathered} {\psi }_k-{\varphi }_k=\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{\left({\lambda }_k-{\lambda }_r\right)}^{-1}\sum _{s=0}^{\infty }{\left\{\left({\lambda }_k-{\theta }_r\right)/\left({\lambda }_k-{\lambda }_r\right)\right\}}^s{𝒫}_r\Delta \left\{{\varphi }_k+\left({\psi }_k-{\varphi }_k\right)\right\}+{𝒫}_k\left({\psi }_k-{\varphi }_k\right) \\ =\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{\left({\lambda }_k-{\lambda }_r\right)}^{-1}{𝒫}_r\Delta {\varphi }_k+\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{\left({\lambda }_k-{\lambda }_r\right)}^{-1}{𝒫}_r\Delta \left({\psi }_k-{\varphi }_k\right)+\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }\sum _{s=1}^{\infty }\left\{{\left({\lambda }_k-{\lambda }_s\right)}^s/{\left({\lambda }_k-{\lambda }_r\right)}^{s+1}\right\}{𝒫}_r\Delta {\psi }_k+{𝒫}_k\left({\psi }_k-{\varphi }_k\right). \end{gathered}$$ (A.3)

Moreover, using Bessel’s inequality, we can bound last three terms in the above equation by $\Vert \Delta {\Vert }^2$.

Appendix B. Some useful lemmas

In this section, we present some useful lemmas. For convenience, let us recall the notation. Assume that $m_i\mathrm{s}$ are all of the same order, viz, $m\equiv m\left(n\right)$. Define, $d_{n1}\left(h\right)=h^2+h\bar{m}/m$ and $d_{n2}\left(h\right)=h^4+h^3\bar{m}/m+h^2\overline{\bar{m}}/m^2$ where ${\bar{m}=\text{lim}\text{sup}}_{n\to \infty }{n}^{-1}\sum _{i=1}^{n}m/m_i$ and ${\overline{\bar{m}}=\text{lim}\text{sup}}_{n\to \infty }{n}^{-1}\sum _{i=1}^n{\left(m/m_i\right)}^2$. Denote ${\delta }_{n1}\left(h\right)={\left\{d_{n1}\left(h\right)\text{log}n/\left(n{h}^2\right)\right\}}^{1/2},{\delta }_{n2}\left(h\right)={\left\{d_{n2}\left(h\right)\text{log}n/\left(n{h}^4\right)\right\}}^{1/2}$ and ${\bar{\delta }}_n\left(h\right)=h^2+{\delta }_{n1}\left(h\right)+{\delta }_{n2}^2\left(h\right)$. Further, $v_{a,b}=\int t^a{K}^b\left(t\right)dt$. Define, $\mathbf{W}={\left(\mathit{\varphi }{\left(t_1\right)}^{\mathrm{T}},\dots ,\mathit{\varphi }{\left(t_m\right)}^{\mathrm{T}}\right)}^{\mathrm{T}}$ be matrix of order $m\times {\kappa }_0$ obtained after stacking all ${\mathit{\varphi }}_k\mathrm{s}$ and random components ${\mathit{\xi }}_i={\left({\xi }_{i1},\dots ,{\xi }_{i{k}_0}\right)}^{\mathrm{T}}$. Further, $\mathit{\xi }$ has mean zero and variance $\mathbf{\Lambda }$ which is a diagonal matrix with components ${\lambda }_1,\dots ,{\lambda }_{k_0}$. The sign ‘≲’ indicates that for two sequence of positive real numbers $b_{n1}$ and $b_{n2}$ we define for large n, $b_{n1}\lesssim b_{n2}$ as $b_{n1}\le C{b}_{n2}$ where $C$ is a positive constant not involving $n$.

Lemma 2. Consider $Z_1,\dots ,Z_n$ be independent and identically distributed random variables with mean zero and finite variance. Suppose that there exists an $M$ such that $P\left(\left|Z_i\right|\le M\right)=1$ for all $i\in \{1,\dots ,n\}$. Let $T_n=n^{-1}\sum _{i=1}^n{Z}_i$. then, $n^{-1}\sum _{i=1}^n{Z}_i=O\left((\text{log}n/n{)}^{1/2}\right.$ almost surely. If $\sqrt{\text{Var}\left(T_n\right)}=O\left\{(\text{log}n/n{)}^{1/2}\right\}$ then $T_n=O(\text{log}n/n)$ almost surely.

Proof: Bernstein’s inequality states that if $Z_1,\dots ,Z_n$ be centered independent bounded random variables with probability 1. Let $T_n=n^{-1}\sum _{i=1}^n{Z}_i$, then let $\text{Var}\left\{T_n\right\}={\sigma }_n^2$. Then for any positive real number $u$, we have $P\left(\left|T_n\right|\ge u\right)\le \text{exp}\left\{-n{u}^2/\left(2{\sigma }_n^2+2Mu/3\right)\right\}$ where $M$ is such that $P\left(\left|Z_i\right|\le M\right)=1$. Moreover, if $T_n$ converges to its limit in probability fast enough, then it converge almost surely in the limit, i.e., if for any $u>0,\sum _{n=1}^{\infty }P\left(\left|T_n\right|\ge u\right)<\infty$ them $T_n$ converges to zero almost surely. Now, choose $u=\sqrt{4{\sigma }_n^2\text{log}n/n}+4M\text{log}n/3n$. Thus, $\sum _{n=1}^{\infty }P\left(\left|T_n\right|\ge u\right)<\sum _{n=1}^{\infty }1/n^2$ which is finite. Therefore, $T_n=O\left(u\right)$ almost surely. Now let ${\sigma }_n\le \sqrt{4M^2\text{log}n/9n}$, we have, $T_n=O(\text{log}n/n)$ and if ${\sigma }_n=O\left(1\right)$ then $T_n=O\left((\text{log}n/n{)}^{1/2}\right)$ almost surely. □

Lemma 3. Suppose $T_{ij}$ are i.i.d. with density $f_T$. Then for fixed $i\in \{1,\dots ,n\}$, any $k$ and $l\ge 1$, under assumptions (C2) and (C6)c, $m_i^{-1}\sum _{j=1}^{m_i}{\varphi }_k\left(T_{ij}\right){\varphi }_l\left(T_{ij}\right)=1(k=l)+O\left({\left(\text{log}{m}_i/m_i\right)}^{1/2}\right)$ almost surely where 1 is the indicator function.

Proof: Observe, $\mathbb{E}\left\{m_i^{-1}\sum _{j=1}^{m_i}{\varphi }_k\left(T_{ij}\right){\varphi }_l\left(T_{ij}\right)\right\}=\int {\varphi }_k\left(t\right){\varphi }_l\left(t\right)dt=1(k=l)$ and,

$$\begin{gathered} \text{Var}\left\{m_i^{-1}\sum _{j=1}^{m_i}{\varphi }_k\left(T_{ij}\right){\varphi }_l\left(T_{ij}\right)\right\}=\mathbb{E}{\left\{m_i^{-1}\sum _{j=1}^{m_i}{\varphi }_k\left(T_{ij}\right){\varphi }_l\left(T_{ij}\right)\right\}}^2-1(k=1) \\ =m_i^{-2}\sum _{j=1}^{m_i}\mathbb{E}\left\{{\varphi }_k^2\left(T_{ij}\right){\varphi }_l^2\left(T_{ij}\right)\right\}+m_i^{-2}\underset{j_1\ne j_2}{\sum _{j_1=1}^{m_i}\sum _{j_2=1}^{m_i}}\mathbb{E}\left\{{\varphi }_k\left(T_{i{j}_1}\right){\varphi }_k\left(T_{i{j}_2}\right){\varphi }_l\left(T_{i{j}_1}\right){\varphi }_l\left(T_{i{j}_2}\right)\right\}-\mathbf{1}(k=l) \\ =m_i^{-2}\sum _{j=1}^{m_i}\mathbb{E}\left\{{\varphi }_k^2\left(T_{ij}\right){\varphi }_l^2\left(T_{ij}\right)\right\}+m_i^{-2}\underset{j_1\ne j_2}{\sum _{j_1=1}^{m_i}\sum _{j_2=1}^{m_i}}\mathbb{E}\left\{{\varphi }_k\left(T_{i{j}_1}\right){\varphi }_l\left(T_{i{j}_1}\right)\right\}\mathbb{E}\left\{{\varphi }_k\left(T_{i{j}_2}\right){\varphi }_l\left(T_{i{j}_2}\right)\right\}-\mathbf{1}(k=l) \\ =\left\{\begin{array}{ll}{m}_i^{-1}\int {\varphi }_k^4(t)dt+\left(m_i-1\right)/m_i{\left(\int {\varphi }_k^2(t)dt\right)}^2-1\hfill & \text{if}k=l\hfill \\ \left.m_i^{-1}\int {\varphi }_k^2(t){\varphi }_l^2(t)dt+\left(m_i-1\right)/m_i\left(\iint {\varphi }_k(t){\varphi }_l(t)\right){\varphi }_k\left(t^{\prime }\right){\varphi }_l\left(t^{\prime }\right)dtd{t}^{\prime }\right)\hfill & \text{if}k\ne l\hfill \end{array}\right. \\ =O\left(1/m_i\right). \end{gathered}$$ (B.1)

Therefore, applying the Lemma 2, the result is immediate. □

Lemma 4. Suppose $T_{ij}$ are i.i.d with density $f_T$. Then for fixed $i\in \{1,\dots ,n\}$, for any $k\ge 1$, under assumptions (C2), (C6)c, $m_i^{-1}\sum _{j=1}^{m_i}{\dot{\mu }}_i\left(T_{ij}\right){\varphi }_k\left(T_{ij}\right)=\int {\dot{\mu }}_i\left(t\right){\varphi }_k\left(t\right)dt+O\left({\left(\text{log}{m}_i/m_i\right)}^{1/2}\right)$ almost surely.

Proof: Observe, $\mathbb{E}\left\{m_i^{-1}\sum _{j=1}^{m_i}{\dot{\mu }}_i\left(T_{ij}\right){\varphi }_k\left(T_{ij}\right)\right\}=\int {\dot{\mu }}_i\left(t\right){\varphi }_k\left(t\right)dt$ and,

$$\begin{gathered} \text{Var}\left\{m_i^{-1}\sum _{j=1}^{m_i}{\dot{\mu }}_i\left(T_{ij}\right){\varphi }_k\left(T_{ij}\right)\right\}\le \mathbb{E}{\left\{m_i^{-1}\sum _{j=1}^{m_i}{\dot{\mu }}_i\left(T_{ij}\right){\varphi }_k\left(T_{ij}\right)\right\}}^2 \\ =m_i^{-2}\sum _{j=1}^{m_i}\mathbb{E}\left\{{\dot{\mu }}_i^2\left(T_{ij}\right){\varphi }_k{\left(T_{ij}\right)}^2\right\}+m_i^{-2}\underset{j_1\ne j_2}{\sum _{j_1=1}^{m_i}\sum _{j_2=1}^{m_i}}\mathbb{E}\left\{{\dot{\mu }}_i\left(T_{i{j}_1}\right){\dot{\mu }}_i\left(T_{i{j}_2}\right){\varphi }_k\left(T_{i{j}_1}\right){\varphi }_k\left(T_{i{j}_2}\right)\right\}=O\left(1/m_i\right), \end{gathered}$$ (B.2)

since $\int {\dot{\mu }}^2\left(t\right){\varphi }_k^2\left(t\right)dt<\infty$. Therefore, applying the Lemma 2, the result is immediate. □

Lemma 5. Define, ${ℳ}_{ir}=\int {\dot{\mu }}_i\left(t\right){\varphi }_r\left(t\right)dt$ and $V_r=\mathbb{E}{\left\{\int \dot{\mu }\left(t\right){\varphi }_r\left(t\right)dt\right\}}^2$ for $r\ge 1$. Then under Conditions (C6)a and (C6)b, for some $\alpha >0$ such that $V_r{\lambda }_r^{-2}{r}^{1+\alpha }\to 0$ as $r\to \infty$ (due to Condition (C6)b, $\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{\left({\lambda }_k-{\lambda }_r\right)}^{-1}{n}^{-1}\sum _{i=1}^n{ℳ}_{ir}{\xi }_{ik}=O\left\{(\text{log}n/n{)}^{1/2}{\lambda }_k^{1/2}{k}^{(1-\alpha )/2}\right\}$ almost surely.

Proof: It is easy to see that, $\mathbb{E}\left\{\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{\left({\lambda }_k-{\lambda }_r\right)}^{-1}{n}^{-1}\sum _{i=1}^n{ℳ}_{ir}{\xi }_{ik}\right\}=0$. Using the spacing condition among the eigen-values in (C6)a, for each $1\le k<r<\infty$ and for nonzero finite generic constant $C_0$,

$${\lambda }_k/\left|{\lambda }_k-{\lambda }_r\right|\le C_{0}r/|k-r|.$$ (B.3)

Similar kind of conditions can be invoked such as the convexity assumption, i.e. ${\lambda }_r-{\lambda }_{r+1}\le {\lambda }_{r-1}-{\lambda }_r$ for all $r\ge 2$. Thus, using Inequality (B.3), for some $\alpha >0$, with condition $V_r{\lambda }_r^{-2}{r}^{1+\alpha }\to 0$ as $r\to \infty$, we can write

$$\begin{gathered} \sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{V}_r{\left({\lambda }_k-{\lambda }_r\right)}^{-2} \\ \lesssim \sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{V}_r{\left\{\text{max}(k,r)/|k-r|\text{max}\left({\lambda }_k,{\lambda }_r\right)\right\}}^2 \\ =\sum _{r\le k/2}{V}_r{\lambda }_r^{-2}{k}^2/(k-r{)}^2+\sum _{r>2k}{V}_r{\lambda }_k^{-2}{r}^2/(k-r{)}^2+\sum _{k/2<r<k}{V}_r{\lambda }_r^{-2}{k}^2/(k-r{)}^2+\sum _{k<r<2k}{V}_r{\lambda }_k^{-2}{r}^2/(k-r{)}^2 \\ \lesssim \sum _{r\le k/2,r>2k}{V}_r{\lambda }_r^{-2}+k^2\sum _{k/2<r<2k}{V}_r{\lambda }_r^{-2}(k-r{)}^{-2}\lesssim 1+k^{1-\alpha }\sum _{k/2<r<2k}(k-r{)}^{-2}\lesssim k^{1-\alpha }. \end{gathered}$$ (B.4)

This follows the line of proofs in Hall and Hosseini-Nasab [15] in different contexts. Thus, using the inequality (B.4), it follows that

$$\text{Var}\left\{\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{\left({\lambda }_k-{\lambda }_r\right)}^{-1}{n}^{-1}\sum _{i=1}^n{ℳ}_{ir}{\xi }_{ik}\right\}=n^{-1}{\lambda }_k\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{V}_r{\left({\lambda }_k-{\lambda }_r\right)}^{-2}=O\left(n^{-1}{\lambda }_k{k}^{(1-\alpha )}\right).$$ (B.5)

Therefore, applying Lemma 2, the proof is immediate. □

Lemma 6. For ${ℳ}_{ir}=\int \dot{\mu }\left(t\right){\varphi }_r\left(t\right)dt$ and ${\eta }_k={\text{min}}_{r\ne k}\left|{\lambda }_k-{\lambda }_r\right|>0$, under Conditions (C6)a and (C6)b, $\sum _{\begin{array}{c}{r}_1\ne 1\\ r_1\ne k\end{array}}^{\infty }\sum _{\begin{array}{c}{r}_2\ne 1\\ r_2\ne k\end{array}}^{{\kappa }_0}\left({\lambda }_k-\right.{\left.{\lambda }_{r_1}\right)}^{-1}{\left({\lambda }_k-{\lambda }_{r_2}\right)}^{-1}{n}^{-1}\sum _{i=1}^n{ℳ}_{i{r}_1}{\xi }_{i{r}_2}=O\left((\text{log}n/n{)}^{1/2}{\kappa }_0^{(3-\alpha )/2}{\lambda }_{{\kappa }_0}^{-1}{\left\{\sum _{r=1}^{{\kappa }_0}{\lambda }_r\right\}}^{1/2}\right)$ almost surely.

Proof: It is not difficult to see that,

$$\mathbb{E}\left\{\sum _{\begin{array}{c}{r}_1=1\\ r_1\ne k\end{array}}^{\infty }\sum _{\begin{array}{c}{r}_2=1\\ r_2\ne k\end{array}}^{k_0}{\left({\lambda }_k-{\lambda }_{r_1}\right)}^{-1}{\left({\lambda }_k-{\lambda }_{r_2}\right)}^{-1}{n}^{-1}\sum _{i=1}^n{ℳ}_{i{r}_1}{\xi }_{i{r}_2}\right\}=0.$$

Moreover, using the spacing condition mentioned in (C6)b, one can derive the upper bound of ${\eta }_k^{-1}$ by

$$\begin{gathered} {\eta }_k^{-1}={\left\{\underset{r\ne k}{\text{min}}\left|{\lambda }_k-{\lambda }_r\right|\right\}}^{-1}=\underset{r\ne k}{\text{max}}{\left|{\lambda }_k-{\lambda }_r\right|}^{-1} \\ \lesssim \underset{r\ne k}{\text{max}}\left\{\max \left(k,r\right)/\left|k-r\right|\text{max}\left({\lambda }_k,{\lambda }_r\right)\right\}\le {\lambda }_k^{-1}k. \end{gathered}$$ (B.6)

Due to the monotonic decreasing property of eigen-values, for fixed $k\in \left\{1,\dots ,{\kappa }_0\right\}$, we have

$$\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{{\kappa }_0}{\lambda }_r{\left({\lambda }_k-{\lambda }_r\right)}^{-2}\lesssim {\eta }_k^{-2}\sum _{r=1}^{{\kappa }_0}{\lambda }_r\lesssim {\lambda }_k^{-2}{k}^2\sum _{r=1}^{{\kappa }_0}{\lambda }_r\lesssim {\lambda }_{{\kappa }_0}^{-2}{\kappa }_0^2\sum _{r=1}^{{\kappa }_0}{\lambda }_r.$$ (B.7)

Therefore, the following holds under similar conditions to obtain the Inequality (B.4),

$$\begin{gathered} \mathbb{E}\left\{\sum _{\begin{array}{c}{r}_1=1\\ r_1\ne k\end{array}}^{\infty }\sum _{\begin{array}{c}{r}_2=1\\ r_2\ne k\end{array}}^{k_0}{\left({\lambda }_k-{\lambda }_{r_1}\right)}^{-2}{\left({\lambda }_k-{\lambda }_{r_2}\right)}^{-2}{\left(n^{-1}\sum _{i=1}^n{ℳ}_{i{r}_1}{\xi }_{{ir}_2}\right)}^2\right\} \\ =n^{-1}\sum _{\begin{array}{c}{r}_1=1\\ r_1\ne k\end{array}}^{\infty }\sum _{\begin{array}{c}{r}_1=1\\ r_1\ne k\end{array}}^{k_0}{\left({\lambda }_k-{\lambda }_{r_1}\right)}^{-2}{\left({\lambda }_k-{\lambda }_{r_2}\right)}^{-2}{V}_{r_1}{\lambda }_{r_2} \\ \lesssim n^{-1}{k}^{1-\alpha }\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{{\kappa }_0}{\lambda }_r{\left({\lambda }_k-{\lambda }_r\right)}^{-2}\lesssim n^{-1}{\kappa }_0^{3-\alpha }{\lambda }_{{\kappa }_0}^{-2}\sum _{r=1}^{{\kappa }_0}{\lambda }_r. \end{gathered}$$ (B.8)

Therefore, applying the Lemma 2, the result is immediate. □

Remark 5. Define, $d_{ik}\left(T_{i{j}_1},T_{i{j}_2}\right)≔{\widehat{\varphi }}_k\left(T_{i{j}_1}\right){\widehat{\varphi }}_k\left(T_{i{j}_2}\right)-{\varphi }_k\left(T_{i{j}_1}\right){\varphi }_k\left(T_{i{j}_2}\right)$. Now replace $𝒢,\theta$ and $\psi$ by $\widehat{ℱ},\widehat{\lambda }$ and $\widehat{\varphi }$ respectively since $\widehat{ℱ}$ be the approximation of $ℱ$ and $\Delta$ is the corresponding perturbation operator in Equation (A.1). Therefore, Lemma 1 immediately implies the following expansion, which is the key fact to represent the objective function in QIF.

$${\widehat{\varphi }}_k-{\varphi }_k=\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{\left({\lambda }_k-{\lambda }_r\right)}^{-1}⟨{\varphi }_r,\Delta {\varphi }_k⟩{\varphi }_r+O\left(\Vert \Delta {\Vert }^2\right),\text{almost}\text{surely},$$ (B.9)

where $\Delta$ be the integral operator with kernel $\widehat{R}-R$.

Appendix C. Proof of Theorem 1

For the $k$-th element of ${\overline{\mathbf{g}}}_n\left({\mathit{\beta }}_0\right)\left(1\le k\le {\kappa }_0\right)$,

$$\begin{gathered} {\overline{\mathbf{g}}}_n^{\left(k\right)}\left({\mathit{\beta }}_0\right)=n^{-1}\sum _{i=1}^n{m}_i^{-2}{\dot{\mathit{\mu }}}_i^{\mathrm{T}}{\widehat{\mathbf{\Phi }}}_k\left({\mathbf{y}}_i-{\mathit{\mu }}_i\right)=n^{-1}\sum _{i=1}^n{m}_i^{-2}{\dot{\mathit{\mu }}}_i^{\mathrm{T}}{\widehat{\mathbf{\Phi }}}_k\mathbf{W}{\mathit{\xi }}_i \\ =n^{-1}\sum _{i=1}^n{m}_i^{-2}{\dot{\mathit{\mu }}}_i^{\mathrm{T}}{\mathbf{\Phi }}_k\mathbf{W}{\mathit{\xi }}_i+n^{-1}\sum _{i=1}^n{m}_i^{-2}{\dot{\mathit{\mu }}}_i^{\mathrm{T}}\left({\widehat{\mathbf{\Phi }}}_k-{\mathbf{\Phi }}_k\right)\mathbf{W}{\mathit{\xi }}_i≔J_k^{n1}+J_k^{n2}. \end{gathered}$$ (C.1)

Now, using Lemmas 3 and 4, the first part of the expression of ${\overline{\mathbf{g}}}_{n,k}\left({\mathit{\beta }}_0\right)$ becomes

$$\begin{gathered} J_k^{n1}=n^{-1}\sum _{i=1}^n{m}_i^{-2}{\dot{\mathit{\mu }}}_i^{\mathrm{T}}{\mathbf{\Phi }}_k\mathbf{W}{\mathit{\xi }}_i=n^{-1}\sum _{i=1}^n{m}_i^{-2}\sum _{j_1=1}^{m_i}\sum _{j_2=1}^{m_i}\sum _{l=1}^{{\kappa }_0}{\dot{\mu }}_i\left(T_{i{j}_1}\right){\varphi }_k\left(T_{i{j}_1}\right){\varphi }_k\left(T_{i{j}_2}\right){\varphi }_l\left(T_{i{j}_2}\right){\xi }_{il} \\ \lesssim n^{-1}\sum _{i=1}^n\left\{{ℳ}_{ik}+O\left((\text{log}m/m{)}^{1/2}\right)\right\}\left\{1+O\left((\text{log}m/m{)}^{1/2}\right)\right\}{\xi }_{ik},\text{where}{ℳ}_{ik}=\int {\dot{\mu }}_i\left(t\right){\varphi }_k\left(t\right)dt \\ =n^{-1}\sum _{i=1}^n{ℳ}_{ik}{\xi }_{ik}\left\{1+O\left((\text{log}m/m{)}^{1/2}\right)\right\}=O\left((\text{log}n/n{)}^{1/2}\left\{1+(\text{log}m/m{)}^{1/2}\right\}\right),\text{almost}\text{surely}. \end{gathered}$$ (C.2)

On the other hand, the last part of ${\overline{\mathbf{g}}}_n^{\left(k\right)}\left({\mathit{\beta }}_0\right)$ can be expressed as

$$J_k^{n2}=n^{-1}\sum _{i=1}^n{m}_i^{-2}{\dot{\mathit{\mu }}}_i^{\mathrm{T}}\left({\widehat{\mathbf{\Phi }}}_k-{\mathbf{\Phi }}_k\right)\mathbf{W}{\mathit{\xi }}_i=n^{-1}\sum _{i=1}^n{m}_i^{-2}\sum _{j_1=1}^{m_i}\sum _{j_2=1}^{m_i}\sum _{l=1}^{{\kappa }_0}{\dot{\mu }}_i\left(T_{i{j}_1}\right)d_{ik}\left(T_{i{j}_1},T_{i{j}_2}\right){\varphi }_l\left(T_{i{j}_2}\right){\xi }_{il},$$ (C.3)

where $d_{ik}\left(T_{i{j}_1},T_{i{j}_2}\right)≔{\widehat{\varphi }}_k\left(T_{i{j}_1}\right){\widehat{\varphi }}_k\left(T_{i{j}_2}\right)-{\varphi }_k\left(T_{i{j}_1}\right){\varphi }_k\left(T_{i{j}_2}\right)$ as defined in Remark 5. Therefore, using the discussion in Remark 5, we can obtain the following expression almost surely.

$$\begin{gathered} d_{ik}\left(T_{i{j}_1},T_{i{j}_2}\right)≔{\widehat{\varphi }}_k\left(T_{i{j}_1}\right){\widehat{\varphi }}_k\left(T_{i{j}_2}\right)-{\varphi }_k\left(T_{i{j}_1}\right){\varphi }_k\left(T_{i{j}_2}\right) \\ =\left\{{\varphi }_k\left(T_{i{j}_1}\right)+\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{\left({\lambda }_k-{\lambda }_r\right)}^{-1}⟨{\varphi }_r,\Delta {\varphi }_k⟩{\varphi }_r\left(T_{i{j}_1}\right)+O\left(\Vert \Delta {\Vert }^2\right)\right\}\left\{{\varphi }_k\left(T_{i{j}_2}\right)+\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{\left({\lambda }_k-{\lambda }_r\right)}^{-1}⟨{\varphi }_r,\Delta {\varphi }_k⟩{\varphi }_r\left(T_{i{j}_2}\right)+O\left(\Vert \Delta {\Vert }^2\right)\right\} \\ -{\varphi }_k\left(T_{i{j}_1}\right){\varphi }_k\left(T_{i{j}_2}\right) \\ =\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{\left({\lambda }_k-{\lambda }_r\right)}^{-1}⟨{\varphi }_r,\Delta {\varphi }_k⟩\left\{{\varphi }_r\left(T_{i{j}_1}\right){\varphi }_k\left(T_{i{j}_2}\right)+{\varphi }_k\left(T_{i{j}_1}\right){\varphi }_r\left(T_{i{j}_2}\right)\right\} \\ +\sum _{r_1\ne k}\sum _{r_2\ne k}{\left({\lambda }_k-{\lambda }_{r_1}\right)}^{-1}{\left({\lambda }_k-{\lambda }_{r_2}\right)}^{-1}⟨{\varphi }_{r_1},\Delta {\varphi }_k⟩⟨{\varphi }_{r_2},\Delta {\varphi }_k⟩{\varphi }_{r_1}\left(T_{i{j}_1}\right){\varphi }_{r_2}\left(T_{i{j}_2}\right)+O\left(\Vert \Delta {\Vert }^2\right) \\ ≔I_{ik}^{n1}\left(T_{i{j}_1},T_{i{j}_2}\right)+I_{ik}^{n2}\left(T_{i{j}_1},T_{i{j}_2}\right)+O\left(\Vert \Delta {\Vert }^2\right). \end{gathered}$$ (C.4)

Thus, almost surely, we can write

$$\begin{gathered} n^{-1}\sum _{i=1}^n{m}_i^{-2}{\dot{\mathit{\mu }}}_i^{\mathrm{T}}\left({\widehat{\mathbf{\Phi }}}_k-{\mathbf{\Phi }}_k\right)\mathbf{W}{\mathit{\xi }}_i=n^{-1}\sum _{i=1}^n{m}_i^{-2}\sum _{j_1=1}^{m_i}\sum _{j_2=1}^{m_i}\sum _{l=1}^{{\kappa }_0}{\dot{\mu }}_i\left(T_{i{j}_1}\right)d_{ik}\left(T_{i{j}_1},T_{i{j}_2}\right){\varphi }_l\left(T_{i{j}_2}\right){\xi }_{il} \\ =n^{-1}\sum _{i=1}^n{m}_i^{-2}\sum _{j_1=1}^{m_i}\sum _{j_2=1}^{m_i}\sum _{l=1}^{{\kappa }_0}{\dot{\mu }}_i\left(T_{i{j}_1}\right)d_{ik}\left(T_{i{j}_1},T_{i{j}_2}\right){\varphi }_l\left(T_{i{j}_2}\right){\xi }_{il} \\ =n^{-1}\sum _{i=1}^n{m}_i^{-2}\sum _{j_1=1}^{m_i}\sum _{j_2=1}^{m_i}\sum _{l=1}^{{\kappa }_0}{\dot{\mu }}_i\left(T_{i{j}_1}\right)\left\{I_{ik}^{n1}\left(T_{i{j}_1},T_{i{j}_2}\right)+I_{ik}^{n2}\left(T_{i{j}_1},T_{i{j}_2}\right)\right\}{\varphi }_l\left(T_{i{j}_2}\right){\xi }_{il}+O\left(\Vert \Delta {\Vert }^2\right) \\ ≔J_{k1}^{n2}+J_{k2}^{n2}+O\left(\Vert \Delta {\Vert }^2\right). \end{gathered}$$ (C.5)

Under assumptions (C1)-(C5), by using Theorem 3.3 in Li and Hsing [21], $\Vert \Delta {\Vert }^2=O\left(h^4+{\delta }_{n2}^2\left(h\right)\right)$ almost surely. Now observe that

$$\begin{gathered} J_{k1}^{n2}=n^{-1}\sum _{i=1}^n{m}_i^{-2}\sum _{j_1=1}^{m_i}\sum _{j_2=1}^{m_i}\sum _{l=1}^{k_0}\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{\left({\lambda }_k-{\lambda }_r\right)}^{-1}{\dot{\mu }}_i\left(T_{i{j}_1}\right)\left\{{\varphi }_r\left(T_{i{j}_1}\right){\varphi }_k\left(T_{i{j}_2}\right)+{\varphi }_k\left(T_{i{j}_1}\right){\varphi }_r\left(T_{i{j}_2}\right)\right\}{\varphi }_l\left(T_{i{j}_2}\right)⟨{\varphi }_r,\Delta {\varphi }_k⟩{\xi }_{il} \\ \lesssim n^{-1}\sum _{i=1}^n{m}_i^{-1}\sum _{\begin{array}{c}1\\ j_1=1\end{array}}^{m_i}\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }\sum _{l=1}^{{\kappa }_0}{\left({\lambda }_k-{\lambda }_r\right)}^{-1}{\dot{\mu }}_i\left(T_{i{j}_1}\right){\varphi }_r\left(T_{i{j}_1}\right)\left\{1(l=k)+O\left((\text{log}m/m{)}^{1/2}\right)\right\}⟨{\varphi }_r,\Delta {\varphi }_k⟩{\xi }_{il} \\ +n^{-1}\sum _{i=1}^n{m}_i^{-1}\sum _{j_1=1}^{m_i}\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }\sum _{l=1}^{k_0}{\left({\lambda }_k-{\lambda }_r\right)}^{-1}{\dot{\mu }}_i\left(T_{i{j}_1}\right){\varphi }_k\left(T_{i{j}_1}\right)\left\{1(r=l)+O\left((\text{log}m/m{)}^{1/2}\right)\right\}⟨{\varphi }_r,\Delta {\varphi }_k⟩{\xi }_{il} \\ \lesssim n^{-1}\sum _{i=1}^n{m}_i^{-1}\sum _{j_1=1}^{m_i}\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{\left({\lambda }_k-{\lambda }_r\right)}^{-1}{\dot{\mu }}_i\left(T_{i{j}_1}\right){\varphi }_r\left(T_{i{j}_1}\right)⟨{\varphi }_r,\Delta {\varphi }_k⟩{\xi }_{ik}\left\{1+O\left((\text{log}m/m{)}^{1/2}\right)\right\} \\ +n^{-1}\sum _{i=1}^n{m}_i^{-1}\sum _{\begin{array}{c}{m}_i=1\end{array}}^{m_1=\begin{array}{c}{k}_0\\ r\ne 1\end{array}}{\left({\lambda }_k-{\lambda }_r\right)}^{-1}{\dot{\mu }}_i\left(T_{i{j}_1}\right){\varphi }_k\left(T_{i{j}_1}\right)⟨{\varphi }_r,\Delta {\varphi }_k⟩{\xi }_{ir}\left\{1+O\left((\text{log}m/m{)}^{1/2}\right)\right\} \\ ≔\left(U_{k1}^n+U_{k2}^n\right)\left\{1+O\left((\text{log}m/m{)}^{1/2}\right)\right\},\text{almost}\text{surely.} \end{gathered}$$ (C.6)

Then applying the triangle inequality, we have

$$\begin{gathered} U_{k1}^n=n^{-1}\sum _{i=1}^n{m}_i^{-1}\sum _{j=1}^{m_i}\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{\left({\lambda }_k-{\lambda }_r\right)}^{-1}{\dot{\mu }}_i\left(T_{ij}\right){\varphi }_r\left(T_{ij}\right)⟨\Delta {\varphi }_k,{\varphi }_r⟩{\xi }_{ik} \\ \lesssim ‖\Delta {\varphi }_k‖\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{\left({\lambda }_k-{\lambda }_r\right)}^{-1}{n}^{-1}\sum _{i=1}^n\left\{{ℳ}_{ir}+O\left((\text{log}m/m{)}^{1/2}\right)\right\}{\xi }_{ik}=‖\Delta {\varphi }_k‖\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{\left({\lambda }_k-{\lambda }_r\right)}^{-1}{n}^{-1}\sum _{i=1}^n{ℳ}_{ir}{\xi }_{ik}\left\{1+O\left((\text{log}m/m{)}^{1/2}\right)\right\}, \end{gathered}$$ (C.7)

where ${ℳ}_{ir}=\int {\dot{\mu }}_i\left(t\right){\varphi }_r\left(t\right)dt$. By Lemma 6 of Li and Hsing [21], under conditions (C1)-(C5), for any measurable bounded function $e$ on [0, 1], $‖\Delta {\varphi }_k‖=O\left(h^2+{\delta }_{n1}\left(h\right)+{\delta }_{n2}^2\left(h\right)\right)\equiv O\left({\bar{\delta }}_n\left(h\right)\right)$ almost surely, where ${\bar{\delta }}_n\left(h\right)=h^2+{\delta }_{n1}\left(h\right)+{\delta }_{n2}^2\left(h\right)$. Thus, in addition with Inequalities (B.5) in Lemma 5, we obtain

$$U_{k1}^n=O\left({\bar{\delta }}_n\left(h\right)(\text{log}n/n{)}^{1/2}{\lambda }_k^{1/2}{k}^{(1-\alpha )/2}\left\{1+(\text{log}m/m{)}^{1/2}\right\}\right)$$

almost surely. Next, under the spacing condition mentioned earlier and in assumption (C6)a, using the Inequality (B.6), recall ${\eta }_k^{-1}\lesssim {\lambda }_k^{-1}k$. Thus, observe that

$$\begin{gathered} U_{k2}^n=n^{-1}\sum _{i=1}^n{m}_i^{-1}\sum _{j=1}^{m_i}\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{{\kappa }_0}{\left({\lambda }_k-{\lambda }_r\right)}^{-1}{\dot{\mu }}_i\left(T_{ij}\right){\varphi }_k\left(T_{i_j}\right)⟨\Delta {\varphi }_k,{\varphi }_r⟩{\xi }_{ir} \\ \lesssim ‖\Delta {\varphi }_k‖\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{{\kappa }_0}{\left({\lambda }_k-{\lambda }_r\right)}^{-1}\left\{n^{-1}\sum _{i=1}^n{ℳ}_{ik}{\xi }_{ir}\right\}\left\{1+O\left((\text{log}m/m{)}^{1/2}\right)\right\}\lesssim ‖\Delta {\varphi }_k‖{\eta }_k^{-1}\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{{\kappa }_0}\left\{n^{-1}\sum _{i=1}^n{ℳ}_{ik}{\xi }_{ir}\right\}\left\{1+O\left((\text{log}m/m{)}^{1/2}\right)\right\}. \end{gathered}$$ (C.8)

Using condition (C6)b, we also have $V_k^{1/2}{\eta }_k^{-1}\lesssim V_k^{1/2}{\lambda }_k^{-1}k=O\left(k^{(1-\alpha )/2}\right)$. Finally, combining with the bounds for $U_{k1}^n,U_{k2}^n$, we have, almost surely,

$$J_{k1}^{n2}=O\left\{(\text{log}n/n{)}^{1/2}{\bar{\delta }}_n(h)\left({\lambda }_k^{1/2}{k}^{(1-\alpha )/2}+{\eta }_k^{-1}{V}_k^{1/2}\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{{\kappa }_0}{\lambda }_r^{1/2}\right)\left(1+(\text{log}m/m{)}^{1/2}\right)\right\}≔O\left({\omega }_{k1}(n,h)\right),$$ (C.9)

where ${\omega }_{k1}(n,h)=(\text{log}n/n{)}^{1/2}{\bar{\delta }}_n(h)k^{(1-\alpha )/2}\sum _{k=1}^{{\kappa }_0}{\lambda }_r^{1/2}\left\{1+(\text{log}m/m{)}^{1/2}\right\}$.

It is easy to see that $\sum _{r=1}^{{\kappa }_0}{\lambda }_r^{1/2}~{\kappa }_0^{-{\tau }_1/2+1}$. Therefore, for $\tau =\alpha +{\tau }_1,{\omega }_{k1}(n,h)~(\text{log}n/n{)}^{1/2}{\bar{\delta }}_n(h){\kappa }_0^{(3-\tau )/2}\left\{1+(\text{log}m/m{)}^{1/2}\right\}$. Similarly, to the derivation of the bound for $J_{k1}^{n2}$, we can write

$$\begin{gathered} J_{k2}^{n2}=n^{-1}\sum _{i=1}^n{m}_i^{-2}\sum _{j_1=1}^{m_i}\sum _{j_2=1}^{m_i}\sum _{l=1}^{{\kappa }_0}{\dot{\mu }}_i\left(T_{i{j}_1}\right)I_{ik}^{n2}\left(T_{i{j}_1},T_{i{j}_2}\right){\varphi }_l\left(T_{i{j}_2}\right){\xi }_{il} \\ =n^{-1}\sum _{i=1}^n{m}_i^{-2}\sum _{j_1=1}^{m_i}\sum _{j_2=1}^{m_i}\sum _{l=1}^{{\kappa }_0}\sum _{\begin{array}{c}{r}_1=1\\ r_1\ne k\end{array}}^{\infty }\sum _{\begin{array}{c}{r}_1=1\\ r_2\ne k\end{array}}^{k0}{\left({\lambda }_k-{\lambda }_{r_1}\right)}^{-1}{\left({\lambda }_k-{\lambda }_{r_2}\right)}^{-1}\langle {\varphi }_{r_1},\Delta {\varphi }_k\rangle \langle {\varphi }_{r_2},\Delta {\varphi }_k\rangle {\dot{\mu }}_i\left(T_{i{j}_1}\right){\varphi }_{r_1}\left(T_{i{j}_1}\right){\varphi }_{r_2}\left(T_{i{j}_2}\right){\varphi }_l\left(T_{i{j}_2}\right){\xi }_{il} \\ \lesssim n^{-1}\sum _{i=1}^n{m}_i^{-1}\sum _{j_1=1}^{m_i}\sum _{l=1}^{{\kappa }_0}\sum _{\begin{array}{c}{r}_1=1\\ r_1\ne k\end{array}}^{\infty }\sum _{\begin{array}{c}{r}_2=1\\ r_2\ne k\end{array}}^{k0}{\left({\lambda }_k-{\lambda }_{r_1}\right)}^{-1}{\left({\lambda }_k-{\lambda }_{r_2}\right)}^{-1}{\varphi }_{r_1},\Delta {\varphi }_k{\varphi }_{r_2},\Delta {\varphi }_k{\dot{\mu }}_i\left(T_{i_1}\right){\varphi }_{r_1}\left(T_{i{j}_1}\right)\left\{1\left(r_2=l\right)+O\left({(\text{log}m/m)}^{1/2}\right)\right\}{\xi }_{il} \\ =n^{-1}\sum _{i=1}^n{m}_i^{-1}\sum _{j_1=1}^{m_i}\sum _{\begin{array}{c}{r}_1=1\\ r_1\ne k\end{array}}^{\infty }\sum _{\begin{array}{c}{r}_2=1\\ r_2\ne k\end{array}}^{k0}{\left({\lambda }_k-{\lambda }_{r_1}\right)}^{-1}{\left({\lambda }_k-{\lambda }_{r_2}\right)}^{-1}{\varphi }_{r_1},\Delta {\varphi }_k{\varphi }_{r_2},\Delta {\varphi }_k{\dot{\mu }}_i\left(T_{i{j}_1}\right){\varphi }_{r_1}\left(T_{i{j}_1}\right){\xi }_{i{r}_2}\left\{1+O\left({(\text{log}m/m)}^{1/2}\right)\right\} \\ \lesssim {‖\Delta {\varphi }_k‖}^2\sum _{\begin{array}{c}{r}_1=1\\ r_1\ne k\end{array}}^{\infty }\sum _{\begin{array}{c}{r}_2=1\\ r_2\ne k\end{array}}^{k0}{\left({\lambda }_k-{\lambda }_{r_1}\right)}^{-1}{\left({\lambda }_k-{\lambda }_{r_2}\right)}^{-1}\left(n^{-1}\sum _{i=1}^n\left\{{ℳ}_{i{r}_1}+O\left({(\text{log}m/m)}^{1/2}\right)\right\}{\xi }_{i{r}_2}\right)\left\{1+O\left({(\text{log}m/m)}^{1/2}\right)\right\} \\ ={‖\Delta {\varphi }_k‖}^2\sum _{\begin{array}{c}{r}_1=1\\ r_1\ne k\end{array}}^{\infty }\sum _{\begin{array}{c}{r}_2=1\\ r_2\ne k\end{array}}^{k0}{\left({\lambda }_k-{\lambda }_{r_1}\right)}^{-1}{\left({\lambda }_k-{\lambda }_{r_2}\right)}^{-1}\times \left\{n^{-1}\sum _{i=1}^n{ℳ}_{i{r}_1}{\xi }_{i{r}_2}\right\}\left\{1+O\left({(\text{log}m/m)}^{1/2}\right)\right\}. \end{gathered}$$ (C.10)

Therefore, Inequality (C.10) immediately follows using Lemma 6,

$$J_{k2}^{n2}=O\left((\text{log}n/n{)}^{1/2}{\bar{\delta }}_n^2(h){\kappa }_0^{(3-\alpha )/2}{\lambda }_{{\kappa }_0}^{-1}{\left\{\sum _{r=1}^{{\kappa }_0}{\lambda }_r\right\}}^{1/2}\left\{1+(\text{log}m/m{)}^{1/2}\right\}\right)≔O\left({\omega }_{k2}(n,h)\right),\text{almost}\text{surely},$$

where

$${\omega }_{k2}(n,h)=(\text{log}n/n{)}^{1/2}{\bar{\delta }}_n^2(h){\kappa }_0^{(3-\alpha )/2}{\lambda }_{{\kappa }_0}^{-1}\sum _{r=1}^{{\kappa }_0}{\lambda }_r\left\{1+(\text{log}m/m{)}^{1/2}\right\}.$$

We observe that $\sum _{r=1}^{{\kappa }_0}{\lambda }_r~{\kappa }_0^{-{\tau }_1+1}$ under assumption (C6)b. Thus, ${\omega }_{k2}(n,h)~(\text{log}n/n){\bar{\delta }}^2\left(h\right){\kappa }_0^{(4-\tau )/2}\left\{1+(\text{log}m/m{)}^{1/2}\right\}$. Since ${\omega }_{n2}=O\left({\omega }_{n1}\right)$ and ${\bar{\delta }}_n^2\left(h\right)=O\left({\omega }_{n1}\right)$, in summary, for each $k\in \left\{1,\dots ,{\kappa }_0\right\}$,

$${\overline{\mathbf{g}}}^{\left(k\right)}\left(\mathit{\beta }\right)=O\left((\text{log}n/n{)}^{1/2}\left\{1+(\text{log}m/m{)}^{1/2}\right\}+{\omega }_{k1}(n,h)\right)$$

almost surely. Since $1/\left(nm\right)=O(1/n)$,

$$\text{AMSE}\left\{{\overline{\mathbf{g}}}^{\left(k\right)}\left({\mathit{\beta }}_0\right)\right\}=O\left(n^{-1}+n^{-1}{\kappa }_0^{3-\tau }{R}_n\left(h\right)\right),$$

where $R_n\left(h\right)=\left\{h^4+\frac{1}{n}+\frac{1}{nmh}+\frac{1}{n^2{m}^2{h}^2}+\frac{1}{n^2{m}^4{h}^4}+\frac{1}{n^{2}mh}+\frac{1}{n^2{m}^3{h}^3}\right\}$. Combining the above conditions, we find that if $a>1/4,{\kappa }_0=O\left(n^{1/(3-\tau )}\right)$ and $n^{-1/4}\lesssim h\lesssim n^{-(a+1)/5}$ then $\text{AMSE}\left\{\overline{{\mathbf{g}}^{\left(k\right)}}\left({\mathit{\beta }}_0\right)\right\}=O(1/n)$. On the other hand, if $a\le 1/4,{\kappa }_0=O\left(n^{4(1+a)/5(3-\tau )}\right)$ and $h\lesssim n^{-1/4}\text{AMSE}\left\{\overline{{\mathbf{g}}^{\left(k\right)}}\left({\mathit{\beta }}_0\right)\right\}=O(1/n)$.

Note that for a three-dimensional array $\left(\partial C/\partial {\beta }_1,\dots ,\partial C/\partial {\beta }_p\right)$ such that the following is a $p\times 1$ vector.

$$\overline{\mathbf{g}}{\left({\mathit{\beta }}_0\right)}^{\mathrm{T}}{\mathbf{C}}^{-1}\left({\mathit{\beta }}_0\right)\dot{\mathbf{C}}\left({\mathit{\beta }}_0\right){\mathbf{C}}^{-1}\left({\mathit{\beta }}_0\right)\overline{\mathbf{g}}\left({\mathit{\beta }}_0\right)$$ (C.11)

Therefore,

$$n^{-1}\dot{\mathit{Q}}\left({\mathit{\beta }}_0\right)=2\dot{\overline{\mathbf{g}}}{\left({\mathit{\beta }}_0\right)}^{\mathrm{T}}{\mathbf{C}}^{-1}\left({\mathit{\beta }}_0\right)\overline{\mathbf{g}}\left({\mathit{\beta }}_0\right)-\overline{\mathbf{g}}{\left({\mathit{\beta }}_0\right)}^{\mathrm{T}}{\mathbf{C}}^{-1}\left({\mathit{\beta }}_0\right)\dot{\mathbf{C}}\left({\mathit{\beta }}_0\right){\mathbf{C}}^{-1}\left({\mathit{\beta }}_0\right)\overline{\mathbf{g}}\left({\mathit{\beta }}_0\right)$$ (C.12)

and

$$n^{-1}\ddot{\mathit{Q}}\left({\mathit{\beta }}_0\right)=2\dot{\overline{\mathbf{g}}}{\left({\mathit{\beta }}_0\right)}^{\mathrm{T}}{\mathbf{C}}^{-1}\left({\mathit{\beta }}_0\right)\dot{\overline{\mathbf{g}}}\left({\mathit{\beta }}_0\right)+r_{n1}+r_{n2}+r_{n3}+r_{n4}$$ (C.13)

where

$$\begin{gathered} r_{n1}=2\ddot{\overline{g}}\left({\mathit{\beta }}_0\right){\mathbf{C}}^{-1}\left({\mathit{\beta }}_0\right)\overline{\mathbf{g}}\left({\mathit{\beta }}_0\right),r_{n2}=-4\dot{\overline{\mathbf{g}}}{\left({\mathit{\beta }}_0\right)}^{\mathrm{T}}{\mathbf{C}}^{-1}\left({\mathit{\beta }}_0\right)\dot{\mathbf{C}}\left({\mathit{\beta }}_0\right){\mathbf{C}}^{-1}\left({\mathit{\beta }}_0\right)\overline{\mathbf{g}}\left({\mathit{\beta }}_0\right), \\ {r}_{n3}=2\dot{\overline{g}}\left({\mathit{\beta }}_0\right){\mathbf{C}}^{-1}\left({\mathit{\beta }}_0\right){\mathbf{C}}^{-1}\left({\mathit{\beta }}_0\right)\dot{\mathbf{C}}\left({\mathit{\beta }}_0\right){\mathbf{C}}^{-1}\left({\mathit{\beta }}_0\right)\overline{\mathbf{g}}\left({\mathit{\beta }}_0\right),r_{n4}=-\overline{\mathbf{g}}{\left({\mathit{\beta }}_0\right)}^{\mathrm{T}}{\mathbf{C}}^{-1}\left({\mathit{\beta }}_0\right)\ddot{\mathbf{C}}\left({\mathit{\beta }}_0\right){\mathbf{C}}^{-1}\left({\mathit{\beta }}_0\right)\overline{\mathbf{g}}\left({\mathit{\beta }}_0\right). \end{gathered}$$ (C.14)

Since $\overline{\mathbf{g}}\left({\mathit{\beta }}_0\right)=O_P\left(n^{-1/2}\right)$ and the weight matrix converges almost surely to an invertible matrix,

$$\overline{\mathbf{g}}{\left({\mathit{\beta }}_0\right)}^{\mathrm{T}}{\mathbf{C}}^{-1}\left({\mathit{\beta }}_0\right)\dot{\mathbf{C}}\left({\mathit{\beta }}_0\right){\mathbf{C}}^{-1}\left({\mathit{\beta }}_0\right)\overline{\mathbf{g}}\left({\mathit{\beta }}_0\right)=o\left(n^{-1}\right),$$

almost surely. Furthermore, $r_{n1}=O\left(n^{-1/2}\right),r_{n2}=o\left(n^{-1/2}\right),r_{n3}=O\left(n^{-1/2}\right)$, and $r_{n4}=O\left(n^{-1}\right)$ almost surely. Combining these bounds, we have $r_n=o\left(1\right)$ almost surely. Therefore, $n^{-1}\dot{\mathit{Q}}-2\dot{\overline{g}}{\left({\mathit{\beta }}_0\right)}^{\text{T}}{C}^{-1}\left({\mathit{\beta }}_0\right)\overline{g}\left({\mathit{\beta }}_0\right)=o_P\left(n^{-1}\right)$ and $\Vert n^{-1}\ddot{\mathit{Q}}-2\dot{\overline{g}}{\left({\mathit{\beta }}_0\right)}^{\mathrm{T}}{\mathbf{C}}^{-1}\left({\mathit{\beta }}_0\right)\dot{\overline{g}}\left({\mathit{\beta }}_0\right)\Vert =o_P\left(1\right)$.

The following lines are based on common steps in the GEE literature that includes Balan and Schiopu-Kratina [2], McCullagh and Nelder [27], Tian et al. [30] among many others. Let ${\mathit{\beta }}_n={\mathit{\beta }}_0+\delta \mathbf{d}$ where set $\delta =n^{-1/2}$. We have to show that for any $ϵ>0$ there exists a large constant $c$ such that

$$\mathbb{P}\left\{\underset{\Vert d\Vert =c}{\text{inf}}Q\left({\mathit{\beta }}_n\right)\ge Q\left({\mathit{\beta }}_0\right)\right\}>1-ϵ.$$ (C.15)

Note that the above statement is always true if $ϵ\ge 1$. Thus, we assume that $ϵ\in \left(\mathrm{0,1}\right)$. Due to Taylor series expansion,

$$Q\left({\mathit{\beta }}_n\right)=Q\left({\mathit{\beta }}_0+\delta \mathbf{d}\right)=Q\left({\mathit{\beta }}_0\right)+\delta {\mathbf{d}}^{\mathrm{T}}\dot{Q}\left({\mathit{\beta }}_0\right)+0.5\delta {\mathbf{d}}^{\mathrm{T}}\ddot{Q}\left({\mathit{\beta }}_0\right)\mathbf{d}+\Vert \mathbf{d}{\Vert }^2{o}_P\left(1\right).$$ (C.16)

Now, observe that, using Equation (C.12),

$$\delta {\mathbf{d}}^{\mathrm{T}}\dot{Q}\left({\mathit{\beta }}_0\right)=\Vert \mathbf{d}\Vert O_P\left(\sqrt{n}\delta \right)+\Vert \mathbf{d}\Vert O_P\left(\delta \right),$$ (C.17)

and

$$0.5{\delta }^2{\mathbf{d}}^{\mathrm{T}}\ddot{\mathit{Q}}\left({\mathit{\beta }}^{*}\right)\mathbf{d}=n{\delta }^2{\mathbf{d}}^{\mathrm{T}}\dot{\overline{\mathbf{g}}}\left({\mathit{\beta }}_0\right){\mathbf{C}}^{-1}\left({\mathit{\beta }}_0\right)\dot{\mathbf{g}}\left({\mathit{\beta }}_0\right)\mathbf{d}+n{\delta }^2\Vert \mathbf{d}{\Vert }^2{o}_P\left(1\right).$$ (C.18)

Therefore, for given $ϵ>0$, there exists a large enough $c$ such that the above equation (C.15) holds. This implies that there exists a $\stackrel{\mathit{ˆ}}{\mathit{\beta }}$ that satisfies $‖\widehat{\mathit{\beta }}-{\mathit{\beta }}_0‖=O_P\left(\delta \right)$. Thus, for large $n$, with probability 1, $Q\left(\mathit{\beta }\right)$ attains the minimal value at $\hat{\beta }$ and therefore, $\dot{Q}=0$.

Appendix D. Proof of Theorem 2

Recall, $C_i=\sum _{k_1=1}^{k_0}\sum _{k_2=1}^{k_0}{\mathbf{\Phi }}_{k_1}{\mathbf{X}}_i{\mathbf{C}}_{k_1,k_2}^{-1}{\mathbf{X}}_i^{\mathrm{T}}{\mathbf{\Phi }}_{k_2}$, where ${\mathbf{C}}_{k_1,k_2}^{-1}$ is the $\left(k_1,k_2\right)$ block of ${\mathbf{C}}_0^{-1}$. Similarly, we can define $\widehat{{𝒞}_i}=\sum _{k_1=1}^{k_0}\sum _{k_2=1}^{k_0}{\widehat{\mathbf{\Phi }}}_{k_1}{\mathbf{X}}_i{\mathbf{C}}_{k_1,k_2}^{-1}{\mathbf{X}}_i^{\mathrm{T}}{\widehat{\mathbf{\Phi }}}_{k_2}$. It is easy to observe that $\widehat{C_i}=C_i+\sum _{k_1=1}^{k_0}\sum _{k_2=1}^{{\kappa }_0}\left({\widehat{\mathbf{\Phi }}}_{k_1}-{\mathbf{\Phi }}_{k_1}\right){\mathbf{X}}_i{\mathbf{C}}_{k_1,k_2}^{-1}{\mathbf{X}}_i^{\mathrm{T}}\left({\widehat{\mathbf{\Phi }}}_{k_2}-\right.\left.{\mathbf{\Phi }}_{k_2}\right)+2\sum _{k_1=1}^{{\kappa }_0}\sum _{k_2=1}^{{\kappa }_0}{\mathbf{\Phi }}_{k_1}{\mathbf{X}}_i{\mathbf{C}}_{k_1,k_2}^{-1}{\mathbf{X}}_i^{\mathrm{T}}\left({\widehat{\mathbf{\Phi }}}_{k_2}-{\mathbf{\Phi }}_{k_2}\right)$. Therefore, $n^{-1}\sum _{i=1}^n{\dot{\mathit{\mu }}}_i^{\text{T}}\left(\widehat{C_i}-C\right)X_i=n^{-1}\sum _{i=1}^n\sum _{k_1=1}^{k_0}\sum _{k_2=1}^{{\kappa }_0}{P}_{i{k}_1}{C}_{k_1,k_2}^{-1}{P}_{i{k}_2}$ and $n^{-1}\sum _{i=1}^n{\dot{\mathit{\mu }}}_i^{\mathrm{T}}\left(\widehat{{\mathit{C}}_i}-C\right){\mathbf{y}}_i=n^{-1}\sum _{i=1}^n\sum _{k_1=1}^{K_0}\sum _{k_2=1}^{K_0}{\mathbf{P}}_{i{k}_1}{\mathbf{C}}_{k_1,k_2}^{-1}{\mathbf{Q}}_{i{k}_2}$ where ${\mathbf{P}}_{i,k}={\dot{\mathit{\mu }}}_i^{\mathrm{T}}{\mathbf{D}}_{ik}{\mathbf{X}}_i$ and ${\mathbf{Q}}_{ik}={\dot{\mathit{\mu }}}_i^{\mathrm{T}}{\mathbf{D}}_{ik}{\mathbf{y}}_i$ with ${\mathbf{D}}_{ik}$ be the difference matrix with $\left(j_1,j_2\right)$-th element is $d_i\left(T_{i{j}_1},T_{i{j}_2}\right)$. Thus, note that, almost surely we have the following relation,

$$\begin{gathered} {\mathbf{P}}_{ik}=m_i^{-2}\sum _{j_1=1}^{m_i}\sum _{j_2=1}^{m_i}{\dot{\mu }}_i\left(T_{i{j}_1}\right)d_i\left(T_{i{j}_1},T_{i{j}_2}\right)x_i\left(T_{i{j}_2}\right) \\ \lesssim m_i^{-2}\sum _{j_1=1}^{m_i}\sum _{j_2=1}^{m_i}{\dot{\mu }}_i\left(T_{i{j}_1}\right)\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{\left({\lambda }_k-{\lambda }_r\right)}^{-1}⟨{\varphi }_r,\Delta {\varphi }_k⟩\left\{{\varphi }_r\left(T_{i{j}_1}\right){\varphi }_k\left(T_{i{j}_2}\right)+{\varphi }_k\left(T_{i{j}_1}\right){\varphi }_r\left(T_{i{j}_2}\right)\right\}+O\left(\Vert \Delta {\Vert }^2\right) \\ \lesssim \sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{\left({\lambda }_k-{\lambda }_r\right)}^{-1}‖\Delta {\varphi }_k‖+O\left(\Vert \Delta {\Vert }^2\right)=O\left(\varpi \right), \end{gathered}$$ (D.1)

since $m_i^{-1}\sum _{j=1}^{m_i}\dot{\mu }\left(T_{ij}\right){\varphi }_k\left(T_{ij}\right)$ and $m_i^{-1}\sum _{j=1}^{m_i}{x}_i\left(T_{ij}\right){\varphi }_k\left(T_{ij}\right)$ are finite, where $\varpi =\sum _{\begin{array}{c}r=1\\ r\ne k\end{array}}^{\infty }{\left({\lambda }_k-{\lambda }_r\right)}^{-1}{\bar{\delta }}_n\left(h\right)+h^2+{\delta }_{n2}^2\left(h\right)$. A similar result can be obtained for ${\mathbf{Q}}_{ik}$. Combining such results, in summary, we have $-2n^{-1}\sum _{i=1}^n{\mathbf{X}}_i^{\mathrm{T}}\widehat{{𝒞}_i}\left({\mathbf{y}}_i-{\mathbf{X}}_i^{\mathrm{T}}\widehat{\mathit{\beta }}\right)=-2n^{-1}\sum _{i=1}^n{\mathbf{X}}_i^{\mathrm{T}}{C}_i\left({\mathbf{y}}_i-{\mathbf{X}}_i^{\mathrm{T}}\widehat{\mathit{\beta }}\right)+O\left({\varpi }_n\right)$. Since, for $n\to \infty ,Q\left(\mathit{\beta }\right)$ attains a minimal value at $\mathit{\beta }=\widehat{\mathit{\beta }}$, we therefore have $\dot{Q}\left(\widehat{\mathit{\beta }}\right)=0$. Thus,

$$\dot{Q}\left(\widehat{\mathit{\beta }}\right)=-2n^{-1}\sum _{i=1}^n{\mathbf{X}}_i^{\mathrm{T}}\widehat{{𝒞}_i}\left({\mathbf{y}}_i-{\mathbf{X}}_i^{\mathrm{T}}\widehat{\mathit{\beta }}\right)=-2n^{-1}\sum _{i=1}^n{\mathbf{X}}_i^{\mathrm{T}}\left(\widehat{C_i}-C_i\right)\left({\mathbf{y}}_i-{\mathbf{X}}_i^{\mathrm{T}}\widehat{\mathit{\beta }}\right)-2n^{-1}\sum _{i=1}^n{\mathbf{X}}_i^{\mathrm{T}}{C}_i\left({\mathbf{y}}_i-{\mathbf{X}}_i^{\mathrm{T}}\widehat{\mathit{\beta }}\right)=0.$$ (D.2)

Therefore, almost surely, we have,

$$\begin{gathered} -2n^{-1}\sum _{i=1}^n{\mathbf{X}}_i^{\mathrm{T}}{C}_i\left({\mathbf{y}}_i-{\mathbf{X}}_i^{\mathrm{T}}\widehat{\mathit{\beta }}\right)+O\left({\varpi }_n\right)=0,-2n^{-1}\sum _{i=1}^n{\mathbf{X}}_i^{\mathrm{T}}{𝒞}_i\left({\mathbf{X}}_i^{\mathrm{T}}{\mathit{\beta }}_0+{\mathbf{e}}_i-{\mathbf{X}}_i^{\mathrm{T}}\widehat{\mathit{\beta }}\right)+O\left({\varpi }_n\right)=0, \\ \sqrt{n}\left(\widehat{\mathit{\beta }}-{\mathit{\beta }}_0\right)\left\{n^{-1}\sum _{i=1}^n{\mathbf{X}}_i^{\mathrm{T}}{C}_i{\mathbf{X}}_i\right\}=n^{-1}\sum _{i=1}^n{\mathbf{X}}_i^{\mathrm{T}}{𝒞}_i{\mathbf{e}}_i. \end{gathered}$$ (D.3)

Now, using the central limit theorem, we can obtain the following.

$$\sum _{i=1}^n{\mathbf{X}}_i^{\mathrm{T}}{C}_i{\mathbf{e}}_i/\sqrt{n}\stackrel{d}{\to }N(0,\mathbf{A}).$$ (D.4)

In addition, by the law of large numbers $n^{-1}\sum _{i=1}^n{\mathbf{X}}_i^{\mathrm{T}}{𝒞}_i{\mathbf{X}}_i\to \mathbf{B}$ in probability. Therefore, using the Slutsky theorem, we complete the proof of Theorem 2.