Fast covariance estimation for multivariate sparse functional data

Abstract

Covariance estimation is essential yet underdeveloped for analysing multivariate functional data. We propose a fast covariance estimation method for multivariate sparse functional data using bivariate penalized splines. The tensor‐product B‐spline formulation of the proposed method enables a simple spectral decomposition of the associated covariance operator and explicit expressions of the resulting eigenfunctions as linear combinations of B‐spline bases, thereby dramatically facilitating subsequent principal component analysis. We derive a fast algorithm for selecting the smoothing parameters in covariance smoothing using leave‐one‐subject‐out cross‐validation. The method is evaluated with extensive numerical studies and applied to an Alzheimer's disease study with multiple longitudinal outcomes.

1. INTRODUCTION

Functional data analysis (FDA) has been enjoying great successes in many applied fields, for example, neuroimaging (Reiss & Ogden, 2010; Lindquist, 2012; Goldsmith, Crainiceanu, Caffo, & Reich, 2012; Zhu, Li, & Kong, 2012), genetics (Leng & Müller, 2006; Reimherr & Nicolae, 2014; 2016), and wearable computing (Morris et al., 2006; Xiao et al., 2015). Functional principal component analysis (FPCA) conducts dimension reduction on the inherently infinite‐dimensional functional data and thus facilitates subsequent modelling and analysis. Traditionally, functional data are densely observed on a common grid and can be easily connected to multivariate data, although the notion of smoothness distinguishes the former from the latter. In recent years, covariance‐based FPCA (Yao, Müller, & Wang, 2005) has become a standard approach and has greatly expanded the applicability of functional data methods to irregularly spaced data such as longitudinal data. Various nonparametric methods have now been proposed to estimate the smooth covariance function, for example, Peng and Paul (2009), Cai and Yuan (2010), Goldsmith et al. (2012), Xiao, Li, Checkley, and Crainiceanu (2018), and Wong and Zhang (2019).

There has been growing interest in multivariate functional data where multiple functions are observed for each subject. For dense functional data, Ramsay and Silverman (2005, Chapter 8.5) proposed to concatenate multivariate functional data as a single vector and conduct multivariate PCA on the long vectors, and Berrendero, Justel, and Svarc (2011) repeatedly applied point‐wise univariate PCA. For sparse and paired functional data, Zhou, Huang, and Carroll (2008) extended the low‐rank mixed effects model in James, Hastie, and Sugar (2000). Chiou, Chen, and Yang (2014) considered normalized multivariate FPCA through standardizing the covariance operator. Petersen and Müller (2016) proposed various metrics for studying cross‐covariance between multivariate functional data. More recently, Happ and Greven (2018) introduced a FPCA framework for multivariate functional data defined on different domains.

The interest of the paper is FPCA for multivariate sparse functional data, where multiple responses are observed at time points that vary from subject to subject and may even vary between responses within subjects. There are much fewer works to handle such data. The approach in Zhou et al. (2008) focuses on bivariate functional data and can be extended to more than two‐dimensional functional data, although model selection (e.g., selection of smoothing parameters) can be computationally difficult and convergence of the expectation–maximization estimation algorithm could also be an issue. The local polynomial method in Chiou et al. (2014) can be applied to multivariate sparse functional data, although a major drawback is the selection of multiple bandwidths. Moreover, because the local polynomial method is a local approach, there is no guarantee that the resulting estimates of covariance functions will lead to a properly defined covariance operator. The approach in Happ and Greven (2018) (denoted by mFPCA hereafter) estimates cross‐covariances via scores from univariate FPCA and hence can be applied to multivariate sparse functional data. Although mFPCA is theoretically sound for dense functional data, it may not capture cross‐correlations between functions because scores from univariate FPCA for sparse functional data are shrunk towards zero.

We propose a novel and fast covariance‐based FPCA method for multivariate sparse functional data. Note that multiple auto‐covariance functions for within‐function correlations and cross‐covariance functions for between‐function correlations have to be estimated. Tensor‐product B‐splines are employed to approximate the covariance functions, and a smoothness penalty as in bivariate penalized splines (Eilers & Marx, 2003) is adopted to avoid overfit. Then, the individual estimates of covariance functions will be pooled and refined. The advantages of the new method are multifold. First, the tensor‐product B‐spline formulation is computationally efficient to handle multivariate sparse functional data. Second, a fast fitting algorithm for selecting the smoothing parameters will be derived, which alleviates the computational burden of conducting leave‐one‐subject‐out cross‐validation. Third, the tensor‐product B‐spline representation of the covariance functions enables a straightforward spectral decomposition of the covariance operator for the multivariate functional data; see Proposition 1. In particular, the eigenfunctions associated with the covariance operator are explicit functions of the B‐spline bases. Last but not the least, via a simple truncation step, the refined estimates of the covariance functions lead to a properly defined covariance operator.

Compared with mFPCA, the proposed method does not rely on scores from univariate FPCA, which could be a severe problem for sparse functional data and hence could better capture the correlations between functions. And an improved correlation estimation will lead to improved subsequent FPCA analysis and curve prediction. The proposed method also compares favourably with the local polynomial method in Chiou et al. (2014) because of the computationally efficient tensor‐product spline formulation of the covariance functions and the derived fast algorithm for selecting the smoothing parameters. Moreover, as mentioned above, there exists an explicit and easy‐to‐calculate relationship between the tensor‐product spline representation of covariance functions and the associated eigenfunctions/eigenvalues, which greatly facilitate subsequent FPCA analysis.

In addition to FPCA, there are also abundant literatures on models for multivariate functional data with most focusing on dense functional data. For clustering of multivariate functional data, see Zhu, Brown, and Morris (2012), Jacques and Preda (2014), Huang, Li, and Guan (2014), and Park and Ahn (2017). For regression with multivariate functional responses, see Zhu et al. (2012), Luo and Qi (2017), Li, Huang, and Zhu, (2017), Wong, Li, and Zhu (2019), Zhu, Morris, Wei, and Cox (2017), Kowal, Matteson, and Ruppert (2017), and Qi and Luo (2018). Graphical models for multivariate functional data are studied in (Zhu, Strawn, & Dunson, 2016) and Qiao, Guo, and James (2019). Works on multivariate functional data include also Chiou and Müller (2014, 2016).

The remainder of the paper proceeds as follows. In Section 2, we present our proposed method. We conduct extensive simulation studies in Section 3 and apply the proposed method to an Alzheimer's disease (AD) study in Section 4. A discussion is given in Section 5. All technical details are enclosed in the Appendix.

2. METHODS

2.1. Fundamentals of multivariate functional principal component analysis

Let p be a positive integer and denote by $\mathcal{T}$ a continuous and bounded domain in the real line $\mathbb{R}$. Consider the Hilbert space $\mathcal{H}:\underset{p}{\underbrace{L^2(\mathcal{T})\times \text{…}\times L^2(\mathcal{T})}}$ equipped with the inner product $<·,·{>}_{\mathcal{H}}$ and norm $\Vert ·{\Vert }_{\mathcal{H}}$ such that for arbitrary functions $\mathbf{f}={\left(f^{(1)},\text{…},f^{(p)}\right)}^{\top }$ and $\mathbf{g}={\left(g^{(1)},\text{…},g^{(p)}\right)}^{\top }$ in $\mathcal{H}$ with each element in $L^2(\mathcal{T})$, $<\mathbf{f},\mathbf{g}{>}_{\mathcal{H}}={\sum }_{k=1}^p\int f^{(k)}(t)g^{(k)}(t)dt$ and $\Vert \mathbf{f}{\Vert }_{\mathcal{H}}=<\mathbf{f},\mathbf{f}{>}_{\mathcal{H}}^{1/2}$. Let ${\left\{x^{(k)}\right\}}_{k=1,\text{…},p}$ be a set of p random functions with each function in $L^2(\mathcal{T})$. Assume that the p‐dimensional vector $\mathbf{x}(t)={\left(x^{(1)},\text{…},x^{(p)}\right)}^{\top }\in {\mathbb{R}}^p$ has a p‐dimensional smooth mean function, $\mu (t)=\mathbb{E}\{\mathbf{x}(t)\}={\left(\mathbb{E}\left\{x^{(1)}(t)\right\},\text{…},\mathbb{E}\left\{x^{(p)}(t)\right\}\right)}^{\top }={\left({\mu }^{(1)}(t),\text{…},{\mu }^{(p)}(t)\right)}^{\top }$. Define the covariance function as $\mathbf{C}(s,t)=\mathbb{E}\left\{(\mathbf{x}(s)-\mu (s)){(\mathbf{x}(t)-\mu (t))}^{\top }\right\}={\left[C_{k{k}^{\prime }}(s,t)\right]}_{1\le k,k^{\prime }\le p}$ and $C_{k{k}^{\text{'}}}(s,t)=\text{Cov}\left\{x^{\left(k\right)}\left(s\right),x^{\left(k^{\text{'}}\right)}\left(t\right)\right\}$. Then, the covariance operator $\Gamma :\mathcal{H}\to \mathcal{H}$ associated with the kernel C(s,t) can be defined such that for any $\mathbf{f}\in \mathcal{H}$, the kth element of Γf is given by

$${(\Gamma \mathbf{f})}^{(k)}(s)=<{\mathbf{C}}_k(s,·),\mathbf{f}{>}_{\mathcal{H}}={\displaystyle \sum _{k^{\prime }=1}^p}\int C_{k{k}^{\prime }}(s,t)f^{(k^{\prime })}(t)dt,$$

where C _k (s,t)=(C _k1(s,t),…,C _kp (s,t))^⊤. Note that Γ is a linear, self‐adjoint, compact, and non‐negative integral operator. By the Hilbert–Schmidt theorem, there exists a set of orthonormal bases ${\{{\Psi }_{\ell }\}}_{\ell \ge 1}\in \mathcal{H}$, ${\Psi }_{\ell }={\left({\Psi }_{\ell }^{(1)},\text{…},{\Psi }_{\ell }^{(p)}\right)}^{\top }$, and $<{\Psi }_{\ell },{\Psi }_{{\ell }^{\prime }}{>}_{\mathcal{H}}={\sum }_{k=1}^p\int {\Psi }_{\ell }^{(k)}(t){\Psi }_{{\ell }^{\prime }}^{(k)}(t)dt=1_{\left\{\ell ={\ell }^{\prime }\right\}}$, such that

$${(\Gamma {\Psi }_{\ell })}^{(k)}(s)={\displaystyle \sum _{k^{\prime }=1}^p}\int C_{k{k}^{\prime }}(s,t){\Psi }_{\ell }^{(k^{\prime })}(t)dt=d_{\ell }{\Psi }_{\ell }^{(k)}(s),$$ (1)

where d _ℓ is the ℓth largest eigenvalue corresponding to Ψ _ℓ . Then, the multivariate Mercer's theorem gives

$$\mathbf{C}(s,t)={\displaystyle \sum _{\ell }^{\infty }}{d}_{\ell }{\Psi }_{\ell }(s){\Psi }_{\ell }^{\top }(t),$$ (2)

where $C_{k{k}^{\prime }}(s,t)={\sum }_{\ell =1}^{\infty }{d}_{\ell }{\Psi }_{\ell }^{(k)}(s){\Psi }_{\ell }^{(k^{\prime })}(t)$. As shown in Saporta (1981), x(t) has the multivariate Karhunen–Loève representation, $\mathbf{x}(t)=\mu (t)+{\sum }_{\ell =1}^{\infty }{\xi }_{\ell }{\Psi }_{\ell }(t)$, where ${\xi }_{\ell }=<\mathbf{x}-\mu ,{\Psi }_{\ell }{>}_{\mathcal{H}}$ are the scores with $\mathbb{E}({\xi }_{\ell })=0$ and $\mathbb{E}({\xi }_{\ell }{\xi }_{{\ell }^{\prime }})=d_{\ell }{1}_{\left\{\ell ={\ell }^{\prime }\right\}}$. The covariance operator Γ has the positive semidefiniteness property; that is, for any $\mathit{a}={(a_1,\text{…},a_p)}^{\top }\in {\mathbb{R}}^p$, the covariance function of a ^⊤ x, denoted by C _a (s,t), satisfies that for any sets of time points $(t_1,\text{…},t_q)\subset \mathcal{T}$ with an arbitrary positive integer q, the square matrix ${[C_{\mathit{a}}(t_i,t_j)]}_{\{1\le i,j\le q\}}\in {\mathbb{R}}^{q\times q}$ is positive semidefinite.

2.2. Covariance estimation by bivariate penalized splines

Suppose that the observed data take the form $\left\{\left(y_{ij}^{(k)},t_{ij}^{(k)}\right):i=1,\text{…},n;k=1,\text{…},p;j=1,\text{…},m_{ik}\right\}$, where $t_{ij}^{(k)}\in \mathcal{T}$ is the observed time point, $y_{ij}^{(k)}$ is the observed kth response, n is the number of subjects, and m _ik is the number of observations for subject i's kth response. The model is

$$y_{ij}^{(k)}=x_i^{(k)}\left(t_{ij}^{(k)}\right)+{ϵ}_{ij}^{(k)}={\mu }^{(k)}\left(t_{ij}^{(k)}\right)+{\displaystyle \sum _{\ell =1}^{\infty }}{\xi }_{i\ell }{\Psi }_{\ell }^{(k)}\left(t_{ij}^{(k)}\right)+{ϵ}_{ij}^{(k)},$$ (3)

where ${\mathbf{x}}_i(t)={\left(x_i^{(1)}(t),\text{…},x_i^{(p)}(t)\right)}^{\top }\in \mathcal{H}$, ${ϵ}_{ij}^{(k)}$ are random noises with zero means and variances ${\sigma }_k^2$ and are independent across i,j, and k.

The goal is to estimate the covariance functions C _{kk ′} . We adopt a three‐step procedure. In the first step, empirical estimates of the covariance functions are constructed. Let $r_{ij}^{(k)}=y_{ij}^{(k)}-{\mu }^{(k)}\left(t_{ij}^{(k)}\right)$ be the residuals and $C_{i{j}_1{j}_2}^{(k{k}^{\prime })}=r_{i{j}_1}^{(k)}{r}_{i{j}_2}^{(k^{\prime })}$ be the auxiliary variables. Note that $\mathbb{E}\left(C_{i{j}_1{j}_2}^{(k{k}^{\prime })}\right)=C_{k{k}^{\prime }}\left(t_{i{j}_1}^{(k)},t_{i{j}_2}^{(k^{\prime })}\right)+{\sigma }_k^2{1}_{\{k=k^{\prime },j_1=j_2\}}$ for 1 ≤ j ₁ ≤ m _ik ,1 ≤ j ₂ ≤ m _{ik ′} . Thus, $C_{i{j}_1{j}_2}^{(kk\prime )}$ is an unbiased estimate of $C_{k{k}^{\prime }}\left(t_{i{j}_1}^{(k)},t_{i{j}_2}^{(k^{\prime })}\right)$ whenever k≠k ′ or j ₁≠j ₂. In the second step, the noisy auxiliary variables are smoothed to obtain smooth estimates of the covariance functions. For smoothing, we use bivariate P‐splines (Eilers & Marx, 2003) because it is an automatic smoother and is computationally simple. In the final step, we pool all estimates of the individual covariance functions and use an extra step of eigendecomposition to obtain refined estimates of covariance functions. The refined estimates lead to a covariance operator that is properly defined, that is, positive semidefinite. In practice, the mean functions μ ^(k)s are unknown, and we estimate them using P‐splines (Eilers & Marx, 1996) with the smoothing parameters selected by leave‐one‐subject‐out cross‐validation; see the Supporting Information for details. Denote the estimates by ${\widehat{\mu }}^{(k)}$. Let ${\widehat{r}}_{ij}^{(k)}=y_{ij}^{(k)}-{\widehat{\mu }}^{(k)}\left(t_{ij}^{(k)}\right)$ and ${Ĉ}_{i{j}_1{j}_2}^{(k{k}^{\prime })}={\widehat{r}}_{i{j}_1}^{(k)}{\widehat{r}}_{i{j}_2}^{(k^{\prime })}$, the actual auxiliary variables.

The bivariate P‐splines model $C_{k{k}^{\prime }}(s,t)$ uses tensor‐product splines $G_{k{k}^{\prime }}(s,t)$ for 1 ≤ k,k ′ ≤ p. Specifically, $G_{k{k}^{\prime }}(s,t)={\sum }_{1\le {\gamma }_1,{\gamma }_2\le c}{\theta }_{{\gamma }_1{\gamma }_2}^{(k{k}^{\prime })}{B}_{{\gamma }_1}(s)B_{{\gamma }_2}(t)$, where ${\Theta }_{k{k}^{\prime }}={\left[{\theta }_{{\gamma }_1{\gamma }_2}^{(k{k}^{\prime })}\right]}_{1\le {\gamma }_1,{\gamma }_2\le c}\in {\mathbb{R}}^{c\times c}$ is a coefficient matrix, {B ₁(·),…,B _c (·)} is the collection of B‐spline basis functions in $\mathcal{T}$, and c is the number of equally spaced interior knots plus the order (degree plus 1) of the B‐splines. Because $C_{k{k}^{\prime }}(s,t)=C_{k^{\prime }k}(t,s)=\text{Cov}\left\{x^{(k)}(s),x^{(k^{\prime })}(t)\right\}$, it is reasonable to impose the assumption that

$${\Theta }_{k{k}^{\text{'}}}={\Theta }_{k^{\text{'}}k}^{\top },$$

so that G _{kk ′} (s,t)=G _{k ′ k} (t,s). Therefore, in the rest of the section, we consider only k ≤ k ′.

Let $\mathbf{D}\in {\mathbb{R}}^{(c-2)\times c}$ denote a second‐order differencing matrix such that for a vector $\mathbf{a}={(a_1,\text{…},a_c)}^{\top }\in {\mathbb{R}}^c$, $\mathbf{D}\mathbf{a}={(a_3-2a_2+a_1,a_4-2a_3+a_2,\text{…},a_c-2a_{c-1}+a_{c-2})}^{\top }\in {\mathbb{R}}^{c-2}$. Also let ‖·‖ _F be the Frobenius norm. For the cross‐covariance function $C_{k{k}^{\prime }}(s,t)$ with k<k ′, the bivariate P‐splines estimate the coefficient matrix Θ _{kk ′} by ${\widehat{\Theta }}_{k{k}^{\prime }}$, which minimizes the penalized least squares

$${\displaystyle \sum _{i=1}^n}\sum _{1\le j_1\le m_{ik}}\sum _{1\le j_2\le m_{i{k}^{\prime }}}{\left\{G_{k{k}^{\prime }}\left(t_{i{j}_1}^{(k)},t_{i{j}_2}^{(k^{\prime })}\right)-{Ĉ}_{i{j}_1{j}_2}^{(k{k}^{\prime })}\right\}}^2+{\lambda }_{k{k}^{\prime }1}\Vert \mathbf{D}{\Theta }_{k{k}^{\prime }}{\Vert }_F^2+{\lambda }_{k{k}^{\prime }2}\Vert \mathbf{D}{\Theta }_{k{k}^{\prime }}^{\top }{\Vert }_F^2,$$ (4)

where ${\lambda }_{k{k}^{\prime }1}$ and ${\lambda }_{k{k}^{\prime }2}$ are two non‐negative smoothing parameters that balance the model fit and smoothness of the estimate and will be determined later. Indeed, the column penalty $\Vert \mathbf{D}{\Theta }_{k{k}^{\prime }}{\Vert }_F^2$ penalizes the second‐order consecutive differences of the columns of Θ _{kk ′} and similarly, the row penalty $\Vert \mathbf{D}{\Theta }_{k{k}^{\prime }}^{\top }{\Vert }_F^2$ penalizes the second‐order consecutive differences of the rows of Θ _{kk ′} . The two penalty terms are essentially penalizing the second‐order partial derivatives of G _{kk ′} (s,t) along the s and t directions. The two smoothing parameters are allowed to differ to accommodate different levels of smoothing along the two directions.

For the auto‐covariance functions C _kk (s,t) with k=1,…,p, we conduct bivariate covariance smoothing by enforcing the following constraint on the coefficient matrix Θ _kk (Xiao et al., 2018):

$${\Theta }_{kk}={\Theta }_{kk}^{\top }.$$ (5)

It follows that G _kk (s,t) is a symmetric function. Then, the coefficient matrix Θ _kk and the error variance ${\sigma }_k^2$ are jointly estimated by ${\widehat{\Theta }}_{kk}$ and ${\widehat{\sigma }}_k^2$, which minimize the penalized least squares

$${\displaystyle \sum _{i=1}^n}\sum _{1\le j_1,j_2\le m_{ik}}{\left\{G_{kk}\left(t_{i{j}_1}^{(k)},t_{i{j}_2}^{(k)}\right)+{\sigma }_k^2{1}_{\{j_1=j_2\}}-{Ĉ}_{i{j}_1{j}_2}^{(kk)}\right\}}^2+{\lambda }_k\Vert \mathbf{D}{\Theta }_{kk}{\Vert }_F^2,$$ (6)

over all symmetric Θ _kk and λ _k is a smoothing parameter. Note that the two penalty terms in Equation (4) become the same when Θ _kk is symmetric, and thus, only one smoothing parameter is needed for auto‐covariance estimation.

2.2.1. Estimation

We first introduce the notation. Let vec(·) be an operator that stacks the columns of a matrix into a column vector and denote by ⊗ the Kronecker product. Fix k and k ′ with k ≤ k ′. Let ${\theta }_{k{k}^{\prime }}=\text{vec}({\Theta }_{k{k}^{\prime }})\in {\mathbb{R}}^{c^2}$ be a vector of the coefficients and $\mathbf{b}(t)={\{B_1(t),\text{…},B_c(t)\}}^{\top }\in {\mathbb{R}}^c$ denotes the B‐spline base. Then,

$$G_{k{k}^{\prime }}(s,t)=\mathbf{b}{(s)}^{\top }{\Theta }_{k{k}^{\prime }}\mathbf{b}(t)={\{\mathbf{b}(t)\otimes \mathbf{b}(s)\}}^{\top }{\theta }_{k{k}^{\prime }}.$$

We now organize the auxiliary responses ${Ĉ}_{i{j}_1{j}_2}^{(kk\prime )}$ for each pair of k and k ′. Let ${\mathbf{r}}_i^{(k)}={\left(r_{i1}^{(k)},\text{…},r_{i{m}_{ik}}^{(k)}\right)}^{\top }\in {\mathbb{R}}^{m_{ik}}$, ${\widehat{\mathbf{C}}}_i^{(k{k}^{\prime })}={\mathbf{r}}_i^{(k)}\otimes {\mathbf{r}}_i^{(k^{\prime })}\in {\mathbb{R}}^{m_{ik}{m}_{i{k}^{\prime }}}$ and ${\widehat{\mathbf{C}}}^{(k{k}^{\prime })}={\left({\widehat{\mathbf{C}}}_1^{(k{k}^{\prime }),\top },\text{…},{\widehat{\mathbf{C}}}_n^{(k{k}^{\prime }),\top }\right)}^{\top }\in {\mathbb{R}}^{N_{k{k}^{\prime }}}$, where $N_{k{k}^{\prime }}={\sum }_{i=1}^n{m}_{ik}{m}_{i{k}^{\prime }}$ is the total number of auxiliary responses for the pair of k and k ′. As for the B‐splines, let ${\mathbf{b}}_i^{(k)}=\left[\mathbf{b}\left(t_{i1}^{(k)}\right),\text{…},\mathbf{b}\left(t_{i{m}_{ik}}^{(k)}\right)\right]\in {\mathbb{R}}^{c\times m_{ik}}$, ${\mathbf{B}}_i^{(kk\prime )}={\left({\mathbf{b}}_i^{(k\prime )}\otimes {\mathbf{b}}_i^{(k)}\right)}^{\top }\in {\mathbb{R}}^{(m_{ik}{m}_{ik\prime })\times c^2}$, and ${\mathbf{B}}^{(kk\prime )}={\left[{\mathbf{B}}_1^{(kk\prime ),\top },\text{…},{\mathbf{B}}_n^{(kk\prime ),\top }\right]}^{\top }\in {\mathbb{R}}^{N_{kk\prime }\times c^2}$.

For estimation of the cross‐covariance functions C _{kk ′} with k<k ′, the penalized least squares in Equation (4) can be rewritten as

$${\left({\widehat{\mathbf{C}}}^{(k{k}^{\prime })}-{\mathbf{B}}^{(k{k}^{\prime })}{\theta }_{k{k}^{\prime }}\right)}^{\top }\left({\widehat{\mathbf{C}}}^{(k{k}^{\prime })}-{\mathbf{B}}^{(k{k}^{\prime })}{\theta }_{k{k}^{\prime }}\right)+{\lambda }_{k{k}^{\prime }1}{\theta }_{k{k}^{\prime }}^{\top }{\mathbf{P}}_1{\theta }_{k{k}^{\prime }}+{\lambda }_{k{k}^{\prime }2}{\theta }_{k{k}^{\prime }}^{\top }{\mathbf{P}}_2{\theta }_{k{k}^{\prime }},$$ (7)

where ${\mathbf{P}}_1={\mathbf{I}}_c\otimes {\mathbf{D}}^{\top }\mathbf{D}$ and ${\mathbf{P}}_2={\mathbf{D}}^{\top }\mathbf{D}\otimes {\mathbf{I}}_c$. The expression in Equation (7) is a quadratic function of the coefficient vector θ _{kk ′} . Therefore, we derive that

$${\widehat{\theta }}_{k{k}^{\prime }}={\left({\mathbf{B}}^{(k{k}^{\prime }),\top }{\mathbf{B}}^{(k{k}^{\prime })}+{\lambda }_{k{k}^{\prime }1}{\mathbf{P}}_1+{\lambda }_{k{k}^{\prime }2}{\mathbf{P}}_2\right)}^{-1}{\mathbf{B}}^{(k{k}^{\prime }),\top }{\widehat{\mathbf{C}}}^{(k{k}^{\prime })}$$

and the estimate of the cross‐covariance function C _{kk ′} (s,t) is ${Ĉ}_{k{k}^{\prime }}(s,t)={\{\mathbf{b}(t)\otimes \mathbf{b}(s)\}}^{\top }{\widehat{\theta }}_{k{k}^{\prime }}$.

For estimation of the auto‐covariance functions, because of the constraint on the coefficient matrix in Equation  (5), let ${\chi }_k\in {\mathbb{R}}^{c(c+1)/2}$ be a vector obtained by stacking the columns of the lower triangle of Θ _kk , and let ${\mathbf{G}}_c\in {\mathbb{R}}^{c^2\times c(c+1)/2}$ be a duplication matrix such that θ _kk =G _c η _k (Seber, 2008, p. 246).

Let ${\mathbf{Z}}_i^{(k)}=\text{vec}({\mathbf{I}}_{m_{ik}})\in {\mathbb{R}}^{m_{ik}^2}$ and ${\mathbf{Z}}^{(k)}={\left({\mathbf{Z}}_1^{(k),\top },\text{…},{\mathbf{Z}}_n^{(k),\top }\right)}^{\top }\in {\mathbb{R}}^{N_{kk}}$. Finally, let ${\beta }_k={\left({\chi }_k^{\top },{\sigma }_k^2\right)}^{\top }\in {\mathbb{R}}^{\tilde{c}}$ with $\tilde{c}=c(c+1)/2+1$. It follows that the penalized least squares in Equation (6) can be rewritten as

$${\left({\widehat{\mathbf{C}}}^{(kk)}-{\mathbf{X}}^{(k)}{\beta }_k\right)}^{\top }\left({\widehat{\mathbf{C}}}^{(kk)}-{\mathbf{X}}^{(k)}{\beta }_k\right)+{\lambda }_k{\beta }_k^{\top }\mathbf{Q}{\beta }_k,$$

where ${\mathbf{X}}^{(k)}=\left[{\mathbf{B}}^{(kk)},{\mathbf{Z}}^{(k)}\right]\in {\mathbb{R}}^{N_{kk}\times \tilde{c}}$ and $\mathbf{Q}=\text{blockdiag}\left\{{\mathbf{G}}_c^{\top }({\mathbf{I}}_c\otimes \mathbf{D}{\mathbf{D}}^{\top }){\mathbf{G}}_c^{\top },0\right\}\in {\mathbb{R}}^{\tilde{c}\times \tilde{c}}$. Therefore, we obtain

$${\widehat{\beta }}_k={\left({\widehat{\chi }}_k^{\top },{\widehat{\sigma }}_k^2\right)}^{\top }={\left({\mathbf{X}}^{(k),\top }{\mathbf{X}}^{(k)}+{\lambda }_k\mathbf{Q}\right)}^{-1}{\mathbf{X}}^{(k),\top }{\widehat{\mathbf{C}}}^{(kk)}.$$

It follows that ${\widehat{\theta }}_{kk}={\mathbf{G}}_c{\widehat{\chi }}_k$, and the estimate of the auto‐covariance function C _kk (s,t) is ${Ĉ}_{kk}(s,t)={\{\mathbf{b}(t)\otimes \mathbf{b}(s)\}}^{\top }{\widehat{\theta }}_{kk}$.

The above estimates of covariance functions may not lead to a positive semidefinite covariance operator and thus have to be refined. We pool all estimates together, and we shall use the following proposition.

Proposition 1

Assume that $C_{k{k}^{\prime }}(s,t)=\mathbf{b}{(s)}^{\top }{\Theta }_{k{k}^{\prime }}\mathbf{b}(t)$. Let $\mathbf{G}=\int \mathbf{b}(t)\mathbf{b}{(t)}^{\top }dt\in {\mathbb{R}}^{c\times c}$ and assume that G is positive definite (Zhou, Shen, & Wolfe, 1998). Then, ${\left[{\mathbf{G}}^{\frac{1}{2}}{\Theta }_{k{k}^{\prime }}{\mathbf{G}}^{\frac{1}{2}}\right]}_{1\le k,k^{\prime }\le p}\in {\mathbb{R}}^{pc\times pc}$ admits the spectral decomposition, ${\sum }_{\ell =1}^{\infty }{d}_{\ell }{\mathbf{u}}_{\ell }{\mathbf{u}}_{\ell }^{\top }$, where d _ℓ is the ℓth largest eigenvalue of the covariance operator Γ, and ${\mathbf{u}}_{\ell }={\left\{{\mathbf{u}}_{\ell }^{(1),\top },\text{…},{\mathbf{u}}_{\ell }^{(p),\top }\right\}}^{\top }\in {\mathbb{R}}^{pc}$ is the associated eigenvector with ${\mathbf{u}}_{\ell }^{(k)}\in {\mathbb{R}}^c$ and such that ${\Psi }_{\ell }^{(k)}(t)=\mathbf{b}{(t)}^{\top }{\mathbf{G}}^{-\frac{1}{2}}{\mathbf{u}}_{\ell }^{(k)}$.

The proof is provided in Appendix A. Proposition 1 implies that, with the tensor‐product B‐spline representation of the covariance functions, one spectral decomposition gives us the eigenvalues and eigenfunctions. In particular, the eigenfunctions ${\Psi }_{\ell }^{(k)}(t)$ are linear combinations of the B‐spline basis functions, which means that they can be straightforwardly evaluated, an advantage of spline‐based methods compared with other smoothing methods for which eigenfunctions are approximated by spectral decompositions of the covariance functions evaluated at a grid of time points.

Once we have ${\widehat{\Theta }}_{k{k}^{\prime }}$, the estimate of the coefficient matrix Θ _{kk ′} , the spectral decomposition of ${[{\mathbf{G}}^{\frac{1}{2}}{\widehat{\Theta }}_{k{k}^{\prime }}{\mathbf{G}}^{\frac{1}{2}}]}_{k{k}^{\prime }}$ gives us estimates ${\widehat{d}}_{\ell }$ and ${\widehat{\mathbf{u}}}_{\ell }={\left\{{\widehat{\mathbf{u}}}_{\ell }^{(1),\top },\text{…},{\widehat{\mathbf{u}}}_{\ell }^{(p),\top }\right\}}^{\top }$. We discard negative ${\widehat{d}}_{\ell }$ to ensure that the multivariate covariance operator is positive semidefinite, and this leads to a refined estimate of the coefficient matrix Θ _{kk ′} , ${\tilde{\Theta }}_{k{k}^{\prime }}={\mathbf{G}}^{-\frac{1}{2}}\left\{{\sum }_{\ell :{\widehat{d}}_{\ell }>0}{\widehat{d}}_{\ell }{\widehat{\mathbf{u}}}_{\ell }^{(k)}{\widehat{\mathbf{u}}}_{\ell }^{(k^{\prime }),\top }\right\}{\mathbf{G}}^{-\frac{1}{2}}$. Then, the refined estimate of the covariance functions is ${\tilde{C}}_{k{k}^{\prime }}(s,t)=\mathbf{b}{(s)}^{\top }{\tilde{\Theta }}_{k{k}^{\prime }}\mathbf{b}(t)$. Proposition 1 also suggests that the eigenfunctions can be estimated by ${\tilde{\Psi }}_{\ell }^{(k)}(t)=\mathbf{b}{(t)}^{\top }{\mathbf{G}}^{-\frac{1}{2}}{\widehat{\mathbf{u}}}_{\ell }^{(k)}$.

For principal component analysis or curve prediction in practice, one may select further the number of principal components by either the proportion of variance explained (PVE) (Greven, Crainiceanu, Caffo, & Reich, 2010) or an Akaike information criterion‐type criterion (Li, Wang, & Carroll, 2013). Here, we follow Greven et al. (2010) using PVE with a value of 0.99.

2.2.2. Selection of smoothing parameters

We select the smoothing parameters in each auto‐covariance/cross‐covariance estimation using leave‐one‐subject‐out cross‐validation; see, for example, Yao et al. (2005) and Xiao et al. (2018). A fast approximate algorithm for the auto‐covariance has been derived in Xiao et al. (2018). So we focus on the cross‐covariance and use the notation in Equation (7). Note that there are two smoothing parameters for each cross‐covariance estimation.

For simplicity, we suppress the superscript and subscript kk ′ in Equation (7) for both $\widehat{\mathbf{C}}$ and B. Let ${\tilde{\mathbf{C}}}_i^{[i]}$ be the prediction of the auxiliary responses ${\widehat{\mathbf{C}}}_i$ from the estimate using data without the ith subject. Let ‖·‖ be the Euclidean norm, and the cross‐validation error is

$$\text{iCV}={\displaystyle \sum _{i=1}^n}{‖{\widehat{\mathbf{C}}}_i-{\tilde{\mathbf{C}}}_i^{[i]}‖}^2.$$ (8)

We shall also now suppress the subscript k from m _ik and kk ′ from N _{kk ′} . Let $\mathbf{S}=\mathbf{B}{({\mathbf{B}}^{\top }\mathbf{B}+{\lambda }_1{\mathbf{P}}_1+{\lambda }_2{\mathbf{P}}_2)}^{-1}{\mathbf{B}}^{\top }\in {\mathbb{R}}^{N\times N}$, ${\mathbf{S}}_i={\mathbf{B}}_i{({\mathbf{B}}^{\top }\mathbf{B}+{\lambda }_1{\mathbf{P}}_1+{\lambda }_2{\mathbf{P}}_2)}^{-1}{\mathbf{B}}^{\top }\in {\mathbb{R}}^{m_i^2\times N}$, and ${\mathbf{S}}_{ii}={\mathbf{B}}_i{({\mathbf{B}}^{\top }\mathbf{B}+{\lambda }_1{\mathbf{P}}_1+{\lambda }_2{\mathbf{P}}_2)}^{-1}{\mathbf{B}}_i^{\top }\in {\mathbb{R}}^{m_i^2\times m_i^2}$.

Then, a shortcut formula for Equation (8) is

$$\text{iCV}={\displaystyle \sum _{i=1}^n}{‖{\left({\mathbf{I}}_{m_i^2}-{\mathbf{S}}_{ii}\right)}^{-1}({\mathbf{S}}_i\widehat{\mathbf{C}}-{\widehat{\mathbf{C}}}_i)‖}^2.$$

Similar to Xu and Huang (2012) and Xiao et al. (2018), the iCV can be further simplified by adopting the approximation ${({\mathbf{I}}_{m_i^2}-{\mathbf{S}}_{ii})}^{-2}={\mathbf{I}}_{m_i^2}+2{\mathbf{S}}_{ii}$, which results in the generalized cross‐validation, denoted by iGCV,

$$\text{iGCV}={‖\widehat{\mathbf{C}}-\mathbf{S}\widehat{\mathbf{C}}‖}^2+2{\displaystyle \sum _{i=1}^n}{\left({\mathbf{S}}_i\widehat{\mathbf{C}}-{\widehat{\mathbf{C}}}_i\right)}^{\top }{\mathbf{S}}_{ii}\left({\mathbf{S}}_i\widehat{\mathbf{C}}-{\widehat{\mathbf{C}}}_i\right).$$ (9)

Although iGCV is much easier to compute than iCV, the formula in Equation (9) is still computationally expensive to compute. Indeed, the smoother matrix S is of dimension 2,500×2,500 if n=100 and m _i =m=5 for all i. Thus, we need to further simplify the formula.

Let ${\mathbf{G}}_n={\mathbf{B}}^{\top }\mathbf{B}$, $\tilde{\mathbf{B}}=\mathbf{B}{\mathbf{G}}_n^{-1/2}\in {\mathbb{R}}^{N\times c^2}$, ${\tilde{\mathbf{B}}}_i={\mathbf{B}}_i{\mathbf{G}}_n^{-1/2}\in {\mathbb{R}}^{m_i^2\times c^2}$, $\mathbf{f}={\tilde{\mathbf{B}}}^{\top }\widehat{\mathbf{C}}\in {\mathbb{R}}^{c^2}$, ${\mathbf{f}}_i={\tilde{\mathbf{B}}}_i^{\top }{\widehat{\mathbf{C}}}_i\in {\mathbb{R}}^{c^2}$, and ${\mathbf{L}}_i={\tilde{\mathbf{B}}}_i^{\top }{\tilde{\mathbf{B}}}_i\in {\mathbb{R}}^{c^2\times c^2}$. Also let ${\tilde{\mathbf{P}}}_1={\mathbf{G}}_n^{-1/2}{\mathbf{P}}_1{\mathbf{G}}_n^{-1/2}\in {\mathbb{R}}^{c^2\times c^2}$, ${\tilde{\mathbf{P}}}_2={\mathbf{G}}_n^{-1/2}{\mathbf{P}}_2{\mathbf{G}}_n^{-1/2}\in {\mathbb{R}}^{c^2\times c^2}$, and $\sum ={\mathbf{I}}_{c^2}+{\lambda }_1{\tilde{\mathbf{P}}}_1+{\lambda }_2{\tilde{\mathbf{P}}}_2$. Then, Equation (9) can be simplified as

$$\text{iGCV}={‖\widehat{\mathbf{C}}‖}^2-2{\mathbf{f}}^{\top }{\sum }^{-1}\mathbf{f}+{\mathbf{f}}^{\top }{\sum }^{-2}\mathbf{f}+2{\displaystyle \sum _{i=1}^n}{\left({\mathbf{L}}_i{\sum }^{-1}\mathbf{f}-{\mathbf{f}}_i\right)}^{\top }{\sum }^{-1}\left({\mathbf{L}}_i{\sum }^{-1}\mathbf{f}-{\mathbf{f}}_i\right).$$ (10)

Note that Σ has two smoothing parameters. Following Wood (2000), we use an equivalent parameterization $\sum =\mathbf{I}+\rho \{w{\tilde{\mathbf{P}}}_1+(1-w){\tilde{\mathbf{P}}}_2\}$, where ρ=λ ₁+λ ₂ represents the overall smoothing level and w=λ ₁ ρ ⁻¹∈[0,1] is the relative weight of λ ₁. We conduct a two‐dimensional grid search of (ρ,w), as follows. For a given w, let Udiag(s)U ^⊤ be the eigendecompsition of $w{\tilde{\mathbf{P}}}_1+(1-w){\tilde{\mathbf{P}}}_2$, where $\mathbf{U}\in {\mathbb{R}}^{c^2\times c^2}$ is an orthonormal matrix and $\mathbf{s}=(s_1,\text{…},s_{c^2})\in {\mathbb{R}}^{c^2}$ is the vector of eigenvalues. Then, ${\sum }^{-1}=\mathbf{U}\text{diag}(\tilde{\mathbf{d}}){\mathbf{U}}^{\top }$ with $\tilde{\mathbf{d}}=1/(1+\rho \mathbf{s})\in {\mathbb{R}}^{c^2}$.

Proposition 2

Let ⊙ stand for the point‐wise multiplication. Then,

$$\text{iGCV}={‖\widehat{\mathbf{C}}‖}^2+{(\tilde{\mathbf{f}}\odot \tilde{\mathbf{d}})}^{\top }(\tilde{\mathbf{f}}\odot \tilde{\mathbf{d}})-2{\tilde{\mathbf{d}}}^{\top }\mathbf{g}-4{\tilde{\mathbf{d}}}^{\top }\mathbf{F}\tilde{\mathbf{d}}+2{\tilde{\mathbf{d}}}^{\top }\left[{\displaystyle \sum _{i=1}^n}\left\{{\tilde{\mathbf{L}}}_i(\tilde{\mathbf{f}}\odot \tilde{\mathbf{d}})\right\}\odot \left\{{\tilde{\mathbf{L}}}_i(\tilde{\mathbf{f}}\odot \tilde{\mathbf{d}})\right\}\right],$$

where ${\tilde{\mathbf{f}}}_i={\mathbf{U}}^{\top }{\mathbf{f}}_i\in {\mathbb{R}}^{c^2}$, $\tilde{\mathbf{f}}={\mathbf{U}}^{\top }\mathbf{f}\in {\mathbb{R}}^{c^2}$, $\mathbf{g}=\tilde{\mathbf{f}}\odot \tilde{\mathbf{f}}-{\sum }_{i=1}^n{\tilde{\mathbf{f}}}_i\odot {\tilde{\mathbf{f}}}_i\in {\mathbb{R}}^{c^2}$, ${\tilde{\mathbf{L}}}_i={\mathbf{U}}^{\top }{\mathbf{L}}_i\mathbf{U}\in {\mathbb{R}}^{c^2\times c^2}$, and $\mathbf{F}={\sum }_{i=1}^n({\tilde{\mathbf{f}}}_i{\tilde{\mathbf{f}}}^{\top })\odot {\tilde{\mathbf{L}}}_i\in {\mathbb{R}}^{c^2\times c^2}$.

The proof is provided in Appendix A. For each w, note that only $\tilde{\mathbf{d}}$ depends on ρ and needs to be calculated repeatedly, and all other terms need to be calculated only once. The entire algorithm is presented in Algorithm 1. We give an evaluation of the complexity of the proposed algorithm. Assume that m _i =m for all i. The first initialization (Step 1) requires O(nm ² c ²+nc ⁴+c ⁶) computations. For each w, the second initialization (Step 2) also requires $O\{n{c}^4\min (m^2,c^2)+c^6\}$ computations. For each ρ, Steps 3–8 require O(nc ⁴) computations. Therefore, the formula in Proposition 2 is most efficient to calculate for sparse data with small numbers of observations per subject; that is, m _i s are small.

2.3. Prediction

For prediction, assume that the smooth curve x _i (t) is generated from a multivariate Gaussian process. Suppose that we want to predict the ith multivariate response x _i (t) at {s _i1,…,s _im } for m ≥ 1. Let ${\mathbf{y}}_i^{(k)}={\left(y_{i1}^{(k)},\text{…},y_{i{m}_{ik}}^{(k)}\right)}^{\top }$ be the vector of observations at $\left\{t_{i1}^{(k)},\text{…},t_{i{m}_{ik}}^{(k)}\right\}$ for the kth response. Let ${\mu }_i^{(k),o}={\left({\mu }^{(k)}\left(t_{i1}^{(k)}\right),\text{…},{\mu }^{(k)}\left(t_{i{m}_{ik}}^{(k)}\right)\right)}^{\top }$ be the vector of the kth mean function at the observed time points. Let ${\mathbf{y}}_i={\left({\mathbf{y}}_i^{(1),\top },\text{…},{\mathbf{y}}_i^{(p),\top }\right)}^{\top }$ and ${\mu }_i^o={\left({\mu }_i^{(1),o,\top },\text{…},{\mu }_i^{(p),o,\top }\right)}^{\top }$. Let ${\mu }_i^n={\left({\mu }^{(1)}(s_{i1}),\text{…},{\mu }^{(1)}(s_{im}),\text{…},{\mu }^{(p)}(s_{i1}),\text{…},{\mu }^{(p)}(s_{im})\right)}^{\top }$ be the vector of mean functions at the time points for prediction.

It follows that

$$\left(\begin{array}{c}{\mathbf{y}}_i\\ {\mathbf{x}}_i\end{array}\right)\sim \mathcal{N}\left\{\left(\begin{array}{c}{\mu }_i^o\\ {\mu }_i^n\end{array}\right),\left(\begin{array}{cc}\text{Cov}({\mathbf{y}}_i)& \text{Cov}{({\mathbf{x}}_i,{\mathbf{y}}_i)}^{\top }\\ \text{Cov}({\mathbf{x}}_i,{\mathbf{y}}_i)& \text{Cov}({\mathbf{x}}_i)\end{array}\right)\right\}.$$

Thus, we obtain

$$\mathbb{E}\left({\mathbf{x}}_i\right|{\mathbf{y}}_i)=\text{Cov}({\mathbf{x}}_i,{\mathbf{y}}_i\left)\text{Cov}{\left({\mathbf{y}}_i\right)}^{-1}\right({\mathbf{y}}_i-{\mu }_i^o)+{\mu }_i^n,$$

$$\text{Cov}\left({\mathbf{x}}_i\right|{\mathbf{y}}_i)=\text{Cov}({\mathbf{x}}_i)-\text{Cov}({\mathbf{x}}_i,{\mathbf{y}}_i)\text{Cov}{\left({\mathbf{y}}_i\right)}^{-1}\text{Cov}{({\mathbf{x}}_i,{\mathbf{y}}_i)}^{\top }.$$

Let ${\mathbf{b}}_i^{(k),o}={\left[\mathbf{b}\left(t_{i1}^{(k)}\right),\text{…},\mathbf{b}\left(t_{i{m}_{ik}}^{(k)}\right)\right]}^{\top }$ and ${\mathbf{b}}_i^n={\left[\mathbf{b}\left(s_{i1}\right),\text{…},\mathbf{b}\left(s_{im}\right)\right]}^{\top }$. Next, let ${\mathbf{B}}_i^o=\text{blockdiag}\left({\mathbf{b}}_i^{(1),o},\text{…},{\mathbf{b}}_i^{(p),o}\right)$ and ${\mathbf{B}}_i^n={\mathbf{I}}_p\otimes {\mathbf{b}}_i^n$. Then, Cov(y _i ) given by ${\mathbf{B}}_i^o\Theta {\mathbf{B}}_i^{o,\top }+\text{blockdiag}({\sigma }_1^2{\mathbf{I}}_{m_{i1}},\text{…},{\sigma }_p^2{\mathbf{I}}_{m_{ip}})$, and Cov(x _i ) and Cov(y _i ,x _i ) are given by ${\mathbf{B}}_i^n\Theta {\mathbf{B}}_i^{n,\top }$ and ${\mathbf{B}}_i^o\Theta {\mathbf{B}}_i^{n,\top }$, respectively. Let $\tilde{\Theta }={\left[{\tilde{\Theta }}_{k{k}^{\prime }}\right]}_{1\le k,k^{\prime }\le p}\in {\mathbb{R}}^{pc\times pc}$. Plugging in the estimates, we predict x _i by

$$\begin{array}{ccc}\hfill {\widehat{\mathbf{x}}}_i& ={\left({\widehat{x}}_i^{(1)}(s_{i1}),\text{…},{\widehat{x}}_i^{(1)}(s_{im}),\text{…},{\widehat{x}}_i^{(p)}(s_{i1}),\text{…},{\widehat{x}}_i^{(p)}(s_{im})\right)}^{\top }\hfill & \hfill \\ \hfill & =\left({\mathbf{B}}_i^n\tilde{\Theta }{\mathbf{B}}_i^{o,\top }\right){\widehat{\mathbf{V}}}_i^{-1}({\mathbf{y}}_i-{\widehat{\mu }}_i^o)+{\widehat{\mu }}_i^n,\hfill \end{array}$$

where ${\widehat{\mu }}_i^o={\left({\widehat{\mu }}^{(1)}\left(t_{i1}^{(1)}\right),\text{…},{\widehat{\mu }}^{(1)}\left(t_{i{m}_{i1}}^{(1)}\right),\text{…},{\widehat{\mu }}^{(p)}\left(t_{i1}^{(p)}\right),\text{…},{\widehat{\mu }}^{(p)}\left(t_{i{m}_{ip}}^{(p)}\right)\right)}^{\top }$ is the estimate of ${\mu }_i^o$, ${\widehat{\mu }}_i^n={\left({\widehat{\mu }}^{(1)}(s_{i1}),\text{…},{\widehat{\mu }}^{(1)}(s_{im}),\text{…},{\widehat{\mu }}^{(p)}(s_{i1}),\text{…},{\widehat{\mu }}^{(p)}(s_{im})\right)}^{\top }$ is the estimate of ${\mu }_i^n$, ${\widehat{\mathbf{V}}}_i={\mathbf{B}}_i^o\tilde{\mathbf{\Theta }}{\mathbf{B}}_i^{o,\top }+\text{blockdiag}\left({\widehat{\sigma }}_1^2{\mathbf{I}}_{m_{i1}},\text{…},{\widehat{\sigma }}_p^2{\mathbf{I}}_{m_{ip}}\right)$. An approximate covariance matrix for ${\widehat{\mathbf{x}}}_i$ is

$$\widehat{\text{Cov}}\left({\mathbf{x}}_i\right|{\mathbf{y}}_i)={\mathbf{B}}_i^n\tilde{\Theta }{\mathbf{B}}_i^{n,\top }-\left({\mathbf{B}}_i^n\tilde{\Theta }{\mathbf{B}}_i^{o,\top }\right){\widehat{\mathbf{V}}}_i^{-1}{\left({\mathbf{B}}_i^n\tilde{\Theta }{\mathbf{B}}_i^{o,\top }\right)}^{\top }.$$

Therefore, a 95% point‐wise confidence interval for the kth response is given by

$${\widehat{x}}_i^{\left(k\right)}\left(s_{ij}\right)\pm 1.96\sqrt{\widehat{\mathrm{Var}}\left(x_i^{\left(k\right)}\left(s_{ij}\right)|{\mathbf{y}}_i\right)},$$

where $\widehat{\mathrm{Var}}\left(x_i^{\left(k\right)}\left(s_{ij}\right)|{\mathbf{y}}_i\right)$ can be extracted from the diagonal of $\widehat{\text{Cov}}\left({\mathbf{x}}_i\right|{\mathbf{y}}_i)$.

Finally, we predict the first L ≥ 1 scores ξ _i =(ξ _i1,…,ξ _iL )^⊤ for the ith subject. Note that ${\xi }_{i\ell }=\int {\mathbf{\Psi }}_{\ell }{(t)}^{\top }\{{\mathbf{x}}_i(t)-\mu (t)\}dt$. With a similar derivation as above, x _i (t)− μ (t) can be predicted by ${\left\{{\mathbf{I}}_p\otimes \mathbf{b}(t)\right\}}^{\top }\tilde{\Theta }{\mathbf{B}}_i^{o,\top }{\widehat{\mathbf{V}}}_i^{-1}({\mathbf{y}}_i-{\widehat{\mu }}_i^o)$. By Proposition 1, the eigenfunctions ${\Psi }_{\ell }^{(k)}(t)$ are estimated by $\mathbf{b}{(t)}^{\top }{\mathbf{G}}^{-\frac{1}{2}}{\widehat{\mathbf{u}}}_{\ell }^{(k)}$, and thus, ${\mathbf{\Psi }}_{\ell }{(t)}^{\top }={\widehat{\mathbf{u}}}_{\ell }^{\top }\{{\mathbf{I}}_p\otimes {\mathbf{G}}^{-\frac{1}{2}}\mathbf{b}(t)\}$. It follows that

$${\widehat{\xi }}_{i\ell }={\widehat{\mathbf{u}}}_{\ell }^{\top }\left({\mathbf{I}}_p\otimes {\mathbf{G}}^{\frac{1}{2}}\right)\tilde{\Theta }{\mathbf{B}}_i^{o,\top }{\widehat{\mathbf{V}}}_i^{-1}({\mathbf{y}}_i-{\widehat{\mu }}_i^o).$$

3. SIMULATIONS

We evaluate the finite sample performance of the proposed method (denoted by mFACEs) against mFPCA via a synthetic simulation study and a simulation study mimicking the Alzheimer's Disease Neuroimaging Initiative (ADNI) data in the real data example. Here, we report the details and results of the former as the conclusions remain the same for the latter, and details are provided in the Supporting Information.

3.1. Simulation settings and evaluation criteria

We generate data by model  (3) with p=3 responses. The mean functions are $\mu (t)={[5\sin (2\pi t),5\cos (2\pi t),5{(t-1)}^2]}^{\top }$. We first specify the auto‐covariance functions. Let ${\Phi }_1(t)={\left[\sqrt{2}\sin (2\pi t),\sqrt{2}\cos (4\pi t),\sqrt{2}\sin (4\pi t)\right]}^{\top }$, ${\Phi }_2(t)={\left[\sqrt{2}\cos (\pi t),\sqrt{2}\cos (2\pi t),\sqrt{2}\cos (3\pi t)\right]}^{\top }$, and ${\Phi }_3(t)={\left[\sqrt{2}\sin (\pi t),\sqrt{2}\sin (2\pi t),\sqrt{2}\sin (3\pi t)\right]}^{\top }$. Also let

$${\Lambda }_{11}=\left(\begin{array}{ccc}3& 0& 0\\ 0& 1.5& 0\\ 0& 0& 0.75\end{array}\right),{\Lambda }_{22}=\left(\begin{array}{ccc}3.5& 0& 0\\ 0& 1.75& 0\\ 0& 0& 0.5\end{array}\right),{\Lambda }_{33}=\left(\begin{array}{ccc}2.5& 0& 0\\ 0& 2& 0\\ 0& 0& 1\end{array}\right).$$

Then, the auto‐covariance functions are $C_{kk}(s,t)={\Phi }_k{(s)}^{\top }{\Lambda }_{kk}{\Phi }_k(t),k=1,2,3$. For the cross‐covariance functions, let $C_{k{k}^{\prime }}(s,t)=\rho {\Phi }_k{(s)}^{\top }{\Lambda }_{kk}^{\frac{1}{2}}{\Lambda }_{k^{\prime }{k}^{\prime }}^{\frac{1}{2}}{\Phi }_{k^{\prime }}(t)$ for k≠k ′, where ρ∈[0,1] is a parameter to be specified. The induced covariance operator from the above specifications is proper; see Lemma 1 in Appendix B. It is easy to derive that the absolute value of cross‐correlation ${\rho }_{k{k}^{\prime }}(s,t)=C_{k{k}^{\prime }}(s,t)/\sqrt{C_{kk}(s,s)C_{k^{\prime }{k}^{\prime }}(t,t)}$ is bounded by ρ. Hence, ρ controls the overall level of correlation between responses: if ρ=0, then the responses are uncorrelated from each other.

The eigendecomposition of the multivariate covariance function gives nine nonzero eigenvalues with associated multivariate eigenfunctions; hence, for ℓ=1,…,9, we simulate the scores ξ _iℓ from $\mathcal{N}(0,d_{\ell })$, where d _ℓ are the induced eigenvalues. Next, we simulate the white noises ${ϵ}_{ij}^{(k)}$ from $\mathcal{N}(0,{\sigma }_{ϵ}^2)$, where ${\sigma }_{ϵ}^2$ is determined according to the signal‐to‐noise ratio $\text{SNR}={\sum }_{\ell }{d}_{\ell }/(p{\sigma }_{ϵ}^2)$. Here, we let SNR=2. For each response, the sampling time points are drawn from a uniform distribution in the unit interval, and the number of observations for each subject, m _ik , is generated from a uniform discrete distribution on {3,4,5,6,7}. Thus, the sampling points vary not only from subject to subject but also across responses within each subject.

We use a factorial design with two factors: the number of subjects n and the correlation parameter ρ. We let n=100,200, or 400. We let ρ=0.5, which corresponds to a weak correlation between responses as the average absolute correlation between responses is only 0.36. Another value of ρ is 0.9, which corresponds to a moderate correlation between responses as the average absolute correlation between responses is about 0.50.

In total, we have six model conditions, and for each model condition, we generate 200 datasets. To evaluate the prediction accuracy of the various methods, we draw 200 additional subjects as testing data. The true correlation functions and a sample of the simulated data are shown in the Supporting Information.

We compare mFACEs and mFPCA in terms of estimation accuracy of the covariance functions, the eigenfunctions and eigenvalues, and prediction of new subjects. For covariance function estimation, we use the relative integrated square errors (RISE). Let ${Ĉ}_{k{k}^{\prime }}(s,t)$ be an estimate of $C_{k{k}^{\prime }}(s,t)$, and then RISE are given by

$$\frac{{\sum }_{k=1}^p{\sum }_{k^{\prime }=1}^p{\int }_0^1{\int }_0^1{\left\{C_{k{k}^{\prime }}(s,t)-{Ĉ}_{k{k}^{\prime }}(s,t)\right\}}^{2}dsdt}{{\sum }_{k=1}^p{\sum }_{k^{\prime }=1}^p{\int }_0^1{\int }_0^1{\left\{C_{k{k}^{\prime }}(s,t)\right\}}^{2}dsdt}.$$

For estimating the ℓth eigenfunction, we use the integrated square errors (ISE), which are defined as

$$\min \left[{\displaystyle \sum _{k=1}^p}{\int }_0^1{\left\{{\Psi }_{\ell }^{(k)}(t)-{\widehat{\Psi }}_{\ell }^{(k)}(t)\right\}}^{2}dt,{\displaystyle \sum _{k=1}^p}{\int }_0^1{\left\{{\Psi }_{\ell }^{(k)}(t)+{\widehat{\Psi }}_{\ell }^{(k)}(t)\right\}}^{2}dt\right].$$

Note that the range of ISE is [0,2]. For estimating the eigenvalues, we use the ratio of the estimate against the truth; that is, ${\widehat{d}}_{\ell }/d_{\ell }$.

For predicting new curves, we use the mean integrated square errors (MISE), which are given by

$$\frac{1}{200p}{\displaystyle \sum _{k=1}^p}{\displaystyle \sum _{i=1}^{200}}\left[{\int }_0^1{\left\{x_i^{(k)}(t)-{\widehat{x}}_i^{(k)}(t)\right\}}^{2}dt\right].$$

For the curve prediction using mFPCA, we truncate the number of principal components using a PVE of 0.99. It is worth noting that if no truncation is adopted, then the curve prediction using mFPCA reduces to curve prediction using univariate FPCA. We shall also consider the conditional expectation method based on the estimates of covariance functions from mFPCA. The method is denoted by mFPCA(CE), and its difference with mFACEs is that different estimates of covariance functions are used.

3.2. Simulation results

Figure 1 gives boxplots of RISEs of mFACEs and mFPCA for estimating covariance functions. Under all model conditions, mFACES outperforms mFPCA, and the improvement in RISEs as the sample size increases is much more pronounced for mFACEs. Under the model conditions with moderate correlations (ρ=0.9), the advantage of mFACEs is substantial even for the small sample size n=100.

Figure 1.

Boxplots of relative integrated square errors (RISEs) of mFACEs and mFPCA for estimating the covariance function

Figures 2 and 3 give boxplots of ISEs and violin plots of mFACEs and mFPCA for estimating the top two eigenfunctions and eigenvalues, respectively. The top two eigenvalues account for about 60% of the total variation in the functional data for ρ=0.5, and it is 80% for ρ=0.9. Figure 2 shows that although the two methods are overall comparable for estimating the first eigenfunction, mFACEs has a much better accuracy for estimating the second eigenfunction than mFPCA. The violin plots in Figure 3 show that mFACEs outperforms mFPCA substantially for estimating both eigenvalues under all model conditions. The mFPCA always underestimates the eigenvalues as the variation of scores from univariate FPCA is smaller than the true variation and hence leads to underestimates of eigenvalues.

Figure 2.

Boxplots of integrated square errors (ISEs) of mFACEs and mFPCA for estimating the top two eigenfunctions

Figure 3.

Violin plots of mFACEs and mFPCA for estimating the top two eigenvalues. The red horizontal lines indicate that the estimates are equal to the truth

Finally, we consider the prediction of new subjects by mFACEs, mFPCA, and mFPCA(CE). We define the relative efficiencies of different methods as the ratios of MISEs with respect to those of univariate FPCA; see Figure 4. Univariate FPCA is implemented in the R package face (Xiao, Li, Checkley, & Crainiceanu, 2018). We have the following findings. Under all model conditions, mFACEs has the smallest MISE, mFPCA(CE) has the second best performance, and mFPCA is close to univariate FPCA. Thus, on average, mFACEs provides the most accurate curve prediction. These results indicate that (a) mFACEs has better covariance estimation than mFPCA(CE) and so is the prediction based on it; (b) compared with mFPCA/univariate FPCA, mFPCA(CE) exploits the correlation information and hence results in better predictions.

Figure 4.

Boxplots of relative efficiency of three methods for curve prediction. The grey horizontal lines indicate the mean integrated square errors (MISEs) for univariate functional principal component analysis (FPCA)

In summary, mFACEs shows competing performance against alternative methods.

4. APPLICATION TO ALZHEIMER'S DISEASE STUDY

The ADNI is a two‐stage longitudinal observational study launched in year 2003 with the primary goal of investigating whether serial neuroimages, biological markers, clinical and neuropsychological assessments can be combined to measure the progression of AD (Weiner et al., 2017). The ADNI‐1 data from the first stage contain 379 patients with amnestic mild cognitive impairment (MCI, a risk state for AD) at baseline who had at least one follow‐up visit. Participants were assessed at baseline, 6, 12, 18, 24, and 36 months with additional annual follow‐ups included in the second stage of the study. At each visit, various neuropsychological assessments, clinical measures, and brain images were collected. The ADNI‐2 data include 424 additional patients suffering from MCI and significant memory concern, with at least one follow‐up visit and longitudinal data collected over 4 years. Thus, for the combined data, the total number of subjects is 803, and the average number of visits is 4.72. The data are publicly available at http://ida.loni.ucla.edu/.

We consider five longitudinal markers commonly measured in studies of AD with strong comparative predictive value (Li, Chan, Doody, Quinn, & Luo, 2017). Among the five markers, Disease Assessment Scale‐Cognitive 13 items (ADAS‐Cog 13), Rey Auditory Verbal Learning Test immediate recall (RAVLT.imme), Rey Auditory Verbal Learning Test learning curve (RAVLT.learn), and Mini‐Mental State Examination (MMSE) are neuropsychological assessments. Functional Assessment Questionnaire (FAQ) is a functional and behavioural assessment. High values of ADAS‐Cog 13 and FAQ indicate a high‐risk state for AD, whereas low values of RAVLT.imme, RAVLT.learn, and MMSE reflect severe cognitive impairment. The longitudinal trajectories in ADNI‐1 and ADNI‐2 are defined on the same time domain with the largest follow‐up time of 96 months from the start of ADNI‐1 (Time 0).

4.1. Multivariate FPCA via mFACEs

We analyse the five longitudinal biomarkers using mFACEs. For better visualization, we plot in Figure 5 the estimated correlation functions ${\rho }_{k{k}^{\prime }}(s,t)=C_{k{k}^{\prime }}(s,t)/\sqrt{C_{kk}(s,s)C_{k^{\prime }{k}^{\prime }}(t,t)}$. The plot indicates two groups of biomarkers, ADAS‐Cog 13 and FAQ, in one group and RAVLT.imme, RAVLT.learn, and MMSE in another group. The biomarkers within the groups are positively correlated and negatively correlated between groups, which makes sense as high values of ADAS‐Cog 13 and FAQ and low values for the other biomarkers suggest of AD. Next, we display in Figure 6 the two estimated (multivariate) eigenfunctions associated with the top two estimated eigenvalues, which account for 69% and 11% of the total variance in the functional part of the data. The eigenfunctions reveal how the five biomarkers covary and how a subject's trajectories of biomarkers deviate from the population mean. Indeed, we see from Figure 6 that the first eigenfunction (solid curves) is below the zero line for ADAS‐Cog 13 and FAQ and above the zero line for the other three biomarkers. This means that the score corresponding to the first eigenfunction might be used as an indicator of AD. Indeed, a negative score for the first eigenfunction means higher‐than‐population‐mean values of the former whereas lower‐than‐population‐mean values of the latter, indicating more severe AD status. The second eigenfunction (dashed curves) for the five biomarkers is below the zero line at first and then above it or the other way around, potentially suggesting of a longitudinal pattern of the AD progression. Specifically, these subjects with a positive score for the second eigenfunction will have higher ADAS‐Cog 13/FAQ and lower RAVLT and MMSE over the months, suggesting of AD progression. Finally, we illustrate in Figure 7 the predicted curves along with the associated 95% point‐wise confidence bands for three subjects. We focus on predicting the trajectories over the first 4 years as there are more observations. We can see that the confidence bands are getting wider at the later time points because of fewer observations.

Figure 5.

Estimated correlation functions for the longitudinal markers. ADAS‐Cog 13, Disease Assessment Scale‐Cognitive 13 items; FAQ, Functional Assessment Questionnaire; MMSE, Mini‐Mental State Examination; RAVLT.imme, Rey Auditory Verbal Learning Test immediate recall; RAVLT.learn, Rey Auditory Verbal Learning Test learning curve

Figure 6.

Estimated top two eigenfunctions for the longitudinal markers. ADAS‐Cog 13, Disease Assessment Scale‐Cognitive 13 items; FAQ, Functional Assessment Questionnaire; MMSE, Mini‐Mental State Examination; RAVLT.imme, Rey Auditory Verbal Learning Test immediate recall; RAVLT.learn, Rey Auditory Verbal Learning Test learning curve

Figure 7.

Predicted subject‐specific curves (red solid line) of the longitudinal markers and associated 95% point‐wise confidence bands (blue dashed line) for three subjects. ADAS‐Cog 13, Disease Assessment Scale‐Cognitive 13 items; FAQ, Functional Assessment Questionnaire; MMSE, Mini‐Mental State Examination; RAVLT.imme, Rey Auditory Verbal Learning Test immediate recall; RAVLT.learn, Rey Auditory Verbal Learning Test learning curve

4.2. Comparison of prediction performance of different methods

We compare the proposed mFACEs with mFPCA and mFPCA(CE) for predicting the five longitudinal biomarkers. The prediction performance is evaluated by the average squared prediction errors (APE),

$${\text{APE}}_k=\frac{1}{n}{\displaystyle \sum _{i=1}^n}\left[\frac{1}{m_{ik}}{\displaystyle \sum _{j=1}^{m_{ik}}}{\left\{y_{ij}^{(k)}-{ŷ}_{ij}^{(k)}\right\}}^2\right],$$

where ${ŷ}_{ij}^{(k)}$ is the predicted value of the kth biomarker for the ith subject at time $t_{ij}^{(k)}$. We conduct two types of validation: an internal validation and an external validation. For the internal validation, we perform a 10‐fold cross‐validation to the combined data of ADNI‐1 and ADNI‐2. For the external validation, we fit the model using only the ADNI‐1 data and then predict ADNI‐2 data. Figure 8 summarizes the results. For simplicity, we present the relative efficiency of APE, which is the ratio of APEs of one method against the mFPCA. In both cases, mFACEs achieves better prediction accuracy than competing methods. Note that mFPCA(CE) outperforms mFPCA for predicting almost all biomarkers. The results suggest that (a) mFACEs is better than competing methods for analysing the longitudinal biomarkers. and (b) exploiting the correlations between the biomarkers improves prediction.

Figure 8.

The internal and external prediction validations for the Alzheimer's Disease Neuroimaging Initiative longitudinal makers. ADAS‐Cog 13, Disease Assessment Scale‐Cognitive 13 items; FAQ, Functional Assessment Questionnaire; MMSE, Mini‐Mental State Examination; RAVLT.imme, Rey Auditory Verbal Learning Test immediate recall; RAVLT.learn, Rey Auditory Verbal Learning Test learning curve

5. DISCUSSION

The prevalence of multivariate functional data has sparked much research interests in recent years. However, covariance estimation for multivariate sparse functional data remains underdeveloped. We proposed a new method, mFACEs, and its features include the following: (a) A covariance smoothing framework is proposed to tackle multivariate sparse functional data; (b) an automatic and fast fitting algorithm is adopted to ensure the scalability of the method; (c) eigenfunctions and eigenvalues can be obtained through a one‐time spectral decomposition, and eigenfunctions can be easily evaluated at any sampling points; and (d) a multivariate extension of the conditional expectation approach (Yao et al., 2005) is derived to exploit correlations between outcomes. The simulation study and the data example showed that mFACEs could better capture between‐function correlations and thus gives improved principal component analysis and curve prediction.

When the magnitude of functional data are quite different, one may first normalize the functional data, as recommended by Chiou et al. (2014). One method of normalization is to rescale the functional data using the estimated variance function ${Ĉ}_{kk}{(t,t)}^{-1/2}$ as in Chiou et al. (2014) and Jacques and Preda (2014). An alternative method is to use a global rescaling factor like ${\left(\int {Ĉ}_{kk}(t,t)dt\right)}^{-1/2}$ as in Happ and Greven (2018). Both methods can be easily incorporated into our proposed method. In our data analysis, we find that the results with normalization are very close to those without normalization; thus, we present the results without normalization.

Because multivariate FPCA is more complex than univariate FPCA, weak correlations between the functions and small sample size may offset the benefit of conducting multivariate FPCA; see Section 7.3 in Wong et al. (2019). Thus, it is of future interest to develop practical tests to determine if correlations between multivariate functional data are different from 0.

The mFACEs method has been implemented in an R package mfaces and will be submitted to CRAN for public access.

DATA AVAILABILITY STATEMENT

Data used in preparation of this article were obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database (http://adni.loni.usc.edu). As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this paper. A complete listing of ADNI investigators can be found at: http://adni.loni.usc.edu/wp‐content/uploads/how_to_apply/ADNI_Acknowledgement_List.pdf.

Supporting information

The Supporting Information contains additional details and results of a simulation that mimics the ADNI data. R package mfaces (https://github.com/cli9/mfaces) contains an open‐source implementation of the proposed method described in the article. Demo zip file contains codes to demonstrate the proposed method with a simulated dataset in Section 3.

Supporting info item

APPENDIX A. PROOFS OF PROPOSITIONS 1 AND 2

Define $\tilde{\mathbf{b}}(t)={\mathbf{G}}^{-\frac{1}{2}}\mathbf{b}(t)$, and then $\int \tilde{\mathbf{b}}(t)\tilde{\mathbf{b}}{(t)}^{\top }dt=\mathbf{I}$. Define ${\check{\Theta }}_{k{k}^{\prime }}(t)={\mathbf{G}}^{\frac{1}{2}}{\Theta }_{k{k}^{\prime }}{\mathbf{G}}^{\frac{1}{2}}$. According to Equation (1),

$$\begin{array}{ccc}\hfill d_{\ell }{\Psi }_{\ell }^{(k)}(s)& ={\displaystyle \sum _{k^{\prime }=1}^p}\int \mathbf{b}{(s)}^{\top }{\mathbf{\Theta }}_{k{k}^{\prime }}\mathbf{b}(t){\Psi }_{\ell }^{(k^{\prime })}(t)dt\hfill & \hfill \\ \hfill & =\tilde{\mathbf{b}}{(s)}^{\top }{\displaystyle \sum _{k^{\prime }=1}^p}{\check{\mathbf{\Theta }}}_{k{k}^{\prime }}\int \tilde{\mathbf{b}}(t){\Psi }_{\ell }^{(k^{\prime })}(t)dt.\hfill \end{array}$$

Thus, ${\Psi }_{\ell }^{(k)}(s)=\tilde{\mathbf{b}}{(s)}^{\top }{\mathbf{u}}_{\ell }^{(k)}$ with ${\mathbf{u}}_{\ell }^{(k)}=d_{\ell }^{-1}\left\{{\sum }_{k^{\prime }=1}^p{\check{\Theta }}_{k{k}^{\prime }}\int \tilde{\mathbf{b}}(t){\Psi }_{\ell }^{(k^{\prime })}(t)\hspace{0.17em}dt\right\}\in {\mathbb{R}}^c$ and ${\mathbf{u}}_{\ell }={({\mathbf{u}}_{\ell }^{(1),\top },\text{…},{\mathbf{u}}_{\ell }^{(p),\top })}^{\top }\in {\mathbb{R}}^{pc}$. As ${\sum }_{k=1}^p\int {\Psi }_{\ell }^{(k)}(t){\Psi }_{{\ell }^{\prime }}^{(k)}(t)dt=1_{\{\ell ={\ell }^{\prime }\}}$, we derive that

$${\mathbf{u}}_{\ell }^{\top }{\mathbf{u}}_{{\ell }^{\prime }}={\displaystyle \sum _{k=1}^p}{\mathbf{u}}_{\ell }^{(k),\top }{\mathbf{u}}_{{\ell }^{\prime }}^{(k)}=1_{\{\ell ={\ell }^{\prime }\}}.$$ (B1)

By Equation (2),

$$C_{k{k}^{\prime }}(s,t)=\tilde{\mathbf{b}}{(s)}^{\top }{\check{\Theta }}_{k{k}^{\prime }}\tilde{\mathbf{b}}(t)=\tilde{\mathbf{b}}{(s)}^{\top }\left\{\sum _{\ell \ge 1}{d}_{\ell }{\mathbf{u}}_{\ell }^{(k)}{\mathbf{u}}_{\ell }^{(k^{\prime }),\top }\right\}\tilde{\mathbf{b}}(t),$$

which gives

$${\check{\Theta }}_{k{k}^{\prime }}=\sum _{\ell \ge 1}{d}_{\ell }{\mathbf{u}}_{\ell }^{(k)}{\mathbf{u}}_{\ell }^{(k^{\prime })}.$$

As 1 ≤ k,k ′ ≤ p, the above is equivalent to

$$\begin{array}{cc}\hfill \underset{:=\check{\Theta }}{\underbrace{\left(\begin{array}{ccc}{\check{\Theta }}_{11}& \text{…}& {\check{\Theta }}_{1p}\\ ⋮& \ddots & ⋮\\ {\check{\Theta }}_{p1}& \text{…}& {\check{\Theta }}_{pp}\end{array}\right)}}& =\sum _{\ell \ge 1}{d}_{\ell }{\mathbf{u}}_{\ell }{\mathbf{u}}_{\ell }^{\top }.\hfill \end{array}$$

Because of Equation (B1), u _ℓ s are orthonormal eigenvectors of $\check{\Theta }$ with d _ℓ s the corresponding eigenvalues. The proof is now complete.

By Equation (10),

$$\text{iGCV}={‖\widehat{\mathbf{C}}‖}^2-2{\mathbf{f}}^{\top }{\sum }^{-1}\mathbf{f}+{\mathbf{f}}^{\top }{\sum }^{-2}\mathbf{f}+2{\displaystyle \sum _{i=1}^n}{\left({\mathbf{L}}_i{\sum }^{-1}\mathbf{f}-{\mathbf{f}}_i\right)}^{\top }{\sum }^{-1}\left({\mathbf{L}}_i{\sum }^{-1}\mathbf{f}-{\mathbf{f}}_i\right).$$ (B2)

Since ${\sum }^{-1}=\mathbf{U}\text{diag}(\tilde{\mathbf{d}}){\mathbf{U}}^{\top }$, we have

$${\mathbf{f}}^{\top }{\sum }^{-1}\mathbf{f}={\tilde{\mathbf{f}}}^{\top }\text{diag}(\tilde{\mathbf{d}})\tilde{\mathbf{f}}={\tilde{\mathbf{d}}}^{\top }(\tilde{\mathbf{f}}\odot \tilde{\mathbf{f}}).$$ (B3)

Similarly,

$${\mathbf{f}}^{\top }{\sum }^{-2}\mathbf{f}={\tilde{\mathbf{f}}}^{\top }\text{diag}({\tilde{\mathbf{d}}}^2)\tilde{\mathbf{f}}={(\tilde{\mathbf{f}}\odot \tilde{\mathbf{d}})}^{\top }(\tilde{\mathbf{f}}\odot \tilde{\mathbf{d}}).$$ (B4)

Next we derive that

$$\begin{array}{c}{\left({\mathbf{L}}_i{\sum }^{-1}\mathbf{f}-{\mathbf{f}}_i\right)}^{\top }{\sum }^{-1}\left({\mathbf{L}}_i{\sum }^{-1}\mathbf{f}-{\mathbf{f}}_i\right)={\left({\mathbf{U}}^{\top }{\mathbf{L}}_i{\sum }^{-1}\mathbf{f}-{\mathbf{U}}^{\top }{\mathbf{f}}_i\right)}^{\top }\text{diag}(\tilde{\mathbf{d}})\left({\mathbf{U}}^{\top }{\mathbf{L}}_i{\sum }^{-1}\mathbf{f}-{\mathbf{U}}^{\top }{\mathbf{f}}_i\right)={\left({\tilde{\mathbf{L}}}_i\text{diag}(\tilde{\mathbf{d}})\tilde{\mathbf{f}}-{\tilde{\mathbf{f}}}_i\right)}^{\top }\text{diag}(\tilde{\mathbf{d}})\left({\tilde{\mathbf{L}}}_i\text{diag}(\tilde{\mathbf{d}})\tilde{\mathbf{f}}-{\tilde{\mathbf{f}}}_i\right).\hfill \end{array}$$

It follows that

$$\begin{array}{c}{\left({\mathbf{L}}_i{\sum }^{-1}\mathbf{f}-{\mathbf{f}}_i\right)}^{\top }{\sum }^{-1}\left({\mathbf{L}}_i{\sum }^{-1}\mathbf{f}-{\mathbf{f}}_i\right)={\tilde{\mathbf{d}}}^{\top }\left[\left\{{\tilde{\mathbf{L}}}_i(\tilde{\mathbf{f}}\odot \tilde{\mathbf{d}})\right\}\odot \left\{{\tilde{\mathbf{L}}}_i(\tilde{\mathbf{f}}\odot \tilde{\mathbf{d}})\right\}+({\tilde{\mathbf{f}}}_i\odot {\tilde{\mathbf{f}}}_i)\right]-2{\tilde{\mathbf{d}}}^{\top }\left\{({\tilde{\mathbf{f}}}_i{\tilde{\mathbf{f}}}^{\top })\odot {\tilde{\mathbf{L}}}_i\right\}\tilde{\mathbf{d}}.\hfill \end{array}$$ (B5)

By combining Equations  (B2), (B3), (B4), and (B5), the proof is complete.

APPENDIX B. LEMMA

Lemma 1

The covariance operator with the covariance functions defined in Section 4.1 is positive semidefinite.

Let $\mathit{a}={(a_1,\text{…},a_p)}^{\top }\in {\mathbb{R}}^p$ and $\tilde{X}={\mathit{a}}^{\top }\mathbf{x}$, and then $\tilde{X}$ is a stochastic process with covariance function

$$\text{Cov}\left\{\tilde{X}(s),\tilde{X}(t)\right\}=\sum _{k{k}^{\prime }}{a}_k{a}_{k^{\prime }}{C}_{k{k}^{\prime }}(s,t)=\sum _{k{k}^{\prime }}{a}_k{a}_{k^{\prime }}\left(\rho +(1-\rho )1_{\{k=k^{\prime }\}}\right){\tilde{\Phi }}_k{(s)}^{\top }{\tilde{\Phi }}_{k^{\prime }}(t),$$

where ${\tilde{\Phi }}_k(s)={\Phi }_k{(s)}^{\top }{\Lambda }_{kk}^{\frac{1}{2}}$. Let $\Psi (s)={\sum }_{k=1}^p{a}_k{\tilde{\Phi }}_k(s)$. Then

$$\text{Cov}\left\{\tilde{X}(s),\tilde{X}(t)\right\}=\rho \Psi {(s)}^{\top }\Psi (t)+(1-\rho )\sum _k{a}_k^2{\tilde{\Phi }}_k{(s)}^{\top }{\tilde{\Phi }}_k(t),$$

which is always positive semidefinite, and the proof is complete.

Li C, Xiao L, Luo S. Fast covariance estimation for multivariate sparse functional data. Stat. 2019:9;e245. 10.1002/sta4.245