Fast Covariance Estimation for High-dimensional Functional Data

Abstract

We propose two fast covariance smoothing methods and associated software that scale up linearly with the number of observations per function. Most available methods and software cannot smooth covariance matrices of dimension J > 500; a recently introduced sandwich smoother is an exception but is not adapted to smooth covariance matrices of large dimensions, such as J = 10, 000. We introduce two new methods that circumvent those problems: 1) a fast implementation of the sandwich smoother for covariance smoothing; and 2) a two-step procedure that first obtains the singular value decomposition of the data matrix and then smoothes the eigenvectors. These new approaches are at least an order of magnitude faster in high dimensions and drastically reduce computer memory requirements. The new approaches provide instantaneous (a few seconds) smoothing for matrices of dimension J = 10,000 and very fast (< 10 minutes) smoothing for J = 100, 000. R functions, simulations, and data analysis provide ready to use, reproducible, and scalable tools for practical data analysis of noisy high-dimensional functional data.

1 Introduction

The covariance function plays an important role in functional principal component analysis (fPCA), functional linear regression, and functional canonical correlation analysis (see, e.g., Ramsay and Silverman 2002, 2005). The major difference between the covariance function of functional data and the covariance matrix of multivariate data is that functional data is measured on the same scale, with sizable noise and possibly sampled at an irregular grid. Ordering of functional observations is also important, but it can easily be handled by careful indexing. Thus, it has become common practice in functional data analysis to estimate functional principal components by diagonalizing a smoothed estimator of the covariance function; see, e.g., Besse and Ramsay (1986); Ramsay and Dalzell (1991); Kneip (1994); Besse et al. (1997); Staniswalis and Lee (1998); Yao et al. (2003, 2005).

Given a sample of functions, a simple estimate of the covariance function is the sample covariance. The sample covariance, its eigenvalues and eigenvectors have been shown to converge to their population counterparts at the optimal rate when the sample paths are completely observed without measurement error (Dauxois et al. 1982). However, in practice, data are measured at a finite number of locations and often with sizable measurement error. For such data the eigenvectors of the sample covariance matrix tend to be noisy, which can substantially reduce interpretability. Therefore, smoothing is often used to estimate the functional principal components; see, e.g., Besse and Ramsay (1986); Ramsay and Dalzell (1991); Rice and Silverman (1991); Kneip (1994); Capra and Müller (1997); Besse et al. (1997); Staniswalis and Lee (1998); Cardot (2000); Yao et al. (2003, 2005). There are three main approaches to estimating smooth functional principal components. The first approach is to smooth the functional principal components of the sample covariance function; for a detailed discussion see, for example, Rice and Silverman (1991); Capra and Müller (1997); Ramsay and Silverman (2005). The second is to smooth the covariance function and then diagonalize it; see, e.g., Besse and Ramsay (1986); Staniswalis and Lee (1998); Yao et al. (2003). The third is to smooth each curve and diagonalize the sample covariance function of the smoothed curves; see Ramsay and Silverman (2005) and the references therein. Our first approach is a fast bivariate smoothing method for the covariance operator which connects the latter two approaches. This method is a fast and new implementation of the ‘sandwich smoother’ in Xiao et al. (2013), with a completely different and specialized computational approach that improves the original algorithm’s computational efficiency by at least an order of magnitude. The sandwich smoother with the new implementation will be referred to as Fast Covariance Estimation, or FACE. Our second approach is to use smoothing spline smoothing of the eigenvectors obtained from a high-dimensional singular value decomposition of the raw data matrix and will be referred to as smooth SVD, or SSVD. To the best of our knowledge, this approach has not been used in the literature for low- or high-dimensional data. Given the simplicity of SSVD, we will focus more on FACE, though simulations and data analysis will be based on both approaches.

The sandwich smoother provides the next level of computational scalability for bivariate smoothers and has significant computational advantages over bivariate P-splines (Eilers and Marx 2003; Marx and Eilers 2005) and thin plate regression splines (Wood 2003). This is achieved, essentially, by transforming the technical problem of bivariate smoothing into a short sequence of univariate smoothing steps. For covariance matrix smoothing, the sandwich smoother was shown to be much faster than local linear smoothers. However, adapting the sandwich smoother to fast covariance matrix smoothing in the ultrahigh dimensions of, for example, modern medical imaging or high density wearable sensor data, is not straightforward. For instance, the sandwich smoother requires the sample covariance matrix which can be hard to calculate and impractical to store for ultrahigh dimensions. While the sandwich smoother is the only available fast covariance smoother, it was never tested for dimensions J > 5,000 and becomes computationally impractical for J > 5,000 on current standard computers. All of these dimensions are well within the range of current high-dimensional data.

In contrast, our novel approach, FACE, is linear in the number of functional observations per subject, provides instantaneous (< 1 minutes) smoothing for matrices of dimension J = 10,000 and fast (< 10 minutes) smoothing for J = 100, 000. This is done by carefully exploiting the low-rank structure of the sample covariance, which allows smoothing and spectral decomposition of the smooth estimator of the covariance without calculating or storing the empirical covariance operator. The new approach is at least an order of magnitude faster in high dimensions and drastically reduces memory requirements; see Table 4 in Section 6 for a comparison of computation time. Unlike the sandwich smoother, FACE also efficiently estimates the covariance function, eigenfunctions, and scores.

Table 4.

Computation time (in seconds) of the SSVD, S-Smooth and FACE methods averaged over 100 data sets on 2.4GHz Mac computers with 8 gigabytes of random access memory. The computation time of the sandwich smoother is also provided except for J = 10,000 and is averaged over 10 datasets only.

J	I	SSVD	S-Smooth	FACE 100 knots	FACE 500 knots	Sandwich 100 knots	Sandwich 500 knots
3,000	50	0.25	1.28	0.34	1.76	47.41	210.41
	500	3.81	13.88	0.89	2.61	50.91	364.39
5,000	50	0.43	2.14	0.50	2.09	251.48	1362.67
	500	6.08	34.63	1.26	3.19	302.34	1743.86
10,000	50	0.86	4.29	0.82	2.92	-	-
	500	12.78	98.41	2.34	4.68	-	-

The remainder of the paper is organized as follows. Section 2 provides the model and data structure. Section 3 introduces FACE and provides the associated fast algorithm. Section 4 extends FACE to structured high-dimensional functional data and incomplete data. Section 5 introduces SSVD, the smoothing spline smoothing of eigenvectors obtained from SVD. Section 6 provides simulation results. Section 7 shows how FACE works in a large study of sleep. Section 8 provides concluding remarks.

FACE and SSVD are now implemented as R functions “fpca.face” and “fpca2s”, respectively, in the publicly available package refund (Crainiceanu et al. 2013).

2 Model and data structure

Suppose that {X_i, i = 1, …, I} is a collection of independent realizations of a random functional process X with covariance function K(s, t), s, t ∈ [0,1]. The observed data, Y_ij = X_i(t_j) + ε_ij, are noisy proxies of X_i at the sampling points {t₁, …, t_J}. We assume that ε_ij are i.i.d. errors with mean zero and variance σ², and are mutually independent of the processes X_i.

The sample covariance function can be computed at each pair of sampling points (t_j, t_ℓ) by K̂(t_j, t_ℓ) = I⁻¹ ∑_i Y_ijY_iℓ. For ease of presentation we assume that Y_ij have been centered across subjects. The sample covariance matrix, K̂, is the J × J dimensional matrix with the (j, ℓ) entry equal to K̂(t_j, t_ℓ). Covariance smoothing typically refers to applying bivariate smoothers to K̂. Let Y_i = (Y_i1, …,Y_iJ)^T, i = 1, …, I, then $\mathbf{K̂}=I^{-1}{\sum }_{i=1}^I{\mathbf{Y}}_i{\mathbf{Y}}_i^T=I^{-1}\mathbf{Y}{\mathbf{Y}}^T$, where Y = [Y₁, …,Y_I] is a J × I dimensional matrix with the ith column equal to Y_i. When I is much smaller than J, K̂ is of low rank; this low-rank structure of K̂ will be particularly useful for deriving fast methods for smoothing K̂.

3 FACE

The FACE estimator of the covariance matrix has the following form

$$\mathbf{K̃}=\mathbf{S}\mathbf{K̂}\mathbf{S},$$ (1)

where S is a symmetric smoother matrix of dimension J × J. Because of (1), we say FACE has a sandwich form. We use P-splines (Eilers and Marx 1996) to construct S so that S = B (B^T B + λP)⁻¹ B^T. Here B is the J × c design matrix {B_k(t_j)}_{1≤j≤J,1≤k≤c}, P is a symmetric penalty matrix of size c × c, λ is the smoothing parameter, {B₁(·), …,B_c(·)} is the collection of B-spline basis functions, c is the number of interior knots plus the order (degree plus 1) of B-splines. We assume that the knots are equally spaced and use a difference penalty as in Eilers and Marx (1996) for the construction of P. Model (1) is a special case of the sandwich smoother in Xiao et al. (2013) as the two smoother matrices for FACE are identical. However, FACE is specialized to smooth covariance matrices and has some further important characteristics.

First, K̃ is guaranteed to be symmetric and positive semi-definite because K̂ is so. Second, the sandwich form of the smoother and the low-rank structure of the sample covariance matrix can be exploited to scale FACE to high and ultra high dimensional data (J > 10, 000). For instance, the eigen-decomposition of K̃ provides the estimates of the eigenfunctions associated with the covariance function. However, when J is large, both the smoother matrix and the sample covariance matrix are high dimensional and even storing them may become impractical. FACE, unlike the sandwich smoother, is designed to obtain the eigendecomposition of K̃ without computing the smoother matrix or the sample covariance matrix.

FACE depends on a single smoothing parameter, λ, which needs to be selected. The algorithm for selecting λ in Xiao et al. (2013) requires O(J²I) computations and can be hard to compute when J is large. We propose efficient smoothing parameter estimation algorithms that requires only O(JIc) computations; see Section 3.2 for details.

3.1 Estimation of eigenfunctions

Assuming that the covariance function K is in L₂([0,1]²), Mercer’s theorem states that K admits an eigendecomposition K(s, t) = ∑_k λ_kψ_k(s)ψ_k(t) where {ψ_k(·) : k ≥ 1} is a set of orthonormal basis of L₂([0,1]) and λ₁ ≥ λ₂ ≥ ⋯ are the eigenvalues. Estimating the functional principal components/eigenfunctions ψ_k’s is one of the most fundamental tasks in functional data analysis and has attracted a lot of attention (Ramsay and Silverman 2005). Typically, interest lies in seeking the first few eigenfunctions that explain a large proportion of the observed variation. This is equivalent to finding the first few eigenfunctions whose linear combination could well approximate the random functions X_i. Computing the eigenfunctions of a symmetric bivariate function is generally not trivial. The common practice is to discretize the estimated covariance function and approximate its eigenfunctions by the corresponding eigenvectors (see, e.g., Yao et al. 2003). In this section, we show that by using FACE we can easily obtain the eigendecomposition of the smoothed covariance matrix K̃ in equation (1).

We start with the decomposition (B^TB)^−1/2P(B^TB)^−1/2 = Udiag(s)U^T, where U is the matrix of eigenvectors and s is the vector of eigenvalues. Let A_S = B(B^TB)^−1/2U. Then ${\mathbf{A}}_S^T{\mathbf{A}}_S={\mathbf{I}}_c$ which implies that A_S has orthonormal columns. It follows that $\mathbf{S}={\mathbf{A}}_S{\mathit{\Sigma }}_S{\mathbf{A}}_S^T$ with Σ_S = {I_c + λdiag(s)}⁻¹. Let $\mathbf{Ỹ}={\mathbf{A}}_S^T\mathbf{Y}$ be a c × I matrix, then $\mathbf{K̃}={\mathbf{A}}_S\left(I^{-1}{\mathit{\Sigma }}_S\mathbf{Ỹ}{\mathbf{Ỹ}}^T{\mathit{\Sigma }}_S\right){\mathbf{A}}_S^T$. Thus only the c × c dimensional matrix in the parenthesis depends on the smoothing parameter; this observation will lead to a simple spectral decomposition of K̃. Indeed, consider the spectral decomposition I⁻¹Σ_SỸỸ^TΣ_S = AΣA^T, where A is the c × c matrix of eigenvectors and Σ is the c × c diagonal matrix of eigenvalues. It follows that K̃ = (A_SA)Σ(A_SA)^T which is the eigendecomposition of K̃ and shows that K̃ has no more than c nonzero eigenvalues (Proposition 1). Because of the dimension reduction of matrices (c × c versus J × J), this eigenanalysis of the smoothed covariance matrix is fast. The derivation reveals that through smoothing we obtain a smoothed covariance operator and its associated eigenfunctions. An important consequence is that the number of elements stored in memory is only O(Jc) for FACE, while using other bivariate smoothers requires storing the J × J dimensional covariance operators. This makes a dramatic difference, allows non-compromise smoothing of covariance matrices, and provides a transparent, easy to use method.

3.2 Selection of the smoothing parameter

We start with the following result.

Proposition 1 Assume c = o(J), then the rank of the smoothed covariance matrix K̃ is at most min(c, I).

This indicates that the number of knots controls the maximal rank of the smoothed covariance matrix, K̃, or equivalently, the number of eigenfunctions that can be extracted from K̃. This implies that using an insufficient number of knots may result in severely biased estimates of eigenfunctions and number of eigenfunctions. We propose to use a relatively large number of knots, e.g., 100 knots, to reduce the estimation bias and control overfitting by an appropriate penalty. Note that for high-dimensional data, J can be thousands or more and the dimension reduction by FACE is size-able. Moreover, as only a small number of functional principal components is typically used in practice, FACE with 100 knots seems adequate for most applications. When the covariance function has a more complex structure or a larger number of functional principal components are needed, one may use a larger number of knots; see Ruppert (2002) and Wang et al. (2011) for simulations and theory. Next we focus on selecting the smoothing parameter.

We select the smoothing parameter by minimizing the pooled generalized cross validation (PGCV), a functional extension of the GCV (Craven and Wahba 1979),

$$\sum _{i=1}^I{\Vert {\mathbf{Y}}_i-\mathbf{S}{\mathbf{Y}}_i\Vert }^2/{\{1-\text{tr}(\mathbf{S})/J\}}^2.$$ (2)

Here ‖·‖ is the Euclidean norm of a vector. Criterion (2) was also used in Zhang and Chen (2007) and could be interpreted as smoothing each sample, Y_i, using the same smoothing parameter. We argue that using criterion (2) is a reasonable practice for covariance estimation. An alternative but computationally hard method for selecting the smoothing parameter is the leave-one-curve-out cross validation (Yao et al. 2005). The following result indicates that PGCV can be easily calculated in high dimensions.

Proposition 2 The PGCV in expression (2) equals to

$$\frac{{\sum }_{k=1}^c{C}_{kk}{(\lambda s_k)}^2/{(1+\lambda s_k)}^2-{\Vert \mathbf{Ỹ}\Vert }_F^2+{\Vert \mathbf{Y}\Vert }_F^2}{{\{1-J^{-1}{\sum }_{k=1}^c{(1+\lambda s_k)}^{-1}\}}^2},$$

where s_k is the kth element of s, C_kk is the kth diagonal element of ỸỸ^T, and ‖·‖_F is the Frobenius norm.

The result shows that ${\Vert \mathbf{Y}\Vert }_F^2,{\Vert \mathbf{Ỹ}\Vert }_F^2$, and the diagonal elements of ỸỸ^T need to be calculated only once, which requires O(IJ + cI) calculations. Thus, the FACE algorithm is fast.

FACE algorithm:

• Step 1. Obtain the decomposition (B^TB)^−1/2P(B^TB)^−1/2 = Udiag(s)U^T.

• Step 2. Specify S by calculating and storing s and A_S = B(B^TB)^−1/2U.

• Step 3. Calculate and store $\mathbf{Ỹ}={\mathbf{A}}_S^T\mathbf{Y}$.

• Step 4. Select λ by minimizing PGCV in expression (2).

• Step 5. Calculate Σ_S = {I_c+λdiag(s)}⁻¹.

• Step 6. Construct the decomposition I⁻¹Σ_SỸỸ^TΣ_S = AΣA^T.

• Step 7. Construct the decomposition K̃ = (A_SA)Σ(A_SA)^T.

The computation time of FACE is O (IJc + Jc² + c³ + ck₀), where k₀ is the number of iterations needed for selecting the smoothing parameter, and the total required memory is O (IJ + I² + Jc + c² + k₀). See Proposition 3 in the appendix for details. When c = O(I) and k₀ = o(IJ), the computation time of FACE is O(JI² + I³) and O(JI + I²) memory units are required. As a comparison, if we smooth the covariance operator using other bivariate smoothers, then at least O(J² + IJ) memory units are required, which dramatically reduces the computational efficiency of those smoothers.

3.3 Estimating the scores

Under standard regularity conditions (Karhunen 1947), X_i(t) can be written as ∑_k≥1 ξ_ikψ_k(t) where {ψ_k : k ≥ 1} is the set of eigenfunctions of K and ${\xi }_{ik}={\int }_0^1{X}_i(s){\psi }_k(s)ds$ are the principals scores of X_i. It follows that Y_i(t_j) = ∑_k≥1 ξ_ikψ_k(t_j) + ε_ij. In practice, we may be interested in only the first N eigenfunctions and approximate Y_i(t_j) by ${\sum }_{k=1}^N{\xi }_{ik}{\psi }_k(t_j)+{\epsilon }_{ij}$. Using the estimated eigenfunctions ψ̂_k’s and eigenvalues λ̂_k’s from FACE, the scores of each X_i can be obtained by either numerical integration or as best linear unbiased predictors (BLUPs). FACE provides fast calculations of scores for both approaches.

Let Ỹ_i denote the ith column of Ỹ. Let ξ_i = (ξ_i1, …,ξ_iN)^T and let Â_N denote the first N columns of A defined in Section 3.1. Let ψ_k = {ψ_k(t₁), …, ψ_k(t_J)}^T and Ψ = [ψ₁, …, ψ_N]. The matrix J^−1/2Ψ is estimated by A_SÂ_N. The method of numerical integration estimates ξ_ik by ${\mathrm{\xi ̂}}_{ik}={\int }_0^1{Y}_i(t){\mathrm{\psi ̂}}_k(t)dt\approx J^{-1}{\sum }_{j=1}^J{Y}_i(t_j){\mathrm{\psi ̂}}_k(t_j)$.

Theorem 1 The estimated principal scores ξ̂_i = (ξ̂_i1, …, ξ̂_iN)^T using numerical integration are ${\mathbf{\xi ̂}}_i=J^{-1/2}{\mathbf{Â}}_N^T{\mathbf{Ỹ}}_i,1\le i\le I$, 1 ≤ i ≤ I.

We now show how to obtain the estimated BLUPs for the scores. Let ${\epsilon }_{ij}=Y_i(t_j)-{\sum }_{k=1}^N{\psi }_k(t_j){\xi }_{ik}$ and ε_i = (ε_i1, …, ε_iJ)^T. Then Y_i = Ψξ_i + ε_i. The covariance var(ξ_i) = diag(λ₁, …,λ_N) can be estimated by J⁻¹ Σ̂_N = J⁻¹ diag(λ̂₁, …,λ̂_N). The variance of ε_ij can be estimated by

$${\mathrm{\sigma ̂}}^2=I^{-1}{J}^{-1}{\Vert \mathbf{Y}\Vert }_F^2-J^{-1}\sum _k{\mathrm{\lambda ̂}}_k.$$ (3)

Theorem 2 Suppose Ψ is estimated by J^1/2 A_SÂ_N, var(ξ_i) = diag(λ₁, …,λ_N) is estimated by Σ̂_N = diag(λ̂₁, …, λ̂_N), and σ² is estimated by σ̂² in equation (3). Then the estimated BLUPs of ξ_i are given by ${\mathbf{\xi ̂}}_i=J^{-1/2}{\mathit{\Sigma ̂}}_N{({\mathit{\Sigma ̂}}_N+J^{-1}{\mathrm{\sigma ̂}}^2{\mathbf{I}}_N)}^{-1}{\mathbf{Â}}_N^T{\mathbf{Ỹ}}_i$, for 1 ≤ i ≤ I.

Theorems 1 and 2 provide fast approaches for calculating the principal scores using either numerical integration or BLUPs. These approaches combined with FACE are much faster because they make use of the calculations already done for estimating the eigenfunctions and eigenvalues. When J is large, the scores by BLUPs tend to be very close to those obtained by numerical integration; in the paper we only use numerical integration.

4 Extension of FACE

4.1 Structured functional data

When analyzing structured functional data such as multilevel, longitudinal, and crossed functional data (Di et al. 2009; Greven et al. 2010; Zipunnikov et al. 2011, 2012; Shou et al. 2013), the covariance matrices have been shown to be of the form YHY^T, where H is a symmetric matrix; see Shou et al. (2013) for more details. We assume H is positive semi-definite because otherwise we can replace H by its positive counterpart. Note that if H₁ is a matrix such that ${\mathbf{H}}_1{\mathbf{H}}_1^T=\mathbf{H}$, smoothing YHY^T can be done by using FACE for the transformed functional data YH₁. This insight is particularly useful for the sleep EEG data, which has two visits and requires multilevel decomposition.

4.2 Incomplete data

To handle incomplete data, such as the EEG sleep data where long portions of the functions are unavailable, we propose an iterative approach that alternates between covariance smoothing using FACE and missing data prediction. Missing data are first initialized using a smooth estimator of each individual curve within the range of the observed data. Outside of the observed range the missing data are estimated as the average of all observed values for that particular curve. FACE is then applied to the initialized data, which produces predictions of scores and functions and the procedure is then iterated. We only use the scores of the first N components, where N is selected by the criterion

$$N=\text{min}\{k:\frac{{\sum }_{j=1}^k{\lambda }_j}{{\sum }_{j=1}^{\infty }{\lambda }_j}\ge 0.95\}.$$

Suppose Ψ̂ is the p × N matrix of estimated eigenvectors from FACE, Σ̂_N = diag(λ̂₁, …,λ̂_N) is the matrix of estimated eigenvalues, and ${\mathrm{\sigma ̂}}_{\epsilon }^2$ is the estimated variance of the noise. Let y_obs denote the observed data and y_mis the missing data for a curve. Similarly, Ψ̂_obs is a sub-matrix of Ψ̂ corresponding to the observed data and Ψ̂_mis is another submatrix of Ψ̂ corresponding to the missing data. Then the prediction (ŷ_mis, ξ̂) minimizes the following

$$\frac{{\Vert {\mathbf{ŷ}}_{\mathit{\text{mis}}}-J^{1/2}{\mathrm{\Psi ̂}}_{\mathit{\text{mis}}}\mathrm{\xi ̂}\Vert }_2^2+{\Vert {\mathbf{y}}_{\mathit{\text{obs}}}-J^{1/2}{\mathrm{\Psi ̂}}_{\mathit{\text{obs}}}\mathrm{\xi ̂}\Vert }_2^2}{2{\mathrm{\sigma ̂}}_{\epsilon }^2}+\frac{1}{2}{\mathrm{\xi ̂}}^T{\mathrm{\Sigma ̂}}_N^{-1}\mathrm{\xi ̂}.$$

Note that if there is no missing data, the solution to this minimization problem leads to Theorem 2. For the next iteration we replace y_mis by ŷ_mis and re-apply FACE to the updated complete data. We repeat the procedure until convergence is reached. In our experience convergence is very fast and typically achieved in fewer than 10 iterations.

5 The SSVD estimator and a subject-specific smoothing estimator

A second approach for estimating the eigenfunctions and eigenvalues is to decompose the sample covariance matrix K̂ and then smooth the eigenvectors. First let ${\mathbf{U}}_y{\mathbf{D}}_y{\mathbf{V}}_y^T$ be the singular value decomposition (SVD) of the data matrix Y. Here U_y is a J × I matrix with orthonormal columns, V_y is an I orthogonal matrix, and D_y is an I diagonal matrix. The columns of U_y contain all the eigenvectors of K̂ that are associated with non-zero eigenvalues and the set of diagonal elements of $I^{-1}{\mathbf{D}}_y^2$ contain all the non-zero eigenvalues of K̂. Thus, obtaining U_y and D_y is equivalent to the eigendecomposition of K̂. Then we smooth the retained eigenvectors by smoothing splines, implemented by the R function “smooth.spline”. SSVD avoids the direct decomposition of the sample covariance matrix and is computationally simpler. SSVD requires O{min(I, J)IJ} computations.

The approach of smoothing each curve and then diagonalizing the sample covariance function of the smoothed curves can also be efficiently implemented. First we smooth each curve using smoothing splines. We use the R function “smooth.spline” which requires only O(J) computations for a curve with J data points. Our experience is that the widely used function “gam” in the R package mgcv (Wood 2013) is much slower and can be computationally intensive with a number of curves to smooth. Then instead of directly diagonalizing the sample covariance of the smoothed curves, which requires O(J³) computations, we calculate the singular value decomposition of the I × J matrix formed by the smoothed curves, which requires only O(min(I,J)IJ) computations. The resulting right singular vectors estimate the eigenfunctions scaled by J^−1/2. Without the SVD step, a brute-force decomposition of the J × J sample covariance becomes infeasible when J is large, such as 5, 000. We will refer to the this approach as S-Smooth, which, to the best of our knowledge, is the first computationally efficient method for covariance estimation using subject-specific smoothing.

We will compare SSVD, S-Smooth and FACE in terms of performance and computation time in the simulation study.

6 Simulation

We consider three simulation studies. In the first study we use moderately high-dimensional data contaminated with noise. We let J = 3,000 and I = 50, which are roughly the dimensions of the EEG data in Section 7.We use SSVD, S-Smooth and FACE. We did not evaluate other bivariate smoothers because we were unable to run them on such dimensions in a reasonably short time. In the second study we consider functional data where portions of the observed functions are missing completely at random (MCAR). This simulation is directly inspired by our EEG data where long portions of the functions are missing. In the last study we assess the computation time of FACE and compare it with that of SSVD and S-Smooth. We also provide the computation time of the sandwich smoother (Xiao et al., 2013). We use R code that is made available with this paper. All simulations are run on modest, widely available computational resources: an Intel Core i5 2.4 GHz Mac with 8 gigabytes of random access memory.

6.1 Complete data

We consider the following covariance functions:

• 1&2Finite basis expansion. $K(s,t)={\sum }_{\ell =1}^3{\lambda }_{\ell }{\psi }_{\ell }(s){\psi }_{\ell }(t)$ where ψ_ℓ’s are eigenfunctions and λ_ℓ’s are eigenvalues. We choose λ_ℓ = 0.5^ℓ−1 for ℓ =1,2,3 and there are two sets of eigenfunctions: case 1: ${\psi }_1(t)=\sqrt{2}\text{sin}(2\pi t),{\psi }_2(t)=\sqrt{2}\text{cos}(4\pi t)$ and ${\psi }_3(t)=\sqrt{2}\text{sin}(4\pi t)$; and case 2: ${\psi }_1(t)=\sqrt{3}(2t-1),{\psi }_2(t)=\sqrt{5}(6t^2-6t+1)$ and ${\psi }_3(t)=\sqrt{7}(20t^3-30t^2+12t-1)$.
• 3Brownian motion. $K(s,t)={\sum }_{\ell =1}^{\infty }{\lambda }_{\ell }{\psi }_{\ell }(s){\psi }_{\ell }(t)$ with eigenvalues ${\lambda }_{\ell }=\frac{1}{{(\ell -1/2)}^2{\pi }^2}$ and eigenfunctions ${\psi }_{\ell }(t)=\sqrt{2}\text{sin}((\ell -1/2)\pi t)$.
• 4Brownian bridge. $K(s,t)={\sum }_{\ell =1}^{\infty }{\lambda }_{\ell }{\psi }_{\ell }(s){\psi }_{\ell }(t)$ with eigenvalues ${\lambda }_{\ell }=\frac{1}{{\ell }^2{\pi }^2}$ and eigenfunctions ${\psi }_{\ell }(t)=\sqrt{2}\text{sin}(\ell \pi t)$.
• 5
  Matérn covariance structure. The Matérn covariance function

  $$C(d;\varphi ,\nu )=\frac{1}{2^{\nu -1}\Gamma (\nu )}{\left(\frac{\sqrt{2\nu }d}{\varphi }\right)}^{\nu }{K}_{\nu }\left(\frac{\sqrt{2\nu }d}{\varphi }\right)$$

  with range ϕ = 0.07 and order ν = 1. Here Kν is the modified Bessel function of order ν. The top three eigenvalues for this covariance function are 0.209, 0.179 and 0.143.

We generate data at {1/J,2/J, …,1} with J = 3,000 and add i.i.d. 𝒩 (0,σ²) errors to the data. We let

$${\sigma }^2={\int }_{s=0}^1{\int }_{t=0}^{1}K(s,t)\mathrm{d}s\mathrm{d}t,$$

which implies that the signal to noise ratio in the data is 1. The number of curves is I = 50 and for each covariance function 200 datasets are drawn.

We compare the performance of the three methods to estimate: (1) the covariance matrix; (2) the eigenfunctions; and (3) the eigenvalues. For simplicity, we only consider the top three eigenvalues/eigenfunctions. For FACE we use 100 knots; for SSVD and S-Smooth we use smoothing splines, implemented through the R function ‘smooth.spline’. Figure 1 displays, for one simulated data set for each case, the true and estimated eigenfunctions using SSVD and FACE, as well as the estimated eigenfunctions without smoothing.

Fig. 1.

True and estimated eigenfunctions for three cases each with one simulated data set. Each row corresponds to one simulated data set. Each box shows the true eigenfunction (blue dot-dashed lines), the estimated eigenfunction using FACE (red solid lines), the estimated eigenfunction using SSVD (cyan dashed lines), and the estimated eigenfunction without smoothing (black dotted lines). We do not show the estimates from S-Smooth and FACE (incomplete data) because they are almost identical to these from FACE and SSVD.

We see from Figure 1 that the smoothed eigenfunctions are very similar and the estimated eigenfunctions without smoothing are quite noisy. The results are expected as all smoothing-based methods are designed to account for the noise in the data and the discrepancy between the estimated and the true eigenfunctions is mainly due to the variation in the random functions. Table 1 provides the mean integrated squared errors (MISE) of the estimated eigenfunctions indicating that FACE and S-Smooth have better performance than SSVD. For case 5, the smoothed eigenfunctions for all methods are far from the true eigenfunctions. This is not surprising because for this case the eigenvalues are close to each other and it is known that the accuracy of eigenfunction estimation also depends on the gap between consecutive eigenvalues; see for example, Bunea and Xiao (2013). In terms of covariance estimation, Table 2 suggests that SSVD is outperformed by the other two methods. However, the simplicity and robustness of SSVD may actually make it quite popular in applications.

Table 1.

100×MISEs of the three methods for estimating the eigenfunctions. The incomplete data has about 13% observations missing.

	Eigenfunction	No smoothing	SSVD	S-Smooth	FACE	FACE incomplete data
Case 1	1	9.19	7.27	7.01	6.86	6.97
	2	16.95	12.12	11.76	11.65	11.96
	3	20.27	6.90	6.74	6.74	6.74
Case 2	1	10.05	6.41	6.39	6.29	6.34
	2	17.38	11.13	10.92	10.37	10.46
	3	19.71	6.75	6.51	6.08	6.23
Case 3	1	3.14	0.58	0.58	0.58	0.58
	2	23.84	4.40	4.37	4.37	4.37
	3	55.51	14.07	13.40	13.41	13.14
Case 4	1	5.09	1.81	1.80	1.80	1.87
	2	20.14	8.23	8.20	8.20	8.67
	3	42.04	19.39	19.39	19.40	20.70
Case 5	1	70.34	64.71	64.71	64.71	65.79
	2	96.39	90.57	90.31	90.38	90.84
	3	93.09	84.15	83.88	83.99	84.66

Table 2.

100×MISEs of the three methods for estimating the covariance function. The incomplete data has about 13% observations missing.

	SSVD	S-Smooth	FACE	FACE incomplete data
Case 1	9.34	8.96	8.94	8.93
Case 2	8.96	8.64	8.62	8.69
Case 3	1.22	0.76	0.76	0.76
Case 4	0.11	0.07	0.07	0.08
Case 5	2.69	1.98	1.98	2.18

Figure 2 shows boxplots of estimated eigenvalues that are centered and standardized, λ̂_k/λ_k − 1. The SSVD method works well for cases 1 and 2, where the true covariance has only three non-zero eigenvalues, but tends to overestimate the eigenvalues for the other three cases, where the covariance function has an infinite number of non-zero eigenvalues. In contrast, the FACE and S-Smooth estimators underestimate the eigenvalues for the simple cases 1 and 3 but are much closer to the true eigenvalues for the more complex cases. Table 3 provides the average mean squared errors (AMSEs) of λ̂_k/λ_k − 1 for k = 1,2, 3, and indicates that S-Smooth and FACE tend to estimate the eigenvalues more accurately.

Fig. 2.

Boxplots of the centered and standardized estimated eigenvalues, λ̂_k/λ_k − 1. The top panel is for case 2, the middle panel is for case 4, and the bottom panel is for case 5. The zero is shown by the solid red line. Case 1 is similar to case 2 and case 3 is similar to case 4, and hence are not shown.

Table 3.

100× average (λ̂_k/λ_k − 1)² of the three methods for estimating the eigenvalues. The incomplete data has about 13% observations missing.

	Eigenvalue	SSVD	S-Smooth	FACE	FACE incomplete data
Case 1	1	4.37	3.99	3.99	4.31
	2	3.43	3.68	3.76	3.96
	3	3.97	4.95	5.03	4.99
Case 2	1	4.40	4.05	4.05	4.10
	2	3.58	3.78	3.81	3.83
	3	3.38	4.02	4.38	4.22
Case 3	1	3.80	3.55	3.55	3.55
	2	9.79	3.38	3.38	3.42
	3	48.27	4.03	4.03	3.96
Case 4	1	4.22	3.81	3.81	3.84
	2	5.65	3.69	3.69	3.64
	3	14.77	3.53	3.53	3.43
Case 5	1	12.45	6.45	6.45	7.05
	2	4.35	2.09	2.09	2.03
	3	3.05	1.64	1.64	1.55

6.2 Incomplete data

In Section 4.2 we extended FACE for incomplete data, and here we illustrate the extension with a simulation.We use the same simulation setting in Section 6.1 except that for each subject we allow for portions of observations missing completely at random. For simplicity we fix the length of each portion so that 0.065J consecutive observations are missing. We allow one subject to miss either 1, 2, or 3 portions with equal probabilities so that in expectation 13% of the data are missing. Note that the real data we will consider later also has about 13% measurements missing.

In Figure 2, boxplots of the estimated eigenvalues are shown. The MISEs of the estimated covariance function and estimated eigenfunctions and the AMSEs of the estimated eigenvalues appear in Tables 2, 1 and 3, respectively. The simulation results show that the performance of FACE degrades only marginally.

6.3 Computation time

We record the computation time of FACE for various combinations of J and I. All other settings remain the same as in the first simulation study and we use the eigenfunctions from case 1. For comparison the computation times of SSVD, S-Smooth and the sandwich smoother (Xiao et al. 2013) are also given. Table 4 summarizes the results and shows that FACE is fast even with high-dimensional data while the computation time of the sandwich smoother increases dramatically with J, the dimension of the problem. For example it took FACE only 5 seconds to smooth a 10,000 by 10,000 dimensional matrix for 500 subjects, while the sandwich smoother did not run on our computer. While SSVD, S-Smooth and FACE are all fast to compute, FACE is computationally faster when I =500.We note that S-Smooth has additional problems when data are missing, though a method similar to FACE may be devised. Ultimately, we prefer the self-contained, fast, and flexible FACE approach.

Although we do not run FACE on ultrahigh-dimensional data, we can obtain a rough estimate of the computation time by the formula O(JIc). Table 4 shows that FACE with 500 knots takes 5 seconds on data with (J, I) = (10000,500). For data with J equal to 100,000 and I equal to 2,000, FACE with 500 knots should take 4 minutes to compute, without taking into account the time for loading data into the computer memory. Our code was written and run in R, so a faster implementation of FACE may be possible on other software platforms.

7 Example

The Sleep Heart Health Study (SHHS) is a large-scale study of sleep and its association with health-related outcomes. Thousands of subjects enrolled in SHHS underwent two in-home polysomnograms (PSGs) at multiple visits. Two-channel electroencephalographs (EEG), part of the PSG, were collected at a frequency of 125Hz, or 125 observations per second for each subject, visit and channel. We model the proportion of δ-power which is a summary measure of the spectrum of the EEG signal. More details on δ-power can be found in Crainiceanu et al. (2009) and Di et al. (2009). The data contain 51 subjects with sleep-disordered breathing (SDB) and 51 matched controls; see Crainiceanu et al. (2012) and Swihart et al. (2012) for details on how the pairs were matched. An important feature of the EEG data is that long consecutive portions of observations, which indicate wake periods, are missing. Figure 3 displays data from 2 matched pairs. In total about 13% of the data is missing.

Fig. 3.

Data for two matched pairs of case and controls in the Sleep Heart Health Study. The red lines are for cases while the black are for controls. For simplicity only the last observation in each minute of the 4-hour interval is shown.

Similar to Crainiceanu et al. (2012), we consider the following statistical model. The data for proportion of δ-power are pairs of curves {Y_iA(t),Y_iC(t)}, where i denotes subject, t = t₁, …, t_J (J = 2,880) denotes the time measured in 5-second intervals in a 4-hour sleep interval from sleep onset, A stands for apneic and C stands for control. The model is

$$\{\begin{array}{c}{Y}_{iA}(t)={\mu }_A(t)+X_i(t)+U_{iA}(t)+{\epsilon }_{iA}(t)\hfill \\ Y_{iC}(t)={\mu }_C(t)+X_i(t)+U_{iC}(t)+{\epsilon }_{iC}(t)\hfill \end{array}$$ (4)

where μ_A(t) and μ_C(t) are mean functions of proportions of δ-power, X_i(t) is a functional process with mean 0 and continuous covariance operator K_X (·, ·), U_iA(t) and U_iC(t) are functional processes with mean 0 and continuous covariance operator K_U(·, ·), and ε_iA(t), ε_iC(t) are measurement errors with mean 0 and variance σ². The random processes X_i,U_iA,U_iC, ε_iA and ε_iC are assumed to be mutually independent. Here X_i accounts for the between-pair correlation of the data while U_iA and U_iC model the within-pair correlation. The Multilevel Functional Principal Component Analysis (MFPCA) (Di et al. 2009) can be used to analyze data with model (4). One crucial step of MFPCA is to smooth two estimated covariance operators which in this example are 2880 × 2880 matrices.

Smoothing large covariance operators of dimension 2880 × 2880 can be computationally expensive. We tried bivariate thin plate regression splines and used the R function ‘bam’ in the mgcv package (Wood 2013) with 35 equally-spaced knots for each axis. The smoothing parameter was automatically selected by ‘bam’ with the option ‘GCV.cp’. Running time for thin plate regression splines was three hours. Because the two covariance operators take the form in Section 4.1 (see the details in Appendix B), we applied FACE, which ran in less than 10 seconds with 100 knots. Note that we also tried thin plate splines with 100 knots in mgcv, which was still running after 10 hours. Figure 4 displays the first three eigenfunctions for K_X and K_U, using both methods. As a comparison, the eigenfunctions using SSVD are also shown. For the SSVD method, to handle incomplete data the SVD step was replaced by a brute-force decomposition of the two 2880 × 2880 covariance operators. Figure 4 shows that the top eigenfunctions obtained from the two bivariate smoothing methods are quite different, except for the first eigenfunctions on the top row. The estimated eigenfunctions using FACE in general resemble those by SSVD with some subtle differences, while thin plate splines in this example seem to over-smooth the data, probably because we were forced to use a smaller number of knots.

Fig. 4.

The eigenfunctions associated with the top three eigenvalues of K_X and K_U for the Sleep Heart Health Study data. The left column is for K_X and the right one is for K_U. The red and green solid lines correspond to the FACE approach using the original and modified GCV, respectively. The black dashed lines are for thin plate splines, and the cyan dotted lines are for SSVD.

The smoothed eigenfunctions from FACE using PGCV (red solid lines in Figure 4) appear undersmooth. This may be due to the well reported tendency of GCV to undersmooth as well as to the noisy and complex nature of the data. A common way to combat this problem is to use modified GCV (modified PGCV for our case) where tr(S) in (2) is multiplied by a constant α that is greater than 1; see Cummins et al. (2001) and Kim and Gu (2004) for such practices for smoothing splines. Similar practice has also been proposed for AIC in Shinohara et al. (2014). We re-ran the FACE method with α = 2 and the resulting estimates (green solid lines in Figure 4) appear more satisfactory. In this case, the direct smoothing approach of the eigenfunctions (Rice and Silverman 1991; Capra and Müller 1997; Ramsay and Silverman 2005) might provide good results. However, the missing data issue and the computational difficulty associated with large J make the approach difficult to use.

Table 5 provides estimated eigenvalues of K_X and K_U. Compared to FACE (with α = 2), thin plate splines over-shrink significantly the eigenvalues, especially those of the between pair covariance. The results from FACE in Table 5 show that the proportion of variability explained by K_X, the between-pair variation, is 14.40/(14.40 + 22.75) ≈ 38.8%.

Table 5.

Estimated eigenvalues of K_X and K_U. All eigenvalues are multiplied by J to refer to the variation in the data explained by the eigenfunctions. The row ‘all’ refers to the sum of all positive eigenvalues.

	Eigenfunction	SSVD	FACE	Thin Plate Splines
KX	1	4.31	3.92	1.91
	2	2.64	2.66	0.50
	3	1.88	1.35	0.31
	all	48.14	14.40	2.81
KU	1	8.84	6.33	6.75
	2	5.69	3.18	2.55
	3	5.03	2.86	2.04
	all	107.95	22.75	12.95

8 Discussion

In this paper we developed a fast covariance estimation (FACE) method that could significantly alleviate the computational difficulty of bivariate smoothing and eigendecomposition of large covariance matrices in FPCA for high-dimensional data. Because bivariate smoothing and eigendecomposition of covariance matrices are integral parts of FPCA, our method could increase the scope and applicability of FPCA for high-dimensional data. For instance, with FACE, one may consider incorporating high-dimensional functional predictors into the penalized functional regression model of Goldsmith et al. (2011).

The proposed FACE method can be regarded as a two-step procedure such as S-Smooth (see, e.g., Besse and Ramsay 1986; Ramsay and Dalzell 1991; Besse et al. 1997; Cardot 2000; Zhang and Chen 2007). Indeed, if we first smooth data at the subject level Ŷ_i = SY_i, i = 1, …, I, then it is easy to show that the empirical covariance estimator of the Ŷ_i is equal to e K̃. There are, however, important computational differences between FACE and the current two-step procedures. First, the fast algorithm in Section 3.2 enables FACE to select efficiently the smoothing parameter. Second, FACE could work with structured functional data and allow for different smoothing for each covariance operator. Third, FACE can be easily extended for incomplete data where long consecutive portions of data are missing while it is unclear how a two-step procedure could be used for such data.

The second approach, SSVD, is very simple and reasonable, though some problems remain open, especially in applications with missing data. Another drawback of SSVD is that the smoothed eigenvectors are not necessarily orthogonal, though the fast Gram-Schmidt algorithm could easily be applied to the smooth vectors. Overall, we found that using a combination of FACE and SSVD provides a reasonable and practical starting point for smoothing covariance operators for high dimensional functional data, structured or unstructured.

In this paper we have only considered the case when the sampling points are the same for all subjects. Assume now for the ith sample that we observe Y_i = {Y_i(t_i1), …,Y_i(t_iJ_i)}^T, where t_ij, j = 1, …,J_i can be different across subjects. In this case the empirical estimator of the covariance operator does not have a decomposable form. Consider the scenario when subjects are densely sampled and all J_i’s are large. Using the idea from Di et al. (2009), we can undersmooth each Y_i using, for example, a kernel smoother with a small bandwidth or a regression spline. FACE can then be applied on the under-smoothed estimates evaluated at an equally spaced grid, {Ŷ₁, …,Ŷ_I}. Extension of FACE to the sparse design scenario remains a difficult open problem.

A Appendix: Proofs

Proof of Proposition 1: The design matrix B is of full rank (Xiao et al. 2012). Hence B^TB is invertible and A_S is of rank c. Σ_S is a diagonal matrix with all elements greater than 0 and Ỹ is of rank at most min(c, I). Hence $\mathbf{K̃}={\mathbf{A}}_S\left(I^{-1}{\mathit{\Sigma }}_S\mathbf{Ỹ}{\mathbf{Ỹ}}^T{\mathit{\Sigma }}_S\right){\mathbf{A}}_S^T$ has a rank at most min(c, I) and the proposition follows.

Proof of Proposition 2: First of all, tr(S) = tr(Σ_S) which is easy to calculate. We now compute ${\sum }_{i=1}^I{\Vert {\mathbf{Y}}_i-\mathbf{S}{\mathbf{Y}}_i\Vert }^2$. Because ${\Vert {\mathbf{Y}}_i-\mathbf{S}{\mathbf{Y}}_i\Vert }^2={\mathbf{Y}}_i^T{(\mathbf{S}-{\mathbf{I}}_J)}^2{\mathbf{Y}}_i=\text{tr}\{{(\mathbf{S}-{\mathbf{I}}_J)}^2{\mathbf{Y}}_i{\mathbf{Y}}_i^T\}$,

$$\sum _{i=1}^I{\Vert {\mathbf{Y}}_i-\mathbf{S}{\mathbf{Y}}_i\Vert }^2=\text{tr}\left\{{(\mathbf{S}-{\mathbf{I}}_J)}^2\sum _{i=1}^I{\mathbf{Y}}_i{\mathbf{Y}}_i^T\right\}=\text{tr}\{{(\mathbf{S}-{\mathbf{I}}_J)}^2\mathbf{Y}{\mathbf{Y}}^T\}.$$

It can be shown that ${\mathbf{S}}^2={\mathbf{A}}_S{\mathit{\Sigma }}_S^2{\mathbf{A}}_S^T$. Hence $\text{tr}({\mathbf{S}}^2\mathbf{Y}{\mathbf{Y}}^T)=\text{tr}({\mathbf{Y}}^T{\mathbf{S}}^2\mathbf{Y})=\text{tr}({\mathbf{Ỹ}}^T{\mathit{\Sigma }}_S^2\mathbf{Ỹ})=\text{tr}({\mathit{\Sigma }}_S^2\mathbf{Ỹ}{\mathbf{Ỹ}}^T)$. Similarly, we derive tr(SYY^T) = tr(Σ_SỸỸ^T). We have $\text{tr}(\mathbf{Y}{\mathbf{Y}}^T)={\Vert \mathbf{Y}\Vert }_F^2$. It follows that

$$\sum _{i=1}^I{\Vert {\mathbf{Y}}_i-\mathbf{S}{\mathbf{Y}}_i\Vert }^2=\text{tr}\left\{{({\mathit{\Sigma }}_S-{\mathbf{I}}_c)}^2\mathbf{Ỹ}{\mathbf{Ỹ}}^T\right\}-{\Vert \mathbf{Ỹ}\Vert }_F^2+{\Vert \mathbf{Y}\Vert }_F^2.$$

Proposition 3 The computation time of FACE is O (IJc + Jc² + c³ + ck₀), where k₀ is the number of iterations needed for selecting the smoothing parameter (see Section 3.2), and the total required computer memory is O (JI + I² + Jc + c² + k₀) memory units.

Proof of Proposition 3: We need to compute or store the following quantities: X, B, B^TB, (B^TB)^−1/2, P, (B^TB)^−1/2P(B^TB)^−1/2, A_S, Ỹ, A, U, and A_SA. For the computational complexity, B^TB, A_S = B(B^TB)^−1/2U, and A_SA require O(Jc²) computations; (B^TB)^−1/2, P, (B^TB)^−1/2P(B^TB)^−1/2, A, and U require O(c³) computations; $\mathbf{Ỹ}={\mathbf{A}}_S^T\mathbf{Y}$ requires O(JIc) computations. So in total, O(JIc + Jc² + c³) computations are required. For the memory burden, the loading of Y requires O(JI) memory units, computer of B and A_SA requires O(Jc) memory units, and other objects require O(c²) memory units.

Proof of Theorem 1:We have ${\mathbf{\xi ̂}}_i=J^{-1/2}{({\mathbf{A}}_S{\mathbf{Â}}_N)}^T{\mathbf{Y}}_i=J^{-1/2}{\mathbf{Â}}_N^T({\mathbf{A}}_S^T{\mathbf{Y}}_i)=J^{-1/2}{\mathbf{Â}}_N^T{\mathbf{Ỹ}}_i$. Proof of Theorem 2: Let Ã_N denote the first N columns of A_SA, then Ã_N = A_SÂ. The estimated BLUPs for ξ_i (Ruppert et al. 2003) is

$${\mathit{\xi ̂}}_i=J^{-1/2}{\mathit{\Sigma ̂}}_N{\mathbf{Ã}}_N^T{({\mathbf{Ã}}_N{\mathit{\Sigma ̂}}_N{\mathbf{Ã}}_N^T+J^{-1}{\mathrm{\sigma ̂}}^2{\mathbf{I}}_J)}^{-1}{\mathbf{Y}}_i.$$

The inverse matrix in the above equality can be replaced by the following (Seber (2007), page 309, equality b(i)),

$${({\mathbf{Â}}_N{\mathit{\Sigma ̂}}_N{\mathbf{Ã}}_N^T+J^{-1}{\mathrm{\sigma ̂}}^2{\mathbf{I}}_J)}^{-1}=\frac{J}{{\mathrm{\sigma ̂}}^2}\{{\mathbf{I}}_N-\frac{J}{{\mathrm{\sigma ̂}}^2}{\mathbf{Ã}}_N{({\mathit{\Sigma ̂}}_N^{-1}+\frac{J}{{\mathrm{\sigma ̂}}^2}{\mathbf{I}}_N)}^{-1}{\mathbf{Ã}}_N^T\}.$$

It follows that

$$\mathit{\xi ̂}=J^{-1/2}\frac{J}{{\mathrm{\sigma ̂}}^2}\mathit{\Sigma ̂}\{{\mathbf{I}}_N-\frac{J}{{\mathrm{\sigma ̂}}^2}{({\mathit{\Sigma ̂}}_N^{-1}+\frac{J}{{\mathrm{\sigma ̂}}^2}{\mathbf{I}}_N)}^{-1}\}{\mathbf{Â}}_N^T{\mathbf{Ỹ}}_i=J^{-1/2}{\mathit{\Sigma ̂}}_N{({\mathit{\Sigma ̂}}_N+J^{-1}{\mathrm{\sigma ̂}}^2{\mathbf{I}}_N)}^{-1}{\mathbf{Â}}_N^T{\mathbf{Ỹ}}_i.$$

B Appendix: Empirical covariance operators for K_X and K_U

Let I denote the number of pairs of cases and controls. For simplicity, we assume estimates of μ_A(t) and μ_C(t) have been subtracted from Y_iA and Y_iC, respectively. Let Y_iA = (Y_iA(t₁), …,Y_iA(t_T))^T and Y_iC = (Y_iC(t₁), …,Y_iC(t_J))^T. By Zipunnikov et al. (2011), we have estimates of the covariance operators,

$${\mathbf{K̂}}_X-\frac{1}{2I}\sum _{i=1}^I({\mathbf{Y}}_{iA}{\mathbf{Y}}_{iC}^T+{\mathbf{Y}}_{iC}{\mathbf{Y}}_{iA}^T),$$

and

$${\mathbf{K̂}}_U=\frac{1}{2I}\sum _{i=1}^I({\mathbf{Y}}_{iA}-{\mathbf{Y}}_{iC}){({\mathbf{Y}}_{iA}-{\mathbf{Y}}_{iC})}^T.$$

Let Y_A = [Y_1A, …,Y_nA], Y_C = [Y_1C, …,Y_nC] and Y = [Y_A,Y_C]. Then Y is of dimension J × 2I. It can be shown that K̂_X = YH_XY^T and K̂_U = YH_UY^T, where

$${\mathbf{H}}_X=\frac{1}{2I}\left(\begin{array}{cc}{\mathbf{0}}_I\hfill & {\mathbf{I}}_I\hfill \\ {\mathbf{I}}_I\hfill & {\mathbf{0}}_I\hfill \end{array}\right),{\mathbf{H}}_U=\frac{1}{2I}\left(\begin{array}{cc}{\mathbf{I}}_I\hfill & -{\mathbf{I}}_I\hfill \\ -{\mathbf{I}}_I\hfill & {\mathbf{I}}_I\hfill \end{array}\right).$$