Localized Functional Principal Component Analysis

Abstract

We propose localized functional principal component analysis (LFPCA), looking for orthogonal basis functions with localized support regions that explain most of the variability of a random process. The LFPCA is formulated as a convex optimization problem through a novel Deflated Fantope Localization method and is implemented through an efficient algorithm to obtain the global optimum. We prove that the proposed LFPCA converges to the original FPCA when the tuning parameters are chosen appropriately. Simulation shows that the proposed LFPCA with tuning parameters chosen by cross validation can almost perfectly recover the true eigenfunctions and significantly improve the estimation accuracy when the eigenfunctions are truly supported on some subdomains. In the scenario that the original eigenfunctions are not localized, the proposed LFPCA also serves as a nice tool in finding orthogonal basis functions that balance between interpretability and the capability of explaining variability of the data. The analyses of a country mortality data reveal interesting features that cannot be found by standard FPCA methods.

1. INTRODUCTION

Functional principal component analysis has emerged as a major tool to explore the source of variability in a sample of random curves and has found wide applications in functional regression, curve classification, and clustering (Castro et al., 1986; Rice and Silverman, 1991; Cardot, 2000; Yao et al., 2005; Ramsay and Silverman, 2005; Hall and Hosseini-Nasab, 2006). In this paper, we consider functional principal component analysis with localized support regions. That is, for a smooth random function X, we look for orthogonal basis functions with localized support regions that explain most of the variance. The main motivation of the localized functional principal component analysis (LFPCA) is to find a parsimonious linear representation of the data that balances the interpretability and the capability of explaining variance of the stochastic process.

The proposed method outputs localized basis functions whose localization level is controlled by a localization tuning parameter. We propose two methods to select the localization parameter, corresponding to two useful applications of our proposed method. First, one can choose the localization parameter by maximizing the explained variance of the random process computed by V-fold cross-validation. Our simulation shows that when the eigenfunctions truly have localized support regions, the proposed LFPCA with a localization parameter chosen by cross-validation significantly improves the estimation accuracy of the eigenfunctions compared to standard FPCA methods. On the other hand, when the original eigenfunctions are not localized, the localization parameter chosen by cross-validation is expected to be very close to zero and it is confirmed by our numerical studies that the performance of the proposed LFPCA is almost identical to standard FPCA methods. The second method of choosing the localization parameter is to seek the most localized basis functions that explain a fixed level of variance. This method is particularly useful when the standard eigenfunctions are not localized, and it makes sense for the proposed LFPCA not to target at the standard eigenfunctions but to balance between interpretability and the capability of explaining variance of the stochastic process. Details can be found in Section 3.2. We consider a country mortality data application to illustrate the second method of choosing the localization parameter. In the country mortality data, the mortality rates at age 60 were recorded from year 1960–2006 for 27 countries around the world. The first three localized basis functions, explaining more than 85% variance, correspond to variational modes around mid 1990s, 1980s, and 1960s, respectively. Another example concerning a growth curve data is presented in the Online Supplementary File. The first two localized basis functions explain more than 85% variance, and clearly indicate that the main variational modes of female height growth are around age 12 and around age 5, perfectly matching the knowledge of growth spurts. These interesting features cannot be revealed by standard FPCA.

Domain localization has been studied by several authors in the functional regression model: E(Y|X) = a + X(t) β(t) dt, where the coefficient function β(t) is desired to be zero outside a subdomain ∈ with the purpose of improved interpretability (James et al., 2009; Zhao et al., 2012; Zhou et al., 2013). Most of these methods turn the problem into a variable selection problem and use LASSO type penalties. In a recent thesis work Lin (2013), interpretable functional principal component analysis is studied, which has a very similar flavor as our proposed LFPCA. In their work, an ℓ₀ penalty is added on eigenfunctions and a greedy algorithm based on basis expansion of curves is developed to solve the non-convex optimization problem. The formulation of localization or domain selection in the context of functional principal component analysis is quite challenging for at least two reasons. First, the eigen problem together with a localization penalty is usually not convex, and in general it is an NP-hard problem to find a global optimum. Second, in order to obtain a sequence of mutually orthogonal eigen-components, a commonly taken procedure is to deflate the empirical covariance operator at step j by removing the effect of the previous j − 1 components (White, 1958; Mackey, 2008). But with the localization penalty in the objective function, this procedure can not guarantee the orthogonality of such sequentially obtained eigen-components. In sequential estimation of principal components, being orthogonal to the first component is a natural requirement when looking for the second component, otherwise the maximization over second direction is not well-defined since the solution would still be the first direction. From a dimension reduction perspective, the orthogonality is also appealing since the resulting k dimensional orthogonal basis leads to very simple calculation for subsequent inferences.

The main contribution of this paper is three-fold. First, we formulate the LFPCA as a convex optimization problem with explicit constraint on the orthogonality of eigen-components. Second, we provide an efficient algorithm to obtain the global maximum of this convex problem. Third, we carefully investigate the estimation error from the discretized data version to the functional continuous version, as well as the complex interaction between the eigen problem and the localization penalty, and prove consistency of the estimated eigenfunctions. The starting point of our method is a sup-norm consistent estimator of the covariance operator, up to a constant shift on the diagonal. For dense and equally spaced observations with or without measurement error, the proposed method can be directly carried out on the sample covariance, i.e., without the need to perform basis expansion, smoothing of the individual curves, or smoothing of the estimated covariance operator. For other designs of functional data, the proposed method is still applicable when an appropriate covariance estimator is available.

Our formulation of LFPCA borrows ideas from recent developments in sparse principal component analysis. In Vu et al. (2013); Lei and Vu (2015), a similar convex framework based on Fantope Projection and Selection has been proposed to estimate a k dimensional sparse principal subspace of a high dimensional random vector (see also d’Aspremont et al. (2007) for k = 1). These sparse subspace methods are useful when the union of the support regions of several leading eigenvectors is sparse. In sparse PCA settings, the notion of sparsity requires the proportion of non-zero entries in the leading eigenvectors to vanish as the dimensionality increases and therefore it makes sense to consider the union of the support regions of several leading eigenvectors to be sparse. However, in functional data settings, the length ratio of a support subdomain over the entire domain is determined by the random curve model and usually a constant, and the union of several leading subdomains can be as large as the entire domain. This is also the reason that we use the notion “localized” instead of “sparse”. It has remained challenging to obtain sparse eigenvectors sequentially that each one is allowed to have a different support region. A particular challenge is the interaction between orthogonality and the sparse penalty. Besides the difference between functional PCA and sparse PCA, one main extension developed in our method is the construction of a deflated Fantope to estimate individual eigen-components sequentially, with possibly different support regions and guaranteed orthogonality. This deflated Fantope formulation is of independent interest in many other structured principal component analysis.

The rest of this paper is organized as follows. In section 2, we introduce the formulation of localized functional principal component analysis. Section 3 derives the solution to the optimization problem and describes the algorithm as well as the selection of tuning parameters. Section 4 contains the consistency results. Section 5 and Section 6 present numerical experiments and data examples to illustrate our method. Section 7 contains some discussions and extensions. Technical details and additional materials are provided in the Online Supplementary File.

2. LFPCA THROUGH DEFLATED FANTOPE LOCALIZATION

We consider a square integrable random process X(t) : ↦ ℝ over a compact interval ⊂ ℝ, with mean and covariance functions μ(t) = EX(t) and Γ(s, t) = Cov(X(s), X(t)), and covariance operator (Γf)(t) = f (s) Γ(t, s)ds. Under the minimal assumption that Γ(s, t) is continuous over (s, t), this operator Γ has orthonormal eigenfunctions ϕ_j (t), j = 1, 2, ··· , with nonincreasing eigenvalues λ_j, satisfying Γϕ_j = λ_j ϕ_j. The well known Karhunen-Loève expansion then gives the representation

$$X(t)=\mu (t)+\sum _{j=1}^{\infty }{\xi }_j{\varphi }_j(t),$$ (1)

where ξ_j, j ≥ 1, is a sequence of uncorrelated random variables satisfying E(ξ_j) = 0 and var(ξ_j) = λ_j, with an explicit representation ξ_j = (X(t) − μ(t))ϕ_j (t)dt. A key inference task in functional principal component analysis (FPCA) is to estimate the leading eigen-functions ϕ_j (t), 1 ≤ j ≤ k, from n independent sample curves.

In practice, the underlying sample curves X_i(t), 1 ≤ i ≤ n, are usually recorded at a grid of points and may be contaminated with additive measurement errors. We start with dense and equally spaced observations

$$Y_{il}=X_i(t_l)+{\epsilon }_{il}=\mu (t_l)+\sum _{j=1}^{\infty }{\xi }_{ij}{\varphi }_j(t_l)+{\epsilon }_{il},i=1,\dots ,n,l=1,\dots ,p,$$ (2)

where ε_il are independent noises with mean zero and variance σ², and t_l, 1 ≤ l ≤ p, are grid points in at which observations are recorded. Starting from such discrete and possibly noisy data, there are different ways of introducing smoothness in the estimation of FPCA. Rice and Silverman (1991); Silverman (1996); Huang et al. (2008), among others, studied approaches where a smoothness penalty on eigenfunctions is integrated in the optimization step of eigen-decomposition. Let S be the p × p sample covariance matrix of the observed vector Y, and v be a p dimensional vector. Rice and Silverman (1991) used a roughening matrix D = Δ^TΔ where Δ ∈ ℝ^(p−2)×p is a second-differencing operator:

$${\mathrm{\Delta }}_{ij}=\{\begin{array}{ll}1,\hfill & \text{if}j\in \{i,i+2\},\hfill \\ -2,\hfill & \text{if}j=i+1,\hfill \\ 0,\hfill & \text{otherwise}.\hfill \end{array}$$

A smoothed eigenfunction estimator is obtained by solving the following eigen problem:

$$max{v}^T(S-{\rho }_{1}D)v,\text{s.t.}{\Vert v\Vert }_2=1,$$ (3)

where ρ₁ is a smoothing parameter, and ||v||₂ is the Euclidean norm of v.

A straightforward approach to localize the estimated eigenfunctions is to add another localization penalty:

$$max{v}^T(S-{\rho }_{1}D)v-{\rho }_2{\Vert v\Vert }_1,\text{s.t.}{\Vert v\Vert }_2=1,$$ (4)

where ρ₂ is a tuning parameter, and ||v||₁ is the ℓ₁ norm of v. However, this is not a convex problem and there are no known algorithms that can efficiently find a global optimum even for the first eigen-component.

Here we propose a novel sequential estimation procedure based on the idea of estimating the rank one projection matrix vv^T. Let 〈A, B〉 = trace(A^T B) for matrices A, B of compatible dimensions. Denote ||H||_1,1 the matrix ℓ₁ norm, which is the sum absolute value of all entries in H. Starting from Π̂₀ = 0, for each j = 1, … , k, the jth localized eigen-component is estimated as follows.

$$\begin{array}{l}{H}_j=argmax\langle S-{\rho }_{1}D,H\rangle -{\rho }_2{\Vert H\Vert }_{1,1},\text{s.t.}H\in {\mathcal{D}}_{{\widehat{\mathrm{\Pi }}}_{j-1}},\\ {\widehat{v}}_j=\text{the}\text{first}\text{eigenvector}\text{of}{H}_j,\\ {\widehat{\mathrm{\Pi }}}_j={\widehat{\mathrm{\Pi }}}_{j-1}+{\widehat{v}}_j{\widehat{v}}_j^T,\end{array}$$ (5)

where, for any p × p projection matrixΠ,

$${\mathcal{D}}_{\mathrm{\Pi }}:=\{H:0\preccurlyeq H\preccurlyeq I,\text{trace}(H)=1,\text{and}\langle H,\mathrm{\Pi }\rangle =0\},$$

and, for symmetric matrices A, B, “A ⪯ B” means that B − A is positive semidefinite.

Problem (5) is a convex relaxation of (4) with integrated orthogonality constraints on the estimated localized eigen-components. With the estimated v̂_j, we can easily obtain an estimate ϕ̂_j(t) of the localized eigenfunction ϕ_j(t) by standard interpolation techniques such as linear interpolation, plus an optional final step of re-orthogonalization and re-normalization.

With appropriately chosen tuning parameters, the performance of the proposed method ties to the maximum entry-wise error of the discretized covariance estimator S (see Section 4 for details). Our presentation will focus on a sample covariance S that is computed from dense and equally spaced observations, but the proposed method is not restricted to a dense regular design as long as a reasonable covariance estimate can be obtained. More discussions can be found in Section 7.

To solve for v̂_j and ϕ̂_j(t), the key step in (5) is to solve

$$\underset{H}{max}\langle S-{\rho }_{1}D,H\rangle -{\rho }_2{\Vert H\Vert }_{1,1},\text{s.t.}H\in {\mathcal{D}}_{\mathrm{\Pi }},$$ (6)

where Π = Π̂_j−1 at step j. In next section we present an algorithm that solves problem (6), with a discussion on the choice of tuning parameters ρ₁ and ρ₂.

In the sparse PCA literature, d’Aspremont et al. (2007); Vu et al. (2013) have considered the following problem,

$$\underset{H}{max}\langle S,H\rangle -\rho {\Vert H\Vert }_{1,1},\text{s.t.}H\in {\mathcal{F}}^d,$$ (7)

where the convex set := {H : 0 ⪯ H ⪯ I, trace(H) = d} is called the Fantope of degree d (Dattorro, 2005), and is the convex hull of all rank d projection matrices. While the convex relaxation given in (7) allows us to estimate d-dimensional sparse principal subspaces, it does not lead to a sequence of mutually orthogonal eigenvectors with different support regions.

To ensure orthogonality among the estimated eigenvectors, we consider := {H : H ∈ , and 〈H, Π 〈 = 0}, which we call the deflated Fantope. It can be naturally generalized to ${\mathcal{D}}_{\mathrm{\Pi }}^{\mathrm{d}}:=\{H:H\in {\mathcal{F}}^d,\text{and}\langle H,\mathrm{\Pi }\rangle =0\}$ to estimate mutually orthogonal principal subspaces. Such a feasibility deflation technique is quite different from the commonly suggested matrix deflation techniques in sequential estimation of eigenvectors (see Mackey (2008) for example).

3. ALGORITHM

3.1 Deflated Fantope Localization using ADMM

The main difficulty in solving problem (6) is the complex interaction between the ℓ₁ penalty and the deflated Fantope constraint. To overcome this difficulty, we write (6) in an equivalent form to separate the ℓ₁ penalty and deflated Fantope constraint:

$$\underset{H,Z}{min}{\mathbb{I}}_{{\mathcal{D}}_{\mathrm{\Pi }}}(H)-\langle S-{\rho }_{1}D,H\rangle +{\rho }_2{\Vert Z\Vert }_{1,1},\text{s.t.}H-Z=0,$$ (8)

where is the convex indicator function, which is ∞ outside and 0 inside . Problem (8) is a convex global variable consensus optimization, which can be solved using alternating direction method of multipliers (ADMM, Boyd et al. (2011)). We describe in Algorithm 1 an ADMM algorithm that solves (8) and hence (6). It extends the FPS algorithm in Vu et al. (2013) to the deflated Fantope.

The two matrix operators used in the algorithm are defined as follows.

Algorithm 1.

Deflated Fantope Localization using ADMM

Require: S = ST, Π, D, ρ1, ρ2 ≥ 0, τ > 0, ε > 0
Z(0) ← 0, W(0) ← 0	▷ Initialization
repeat r = 1, 2, …
H(r) ← [Z(r−1) − W(r−1) + (S − ρ1D)/τ]	▷ Deflated Fantope projection
Z(r) ← (H(r) + W(r−1))	▷ Elementwise soft thresholding
W(r) ← W(r−1) + H(r) − Z(r)	▷ Dual variable update
until ${\Vert H^{(r)}-Z^{(r)}\Vert }_F^2\vee {\tau }^2{\Vert Z^{(r)}-Z^{(r-1)}\Vert }_F^2\le {\epsilon }^2$	▷ Stopping criterion
return Z(r)

1. Soft-thresholding operator: for any a > 0,

   $${\mathcal{S}}_a(x)=\text{sign}(x)max(\mid x\mid -a,0).$$
2. Deflated-Fantope-projection operator: For any p×p symmetric matrix A and projection matrixΠ,

   $${\mathcal{P}}_{{\mathcal{D}}_{\mathrm{\Pi }}}(A):=arg\underset{B\in {\mathcal{D}}_{\mathrm{\Pi }}}{min}{\Vert A-B\Vert }_F^2$$

   is the Frobenius norm projection of A onto the deflated Fantope .

A non-trivial subroutine in Algorithm 1 is to calculate the deflated-Fantope-projection (A) for a symmetric matrix A. The following lemma gives a close-form characterization of the deflated-Fantope-projection operator.

Lemma 3.1

Let Π = VV^T, where V is a p × d matrix with orthonormal columns. Let U be a p × (p − d) matrix that forms an orthogonal complement basis of V. Then

$${\mathcal{P}}_{{\mathcal{D}}_{\mathrm{\Pi }}}(A)=U\left[\sum _{i=1}^{p-d}{\gamma }_i^{+}(\theta ){\eta }_i{\eta }_i^T\right]U^T,$$

where ${({\gamma }_i,{\eta }_i)}_{i=1}^{p-d}$ are eigenvalue-eigenvector pairs of $U^{T}AU:U^{T}AU={\sum }_{i=1}^{p-d}{\gamma }_i{\eta }_i{\eta }_i^T$, and ${\gamma }_i^{+}(\theta )=min(max({\gamma }_i-\theta ,0),1)$, with θ chosen such that ${\sum }_{i=1}^{p-d}{\gamma }_i^{+}(\theta )=1$.

The next theorem ensures the convergence of our algorithm to a global optimum of problem (6). The proofs of Lemma 3.1 and Theorem 3.2 are given in the Online Supplementary File.

Theorem 3.2

In Algorithm 1, Z^(r) → H^*, H^(r) → H^* as r → ∞, where H^* is a global optimum of problem (6).

In the proof of Theorem 3.2 we will see that the auxiliary number τ used in Algorithm 1 plays a role that is similar to the step size commonly seen in iterative convex optimization solvers. The particular choice of τ does not affect the theoretical convergence of the ADMM algorithm. There are some general guidelines on the practical choice of τ and we refer to Boyd et al. (2011) for further details.

3.2 Choice of Tuning Parameters

The optimization problem (5) involves two tuning parameters: ρ₁ controls the roughness of eigenfunctions; and ρ₂ controls the localization of the eigenfunctions. We present a two-step approach where ρ₁ is chosen first and kept the same for all 1 ≤ j ≤ k, and then ρ₂ is determined sequentially for each eigenfunction ϕ_j(t) and denoted by ρ_2,j. The two parameters can be chosen together by straightforward modification but computationally it would be a bit intensive.

The choice of the smoothing parameter ρ₁ has been discussed in Rice and Silverman (1991), and they recommended cross-validation or manual selection. In our simulation and data analysis, we have used V-fold cross-validation. First the data is divided into V folds, denoted by , , … , . Let $H_j^{(-v)}({\rho }_1,{\rho }_2)$ be the estimated H_j in (5) using data other than with tuning parameters ρ₁ and ρ₂. Let S^v be the discrete covariance estimated from data . The smoothing parameter is chosen by maximizing the cross-validated inner product of $H_1^{(-v)}$ and S^v:

$${\widehat{\rho }}_1=\underset{\rho \in {\mathcal{A}}_1}{argmax}\sum _{v=1}^V\langle H_1^{(-v)}(\rho ,0),S^v\rangle ,$$ (9)

where is a candidate set of ρ₁ and empirically we found that a sequence between 0 and p times the largest eigenvalue of S works well.

In the following, we present two methods for the choice of ρ₂ given a pre-chosen smoothing parameter ${\rho }_1^{\ast }$. The first method is to choose ρ_2,j by maximizing the cross-validated inner product of $H_j^{(-v)}$ and S^v:

$${\widehat{\rho }}_{2,j}=\underset{\rho \in {\mathcal{A}}_{2,j}}{argmax}\sum _{v=1}^V\langle H_j^{(-v)}({\rho }_1^{\ast },\rho ),S^v\rangle ,j=1,2,\dots ,k,$$ (10)

where is a candidate set for ρ_2,j, and we propose to use a sequence between 0 and the 95% quantile of absolute values of off-diagonal entries in S_j, with S_j = (I − Π̂_j−1)S(I − Π̂_j−1).

The V-fold cross-validation approach is expected to give a ρ̂₂ that indicates the true localization level of the eigenfunctions. The criterion in our proposed cross-validation corresponds to maximizing 〈Σ, Ĥ(ρ₂)〉, where Σ is the discretized population covariance and is substituted by the test sample covariance in practice. When the true eigenvector v is localized, 〈Σ, Ĥ(ρ₂)〉 shall be maximized at approximately the value of ${\rho }_2^{\ast }$ which corresponds to the ideal localization level of the eigenfunction, i.e., ${\Vert \widehat{H}({\rho }_2^{\ast })\Vert }_{1,1}\approx {\Vert {vv}^T\Vert }_{1,1}$. We shall expect 〈Σ, Ĥ(ρ₂)〉 as a function of ρ₂ to be (i) monotonically increasing on [0, ${\rho }_2^{\ast }$] as the search area gradually expands to cover the true eigenvector, and (ii) monotonically decreasing on [ ${\rho }_2^{\ast }$, ∞) as the search area goes unnecessarily larger so that the estimation becomes more noisy. When the true eigenvector v is not localized, then we shall expect 〈Σ, Ĥ(ρ₂)〉 to be monotonically decreasing as ρ₂ increases. The numerical study confirms the good performance of the cross-validation method. See Section 5 for more details.

In some applications, we may not want to target the standard eigenfunction, but instead we may want to find orthogonal linear expansions that balance the interpretability (localization) and the capability of explaining the variance of the process. We therefore propose a second method of choosing ρ₂, which is based on the notion of fraction of variance explained (FVE). For a p-dimensional vector v of unit length and a sup-norm consistent estimator S of the covariance operator,

$$\mathit{FVE}(v)=v^{T}Sv/\mathit{totV}(S),$$ (11)

where totV (S) is the sum of positive eigenvalues of S. We note that for dense and equally spaced observations with measurement error, the FV E(v) defined above is not directly applicable to a sample covariance S because the sample covariance is sup-norm consistent up to a shift σ² on the diagonal, where σ² is the error variance. To avoid serious bias by the nugget effect on the diagonal, one may use the eigenvalues from the smoothed covariance. Another practical way is to approximate totV (S) by the sum of the first M leading eigenvalues of S, for a finite number M. For a reasonable error level, the nugget effect bias Mσ² is small compared to the sum of the first M eigenvalues of S, which is of order p (Kneip et al., 2011), while the remaining true eigenvalues beyond M are usually very small because the smoothness of X(t) ensures fast decay of eigenvalues. In numerical experiments where FV E is needed for determining the number of principal components to be included, we use M = min(20, p − 2).

To sequentially select the sparsity parameter ρ₂ for the jth eigenfunction. Suppose that we have estimated v̂_i for 1 ≤ i ≤ j − 1. Let ${\widehat{v}}_j({\rho }_1^{\ast },\rho )$ be the solution of (5) by using a fixed ${\rho }_1^{\ast }$ and ρ₂ = ρ, with Π̂_j−1 being the projector of the subspace spanned by (v̂_i : 1 ≤ i ≤ j − 1). We can define

$$\mathit{rFVE}(\rho )=\frac{\mathit{FVE}({\widehat{v}}_j({\rho }_1^{\ast },\rho ))}{\mathit{FVE}({\widehat{v}}_j({\rho }_1^{\ast },0))}=\frac{{\widehat{v}}_j^T({\rho }_1^{\ast },\rho )S{\widehat{v}}_j({\rho }_1^{\ast },\rho )}{{\widehat{v}}_j^T({\rho }_1^{\ast },0)S{\widehat{v}}_j({\rho }_1^{\ast },0)},$$ (12)

and choose ρ_2,j as

$$max\{\rho \in {\mathcal{A}}_{2,j}:\mathit{rFVE}(\rho )\ge 1-a\},$$ (13)

where a ∈ [0, 1) is the proportion of FVE that one chooses to sacrifice in return of localization. For any a ∈ [0, 1), a ρ satisfying (13) always exists, because rFV E(ρ) = 1 for ρ = 0 and rFV E(ρ) ∈ [0, 1) for ρ > 0. Equation (12) also suggests that rFV E(ρ) can be calculated without computing totV (S). Although the first localized basis function explains less variance than the standard eigenfunction, the lost proportion is likely to be picked up by the second component, and we are still able to explain a large proportion of the total variance with a small number of components. We illustrate this method with real data analyses in Section 6.

4. ASYMPTOTIC PROPERTIES

In this section we establish the ℓ₂ consistency of the proposed estimator in an asymptotic setting where both the sample size n and the number of grid points p increase. We will first provide sufficient conditions on the tuning parameters ρ₁ and ρ₂ such that the LFPCA estimate is consistent. Our second result provides further insights on how the localization penalty ρ₂ affects the rate of convergence. We make the following assumptions.

• A1
  The input matrix S in (5) satisfies sup-norm consistency up to a constant shift on the diagonal: for some constant α ≥ 0 and a sequence e_n = o(1),

  $$\underset{1\le l,l^{\prime }\le p}{max}\mid S(l,l^{\prime })-\mathrm{\Gamma }(t_l,t_{l^{\prime }})-\alpha \mathbf{1}(l=l^{\prime })\mid =O_P(e_n),\text{as}(n,p)\to \infty .$$

Remark

Assumption (A1) puts a mild condition on the input matrix S that can be satisfied by many standard estimators. Consider functional data with dense and equally spaced observations. If S is the sample covariance estimator from the raw data, standard large deviation bounds such as Bernstein’s inequality (Van Der Vaart and Wellner (1996), Chapter II) imply that Assumption (A1) holds with $e_n=\sqrt{logp/n}$ if log p/n → 0 and the random curve X(t) as well as the observation error in model (2) has sub-Gaussian tails; see also Kneip et al. (2011). In this case α = σ², the noise variance. If a smoothed covariance estimator S is used, the sup-norm rate can be $\sqrt{logn/n}$ (Li et al., 2010). The convergence in other norms such as Frobenius norm can be found in (Hall and Hosseini-Nasab, 2006; Hall et al., 2006; Bunea and Xiao, 2015). In general, the consistency result does not really depend on the observational design as long as we can get an estimate of the covariance operator whose sup-norm error vanishes as n and p increase. More discussions about cases where a sample covariance is not feasible can be found in Section 7.

• A2There is a positive integer k such that the eigenvalues of Γ satisfies λ₁ >, …, > λ_k > λ_k+1 ≥, …, ≥ 0, with positive eigen-gap δ := min_1≤j≤k(λ_j − λ_j+1) > 0.
• A3
  Γ is Lipschitz continuous:

  $$\mid \mathrm{\Gamma }(s,t)-\mathrm{\Gamma }(s^{\prime },t^{\prime })\mid \le Lmax(\mid s-s^{\prime }\mid ,\mid t-t^{\prime }\mid ),\forall s,s^{\prime },t,t^{\prime }.$$
• A4
  The k leading eigenfunctions of Γ have Lipschitz first derivatives:

  $$\mid {\varphi }_j^{\prime }(t)-{\varphi }_j^{\prime }(s)\mid \le L\mid t-s\mid ,\forall 1\le j\le k.$$

Theorem 4.1 (ℓ₂ consistency)

Under assumptions (A1–A4), if ρ₁/p⁵ → 0 and ρ₂ → 0 as (n, p) → ∞, then for k as defined in A2, we have

$$\underset{1\le j\le k}{sup}{\Vert {\widehat{\varphi }}_j(t)-{\varphi }_j(t)\Vert }_2\stackrel{P}{\to }0.$$

The proof of Theorem 4.1 is given in the Online Supplementary File. Here we outline the proof, highlighting some key technical challenges.

Let u_j be the unit vector obtained by discretizing and re-normalizing the eigenfunction ϕ_j(t). We will prove Theorem 4.1 by proving

$$\underset{1\le j\le k}{sup}{\Vert {\widehat{v}}_j-u_j\Vert }_2\stackrel{P}{\to }0.$$

Let Σ be the discretized covariance operator, with a possible constant shift a on the diagonal. Roughly speaking, v̂_j and u_j approximate the jth eigenvector of S and Σ, respectively. We hope to establish the following inequality based on the standard Davis-Kahan sin Θ theorem (Bhatia (1997), Theorem VII.3.1),

$${\Vert {\widehat{v}}_j{\widehat{v}}_j^T-u_j{u}_j^T\Vert }_F\le \frac{c}{\delta p}{\Vert S-\mathrm{\sum }\Vert }_F\le \frac{c}{\delta }{\Vert S-\mathrm{\sum }\Vert }_{\infty ,\infty },$$ (14)

provided that S and Σ have eigengap of order δp (Lemma C.2), where ||A||_F = 〈A, A〉^1/2 is the Frobenius norm and ||A||_∞,∞ is the maximum absolute value of all entries in A.

However, to rigorously obtain an approximated version of (14) is non-trivial. First, u_j is not an eigenvector of Σ because of the discretization error. The discretization error will be explicitly tracked in all subsequent analysis when comparing u_j with v̂_j, for example, in the characterization of population PCA problem (Lemma C.4). Second, v̂_j is obtained by solving a penalized eigenvector problem over the deflated Fantope, and hence is not directly comparable to its ideal theoretical counterpart u_j, which may not be in the feasible set of problem (5). To overcome this difficulty, we will consider a modified version of u_j that is feasible for (5) but still possesses similar smoothness as well as proximity to the true eigenvector of Σ. Furthermore, the sequential estimation procedure (5) involves deflation based on estimated projection matrix Π̂_j−1, which carries over estimation error from previous steps. We will use an induction argument to control the sequential error accumulation.

Due to the sequential error accumulation, the convergence rate involves ρ₁ and ρ₂ in a complex way. To provide insights for our localized estimation procedure, the following theorem shows how the rate of convergence depends on ρ₂ when the smoothing parameter ρ₁ = 0. The proof is included in the Online Supplementary File.

Theorem 4.2 (Rate of convergence)

Under assumptions (A1–A4), if ρ₁ = 0 and ρ₂ → 0 as (n, p) → ∞, then for k as defined in A2, we have

$$\underset{1\le j\le k}{sup}{\Vert {\widehat{\varphi }}_j(t)-{\varphi }_j(t)\Vert }_2=O_P(e_n+{\rho }_2+p^{-1}).$$

The three parts in the rate of convergence correspond to covariance estimation error, bias caused by localization penalty, and discretization error, respectively. According to the discussion after Assumption A1, the sup-norm covariance estimation error e_n can be made as small as $\sqrt{logp/n}$ or $\sqrt{logn/n}$ depending on the estimating method used and observation scheme. Thus if p grows at the same or higher order than $\sqrt{n}$, and ρ₂ = O(e_n), our LFPCA estimate achieves an error rate of e_n within a logarithm factor from the standard FPCA error rate.

5. NUMERICAL STUDY

To illustrate our methods for localized functional principal component analysis, we conduct simulations under two scenarios, Simulation I: localized case and Simulations II: non-localized case. For Simulation I, data {Y_il, i = 1, …, n, l = 1, …, p} are generated according to model (2), where t_l are equally spaced observational points on [0, 1]. We set μ(t) = 0, ξ_ij ~ N(0, λ_j), independent, with λ_j taken from (4², 3², 2.5², 1.25², 1, 0.75², 0.5², 0.25²) and λ_j = 0 for j > 8, and the measurement errors ${\epsilon }_{il}\stackrel{\mathit{iid}}{\sim }N(0,{\sigma }^2)$. We generate the eigenfunctions as follows. Let ϕ̃₁(t) = B₃(t), ϕ̃₂(t) = B₆(t) and ϕ̃₃(t) = B₉(t), where B_b(t) is the bth cubic B-spline basis on [0, 1], with 8 equally spaced interior knots. For j > 3, ${\stackrel{\sim }{\varphi }}_j(t)=\sqrt{2}\text{cos}((j+1)\pi t)$ for odd values of j and ${\stackrel{\sim }{\varphi }}_j(t)=\sqrt{2}\text{sin}(j\pi t)$ for even values of j, 0 ≤ t ≤ 1. Then ϕ_j(t), 1 ≤ j ≤ 8, are obtained by applying Gram-Schmidt orthonormalization on the set of ϕ̃_j(t), 1 ≤ j ≤ 8. For Simulation II, we use ${\varphi }_j(t)=\sqrt{2}\text{cos}((j+1)\pi t)$ for odd values of j and ${\varphi }_j(t)=\sqrt{2}\text{sin}(j\pi t)$ for even values of j, 0 ≤ t ≤ 1, for 1 ≤ j ≤ 8. The rest is the same as simulation I.

We investigate the performance of the proposed LFPCA under varying combinations of sample size n, number of observations per curve p, and noise level σ². More importantly, we compare the estimates given by four different methods (i) (ρ₁, ρ₂) = (0, 0) corresponds to the ordinary PCA estimation directly obtained from the sample covariance; (ii) (ρ₁, ρ₂) = (ρ̂₁, 0) corresponds to the smoothed eigenfunction estimation without localization, where ρ̂₁ was chosen by 5-fold cross-validation as discussed in Section 3.2. Empirically, we found these estimated eigenfunctions almost identical to those estimated from a smoothed covariance function or pre-smoothed individual curves; (iii) NonSeq corresponds to the subspace method developed in Vu et al. (2013). For comparison purpose, we incorporate the roughening matrix, i.e., use S − ρ₁D as input matrix in (7) with ρ̂₁ chosen by 5-fold cross-validation, the same as used in (ii) and (iv). The sparse tuning parameter is chosen by 5-fold cross validation ${\stackrel{\sim }{\rho }}_2=arg{max}_{\rho \in {\mathcal{A}}_2}{\sum }_{v=1}^V\langle H^{(-v)}({\rho }_1,\rho ),S^v\rangle$. We also note that their proposed method only outputs k basis vectors of the k-dimensional subspace, and one needs to rotate that basis to obtain eigenvectors. (iv) (ρ₁, ρ₂) = (ρ̂₁, ρ̂₂) corresponds to the proposed LFPCA with tuning parameters selected by 5-fold cross-validation as detailed in (9) and (10). Each setting is repeated 200 times to assess the average performance. The number of included components k is chosen to account for at least 85% of the total variance, i.e. ${\sum }_{j=1}^k\mathit{FVE}({\widehat{v}}_j)\ge 85\%$, where FVE is defined in (11). The selected number of k is quite robust among all simulation settings, and the average number over 200 simulations is 3.01.

The estimated eigenfunctions ϕ_j(t), j = 1, 2, 3, from a typical run of Simulation I with p = 100, n = 100, and σ = 1 are visualized in Figure 1. The four columns correspond to results given by the four methods described above. One can clearly see the improvement by adding smoothing and localization penalties. The 5-fold cross-validation choice (ρ̂₁, ρ̂₂) leads to almost perfect recovery of the true eigenfunctions. The fitted curves ${\widehat{X}}_i(t)=\widehat{\mu }(t)+{\sum }_{j=1}^k{\widehat{\xi }}_{ij}{\widehat{\varphi }}_j(t)$ for nine randomly chosen subjects are shown in Figure 2, where ϕ̂_j(t) are obtained using (ρ̂₁, ρ̂₂). It demonstrates accurate recovery of the true curves X_i(t).

Figure 1.

True (blue-solid) and estimated (red-dashed) eigenfunctions ϕ_j(t), j = 1, 2, 3, in one run of Simulation I, with n = 100, p = 100 and σ = 1, by four different methods as described in Section 5. Tuning parameters are chosen by 5-fold cross-validation.

Figure 2.

Noisy observations and recovered functions X̂_i(t) (red-solid) for nine randomly selected subjects, as obtained in one run of simulation I with n = 100, p = 100 and σ = 1, using (ρ̂₁, ρ̂₂) chosen by 5-fold cross-validation.

To better quantify the performance of estimating ϕ_j(t) we report the ℓ₂ distance ||ϕ_j(t) − ϕ̂_j(t)||₂. The medians of the errors over 200 simulation runs are reported in Table 1. The results are quite similar for different levels of p and σ, so only results for p = 100 and σ = 1 are reported with various sample size n. The errors are found to decline with increasing sample size n, as expected. For Simulation I: localized case, the proposed LFPCA with ρ̂₂ chosen by cross-validation significantly outperforms other methods. The ρ̂₂ chosen by cross-validation well approximates the true localization level of the eigenfunctions; see Figure 3 for an illustration and detailed discussions can be found in section 3.2. The NonSeq, a subspace method proposed by Vu et al. (2013), does not perform well since the union of the subdomains under consideration is the entire domain, not ‘sparse’ at all in their setting. The results demonstrate the advantage of our proposed sequential method. For Simulation II: non-localized case, 5-fold cross-validation method combined with the proposed LFPCA choose ρ̂_2j to be very close to 0, and as expected, the ℓ₂ errors of the four methods are almost identical.

Table 1.

Results for simulation: reporting the median of errors (||ϕ_j − ϕ̂_j||₂) for ϕ_j, j = 1, 2, 3, (with median absolute deviations in parentheses) over 200 simulation runs, with σ = 1, p = 100, and varying sample sizes n, where NonSeq is the subspace method developed in Vu et al. (2013).

		Simulation I: Localized			Simulation II: Non-localized
		n = 50	n = 100	n = 200	n = 50	n = 100	n = 200
ϕ1	(0, 0)	0.22 (0.09)	0.18 (0.06)	0.13 (0.04)	0.22 (0.08)	0.15 (0.05)	0.12 (0.04)
	(ρ̂1, 0)	0.22 (0.09)	0.18 (0.07)	0.13 (0.04)	0.21 (0.08)	0.15 (0.06)	0.11 (0.04)
	NonSeq	0.22 (0.09)	0.18 (0.07)	0.13 (0.04)	0.21 (0.08)	0.15 (0.06)	0.11 (0.03)
	(ρ̂1, ρ̂2)	0.12 (0.05)	0.10 (0.03)	0.06 (0.02)	0.22 (0.08)	0.15 (0.05)	0.13 (0.03)
ϕ2	(0, 0)	0.42 (0.16)	0.30 (0.13)	0.20 (0.07)	0.42 (0.16)	0.29 (0.12)	0.19 (0.07)
	(ρ̂1, 0)	0.41 (0.16)	0.30 (0.13)	0.20 (0.07)	0.41 (0.15)	0.28 (0.11)	0.19 (0.06)
	NonSeq	0.40 (0.16)	0.30 (0.12)	0.20 (0.07)	0.41 (0.15)	0.28 (0.11)	0.19 (0.07)
	(ρ̂1, ρ̂2)	0.26 (0.17)	0.14 (0.07)	0.11 (0.05)	0.41 (0.17)	0.28 (0.11)	0.20 (0.07)
ϕ3	(0, 0)	0.37 (0.12)	0.27 (0.11)	0.18 (0.07)	0.31 (0.10)	0.26 (0.09)	0.18 (0.06)
	(ρ̂1, 0)	0.36 (0.12)	0.27 (0.11)	0.18 (0.08)	0.30 (0.10)	0.26 (0.09)	0.17 (0.06)
	NonSeq	0.33 (0.14)	0.27 (0.11)	0.18 (0.07)	0.30 (0.10)	0.26 (0.09)	0.18 (0.06)
	(ρ̂1, ρ̂2)	0.24 (0.15)	0.14 (0.08)	0.09 (0.04)	0.31 (0.11)	0.26 (0.09)	0.18 (0.06)

Figure 3.

Performance of the 5-fold cross-validation to choose ρ₂ for ϕ₁. The top panel is for Simulation I: localized case: The peak of the cross-validated inner product 〈H, S〉 (dashed curve, left y label) corresponds to ρ̂₂ = 3.8. The estimated ϕ₁ is more localized as ρ₂ increases, and an ideal ρ₂ is where the ||·||_1,1 of estimated H (solid curve, right y label) meets that of true discretized projection matrix corresponding to ϕ₁ (indicated by the horizontal line). The bottom panel is for Simulation II: non-localized case: ρ̂₂ = 1.2.

6. APPLICATION TO COUNTRY MORTALITY DATA

The analysis of human mortality is important in assessing the future demographic prospects of societies, and quantifying differences between countries with regard to the overall public health measure. Functional data analysis approaches have been previously applied to study mortality data (Hyndman et al., 2007; Chiou and Müller, 2009; Chen and Müller, 2012). To study the variational modes of mortality rates across countries over years, we applied the proposed LFPCA method to period life tables for 27 countries, with rates of mortality at age 60 available for each of the calendar years from 1960 to 2006. The data were obtained from the Human Mortality Database (downloaded on March 1, 2011), maintained by University of California, Berkeley (USA), and Max Planck Institute for Demographic Research (Germany). The data is available at www.mortality.org or www.humanmortality.de, with detailed description in Wilmoth et al. (2007).

Let X_i(t) denote the mortality rate in the ith country for subjects at age 60 during calendar year t, where 1960 ≤ t ≤ 2006. We directly compute the sample covariance matrix S from the observed data and apply the proposed algorithm to solve problem (5). The ρ̂₁ chosen by 5-fold cross-validation (9) is always used to ensure a relatively smooth estimate of the eigenfunction, and the solution path along different levels of localization is investigated. The 5-fold cross-validation method gave ρ̂₂ = 0, indicating the eigenfunctions are not exactly localized. The estimated eigenfunctions without localization penalty are given in the top row of Figure 4. We then choose ρ̂₂ by the second method as defined in (13) with a = 30%. The estimated localized basis functions, as visualized in the bottom row of Figure 4, reveal several interesting features. The first localized basis function ϕ₁(t), explaining 54% of the total variance, indicates that a big variation of the mortality functions X_i(t) around their mean function happens around mid 1990s. The second basis function ϕ₂(t) with a mode around 1980s accounts for 21.2% of the total variation. The third basis function ϕ₃(t) characterizes variation of mortality around 1960s. Although the first localized basis function explains less variance than the first leading eigenfunction, only retaining 70% of the capability in return of localization, the lost proportion is picked up by the second component. The second component could have explained 30.3% of the variance without localization. Therefore, we only need three localized eigenfunctions to account for more than 85% of the total variance. Plots of estimated eigenvalues using other values of a are provided in the Online Supplementary File.

Figure 4.

Top Row: Estimated eigenfunctions for the mortality data, ρ̂₁ is chosen by 5-fold cross validation; Bottom Row: Estimated orthogonal basis functions, ρ̂₂ is chosen to maintain rFVE at 70% in (12), and the number of components k = 3 is chosen to explain at least 85% of the total variance.

7. DISCUSSION

In this paper, we propose a localized functional principal component analysis through a De-flated Fantope Localization method, where sequentially obtained eigenfunctions have guaranteed orthogonality and are allowed to be supported on different localized subdomains. As mentioned in Section 2, the deflated Fantope can easily be generalized to a d-dimensional version ${\mathcal{D}}_{\mathrm{\Pi }}^d$. In some applications, one might be interested to estimate mutually orthogonal principal subspaces with dimensions d₁, …, d_k, and each principal subspace is only supported on a subdomain ⊂ .

Throughout the paper, we mainly focus on a dense and regular functional data design where p equally spaced observations are recorded on each curve. Most commonly seen functional data have this design and a sample covariance can be easily computed from the discrete and possibly noisy observations. Our proposed formula (5) takes the sample covariance S and outputs smooth and localized estimates of eigenfunctions. In fact the proposed method puts rather minimal requirement on the input matrix S and does not rely on the design of observations. Consider the discretized version of 3 by evaluating on p×p equally spaced grid points. The estimation error of the eigenfunctions directly ties to the maximum entry-wise error of the input matrix S. Here we briefly discuss several scenarios where a sample covariance is not feasible. (i) For dense but irregularly observed functional data, one can simply smooth each curve (Ramsay and Silverman, 2005) or interpolate between points to get p equally spaced observations, and then a p×p covariance estimate S can be computed. This is what we have done for the Berkeley growth data given in the Online Supplementary File. (ii) For sparse functional data where the observations are recorded at random and sparse time points, individual smoothing or interpolation is impossible. But a uniformly consistent covariance estimation is possible by, for example, two-dimensional smoothing methods (Yao et al., 2005; Li et al., 2010). Our proposed LFPCA can then be applied by taking S as the discretized version of the smooth covariance estimator. (iii) For ultra dense and noisy data, the independent measurement errors accumulate if one uses sample covariance computed from the raw measurements. Moreover, using a large p × p matrix is not computationally efficient. We recommend performing pre-smoothing or pre-binning on individual curves and choosing a grid with a moderate size p.

We proposed two methods of choosing the localization parameter ρ₂. When the cross-validation method chooses ρ₂ = 0, it roughly means that the true eigenfunctions are not localized. Then for a fixed number of a, we find a set of orthogonal basis functions that retain the ability to explain a fair amount of variance (at least a (1 − a) proportion) and are localized. In this case, the outcome would depend on the choice of a and they should not be interpreted as the true eigenfunctions. Rather, these localized basis functions and the corresponding projection scores have ready interpretation with domain knowledge.