Localized Sparse Principal Component Analysis of Multivariate Time Series in the Frequency Domain

Abstract

Principal component analysis has been a main tool in multivariate analysis for estimating a low dimensional linear subspace that explains most of the variability in the data. However, in high-dimensional regimes, naive estimates of the principal loadings are not consistent and difficult to interpret. In the context of time series, principal component analysis of spectral density matrices can provide valuable, parsimonious information about the behavior of the underlying process, particularly if the principal components are interpretable in that they are sparse in coordinates and localized in frequency bands. In this paper, we introduce a formulation and consistent estimation procedure for interpretable principal component analysis for high-dimensional time series in the frequency domain. An efficient frequency-sequential algorithm is developed to compute sparse-localized estimates of the low-dimensional principal subspaces of the signal process. The method is motivated by and used to understand neurological mechanisms from high-density resting-state EEG in a study of first episode psychosis.

1. Introduction

Since its first descriptions by Pearson (1901), principal component analysis (PCA) has been one of the main multivariate analysis techniques for dimension reduction and feature extraction. PCA has become an essential tool for not just independent and identically distributed (iid) multivariate data, but also for serially correlated multivariate time series data in both the time and frequency domains. In the frequency domain, PCA as a sequential method for finding directions of maximum variability appeared in the work of Brillinger (1964). The principal component series are formulated through an optimal linear filtering that transmits a $p$-dimensional signal through a $d$-dimensional channel and recovers it with minimum loss of information. A foundational discussion of theory and applications of PCA in frequency domain can be found in Brillinger (2001); recent applications of this framework include uncovering non-coherent block structures (Sundararajan, 2021), time-frequency analysis (Ombao et al., 2005) and change point detection (Jiao et al., 2021).

PCA for the frequency domain analysis of high-dimensional multivariate time series faces several challenges. The first challenge, which is not unique to frequency domain PCA and is a challenge for PCA in general, is high-dimensionality. When the dimension is fixed, sample eigenvectors, and consequently sample estimates of the principal components, are consistent and asymptotically normally distributed. However, in high-dimensional regimes, where the dimension of the random variable grows, sample PCs fail to be consistent. For the single spiked covariance model in the iid multivariate setting, it has been shown that the leading eigenvector of the sample covariance matrix can actually be orthogonal to the leading eigenvector of the population covariance matrix if its corresponding eigenvalue is not sufficiently large (Paul, 2007).

To obtain consistent estimates of PCs, Johnstone and Lu (2009) proposed to obtain PCs that have sparse representation in an orthonormal system. Reviews of sparse PCA can be found in Johnstone and Paul (2018) and in Zou and Xue (2018). Aside from the theoretical benefits or necessities for PCA in high-dimensions, sparsification also provides interpretation that is essential for effective data analysis. For example, consider our motivating application of resting state EEG data that is discussed in Section 7. Figure 1 displays subsets of EEG signals from 9 channels, or locations in the brain, that are part of 64 channel recording, from two individuals, one who is experiencing a first psychotic episode (FEP) and one who is a healthy control (HC), for one minute while resting with their eyes open. We desire a frequency domain analysis of each of these data that can provide insights into underlying dependence structure of the signals at each frequency across and within each location in the brain. This inherently requires low-dimensional representations that are interpretable as combinations of power at important frequencies, which requires localization in frequency, and within certain channels or regions of the brain, which is equivalent to sparsity within coordiantes.

Figure 1:

Resting state EEG data from two individuals, a healthy control (HC, left) and a person experiencing a first psychotic episode (FEP, right), from a subset of 9 channels from a 64-channel montage.

Various formulations of the PC problem, alternative methods of imposing sparsity, and convex relaxation of the corresponding optimization problems have inspired a notable volume of research. For iid multivarite data, these methods include diagonal thresholding for spiked covariance models (Johnstone and Lu, 2009), regularized regression (Zou et al., 2006), low rank matrix approximation with sparsity constraint (Shen and Huang, 2008; Witten et al., 2009), maximizing variability over sparse subspaces (d’Aspremont et al., 2004; Vu et al., 2013), and subspace estimation via orthogonal iteration with sparsification (Ma, 2013; Yuan and Zhang, 2013). Literature on the analysis of sparse PCA for serially dependent data is dearth compared to that for iid multivarite observations. In the time domain, sparse principal subspace estimation under vector autoregressive models were studied by Wang et al. (2013). Wang et al. (2014) developed a two step method of Fantope projection and selection (FPS) (Vu et al., 2013) followed by an orthogonal iteration method with sparsification (SOAP) and studied its theoretical properties for non-Gaussian and dependent data. Sparse PCA is even less explored in the frequency domain. To the best of our knowledge, the only previous method for sparse PCA in the frequency domain is a two-step FPS followed by SOAP approach introduced by (Lu et al., 2016).

Being equivalent to finding a sequence of principal subspaces of spectral matrices across frequency, high-dimensional spectral domain PCA has an additional layer of complexity compared to the iid multivaraite case that only considers the principal subspace of a single matrix. This additional complexity presents two types of challenges. First, the ability to consistently estimate a $d$-dimensional principal subspace relies on the first $d$ eigenvalues being sufficiently larger than the remaining, which is often referred to as having a sufficient eigengap. This presents theoretical challenges in a PCA for the entire spectrum across all frequencies since, although there can be a sufficient eigengap at certain frequencies of interest, one is unable to reliably estimate principal subspaces of spectral matrices at frequencies with low power where the eigengap is small. A second challenge emerges with regards to interpretation. The challenge of interpretation is typically addressed by applied researchers by summarizing frequency-domain information by collapsing power within a finite number of pre-defined frequency bands. Although historically derived, pre-defined frequency bands have been shown to be associated with a variety of scientific mechanisms, they are not optimal for parsimoniously summarizing and describing information in any given signal. There has been considerable recent research into methods to address this challenge and to learn frequency bands for a given time series that are optimal in some sense (Bruce et al., 2020; Tuft et al., 2023). However, to the best of our knowledge, there exist no approach to provide a frequency-domain PCA of a stationary time series that is interpretable in that principal components are localized in frequency band.

The concept of localization to improve interpretation has been developed within the context of functional data analysis, where it is often referred to as “interpretable functional data analysis” (James et al., 2009; Zhang et al., 2021). Although functional data analytic methods have been developed for the frequency-domain analysis of replicated time series where several independent time series realizations are observed (Krafty et al., 2011; Krafty, 2015), it should be noted that a different setting and question is considered in this article. This article considers the analysis of a single realization of a multivariate time series, so that we consider PCA in the sense of Brillinger (1964) that involves principal subspaces of spectral matrices as operators on finite complex vector spaces, and not PCA in the functional sense that considers principal subspaces of functional operators over a continuous domain of frequency.

The broad contribution of this paper is the introduction of the first method for conducting interpretable frequency-domain PCA that is both sparse among variables as well as localized in bands of frequency. We formulate a definition for the PCA of multivariate time series that contains a low-rank signal of interest with sparse, continuous, localized principal subspaces. We propose a sequential algorithm for estimating the principal subspaces. The approach estimates the sparse $d$-dimensional principal subspace itself, thus avoiding some of the challenges associated with deflation that is required in approaches to sparse PCA that sequentially estimate one-dimensional subspaces. Through simulation, we have studied the performance of the proposed algorithms on estimation of the underlying principal subspaces as well as parameter selection. On the theoretical side, we have established the consistency of the estimated principal subspaces in high-dimensions. The proof builds upon and extends the arguments presented in Wang et al. (2014) and Lu et al. (2016) to account for smoothness of the principle subspace, including a new concentration inequality that shares information across frequency.

The remainder of this paper is organized as follows. In Section 2, a formulation of the localized and sparse principal subspaces of the underlying process is described. Section 3 is devoted to the development of the estimation procedure. The theoretical analysis of the estimator is covered in Section 4 and the selection of tuning parameters is discussed in Section 5. Empirical properties are investigated through simulation studies in Section 6 and through the analysis of the motivating EEG data in Section 7. Section 8 provides a discussion of limitations and future directions. Proofs of theoretical results and additional details with regards to the algorithms, simulation results and data analysis are provided in Supplementary Material.

Notation:

Let $\mathbf{A}=\left[A_{i,j}\right]\in {\mathbb{C}}^{p\times p}$. We denote conjugate transpose of $\mathbf{A}$ by ${\mathbf{A}}^{†}$ and will use it to represent transpose of a real valued matrix as well. Let $u\in {\mathbb{C}}^p$, the ${\ell }_2$-norm of $u$ is defined as $\Vert u{\Vert }_2=\sqrt{u^{†}u}$ and the ${\ell }_0$-norm of $u$ is the number of non-zero elements of $u$. For matrices ${\mathbf{A}}_1$ and ${\mathbf{A}}_2$, we define the inner product as $⟨{\mathbf{A}}_1,{\mathbf{A}}_2⟩=\text{tr}\left({\mathbf{A}}_1^{†}{\mathbf{A}}_2\right)$ and $\Vert \mathbf{A}{\Vert }_F=\sqrt{⟨\mathbf{A},\mathbf{A}⟩}$, where $\text{tr}\left(\mathbf{A}\right)$ is the trace of $A$. In this paper, $\Vert \mathbf{A}{\Vert }_{\mathrm{2,0}}$ determines the number of non-zero rows of $\mathbf{A}$ and $\Vert \mathbf{A}{\Vert }_{\mathrm{1,1}}≔\sum _{i,j}\left|{\mathbf{A}}_{i,j}\right|$. In addition, for Hermitian matrices $\mathbf{A}$, $\mathbf{B}\in {\mathbb{C}}^{p\times p}$, $\mathbf{A}\preccurlyeq \mathbf{B}$ if and only if $\mathbf{B}-\mathbf{A}$ is positive definite. We denote the real and imaginary parts by $\mathfrak{R}(\cdot )$ and $\mathfrak{I}(\cdot )$, respectively, and we define the unit ball in ${\mathbb{C}}^p$ by ${\mathbb{S}}^{p-1}\left(\mathbb{C}\right)=\left\{v\in {\mathbb{C}}^p\mid \Vert v{\Vert }_2=1\right\}$.

2. Localized Sparse Principal Components

2.1. Principal Components in Frequency Domain

Let $\{X(t):t\in \mathbb{Z}\}$ be a $p$-dimensional stationary time series with mean vector $\mathbb{E}\left[X\right]=\mu$, auto-covariance matrix $𝚪(h)=\mathbb{E}\left\{[X(t+h)-\mu ][X(t)-\mu {]}^{†}\right\},h=0,\pm 1,\dots$, and spectral density matrix $\mathbf{f}(\omega )=(2\pi {)}^{-1}\sum _{h=-\infty }^{\infty }𝚪(h)\text{exp}\{-i\omega h\},-\infty <\omega <\infty$, that is continuous as a function of frequency. Consider the decomposition $X(t)=\vartheta (t)+\epsilon (t)$, $t\in \mathbb{Z}$, where $\vartheta$ is the time series that is the closest time series to $X$ in terms of mean square error that can be obtained after compressing then reconstructing $X$ through a $d$-dimensional linear filter. Formally, $\vartheta$ is defined by the $d\times p$ filter $\{\mathbf{b}(h)\}$ and the $p\times d$ filter $\{\mathbf{c}(h)\}$ such that $\vartheta (t)={\mu }_{\vartheta }+\sum _h\mathbf{c}(t-h)\zeta (h)$ and $\zeta (t)=\sum _h\mathbf{b}(t-h)X(h)$ minimizes

$$\mathbb{E}\left\{[X(t)-\vartheta (t){]}^{†}[X(t)-\vartheta (t)]\right\}$$ (1)

over all possible $d\times p$ and $p\times d$ filters.

Let $\mathbf{B}(\omega )=\sum _h\mathbf{b}(h)\text{exp}\left\{i\omega h\right\}$ and $\mathbf{C}(\omega )=\sum _h\mathbf{c}(h)\text{exp}\left\{i\omega h\right\}$ be the corresponding transfer functions. The next theorem, presented in Brillinger (2001), identifies the optimal transfer functions that minimizes Equation (1).

Theorem 1. Let $\{X(t)\}$ be a $p$-dimensional weakly stationary time series with mean vector $\mu$, an absolutely summable autocovariance function $𝚪(h)$, and spectral density matrix $\mathbf{f}(\omega ),-\infty <\omega <\infty$. Then the ${\mu }_{\vartheta },\{\mathbf{b}(h)\}$, and $\{\mathbf{c}(h)\}$ that minimizes (1) are given by ${\mu }_{\vartheta }=\mu -\left[\sum _h\mathbf{c}(h)\right]\left[\sum _h\mathbf{b}(h)\right]\mu$, $\mathbf{b}(h)=(2\pi {)}^{-1}{\int }_0^{2\pi }\mathbf{B}(\alpha )\text{exp}\{ih\alpha \}$ $d\alpha$, and $\mathbf{c}(h)=(2\pi {)}^{-1}{\int }_0^{2\pi }\mathbf{C}(\alpha )\text{exp}\{ih\alpha \}$ $d\alpha$, where $\mathbf{C}(\omega )=\left[U_1(\omega )\dots U_d(\omega )\right]$, $\mathbf{B}(\omega )={\mathbf{C}}^{†}(\omega )$, and $U_j(\omega )$ is the $j$-th eigenvector of $\mathbf{f}(\omega ),j=1,\dots ,p$. In addition, if ${\lambda }_j(\omega )$ denotes the corresponding eigenvalue, $j=1,\dots ,p$, then the minimum obtained is ${\int }_0^{2\pi }\sum _{j>d}{\lambda }_j(\alpha )$ $d\alpha$.

Note that if we denote $\mathbf{A}(\omega )=\mathbf{C}(\omega )\mathbf{B}(\omega )$, then

$$\begin{gathered} \mathbb{E}\left\{[X(t)-\vartheta (t){]}^{†}[X(t)-\vartheta (t)]\right\}={\int }_0^{2\pi }\text{tr}\left\{[I-\mathbf{A}(\omega )]\mathbf{f}(\omega )[I-\mathbf{A}(\omega ){]}^{†}\right\}d(\omega ) \\ \mathbf{A}(\omega )=U_1(\omega )U_1(\omega {)}^{†}+\cdots +U_d(\omega )U_d(\omega {)}^{†}. \end{gathered}$$

In other words, for each $\omega \in \left[\mathrm{0,1}\right)$, $\mathbf{A}\left(\omega \right)$ is a rank $d$ projection matrix. This indicates that the minimizer of ${\int }_0^{2\pi }\text{tr}\left\{[I-\mathbf{A}(\omega \left)\right]\mathbf{f}\left(\omega \right)[I-\mathbf{A}(\omega ){]}^{†}\right\}d\left(\omega \right)$ over the space of rank-$d$ projection matrices ${𝒢}_d$ is equivalent to the solution to the maximization problem

$$\underset{A\in {𝒢}_d}{\text{max}}{\int }_0^{2\pi }\text{tr}\left[\mathbf{f}\left(\omega \right)\mathbf{A}\left(\omega \right)\right]d\left(\omega \right).$$ (2)

The focus of this article is the interpretation and estimation of the principal subspace spanned by the orthogonal directions $U_j\left(\omega \right)$, considered with reference to the eigenvalues ${\lambda }_j\left(\omega \right)$. It should be noted that the power spectrum of $\vartheta$ can be represented as ${\mathbf{f}}_{\vartheta }\left(\omega \right)=\sum _{j=1}^d{\lambda }_j\left(\omega \right)U_j\left(\omega \right)U_j(\omega {)}^{†}$. The principal time series ${\zeta }_j\left(t\right)$, $j=1,\dots ,d$, are uncorrelated time series with power spectra ${\lambda }_j\left(\omega \right)$ that represent parsimonious underlying latent mechanisms that account for most of the information in $X\left(t\right)$. The orthogonal directions $U_j\left(\omega \right)$ describe how these latent time series relate to and can be interpreted from the perspective of the $p$-dimensional space. For example, in our analysis of the EEG data that is presented in Section 7, $U_j$ and ${\lambda }_j$ are dominated by information within a subset of the slow delta frequencies less than 4 Hz, which has been shown to be most prominent during times of and can be used to electrophysiologically quantify rest, and within a subset of the theta frequencies between 4 – 7 Hz, which has been shown to be associated with attention control. The principal time series ${\zeta }_j$ represent uncorrelated relative expression of these two mechanisms, and the principal directions $U_j$ indicate how these mechanisms are expressed in each location of the brain.

2.2. Sparsity

This article is concerned with PCA where principal subspaces are sparse in variates. This assumption is essential both to make estimation tractable in the high-dimensional setting, as well as for interpretation. For example, in our motivating application, we desire a parsimonious interpretation where a component represents power in only certain regions of the brain by estimating principal subspaces that are sparse in the following sense.

Definition 1 (Subspace Sparsity). Let $𝒰$ be a $d$-dimensional subspace of ${\mathbb{C}}^p$ and $\mathbb{U}$ be the set of $p\times d$ orthonormal matrices whose columns span $𝒰$. Let $𝚷={\mathbf{U}\mathbf{U}}^{†}$, $\mathbf{U}\in \mathbb{U}$ be the unique (orthogonal) projection matrix onto the subspace $𝒰$. We define the sparsity level of $𝒰$ as $s=\left|\text{supp}\right[\text{diag}\left(𝚷\right)\left]\right|$.

We desire a principal component analysis under the assumption that the principal subspace that is spanned by $U_1\left(\omega \right),\dots ,U_d\left(\omega \right)$ is sparse with sparsity level $s^{*}$.

2.3. Frequency Localization

In addition to sparsity, we also desire a PCA that is localized in the frequency domain in that only the most relevant frequencies are retained. Frequency localization is important for two reasons. In terms of estimation, the ability to consistently estimate the $d$-dimensional principal subspace of a matrix depends on the difference between the $d$th and $d+1$ st eigenvalues. In many applications, including our motivating EEG example where there is low signal at higher frequencies, this eigengap is not sufficiently large for spectral matrices at many frequencies. In terms of interpretation and applications, we desire a parsimonious decomposition of information in that principal components can be interpretable as having support withing certain ranges or bands of frequency. This desire also mitigates the issues caused by the inability to consistently estimate principal subspaces at frequencies with insufficient eigengaps as said information is not of practical interest. Formally, we desire a frequency localization procedure that identifies frequencies with sufficient power by finding $\Omega \subset \left[\mathrm{0,2}\pi \right)$ such that the power of $X$ at frequency $\omega$ that is accounted for by $\vartheta$, or $\sum _{j=1}^d{\lambda }_j\left(\omega \right)$, is greater than some threshold for all $\omega \in \Omega$. Although this threshold can be selected either subjectively or based on existing scientific knowledge, in Section 5 we discuss a data driven procedure for selecting this threshold relative to the variance of the remainder ${ϵ}_t$ and present details in Appendix A.

2.4. Frequency Bands and Continuity of Principal Subspaces

Continuity of the power spectrum is a common theoretical assumption that is justifiable in most practical applications, including for EEG. We utilize the assumption that principal subspaces are continuous as a function of frequency in two ways. First, it allows for the sharing of information in adjacent frequencies, which improves estimation, especially in areas with small eigen-gap. Second, we utilize the continuity of the principal subspaces to improve interpretation. When combined with frequency localization, regularizing the continuity of the principal subspace results in estimates of the principal subspaces that are not localized in individual frequencies, but in bands of frequencies.

2.5. Optimization Problem

We combine Equation (2) to formulate the PCA problem with appropriate regularizations to obtain estimates of the principal subspaces that are localized, sparse, and preserve smoothness presented in the underlying principal subspaces, as a function of frequency. In order to obtain a sparse solution, we can add the ${\ell }_0$ penalty term for each frequency component, i.e. via ${‖\mathbf{V}\left({\omega }_{\ell }\right)‖}_{\mathrm{2,0}}\le s^{*}$. To obtain a localized solution, we propose to discretize the objective function and add a shrinkage parameter, ${\beta }_{\ell }$, for each frequency ${\omega }_{\ell }$. To control variation in estimated principal subspaces at consecutive fundamental frequencies, an intuitive approach can be to add the constraint ${‖{𝚷}_{\ell }-{𝚷}_{\ell +1}‖}_F<\epsilon$ for all $\ell =1,\dots ,n/2-1$. We relax this constraint to obtain a computationally feasible, sequential optimization problem.

Given a realization $X\left(1\right),\dots ,X\left(n\right)$ of the time series $\left\{X\right(t):t\in \mathbb{Z}\}$, we consider the frequency transformation of the data given by

$${\mathbf{f}}_n\left({\omega }_{\ell }\right)=\sum _{t=-M}^M{\stackrel{\mathbf{ˆ}}{\mathbf{R}}}_t\text{exp}\left\{-2\pi i\omega t\right\},$$ (3)

where ${\stackrel{\mathbf{ˆ}}{\mathbf{R}}}_t=\frac{1}{n}\sum _{k=1}^{n-t}X(k+t)X(t{)}^{†}$ with the dependence of $M$ on $n$ implicit so that $M$ can grow with $n$ as $n\to \infty$. Note that, when $M=n/2,{\mathbf{f}}_n$ is the standard periodogram and when $M<n/2$, ${\mathbf{f}}_n$ is a truncated periodogram. In establishing the theoretical properties of our proposed estimator, we specify an appropriate rate at which $M$ can grow relative to $n$ and $p$ such that we maintain consistency of the estimator.

With the aforementioned considerations, we can consider estimating the sparse and localized principal components through solving the following optimization problem.

$$\begin{gathered} \underset{\begin{array}{l}{\beta }_{\ell },V\left({\omega }_{\ell }\right)\\ \ell =1,\dots ,n/2\end{array}}{\text{maximize}}\left\{\sum _{\ell =1}^{n/2}{\beta }_{\ell }\text{tr}\left[{\mathbf{f}}_n\left({\omega }_{\ell }\right)\mathbf{V}\left({\omega }_{\ell }\right)\mathbf{V}{\left({\omega }_{\ell }\right)}^{†}\right]\right\} \\ \text{subject to:}\mathbf{V}\left({\omega }_{\ell }\right)\text{orthonormal},{‖\mathbf{V}\left({\omega }_{\ell }\right)‖}_{\mathrm{2,0}}\le s^{*}\text{and} \\ {‖\mathbf{V}\left({\omega }_{\ell }\right)\mathbf{V}{\left({\omega }_{\ell }\right)}^{†}-\mathbf{V}\left({\omega }_{\ell +1}\right)\mathbf{V}{\left({\omega }_{\ell +1}\right)}^{†}‖}_F<\epsilon ,\ell =1,\dots ,n/2-1\text{and} \\ \sum _{\ell =1}^{n/2}{\beta }_{\ell }\le \eta \text{and}0\le {\beta }_{\ell }\le 1,\ell =1,\dots ,n/2. \end{gathered}$$ (4)

Note that the constraint set in Equation (4) is non-convex and solving it directly is a challenging problem. We approach the problem through relaxing the constraint set, reformulating the problem as a different regularization problem and propose a sequential procedure to obtain a solution. We show in Section 4 that, for large enough $n$, the solution obtained through the proposed sequential procedure is feasible for (4), with high probability, and is a consistent estimate of the underlying principal subspaces.

Let $\Theta =\left\{\left({𝚷}_1,\dots ,{𝚷}_{n/2}\right):{‖{𝚷}_{\ell }-{𝚷}_{\ell +1}‖}_F<\epsilon \right.$ for all $\left.\ell =1,\dots ,n/2-1\right\}$. We can show that ${‖{𝚷}_{\ell }-{𝚷}_{\ell +1}‖}_F=\sqrt{2}{‖{𝚷}_{\ell +1}-{𝚷}_{\ell }{𝚷}_{\ell +1}{𝚷}_{\ell }‖}_F$, see Appendix B.2.1 for details. Thus, for any $\left({𝚷}_1,\dots ,{𝚷}_{n/2}\right)\in \Theta$, $\sum _{\ell }\text{tr}\left\{{\beta }_{\ell }{\mathbf{f}}_n\left({\omega }_{\ell +1}\right)\left[{𝚷}_{\ell +1}-{𝚷}_{\ell }{𝚷}_{\ell +1}{𝚷}_{\ell }\right]\right\}\le E$ for an appropriate $E$ (e.g, $E=2n\sqrt{d}\underset{\ell }{\text{sup}}\left\{{\lambda }_1\left({\omega }_{\ell }\right)\right\}\epsilon$), and we define the relaxed constraint set $\tilde{\Theta }≔\left\{\left({𝚷}_1,\dots ,{𝚷}_{n/2}\right):\sum _{\ell }\text{tr}\left\{{\beta }_{\ell }{\mathbf{f}}_n\left({\omega }_{\ell +1}\right)\left[{𝚷}_{\ell +1}-{𝚷}_{\ell }{𝚷}_{\ell +1}{𝚷}_{\ell }\right]\right\}\le E\right\}$. While the constrained set $\Theta$ restricts the distance between principle subspaces at all pairwise adjacent frequencies, this relaxed set $\tilde{\Theta }$ restricts the sum of the distances across selected frequencies between the projection of the periodogram at a frequency onto its principle subspace and onto the principle subspace at the previous frequency. Relaxing the constraint to $\tilde{\Theta }$ enables us to consider (4) as

$$\begin{gathered} \underset{\begin{array}{c}{\beta }_{\ell },V\left({\omega }_{\ell }\right)\\ \ell =1,\dots ,n/2\end{array}}{\text{maximize}}\left\{\sum _{\ell =1}^{n/2}{\beta }_{\ell }\text{tr}\left[{\mathbf{f}}_n\left({\omega }_{\ell }\right)\mathbf{V}\left({\omega }_{\ell }\right)\mathbf{V}{\left({\omega }_{\ell }\right)}^{†}\right]-\theta \sum _{\ell =1}^{n/2-1}\text{tr}\left[{\beta }_{\ell }{\mathbf{f}}_n\left({\omega }_{\ell +1}\right)\left({𝚷}_{\ell +1}-{𝚷}_{\ell }{𝚷}_{\ell +1}{𝚷}_{\ell }\right)\right]\right\} \\ \text{subject to:}\mathbf{V}\left({\omega }_{\ell }\right)\text{orthonormal},{‖\mathbf{V}\left({\omega }_{\ell }\right)‖}_{\mathrm{2,0}}\le s^{*}\text{and} \\ \sum _{\ell =1}^{n/2}{\beta }_{\ell }\le \eta \text{and}0\le {\beta }_{\ell }\le 1,\ell =1,\dots ,n/2, \end{gathered}$$

where the relation between $\theta$ and $E$ depends on the data. Simple calculations show that $\text{tr}\left\{{\mathbf{f}}_n\left({\omega }_{\ell +1}\right){𝚷}_{\ell +1}-\theta {\mathbf{f}}_n\left({\omega }_{\ell +1}\right)\left[{𝚷}_{\ell +1}-{𝚷}_{\ell }{𝚷}_{\ell +1}{𝚷}_{\ell }\right]\right\}=\text{tr}\left\{\left[(1-\theta ){\mathbf{f}}_n\left({\omega }_{\ell +1}\right)+\theta {𝚷}_{\ell }{\mathbf{f}}_n\left({\omega }_{\ell +1}\right){𝚷}_{\ell }\right]{𝚷}_{\ell +1}\right\}$. This enables us to rewrite the above optimization problem as the following problem

$$\begin{gathered} \underset{\begin{array}{c}{\beta }_{\ell },V\left({\omega }_{\ell }\right)\\ \ell =1,\dots ,n/2\end{array}}{\text{maximize}}\left\{\sum _{\ell =1}^{n/2}{\beta }_{\ell }\text{tr}\left[{\mathbf{f}}^{\left(\theta \right)}\left(\omega \right)\mathbf{V}\left({\omega }_{\ell }\right)\mathbf{V}{\left({\omega }_{\ell }\right)}^{†}\right]\right\} \\ \text{subject to:}\mathbf{V}\left({\omega }_{\ell }\right)\text{orthonormal},{‖\mathbf{V}\left({\omega }_{\ell }\right)‖}_{\mathrm{2,0}}\le s^{*}\text{and} \\ \sum _{\ell =1}^{n/2}{\beta }_{\ell }\le \eta \text{and}0\le {\beta }_{\ell }\le 1,\ell =1,\dots ,n/2, \end{gathered}$$ (5)

where ${\mathbf{f}}^{\left(\theta \right)}\left({\omega }_{\ell +1}\right)=(1-\theta ){\mathbf{f}}_n\left({\omega }_{\ell +1}\right)+\theta {𝚷}_{\ell }{\mathbf{f}}_n\left({\omega }_{\ell +1}\right){𝚷}_{\ell }$ and ${\mathbf{f}}^{\left(\theta \right)}\left({\omega }_1\right)={\mathbf{f}}_n\left({\omega }_1\right)$, and we suppressed the dependence of ${\mathbf{f}}^{\left(\theta \right)}$ on $n$ to ease notation. Note that, since $\Theta \subset \tilde{\Theta }$, a solution to (5) may not be feasible for (4). However, as will be shown in Section 4, for large enough $n$ the solution obtained falls in $\Theta$ with high probability.

Observe that ${\mathbf{f}}^{\left(\theta \right)}$ regularizes the object of interest, the principal subspace, by shrinking the principal subspace at frequency ${\omega }_{\ell +1}$ toward the principal subspace at frequency ${\omega }_{\ell }$, which implicitly encourages the estimated principal subspaces at consecutive frequencies to stay close together. The parameter $\theta$ controls the amount of information that is pooled from the previous frequency and implicitly controls the total variation in the estimated principal subspaces. In the extreme case where $\theta =0$, no sharing of information is incorporated into subspace estimation, and when $\theta =1$, the principal subspace at each frequency is projected on the space spanned by the corresponding previous frequency resulting in recovering the estimated principal subspace obtained at ${\omega }_1$.

3. Estimation Procedure

We propose to solve the optimization problem in (5) sequentially as follows. First, we estimate $\mathbf{V}\left({\omega }_1\right),\dots ,\mathbf{V}\left({\omega }_{n/2}\right)$ sequentially, where, at frequency ${\omega }_{\ell +1}$ and conditional on estimates $\stackrel{\mathbf{ˆ}}{\mathbf{V}}\left({\omega }_1\right),\dots ,\stackrel{\mathbf{ˆ}}{\mathbf{V}}\left({\omega }_{\ell }\right)$ for $\mathbf{V}\left({\omega }_1\right),\dots ,\mathbf{V}\left({\omega }_{\ell }\right)$, we obtain an estimate for $\mathbf{V}\left({\omega }_{\ell +1}\right)$ through solving

$$\underset{\mathbf{V}\left({\omega }_{\ell +1}\right)}{\text{max}}\text{tr}\left[{\stackrel{\mathbf{ˆ}}{\mathbf{f}}}^{\left(\theta \right)}\left({\omega }_{\ell +1}\right)\mathbf{V}\left({\omega }_{\ell +1}\right)\mathbf{V}{\left({\omega }_{\ell +1}\right)}^{†}\right],\text{s.t.}\mathbf{V}\left({\omega }_{\ell +1}\right)\text{orthonormal},{‖\mathbf{V}\left({\omega }_{\ell +1}\right)‖}_{\mathrm{2,0}}\le s^{*},$$ (6)

where ${\stackrel{\mathbf{ˆ}}{\mathbf{f}}}^{\left(\theta \right)}\left({\omega }_{\ell +1}\right)=(1-\theta ){\mathbf{f}}_n\left({\omega }_{\ell +1}\right)+\theta {\stackrel{\mathbf{ˆ}}{𝚷}}_{\ell }{\mathbf{f}}_n\left({\omega }_{\ell }\right){\stackrel{\mathbf{ˆ}}{𝚷}}_{\ell }$, ${\stackrel{\mathbf{ˆ}}{𝚷}}_{\ell }=\stackrel{\mathbf{ˆ}}{\mathbf{V}}\left({\omega }_{\ell }\right){\stackrel{\mathbf{ˆ}}{\mathbf{V}}}^{†}\left({\omega }_{\ell }\right)$, and ${\stackrel{\mathbf{ˆ}}{\mathbf{f}}}^{\left(\theta \right)}\left({\omega }_1\right)={\mathbf{f}}_n\left({\omega }_1\right)$. Note that substituting ${𝚷}_{\ell }$ by its estimate is justified by the consistency of the estimated principal subspaces established in Theorem 2. Next, we solve

$$\underset{\underset{{\sum }_{\ell =1}^{n/2}{\beta }_{\ell }\le \eta \text{and}0{\le \beta }_{\ell }\le 1,\ell =1,\dots ,n/2}{{\beta }_1,\dots ,{\beta }_{n/2}}}{\text{maximize}}\left\{\sum _{\ell =1}^{n/2}{\beta }_{\ell }\text{tr}\left[{\stackrel{\mathbf{ˆ}}{\mathbf{f}}}^{\left(\theta \right)}\left({\omega }_{\ell }\right)\stackrel{\mathbf{ˆ}}{\mathbf{V}}\left({\omega }_{\ell }\right)\stackrel{\mathbf{ˆ}}{\mathbf{V}}{\left({\omega }_{\ell }\right)}^{†}\right]\right\},$$ (7)

which as shown in Proposition 3.1, maintains only the $\eta$ frequency components with highest objective function. We solve the former problem sequentially by utilizing the sparse orthogonal iterated pursuit (SOAP) proposed by Wang et al. (2014), where the initial value at ${\omega }_1$ is obtained by the Fantope projection and selection (FPS) and the estimate obtained at each frequency is used as the initial value for the next frequency component afterwards.

3.1. Solution of the Sparse Principal Components

Although the solution to (6) for each $\ell =1,\dots ,n/2$ can attain the optimal statistical rate of convergence (Vu and Lei, 2013), it is NP-hard to compute (Moghaddam et al., 2005). Extensive research has been done to design a computationally feasible algorithm that enjoys optimal statistical convergence rate, a review of which can be found in Section D of Zou and Xue (2018).

The optimization problem (6) can be solved directly by the orthogonal iteration algorithm (Golub and Van Loan, 2013) combined with a sparsification step. In addition, to ensure the solution attains optimal statistical rate of convergence, the initial value should fall within an appropriate distance from the solution. The initial estimate is obtained by applying the FPS method that solves a convex relaxation of the problem (6). The two steps of (relaxed step) FPS followed by (tightened step) SOAP are explained below.

We first introduce the FPS algorithm for estimating the sparse PCs of a real matrix $𝚺$, where we maximize $\text{tr}\left[𝚺{\mathbf{V}}^{†}\right]$ subject to $\mathbf{V}$ being orthonormal and $\Vert \mathbf{V}{\Vert }_{\mathrm{2,0}}\le s^{*}$. We then extend it so that it can estimate the sparse principal subspace of a complex valued spectral density matrix. Note that we can relax the constraint set of (6) to obtain a convex relaxation of the problem. To do so, let $𝚷={\mathbf{V}\mathbf{V}}^T$ and note that since $\mathbf{V}$ is an orthonormal matrix, $𝚷$ is the projection matrix onto a $d$-dimensional subspace of ${\mathbb{R}}^p$ (in the real case). In addition, we know that $𝚷$ has exactly two eigenvalues, 1 with multiplicity $d$ and 0 with multiplicity $p-d$. Such constraint on eigenvalues of $𝚷$ can be relaxed to $\text{tr}\left(𝚷\right)=d$ and $0\preccurlyeq 𝚷\preccurlyeq {\mathbb{I}}_p$. In addition, we relax the constraint $\Vert 𝚷{\Vert }_{\mathrm{2,0}}\le s^{*}$ to $\Vert 𝚷{\Vert }_{\mathrm{1,1}}\le s^{*}$. Note that the constraint set $\left\{\Vert \mathbf{V}{\Vert }_{\mathrm{2,0}}\le s^{*}\right\}$ is not convex, while the set $𝒜=\left\{𝚷:𝚷\in {\mathbb{R}}^{p\times p},\text{tr}\left(𝚷\right)=d,0\preccurlyeq 𝚷\preccurlyeq {\mathbb{I}}_p\right\}$ is convex.

The relaxed convex optimization problem can be equivalently expresses as

$$\text{minimize}\left\{\text{tr}\left(𝚺𝚷\right)+\rho \Vert 𝚽{\Vert }_{\mathrm{1,1}}\mid 𝚷=𝚽,𝚷\in 𝒜,𝚽\in {\mathbb{R}}^{p\times p}\right\},$$ (8)

with the Lagrangian $ℒ(𝚷,𝚽,𝚯)=\text{tr}\left(𝚺𝚷\right)+\rho \Vert 𝚽{\Vert }_{\mathrm{1,1}}-\text{tr}\left[𝚯\right(𝚷-𝚯\left)\right]$, $𝚷\in 𝒜,𝚽\in {\mathbb{R}}^{p\times p},\Theta \in {\mathbb{R}}^{p\times p}$, and be solved by the alternating direction of multiplier (ADMM), which iteratively minimizes the augmented Lagrangian,

$$ℒ(𝚷,𝚽,𝚯)+\beta /2\Vert 𝚷-𝚽{\Vert }_F^2$$ (9)

with respect to $𝚷$ and $𝚽$ and updating the dual variable $𝚯$. We only need to iterate the algorithm enough so that the calculated $𝚷$ at iteration $T$, ${𝚷}^{\left(T\right)}$, falls within the basin of attraction of the SOAP algorithm. A detailed description of the ADMM step can be found in Appendix A. Then, the top $d$ leading eigenvectors of ${𝚷}^{\left(T\right)}$ are used as the initial value in the SOAP algorithm.

To apply the FPS algorithm to complex valued metrics we invoke to Lemma B.4.3 from the Appendix B that describes the isomorphism between complex matrices and real matrices. Let ${}_{}{}^{\left(R\right)}\hat{f}\left(\theta \right)$ be the associated $\left(2p\right)\times \left(2p\right)$ real matrix to ${\stackrel{\mathbf{ˆ}}{\mathbf{f}}}^{\left(\theta \right)}$. We propose to apply the FPS algorithm to ${}_{}{}^{\left(R\right)}\hat{f}\left(\theta \right)\left({\omega }_{\ell }\right)$ and estimate the $2d$-dimensional principal subspace of ${}_{}{}^{\left(R\right)}\hat{f}\left(\theta \right)\left({\omega }_{\ell }\right)$ to obtain the initial estimate of $d$ leading eigenvectors of $\mathbf{f}\left({\omega }_{\ell }\right)$. As will be shown in Theorem (2), part (I), the estimated subspace obtained in this manner will be consistent.

In SOAP, the orthogonal iteration method is followed by a truncation step to enforce row-sparsity and further followed by taking another re-normalization step to enforce orthogonality. More precisely, at the $t$-th iteration of the algorithm the following operations are performed

• Orthogonal iteration: ${\tilde{\mathbf{V}}}^{(t+1)}\leftarrow {\stackrel{\mathbf{ˆ}}{\mathbf{f}}}^{\left(\theta \right)}\left({\omega }_{\ell }\right){\mathbf{U}}^{\left(t\right)};{\mathbf{V}}^{(t+1)},{\mathbf{R}}_1^{(t+1)}\leftarrow \text{QR}\left({\tilde{\mathbf{V}}}^{(t+1)}\right)$

• Truncation/re-normalization: ${\tilde{\mathbf{U}}}^{(t+1)}\leftarrow \text{Truncate}\left({\mathbf{V}}^{(t+1)},\hat{s}\right);{\mathbf{U}}^{(t+1)},{\mathbf{R}}_2^{(t+1)}\leftarrow \text{QR}(\tilde{\mathbf{U}}{)}^{(t+1)}$

where columns of ${\mathbf{U}}^{\left(t\right)}$ contain the estimated first $d$ eigenvectors of $\mathbf{f}\left({\omega }_{\ell }\right)$ and the truncation operator sets the $p-\hat{s}$ rows with the smallest modulus to zero.

3.2. Solution of the Linear Programming Problem

Let $\stackrel{\mathbf{ˆ}}{\mathbf{V}}\left({\omega }_{\ell }\right),\ell =1,\dots ,n/2$ be the maximizer of $\text{tr}\left\{\left[{\stackrel{\mathbf{ˆ}}{\mathbf{f}}}^{\left(\theta \right)}\left({\omega }_{\ell }\right)\right]\mathbf{V}\left({\omega }_{\ell }\right)\mathbf{V}{\left({\omega }_{\ell }\right)}^{†}\right\}$ such that $V\left({\omega }_{\ell }\right)$ is orthonormal and ${‖\mathbf{V}\left({\omega }_{\ell }\right)‖}_{\mathrm{2,0}}\le s^{*}$. Since ${\stackrel{\mathbf{ˆ}}{\mathbf{f}}}^{\left(\theta \right)}\left({\omega }_{\ell }\right),\ell =1,\dots ,n/2$ are positive definite, $h_{\ell }≔\text{tr}\left[{\stackrel{\mathbf{ˆ}}{\mathbf{f}}}^{\left(\theta \right)}\left({\omega }_{\ell }\right)\stackrel{\mathbf{ˆ}}{\mathbf{V}}\left({\omega }_{\ell }\right)\stackrel{\mathbf{ˆ}}{\mathbf{V}}{\left({\omega }_{\ell }\right)}^{†}\right]>0$. Thus, we can write (7) as

$$\underset{\underset{{\sum }_{\ell =1}^{n/2}{\beta }_{\ell }\le \eta \text{and}0{\le \beta }_{\ell }\le 1,\ell =1,\dots ,n/2}{{\beta }_1,\dots ,{\beta }_{n/2}}}{\text{maximize}}\left\{\sum _{\ell =1}^{n/2}{\beta }_{\ell }{h}_{\ell }\right\}.$$ (10)

Note that, since $h_{\ell }>0$, $\ell =1,\dots ,n/2$, the objective function is monotonically increasing in ${\beta }_{\ell }$, $\ell =1,\dots ,n/2$ and therefore attains its maximum on the boundary of the constraint set. The algorithm selects the $\eta$ largest $h_j$’s and set the coefficients of the $n/2-\eta$ smallest $h_j$’s to zero.

Proposition 3.1. Let $\eta \in \mathbb{N}$, $\eta <K$ for some $K\in \mathbb{N}$, and $h_1,\dots ,h_K\in {\mathbb{R}}^{+}$. In addition, let $h_{\left(1\right)}\ge \cdots \ge h_{\left(K\right)}$ be the sorted $h_j^{\prime }s$ in decreasing order and ${\beta }_{\left(1\right)},\dots ,{\beta }_{\left(K\right)}$ be the corresponding coefficients in (10). Then

$$\underset{\underset{{\sum }_{\ell =1}^K{\beta }_{\ell }\le \eta \text{and}0{\le \beta }_{\ell }\le 1,\ell =1,\dots ,K}{{\beta }_{1,\dots ,{\beta }_K}^K}}{\text{maximize}}\left\{\sum _{\ell =1}^K{\beta }_{\ell }{h}_{\ell }\right\}.$$ (11)

is attained at ${\beta }_{\left(1\right)}=\cdots ={\beta }_{\left(\eta \right)}=1,{\beta }_{(\eta +1)}=\cdots ={\beta }_{\left(K\right)}=0$.

3.3. LSPCA Algorithm

Below, we summarize the estimation procedure described above and will refer to it as the localized sparse principal component analysis (LSPCA) algorithm.

4. Theoretical Analysis

To investigate theoretical properties, we first introduce the notion of distance between subspaces, in addition to several key quantities that will be used in the theoretical analysis, then we present the model and assumptions under which theoretical properties were derived. Finally, we present the rate of convergence of the estimated localized sparse principal components.

• Subspace distance: Let $𝒰$ and ${𝒰}^{\prime }$ be two $d$-dimentional subspaces of ${\mathbb{C}}^p$. Denote the projection matrices onto them by $𝚷$ and ${𝚷}^{\prime }$, respectively. We define and denote the distance between $𝒰$ and ${𝒰}^{\prime }$ by $𝒟\left(𝒰,{𝒰}^{\prime }\right)={‖𝚷-{𝚷}^{\prime }‖}_F$.

• Principal subspace notations: Let ${𝒰}_{\ell }^{*}$ be the $d$-dimensional principal subspace of $\mathbf{f}\left({\omega }_{\ell }\right)$ for each fundamental frequency ${\omega }_{\ell }=\ell /n$, $\ell =2,\dots ,n/2$ and ${𝒰}^{\left(t\right)}\left({\omega }_1\right)$ be the $d$-dimensional subspace spanned by the top $d$ eigenvectors of ${\overline{𝚷}}^{\left(t\right)}$ obtained at the $t$-th iteration of the ADMM algorithm presented in the Appendix A.

• Minimum number of iterations and data points: Let $\gamma ={\text{sup}}_{\omega \in \left[\mathrm{0,1}\right]}\frac{3{\lambda }_{d+1}\left(\omega \right)+{\lambda }_d\left(\omega \right)}{{\lambda }_{d+1}\left(\omega \right)+3{\lambda }_d\left(\omega \right)}$ and $R=\text{min}\left\{\sqrt{\frac{d\gamma \left(1-{\gamma }^{1/2}\right)}{2}},\frac{\sqrt{2\gamma }}{4}\right\}$. The minimum number of iterations of the ADMM, $T_{min}$, the SOAP, ${\tilde{T}}_{min}$, and the minimum data points, $n_{min}$, are

  $$T_{\text{min}}=⌈\frac{{\zeta }_1^2}{{\left(R-{\zeta }_2\right)}^2}⌉;{\tilde{T}}_{\text{min}}=⌈\frac{4\text{log}(R/\xi )}{\text{log}(1/\gamma )}⌉;n_{\text{min}}=C\frac{\left(s^{*}\right)\text{log}\left(p\right)}{R^2}{\left(\frac{{\lambda }_1\left({\omega }_1\right)}{{\lambda }_d\left({\omega }_1\right)-{\lambda }_{d+1}\left({\omega }_1\right)}\right)}^2,$$ (12)

  where ${\zeta }_1=\frac{{\tilde{C}}^{\prime }{\lambda }_1\left({\omega }_1\right)}{{\lambda }_d\left({\omega }_1\right)-{\lambda }_{d+1}\left({\omega }_1\right)}.s^{*}\sqrt{\frac{\text{log}\left(p\right)}{n}}$, and ${\zeta }_2=\frac{{\tilde{C}}^{″}\sqrt{M\left(n\right){\lambda }_1\left({\omega }_1\right)}}{\sqrt{{\lambda }_d\left({\omega }_1\right)-{\lambda }_{d+1}\left({\omega }_1\right)}}{\left(\frac{d\cdot p^2\text{log}\left(p\right)}{n}\right)}^{1/4}\frac{1}{\sqrt{t}}$.

4.0.1. Model Assumptions

let ${ℳ}_d\left(f,d,s^{*}\right)$ be the class of $p$-dimensional stationary time series $\left\{X\right(t):t\in \mathbb{Z}\}$ satisfying the following assumptions.

Assumption 1. For all $\omega \in \left[\mathrm{0,1}\right)$, the $d$-dimensional principal subspace of $\mathbf{f}\left(\omega \right)$ is continuous as a function of $\omega$, is $s^{*}$-sparse and these principal subspaces share the same support. In addition, we assume that ${\text{inf}}_{\omega \in [0,.5]}{\lambda }_d\left(\omega \right)-{\lambda }_{d+1}\left(\omega \right)>\delta >0$ for some constant $\delta$.

Assumption 2. There exists constants $c_1$ and ${\gamma }_1\ge 1$ such that for all $h\ge 1$, the $\alpha$-mixing coefficient satisfies $\alpha \left(h\right)\le \text{exp}\left\{-c_1{h}^{{\gamma }_1}\right\}$.

Assumption 3. There exists positive constants $c_2$ and ${\gamma }_2$ such that for all $v\in {\mathbb{S}}^{p-1}\left(\mathbb{C}\right)$ and all $\lambda \ge 0$, we have $\mathbb{P}\left(\left|v^{*}X\left(t\right)\right|\ge \lambda \right)\le 2\text{exp}\left\{-c_2{\lambda }^{{\gamma }_2}\right\}$ for all $t\in \mathbb{Z}$.

Assumption 4. Define $\gamma$ via $\frac{1}{\gamma }=\frac{1}{{\gamma }_1}+\frac{2}{{\gamma }_2}$, where ${\gamma }_1$ and ${\gamma }_2$ are given in Assumptions (6) and (7). We assume that $\gamma <1$.

Assumption 1 is made to ensure consistency of the estimated principal subspaces as well as enhancing interpretability. Moreover, continuity of the principal subspace allows us to solve the optimization problem in (5) sequentially, where the solution obtained are feasible with high probability for large enough $n$. Assumption 2 ensures the process we consider has short range dependence. An example of processes satisfying this assumption is the class of 1-Lipschitz functions of linear processes with absolutely regular innovations (Merlevède et al., 2011). Assumption 3 ensures the processes we consider are not heavy tailed. We leave spectrum estimation in the frequency domain for heavy tailed and long memory processes for a later investigation. Assumption 4 is required for applying the deviation inequality derived in Merlevède et al. (2011).

Theorem 2. Let $\left\{X\right(t):t=1,\dots ,n\}$ be a realization of a weakly stationary time series that follows ${ℳ}_d\left(f,d,s^{*}\right)$ with $n>n_{\text{min}}$. Let the regularization parameter in (8) be $\varrho =C{\lambda }_1\left({\omega }_1\right)\sqrt{\text{log}\left(p\right)/n}$ for a sufficiently large constant $C$, and the penalty parameter $\beta$ in (9) be $\beta =\sqrt{2}p\cdot \varrho /\sqrt{d}$.

1. The iterative sequence of $d$-dimensional subspace ${\left\{{𝒰}^{\left(t\right)}\left({\omega }_1\right)\right\}}_{t=1}^T$ satisfies

   $$𝒟\left[{𝒰}^{\left(t\right)}\left({\omega }_1\right),{𝒰}^{*}\left({\omega }_1\right)\right]\le \frac{\widetilde{\stackrel{~}{C^{\prime }}}{\lambda }_1\left({\omega }_1\right)}{{\lambda }_d\left({\omega }_1\right)-{\lambda }_{d+1}\left({\omega }_1\right)}\cdot s^{*}\sqrt{\frac{\text{log}\left(p\right)}{n}}+\frac{\widetilde{\stackrel{~}{C^{″}}}\sqrt{M{\lambda }_1\left({\omega }_1\right)}}{\sqrt{{\lambda }_d\left({\omega }_1\right)-{\lambda }_{d+1}\left({\omega }_1\right)}}{\left[\frac{d\cdot p^2\text{log}\left(p\right)}{n}\right]}^{1/4}\frac{1}{\sqrt{t}}$$ (13)

   with high probability, where $\widetilde{C^{\prime }}$ and $\widetilde{\stackrel{~}{C^{″}}}$ are constants.
2. Let ${𝒰}_{\ell }^{(T+\tilde{T})}$ be the space spanned by the columns of the estimator obtained from the Algorithm LSPCA after $T\ge T_{\mathit{\text{min}}}$ iterations in Algorithm ADMM followed by $\tilde{T}\ge {\tilde{T}}_{\text{min}}$ iterations of Algorithm SOAP. By taking the sparsity parameter $\hat{s}$ in Algorithm SOAP such that $\hat{s}=C\text{max}\left\{\left[\frac{4d}{{\left({\gamma }^{-1/2}-1\right)}^2}\right],1\right\}{s}^{*}$, for some integer constant $C\ge 1$, and a fixed $\theta \in \left[\mathrm{0,1}\right)$, the final estimator ${\widehat{𝒰}}_{\ell }={𝒰}_{\ell }^{(T+\tilde{T})}$ satisfies

   $$𝒟\left({𝒰}_{\ell }^{*},{\widehat{𝒰}}_{\ell }\right)\le C^{‴}\frac{{\gamma }^{1/2}}{1-{\gamma }^{1/4}}\Delta (2\hat{s})$$ (14)

   with high probability, for all $\ell =\mathrm{1,2},\dots ,n/2$, where

   $$\begin{array}{c}\Delta (s)≔\underset{{\omega }_{\ell }}{\text{sup}}\frac{\sqrt{2d}\left[\left(\text{exp}\left(-c_{0}M\right)\vee M\sqrt{\frac{s^{*}\text{log}\left(p\right)}{n}}\right)+2\theta {\lambda }_1\left({\omega }_{\ell +1}\right)\left[𝒟\left({𝒰}_{\ell }^{*},{𝒰}_{\ell +1}^{*}\right)+\alpha \right]\right]}{\frac{1}{2}\left[{\lambda }_d\left({\omega }_{\ell +1}\right)-(1-\theta ){\lambda }_{d+1}\left({\omega }_{\ell +1}\right)\right]-2\theta {\lambda }_1\left({\omega }_{\ell +1}\right)\alpha },\end{array}$$ (15)

   $\alpha ={\text{sup}}_{{\omega }_{\ell }}\frac{c_1\sqrt{2d}}{{\lambda }_d\left({\omega }_{\ell }\right)-{\lambda }_{d+1}\left({\omega }_{\ell }\right)}\left\{\text{exp}\left[-c_{0}M\right]\vee M\sqrt{\frac{s^{*}\text{log}\left(p\right)}{n}}\right\}$, and $c_0$, $c_1$ are constants.

Theorem 2 can be seen as an extension of the result in Wang et al. (2014) that is also an extension of the Davis-Kahan theorem quantifying the precision of the estimated principal subspaces and its dependence on the sample size, dimension, and the magnitude of the perturbation relative to the eigengap of the covariance type matrix in an iid sampling scheme. In particular, Equations (14) and (15) reveal how convergence of the estimated principal subspaces depends on the eigengap ${\lambda }_d\left(\omega \right)-{\lambda }_{d+1}\left(\omega \right)$, sample size, and dimension. Observe that the numerator of (15) is a combination of a term proportional to $2\theta {\lambda }_1\left({\omega }_{\ell +1}\right)𝒟\left({𝒰}_{\ell }^{*},{𝒰}_{\ell +1}^{*}\right)$ and a term proportional to $\left\{\text{exp}\left[-c_{0}M\right]\vee M\sqrt{\frac{s^{*}\text{log}\left(p\right)}{n}}\right\}$. As the proof of Lemma B.2.1 in Appendix B indicates, the former term is an upper bound for ${‖{\mathbf{f}}^{\left(\theta \right)}\left(\omega \right)-\mathbf{f}\left(\omega \right)‖}_{op}$, which represents the extend of the bias introduced by the information sharing step in the proposed principal component analysis. Note that the bias vanishes as $n\to \infty$, since $𝒟\left({𝒰}_{\ell }^{*},{𝒰}_{\ell +1}^{*}\right)$ converges to zero by the continuity of the principal subspaces. The later term is an upper bound on the sparse operator norm of ${\mathbf{f}}_n\left(\omega \right)-\mathbf{f}\left(\omega \right)$. The upper bound obtained holds with high probability and vanishes when $M\to \infty$ and $M\sqrt{\text{log}\left(p\right)}/n\to 0$ as $n,p\to \infty$, with the dependence of $M$ on $n$ implicit. This indicates that, to achieve consistency in estimation of the principal subspaces, the parameter $M$, should grow with $n$ and $p$ at the rate such that $M\sqrt{\text{log}\left(p\right)}/n\to 0$. Finally, Theorem 2 guaranties that for a sufficiently large $n$, $p$ and $M$, $𝒟\left({𝚷}_{\ell },{\stackrel{\mathbf{ˆ}}{𝚷}}_{\ell }\right)<ϵ/3$ for a given $ϵ$. This, along with the smoothness of principal subspaces as a function of frequency, implies $‖{\stackrel{\mathbf{ˆ}}{𝚷}}_{\ell }-{\stackrel{\mathbf{ˆ}}{𝚷}}_{\ell +1}‖<ϵ$ with high probability. This confirms that, for sufficiently large $n,p$, and $M$ the estimates obtained by the LSPCA is feasible for (4), with high probability.

5. Tuning Parameter Selection

In this section we outline a procedure for the selection of four parameters: the dimension of the principal subspaces $d$, the sparsity parameter $\hat{s}$, the localization parameter $\eta$, and the smoothing parameter $\theta$. A detailed description is provided in Appendix A. Given the complex nature of the problem that makes the empirical joint selection of the parameters infeasible, we propose the selection of each parameter individually, conditional on the other parameters, and iteratively.

For the dimension of the principal subspaces $d$, we inspect the scree plot or equivalently the plot of the proportion of variance explained. For the localization parameter $\eta$, we propose to use information criteria based on the log-Whittle likelihood. More precisely, let $h_{\ell }$ be the maximum total power in a $d$-dimensional subspace at frequency $\ell$ defined in Section 3.2, $\left\{h_{\left(1\right)},\dots ,h_{(n/2)}\right\}$ be their order statistics, $\check{L}=\left\{{\ell }_{\left(1\right)},\dots ,{\ell }_{(n/2)}\right\}$ be the corresponding indices of the Fourier frequencies of these order statistics, and ${\check{L}}_{\eta }=\left\{{\ell }_{\left(1\right)},\dots ,{\ell }_{\left(\eta \right)}\right\}\subset \check{L}$ be the the Fourier frequency indices of the top $\eta$ order statistics. The log-Whittle likelihood is estimated by $\text{log}\left(ℒ\right)=-\sum _{\ell =1}^{n/2}\left\{p\text{log}\pi +\text{log}\left|{\stackrel{\mathbf{ˆ}}{\mathbf{G}}}_{\ell }\right|+\left[d_X\left({\omega }_{\ell }\right)\right]{\left[{\stackrel{\mathbf{ˆ}}{\mathbf{G}}}_{\ell }\right]}^{-1}{\left[d_X\left({\omega }_{\ell }\right)\right]}^{†}\right\}$, where ${\stackrel{\mathbf{ˆ}}{\mathbf{G}}}_{\ell }=\mathbb{I}\left\{\ell \in {\check{L}}_{\eta }\right\}{\stackrel{\mathbf{ˆ}}{\mathbf{f}}}_{\vartheta }\left({\omega }_{\ell }\right)+\stackrel{\mathbf{ˆ}}{𝚺},{\stackrel{\mathbf{ˆ}}{\mathbf{f}}}_{\vartheta }\left({\omega }_{\ell }\right)={\mathbf{f}}_n\left({\omega }_{\ell }\right){\hat{U}}_1\left({\omega }_{\ell }\right){\hat{U}}_1{\left({\omega }_{\ell }\right)}^{†}+\cdots +{\mathbf{f}}_n\left({\omega }_{\ell }\right){\hat{U}}_d\left({\omega }_{\ell }\right){\hat{U}}_d{\left({\omega }_{\ell }\right)}^{†}$, ${\hat{U}}_j\left({\omega }_{\ell }\right)$ are obtained from the LSPCA algorithm, $\stackrel{\mathbf{ˆ}}{𝚺}=\sum _{\ell \in \check{L}/{\check{L}}_{\eta }}{d}_X\left({\omega }_{\ell }\right)d_X{\left({\omega }_{\ell }\right)}^{†}/\left(\left|\check{L}\right|-\left|{\check{L}}_{\eta }\right|\right)$, and $d_X\left({\omega }_{\ell }\right)$ is the discrete Fourier transform of the data at ${\omega }_{\ell }$. We use this to define standard information criteria for $\eta$ including $AIC=-2\text{log}\left(ℒ\right)+2\eta$, $\mathit{\text{AICc}}=-2\text{log}\left(ℒ\right)+2\eta +\frac{2{\eta }^2+2\eta }{n-\eta -1}$, and $BIC=-2\text{log}\left(ℒ\right)+\text{log}\left(n\right)\eta$. The localization parameter $\eta$ is selected to minimize the information criteria. For both the sparsity parameter $s$ and the smoothing parameter $\theta$, we propose to use $k$-folds cross validation, with the Mahalanobis distance to evaluate the performance of fitted model in the validation step. For time series with short length, one might consider alternative methods such as information criterion for selecting $s$ and $\theta$.

6. Simulation Studies

6.1. Setting

In this section, we explore, numerically, the performance of the LSPCA algorithm, effect of smoothing, and performance of the parameter selection procedures and compare the results with the classical frequency domain PCA developed in Brillinger (2001). Let $a\left(t\right)$ be a linear filter with frequency response $\mathbf{I}\left(\Omega \right)$, where $\mathbf{I}\left(\Omega \right)$ is the indicator function of the set $\Omega =[.05,.25]$. We consider processes $X\left(t\right)={\left[X_1\left(t\right),\dots ,X_p\left(t\right)\right]}^T$, $t=1,\dots ,n$ that are constructed from independent processes $Y_1,\dots ,Y_5$ such that $\left\{Y_k\left(t\right):t\in \mathbb{Z}\right\}~AR\left(4\right)$. More precisely, let $\star$ represents the convolution operator, we define $X_1\left(t\right)=a\left(t\right)\star (1/c)Y_1\left(t\right)$, $X_j\left(t\right)=a\left(t\right)\star \left[{\pi }_j^{\left(x\right)}{X}_1\left(t\right)+(1/c)Y_j\left(t\right)\right]+W_{j1}\left(t\right)$, $j=2,\dots ,5$ with ${\pi }_2^{\left(x\right)}=2.2,{\pi }_3^{\left(x\right)}=1.2,{\pi }_4^{\left(x\right)}=1.25$, ${\pi }_5^{\left(x\right)}=2.25$ and $X_j\left(t\right)~WN\left(\mathrm{0,1}\right)$, $j=6,\dots ,p$, where $Y_j\left(t\right)=\sum _{k=1}^4{\pi }_{jk}{Y}_j(t-k)+W_{j0}\left(t\right)$, $W_{j0}\left(t\right)~N(\mathrm{0,1}/4),j=1,\dots ,5$ such that ${\pi }_{j1}={\alpha }_1+{\varphi }_{j1}$, ${\pi }_{j2}={\alpha }_2-{\alpha }_1{\varphi }_{j1}+{\varphi }_2$, ${\pi }_{j3}=-\left({\varphi }_{j1}{\alpha }_2+{\varphi }_2{\alpha }_1\right)$, ${\pi }_{j4}=-{\varphi }_{j2}{\alpha }_2$, ${\alpha }_1=1/20$, ${\alpha }_2=-1/1.15$, ${\varphi }_2=-0.75$, ${\varphi }_{11}=1.5$, ${\varphi }_{21}=1.55$, ${\varphi }_{31}=1.45$, ${\varphi }_{41}=1.65$, ${\varphi }_{51}=1.35$, and $W_{j0}\left(t\right),j=1,\dots ,5$ are independent white noise processes with mean zero and variance 1 and are independent of $W_{j1}\left(t\right),j=2,\dots ,5$.

The processes have $d=1$ dimensional principle subspaces. The parameter $c$ controls the eigengap such that increases in $c$ reduces the power in the first 5 coordinates, which weakens the signal strength and decreases the eigengap. The modulus of its leading eigenvector is displayed in the top left panel of Figure 2 for $c=3$. The accuracy of the estimated principal subspace at any frequency, say $\omega$, depends on various factors including the dimension $\left(p\right)$, sample size $\left(n\right)$, and the eigengap. We considered two values of the dimension $p=\mathrm{64,128}$, three sample sizes $n=\mathrm{1024,2048,4096}$, and two values of signal strength $c=\mathrm{1,3}$. One hundred realizations of the process for all combinations of $n$, $p$, and $c$ were generated, and the first eigenvector of the spectral density matrix at each frequency were estimated using the LSPCA algorithm and the classical method of Brillinger (2001). We evaluated the performance of the estimators using mean squared estimation error (MSEE) defined as $\sum _{k=1}^{100}𝒟\left(𝒰-{\widehat{𝒰}}^{\left(k\right)}\right)/100$, where $𝒰$ is the true 1-dimensional principal subspace and ${\widehat{𝒰}}^{\left(k\right)}$ is the estimated one obtained from the $k$-th run of the simulation, and $𝒟$ is the distance defined in Section 4.

Figure 2:

Top left panel represents the population leading eigenvector; Top right panel represents the classical estimate of the leading eigenvector; Bottom left panel represents the sparse estimate of the leading eigenvector with $\theta =0$; Bottom right panel represents the sparse estimate of the leading eigenvector with $\theta =0.6$.

6.2. Results

We present three sets of results here; additional results related to computation time, sparsity, and localization are provided in Appendix C. The first set of results are illustrative results from one realization with $p=64$, $n=1024$ and $c=3$. The top right panel of Figure 2 displays the estimated modulus of the first principal component estimated from the classical estimator of Brillinger (2001). The high variability of the classical estimator is illustrated, which inhibits the separation of signal within the five channels in the support $\Omega$ from noise. The bottom panels of Figure 2 provide a visual illustration of the role of the sharing of information across principle subspaces at adjacent frequencies via $\theta$. Although the results of Section 4 indicate consistency even with $\theta =0$, we see that without sharing information across frequency in the lower left panel where $\theta =0$ at certain frequencies, other coordinates are selected rather than the five that have true signal. The result is that supported subspaces cannot be interpreted as a continuous bands. The estimate under $\theta =0.6$ is provided in the lower right panel, where it can be seen that the sharing of information not only improves estimation in finite samples, but provides essential interpretation where frequency is localized into bands.

The second set of results investigates the stability of the iterative procedure for selecting tuning parameters and the relative estimation at frequencies within the support $\Omega$ and outside the support ${\Omega }^c=[0,.5]\setminus \Omega$ for $p=64$, $n=1024$, $c=3$. Figure 3 illustrates side by side boxplots of the mean estimation error of the LSPCA and the classical PCA over $\Omega$ and ${\Omega }^c$ after 1–4 iterations of the tuning parameter selection procedure. Plots illustrates that, after two iterations of the parameter selection and estimation procedure, the estimated principal subspaces stabilizes, suggesting performing the iterative scheme in LSPCA for two iterations. In addition, all plots confirm that the LSPCA improves estimation of the underlying principal subspaces in $\Omega$ compared to the classical approach. Moreover, over ${\Omega }^c$, where the underlying principal subspaces are not sparse and signal strength is low, both estimates obtained by the classical PCA and the LSPCA do not perform well. It can be seen that imposing sparsity through LSPCA improves estimation within regions of scientific interest $\Omega$ while increasing error in regions that are not of scientific interest ${\Omega }^c$.

Figure 3:

Side by side box plots of the MSEE of the LSPCA and the classical PCA for 4 times iteration of the parameter selection and estimation of the 1-dimensional principal subspaces over $\Omega$ and ${\Omega }^c$.

The third set of results investigates the relative effects of sample size $n$, dimension $p$ and signal strength $c$ for estimation within $\Omega$ as summarized in Figure 4. We found that LSPCA outperforms classical PCA in all settings, and confirms the results in Section 4 in that estimation error improves for higher sample sizes $n$, smaller dimensions $p$ and stronger larger eigengaps/smaller $c$.

Figure 4:

Side by side box plots of the MSEE of the LSPCA illustrating the effect of increasing sample size, dimension, and signal strength on estimation of the 1-dimensional principal subspaces over $\Omega$ and in comparison with the the classical principal subspace estimation.

7. Data Analysis

Evidence suggests that electrophysiological activity at different frequencies and locations of the brain can be biomarkers for schizophrenia (Renaldi et al., 2019; Zhang et al., 2021). To illustrate the use of LSPCA to obtain interpretable frequency-channel analyses, we apply it in separate analyses of 64-channel EEG recording from two individuals. One is from a patient who is experiencing an episode of psychosis for the first time (FEP), which can be a precursor to the eventual development of schizophrenia, and has been emitted to a psychiatric emergency department. The second is from a healthy control with no history of mental illness (HC). During the recording, participants sat in a chair and relaxed with their eyes open. Data were recorded using a f10–10 system and were initially sampled at a rate of 250 Hz for one minute. Pre-processing consisted of down-sampling to 64 Hz and filtering using a 1 Hz high-pass filter and 58 Hz low-pass filter; removal of segments with large artifacts such as muscle activity or movements by a trained EEG data manager; and further removal of subtle artifacts such as ocular movement and cardiac signals via independent component analysis (Delorme and Makeig, 2004).

We applied LSPCA to understand personal electrophysiological activity by analyzing each subject’s data separately. We made exploratory comparisons and discuss future formal group analyses in Section 8. The parameter selection procedure selected a sparsity level of $\hat{s}=8$ for both subjects, smoothing parameters of $\theta =0.2$ and $\theta =0.6$, and localization parameters of $\eta =41$ and $\eta =52$ for the FEP and HC participants, respectively. Inspection of scree plots at all frequencies suggests that $d=2$. Two aspects of the results are presented here; additional explorations, including analyses of real and complex structures and of coherence, are provided in Appendix D.

First, we investigated the modulus of PC loadings of the $d=2$ components as functions of coordinate/channel and frequency for the FEP participant and the healthy control in Figure 5. Principal subspaces for both participants are localized within the union of a band of very-low frequencies that is contained within the traditional delta band of frequencies less than 4 Hz, and with a band that is contained in the traditional theta band of frequencies between 4 – 8 Hz. As power within the delta band is characteristic of unconscious processes and elevated during rest, and power within the theta band is involved in cognitive processes such as attention control that is elevated when eyes are open, this localization is not unexpected. However, as opposed to collapsing power within the historically defined delta and theta bands, the boundaries of which are displayed in Figure 5, the data-driven LSPCA identified narrower, more parsimonious, person-specific bands.

Figure 5:

Top left panel Modulus of the first PC loadings of HC subject; Top right pane: Modulus of the second PC loadings of HC subject. Bottom left panel Modulus of the first PC loadings of FEP subject; Bottom right pane: Modulus of the second PC loadings of FEP subject. Vertical lines denote the boundaries of the traditional delta and theta bands.

Next, we explored the spatial localization of power within these identified bands. Figure 6 displays the diagonal elements of $\sum _{\omega \in B}{\stackrel{\mathbf{ˆ}}{\mathbf{f}}}_{\vartheta }\left(\omega \right)$ where B is the localized frequency band for both subjects. We see that power in both the delta and theta band are concentrated in the central regions of the HC; theta power in the FEP participant is also concentrated in the central region. However, delta power for the FEP participant is primarily located in the frontopolar and parietal regions. This difference in distribution of delta brain activity between the FEP and HC participants is consistent with the findings of Renaldi et al. (2019), who reported significantly higher delta power in the frontal and posterior regions in FEP compared to HC.

Figure 6:

Active channels in the FEP (top row) and HC (bottom row) participants for the lower frequency band (left column) and higher frequency band (right column).

8. Discussion

This article introduced what is, to the best of our knowledge, the first approach to conducting a PCA on a high-dimensional stationary time series whose principal subspace is sparse among variates, localized within frequency, and smooth as a function of frequency. The method is by no means exhaustive and can potentially be extended to more complex scenarios. The first of these is to nonstationary time series. Although the developed LSPCA routine could be applied directly using quadratic time-frequency transformations such as the local Fourier periodogram or SLeX periodogram, it is not yet obvious how to impose sparsity, frequency localization, or smoothness on the time-frequency subspaces that accounts for temporal ordering and information. A second extension is to the replicated time series setting in which a joint analysis is conducted on data where multivariate time series are observed for multiple subjects. As opposed to the analysis presented in Section 7, where separate PCAs were conducted individually for two separate subjects, a PCA for replicated time series will find optimal eigenspaces for describing mutual and subject specific information. The proposed LSPCA offers a regularized estimation approach. One might desire a confidence-based procedure that provides inference with regards to included frequency bands and retained channels. Although the excursion set method that has been used to obtain inference for spatial clusters in image data appears to provide a natural solution for conducting inference with regards to frequency localization (Maullin-Sapey et al., 2023), how to do this while imposing sparsity and extract the low-dimensional principal subspace could prove to be challenging. Lastly, PCA is just one of many popular tools for extracting low-dimensional structures in time series data, each with different goals. Another popular tool, dynamic factor modeling, finds parsimonious dynamic factors that represent comovements of the variates. It has an elegant formulation that involves the eigendecomposition of spectral matrices and has favorable statistical properties when eigenvalues diverge (Forni et al., 2000). Future work can investigate the extension of LSPCA to dynamic factor models when largest eigenvalues are bounded.