Time-varying dynamic network model for dynamic resting state functional connectivity in fMRI and MEG imaging

Abstract

Dynamic resting state functional connectivity (RSFC) characterizes fluctuations that occur over time in functional brain networks. Existing methods to extract dynamic RSFCs, such as sliding-window and clustering methods that are inherently non-adaptive, have various limitations such as high-dimensionality, an inability to reconstruct brain signals, insufficiency of data for reliable estimation, insensitivity to rapid changes in dynamics, and a lack of generalizability across multiply functional imaging modalities. To overcome these deficiencies, we develop a novel and unifying time-varying dynamic network (TVDN) framework for examining dynamic resting state functional connectivity. TVDN includes a generative model that describes the relation between a low-dimensional dynamic RSFC and the brain signals, and an inference algorithm that automatically and adaptively learns the low-dimensional manifold of dynamic RSFC and detects dynamic state transitions in data. TVDN is applicable to multiple modalities of functional neuroimaging such as fMRI and MEG/EEG. The estimated low-dimensional dynamic RSFCs manifold directly links to the frequency content of brain signals. Hence we can evaluate TVDN performance by examining whether learnt features can reconstruct observed brain signals. We conduct comprehensive simulations to evaluate TVDN under hypothetical settings. We then demonstrate the application of TVDN with real fMRI and MEG data, and compare the results with existing benchmarks. Results demonstrate that TVDN is able to correctly capture the dynamics of brain activity and more robustly detect brain state switching both in resting state fMRI and MEG data.

1. Introduction

The human brain’s functional activity can be described as highly dynamic functional networks arising from a structural network whose fluctuations over time form the basis for complex cognitive functions and consciousness (Bassett et al., 2011; Deco and Jirsa, 2012; Duan et al., 2020; Liu et al., 2022; 2022; Pasquini et al., 2020; Shine et al., 2015). This view of brain function highlights the importance of time sensitive descriptions of brain network activity for understanding the functional relevance of alterations in the network function that may underlie different behavioral states and conditions(Varela et al., 2001). Recent experiments using fMRI data have demonstrated that global brain signals transition between states of high and low connectivity strength over time (Zalesky et al., 2014) and these fluctuations are related to coordinated patterns of network topology (Betzel et al., 2016). Studies suggest that dynamic fluctuations in the network structure also relate to fluctuations in the cognitive function (Shine et al., 2015). Therefore, analyses of functional neuroimaging data to examine time-varying reconfiguration of the global network structure may provide a unique opportunity to gain insights into the dynamics of functional brain networks, their association with behavioral states, and their alterations in disease and therapeutic interventions.

To appropriately describe synchronous temporal fluctuations in neuroimaging data, many data driven approaches have been used, especially with the resting state functional connectivity (RSFC) which describes how brain activity is correlated across regions when an explicit task is not being performed. Many studies have shown that this functional connectivity provides a powerful and informative framework for exploring brain organization (Bullmore and Sporns, 2009; Greicius, 2008; Shine et al., 2015). RSFC studies have been described both for blood-oxygen level-dependent (BOLD) data measured with functional magnetic resonance imaging (fMRI) (Biswal et al., 1997; Calhoun et al., 2001; Greicius et al., 2003) and for faster time scale neural oscillatory network changes measured with magnetoencephalography (MEG) (Englot et al., 2015; Ranasinghe et al., 2017) or electroencephalography (EEG) imaging (Brookes et al., 2011; Dominguez et al., 2013; Hohlefeld et al., 2013). Approaches for RSFC analyses include seed-based correlations (Lv et al., 2018), independent component analysis (Beckmann et al., 2005) and dynamic mode decomposition (Brunton et al., 2016; Kutz et al., 2016). Recent work has also focused on recovering the static RSFC from the underlying structural connectivity via graph methods like the network diffusion model (Abdelnour et al., 2014) and algebraic spectral graph expansions (Abdelnour et al., 2018; Becker et al., 2018; Meier et al., 2016; Tewarie et al., 2020). of dynamic changes in functional network architecture. To date, most existing statistical techniques for RSFC have assumed that the functional connectivity structure is stationary over a dataset, which is in direct contrast to emerging data that suggest the strength of connectivity between regions is variable over time. Therefore, the development of statistical methods that enable exploration of dynamic changes in the functional connectivity is currently of great importance to the neuroscience community.

The extension of current techniques to capture the dynamic changes in RSFC during the scan period is a lively yet evolving topic. It is well known that the brain at rest is in fact quite dynamic, with RSFC capable of changing over a matter of seconds to minutes (Hutchison et al., 2013). This time varying pattern, namely the dynamic functional connectivity, has been shown to constitute novel imaging biomarkers for identifying neurological dysfunctions such as schizophrenia, autism and various forms of dementia (Damaraju et al., 2014; Filippi et al., 2019; Long et al., 2020; Ma et al., 2014; Mash et al., 2019; Pasquini et al., 2020; Rashid et al., 2016; 2014; Schumacher et al., 2019). For instance, dynamic FC may underlie the neuropathology of major depressive disorder (Long et al., 2020), can assess the abnormal brain states for schizophrenia (Duan et al., 2020) and it can identify early mild cognitive impairment for dementia (Wee et al., 2016) and distinguish Alzheimer’s Disease (AD) patients from healthy controls (Schumacher et al., 2019). Thus, the dynamic component of RSFC may serve as an additional biomarker of neurological disorders - a key motivation of current work.

Currently, the most common approach to extract dynamic RSFCs relies on the sliding-window method, which generally consists of two steps: (1) divide signals into segments of the equal duration; (2) implement the traditional seed based method (Biswal et al., 1995; Fox et al., 2005), independent component analysis (Allen et al., 2014; Calhoun et al., 2001; van de Ven et al., 2004), or the dynamic mode decomposition method (Brunton et al., 2016; Kutz et al., 2016) on the segments sequentially. While the sliding-window method is practically attractive since it enables the use of earlier static methods in the dynamic context, it presents several limitations and trade-offs, which will be discussed in detail in Section 4.1. One notable issue is that current methods for dynamic functional connectivity (FC) analysis do not account for biological constraints or biophysically realistic models of brain activity and state switches. This represents a lost opportunity to overcome some of the limitations noted in Section 4.1. Here we propose a novel model for extracting dynamic FC that relies on discrete and discontinuous “state changes” in brain activity. Indeed, there is mounting evidence that the brain’s dynamics results from its cycling through a number of brain-states, i.e., the transient, patterned, quasi-stable states or patterns of the brain activity (Coquelet et al., 2021; Croce et al., 2020; Michel and Koenig, 2018), separated by brain state switches, such that while the FC during brain states may be considered stationary, FC during the transitions between brain states are subject to discontinuous, abrupt or non-smooth events (Li et al., 2013; Saper et al., 2010; Vidaurre et al., 2017). In addition to being more biologically realistic, this approach allows us to benefit from several constraints, especially the concept that the spatial features of brain activity might be stationary, while the coupling between these stationary structures might be temporally dynamic. For instance, the spatial structures may arise from the underlying structural connectivity, while the temporal parameters describe the dynamic switching between brain networks over time. Therefore, while the spatial structure of the FC patterns is considered stationary due to the linkage with the structure of the brain, how these spatial features work together is allowed to vary over time. It is further possible to constrain the dynamics of the temporal parameters. Rather than randomly or continuously traversing through the latent state-space, RSFCs most likely undergo discrete and discontinuous shifts, resulting in the concept of “brain states” (Li et al., 2013; Saper et al., 2010; Vidaurre et al., 2017). Hence we recommend to impose piece-wise constancy to these temporally changing coefficients. We show that using these powerful constraints, it is possible to overcome the trade-offs and limitations currently pertinent to the dynamic RSFC analysis.

We present a unified solution for extracting dynamic FCs from both fMRI and MEG data, which directly addresses these limitations. We call this method the time-varying dynamic network (TVDN) framework. We develop a novel automatic and provably statistically optimal inference algorithm based on the TVDN model to infer the dynamics that underlie the model. We extract the stationary spatial features and detect the dynamic brain state switches adaptively. The algorithm is able to divide the brain signals into uneven segments, each of which contains brain activities in a stationary brain state. Once the parameters have been successfully inferred, the entire spatio-temporal noise-free imaging signal can be reconstructed through a high dimensional linear forward model - a feature that is rarely available in current methods. The algorithm involves a few tuning or hyper-parameters, which are automatically selected to minimize the uncertainties of the number of switches across independent samples. We expect that the presented TVDN framework will prove effective in robustly generating dynamic FC features that will serve as useful biomarkers of neurological and neurodegenerative diseases.

2. Materials and methods

The dynamic FC contains spatial and temporal components (Lang et al., 2012). The spatial features of the dynamic FC capture the links among brain regions (Alexander-Bloch et al., 2010; Brier et al., 2014; Geerligs et al., 2015; Sanz-Arigita et al., 2010; Van Den Heuvel et al., 2009). The temporal features characterize the state changes of brain activity (Di et al., 2013; Gonzalez-Castillo et al., 2015; Kitzbichler et al., 2011; Moussa et al., 2011; Shirer et al., 2012). Furthermore, the spatial features are constrained by the stable brain structures, and hence they must be consistent over the signal sampling time and across the image modalities. Moreover, different modalities have distinct temporal resolutions, and therefore the temporal features are distinct across the modalities. Considering these characteristics of the spatial and temporal features, we develop a novel methodology to extract the time invariant spatial features and time varying temporal features. Fig. 1 shows a flowchart of the estimation procedure. The purple oval represents TVDN inputs, and red ovals represent TVDN outputs. The blue rectangles represent the building blocks of TVDN, which we discuss in detail in Section 2.3.3.

Fig. 1.

TVDN pipline. The purple ovals represent TVDN inputs, and red oval represents TVDN outputs. The blue rectangles represent the building blocks of TVDN. Two multimodality kernel examples are provided and will be discussed in Section 2.3.3.

2.1. Time-varying dynamic network

Let X_i(t) be the brain signal at time t on the ith brain region of interest (ROI), $X_i^{\prime }(t)$ be the derivative of X_i(t) with respect to t with t ∈ [0, T], representing the increment of brain activity at time t. Furthermore, let d be the number of ROIs, we write $X(t)={\{X_1(t),\dots ,X_d(t)\}}^{\text{T}}$, and $X^{\prime }(t)={\{X_1^{\prime }(t),\dots ,X_d^{\prime }(t)\}}^{\text{T}}$. In practice, instead of the true signal, we observe a noisy signal at n discrete acquisition time points. Denote t_j as the jth acquisition time, to accommodate the noisy data, we write

$$Y_j=X(t_j)+{\mathit{ϵ}}_j,$$ (1)

where ϵ_j, j = 1, …, n, are independent mean zero random errors. Furthermore, we assume

$$X^{\prime }(t_j)=A(t_j)X(t_j),$$ (2)

where A(t_j) is a time-varying unknown matrix of size d × d. We name model (1) and (2) together the time-varying dynamic network (TVDN) model, where the dynamics of the resting state functional connectivity are captured by the time-varying matrix A(t). Model (2) is a direct extension of the dynamic mode decomposition model (Brunton et al., 2016; Kutz et al., 2016). To see the connection, first note that (2) is equivalent to $X(t+dt)=\{A(t)dt+I\}X(t)$, where dt is the unit measurement time and I is an identity matrix. When A(t) is a fixed matrix over time, we can consider A(t)dt + I as a constant matrix. Then model (2) reduces to the dynamic mode decomposition model extensively studied in Brunton et al. (2016) and Kutz et al. (2016). Furthermore, when A(·) is a fixed matrix where A(·) = −βℒ, then the model (2) is also a network diffusion model (Abdelnour et al., 2014) that explains how brain activations from different ROIs are coupled together to generate new signals via the structural connectivity given by the matrix Laplacian ℒ and the diffusivity constant β. Other algebraic graph relationships have also been proposed, such that A may be given by the eigenvectors of the structural (Abdelnour et al., 2018) or functional connectivity matrix (Becker et al., 2018), after a suitable transformation of the eigenvalues. While these approaches do not readily accommodate time-varying features of A, they point to an important property of the eigenvectors of A, which may be considered as resting state networks (RSN) (Abdelnour et al., 2018; 2014). Because these RSNs represent static brain connections or other non-dynamic brain substrates, we propose the following constraints that together constitute the TVDN model:

1. The eigen-decomposition of A(t_j) is in the form of

   $$A(t_j)=U\Lambda (t_j)U^{-1},$$

   where we fix the eigenvector U but allow the eigenvalues to depend on time. Under this formulation, U may be considered as a set of spatial features that are stationary over time. The absolute magnitudes of the (time-varying) eigenvalues govern the relative importance of each of the RSNs. This constraint reflects the biological concept that the spatial features of brain activity might be stationary, while the coupling between these stationary structures might be temporally dynamic.
2. We then impose the condition that dynamicity in FCs arises from discrete and potentially discontinuous shifts (“brain state switches”) in activity, which we accommodate by allowing the eigenvalues, i.e. diagonal elements of Λ(t), to be piece-wise constant functions of time, reflecting the phenomenon that the brain has a tendency to stay within, with sporadic cycling between the RSNs (Vidaurre et al., 2017). Therefore, let ${\mathit{\tau }}_0=({\tau }_k,k=0,1,\dots ,M,M+1;{\tau }_0=0,{\tau }_{M+1}=T)$ be a set of true switching points, dividing the signal to M + 1 stationary segments, we write

   $$\Lambda (t)=\Lambda ({\tau }_k)\text{if}{\tau }_{k-1}<t\le {\tau }_k,\text{and}\Lambda ({\tau }_k)\ne \Lambda ({\tau }_{k+1}),\Lambda ({\tau }_k)\ne \Lambda ({\tau }_{k-1}).$$

   Such formulation suggests that when $t\in ({\tau }_{0k-1},{\tau }_{0k}],\Lambda (t)$ has a constant value at Λ(τ_0k). And the values of Λ(·) are different at distinct time points that fall into two consecutive segments constructed by the switching points.
3. The number of nonzero eigenvalues in Λ(t), t ∈ [0, T] represents the number of intrinsic brain states in the brain activity data. It is well known that only a few RSNs are typically operational in the brain, and the canonical RSFC can be well captured by 7 – 20 such RSNs (Yeo et al., 2011). In prior graph theoretic models also, A(·) are assumed to be low rank matrices (Abdelnour et al., 2018; Raj et al., 2019). It is therefore plausible to assert that the number of such RSNs or brain state is quite small. Hence our final constraint is that

   $$rank\{A(t)\}=rank\{\Lambda (t)\}\le r\ll d,$$

   where r is the maximal rank for A(t), t ∈ [0, T], representing the number of distinct brain states.

2.2. Model interpretations

Since A(t) is constant in a given segment, the solution of the ordinary differential Eq. (2) in the kth segment is given by $X(t)=U\exp \{\Lambda ({\tau }_k)(t-{\tau }_{0k-1})\}{U}^{-1}{X}_{0k}$, where X_0k is the initial value at the kth segment and Λ(t) is a constant matrix in the kth segment. Let us define the real and imaginary components of the jth eigenvalue as ${\lambda }_j={\gamma }_j+i2\pi f_j$. Then the underlying signal in the kth segment satisfies

$$X(t)=\sum _{j=1}^r{U}_j\exp \{({\gamma }_j+i2\pi f_j)(t-{\tau }_{0k-1})\}{\{{(U^{-1})}^j\}}^{\text{T}}{X}_{0k},$$ (3)

where (U^−1j is the jth row of U⁻¹. The real term γ_j is interpreted as a coefficient that determines the growth or decay of the signal during this segment, and the imaginary component f_j is interpreted as the oscillation frequency of the mode (Kunert-Graf et al., 2019) in cycles per sample interval, which is 2 seconds for fMRI data and 1/60 seconds for MEG data). Therefore, when an estimator for $\mathbf{\Lambda }(t)$, say $\widehat{\Lambda }(t)$, is available for the kth segment, we can directly infer the grow/decay constant in the segment as $\mathrm{Re}\{\widehat{\mathbf{\Lambda }}(t)\}f$ and the signal frequency as $\mathrm{Im}\{\widehat{\mathbf{\Lambda }}(t)\}f/2\pi$, where f is the sampling frequency of the signals.

It is worth mentioning that in situations where mulitple modalities are available for the same subject, e.g. fMRI and MEG, the spatial features U may be considered to be shared between the modalities. In those cases TVDN will be able to aggregate the signals to generate a common estimator for U across the modalities. Then this common estimator will be used to obtain the modality specific temporal features Λ(t), where the information borrowing is clearly embedded in the estimation through sharing U. Augmentation with multi-modal data can potentially improve estimation accuracy.

2.3. Estimation of the spatial and temporal features

We estimate the spatial feature U through a kernel based method and detect the critical points for brain state switches via a switch detection algorithm.

2.3.1. Notation

Let t₁, …, t_n denote n signal acquisition time points. We denote M_r and M_r×r be the first r column and the first r × r block of M, respectively. Furthermore, define $\parallel ℳ\parallel$ be the cardinality of an arbitrary set $ℳ$. Let ‖M‖_F be the Frobenius norm of matrix M. For a vector a, let $\Vert a{\Vert }_1,\Vert a{\Vert }_2$ be its L₁, L₂ norm, respectively. Let M⁻¹ be the generalized inverse of matrix M.

2.3.2. B-spline smoothing

To obtain a proper estimator for X(t) and X′(t) from the noisy observation Y(t), we first de-noise the signals through the B-spline smoothing as follows

$$\begin{gathered} \widehat{\mathbf{\Gamma }}={\text{argmin}}_{\Gamma }\sum _{j=1}^n{\Vert Y_j-\Gamma B(t_j)\Vert }_2^2, \\ =\sum _{j=1}^n{Y}_{j}B{(t_j)}^{\text{T}}{\left\{\sum _{i=1}^{n}B(t_j)B{(t_j)}^{\text{T}}\right\}}^{-1} \end{gathered}$$

and

$$\widehat{X}(t_j)=\widehat{\Gamma }B(t_j),{\widehat{X}}^{\prime }(t_j)=\widehat{\Gamma }{B}_r^{\prime }(t_j),$$

where B(·) is the bth order B-spline basis with N interior knots and $\widehat{X}$ and ${\widehat{X}}^{\prime }$ are the smoothed version of X and X′, respectively. The B-spline estimation generates an estimator of A(t). When the sample size increases, this estimator will approach to A(t) consistently as shown in Theorem 1 in Appendix E.2. This leads to the consistent estimations of U and Λ(t) in the subsequent procedures. The de-noising step is necessary to generate a good estimator for X′(t). In practice, people can use other de-noising techniques, such as using Fourier-base or eigen-base expansion to approximate X(t). We choose B-spline method because it has well established statistical properties.

2.3.3. Estimation of the spatial features U

On these smoothed signals, we estimate the matrix A(t_s) at any time point of interest t_s by minimizing

$$\mathbf{ℒ}\{A(t_s)\}=\sum _{j=1}^n{\left\{{\widehat{X}}^{\prime }(t_j)-A(t_s)\widehat{X}(t_j)\right\}}^{\text{T}}\left\{{\widehat{X}}^{\prime }(t_j)-A(t_s)\widehat{X}(t_j)\right\}{K}_h(t_j-t_s),$$ (4)

where $K_h(|x|)=1/hK(|x|/h)$ is a kernel function with h be the bandwidth. Here $K_h(|x|)$ is a deceasing function of |x|. Hence when estimating $A(t_s),K_h(|x|)$ weighs the samples higher when they are closer to t_s. The width h of the kernel controls how “local” the estimator of A(t) is; if it is large, then the estimator of A(t) would hardly change over time, and it would reduce to the dynamic mode decomposition model (Kunert-Graf et al., 2019). A typical choice of K_h(|x|) is the Gaussian density function with the standard deviation h. The bandwidth h is often selected to satisfy h → 0 when n → ∞ so that even if n grows, when estimate A(t_s), the amount of information used in the estimation remains fixed. In Fig. 2, we show the weights K_h(t − 180) across t ∈ [1, 360] in seconds for fMRI and weights K_h(t − 30) across t ∈ [1, 60] in seconds for MEG when the rule-of-thumb bandwidth h (page 48 in Silverman (1986)) was selected.

Fig. 2.

The weights K_h(t − 180) across t ∈ [1, 360] in seconds for fMRI (left) and weights K_h(t − 30) across t ∈ [1, 60] in seconds for MEG (right) when the rule-of-thumb bandwidth h.

Minimizing $\mathbf{ℒ}\{A(t_s)\}$ has a close form solution for A(t_s) as

$$\sum _{j=1}^n{\widehat{X}}^{\prime }(t_j)\widehat{X}{(t_j)}^{\text{T}}{K}_h(t_j-t_s){\left[\sum _{j=1}^n\{\widehat{X}(t_j)\widehat{X}{(t_j)}^{\text{T}}\}{K}_h(t_j-t_s)\right]}^{-1},$$

which is the Nadaraya-Watson estimator (Nadaraya, 1964; Watson, 1964) regularly used to estimate functions at specific time points.

To account for the fact that $M={\sum }_{j=1}^n\{\widehat{X}(t_j)\widehat{X}{(t_j)}^{\text{T}}\}{K}_h(t_j-t_s)$ can be a low rank matrix, we replace the matrix inverse above with a truncated-rank inverse such that all eigenvalues of M below a threshold value are set to zero and removed from the pseudo-inverse. Formally, define a truncation function ρ_λ as ${\rho }_{\lambda }(M)=B_1\text{diag}\{{\sigma }_{j}I({\sigma }_j>\lambda ),j=1,\dots ,\min (m,n)\}{B}_2^{\text{T}}$, where B₁, B₂ are the left and right singular vectors and σ_j is the jth singular value of M. Here we choose a truncation threshold $b_0=O(h^{2}r+n^{-1/2}{N}^{1/2}{h}^{2}d)$, which is in the order of ${\Vert \widehat{X}W{\widehat{X}}^{\text{T}}-XW{X}^{\text{T}}\Vert }_F/n$ with $W=\text{diag}\{K_h(t_j-t_s),j=1,\dots ,n\}$ for a specific time point t_s and N be the number of B-spline basis. Here, as shown in Theorem 1 in Appendix E.2, the first order h²r comes from the kernel smoothing and the second term $n^{-1/2}{N}^{1/2}{h}^{2}d$ comes from the B-spline smoothing. When n → ∞ and $n^{-1/2}{N}^{1/2}\to 0$ as n → ∞, the error ${\Vert \widehat{X}W{\widehat{X}}^{\text{T}}-XW{X}^{\text{T}}\Vert }_F/n$ and quantity b₀ go to 0 if sufficient samples are collected overtime. Thus, the estimation of A(t_s) is

$$\widehat{A}(t_s)=\sum _{j=1}^n{\widehat{X}}^{\prime }(t_j)\widehat{X}{(t_j)}^{\text{T}}{K}_h(t_j-t_s){\rho }_{b_0}^{-1}\left[\sum _{j=1}^n\{\widehat{X}(t_j)\widehat{X}{(t_j)}^{\text{T}}\}{K}_h(t_j-t_s)\right].$$

The truncation function is specially designed for the high correlated sequences where the XWX^T is a low rank matrix, but $\widehat{X}W{\widehat{X}}^{\text{T}}$ can be full ranked due to the additional estimation error ${\Vert \widehat{X}W{\widehat{X}}^{\text{T}}-XW{X}^{\text{T}}\Vert }_F/n$. Using the truncation function helps to remove spurious eigenvalues, which not only improves the estimation accuracy but also stabilizes the computation.

When X is full row rank matrix, let $M_S\subseteq \{1,\dots ,n\}$ be the set of time indices, we can show that ${\Vert {\sum }_{s\in M_S}\widehat{A}(t_s)-A(t_s)\Vert }_F/\left|M_S\right|=O_p(h^{2}r+n^{-1}{N}^{1/2}dr)$, which goes to 0 when the sample size increases under mild conditions. Here, the first term h²r is the order of the estimation error in the kernel regression procedure and the second term $n^{-1}{N}^{1/2}dr$ is the order of the error from the B-spline smoothing procedure. When h → 0 and $n^{-1}{N}^{1/2}\to 0$ as n → ∞, the estimation error of ${\sum }_{s\in M_S}\widehat{A}(t_s)/\left|M_S\right|$ vanishes along with the increment of the sample size. This suggests that we need sufficient samples to recover the underlying true parameters. This fact also explains the phenomenon in the real data analysis that the reconstruction of the MEG signals is better than that of the fMRI signals. Therefore, we extract the estimator for U as the eigenvector of ${\sum }_{s\in M_S}\widehat{A}(t_s)$, denoted as $\widehat{\mathbf{U}}$.

2.3.4. Brain-state switch detection

Define M_r. as the first r rows of matrix M, and M_r×r as the first r × r block matrix of M. Because

$$X^{\prime }(t)=A(t)X(t)=U_r{\Lambda }_{r\times r}(t){(U^{-1})}_{r.}X(t),$$

after obtaining $\widehat{\mathbf{U}}$ we reduce the data dimension as

$${(U^{-1})}_{r.}{X}^{\prime }(t)={\Lambda }_{r\times r}(t){(U^{-1})}_{r.}X(t).$$ (5)

It is worth mentioning that multiplying both sides by ${(U^{-1})}_{r\cdot }X(t)$ projects the d dimensional ROIs to a lower r dimensional space. Furthermore, Λ_r×r(t) is a diagonal matrix, which contains only r unknown parameters. Such dimension reduction procedure is crucial for speeding up and stabilizing the switch detection algorithm, which makes the brain state switch detection practically feasible for signals from a large number of ROIs.

Let ${\tilde{X}}^{\prime }(t)={(U^{-1})}_{r\cdot }{X}^{\prime }(t)$ and $\tilde{X}(t)={(U^{-1})}_{r\cdot }X(t)$, we obtain the estimator for true switch number M and locations τ_k’s through minimizing a modified Bayesian information criteria (MBIC) defined as

$$\text{MBIC}(\mathit{\tau },M)=\sum _{k=0}^M\mathbf{ℒ}({\tau }_k+1,{\tau }_{k+1})+2r\log {(n)}^{\kappa }(M+1),$$

where κ is a constant, and

$$\mathbf{ℒ}({\tau }_k+1,{\tau }_{k+1})=-\sum _{s={\tau }_k+1}^{{\tau }_{k+1}}\log [\Phi \{{\tilde{X}}^{\prime }(s)-{\widehat{\Lambda }}_k\tilde{X}(s),0,{\widehat{\Sigma }}_k\}]$$

with Φ(x, a, s) be the multivariate normal density function with mean a and variance covariance matrix s evaluated at x. To solve the minimization problem, we iterate over all possible segmentations of the sequence. For the samples in a given segment, say $s\in ({\tau }_k+1,{\tau }_{k+1}]$, we obtain ${\widehat{\Lambda }}_k$ as

$${\widehat{\Lambda }}_k={\text{argmin}}_{{\Lambda }_k}\sum _{s={\tau }_k+1}^{{\tau }_{k+1}}{\{{\tilde{X}}^{\prime }(s)-{\Lambda }_k\tilde{X}(s)\}}^{\text{T}}\{{\tilde{X}}^{\prime }(s)-{\Lambda }_k\tilde{X}(s)\},$$

subject to the fact that Λ_k is a r × r diagonal matrix, and obtain ${\widehat{\Sigma }}_k$ as the estimated sample covariates defined as

$${\widehat{\Sigma }}_k=\sum _{s={\tau }_k+1}^{{\tau }_{k+1}}\{{\tilde{X}}^{\prime }(s)-{\widehat{\Lambda }}_k\tilde{X}(s)\}{\{{\tilde{X}}^{\prime }(s)-{\widehat{\Lambda }}_k{\tilde{X}}_{(s)}\}}^{\text{T}}/({\tau }_{k+1}-{\tau }_k).$$

We then find the best segmentation, that is the best (τ, M) that minimizes MBIC (τ, M) as the estimated locations and number of the switch points. In short, we obtain the estimators as

$$(\widehat{\mathit{\tau }},\widehat{M})={\text{argmin}}_{\tau ,M}\text{MBIC}(\mathit{\tau },M).$$

We employ the dynamic programming algorithm as detailed in Algorithm 1 following Jackson et al. (2005); Killick et al. (2012) to efficiently evaluate all possible segmentations and obtain $\widehat{\mathit{\tau }}$ and $\widehat{M}$. The dynamic programming algorithm finds the optimal value recursively, avoiding re-computing the ℒ over overlapped segments (Bellman and Roth, 1969; Bement and Waterman, 1977; Du et al., 2016a; Yau and Zhao, 2016). The computational cost is O(rn² M_max) and storage is O(rnM_max), where M_max is the maximum number of switches points in the signal (Du et al., 2016b).

Algorithm 1:

Dynamic programming algorithm.

Input: (1) Lmin, the minimum distance between two change points; (2) Mmax, the maximum number of changepoints; (3) κ value.

1. For 0 $\le i\le n-L_{\min }+1$ and $i+L_{\min }\le j\le n+1$, calculate $\mathbf{ℒ}(t_i,t_j)$.

2. Initialize $\mathit{H}\left(t_j\mid 0\right)=\mathbf{ℒ}(t_{j+1},t_{n+1}),j=0,\dots ,n-L_{\min }+1$.

3. For $1\le s\le M_{\max }$ and $0\le i\le n-s-L_{\min }$, update

$\mathit{H}\left(t_i\mid s\right)=\underset{i+L_{\min }\le j\le n-(s-1)}{\min }\left\{\mathbf{ℒ}(t_i+1,t_j)+\mathit{H}\left(t_j\mid s-1\right)\right\}.$

Record the locations of s change points that yield $\mathit{H}\left(t_0\right|s)$, denoted by ${\widehat{𝒥}}_s$.

4. For 1 ≤ s ≤ Mmax, find

$\widehat{\mathit{M}}={\text{argmin}}_s\mathit{H}\left(t_0\mid s\right)+2r\log {(n)}^{\kappa }(s+1)$.

The corresponding estimated switch point set is $\widehat{\mathit{\tau }}={\widehat{𝒥}}_{\widehat{M}}$.

Output: $\widehat{\mathit{\tau }}$ and $\widehat{\mathit{M}}$.

2.4. Competing methods: Sliding window approaches

We also implement the sliding window approaches: time-varying seed based (TVCOR), time-varying principal component analysis (TVPCA) and time-varying dynamic mode model (TVDMD). We perform TVCOR, TVPCA and TVDMD as follows where we construct windows in different sizes sliding by four frames in each step. Let S_l be the set of time point indices in the sliding-window l, l = 1, …, L. For matrix (Y_j, t_j ∈ S_l), TVCOR calculates the pairwise correlation between signals from different ROIs, TVPCA extracts the principal components, TVDMD extracts the dynamic modes (Brunton et al., 2016; Kunert-Graf et al., 2019) from the brain signals. Next we vectorize the resulting correlations, principal components and dynamic modes and cluster them into four clusters, corresponding to the number of true segments in the simulation. Finally, we obtain the switch locations as the time points where the vectorized correlations, principal components and dynamic modes switch the cluster memberships.

3. Results

3.1. Simulation study

We construct $A(t_j)=U\Lambda (t_j)U^{-1}$ for j = 1, …, 180, where Λ(s) is diagonal matrix whose diagonal terms are eigenvalues and U is the matrix whose columns are eigenvectors and eigenvectors estimated from a functional magnetic resonance imaging (fMRI). Here Λ(s) is a rank six matrix, which contains three switches at the 50, 99, 144 th time points. We simulate data from model (1) and (2), where ${\mathit{ϵ}}_j=U\Lambda (1)U^{-1}{\mathit{\xi }}_j/10,j=1,\dots ,n$ and ξ_j is a sparse error vectors with 10% nonzero entries. Each nonzero element in ξ_j is independently generated from a normal distribution with standard error (t₂ − t₁)/8. We simulate the data 100 times with the same set of parameters. Then we implement TVDN to obtain the estimated spatial feature $\widehat{\mathit{U}}$, the switch locations and the temporal features $\widehat{\Lambda }(s)$, where we select κ = 1.53 throughout the simulations so that the algorithm detects the correct number of switch points in over 80% of the simulated samples.

We plot the estimated switches in Fig. 3(a). The result shows that TVDN captures the brain state switch accurately. To illustrate the estimation results, we reconstruct the data by using estimated spatial and temporal features. We show the mean of the estimators at selected brain regions and the 95% empirical confidence interval, that includes 2.5% and 97.5% quantiles of the estimators over 100 simulations in Fig. 3(b). Fig. 3(b) shows that TVDN recovers the original noiseless sequence and the confidence intervals cover the true signals. We also add an additional simulation scenario when the switch points are distributed unevenly across the time in Fig. 12(b) in Appendix B. The result shows that TVDN detects the correct switch points in most of the simulations (82%). Finally, we examine the effect of B-spline knots selections on TVDN reconstruction errors Fig. 12(c) in Appendix B by plotting the distributions of the reconstruction errors across different B-spline knots selections. The results show that TVDN is insensitive to the selection of the B-spline knots so that the distributions of the reconstruction errors are consistent across different selections of the number of B-spline knots.

Fig. 3.

The simulation results with three switches. TVDN detects the true brain state switches and can reconstruct the true signal. (a) Switch times. Red lines are the true switch times and the dots are the estimated locations. (b) The black lines are the true X(t) at four selected regions. The red solid and dash curves are the mean and median of the estimators and above and below blue curves are the 95% empirical confidence intervals. The figures from left to right represent the results of the estimators whose mean squared errors fall at the 0%, 25%, 50% and 75% quantiles of the mean squared errors across all simulations.

In the left panel of Fig. 4(a), we plot the reconstruction errors defined as

$$\sum _{s=1}^n{\Vert Y_s-\exp \left\{{\int }_0^{t_s}\widehat{A}(u)du\right\}{Y}_1\Vert }_2^2,$$ (6)

when different ranks for $\widehat{\mathbf{U}}$ are selected in the estimations. The results show that the estimation error substantially drops from the setting when r = 4 to the setting when r = 6. Furthermore, when r > 6, the reconstruction error starts to incline. The convex phenomenon attributes to the tradeoff between the dimension reduction described in (5) and switch detection accuracy: when selecting a larger r, the transformation ${(U^{-1})}_r.X(t)$ contains more information in X(t), but it increases the estimation errors for the brain state switch detection algorithm; on the other hand, selecting a smaller r improves the switch detection accuracy, but ${(U^{-1})}_r.X(t)$ contains less information in the original data. On the right panel in Fig. 4(a), we show the MBIC values when selecting $\kappa =1.53$, which reach the minimum when three switches are selected.

Fig. 4.

The simulation results with three switches. Here A(t) has six ranks. (a) Left: the boxplot of the reconstruction error in (6) from 100 simulations when choosing different ranks for $\widehat{A}(t)$ in the estimation. Right: the boxplot of the MBIC values at different number of switches when κ = 1.53. (b) Hausdorrff distance between true switches and the estimated switches. The window sizes (wsize) selected are 10 (left) and 20 (right) for the sliding-window based methods. (c) Chang point locations for window sizes 10 (top) and 20 (bottom) for the TVCOR (left), TVPCA (middle) and TVDMD (right) methods.

We compare TVDN with the sliding window approaches: TVCOR, TVPCA and TVDMD. The detailed implementations of these competing methods are described in Section 2.4 in Appendix. We select six (the rank of A(s)) principal components and dynamic modes throughout the simulations.

Fig. 4 (b) plots the distribution of the Hausdorrff distance between the true and estimated switches for the different methods. A smaller Hausdorrff distance implies a better estimation. It can be seen that the switches from TVDN have the smallest Hausdorrff distance with the truth. There are several occasions that the sliding-window approaches outperform the TVDN method. This is because we specify the true number of segments in the sliding-window approaches, while we leave this parameter unknown in the TVDN approach and allow TVDN to choose it adaptively. To illustrate the pattern in more details, we plot the resulting switch locations from the sliding-window methods in Fig. 4(c). Fig. 4(c) shows that none of the three methods correctly identifies the switches. In addition, the sliding-window based methods are sensitive to the window size changes, which leads to substantial different results when varying the window sizes.

Finally, we adopt the same simulation procedure while assume a time invariant A(s). Then we implement TVDN on the resulting high dimensional sequences and reconstruct the observed data. We show the mean of the estimators and 95% confidence intervals in Fig. 12 in Appendix. The results show that even if A(s) is stationary over time, TVDN correctly extracts the spatial and temporal features from the brain signals.

3.2. TVDN results for resting state fMRI data

We present the TVDN detection results from one fMRI sequence in Fig. 5(a). We also obtain the growth/decay constant $\mathrm{Re}\{\widehat{\Lambda }(\cdot )\}f$ and the signal frequency as $\text{Im}\{\widehat{\mathbf{\Lambda }}(\cdot )\}f/2\pi$, where f = 0.5 Hz is the sampling frequency of the fMRI signal. It can be seen from Fig. 5(b) that the resting state fMRI brain signals are active in the frequency range between 0.001 and 0.007 Hz. We calculated the Pearson correlation between the weighted spatial features, that is, column sum of $\widehat{U}\widehat{\Lambda }(t)$, and the seven canonical networks from Yeo et al. (2011)’s independent component analysis. As shown in Fig. 5(c), the subject’s weighted spatial features have the strongest correlation with the limbic network in the first segment (0.38), with the ventral attention network in the second (0.443), third (0.415), sixth (0.432) and eighth (0.32) segments, with the dorsal attention network (0.24) in the fourth segment. This changing correlation pattern is indicative of brain state switches over time, demonstrating that different functional networks are operational at various times. In order to visualize the changing spatial patterns, we plotted the weighted spatial features across the segments on the brain surface in Fig. 5(d). The results again illustrate that the spatial pattern reflecting brain state switches among frontal, parietal and occipital lobes over time. We also present the estimated spatial eigen-mode, that is the modulus of the estimated U matrix in (e).

Fig. 5.

The results from the first fMRI dataset. (a) The real sequences with switch locations (black dash lines) detected by TVDN. (b) Changes of growth/decay constant $(\mathrm{Re}\{\widehat{\mathbf{\Lambda }}(\cdot )\}f)$, changes of the frequencies $(\text{Im}\{\widehat{\Lambda }(\cdot )\}f/2\pi )$. (c) The Pearson correlation between the weighted spatial features and the seven canonical networks. (d) The weighted spatial features across different segments detected by TVDN. (e) The static spatial features i.e. the moduli of first r columns of the U matrix.

We also plot, in Fig. 13 in Appendix, the pair-wise connectivity measure in each segment, defined by $\exp (-{\Vert x_1-x_2\Vert }_2)$, where x₁, x₂ represent signal sequences from two brain regions. Fig. 13 shows that the connectivity increases gradually over time. For comparison we show analogous results from TVDOR, TVPCA and TVDMD methods with different window sizes in Fig. 14 in the Appendix. The latter results suggest that these existing sliding-window methods are sensitive to tuning parameters and do not give coherent switch times when different window sizes are selected. Another representative example similar to the above is given in Fig. 15; its connectivity measures in Fig. 16 and the results from competing methods 17 in Appendix.

3.3. TVDN results for resting state MEG data

We evaluated TVDN on resting state MEG data, where we consider series of de-trended MEG source signals with d = 68 ROIs. Note that for MEG data, we did not filter the source signals because the high time resolution MEG data contain clear fluctuation trends that are not overwhelmed by the noise.

We obtain the detection results as shown in Fig. 6. We also obtain the growth/decay constant $\mathrm{Re}\{\widehat{\Lambda }(\cdot )\}f$ and the signal frequency as $\text{Im}\{\widehat{\mathbf{\Lambda }}(\cdot )\}f/2\pi$, where f = 60 Hz is the sampling frequency for the MEG signal. The results in Fig. 6(a) show that there are seven switches in the signal. In addition, the brain is active in the frequency range between 0 to 6 Hz as shown in Fig. 6(b). We also plot the correlation between the weighted spatial features and the seven canonical network in Fig. 6(c). It can be seen that the subject’s weighted spatial features have the strongest correlation with the visual network in the first segment (0.31), with the dorsal attention network in the second (0.32), third (0.45), fifth (0.28) and sixth (0.44) segments, with limbic network (0.29) in the seventh segment, and with frontopartietal network (0.62) in the eighth segment. These correlations are larger than those from the resting state fMRI (Fig. 5). Finally, we view the weighted spatial features across the segments in Fig. 6(d). We also present the estimated spatial eigen-mode, that is the modulus of the estimated U matrix in (e). In addition, we plot, in Fig. 18 in Appendix, the pair-wise connectivity measure in each segment, which shows that the connectivity increases from the first to the third segment, decreases from the fourth to the sixth segment, and increases again until the end of the time. We further show the results from the existing sliding-window methods in Fig. 19 in the Appendix, which demonstrates that the sliding-window methods are sensitive to the window size selection. Another representative example is given in Fig. 20, the corresponding connectivity measures in Fig. 21 and results from competing methods in Fig. 22 in Appendix.

Fig. 6.

The results from the first resting state MEG dataset. (a) The real sequences with switch locations detected by TVDN. The dash lines are the detected brain state switches. (b) Changes of growth/decay constant $(\mathrm{Re}\{\widehat{\mathbf{\Lambda }}(\cdot )\}f)$, changes of the frequencies $(\text{Im}\{\widehat{\Lambda }(\cdot )\}f/2\pi )$. (c) The Pearson correlation between the weighted spatial features and the seven canonical networks. (d) The weighted spatial features across different segments detected by TVDN. (e) The static spatial features i.e. the moduli of first r columns of the U matrix.

3.4. TVDN results for task based MEG data

To validate the accuracy of the brain state switch detection, we evaluate TVDN on MEG recordings during a simple eyes-open to eyes-close task-switching experiment, where six eye close and open tasks blocks were performed within one minute and the switch times were manually labeled. In Fig. 7, we show the detection results based on the MEG data from two subjects. Clearly, the switch locations from TVDN are very close to the manually labeled ones, which suggests TVDN can correctly identify the brain state switch times. Take the first sample as an example, we obtain the growth/decay constant and the signal frequency with f = 120 Hz be the sampling frequency for the two task based MEG signals. The brain is active in the frequency between 0 to 12 Hz as shown in Fig. 23 (a) in Appendix, which is higher than that from the resting state MEG. Furthermore, we obtain the band passed signals in alpha band (8–12 Hz), and re-estimated the U and Λ based on the filtered the signals. We then calculate the Pearson correlation between the re-estimated weighted spatial features and the seven canonical networks. As shown in Fig. 23(b) in Appendix, although the correlations with the visual network change over time, the switch patterns do not exactly follow the eyes-open and eyes-close states. This implies there are brain state changes that are unrelated to the visual network during the data acquisition period. Moreover, the brain views of the re-estimated weighted spatial features in Fig. 23(c) illustrate that the brain state in alpha band switches in between inferior parietal and supra marginal in the parietal lobe in most of the segment, while it switches to occipital lobe at the end of the time. We also plot in Fig. 24 in Appendix the pair-wise connectivity measure in each segment base on the unfiltered signal, which shows that the connectivity decreases from the first to the fourth segment, increases from the fourth to the fifth segment, and decreases again to the end of the time. We further show the results from the TVCOR, TVPCA, and TVDMD methods in Fig. 25 in Appendix, which suggests none of the methods provides robust result across the selected window sizes. Finally, base on the second sample, we show the growth/decay constants, signal frequency, correlations with the canonical networks and brain views in Fig. 26, the connectivity measures in Fig. 24 and the results from competing methods in Fig. 28 in Appendix.

Fig. 7.

TVDN captures the task-switching dynamics in two eyes-open to eyes-close task-switching MEG records. The the real sequences with switch locations detected by TVDN. The black dashed lines are the detected brain state switches. The red solid lines are the manually labeled switch times.

3.5. Comparison to benchmark methods

We implemented TVDN on 103 fMRI datasets. The distribution of the number of switches and ranks are displayed in Fig. 8 (a) and (b), respectively, which show around 50% samples have eight switches and over 65% samples have seven distinct brain states (ranks) in the resting state.

Fig. 8.

The distribution of the spatial features are similar from different approaches (c) and the related prediction errors from TVDN are smaller compare with those from TVDMD, the only existing method that allows for the reconstruction of the signals (d). (a) The distribution of the number of switches across samples. (b) The distribution of the brain state across samples. (c) The distributions of the maximum correlation between the spatial features from TVPCA, TVDMD, and TVDN methods with the canonical networks. (d) The average related prediction errors from the TVDN and TVDMD methods. The shaded area is the 95% confidence band from the TVDN method. The 95% confidence band from TVDMD covers the entire plotted area, which we do not show.

We further evaluated the correlations between TVDN spatial features with the seven canonical networks from Yeo et al. (2011)’s independent component analysis under selected κ and r. We extract the spatial features as the moduli of the first r columns of $\widehat{\mathbf{U}}$ from each subject, and project them to [0, 1] interval. We also implemented the TVPCA, TVDMD methods to obtain the corresponding principal components and dynamic modes from each segment as the spatial features, and calculated their correlations with the canonical networks. We plot the distributions of the maximum correlations between the canonical networks and the spatial features from TVDN, TVPCA and TVDMD across 103 samples in Fig. 8(c). It can be seen that although TVDN has far fewer spatial features compared with TVPCA and TVDMD (each subject has only r spatial features), the distribution of the maximum correlation is similar with those from TVPCA and TVDMD. In addition, we plot the prediction errors versus the number of switches from TVDN and TVDMD in Fig. 8(d). To obtain the prediction error, for each segment in between two consecutive switch points, we use the first half of the fMRI records as the training data to estimate A(t) in the segments. Then we use the rest of the signals as the testing data, and calculate the related prediction errors defined as

$$N_{\mathit{test}}^{-1}\sum _s{\Vert Y_s-\exp \left\{{\int }_0^s\widehat{A}(u)du\right\}{Y}_{s_0}{\Vert }_2/\Vert Y_s\Vert }_2,$$

where Y_s is the sth observed signals, Y_s0 is first signal in the testing sample, N_test is the total number of testing sample (half of the signal length) and the summation is over the test signals. We average the corresponding prediction errors across the segments and individuals for the TVDN and TVDMD methods. For the TVDN method, we further construct the 95% confidence bands of the prediction errors as 2.5% (lower) and 97.5% (upper) quantiles of the errors in the 103 study samples. We do not show the 95% confidence band from TVDMD because it almost covers the entire plotted area. Fig. 8(d) shows that TVDN has smaller prediction error than TVDMD, especially when the number of switches is larger than four. It also suggests that when the number of switches is small, each segment contains sufficient samples to recover the large number of parameters in TVDMD. Therefore, when there are less than four switches, TVDN and TVDMD perform equally well in prediction as the confidence band cover both curves. However, when the number of switches is moderately large, the sample in each segment is no longer enough to provide accurate estimations for TVDMD parameters. Therefore, the more parsimonious TVDN method yields substantially smaller prediction errors than the TVDMD method.

To illustrate the robustness of TVDN, in Fig. 9, we plot the distribution of the number of switches from TVDN and the sliding-window methods when different kernel bandwidths and window sizes are selected, respectively. Note that the kernel bandwidth in TVDN serves the same function as the window sizes in the sliding-window methods. For each window size, we adjust the kernel bandwidth so that the lower 2.5% and upper 97.5% of the Gaussian kernel correspond to the left and right endpoints of the window, respectively. It can be seen that TVCOR, TVPCA and TVDMD are sensitive to the window size selection – the larger the window size, the smaller the number of detected brain switches. In contrast, TVDN is robust to the kernel bandwidth selection, with only small shifts of the distribution center with increasing kernel bandwidth.

Fig. 9.

TVDN’s brain state switch detection is robust to the kernel bandwidth selection but the sliding-window methods are sensitive to the window size selection. The distributions of the switch points when different window sizes (wsize) are chosen for the sliding-window methods and different kernel bandwidths are chosen for TVDN. The kernel bandwidths are adjusted so that the lower 2.5% and upper 97.5% quantiles of the Gaussian kernel correspond to the left and right endpoints of the window, respectively.

To show the reproducibility of our method, we split the 103 subjects in the fMRI study to two samples with approximately equal sample sizes (52/51). We then implement TVDN on the two samples separately and study the distributions of the number switch points, the ranks, and the correlation of the spatial features with the canonical networks. We present the results in Fig. 10. These distributions are coherent in the two samples, which demonstrates the reproducibility of TVDN.

Fig. 10.

The distributions of the number switch points (a), the ranks (b), and the correlation of the spatial features with the canonical networks (c) are the same in the first and second halves of the fMRI samples.

3.6. Testing null hypothesis of static functional connectivity

TVDN is designed to estimate the time-varying functional connectivity, it inevitably returns time-resolved estimates of functional connectivities that vary to some degree with time. It is important to evaluate whether the estimated time varying functional connectivities significantly deviate from those that might have been obtained from time series generated by a process that lacks state switching (Lurie et al., 2020). To this end, we develop a testing procedure to test that whether a sequence contains switch points. More specifically, when a sequence has been divided to multiple segments after the TVDN detection, we use the first half of the signals in each segment as the training data and the rest of the signals as the testing data. Under the alternative hypothesis that there is at least one switch point, we use the first half of the signals in each segment to estimate A(t) based on the model X′(t) = A(t)X(t). We then predict the second half of the signals using the segment-specific estimator for A(t), and calculate the prediction error as

$$N_{test}^{-1}\sum _{k=1}^{\widehat{M}}\sum _{s=1}^{n_k}{\Vert Y_{ks}-\exp \{\widehat{A}({\widehat{\tau }}_k)s\}{Y}_{k0}\Vert }_2/{\Vert Y_{ks}\Vert }_2,$$

where $\widehat{\mathit{M}}$ is estimated number of switch points, n_k is the number of testing sample in the kth segment, ${\widehat{\tau }}_k$ the kth estimated switch points, $\widehat{A}({\widehat{\tau }}_k)$ is the estimator for $A({\widehat{\tau }}_k)$ based on the training data in the kth segment, Y_k0 is the initial observed value in the kth segment from the testing data and N_test is the total sample size of the testing data. Note that the summation is taken over the testing data on the kth segments. Under the null hypothesis, we combine the training data from all segments, and perform a resampling procedure to construct the null distribution of the prediction error. More specifically, we sample paires of estimated X′(t) and X(t) with replacement from training data and estimate A matrix based on the static model X′(t) = AX(t). We then use the estimated A to predict the signals in the testing sample and calculate the prediction errors as

$$N_{\mathit{test}}^{-1}\sum _{k=1}^{\widehat{M}}\sum _{s=1}^{n_k}{\Vert Y_{ks}-\exp \{\widehat{A}s\}{Y}_{k0}\Vert }_2/{\Vert Y_{ks}\Vert }_2,$$

where $\widehat{A}$ is estimator for A using the sampled training data. We repeat this procedure 100 times and obtain the p-value as the percentage of prediction errors from null distribution that are less than the prediction error under the alternative. We show the 100 prediction errors under the null hypothesis versus the prediction error under the alternative hypothesis for the two resting state and two eye open/closed MEG data in Fig. 11 (a)–(d). The results show that all p-values are less than 0.05, suggesting there is at least one switch point in every sequence. Furthermore, we plot the p-values for the fMRI data from 103 healthy subjects in Fig. 11(e). The results show that all p-values are less than 0.05, suggesting that every fMRI signal has at least one switch point.

Fig. 11.

The p-values of testing whether there are switch points. (a), (b) p-values from two MEG resting state data examples. (c), (d) p-values from two eye open/closed data examples. (e) p-values from 103 fMRI data. The red dots in (a)–(d) are the log of the prediction errors across 100 simulations under the null hypothesis. The black line is the log of the prediction error of TVDN method under the alternative hypothesis. The red dots in (e) are the percentage of errors under null hypothesis that is less than the one from TVDN method. The black line is 0.05 cutoff value.

4. Discussion and conclusion

We proposed a novel biologically-constrained model of the brain state evolution during resting-state functional recording, called the TVDN model. We presented an optimal algorithm to infer the model’s parameters and to extract the spatial and temporal features from resting state brain signals. The method relies on the assumption that while the spatial signatures of RSFC, given by the eigenvectors of the forward model, are static, the evolution of temporal features, given by the eigenvalues, is dynamic within the recording duration. We developed an eigenvector estimation technique to extract consistent spatial features across signal acquisition times. In addition, we proposed a dynamic programming based algorithm to detect temporal switches adaptively based on the signal oscillation patterns, under the biologically-inspired assumption that state transitions are abrupt rather than smooth in time. Using the inferred spatial and temporal features, we can reconstruct the underlying mean signals that generate the noisy observations. This may be considered a model-based smoothing operation, with several potential applications. Thus, our method is a legitimate generative model of dynamic functional activity in the brain. In addition, the ability to reconstruct noiseless signals gives the algorithm an opportunity to tune its parameters using a reconstruction error metric to be minimized. We evaluated the method on thorough simulated data, followed by a rigorous characterization of its performances on empirical fMRI and MEG data from the BIL laboratory at UCSF. The simulation study shows that TVDN captures the true brain switch locations and is able to recover the true signal that generates the observed ones. In the empirical study, for comparison we implemented several competing techniques, including TVCOR, TVPCA and TVDMD methods. Compared with competing methods, TVDN produces smaller set of spatial features but their correlations with the seven canonical networks have the same distributions as those from the TVCOR, TVPCA and TVDMD methods. This suggests the smaller set of spatial features from TVDN is sufficient to explain the brain connection patterns. Furthermore, TVDN provides more robust temporal features, which are adaptive to the signals and noises from different data and are insensitive to the tuning parameters, such as kernel bandwidth. In addition, the evaluation on the eye-opening-closing task data shows that TVDN captures the brain state switches accurately. More importantly, TVDN has significantly smaller prediction errors than TVDMD does when predicting “future” activity in the same segment. Last but not least, the resulting temporal features include instantaneous estimates of the active oscillation frequency of functional activity, thus imparting the method with attributes of a model-based alternative to conventional time-frequency analysis.

The ultimate solution to improving the estimation accuracy on fMRI data is to integrate multi-modality data into the analysis. It is therefore highly advantageous that TVDN is naturally able to handle multimodality data. To understand this aspect intuitively, note that the stationary spatial features are by design modality invariant, and can be shared across multiple modalities. This imparts the TVDN framework with the ability to integrate information from both fMRI and MEG to estimate the spatial features. For example, we could train TVDN on concatenated data (over time) from different modalities to obtain shared spatial features. These shared spatial features can then be used to estimate the modality-specific temporal features, using information from both fMRI and MEG at each step, which will certainly improve estimation accuracy. While in this study we have shown how TVDN can operate seamlessly on both fMRI and MEG, we have not integrated the two for the current analysis because the data from paired samples are not available. Evaluating its performance on synchronized multi-modality data would require larger collaborative studies involving both the fMRI and MEG centers.

One question of clinical interest is whether the dynamic RSFC predicts clinical outcomes, such as cognitive scores and disease risk. To address the question, the first and foremost step is to extract subject-specific dynamic RSFC features. However, the dynamic RSFC features from existing sliding-window methods give a set of RSNs of varying numbers across subjects, which makes it difficult to explicitly define unique spatial and temporal features for each subject. In contrast, TVDN extracts subject-specific dynamic RSFCs from both the fMRI and MEG data, which generates explicit spatial and temporal features that can be directly used to predict clinical outcomes. Evaluate the relationship between the dynamic RSFC features and clinical outcomes may potentially generate novel biomarkers for disease prediction.

4.1. Related methods

Sliding-window approaches are the most popular methods to extract dynamic RSFC from brain imaging data. However, the most popular seed-based sliding window approaches do not typically allow for reconstructing the original brain signals in time or space, since they do not require a model of signal generation. And the temporal resolution of the inferred dynamic FC is inherently limited by the window length, which in turn is constrained by the requirement to have sufficient samples and signal-to-noise ratio within each window. In practice, this tradeoff means that only slow changes in brain dynamics can be detected or tracked. Furthermore, in almost all current implementations, the sliding-window width is typically pre-specified and is not adaptable to the signal statistics or sampling noise in real time. In addition, they do not generate common features from the multiple modalities that may be available from a single subject (e.g. fMRI and MEG). This impedes information sharing across modalities and precludes benefiting from shared or redundant information between modalities.

Moreover, these methods typically suffer from very high data dimensionality, since at each window, the brain state is given by an entire network or several high-dimensional independent components - with no a priori notion of which features are actually evolving and which are static. Therefore the ability to detect discrete brain state switches then becomes dependent on the ability of unsupervised clustering algorithms like k-means or hierarchical clustering, to overcome the so-called“curse of dimensionality”. Finally, most dynamic extensions to static FC methods are purely data-driven and are not informed by biologically plausible modes of dynamicity in the brain, since they do not constrain which brain signal features can change dynamically and how - this aspect is discussed below. Therefore, sliding-window approaches present several limitations that must be overcome to gain further progress in critical neuroscience and clinical applications. Several extensions of current methods have been proposed to address some of these limitations. To improve the sliding-window seed-based correlation approach, Faghiri et al. (2020) proposed a new metric replacing Pearson correlation between signals. Furthermore, Vergara et al. (2020) proposed a robust method to determine the number of brain states from the sliding window methods. Hidden Markov models (Baum and Petrie, 1966) are an another robust alternative to capture brain state switches in the frequency domain. Vidaurre et al. (2017) and Quinn et al. (2018) used group level data to estimate the model parameters, assuming that study subjects share the same latent structure. Vidaurre et al. (2016) combined multivariate auto-regression model (Penny and Roberts, 2002) and hidden Markov model to obtain brain transitions, assuming that brain oscillations depend on the signals in a short time period prior to the current time. However, neither the improved sliding window methods nor the hidden Markov models are able to extract both the static spatial and dynamic temporal features from non-stationary time series. It is possible that further extension of these methods to account for both static and dynamic features will prove worthwhile, but out of scope of the current work. Furthermore, because the HMM model involves a large number of parameters, ad hoc dimension reduction procedures are always performed to reduce the computation burden. For example, the computational time of the HMM method proposed in Vidaurre et al. (2018) grows polynomially with the number of states. When studying the whole brain signals, the number of regions of interest was reduced by using the principal component method, the number of latent state was restricted to be a small number, and the time series were split to segments using sliding windows to facilitate the computation. The computational time of TVDN grows linearly with the number of ROIs (please see the discussion in the last paragraph of Section 2.3.4), which avoids restricting the number of brain states and does not require to use ad hoc sliding windows to split the time series.

4.2. Limitations and future directions

In our implementation, the tuning parameters of TVDN for fMRI were selected to minimize the average reconstruction error, and for MEG they were selected to minimize prediction error from cross validation among brain regions. A better tuning strategy might be to use cross-validation across individuals, where the data are split between training and testing individuals, and the tuning parameters are selected to minimize the prediction error in the testing individuals. However, because our switch detection relies on the entire time series of the whole brain, there is no existing method to split the study samples and validate parameter selection in the temporal switch detection procedure. Furthermore, a smaller prediction error may not necessarily imply a better prediction of disease states, like neurodegenerative disease risk. When the disease outcomes are available, an appropriate parameter selection strategy would be to select the tuning parameters that minimize the disease prediction error. Additional data and further research along these lines are ongoing in our laboratory.

Generally, fMRI signals have lower signal-to-noise ratio and temporal resolution than source-reconstructed MEG signals, which limits the former’s sample size available for parameter estimation. Hence, the brian state switching patterns extracted by TVDN from MEG appear clearer than those from fMRI, with larger correlation with canonical networks. This suggests that MEG imaging could be a more informative technique to capture dynamic RSFC than fMRI. It is possible that alternative smoothing approaches other than the one taken here might prove more effective on fMRI. It is possible that deconvolution of the hemodynamic response function might be helpful on fMRI data, an aspect that was not considered here.

Appendix A. Additional methods and data preprocessing procedure

A1. Tuning parameter selection

We choose rank r so that the first r moduli of the eigenvalues of ${\sum }_t\widehat{A}(t)$ comprise 80% of the total sum of them, where the summation is taken over a random subset of times. Furthermore, we select the bandwidth for kernel in (4) to be the rule-of-thumb bandwidth times 0.5. Moreover, for the resting state fMRI data, we select κ to minimize the variation of the number of switches across the subjects. For the resting state MEG data we select κ through resampling over acquisition time. More specifically, for a given κ, we select five subsamples, where the jth sample contains data at times 5t + j with j = 1, …, 5 and t = 1, …, (n − 5)/5. Since the five sequences are in conjunction with each other, they should have similar numbers of switches.

A2. Data and preprocessing

A2.1. fMRI data

Resting state fMRI data from 103 health subject were acquired at the UCSF Neuroimaging Center using a Siemens 3T TIM TRIO scanner using a T2* -weighted AC-PC aligned echo planar imaging (EPI) sequence with the following parameters: TR = 2000 ms, TE = 29 ms, flip angle = 75, FOV = 240 x 240, slice thickness = 3.5 mm. Each fMRI was recorded over six minutes with 0.5 Hz sampling rate. Preprocessing included slice-timing correction (Cox and Hyde, 1997), image realignment to correct for motion (Jenkinson and Smith, 2001), and intensity normalization. The head-motion parameters were estimated before any spatiotemporal filtering was used (Jenkinson et al., 2002). After regression of nuisance signals, fMRI was coregistered on the T1-weighted anatomical image, and the resulting time-series were normalized to MNI space with the non-linear registration from ANTS (Avants et al., 2009). Following time series extraction, data were detrended and a bandpass filter was applied between 0.009 and 0.08 Hz. To remove the boundary effect from the filtering procedure, we removed the first 25 sampling points. Hence, the total length of the signal is 155.

A2.2. MEG data

MEG data were acquired in the Biomagnetic Imaging Laboratory at University of California, San Francisco (UCSF) with an Omega 2000 whole-head MEG system from CTF Inc. (Coquitlam, BC, Canada) with 1200 Hz sampling rate. For resting state data analysis, two subjects were instructed simply to keep their eyes closed and stay awake. We collected 4 trials per subject, each trial of 1-min length with a sampling rate of 1200 Hz. We randomly chose 10 seconds or equivalently 12000 time samples for brain source reconstructions for each subject. Additionally, for one subject, MEG data were collected across two sessions for an eye-opening-closing task. To measure eye opening and closing, two pairs of Electrooculography (EOG) electrodes were placed to the left and right of the eye during MEG scans. A potential difference was recorded when the subject blinked eyes and a signal peak occurred in the EOG channel of scanned data. We manually labeled EOG peaks to indicate time periods of eye opening and closing for TVDN analyses. Across both resting and eye-opening task, all MEG sensor locations were co-registered to each subject’s anatomical MRI scans. The leadfield for each subject was calculated in NUTMEG (Dalal et al., 2004) using a single-sphere head model (two spherical orientation leadfields) and an 8 mm voxel grid. Each column was normalized to have a norm of unity. The data were digitally filtered to remove DC offset and any other noisy artifact outside of the 1 to 45 Hz bandpass range.

To infer the neuronal activity in the source space from the MEG recordings, which were in sensor space, source localization was performed using time-frequency optimized adaptive beamforming (Dalal et al., 2004) using the custom-built open source NUTMEG software tool. Since this study focuses on the cortical areas and only the sources belonging to the 68 cortical regions were selected based on the Desikan-Killiany parcellations. The time-course of activity in each of the 68 brain regions was estimated by averaging the time-course of source activity estimated from voxels within a 20 mm radius of its centroid.

The resting state MEG data were downsampled to 600 Hz, while the eye-opening-closing MEG data were downsampled to 1200 Hz in our analysis.

Appendix C. fMRI additional results

Fig. 12.

(a) The simulation results with no switch. Here A(t) has six ranks. The black lines are the true X(t) at four selected regions. The red solid and dash curves are the mean and median of the estimators and above and below blue curves are the 95% empirical confidence intervals. The figures from left to right represent the results of the estimators whose mean squared errors follow on the 0%, 25%, 50%,75% quantiles of the mean squared errors across all simulations. The confidence intervals are narrow because the simulated random errors have small variabilities. (b) The switch points detection results for the setting when the difference between the switch points are significant different. (c) The reconstruction errors are insensitive to the number of B-spline knots selection.

Fig. 13.

The pairwise connectivity over 90 ROIs from the first fMRI example.

Fig. 14.

The brain state switch detection is not robust across different window size selections for the sliding-window approaches on the first fMRI example.

Appendix D. MEG additional results

Fig. 15.

The results from the second fMRI sample. (a) The real sequences with switch locations (black dash lines) detected by TVDN. (b) Changes of growth/decay constant $(\mathrm{Re}\{\widehat{\mathbf{\Lambda }}(\cdot )\}f)$, changes of the frequencies $(\text{Im}\{\widehat{\Lambda }(\cdot )\}f/2\pi )$. (c) The Pearson correlation between the weighted spatial features and the seven canonical networks. (d) The weighted spatial features across different segments detected by TVDN. (e) The static spatial features i.e. the moduli of first r columns of the U matrix.

Fig. 16.

The pairwise connectivity over 90 ROIs from the second fMRI example.

Fig. 17.

The brain state switch detection is not robust across different window size selections for the sliding-window approaches on the second fMRI example.

Fig. 18.

The pairwise connectivity over 68 ROIs from the first MEG resting state example.

Fig. 19.

The brain state switch detection is not robust across different window size selections for the sliding-window approaches on the first MEG resting state example.

Fig. 20.

The results from the second resting state MEG record. (a) The real sequences with switch locations (black dash lines) detected by TVDN. (b) Changes of growth/decay constant $(\mathrm{Re}\{\widehat{\mathbf{\Lambda }}(\cdot )\}f)$, changes of the frequencies $(\text{Im}\{\widehat{\Lambda }(\cdot )\}f/2\pi )$. (c) The Pearson correlation between the weighted spatial features and the seven canonical network. (d) The weighted spatial features across different segments detected by TVDN. (e) The static spatial features i.e. the moduli of first r columns of the U matrix.

Fig. 21.

The pairwise connectivity over 68 ROIs from the second MEG resting state example.

Fig. 22.

The brain state switch detection is not robust across different window size selections for the sliding-window approaches on the second MEG resting state example.

Fig. 23.

The results from the first eye-opening-closing MEG record. (a) Changes of growth/decay constant $(\mathrm{Re}\{\widehat{\mathbf{\Lambda }}(\cdot )\}f)$, changes of the frequencies $(\text{Im}\{\widehat{\Lambda }(\cdot )\}f/2\pi )$. (b) The Pearson correlation between the weighted spatial features and the seven canonical networks. (c) The weighted spatial features across different segments detected by TVDN with eye-opening-closing labels. (e) The static spatial features i.e. the moduli of first r columns of the U matrix.

Fig. 24.

The pairwise connectivity over 68 ROIs from the first MEG eyes-open and eyes-close example.

Fig. 25.

The brain state switch detection is not robust across different window size selections for the sliding-window approaches on the first MEG eye-opening-closing example.

Fig. 26.

The results from the second eye-opening-closing MEG record. (a) Changes of growth/decay constant $(\mathrm{Re}\{\widehat{\mathbf{\Lambda }}(\cdot )\}f)$, changes of the frequencies $(\text{Im}\{\widehat{\Lambda }(\cdot )\}f/2\pi )$. (b) The Pearson correlation between the weighted spatial features and the seven canonical networks. (c) The weighted spatial features across different segments detected by TVDN with eye-opening-closing labels. (e) The static spatial features i.e. the moduli of first r columns of the U matrix.

Fig. 27.

The pairwise connectivity over 68 ROIs from the second MEG eyes-open and eyes-close example.

Fig. 28.

The brain state switch detection is not robust across different window size selections for the sliding-window approaches on the second MEG eye-opening-closing example.

Appendix E. Statistical consistency

We show the statistics consistency of estimating A(t) and U in the following sections. These theoretic results support the statistical convergence rates of $\widehat{A}(t)$ and $\widehat{\mathbf{U}}$ presented in Section 2.3.3. First we list the regularity conditions that are necessary to prove the statistical consistency.

E1. Regularity conditions

• A1In the kernel function $K_h(t)=K(t/h)/h$, K is a second order symmetric kernel function that satisfies $\int K(t)dt=1,\int K^2(t)dt<\infty$, and $\int t^{2}K{(t)}^{2}dt<\infty$. h satisfies h → 0 when n → ∞.
• A2X_i(t) is bounded on [0,1].
• A3Define the knots $t_{-b+1}=\cdots =t_0=0<t_1<\dots <t_N<1=t_{N+1}=\cdots =t_{N+b}$, where N is the number of interior knots and [0, 1] is divided into N + 1 subintervals. N satisfies N → ∞, N⁻¹ n(logn)⁻¹ → ∞ when n → ∞.
• A4And for t in $({\tau }_{0k},{\tau }_{0k+1}]\Vert A(t){\Vert }_{op}({\tau }_{0k+1}-{\tau }_{0k})/n\le 1,k=1,\dots ,{\mathit{M}}_0$.

• Assume $X_i(t)\in C^q([0,T])$, there is N + b dimensional γ_0i, and a bth order Bspline such that ${\sup }_{x\in [0,T]}\left|B^{\text{T}}(t){\mathit{\gamma }}_{0i}-X_i(t)\right|=O_p(N^{-s})$, And denote ${\Gamma }_0={({\mathit{\gamma }}_{0i},i=1,\dots ,d)}^{\text{T}}$.

• A6Let h_p be the distance between the (p + 1)st and pth interior knots and let $h_b=\underset{b\le p\le N+b}{\max }{h}_p,h_s=\underset{b\le p\le N+b}{\min }{h}_p$. There exists a constant $c_{h_b},0<c_{h_b}<\infty$ such that $h_b/h_s<c_{h_b}$. Therefore, $h_b=O(N^{-1}),h_s=O(N^{-1})$.

E2. Theorem of the statistical consistency

We show the consistency of the $\widehat{A}(t)$ and $\widehat{\mathbf{U}}$ in Theorem 1. Below we first show that the brain activity X_i(t) is a smooth function and can be approximated consistently with B-spline function in Proposition 1. By utilizing the results from Proposition 1 and Lemma 1, Theorem 1 and Remark 1 establish the consistency of $\widehat{A}(t)$ and $\widehat{\mathbf{U}}$.

Proposition 1. Assume Condition (A4) holdes, for given s, there is a Bspline function ${\sup }_{t\in [0,1]}\left|B{(t)}^{\text{T}}{\mathit{\gamma }}_{0i}-X_i(t)\right|=O_p(N^{-s})$ with order b > s.

Proof. By the fact that each element in A(s) is a piece-wise constant function we can write

$$A(t)=\sum _{k=0}^{M_0}{A}_k\mathit{I}({\tau }_{0k}/n<t\le {\tau }_{0k+1}/n),$$

where τ₀₀ = 0 and τ_0M0+1 = n. We first show that for $t\in ({\tau }_{0k}/n,{\tau }_{0k+1}/n]$ by the induction

$$X_i(t)=\exp \left\{\sum _{l=0}^{k-1}{A}_l({\tau }_{0l+1}-{\tau }_{0l})/n+A_k(t-{\tau }_{0k}/n)\right\}{X}_0.$$ (7)

First (7) holds for $t\in ({\tau }_{00}/n,{\tau }_{01}/n]$ because for any constant matrix M, X′(t) = MX(t) as a closed form solution as

$$X(t)=\exp (Mt)X_0.$$

Suppose (7) holds for $t\in ({\tau }_{0k}/n,{\tau }_{0k+1}/n]$, we show that it holds for $t\in ({\tau }_{0k+1}/n,{\tau }_{0k+2}/n]$. For any $t\in ({\tau }_{0k+1}/n,{\tau }_{0k+2}/n]$, we have

$$\begin{gathered} X(t)=\exp \{A_{k+1}(t-{\tau }_{0k+1}/n)\}X({\tau }_{0k+1}/n) \\ =\exp \{A_{k+1}(t-{\tau }_{0k+1})/n)\} \\ \exp \left\{\sum _{l=0}^{k-1}{A}_l({\tau }_{0l+1}-{\tau }_{0l})/n+A_k({\tau }_{0k+1}-{\tau }_{0k})/n\right\}{X}_0 \\ =\exp \left\{\sum _{l=0}^k{A}_l({\tau }_{0l+1}-{\tau }_{0l})/n+A_{k+1}(t-{\tau }_{0k+1}/n)\right\}{X}_0, \end{gathered}$$

which is the same as the relation in 7 . Hence, (7) holds.

By the Taylor expansion, $t\in ({\tau }_{0k}/n,{\tau }_{0k+1}/n]$ we have

$$\begin{gathered} X(t)=X({\tau }_{0k}/n)+\sum _{l=1}^s\frac{A_k^l{(t-{\tau }_{0k}/n)}^l}{l!}X({\tau }_{0k+1}/n) \\ +\sum _{l=s}^{\infty }\frac{A_k^l{(t-{\tau }_{0k}/n)}^l}{l!}X({\tau }_{0k+1}/n) \\ =X({\tau }_{0k}/n)+\sum _{l=1}^s\frac{A_k^l{(t-{\tau }_{0k}/n)}^l}{l!}X({\tau }_{0k+1}/n)+R_s \end{gathered}$$

where $R_{s_1}={\sum }_{l=s_1}^{\infty }\frac{A_k^l{(t-{\tau }_{0k}/n)}^l}{l!}X({\tau }_{0k+1}/n)$. Now by the Condition (A4) that ${\Vert A_k\Vert }_{op}({\tau }_{0k+1}-{\tau }_{0k}/n)\le 1$, we have there is a s₁ such that

$$\begin{gathered} {\Vert R_{s_1}\Vert }_2\le {\Vert \sum _{l=s_1}^{\infty }\frac{A^l{(t-{\tau }_{0k}/n)}^l}{l!}X({\tau }_{0k+1}/n)\Vert }_2 \\ \le {(t-{\tau }_{0k}/n)}^{s_1}{\Vert A_k^{s_1}\Vert }_{op}{\Vert X({\tau }_{0k+1}/n)\Vert }_2\sum _{l=1}^{\infty }1/k! \\ ={(t-{\tau }_{0k}/n)}^{s_1}{\Vert A_k^{s_1}\Vert }_{op}{\Vert X({\tau }_{0k+1}/n)\Vert }_{2}e \\ =O_p(N^{-s}). \end{gathered}$$

Furthermore, because for any s₁ degree polynormal function there is a b ≥ s₁ − 1 order exact Bspline representation (De Boor, 1978), we have conclude that there is a Bspline function $\left|B{(\cdot )}^{\text{T}}{\mathit{\gamma }}_{0i}-X_i(t)\right|=O_p(N^{-s})$ with order b > s for t ∈ [0, 1]. □

Lemma 1. Assume B_k, k = 1, …, d are B-spline bases with equally distributed knots. There is a constant D_b > 0 such that for each spline ${\sum }_{k=1}^d{c}_k{\mathit{B}}_k(t)$, and for each 1 ≤ p ≤ ∞

$$D_b{\Vert c^{\prime }\Vert }_p\le {\left[{\int }_0^1{\left\{\sum _{k=1}^d{c}_k{B}_k(t)\right\}}^{p}dt\right]}^{1/p}\le {\Vert c^{\prime }\Vert }_p,$$

where $c^{\prime }={\{c_k{\{(t_k-t_{k-r})/b\}}^{1/p},k=1,\dots ,d\}}^{\text{T}}$, where d is the number of basis and b is the distance between B-spline knots.

Proof. This is a direct consequence of Theorem 5.4.2 on page 145 in DeVore and Lorentz (1993). □

Theorem 1. Assume Conditions (A1)–(A6) hold. Let $U_x\in {ℝ}^{d\times d},V_x\in {ℝ}^{d\times n}$ and ${\Sigma }_x\in {ℝ}^{d\times d}$ be the left, right singular vectors and singular value matrix of the rank q matrix X, with q ≤ r.

Assume $E\{\Vert X(t){\Vert }_2\}=O_p(1)$ and $E\{\Vert \mathit{ϵ}(t){\Vert }_2\}=O_p(1)$. Suppose

$$\begin{gathered} \sqrt{n{N}^{-1}}\{\widehat{X}(t)-X(t)\}=G_1(t)\{1+o_p(1)\} \\ \sqrt{n}{N}^{-3/2}\{{\widehat{X}}^{\prime }(t)-X^{\prime }(t)\}=G_2(t)\{1+o_p(1)\} \end{gathered}$$

$$\begin{gathered} G_1(t)=\sqrt{n{N}^{-1}}\sum _{i=1}^n\{Y_i-X_i(t_i)\}B{(t_i)}^{\text{T}}{\left\{\sum _{i=1}^{n}B(t_i)B{(t_i)}^{\text{T}}\right\}}^{-1}B(t) \\ {G}_2(t)=\sqrt{n}{N}^{-3/2}\sum _{i=1}^n\{Y_i-X_i(t_i)\}B{(t_i)}^{\text{T}}{\left\{\sum _{i=1}^{n}B(t_i)B{(t_i)}^{\text{T}}\right\}}^{-1}{B}^{\prime }(t) \end{gathered}$$

$G_1(t),G_2(t)$ are mean 0 Gaussian vectors with each element to be of order $O_p(1)$.

$$\begin{gathered} \widehat{A}(t_s)\equiv \sum _{j=1}^n{K}_h(t_j-t_s){\widehat{X}}^{\prime }(t_j)\widehat{X}{(t_j)}^{\text{T}}{\rho }_{C\{h^2+n^{-1/2}{N}^{1/2}{h}^{2}d\}}^{-1} \\ \left\{\sum _{j=1}^n{K}_h(t_j-t_s)\widehat{X}(t_j)\widehat{X}{(t_j)}^{\text{T}}\right\}, \end{gathered}$$

for some constant C > 0. And if |M_S| = O(n), when h → 0, we have

$${\Vert {\left|M_S\right|}^{-1}\sum _{s\in M_S}\left\{\widehat{A}(t_s)-A(t_s)U_{xq}{U}_{xq}^{\text{T}}\right\}\Vert }_F=D(h^{2}r+n^{-1/2}{N}^{1/2}d)$$

for some constant D > 0 and ${\widehat{\mathbf{U}}}_r$ satisfies

$$\Vert {\widehat{U}}_r-U_{pr}{\Vert }_F=O_p(h^{2}r+n^{-1}{N}^{1/2}d),$$

where U_pqr is the eigenvector of ${\sum }_{s\in M_S}A(t_s)U_{xq}{U}_{xq}^{\text{T}}$.

Proof. Let W = diag{K_h(t_j − t_s), j = 1, …, n}, for each t _s, we have

$$\begin{gathered} n^{-1}\sum _{j=1}^n{K}_h(t_j-t_s){\widehat{\mathbf{X}}}^{\prime }(t_j)\widehat{\mathbf{X}}{(t_j)}^{\text{T}} \\ =n^{-1}\sum _{j=1}^n{K}_h(t_j-t_s)X^{\prime }(t_j)X{(t_j)}^{\text{T}}+R_1(t_s)+R_2(t_2) \\ =n^{-1}\sum _{j=1}^n{K}_h(t_j-t_s)A(t_j)X(t_j)X{(t_j)}^{\text{T}}+R_1(t_s)+R_2(t_2) \\ =n^{-1}\sum _{j=1}^n{h}^{-1}K\{(t_j-t_s)/h\}A(t_j)X(t_j)X{(t_j)}^{\text{T}}+R_1(t_s)+R_2(t_2) \\ ={\int }_0^T{h}^{-1}K\{(t-t_s)/h\}A(t)X(t)X{(t)}^{T}d{P}_n(t)+R_1(t_s)+R_2(t_2) \\ ={\int }_0^{T}K(u)A(t_s+hu)X(t_s+hu)X{(t_s+hu)}^{\text{T}}d{P}_n(u)+R_1(t_s)+R_2(t_2) \\ ={\int }_0^{T}K(u)\{A(t_s)+A^{\prime }(t_s)hu+A^{″}(t_s)h^2{u}^2\{1+o_p(1)\}\}X(t_s+hu)X{(t_s+hu)}^{\text{T}}d{P}_n(u)+R_1(t_s)+R_2(t_2) \\ =A(t_s){\int }_0^{T}K(u)X(t_s+hu)X{(t_s+hu)}^{\text{T}}d{P}_n(u)+h^2{R}_k{Q}_1\{1+o_p(1)\}+R_1(t_s)+R_2(t_s) \\ =A(t_s)XW{X}^{\text{T}}/n+h^2{R}_k{Q}_1\{1+o_p(1)\}+R_1(t_s)+R_2(t_2), \end{gathered}$$ (8)

where P_n is the empirical measure of t,

$$Q_1=\{A^{\prime }(t_s)X^{\prime }(t_s)X{(t_s)}^{\text{T}}+A^{\prime }(t_s)X(t_s)X^{\prime }{(t_s)}^{\text{T}}+A^{″}(t_s)X(t_s)X{(t_s)}^{\text{T}}\},{\Vert Q_1\Vert }_F=O_p(r),$$

$$\begin{gathered} R_1(t_s)\equiv n^{-1}\sum _{j=1}^n{K}_h(t_j-t_s){\widehat{X}}^{\prime }(t_j){\{\widehat{X}(t_j)-X(t_j)\}}^{\text{T}} \\ =n^{-1}\sum _{j=1}^n{K}_h(t_j-t_s)[X^{\prime }(t_j)+n^{-1/2}{N}^{3/2}{G}_2(t_j)\{1+o_p(1)\}] \\ \times {[n^{-1/2}{N}^{1/2}{G}_1(t_j)\{1+o_p(1)\}]}^{\text{T}}, \end{gathered}$$ (9)

$$\begin{gathered} R_2(t_s)\equiv n^{-1}\sum _{j=1}^n{K}_h(t_j-t_s)\{{\widehat{X}}^{\prime }(t_j)-X^{\prime }(t_j)\}X{(t_j)}^{\text{T}} \\ =n^{-1}\sum _{j=1}^n{K}_h(t_j-t_s)n^{-1/2}{N}^{3/2}{G}_2(t_j)\{1+o_p(1)\}X{(t_j)}^{\text{T}}, \end{gathered}$$ (10)

and recall that

$$R_k={\int }_0^{T}K(u)u^{2}d{P}_n(u).$$

Similarly we have

$$\begin{gathered} n^{-1}\sum _{j=1}^n{K}_h(t_j-t_s)\widehat{X}(t_s)\widehat{X}{(t_s)}^{\text{T}} \\ =n^{-1}\sum _{j=1}^n{K}_h(t_j-t_s)X(t_s)X{(t_s)}^{\text{T}}+R_3(t_s)+R_4(t_s) \\ ={\int }_0^{T}K(u)X(t_s+hu)X{(t_s+hu)}^{\text{T}}d{P}_n(u)+h^2{R}_{k}Q\{1+o_p(1)\}+R_3(t_s)+R_4(t_s) \\ =XW{X}^{\text{T}}/n+h^2{R}_{k}Q\{1+o_p(1)\}+R_3(t_s)+R_4(t_s), \end{gathered}$$

where

$$Q=\{X^{\prime }(t_s)X^{\prime }{(t_s)}^{\text{T}}+X(t_s)X^{″}{(t_s)}^{\text{T}}+X^{″}(t_s)X{(t_s)}^{\text{T}}\},\Vert Q{\Vert }_F=O_p(r)$$

$$\begin{gathered} R_3\equiv n^{-1}\sum _{j=1}^n{K}_h(t_j-t_s)\{\widehat{X}(t_s)-X(t_s)\}X{(t_s)}^{\text{T}} \\ =n^{-1}\sum _{j=1}^n{K}_h(t_j-t_s)n^{-1/2}{N}^{1/2}{G}_1(t_s)\{1+o_p(1)\}X{(t_s)}^{\text{T}} \end{gathered}$$

$$\begin{gathered} R_4\equiv n^{-1}\sum _{j=1}^n{K}_h(t_j-t_s)\widehat{X}(t_s){\{\widehat{X}(t_s)-X(t_s)\}}^{\text{T}} \\ =n^{-1}\sum _{j=1}^n{K}_h(t_j-t_s)[X(t_s)+n^{-1/2}{N}^{1/2}{G}_1(t_s)\{1+o_p(1)\}] \\ \times n^{-1/2}{N}^{1/2}{G}_1{(t_s)}^{\text{T}}\{1+o_p(1)\}. \end{gathered}$$

By the asymptotic bias of the kernel regression estimator, and the fact that R₃, R₄ are rank d matrices, we have

$${\Vert R_3\Vert }_F=C_1{n}^{-1/2}{N}^{1/2}{h}^{2}d,{\Vert R_4\Vert }_F=C_2{n}^{-1/2}{N}^{1/2}{h}^{2}d$$

for positive constants C₁, C₂, where the last equality holds by the order of the asymptotic bias of the kernel estimator. Furthermore, let Σ_xq be the upper q × q diagonal matrix of Σ_x and V_xq be first q column of V_x, we can write

$$XW{X}^{\text{T}}=U_x\left\{\begin{array}{cc}\left({\Sigma }_{xq}{V}_{xq}^{\text{T}}W{V}_{xq}{\Sigma }_{xq}\right)& 0\\ 0& 0\end{array}\right\}{U}_x^{\text{T}}.$$

Hence, as h → 0, there is a C such that

$$\begin{gathered} {\rho }_{C\{h^2+n^{-1/2}{N}^{1/2}{h}^{2}d\}}^{-1}\left\{n^{-1}\sum _{j=1}^n{K}_h(t_j-t_s)\widehat{X}(t_j)\widehat{X}{(t_j)}^{\text{T}}\right\} \\ ={(XW{X}^{\text{T}}/n)}^{-1} \\ =U_x\left\{\begin{array}{cc}{\left({\Sigma }_{xq}{V}_{xq}^{\text{T}}W{V}_{xq}{\Sigma }_{xq}/n\right)}^{-1}& 0\\ 0& 0\end{array}\right\}{U}_x^{\text{T}}. \end{gathered}$$ (11)

Combine (8) and (11), and Condition that A(t_s) is rank r matrix, we have

$$\begin{gathered} \widehat{A}(t_s)-A(t_s)U_{xq}{U}_{xq}^{\text{T}} \\ =h^2{R}_k{Q}_1\{1+o_p(1)\}{(XW{X}^{\text{T}}/n)}^{-1} \\ +R_1(t_s){(XW{X}^{\text{T}}/n)}^{-1}+R_2(t_2){(XW{X}^{\text{T}}/n)}^{-1} \end{gathered}$$ (12)

$$=O_p(h^2{r}^{1/2})+R_1(t_s){(XW{X}^{\text{T}}/n)}^{-1}+R_2(t_2){(XW{X}^{\text{T}}/n)}^{-1}$$ (13)

The last equality holds because Q₁ (XWX^T /n)⁻¹ is a rank r matrix and ${\Vert Q_1{(XW{X}^{\text{T}}/n)}^{-1}\Vert }_F=\sqrt{r}{\Vert Q_1{(XW{X}^{\text{T}}/n)}^{-1}\Vert }_2=O_p(\sqrt{r})$ by the fact that $\Vert X(t){\Vert }_2=O_p(1)$ and in turn ${\Vert X^{\prime }(t)\Vert }_2=\Vert A(t)X(t){\Vert }_2=\Vert A(t){\Vert }_{op}\Vert X(t){\Vert }_2=O_p(1)$. Furthermore,

$$\begin{gathered} {\Vert {\left|M_S\right|}^{-1}\sum _{s\in M_S}{R}_1(t_s){(XW{X}^{\text{T}}/n)}^{-1}\Vert }_F \\ =\Vert n^{-1}\sum _{j=1}^n{\left|M_S\right|}^{-1}\sum _{s\in M_S}{K}_h(t_j-t_s)[X^{\prime }(t_j)+n^{-1/2}{N}^{3/2}{G}_2(t_j)\{1+o_p(1)\}] \\ \times {[n^{-1/2}{N}^{1/2}{G}_1(t_j)\{1+o_p(1)\}]}^{\text{T}}{(XW{X}^{\text{T}}/n)}^{-1}{\Vert }_F \\ =\Vert n^{-1}\sum _{j=1}^n{f}_t(t_j)[X^{\prime }(t_j)+n^{-1/2}{N}^{3/2}{G}_2(t_j)\{1+o_p(1)\}] \\ {[n^{-1/2}{N}^{1/2}{G}_1(t_j)\{1+o_p(1)\}]}^{\text{T}} \\ \times {(XW{X}^{\text{T}}/n)}^{-1}{\Vert }_F \\ =D_1{n}^{-1/2}{N}^{-1/2}q, \end{gathered}$$

where D₁ is a positive constant and f_t(·) is the density function for t. The third line holds because ${\left|M_S\right|}^{-1}{\sum }_{s\in M_S}{K}_h(t_j-t_s)$ is a consistant estimator for f_t(t_j). The last equality holds by the arguments as follows. First let

$$Q_a(t)=f_t(t)[X^{\prime }(t)+n^{-1/2}{N}^{3/2}{G}_2(t)\{1+o_p(1)\}]$$

and q_k(t) be its kth element. Then by the definition of G₁ in the Theorem statement, the second to the last equality can be written as

$$\begin{gathered} \Vert n^{-1}\sum _{j=1}^n{Q}_a(t_j)B{(t_j)}^{\text{T}}{\left\{\sum _{i=1}^{n}B(t_i)B{(t_i)}^{\text{T}}\right\}}^{-1} \\ \sum _{i=1}^{n}B(t_i){\{Y_i-X(t_i)\}}^{\text{T}}{(XW{X}^{\text{T}}/n)}^{-1}{\Vert }_F \\ \times \{1+o_p(1)\} \\ =\Vert {\int }_0^T{Q}_a(t)B{(t)}^{\text{T}}dt{\left\{\sum _{i=1}^{n}B(t_i)B{(t_i)}^{\text{T}}\right\}}^{-1} \\ \sum _{i=1}^{n}B(t_i){\{Y_i-X(t_i)\}}^{\text{T}}{(XW{X}^{\text{T}}/n)}^{-1}{\Vert }_F\{1+o_p(1)\}. \\ \le {\Vert {\int }_0^T{Q}_a(t)B{(t)}^{\text{T}}dt{\left\{\sum _{i=1}^{n}B(t_i)B{(t_i)}^{\text{T}}\right\}}^{-1}\sum _{i=1}^{n}B(t_i){\{Y_i-X(t_i)\}}^{\text{T}}\Vert }_{op} \\ \times {(XW{X}^{\text{T}}/n)}^{-1}{}_F\{1+o_p(1)\}. \end{gathered}$$

Now because ${\int }_0^T{q}_k(t)B_l(t)dt=q_k(t^{\ast }){\int }_0^T{B}_l(t)dt=O_p(t_k-t_{k-b})=O_p(N^{-1})$ by the mean value theorem and Lemma 1 with p = 1 . Therefore, each element in

$${\int }_0^T{Q}_a(t)B{(t)}^{\text{T}}dt{\left\{\sum _{i=1}^{n}B(t_i)B{(t_i)}^{\text{T}}\right\}}^{-1}$$

is of order O_p(N⁻¹). Furthermore, each element in G₁ is of order O_p(1), and hence each element in

$${\left\{\sum _{i=1}^{n}B(t_i)B{(t_i)}^{\text{T}}\right\}}^{-1}\sum _{i=1}^{n}B(t_i){\{Y_i-X(t_i)\}}^{\text{T}}$$

is of order n^−1/2N^1/2 by the definition of G₁ in the theorem statement. Therefore,

$$\begin{gathered} {\Vert n^{-1}\sum _{j=1}^n{Q}_a(t_j)B{(t_j)}^{\text{T}}{\left\{\sum _{i=1}^{n}B(t_i)B{(t_i)}^{\text{T}}\right\}}^{-1}{\{Y_i-X(t_i)\}}^{\text{T}}\Vert }_{\max } \\ =O_p(n^{-1/2}{N}^{1/2}{N}^{-1})=O_p(n^{-1/2}{N}^{-1/2}) \end{gathered}$$

and hence

$$\begin{gathered} {\Vert n^{-1}\sum _{j=1}^n{Q}_a(t_j)B{(t_j)}^{\text{T}}{\left\{\sum _{i=1}^{n}B(t_i)B{(t_i)}^{\text{T}}\right\}}^{-1}{\{Y_i-X(t_i)\}}^{\text{T}}\Vert }_{op} \\ =O_p(d{n}^{-1/2}{N}^{-1/2}). \end{gathered}$$

Now combine with the fact that (XWX^T /n)⁻¹ is a rank q matrix, we obtain the result in the last equality.

Similarly, we have

$${\left|M_S\right|}^{-1}{\Vert \sum _{s\in M_S}{R}_2(t_s){(XW{X}^{\text{T}}/n)}^{-1}\Vert }_F=D_2{n}^{-1/2}{N}^{1/2}d$$

for some positive constants D₂. Combine with (12), we have

$${\left|M_S\right|}^{-1}{\Vert \sum _{s\in M_S}\widehat{A}(t_s)-\sum _{s\in M_S}A(t_s)U_{xq}{U}_{xq}^{\text{T}}\Vert }_F=D(h^{2}r+n^{-1/2}{N}^{1/2}d)$$

for positive constant D. The first r column of the eigenvector of ${\left|M_S\right|}^{-1}{\sum }_{s\in M_S}A(t_s)U_{xq}{U}_{xq}^{\text{T}}$ is U_p respectively. Hence $\Vert {\widehat{U}}_r-U_{pr}\Vert =O_p(r{h}^2+n^{-1/2}{N}^{1/2}d)$ as h → 0. This proves the result. □

Remark 1. Theorem 1 shows that when XWX^T is a full rank matrix, ${\sum }_{s\in M_S}\widehat{A}(t_s)/\left|M_S\right|$ converges to ${\sum }_{s\in M_S}A(t_s)/\left|M_S\right|$ consistently. Hence ${\widehat{U}}_r\to U_r$ with probability one. If XWX^T is a low rank matrix, ${\sum }_{s\in M_S}\widehat{A}(t_s)/\left|M_S\right|$ converges to a projection of ${\sum }_{s\in M_S}A(t_s)/\left|M_S\right|$ on the sub-space of R^q, where q is the rank of XWX^T. In practice, if XWX^T is a low rank matrix, we can first project X to full rank sub-space and perform the TVDN algorithm on the projected signals. The conditions

$$\begin{gathered} \sqrt{n{N}^{-1}}\{\widehat{X}(t)-X(t)\}=G_1(t)\{1+o_p(1)\} \\ \sqrt{n}{N}^{-3/2}\{{\widehat{X}}^{\prime }(t)-X^{\prime }(t)\}=G_2(t)\{1+o_p(1)\} \end{gathered}$$

are the general properties of B-spline estimator as shown in (Jiang et al., 2019; 2015). We use the result without proof.