Statelets: Capturing recurrent transient variations in dynamic functional network connectivity

Abstract

Dynamic functional network connectivity (dFNC) analysis is a widely used approach for capturing brain activation patterns, connectivity states, and network organization. However, a typical sliding window plus clustering (SWC) approach for analyzing dFNC models the system through a fixed sequence of connectivity states. SWC assumes connectivity patterns span throughout the brain, but they are relatively spatially constrained and temporally short‐lived in practice. Thus, SWC is neither designed to capture transient dynamic changes nor heterogeneity across subjects/time. We propose a state‐space time series summarization framework called “statelets” to address these shortcomings. It models functional connectivity dynamics at fine‐grained timescales, adapting time series motifs to changes in connectivity strength, and constructs a concise yet informative representation of the original data that conveys easily comprehensible information about the phenotypes. We leverage the earth mover distance in a nonstandard way to handle scale differences and utilize kernel density estimation to build a probability density profile for local motifs. We apply the framework to study dFNC of patients with schizophrenia (SZ) and healthy control (HC). Results demonstrate SZ subjects exhibit reduced modularity in their brain network organization relative to HC. Statelets in the HC group show an increased recurrence across the dFNC time‐course compared to the SZ. Analyzing the consistency of the connections across time reveals significant differences within visual, sensorimotor, and default mode regions where HC subjects show higher consistency than SZ. The introduced approach also enables handling dynamic information in cross‐modal and multimodal applications to study healthy and disordered brains.

We proposed a novel method for analyzing dynamic functional connectivity via extracting high‐frequency texture from the connectivity space. The analysis of those motifs enables measuring the characteristics of brain circuitry and network organization. The experiments don't he summary motifs facilitate the observation of distinguishing connectivity signatures and the interplay among the hubs to process information

1. INTRODUCTION

Schizophrenia (SZ) is a neuropsychiatric disorder characterized by diverse cognitive impairments and a decline in personal and social functioning. The decomposition of brain images into meaningful independent components (ICs) and generating biomarkers helps analyze SZ to a greater extent (Calhoun, Liu, & Adalı, 2009; Du et al., 2019; Erhardt et al., 2011; Liang et al., 2006; Rahaman et al., 2019; Zhou et al., 2008). Nevertheless, to reason about the neuropsychiatric disorders and studying the brain remains challenging because of the heterogeneous nature of these diseases (Alnæs et al., 2019; Tsuang, Lyons, & Faraone, 1990). Thus, studies employ subgrouping/clustering of the subjects to minimize the dissimilarity in the population and make the investigation more viable (Luchins, Weinberger, & Wyatt, 1979; Rahaman et al., 2020; Scarr et al., 2009). The human brain is considered an interconnected dynamical system with real‐time interaction between different nodes of the brain network (Bassett & Sporns, 2017; Betzel & Bassett, 2017). These dependencies among the neuronal populations or brain regions are described as functional connectivity (FC) and generally computed using Pearson correlation of time courses (TCs) in task‐based or resting‐state functional magnetic resonance Imaging (fMRI) (Buckner, Krienen, & Yeo, 2013; Power et al., 2011). Recent empirical evidence has suggested that this dynamic spatiotemporal configuration better models brain activities and network organization (Liu et al., 2017; Ma et al., 2014; Sakoğlu et al., 2010; Vergara et al., 2018; Zhi et al., 2018). Studies have hypothesized that SZ is a disorder of disrupted cognition, network dysconnectivity, and lack of functional integration (Friston & Frith, 1995; Lynall et al., 2010; Stephan, Baldeweg, & Friston, 2006). Moreover, not all cognitive processes occur within a single brain component, instead of requiring dynamic reconfiguration of neural resources such as the brain network's nodes and connections (Alavash, Tune, & Obleser, 2019; Petersen & Sporns, 2015). As such, research into the brain's static and dynamic functional network connectivity (dFNC), in which nodes are distributed maps, for example, IC maps, may provide crucial insights into functional integration and its disorder due to a heterogeneous neuropsychiatric disorder such as SZ.

dFNC is defined as the temporal interdependence among intrinsic connectivity networks (ICNs) extracted from independent component analysis (ICA) (Allen et al., 2014; Jafri et al., 2008; Rashid et al., 2014). dFNC study provides an ability to track time‐varying transitions in connectivity strength, thus moving beyond the limiting assumption of a single static connectivity pattern (Rashid et al., 2014; Vergara et al., 2018). However, a typical dFNC analysis assumes fixed discrete states with varying occupancy over time and does not capture continuous transient information. Studies often analyze dFNC through connectivity states and estimate those states via clustering the windowed FNC sliding across time (Miller et al., 2016; Rashid et al., 2014; Saha et al., 2019a). The obtained states essentially fuse all homogenous connectivity patterns ignoring their order and timing. Additionally, these states explore connectivity patterns that span throughout the brain and prevail for a longer timescale. In practice, connectivity signatures are more spatially constrained and exist at a shorter timescale (Grandjean et al., 2017; Hilger et al., 2020; Liao et al., 2017; Miller et al., 2016). That is, transformations manifest over a slower timescale, and dynamics can adjust functional network topology accordingly (Aine et al., 2017; Bullmore & Sporns, 2009). So, these methods are less suitable for capturing dynamic changes in the dFNC TC. Specially, those which manifest over a short and continuous timescale. Consequently, the methods also incompetent to analyze dynamic properties of dFNC, such as synchronizability and intermittent connectivity, which are also affected by neuropsychiatric disorders like SZ (Siebenhühner et al., 2013; Yu et al., 2012).

Addressing the shortcomings of the existing methods, an increasing number of recent studies have started to investigate connectivity time series in a reformed timescale through bipartitions (Sporns et al., 2020) or network dynamics (Esfahlani et al., 2020; Faskowitz et al., 2020; Morioka, Calhoun, & Hyvärinen, 2020; Sporns et al., 2020). We further advance this agenda by introducing a robust approach to decompose FC into a more granular scale by tracking the most recurring patterns of the dFNC TC. To capture brief, repetitive co‐fluctuations, we focus on the time‐series “motif”—a previously unknown but recurring time‐series pattern (Lin et al., 2002). In data mining, time series motifs as such signatures are deemed powerful tools for modeling and analyzing dynamical systems (Mueen, 2014; Torkamani & Lohweg, 2017). The dFNC TC represents the link between a pair of nodes during a time interval; intuitively, we are interested in the frequently occurring behavior of the signal rather than a few sporadic episodes. Motif extraction from each connection of FC can easily yield an extensive collection of variable‐length shapes even for a moderate connectivity dataset. A comprehensive collection of samples dramatically complicates the task of summarizing the transient dynamics into a few interpretable trends; thus, predicting the overall dynamical system's behaviors is still similar to looking for a needle in a haystack.

This article addresses the tradeoff between the desire to model transient dynamics and the need to constrain the set of extracted features to a reasonable size. Summarization helps to distill useful information and general trends from the data making them interpretable. Studies that propose summarizing time series are mostly domain‐specific and lack generalization (S. Ahmad, Taskaya‐Temizel, & Ahmad, 2004; Kacprzyk, Wilbik, & Zadrożny, 2008; Sripada et al., 2003). This article offers a novel probabilistic pattern summarization framework called “statelets,” highlighting the dynamics of capturing FC of the brain. Statelets are driven by the desire to build a general summarization method and rely on an efficient earth mover distance (EMD) (Rüschendorf, 1985) implementation supporting kernel density estimation (KDE) in motif space. We demonstrate how the EMD may be an effective similarity metric for motif comparison. EMD provides a scale‐independent comparison between signatures that can handle variable‐length substructures and account for partial matching (Rubner, Tomasi, & Guibas, 2000).

Results present the summary prototypes of both connectivity dynamics. The statelets from both groups pose substantial group differences in multiple dynamic properties of the brain's functional system. The connection rank computed based on the probability density (PD) within the groups reveals unique co‐fluctuations among functional brain networks and their corresponding dynamic interplay (Figure 7). Connections identified in SZ patients show reduced modularity relative to healthy controls (HCs) (Figure 9), and in addition, HC statelets are significantly more recurring than in SZ (Figure 10). An experiment of the transitivity of time decay (TD) graphs indicates that HC networks are more often in sync and for more extended periods, consistent with increased inter‐communication to SZ (Figure 11). Finally, statelet‐wise subgrouping of the dynamics reports salient and stable group differences in sensorimotor (SM), cognitive control (CC), and cerebellar (CB) domains, which are not observed in earlier studies. The main contributions of our article can be highlighted as follows:

• • Propose a new statelet approach to address the limitations of current sliding window plus clustering (SWC) methods for dFNC analysis.

• • Adapt the EMD as a distance metric for comparing time series motifs.

• • Use time‐series motifs for capturing transient recurring co‐fluctuation of neuronal populations and unwarp the FC in a finer time scale.

• • Develop a novel probabilistic motifs summarization framework for providing the synopsis of the dynamic through a subset of connectivity prototypes.

• • Use PD of motifs to assess spatial consistency across a group.

• • Introduce a TD metric to investigate the summary signatures' temporal consistency.

• • Enable measurement of the brain's dynamic properties: an inherent dynamical system.

• • Show significant differences in SZ patients who generally show less frequent and shorter statelets.

2. DEFINITIONS AND BACKGROUND

2.1. Definition 1: Dynamic functional network connectivity

dFNC is a time‐varying correlation typically computed using a sliding window (block of time points, for example, 30–60 s) technique among ICs of the brain. The correlation value at each window also quantifies the FC strength between a pair of brain networks within the time frame.

2.2. Definition 2: dFNC time course

A dFNC time series, T = w ₁, w ₂, w ₃, …, w _n is a sequential set of n real values, where w _i represents the correlation between a pair of ICs of the brain for a certain period. w stands for a sliding window of a certain size. There is one such TC for each pair of components (connections).

2.3. Definition 3: Connections/pair

A connection is a functional association between a pair of independent neural components. We use “connection” and “pair” interchangeably in our writing.

2.4. Definition 4: Subsequence

Given a time series T of length n, a subsequence S is a subset of length m ≤ n contiguous indices of T.

2.5. Definitions 5: Motif

Given a time series T of length n, a motif H _i,m is a subsequence of T with length m having the minimum average distance from ⁿ C _m  − 1 other m‐length subsequences of T. H _i,m is the most recurring shape in T of length m.

2.6. Definition 6: Statelets

Given a collection (constant/variable‐lengths) of time series motifs G and a positive real number d (size parameter), Statelets S is a subset of size d of state‐shape prototypes evaluated using the proposed summarization framework.

3. DATA COLLECTION AND PREPROCESSING

We used an existing SZ dataset for generating the dFNC TCs in this project (Keator et al., 2016). The data repository has resting‐state functional magnetic resonance imaging data collected from 163 HCs (117 males, 46 females; mean age 36.9) and 151 age‐ and gender‐matched patients with SZ (114 males, 37 females; mean age 37.8) during the eyes‐closed condition. Collected data pass‐through data quality control (explained in Allen et al., 2014; Damaraju et al., 2014). The participant's consent was obtained before scanning following the Internal Review Boards of affiliated institutions. Data were collected with a repetition time (TR) of 2 s on 3T scanners. Imaging data for six of the seven sites were collected on a 3T Siemens Tim Trio System and a 3rGeneral Electric Discovery MR750 scanner at one site. Resting‐state fMRI scans were acquired using a standard gradient‐echo echo‐planar imaging paradigm: Field of view of 220 × 220 mm (64 × 64 matrices), TR = 2 s, Echo time  = 30 ms, Fractional anisotropy  = 770, 162 volumes, 32 sequential ascending axial slices of 4 mm thickness and 1 mm skip. Subjects had their eyes closed during the resting state scan.

Data preprocessing, quality control, and dFNC approximation follow the standard pipeline described in (Rashid et al., 2014). The preprocessing follows a standard pipeline that has been adapted in several previous studies (Agcaoglu et al., 2020; Damaraju et al., 2014; Espinoza et al., 2019; Rashid et al., 2014, 2016). The preprocessing steps include image alignment (motion correction), slice timing correction, spatial normalization, despiking, and smoothing, which are commonly used for fMRI preprocessing. Depending on the MRI scanning protocols and collected data, the preprocessing parameters were selected and described in these studies and their supplementary (Allen et al., 2011, 2014; Damaraju et al., 2014). First, rigid body motion correction has been done using the INRIAlign toolbox in SPM to correct for subject head motion, followed by a slice‐timing correction to account for timing differences in slice acquisition (Freire & Mangin, 2001). Then, the scans went through a 3dDespike algorithm to regress out the outlier effect and warped to a Montreal Neurological Institute (MNI) template and resampled to 3 mm³ isotropic voxels. Instead of Gaussian smoothing, we smoothed the data to 6 mm full width at half maximum (FWHM) using the BlurToFWHM algorithm, which performs smoothing by a conservative finite difference approximation to the diffusion equation. Additionally, the voxel TC was variance normalized before performing the ICA (Hyvärinen & Oja, 2000), as this has shown better to decompose subcortical (SB) sources in addition to cortical networks. Group ICA (Calhoun et al., 2001) was performed on the preprocessed data and identified 100 ICNs. Subject‐specific spatial maps (SMs) and TCs were obtained using the spatiotemporal regression back reconstruction approach implemented in GIFT software (Calhoun et al., 2001). To ensure estimation stability, we repeated the ICA algorithm 20 times in ICASSO, and aggregate SMs were estimated as the modes of component clusters. Subject‐specific SMs and TCs were obtained using the spatiotemporal regression back reconstruction (Calhoun et al., 2001; Erhardt et al., 2011) implemented in GIFT. Following ICA, we obtained one sample t‐test map for each SM across all subjects and threshold these maps to obtain regions of peak activation clusters for each component; we also computed the mean power spectra of the corresponding TCs. These heuristics curated a group of components as ICNs if their peak activation clusters fell on gray matter and showed less overlap with known vascular, susceptibility, ventricular, and edge regions corresponding to head motion. We also ensured that the mean power spectra of the selected ICN TCs showed higher low‐frequency spectral power. This selection procedure resulted in 47 ICNs out of the 100 ICs obtained. The cluster stability/quality (I _q) index for these ICNs over 20 ICASSO runs was very high (I _q > 0.9) for all of the components, except an ICN that resembles a language network (I _q = 0.74). The subject‐specific TCs corresponding to the ICNs selected were detrended, orthogonalized with respect to estimated subject motion parameters, and then despiked. The despiking procedure involved detecting spikes as determined by AFNI's 3dDespike algorithm and replacing spikes by values obtained from the third‐order spline fit to neighboring clean portions of the data. The despiking process reduces the impact/bias of outliers on subsequent FNC measures. The 47 components are organized into modular partitions using the Louvain algorithm of the brain connectivity toolbox. We computed functional network connectivity (FNC), defined as pairwise correlation between ICN TCs. To compute the time varying FNC between the ICN TCs defined as pairwise correlation dFNC between two ICA TCs referred to dFNC was evaluated using a sliding window correlation approach with a window size of 22 TR (44 s) in steps of 1 TR (Allen et al., 2014; Calhoun et al., 2001). The window constituted a rectangular window of 22 time points convolved with Gaussian of sigma 3 TRs to obtain tapering along the edges (Allen et al., 2014). We compute the time‐varying FNC for each connection between a pair of nodes (ICNs) and generate a time series. We estimated covariance from the regularized inverse covariance matrix (Smith et al., 2011; Varoquaux et al., 2010) using the graphical LASSO framework (Friedman, Hastie, & Tibshirani, 2008). The regularization parameter was optimized for each subject by evaluating the log‐likelihood of the subject's unseen data in a cross‐validation framework. Consistent for all post‐hoc steps for extracting and validating. Subject‐wise dFNC values were Fisher‐Z transformed and residualized with respect to age, gender, and site. After computing dFNC values for each subject, covariance values were Fisher‐Z transformed and residualized with respect to age, gender, and site using the reduced model determined from our static FNC (sFNC) analysis. The mean dFNC matrix was computed over all subjects. This connectivity represents the association/relation between two parts of the brain and how this relation evolves with time. In other words, we can interpret this connection as an abstraction of communication between different brain hubs to process information. As such, both highly positive and negative correlations are informative to characterize the relations in two distinct directions. The hypothesis is the presence of these connections make the neural system process information and generate response in a healthy manner. Likewise, having disruption in those connections can cause symptoms present in several mental disorder like SZ. Our study aims to extract most dominant repetitive patterns of these connections by decomposing the time series in a granular scale. Then, summarize these patterns as a global representative set (Statelets) that helps leveraging this knowledge to measure different dynamic features of brain in post hoc study. More details about the components and the dFNC computation is available in this study and its supplementary (Damaraju et al., 2014).

4. EARTH MOVER DISTANCE

We implement the EMD for computing distances between motifs. EMD usually measures the dissimilarity between two probability distributions (Levina & Bickel, 2001; Vallender, 1974). Also, EMD is effectively adaptable for time series subsequence comparison. It corresponds to the minimum amount of work required to make two subsequences look‐alike (Champion, De Pascale, & Juutinen, 2008). For p and q, two given signatures, F(p, q) represents the set of all possible flow between p and q. Then, the work is defined as follows,

$$\text{Work}\left(F,p,q\right)=\sum _{I=1}^m\sum _{j=1}^n{f}_{\mathit{ij}}{d}_{\mathit{ij}}$$ (1)

here $d_{\mathit{ij}}$ is some measure of dissimilarity (e.g., Euclidean distance), $f_{\mathit{ij}}$ is the optimal flow that minimizes the cost between $p_i$ and $q_j$ (ith time points). The terms m and n denote the length of the signatures, respectively. The heuristic minimizes the amount of work done to compute the EMD between p and q given by the following equation (Andoni, Indyk, & Krauthgamer, 2008; Champion et al., 2008).

$$\mathrm{EMD}\left(\mathrm{p},\mathrm{q}\right)=\frac{\mathrm{mi}{\mathrm{n}}_{\mathrm{F}=\left({\mathrm{f}}_{\mathrm{ij}}\right)\in \mathrm{F}\left(\mathrm{p},\mathrm{q}\right)}\text{Work}\left(\mathrm{F},\mathrm{p},\mathrm{q}\right)}{\text{Total flow}}$$ (2)

The above equations provide the canonical formulation for computing EMD between signatures of any dimensions; however, we opt to compare shapes from one‐dimensional time series. Hence, we propose a more straightforward implementation in the next paragraph where we are not required to use an off‐the‐shelf distance measure.

4.1. Implementation in our study

We use a particular case of EMD for a one‐dimensional time series, which parses through the vectors and keeps track of how much flow occurs between consecutive time points. Here, flow is the difference between the amplitudes of two‐time series at a given time point. The method recursively accumulates the absolute work done at each time point, and finally, the summation over the time points corresponds to the EMD distance (see Equations 5 and 6). Equations 3 and 4 normalize shapes p and q, respectively.

$$A_i={\widehat{p}}_i–\min \left(p\right),\text{where}{\widehat{p}}_i=\frac{{\mathrm{p}}_i}{{\sum }_{j=1}^{\mathit{\text{length}}\left(p\right)}{p}_j}$$ (3)

$$B_i={\widehat{q}}_i–\min \left(q\right),\text{where}{\widehat{q}}_i=\frac{{\mathrm{q}}_i}{{\sum }_{j=1}^{\mathit{\text{length}}\left(q\right)}{q}_j}$$ (4)

$$\mathrm{EM}{\mathrm{D}}_{\mathrm{i}+1}=\mathrm{EM}{\mathrm{D}}_{\mathrm{i}}+\left({\mathrm{A}}_{\mathrm{i}}-{\mathrm{B}}_{\mathrm{i}}\right),\text{where}\mathrm{EMD}0=0$$ (5)

$$\text{Distance}\left(\mathrm{P},\mathrm{Q}\right)=\sum \left|\mathrm{EM}{\mathrm{D}}_{\mathrm{i}}\right|$$ (6)

5. OUR PROPOSED FRAMEWORK

Our architecture performs two fundamental steps to estimate state‐shape prototypes (statelets) from time‐series dataset, (A) motif discovery and (B) summarization. Figure 3 depicts the subprocesses required to perform these steps.

5.1. Step 1: Motif discovery

Each dFNC time series represents the FC between two distinct brain networks. Therefore, it is not feasible to impose a common motif length for all the time series in the dataset, like most other motif discovery methods. The proposed heuristic takes a range [R1 R2] and suggests the perfect length to explore the time series within that boundary. In our case, dFNC is collected using a window size of 22 time points. So, the lower bound for R1 > 22. The upper bound for R2 should be confined within half of the signal length for a sensible parcellation of dFNC. Since the study aims to observe transient signatures in the data, we select the range [30 50] in our analysis. Figure 1a demonstrates the step for performing motif discovery. At first, the process creates all the subsequences of different lengths and then passes these candidates through the following subroutines (Matlab scripts).

FIGURE 1.

Our proposed methodology for time series motifs discovery and summarization. (a) Step 1: Motif extraction using EMD as a similarity metric. The subroutine takes out the most repetitive pattern (possibly with multiple occurrences) of a given time course, (b) Step 2: Summarization of motifs using their probability density computed by a kernel density estimator (KDE). It takes a bag of varying length motifs and generates a concise smaller collection of the most frequent shapes/patterns representing the functional system (SZ/HC). We defined these prototypes as the statelets. The EMD distance matrix is used for both performing the tSNE and computing probability density (PD) of the motifs. The relevant processing blocks and their intuitions in Step 2 are described elaborately in Section 5.2

5.1.1. EMD layer

For a given candidate of length $l_i$, the algorithm searches through the timepoints by taking all other candidates of that length and computing the EMD between them. We can consider the selected candidate as a kernel here, and the idea is to apply that kernel on the input signal to compute distances.

5.1.2. Pooling local minima

The module pools a subset of top matches (minimum distance) instead of just one across the signal. Since motifs are repeating signatures, intuitively, we scrutinize the top matches only. It decomposes the input TC into several subsections and pools the minimum from each subsection.

5.1.3. Mean of LM's and global minimum

This layer computes the mean of those local minima, representing the overall performance by that specific candidate. After finishing computation for all the candidates of $l_i$, the method selects the global minimum, which corresponds to the best acquirable matching (BAM) score.

5.1.4. The best length selection

To extract most recurring transient patterns, a recommender unit optimizes two objectives using the BAM scores considering all different lengths (i) minimizing the motif's length and (ii) maximizing the similarity score. The recommender unit returns the best length for decomposing a time series.

5.1.5. Motif extraction

We have already collected information from the length selection process, that is, distance, occurrences required for motifs extraction. So, using the leading candidate as the benchmark, the subroutine collects similar nonoverlapping occurrences of local motifs across the TC and stores these motifs for creating global dominants across the subjects for a given pair.

This analysis compares the dFNC time‐series datasets across the two groups, HC and SZ. After discovering the motifs from all the dFNC time series, we have a large population of variable length patterns. Next, a group‐wise summarization is performed on these motifs.

5.2. Step 2: Summarization

The decomposition technique (motif discovery) transforms the FC into a diverse connectivity signatures manifold. Each pair of components (connection) discovers a group of shapes from all the subjects. Summarization aims to fetch the representatives of the motif's collection. The intuition is to find highly persistent patterns from all the distinctive subgroups of the population to ensure that statelets approximately capture all the diversities in the given stack of motifs. The persistence is measured from the shape's PD across the given trajectory, and the subgrouping is unveiled through the t‐distributed stochastic neighbor embedding (tSNE). Our summarization scheme weighs the tSNE points using the corresponding PD. Then, an off‐the‐shelf peak technique locates the expected prototypes with standard smoothing and proximity parameters. To run the subprocesses, the probabilistic summarization runs a module called “FindDominants”: a suite of Matlab and Python scripts consisting of four steps (a, b, c, and d) mentioned in Figure 1b to identify pairwise dominant motifs. We replicate the same analysis for both subject groups SZ and HC.

5.2.1. EMD matrix

We need the distance from each motif to all others to measure the density and to subgroup them using tSNE. Here, we use EMD again for computing the distance matrix. First, we normalize the TCs to make our distance invariant to scale and offset. Then, calculate cross EMD across the motifs, which creates a square matrix of n × n dimension where n is the number of shapes in the given collection.

5.2.2. Kernel density estimator

Our method applies KDE to calculate the PD of data instances (motifs) in the population. We incorporate a Gaussian kernel with optimal bandwidth (Sheather & Jones, 1991; Silverman, 1986) in KDE. In this setting, kernel density at a point x is given by Equation 7,

$$p\left(x\right)=\frac{1}{\mathrm{n}}\sum _{\mathrm{i}=1}^{\mathrm{n}}{\mathrm{K}}_{\mathrm{h}}\left(\mathrm{x}-{\mathrm{x}}_{\mathrm{i}}\right)$$ (7)

For ‘D’ dimensions, the formulation becomes,

$$p\left(x\right)=\frac{1}{\mathrm{n}}\sum _{\mathrm{i}=1}^{\mathrm{n}}\frac{1}{{\mathrm{h}}^{\mathrm{D}}}\mathrm{K}\left(\frac{\mathrm{x}-{\mathrm{x}}_{\mathrm{i}}}{\mathrm{h}}\right)$$ (8)

where n is the number of samples, $k_h$ is the kernel with a smoothing parameter, h is called the bandwidth. The kernel is a non‐negative function and h > 0. The common practice uses the Gaussian kernel for a smoother density model, simplifying the following kernel density model.

$$\mathrm{p}\left(\mathrm{x}\right)=\frac{1}{\mathrm{n}}\sum _{\mathrm{i}=1}^{\mathrm{n}}\frac{1}{{\left(2\mathrm{\pi }{\mathrm{h}}^2\right)}^{\frac{\mathrm{D}}{2}}}{\mathrm{e}}^{-\frac{{\left|\mathrm{x}-{\mathrm{x}}_{\mathrm{i}}\right|}^2}{2{\mathrm{h}}^2}}$$ (9)

here $h$ is the SD of the Gaussian components. Another crucial part of the process is to select an appropriate bandwidth (h) for the density, and several strategies for selecting $h$ (I. A. Ahmad & Amezziane, 2007; Turlach, 1993). Our approach incorporates the rule‐of‐thumb bandwidth estimator for Gaussian (Scott, 2012). The optimal choice for $h$ is following,

$$\mathrm{h}={\left(\frac{4{\widehat{\mathrm{\sigma }}}^5}{3\mathrm{n}}\right)}^{\frac{1}{5}}\approx 1.06\widehat{\mathrm{\sigma }}{\mathrm{n}}^{\frac{-1}{5}}$$ (10)

here $\widehat{\sigma }$ is the SD of the samples, and $n$ is the number of total samples. The term $\left(x-x_i\right)$ in the equation corresponds to the distance between the sample $x$ and $x_{i\in n}$. So, we assign EMD distances between the shapes computed in Section 5.2.1 to assign the value of the term $\left(x-x_i\right)$ in the equation. Using Equation (9), KDE generates PD for all motifs in the collection.

5.2.3. tSNE using EMD

tSNE is a powerful and flexible visualization tool for high‐dimensional data by giving each datapoint a location in a two‐ or three‐dimensional map (Gisbrecht, Schulz, & Hammer, 2015; Maaten & Hinton, 2008). Decentralized stochastic neighbor embedding (dSNE) separates the data subgroups using their distance metrics (Saha et al., 2017, 2019b). In our case, tSNE considers each motif as a data point and uses the EMD matrix (computed in Section 5.2.1) to select the neighbors for the embeddings. Next, the method weighs all the points using their corresponding PD. Figure 2 presents an example of how tSNE reveals intrinsic subgroups of data collection. The color indicates the PD. The brighter, the higher. The visualizations in the subsequent steps are generated using this same plot.

FIGURE 2.

An example of tSNE using EMD distance on a collection of real motifs. Data points represent the motifs weighted by their probability density computed using KDE. X and Y axis stand for the horizontal and vertical coordinates of each point, respectively

5.2.4. Two‐dimensional mapping

Although the above tSNE map unveils the subgroups, we still need to mechanistically locate the summary prototypes. To employ the peak finder, we need to map the tSNE plot (Figure 2) to a standard discrete two‐dimensional (2D) coordinate system, for example, a 2D grid. This subroutine assigns all tSNE points to a two‐dimensional discrete system and accumulates the PD of all closely neighboring shapes onto a single cell. The strive is to make dominant points more distinguishable by increasing the frequency in the corresponding vicinity. This approach discretizes the range of coordinates into a set of integer intervals. As a result, multiple points from the tSNE plot are stacked together into a single cell, as shown in Figure 3 to summarize the PD of a close neighborhood.

FIGURE 3.

Mapping tSNE points to a 2D matrix for accumulating PD's of close neighbors. Therefore, we observe the higher density data with a brighter color in the figure

5.2.5. Gaussian blur and peak finding

Defocusing noise helps identify significant features/data points representing the subgroup. We apply a Gaussian blur to the tSNE image to facilitate the estimation of the peaks. Figure 4 displays how the blurring effect changes the focus toward relevant loci of the image. Then, we use a two‐dimensional peak finder to compute the high‐density data points. The peak finder extracts a characteristic shape from each subgroup with a higher PD. The spikes in Figure 5 indicate the representative motifs (statelets) from the whole population. It visualizes one spike per high‐density subspace. It visualizes how the summarization scheme covers the whole state‐shape trajectory and approximates the synopsis. For more clarity, the method restricts the peak finding subroutine by calibrating the smoothing parameter to avoid creating a pile of similar patterns. The objective is to find a diversified set of shapes that reasonably approximates the overall dynamics.

FIGURE 4.

After applying a Gaussian blur on the 2D image to defocus less dense data points

FIGURE 5.

A three‐dimensional view of the tSNE plot after marking the peaks extracted by a 2D peak finder. The peak finder selects at least one peak from each high‐density region. Later, we use peak's tSNE coordinates to determine the real motifs it represents

6. EXPERIMENTAL RESULTS

We divide the dataset into 163 HC subjects and 151 SZ patients (SZ) groups. Each subject has ⁴⁷ C ₂ = 1,081 pairs of components or 1,081 connections: 1081 dFNC TCs. For each connection, we take TCs from all the subjects within that group (HC/SZ). Our method evaluates a subset of statelets from both groups of subjects (SZ/HC). The PD of these high dimensional state shapes denotes the consistency of these statelets across connections among multiple brain regions. In Figure 6, we show the most frequent statelets from both groups. We used the Jonker‐Volgenant algorithm (Cao, 2013) for a linear assignment problem that optimizes the mapping of tSNE coordinates to a two‐dimensional grid preserving the matching shapes in the same neighborhood constraint. In Figure 7, we compute the statelet's PD that denotes the consistency of these statelets across connections among multiple brain regions. Each bar indicates how frequently a connection's statelet has appeared in the group dynamics. The frequencies are also sorted based on their relative rank computed on their frequency in both patient and control groups. This serves to analyze the connection's readiness in the functional dynamics. We can see that the pair (connections) rank is very different in each group, suggesting a diverse influence of a connection driving the dynamics of patients versus controls. The observation indicates two distinct subsets of connection drive the group dynamics and evident a high‐level group difference in terms of neural components participating in the information processing. Further analysis is required to shed more light on these differences. Another observation is frequency of the statelets in HC connections is higher than in the SZ. To test the statistical significance of this difference between the two distributions, we compute a Kendall tau correlation coefficient, which is traditionally used to measure the ordinal association between two entities (Kendall, 1938; Puka, 2011). The hypothesis test evaluates $\tau$ = 0.045 with a p‐value = 0.025 indicating a statistically significant difference.

FIGURE 6.

A subset of dominant motifs from SZ and HC dynamics. The motifs are detrended and visualized using tSNE followed by the Jonker‐Volgenant algorithm for the linear assignment problems. A similar type of shape is embedded into the same neighborhood. We can see a few potential subgroups of motifs. The blank space in the figure was generated because we used a larger 2D grid than the number of motifs to display. So, the algorithm optimizes the location for each motif from the 2D coordinate system to assign the matching patterns in the nearby vicinity. The X axis is time, and the Y axis represents the dynamic functional connectivity strength

FIGURE 7.

Each connection's probability density (PD) demonstrates how frequently the statelets extracted from a connection appear in the group dynamic. Each connection has two density values: one per subject group. Blue demonstrates PD in controls (HC) and red in the patients (SZ). We sort the connections low to high according to their PD in the SZ group (left subplot), and HC sorts the right subplot. Y axis represents the order of the links after sorting, and X axis depicts their corresponding PD. We observe that the rank of connections differs in both subject's groups in terms of their PD; the PD difference is statistically significant and visually evident in the plot

6.1. Dynamic features analysis

We focus more on the dynamic features of the neural system, which have been primarily unexplored in previous studies of dFNC. The dynamic properties of these representative shapes show significant group differences between HC and SZ phenotypes.

6.1.1. Recurrence

To quantify the recurrence of statelets, we use passage coding by convolving the group‐wise most frequent statelet with each connectivity time series. Passage coding annotates a time series with the corresponding convolution score and visually perceivable the repetitions. Figure 8 presents the occurrences of the group statelets in individual dynamics (randomly selected three subjects from each group). We investigated the recurrence of statelets for all the subjects and got consistent results, as shown in Figure 8. Statelets are designed to be the most generalized envelopes of brain connectivity time series; thus, higher recurrence endorses better convergence to the standard form of connectivity. The agreement toward regular co‐fluctuations by the connection aids synchronization in the brain circuitry. In the SZ dynamic, the repetitions of statelets are significantly lower than HC. That leads to a weaker association between the brain components in SZ dynamics to perform smoother cognition. Also, fewer connections are present in the SZ dynamics than HC, which indicates the absence of necessary communication among the subunits of the brain to process brain signals. Overall, the recurrence helps understand the pattern of communication (e.g., message passing) between the modules of the brain. Apparently, a more connected brain graph with the edges characterized by a recurrent communication scheme is required to produce the necessary cognitive response. This recurrence analysis indicates the absence of the necessary repetition of communications between the submodules of the brain in the SZ dynamic. This provides a ground for further study to explore the reasoning behind these abrupt connectivity patterns. Also, more analyses are required to identify why these disruptions are being triggered.

FIGURE 8.

Each graph represents the occurrences of the most recurring statelets over each subject's time course. These are three randomly selected subjects from the HC (top) and SZ (bottom) groups. We convolve the group statelets with the subject dynamics (all the dFNC time courses, 1,081) to investigate the recurrence of the statelets over time. Then, we sorted the pairs based on the dominant shape's first occurrence in their time series. Consequently, the early the statelets appear in a pair's time course, the higher the pair/connection in that subject's dynamics—we threshold these convolutions matching scores at 0.8 for both groups to track down the strong appearances only. The color intensity corresponds to how strongly/weakly the shape appears in that part of the course. The color bar is identical for all the reference subjects in the figure

6.1.2. Modularity

The line graphs in Figure 11 illustrate the ensemble behavior of the connections in both dynamics (SZ/HC). The value represents the number of connections with similar prototypes at each time point. Throughout the scan session, comparatively more HC connections are activated in identical time steps than patients. This characterizes the higher modularity in the HC functional network/graph (the connections between the functional components of the brain). Modularity is imperative because it quantifies the strength of the network's division into modules. Each module consists of a subset of nodes, independent brain components in our case. These modules help data transmission by performing specific tasks independently. Results show individuals with higher baseline modularity exhibit more significant improvement with cognitive training, suggesting that a more modular baseline network state may contribute to greater adaptation in response to cognitive training (Arnemann et al., 2015). Also, cognitive ability is influenced by the relative extent of integration and segregation of functional networks (i.e., modules) distributed across distant brain regions (Chen, Abrams, & D'Esposito, 2006; Stevens et al., 2012). Subsequently, the difference in modularity is informative about the cognitive symptoms of SZ. In addition, the graph shows SZ connections become modularized at the end of the time series, whereas the HC ensembles are reduced by their size (Figure 9). This ensemble activation also demonstrates a distinct synchronization pattern between SZ and HC brains and suggests an interchanging control by various neural components.

FIGURE 9.

Based on probability density ranking (in Figure 7), we computed a collective appearance of the connections across all the subjects. The X axis shows the time steps, and the Y axis corresponds to the number of pairs connections that show the first statelet at that step, which indicates the activation of the pair

6.1.3. Consistency graphs

To measure the statelets' temporal consistency in the connectivity time series, we introduced a metric called TD. The primary motivation behind introducing TD is to approximate the temporal consistency of the statelets. Also, measuring how frequently these generalized dominant patterns appear in the timeline. The metric indicates how temporally localized or widespread the statelets are. TD is defined as follows.

TD: a temporal consistency measure—The TD approximates how consistently a pair's connectivity exhibits the dominant motif in the TC. The metric captures the temporal information of the summary shapes and integrates it across a group dynamic. The following equation describes TD.

$$\mathrm{TD}=\frac{1}{\left|\mathrm{S}\right|}\sum _{\mathrm{i}=1}^{\left|\mathrm{S}\right|}\frac{1}{{\mathrm{T}}_{\mathrm{i}}}$$ (11)

Here, |S| = The total number of subjects in the group SZ/HC and T _i the index at which the pair has appeared for the first time in a subject (T _i  = 1 to 136). If the pair is not present in a subject's dynamic, then T _i = $\infty$ thus, TD for that pair in a subject is zero. We calculate the TD for each connection in a subject's dynamic and lately compute the average across the subjects within a group (SZ/HC). We used the meantime decay computed on the group dynamic for the group representation. We consider the appearance of a connection in a subject's dynamic based on the first occurrence of group statelet on that time series. The occurrence is calculated based on a thresholded convolution score. We also sorted the connection based on their first initialization in the subject's dFNC. The intuition was to investigate which connections are responsible for triggering the convergence toward statelets and how it propagates the dominance across the TC. The expected time point at which the connection is initiated in the dynamic is inverted. So, if the TD is very localized, then the mean is very close to that time point and has a higher value. In essence, it quantifies the temporal uniformity of the statelets in a subject's dynamic. Since statelets are the most repeated, stable, and representative subsequences of the time series, this attribute can potentially explain the connection's affinity toward a more generic connectivity state contributing to the dynamics. Also, TD estimates how quickly/lately the connectivity rectifies their initial disparity and converges to routine cognition. It characterizes a connection as consistently late (low TD value), consistently early (high TD value), pseudo‐random positioning (medium TD value, e.g., 0.10 to 0.15), consistent but sparse in the group dynamics (0.5–0.9), etc.

Figure 10 demonstrates the average TD graphs computed on both dynamics. Here, the vertices are independent brain networks, and the edge represents the connection between two functional networks. The weight of the edge stands for the TD; the darker, the higher. As we can see, HC functional networks are firmly connected, and most of the connections (out of 1,081) pass the TD benchmark. On the other hand, only a small subset of SZ connections converges to their group statelets. This provides evidence that the consistency of SZ network connections is considerably lower than HC. The functional connections between different brain components show more substantial temporal consistency in HC. SZ statelets are more dispersed along the timeline, consequently lacking the expected connectivity signature in the active channels. It might impact overall information processing due to a deficiency in temporally sensitive message passing in the functional network. To further probe the interconnections, we computed the transitivity of each subject's TD graph. Figure 11 shows the histograms of transitivity of subject‐wise TD graphs in both groups. Transitivity symbolizes the global probability of the network (connectivity graph) to have adjacent nodes interconnected, revealing the existence of tightly connected communities, e.g., clusters and subgroups. Most of the SZ graphs have zero or very low transitivity compared to HCs. We compute a two‐sample t test on the transitivity scores from both groups, and the p‐value, p = 9.23e‐88, indicates the difference is highly statistically significant. Higher transitivity suggests that modules in the network consistent with prior work showing increased modularity are linked to improved cognitive performance (Arnemann et al., 2015). The analysis indicates that HC networks intercommunicate more consistently and keep the channels up than SZ. This result is consistent with previous work (Allen et al., 2014; Karlsgodt, Sun, & Cannon, 2010; Rashid et al., 2014), which suggested that patients exhibit more erratic and less efficient communication among brain regions than controls. Figure 12 shows the average TD of each connection in both SZ and HC groups. The rightmost map shows HC‐SZ group differences. All the pairs in HC show a more substantial TD than SZ, especially in visual, SM, and default mode (DM) regions. Notably, HC subjects show higher TD than patients with SZ, and the apparent group differences present almost all over the brain. That also makes the data more useful for classification. Intuitively, we feed subject‐wise TD data to multiple classifiers and get good accuracy in every case. To avoid bias, the statelets are extracted from the whole dataset without using group (SZ/HC) information (unsupervised). Thus, the classification accuracies are reasonably comparable. For clarity, we can consider it a dimensionality reduction method like principal component analysis (PCA) (Wold, Esbensen, & Geladi, 1987) for better features selection. We compare the performance between running the models directly on the dFNC matrix and methods like PCA (Wold et al., 1987) for better features selection. We compare the performance between running the models directly on the dFNC matrix and the subject‐wise TD. We use multiple classifiers to test TD and observe significant accuracy improvement for more robustness. Figure 13 shows how the models perform in both cases. We can see a simple SVM or logistic regression model using TD outperforms all other methods, including LSTM with attention.

FIGURE 10.

Time decay graphs from both groups. The nodes are functional networks, edges correspond to the connection between them (maximum 1,081 possible), and the weight represents the mean time decay (TD) of a connection within a group dynamic. We compute color scaling from the 95 percentiles of the total values. After thresholding at average group mean (SZ group mean + HC group mean)/2, 1,061 edges survive in the HC group and only 16 edges in the SZ group

FIGURE 11.

Histogram of transitivity from subject‐wise time decay graphs. It refers to the extent to which the relation between two nodes in a network connected by an edge is transitive. A significant portion of SZ subjects shows 0 transitivity, which means the connections are less consistent across different subjects. We show the differences are statistically significant using a two‐sample t‐test on both distributions. Transitivity is also related to the clustering coefficient

FIGURE 12.

Pairwise mean time decays in healthy control (HC) and schizophrenia (SZ) groups. We run a two sample t test on the pairwise time decay values to check the statistical significance of their HC‐SZ group differences. The rightmost subfigure represents the FDR corrected t values

FIGURE 13.

Classification accuracy for different methods. First, three methods were applied to the dFNC matrix and LSTM with the attention model applied to the dFNC time course. The last two methods use time decay (TD) for classification. We run the models on time decay information of all the subjects for 100 repeated iterations, and the accuracies are mean across the iterations. We train the model on 200 random samples in each iteration and cross‐validate them using the remaining 114 subjects

dFNC states from statelets—In this cross‐group analysis, we focus on obtaining statelets across the dataset without separating SZ and HC subjects into two different bins. The major objectives can be articulated (i) extracting cross‐group patterns, (ii) subgrouping the subjects based on statelets, (iii) experimenting with the statelet's connectivity strength and evaluating domain‐wise group differences. It is identical to the group‐wise analysis motif extraction followed by probabilistic summarization. The key difference is we send all the time series (from both subject's group HC/SZ) together through our statelet framework. Then, using approximated statelets as the representative, we partition the collection of shapes into several subgroups (connectivity states). After that, we use the EMD of extracted motifs to determine their association toward a specific subgroup/state—assign motifs to the nearest (lowest EMD) state. Statelets characterize the intrinsic cluster in the motifs extracted from dFNC TCs. The centroids of these clusters are referred to as brain states. We represent these connectivity patterns extracted from the dFNC TC as dFNC states. The centroids of these states represent patterns of FNC (temporal coherence) that individuals move between throughout the course of the experiment. We evaluate the HC‐SZ group differences in connectivity strength across the subjects within a state. For measuring the statistical significance of these differences, we use a two‐sample t test. The t‐values provide significant group differences in connectivity strength throughout the distinct regions of the brain (Figure 14). HC subjects in state 1 show stronger connections than SZ, and differences are statistically significant in SM, CC, and DM regions. In states 2 and 3, the differences are mostly HC < SZ; in state 3, group differences are statistically significant across the brain but similar strength in the connections between SB and other domains. State 4 shows mostly weaker differences except for a few auditory (AUD) and SB components, demonstrating SZ subjects are more strongly connected than HC in those regions. Two connections between AUD and DM show a statistically significant difference in state 5, and state 6 exhibits HC > SZ in the CC region. Above all, states 7, 8, and 9 show significant connectivity differences in CC and DM regions. Specifically, state 7 depicts HC > SZ and 8 and 9 shows HC < SZ group distinction. In an earlier study (Rashid et al., 2014), strong group differences are mostly in visual and AUD, but we got differences in SM, CC, and CB. We got both HC < SZ and SZ < HC in SM, CC, and DM. We observe the most significant pattern of group differences here in CC, DM, and CB regions. This triangle shows the variation in connectivity between patients and controls across multiple states with an alternating directionality, which indicates a potential biomarker for specific subtypes of SZ.

FIGURE 14.

HC—SZ group differences in terms of max connectivity strength. State‐wise group differences in functional connectivity (FC). We have both SZ and HC subjects' groups at each state. For each pair of components, we have a subset of statelets from HC subjects and a subset from SZ subjects. Then, we compute the maximum connectivity strength of those statelets from both subgroups. A two‐sample t test using a null hypothesis of “No group difference” compares patients' max connectivity versus controls. A higher t value indicates the rejection of the null hypothesis irrespective of their sign. However, the sign of t values represents the directionality of the group difference. The pair matrix (47 × 47) is labeled into seven different domains subcortical (SB), auditory (AUD), visual (VIS), sensorimotor (SM), cognitive control (CC), defaultmode (DM), cerebellar (CB), respectively. White cells in the matrix indicate either the absence of that pair or nonsignificant group differences for this pair within a state. The upper triangle represents the FDR corrected differences

7. CONCLUSION

We proposed a novel method for analyzing dynamic FC via extracting high‐frequency texture from the connectivity space. To our best knowledge, it is the first pattern mining application on dFNC data. The proposed framework addresses issues in current dFNC analysis methods by modeling the dynamics through brief connectivity shapes. The analysis of those motifs enables measuring the characteristics of brain circuitry and network organization. The major contributions of our study are two‐folded, an unsupervised method for motifs discovery using EMD as a distance metric and a probabilistic summarization of these patterns. Because of the complexity of the data, it was necessary to summarize the information into a predictable and concise subspace. To do this, we included a representative snippet of the human brain connectivity graph, using the set of components as reduced space to show how it behaves over time. The experiments facilitate the observation of distinguishing connectivity signatures and the interplay among the hubs to process information. Statelets provide this summary that includes abstraction, analytics, and current trends at a glance. The statelet approach seeks an improved understanding of connectivity states and the mechanism through which their dynamics vary across individuals. Results demonstrate how these state movies help to investigate the dynamic properties of an inherently dynamic system (i.e., brain). Our approach is more robust to noise while observing short‐length dynamic features of the functional dynamics of the brain. These connectivity shapes from SZ and HC dynamics help create a global contrast between healthy and diseased brains and illustrate crucial group differences in several dynamic properties such as recurrence, modularity, and synchronizability. Our method has some limitations and potential future improvements too. That includes high time complexity and parameters tuning. We are working on a dynamic programming implementation of motif extractions to reduce the time complexity to a polynomial order—also, a framework for automatic fine‐tuning of the model's free parameters. However, statelets provide shape‐based analytics and outlines of the dynamics for the first time. It enriches our understanding of how networks communicate in spontaneous brain activity and the pattern of binding/impairment between them to maximize information transfer and minimize connection costs. We believe this would fill in a gap in the field that will help us understand the dynamics better, possibly providing an improved way to study neuropsychiatric disorders more effectively.

CONFLICT OF INTERESTS

There is no conflict of interest.

AUTHOR CONTRIBUTIONS

Vince D. Calhoun and Md Abdur Rahaman proposed the idea of statelets. Md Abdur Rahaman and Sergey M. Plis developed the methodology. Md Abdur Rahaman ran the experiments and drafted the manuscript. Sergey M. Plis helped to design a few experiments and edited the manuscript. Vince D. Calhoun supervised the whole project, thoroughly edited the article, and gave valuable feedback on the results. Eswar Damaraju provided the data and performed the preprocessing. Eswar Damaraju and Debbrata K. Saha helped run the experiments and significantly contributed to the manuscript. All authors have approved the final version of the submission.

Rahaman, M. A. , Damaraju, E. , Saha, D. K. , Plis, S. M. , & Calhoun, V. D. (2022). Statelets: Capturing recurrent transient variations in dynamic functional network connectivity. Human Brain Mapping, 43(8), 2503–2518. 10.1002/hbm.25799

DATA AVAILABILITY STATEMENT

The dataset used in this study was collected ten years ago and included no data‐sharing agreement. Based on the IRB data privacy agreement, we are not allowed to share any subject‐specific data. However, the resting‐state fMRI data can be requested from the following link (cf. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4651841/). The pre‐processing pipelines are public, as referred to in the manuscript. All the pre‐processing tools are available online at http://trendscenter.org/software and the statelet approach will also be released within the dynamic functional network connectivity toolbox within the GIFT software package.