Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2021 May 1.
Published in final edited form as: J Neurosci Methods. 2020 Feb 25;337:108651. doi: 10.1016/j.jneumeth.2020.108651

Determining the Number of States in Dynamic Functional Connectivity Using Cluster Validity Indexes

Victor M Vergara 1, Mustafa Salman 1,3, Anees Abrol 1, Flor A Espinoza 1,2, Vince D Calhoun 1,3
PMCID: PMC7125031  NIHMSID: NIHMS1575458  PMID: 32109439

Abstract

Background

Clustering analysis is employed in brain dynamic functional connectivity (dFC) to cluster the data into a set of dynamic states. These states correspond to different patterns of functional connectivity that iterate through time. Although several clustering validity index (CVI) methods to determine the best clustering partition exists, the appropriateness of methods to apply in the case of dynamic connectivity analysis has not been determined.

New Method

Currently employed indexes do not provide a crisp answer on what is the best number of clusters. In addition, there is a lack of CVI testing in the context of dFC data. This work tests a comprehensive set of twenty four cluster validity indexes applied to addiction data and suggest the best ones for clustering dynamic functional connectivity.

Results

Out of the twenty four considered CVIs, Davies-Bouldin and Ray-Turi were the most suitable methods to find the number of clusters in both simulation and real data. The solution for these two CVIs is to find a local minimum critical point, which can be automated using computational algorithms.

Comparison with Existing Methods

Elbow-Criterion, Silhouette and GAP-Statistic methods have been widely used in dFC studies. These methods are included among the tested CVIs where the performances of all twenty four CVIs are compared.

Conclusions

Davies-Bouldin and Ray-Turi CVIs showed better performance among a group of twenty four CVIs in determining the number of clusters to use in dFC analysis.

Keywords: Magnetic resonance imaging, dynamic functional network connectivity, clustering analysis, clustering validity index, k-means clustering, Elbow-Criterion, GAP-Statistics, Davies-Bouldin, Calinski-Harabasz, Silhouette, Wemmert-Gancarski, Ray-Turi, Ratkowsky-Lance, SD Dis, Dunn, Point-Biserial, McClain-Rao, Xie-Beni, PBM, Ball-Hall, Banfeld-Raftery, Ksq DetW, Log Det Ratio, Log SS Ratio, Scott-Symons, SD Scat, Trace W, Trace Wib and Det Ratio

Graphical abstract

graphic file with name nihms-1575458-f0001.jpg

Introduction

Functional magnetic resonance imaging (fMRI) is a non-invasive method widely used to acquire temporal information of neuronal activity through the brain. Techniques that can extract information from fMRI are promising for developing biomarkers that may be useful in the clinical setting (Fox and Greicius, 2010). It has been found that brain areas tend to activate with a certain degree of temporal synchronicity, providing a mechanism for the coordinated functioning of the brain (Fries, 2015; Singer, 1999). A pair of brain areas exhibiting synchronous co-activation is said to be functionally connected. Abnormal functional connectivity has been observed in many neurodegenerative diseases revealing the importance of this feature to understand the underpinnings of brain dysfunctions (Agosta et al., 2017; Espinoza et al., 2018). In addition to the temporal variation of localized activity, brain dynamicity includes time changes of functional connectivity, a feature that has been termed dynamic functional connectivity (dFC) or in the case of dynamics among brain networks/sources instead of among fixed seeds this is called dynamic functional network connectivity (dFNC). Research has shown different patterns of whole-brain dFC over time during a given brain scan. A highly replicable set of these patterns has been observed through diverse sets of patient cohorts in the literature (Abrol et al., 2017; Calhoun et al., 2014). Each of these connectivity patterns is linked to a specific brain state representing a particular brain functional connectivity pattern. Our group has consistently observed a minimum of four whole-brain states (Damaraju et al., 2014; Vergara et al., 2018b), but this number can be dependent on the study, for example Allen et al. found seven states in their analysis (Allen et al., 2014). An important problem is to find the optimal number of states because the final analysis outcome is impacted by this estimation. Further research for the estimation of the number of dFC states is thus important for the research community.

One of the simplest and efficient ways of estimating the dFC state at a particular moment in time is to parcel the brain in relevant regions of interest (ROIs) and estimate the set of short-time correlations characterized by all possible pairs of ROIs. The shortness of the time interval used is a topic of debate in the literature (Zalesky and Breakspear, 2015; Zalesky et al., 2014), but practical intervals range from 40 sec to 100 sec (Leonardi and Van De Ville, 2015; Sakoğlu et al., 2010; Vergara et al., 2019a). The observed dFC pattern is thought to represent an instantaneous brain connectivity state. The time evolution of the states can be extracted by estimating dFC patterns at different time points. Comparing similarities and differences of the states as they evolve through time is not an easy task. Usually, an unsupervised clustering technique, such as k-means clustering, is employed to identify dFC states and their sequence through time. One central problem is the estimation of the optimal number of states commonly parameterized by the variable k. In dFC-fMRI analysis, the most used method is to run k for a range of cluster numbers, estimate a cluster validity index (CVI) for each k to numerically validate each partition, then employ a selection criterion for k which in most cases involves finding a critical point in the plot CVI vs. k. One of the earliest CVI in dFC is the widely use Elbow-Criterion (Abrol et al., 2017; Allen et al., 2014; Damaraju et al., 2014; Espinoza et al., 2019; Kodinariya and Makwana, 2013) that requires finding the elbow point where increasing k does not bring a considerable change in the CVI. By visually assessing the Elbow-Criterion plot, we hope to balance the trade-off between compactness and separation of the clusters. The result relies on the expertise of the user. This approach has been criticized because of its substantial subjectivism as cluster indexes are intended as a mathematically objective tool (Arbelaitz et al., 2013). Another two CVIs used in the field are Silhouette (Rousseeuw, 1987) and GAP-Statistics (Tibshirani et al., 2001) implemented in the fMRI processing software Group ICA of fMRI Toolbox (GIFT) (Rachakonda et al., 2007). The optimal number of clusters for these last two methods is indicated by the maximum CVI value in the range of k. A first dilemma occurs when the available CVIs do not agree in the number of clusters especially if the discrepancy is large. In this context it would be useful to have evidence of the performance of each CVI before adopting a final decision. A comparison of CVI performance applied to dFC data is missing in the fMRI literature.

This work aims at assessing alternative clustering validity index techniques to be used as regular dFC processing. The first stance is the large availability of techniques looking into finding an optimal number of clusters. However, none of these techniques has proven effective for all kinds of data (Pal and Biswas, 1997). Comparisons among available techniques have been performed in the literature (Brun et al., 2007; Guyon et al., 2009; Halkidi et al., 2001). Although some work explored a large variety of techniques (Arbelaitz et al., 2013), we decided to focus on widely used software implementations. Thus, we evaluated 24 different methods as applied to simulations with different noise levels and real dFC data in an addiction sample. Final recommendations are presented for future work.

Material and Methods

Clustering Validity Preamble

The dFC data consists of temporal sequences of connectivity values. Consider the two connectivity sequences of Figure 1. In this problem with two dFC sequences X and Y, an X-Y Plot is created where time is ignored such that a pattern of clusters emerges. Figure 1a presents an example where four clusters can be visually identified from the X-Y Plot and labeling each X-Y time point is not a difficult problem. This example was built using four separate 2D Gaussians with the same variance but different X-Y means. Figure 1b exemplifies a more difficult task where the variance is high, and the X-Y Plot resembles a single cluster. In this case it is difficult to think that the ground truth is not one single cluster. Labeling the imposed ground truth reveals the four clusters, otherwise obscured by noise. We are looking for CVI techniques to identify the optimal number of clusters even in cases where noise increases the difficulty of performing this task. There are many CVIs proposed in the literature with performances that depend on the data set (Arbelaitz et al., 2013; Halkidi et al., 2001). We chose to test the set of CVIs included in the software package R (version 3.5.1) given it is widely used and easily accessible (https://www.r-project.org/). The included CVIs are Ball-Hall (Ball and Hall, 1965), Banfeld-Raftery (Banfield and Raftery, 1993), Calinski-Harabasz (Calinski and Harabasz, 1974), Davies-Bouldin (Davies and Bouldin, 1979), Det_Ratio (Scott and Symons, 1971), Dunn (Dunn, 1974), Ksq_DetW (Marriott, 1971), Log_Det_Ratio (Scott and Symons, 1971), Log_SS_Ratio (Hartigan, 1975), McClain-Rao (McClain and Rao, 1975), PBM (Pakhira et al., 2004), Point-Biserial (Milligan, 1981), Ray-Turi (Ray and Turi, 1999), Ratkowsky-Lance (Ratkowsky and Lance, 1978), Scott-Symons (Scott and Symons, 1971), SD_Scat (Halkidi et al., 2001), SD_Dis (Halkidi et al., 2001), Silhouette (Rousseeuw, 1987), Trace_W (Edwards and Cavalli-Sforza, 1965), Trace_WiB (Friedman and Rubin, 1967), Wemmert-Gancarski (Desgraupes, 2013) and Xie-Beni (Xie and Beni, 1991). Mathematical descriptions of all these indexes can be found in the documentation for the R package written by Bernard Desgraupes (Desgraupes, 2013). We also included the Elbow-Criterion, not included in R, given its background and usage in the dFC literature. The Elbow-Criterion is a method used in fMRI functional connectivity (Damaraju et al., 2014; Espinoza et al., 2019; Vergara et al., 2018a) defined as the ratio of within cluster to between cluster distances (Allen et al., 2014). Since the GIFT software (Rachakonda et al., 2007) includes Silhouette (already in the R software) and GAP-Statistics, we added the GAP-Statistics to the list of revisited methods for a total of twenty four CVIs.

Figure 1.

Figure 1.

Open in a new tab

A two-dimensional example of clustering using two dFC sequences. In panel a) the clusters are easy to identify even if the labels are not indicated. In panel b) cluster separation is not easy to identify from the X-Y Plot.

The task at hand is to use a CVI method to assess the validity of a clustering membership. Figure 1b displays the correct clustering membership only because it was known a priori, otherwise, it would have been very difficult to decide that four clusters is the ground truth of this simulation. A practical CVI should provide numeric evidence about the optimal number of clusters for a particular data set. In Figure 2, we have applied three cluster validation methods (Davies-Bouldin, Calinski-Harabasz and Elbow-Criterion) to the data from Figure 1b to have an idea of what to expect. Clustering membership was determined by running a k-means algorithm for k values (k is the number of clusters requested from k-means) between 2 and 6. Davies-Bouldin is one in a series CVIs where the minimum value indicates the optimal number of clusters. Calinski-Harabasz belongs to another set of CVIs where the maximum value points to the optimal number of clusters. In the case of the Elbow-Criterion, the optimal number of clusters is obtained by the elbow point where increasing k does not produce a significant change. In each instance, the clustering algorithms successfully indicated four clusters.

Figure 2.

Figure 2.

Open in a new tab

This figure shows Davies-Bouldin (DB), Calinski-Harabasz (CH) and Elbow-Criterion applied to the data from Figure 1b. All methods are able to find the true number of clusters. The DB index indicates the number of clusters by the minimum value. The maximum CH value indicates the optimal partition. The Elbow-Criterion requires finding the “elbow” point where increasing the number of clusters do not produce considerable improvement.

Simulations

Two simulations were performed. The first simulation consisted of repeating the example of Figure 1, but with an increasing noise variance. The signal variance, in this case, was calculated by the variance of four centroids taken as the ground truth. The four centroids were located at the corners of an imaginary square, as seen in Figure 3. A total of 1000 points were generated and assigned to one of the centroids resulting in four groups of 250 individuals for 4 clusters. Noise was added such that each simulated point corresponded to its centroid location plus a random deviation. The noise followed a circular (covariance equal to zero) 2D Gaussian distribution with variable variance depending on the simulated signal to noise ratio. Four different signal-to-noise ratio (SNR) values were simulated −10, −3, 3 and 10 dB. Five different k-means were performed with k from 2 to 6. All CVIs were tested in their ability to find the ground truth imposed of k = 4 clusters.

Figure 3.

Figure 3.

Open in a new tab

Simulation results for four clusters and different noise levels. This figure employs the four clusters from the example in Figure 1. Only 10 CVIs successfully indicated four clusters on at least one of the SNR levels. CVIs are displayed in order of success with the PBM being the most successful.

The next simulation is motivated by the possibility that certain patterns in the data might affect the estimation of the optimal number of clusters made by the CVIs. In this case, we have added four extra clusters located at the corners of a second imaginary square, but the position of these squares is clearly separated. The pattern of centroids is displayed in Figure 4, where two imaginary squares can be recognized. We use k-means with k from 2 to 10 given that ground truth was eight clusters. Noise strength and SNR adjustments followed the same procedure as in the previous simulation.

Figure 4.

Figure 4.

Open in a new tab

Simulation using eight cluster centers and a SNR = 10 dB. The position of the eight cluster centers are designed to confuse the CVIs into detecting only two clusters. In panel a) the eight centroids and the different k-means outcomes are displayed. The Elbow-Criterion is unable to detect the ground truth, but the result is shown for reference. In b), we present those CVIs that indicated either k=2 as the optimal partition or indicated k=8 as a local minimum for a second-best optimal. In c) we display examples using the first three CVIs where it is clear that the ground truth is indicated by the local minimum at k=8. Note that optimal points are minimum for Davies-Bouldin and maximum for Calinski-Harabasz and Silhouette.

Real Data 1

The sample cohort consists of 986 subjects from several addiction studies. Subsets of this cohort have been previously employed in addiction research (Vergara et al., 2017a; Vergara et al., 2018c). Before scanning, subjects were excluded if reported injury to the brain, brain-related medical problems, and bipolar or psychotic disorders existed. Since we only use this data to explore clustering results, addiction status and other demographic data were deemed unimportant for our current aims.

In this paragraph we provide a summary of the data collection protocol while a full description can be found in the dynamic connectivity report (Vergara et al., 2018c). All images were collected on a 3 Tesla Siemens Trio scanner. Each participant completed a 5-minute resting state run using a single-shot, gradient-echo echo-planar pulse sequence (TR = 2000 ms; TE = 29 ms; flip angle = 75°; FOV = 240 mm; matrix size = 64 × 64). Foam padding and paper tape were used to restrict motion within the scanner. Thirty-three contiguous, axial 4.55-mm thick slices were selected to provide whole-brain coverage (voxel size: 3.75 × 3.75 × 4.55 mm) during the resting state scan. The first five images were eliminated to account for T1 equilibrium effects; 145 images were selected for further analysis. Presentation software (Neurobehavioral Systems) was used for stimulus presentation and synchronization of stimuli with the MRI scanners. Subjects were instructed to passively stare at a foveally presented fixation cross (visual angle = 1.02°) for approximately 5 minutes and to keep head movement to a minimum.

Preprocessing and other analyses were similar to our previous publication (Vergara et al., 2017b). Data were pre-processed using statistical parametric mapping (Friston, 2003) (SPM: http://www.fil.ion.ucl.ac.uk/spm) including slice-timing correction, realignment, co-registration, and spatial normalization and then transformed to the Montreal Neurological Institute standard space. For despiking, we utilized the command 3dDespike from the software Analysis of Functional NeuroImages (AFNI). The time courses were orthogonalized with respect to the following: i) linear, quadratic, and cubic trends; ii) the six realignment parameters; and iii) realignment parameters derivatives. A full width at half maximum Gaussian kernel of 6 mm was then used for smoothing. Data from all subjects were subject to a gICA decomposition (Calhoun et al., 2001; Calhoun and Adali, 2012) using the GIFT software (GIFT: http://trendscenter.org/software/gift/) to obtain a set of functionally independent components. The number of components was determined to be 100 using a modified version of ICASSO (Himberg et al., 2004; Ma et al., 2011). Artifactual components were previously detected and discarded as described in (Vergara et al., 2018a). A total of 39 resting state networks (RSNs) were used in this work corresponding to the retained gICA components. Spatial maps and MNI coordinates for the retained RSNs are displayed in Supplementary Figure 1.

Real Data 2

A total number of 7000 resting state fMRI scans were included for this dataset. The scans were collected from several studies performed in New Mexico and Colorado. This dataset has been previously used to study functional connectivity replicability (Abrol et al., 2017). Informed consent was received from all individuals (aged between 13 and 75 years), as per the institutional guidelines practiced at the University of New Mexico (UNM) and the University of Colorado Boulder (UC, Boulder). After anonymizing the data, the only information available was age and gender which does not play a role in the analysis pursued in this work.

Scans were acquired using 3-T Siemens TIM Trio MRI scanners with 12 channel radio frequency coils at the Mind Research Network (MRN) in association with UNM, or using the same hardware scanner at UC, Boulder. Both scanners used the exact same acquisition parameters (except for the repetition time) for most of the subjects. T2*-weighted functional images were acquired using a gradient-echo EPI sequence with TE = 29 ms, TR = 2s (6992 scans) or 1.3s (8 scans), flip angle = 75°, slice thickness = 3.5 mm, slice gap = 1.05 mm, field of view = 240 mm, matrix size = 64 × 64, voxel size = 3.75 mm × 3.75 mm × 4.55 mm. The sampling rates of the scans were matched to 2s before dynamic functional connectivity analysis was performed. The scans had variable length. The minimum scan length was 150 TRs. For this reason, we only included the first 150 time-points of all scans and discarded the rest. The first three images in the scans were eliminated to avert T1 equilibration effects. The following preprocessing steps are: realignment using INRIalign, timing correction of slices with the middle slice fixed as reference, spatial normalization of data into the Montreal Neurological Institute (MNI) space, re-slicing of data into cubic voxels of side 3 mm, and data smoothing using a Gaussian Kernel with the full-width at half-maximum (FWHM) set to 10 mm.

To study replicability, we created 28 groups of 250 scans each. Each group of 250 was independently analyzed using a gICA with 130 components at the group level and 100 at the single scan level. Component reliability was assessed using ICASSO (Himberg et al., 2004; Ma et al., 2011). The set of components were somewhat different for each of the 28 groups. RSNs were selected using the same criteria as that using in Dataset 1. After selecting RSNs of interest, the 37 components with highest correlation values or more specifically above the first quartile correlation threshold value of 0.65 and global correlation threshold value of 0.4 for all sample decompositions were retained for further dynamic FNC analysis.

Dynamic FNC

The dFNC analyses were performed using the dynamic FNC Toolbox (dFNC v1.0a) available in the GIFT package. A fifth-order Butterworth band-pass filter [0.01 0.15] Hz was applied to the time courses extracted from gICA. We explored two versions of the sliding window approach:

  1. Sliding Window Correlation (SWC): This approach is the nominal method employed to estimate how correlation changes with time (Allen et al., 2014). This method has been criticized as it suffers from spurious fluctuations in detriment of the correlation estimate (Leonardi and Van De Ville, 2015).

  2. Average Sliding Window Correlation (ASWC): This approach consists on applying a moving average filter to the SWC estimation to reduce the spurious fluctuations and improve the estimation of correlation (Vergara et al., 2019a). This method provides a cleaner signal which may be important in resolving clusters by the clustering algorithm.

Since the main problem in SWC and ASWC is the selection of an optimal configuration of window lengths, we repeated all dFNC analysis with window lengths 20, 40, 60, 80, and 100 sec. The next step is to cluster the data extracting dFNC states. We employed k-means clustering with k between 2 and 10. Plots were obtained for each of the considered CVIs. Finding the number of clusters was decided by looking at critical points (minimums or maximums) of the CVIs.

Results

The results obtained are based on the CVIs available in R. However, there were several CVIs that did not converge (possibly due to excessive computational demand and/or suboptimal library implementation) despite a long run-time and did not establish a reliable result. For this reason, a total of 22 “well behaved” CVIs were included from the R software in addition to GAP-Statistics and Elbow-Criterion. The Silhouette CVI is included in both R and GIFT, but we employed the R implementation as it was programmatically accessible for scripting.

Simulation 1

This simulation shows results for an increasing noise variance. The results are displayed in Figure 3. Not all CVIs were able to find the ground truth even at the highest SNR tested of 10 dB. Most CVIs failed to find the ground truth for the lowest SNR tested of −3 dB. Only the PBM and the Elbow-Criterion CVIs were able to agree with the ground truth on all four SNR levels tested indicating high resilience to Gaussian deviations from the centroids. Davies-Bouldin and Ray-Turi CVIs were next in performance because they could resolve the ground truth except for the case of lowest SNR.

Simulation 2

The second simulation was designed to replicate a pattern that is observed in the real data where some CVIs exhibit two critical points. This instance is a modification of the simulation in Figure 3 but using a ground truth of eight clusters with specific centroid positions. We imposed an SNR of 10 dB, where the variance is adjusted based on the variance of the simulated centroids. Results are displayed in Figure 4, where only the successful CVIs were included. Despite the ground truth of eight clusters (k=8), a visual inspection in Figure 4a can subjectively suggest an optimal of two clusters (k=2). The ground truth of eight clusters is not evident in this simulation. However, seven CVIs exhibited a local critical point that coincides with the imposed ground truth of k=8. Because of its background in functional connectivity, the Elbow-Criterion (Allen et al., 2014) plot has been included in Figure 4a. It is noticeable that the Elbow-Criterion suggests an optimal number of clusters of k=6 but misses the two points of interest (k=2) and (k=8). The Elbow-Criterion was excluded from further analysis because it failed to detect the two points of interest in this simulation.

Figure 4b displays the successful CVIs that were able to resolve at least one of the two points k=2 or k=8. We will focus our attention on the seven CVIs (Davies-Bouldin, Calinski-Harabasz, Silhouette, Ray-Turi, Wemmert-Gancarski, Dunn, and Xie-Beni) having a critical point that coincided with k=8, thus validating their sensitivity in detecting the ground truth. The reason for focusing on these CVIs will become more evident after analyzing the real data as the results tend to exhibit similar behavior. For illustrative purposes, Figure 4c shows examples of the first three CVIs where the local critical point at k=8 is evident as a maximum or minimum depending on the CVI properties. The other four CVIs successfully detecting k=8 are not included in Figure 4c, but their plots followed similar patterns.

Real Data 1

As illustrated in Simulation 1, the performance of the CVI methods depends on the SNR level. Similar to the simulations, the signal power was determined by the variance of the estimated centroids. The noise power was estimated in two steps: first calculating differences between each point and its corresponding centroid, second calculating the variance of the calculated differences. The SNR estimation results are displayed in Figure 5. SNR levels in SWC and ASWC real data approximately ranged from −15 dB to −1 dB. This SNR range is on the lower end of that included in the simulation of Figure 3. The window length of 10 sec exhibits an SNR below the simulated value −10 dB suggesting that most CVIs would fail if their behavior is similar to the simulation in Figure 3. In the same line, the Xie-Beni CVI might not produce an accurate result because the same simulation suggested it might need a high SNR of 3 dB. Notice that ASWC data showed higher SNR than SWC. For this reason, we expect CVIs applied to the ASWC data to have a better performance than using the SWC data.

Figure 5.

Figure 5.

Open in a new tab

Signal to Noise Ratio in decibels (SNR dB) for the real data. The calculation of signal and noise power is the same as that used in the simulations. SWC exhibits lower SNR than ASWC.

CVI results for the addiction dataset are displayed in Figure 6. We selected seven CVIs that showed good performance in the simulation of Figure 4: Davies-Bouldin, Ray-Turi, Xie-Beni, Wemmert-Gancarski, Calinski-Harabasz, Silhouette, and Dunn. Only the Davies-Bouldin and Ray-Turi CVIs resulted in a very regular pattern of two minima (one at two and one at seven clusters) which is similar to the simulated pattern of Figure 4. Xie-Beni, Dunn and Wemmert-Gancarski CVIs followed in performance but exhibited irregular patterns of more than two critical points where the outcomes were variable depending on preprocessing (ASWC and SWC) and window length. Calinski-Harabasz and Silhouette CVIs exhibited a monotonically decreasing value where the winning number of clusters was always k=2 and a local maximum was not observed. The outcome of these last two CVIs was not considered a complete success. The main reason is that two clusters might not be enough to describe the complexity of patterns in the data. Figure 7 shows the relationships between centroids of clustering with k=2 (2Cp-1 and 2Cp-2) and k=7 (7Cp-1, 7Cp-2, 7Cp-3, 7Cp4, 7Cp5, 7Cp-6, and 7Cp-7). As a baseline of cluster separation consider that correlation between 2Cp-1 and 2Cp-2 is equal to 0.85, thus any correlation lower than this value suggests cluster differentiation. While their patterns are different, correlation analysis indicates that 7Cp-4 (ρ=0.99) and 7CP-6 (ρ=0.93) are the most similar to 2Cp1. Similarly, centroids 7Cp1 (ρ=0.95), 7Cp-3(ρ=0.97) and 7Cp-5(ρ=0.95) are the closest to 2Cp-2. Centroid 7Cp-2 seems equally different from k=2 centroids 2Cp1 (ρ=0.81) and 2Cp2 (ρ=0.85) suggesting the existence of a very distinct cluster. Correlation results help us demonstrate that k=2 would not be a complete description of the data, but the real validation of the optimal number of clusters is better obtained through the CVI results. Although not included among the most successful CVIs, we provide the Elbow-Criterion results from real data in Supplementary Figure 2 revealing the ineffectiveness of this method as previously observed in Simulation 2.

Figure 6.

Figure 6.

Open in a new tab

Clustering index analysis for the addiction dataset. Seven CVIs were selected from the simulations. The plot show results for different preprocessing (SWC and ASWC) and sliding window lengths (20, 40, 60, 80, and 100 sec).

Figure 7.

Figure 7.

Open in a new tab

Correlations of centroids (Cp) for a 2-cluster partition against a 7-cluster partition for the addiction dataset. The correlation between 2Cp-1 and 2Cp-2 is equal to 0.85 to have a baseline of cluster separation. These two options are defined by the two local minimum in Figure 4. Although 2 clusters is the global minimum, the 7-cluster partition (the second minimum) reveals more richness of connectivity patterns in the data that cannot be observed in the 2 clusters option.

Real Data 2

The 7000 scans were used to explore reliability of the results by creating 28 different groups of scans. The SNR levels of these data are lower than the addiction dataset as shown in Figure 5. Similar to the results from the first dataset, the SNR levels from the ASWC method are generally higher thus suggesting that CVIs results might be more consistent than the SWC results. Figure 8 presents the average CVI plots for all seven CVIs tested. Again, in this dataset, Davies-Bouldin and Ray-Turi generally present two possible candidates k=2 (global minimum) and k=4 (first local minimum) for the optimal number of clusters. For these two CVIs, the ASWC results were consistent across all 5 window lengths while the SWC result for 20 sec window length fail to detect the k=4 point. Xie-Beni and Dunn results are not consistent, and the answer is different depending on the window length. The other CVIs show a decreasing trend with a single global answer at k=2 and no sensitivity to k=4 or any other k.

Figure 8.

Figure 8.

Open in a new tab

Results for the 7000 fMRI samples data. The data was distributed on 28 groups of 250 samples, each one was processed separately. The CVI plots correspond to the mean values and the error bars to the one standard deviation over the result from the 28 groups.

We focus on the two CVIs that consistently detected the local minimum at k=4 for the average results. However, drilling down into each replication analysis we can see that k=4 was not detected in all 28 data groups. Per group results are provided in Supplementary Figure 3 and Supplementary Figure 4. Table I shows the number of groups that agreed with the mean CVI result. The first observation is that data analyzed with ASWC shows higher consistency across data subsets than SWC counterparts. This might be expected since ASWC increases the SNR reducing the power of artifact fluctuations (Vergara et al., 2019a) that have been determined to exist in SWC (Hindriks et al., 2016; Leonardi and Van De Ville, 2015). The cases that deviate the most from Figure 8 occur for SWC and small window lengths. In several data groups at small window lengths the disagreement occurs because of a monotonic decrease where there seems to be no minimum point for the CVI plot. In other cases, the local minimum of the CVI is different from that in Figure 8 (k=4). Excluding the window length of 20 sec, the CVIs achieve an 86% rate of consistency (k=4) for window lengths from 40 sec to 100 sec.

Table I.

Number of CVI analyses that agreed with the average plot in Figure 8.

WinLen 20 sec WinLen 40 sec WinLen 60 sec WinLen 80 sec WinLen 100 sec
Davies-Bouldin SWC 5 18 24 27 24
ASWC 23 25 25 25 24
Ray-Turi SWC 6 19 24 27 24
ASWC 23 25 26 25 24
Open in a new tab

Discussion

The aim of this work is to test the use of available CVIs as techniques that provide the number of dFNC states. For this purpose, we employed simulated and fMRI data. The procedure required running the k-means clustering algorithm with different values of k where k represents the number of clusters. A CVI assessment is then obtained for each k. The optimal k corresponds to a maximum or minimum point depending on the characteristics of the CVI. Using this procedure, we assessed the performance of 22 CVIs that are readily available to the research community through the software package R. A small set of seven candidate CVIs were selected that demonstrated good performance during numerical simulations. Designing the simulated data was intended to replicate two important characteristics of the real data. First, different nuisance strengths that may arise during preprocessing due to the dynamic connectivity estimation were simulated by adding Gaussian noise. Second, real data might exhibit more than one solution for the optimal k. A practical number of clusters from resting state fMRI data is usually larger than three (Abrol et al., 2017), but most CVIs suggest two. However, two clusters might not describe the whole richness of patterns in the data as exemplified in Figure 7 and a second solution designated by a local minimum with k>2 indicated a better option. In this analysis, CVIs that are sensitive to the second solution for k were regarded as high performers. We selected the seven best CVIs based on these simulations and continued with further tests by applying the chosen CVIs to a dataset acquired from substance abuse subjects. Only two CVIs exhibited good performance and consistency in the real data exhibiting the simulated characteristics. We used a replication data set of 7000 fMRI scans to test the reliability of the two high-performing CVIs resulting in a replication outcome of 86%. Thus, the recommendation for dFC data is to use either Davies-Bouldin or Ray-Turi CVI.

The most important observation in the simulations is the general failure to resolve a minimum for the noisiest case. Notice that only the PBM index and the Elbow-Criterion were able to resolve the number of clusters for SNR= −10 dB in Figure 3. This behavior makes it evident that high noise power can hinder the performance of the CVIs. The next SNR level of −3 dB showed that only four CVIs were effective. We accepted the 11 methods that showed results up to 10 dB, thus discarding almost half of the tested CVI set. Real data exhibited several SNRs ranging from −10 dB to 3 dB, which coincided with the tested SNR range in the simulations. Recall that SNR is based on distances to corresponding centroids (as described for real data in the Results section) where the centroid location corresponds to the clustering outcome. Figure 5 shows that SNR in real data is highly dependent on the window length used, suggesting that the non-dFNC variability is linked to windowing. As it has been determined before, small window lengths produce a series of artifactual fluctuations that contaminates the real dFC signal (Leonardi and Van De Ville, 2015). The strength of these artifacts is the most plausible cause for the behavior demonstrated in Figure 5, where smaller window lengths correspond to lower SNR. To avoid strong artifacts it has been suggested that the window length be adjusted to 1/fmin, where fmin is the lowest frequency in the time courses spectrum. In our analysis, 1/fmin = 100 sec and the results for this window length coincide with low noise simulation behavior. The outcome was different for the ASWC preprocessing where a reduced disparity among the different window length results was observed in the SNR results of Figure 5 and almost all CVIs of Figure 6 and Figure 8. These results suggest low noise/nuisance strength when compared with the high SNR simulation results in Figure 3. The ASWC outcome is thus consistent with the theoretical noise reduction achieved when averaging the SWC windows (Vergara et al., 2019a). Notice that the simulations assume additive white Gaussian noise, but the true distribution of the contaminant signal in the real data is unknown. There is the possibility of respiratory and cardiac variance as contaminants (Cordes et al., 2001), but their influence was less of an issue in real data since well the behaved ASWC results could have corrected the spurious fluctuations present in SWC (Leonardi and Van De Ville, 2015).

The most revealing observation was the appearance of two different points of interest in real data. Figure 6 shows that sensitive CVIs points to 2 clusters as the best partition, but also exhibit a local minimum at 7 clusters which is a more practical result and closer to the original number of clusters previously used in a dynamic connectivity analysis of this dataset (Vergara et al., 2018c). Figure 8 shows that this CVI behavior of exhibiting a local minimum can be replicated across different datasets. Different from the addiction data, the optimal local minimum was 4 clusters. The discrepancy between the two datasets is likely due to differences in the preprocessing features and the selected RSNs. One important step in the second dataset was to discard the gICA components that could not be matched across the 28 data subsets. While this work does not focus on the relationship between the number of clusters and specific components, we will argue that component matching removed information enough to reduce the number of clusters to 4. The main reason for our second simulation (see Figure 4) with eight centroids was to demonstrate that CVIs may not converge to the ground truth and indicate a lower number of clusters, representing a local minimum. Not all of the 7 CVIs from the second simulation were able to perform well in real data. Likely, the distribution of noise and nuisances is not Gaussian, white or independent. Of the 7 CVIs, only two performed well in the real data since because the real noise is different from that in Figure 4. These two CVIs (Davies-Bouldin and Ray-Turi) are more promising than the others for the dynamic connectivity data.

There are a few considerations in selecting k that have an impact on dynamic functional connectivity estimates. Starting from the Elbow-Criterion, this method does not point to a single value because of its monotonically descending behavior. Notice the example in Figure 4 showing the possibility that the Elbow-Criterion points to a solution in between low and high number of clusters. The problem in selecting a middle point like the one on Figure 4 is that the true structure of the data might not be represented. On the other hand, the CVIs that passed the tests in this work indicate that there might be more than one point of interest for k which is not possible to see from the Elbow-Criterion. The analysis performed here suggests that the first optimal partition is almost always just two clusters. While this number may be considered in some context as too low to truly represent the data, it is plausible that two clusters represent most of the data. However, the local minimum appearing on the CVIs validated in this work suggests a more precise account of a higher level of structure in the data. As the real data analysis revealed, the local minimum result can vary across datasets. The cause of this variability may be rooted in aspects such as the data preprocessing, the neuropathology studied or the exclusion of some brain areas from the analysis. As the number of clusters increases, there is the possibility of finding a second local minimum. The disadvantage of increasing the number of clusters is the reduction on cluster size causing underrepresentation of the data. In addition, the distance between clusters decreases creating multidimensional regions of confusion where k-means have trouble assigning a membership. It is likely that k values larger than the first local minimum will result in CVI variability (with local minima and maxima) contaminated by the increasing size of the regions of confusion resulting in diminish information about the real structure of the data. The best example in our analysis is the results in Figure 8 where for k>4 there are small variations that are not reproducible across window lengths or data groups. However, the solution k=4 was reproducible for Davies-Bouldin and Ray-Turi CVIs. The proposed data-driven approach also provides a benchmark to compare results with the validated indexes in future neuroscience studies based on similar methods. Given the widespread use of time-varying functional connectivity approaches based on clustering, our work assumes critical importance in validating the estimated brain dynamics in future studies. We conclude that this correction could have a direct impact on brain state characterization post clustering, and help trace more precise information about the functional connectivity of the brain. Determining the causes for different characteristics of data structure is a topic for future work. This analysis is only concerned with determining the effectivity and sensitivity of methods to detect the existence of structure in the form of data clusters.

The application of sliding window methods to estimate functional connectivity has been debated with several studies citing potential pitfalls of the technique, and also emphasizing on the importance of comparing against null models (Hindriks et al., 2016; Lindquist et al., 2014; Shakil et al., 2015; Shakil et al., 2016). However, a sizable literature (Abrol et al., 2017; Calhoun et al., 2014; Hutchison et al., 2013; Preti et al., 2017) has clearly demonstrated the utility of this method in characterizing an extensive range of brain imaging research spheres including, but not limited to, pathophysiology (Espinoza et al., 2018; van der Horn et al., 2019; Vergara et al., 2019b), consciousness levels (Mooneyham et al., 2017), behavior (Schulz and Huston, 2002), substance abuse (Vergara et al., 2018c) and demographic variation (Lin et al., 2018). Inarguably, there can be several different methods of characterizing the temporal dynamics of the brain and providing useful decompositions to illuminate brain function. Furthermore, each of these time-varying functional connectivity assessments demand appropriate methodological and parametrical choices. Our work focuses on optimizing/validating a key parametric choice (i.e. the model order in k-means clustering) in one of the most widely used time-varying functional connectivity estimation methods (i.e. brain state estimation with sliding window technique) and post-decomposition clustering methods (k-means clustering) in recent literature. Indeed, the comparison of different validation indexes tested on the k-means clustering algorithm could be extended to other clustering approaches and such a comparison would make an interesting topic for future work.

One limitation of this work is the use of window lengths of 100 sec or smaller. The reason for choosing this upper limit is the known fact that large window lengths decrease the ability to resolve functional connectivity (Vergara et al., 2019a). As example we can point the Davies-Bouldin result for ASWC in Figure 6 showing a local minimum at k=4 for a window length of 100 sec and it showed k=7 as a second local minimum. We explored this point a bit and calculated Davies-Bouldin plots for longer window lengths in a complementary analysis with results that are provided in Supplementary Figure 2. The outcome k=4 was consistently the first local minimum for the window lengths larger than 100 sec. Likely, increasing the window length results in temporal overlaps that strengthens similarities within clusters from low k. Larger window lengths (>100 sec) have been used in a previous study where the analysis focused on the default mode network (Savva et al., 2019). The results of that study were visible at 120 sec, but concentrated in specific brain areas. The current work is different because the analysis includes the whole brain where a larger amount of information may translate into more variations on small time scales. The application of the two CVIs previously recommended can be automated by applying algorithms to find multiple local minima (Larson and Wild, 2015). Although most of our discussion so far assumed interest in a single solution for the number of clusters k, we can also consider that real data might be described by more than one clustering partition. There is evidence that different cluster partitions might not follow a strict hierarchy in fMRI functional connectivity data (Yeo et al., 2011). While more than one clustering partition was analytically deemed feasible, the different solutions for k (k=7 and k=17 in (Yeo et al., 2011)) did not result in a clear hierarchical distribution suggesting that two analyses should be performed, one for each k. In the same line, Figueroa et al. (Figueroa et al., 2019) found functional connectivity effects associated with major depressive disorder at different k values of the k-means algorithm. While biomarker discovery is not the scope of the current work, the existence of more than one critical point in the CVI plots (such as those in Figure 4c and Figure 6) suggests the value of not restricting our analysis to a single clustering partition.

Supplementary Material

1
2
3
4
5

Highlights.

  • We seek to improve finding the number of Dynamic Functional Network Connectivity (dFNC) clusters in the data.

  • While one of the most used techniques is the Elbow-Criterion, we present evidence that other cluster validity index (CVI) methods are proficient.

  • We tested the performance of twenty four CVIs applied to dFNC data.

  • Simulation and real data comparisons indicate that two CVIs are proficient for dFNC analysis: Ray-Turi and Davies-Bouldin.

Acknowledgements

This work was funded by the following NIH grants P20GM103472/1R01EB006841/R01REB020407 and National Science Foundation (#1539067) to V.C.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Author Disclosure Statement

No competing financial interests exist.

References

  1. Abrol A, Damaraju E, Miller RL, Stephen JM, Claus ED, Mayer AR, Calhoun VD, 2017. Replicability of time-varying connectivity patterns in large resting state fMRI samples. Neuroimage 163, 160–176. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Agosta F, Galantucci S, Filippi M, 2017. Advanced magnetic resonance imaging of neurodegenerative diseases. Neurol Sci 38, 41–51. [DOI] [PubMed] [Google Scholar]
  3. Allen EA, Damaraju E, Plis SM, Erhardt EB, Eichele T, Calhoun VD, 2014. Tracking whole-brain connectivity dynamics in the resting state. Cereb Cortex 24, 663–676. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Arbelaitz O, Gurrutxaga I, Muguerza J, Pérez JM, Perona I, 2013. An extensive comparative study of cluster validity indices. Pattern Recognition 46, 243–256. [Google Scholar]
  5. Ball GH, Hall DJ, 1965. ISODATA, a novel method of data analysis and pattern classification. Stanford research inst; Menlo Park CA. [Google Scholar]
  6. Banfield JD, Raftery AE, 1993. Model-based Gaussian and non-Gaussian clustering. Biometrics, 803–821. [Google Scholar]
  7. Brun M, Sima C, Hua J, Lowey J, Carroll B, Suh E, Dougherty ER, 2007. Model-based evaluation of clustering validation measures. Pattern Recognition 40, 807–824. [Google Scholar]
  8. Calhoun V, Adali T, Pearlson G, Pekar J, 2001. A method for making group inferences from functional MRI data using independent component analysis. Human brain mapping 14, 140–151. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Calhoun VD, Adali T, 2012. Multisubject independent component analysis of fMRI: a decade of intrinsic networks, default mode, and neurodiagnostic discovery. IEEE Rev Biomed Eng 5, 60–73. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Calhoun VD, Miller R, Pearlson G, Adali T, 2014. The chronnectome: time-varying connectivity networks as the next frontier in fMRI data discovery. Neuron 84, 262–274. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Calinski T, Harabasz J, 1974. A dendrite method for cluster analysis. Communications in Statistics - Theory and Methods 3, 1–27. [Google Scholar]
  12. Cordes D, Haughton VM, Arfanakis K, Carew JD, Turski PA, Moritz CH, Quigley MA, Meyerand ME, 2001. Frequencies contributing to functional connectivity in the cerebral cortex in “resting-state” data. AJNR Am J Neuroradiol 22, 1326–1333. [PMC free article] [PubMed] [Google Scholar]
  13. Damaraju E, Allen EA, Belger A, Ford JM, McEwen S, Mathalon DH, Mueller BA, Pearlson GD, Potkin SG, Preda A, Turner JA, Vaidya JG, van Erp TG, Calhoun VD, 2014. Dynamic functional connectivity analysis reveals transient states of dysconnectivity in schizophrenia. Neuroimage Clin 5, 298–308. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Davies DL, Bouldin DW, 1979. A Cluster Separation Measure. IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-1, 224–227. [PubMed] [Google Scholar]
  15. Desgraupes B, 2013. Clustering indices. University of Paris Ouest-Lab Modal’X 1, 34. [Google Scholar]
  16. Dunn JC, 1974. Well-separated clusters and optimal fuzzy partitions. Journal of cybernetics 4, 95–104. [Google Scholar]
  17. Edwards AW, Cavalli-Sforza LL, 1965. A method for cluster analysis. Biometrics, 362–375. [PubMed] [Google Scholar]
  18. Espinoza FA, Liu J, Ciarochi J, Turner JA, Vergara VM, Caprihan A, Misiura M, Johnson HJ, Long JD, Bockholt JH, Paulsen JS, Calhoun VD, 2019. Dynamic functional network connectivity in Huntington’s disease and its associations with motor and cognitive measures. Hum Brain Mapp. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Espinoza FA, Turner JA, Vergara VM, Miller RL, Mennigen E, Liu J, Misiura MB, Ciarochi J, Johnson HJ, Long JD, Bockholt HJ, Magnotta VA, Paulsen JS, Calhoun VD, 2018. Whole-Brain Connectivity in a Large Study of Huntington’s Disease Gene Mutation Carriers and Healthy Controls. Brain Connect 8, 166–178. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Figueroa CA, Cabral J, Mocking RJT, Rapuano KM, van Hartevelt TJ, Deco G, Expert P, Schene AH, Kringelbach ML, Ruhe HG, 2019. Altered ability to access a clinically relevant control network in patients remitted from major depressive disorder. Hum Brain Mapp 40, 2771–2786. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Fox MD, Greicius M, 2010. Clinical applications of resting state functional connectivity. Front Syst Neurosci 4, 19. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Friedman HP, Rubin J, 1967. On some invariant criteria for grouping data. Journal of the American Statistical Association 62, 1159–1178. [Google Scholar]
  23. Fries P, 2015. Rhythms for Cognition: Communication through Coherence. Neuron 88, 220–235. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Friston KJ, 2003. Statistical parametric mapping. Neuroscience Databases. Springer, pp. 237–250. [Google Scholar]
  25. Guyon I, Von Luxburg U, Williamson RC, 2009. Clustering: Science or art. NIPS 2009 workshop on clustering theory, pp. 1–11. [Google Scholar]
  26. Halkidi M, Batistakis Y, Vazirgiannis M, 2001. On clustering validation techniques. Journal of Intelligent Information Systems 17, 107–145. [Google Scholar]
  27. Hartigan JA, 1975. Clustering algorithms.
  28. Himberg J, Hyvärinen A, Esposito F, 2004. Validating the independent components of neuroimaging time series via clustering and visualization. Neuroimage 22, 1214–1222. [DOI] [PubMed] [Google Scholar]
  29. Hindriks R, Adhikari MH, Murayama Y, Ganzetti M, Mantini D, Logothetis NK, Deco G, 2016. Can sliding-window correlations reveal dynamic functional connectivity in resting-state fMRI? Neuroimage 127, 242–256. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Hutchison RM, Womelsdorf T, Allen EA, Bandettini PA, Calhoun VD, Corbetta M, Della Penna S, Duyn JH, Glover GH, Gonzalez-Castillo J, 2013. Dynamic functional connectivity: promise, issues, and interpretations. Neuroimage 80, 360–378. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Kodinariya TM, Makwana PR, 2013. Review on determining number of Cluster in K-Means Clustering. International Journal 1, 90–95. [Google Scholar]
  32. Larson J, Wild SM, 2015. A batch, derivative-free algorithm for finding multiple local minima. Optimization and Engineering 17, 205–228. [Google Scholar]
  33. Leonardi N, Van De Ville D, 2015. On spurious and real fluctuations of dynamic functional connectivity during rest. Neuroimage 104, 430–436. [DOI] [PubMed] [Google Scholar]
  34. Lin SJ, Vavasour I, Kosaka B, Li DKB, Traboulsee A, MacKay A, McKeown MJ, 2018. Education, and the balance between dynamic and stationary functional connectivity jointly support executive functions in relapsing-remitting multiple sclerosis. Hum Brain Mapp 39, 5039–5049. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Lindquist MA, Xu Y, Nebel MB, Caffo BS, 2014. Evaluating dynamic bivariate correlations in resting-state fMRI: a comparison study and a new approach. Neuroimage 101, 531–546. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Ma S, Correa NM, Li X-L, Eichele T, Calhoun VD, Adali T, 2011. Automatic identification of functional clusters in FMRI data using spatial dependence. Biomedical Engineering, IEEE Transactions on 58, 3406–3417. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Marriott F, 1971. Practical problems in a method of cluster analysis. Biometrics, 501–514. [PubMed] [Google Scholar]
  38. McClain JO, Rao VR, 1975. Clustisz: A program to test for the quality of clustering of a set of objects. JMR, Journal of Marketing Research (pre-1986) 12, 456. [Google Scholar]
  39. Milligan GW, 1981. A Monte Carlo study of thirty internal criterion measures for cluster analysis. Psychometrika 46, 187–199. [Google Scholar]
  40. Mooneyham BW, Mrazek MD, Mrazek AJ, Mrazek KL, Phillips DT, Schooler JW, 2017. States of Mind: Characterizing the Neural Bases of Focus and Mind-wandering through Dynamic Functional Connectivity. J Cogn Neurosci 29, 495–506. [DOI] [PubMed] [Google Scholar]
  41. Pakhira MK, Bandyopadhyay S, Maulik U, 2004. Validity index for crisp and fuzzy clusters. Pattern Recognition 37, 487–501. [Google Scholar]
  42. Pal NR, Biswas J, 1997. Cluster validation using graph theoretic concepts. Pattern Recognition 30, 847–857. [Google Scholar]
  43. Preti MG, Bolton TA, Van De Ville D, 2017. The dynamic functional connectome: State-of-the-art and perspectives. Neuroimage 160, 41–54. [DOI] [PubMed] [Google Scholar]
  44. Rachakonda S, Egolf E, Correa N, Calhoun V, 2007. Group ICA of fMRI toolbox (GIFT) manual. https://www.nitrc.org/docman/view.php/55/295/v1.3d_GIFTManual.pdf [cit. 2011-11-5].
  45. Ratkowsky D, Lance G, 1978. Criterion for determining the number of groups in a classification.
  46. Ray S, Turi RH, 1999. Determination of number of clusters in k-means clustering and application in colour image segmentation. Proceedings of the 4th international conference on advances in pattern recognition and digital techniques Calcutta, India, pp. 137–143. [Google Scholar]
  47. Rousseeuw PJ, 1987. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics 20, 53–65. [Google Scholar]
  48. Sakoğlu Ü, Pearlson GD, Kiehl KA, Wang YM, Michael AM, Calhoun VD, 2010. A method for evaluating dynamic functional network connectivity and task-modulation: application to schizophrenia. Magnetic Resonance Materials in Physics, Biology and Medicine 23, 351–366. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Savva AD, Mitsis GD, Matsopoulos GK, 2019. Assessment of dynamic functional connectivity in resting-state fMRI using the sliding window technique. Brain Behav 9, e01255. [DOI] [PMC free article] [PubMed] [Google Scholar]
  50. Schulz D, Huston JP, 2002. The sliding window correlation procedure for detecting hidden correlations: existence of behavioral subgroups illustrated with aged rats. Journal of neuroscience methods 121, 129–137. [DOI] [PubMed] [Google Scholar]
  51. Scott AJ, Symons MJ, 1971. Clustering methods based on likelihood ratio criteria. Biometrics, 387–397. [Google Scholar]
  52. Shakil S, Keilholz SD, Chin-Hui L, 2015. On frequency dependencies of sliding window correlation. 363–368. [Google Scholar]
  53. Shakil S, Lee CH, Keilholz SD, 2016. Evaluation of sliding window correlation performance for characterizing dynamic functional connectivity and brain states. Neuroimage 133, 111–128. [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Singer W, 1999. Neuronal Synchrony: A Versatile Code for the Definition of Relations? Neuron 24, 49–65. [DOI] [PubMed] [Google Scholar]
  55. Tibshirani R, Walther G, Hastie T, 2001. Estimating the number of clusters in a data set via the gap statistic. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 63, 411–423. [Google Scholar]
  56. van der Horn HJ, Vergara VM, Espinoza FA, Calhoun VD, Mayer AR, van der Naalt J, 2019. Functional outcome is tied to dynamic brain states after mild to moderate traumatic brain injury. Hum Brain Mapp. [DOI] [PMC free article] [PubMed] [Google Scholar]
  57. Vergara VM, Abrol A, Calhoun VD, 2019a. An average sliding window correlation method for dynamic functional connectivity. Hum Brain Mapp. [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. Vergara VM, Liu J, Claus ED, Hutchison K, Calhoun V, 2017a. Alterations of resting state functional network connectivity in the brain of nicotine and alcohol users. Neuroimage 151, 45–54. [DOI] [PMC free article] [PubMed] [Google Scholar]
  59. Vergara VM, Mayer A, Kiehl KA, Calhoun VD, 2018a. Dynamic functional network connectivity discriminates mild traumatic brain injury through machine learning. NeuroImage: Clinical. [DOI] [PMC free article] [PubMed] [Google Scholar]
  60. Vergara VM, Mayer AR, Damaraju E, Kiehl KA, Calhoun V, 2017b. Detection of Mild Traumatic Brain Injury by Machine Learning Classification Using Resting State Functional Network Connectivity and Fractional Anisotropy. J Neurotrauma 34, 1045–1053. [DOI] [PMC free article] [PubMed] [Google Scholar]
  61. Vergara VM, Mayer AR, Kiehl KA, Calhoun VD, 2018b. Dynamic functional network connectivity discriminates mild traumatic brain injury through machine learning. Neuroimage Clin 19, 30–37. [DOI] [PMC free article] [PubMed] [Google Scholar]
  62. Vergara VM, van der Horn HJ, Mayer AR, Espinoza FA, van der Naalt J, Calhoun VD, 2019b. Mild Traumatic Brain Injury Disrupts Functional Dynamic Attractors of Healthy Mental States.
  63. Vergara VM, Weiland BJ, Hutchison KE, Calhoun VD, 2018c. The Impact of Combinations of Alcohol, Nicotine, and Cannabis on Dynamic Brain Connectivity. Neuropsychopharmacology 43, 877–890. [DOI] [PMC free article] [PubMed] [Google Scholar]
  64. Xie XL, Beni G, 1991. A validity measure for fuzzy clustering. IEEE Transactions on Pattern Analysis & Machine Intelligence, 841–847. [Google Scholar]
  65. Yeo BT, Krienen FM, Sepulcre J, Sabuncu MR, Lashkari D, Hollinshead M, Roffman JL, Smoller JW, Zollei L, Polimeni JR, Fischl B, Liu H, Buckner RL, 2011. The organization of the human cerebral cortex estimated by intrinsic functional connectivity. J Neurophysiol 106, 1125–1165. [DOI] [PMC free article] [PubMed] [Google Scholar]
  66. Zalesky A, Breakspear M, 2015. Towards a statistical test for functional connectivity dynamics. Neuroimage 114, 466–470. [DOI] [PubMed] [Google Scholar]
  67. Zalesky A, Fornito A, Cocchi L, Gollo LL, Breakspear M, 2014. Time-resolved resting-state brain networks. Proceedings of the National Academy of Sciences 111, 10341–10346. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

1
NIHMS1575458-supplement-1.docx (2.2MB, docx)
2
NIHMS1575458-supplement-2.docx (379.3KB, docx)
3
NIHMS1575458-supplement-3.docx (3.1MB, docx)
4
NIHMS1575458-supplement-4.docx (2.7MB, docx)
5
NIHMS1575458-supplement-5.zip (99.5MB, zip)

RESOURCES