The Temporal Dynamics of Spontaneous Emotional Brain States and Their Implications for Mental Health

Abstract

Temporal processes play an important role in elaborating and regulating emotional responding during routine mind wandering. However, it is unknown whether the human brain reliably transitions among multiple emotional states at rest and how psychopathology alters these affect dynamics. Here, we combined pattern classification and stochastic process modeling to investigate the chronometry of spontaneous brain activity indicative of six emotions (anger, contentment, fear, happiness, sadness, and surprise) and a neutral state. We modeled the dynamic emergence of these brain states during resting-state fMRI and validated the results across two population cohorts—the Duke Neurogenetics Study and the Nathan Kline Institute Rockland Sample. Our findings indicate that intrinsic emotional brain dynamics are effectively characterized as a discrete-time Markov process, with affective states organized around a neutral hub. The centrality of this network hub is disrupted in individuals with psychopathology, whose brain state transitions exhibit greater inertia and less frequent resetting from emotional to neutral states. These results yield novel insights into how the brain signals spontaneous emotions and how alterations in their temporal dynamics contribute to compromised mental health.

INTRODUCTION

As people go about their daily lives, they cycle through many emotions that are triggered by a combination of external stimuli and internal thoughts. Mental processes determine the intensity and duration of these affective experiences through both effortful and automatic influences. While sustaining positive emotions like happiness and contentment contribute to well-being, excess euphoria can lead to heightened risk-taking and manic episodes. Similarly, negative emotions like fear in situations of impending threat promote vigilance and strategic coping, whereas unregulated anxiety impairs cognitive functioning even in safe environments. Emotions are thus dynamical systems (Lewis, 2005), and a key component to healthy emotion regulation is the ability to experience specific emotions when they are conducive to goal-oriented behavior and to terminate them and shift to other affective states to avoid maladaptive consequences (Cole, Ramsook, & Ram, 2019).

Despite the potential importance of these dynamics to mental health and well-being, research on the temporal properties of emotions, or “affective chronometry,” is sparse but growing (Davidson, 2015). Conceptual frameworks in this field emphasize several temporal parameters that are thought to play key roles in emotion processing. For instance, emotional reactivity is characterized not only by the threshold for eliciting a response and the response amplitude but also its duration, which can be further decomposed into the rise time to the peak response, maintenance time of the peak response, and recovery time back to baseline (Davidson, 1998). Emotions are also characterized by variability in their quality and/or intensity over time in response to changing stimulus contingencies, appraisals, goal pursuits, and social interactions. Such variation occurs over multiple timescales, from those that are initiated within seconds of an eliciting event to those measured over hours, days, or weeks. By contrast, “emotional inertia” refers to the resistance of emotions to changes over time, which leads to a carryover of emotional experience despite changing environmental demands or intrinsic motivations (Kuppens & Verduyn, 2017). Finally, there may be characteristic sequencing or temporal co-occurrences of specific emotions, either because of their inherent functional relationships (e.g., relief following the termination of anxiety), their natural comingling (e.g., anger that accompanies contempt), or poor differentiation among them (confusion between similar emotions, such as shame and guilt; Kuppens & Verduyn, 2017).

Behavioral studies have examined these temporal relationships using ecological momentary assessment (EMA), which longitudinally samples emotional experience in the real world at prescribed temporal intervals over several days or weeks. A meta-analysis of 79 EMA studies found that psychological health was associated with emotional experience patterns that had less inertia, were less variable, and were more stable, particularly for negative emotions (Houben, Van Den Noortgate, & Kuppens, 2015). These dynamic measures may be important for predicting affective trajectories over time and for distinguishing affective disorders from one another. For instance, one study found that greater liability and negative affect in 7-year-olds predicted poorer adaptive self-regulation at 8 years of age and internalizing symptoms at 9 years of age (Kim-Spoon, Cicchetti, & Rogosch, 2013). In adults, instability of reported affect over a 10-day period was related to individual differences in trait anxiety, whereas average levels of positive and negative affect were related to individual differences in depression (Heller, Fox, & Davidson, 2019). Theoretical models, with substantial empirical support, implicate emotional instability in borderline personality disorder, whereas emotional inertia is generally associated with stress and depressive symptoms (Kuppens, Allen, & Sheeber, 2010; Trull et al., 2008; Tull, Barrett, McMillan, & Roemer, 2007; Suls, Green, & Hillis, 1998). Counter to this body of evidence, a recent meta-analysis of a smaller subset of 15 EMA studies found that dynamic measures may not add much value relative to measures of mean affect levels to predict overall psychological well-being (Dejonckheere et al., 2019). The authors stated that some of the studies included in their meta-analysis may have sampled emotions too sparsely (some only sampled once per day) and that typical subjective measures of well-being emphasize general affect over a broad timescale, such as assessing average sadness over a 2-week period, which emphasizes global, static facets of affective experience. The authors also suggested that machine learning tools may provide a promising means forward to identify more useful features of affective patterning in complex time series data.

Thus, there is a need for research paradigms to sample affective states at a more refined temporal scale, to innovate more objective measures of affect that go beyond general symptom questionnaires, and to utilize more sensitive quantitative metrics of temporal dynamics. Wichers has argued that smaller units of experiential variation at the “microscale” of minutes to hours would establish the smallest building blocks of momentary experiences that contribute to the formation and long-term trajectory of psychopathology (Wichers, 2014). She further postulates that reinforcing loops and continuous interplay among different momentary affective states in response to emotion elicitors can alter the balance between healthy and maladaptive outcomes in the long run and that these dynamics may be influenced by individual differences in genetic or early-life experiential risk factors for psychopathology. Ruminative thinking may play a key role in altering the natural course of emotion dynamics by extending the duration of self-reflective negative thoughts via an impaired disengagement mechanism (Koster, De Lissnyder, Derakshan, & De Raedt, 2011). Although these ideas were initially applied to explain major depressive disorder, extensions of this argument can be readily made for other facets of psychopathology, including the role of worry in anxiety disorders.

One potential fruitful avenue is to incorporate neurobiological measurements in these studies to derive more objective indices of emotional state dynamics as they naturally unfold over shorter timescales. Communicated aspects of subjective emotional experience may only partially overlap with physiological indices (Scherer, 2009), and affective self-reports are subject to bias, social desirability, and other experimenter demand characteristics. Moreover, behavioral sampling by EMA does not provide a continuous measurement of affect change. Nonetheless, most neuroimaging studies of emotion focus only on the immediate response to brief emotional stimuli like faces or scenes that likely reflect stimulus evaluative processes rather than induced experiential aspects of emotion. Electrophysiological studies have examined temporal brain dynamics involved in the initial orienting, appraisal, and regulation of the emotional elicitor that occur within hundreds of milliseconds to seconds of stimulus onset (e.g., Luo, Jiang, Chen, Zhang, & You, 2019; Paul, Simon, Kniesche, Kathmann, & Enrass, 2013; Pourtois, Delplanque, Michel, & Vuilleumier, 2008). Some fMRI studies have shown carryover effects that reflect residual processing of an emotional event several seconds after its termination (e.g., Heller et al., 2015; Walter et al., 2009; Siegle, Steinhauer, Thase, Stenger, & Carter, 2002) or the lingering impact of an emotional state minutes later during a subsequent cognitive task (Tambini, Rimmele, Phelps, & Davachi, 2017; Wang, LaBar, & McCarthy, 2006). Even less is known about the emergence and sequencing of intrinsic emotional states during unconstrained periods of spontaneous thought (Kragel, Knodt, Hariri, & LaBar, 2016), although it is hypothesized that, during these periods of quiescent self-reflection, ruminative thinking impacts negative affect perseveration in depression, anxiety, and other disorders (Northoff, Wiebking, Feinberg, & Panksepp, 2011; Nolen-Hoeksema, Wisco, & Lyubomirsky, 2008).

Despite the initial advances in characterizing emotional brain dynamics, it remains unclear how to detect and quantify transitions among discrete emotional states in the brain over a timescale of tens of seconds to minutes and whether the temporal patterns relate to psychopathology risk. To address this gap in the literature, we undertook an affective neurocomputing approach to track how the brain cycles among various emotions while participants' mind wander at rest. We built upon our previous fMRI work that used machine learning to decode six emotions (anger, contentment, fear, happiness, sadness, and surprise) and a neutral state from film and music inductions (Kragel & LaBar, 2015). This prior study established the reliable separability of cortical-limbic-subcortical networks for each affective state using cross-subject validation. Activity in these distributed networks that informed the classifier's performance tracked the online subjective experience of the emotions elicited by the stimuli. We then applied the emotion-specific maps to model the spontaneous elicitation of the same brain states during resting-state fMRI (Kragel et al., 2016). We found that fluctuations in resting-state fMRI activity can be decoded into discrete affective states that exhibit spatiotemporal coherence and predict subjective reports of feelings that emerged during the resting-state scan. Finally, we showed that the frequency distributions of sad and fear brain states during the resting-state period predicted individual differences in self-reported depression and state anxiety, respectively, immediately after the scanning session.

In this study, we extended this line of research by modeling the intrinsic brain dynamics associated with the same affective states as a discrete-time Markov process. The model generates probabilities that reflect the likelihood that an individual will remain in the same affective state from one brain sample (repetition time [TR]) to the next in the resting-state time series (self-transitions), as well as the probabilities that reflect the likelihood that they will transition from one affective state to another (other-transitions) for all possible combinations in the set. Importantly, this modeling approach does not assume that the affective states individuals traverse at rest are quantitatively similar to prototypical emotional episodes, either in terms of brain activity or subjective experience. Rather, it assumes that relative engagement of the cortical-limbic-subcortical networks that predict each affective state exhibit reliable dynamics at rest, just as activity in motor cortex exhibits intrinsic activity that is low amplitude (when compared with overt motor behavior) but is nonetheless highly correlated (Biswal, Yetkin, Haughton, & Hyde, 1995).

We quantified the extent to which each brain state acts as transition hub in the emotion network by computing the proportion of transitions to each state that originated from a different state, as opposed to a self-transition, with higher values implicating greater hubness. We introduce a novel affective chronometry concept—emotional resetting—as the probability of transitioning from any emotional brain state to a neutral brain state. Unlike some return-to-baseline measures, emotional resetting emphasizes probabilities rather than temporal durations, and the baseline is specified as a neutral brain state, rather than any affective state that happened to precede the emotion change. We conceptualize emotional resetting as playing an important regulatory function by limiting the persistence of affective states and by permitting the system to reestablish a common baseline state, consistent with theoretical models of emotion dynamics (Lewis, 2005).

We developed the emotion dynamics model in a local convenience sample, the Duke Neurogenetics Study (DNS), that was used in our initial resting-state analyses (Kragel et al., 2016). To determine the generalizability of the model, we tested it across two out-of-sample cohorts—a hold-out testing set from the DNS and a similarly sized subset of the Nathan Kline Institute Rockland Sample (NKI-RS; Nooner et al., 2012). To evaluate the potential utility of the model to capture brain dynamics that differentiate individuals with psychopathology from mentally healthy individuals, we conducted additional analyses on subsets of participants in the two cohorts based on their clinical characteristics in a transdiagnostic manner. We hypothesized that our stochastic model would provide a better fit to the test data than a null model that preserves the autocorrelation in the time series, thus demonstrating good out-of-sample validation. We further hypothesized that, compared with healthy participants, individuals with a mental health diagnosis would present with relatively more emotional inertia and less emotional resetting in their spontaneous brain dynamics.

METHODS

Participants

Model Development Sample

The sample used for model development came from the DNS, which assesses a wide range of behavioral and biological traits among young adult university students. A total of 499 participants from this study (mean age = 19.65 years, SD = 1.22 years; 274 women) had usable data from two consecutive 256-sec resting-state fMRI scans (512-sec total scan duration) as part of the protocol and were included in the analyses. The sample was <1% Native American, 27% Asian, 11% Black, 51% White, 8% biracial/multiracial, and 3% other. All participants provided informed consent in accordance with Duke University guidelines. The DNS sample includes mentally healthy participants, as well as those with an Axis I disorder or select Axis II disorders (antisocial personality disorder and borderline personality disorder), assessed with the electronic Mini International Neuropsychiatric Interview (Sheehan et al., 1998) and Structured Clinical Interview for the Diagnostic and Statistical Manual of Mental Disorders (4th edition; DSM-IV) subtests (American Psychiatric Association, 1994). Of the participants included in this study, 121 (24.2%) met DSM-IV criteria for current or past history of at least one mental health disorder and/or exhibited current mood/depressive symptoms (9 agoraphobia, 63 alcohol abuse/dependence, 1 anorexia nervosa, 1 antisocial personality disorder, 1 bipolar II disorder, 6 bipolar disorder not otherwise specified (NOS), 2 borderline personality disorder, 4 bulimia nervosa, 17 cannabis abuse/dependence, 6 generalized anxiety disorder, 15 hypomanic episode, 16 major depressive disorder, 24 major depressive episode, 1 manic episode, 3 mood disorder with psychotic features, 6 obsessive compulsive disorder, 8 panic disorder, 1 psychotic disorder, 1 seasonal affective disorder, 4 social phobia, and 34 suicidality). A random subset of participants was used for model development (n = 200), with the remaining 299 participants held out for as an independent validation sample. Although no a priori power analysis was conducted due to the novel, data-driven nature of the project aims and our goal of validation in independent cohorts, the database contained a large sample size for resting-state fMRI analyses, and we included all participants in the database with usable data at the time the study commenced.

External Validation Sample

A total of 207 participants (mean age = 35.01 years, SD = 20.13 years; 87 women) were included from an early release of the NKI-RS, which investigates developmental trajectories for risk and resilience across the life span (Nooner et al., 2012). The sample was <1% Native American, 7% Asian, 28% Black, <1% Native Hawaiian, and 64% White. All participants from this study provided informed consent, either at the Nathan Kline Institute or Montclair State University. Mental health disorders were assessed using the research version of the Structured Clinical Interview for DSM-IV-TR (First, Spitzer, Gibbon, & Williams, 2002). We included all participants in the database release at the time our study commenced with useable resting-state data and a minimum scan duration that matched that from the DNS (the NKI-RS participants underwent a 650-sec duration scan). We chose this database because it is a community-based sample who are somewhat older than the DNS participants, which provides a good test case for evaluating the reliability of the model. Twenty-six participants (12.6%) from this sample met DSM-IV criteria for current and/or past history of one or more mental health disorders (2 alcohol abuse/dependence, 1 anxiety disorder NOS, 3 attention deficit hyperactivity disorder, 2 bipolar i disorder, 1 body dysmorphic disorder, 1 delusional disorder, 1 eating disorder NOS, 11 major depressive disorder/depressive disorder NOS, 2 panic disorder, 4 posttraumatic stress disorder, 2 psychotic disorder NOS, 1 reading disability, 3 social phobia, 1 specific phobia, 4 substance use). Two participants had missing diagnostic information and were thus excluded from the analyses comparing healthy and clinical subgroups.

Data Acquisition

Model Development and Internal Validation Sample

Participants were scanned on one of two identical 3-T General Electric MR 750 systems with 50-mT/m gradients and an eight-channel head coil for parallel imaging. For the two consecutive 256-sec resting-state scans (128 functional volumes each), a series of interleaved axial functional slices aligned with the anterior commissure–posterior commissure plane were acquired for whole-brain coverage using an inverse-spiral pulse sequence to reduce susceptibility artifact (TR = 2000 msec, echo time = 30 msec, α = 60°, field of view = 240 mm, voxel size = 3.75 × 3.75 × 4 mm, 34 contiguous slices). Four initial radiofrequency excitations were performed and discarded to achieve steady-state equilibrium. Participants were shown a blank gray screen and instructed to lie still with their eyes open, think about nothing in particular, and remain awake.

External Validation Sample

Participants in the NKI-RS sample underwent a scan session using a Siemens TIM Trio 3.0 T MRI scanner with a 12-channel head coil. Resting-state fMRI scans were collected using an EPI sequence with the following parameters: TR/echo time = 2500 / 30 msec, flip angle = 80°, field of view = 216 × 216 mm², voxel size = 3.0 × 3.0 × 3.0 mm², distance factor = 10%, number of slices = 38. Each scan session was 650 sec long and comprised 260 functional volumes. Participants were instructed to keep their eyes closed, relax their minds, and not to move.

Preprocessing and Analysis of fMRI Data

All preprocessing was conducted using SPM8 (Wellcome Department of Imaging Neuroscience). Images for each participant were slice-time-corrected, realigned to the first volume in the time series to correct for head motion, spatially normalized into a standard stereotactic space (Montreal Neurological Institute template) using a 12-parameter affine model (final resolution of functional images = 2 mm isotropic voxels), and smoothed with a 6-mm full-width at half-maximum Gaussian filter. Low-frequency noise was attenuated by high-pass filtering with a 0.0078 Hz cutoff. Time series for each voxel were centered to have zero mean and scaled to have unit standard deviation.

Preprocessed resting-state time series were classified into multiple distinct emotional states as described in detail previously (Kragel et al., 2016; Kragel & LaBar, 2015). Briefly, emotion-specific information maps were derived from a study that used instrumental music and film clips to induce the six emotions of interest plus a neutral state (Kragel & LaBar, 2015). Parameter estimates were beta-weights in the general linear model of each induction trial for each voxel (mean-centered, gray matter masked, and unsmoothed). Decoding was performed via partial least squares discriminant analysis (PLS-DA). The PLS regression coefficients, which indicate the importance of each voxel in informing the classifier's emotion category predictions, were then remapped to standard space and corrected for multiple comparisons. This analysis yielded a distributed cortico-limbic-subcortical network that characterized importance maps for each emotion classification (see Figure S1 in Kragel et al., 2016). The resting-state classifications were achieved by computing the dot product between these voxel-wise parameter estimates for the seven emotion-specific maps and the preprocessed resting-state fMRI data at each acquisition time point (Kragel et al., 2016). We employed a one-versus-all approach for multiclass classification, where categorizations are performed by identifying the model (out of 7) with the highest dot product (no minimum difference is required). In this way, each time point was labeled as the emotion category with the largest dot product value. Because the dot product is influenced by both the magnitude and similarity of the two vectors used in its computation, brain states that are classified into each emotion category will tend, on average, to resemble the pattern of weights that define each classification model. The values reported in Kragel et al. (2016) for the DNS sample were used in the present analyses, and the same methods were applied here to the data from the NKI-RS sample.

We note that in no cases were the dot products identical. The average difference between the maximum dot product and the second highest dot product was 0.839 (SD = 0.107) in the DNS sample (n = 499) and 0.1441 (SD = 0.0247) in the NKI-RS (n = 207). Thus, the average observed differences in classifier outputs were “very large” with Cohen's d = 7.84 for the DNS and d = 5.8 for the NKI-RS. We also note it is unlikely that a small number of voxels contribute to the dot product used for classification. The PLS-DA algorithm reduces model complexity by identifying a small number of latent dimensions that span all inputs (voxels); thus, the contribution of individual voxels is combined with that from thousands of other voxels in the model. In addition, the resting-state time series of each voxel is standardized to have zero mean and unit variance, which reduces differences in scale across voxels, making it less likely for a small number of voxels to have an undue influence. Resting-state data are characterized by structured correlations at multiple spatial scales (Hay, Ritter, Lobaugh, & McIntosh, 2017); the high covariance between regions also makes it unlikely that isolated regions are the primary contributors to spontaneous emotion signaling at rest.

The present work is based on a supervised procedure that requires brain states are assigned labels to train classification models. However, there is no assumption that the labels used as “ground truth” to train the classification models are correct in some theoretical sense. There is an assumption that these brain states differ in some way that is consistent across individuals, but the precise nature of this difference is not specified. Brain states could share labels because they are members of a classically defined natural kind (Ekman & Cordaro, 2011), because they cause coordinated behavioral and body changes (Anderson & Adolphs, 2014), because they are sufficiently similar to emotion prototypes (Fehr & Russell, 1984), because they involve distinctive patterns of cognitive appraisal (Moors, Ellsworth, Scherer, & Frijda, 2013), or because they are grouped that way by situated conceptualizations/categorizations (Barrett, 2013, 2017). Any of the above explanations could be the reason for category membership, and for this reason, the ground truth labels can be considered theoretically unbiased. Whether the brain-based emotion classifiers are more or less consistent with any particular theoretical account is not an assumption but is an open empirical question (Azari et al., 2020; Kragel, Koban, Barrett, & Wager, 2018).

Modeling Emotional Brain States as a Markov Process

We modeled intrinsic emotional brain dynamics as a Markov process (Figure 1). Markov processes have two key characteristics: They are random and memoryless. Markov processes are random because their outputs are not determined by their parameters and initial conditions. Unlike task-based fMRI, which produces signals that are partially determined by experimental manipulations, intrinsic brain activity measured using resting-state fMRI is inherently less structured and is more likely to be characterized as a stochastic process. Markov processes are memoryless because future states of a system only depend on the present but not past states of the system. Using the DNS model development sample (n = 200), we estimated parameters of a discrete-time Markov chain (i.e., the 7 × 7 transition matrix P describing the probability of brain states changing from one emotional state to another) from the sequence of resting-state fMRI emotion classifications for each participant. This was accomplished by using the maximum likelihood estimation of the transition matrix (Anderson & Goodman, 1957), which corresponds to the sample proportion of transitions from each state to all possible states of the system. The estimated probability of transitioning from state i to state j (P_i,j) is the number of observed transitions from i to j divided by the total number of observed transitions out of state i. Computing estimates for all possible transitions yields the full transition matrix P. We used this procedure and created a single group-averaged model by averaging transition matrices across participants.

Figure 1. .

Modeling the intrinsic dynamics of emotional brain states. (A) Resting-state fMRI data were acquired and preprocessed, producing multidimensional time series with hundreds of thousands of features and several hundred time points for each participant. (B) Voxel-wise maps for each of the seven emotion categories, derived from a machine-learning analysis of a prior emotion induction study (Kragel & LaBar, 2015), were applied to the preprocessed resting-state fMRI data at each time point (TR) by computing their dot product, thereby reducing the data to a seven-dimensional time series (Kragel et al., 2016). (C) Each time point is classified by identifying the emotion category that yields the highest dot product, which generates a sequence that reflects the best evidence for the brain being in one of the seven affective states at each resting-state time point in the series. The classification sequence for one participant chosen at random is illustrated in the figure; based on the classifier's predictions, this individual was most often in a sad brain state (purple lines) toward the beginning of the resting-state scan and then shifted to being most prominently in a neutral brain state (pink lines) later on. Each line represents one TR, and all 256 TRs from the resting-state period are concatenated in the image and aligned left-to-right from the beginning to the end of the scan. Note that once a change in emotional state occurs, it usually lasts for several seconds before shifting again. Maximum likelihood estimation is used to produce a transition matrix (i.e., parameter values for a discrete-time Markov chain) that specifies the probability of transitioning from one emotional brain state to another, which is then averaged across all participants in the model development sample (n = 200), which were then tested in independent data sets. The transition matrix from one individual participant is shown in the figure.

To estimate the reliability of model estimates, a split-half analysis was performed using the first and second halves of resting-state data from the DNS training data (i.e., the first and second runs of scanning). The intraclass correlation coefficient (ICC) was used as an estimate of model reliability for parameters of group-level and individual subject models (Shrout & Fleiss, 1979). Bootstrap confidence intervals for the ICC were estimated by randomly resampling participants with replacement.

Validation Using Bayesian Model Comparisons

The Markov chain was evaluated in multiple independent validation data sets using Bayesian information criterion (BIC) analysis. For each participant in each test set, the BIC was estimated as BIC = −2ln(L) + kln(n), where L is the model log likelihood, k is the number of model parameters (in this case, there are 7 × 6 = 42 unique parameters, as all the transition probabilities sum to 1), and n is the number of observations (256 acquisition volumes for the DNS and 205 for NKI-RS). The log likelihood of an observed sequence of test data is defined as L = ${\sum }_{i=1}^n$ log P_θ(X_i|X_i−1), where P_θ are elements of the transition matrix, corresponding to the observed transitions in independent data.

As a baseline for comparison, synthetic null models were created by performing wavelet resampling of fMRI time series (Kragel et al., 2016; Polyn, Natu, Cohen, & Norman, 2005), which preserves the autocorrelation present in the fMRI signal, and repeating the model estimation procedure. This procedure was repeated 1000 times to estimate models that would be expected based on autocorrelation alone. ΔBIC was computed as the difference in BIC between the group-averaged Markov chain estimated on real time-series and the distribution of BIC values based on synthetic autocorrelation. BIC was computed separately on each random iteration by computing the average log likelihood in each validation sample. The mean and 95th percentile of ΔBIC were computed from these two distributions (displayed in Figure 2). ΔBIC values exceeding 10 are generally considered as very strong evidence in favor of one model over another (Kass & Raftery, 1995). For completeness, we report both differences in log likelihood (ΔL) and BIC (ΔBIC).

Figure 2. .

Markov models characterize emotional brain dynamics in multiple independent samples. (A) The transition matrix derived from 200 participants in the DNS. The color map indicates the probability of transitioning from one brain state (indicated by the index of the row, state i) to another (indicated by the column, state j). Greater probabilities along the diagonal show that the brain tends to stay in the same emotional state over consecutive time points while at rest, while off-diagonal probability values indicate transitions between states. (B) BIC analysis indicates that the empirical model shown in A predicts sequences of emotional brain states in two independent samples better than models based on autocorrelation alone (distributions for 1,000 autocorrelation models are shown). (C) Neutral states serve as transition hubs in the network of emotions. Across the resting-state fMRI session, individuals were, on average, more likely to transition from any emotional brain state to a neutral brain state than to any other emotional brain state in the network. The network is visualized using a force-directed graph layout (Hu, 2005), with emotional states depicted as gray circles and directed edges as colored lines oriented clockwise from source to destination. Colors use the same labeling scheme as in Figure 1B. The width of the edges indicates the strength of the transition probabilities. Note that neutral states are relatively equidistant from all emotional starting points in the graph.

Following model comparisons, we evaluated how brain dynamics differed from the null autocorrelation model by testing for differences in the magnitude of parameter estimates in the empirical Markov model. We tested for column-wise differences in off-diagonal parameters of the transition matrix because the synthetic autocorrelation model predicts that transitions among different states are equally likely. The proportion of off-diagonal transitions were compared between different emotional states using Friedman's ANOVA because the ratios being compared have a restricted range between 0 and 1 and do not meet the distributional assumptions of parametric ANOVA.

Estimating Transient Changes in Emotional Brain States

Given the transition matrix P estimated from the DNS training data, we estimated the probability of transitioning from state i to state j after n time points as $P_{i,j}^n$ (Gallager, 2013). This procedure was repeated for all initial states for n = 8 time steps (as probability estimates stabilized at this number of time steps) to produce transient probability estimates of all emotional states (these curves are shown in Figure 3). To evaluate the fidelity of these predictions, Spearman correlations were computed between model predictions and observed probabilities for each of the eight time steps, evaluating the similarity in the rank probability of the seven emotional states. This analysis was performed for both of the test data sets (held-out DNS participants, n = 299, and NKI-RS, n = 207), and comparisons were made against outputs from the null autocorrelation model above. Differences in correlation coefficients (between the empirical Markov model and the synthetic autocorrelation model) were made by taking the Fisher transform of Spearman correlations and performing paired t tests.

Figure 3. .

Predicted and observed probabilities of being in an emotional brain state during rest. Colored curves indicate the probability of being in one of seven emotional states over time after starting in a given state (organized by columns). For instance, the first column of curves shows the probability of being in each emotional state after starting from a state of contentment. Note that there is a tendency for the brain to remain in the same emotional state (self-transition) for several seconds. When starting from an emotional state, if the brain transitions to another state (other-transitions), it is most likely to transition to a neutral state (indicated in pink) rather than another emotional state. Thin lines depict the standard error of observed data.

Evaluating State Dynamics and Their Relation to Psychopathology

One feature of a dynamic model is the tendency for a system to transition to the same baseline state, as opposed to transitioning rapidly among multiple states. States that behave in this way act as hubs by facilitating transitions among emotions in an organized way. We quantified the extent to which emotional brain states acted as hubs by computing the proportion of transitions to each state that originated from a different emotional state, as opposed to a self-transition. Given evidence that emotion dynamics play an important role in mental health and well-being (Houben et al., 2015), we additionally examined the relationship between parameters of Markov models and psychopathology, dichotomized as “present” versus “absent” (for criteria and methods for diagnosis, see the Participants section). We further evaluated emotional inertia by comparing the likelihood of self-transitions, which reflects the propensity to remain in the same emotional state over time, as a function of mental health diagnosis. Comparisons of the proportion of off-diagonal transitions to neutral states (emotional resetting) and emotional intertia between clinically diagnosed and healthy participants were made using Wilcoxon rank sum tests. This test was chosen because the proportions being compared have a restricted range between 0 and 1 and do not meet the distributional assumptions of a two-sample t test. Because of the transdiagnostic nature of the mental health status grouping, the presence of more than one disorder in many participants, frequent transitions between disorders across the lifetime (Caspi et al., 2020), and the lower number of participants with psychopathology in the sample, we averaged the inertia scores across all emotions and diagnoses.

RESULTS

Emotional brain dynamics were better characterized by our discrete-time Markov chain model than a model based on the inherent autocorrelation present in fMRI measures of resting-state brain activity (Figure 2A). The average log likelihood of the empirical Markov model was higher than the autocorrelation model for test data in independent participants from the DNS sample (ΔL = 357.28, 95% CI [81.90, 899.4]) and NKI-RS data set (ΔL = 366.15, 95% CI [116.8, 876.9]). A BIC analysis that compared models based on their likelihood, while penalizing for complexity, provided strong support (ΔBIC scores > 10) for the empirically derived Markov chain in both test sets. The BIC of Markov chains derived from real time series were substantially larger than those of a distribution of synthetic models approximating the autocorrelation in the time-series data (DNS: ΔBIC = 714.57, 95% CI [163.8, 1798.8]; NKI-RS: ΔBIC = 732.31, 95% CI [233.6, 1753.4]; Figure 2B). These results indicate that the human brain exhibits emotional brain dynamics that are present in multiple independent data sets and that they cannot simply be described in terms of temporal autocorrelation inherent in fMRI data.

To quantify which aspects of brain dynamics differ from pure autocorrelation, we tested for differences in the magnitude of parameter estimates in our Markov model. In particular, we evaluated whether transitions from one emotional state to another differ from one another (i.e., testing for column-wise differences in off-diagonal parameters of the transition matrix), as autocorrelation alone would predict that transitions among different states are equally likely. Examining differences in these parameter values in the DNS development sample revealed strong differences between emotions (χ² = 174.03, p ≤ .0001, Friedman's ANOVA). Post hoc comparisons revealed that the proportion of transitions to neutral states were the greatest (M = .60, SD = .11) compared with the proportion of transitions to other states (M = .51, SD = .07; all pairwise comparisons p < .0001, corrected by Tukey–Kramer tests). This result indicates that even though the emotion-specific information maps were not optimized to do so, they exhibit intrinsic dynamics that are consistent with classic views positing that emotions emerge independently against an affectively neutral background state (Lewis, 2005). Neutral brain states thus serve as a common transit hub between emotions (Figure 2C).

To assess how well the Markov model accounts for transient changes in emotional states, we used it to estimate the probability of transitioning to each emotional state from every possible initial state. This was accomplished by exponentiating the transition matrix estimated from our training data, with an exponent equal to the number of time steps in the future. This procedure yielded a series of probabilistic estimates of transitions from one state to another at a future time (Figure 3). These profiles of transitions were similar to those observed in the test data sets (DNS: mean Spearman's r = .44, t(298) = 62.02, p < .0001; NKI-RS: mean Spearman's r = .42, t(206) = 43.08, p < .0001), more so than simple autocorrelation models (DNS: Δr = .17, t(297) = 21.74, p < .0001; NKI-RS: Δr = .10, t(206) = 13.69, p < .0001). Thus, our Markov model captures moment-to-moment changes between brain states that generalize across samples and captures information beyond time-invariant or static differences in the activation of emotion-related neural substrates.

To establish the reliability of the Markov model, we conducted a test–retest reliability analysis examining the similarity of model parameters estimated on the first and second scanning runs of the resting-state period in the model development sample, as participants were run in two consecutive 256-sec blocks (for more details, see the Methods section). This analysis revealed excellent levels of reliability for the group level model, ICC(2, 1) = .9509, 95% CI [.94, .97], which was computed as the average transition matrix of 200 participants. When model parameters are based upon transitions from a single participant, they are more variable but still have fair reliability, ICC(2, 1) = .48, 95% CI [.43, .49]. Repeated random subsampling of the group model revealed that test–retest reliability reaches ICC(2, 1) = .9 when models are averaged across 17 participants and plateaus at approximately ICC(2, 1) = .95 at 100 participants.

Given the evidence that more variable and less structured emotion dynamics are associated with lower levels of well-being and impaired mental health (see the Introduction section), we additionally evaluated whether the structure of intrinsic emotional brain dynamics differed as a function of mental health status based on DSM-IV criteria (see the Methods section for details). In particular, we tested whether the proportion of transitions from emotional to neutral brain states (what we call “emotional resetting”) differed as a function of clinical status. Tests for differences revealed that transitions to neutral states were less common in individuals with psychopathology (DNS: n = 68, M = 0.57, SD = 0.01; NKI-RS: n = 26, M = 0.52, SD = 0.02) compared with healthy individuals (DNS: n = 231, M = 0.60, SD = 0.006; NKI-RS: n = 179, mean proportion = .59, SD = .008) in both validation samples (DNS: Wilcoxon rank sum test: z = −2.17, p = .03; NKI-RS: z = −2.61, p = .009; Figure 4A). These results suggest that emotional resetting may serve an affect regulation function. Furthermore, we tested whether individuals with psychopathology presented with greater inertia in their emotional brain patterning. Using self-transition probabilities as an index of inertia, we found that individuals with psychopathology (DNS: n = 68, M = 0.49, SD = 0.06; NKI-RS: n = 26, M = 0.48, SD = 0.08) had greater inertia than healthy individuals (DNS: n = 231, M = 0.47, SD = 0.06; NKI-RS: n = 179, M = 0.44, SD = 0.07) in both validation samples (DNS: Wilcoxon rank sum test: z = 2.02, p = .04; NKI-RS: z = 3.59, p < .0001; Figure 4B). Because of comorbidities and the limited number of patients per diagnostic category, it was not possible to further decompose these effects by diagnosis (e.g., to test whether inert sadness was selectively associated with depression).

Figure 4. .

Emotional resetting and inertia distinguish resting-state brain activity patterns in healthy individuals from those with mental health disorders. (A) Across two validation cohorts, healthy individuals exhibited a greater likelihood to transition from an emotional brain state to a neutral one (emotional resetting) than individuals with a mental health diagnosis. Proportions indicate the number of transitions to neutral states from emotional states divided by the total number of transitions to neutral. (B) Across two validation cohorts, individuals with a mental health diagnosis exhibited greater inertia in their spontaneous brain activity patterns, as quantified by a propensity to remain in the same emotional brain state over time (i.e., ${\sum }_{i=1}^7$ P_i,i for emotion states i in transition matrix P). Proportions represent the number of self-transitions across all emotions. Circles correspond to individual participants. Dark gray areas indicate the sample standard deviation. Light gray areas indicate the SEM. Solid black lines denote sample means. *p < .05, **p < .01, ***p < .0001.

As discussed in the Introduction section, it has been recently called into question whether emotion dynamics as measured by EMA predict well-being above and beyond static behavioral measures of affect (Dejonckheere et al., 2019). Based on this concern, we additionally ran a set of control analyses that accounted for the contribution of static measures of neutral signaling in the brain. We fit logistic regression models predicting clinical status on the basis the proportion of transitions to neutral brain states that originate from emotional brain states (emotional resetting) and two additional covariates: (1) the overall frequency of neutral states and (2) the variance of neutral states, which could potentially mediate the effect of dynamics. This analysis revealed that emotional resetting uniquely contributed to the prediction of mental health (DNS: $\widehat{\mathrm{\beta }}$ = −11.68, SEM = 5.52, t(295) = 2.11, p = .03; NKI-RS: $\widehat{\mathrm{\beta }}$ = −6.17, SEM = 2.03, t(203) = 3.05, p = .002). Thus, the relationship between transitions from emotional to neutral brain states and mental health is not likely a byproduct of healthy individuals having less expressive or variable emotional brain systems at rest but is a meaningful property of emotion dynamics that may have a regulatory function.

DISCUSSION

The ebb and flow of thoughts and memories during resting wakefulness triggers emotions whose spontaneous expression may yield insights into individual differences in affect and risk for psychopathology. However, it has been challenging to derive objective metrics of the brain dynamics that mediate such changes in incidental affect, which take place in the absence of external stimulation. Here, we leveraged our prior machine-learning approach to fMRI-based decoding of discrete emotions and combined it with stochastic modeling to characterize how the brain transitions among several emotional states while participants mind wander at rest.

Using a Bayesian model comparison that penalizes complexity, we found that our Markov process model characterizes resting-state emotional brain dynamics in two large out-of-sample cohorts. To provide a robust estimate of generalizability, we validated the model across cohorts that differ in age and education level, and where the resting-state fMRI data were collected on different MRI scanners using different imaging parameters and resting-state paradigms (e.g., eyes open/eyes closed). Our model performed better than a synthetic “null” model based on the inherent autocorrelation in fMRI time series data. This improvement in performance indicates that our model captures regularities in how the brain transitions between emotional states rather than simply capturing a general tendency of the brain to persist in the same state over time.

Overall, self-transitions were more likely than transitioning to another state, which implies that spontaneous emotions have an intrinsic temporal duration in their brain signaling. Our Markov process model predicts that states tend to last for about 5–15 sec before transitioning to another state. These average durations do not necessarily correspond to subjective experience during the resting state, as individual participants may repeatedly return to a specific brain state over time after brief excursions to different states (see Figure 1C, where the individual gradually transitioned from a predominantly sad brain state to a predominantly neutral brain state over a period of minutes). Subjective experience and self-report measures of affect are known to involve variable weight of events that take place over extended timescales, with an emphasis on recent and extreme events (e.g., Varey & Kahneman, 1992). Our Markov model was designed to describe brain dynamics on the order of seconds, rather than that of self-report measures of affective experience which fluctuate over longer timescales. Accordingly, we believe our model captures fluctuations in ongoing neural processes that contribute to conscious experience (e.g., changes in “first-order” representations, see LeDoux & Lau, 2020; LeDoux & Brown, 2017) rather than characterizing the dynamics of experience directly. Other modeling schemes that incorporate time continuously (Suchard, Weiss, & Sinsheimer, 2001) and/or model hierarchical temporal structure (Fine, Singer, & Tishby, 1998) may produce dynamics that more closely correspond to those present in self-report measures of emotion.

When other-transitions did occur, they most often went to a neutral state irrespective of the specific emotional start state—an affective chronometry feature that we introduce here as “emotional resetting.” Considering affective state dynamics as a graph derived from the transition matrix (Dabrowski & Hunt, 2011; Jarvis & Shier, 1999) permits relating states and transitions in a Markov process to nodes and edges in a network from a graph theoretic perspective. Neutral brain states occupied a central hub position in the affective state network (Figure 2C). Hubs serve to efficiently connect otherwise distant nodes in a network. They also confer resilience against random failures in networks, because many nodes can be disrupted without altering the overall network structure. At the same time, hubs can potentially lead to catastrophic failures if they are damaged because of their central position. In the case of emotional brain dynamics, disrupting the neutral hub would impair the ease with which brain states fluctuate between emotions, which would be relatively spared if failures occurred in other nodes. This network configuration is consistent with dynamical systems models of emotion and large-scale neural networks, which often efficiently organize around a baseline attractor state or small number of metastates (Vidaurre, Smith, & Woolrich, 2017; Lewis, 2005).

Emotional resetting to a neutral state was less likely to occur in individuals diagnosed with a mental health condition (Figure 4A), which suggests that this characteristic of brain dynamics may play a role in adaptive emotion regulation. Indeed, emotional resetting predicted mental health status over and above the mean frequency and variance of neutral states. Hence, this dynamic feature associated with psychopathology is not likely to be explained by a generic failure to engage or modify neutral states. This result stands in contrast to a meta-analysis of 15 EMA studies, which concluded that dynamical measures did not add uniquely to predictions of emotional well-being (Houben et al., 2015). This discrepancy might relate to the importance of sampling emotions at a more refined temporal scale than typical EMA studies, the value added of continuous neurophysiological measures relative to spaced self-report data, and/or the improved sensitivity afforded by our combined machine learning and computational modeling approach.

By contrast, emotional inertia was more characteristic of individuals with psychopathology than healthy participants (Figure 4B). We speculate that repetitive thinking in the form of rumination and/or worry may play a role in prolonging the duration of these emotional brain states in individuals with psychopathology. Basic research on emotion duration estimates has shown that rumination is correlated with the duration of discrete emotions present in autobiographical memories (Verduyn & Lavrijsen, 2015). Ruminative thinking is applied to both positive and negative emotions as a form of emotion regulation; it is exacerbated in individuals with depression, anxiety, and other psychiatric disorders, and it predicts depression onset and interacts with negative cognitive styles to predict depressive symptom duration (Nolen-Hoeksema et al., 2008). Resting-state fMRI provides an opportunity for such repetitive thought processes to emerge in the context of a prolonged period of unconstrained task demands. The opposing patterning of emotional resetting and emotional inertia across individuals with and without psychopathology support theories emphasizing a key role for momentary affective state dynamics, interplay, and reinforcing loops at this timescale for informing psychiatric trajectories (Wichers, 2014). Overall, our findings indicate that it is possible to decode and predict the spontaneous affective dynamics of the human brain over a period of minutes and that such metrics may have utility in clinical applications. It may be particularly interesting to incorporate these neural metrics as outcome measures in affective intervention trials, as there are currently no well-validated methods for assessing changes in brain activity that index the spontaneous fluctuations of specific emotions.

Most prior neuroimaging dynamics work has used fMRI, EEG, or MEG to measure changes to eliciting emotional stimuli at very short timescales (hundreds of microseconds to a few seconds), which likely reflect initial cognitive and affective processes recruited to attend and appraise external emotional stimuli. Here, we extend such analyses to discrete emotions elicited by spontaneous thought over a timescale of seconds to minutes that more closely approximate the time frame of emergent subjective affective states. Recent advances in computational modeling, machine learning, and multivariate analytical approaches have enabled new insights into the neurocognitive processes that mediate facets of emotional experience while individuals engage with dynamic naturalistic stimuli like movies, music, narratives, and videogames (for reviews, see Jääskeläinen, Sams, Glerean, & Ahveninen, 2021; Kragel & LaBar, 2016). Although these efforts demonstrate that it is possible to track affective features from complex, temporally extended naturalistic stimuli, the studies have largely neglected to characterize the dynamic transitions from one emotion to another and have left open the question of modeling intrinsic dynamics during mind-wandering in the absence of exteroceptive stimuli. Our results contribute to the growing body of research highlighting the importance of dynamic constructs and computational modeling approaches in elucidating brain function relevant for emotion processing and for differentiating psychiatric illness.

Our study had several limitations. We used data from existing convenience samples to develop and test the reliability of the computational model. Because of the smaller proportion of the study population with mental health diagnoses, the variability of clinical presentation profiles, and the co-occurrence of multiple diagnoses, we could not evaluate the disorder specificity within the clinical sample, the association of specific disorders with alterations in processing specific emotions, or any gender differences. Although assessments revealed excellent levels of reliability even at relatively small sample sizes (n = 17), evaluating dynamics for individual participants will require improvements, either by increasing the duration of resting-state scans, combining data across resting-state and task scans, further reducing potential sources of error (e.g., subject motion or thermal noise), or adapting model estimation procedures to optimize reliability explicitly (Elliott, Knodt, & Hariri, 2021). Additionally, the archival data sets we used for analyses did not contain an independent measure of emotional state (e.g., self-report or recording of autonomic nervous system activity) that could be used to corroborate the emotion dynamics measured from brain activity. Finally, we only tested a small number of basic emotions. Work examining a larger number of emotion categories have revealed gradients of experience (e.g., from anxiety to fear to horror) that are well suited for dynamic analysis (Cowen & Keltner, 2017). Despite these limitations, we feel that our affective computing approach takes an important step forward in characterizing how the human brain spontaneously transitions across emotions at rest, with implications for identifying profiles of maladaptive intrinsic dynamics that contribute to psychopathology.

Acknowledgments

This work was supported by NIH R01 DA033369 and NIH R01 AG049789 to A. R. H. and NIH R21 MH098149 and NIH R01 MH124112 to K. S. L.

Reprint requests should be sent to Kevin S. LaBar, Department of Psychology and Neuroscience, Center for Cognitive Neuroscience, Duke University, Box 90999, Durham, NC 27708-0999, or via e-mail: klabar@duke.edu.

Author Contributions

Philip A. Kragel: Conceptualization; Data curation; Formal analysis; Investigation; Methodology; Validation; Visualization; Writing—Original draft; Writing—Review & editing. Ahmad R. Hariri: Data curation; Funding acquisition; Investigation; Project administration; Resources; Supervision; Writing—Original draft; Writing—Review & editing. Kevin S. LaBar: Conceptualization; Funding acquisition; Project administration; Resources; Supervision; Writing—Original draft; Writing—Review & editing.

Funding Information

Ahmad R. Hariri, National Institute on Drug Abuse (https://dx.doi.org/10.13039/100000026), grant number: 1 R01 DA033369. Ahmad R. Hariri, National Institute on Aging (https://dx.doi.org/10.13039/100000049), grant number: 1 R01 AG049789. Kevin S. LaBar, National Institute of Mental Health (https://dx.doi.org/10.13039/100000025), grant numbers: 1 R01 MH124112, 1 R21 MH098149.

Diversity in Citation Practices

Retrospective analysis of the citations in every article published in this journal from 2010 to 2021 reveals a persistent pattern of gender imbalance: Although the proportions of authorship teams (categorized by estimated gender identification of first author/last author) publishing in the Journal of Cognitive Neuroscience (JoCN) during this period were M(an)/M = .407, W(oman)/M = .32, M/W = .115, and W/W = .159, the comparable proportions for the articles that these authorship teams cited were M/M = .549, W/M = .257, M/W = .109, and W/W = .085 (Postle and Fulvio, JoCN, 34:1, pp. 1–3). Consequently, JoCN encourages all authors to consider gender balance explicitly when selecting which articles to cite and gives them the opportunity to report their article's gender citation balance.