A computational model for learning from repeated traumatic experiences under uncertainty

Abstract

Traumatic events can lead to lifelong inflexible adaptations in threat perception and behavior which characterize posttraumatic stress disorder (PTSD). This process involves associations between sensory cues and internal states of threat and then generalization of the threat responses to previously neutral cues. However, most formulations neglect adaptations to threat that are not specific to those associations. In order to incorporate non-associative responses to threat, we propose a computational theory of PTSD based on adaptation to the frequency of traumatic events using a reinforcement learning momentum model. Recent threat prediction errors generate momentum that influences subsequent threat perception in novel contexts. This model fits primary data acquired from a mouse model of PTSD, in which unpredictable footshocks in one context accelerate threat learning in a novel context. The theory is also consistent with epidemiological data showing that PTSD incidence increases with the number of traumatic events, as well as the disproportionate impact of early life trauma. Since the theory proposes that PTSD relates to the average of recent threat prediction errors rather than the strength of a specific association, it makes novel predictions for the treatment of PTSD.

Introduction

Computational psychiatry seeks to define psychiatric disorders in terms of fundamental algorithms for survival rather than only as pathological states (1–3). Quantitative models may allow personalization of mental health care, insight into the nature of the disorder, or predict the trajectory of symptoms (4–6. For example, depression has been conceived as an adaptation to periods of low reward availability (7). Similarly, hallucinations have been conceptualized as resulting from excessive weighting of prior expectations for auditory stimuli in a Bayesian model (8–9). One approach to describing a computational function of a neural system is using David Marr’s three levels of analysis (10) (Figure 1A), which seeks to map connections between computational goals, algorithmic procedures to achieve them, and the neurobiological substrate underlying these processes.

Figure 1 –

David Marr’s Levels of Analysis for computational neuroscience as applied to PTSD. (A) Definition of the three levels of analysis from ref. 7. (B) Application of those levels to associative learning (left) and non-associative learning (right) in PTSD. (left) Associative learning is a well-characterized system with a clear computational goal of ethological relevance (Computational), a mathematically defined formal model (Algorithm), and neural circuit mechanisms (Implementation). (right) Non-associative learning is less well-understood but in this study conceptualizes linkages between non-associative Bayesian estimation of trauma and RL models.

Posttraumatic stress disorder has a computational description that organizes theory and neurobiological data across Marr’s three levels - associative fear learning (Figure 1B, refs. 11–15). Threat learning models have been successfully applied to PTSD and underlie current conceptualizations of the disorder and treatment options (16–17). PTSD is seen as an extreme outcome of associative fear learning, which in turn is a fundamental mechanism for predicting threats based on previous experience (18). In this model, PTSD occurs when life-threatening situations create potent associations between sensory reminders of the traumatic event and the emotional experience of fear (17). The intensity of this association then motivates a person to avoid (19) future trauma cues, limits extinction of the fear memory (20), and supports the subsequent formation of new fear memories via generalization and second-order conditioning (21). This process can be described mathematically, enabling learning parameters to be precisely measured during new associative learning in a laboratory setting (18). The precision with which associative learning can be controlled has enabled neurobiological studies into circuit mechanisms in both humans and animals (22).

In contrast, non-associative learning – increases (sensitization) or decreases (habituation) in response to a repeated stimulus (23) – is a prominent component of PTSD that lacks a formal algorithmic description (Figure 1B). In humans, repeated traumatic events increase the probability of developing PTSD and may change the nature of the disorder (24–25). However, the computational and neural processs ethat mediates this gradual increase in PTSD risk with trauma burden are not well understood. Core PTSD symptoms, such as hyperarousal, inherently involve an exaggerated response to sensory cues – importantly, these cues need not be associated with the traumatic event to trigger the response (26) but may result from sensitization of neuromodulatory systems (27, but see also 28 for habituation responses in PTSD). Neurobiological studies in animals have shown that stress enhances both innate defensive behaviors (29) and learning about unrelated fear cues (30). There are conceptual models of how habituation and sensitization occur (Dual Process Theory, ref. 31; Wagner-Koniorsky Theory, ref. 32), which center the role of arousal in changing the response to a stimulus with repetition. However, these models lack the algorithmic detail and clear relation to survival value t Rescorla-Wagner and related reinforcement learning (RL) models (33). This has limited the ability to parametrically manipulate and therefore understand non-associative learning in PTSD patients and animal models.

These non-associative adaptations are essential because an organism must estimate the threat of violence to adapt to it. This process of estimation must necessarily involve information gathered across timescales, since threat may increase suddenly or may increase over long periods (39). Longer timescale estimation of threat involves integrating experience in disparate environments.

To consider a concrete example: predator attacks are stereotyped events which carry a significant probability of death (20% for mice exposed to an owl, ref. 40). Life History Theory explains this by positing that stressful experiences in childhood provide information about organismal strategies that will be adaptive in the adult environment (42). Childhood traumatic experiences have a strong impact on adult brain structure and function which influence the development of PTSD (41). Previous approaches to computational modeling of PTSD have focused on defining changes in associative learning after traumatic experience (11–15). PTSD is thus framed as a consequence of underlying mechanisms for predicting threat based on previous associations. In contrast, we were interested in defining how PTSD arises from the combination of associative learning and an agent’s estimation of the frequency of traumatic events. An agent has to estimate the statistical relationship between experiences of trauma based on prior experience which can be considered as a Bayesian process.

Here, we conceive of that Bayesian process as linking non-associative and associative learning in simulations and in model-fitting to primary mouse behavioral data in a model of PTSD. First, we posit an ecological role for non-associative learning in estimating the frequency of predator attacks (or other violence) across life history. In these simulations, we apply a Bayesian approach to understand how well an ideal agent could estimate predation risk from its own life history and consider the relationship to early life stress. Second, we acquired behavioral freezing data in a mouse model of PTSD which is characterized by non-associative sensitization to threat in a new context. We compare this behavioral data with classical RL models and with a recently developed RL momentum model (7,34–36).

Results

This study utilizes a combination of simulation and analysis of primary mouse stress data in a mouse behavioral model to understand PTSD behavior. A Bayesian model is used to understand, via simulation, how an agent can estimate the rate of traumatic attacks over its lifetime. The mouse behavioral model is stress-enhanced fear learning (SEFL), in which unpredictable footshocks in one context lead to longlasting sensitization to another weak footshock in a new context (Supplementary Figure 1). Reinforcement learning models can be used to compare learning processes and to integrate non-associative learning (about the frequency of threat) with associative learning (about the associations of threat; refs. 7,34–36). Further simulations of this model have implications for treatment and future research into the neurobiology of PTSD.

PTSD as trauma rate estimation (Simulation)

If the probability of death is high, then the animal will experience few attacks before dying (Figure 2B). In this information-poor environment, the animal must maximize the available information in estimating the rate of such attacks. In order to determine how well an ideal observer could do under such conditions, we constructed a simple probabilistic model with a fixed probability of attacks $p_a$ and probability of dying per attack $p_d$ at each time point (Figure 2A).

Figure 2 –

A Bayesian observer can measure the rate of traumatic attacks more easily than lethality. (A) Schematic of a simplified doubly stochastic model of attacks (i.e., traumatic events). Attacks occur stochastically at each timepoint with a fixed probability $p_a$ . Conditional on attacks occurring, agents die with probability $p_d$. If agents survive, they estimate the ongoing probability of attacks according to Bayes’ rule. (B) Agents must estimate $p_a$ and $p_d$ in an information-poor environment. The number of attacks experienced by the typical agent is low, usually 3–5 over the course of a lifetime for $p_a=0.2$, a typical value for the lethality of predator attacks (30). (C) The Bayesian estimator of $p_a$ and $p_d$ for a typical example sequence of attacks shows tight convergence for $p_a$ and non-convergence for $p_d$. The wider spread in $p_d$ is indicative of the inability of the Bayesian agent to estimate the lethality of attacks. (D) As the agent continues over its lifetime (red to blue map), the estimate of $p_a$ slowly narrows (vertical lines, 95% intervals). Greater time allows the agent to accumulate greater evidence about the true value of $p_a$.

Even with a flat prior, $p_a$ could be estimated with a high degree of precision by the end of the lifetime of the animal (Figure 2C). Variance in $p_a$ decreases progressively over the lifetime of the animal (Figure 2D). In contrast, infering the probability of dying on each attack ($p_d$) with as much certainty (Figure 2C; Supplementary Figure 1). This observation has an intuitive explanation – the organism knows it survived n attacks, but not how many organisms did not survive. The limited estimability of $p_d$ has further implications – each attack contributes to attack rate estimate (${\widehat{p}}_a$) and therefore the behavior of the agent. However, since individual attacks are of less defined lethality, the organismal response to them ought to be stereotyped.

The disproportionate impact of early life stress (ELS) on adult behavior (39) is explained by the Bayesian trauma rate model. We evaluated the Bayesian trauma rate estimator in two scenarios with the same total number of traumatic events, one in which traumas occur early in life (ELS) and one in which they are spread across the lifespan (Figure 3A). The number of attacks was equal in both cases. Variance in ${\widehat{p}}_a$ decreases with time in both models, as traumatic events reduce uncertainty in the true rate of violence (Figure 3B). However, over the course of the lifespan the ELS model shows a higher estimated rate of violence (${\widehat{p}}_a$). Thus, the increased response to ELS does not require specialized critical period mechanisms, but instead arises naturally in a normative estimator of violence rate.

Figure 3 –

Early life traumas have a disproportionate effect on the estimated attack rate. (A) Characteristic examples of two distributions of attack frequencies. In the random attack model (uniform attack probability), attacks are uniformly distributed across the lifespan. In the early life stress model, an identical number of attacks are uniformly distributed across the first half of the lifespan. (B) Two Bayesian agents (one for early life stress, one for random attack) posterior distributions for attack rate sequentially measured across the lifespan, for the random and early attack models. The discrepancy between estimated and true attack rate (p_a=0.01) is greatest at the start of life due to a higher density of attacks in the early life stress model. Over the course of the lifespan, these two models converge.

RL momentum model (Behavior data and simulation)

In this section, we compare a recently proposed RL momentum model (7,34–36) with a new data behavioral freezing behavior in a mouse PTSD model. Classical RL models, such as temporal difference learning (Figure 4A), enable an organism to associate threatening experiences with the context in which they are experienced. However, threats in one context do not influence threats in another (Figure 4A). In contrast, in the RL-momentum model, traumatic events occurring close in time but in unrelated environments contribute to a slowly varying momentum term (Figure 4B). Momentum carries information about recent threats, allowing the agent to correctly assess risk in a changing environment. We determined how the ideal length of time for momentum to persist depends on how long threats persist using simulation (Figure 4C). When attacks are uncorrelated in time, there is no advantage to momentum and the optimal momentum learning rate ($\nu$) is zero. When attacks are correlated (Figure 4C, light blue), a substantial improvement in threat estimation can be obtained by including the momentum parameter. The long-time scale of optimal threat adaptation offers a potential explanation for the persistence of PTSD symptoms. If threat momentum, rather than the specific association with the initial traumatic event, were the source of PTSD symptoms, then this would have substantial implications for the understanding of PTSD.

Figure 4 –

Reinforcement learning with momentum allows improved estimation of autocorrelated attack rates. (A) Single traumatic events occur in different environmental states (contexts), leading to increased associated threat according to the RL model. (B) In the RL momentum model, the same series of attacks produces momentum which couples threat across contexts. Context C threat is due to momentum since the animal receives no footshocks in that state. (C) The momentum learning rate term of the RL momentum model enables extraction of information about fluctuating attack rates. Autoregressive attack rates were produced as shown in figure 3 to produce n=10000 simulated attack sequences (light blue, highest autoregression to dark blue, lowest autoregression). All attacks occur in a different context. In the absence of momentum, the agent cannot extract information about fluctuations in underlying attack rate. With higher momentum, the agent can extract information about the underlying attack rate fluctuations.

To test this idea, we induced stress in a mouse model of PTSD (Stress-Enhanced Fear Learning; SEFL) and compared the performance of temporal difference learning (RL model) and a momentum model (RL momentum model) in explaining defensive behavior (Figure 5). In this model, mice receive unpredictable footshocks in one context (Context A) and then show sensitized threat responses to a single footshock in another context (Context B) later (Figure 5A, top). The RL momentum model fits the observed freezing behavior (Figure 5A, bottom) well, showing a disproportionate freezing response to the single footshock in a novel context. This sensitized freezing behavior can be explained by the momentum term in the model, which links the threat prediction errors produced across contexts.

Figure 5 –

RL momentum fits threat behavioral data in a mouse model of PTSD. (A) Example mouse behavioral data across three days of in the stress-enhanced fear learning model of PTSD (upper), along with RL momentum fit to behavioral data (lower). (upper left) Freezing across 90 minutes (red) of exposure to 15 unpredictable footshocks (black; 1mA, 1s). (upper center) Freezing across subsequent exposure to 1 uncued footshock in a new context. (upper right) Freezing during re-test in the new context (lower left) Threat according to maximum likelihood model fit of the RL momentum model (threat associated with context A – blue, context B- green) on day 1, (lower center) day 6, and (lower right) day 7. (B-C) Averaged freezing data across (n=17 stressed, n=18 controls) on days 6 (B) and 7 (C). (D) Model comparison between classic RL model and RL momentum model for SEFL mice (n=17 stressed, n=18 controls). Bayes information criterion (BIC) was calculated (see Methods) for maximum likelihood fits of the RL model and RL momentum model for either unstressed animals (0 shocks on day 1) or stressed animals (15 shocks on day 1). Difference in BIC between the two models is shown for individual animals (gray dots; black dot for example data from (A)), mean BIC difference per condition as bars (blue – unstressed, pink – stressed).

We compared Maximum Likelihood fits between the RL and RL momentum models (n=18 unstressed, n=17 stressed mice), using the Bayes Information Criteria (BIC; Figure 5B). When the momentum learning parameter ($\nu$) is zero, the two models are equivalent, but the the RL momentum model has a greater number of parameters (4 for RL momentum, 2 for RL model). Since the BIC penalizes the number of parameters, this produces model fits where the RL model is preferred (for unstressed mice, RL model was preferred in 17/18 animals). For stressed mice, however, the BIC strongly favored fits from the RL momentum model (14/17 animals). The RL momentum model predicts greater freezing in a novel context in stressed animals than the RL model, which accounts for the improved predictions over the RL model. This model also showed improved predictions as compared with an application of the Bayesian model of attack rate estimation (Supplementary Figure 2).

Implications for the treatment of PTSD (Simulation)

The RL-momentum model of PTSD presents an additional learning mechanism by which PTSD symptoms may be ameliorated. In the classical RL model of PTSD, extinction learning (Figure 6A) works to reduce PTSD by generating small prediction errors when the agent is re-exposed to the traumatic context. This approach underlies evidence-based psychological therapies for PTSD, such as prolonged exposure and cognitive reprocessing therapy. The RL momentum model retains extinction of learned associations, but the threat prediction errors generated by extinction also generate negative momentum that reduces responses to novel threats (Figure 6B). This model also offers a novel perspective on treatment failure of exposure therapy in PTSD.

Figure 6 –

RL momentum model offers a new perspective on mechanisms of extinction and symptom exacerbation in PTSD. (A) RL model: Two traumatic events in an initial context (context A; blue highlight) produce threat learning associated with that context (blue line) but no threat associated with a novel context (context B; green line) during exposure to that context (green highlights). Extinction occurs when exposure to the initial context A after the traumatic events causes threat prediction errors which decrease threat associated with context A (blue highlights, second and third exposures). (B) RL momentum model: Two traumatic events in initial context produce a momentum which increases threat in a novel context (green line). Re-exposure to initial threat context (context A; blue highlights) reduces threat associated with context A (blue line) but also reduces threat momentum (green line). Green dotted line shows counterfactual threat momentum if no re-exposure to context A had occurred). (C) RL momentum model demonstrates a novel explanation for relapse during exposure therapy. Exposure to smaller stressors (small lines) in a novel context increases threat associated with context B (green line) but also, via the momentum term, increases threat associated with the initial traumatic context A (blue line).

Current learning-based accounts of this phenomenon posit that individuals may experience extinction renewal or extinction resistance, in which either extinction fails to occur or in which the extinction memory may be specific to the context in which it was generated (e.g., the therapy session). In contrast, the RL momentum proffers a simple explanation – unrelated mild stressors generate threat momentum, which increases threat associated with the original traumatic context (Figure 6C). Similarly, an implication of this model is that exposure to novel threats independent of the traumatic context could reduce threat momentum. For example, an agent encountering an intense innate threat (e.g., standing on the side of a high cliff) without injury might experience a strong negative prediction error which would reduce threat momentum for the same reason as exposure to a cue associated to a traumatic event.

Discussion

We formulated PTSD as a learning process directed at estimating both the rate of trauma and the specific associations with the trauma. To do so, we combined simulations of two distinct models with model fitting to primary behavioral data. The Bayesian formulation of this problem treated the agent experiencing trauma as an ideal observer. We found that the rate of traumatic events could be estimated well, but that the lethality of traumatic events cannot be estimated even by an ideal observer. Early life trauma had disproportionate impact in this model even without specialized mechanisms for amplifying early life experience. We applied the reinforcement learning momentum model to PTSD, and found that RL-momentum performs well when violence is clustered in time. The slower the change in trauma rate, the more momentum contributes to optimal learning from traumatic stress. This model also offers a novel conceptualization of extinction learning, and suggests that exposure to unassociated strong threats could affect threat momentum. Understanding the impact of innate danger on threat momentum requires further modeling and empirical investigation, since exposure to innate threat could lead to either positive or negative changes in threat momentum.

Previous formal approaches to learning in PTSD have focused primarily on associative mechanisms. A simple model of a Bayesian observer experiencing potentially lethal attacks provided here demonstrates several features which match the epidemiology of PTSD. For example, the question of why early life experiences lead to such profound effects on behavior is explained as a natural outcome of estimation bias induced by clustered trauma in early life. These obsevations may explain experimental observations of sensitization to new threats by previous stress are often used to model PTSD (26,29–30). We show that stress sensitization of threat, a model of PTSD, is well fit by the RL-momentum model. However, our ability to precisely fit the parameters of the RL-momentum model is limited by the binary nature of the stress in this dataset. Full validation and parameter-fitting for the RL-momentum model will require more precise manipulations of the sequence of threat prediction errors over time.

A further limitation of this study is that we did not consider parameter regimes that may give rise to habituation (decrease in response to repeated stimuli). Both sensitization and habituation can occur in the RL-momentum model, depending on chosen parameters (7). In PTSD, habituation has recently been suggested as an outcome of repeated trauma (44), and may relate to the numbing symptoms in PTSD. Habituation and sensitization have been thought of as separate processes which competitively modulate responses to repeated stimuli (45). PTSD can involve both numbing and hyperarousal emotional reactions to stimuli after traumatic stress (45–47). A more complete model of the impact of a sequence of threat prediction errors on subsequent emotional responses may explain this apparent contradiction.

Future progress in understanding the role of non-associative learning in PTSD may depend on measuring the neural substrate of threat momentum (or estimated attack rate in the Bayesian model). Applying David Marr’s three levels of analysis to non-associative learning from threat (Figure 1), we have defined the computational problem (“predicting future threats based on a sequence of attacks”) that must be solved. We have compared two algorithms for accomplishing this goal: Bayesian attack rate estimation and RL momentum (Supplementary Figure 2). We find the RL momentum model offers a formal mathematical approach at the implementation level which explains clinical features of PTSD and behavior in a mouse model of PTSD. However, the implementation level of the RL momentum has not been identified.

Identifying PTSD with threat momentum may facilitate future neurobiological and translational studies of PTSD. Extensive work has shown that patients with PTSD have different learning rate parameters during fear and extinction learning (11–15) than controls. This study extends these findings by offering a model of how the sequence of threat prediction errors may generate other associative learning alterations in PTSD. The neurobiological correlates of threat momentum would be slowly varying summing functions of previous threat prediction errors which sensitize defensive behaviors, such as neuromodulatory systems (29) or molecular switches leading to persistent neural changes. Future extensions of this approach may link effects of arousal on learning rates (rather than overall threat) to averaged recent threat prediction errors, similar to Pearce-Hall learning (49). Thus, the present study may facilitate future work linking non-associative and associative mechanisms in PTSD. Such links are evident in behavioral and epidemiological data and have plausible biological mechanisms, but have previously lacked a computational model to facilitate the design of future experiments.

Methods

In this study, mouse freezing behavior was induced by contextual fear conditioning after a stress manipulation that models sensitization effects of PTSD. This behavior was compared with predictions of reinforcement learning models to identify model-based descriptions of the stress-induced effects on contextual freezing. Simulations were used to model longer lifetime estimation of danger in an agent experiencing traumatic attacks over time. These simulations were then fit to the estimation of a Bayesian model for these attacks to better understand long lifetime estimation problems related to PTSD.

Animal Behavior.

All procedures were carried out in accordance with the ethical guidelines of the National Institutes of Health and were approved by the Institutional Animal Care & Use Committee of Yale University. 8–12 week old C57Bl/6 male mice were stressed using using the Stress-Enhanced Fear Learning model (30), which has been shown to lead to long-lasting enhancement of fear and anxiety behaviors in both mice (30) and rats (38). This model consists of 15 unpredictable footshocks (1mA, 1s) with random intershock intervals between 4 and 8 minutes. For contextual fear experiments, a second context (Context B) was used on day 6, in a separate room with different ambient auditory, visual, tactile, and olfactory characteristics. On Day 6, a single 1mA 1s shock was administered after 5 minutes, and then freezing was assessed for 5 more minutes. On day 7, mice were returned to Context B for 10 minutes. MedAssociates boxes were used for all footshock experiments, and freezing was assessed as complete cessation of movement other than breathing (motion <18 a.u.) with automated VideoFreeze software.

Bayesian attack model.

At each time step, threat events (attacks) are binomially distributed with probability of attack $p_a=0.01$ for 700 time steps (Figure 2a). Deaths occur with probability $p_d=0.2$ contingent on an attack occurring. Agents that die are not included in the analysis, and in any comparisons attack rates are equalized to isolate the effect of estimators. The agent’s estimate of $p_a$ and $p_d$ is derived from the sequence of attack observations (${\mathit{x}}_{\mathit{t}}=0,0,1\dots 0$) according to Bayes’ rule

$$p\left(p_a,p_d\mid {\mathit{x}}_{\mathit{t}}\right)=\frac{p\left({\mathit{x}}_{\mathit{t}}\mid p_a,p_d\right)p\left(p_a,p_d\right)}{\int p\left({\mathit{x}}_{\mathit{t}}{p}_a,p_d\right)d{\mathit{x}}_{\mathit{t}}}$$ (1)

Bayesian model fitting procedure.

The model is fit using a Markov Chain Monte Carlo sampler with a flat prior at time t=0 with independent fitting performed at each time point for the events (attacks) up to that point. Specifically, an affine invariant ensemble MCMC sampler (MCMC Hammer, ref. 37) toolbox for Matlab with 30 walkers was used to estimate the posterior for these two parameters ($p_a$ and $p_d$), given the likelihood function

$$\ln (\left(\mathcal{L}\left({\mathit{x}}_{\mathit{t}},p_a,p_d\right)\right)={\text{Σ}}_{attacks}ln\left(p_a\left(1-p_d\right)\right)+{\text{Σ}}_{non-attacks}ln\left(1-p_a\right)+{\text{Σ}}_{death}\ln \left(p_a\ast p_d\right)$$ (2)

MCMC estimation is conducted with the ‘gwmcmc’ function in Matlab using the MCMC Hammer toolbox with ‘burnin’ .3 and ‘stepsize’ 2.

Autoregressive time series.

Autocorrelated attack rate time series were generated for an AR(1) autoregressive process

$$p_{a,t}=c{p}_{a,t-1}+\mathcal{N}(0,0.1),$$ (3)

where $p_{a,t}$ is the attack rate at time $t,c$ is a constant equal to the correlation of successive time steps, and $\mathcal{N}(\mu ,\sigma )$ is normally distributed noise with mean $\mu$ and standard deviation$\sigma$. Simulations used the arima function in Matlab. N=10,000 simulated lifetime attack rate time series were generated, then for each an agent’s experienced attack time series was generated and the MCMC Hammer estimator was then used to progressively estimate attack rates as above. At each time step, the estimator was progressively estimated to generate the available estimate for an ideal Bayesian observer with information available up to that time.

Reinforcement learning models.

In temporal difference learning, threat at time t in context c ($T_{c,t}$) is learned from a sequence of unconditioned stimuli ($u_t$) which produce prediction errors according to

$$T_{c,t}=T_{c,t-1}+\alpha \left(u_t-{\gamma }_1{T}_{c,t-1}\right),$$ (4)

where $\alpha$ is a learning rate and ${\gamma }_1$ is a decay rate constant. Equation 4 is referred to as RL model in the Results section, and describes the formation of associative threat learning. The addition of a momentum term (7) allows prediction errors from different states to influence one another according to an RL momentum model

$$T_{c,t}=T_{c,t-1}+\alpha \left(u_t-{\gamma }_1{T}_{c,t-1}\right)+f{m}_t,$$ (5)

where f is a scaling constant and $m_t$ is the momentum at time t. This momentum term is defined by

$$m_t=m_{t-1}+{\gamma }_2{\text{Σ}}_{c=\{A,B,\dots \}}\alpha \left(u_t-T_{c,t-1}\right)$$ (6)

in which the sum of decayed prediction errors across all contexts $c=\{A,B,\dots \}$ with momentum decay constant${\gamma }_2$. This can lead to either oscillatory behavior or slow summation of prediction errors across states depending on ${\gamma }_2$. Reinforcement learning models (RL – equation 4, RL with momentum – equation 5) were fit to smoothed freezing (sliding window, 15s) on days 1, 6, and 7. Inputs to the model were shock times and threat was fit for both Context A and Context B. Parameters for each model were fit using maximum likelihood estimation in Matlab, with threat variable rescaled to (0.1,0.9) to match freezing probability. Maximum likelihood fit was compared by calculating the Bayes Information Criterion (BIC) for RL and RL with momentum models at the single animal level for both stressed and unstressed mice.