Latent state and model-based learning in PTSD

Abstract

Posttraumatic stress disorder (PTSD) is characterized by altered emotional and behavioral responding following a traumatic event. In this article, we review the concepts of latent state and model-based learning (i.e., learning and inferring abstract task representations) and discuss their relevance for clinical and neuroscience models of PTSD. Recent data demonstrate evidence for brain and behavioral biases in these learning processes in PTSD. These new data potentially recast excessive fear towards trauma cues as a problem in learning and updating abstract task representations, as opposed to traditional conceptualizations focused on stimulus-specific learning. Biases in latent state and model-based learning may also be a common mechanism targeted in common therapies for PTSD. We highlight key knowledge gaps that need to be addressed to further elaborate how latent state learning and its associated neurocircuitry mechanisms function in PTSD and how to optimize treatments to target these processes.

Computational models of learning and decision-making in PTSD

Posttraumatic stress disorder (PTSD) is a heterogeneous mental health disorder generally characterized by intrusive recollections of the traumatic event, avoidance of the trauma memories and stimuli that elicit the trauma memories, broad changes in mood and cognition, and hyperarousal. PTSD is unique among mental health disorders by necessitating a precipitating environmental event in its definition: specifically, exposure to a traumatic event is required to meet diagnostic criteria. This requirement for a persistent and maladaptive reaction to an environmental event (or events) has intuitively led to reliance on basic models of stress responding and Pavlovian fear learning for understanding the development of PTSD in humans[1].

Although there have been advancements using basic stress models of PTSD [1–3], a somewhat parallel line of research attempts to explain multifaceted PTSD-related phenomena by appealing to computational learning models. These approaches fit within the broader scope of computational psychiatry, which seeks to apply computational science methodology toward understanding neural circuits and behavioral endophenotypes that distinguish or cut across diagnostic categories [4,5]. Investigation of computational biases in learning and decision-making in PTSD are growing (Box 1), with the promise of allowing better and more precise predictions about vulnerability, etiology, diagnosis, prognosis, and ultimately inform the development of better treatment options. Here, we focus on a specific emerging area of the learning and decision-making literature, latent state and model-based learning, and discuss their underlying neurocircuitry mechanisms, relationships with PTSD neurocircuitry and associated behavioral impairments, and emerging evidence for impairment in PTSD. We also discuss how these concepts may inform understanding of learning processes targeted by the dominant psychological treatments for PTSD.

Box 1. Computational modeling of learning and decision-making in PTSD.

Valuation.

Valuation is the primary computation of most reinforcement learning models, which refers to estimating the degree to which a cue is associated with an outcome in associative learning or the expected value of an action in instrumental learning. PTSD has been associated with decreased encoding of social reward value in a frontoparietal network compared to controls [95], as well as increased encoding of the absolute magnitude of subjective value for wins and losses in the ventral striatum among individuals with PTSD [96]. One study among combat veterans using a Pavlovian fear learning reversal task [97] found that encoding of threat value in the amygdala was negatively correlated with PTSD symptom severity[98]. By contrast, a separate study[32] of Pavlovian fear and extinction learning demonstrated that greater PTSD symptoms were associated with greater threat value encoding in a frontoparietal network during acquisition, yet with lesser threat encoding in a dorsomedial PFC network during extinction. These inconsistencies suggest a clear need for further research of value encoding in PTSD.

Prediction error.

The prediction error (PE) refers to a discrepancy between observed and expected outcome and is a fundamental teaching signal for learning the value of a cue or behavior[20,99]. PTSD has been associated with decreased encoding of reward prediction errors in medial PFC / striatum and anterior insula networks [100] and left temporoparietal junction[95]. During social reward learning, traumatized youth demonstrate decreased encoding of negative social reward prediction errors in a salience network [101,102]. Research has also failed to identify altered encoding of prediction errors during reward loss learning[103] and threat reversal learning[98] in PTSD. The inconsistency in biases in prediction error encoding in PTSD is noteworthy given the emphasis on generating prediction errors in many models of exposure therapy[104], suggesting a clear need for additional research.

Associability.

Associability refers to trial-by-trial salience for a given cue, gating the amount of learning that occurs for that cue based on the current statistics (e.g., variance) of the learning environment [105,106]. PTSD symptoms in two separate samples of combat veterans have been associated with heightened associability weights [98,103]. However, in one sample PTSD was associated with heightened encoding the anterior insula and ventral striatum extending towards dorsal amygdala [103], yet decreased encoding of associability in the striatum in the other sample [98]. Task differences may explain these discrepant results and point toward need of more research.

Latent State and Model-Based Learning

Latent state learning broadly refers to learning the abstract structure of an environment that governs the specific relationships between cues, behaviors, and outcomes, and is a form of model-based learning [6–10]. In early instances, the ‘model’ in model-based learning generally referred to the learner forming an internal model of the state transition probabilities in a task, allowing the learner to prospectively plan behaviors to traverse the state space of the task as needed to optimally pursue reward [11–13]. Model-based learning has often been discussed as an alternative learning system in contrast to modelfree learning [11,14]; however, more contemporary analyses make a less firm distinction between modelbased and model-free learning [15,16]. Nonetheless, the distinction between model-based and modelfree learning provides a useful heuristic with which to highlight some important aspects of model-based learning and its neurocircuitry encoding. Specifically, model-based learning involves the learner forming an internal model of the environment that governs relationships between contexts, cues, behaviors, and outcomes, thereby enabling prospective and flexible behavior in pursuit of goals.

Within this larger and generic category of model-based learning, the more specific class of latent state learning models have been developed. These models were initially geared towards solving a specific type of learning problem: understanding rapid renewal of responding following extinction that is not predicted by traditional model-free models [9,10,17]. Relevant to PTSD, this problem can be cast in the context of the rapid return of fear following extinction[18,19]. Standard models (e.g., Rescorla-Wagner [20]), that do not explain this behavior well, would predict gradual increases in fear responding as a cue predicts a shock in context A, followed by gradual reductions in responding in context B when the cue no longer predicts shock (i.e., extinction), followed again by gradual increases in responding following a return to context A where the cue again predicts shock. In reality, while there are gradual increases and decreases in context A and B during learning, a return to context A produces a sharp renewal of responding. This rapid renewal of responding is decidedly not predicted by standard ‘model-free’ learning that simply updates cached values of a cue or behavior based on trial-and-error [9]. Latent state models, by contrast, posit that the learner attributes cues, behaviors, and outcomes in a given context to an abstract state (or alternatively latent ‘causes’ [7,8]), allowing the learner to form a distinct cue/behavioroutcome relationship for each abstract state (e.g., unique associations are separately stored for a fear acquisition and fear extinction contexts). The learner infers which state is most probable given current evidence, and belief in a given state’s probability then governs whether the outcome is expected or not. The rapid renewal of fear responding would then be explained by the learner inferring a state of the world in which the cue predicts danger, thereby enabling rapid recovery of fear responding.

This concept of the learner forming distinct memories for the acquisition vs extinction associations is not new to the learning and memory literature [21], but these were among the first computational approaches to formalize a mathematical account explaining this phenomenon. It is also noteworthy that rapid renewal of responding after extinction that latent state learning models attempted to solve is highly similar to the observation of poor reversal learning following damage to the orbitofrontal cortex (OFC) [22,23]. Indeed, impaired representation of latent states in OFC has subsequently emerged as a dominant model explaining this behavioral effect as well [17]. Thus, latent state learning models have emerged as a powerful explanation for a broad array of learning and behavioral phenomena [6–8]. Recent models have been developed that specify computations for how many, and which, latent states a learner may believe are active at a given time [6,7]. Within the context of PTSD, these models enable tracking not just of fear or reward value associated with a cue or behavior, but tracking the individual’s model of the world that explains the value of the cue or behavior. Related developments are computational models explaining contextual inference [24,25]; that is, how a learner uses currently available information to infer abstract states and contexts and retrieve the appropriate memory. Importantly, and to be discussed in more detail below, conceptualizing the phenomenon that fear often returns following extinction from a latent state perspective reframes the problem for clinician/scientists concerned with extinguishing fear. That is, instead of asking the question of how to decrease fear responding to a threat cue, the question becomes how to increase the likelihood that the leaner infers a state of the world in which that cue no longer predicts danger. This is a subtle but important difference and has broader implications for treatment of PTSD beyond just extinction learning and exposure therapy.

Latent State and Model-Based Learning in PTSD

Emerging data suggests altered latent state learning in PTSD. One initial study among youth with varying severities of interpersonal violence compared fit of a latent state learning model to a standard Rescorla-Wagner (RW) model to behavior during a three arm bandit reward learning task, in which participants had to track which of the three arms were more likely to deliver reward and the contingencies changed every 30 trials [26]. Overall model fit to these data was better for the latent state model; further, estimates of updates in the learner’s beliefs about the latent states corresponded well with the changes in the structure of the task’s contingencies. Youth with severe histories of sexual abuse were found to demonstrate greater use of the RW model, versus the latent state mode, and they similarly demonstrated decreased frontoparietal encoding of updates in latent state beliefs. Consistent with the role of the frontoparietal network in updating abstract models of the learning environment [12], these data suggest severely traumatized youth are impaired in tracking latent characteristics of the learning environment. This impairment possibly suggests a higher order mechanism for broadly impaired learning and decision-making typically observed in trauma-exposed samples (e.g., impaired reward, extinction, and context learning in PTSD [27–29]).

Similar results have been found among adults with PTSD in the context of learning and extinction tasks. One behavioral study among trauma-exposed adults with varying PTSD symptom severities investigated latent state learning during a reward loss learning and extinction paradigm. Results suggested that those with greater PTSD symptom severity were less likely to attribute changes in task structure (i.e., an objective change in the latent structure of the learning environment) to different latent states [30], consistent with the view of decreased tracking and updating of the latent structure of the learning environment. A neuroimaging study of fear conditioning and extinction, both occurring on the same day, among adult women with PTSD [31] found better fit of a latent state learning model compared to a hybrid associability model and a standard RW model based on skin conductance data. Imaging data did not reveal that tracking of latent state changes in any neurocircuitry network was related to PTSD symptom severity. Instead, greater tracking of threat value (as estimated from the latent state model) in the frontoparietal network during conditioning and lesser tracking of threat value in a dorsomedial network during extinction were associated with PTSD severity. Overall, these emerging data [30–32] point towards altered tracking and updating of latent states in PTSD samples.

Two recent behavioral studies [33,34] also investigated the impact of PTSD on a more traditional form of model-based learning [11,16]. Here, model-based learning was demonstrated through use of the state transition probabilities of the task [11]. Individuals with PTSD demonstrated less model-based learning compared to controls [33]. Moreover, a subsequent study demonstrated that the presence of threat-related stimuli further disrupted use of the state transition probabilities when making decisions for reward among individuals with elevated PTSD symptoms [34]. These studies provide additional support for the hypothesis of decreased ability to track and update the latent structure of the learning environment in PTSD.

Neurocircuitry of Latent-State and Model-Based Learning and its Overlap with Neurocircuitry of PTSD

In addition to the growing evidence suggesting impaired ability to track and update latent characteristics of the environment in PTSD [26,30,32–34], there is also considerable overlap in the neurocircuitry theorized to mediate latent state learning and neurocircuitry disruptions implicated in PTSD. Learning and building an internal model of the abstract structure of the environment is a concept closely related to a cognitive map, and can be linked with building a cognitive map of the current task space (in contrast to a cognitive map of physical space) [35–41]. As such, two main neural structures strongly implicated in supporting latent state learning are the orbitofrontal cortex (OFC) and hippocampus (HC) [17,35–37,39,42]. Indeed, human neuroimaging studies demonstrate that latent states of a task (e.g., whether faces vs houses are needed for the current decision) can be decoded from multivariate patterns of activity in the OFC [43] and HC [44].

A somewhat separate body of research also demonstrates the important role of the lateral prefrontal cortex (PFC) and a broader frontoparietal network in representing the abstract structure of the environment [12,45,46]. Initial demonstrations came from human neuroimaging studies where activity in dorsolateral and parietal structures tracked state prediction errors (i.e., surprise in a given state of the task due to current estimates of state- and action- transition probabilities), rather than reward prediction errors (i.e., surprise in reward outcomes due to current reward expectations) [12,13]. Transcranial magnetic stimulation to disrupt activity in the dorsolateral PFC in humans decreases model-based learning [47], suggesting causal evidence for the lateral PFC in representing the abstract state space of the task. The role of frontoparietal structures in representing and updating latent states of the environment would also be consistent with the role of these structures in general executive function, planning, and working memory, all of which are sub-processes of model-based learning [48,49]. The separate neural systems implicated in model-based and latent state learning, HC and OFC vs frontoparietal network, are not incompatible. Indeed, emerging evidence suggests both systems are working together to support the numerous computations needed to code abstract states of the environment and inform behavior [50] (e.g., representing states in the OFC and HC vs action planning towards goals in the PFC).

The main neurocircuitry implicated in latent state and model-based learning overlaps with neurocircuitry implicated in PTSD. Though traditionally conceptualized as a fear-based disorder with emphasis on fear and salience circuitry [1,19,51–54], PTSD neurocircuitry models also have a long history of emphasizing dysfunction of HC and ventromedial PFC (including the more specific OFC) [19,54]. However, these PTSD models have traditionally focused on the role of HC and ventromedial PFC dysfunction in explaining impaired fear extinction and fear inhibition in PTSD [19,51,55]. Nonetheless, HC and vmPFC dysfunction in PTSD would be consistent with higher-order impairments in tracking and updating abstract representations of the environment, which could then be responsible for downstream impairments in extinction learning and fear inhibition (Figure 1). Similarly, impaired context processing due to HC dysfunction has recently been discussed as a primary mechanism of dysfunction in PTSD [27], which can clearly be recast from a computational perspective as difficulty tracking and updating the abstract structure of learning environments.

Figure 1.

Depiction of possible model by which different neurocircuits implicated in PTSD interact and explain different behavioral and clinical phenomena. Histories of traumatic event and/or stressor exposure exerts independent effects on function and structure of circuitry implicated in threat and salience detection (amygdala (Amy), anterior insula (AI), dorsal anterior cingulate cortex (dACC)), reward processing and action selection (dorsal and ventral striatum), and tracking and updating abstract representations of the environment (hippocampus (HC), orbitofrontal cortex (OFC), and lateral PFC). We hypothesize reciprocal interactions between these neurocircuits, associated higher-order functions, and downstream behavioral targets. For example, heightened threat detection and impaired updating of task representations both likely impact reward learning in PTSD; decreased reward processing and positive affect similarly likely impact updating task representations and anxiety and stress responding. The novel contribution of incorporating impaired updating and tracking of abstract task representations is that this higher-order process explains a wider array of downstream targets than predicted in traditional PTSD neurocircuitry models, which emphasize the role of the HC and OFC mostly in fear inhibition and fear extinction. Importantly, if it is the case that impaired latent state and model-based learning operate as higher-order functions explaining deficits in fear inhibition and fear extinction, then it would be necessary to target the updating of abstract models of the environment rather than simply targeting stimulus-specific responding. Targeting the abstract models of the environment would also engender broader improvements in overall functioning in PTSD (e.g., cognitive flexibility, context processing, cognitive re-appraisal, etc) beyond just fear extinction and fear inhibition.

Beyond the traditional structures of HC and vmPFC, additional neurocircuitry implicated in PTSD also supports impaired tracking and updating of task representations. Meta-analyses provide robust evidence for decreased frontoparietal activation across cognitive tasks in PTSD [56]. While altered activity and connectivity of frontoparietal regions in PTSD have traditionally been conceptualized as explaining executive function, working memory, and attention deficits in PTSD [56–58], they would also be consistent with altered tracking of abstract task representations. Given that basic neurocognitive processes (working memory, attention, etc.) underlie the more complex operation of tracking and updating task representations [48,59,60], a plausible hypotheses is that altered frontoparietal encoding of latent state learning in PTSD is due to these more basic neurocognitive disruptions.

Alterations in neurocircuitry supporting latent state and model-based learning would also generally suggest impairment in a broader array of learning and decision-making contexts beyond just fear extinction/fear inhibition, consistent with a recent meta-analysis [61]. For example, research provides strong support for altered reward learning in PTSD [28,33,62]. Beyond the direct impact of trauma and stressor exposure on striatal and reward function [63–66], it would be expected that higher-order impairments in tracking abstract task representations would have an additional downstream impact on striatal processing of reward and subsequent reward learning. Overall, neurocircuitry mediating latent state and model-based learning clearly overlaps with neurocircuitry dysfunction, and downstream behavioral impairments implicated in these neurocircuits (e.g., cognitive re-appraisal, context processing, reward learning, fear extinction, etc.), that are widely implicated in PTSD.

By contrast, certainly not all behavioral and neurocircuitry impairments observed in PTSD would fit with, or be predicted by, impairments in latent state and model-based learning. Heightened fear responding, including fear acquisition learning, and associated hyperactivity in fear processing and salience detection neurocircuitry, would seemingly not be expected in PTSD based solely on impairments in learning the abstract structure of environments. Indeed, one needed area of future research will be investigating how stressor and threat exposure, either acutely or chronically, impacts latent state and model-based learning. For example, a reasonable alternative hypothesis is that it is the state of heightened fear/stress that causes impairments in higher-order learning seen in PTSD. Interestingly, emerging research suggests that acute stress increases cognitive flexibility (i.e., a construct explained by latent state learning in recent OFC models [17]), whereas chronic stress decreases cognitive flexibility [67], possibly consistent with impairments in latent state and model-based learning seen in PTSD samples.

Another interesting hypothesis is that due to interacting dysfunction in the neurocircuits supporting fear learning and neurocircuits supporting the abstract representations of environmental models, individuals with PTSD may have overly fixed or rigid maladaptive world models related to threat or danger. Possibly consistent with this, a recent animal model study demonstrated that a fear conditioning event creates a biased pattern of hippocampal replay, such that replay sequences begin with the animals current location and extend towards the location of shock [68]. Relatedly, chronic stress appears to alter HC discrimination of familiar and novel contexts, possibly consistent with impaired representation of (spatial) cognitive maps [69]. These data are consistent with a hypothesis of latent models of the world that become biased due to fear- and stress-related learning experiences in PTSD. It will be important for future research to investigate threat-related impacts on forming and updating of abstract task representations, as this line of research will help tie canonical neurocircuitry in PTSD (amygdala, dorsal anterior cingulate cortex, anterior insula) together with neurocircuitry of latent state and model-based learning. Figure 1 depicts one possible model integrating latent state learning and associated impairments into canonical PTSD neurocircuitry and behavioral impairments.

Implications for Enhancing Extinction and Exposure Therapy

One of the goals of applying computational models to laboratory learning and decision-making tasks is to inform novel treatment development. Latent state learning models reconceptualize the problem of reducing fear towards a learned cue, thereby enabling novel insights and stimulating novel intervention adjuncts. A traditional view of the problem of learned (or generalized) fear towards a trauma cue is to emphasize the threat value of the given cue without consideration of a larger world model governing that threat value. That is, more consistent with a ‘model-free’ conceptualization, some efforts have focused on enhancing prediction errors during extinction or exposure therapy to magnify the decrease in threat valuation, thereby presumably leading to decreased fear responding upon subsequent cue presentations [70–73]. However, as extinction learning does not (except in specific and unique situations [74]) modify the original fear memory, the implication is that efforts simply focused on decreasing threat valuation might have limited potential over longer time scales. Indeed, larger predictions errors during extinction can actually lead to greater fear recall at subsequent tests under certain conditions [75]. Rather, consistent with contemporary models [6,7,9,24,76], the goal for extinction learning or exposure therapy from a computational perspective might instead be recast to focus on maximizing the likelihood of the learner inferring an abstract model of the environment during an encounter with a trauma cue in which that trauma cue has low threat value. That is, the goal is less about changing the learner’s threat association with a given cue and more about the learner’s abstract models that govern associations with the cue and the inferential process by which the learner infers, and adjudicates between, these abstract models (e.g., a bias towards inferences of a danger vs safe latent state would accordingly bias threat responding).

This re-conceptualization of the goal for extinction learning and exposure therapy seems consistent with clinical evidence regarding the role of maladaptive traumatic beliefs (i.e., maladaptive beliefs about the self, others, and the world that may have developed due to, or have been reinforced by, the traumatic experience [77]) in exposure therapy for PTSD. Despite exposure therapy for PTSD not directly targeting or focusing on maladaptive traumatic beliefs, exposure therapy nonetheless does result in improvements in traumatic beliefs (e.g., individuals report less belief that the world is a dangerous place or that people cannot be trusted) [78–81]. Moreover, session-to-session improvements in traumatic beliefs during exposure therapy predict subsequent decreases in PTSD symptoms, but not vice versa [78,79]. As such, even though exposure therapy makes no direct attempt to modify these larger maladaptive world beliefs, repeated exposure to corrective information appears to elicit updates to larger abstract models and schemas about the self and the world. Relatedly, exposure therapy has been found to increase activity in anterior prefrontal cortex during cognitive re-appraisal, and anterior prefrontal cortex is linked with representing schema-based knowledge [82], possibly consistent with a role for exposure therapy in enabling abstract model updates. Overall, there appears to be convergent clinical data that supports a recasting of the focus of extinction learning and exposure therapy from stimulus-specific associations to the larger abstract world models that may govern these downstream associations.

Emerging basic computational work on learning and memory captures the contextual inferential processes driving the formation, updating, and retrieval of abstract models that govern specific memory retrieval [24,25]. However, there is a significant knowledge gap in the literature informing precisely how to modify a specific learning experience (e.g., extinction training or exposure therapy) to drive specific changes in abstract world models that generalize to novel encounters with similar cues. Similarly, there is a knowledge gap in informing how to drive a specific inferential process whereby the learner infers one model (e.g., emphasizing safe associations with cues) vs another model (e.g., emphasizing danger associations with cues). While there have been computationally-informed attempts to enhance extinction learning (e.g., deepened extinction [83,84], gradual extinction [75], replacing US with a novel neutral outcome [85]), it is still not clear how to inform clinical efforts directed at this problem. For example, how does one tailor an exposure therapy exercise so that an unpredicted safe outcome results in a fundamental change to an abstract world model rather than being assimilated into the existing model (see Figure 2)? Following the accumulation of safe experiences during exposure therapy with trauma-related stimuli, and presumably the formation of an abstract model that accommodates these experiences, what determines whether the learner infers the new model vs the old model upon encounter to a novel trauma cue? What is the relevant therapeutic benefit of forming a new abstract model vs updating an old model? Answers to these questions have clear relevance to translation to clinical practice and require careful experiments of laboratory analogues of exposure therapy. It is accordingly quite relevant that most laboratory paradigms of fear conditioning and extinction generally fail to capture important aspects of exposure therapy for PTSD (Box 2). This creates a problem for computational models that explain laboratory conditioning and extinction behavior to also capture and explain real-world exposure therapy.

Figure 2.

Depiction of abstract model updating in the context of traumatic beliefs relevant to PTSD. Following a traumatic experience, here an attack by a dog, the individual may form an abstract model about danger that applies to the broad category of dogs (i.e., the prototypical dog is dangerous). Upon exposure to a new dog, this traumatic belief is inferred, thereby leading to expectations of danger and eliciting fear conditioned responses (CRs). Upon an unexpected safe encounter with this new dog, there are different ways in which the individual’s model may be updated to account for it. One possibility, which is less likely to be helpful therapeutically, is the individual might assimilate this discrepant safe encounter into the existing traumatic belief by allowing for statistical outliers, resulting in the traumatic belief still being inferred in response to most dogs. Upon an encounter with a subsequent novel dog the old dangerous belief might be inferred, thereby leading to continued expectation for danger and increased fear CRs. A different possibility, which is more likely to have a positive therapeutic impact, is the individual might accommodate the traumatic belief to account for the safe encounter by forming a new abstract model about the environment in which most dogs are safe and the dangerous dog from the original traumatic event is the outlier. Upon an encounter with a subsequent dog the new safe belief might be inferred, thereby leading to decreased expectation for danger and decreased fear CRs.

Box 2. Mismatch between real-world exposure therapy and laboratory fear conditioning and extinction models.

Degree of fear reduction.

Fear responding during exposure therapy rarely results in previously feared stimuli being indistinguishable from never feared stimuli. During exposure therapy, a common index of emotional responding is the Subjective Units of Distress (SUDs) rating[107,108] using a 0–100 scale. PTSD studies of SUDs rating during exposure therapy often find that SUDs peak within a session around 80, yet the average reduction from peak to end within a session is only around 20 SUDs increments[109,110]. Indeed, achieving a moderate, but tolerable, amount of distress towards a vivid trauma memory is considered a success when treating PTSD. It will be interesting and important for laboratory-based fear extinction methodologies to mimic this partial reduction of fear responding.

Behavioral history with fear stimuli.

Most laboratory fear conditioning and extinction paradigms present stimuli to participants who passively observe the stimuli and do not interact with the stimuli. By contrast, patients presenting to therapy for PTSD by definition have significant histories of avoiding and interacting with traumatic stimuli[111]. Indeed, experimental work in healthy humans has demonstrated that manipulating the engagement of a class of avoidance behaviors (e.g., broad avoidance of contaminants) for two weeks results in a subsequent increase in anxiety[112]. These data suggest that the learning history of avoidance behaviors with which patients with PTSD present has also influenced their emotional responding to trauma cues. As such, laboratory extinction paradigms where the learner has no history of avoiding the conditioned stimuli may not generalize as well to real-world exposure therapy. There has been a recent resurgence in basic research on avoidance behavior that may continue to fill in this gap[113–116].

Complex emotional responses.

Emotional responding to trauma cues is considerably more complex than is modeled in laboratory tasks. In contrast to anticipatory anxiety that may be elicited in response to CSs that predict an electric shock, emotional responding to trauma cues often involves a broad array of emotional responses including fear, shame, disgust, anger, and sadness. It is unclear from either a basic neuroscientific or computational perspective whether learning towards CSs that predict relatively simplistic emotional responses to electric shock generalize to traumatic cues and complex emotional responses. Adding additional complexity, individuals with PTSD are also typically in chronic mood states characterized by anxiety and depression, which can affect fear learning[117–121]. Extending and applying work on complex emotions and mood to the domain of exposure therapy will be important.

Latent State and Model-Based Learning applied to Cognitive Processing Therapy

Latent state and model-based learning may also serve as a conceptual bridge connecting a mechanism of action in exposure therapy with the other gold standard, but procedurally very different, treatment for PTSD, Cognitive Processing Therapy. The neuroscience literature of PTSD tends to be dominated by constructs related to fear conditioning and extinction. Indeed, an overly literal interpretation of the basic neuroscience literature of PTSD might lead one to conclude that the only reasonable treatment that would work for people with PTSD are exposure-based treatments or other interventions that directly target fear memories. In fact, a non-exposure-based treatment, Cognitive Processing Therapy (CPT), demonstrates equivalent efficacy for PTSD symptom reduction [86]. In contrast to exposure therapy, CPT focuses explicitly on identifying and modifying traumatic beliefs. Supporting the specific efficacy of targeting these beliefs, a careful dismantling study demonstrated that CPT retained its efficacy when removing any exposure therapy elements from the treatment and solely focused on identifying and modifying these beliefs [87].

Exposure therapy as a clinical procedure strongly draws on the basic science of fear extinction. Analogously, cognitive restructuring procedures, such as those used in CPT, similarly can at least partially be modeled through the experimental laboratory paradigm of cognitive re-appraisal [88]. There was early speculation that fear extinction and cognitive re-appraisal relied on shared underlying neurocircuitry; namely, ventromedial PFC regulation of amygdala[89]. However, subsequent meta-analyses failed to support ventromedial PFC engagement during cognitive re-appraisal and instead demonstrated strong engagement of frontoparietal networks, which is more consistent with the higher order executive functions engaged during cognitive re-appraisal[90]. As such, the fact that CPT reduces PTSD symptoms as well as exposure-based therapy cannot be explained away by arguing that CPT is simply a different way of engaging the same extinction neurocircuitry mechanisms as exposure therapy (though we agree there are likely some convergent distal therapeutic effects on other overlapping circuits between CPT and prolonged exposure).

Instead, we suggest that traumatic beliefs, and their targeting through CPT, can be understood from a computational perspective by incorporating concepts related to latent state and model-based learning [7,24]. First, latent state learning and cognitive reappraisal engage overlapping neurocircuitry. As noted above, meta-analyses demonstrate strong engagement of frontoparietal structures during cognitive re-appraisal, which overlaps well with neurocircuitry engaged during model-based learning [12,13,45]. Second, the clinical terminology of “beliefs” and “stuck points” can readily be re-cast as a model of the environment. Indeed, recent work has explicitly linked concepts related to cognitive schemas, which formed the basis for CPT conceptualizations of PTSD [91], with computational perspectives of latent state and model-based learning [92–94]. For example, a common belief among individuals with PTSD is “people cannot be trusted”, which is a shorthand abstraction of a more complicated model of the environment that makes predictions about states (e.g., people’s intentions/motives during interactions), behaviors (e.g., asking for emotional support), and outcomes (e.g., betrayal). It is reasonable to hypothesize that targeting these beliefs/schemas in CPT involves making updates to this abstract model. Finally, as discussed above, emerging animal work on the influence of stress and fear conditioning on cognitive map representations in the hippocampus [68,69] could suggest a neurocircuitry mechanism for rigid world beliefs related to danger in PTSD that are targeted in CPT. Conceptualizing CPT targets from a model-based learning perspective will hopefully motivate more specific laboratory-based experiments and attempts to apply computational models to this more specific domain that is as relevant to PTSD as fear learning and extinction.

Concluding Remarks

We have discussed emerging areas of computational and cognitive neuroscience, latent state and model-based learning, evidence for behavioral and brain biases in these processes, and implication these concepts and biases in PTSD have for neurocircuitry and treatment models of PTSD. Incorporation of these concepts into existing models currently raises more questions than answers (see Outstanding Questions). We argue that focused research that further elaborates whether and in what capacity latent state learning is altered in PTSD and how to target these processes in treatment will ultimately lead toward both better scientific understanding and ability to effectively treat PTSD.

Outstanding Questions.

How do biases in latent state and model-based learning fit into existing PTSD neurocircuitry and clinical models? One possibility is that biased tracking and updating of abstract task representations operates as a higher-order function that explains some existing observations (e.g., impaired fear extinction) but not others (e.g., heightened fear learning). Other possibilities include a chronically heightened state of stress, and associated impairments in executive function, and/or specific trauma exposures instead producing a downstream bias in abstract task representation learning. The reciprocal relationships between tracking the abstract characteristics of the environment and other widely documented processes in PTSD will need to be mapped and sorted with respect to cause vs consequence.

There is substantial heterogeneity in PTSD. To what degree do biases in latent state and model-based learning, and their neurocircuitry encoding, differ among individuals with PTSD, and how do individual differences in these biases relate to different symptom profiles and treatment needs?

How does the context and valence of the learning task determine the degree of alteration in tracking and updating of abstract representations? For example, as PTSD is associated with heightened fear learning, would PTSD be associated with better latent state learning in a task context that required tracking and updating latent states for fear associations?

How well do laboratory tasks and the targeted learning processes map on to the related procedures and processes during real-world therapy? For example, to what degree does exposure therapy or cognitive processing therapy rely on the basic mechanisms of extinction learning and cognitive re-appraisal vs latent state learning? Translation of the emergent elegant computational work to real-world therapy will depend on collaboration between the fields of computational and clinical sciences to develop laboratory tasks and models that generalize to clinical settings.

Highlights.

Recent research suggests evidence for biases in latent state and model-based learning in PTSD

Biases in latent state and model-based learning potentially explain a larger set of behavioral and neurocircuitry findings in PTSD

Biases in latent state and model-based learning potentially operate as a common target of both exposure therapy and cognitive processing therapy for PTSD

More research is needed to inform how to drive learning towards specific model updates during treatment

More research is needed integrating biases in latent state and model-based learning within existing neurocircuitry and behavior models of PTSD