Neural computations of threat

Abstract

A host of learning, memory, and decision-making processes form the individual’s response to threat, and may be disrupted in anxiety and post-trauma psychopathology. Here we review the neural computations of threat, from the first encounter with a dangerous situation, through learning, storing, and updating cues that predict it, to making decisions about the optimal course of action. The overview highlights the interconnected nature of these processes, and their reliance on shared neural and computational mechanisms. We propose an integrative approach to the study of threat-related processes, in which specific computations are studied across the various stages of threat experience, rather than in isolation. This approach can generate new insights about the evolution, diagnosis and treatment of threat-related psychopathology.

NEURAL COMPUTATIONS OF THREAT: LEARNING, MEMORY AND DECISION MAKING

How does the brain compute threat? When facing danger, an organism is tasked with learning precursors of threat, remembering those precursors for extended periods and across contexts, and making optimal decisions under stress. Anxiety and trauma can impact each of these cognitive processes. Here we define threat as an organism, a thing, or a situation that is likely to inflict damage on an organism’s physical or mental wellbeing. We review the neural computations underlying adaptive and maladaptive threat learning, memory and decision making. Beginning with an encounter with a threatening situation, we follow the associations born out of this event, the elaboration of these associations, the formation and reformation of threat memories, and how threat experience shapes decision making (Figure 1). For each phase, we describe the neural computations performed on incoming, retrieved or projected information, and their manifestation in post-traumatic stress disorder (PTSD) and anxiety disorders.

Figure 1. The stages of threat experience.

Experiencing a life-threatening event, in this case, an aversive emotional memory of a car explosion (panel A), may result in associative learning (panel B) where a neutral stimulus (the blue car) becomes threatening as it predicts danger (explosion). The learned association then competes with or influences new associations (panel C). For example, generalization of the association to other stimuli may occur (panel C top), or extinction learning (panel C bottom), where repeated exposure to blue cars diminishes the threat response. A more permanent way of diminishing the learned threat response is by modifying the original association through reconsolidation updating (panel D). A reminder cue may trigger the memory and destabilize it, requiring re-stabilization (reconsolidation) to return it to a stable state. In the course of destabilization, updates may occur in several ways, such as extinction (top), counterconditioning (middle, car associated with a positive outcome, such as a wedding), or sensorimotor interference (depicted here as a Tetris game). The new information these processes provide is incorporated into the memory (extinction, counterconditioning), or depletes neural resources of reconsolidation (sensorimotor interferences). Finally, threat learning interacts with processes of decision making and attitudes toward loss, risk and ambiguity (panel E). For example, when facing a choice between riding a car or a bicycle, threat-related processes may bias the choice toward the less threatening option. The depiction of the stages of threat experience (A-E) does not mean to indicate any sequential order or independence. The stages are intertwined throughout the threat experience.

The juxtaposition of the various stages of threat experience highlights the interconnected nature of these processes, with common computations and overlapping neural regions (Figure 2; Box 1). At the heart of this process are computations during ambiguous situations, where uncertainty could be reduced through information gathering, proactively anticipating consequences, and the retrieval and updating of relevant memories, for the purpose of making predictions and choices more accurate. We use the term ‘computation’ in accordance with Marr’s three levels [1], whereby the term refers to the goal of the computation and the logic by which it can be carried out. We argue that the different stages of threat experience share computational goals, and therefore the algorithmic and implementation levels could also overlap (Box 2). This set of computations ought to consist of the values of predictive cues, actions and outcomes, prediction error that drives learning, dynamically adjusted learning rates, and the uncertainty surrounding these estimations. These values could be learned through different policies such as trial and error or by learning about the structure of the environment.

Figure 2. Neural basis of threat.

Brain schema depicting working hypotheses, based on extant evidence, for brain regions and neural circuits involved in threat reactivity, learning, and decision making. Rather than different regions uniquely engaged in separate processes, review of the evidence suggests that most regions are engaged in more than one function, and that the concepts of learning, memory and decision making are difficult to isolate behaviorally and computationally. For example, regions typically involved in physiological reactivity such as the PAG, also contribute to the formation of Pavlovian association by providing a teaching signal to the amygdala. Regions typically assigned to associative learning, such as the amygdala, hippocampus and ventral striatum, are also involved in decision making. Regions often assumed a role in decision making (valuation and choice), such as the OFC, vmPFC, and dACC, PPC and lateral PFC are also involved in learning. The insula has been implicated in decision making, learning and reactivity. Together, these regions converge into a global network that conducts a similar set of computations across various phases of experience. The separate investigation of specific types of learning, memory and decision-making processes is thus not conducive for a comprehensive understanding of a unified global network. A refined approach would first define the computational problem, for example, how the brain predicts outcomes under uncertainty in a volatile environment given certain stimuli or actions; and then examine how each region in this global network contributes to these computations (e.g., informing value, tracking associability, computing prediction error). This process should then be iterated across the various stages of threat experience (e.g., initial encounter, memory retrieval, decision making), as well as other domains such as reward.

Box 1. The neural mechanisms of threat detection and modification.

Innate and learned threats processes are distinguishable in the brain. The neural circuits of innate threats can be described by three main functional units: a detection unit subserved by sensory systems that gather sensory information signaling the presence of threat; an integration unit where sensory information converges, directing the recruitment of downstream structures that produce the adaptive response, with the integration occurring at the level of the amygdala and hypothalamus; and an output unit comprised of brain stem structures, including the periaqueductal gray, directly producing adaptive physiological and behavioral responses to the threatening stimuli [7, 164]. The experience of innate threat instructs a learning process that forms a memory of the threatening event.

The neural mechanisms of threat acquisition, extinction, and other forms of threat modulation (Box Fig. 1), are centered around the routing of information to and from the amygdala and within amygdala nuclei [165]. During threat conditioning, sensory inputs arrive to the amygdala either through a thalamo-cortico-amygdala pathway, or a direct thalamo-amygdala pathway. Those sensory inputs, signaling the neutral (to-be-conditioned) stimulus and the aversive outcome, converge onto neurons in the lateral amygdala (LA). The stimulus-outcome convergence induces long-term potentiation of stimulus input synapses, such that when the stimulus later appears alone, its input will sufficiently drive LA outputs, triggering the threat response. Within the amygdala, the LA relays information directly to the central nucleus (CeA) or via the basal nucleus. There is also evidence that the basal/lateral nuclei (BLA) and CeA process information in parallel and not only serially [155]. The CeA is the major output structure of the amygdala. CeA projections to the hypothalamus, periaqueductal gray (PAG) and other regions, mediate the behavioral and physiological threat response (freezing, change in heart rate and blood pressure, and release of stress hormones) [23, 166–169].

LA neurons’ aversive responses correspond to prediction error encoding, because these cells respond strongly to unexpected aversive outcomes, but reduce their firing when the outcomes are predicted by conditioned stimuli. The prediction error signal is created by an amygdala-PAG feedback circuit. A conditioned stimulus recruits the CeA to activate a specific population of PAG neurons, which in turn inhibits aversive signaling before it reaches the LA, thereby resetting threat learning levels and controlling conditioned threat behaviors [29, 170–172].

Along the borders of the BLA and CE lay islands of inhibitory (GABAergic) neurons. These are the intercalated cell masses (ITC) that exert inhibitory control over the amygdala, as part of an ‘off switch’ system. The major amygdala nuclei further divide into internal partitions. For example, the CeA divides into lateral and medial parcels. The lateral part of the CeA continuously inhibits the medial part, keeping amygdala’s output under control. BLA overturns this effect by projecting to the lateral CeA via ITC cells. The medial CeA, consequently, now free of inhibition, enables the threat response [173, 174]. In addition to the threat responsive population of ‘on’ cells in the amygdala, there are also ‘off’ cells that are responsive to stimuli that signal extinction, as well as BLA to nucleus accumbens projecting neurons that signal reward [123, 175, 176].

Amygdala threat responses are short-lived: they last only a few hundred milliseconds and therefore cannot be responsible for the stimulus-evoked sustained threat response, which typically lasts at least a few seconds. In rodents, the dorsal part of the medial PFC, the prelimbic PFC, is the region that maintains and prolongs threat responses. The adjacent infralimbic PFC mediates the diminution of the threat response seen following extinction [177–180]. The putative human homologues of these regions are dorsal anterior cingulate cortex (dACC) and ventromedial PFC (vmPFC), respectively [181]. Retrieval of extinction memory involves potentiated inhibitory circuits in BLA and increased medial PFC output to amygdala [173, 175, 182]. Inputs from the hippocampus, insula and the thalamus, among other regions, further modulate the amygdala’s threat response [177, 178, 183].

Regions of the PFC arbitrate between freezing and avoidance responses. The prelimbic gates the impact of BLA inputs to ventral striatum during avoidance [48]. BLA projections to ventral striatum, and prelimbic PFC projections to BLA, both facilitate avoidance; whereas prelimbic PFC projections to ventral striatum diminish avoidance [184]. The infralimbic PFC suppresses freezing mediate by amygdala’s CE [185]. The retrieval of avoidance memory relies on prelimbic PFC projections to BLA [44], and avoidance extinction relies on projections from the infralimbic PFC to BLA and ventral striatum [186]. Consistent with these findings, research in humans showed that the degree to which amygdala and striatum were synchronized with regions in the medial PFC during avoidance learning predicted avoidance success [46]. Striatal activation differentiated between participants that exerted control over conditioned stimuli versus those that did not, and corresponded to diminished return of threat responses [50, 51, 187]. Theoretical formulations suggest that individuals’ estimates of agency based on past experience with controllable and uncontrollable outcomes, adaptively calibrates their proactive or reactive behavioral strategies [188].

Box Figure 1. The neurocircuitry of threat learning and modification.

A schematic representation of the inputs into the amygdala, information transfer within the amygdala, and amygdala outputs, and their putative functional role in threat learning and regulation. B, basal amygdala; LA, lateral amygdala; CE, central amygdala; dACC, dorsal anterior cingulate cortex; vmPFC, ventromedial prefrontal cortex; PL, prelimbic prefrontal cortex; IL, infralimbic prefrontal cortex; CS, conditioned stimulus; US, unconditioned stimulus.

Box 2. PTSD and anxiety.

Post-traumatic stress disorder (PTSD) was first introduced as a diagnosis in 1980, in the third edition of the American Psychiatric Association’s (APA) Diagnostic and Statistical Manual of Mental Disorders (DSM-III). The initial definition attracted controversy and was revised over the years [189]. The most substantial conceptual change occurred in the latest DSM-5, with the removal of PTSD from the category of anxiety disorders [190]. Instead, the disorder was placed in a new diagnostic category named “Trauma and Stressor-related Disorders.” This new categorization is unique among psychiatric disorders: while all other DSM diagnostic categories are conceptually grouped by symptom characteristics, this is the only category that requires an exposure to a stressful event as a precondition.

Accordingly, most fundamental to the nosology of PTSD is criterion A -- exposure to a traumatic event. The definition of trauma includes actual or threatened death, serious injury, or sexual violence [72]. This specific definition indicates that not all stressful events (e.g., psychological stressors like losing a job or a divorce) qualify as trauma. Exposure to trauma, according to criterion A, consists of not only direct personal exposure, but also witnessing trauma to others, or indirectly experiencing trauma through a traumatic experience of a close individual. The assessment of PTSD symptoms is valid only if criterion A is met. The symptoms must begin or worsen following the traumatic event, without assuming any causal or etiological inference. The symptom groups are: intrusions, avoidance, negative alterations in cognition and mood, and alterations in arousal and reactivity. This new organization emphasizes avoidance, now making it a requirement in order to meet diagnostic criteria for PTSD.

One reason for the separation of PTSD from anxiety is the ample evidence that PTSD involves emotions outside the range of fear and anxiety (such as anger, guilt, or shame). To be diagnosed with an anxiety disorder, a person must experience fear or anxiety that are out of proportion and impair normal function. Fear refers to an emotional response to an immediate threat, often triggering a fight or flight reaction. Anxiety is more diffused, referring to the anticipation of a future threat, typically manifested in muscle tension and avoidance behaviors.

The neural correlates of PTSD and anxiety largely involve structural and functional aberrations in amygdala, prefrontal cortex and hippocampus. PTSD patients exhibit exaggerated amygdala reactions to negative and trauma-related stimuli, hypoactivation of the vmPFC, and impaired hippocampal-dependent context learning, as well as neuroendocrine dysregulation [191]. Animal models differentiate neural circuits underlying response to immediate present threat versus uncertain threats (i.e., anxiety). Uncertain threats (for example, unpredictable shocks) engage the bed nucleus of stria terminalis (BNST), which mediates the transfer of information between the amygdala and ventral striatum, and modulates defensive reactions [192].

The first line of defense against PTSD is prolonged exposure therapy [193, 194] – the repeated exposure to trauma-related cues – leading to desensitization (akin to laboratory procedures of extinction). The treatment is ineffective in about 20–30% of patients, and approximately 20% fail to complete the full course of treatment. There is little empirical evidence to support pharmacological treatments for PTSD [195, 196], which are typically selective serotonin reuptake inhibitors. The development of novel effective treatments is desperately needed. Possible directions consider drug-assisted behavioral therapy [197], reconsolidation-based pharmacological and behavioral treatments [198–200], a synergistic approach combining multiple behavioral and neural processes [76, 87] and the temporal progression of treatment [84].

Although these computations are deployed throughout the threat experience, the information they process differs depending on the phase, be it the initial encounter, subsequent learning, memory retrieval, or decision making. This approach generates interesting predictions about how clusters of symptoms may organize, and proposes considerations for diagnosis and treatment, as the following sections elaborate. This overview suggests that rather than treating anxiety and PTSD as disorders of multiple distinct processes – heightened emotional reactivity, aberrant learning, impaired inhibition, over-generalization, hyper-avoidance, maladaptive memories, or biased decision-making – a unifying approach may prove more efficient. From a computational standpoint, anxiety and PTSD are disorders of neural prediction – the estimation of future threats.

THE STAGES OF THREAT EXPERIENCE

The experience of imminent threat

Encountering a life-threatening situation engages neural computations that consider information about the environment and the source or threat. These computations design defensive policies and select adaptive responses for execution. The threat imminence continuum model [2], which maps defensive behaviors onto levels of threat imminence (how far a predator is in time and space), provides a platform of prey-predator relations for assessing the neural circuits and computations for survival [3, 4]. In the first ‘safe’ stage, there is no threat, an encounter with a predator may occur in the distant future. Individuals may experience occasional anxiety, and flash forward toward possible future threats. Cognitive control and emotion regulation could keep this process in check. Next is the ‘pre-encounter threat’– the predator is not present, but may surface at any moment. Individuals may experience anticipatory anxiety and exhibit vigilance and preparatory behaviors. In the more dangerous ‘post-encounter threat’ the prey, not yet detected, observes the predator. This step generates encounter anxiety, involving close inspection and anticipation of the predator’s moves, strategic freezing to avoid detection and gather information, and avoidance estimation. Finally, the prey is under most extreme danger during the ‘circa-strike’ phase, when the predator is attacking. Within that attack mode, the predator could be distant enough to allow a feeling of fear and rapid thoughts examining the situation and assessing escape routes. Fight or flight ensues as the predator gets closer yet without contact. The final point of contact provokes hard-wired, fast, often poorly-executed reactions of freezing and panic [5].

A hierarchical neuroanatomical organization traces the threat imminence continuum. As the predator approaches, brain activity shifts from the prefrontal cortex (PFC) to midbrain. Two parallel paths support defensive approach and avoidance, originating from PFC areas through cingulate cortex areas, to hippocampus, amygdala, and hypothalamus, terminating on midbrain periaqueductal gray (PAG) and dorsal raphe nucleus [3]. Converging neuroanatomical and functional evidence across species, including rodents, non-human primates and humans, supports this organizational scheme, attributing feelings of anxiety and fear and cognitive regulation to higher order cortical areas, and freezing, escaping and panic to amygdala, hypothalamus and PAG, respectively [3, 6–8].

A proximal threat engages rapid, reflexive and narrowly targeted actions, and therefore has limited computational resources. Decisions during this phase likely rely on model-free computations, sustaining the repetition of previously reinforced actions. The more distal points of encounter allow time to assess the environment and consider alternative courses of action. It is plausible that, in addition to model-free responses, such threats also initiate model-based computations – the prospective anticipation of action consequences using a constructed map or a model. This computational policy favors accuracy and strategic planning [9, 10], and allows “offline” testing of potential courses of action using mental simulations. Computational methods such as Dyna [11] aim to identify the optimal policy for a particular situation, by simulating actions and their consequences within an internal model of the environment. The mental simulations provide data for training the model and improving predictions in the absence of actual threat experiences.

People that experienced a traumatic event and went on to develop post-traumatic stress, and people suffering from anxiety disorders (Box 2), often exhibit some of the same behaviors along the threat imminence continuum, albeit in an excessive manner and inappropriate contexts. For example, according to the DSM-5 [12], among the criteria for a PTSD diagnosis are hypervigilance and avoidance as in the anticipatory anxiety phase. Anxiety disorders are accompanied by frequent intermittent anxiety, involving rumination, worry and over-strategizing. This suggests that PTSD and anxiety disorders may involve deficient computations of spatiotemporal threat, particularly influencing the reliance on model-based planning during anticipatory and encounter anxiety. Consistent with this idea, high-trait anxiety affects escape decisions from a virtual predator, but only when the threat is distal, rather than imminent [13]. Other studies, in the domain of reward learning, demonstrate the vulnerability of model-based computations to stress [14], particularly in depression [15], and to lifetime stress [16], as well as to self-reported intrusive thought [17], a symptom of generalized anxiety and PTSD. Additional research is required to examine how greater reliance on one system versus another relates to vulnerability or resilience to traumatic stress, and which clusters of symptoms correspond to these underlying behavioral policies.

Beyond direct experience, individuals can also be impacted by second-hand, vicarious experience, by witnessing the experience of others or through verbal instructions (Box 2). This form of social behavior capitalizes on existing neural mechanisms of direct learning, in addition to processing of social information, and is evident across species [18–20]. Socially-formed associations can also shape subsequent decision making the same way direct learning does [21].

Experiencing imminent threat may result in a cascade of (i) formation of threat associations; (ii) post-association learning; (iii) storing and updating of these associations; and (iv) decision-making under threat (Figure 1). It is possible that the particular processes occurring in the first stage would impact the four following processes. For example, a less imminent threat encounter might result in a weaker memory that is easier to extinguish; a highly stressful encounter with more imminent threat could produce a memory that is reconsolidation resistant; the degree of uncertainty during initial encounters might shape the flexible update of initial learning etc. The next sections examine the cascade of processes following initial threat learning.

Backpropagation of threat

The computational processes during imminent threat focus on the immediate needs and current environment of prey. But they also rely on prior learning, such as previously acquired associations, reinforced actions and learned models, called upon in the service of the moment. The merging of past and present experience yields new associations and updated models. In the simplest form, predation becomes associated with neutral stimuli in the environment. The threat of the predator backpropagates to the stimuli that predict it, a process known as Pavlovian conditioning [22, 23].

Various learning models are used in this research, aiming to capture various types of information. Theories of associative learning, such as the Rescorla-Wagner (RW) and reinforcement learning models, envision that learning is driven by surprise, formalized as prediction error – the difference between the outcome expected and the outcome received. The predictive value of the stimulus - the level of threat that it predicts - changes proportionally to the magnitude of the error at a rate that is not in itself influenced by the learning [9, 24]. Overgeneralization of the learning to cues that were not associated with threat [25] may be a hallmark of anxiety disorders [26].

The temporal difference model extends the RW model, allowing predictions of accumulated discounted future outcomes, rather than the immediate only [9]. Other theories, such as the Pearce-Hall model, focus on the predictive efficiency of the cue. Here, to learn cue-reinforcer associations, individuals track a quantity termed associability, which reflects the degree to which a cue has previously been accompanied by surprise (positive or negative prediction error). The associability of a cue gates the amount of future learning about that cue, depending on whether it has previously been a poor or a reliable predictor of an outcome. In this way, associability accelerates learning to cues whose predictions are poor and decelerates it when predictions become reliable [27].

Several brain regions play a role in associative learning (Figure 2; Box 1). There is evidence for encoding of aversive prediction errors in activation patterns in the amygdala [28–31] and the striatum [32–34]. The striatal activation likely results from dopaminergic inputs from ventral tegmental area (VTA) [35, 36], although the evidence for striatal role in aversive prediction error may not be as strong as the evidence for its role in reward prediction error [37]. Both regions have also been implicated in tracking of associability [33, 34, 38, 39].

An augmented ‘hybrid’ Rescorla-Wagner model controls learning rates dynamically based on the Pearce-Hall learning rule. In humans undergoing threat conditioning, measured by skin conductance response (SCR; a measure of autonomic nervous system activity), the hybrid model better captured cue-specific associabilities, over and above value expectations [39–41]. A study in combat veterans used the hybrid model to estimate the influence of prediction errors on cue-specific associabilities in each learning trial [34]. These subject-specific parameters positively corresponded to PTSD symptoms (measured by CAPS, the Clinician Administered PTSD Scale). Thus, by assigning more weight to prediction errors, the more trauma-affected individuals exaggerated their adjustment to cues that did not predict what they had expected [34, 42]. Amygdala and striatal tracking of cue value throughout learning negatively corresponded to CAPS (less faithful neural representation of value related to worse diagnosis). Striatum tracking of associability negatively correlated with CPAS and partially mediated the positive relationship between prediction error weight and CAPS [34].

In parallel to the passive formation of cue-outcome Pavlovian associations, individuals can readily associate their actions with outcomes, a process termed instrumental (or operant) conditioning [43–45]. A classic example of instrumental conditioning in the context of threat is avoidance learning (Box 1), where an animal learns to prevent or minimize contact with an aversive outcome (such as electric shocks) or the stimuli that predict it [46]. Animal studies identify at least two opposing pathways subserving active avoidance learning: a lateral amygdala – basal amygdala – nucleus accumbens pathway required for active avoidance; and a competing lateral amygdala – central amygdala – PAG pathway mediating freezing to conditioned stimuli. The infralimbic and prelimbic PFC subregions serve as the arbitrators, mediating the transition from reaction to action by suppressing freezing and facilitating avoidance [47, 48] (Box 1). Human studies are consistent with these findings, demonstrating the involvement of amygdala and striatum and their interactions with medial PFC [49–51].

Avoidance by itself is an adaptive response to danger, but unwarranted and excessive avoidance is a hallmark of PTSD and anxiety disorders [47, 52]. Avoidance symptoms can be subdivided into passive and active. Passive avoidance is the lack of action (such as strategic freezing) whereas active avoidance involves emitting an action that circumvents the aversive outcome. The fact that active avoidance is a form of learning driven by the absence of the aversive reinforcer poses a challenge for learning theories. The psychologists O.H. Mowrer and Neal Miller provided a conceptual framework for active avoidance as a two-factor learning process, where threat is first acquired through Pavlovian conditioning, and then actions that reduce the conditioned threat are reinforced through instrumental conditioning [46, 53, 54]. Reinforcers of avoidance behavior include negative reinforcement by removing threat-associated cues or positive reinforcement by cues associated with safety [47, 48, 55].

In the context of both Pavlovian and instrumental conditioning, the models we have surveyed explain predictions of expected threat, but ignore uncertainty about these predictions (Box 3). Instead of point estimates, Bayesian learning models include estimates of prediction uncertainty that rule a dynamic learning rate [41, 56–58]. In stable environments, experiences from the distant past are informative for predicting the future, and transient changes should be largely ignored. Natural environments, however, are seldom stable; rather, unexpected uncertainty (Box 2) may arise, when the probabilistic structure of the environment changes abruptly [59–61]. When action-outcome contingencies change (for example, the outcome probability substantially drops), only recent experiences should inform learning, to allow quick adaptation to the changing conditions. Indeed, human participants are able to incorporate estimates of unexpected uncertainty in their learning [62–64]. Individuals adjust their learning rate in response to variations in volatility – the frequency of changes in action-outcome contingencies (or the mean level of unexpected uncertainty; [65]), separately for potential rewards and punishments [66]. Pupil dilation and BOLD signals in Locus Coeruleus, which are indicators of arousal, reflect subjective estimates of unexpected uncertainty [63, 67]. Activity in the anterior cingulate cortex tracks subjective estimates of volatility, and reflects individual differences in learning rate [65]. Using a hierarchical Bayesian learning model, estimates of subjective uncertainty during a probabilistic choice task predicted subjective stress and arousal, exemplifying a tight link between stress responses and environmental uncertainty [68]. These studies suggest that conditioned threat responses do not simply correspond to the outcome prediction, but rather to the degree of uncertainty surrounding that prediction.

Box 3. Types of uncertainty.

Learning and decisions about threats involve multiple forms of uncertainty. The learning literature distinguishes between expected and unexpected uncertainty [201].

Expected uncertainty – also known as irreducible uncertainty [60, 202] - arises from the probabilistic nature of outcomes in a familiar environment. In the laboratory, expected uncertainty occurs when outcome probabilities are fixed and well learned (for example when each cue presentation is associated with an 80% chance for an electric shock). While the outcomes themselves are uncertain (shock or no shock), their variance is expected, and, assuming a stable environment, these outcomes do not provide useful information, and should not drive learning.

Before learning is well established, the lack of experience gives rise to estimation uncertainty [60, 203] – uncertainty about the probabilistic structure of the environment. This type of uncertainty can be reduced with additional evidence, and signals how much learning is required. In the economics literature, irreducible uncertainty and estimation uncertainty are termed ‘risk’ and ‘ambiguity’ respectively (Box 4).

Unexpected uncertainty arises when expectations about the statistical structure of the environment are violated (for example, if the shock probability abruptly drops to 20%). Determining that the uncertainty is unexpected, rather than part of the stochastic nature of the stable environment, is challenging [59]. Expected uncertainty can provide a baseline for the level of uncertainty against which surprising events should be compared [61], considering the learner’s belief about the world [204]. Changes in the probabilistic structure of the environment, for example in a volatile environment, should lead to increased learning rate, such that predictions rely more on recent events. Normative statistical models, such as Bayesian models, and their approximations, provide accurate predictions for behavior [56, 60, 62, 65], but include complex computations, and it is not clear how these computations are implemented by the brain [61]. One model, synaptic metaplasticity (the ability to change synaptic states without measurable changes in synaptic efficacy), was proposed as a biologically plausible mechanism for adjusting learning rates based on unexpected uncertainty [205].

Individuals with trait anxiety learn more from recent punishments, compared to healthy controls [69], but are slow to adapt their learning rates in response to changes in threat volatility, and show reduced pupil response to volatility [70]. Social contexts may exacerbate such reduced adaptability [71]. Inappropriate adjustment to changes in probability structure may also lead to poor decision making, and contribute to increased symptoms if aversive outcomes are perceived as less predictable and less avoidable [70].

Overall, extant evidence suggests that aberrant neural computations of value, prediction error, associability and estimations of uncertainty, are related to anxiety and PTSD. The models described above capture two important facets of associative learning: reinforcement learning formalizes predictions of long-term accumulated outcomes; and Bayesian models, track uncertainty around learned associations. Models that merge the two computations have also been proposed [57].

AMG, amygdala; BNST, bed nucleus of stria terminalis; dACC. dorsal anterior cingulate cortex; Hip, hippocampus; Hyp, hypothalamus, Ins, insula; PAG, periaqueductal gray; PFC, prefrontal cortex;; dmPFC, dorsomedial PFC; dlPFC, dorsolateral PFC; vlPFC, ventrolateral PFC; OFC, orbitofrontal cortex; PCC, posterior cingulate cortex; PPC, posterior parietal cortex; SN, substantia nigra; VS, ventral striatum; VTA, ventral tegmental area.

Flexible threat associations

Following the encounter with danger, an individual will emit defensive responses triggered by the conditioned stimuli, but not for long. Eventually, for adaptive energy maintenance, those defensive responses will dissipate and new learning will take their place. The inappropriate lingering of learned defensive responses is part of a major PTSD symptom cluster in the DSM-V [72], defined as alteration in arousal and reactivity (e.g., hypervigilance), that began or worsened after the trauma. A prime mechanism that counteracts threat conditioning is extinction learning – the decline in responding to a stimulus that previously signaled danger, following repeated nonconsequential exposures [73]. Extinction can also be learned vicariously [74] or through imagination of the conditioned cues [75] by capitalizing on neural mechanisms of direct learning.

Formalizing an associative learning model to describe extinction has been challenging. The Rescorla-Wagner model, for example, views extinction as unlearning of associative contingencies due to the omission of outcome; but this fails to account for the return of extinguished responses under various circumstances (e.g., spontaneous recovery, renewal, reinstatement). The Pearce-Hall model, alternatively, classifies extinction as new learning where omission of the outcome in the presence of the conditioned stimulus creates a second association, such that threat and extinction associations compete for expression. However, it is possible that a mixed model, assuming cooperation between unlearning and new learning, best describes an individual’s internal representation [76]. The latent cause model [77] captures that cooperation: individuals update the associative weight between the stimulus and the outcome (i.e., unlearning) given small prediction errors, but infer that a second rule is likely in effect (i.e., new learning) given large prediction errors. In this way, an individual does not simply learn a single stimulus-outcome association, but rather parses experiences into latent causes, each with its own associative weight, thus constructing a structure of the environment. During extinction, unlearning would dominate over new learning by minimizing deviations from the individual’s expectations, such as transitioning from conditioning to extinction by gradually changing association strengths over time [78]. The model predicts that individuals who assume a single latent cause, and only update the initial threat memory, will show weaker recovery of the extinguished response. Individuals who form a new extinction memory, segmenting their experience into two latent causes, are more likely to show greater recovery from extinction [79].

How the brain implements structure learning is unclear. A possible circuit mechanism for structure learning might suggest that latent cause representations, possibly originating from the orbitofrontal cortex, activate hippocampal dentate gyrus cells, reflecting the likelihood of the active cause. The ventral tegmental area computes the discrepancy (i.e., prediction error) between the associative weights that the inferred cause predicts and the observed contingencies. The prediction error signal is then transmitted via dopaminergic projections to the amygdala and hippocampus, striatum and prefrontal cortex, thereby triggering memory updating or new learning – small prediction errors alter the associative weight of the inferred cause, whereas large prediction errors trigger neurogenesis of dentate gyrus granule cells to generate a new latent cause [80, 81].

Beyond extinction, the power of the latent cause and other unifying models lies in their ability to explain a multitude of post-conditioning phenomena, such as generalization of the response to other stimuli, blocking learning to other stimuli, second-order conditioning, counterconditioning and more [57, 80, 82, 83] thereby generalizing the Rescorla-Wagner model and addressing its explanatory limitations. Further model development is required to capture the effect of multiple extinction sessions, which may diverge into a separate circuit mechanism [84], and capture clinical therapeutic processes more faithfully [85]. Another consideration is the lingering impact of PTSD, which can last years and even decades. The longevity of the disorder is difficult to explain using current learning models, each capturing a sliver of the symptoms. A synergistic model, describing how multiple learning phenomena work together may explain the persistence of defensive motivational states [86, 87].

Lifecyle of threat memory

Aversive events serve as teaching signals instructing synaptic plasticity in the amygdala, resulting in threat memory storage (Box 1). Interventions such as amnesic agents (e.g., protein synthesis inhibitors) or electroconvulsive shock stimulation, interrupt consolidation, the stabilization period following the acquisition of a new memory. Once a memory has formed, however, it goes into a lifecycle of oscillations between periods of neural stability and instability (Figure 3). Memory reactivation and destabilization trigger those periods of instability, providing opportunities for modification. The same interventions that disrupt consolidation also interrupt – and therefore support the existence of – a destabilization period following reactivation of old memories. This phenomenological similarity inspired the term reconsolidation, referring to the active process necessary to restabilize a memory after it has been reactivated and destabilized [88, 89].

Figure 3. Lifecycle of threat memory.

Memory encoding is the strengthening of neural connections through long-term synaptic restructuring; a process occurring when different regions co-activate. A memory trace, or engram, is therefore not a physical entity like a stored object, but rather the disposition of neural circuits to fire upon triggering by a certain reminder. During reactivation, a memory becomes active when the engram ensemble fires. Reactivated memory is eligible for retrieval, behavioral expression, and/or destabilization. This means that memories that are retrieved and expressed are not necessarily destabilized, and that destabilization can even occur covertly without behavioral expression. Destabilization involves a cascade of cellular and molecular processes (such as protein degradation) that instigate the transition of the engram from a stable to an unstable state. At this point, the memory becomes susceptible to modification. Among the computational principles that govern destabilization is trace dominance: a memory will be more malleable to amnestic manipulations to the extent that it has control over behavior at the time of treatment. Prediction error is one of the parameters that influence trace dominance and facilitates destabilization. Memory restabilization (reconsolidation) requires protein synthesis among other cellular/molecular events. Successful reconsolidation restabilizes the ensemble and restores the memory into its inactive state. Manipulations that interrupt reconsolidation (pharmacological agents, behavioral interference, memory enhancers) will reroute the memory toward erasure, strengthening, weakening, or updating. A memory that has not been erased may cycle into another sequence of reactivation, destabilization and reconsolidation. LTM, long-term memory.

The instability period, when old memories resurface, presumably serves the purpose of incorporating new relevant information into the memory, instead of forming a new separate memory of present events [90, 91]. Experimental evidence supporting this theory introduced extinction learning following the reactivation of a conditioned threat stimulus, leading to long-term reduction in conditioned threat responses [92–96]. Another form of intervention used counterconditioning, which was demonstrated in the context of appetitive conditioning. Here, a drug-related conditioned stimulus was repeatedly paired with an aversive outcome (e.g., a disgusting image) following reactivation [97, 98]. Other manners of non-invasive interferences could be effective, including those that deplete the neural resources needed for reconsolidation, such as extensive sensory motor tasks [99] (Figure 1).

How is new learning following reactivation different from a standard associative learning session? Why would reactivation-extinction, for example, lead to a more permanent reduction of threat responses while standard extinction allows their return? The latent cause model, described above, offers a theoretical solution by formalizing the dynamic interplay between learning and memory [80]. Applying the latent cause theory to the case of post-retrieval memory modification stipulates that when a memory is retrieved, the brain assumes that the previously inferred cause (originally assigned to the remembered event) is once again active. This inference makes the memory eligible for updating because new information, now attributed to the original cause, merges and thus changes the original memory. For example, the associative weights of a cue-outcome association will permanently decrease due to the merging of extinction learning; in other words, the memory has been updated and changed form. If the brain otherwise infers a new cause for the surprising event, a new memory will be formed.

One of the parameters that nudges latent cause assignment, as featured in the model, is the duration of the reminder cue. Reminder duration can be conceptualized as the length of exposure or multiple repeated exposures with short gaps [80]. A brief exposure to the reminder favors assignment of the reminder to the initial threat learning cause. With longer reminder durations, prediction errors accrue, facilitating the inference of a new latent cause [100]. The model explains many of the boundary conditions of reconsolidation updating that have been observed in laboratory experiments, and makes testable predictions as to when a memory will or will not maintain its original form [80].

One of the most important goals of clinicians treating anxiety and PTSD is facilitating a change that is enduring in patients. Achieving enduring change that is not easily prone to relapse, conceivably requires three essential components: reactivating the problematic memories along with the emotions they elicit; altering those memories by having a corrective emotional experience during reconsolidation; and, building enduring semantic structures onto the updated memories, by implementing new behaviors and ways of engagement with the world [101, 102]. Indeed, some forms of therapy, such as coherence therapy, are built on the principles of memory reconsolidation and are designed to maximally optimize this process [103–105].

Decision making under threat

When a threat is detected or remembered, a rapid decision-making process ensues, resulting in an approach (e.g., attack) or avoid reaction. Long after the imminent threat has dissipated, decisions about cues that predict threat likely share some of the mechanisms with those initial choices. These decisions may be adaptive (e.g., approaching a threat that could be overcome; avoiding a real threat or a cue that predicts it) or maladaptive (e.g., avoiding a threat that could be overcome).

The decisions that individuals make depend on their available courses of action, the potential outcomes of each action, and the likelihood of each outcome. Common models of choice posit that decision makers integrate the various properties of each option to compute its idiosyncratic subjective value, and then choose the most valuable option [106]. Although this may seem straightforward, we do not fully understand how these computations are implemented in the brain. We do know that activation patterns in a network of multiple brain areas reflect subjective values that are inferred from behavior (Figure 2). Activity in two areas in particular, the ventromedial prefrontal cortex (vmPFC) and the ventral striatum, is consistent with encoding subjective values across different categories and under varying conditions [107, 108]. Other putative value regions include more dorsal regions of the medial prefrontal cortex [107, 109], the orbitofrontal cortex (OFC, [110, 111]), the posterior cingulate cortex (PCC, [112], and the posterior parietal cortex (PPC,[113]).

While it is likely that activity in these areas encodes the value of potential rewards, the evidence for value encoding of punishments, or threats, is less robust. Some neuroimaging studies in humans report overlapping representations of positive and negative values [113–115]. Other studies describe distinct representations of rewards and punishments in different brain areas, with more medial representations of reward value, and more lateral representations of the value of punishments [116–118]. Single-unit studies in animals also report both distinct [35, 119] and overlapping [120] representations of rewards and punishments in dopaminergic midbrain, habenula, and medial prefrontal cortex. In addition, some brain areas, including the vmPFC, OFC and PPC, encode aversive values in a monotonic manner (with decreasing activation for more aversive values), whereas other areas encode value in a u-shaped manner, with high activation for both high rewards and high punishments, consistent with salience representation. The salience network includes the ventral striatum, dorsal anterior cingulate cortex, anterior insula, temporoparietal junction [107, 110, 113, 121], and the amygdala [122, 123]. Thus, the ventral striatum is the only region exhibiting both monotonic and u-shaped representations of value, indicating a dual role for this structure in encoding value and salience [36, 107, 110, 121]. Whether the brain first signals the stimulus’ salience (how important it is), to orient attention, and then determines its valence (positive or negative), or whether value is computed first, with salience information (its absolute value) extracted later, or in parallel, is an open question.

The likelihoods for potential outcomes are an important factor affecting the subjective value of available options. For example, a 10% chance of sustaining an injury is not as bad as a 50% chance of sustaining the same injury. Only seldom, however, are these likelihoods completely known – a type of uncertainty known as risk or “irreducible uncertainty”. In most cases, likelihoods cannot be precisely estimated; rather, there is some ambiguity or “estimation uncertainty” around those likelihoods (Box 3). Repeated sampling of the environment and experiencing various outcomes can reduce ambiguity. Subjective values are influenced not just by the objective levels of risk and ambiguity around potential outcomes, but also by an individual’s subjective perception of risk and ambiguity and their attitudes towards these sources of uncertainty. Since risk and ambiguity attitudes are only weakly correlated across individuals, they likely involve somewhat separate cognitive mechanisms (Box 4).

Box 4. Attitudes towards risk and ambiguity.

Since most decisions are made under partly ambiguous conditions - where likelihood estimates exist, but are not precise – the individual’s behavior under uncertainty is modulated both by their risk attitude and by their ambiguity attitude [206]. To estimate these attitudes in the laboratory, researchers employ simple tasks in which participants are required to make a series of choices between various uncertain and certain options. To elicit risk preferences, the choices are between options whose outcomes and outcome probabilities are fully known (e.g. 50% chance to win $10), where one option is “riskier” and the other is “safer” [207]. A risky option is one that offers a small probability for a high reward; a safe option would offer a smaller reward, but at a higher probability. For example, a 50% chance to win $10 offers $5 on average, but at a higher risk compared to a sure bet of $5. A risk-averse individual would prefer options that offer smaller amounts at higher probabilities, over ones that offer higher amounts at lower probabilities, even if the expected value (the product of the probability and amount) of the latter is higher. A risk-seeking individual would show an opposite preference.

Most people tend to be risk averse when making choices between moderate potential monetary gains [208, 209]. There is high variability across individuals, however, in the degree of risk aversion, and a minority of people exhibit risk neutrality (i.e. they choose based on the expected value alone) or even risk seeking [210]. This is important, because it means that individual risk attitudes may be associated with particular personality traits or psychopathological symptoms. The picture is a bit more complex, however, because preferences change when the choice is between losses, rather than gains. Here too, there is wide variability in individual preferences, but most people tend to exhibit risk seeking, rather than risk aversion. For example, in a choice between losing $8 for sure and taking a 50% chance of losing $20, many are likely to take the risk [208]. Importantly, an individual’s attitude towards risk in the gain domain, does not predict their attitude to risk in the domain of losses (Box Fig. 1) [209, 210], suggesting that these are two separate characteristics, that may be differentially associated with personality traits and clinical symptoms.

To elicit ambiguity attitudes, participants are also asked to make choices when some (or all) of the information about outcome probabilities is withheld. Ambiguity aversion is commonly observed when the choice is between potential gains [211–213]. In the domain of losses, there is some evidence for reduced ambiguity aversion, or even no effect of ambiguity [210, 214]. Importantly, risk and ambiguity attitudes are only weakly, if at all, correlated across individuals (Box Fig. 1) [124, 133, 141, 210, 215, 216] suggesting that they rely on cognitive mechanisms that are at least partly separable. The individual’s behavior when making choices between potential gains also does not predict their choices between potential losses. This means that there is likely no single unified trait of “uncertainty attitude”. Rather, it seems that how the individual copes with uncertainty is a complex process, affected by several attitudes, which are largely independent.

A question of ongoing research is to what extent risk and ambiguity attitudes elicited in the laboratory reflect behavior outside of the laboratory [217]. While there is some evidence for consistent risk attitudes across domains [125, 218], task specifics may affect estimates of these attitudes [219, 220] (although this may arise from changes in the perceived risk, rather than the attitude towards risk [221]). It is also not clear to what extent risk and ambiguity attitudes are stable across time (i.e. reflect a personality trait) and to what extent they reflect state variables, although it is possible that risk attitudes are more stable, and ambiguity attitudes more transient [216].

Box Figure 1. Risk and ambiguity attitudes in the gain and loss domain comprise four largely independent decision characteristics.

Data reanalyzed from [186]. N = 99 participants from the general population between the ages of 12 and 77.

How potential outcomes and their likelihood estimates are integrated in the brain is not completely clear. The integrated subjective value, reflecting an individual’s uncertainty attitudes, is encoded, as expected, in the valuation network [109, 124, 125]. There is also ample evidence that uncertainty is reflected by neural activity in several brain areas, including ventral striatum [126–131], PPC [132–134], anterior insula [127, 131, 132, 135, 136], the lateral OFC and ventrolateral PFC [127, 129, 133, 136, 137], and the ACC [138]. Some of these studies highlight potential differences between neural activation patterns encoding risk and ambiguity [130, 133, 137]. Activity in PPC [133, 134, 137] as well as PPC’s structure [139, 140] reflect individual risk attitudes. In some studies, activity in vlPFC reflects ambiguity attitudes and not risk attitudes [130, 133, 137], but other studies report correlation with risk attitudes [129, 138]; and a study in lesion patients also implicates vlPFC in processing both risk and ambiguity [141]. Finally, the structural and functional connectivity of the amygdala is associated with individual risk attitudes [142]. Taken together, while there is abundant evidence for multiple representations of uncertainty in the brain, the underlying neural substrates of those representations, and whether different types of uncertainty are represented separately, is yet unclear.

In addition to the integration of outcomes with their likelihoods, the decision-maker also needs to integrate the values of potential rewards and punishments. For example, using the car (Figure 1) will be fast and convenient, but will also bring back memories of the aversive event. The weight given to potential gains, compared to potential losses (akin to threats) – or the degree of loss aversion [143] – varies substantially across individuals, and serves as another source for individual differences in decision making. In humans, subjective value representations in the vmPFC and ventral striatum integrate over potential appetitive and aversive outcomes of available options [114, 144], and reflect the individual’s degree of loss aversion, as estimated from their choice behavior [114]. Research in animals also implicates the amygdala in value integration. While activity of amygdala neurons reflects the value of both potential rewards and potential punishments [145], lesion studies in rats suggest that basolateral amygdala (BLA) has a specific role in integrating rewards with punishments. BLA lesion led to increased choices of large rewards accompanied by potential punishments, but did not impair sensitivity to the potential punishments [146]. Optogenetic inhibition of BLA during the simultaneous receipt of reward and punishment also increased risk taking, but inhibition during the deliberation phase had an opposite effect [147], suggesting a heterogeneous role for the amygdala in value integration, which may rely on the orchestrated activity of separate neuronal populations. Interestingly, unpredictable outcomes - even in the absence of interaction with motivational value - lead to sustained activity in the amygdala, in both humans and mice [148].

Thus, decision making as a whole, integrates outcomes and their likelihoods, expected rewards and punishment and their weights (loss aversion), various sources of uncertainty (irreducible or risk, and reducible or ambiguity), and the individual’s attitudes toward uncertainty (risk and ambiguity aversion). The decision maker then needs to compare the integrated values of different options in order to reach a choice – how this is done is a subject of ongoing research. It is possible that integrated values are compared downstream of value computations [149], or that the choice process is an inherent part of iterative value computations [150, 151].

Following a traumatic event, changes in any of the computations of valuation and decision-making may be observed [128]. In rodents, there is some evidence for malleability of valence encoding. During acute stress, the presentation of rewards induces punishment-like responses in the lateral habenula (that is, a switch from the typical decreased firing rate into increased firing rate), consistent with a shift from value to salience encoding [152]. A different type of response shift occurs in the nucleus accumbens, where neurons switch their preference from rewards to punishments in a stressful environment [153].

In combat veterans suffering from PTSD, the ventral striatum appears to shift from value to salience encoding [154]. These individuals also exhibit enhanced aversion to ambiguity around potential losses in a simple monetary choice task, compared to combat veterans who did not develop PTSD [155]. And, in a trauma-exposed community sample, self-reported intolerance of uncertainty correlated with the severity of PTSD symptoms [156]. The evidence from animal research suggests causality, i.e., that at least some of the changes in computations of valuation and decision making in humans may result from the trauma. However, it is also possible that those observed differences reflect predisposition for developing PTSD, which preceded the traumatic events. Abnormally enhanced intolerance of uncertainty may constitute a transdiagnostic factor across anxiety disorders [157], given that anxiety-related pathologies, including obsessive compulsive disorder (OCD) [158], generalized anxiety disorder (GAD) [159], and social anxiety [160] also evince increased intolerance of uncertainty.

The intolerance of uncertainty observed when anxious individuals and those with trauma-related psychopathology make choices, likely affects how they learn. An intriguing question is to what extent variations in uncertainty attitudes shape learning about threats. For example, in Pavlovian and instrumental conditioning, outcomes are used to reduce ambiguity about the probabilistic structure of the environment. Ambiguity aversion may hinder learning, because the individual will make effort to avoid the ambiguous situation; alternatively, it may strengthen acquisition, if the individual will be increasingly motivated to reduce the level of ambiguity. Similarly, extinction, generalization, relearning and other post-association processes all require the individual to cope with varying levels of uncertainty. Whether the individual’s approach to uncertainty in the context of decision making also underlies their handling of uncertainty in the course of learning is an open question.

CONCLUDING REMARKS

A common method of investigation segments threat experience into separate stages and processes, studying and diagnosing them independently. In reality, however, learning, memory and decision making never occur in isolation and are constantly intertwined. Formalizing first principles may capture the basic computational processes that are a common thread, while considering the stages of threat experience as the settings for these computations.

As described in the sections above, a fine-grained look at the initial encounter with imminent threat reveals a series of computations involving predictions about the outcomes of environmental cues and actions in an uncertain environment. New cue-outcome and action-outcome associations – arising via accumulated trial and error or models of the environment – are stored in memory, guiding future behaviors. Newly formed associations will go on to compete with, or facilitate, other associations, which together will adjust the individual’s behavior to a changing environment. The post-association phase further involves additional computations that segment the stream of experience into distinct clusters, or hidden causes. This type of computation plays a major role upon retrieval of learned associations, as it determines whether a certain memory will be updated or remain unchanged, essentially arbitrating between new learning and unlearning. Finally, choices based on learned associations incorporate estimations of uncertainty to predict and optimize future outcomes.

PTSD and anxiety disorders may be linked to aberrant computations at any stage of threat processing [161]. Improper computations may be specific to threat or extend to other domains, such as reward processing or social interactions. Improper computations may also be limited to a particular processing stage (e.g., learning) or shared by several stages (e.g., learning and decision making). For example, the increased adjustment to surprising cues in PTSD may be specific to threat learning, or reflect a general learning deficiency, which is also at play when learning about rewards. The intertwined nature of the various cognitive processes, the overlap in their neural circuits, and the likely contribution of interactions between these processes to mental disorders, also reinforce the significance of studying whole-brain computations and connectivity patterns [162].

To decipher the neural mechanisms of disorders such as PTSD, and to translate insights into the clinic (Box 5), comprehensive approaches across domains and processes should be employed. Throughout the stages of threat experience, an individual confronts two core computational problems: making predictions about long-term accumulated outcomes, and tracking uncertainty in the environment. These computations are first employed in the initial assessment of the threat; next, in the use of cues and actions to confront it, and to appropriately update memories in the aftermath, and finally in the decisions shaped by this experience. To mimic this in the laboratory, we could identify and track a specific set of computations (for example, computation of estimation uncertainty, or ambiguity), and examine it during tasks of learning, memory and decision making, under threat as well as reward experience. This approach could identify whether dysfunction lies within the computation itself and is domain general, or within a specific domain, possibly due to disruptions in incoming and processing of a specific type of information. Such approaches, in conjunction with longitudinal designs [163] are likely to generate new insights about threat computations in the healthy brain, and about disruptions in these computations in disease (see Outstanding Questions).

Box 5. From algorithms to feelings.

In this review, we have approached threat experience as a computational process involving sensory and internal inputs computed into a behavioral output. Where should we place the subjective feeling of fear within this framework? The felt quality of emotion, the feeling of threat, is a conscious mental state. This creates a gap that is difficult to fill, between what we can measure and the conscious experience [222]. A class of theories approaches consciousness by differentiating first order representations – mental representations of our situation or states in the world (e.g., visual perception, threat) – which could be either cortical or subcortical representations, versus higher-order representations, which are representations of other representations, often attributed to the function of cortical regions [223]. From this perceptive, our review of the evidence suggests that threat computations belong to the category of first-order representations, involving not only a subcortical defensive circuitry but also computations of value, state, uncertainty, volatility and more, occurring non-consciously and engaging multiple frontal and parietal cortical regions. By contract, the feeling of fear belongs to higher-order representations, engaging other circuits and integrating several processes, including non-conscious memories, pre-existing schemas, and mental models [224, 225]. Considering fear as a representation or output of a more basic computational operation, inspired calls for revisiting our terminology to reflect the subject of investigation: using ‘threat’ to describe the inner working of defensive survival circuits, and ‘fear’ as a category of conscious experience [226].

Conscious, verbal, descriptions of emotions in terms of their valence and arousal resonate with non-conscious computations of utility and vigor [191, 227, 228], but there are no specific computations that map onto the conscious experience of fear. Nevertheless, conscious experience may influence decisions and shape the output of neural computations. For example, informing individuals about aversive contingencies influenced activation in the striatum and OFC to feedback-driven learning. Amygdala responses resisted verbal warnings and changed only when individuals had direct experience of the relationships between cues and outcomes [229]. Remembered feelings could influence the arbitration of choices by creating an anticipation about how one might feel upon incurring an outcome [230, 231]. Moods can bias the perceptions of outcomes and sway decisions. For example, a negative outcome would be perceived as worse when one is in a bad mood. Moods could be formalized as the cumulative impact of differences between actual and expected outcomes (i.e., a running average of recent prediction errors). By reflecting the momentum of an outcome in a particular environment, mood helps an individual to account for the statistics of the environment. This could be beneficial for steering away from an environment that induces bad mood, for example, since the magnitude and frequency of negative surprise increasingly grow [232].

Understanding the distinction between the feeling of fear and the neural computations of threat, can inform our approach to PTSD and anxiety disorders. Identifying from which level do symptoms arise can advise pharmacological and behavioral treatment. The interactions between feelings and emotions could refine the assessment of aberrant behaviors as well as facilitate treatment [192].

Outstanding questions.

• What explains the lingering effects of PTSD? Current learning models cannot explain the longevity of the disorder. A comprehensive model encompassing multiple learning, memory and decision-making features may be needed.

• To what extent are model-based, versus model-free, computations involved in the development and maintenance of post-trauma and anxiety symptomatology?

• How are value computations in the brain used to generate choice? Subjective values of threats, as well as rewards, are encoded in a network of brain areas, but whether and how these values are compared to produce choice is not clear.

• What is the relationship between neural representations of value and salience? Are they computed in parallel, or is one estimated first and then used to calculate the other?

• Are the neural mechanisms that encode uncertainty shared across learning, memory and decision-making processes, as well as across both threats and rewards?

• To what extent do aberrant computations confer variability to traumatic stress and anxiety and to what extent do they result from stressful experiences?

Highlights.

• The response to threat consists of multiple learning, memory and decision-making processes.

• These processes may be disrupted in anxiety and trauma-related disorders.

• We describe five stages of processing: experience of imminent threat; formation of threat associations; post-association learning; storing and updating of these associations; and decision-making under threat.

• These stages rely on overlapping computations and shared neural circuits.

• We propose that to reach a fundamental understanding of anxiety and trauma-related disorders, these processes should be studied together, rather than in isolation.

GLOSSARY

Ambiguity

a type of uncertainty in which likelihoods for potential outcomes are not precisely known, equivalent to estimation uncertainty

Associability

the degree to which a cue has previously been accompanied by a surprising outcome. Can be used to gate the learning rate in models of reinforcement learning

Avoidance learning

learning a response to evade aversive outcomes or the stimuli that predict them

Consolidation

a period in which a newly acquired memory is stabilized

Counterconditioning

pairing a threat-conditioned stimulus with a rewarding outcome to weaken a learned association

Destabilization

the shift of a reactivated memory into an unstable state, making it susceptible to change

Extinction learning

repeated presentations of a previously conditioned stimulus, in the absence of the associated outcome, result in temporary decline of memory expression

Instrumental conditioning

learning to associate actions with aversive or appetitive outcomes

Loss aversion

the relative weight that the decision maker assigns to losses compared to gains

Model-based computations

learning algorithms that rely on internal models of the environment, including action-state transition probabilities

Model-free computations

learning algorithms that learn the expected value of actions or stimuli based on sampling and direct experience of outcomes

Pavlovian conditioning

learning to associate stimuli with aversive or appetitive outcomes

Prediction error

the difference between the obtained and expected outcomes, used to drive learning in reinforcement-learning models

Predictive value

in the context of models of aversive learning, the degree of threat predicted by a cue

Reactivation

exposure to memory reminders, which may lead to destabilization of the neural representation of the memory

Reconsolidation

the process of restabilization a destabilized memory, updating it with new information. Disruption of the reconsolidation process may result in memory impairment

Risk

a type of uncertainty in which likelihoods for potential outcomes are fully known. In the context of learning, this is also known as expected or irreducible uncertainty

Subjective value

the utility of an option to the decision maker, integrating over all of the option’s properties, including potential outcomes and their likelihoods, as subjectively perceived by the decision maker

Threat imminence continuum

mapping of defensive behaviors into stages of threat, ranging from a ‘safe’ stage (no threat) to the most extreme ‘circa-strike’ stage (an attack takes place)

Unexpected uncertainty

a surprising change in the probabilistic structure of the environment

Volatility

frequency of changes in probabilistic structure of the environment. Learning rate should be higher in volatile, compared to stable, environments