Reward Processes in Extinction Learning and Applications to Exposure Therapy

Abstract

Anxiety disorders are common and highly distressing mental health conditions. Exposure therapy is a gold-standard treatment for anxiety disorders. Mechanisms of Pavlovian fear learning, and particularly fear extinction, are central to exposure therapy. A growing body of evidence suggests an important role of reward processes during Pavlovian fear extinction. Nonetheless, predominant models of exposure therapy do not currently incorporate reward processes. Herein, we present a theoretical model of reward processes in relation to Pavlovian mechanisms of exposure therapy, including a focus on dopaminergic prediction error signaling, coinciding positive emotional experiences (i.e., relief), and unexpected positive outcomes. We then highlight avenues for further research and discuss potential strategies to leverage reward processes to maximize exposure therapy response, such as pre-exposure interventions to increase reward sensitivity or post-exposure rehearsal (e.g., savoring, imaginal recounting) strategies to enhance retrieval and retention of learned associations.

1. INTRODUCTION

Anxiety disorders are among the most common and most disabling mental health conditions (Yang et al., 2021). Psychological treatments for anxiety are widely studied and supported (Bandelow et al., 2018; Carpenter et al., 2018; Van Dis et al., 2020), with exposure therapy considered a gold-standard (Rauch et al., 2012; Abramowitz et al., 2019). However, despite the considerable effectiveness of exposure therapy, approximately 50% of individuals do not attain clinically significant treatment response (Loerinc et al., 2015). Understanding the factors associated with positive outcomes from exposure therapy may fuel future innovations in the treatment of anxiety disorders.

Current theories regarding exposure therapy derive, in part, from models of Pavlovian fear learning. As individuals learn that a neutral cue (conditional stimulus; CS) predicts an innately aversive outcome (unconditional stimulus; US_av), conditional responses correspond with elevations in measures such as peripheral physiology and subjectively reported fear (Lonsdorf et al., 2017). The prediction that a CS will lead to a US_av can lead to avoidance of the CS, which prevents new inhibitory learning from taking place and maintains CS-US_av associations. Exposure therapy is designed to directly counteract this avoidance and enable individuals to learn that a feared CS does not lead to the US_av as often as originally predicted, at least in the exposure therapy context (Craske et al., 2022).

Reward processes may further inform mechanisms of Pavlovian fear learning that are relevant to exposure therapy. For example, avoidance is thought to prompt a positive emotion of relief (Carver, 2009; Deutsch et al., 2015), which overlaps with dopaminergic mechanisms of reward signaling and may reinforce continued avoidance of the CS (San Martín et al., 2020; Vervliet et al., 2017). However, relief is also thought to occur when a CS surprisingly does not lead to a US_av (i.e., extinction), thereby reinforcing approach to the CS (Vervliet et al., 2017). Furthermore, a feared CS may sometimes precede an innately appetitive (i.e., positive) outcome (US_app), which can similarly reinforce approach to the CS (i.e., counterconditioning, see Keller et al., 2019). Approach to the CS is critical to fear extinction, which by definition involves repeated nonreinforced “exposures” to the CS. Interventions to increase reward signaling and positive emotions (e.g., relief) may therefore have the potential to enhance exposure therapy treatment outcomes (Craske et al., 2016; Zbozinek & Craske, 2017a).

Nonetheless, current theories of Pavlovian fear learning and exposure therapy rarely incorporate reward processes. Herein, we argue for several mechanisms by which reward processes may influence Pavlovian fear learning and extinction. We discuss the relevance of these theories to exposure-based treatments of anxiety disorders, particularly considering growing evidence that individuals with both anxiety and depressive disorders (often comorbid with anxiety disorders; Kessler et al., 2012) exhibit low levels of positive affect, a correlate of reward hyposensitivity (Eisner et al., 2009; Grillo, 2016; Guineau et al., 2023; Kashdan, 2007; Vinograd et al., 2021; Winer et al., 2017). We then highlight potential modifications to existing exposure protocols, emphasizing avenues for leveraging reward processes to maximize Pavlovian extinction learning processes. We conclude by offering future research directions to test the theories presented in this paper.

2. PAVLOVIAN FEAR ACQUISITION AND EXTINCTION

Associative learning models can be used to describe how fear and anxiety are initially acquired (Pittig et al., 2018). This is demonstrated in experimental fear conditioning paradigms (Lonsdorf et al., 2017) where, during acquisition, presentation of a CS (e.g., a geometric shape) may be repeatedly followed by a US_av (e.g., an electric shock). This forms an association (or a representation in memory) that the geometric shape leads to electric shock (CS-US_av), which can result in conditional fear responding to the geometric shape and avoidance behavior.¹

Following fear acquisition, one behavioral response is to prevent US_av encounters (i.e., avoidance). This can occur prior to the presentation of a CS that predicts a US_av or can occur after CS presentation but prior to the delivery of the US_av. In the short-term, avoidance appears protective by preventing the US_av from occurring. In the short- and long-term, avoidance prevents opportunities to learn whether a CS does or does not predict the US_av (Lovibond, 2006). In the absence of such learning, the original excitatory CS-US_av association, as well as the conditional fear response, is maintained (Lovibond et al., 2009; Pittig, 2019; Wong et al., 2022). With repetition, avoidance may become habitual or overgeneralized (across CSs or contexts) and continue even in situations where the CS no longer reliably predicts the US_av (LeDoux et al., 2016) and when the conditional response has subsided. These effects can explain why avoidance can continue in the absence of fear.

In extinction learning, the feared CS is repeatedly presented in the absence of the US_av, which leads to the formation of an inhibitory CS-noUS_av association in memory. From the prior example, repeated presentation of the geometric shape without electric shock leads to a new memory representation of the geometric shape no longer being a predictor of electric shock, corresponding with reduced conditional fear responding to the geometric shape. A critical aspect of this learning process is prediction error, presented in the original Rescorla-Wagner model (Rescorla & Wagner, 1972). When the CS is presented, predictions are formed about the likelihood of US_av occurrence. The prediction error that derives from the absence of the expected US_av drives extinction learning.² Mathematically, the Rescorla-Wagner model predicts that greater prediction error should correspond with greater associative learning (Rescorla & Wagner, 1972). In the inhibitory retrieval model of extinction learning (Bouton, 1993, 2002), the CS-noUS_av association does not erase the original CS-US_av association but is instead formed as an ‘inhibitory’ association that competes with the original association. This inhibitory association is contextually gated, meaning that the inhibitory CS-noUS_av association is generally specific to the context in which it is formed (Bouton, 2004).³ Return of conditional fear can therefore occur when the CS is presented in a context that differs from the context in which the inhibitory association was formed, thereby reflecting that the original CS-US_av association remains intact, even after successful extinction (Bouton, 2002, 2004). The combination of prediction error as the driver of extinction learning and competing inhibitory associations that are contextually gated forms the theoretical basis for the inhibitory retrieval model of exposure therapy (Craske et al., 2008; 2014; 2022).

One alternative is the latent cause model (Gershman & Niv, 2012), which proposes that distinct learning phases result in distinct memory states, whereas similar learning phases can be integrated into a single memory state. The latent cause model therefore argues that the original CS-US_av association may be altered by making acquisition and extinction phases more similar, such as gradually reducing the rate of US_av reinforcement during extinction (Niv, 2019). This has been shown to reduce spontaneous recovery and reinstatement relative to standard extinction in rodents (Gershman et al., 2013). Relatedly, there is some evidence to suggest that, consistent with both the inhibitory retrieval model and the latent cause model, occasional presentations of the US_av during extinction (i.e., occasional reinforced extinction) may reduce spontaneous recovery (Culver et al., 2018; Thompson et al., 2018), though the effects are not always replicated (Quintero et al., 2022). The latent cause model may further explain individual differences in extinction learning. For example, individuals who assign acquisition and extinction trials to separate states are more likely to show spontaneous recovery than individuals who assign both acquisition and extinction to a single state (Gershman and Hartley, 2015).

While the inhibitory retrieval model has been applied in clinical contexts (Craske et al., 2022), direct clinical translation of the latent cause model remains untested. In the following sections, we therefore discuss extinction learning, reward processes, and clinical applications of these concepts from an inhibitory retrieval perspective, although we additionally consider areas for further study that align with a latent cause perspective.

3. REWARD PROCESSES DURING PAVLOVIAN FEAR LEARNING

The complex process of learning, remembering, and updating CS associations can be conceptualized according to a sequence of critical events (Figure 1). Below, we consider how reward processes relate to each event along this sequence.

Figure 1.

Sequence of critical events during Pavlovian fear learning following an encounter with a CS. In a given encounter, the CS may terminate before, during, or after the US occurs.

3.1. Avoidance Behavior and Relief

Since avoidance can prevent extinction learning (Figure 1), it is critical to understand the mechanisms by which avoidance is reinforced. To address this need, recent studies have begun exploring relief, a positively valenced emotional construct that is likely to be influenced by many internal and external factors (Barrett & Westlin, 2021; Deutsch et al., 2015) and is closely tied to reward processes (e.g., self-reported reward-responsiveness, motivation, dopaminergic signaling) (Deutsch et al., 2015). In the context of Pavlovian fear learning, the absence of a US_av following a CS is theorized to activate reward-related processes that correspond with the emotional output of relief (Deutsch et al., 2015)⁴. Clinically, leaving a social interaction where rejection is anticipated may lead to a self-reported experience of relief and thus reinforce continued avoidance of social interactions. In support of this theory, a recent study showed that relief is positively valenced and that relief response to threat omission elicited a similar pattern of reward response to monetary gain across behavioral, self-report, and physiological measures (Leng et al., 2023).

Indeed, studies using active avoidance conditioning paradigms (i.e., where during or after CS presentation participants may select an avoidance response to cancel US_av delivery) have supported the theory of relief as a rewarding, positive reinforcer of avoidance. For example, avoidant choices during a CS presentation to prevent US_av occurrence are associated with self-reported relief, with stronger expectancies of US_av prior to avoidance leading to greater relief magnitude, also referred to as relief-pleasantness (San Martín et al., 2020; Vervliet et al., 2017). Moreover, using similar paradigms, relief-pleasantness following avoidance behavior has been shown to predict greater future avoidance behavior (Papalini et al., 2021). Thus, researchers have hypothesized that relief positively reinforces avoidance behavior (Papalini et al., 2021; San Martín et al., 2020; Vervliet et al., 2017), and thus may maintain the CS-US_av relationship and thereby perpetuate fear of the CS (Lovibond et al., 2009; Pittig, 2019; Wong et al., 2022).

Other studies have linked the concept of relief to subjective perceptions of certainty and controllability. For instance, one study found that anxious individuals showed lower confidence in their avoidance responses and therefore experienced greater relief-pleasantness following successful avoidance (De Kleine et al., 2023). Similarly, another study found that limiting the number of avoidance responses available prior to a CS corresponded with lower confidence in avoidance responses, thereby increasing relief-pleasantness following the non-occurrence of the US_av (Cobos et al., 2022). It may therefore be possible to reduce relief-related reinforcement of avoidance behaviors by 1) increasing the availability of avoidance behaviors throughout an encounter with a CS, and 2) increasing confidence that the available avoidance behaviors will be successful. By the same token, however, increasing avoidance opportunities and confidence in avoidance ability may serve as reliable safety signals (see 3.2.4 Incentives to Reduce Avoidance Behavior for further discussion), reduce US expectancy, and therefore obstruct inhibitory learning during extinction and exposure. Thus, these possibilities require additional research.

3.2. Fear Extinction

Herein we consider ways in which reward processes relate to US outcomes and subsequent prediction error signaling during fear extinction (Figure 1). These include 1) during US_av omissions (CS-noUS_av) that promote extinction learning, 2) during US_av occurrences that are less aversive than expected, and 3) during US_app occurrences (CS-US_app) that may further promote extinction learning (e.g., counterconditioning). We then consider strategies for incentivizing approach behavior to the US_av via simultaneous delivery of a US_app.

3.2.1. Prediction Error: Unexpected US_av Omissions

Dopaminergic signaling has been shown to be critical to prediction error processes in appetitive conditioning paradigms, where the unconditional stimulus is rewarding (e.g., food, money). Specifically, the unexpected occurrence of a US_app is associated with dopamine signaling in the nucleus accumbens from neurons in the ventral tegmental area (Schultz, 2016). A number of rodent studies support the role of dopaminergic processes in fear extinction as well. Specifically, a dopaminergic response has been identified during omission of an expected US_av in fear extinction (Kalisch et al., 2019; Gentry et al., 2019; Salinas-Hernández & Duvarci, 2021).⁵ Further, the magnitude of dopamine signaling has been associated with the strength of fear extinction learning, measured by conditional freezing behavior in mice (Salinas-Hernández et al., 2018). Inhibition of dopaminergic signaling impairs fear extinction, whereas activation accelerates fear extinction (Luo et al., 2018; Salinas-Hernández et al., 2018). Activation of dopaminergic signaling immediately after fear extinction has been associated with enhanced extinction memory storage and retrieval (Kalisch et al., 2019; Salinas-Hernández & Duvarci, 2021).

This set of animal findings suggests that omission of an expected US_av during fear extinction shares biological processes with the unexpected occurrence of a US_app during appetitive conditioning (Gentry et al., 2019). Furthermore, the basolateral amygdala (BLA) receives dopaminergic inputs from the ventral tegmental area and substantia nigra (Rosenkrantz & Grace, 1999; Pezze & Feldon, 2004; Salinas-Hernández & Duvarci, 2021), and specific subpopulations of BLA neurons are associated with reward-seeking behaviors (Kim et al., 2016). In a study of fear extinction among mice, these same BLA neurons were further found to be both “necessary and sufficient” for storing of extinction memories, which has been posited as evidence that extinction memories may be stored as rewarding information (Zhang et al., 2020). Together, these findings suggest that reward and fear extinction processes may rely on common dopaminergic mechanisms of appetitive learning, and further support the assertion that relief during unexpected US_av omission elicits a rewarding appetitive response (Leng et al., 2023). However, the directional nature of the relationship between prediction error learning and relief remains unclear. It is possible that 1) greater prediction error learning produces a greater positive emotional output (i.e., relief-pleasantness), 2) greater relief-pleasantness produces greater prediction error learning, or 3) the two processes are distinct but emerge from common dopaminergic mechanisms of extinction learning. Further research is needed to clarify the precise relationship among dopamine signaling, prediction error learning, and relief-pleasantness during unexpected US_av omission.

In addition to the dopamine system, other neural systems associated with reward are implicated in fear extinction. For example, studies have identified overlap between endogenous opioids and fear learning and extinction (Meier et al., 2021). Inhibition of opioid receptors has been shown to sustain fear conditioned responses during fear acquisition (Eippert et al., 2009) and impair fear extinction learning (Bengoetxea et al., 2020; McNally, 2009), which suggests that activation of opioid receptors may be critical to the prediction error process during fear extinction (McNally, 2009). Opiodergic signaling has been implicated in studies of relief from physical pain, such that blocking endogenous opioids reduces self-reported relief-pleasantness (Sirucek et al., 2021) (although there is some evidence suggesting dopaminergic, rather than opiodergic, modulation, see Desch et al., 2023). It has been similarly posited that blocking opioid receptors prevents the rewarding experience of relief associated with unexpected US_av omission (McNally & Westbrook, 2003), although empirical evidence during human fear extinction is needed. Additional research should evaluate the relationship between relief following US_av omission and activation of the opioidergic system.

As previously discussed, the subjective experience of relief has most often been studied in relation to avoidance conditioning paradigms that result in prevention of potential US_av delivery (Papalini et al., 2021; San Martín et al., 2020; Vervliet et al., 2017), and subjective relief correlates with and predicts future avoidance behavior (Papalini et al., 2021). However, when approaching the fearful CS corresponds with unexpected omission of a US_av (i.e., fear extinction), this may similarly result in the subjective experience of relief. In support of this notion, several studies using both fear extinction and avoidance conditioning paradigms have shown that greater prediction error on a given trial corresponds with greater relief-pleasantness at the end of that trial (San Martín et al., 2020; Vervliet et al., 2017; Willems & Vervliet, 2021). Relatedly, as prediction error declines across fear extinction trials (an indication of successful extinction learning) or across repeated avoidance trials (an indication of successful learning of avoidance contingencies), self-reported relief-pleasantness also declines (San Martín et al., 2020; Vervliet et al., 2017).

Although avoidance conditioning paradigms share similarities with fear conditioning and extinction paradigms, there are key differences in the role of prediction error. In avoidance conditioning, the mismatch between an expected US_av and its unexpected omission leads to learning that an active avoidance behavior (e.g., pressing a button) prevents US_av occurrence, which can reinforce further avoidance behavior and prevent CS-US_av extinction learning. In fear extinction, the mismatch between an expected US_av and its unexpected omission leads to learning that approaching the CS does not lead to US_av occurrence, which can reinforce further approach behavior and thus support extinction learning. For the purposes of clinical application, there is a need to find therapeutic strategies for 1) reducing the relief experienced during anticipatory or active avoidance, and 2) increasing the relief experienced at the point of unexpected US omission during an exposure.

3.2.2. Prediction Error: US_av Aversiveness

Conditional fear following a CS is related to the aversiveness of a predicted US_av (Hosoba et al., 2001) and reduces when US_av intensity is reduced (Du et al., 2015). When an individual experiences a US_av, this occurrence provides an opportunity for the individual to learn that the feared US_av is not as bad as anticipated, thereby reducing conditional fear to the CS.⁶ On such occasions, individuals may experience partial US_av omission (i.e., the US_av was not fully omitted, but it did not fully occur as anticipated). Similar to other prediction error mechanisms of learning, it is possible that the mismatch between expected US_av aversiveness and actual US_av aversiveness corresponds with a rewarding relief signal to drive extinction learning.⁷ For example, if a socially anxious individual experiences rejection from their peers, and if that rejection was less aversive than expected, the individual may experience relief and begin to show reduced fear during future encounters with their peers. Of note, occasional exposure to the US_av during extinction has been hypothesized to reduce US_av aversiveness (Culver et al., 2018), which has been shown to reduce reacquisition of fear in some studies (Craske et al., 2022) and may relate to prediction error mechanisms. Additional research is needed to characterize the extent to which prediction error mechanisms of US_av aversiveness relate to relief signaling and therefore support fear extinction.

3.2.3. Prediction Error: Counterconditioning

Counterconditioning involves repeated pairing of the CS that had been originally paired with a US_av with a US_app instead and has been shown to lessen conditional fear responses (de Jong et al., 2000). Counterconditioning with a US_app is likely to involve reward-related processes, although other processes may exist as well (e.g., formation of associations that inhibit original excitatory fear associations; Keller et al., 2020). Specifically, pairing a CS that is expected to predict a US_av with US_app may boost reward processes because of augmentation of the mismatch between expected and actual (US_av omission and US_app receipt) outcomes.

Yet, evidence for the efficacy of counterconditioning of fear conditioning is mixed (for a review, see Keller et al., 2020). On the one hand, some evidence suggests that counterconditioning may lead to better extinction learning and less spontaneous recovery of fear (i.e., return of fear with the passage of time) compared to standard extinction procedures. For example, counterconditioning has been associated with greater reduction of US_av expectancies during extinction and reduced spontaneous recovery (Kang et al., 2018), as well as weaker conditional (physiological) responses to the extinguished CS during extinction recall (Keller & Dunsmoor, 2020). However, other evidence has shown that, compared to standard extinction, counterconditioning does not reduce spontaneous recovery or reinstatement of fear (i.e., return of fear following uncued US occurrence) (van Dis et al., 2019; Quintero et al., 2023; Chen et al., 2022), and some evidence even suggests that conditional fear is more readily renewed following counterconditioning compared with standard extinction (Holmes et al., 2016). This is thought to occur because counterconditioning is more context-dependent than standard extinction (Holmes et al., 2016). Additionally, these studies demonstrated mixed results regarding valence ratings to the CS, with one study finding increased positive valence of the CS directly following counterconditioning (van Dis et al., 2019) and another study finding no change to CS valence (Kang et al., 2018). Methodological differences may account for discrepant findings, as studies have used a range of appetitive unconditional stimuli (e.g., sucrose, cartoons, positive film clips, pleasant music) and aversive unconditional stimuli (e.g., electric shocks, aversive sound). It may be particularly important to consider the aversiveness of the US_av and positivity of the US_app in counterconditioning experiments to ensure sufficient contrast between the stimuli (Quintero et al., 2023).

Whereas both extinction and counterconditioning appear to recruit the dopamine system, no studies to our knowledge have examined the connection between counterconditioning and subjective reporting of positive emotion. While prediction error during CS-noUS_av learning is associated with the positive emotion of relief, counterconditioning may similarly relate to positive emotions associated with the receipt of rewards (i.e., enjoyment). It is conceivable that equivocal findings in the counterconditioning literature are attributable to variations in US_app stimuli and associated dopaminergic reward signaling. It may be possible for counterconditioning to outperform extinction (i.e., lead to greater reductions in spontaneous recovery and reinstatement) if the occurrence of an unexpected US_app activates positive emotional states over and above relief. Additional research is needed to explore the role of inter-individual differences in reward signaling and subjective experiences of positive emotion during counterconditioning.

3.2.4. Incentives to Reduce Avoidance Behavior

Prevention of avoidance is crucial as it is necessary for extinction learning and exposure therapy. Thus, research on explicit prevention of avoidance may lead to strategies for maximizing learning in extinction and exposure therapy. Several studies have examined co-occurring rewards to incentivize approach behavior (Dibbets & Fonteyne, 2015; Pittig, Hengen, et al., 2018; Pittig & Dehler, 2019). For example, participants are more likely to approach an aversive CS (e.g., image of a spider) in the presence of monetary or social rewards (Pittig et al., 2018; Pittig & Dehler, 2019). Notably, the receipt of reward does not appear to reduce fear in response to the CS (Pittig & Dehler, 2019), and reward magnitude appears to be weighted less heavily than threat probability during approach-avoidance decisions (Boschet et al., 2022). Nonetheless, in a fear extinction paradigm where the CS is no longer paired with the US_av, such increased approach behavior can facilitate extinction by creating more prediction error learning opportunities. Furthermore, evidence suggests that use of external rewards to incentivize reduced avoidance during extinction may also lead to reduced future avoidance and return of fear compared to traditional extinction where avoidance options are removed (Pittig & Wong, 2022).

Although research is more limited, evidence suggests that external rewards can reduce the use of safety signals, which is another form of avoidance. Safety signals reduce US_av expectancy or directly predict US absence when present (Salkovskis, 1991; Wells et al., 1995; Wong et al., 2022). For example, presence of an anxiolytic medication (safety signal) may reduce expectancies that sensations of lightheadedness (CS) will lead to passing out (US_av), as is commonly feared in the case of panic disorder. Through this process, the feared CS-US_av association is maintained as opportunities for extinction learning via prediction error are reduced or removed (Lovibond et al., 2000, 2009). As such, safety signals are generally discouraged during exposure therapy.⁸ It is possible that these principles may be leveraged to overcome safety signal use. For example, in a fear conditioning safety behavior experiment that tested the impact of a rewarding US_app, participants were less likely to select a safety button that blocked a US_av when forgoing the safety behavior was paired with a US_app, which in turn facilitated more rapid extinction learning (Pittig, 2019). Considerations for clinical applicability are discussed below.

4. INDIVIDUAL DIFFERENCES: MODERATORS OF REWARD, LEARNING, AND MEMORY

Broadly speaking, reward processes are considered central to the experience of positive emotions (Becker et al., 2019). Initial genome-wide association studies have begun to identify associations between genetic markers of positive affect and reward processes. For example, genetic markers linked to trait positive emotion have been associated with greater activation of the ventral striatum in response to positive facial images (Wingo et al., 2016). Another study showed similar findings for receipt of reward during a probabilistic decision-making task but failed to replicate effects using positive food-related images (Lancaster et al., 2017). Thus, further study is needed (Bondy & Bogdan, 2022; Kujawa et al., 2020). Neural measures of reward sensitivity, including activation of the ventral striatum in response to reward anticipation and receipt of reward during a card-guessing task, also correlate with state positive affect (Forbes et al., 2010). Furthermore, individual differences in positive affect such as anhedonia (characterized by low positive affect and limited interest or joy in activities), are associated with deficits in reward processes including both reward motivation and consumption (Borsini et al., 2020; Huys et al., 2013; Slaney et al., 2022). The below sections detail potential avenues for understanding how individual differences in positive emotions, a correlate of reward processes, relate to fear extinction.

4.1. Positive Affect & Reward Processes during Extinction

As previously stated, unexpected omission of a US_av following CS occurrence (e.g., engaging in a social interaction and not being rejected) is thought to trigger the positive emotional state of relief. As with avoidance behavior, the magnitude of this experience (i.e., relief-pleasantness) tracks closely with prediction error and is therefore considered central to the formation of CS-noUS associations as they begin to compete with CS-US_av associations (Vervliet et al., 2017). For example, maximal relief-pleasantness is reported when there is greater certainty that a CS will lead to a US_av, and no US_av occurs (Willems & Vervliet, 2021). Reported relief-pleasantness then declines during subsequent extinction trials in parallel with decreases in CS-US_av predictions across trials (Vervliet et al., 2017). The role of individual differences in relief-pleasantness is beginning to be explored. For example, elevated anhedonia is associated with reduced self-reported relief-pleasantness during avoidance conditioning (Leng et al., 2022, 2024). Furthermore, elevated anhedonia is associated with distinct patterns of brain activity during fear extinction, including regions of canonical threat neurocircuitry such as the dorsal anterior cingulate, anterior insula, and amygdala that may further relate to individual differences in relief-pleasantness (although this remains an area of ongoing research) (Rosenberg et al., 2022; Young et al., 2021). However, to further examine the directions of these relationships, future studies should specifically test whether individual differences in self-reported relief-pleasantness can predict subsequent learning (e.g., prediction error on subsequent extinction trials).

4.2. Positive Affect & Processes in Learning and Memory that Inform Fear Extinction

Existing studies have demonstrated a connection between positive affect and broader learning and memory principles, which may in turn be applied to extinction. For example, although not specifically tested in fear conditioning and extinction paradigms, positive affect has been associated with enhanced encoding, rehearsal, and retrieval processes (Zbozinek & Craske, 2017a), described in more detail in the following sections. Regarding extinction, these memory processes are essential to the formation and retention of inhibitory CS-noUS associations (Figure 1) and therefore suggest the potential importance of positive affect during extinction learning.

4.2.1. Encoding & Consolidation

Following the initial learning of associations (e.g., CS-US_av, CS-noUS_av, CS-US_app), newly learned information is encoded and consolidated in memory for later retrieval. Variability in how deeply associations are initially encoded influences how these associations are subsequently stored or retrieved (Craik, 2002). Shallow encoding typically involves sensory or surface features (e.g., color, brightness) whereas deep encoding typically involves more complex features (e.g., meaning, inference) and requires greater attentional processing of the encoded stimulus (Craik & Lockhart, 1972; Craik, 2002; Ekuni et al., 2011).

Research suggests that both negative and positive mood can influence encoding processes. Negative mood is associated with more shallow encoding and less cognitive effort expenditure during learning (Bolte et al., 2003; Ellis et al., 1984; Hartlage et al., 1993; Leight & Ellis, 1981) as well as a tendency to engage in referential processing (i.e., processing new information independently of prior knowledge) (Storbeck & Clore, 2008). This is relevant to fear extinction learning as more shallow encoding of CS-noUS relationships may impact subsequent retrieval of extinction. Furthermore, referential processing may disrupt the extinction learning process, as individuals may be slower to update predictions in light of newly learned information. Alternatively, positive mood is associated with semantic learning processes that increase the depth of encoding and enhance long-term retention of learned information (Federmeier et al., 2001; Hänze & Hesse, 2008; Isen, 1987), which may be particularly important for retention of new CS-noUS associations. Positive mood also enhances relational processing (i.e., the integration of new information with prior knowledge) (Clore & Huntsinger, 2007) (Storbeck & Clore, 2008) in ways that may improve updating of expectancies during extinction learning. One potential example relates to the valence of the CS. During fear acquisition, the CS tends to take on a negative valence, given its association with a US_av (Hermans et al., 2002). During extinction, the induction of positive mood can increase positive valence of the CS (Zbozinek et al., 2015). Since the omission of a US_av is encoded as rewarding, increased positive valence of the CS may serve to augment extinction learning as more positively valenced cues may be more readily associated with positive outcomes, in part due to enhanced relational processing.

4.2.2. Rehearsal & Retrieval

Rehearsal of learned information can enhance consolidation and retrieval of long-term memories. In the inhibitory retrieval model of exposure therapy, this strategy takes the form of verbally and imaginally rehearsing the absence of the expected US_av (Craske et al., 2022; McGlade & Craske, 2021). Mental rehearsal of the CS-US_av association can maintain conditional fear (Joos et al., 2012, 2013), whereas repeated mental rehearsal of the CS-noUS in exposure therapy can reduce fear (McGlade & Craske, 2021). Furthermore, rehearsing a subset of learned information has been shown to increase memory while decreasing memory for non-rehearsed information belonging to the same category (i.e., retrieval-induced forgetting) (Anderson, 2003; Anderson et al., 1994, 2000). Thus, by intentionally engaging in rehearsal of learned information, it may be possible to selectively strengthen the consolidation and later retrieval of specific memories, such as CS-noUS_av. Similarly, such a strategy could be implemented for selective strengthening of an appetitive CS-US_app association.

Positive mood is thought to be associated with deeper mental rehearsal of stored information, thereby enhancing consolidation and retrieval of long-term memories (Craik, 2002; Craik & Lockhart, 1972). Positive mood has been found to enhance retrieval-induced forgetting effects (Bäuml & Kuhbandner, 2007), possibly because of more relational processing (i.e., relating new information to previously learned information) than negative mood. Furthermore, retrieval of stored memories is often congruent with positive or negative mood (Matt et al., 1992), such that depressed individuals tend to recall information that is depression-congruent (Watkins, 1992). Conversely, memory retrieval may be biased toward CS-noUS_av or CS-US_app associations when mood is positive. Indirect support of this notion includes evidence for higher positive affect to be associated with less reacquisition (Dour et al., 2016; Zbozinek & Craske, 2017b) and reinstatement (Zbozinek et al., 2015) of conditional fear following extinction, although discrepant findings (e.g., van Dis et al., 2019) motivate further research in this area.

If the absence of the US_av during fear extinction is encoded as positive and rewarding, then perhaps positive mood enhances mood-congruent rehearsal of extinction learning (i.e., CS-noUS_av) relative to original fear memories (i.e., CS-US_av). In turn, if positive mood enhances rehearsal of positive (i.e., CS-noUS_av) but not negative (i.e., CS-US_av) associations, positive mood may therefore facilitate a decreased strength of retrievability of negative fear associations over time via retrieval-induced forgetting mechanisms. Alternatively, if negative mood is associated with mood-congruent rehearsal of original fear memories (i.e., CS-US_av) relative to extinction memories (i.e., CS-noUS), negative mood inductions may increase the retrievability of fear associations. These possibilities could be tested by implementing a positive versus negative mood induction between fear extinction and reacquisition or between fear extinction and extinction recall.

5. CLINICAL IMPLICATIONS

In the section that follows, we provide an outline for future study of potential clinical applications of the relationships between reward processes and extinction learning. Specifically, we identify potential methods to be tested that could maximize prediction error learning and target reward processes, rehearsal and retrieval, and positive affect to facilitate improvements in exposure therapy outcomes.

5.1. Exposure Therapy as an Application of Extinction Learning

The associative learning model of fear acquisition and extinction can be applied clinically to inform the treatment of anxiety disorders via exposure therapy (Craske et al., 2018; Mineka & Oehlberg, 2008). Exposure therapy techniques have evolved over the years from initial systematic desensitization approaches (Wolpe, 1961, 1968) to emotional processing theories (Foz & Kozak, 1986; 1996), and more recently, inhibitory retrieval models of exposure therapy that emphasize maximizing prediction error (i.e., expectancy violation: the violation of expected US_av occurrence) (Craske et al., 2022). This is achieved through exposures designed to test out predictions and provide potent mismatches between expectancies and outcomes, drawing upon principles such as attentional salience, removal of conditional inhibitors or safety signals, deepened extinction, occasional reinforced extinction (when appropriate), and mental rehearsal for consolidation of extinction learning. Additional strategies are utilized to offset contextual specificity of extinction learning, such as stimulus and contextual variation throughout exposure (Craske et al., 2022).

5.2. Prediction Error during Exposure Therapy

Current exposure approaches based on the inhibitory retrieval model emphasize maximizing expectancy violation (the clinical application of prediction error) across exposures to promote inhibitory learning and methods for promoting generalization and retrieval of the learning across time and context (see Craske et al., 2022 for a review). Building upon this existing framework, strategies aimed at maximizing reward sensitivity may serve to further support expectancy violation, as well as generalization and retrieval of learning, during exposure therapy (Figure 2, Table 1).

Figure 2.

Theoretical model of reward-related processes before and during exposure therapy.

Table 1.

Reward-focused interventions and their potential applications in exposure therapy.

Possible Strategies to Target Reward Processes	Timing of Intervention
	Prior to Exposure	During Exposure	Post Exposure
Positive Mood Induction (e.g., imagining of future positive outcomes, exposure to positive images, or exposure to positive memories)	X
Positive Affect Treatments (e.g., PAT or APT to increase overall reward sensitivity and enhance subsequent learning and memory processes)	X	X
Reduce Reward Signal Associated with Avoidance (i.e., decreasing reinforcing strength of using safety signals)		X*
Testosterone/Estradiol Administration	X*	X*
Selection of Potentially Rewarding Exposures (e.g., social encounters with possibility of deepening meaningful relationships)		X
Therapist Reinforcement (e.g., in-session praise of approach behavior)		X
Connecting Exposures to Eudaimonic Rewards (e.g., sense of achievement, mastery, values)		X	X
Positive-Focused Rehearsal Strategies (e.g., savoring, imaginal recounting to maximize extinction and retrieval)		X	X

^*As direct evidence is currently lacking, these approaches should be considered highly tentative at this time.

In exposure therapy, individuals are specifically instructed to reduce avoidance in order to engage in new learning (Craske et al., 2022). As described in 3.2.1 Prediction Error: Unexpected US_av Omissions, relief following unexpected US_av omissions may be central to prediction error mechanisms of extinction learning. If so, methods that maximize the relieving or rewarding experience associated with prediction error (i.e., absence of the feared outcome) could be explored. For example, savoring (i.e., using first-person, present-moment language to rehearse the most positive aspects of an event) (see Sandman & Craske, 2022) during the post-exposure consolidation phase may encourage re-experiencing of the moment at which the expected feared outcome did not occur (e.g., savoring the positive experience of relief during a public speaking exposure upon realizing that the audience was not rejecting). This process could deepen inhibitory CS-noUS_av learning and therefore strengthen extinction learning during exposure therapy.

Relatedly, studies of depression have highlighted a tendency for depressed individuals to update their beliefs from moderate expectancy violations more so than from maximal expectancy violations (Kube et al., 2022). This suggests a non-linear relationship between US expectancy and expectancy violation among depressed individuals (Kube et al., 2020), which contrasts with traditional linear models of expectancy violation (e.g., Rescorla & Wagner, 1972). These results have been interpreted as depressed individuals tending to overlook or minimize positive information that disconfirms negative expectations (i.e., cognitive immunization theory) (Liknaitzky et al., 2017; Everaert et al., 2018; Kube & Glombiewski, 2021; Kube et al., 2022; Kube & Rozenkrantz, 2021). These findings are consistent with the observation that depressed individuals tend to show lower positive affect following surprising positive events as they occur naturalistically in daily life (Villano & Heller, 2024). Some evidence also supports a sigmoidal relationship between reward expectancy and reward positivity, which allows there to be little difference in the positivity of reward when reward is unexpected versus very unexpected (Williams et al., 2017). If the most extreme expectancy violations are not accompanied by similarly heightened reward signaling, which may be the case as a function of reward hyposensitivity that is common to depression (Wang et al., 2022) and anhedonia (Borsini et al., 2020; Nusslock & Alloy, 2017; Slaney et al., 2022), then these exposures may be most susceptible to cognitive immunization effects (e.g., “This is too good to be true”).

The latent cause model may offer one alternative explanation for these phenomena, as maximizing expectancy violation may prompt individuals to seek alternative explanations for US_av non-occurrence (i.e., formation of a novel latent state) (Niv, 2019). If so, there may be an optimal level of expectancy violation that promotes new learning but is unlikely to create a novel latent state, therefore enabling effective extinction of the original CS-US_av association. Furthermore, it is possible that individuals with depression (or elevated anhedonia symptoms) are particularly likely to seek latent causes for unexpected threat omissions, contributing to the cognitive immunization effects described above. Additional work is needed to test these possibilities and to identify other strategies for integrating these principles clinically (i.e., to enhance extinction mechanisms of exposure therapy) (Craske et al., 2022).

Alternatively, individuals with heightened anhedonia or comorbid anxiety and depressive disorders may be particularly likely to benefit from strategies designed to maximize positive emotions during exposures. This may counteract cognitive immunization effects by instead encouraging the capturing and maximizing of the reward signal as it occurs. One such strategy involves the previously described savoring of the absence of the aversive outcome or presence of a rewarding outcome in the post-exposure consolidation phase. Notably, experiential processing of positive autobiographical memories (first person, present tense, with a focus upon sensory and situational details) was found to increase positive affect and reduce tendencies to dampen or dismiss the positive experience relative to analytical or naturalistic processing (Sandman & Craske, under review). Such a strategy may disrupt cognitive immunization effects and promote expectancy violation. A second strategy entails positive mood inductions prior to exposure trials, such as by repeated imagining of future positive outcomes (Zbozinek et al., 2015), exposure to positive images (Zbozinek & Craske, 2017), or exposure to positive autobiographical memories (Sandman & Craske, under review). Though some of these strategies have been linked to improved long-term extinction learning (Zbozinek et al., 2015; Zbozinek & Craske, 2017), it is possible that positive mood induction also increases the reward signal during exposures (thereby deepening CS-noUS_av learning). Third, therapeutic strategies designed to increase overall reward sensitivity, such as Positive Affect Treatment (PAT) (Craske et al., 2019; 2023), may increase all positive emotions, including relief, and therefore facilitate expectancy violation during exposure. Thus, first conducting treatments such as PAT – especially among individuals with anxiety and low reward processing and low positive affect – may enhance the effects of subsequent exposure therapy. Future research is needed to clarify the possible impact of reward- and positive affect-enhancing strategies before and during exposure therapy.

Finally, when a predicted US_av does not occur and a US_app occurs instead (i.e., counterconditioning), the formation of this novel CS-US_app association may then further inhibit CS-US_av associations during extinction. For example, in addition to the absence of rejection during a speech exposure, a person with social anxiety may receive positive feedback during their presentation. Although there is mixed evidence regarding counterconditioning effects versus standard extinction related to spontaneous recovery or reinstatement of fear (see 3.2.3 Prediction Error: Counterconditioning), it may nonetheless be beneficial for therapists to maximize US_app occurrences during therapy. For example, therapists may 1) aid in the selection of exposures that have an increased possibility of leading to a powerful US_app (e.g., attending a personally-meaningful social gathering), and 2) apply strategies to amplify reward responsiveness (e.g., PAT; Craske et al., 2019; Craske et al., 2023) following US_app occurrences. Future research is needed to determine optimal strategies for incorporating counterconditioning principles in exposure therapy.

5.3. Leveraging Reward Processes to Facilitate Exposures

Exposure to the CS is, of course, a necessary step of exposure therapy. Indeed, individuals with anxiety demonstrate greater relief following avoidance (De Kleine et al., 2023), underscoring the need for creative strategies for increasing approach behavior. Building upon prior work in laboratory studies (see 3.2.4 Incentives to Reduce Avoidance Behavior), external and internal rewards may be incorporated clinically to facilitate approach behavior in myriad ways.

It may be possible to leverage rewards during exposure to compete with US_av occurrence. For example, therapists commonly apply positive reinforcement techniques (e.g., praise) to support therapeutic alliance and engagement (Tsascher et al., 2012). Such strategies may be especially applicable during exposures (e.g., praise for remaining in the threatening context without escaping, reducing reliance on safety behaviors, or maintaining attention to the CS). Praise can function as a competing US_app and reinforce future approach behaviors even when a US_av has occurred (similar to Pittig & Dehler, 2019), thereby enabling future opportunities for CS-noUS_av learning. Furthermore, social support figures have been theorized as “prepared fear suppressors” that can signal accessibility of survival resources and reduce aversiveness of the US_av without acting as a safety signal (i.e., without reducing the likelihood of US_av occurrence) (Hornstein et al., 2022). It is possible that, in addition to suppressing fear, social support figures may further support extinction by encouraging approach behaviors (i.e., social support as a US_app). The function of social support as a prepared fear suppressor, in addition to reinforcement of approach behavior, may help explain why it is associated with sustained fear extinction (Hornstein et al., 2022). Further research is needed to understand mechanisms of social support to enhance exposure, while minimizing social support as a safety signal.

Although initial CS-US_app associations typically involve hedonic rewards (i.e., short-term enjoyment or pleasure), another possibility is to incentivize engagement in exposures in service of longer-term outcomes (i.e., goals, values-based living, sense of achievement or mastery) (Papageorgiou & Karekla, 2023) that contribute to eudaimonic well-being, which in turn is associated with physical and psychological health benefits (Cole et al., 2015; Fredrickson et al., 2013, 2015; Kitayama et al., 2016). For example, individuals with social anxiety may continue to attend social engagements if socializing also enables the goal of building more positive, long-term relationships. There is some preliminary evidence for this notion, since greater personal values and self-efficacy were found to predict self-reported willingness to engage in social interactions despite fear or anxiety (Lee & Yeghiazarian, 2021). However, it remains unclear whether approaches that emphasize values and long-term goals are more effective than exposure approaches without that emphasis, and thus studies that directly compare such approaches are needed.

Although rewards have been shown to incentivize CS approach, even when the CS strongly predicts the US_av (Pittig, 2019; Pittig & Dehler, 2019), it is important to note that these effects might be lessened in clinical anxiety, as individuals with elevated anxiety tend to avoid more than healthy control participants (Pittig et al., 2021; Pittig & Scherbaum, 2020) when presented with a competing US_app. Given the finding that reward information is weighed less than threat information in favor of avoidance during approach-avoidance conflicts (Boschet et al., 2022), perhaps anxious individuals show even greater avoidance in these circumstances due to the heightened rewarding value of the avoidance behavior itself (De Kleine et al., 2023). Thus, applying therapeutic strategies designed to enhance attention and sensitivity to reward in new contexts (e.g., PAT, as described above) prior to exposure may boost the potency of the competing US_app, thereby facilitating a greater willingness to approach exposures.

5.4. Rehearsal & Retrieval

Mental rehearsal of the inhibitory CS-noUS_av association immediately following exposure and between exposure sessions has been shown to facilitate greater symptom improvement among a sample of individuals with spider phobia (McGlade & Craske, 2021). Building upon this work to incorporate reward-relevant processes, strategies could be implemented immediately following or between exposure sessions that emphasize rehearsal and retrieval of positively valenced exposure information. Examples of such strategies include imaginal reexperiencing and description of positive events during exposure trials, akin to strategies within PAT (Craske et al., 2019, 2023) and positive memory specificity training with regards to the experience during exposure (Arditte Hall et al., 2018; Chen et al., 2021). Imaginal recounting or memory specificity training could 1) enhance positive emotions following US non-occurrence or US_app delivery during the post-exposure consolidation phase, 2) serve as a form of mental rehearsal, thereby improving later retrieval of the newly learned inhibitory CS-noUS relationship, or 3) serve as a form of retrieval-induced forgetting, which could facilitate extinction of the CS-US_av association and improve memory for the CS-noUS association. To test these hypotheses, future studies may examine whether positive-focused imaginal recounting immediately following exposure or repeatedly between exposure sessions leads to improved treatment response.

5.5. Positive Affect

As previously stated, positive affect is linked to basic encoding, rehearsal, and retrieval learning processes in myriad ways. Clinicians may be able to apply this in exposure therapy to specifically target extinction learning processes (Craske et al., 2016; Zbozinek & Craske, 2017a). The association between low positive mood and poorer treatment outcomes (Craske et al., 2016; Sandman & Craske, 2022) may occur, in part, because low mood is associated with reduced effort during learning and shallowly encoded associations (Bolte et al., 2003; Ellis et al., 1984; Hartlage, et al., 1993; Leight & Ellis, 1981). Furthermore, it is possible that negative mood may lead to recall of memories in line with the feared outcome (e.g., a memory of a social interaction that preceded rejection), whereas positive mood may instead lead to recall of memories in line with extinction (e.g., a memory of a social interaction that preceded a new friendship). Thus, interventions that generally aim to increase positive mood could be applied prior to or in conjunction with exposure therapy to 1) deepen encoding of new associations as they are learned, 2) augment retrieval of extinction-congruent memories, which are associated with positive emotional experiences (e.g., relief or reward consumption), and 3) impede retrieval of fear-congruent memories (i.e., retrieval-induced forgetting), which are associated with negative emotional experiences (e.g., disappointment or loneliness). Strategies can be drawn from existing interventions such as PAT, which aims to increase positive affect and reward sensitivity by increasing engagement in positive activities, increasing attention to positive experiences, and increasing cultivation and savoring of positive experiences (Craske et al., 2019, 2023), or Amplification of Positivity Treatment (APT), which aims to increase positive affect by increasing engagement in positive thinking, emotions, and behavior (Kryza-Lacombe et al., 2021; Taylor et al., 2017, 2020, 2024). For example, prior to a course of exposure therapy, clinicians may implement strategies to generally increase positive affect. During a course of exposure therapy, clients may be instructed to more specific strategies, such as 1) intentionally attending to the positive emotions of relief and enjoyment, 2) applying strategies to prolong or deepen these experiences (e.g., increasing the duration of the positive social interaction), or 3) limiting distractions from the positive experience (i.e., maintain absorption).

5.6. Additional Considerations – Sex Differences

Though findings are mixed, there is some evidence to suggest that sex assigned at birth is associated with differences in reward processes (Bangasser & Cuarenta, 2021; Dhingra et al., 2021; Warthen et al., 2020) and extinction learning (Velasco et al., 2023), which in turn may relate to exposure therapy. One possibility is that sex hormones modulate reward sensitivity and thereby have downstream impacts on extinction and exposure outcomes. For example, administration of testosterone has been associated with increased reward sensitivity (Hermans et al., 2010) and approach behavior (Enter et al., 2016). Clinically, testosterone administration has also been associated with heightened initial fear during an exposure, followed by greater fear reduction at a subsequent exposure for public speaking anxiety (Hutschemaekers et al., 2021). Findings are somewhat mixed for estradiol (Hornung et al., 2020), though some studies have found that elevated estradiol (Diekhof, 2018) and estradiol administration are associated with increased reward sensitivity (Bayer et al., 2020; Veselic et al., 2021). Clinically, low estradiol due to hormonal contraceptive usage was associated with greater fear and avoidance after an exposure for spider phobia (Graham et al., 2018) and poor extinction recall after exposure for women with obsessive-compulsive disorder only (Levy et al., 2024). However, Levy and colleagues (Levy et al., 2023) failed to find differences in exposure outcomes between women with low versus high estradiol.

Together, these findings may further support the notion that threat and reward processes rely on overlapping biological systems. Alternatively, sex hormones may be associated with both threat and reward processes through independent pathways, such that manipulations of sex hormones may selectively impact threat versus reward processing depending on the circumstance. Future research is needed to further explore the extent to which these systems overlap. Furthermore, research is needed to test the potential roles of sex assigned at birth, individual differences in sex hormones, and administration of sex hormones as moderators of exposure outcomes (Velasco et al., 2023). Given these studies have not yet been conducted, existing findings at least highlight the need for clinician consideration of potential individual differences in exposure outcomes based on the presence of sex hormones.

6. CONCLUSION

In summary, we have presented a theoretical model of how reward processes relate to Pavlovian fear learning and may be leveraged in the treatment of anxiety disorders. Rewards can incentivize approach behavior to feared stimuli, thereby facilitating opportunities for extinction. Additionally, positive emotional experiences during extinction, such as relief, are thought to rely on dopaminergic reward signaling during formation of inhibitory CS-noUS_av or CS-US_app associations. Furthermore, individual differences in positive emotional experiences may moderate encoding, rehearsal, and retrieval of extinction memories (Zbozinek & Craske, 2017a). Given that reward deficits are associated with anxiety-related disorders (Eisner et al., 2009; Kashdan, 2007; Vinograd et al., 2021) and are associated with poor treatment outcomes (Craske et al., 2016; Sandman & Craske, 2022), we contend that reward processes remain understudied in exposure. Thus, future research is needed to elucidate how these processes may be leveraged during exposure therapy to maximize therapy effectiveness and inform treatment selection and personalization.

HIGHLIGHTS.

• Pavlovian fear learning, and particularly extinction, involves reward processes.

• Reward processes are rarely incorporated in conceptualizations of exposure therapy.

• Incorporating reward strategies may maximize the effectiveness of exposure therapy.

• Additional research at the intersection of threat and reward learning is warranted.