Stress increases aversive prediction error signal in the ventral striatum

Abstract

From job interviews to the heat of battle, it is evident that people think and learn differently when stressed. In fact, learning under stress may have long-term consequences; stress facilitates aversive conditioning and associations learned during extreme stress may result in debilitating emotional responses in posttraumatic stress disorder. The mechanisms underpinning such stress-related associations, however, are unknown. Computational neuroscience has successfully characterized several mechanisms critical for associative learning under normative conditions. One such mechanism, the detection of a mismatch between expected and observed outcomes within the ventral striatum (i.e., “prediction errors”), is thought to be a critical precursor to the formation of new stimulus–outcome associations. An untested possibility, therefore, is that stress may affect learning via modulation of this mechanism. Here we combine a translational model of stress with a cognitive neuroimaging paradigm to demonstrate that stress significantly increases ventral striatum aversive (but not appetitive) prediction error signal. This provides a unique account of the propensity to form threat-related associations under stress with direct implications for our understanding of both normal stress and stress-related disorders.

Learning to associate cues with threat is adaptive because it allows future threat to be predicted and avoided (1). However, such associative learning also may lead to haunting and debilitating memories; the smell of stir-fry may evoke painful intrusive memories in Vietnam veterans. How aversive information is integrated to guide adaptive or maladaptive behavior, however, is not well understood. Here, we report that stress increases a key learning mechanism: the neural detection of a mismatch between an expected and observed aversive outcome within the ventral striatum, commonly referred to as the aversive prediction error (PE) signal (1, 2).

Neurocomputational and neuroeconomic accounts of cognition posit that stimulus–outcome associations during conditioning are formed via temporal difference learning (1, 3, 4), in which learning depends upon comparing expectation to what is currently happening. New associations, it is argued, are driven by a difference between predicted and actual outcomes (i.e., “prediction errors”), with greater mismatch between expected and actual outcomes evoking greater PE, resulting in greater predictive learning. This mismatch gives rise to phasic dopamine release in the ventral striatum (1, 3–5) for both appetitive and aversive stimuli (1, 2, 6, 7). Little is known, however, about how PE processing might be affected by an organism’s emotional state. Stress, for instance, is well known to facilitate aversive conditioning in both humans and animals (8, 9), raising the possibility that aversive PEs also might be increased by stress. This hypothesis, however, is untested. Therefore, here we used a translational stress induction method in healthy humans (8, 10–12) to test the hypothesis that stress elevates aversive ventral striatum PE signal.

Stress levels were modulated within subjects (n = 24) using a paradigm adapted from rodent studies: threat of foot shock (8, 10–12). Its impact upon the neural substrates of PE processing was established by recording striatal blood-oxygen-level-dependent (BOLD) signal from individuals completing a simple task designed to elicit matched appetitive (happy face) and aversive (fear face) prediction errors in the absence of behavioral confounds (Fig. 1A).

Fig. 1.

(A) The “What’s in the box?” task was completed under safe and threat conditions. Subjects were asked to predict what face would follow the box cue (fearful or happy face representing the face’s reaction to the contents of the box). On 25% of trials, the face feedback was unexpected and elicited a PE (e.g., a fear face following 75% happy face triangle cue). (B) Raw PE values calculated according to the formula in the text for a representative subject. The arrow from A to B shows the link between the illustrated “safe fear” PE and the corresponding parametric value. Facial images from ref. 34. SF, safe-fear; SH, safe-happy; TF, threat-fear; TH, threat-happy.

Separate positively signed appetitive and aversive PE values (Fig. 1B) then were used as regressors in functional MRI analysis. We restricted our search to the striatum on the basis of prior work identifying dissociable striatal regions involved in Pavlovian and instrumental PE responses (ventral and dorsal striatum respectively) (5); predicting that this task would be associated with PE signal in the ventral striatal region only.

Thus, taking the evidence that stress facilitates aversive conditioning (8, 9), we predicted that threat of shock would increase aversive PE response in the ventral striatum.

Results

As predicted, subjects reported being significantly more anxious (F_1,23 = 255, P < 0.001) during the threat (5.4/10) relative to the safe conditions (2.1/10). Critically, this provoked a threat*valence interaction in the right ventral striatum PE node (5) [peak voxel xyz = 28, 10, −10; P(family-wise error [FWE]_voxel-level) = 0.022/P(FWE_{cluster-level}) = 0.038; Fig. 2A], which was driven by significantly increased aversive PE signal under threat relative to safe (F_1,23 = 5.8, P = 0.02) (Fig. 2B) but no comparable change for appetitive PE signal (F_1,23 =1.0, P = 0.33). Moreover, this induced a significant bias toward aversive relative to appetitive PE under threat (F_1,23 = 10, P = 0.004) that was not present under safe (F_1,23 = 0.6, P = 0.46). A comparable bias in aversive PE signal was seen in the left ventral striatal, region but the threat*valence interaction was at trend level [xyz = −26, 10, −10; P(_uncorrected) = 0.008]. Moreover, there were no significant interactions in the dorsal striatal region. Furthermore, subsequent exploratory whole-brain analysis revealed additional threat*valence interactions in PE signal in the insula, parahippocampal, and supramarginal cortical regions (Table 1 and Fig. S1).

Fig. 2.

(A) Threat*valence interaction in striatal PE signal (peak xyz = 28, 10, −10; t-values, 1–2). (B) PE signal extracted from ventral striatal node (error bars represent SEM; *P < 0.05) (1). Facial images from ref. 34.

Table 1.

Exploratory whole-brain threat*valence interaction

k (cluster)	T	P	Peak	Label
173	3.54	0.000	36, −20, 14	R insula
158	3.40	0.000	−52, −50, 30	L supramarginal gyrus
21	3.12	0.001	24, −40, −6	R parahippocampal gyrus
37	3.05	0.001	30, 8, −4	R putamen

Clusters significantly activated at P < 0.005 uncorrected.

Discussion

Consistent with hypotheses, stress increased aversive PE signal within the ventral striatum while having no significant effect upon appetitive PE signals.

This suggests that the well-established facilitation of aversive conditioning under stress in both humans and animals (8, 9) may be driven by modulation of the PE stage of the conditioning mechanism. This, in turn, is consistent with prior work demonstrating that both threat of shock (10) and posttraumatic stress disorder (PTSD) (13, 14) increase the magnitude of the mismatch negativity (MMN) signal, a neural response to neutral stimulus deviance that may be conceived of as a valence nonspecific PE signal (15, 16). In fact, the region previously associated with elevated MMN under threat of shock falls just below the ventral striatum (10) and might plausibly represent the same functional component. The present findings thus suggest that this signal may be especially pronounced for aversive mismatches, adding a valence component to the stress-potentiated mismatch detection/prediction error signal.

Perhaps more importantly, however, the present findings provide a mechanism for the propensity to form threat-related emotional memories during stress (8) and in stress-related disorders (9). PEs are thought to drive the formation of stimulus–outcome associations (1, 3, 4). As such, the present findings suggest that stress ultimately may bias the formation of stimulus–threat over stimulus–reward associations via altered PE processing. From an adaptive standpoint, it makes sense to specifically bias the prediction of threats under conditions of stress, because such learning ultimately may allow the organism to avoid threats and place itself out of harm’s way (hence also reducing stress).

Excitingly, recent rodent research has provided a potential pharmacological mechanism by which this modulation may occur (17). In particular, it has been shown that stress switches the role of corticotropin-releasing hormone (CRH) in the striatum from driving corticotropin-releasing factor receptor (CRFR) 1 and CRFR2 receptor–linked appetitive responding to aversive responding, driving “a diametric change in the emotional response to acute stressors” (17). This is particularly relevant because we know that the anxiogenic effects of the translational threat-of-shock paradigm used here are CRH dependent (18). Thus, the altered PE signal in this study may be the result of a stress-driven shift in the response of the striatum to CRH.

Moreover, the translational model of stress we used is thought to tap into processes contributing to pathological anxiety (11, 12). By tying this translational model to computational models of learning, we therefore open the door to uncovering factors involved in the etiology and maintenance of anxiety disorders from a neurocomputational perspective (19). Elevated aversive PE signal, for instance, might be causally related to phenomena such as emotional responses to trauma reminder in PTSD [i.e., debilitating stimulus–threat associations (9)] or contribute to hyperarousal symptoms in stress-related disorders. Thus, going forward, elevated striatal aversive PE responses may prove a valuable putative biomarker for anxiety disorders and/or for testing the efficacy of treatments for anxiety (20). Indeed, the rodent study highlighted above suggests the CRH pathways may be promising targets for treating such symptoms (17).

It is worth mentioning that some prior studies examined the impact of various other types of stress on motivational behavior. For instance, social stress reduces the use of aversive information (21), whereas cold pressor tests may improve appetitive learning (22). Moreover, the threat-of-shock manipulation used here may reduce reward responsiveness on signal detection tasks (without aversive stimuli) (23, 24). One recent study examined the impact of stress induced by violent movies (25) on viewing morphing emotional faces in female subjects and revealed increased amygdala sensitivity (and reduced selectivity) under stress, as well as a stress*face emotion interaction in the striatum. Note that in this paper (as well as those adopting social stressors), however, subjects underwent the stress manipulation first and then completed the task. As such, it is unclear whether these studies were looking at the effect of stress (as we do) or recovery from stress. Moreover, none of these studies examined neural substrates of PEs, and there are large task-demand differences across studies. Nevertheless, it is clear that stress (and recovery from stress) may have a wide variety of effects upon cognition. It also should be noted that we did not acquire physiological responses in this study (e.g., heart rate or skin conductance). The stress paradigm was adapted from a well-validated psychophysiological threat paradigm that reliably increases physiological concomitants of stress and anxiety (26), but acquisition of such variables in future research would be valuable.

In sum, we present evidence that stress-potentiated aversive PE signal provides a plausible and unique mechanism by which stressful situations may ultimately lead to increased threat-related associations in healthy and pathologically anxious individuals. The present findings thus set the stage for future research recruiting more complex tasks and models as well as psychiatric populations to more precisely delineate the impact of stress upon aversive and appetitive conditioning. Given the enormous and growing psychological, social, and economic impact of stress-related disorders (27), such an understanding is of clear value.

Materials and Methods

Right-handed volunteers (n = 24, 16 males) aged between 18 and 50 (mean 28) successfully passed a screening procedure. Physical and mental health of the participants was determined by a physical examination performed by a physician, a clinical interview conducted by a trained psychologist using the Structured Clinical Interview for the Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV) (28), and self-report of medication and drug use confirmed by urine toxicology analysis. Exclusion criteria included psychotropic drug exposure within 3 wk, major medical or neurological illness, illicit drug use or alcohol abuse within 1 y, lifetime history of alcohol or drug dependence, psychiatric disorders, current pregnancy or breast feeding, structural brain abnormalities on MRI, and general MRI exclusions. Participants provided informed consent as approved by the National Institute of Mental Health Combined Neuroscience Institutional Review Board and were monetarily compensated for their time.

Task.

Subjects completed a simple cue–outcome “What’s in the box?” paradigm (Fig. 1A), in which boxes were associated with learned emotional outcomes [fearful or happy faces matched for arousal per ratings of the same faces in a prior study (29)] according to a partial (75%) probabilistic reinforcement strategy. There were four boxes (star, oval, diamond, and triangle), two of which (order counterbalanced) were followed by 75% happy faces and two by 75% fearful faces. On each trial, subjects were presented with a box (1,000 ms), which then “opened” (1,000–3,000 ms jitter) and was followed by a fearful or happy face (500 ms). On 25% of trials, this feedback was reversed, leading to either a fear or a happy face PE. Faces were used because they are naturalistic emotional stimuli with privileged aversive and appetitive responses (relative to more secondary, conditioned rewards and punishments, such as money) and because they provide a direct link to our recent work using the same facial stimuli during perception tasks (11, 29). [As an aside, we also showed previously that simple “smiley” cartoon happy and sad faces can elicit dissociable striatal PEs (6) that are differentially sensitive to dopamine (30), serotonin (31, 32), and major depression (33). Thus, even weak secondary emotional feedback may elicit these effects.] Subjects were told that the faces represented people’s reaction to what they saw in the box. Trials were presented in a pseudo-random counterbalanced order with 1,000–3,000 ms jittered fixation between trials. To ensure subjects attended to the task and to help identify nonresponders, subjects also were required to make a button-press feedback prediction during the box stimulus. Specifically, they were asked to predict what type of face (fearful or happy) would follow the box stimulus. Reaction time (RT) was defined as average response time to each cue pooled across valence and condition (e.g., threat happy trials); accuracy was defined as correct responses to the same cues divided by the total number of each cue (e.g., total correct safe fear/total safe fear trials).

For subjects to learn the correct conditioned stimulus (CS; Box)-faces contingencies, before the main task subjects went through a training phase in which the 75% contingencies were set at 100% (e.g., cues associated with 75% happy faces were reinforced with 100% happy faces). Subjects responded to the cues during this training, and correct learning was confirmed by asking subjects to explicitly list, following this training phase, the box-face stimulus–outcome contingencies. As such, the “learning” component of the main task involved learning, with each successive unexpected feedback, that the association between the cue and the outcome was declining. Note, however, that by design the contingencies never dropped below 50% so that although subjects were constantly learning new information about the predictive value of cues, the contingencies never reversed. This ensured that the activity associated with the PEs was not confounded by behavioral stimulus–response changes (Table 2) (as may be a problem in reversal-learning tasks). However, given that subjects were asked for a response, we explored the possibility of instrumental response biases via a “win-stay, lose-shift” analysis (see ref. 6 for more details) on the one trial following unexpected feedback. There was no threat*valence interaction on post–unexpected-feedback trials stratified either by previous trial valence (RT, P = 0.3; Accuracy, P = 0.7) or by current trial valence (RT, P = 0.3; accuracy, P = 0.6), indicating that subjects correctly maintained cue–outcome contingencies equally across valences and did not alter their behavior in response to the unexpected feedback. Moreover, across all trials, there were no generic behavioral confounds; there was no threat*valence interaction in either RT (F_1,23 = 1.7, P = 0.2) or accuracy (F_1,23 = 0.9, P = 0.4) during the prediction phase of the task.

Table 2.

Behavioral measures

Condition	Feedback	RT (SEM)	Accuracy (SEM)
Threat	Happy	603 (13)	0.93 (0.02)
	Fear	598 (13)	0.92 (0.02)
Safe	Happy	613 (13)	0.91 (0.02)
	Fear	616 (13)	0.89 (0.03)

Computational Model.

PE values were computed according to the established formula

where the PE for the current trial_(t) equaled the feedback value for the current trial minus the expected value (V) of the outcome according to

where the V of the current trial_(t) equals the V of the previous trial_(t-1) plus the PE of the previous trial multiplied by α set at 0.2 (1). In our model, we assigned value to unexpected outcomes only; expected outcomes were assigned a value of zero, and unexpected outcomes were assigned a value if 1. Positive PEs were calculated for fear vs. happy/safe vs. threat conditions separately (four total; e.g., threat fear). Given that subjects went through a training phase with 100% contingencies, the expected value of the first trial was set at zero (i.e., no expected change in outcome). Moreover, subjects received no unexpected feedback on either of the first two trials (i.e., no actual change in outcome); as such, the PE value of the first trial was zero. Consistent with prior studies, a learning rate of 0.2 was used because it is thought to fall within the naturalistic range of striatal dopamine neurons (1, 4). We also tested a learning rate of 0.1 and a categorical model defining trials as expected and unexpected (broadly equivalent to a learning rate of 1), but responses were maximal for the learning rate shown (see Table S1). It also should be noted that this model likely is an oversimplification because, for example, it does not take into account learning changes over time. However, defining such changes is difficult for this task and we would rather have a model that is overly simplistic than overfitted. Future work will develop better models, but the present model may be seen as a foundation

Manipulation.

During the main task, subjects were informed that they were at risk for shock half the time and that they were safe from shock the rest of the time. The conditions were signaled by a 4,000-ms red (shock) or blue (safe) screen as seen in Fig. 1A and were divided into eight alternating threat (four) and safe (four) conditions per run, with 10 trials per condition. While in the scanner, subjects completed one run of the task in which they got shocked on one of the threat conditions (reinforcement task), followed by one run of the task that included no shocks and during which echo-planar imaging (EPI) images were acquired. These two tasks then were repeated such that EPI images were acquired for a total of two runs of the task. Therefore, there were two total scanned runs leading to a total of 160 trials (80 per scan run), 40 of which were PE trials. This design maintained anxiety during the threat conditions (18) but, because no shocks were given during image acquisition, avoided movement artifacts associated with the shocks. Thirty seconds of fixation was included at the start and end of the task to act as a baseline for image analysis. Stress levels were determined by asking subjects to rate how anxious they felt during each condition on a scale of 1 (not at all) to 10 (extremely).

Functional Imaging.

A GE Signa HDXT 3-Tesla 940 scanner was used to acquire structural and functional images. The functional sequence comprised two EPI sessions of 284 volume acquisitions: flip angle 90°; repetition time = 2,000 ms; echo time = 30 ms; field of view (FOV) = 22 × 22 cm; slice thickness = 3.5 mm; slice spacing = 0 mm; and matrix = 64 × 64 sagittal slices with ASSET to increase coverage area. The first 10 volumes from each session were discarded to allow for magnetization equilibrium before acquisition. The structural sequence comprised a magnetization-prepared rapid gradient echo (MPRAGE) anatomical reference image: flip angle 10°; repetition time = 7,200 ms; echo time = 3,000 ms; inversion time = 450; FOV = 24 × 24 cm; slice thickness = 1.0 mm; slice spacing = 0 mm; and matrix = 224 × 224 for spatial coregistration and normalization.

Image Analysis.

Images were preprocessed and analyzed using SPM8 (Wellcome Trust Center for Neuroimaging, www.fil.ion.ucl.ac.uk/spm/). Preprocessing consisted of within-subject realignment, coregistration, segmentation, spatial normalization, and spatial smoothing. Functional scans were coregistered to the MPRAGE structural image, which was processed using a unified segmentation procedure combining segmentation, bias correction, and spatial normalization; the same normalization parameters then were used to normalize the EPI images. Finally, the EPI images were smoothed with a Gaussian kernel of 8 mm full width at half-maximum. The canonical hemodynamic response function and its temporal derivative were used as covariates in a general linear model.

At the first level, stimulus outcome (face) onsets were entered separately for the safe and threat trials (irrespective of valence) with separate reward and punishment PE values included as parametric modulators of each condition. Note that PEs were modeled at outcome, because that is the point at which predictions are confirmed. At the second level, a 2 × 2 threat*valence flexible factorial design was created from the four parametric modulator PE regressors (threat fear PE, threat happy PE, safe fear PE, and safe happy PE). Peak ventral striatum activity (FWE corrected) was determined within a 7-mm diameter small-volume corrected sphere surrounding the left and right ventral striatal Pavlovian PE peaks highlighted by O’Doherty et al. (5) (right xyz = 26, 6, –8; left xyz = –26, 8, –4). Follow-up analysis was performed using a custom script to extract mean signal across this region of interest (ROI) for each of the four PE regressors and analyzing in a 2 × 2 threat*valence general linear model in SPSS. Subsequent simple effects analysis was Bonferroni corrected. A comparable analysis also was performed for a 7-mm sphere surrounding the dorsal instrumental PE peak (5) (xyz = –8, 22, 0). Analysis of the stimulus onsets in the categorical model (i.e., onset of the face stimuli independent of the parametric modulator) revealed altered activity in regions previously associated with the processing of face stimuli, including the fusiform gyrus and superior temporal sulcus. In the contrast of stress vs. safe, there was significant activity (P_{whole-brain uncorrected} < 0.001) most strongly in the right midoccipital cortex (peak xyz = 46, −64, 28) and the dorsal medial prefrontal cortex (peak xyz = −18, 30, 46), a region we previously implicated in stress responding (11). No regions were more active for safe than stress. The PE contrast revealed no differences between reward and punishment, but a significant increase in a region adjacent to the insula (peak xyz = −42, −4, 12) was present for punishment vs. reward. The threat*valence interaction also was faintly trending in a right (but not left) amygdala anatomical ROI (peak xyz = 22, 0, −20; P_uncorrected = 0.012).