The role of the striatum in aversive learning and aversive prediction errors

Abstract

Neuroeconomic studies of decision making have emphasized reward learning as critical in the representation of value-driven choice behaviour. However, it is readily apparent that punishment and aversive learning are also significant factors in motivating decisions and actions. In this paper, we review the role of the striatum and amygdala in affective learning and the coding of aversive prediction errors (PEs). We present neuroimaging results showing aversive PE-related signals in the striatum in fear conditioning paradigms with both primary (shock) and secondary (monetary loss) reinforcers. These results and others point to the general role for the striatum in coding PEs across a broad range of learning paradigms and reinforcer types.

1. Introduction

The rapid growth of Neuroeconomics has yielded many investigations on valuation signals, absolute and temporal preferences and risky decision making under uncertainty (for reviews see Glimcher & Rustichini (2004), Camerer et al. (2005) and Sanfey et al. (2006)). One key theme emerging is the link between certain neural structures, specifically the striatum, and the subjective value an individual assigns to a particular stimulus (Montague & Berns 2002; Knutson et al. in press). This subjective value is represented prior to choice, encouraging exploratory or approach behaviour, and is dynamically updated according to a previous history, such as when an outcome deviates from expectation. Currently, most of these neuroimaging studies focus on reward processing and use instrumental designs where outcomes are contingent on specific actions and subjective value deviates from positive expectations. As a consequence, the striatum's role in affective learning and decision making in the context of Neuroeconomics has been primarily associated with the domain of reward-related processing, particularly with respect to prediction error (PE) signals.

It is unclear, however, whether the role of the striatum in affective learning and decision making can also be expanded to general aversive processing and learning signals necessary for aversive learning. This is a topic of growing interest in Neuroeconomics given classic economic topics such as loss and risk aversion (Tom et al. 2007), framing and the endowment effect (De Martino et al. 2006) and even social aversions that may arise due to betrayals of trust (Baumgartner et al. 2008). To date, most of our knowledge of the neural basis of aversive processing comes from a rich literature that highlights the specific contributions of the amygdala in the acquisition, expression and extinction of fear (for reviews see LeDoux 1995). In contrast to the instrumental paradigms typically used in Neuroeconomic studies of decision making, most investigations of aversive learning have used classical or Pavlovian conditioning paradigms, in which a neutral stimulus (the conditioned stimulus, CS) comes to acquire aversive properties by simple pairing with an aversive event (the unconditioned stimulus, US).

Although the amygdala has been the focus of most investigations of aversive learning, whereas the striatum is primarily implicated in studies of reward, there is increasing evidence that the role of both of these structures in affective and particularly reinforcement learning is not so clearly delineated. The goal of the present paper is to explore the role of the striatum in aversive processing. Specifically, we focus on the involvement of the striatum in coding PEs during Pavlovian or classical aversive conditioning. To start, we briefly review the involvement of the amygdala, a structure traditionally linked to fear responses, in affective learning. This is followed by a consideration of the role of the striatum in appetitive or reward conditioning. Finally, we discuss the evidence highlighting the contributions of the striatum in aversive processing, leading to an empirical test of the role of the human striatum in aversive learning. One key finding consistent across the literature on human reward-related processing is the correlation between striatal function and PE signals, or learning signals that adjust expectations and help guide goal-directed behaviour (Schultz & Dickinson 2000). However, if the striatum is involved in affective learning and decision making and not solely reward-related processing, similar signals should be expected during aversive conditioning. We therefore propose to test an analogous correlation between PE signals and striatal function with two separate datasets from our laboratory on aversive learning with either primary (Schiller et al. in press) or secondary (current paper) reinforcers using a classical fear conditioning paradigm, which is usually linked to amygdala function (Phelps et al. 2004).

(a) Amygdala contributions to affective learning

In a typical classical fear conditioning paradigm, a neutral event, such as a tone (the CS), is paired with an aversive event, such as a shock (the US). After several pairings of the tone and the shock, the presentation of the tone itself leads to a fear response (the conditioned response, CR). Studies investigating the neural systems of fear conditioning have shown that the amygdala is a critical structure in its acquisition, storage and expression (LeDoux 2000; Maren 2001). Using this model paradigm, researchers have been able to map the pathways of fear learning from stimulus input to response output. Although the amygdala is often referred to as a unitary structure, several studies indicate that different subregions of the amygdala serve different functions. The lateral nucleus of the amygdala (LA) is the region where the inputs from the CS and US converge (Romanski et al. 1993). Lesions to the LA disrupt the CS–US contingency, thus interfering with the acquisition of conditioned fear (Wilensky et al. 1999; Delgado et al. 2006). The LA projects to the central nucleus (CE) of the amygdala (Pare et al. 1995; Pitkanen et al. 1997). Lesions of CE block the expression of a range of CRs, such as freezing, autonomic changes and potentiated startle, whereas damage to areas that CE projects to interferes with the expression of specific CRs (Kapp et al. 1979; Davis 1998; LeDoux 2000). The LA also projects to the basal nucleus of the amygdala. Damage to this region prevents other means of expressing the CR, such as active avoidance of the CS (Amorapanth et al. 2000).

Investigations of the neural systems of fear conditioning in humans have largely supported and extended these findings from non-human animals. Studies in patients with amygdala lesions fail to show physiological evidence (i.e. skin conductance) of conditioned fear, although such patients are able to verbally report the parameters of fear conditioning (Bechara et al. 1995; LaBar et al. 1995). This explicit awareness and memory for the events of fear conditioning is impaired in patients with hippocampal damage, who show normal physiological evidence of conditioned fear (Bechara et al. 1995). Brain imaging studies show amygdala activation to a CS that is correlated with the strength of the CR (LaBar et al. 1998). Amygdala activation during fear conditioning occurs when the CS is presented both supraliminally and subliminally (Morris et al. 1999), suggesting awareness and explicit memory are not necessary for amygdala involvement. Technical limitations largely prevent the exploration of roles for specific subregions of the amygdala in humans, but the bulk of evidence suggests that this fear-learning system is relatively similar across species.

Although most investigations of amygdala function focus on fear or the processing of aversive stimuli, it has been suggested that different subregions of the amygdala may also play specific roles in classical conditioning paradigms involving rewards. When adapted to reward learning, the CS would be a neutral stimulus, such as the tone, but the US would be rewarding, for example a food pellet. Owing to the appetitive nature of the US, this type of reward-related learning is often referred to as appetitive conditioning (Gallagher et al. 1990; Robbins & Everitt 1996). It is hypothesized that the basolateral nucleus of the amygdala (BLA) may be particularly important for maintaining and updating the representation of the affective value of an appetitive CS (Parkinson et al. 1999, 2001), specifically through its interactions with the corticostriatal and dopaminergic circuitry (Rosenkranz & Grace 2002; Rosenkranz et al. 2003). Accordingly, the BLA may be involved in specific variations of standard appetitive conditioning paradigms, such as when using a secondary reinforcer as a US, coding that that value of the US has changed after the conditioning paradigm, and interactions between Pavlovian and instrumental processes (Gallagher et al. 1990). Interestingly, because similar effects were found following disconnection of the BLA and the nucleus accumbens (NAcc), amygdala–striatal interactions appear to be critical for processing of information about learned motivational value (Setlow et al. 2002). A number of studies have shown that the CE may be critical for some expressions of appetitive conditioning, such as enhanced attention (orienting) to the CS (Gallagher et al. 1990), and for controlling the general motivational influence of reward-related events (Corbit & Balleine 2005). In humans, there is some evidence for amygdala involvement in appetitive conditioning. For example, patients with amygdala lesions are impaired in conditioned preference tasks involving rewards (Johnsrude et al. 2000). In addition, neuroimaging studies have reported amygdala activation during an appetitive conditioning task using food as a US (Gottfried et al. 2003).

(b) Striatal contributions to affective learning

The striatum is the input of the basal ganglia and consists of three primary regions encompassing a dorsal (caudate nucleus and putamen) and ventral (NAcc and ventral portions of caudate and putamen) subdivision. A vast array of research exists highlighting the role of the striatum and connected regions of the prefrontal cortex during affective learning essential for goal-directed behaviour (for a review see Balleine et al. 2007). These corticostriatal circuits allow for flexible involvement in motor, cognitive and affective components of behaviour (Alexander et al. 1986; Alexander & Crutcher 1990). Anatomical tracing work in non-human primates also highlights the role of midbrain dopaminergic structures (both substantia nigra and ventral tegmental area) in modulating information processed in corticostriatal circuits. Such work suggests that an ascending spiral of projections connecting the striatum and midbrain dopaminergic centres creates a hierarchy of information flow from the ventromedial to the dorsolateral portions of the striatum (Haber et al. 2000). Therefore, given its connectivity and anatomical organization, the striatum finds itself in a prime position to influence different aspects of affective learning, ranging from basic classical and instrumental conditioning believed to be mediated by more ventral and dorsomedial striatum regions (e.g. O'Doherty 2004; Voorn et al. 2004; Delgado 2007) and progressing to procedural and habitual learning thought to be dependent on dorsolateral striatum (e.g. Jog et al. 1999; Yin et al. 2005, 2006). This flow of information would allow an initial goal-directed learning phase that slowly transfers to habitual processing (Balleine & Dickinson 1998).

Since the goal of this paper is to examine whether the role of the striatum during reward-related processing extends to aversive learning particularly in the context of PEs, we will consider striatal function during similar appetitive and aversive classical conditioning paradigms. In our review, we highlight the contributions of the striatum to affective learning by first discussing the role of the striatum in appetitive or reward conditioning that has more often been linked to the integrity of corticostriatal systems. We then consider the role of the striatum in aversive learning, a domain particularly linked to amygdala function as previously discussed.

(i) Appetitive conditioning in the striatum

Neurophysiological evidence outlining the mechanisms of associative learning of rewards has been elegantly demonstrated by Schultz et al. (1997). According to this research, phasic signals originating from dopamine (DA) neurons in the non-human primate midbrain are observed upon unexpected delivery of rewards, such as a squirt of juice (the US). After repeated pairings with a visual or auditory cue (the CS), responses of DA neurons shift to the onset of the CS, rather than the delivery of the liquid. That is, DA responds to the earliest predictor of the reward, a signal that can be modulated by magnitude (Tobler et al. 2005) and probability (Fiorillo et al. 2003) of the rewarding outcome. Additionally, omission of an expected reward leads to a depression in DA firing. These findings and others (e.g. Bayer & Glimcher 2005) led researchers to postulate that dopaminergic neurons play a specific role in reward processing, but not as a hedonic indicator. Rather, the dopaminergic signal can be thought of as the coding for ‘PEs’, i.e. the difference between the reward received and the expected reward (Schultz & Dickinson 2000), a vital signal to learning and shaping of decisions.

As previously mentioned, both dorsal and ventral striatum are innervated by dopaminergic neurons from midbrain nuclei, contributing to the involvement of the striatum in reward-related learning (Haber & Fudge 1997). Infusion of DA agonists in the rodent striatum, for example, leads to enhanced reward conditioning (Harmer & Phillips 1998). Further, increases in DA release, measured through microdialysis, have been reported in the ventral striatum not only when rats self-administer cocaine (the US), but also when they are solely presented with a tone (the CS) that has been previously paired with cocaine administration (Ito et al. 2000). Consistent with these studies, lesions of the ventral striatum in rats impair the expression of behaviours indicating conditioned reward. For instance, rats with ventral striatum lesions are less likely to approach a CS-predicting reward than non-lesioned rats (Parkinson et al. 2000; Cardinal et al. 2002). Similarly, upon establishing place preference using classical conditioning by exposing hungry rats to sucrose in a distinctive environment, lesions of the ventral striatum abolish this learned response (Everitt et al. 1991). Consistent findings were also demonstrated in non-human primates (e.g. Apicella et al. 1991; Ravel et al. 2003). Striatal neurons show an increased firing rate during presentation of cues that predict a reward, selectively firing at reward-predicting CSs after learning (Schultz et al. 2003). Moreover, associations between actions and rewarding outcomes were also found to be encoded in the primate caudate nucleus (Lau & Glimcher 2007).

Consistent with the findings from these animals models, brain imaging studies in humans have widely reported activation of the striatum during appetitive conditioning tasks with both primary (e.g. O'Doherty et al. 2001; Pagnoni et al. 2002; Gottfried et al. 2003) and secondary (e.g. Delgado et al. 2000; Knutson et al. 2001b; Kirsch et al. 2003) reinforcers. For instance, in a probabilistic classical conditioning paradigm with instruction (i.e. participants are aware of the contingency), activation of the ventral caudate nucleus is observed when comparing a conditioned reinforcer paired with a monetary reward ($4.00) with a non-predictive CS (Delgado et al. 2008; figure 1). This region of interest (ROI) is similar in location to a ventral caudate ROI identified in a classical conditioning paradigm with food, or primary rewards (O'Doherty et al. 2001). Interestingly, a dichotomy between dorsal and ventral striatum has been suggested in human conditioning studies, building on the ‘actor–critic’ model (Sutton & Barto 1998). According to this model, the ‘critic’ learns to predict future rewards whereas the ‘actor’ processes outcome information to guide future behaviour. While some studies suggest that the ventral parts of the striatum are involved in both classical and instrumental conditioning, in turn serving in the role of the critic (O'Doherty 2004), activity in the dorsal striatum resembles the actor, being linked primarily to instrumental conditioning (Elliott et al. 2004; O'Doherty 2004; Tricomi et al. 2004), when rewards are contingent on a purposeful action and inform future behaviour (Delgado et al. 2005).

Figure 1.

(a,b) Striatal responses during reward conditioning with secondary reinforcers. In this paradigm, the participants are presented with two conditioned stimuli that predict a potential reward (CS+, blue bar) or not (CS−, yellow bar). Adapted with permission from Delgado et al. (2008). ROI, region of interest.

(ii) Temporal difference learning in the striatum

Computational models have been particularly influential in understanding the role of the striatum in reward-related learning. Theoretical formulations of reinforcement learning suggest that learning is driven by deviation of outcomes from our expectations, namely PEs. These errors are continuously used to update the value of predictive stimuli (Rescorla & Wagner 1972). Based on this, the temporal difference (TD) learning rule (Sutton & Barto 1990) has been shown to account for the previously discussed electrophysiological data from appetitive conditioning (Montague et al. 1996; Schultz et al. 1997).

The PE signal, which is the key component of reinforcement learning model, can be used to both guide learning and bias action selection. Simply put, positive PEs occur when an unexpected outcome is delivered, while negative PEs occur when an expected outcome is omitted. When the delivery of an outcome is just as expected, a PE signal is zero. This model has been robustly tested in human and non-human animals with reward paradigms. It has been examined to a lesser extent in paradigms involving aversive learning, where some confusion can arise since a PE in an aversive context (i.e. non-delivery of an expected punishment) could be viewed as a positive outcome. Yet, in a TD model where outcomes are treated as indicators, regardless of their valence, PE signals are always negative in this case of the unexpected omission of the outcome (positive or negative). Here, we consider neuroimaging studies of PEs during human reward learning, before discussing PEs during aversive learning in the later sections.

Sophisticated neuroimaging studies incorporating TD learning models and neural data during appetitive conditioning or reward learning have started to identify the neural correlates of PE signals in the human brain (e.g. McClure et al. 2003; O'Doherty et al. 2003; Schönberg et al. 2007; Tobler et al. 2007). The first reports used classical conditioning studies with juice rewards and found that activation, indexed by blood-oxygen-level-dependent (BOLD) signals, in the ventral (O'Doherty et al. 2003) and dorsal (McClure et al. 2003) putamen correlated with a PE signal. Interestingly, the location within the striatum (putamen, NAcc and caudate) varies across paradigms (e.g. classical, instrumental) and even with different types of stimuli (e.g. food, money). PEs in the human striatum have also been observed to correlate with behavioural performance in instrumental-based paradigms (for a review see O'Doherty 2004). In most of these paradigms, PE signals were observed in both dorsal and ventral striatum and were stronger in the participants who successfully learn (Schönberg et al. 2007), while being dissociable from pure goal values or the representation of potential rewards by a stimulus or action (Hare et al. 2008). Finally, extensions of PE models to more social situations are observed in both ventral and dorsal striatum with social stimuli such as attractive faces (Bray & O'Doherty 2007), as well as trust and the acquisition of reputations (King-Casas et al. 2005).

Although the neurophysiological data implicate midbrain DA neurons in coding a PE signal, functional magnetic resonance imaging (fMRI) investigations often focus on the dopaminergic targets such as the striatum. This is primarily due to the difficulty in generating robust and reliable responses in the midbrain nuclei, and the idea that the BOLD signals are thought to reflect inputs into a particular region (Logothetis et al. 2001). Notably, a recent fMRI study used high-resolution fMRI to investigate the changes in the human ventral tegmental area according to PE signals (D'Ardenne et al. 2008). The BOLD responses in the ventral tegmental area reflected positive PEs for primary and secondary reinforcers, with no detectable responses during non-rewarding events. In sum, there is considerable evidence that corticostriatal circuits, modulated by dopaminergic input, are critically involved in appetitive or reward conditioning, and are particularly involved in representations of PE signals, guiding reward learning.

(iii) Aversive processing and the striatum

Evidence for the role of striatum in affective learning is not strictly limited to appetitive conditioning, but was also demonstrated in various tasks involving aversive motivation (for reviews, see Salamone (1994), Horvitz (2000), Di Chiara (2002), White & Salinas (2003), Pezze & Feldon (2004), McNally & Westbrook (2006) and Salamone et al. (2007)). Animal research on aversive learning has implicated, in particular, the DA system in the striatum. In the midbrain, DA neurons appear to respond more selectively to rewards and show weak responses, or even inhibition, to primary and conditioned aversive stimuli (Mirenowicz & Schultz 1996; Ungless et al. 2004). However, elevated DA levels were observed in the NAcc not only in response to various aversive outcomes, such as electric shocks, tail pinch, anxiogenic drugs, restraint stress and social stress (Robinson et al. 1987; Abercrombie et al. 1989; McCullough & Salamone 1992; Kalivas & Duffy 1995; Tidey & Miczek 1996; Young 2004), but also in response to CSs predictive of such outcomes or exposure to the conditioning context (Young et al. 1993, 1998; Saulskaya & Marsden 1995b; Wilkinson 1997; Murphy et al. 2000; Pezze et al. 2001, 2002; Josselyn et al. 2004; Young & Yang 2004). DA in the NAcc is also important for aversive instrumental conditioning as seen during active or passive avoidance and escape tasks (Cooper et al. 1974; Neill et al. 1974; Jackson et al. 1977; Schwarting & Carey 1985; Wadenberg et al. 1990; McCullough et al. 1993; Li et al. 2004). The levels of striatal DA reflect the operation of various processes including the firing of DA neurons, DA reuptake mechanisms and activation of presynaptic glutamatergic inputs, which operate on different time scales. It should be noted, however, that the striatum receives inputs from other monoaminergic systems, which might also convey aversive information, either on its own or through complex interactions with the DA system (Mogenson et al. 1980; Zahm & Heimer 1990; Floresco & Tse 2007; Groenewegen & Trimble 2007).

Lesions of the NAcc or temporary inactivation studies have yielded a rather complex pattern of results, with different effects of whole or partial NAcc damage on cue versus context conditioning (Riedel et al. 1997; Westbrook et al. 1997; Haralambous & Westbrook 1999; Parkinson et al. 1999; Levita et al. 2002; Jongen-Relo et al. 2003; Schoenbaum & Setlow 2003; Josselyn et al. 2004). This has been taken to suggest that the NAcc subdivisions, namely the ventromedial shell and the dorsolateral core, make unique contributions to aversive learning. Accordingly, the shell might signal changes in the valence of the stimuli or their predictive value, whereas the core might mediate the behavioural fear response to the aversive cues (Zahm & Heimer 1990; Deutch & Cameron 1992; Zahm & Brog 1992; Jongen-Relo et al. 1993; Kelley et al. 1997; Parkinson et al. 1999, 2000; Pezze et al. 2001, 2002; Pezze & Feldon 2004).

In addition to the ventral striatum, evidence also exists linking the dorsal striatum with aversive learning. Specifically, lesions to this region have been shown to produce deficits in conditioned emotional response, conditioned freezing and passive and active avoidance (Winocur & Mills 1969; Allen & Davison 1973; Winocur 1974; Prado-Alcala et al. 1975; Viaud & White 1989; White & Viaud 1991; White & Salinas 2003).

Consistent with the rodent data, human fMRI studies also identify the striatum in classical and instrumental learning reinforced by aversive outcomes. Although the striatum is rarely the focus of neuroimaging studies on fear conditioning and fear responses, a number of studies using shock as a US report activation of the striatum in these paradigms, in addition to amygdala activation (Buchel et al. 1998, 1999; LaBar et al. 1998; Whalen et al. 1998; Phelps et al. 2004; Shin et al. 2005). Striatal activation has also been reported in expectation of thermal pain (Ploghaus et al. 2000; Seymour et al. 2005), and even monetary loss (Delgado 2007; Seymour et al. 2007; Tom et al. 2007). The striatum has also been reported during direct experience with noxious stimuli (Becerra et al. 2001) and avoidance responses (Jensen et al. 2003). Aversive-related activation has been observed throughout the striatum, and the distinct contribution of the different subdivisions (dorsal/ventral) has not been clearly identified to date, potentially due to limitations in existing fMRI techniques. However, activation of the striatum during anticipation of aversive events is not always observed (Breiter et al. 2001; Gottfried et al. 2002; Yacubian et al. 2006), with some reports specifically suggesting that the ventral striatum is solely involved in the appetitive events, and not responsive during anticipation of monetary loss (Knutson et al. 2001a).

(iv) PE and striatum during aversive processing

It is generally agreed that the striatum is probably involved in the processing of aversive PE in learning paradigms. However, the nature of the aversive PE signal is still under debate. For example, the neurotransmitter systems carrying the aversive PE signal are unclear. One possibility is that the serotonin released from dorsal raphe nucleus may be the carrier of the aversive PE and act as an opponent system to the appetitive dopaminergic system (Daw et al. 2002). However, there is evidence that dopaminergic modulations in humans affect the PE-related signals not only in appetitive (Pessiglione et al. 2006) but also in aversive conditioning (Menon et al. 2007). Consistent with this finding, there are numerous demonstrations that DA release increases over baseline during aversive learning in rodents (Young et al. 1993, 1998; Saulskaya & Marsden 1995a; Wilkinson 1997; Murphy et al. 2000; Pezze et al. 2001, 2002; Josselyn et al. 2004; Young 2004). These data suggest the possibility that DA codes both appetitive and aversive PEs.

Another issue under debate is where these aversive PEs are represented in the brain. Converging evidence from experiments in humans that adopt fMRI as the major research tool has suggested that aversive and appetitive PEs both may be represented in the striatum (Seymour et al. 2005, 2007; Kim et al. 2006; Tom et al. 2007), albeit spatially separable along its axis (Seymour et al. 2007). However, in a study using an instrumental conditioning paradigm (Kim et al. 2006), researchers failed to find aversion-related PE signals in the striatum, while observing them in other regions including the insula, the medial prefrontal cortex, the thalamus and the midbrain.

A related question is how exactly does the BOLD signal in the striatum correspond to the aversive error signal? Studies of PEs for rewards have shown that outcome omission (i.e. negative PE) results in deactivation of striatal BOLD signal (e.g. McClure et al. 2003; O'Doherty et al. 2003; Schönberg et al. 2007; Tobler et al. 2007). It might be argued that reward omission is equivalent to the receipt of an aversive outcome. Accordingly, one might expect that in the case of aversive outcomes, positive PEs would similarly result in deactivation. An alternative hypothesis would be that positive and negative PEs are similarly signed for both appetitive and aversive outcomes. In support of the latter hypothesis, a number of studies suggest that the same relation existing between striatal BOLD signals and appetitive PEs also applies for aversive PEs (Jensen et al. 2003; Seymour et al. 2004, 2005, 2007). For example, using a high-order aversive conditioning paradigm, Seymour et al. (2004) showed that the BOLD responses in the ventral striatum increase following unexpected delivery of the aversive outcome, and decrease following unexpected omission of it.

An interesting question arises in aversive conditioning tasks by considering the consequences of this omission, i.e. the by-product of relief and its rewarding properties. To examine this using a classical aversive conditioning procedure, the participants experienced prolonged experimentally induced tonic pain and were conditioned to learn pain relief or pain exacerbation (Seymour et al. 2005). The appetitive PE signals related to relief (appetitive learning) and the aversive PE signals related to pain (aversive learning) were both represented in the striatum. These findings support the idea that the striatal activity is consistent with the expression of both appetitive and aversive learning signals.

Finally, when examining the neural mechanisms mediating the aversive PE signal, it is important to take into consideration the type of learning procedure and the type of reinforcement driving it. It is possible that the use of either primary or secondary aversive reinforcers, in either classical or instrumental paradigms, is the cause for some reported inconsistencies in the aversive learning and PE literature. For example, Seymour et al. (2007) used a secondary reinforcer (monetary loss or gain) in a probabilistic first-order classical delay conditioning task, but used a high-order aversive conditioning task (Seymour et al. 2004) when examining the effect of a primary reinforcer (thermal pain). Jensen et al. (2003) conducted a direct comparison between classical and avoidance conditioning but focused on primary reinforcers (electric shock). Given these discrepancies a more careful examination of the differences and commonalities between the processing of primary versus secondary reinforcers during the same aversive learning paradigm may yield useful insight into the role of the striatum in encoding aversive PEs.

(c) Experiment: PEs during classical aversive conditioning with a monetary reinforcer

The goal of this experiment was to examine the representation of aversive PEs during learning with secondary reinforcers (money loss) using a classical conditioning task typically used in aversive conditioning studies. Further, we compare the results with a previous study in our laboratory, which used a similar paradigm with a primary reinforcer (Schiller et al. in press). Unlike previous studies that suggested similarities between primary and secondary reinforcers during aversive learning in the striatum (Seymour et al. 2004, 2007), this experiment takes advantage of more similar paradigms previously used with electric shock (primary reinforcer) to investigate similarities in PE representations with monetary loss (secondary reinforcer), allowing more direct comparisons to be drawn regarding the underlying neural mechanisms of aversive learning.

The data from our study with a primary reinforcer are published elsewhere (Schiller et al. in press) and will only be summarized here. We examined the role of striatum in aversive learning using a typical primary reinforcer (mild shock to the wrist) using a fear discrimination and reversal paradigm (Schiller et al. in press). During acquisition, the participants learned to discriminate between the two faces. One face (CS+) co-terminated with an electric shock (US) on approximately one-third of the trials and the other face (CS−) was never paired with the shock. Following acquisition, with no explicit transition, a reversal phase was instituted where the same stimuli were presented only with reversed reinforcement contingencies. Thus, the predictive face was no longer paired with the shock (new CS−), and the other face was now paired with the shock on approximately one-third of the trials (new CS+). Using fMRI, we examined which regions correlated with the predictive value of the stimuli as well as with the errors associated with these fear predictions. For the latter, we used a PE regressor generated by the TD reinforcement-learning algorithm (see below). We found robust striatal activation, located at the left and right striatum, tracking the predictive value of the stimuli throughout the task. This region showed stronger responses to the CS+ versus CS−, and flexibly switched this responding during the reversal phase. Moreover, we found that the striatal activation correlated with the PEs during fear learning and reversal. The region that showed PE-related activation was located in the head of the caudate nucleus (Talairach coordinates: left: x, y, z=−7, 3, 9; right: x, y, z=9, 5, 8).

In the present study, we further characterize the role of the striatum during aversive learning by investigating PE signals during learning with a secondary aversive reinforcer, namely, monetary loss. The acquisition phase was similar to the one described above. Also similar was the use of the TD learning rule to assess the neural basis of PEs during aversive conditioning.

2. General methods

(a) Participants

Fourteen volunteers participated in this study. Although behavioural data from all the 14 participants are presented, 3 participants were removed from the neuroimaging analysis due to excessive motion. The participants responded to posted advertisement and all the participants gave informed consent. The experiments were approved by the University Committee on Activities Involving Human Subjects.

(b) Procedure

The experiment consisted of two parts (figure 2): a gambling session (adapted from Delgado et al. 2000) and an aversive conditioning session (adapted from Delgado et al. 2006). The goal of the gambling session was to endow the participants with a monetary sum that would be at risk during the aversive conditioning session.

Figure 2.

Experimental paradigm. The experimental task consisted of two parts: (a) a gambling session to allow participants to earn a monetary endowment (adapted from Delgado et al. 2000) and (b) an aversive conditioning paradigm where presentation of the unconditioned stimulus (−$2.00) led to monetary detractions from the total sum earned during the gambling session (adapted from Delgado et al. 2006).

In the gambling session, the participants were told they were playing a ‘card-guessing’ game, where the objective was to determine whether the value of a given card was higher or lower than the number 5 (figure 2a). During each trial, a question mark was presented in the centre of the ‘card’, indicating that the participants had 2 s to make a response. Using a MRI compatible response unit, the participants made a 50/50 choice regarding the potential outcome of the trial. The outcome was either higher (6, 7, 8, 9) or lower (1, 2, 3, 4) than 5. The outcome was then displayed for 500 ms, followed by a feedback arrow (which indicated positive or negative feedback) for another 500 ms and an inter-trial interval of 13 s before the onset of the next trial. A correct guess led to the display of a green upwards arrow indicating a monetary reward of $4.00 (reward trials), while an incorrect guess led to the display of a red downwards arrow indicating a monetary loss of −$2.00 (punishment trials). In some trials, irrespective of guess, the outcome was ‘5’ and led to the display of a blue dash, resulting in no monetary gain or loss (neutral trials). Each trial was presented for 16 s and there were three blocks of 18 trials each (total trials: 21 reward, 21 punishment and 12 neutral). Unbeknownst to the participants, the outcomes were predetermined ensuring a 50 per cent reinforcement rate and a fixed profit across the participants. At the end of the gambling session, a screen appeared congratulating the participant for earning the sum of $42.00 and informing them that the second part was about to start.

In the second part of the experiment, the participants were exposed to an aversive conditioning session with monetary reinforcers (figure 2b). The participants were presented with blue and yellow squares (the conditioned stimuli: CSs) for 4 s, followed by a 12 s inter-trial interval. The unconditioned stimulus (US) was loss of money, depicted by the symbol −$2.00 written in red ink and projected inside the square for 500 ms. In this partial reinforcement design, one coloured square (e.g. blue) was paired with the monetary loss (CS+) on approximately 36 per cent of the trials, while another coloured square (e.g. yellow) was never paired with the US (CS−). The participants were instructed that they would see different coloured squares and occasionally an additional −$2.00 sign indicating that $2.00 were to be deducted from their total accumulated during the gambling session. They were not told about the contingencies. There were 54 total trials broken down evenly into three blocks of 18 trials. There were 21 CS− trials and 33 CS+ trials, of which 12 were paired with the US.

At the end of the aversive conditioning session, the monetary penalties accumulated resulted in a total of $24.00. The participants then performed a final round of the gambling game to ensure that each participant was paid $60.00 in compensation following debriefing.

(c) Physiological set-up, assessment and behavioural analysis

Skin conductance responses (SCRs) were acquired from the participant's middle phalanges of the second and third fingers in the left hand using BIOPAC systems skin conductance module. Shielded Ag–AgCl electrodes were grounded through an RF filter panel and served to acquire data. AcqKnowledge software was used to analyse SCR waveforms. The level of SCR was assessed as the base to peak difference for an increase in the 0.5–4.5 s window following the onset of a CS, the blue or yellow square (see LaBar et al. 1995). A minimum response criterion of 0.02 μS was used with lower responses scored as 0. The responses were square-root transformed prior to statistical analysis to reduce skewness (LaBar et al. 1998). Acquired SCRs through the three blocks of aversive conditioning were then averaged per participant, per type of trial. The trials in which the CS+ was paired with $4.00 were separated into time of CS+ presentation and time of US presentation, so only differential SCR to the CS+ itself was included. Two-tailed paired t-tests were used to compare the activity of CS+ versus CS− trials to demonstrate effective conditioning.

(d) fMRI acquisition and analysis

A 3T Siemens Allegra head-only scanner and a Siemens standard head coil were used for data acquisition at NYU's Center for Brain Imaging. Anatomical images were acquired using a T1-weighted protocol (256×256 matrix, 176 1-mm sagittal slices). Functional images were acquired using a single-shot gradient echo planar imaging sequence (TR=2000 ms, TE=20 ms, FOV=192 cm, flip angle=75°, bandwidth=4340 Hz per pixel and echo spacing=0.29 ms). Thirty-five contiguous oblique-axial slices (3×3×3 mm voxels) parallel to the AC–PC line were obtained. Analysis of imaging data was conducted using Brain Voyager software (Brain Innovation, Maastricht, The Netherlands). The data were initially corrected for motion (using a threshold of 2 mm or less), and slice scan time using sinc interpolation was applied. Further spatial smoothing was performed using a three-dimensional Gaussian filter (4 mm FWHM), along with voxel-wise linear detrending and high-pass filtering of frequencies (three cycles per time course). Structural and functional data of each participant were then transformed to standard Talairach stereotaxic space (Talairach & Tournoux 1988).

A random effects analysis was performed on the functional data using a general linear model (GLM) on 11 participants. There were 12 different regressors: 3 at the level of the CS (CS−, CS+ and CS+-US; the trials paired with US); 2 at US onset (US or NoUS); 1 PE regressor; and 6 motion parameter regressors of no interest in x, y, z dimensions. The main statistical map of interest (correlation with PE) was created using a threshold of p<0.001 along with a cluster threshold of 10 contiguous voxels.

The PE regressor that provided the main analysis of interest was based on traditional TD learning models and is the same as the one used in the aversive conditioning study with primary reinforcers described earlier (Schiller et al. in press). In TD learning, the expectation $\hat{V}$ of the true state value V(t) at time t within a trial is a dot product of the weights w_i and an indicator function x_i(t) that equals 1 if a conditioned stimulus (CS) is present at time t, or 0 if it is absent,

$$\hat{V}(t)=\sum _i{w}_i{x}_i(t).$$ (2.1)

At each time step, learning is achieved by updating the expectation value of each time point t within that trial by continuously comparing the expected value at time t+1 to that at time t, which results in a PE,

$$\delta (t)=r(t)+\gamma \hat{V}(t+1)-\hat{V}(t),$$ (2.2)

where r(t) is the reward harvested at time t. In aversive conditioning, we usually treat aversive stimuli as reward and assign positive value to the aversive reinforcer. Discount factor γ is used to take into account the fact that reward received earlier is more important than the one received later on. Usually, γ is set such that 0<γ<1. In the results reported here, γ=0.99. The weights are then updated from trial to trial using a Bellman rule,

$$w_i\leftarrow w_i+\lambda \sum _i{x}_i(t)\delta (t),$$ (2.3)

where λ is the learning rate and set to be 0.2 in our study. We assigned CSs and outcome as adjacent time points within each trial and set the initial weights for each CS to be 0.4 as used in a variety of aversive conditioning paradigms. With these parameters (λ=0.2, γ=0.99 and w_i=0.4), we calculated PEs using the updating rules (2.1) and (2.2) and generated the actual regressors for the fMRI data analysis.

3. Results

(a) Physiological assessment of aversive conditioning

Analysis of the SCR data assessed the success of aversive conditioning with monetary reinforcers (figure 3). The participants' SCR to CS+ trials (M=0.33, s.d.=0.25) was significantly higher than for CS− trials (M=0.15, s.d.=0.07) over the course of the experiment (t(13)=3.48, p<0.005). Conditioning levels were sustained across the three blocks as no differences were observed in the CR (the difference between CS+ and CS− trials) between blocks 1 and 2 (t(12)=1.53, p=0.15) or blocks 1 and 3 (t(12)=0.97, p=0.35) with one participant removed for showing no responses during block 3. Finally, removal of the three participants due to motion does not affect the main comparison of CS+ and CS− trials (t(10)=5.49, p<0.0005).

Figure 3.

SCRs during the aversive conditioning: SCR data suggest successful aversive conditioning with a secondary reinforcer such as monetary losses. Blue bar, CS+; yellow bar, CS−.

(b) Neuroimaging results

The primary contrast of interest was a correlation with PE as previously described. A statistical parametric map contrasting the PE regressor with fixation provided the main analysis (p<0.001, cluster threshold of 10 contiguous voxels). This contrast led to the identification of five regions (table 1), including the medial prefrontal cortex, midbrain and a region in the anterior striatum in the head of the caudate nucleus (figure 4). The observation of PE signals in the striatum during aversive conditioning with secondary reinforcers is consistent with the previous accounts of striatum involvement in PEs, irrespective of learning context (appetitive or aversive).

Table 1.

PE during aversive conditioning.

region of activation	laterality	Talairach coordinates			no. of voxels
		x	y	z
medial prefrontal cortex (BA 9/24)	right	24	10	35	337
cingulate gyrus (BA 24/32)	left	−13	37	14	869
cingulate gyrus (BA 24/32)	right	16	30	8	280
caudate nucleus	right	13	20	4	329
midbrain	right	6	−24	4	334
medial temporal lobe (BA 41/21)	left	−40	−31	2	524

Figure 4.

Activation of striatum ROI defined by a contrast of PE regressor and fixation. The ROI is located in the right hemisphere, in the anterior portion of the head of the caudate nucleus (x, y, z=13, 20, 4).

4. Discussion

The striatum, although commonly cited for its role in reward processing, also appears to play a role in coding aversive signals as they relate to affective learning, PEs and decision making. In this present study, we used a classical fear conditioning paradigm and demonstrated that BOLD signals in the striatum, particularly the head of the caudate nucleus, are correlated with predictions errors derived from a TD learning model, similar to what has been previously reported in appetitive learning tasks (e.g. O'Doherty et al. 2003). This role for the striatum in coding aversive PEs was observed when conditioned fear was acquired with monetary loss, a secondary reinforcer. These results complement our previously described study that used a similar PE model and paradigm, albeit with a mild shock, a primary reinforcer (Schiller et al. in press). These results and others point to the general role for the striatum in coding PEs across a broad range of learning paradigms and reinforcer types.

Our present results, combined with the results of Schiller et al. (in press), demonstrate aversive PE-related signals with both primary and secondary reinforcers and suggest a common role for the striatum. There were some differences between the two studies, however, with respect to results and design. In the primary reinforcer study, the region of the striatum correlated with PEs was bilateral and located in a more posterior part of the caudate nucleus (x, y, z=9, 5, 8; Schiller et al. in press). In the current study with secondary reinforcers, the correlated striatal region was in more anterior portions of the caudate nucleus, and unilateral (right hemisphere, x, y, z=13, 20, 4). These anatomical distinctions between the studies raise the possibility that different regions of the striatum may code aversive predictions errors for primary and secondary reinforcers, similar to the division within the striatum that has been suggested when comparing appetitive and aversive PEs (Seymour et al. 2007). One potential explanation for the more dorsal striatum ROI identified in the aversive experiments is that the participants may possibly be contemplating ways of avoiding the potentially negative outcome, leading to more dorsal striatum activity previously linked to passive or active avoidance (Allen & Davison 1973; White & Salinas 2003). However, given the nature of neuroimaging data acquisition and analysis techniques, such a conclusion would be premature, as would any conclusion with respect to parcellation of function within subdivisions of the striatum based on the paradigms discussed. Additionally, while the studies were comparable in terms of design, there were distinct differences besides the type of reinforcer that discourages a careful anatomical comparison. Such differences include the timing and amount of trials, the experimental context (e.g. gambling prior to conditioning, reversal learning) and potential individual differences across the participants. Future studies will need to explore within subject designs (e.g. Delgado et al. 2006), with similar paradigms and instructions, and perhaps high-resolution imaging techniques to fully capture any differences in coding aversive PEs-related signals for primary and secondary reinforcers within different subsections of the striatum.

Interestingly, the amygdala, the region that is primarily implicated in the studies of classical fear conditioning, did not reveal BOLD responses correlated with PEs. In our primary reinforcer study previously described (Schiller et al. in press), an examination of the pattern of amygdala activation revealed anticipatory BOLD responses, similar to the pattern observed in the striatum; however, only striatal activation correlated with PEs. Although striatal signals have been shown to be correlated with PEs in a range of neuroimaging studies (see above), very few have reported PE-related signals in the amygdala (Yacubian et al. 2006). A recent electrophysiological study in monkeys found that responses in the amygdala could not differentiate PE-related signals from other signals such as CS value, stimulus valence and US responses (Belova et al. 2007). While the amygdala clearly plays a critical role in aversive learning, the computations underlying the representation of value by amygdala neurons may not be fully captured by traditional TD learning models. Alternatively, modulation of different parameters within a model (e.g. stimulus intensity) or consideration of task context (e.g. avoidance learning) may be revealed to be more sensitive to amygdala activity.

Both striatum and amygdala are intrinsically involved in affective learning, but may differ with respect to their involvement. A vast array of evidences implicates both structures in general appetitive and aversive learning (for a review see O'Doherty (2004) and Phelps & LeDoux (2005)) with variations due to task context and type or intensity of stimuli (Anderson et al. 2003). Thus, it is possible that differences between the striatum and amygdala may be observed during a direct comparison of primary and secondary reinforcers. Further, as previously discussed, the present results suggest that striatum and amygdala differences may arise in the context of processing PEs. Future studies may focus on direct similarities and differences between these two structures in similar paradigms and using within-subjects comparison, varying both the valence (appetitive and aversive), intensity (primary and secondary reinforcer) and type of learning (classical and instrumental) to fully understand how these two structure may interact during affective learning.

Neuroeconomic studies of decision making have emphasized reward learning as critical in the representation of value driving choice behaviour. However, it is readily apparent that punishment and aversive learning are also significant factors in motivating decisions and actions. As the emerging field of Neuroeconomics progresses, understanding the complex relationship between appetitive and aversive reinforcement and the computational processes underlying the interacting and complementary roles of the amygdala and striatum will become increasingly important in the development of comprehensive models of decision making.