The primate amygdala represents the positive and negative value of visual stimuli during learning

Abstract

Visual stimuli can acquire positive or negative value through their association with rewards and punishments, a process called reinforcement learning. Although we now know a great deal about how the brain analyses visual information, we know little about how visual representations become linked with values. To study this process, we turned to the amygdala, a brain structure implicated in reinforcement learning^1–5. We recorded the activity of individual amygdala neurons in monkeys while abstract images acquired either positive or negative value through conditioning. After monkeys had learned the initial associations, we reversed image value assignments. We examined neural responses in relation to these reversals in order to estimate the relative contribution to neural activity of the sensory properties of images and their conditioned values. Here we show that changes in the values of images modulate neural activity, and that this modulation occurs rapidly enough to account for, and correlates with, monkeys’ learning. Furthermore, distinct populations of neurons encode the positive and negative values of visual stimuli. Behavioural and physiological responses to visual stimuli may therefore be based in part on the plastic representation of value provided by the amygdala.

The complex anatomical connections of the amygdala, a collection of nuclei located deep in the medial temporal lobe, make it a prime candidate for providing a representation of the value of visual stimuli. The amygdala receives inputs from the visual system and from other sensory systems that represent reinforcing stimuli^6–8. In addition, the amygdala is likely to receive error signals that represent stimuli in relation to expectations and that may be essential in creating an updated representation of value. The source of error signals for aversive learning has not been identified; however, midbrain dopamine neurons might supply such error signals for appetitive learning⁹. These signals could influence amygdala neural responses either directly⁷ or indirectly through other brain structures such as the prefrontal cortex^6,10. The convergence of information from each of these input pathways, perhaps combined with processing that occurs through intrinsic connections within the amygdala, may form a representation of visual stimulus value.

Unlike the anatomy of the amygdala, the physiological properties of amygdala neurons—especially with respect to learning the value of visual stimuli—remain poorly understood. We therefore recorded the activity of single amygdala neurons while monkeys learned the positive or negative value of new, abstract images during a trace-conditioning procedure¹¹ (Fig. 1a). In this task, a brief time interval (the trace interval) separated the presentation of a conditioned stimulus (CS) from an unconditioned stimulus (US). In each experiment, we assigned every CS a value: positive, negative or non-reinforced. After the image viewing and trace intervals, we delivered a liquid reward (after positive images), an aversive air-puff directed at the face (after negative images) or nothing (after non-reinforced images). The monkeys learned to associate each image with its value: during the trace interval they licked at a spout in anticipation of a liquid reward, or closed their eyes—a defensive behaviour—in anticipation of an air-puff. (We refer to the monkeys’ eye closures as blinks, but the eyes typically stayed closed for longer than a blink, suggesting that this behaviour was purposeful.)

Figure 1. Task and brain MRI.

a, Sequence of events for the three trial types. Top and bottom squares, image values reverse. Middle square, image always non-reinforced. b, Coronal MRI acquired with a two-dimensional (2D) spoiled gradient recalled acquisition (SPGR) sequence in monkey V, showing the artefact from a tungsten microelectrode dorsal to the amygdala (outlined in white). c–f, Coronal MRI with 2D inversion recovery (IR) sequence (c, arrows point to the electrode). Magnified images show the probable border of the lateral nucleus (arrow in d) and recording site locations (slice in f is immediately posterior to e). We collapsed recording sites spanning 2 mm in the anterior–posterior dimension onto each image slice, in many cases resulting in the superposition of multiple cells with different properties (see key above f for symbols denoting properties; ‘ + ’ denotes positive value-coding, ‘ − ’ denotes negative value-coding, and ‘no’ symbol indicates no value-coding). Recording sites from monkey P occurred in an overlapping region of the amygdala.

After the monkeys had learned the associations, without warning we reversed the values of the positive and negative images, and the monkeys learned the new values. Unlike a task in which subjects learn to avoid aversive USs, in our task, CSs predicted USs with 100% certainty. We tested every neuron during both appetitive and aversive conditioning for the same CS, allowing us to determine whether distinct amygdala neurons respond preferentially to positive or negative value for each CS. We report that amygdala neurons encode—in a flexible manner and in separate populations of neurons—the positive or negative value of conditioned images. Furthermore, these signals predict when monkeys learn, and develop rapidly enough to account for learning.

Anticipatory licking and blinking behaviour provided a measure of monkeys’ learning about the positive and negative association of images. In the experiment shown in Fig. 2a, b, licking response rates were greater when an image was positive than when the same image was negative. Blinking responses showed the opposite trend. We used a change-point test to identify when the response rate changed significantly in relation to a value reversal (P < 0.05), represented graphically by a change in slope of the cumulative record of responses¹². For image 1, the licking rate decreased significantly at trial 52 and blinking rate increased at trial 48 (22 and 18 trials after reversal, respectively). Because the monkey still licked sporadically when image 2 was negative, the change-point algorithm identified trial 29 as when licking began to increase, slightly before the reversal. Blinking decreased significantly starting at trial 35.

Figure 2. Behaviour and neural activity from a single amygdala neuron during learning.

a, b, Cumulative (curves) and trial-by-trial (tick marks) measures of licking (red) and blinking (blue), plotted as a function of trial number for images 1 and 2. Black dots indicate change points. Vertical green lines indicate value reversals. c–f, Rasters and peri-stimulus time histograms (PSTHs), truncated at US delivery, for the amygdala cell recorded during the same experiment. Each row of dots represents the timing of action potentials during one trial. PSTHs sum activity across trials and were smoothed by taking a 10-ms moving average of activity. Blue ticks indicate fixation point onset. Red ticks indicate visual stimulus onset/offset. g, h, Spike count and cumulative spike count during the trace interval, plotted as a function of trial number for images 1 and 2. Red dots indicate change points.

The primary objectives of our study were (1) to determine whether positive and negative conditioned visual stimuli engage the same or different populations of amygdala neurons, (2) to compare quantitatively the time course of how neural activity and behaviour change upon image value changes, and (3) to characterize the relative contributions of conditioned value and image identity to neuronal responses. In each of two monkeys (monkey V and monkey P), we used magnetic resonance imaging (MRI) to localize our recording electrode to the amygdala (Fig. 1b–d). We obtained complete data sets during monkeys’ performance of initial and reversal learning for 196 neurons. To determine the extent to which amygdala neural activity reflected the conditioned values of images, or the images’ sensory properties, we examined neural activity in relation to image value reversals. For the amygdala neuron recorded during the behaviour depicted in Fig. 2a, b, neural activity during the trace interval was higher when images had a positive value than when the same images had a negative value (Fig. 2c–f), typical of a value-coding neuron. Other cells showed value-coding during the visual stimulus interval (see Supplementary Note 1 and Supplementary Fig. 1).

To characterize how quickly neural activity changed in relation to reversal of image value, we applied the change-point test¹² to trace interval neural responses (Fig. 2g, h). For image 1, neural activity began to decrease at trial 38 (8 trials after reversal), before either licking or blinking behaviour changed. For image 2, neural activity increased at trial 33, preceding the change in blinking behaviour. Overall, 100/196 (51%) of neurons changed activity in relation to value during the visual stimulus and/or trace intervals. For the cell shown in Fig. 2, a two-way analysis of variance (ANOVA) verified the influence of image value on neural responses (65% of variance accounted for by image value, P < 0.05), as well as a much smaller effect of image identity (8% of variance accounted for by image identity, P < 0.05). Across our population of neurons, image value often accounted for a statistically significant percentage of the variance (Fig. 3a, b; P < 0.05, two-way ANOVA). In addition, image identity had a significant effect in many cells, often overlapping with the representation of value at the single-cell level. The inset histograms show the difference in the strength of identity and value coding for all neurons with a significant effect of image identity, value or both factors. The two distributions are significantly different, with the trace interval showing significantly more value coding compared to image identity coding than the visual stimulus interval (P < 10⁻⁵, Wilcoxon test; visual stimulus median difference − 1.2; trace median difference 11.8). Thus, the representation of image identity weakens substantially after an image disappears, and neural activity then predominantly reflects the learned value of images.

Figure 3. Amygdala neurons encode the positive and negative value of visual stimuli.

a, b, Image identity coding index (II_CI) plotted against image value coding index (IV_CI) for the visual stimulus (VS) and trace intervals. Green, blue and red dots, P < 0.05 for II_CI, IV_CI or both indices, respectively (two-way ANOVA). Yellow dots, not significant. Inset histograms show II_CI subtracted from IV_CI for non-yellow data points. See Supplementary Fig. 9. c, Encoding of positive and negative value shown by ROC analysis. d, Development of value signals as a function of time. Each row in the colour map represents value coding for a neuron during presentation of a single image, with positive and negative cell rows sorted in opposite order according to the first post-visual-stimulus data point significantly different from 0.5 (P < 0.05, permutation test). White curves show mean ROC values across the positive- and negative-coding populations. Time 0 is the start of the bin spanning from 0–99 ms after visual stimulus onset.

To further characterize the representation of value in the amygdala, we determined quantitatively whether neurons responded more strongly when an image had a positive or negative value. We used a receiver operating characteristic (ROC) analysis¹³ to estimate the extent to which activity before and after an identified change point differed (Fig. 3c). Each data point from this analysis reflects the change in activity for a single image in a single interval. By convention, we used the negative value responses as the reference distribution, so ROC values >0.5 indicated cells that responded more when an image was positive than when the same image was negative. All ROC values except two were significantly different from 0.5 (P < 0.05, permutation test). The bi-modally distributed data indicate that some amygdala neurons prefer negative conditioned stimuli, and others prefer positive conditioned stimuli (see Supplementary Note 2).

The activity of neurons encoding positive and negative values was not related to motor responses (licking and blinking; see Supplementary Figs 3, 4). Moreover, the development of value coding was not an artefact of using different modalities for our reinforcing stimuli (although the air-puff and liquid reward both have auditory and somatosensory components). Many neurons encoding positive image value responded to air-puffs, and many neurons whose activity reflected negative image value responded to rewards (Table 1 and Supplementary Fig. 5). Thus, neurons encoding value received information about both kinds of reinforcing stimuli. Finally, neurons representing positive and negative values did not demonstrate clear anatomical clustering (Fig. 1e, f).

Table 1.

Responses to reinforcing stimuli in amygdala neurons that code value

	Excitatory response to air-puff	Inhibitory response to air-puff	Excitatory response to reward	Inhibitory response to reward	No response to reward or air-puff
Cells coding positive CS value (n = 61)	34	12	23	6	9
Cells coding negative CS value (n = 39)	18	8	11	7	7

Excitatory and inhibitory responses to air-puffs and rewards for all cells encoding the positive or negative value of images. The number of cells adds up to more than the number of value-coding neurons because many cells were modulated by both air-puff and reward.

We next characterized how neurons represent value across time during trials. An ROC analysis applied to consecutive overlapping time-windows of 100 ms (advanced in steps of 20 ms), revealed that different neurons encoded image value during different overlapping time segments (Fig. 3d). Some neurons represented value exclusively during the visual stimulus or trace intervals, but other cells sustained a representation across some or all of both intervals. Across the population of neurons, signals representing value developed rapidly after visual stimulus onset and were maintained through the trace interval (Fig. 3d; for positive-coding cells, the first bin significantly greater than 0.5 occurred 80–180 ms after visual stimulus onset, P < 0.05 by t-test; for negative-coding cells, the first significant bin was 120–220 ms after visual stimulus onset). This temporally extended representation of value may be useful for the development of adaptive behaviour—such as licking and blinking—that anticipates reinforcement. Furthermore, the representation of value over time could have an important role in reinforcement learning: computations of error signals may depend on a comparison of neural signals that represent value as a function of time⁹.

If decisions to lick or blink in anticipation of a reward or punishment are based in part on the representation of value we describe in the amygdala, then, on average, neural activity should change at the same time as learning occurs. To compare behavioural learning of the image value reversals to changes in neural activity, we plotted the change-point for neural activity against the trial when behaviour changed (Fig. 4a, b; each data point compares a neural activity change point with either a blinking or licking change point; the distributions of licking and blinking change points were not significantly different from one another; P > 0.25, t-test). Learning was significantly correlated with changes in neural activity during both the visual stimulus and trace intervals (visual stimulus interval: monkey V, P = 0.02, r² = 0.06; monkey P, P < 10⁻⁵, r² = 0.44; trace interval: monkey V, P < 10⁻⁵, r ² = 0.40; monkey P, P < 10⁻⁵, r² = 0.32). The difference between neural and behavioural change points was near zero (Fig. 4 histograms; monkey V: visual stimulus mean difference 1.4 trials, trace mean difference − 0.8 trials; monkey P: visual stimulus mean difference − 0.5 trials, trace mean difference 0.3 trials). For all comparisons, the onsets of changes in behavioural and neural responses were statistically indistinguishable (paired t-test, P > 0.05). In addition, the distributions comparing change points during the visual stimulus and trace intervals were not significantly different (P > 0.1, t-test).

Figure 4. The relationship between changes in neural activity and behavioural responses.

a, b, Onset of changes in neural activity plotted against the onset of changes in licking or blinking for visual stimulus (VS) and trace interval activity. Histograms show the difference between neural and behavioural change points. Blue, data and regression lines for monkey V; red, data and regression lines for monkey P. c, Average normalized neural activity and behavioural responses plotted as a function of trial number relative to the reversal in image value. Shaded regions indicate 95% prediction intervals for best-fit Weibull functions. d, Similar representation as in c, applied to the non-reinforced images. Shaded regions show s.e.m. of data points.

These results indicate that amygdala neurons can rapidly adjust their activity to reflect image value; these changes in activity both predict when monkeys learn and develop, on average, as fast as monkeys learn. The fact that some cells change their activity more rapidly than monkeys learn suggests that monkeys cannot selectively ‘listen’ to these neurons, or they would demonstrate faster learning. Learning may be better accounted for if monkeys must evaluate signals from the population of neurons, which includes neurons that change their activity more slowly.

The data presented so far compare the onset of changes in behavioural and neural responses. We next compared the time course of average neural and behavioural responses across the 100 neurons encoding image value (as identified by the change-point test). We normalized and then averaged neural activity and behaviour from the 20 trials before and after the value reversal of each image. To describe the behavioural and neural changes around reversals, we fitted the behavioural and neural data with sigmoid functions (Fig. 4c, see Supplementary Fig. 6). The 95% prediction intervals for these functions overlapped, indicating that the trajectories of changes in behavioural responses and neural activity were similar under this model. In contrast, the same analysis on the non-reinforced images for the same cells showed no change in activity or behaviour in relation to the reversal of the positive and negative images (Fig. 4d). These data demonstrate that, on average, neurons begin to change their activity within very few trials of a change in image value, and that—across the population of neurons—the rate of changes in activity closely tracks behavioural learning.

The rapid appearance of a representation of visual stimulus value in the amygdala raises the possibility that this representation is formed locally. If a representation of value is first formed elsewhere in the brain, it would have to provide signals that represent value at earlier time points within trials, and activity would have to change even more rapidly across trials compared to what we observe in the amygdala. Moreover, the correlation between changes in amygdala activity and behaviour strongly suggests that the amygdala representation is used by the monkey, regardless of its source. Nonetheless, other brain structures such as the orbitofrontal cortex probably have critical roles in learning on our task. The amygdala seems to update its representation as quickly as the orbitofrontal cortex changes its activity during reversal learning (using a different task, in which monkeys learn to avoid aversive stimuli, and different analytic methods)¹⁴. Other studies that have not used aversive stimuli also suggest that the orbitofrontal cortex provides a representation of reward value^15–17. Understanding the complex interactions between the amygdala and orbitofrontal cortex (and other structures) will require substantial investigative effort in order to tease apart the distinct contributions of each brain area to reinforcement learning. In rodents, some evidence suggests that the physiological properties of amygdala and orbitofrontal neurons are interdependent^18,19.

Human and non-human primate behaviour is strongly influenced by visual processing and, therefore, the assignment of value to visual stimuli is a fundamental process that underlies many of our actions. In lower mammals, previous studies have investigated amygdala neurophysiology in relation to learning; however, these studies have used auditory or olfactory conditioned stimuli, and tasks that either do not include appetitive conditioning or that involve learning to avoid an aversive unconditioned stimulus^1,20. The primate amygdala has extensive interconnections with the visual cortex²¹, and previous lesion studies in monkeys and neuroimaging studies in humans have suggested that the amygdala is involved in updating a representation of the value of visual stimuli^3,22,23 (see Supplementary Note 3). Previous neurophysiological recordings in monkey amygdala have used different methodological procedures than those here, and provide conflicting data regarding whether the amygdala contains an updated representation of visual stimulus value^24–26.

Our data establish that distinct amygdala neurons encode the positive and negative learned value of images. The representation of value is often linked to image selectivity initially, but this representation becomes transformed over time so that value, and not image selectivity, is predominantly encoded after visual stimuli disappear. Signals that reflect value develop rapidly, and evolve as quickly, on average, as monkeys learn. Furthermore, the timing of changes in amygdala activity predicts when monkeys learn. Thus, in principle, decisions to either lick or blink during performance of our task could be based on the plastic representation of visual stimulus value provided by the amygdala.

Reinforcement learning, a process by which sensory stimuli become associated with positive or negative values, is intimately related to physiological and behavioural responses thought to represent emotions, because such responses frequently occur as a reaction to sensory stimuli endowed with value. Amygdala neural signals that reflect CS value may influence a variety of interconnected brain structures—including the ventral striatum, prefrontal cortex, sensory cortices, the hippocampus and rhinal cortices, and subcortical brain areas—involved in aspects of emotional responses and reinforcement learning. Different amygdala nuclei have complex patterns of connectivity with each of these structures^6,7,10,21,27, suggesting that neural signals in the amygdala may be used for very different purposes depending upon their anatomical destination. For example, signals that reflect CS value may influence attention²⁸ or computations of error signals⁹ through projections from the amygdala central nucleus to the ventral tegmental area and substantia nigra²⁹. In addition, value signals could influence behavioural and physiological responses to visual stimuli through other subcortical projections from the central nucleus³⁰, and through projections from the basal and accessory basal nuclei to the prefrontal cortex and ventral striatum^6,7,10. The amygdala neural signals we describe—which reflect the learned value of visual stimuli—are therefore only one component of the intricate neural circuitry that mediates reinforcement learning and our emotional lives. To understand how this circuitry operates, future studies must identify how dynamic interactions between the amygdala and interconnected brain structures function to support a wide range of emotional and cognitive behaviours.

METHODS

Behavioural task

In every experiment, monkeys learned the positive, negative or non-reinforced value of three new images through a trace-conditioning procedure. In individual trials, monkeys centred their gaze at a fixation point (FP) for 1 s and viewed an image for 300 or 350 ms, determined empirically for each monkey to minimize broken fixation behaviour (see Supplementary Fig. 7). After a 1,500-ms trace interval during which monkeys were free to move their eyes, we delivered a US (for positive images, a 0.2–0.9 ml liquid reward; for negative images, a 50–100 ms, 40–60 psi aversive air-puff directed at the face; for non-reinforced images, nothing). We reversed image value assignments without warning at least 8 trials after monkeys learned the initial contingencies, as assessed by viewing licking and blinking data during the experiment. All trials were pseudorandomly interleaved. Images were fractal patterns constructed with a custom-written Matlab (Mathworks) program. Images had no inherent value and were easily distinguishable from each other and from recently shown images on previous days.

Data collection

We recorded neural activity from the right amygdala of two rhesus monkeys (Macaca mulatta) weighing 4–10 kg. All animal procedures conformed to NIH guidelines and were approved by the Institutional Animal Care and Use Committees at New York State Psychiatric Institute and Columbia University. We positioned recording chambers based on anatomical information acquired using MRI. Conventional metal microelectrodes (1–4) (FHC Instruments) were individually controlled and advanced into the brain through dura-puncturing guide tubes using either a motor-controlled hydraulic microdrive (Narishige) or a motorized multi-electrode drive (NAN). Signal amplification, filtering, digitizing of spike waveforms, and spike-sorting using a principal component analysis platform (online, with offline verification) were accomplished using the Plexon system. While we searched for neurons, monkeys performed either no task or a fixation task without images present. We studied all well-isolated neurons (196 neurons in total: 92 from monkey V, 104 from monkey P). We reconstructed recording sites using MRI combined with laboratory notes. Neurons were located in the middle 50% of the anterior–posterior extent of the amygdala, in the lateral 75% of the medial–lateral extent, and in the dorsal 75% of the dorsal–ventral extent. All neurons were probably located in either the basal, accessory basal, lateral or central nucleus, or the intercalated masses.

Licking was measured by the interruption of an infrared beam by the tongue extending to the reward tube, and blinking by the loss in signal from an infrared eye tracker (ASL), which corresponded to blinks as verified by a video camera feed. Licks or blinks occurring during the 500 ms preceding US delivery were scored as responses.

Data analysis

Based on an analysis of visual response latency (see Supplementary Fig. 8), we divided the trial into visual stimulus (90 ms after image appearance until 90 ms after image disappearance) and trace (90 ms after image disappearance until US delivery) intervals. Our data analysis had four principal goals. (1) To find the trial closest to image value reversals at which neural and behavioural (licking, blinking) responses began to change, we used a change-point test¹² (P < 0.05, correcting for multiple comparisons). (2) To characterize the relative contribution of image identity (that is, the sensory properties of images) and value on neural activity, we performed a two-way ANOVA. We defined three indices, corresponding to the percentage of variance accounted for by image identity, image value and their interaction (image identity, image value, and image value–identity interaction coding indices). (3) To determine whether amygdala neurons responded more when an image had a positive or negative value, we compared spike counts from the 20 trials on either side of an identified change point using a receiver operating characteristic (ROC) analysis¹³ on neurons categorized as value-coding (209 intervals from 100 cells coding image value during the visual stimulus, trace or both intervals; 72/100 cells changed activity in opposite directions for both images upon reversal; 28/100 neurons changed responses for one image, but two-way ANOVA showed significant effects of image value). We also characterized the development within trials of signals representing value, computing an ROC value in 100-ms windows moved in 20-ms steps across the trial (performed on 172 image/cell combinations from the 100 value-coding cells). (4) To compare the trajectories of changes in neural and behavioural responses, we normalized and averaged neural activity and behaviour from each of the 20 trials before and after image value reversal across the 100 value-coding neurons. We fitted the behavioural and neural data with a Weibull function:

$$f(x)=u+(1-l-u)exp{(-x/\alpha )}^{\beta }$$ (1)

which modelled normalized response as a function of trial number. We performed a similar normalization and averaging procedure for non-reinforced trials, analysing the 20 trials before and after the trial that the other images reversed. Details of all methods are described further in Supplementary Methods.

Supplementary Material

Fig1-9

Supplementary Information is linked to the online version of the paper at www.nature.com/nature.