Flexible Neural Representations of Value in the Primate Brain

Abstract

The amygdala and orbitofrontal cortex (OFC) are often thought of as components of a neural circuit that assigns affective significance—or value—to sensory stimuli so as to anticipate future events and adjust behavioral and physiological responses. Much recent work has been aimed at understanding the distinct contributions of the amygdala and OFC to these processes, but a detailed understanding of the physiological mechanisms underlying learning about value remains lacking. To gain insight into these processes, we have focused initially on characterizing the neural signals of the primate amygdala, and more recently of the primate OFC, during appetitive and aversive reinforcement learning procedures. We have employed a classical conditioning procedure whereby monkeys form associations between visual stimuli and rewards or aversive stimuli. After learning these initial associations, we reverse the stimulus-reinforcement contingencies, and monkeys learn these new associations. We have discovered that separate populations of neurons in the amygdala represent the positive and negative value of conditioned visual stimuli. This representation of value updates rapidly upon image value reversal, as fast as monkeys learn, often within a single trial. We suggest that representations of value in the amygdala may change through multiple interrelated mechanisms: some that arise from fairly simple Hebbian processes, and others that may involve gated inputs from other brain areas, such as the OFC.

EMOTION, VALUATION, AND REINFORCEMENT LEARNING

In humans, the regulation of emotion is extremely flexible, adapting to different sensory cues, social situations, and cognitive operations, such as the application of rules. How does the brain mediate these different aspects of emotional processing? Most prior efforts to understand emotion at the neural level have employed rodents, often using fear-conditioning and related behavioral paradigms. In human and non-human primates, however, a more flexible control of emotion is thought to be conferred by interactions between the amygdala and prefrontal cortex (PFC).¹ Indeed, in primates, as compared to non-primates, there is an extensive elaboration of the PFC and its connections with the amygdala.² In addition, because the visual system is a dominant sensory modality in primates, there are dense connections among the amygdala, PFC, and visual system. Thus, although many aspects of the function and organization of the amygdala and interconnected structures are conserved across species, amygdala function in primates likely expands upon and differs from processing in rodents in significant ways. For these reasons, it is important to elucidate the complex neural circuitry that regulates emotion in the rhesus monkey. Their rich behavioral and cognitive repertoire makes rhesus monkeys ideal for helping to fill a critical gap between studies in rodents and humans.

One way to approach these questions is to exploit different conditioning procedures developed by experimental psychologists to investigate the neural basis of appetitive and aversive reinforcement learning. During reinforcement learning, subjects learn that particular sensory stimuli are associated with rewards and punishments. Emotional responses, such as excitement or fear, commonly occur upon exposure to sensory stimuli that have been endowed with affective value through reinforcement learning.

Associative learning, such as that which links a sensory stimulus with punishment or reward, is frequently assumed to arise gradually via changing synaptic weights; computational models of reinforcement learning have often worked from this assumption.³ The seminal Rescorla–Wagner model posits that the value representation should be updated on a trial-by-trial basis by “error signals”—that is, signals reflecting the difference between expected and received reinforcement—and recent theories (e.g., temporal difference [TD] models), have extended this model so that reinforcement learning may be described quantitatively in real time.^4-6 TD models require a neural representation of value as a function of time, with error signals being computed continuously by taking the difference in the value of situations or “states” at successive time steps. However, more complex forms of conditioning that often occur in nature—such as conditioning in which the value of stimuli changes depending upon context—require representations of value that can be flexibly activated depending on contextual cues or other information. Consider the game of black-jack, whereby being dealt the same card, such as a king, can be rewarding (if it makes a total of 21) or punishing (if it makes a total greater than 21) depending upon the cards dealt earlier in the hand, which define a context that dictates whether receiving the king would be good or bad. TD and other reinforcement learning models, in their simplest form and on their own, may not be able to explain this kind of flexible control of neural activity reflecting value. Instead, such models would need to be extended to incorporate information, such as rules, so that a potentially infinite number of contexts can be interpreted appropriately; furthermore, describing these processes may require mechanisms that operate on a short timescale to gate, or regulate, neural representations of value. Finally, beyond model development, we still need to understand how the brain actually represents value in a flexible manner.

As a first step to understand how value may be represented flexibly, and building upon the theoretical framework of reinforcement learning,^4,5,7 we have targeted two brain areas, the amygdala and orbitofrontal cortex (OFC). We focused on these brain areas because of their known anatomic connectivity and because of a long history of research linking them to reinforcement learning and emotional processes.^8-10 The amygdala is a structurally and functionally heterogeneous collection of nuclei lying in the anterior medial portion of each temporal lobe.^11,12 Sensory information is provided to the amygdala from advanced levels of sensory cortices, the olfactory system, and polysensory brain areas such as perirhinal cortex.¹¹ This information enters the amygdala primarily, but not exclusively, in the lateral nucleus, and then flows—either directly or through multiple synapses—to the basal, accessory basal, central, and other more medial nuclei. Output from the amygdala is directed to a wide range of target structures, including PFC, sensory cortices, the hippocampus, perirhinal cortex, entorhinal cortex, the striatum, and the basal forebrain; and also to subcortical structures responsible for physiological responses related to emotion, such as autonomic responses, hormonal responses, and startle. In general, subcortical projections originate from the central nucleus, and projections to cortex and the striatum originate from the basal, accessory basal and in some cases the lateral nuclei.

Modulation of intrinsic processing in the amygdala can occur via multiple pathways. In particular, OFC projects to numerous amygdala nuclei, including the basal, accessory basal, intercalated masses, and lateral nuclei.¹³ Other inputs, such as dopaminergic input from the ventral tegmental area and the substantia nigra and serotonergic input from the raphe nuclei, also probably modulate intrinsic amygdala processing, though this has primarily been studied in the rodent.¹¹ Each of these processing streams, as well as others not mentioned, may modulate amygdala processing substantially, potentially helping induce plastic changes important to learning and memory formation, as well as facilitating the expression of physiological and behavioral responses. Thus the amygdala can receive information from all sensory modalities about conditioned and unconditioned stimuli (CSs and USs), and also from structures that might transmit instructive or supervisory signals to the amygdala, such as midbrain dopamine neurons or PFC, especially the OFC.^13,14

THE ROLE OF OFC AND THE AMYGDALA

Prior neurophysiological studies in rodent amygdala have suggested that the amygdala supplies neural signals involved in associative learning.^15-25 Most of these studies have been conducted in the context of fear conditioning or related tasks, and they have revealed that amygdala neural activity changes during fear conditioning. Compared with investigations in rodents, only a few studies have investigated amygdala neurophysiology in primates,^26-32 and results were often conflicting. Prior to our recent paper,³³ there had been no systematic study of the neurophysiological properties of primate amygdala neurons during classical conditioning with a well-controlled experimental design.

The OFC, which comprises much of the ventral surface of the frontal lobe, is an area that also has frequently been implicated in the control of emotional behavior. It receives extensive innervation from the amygdala, hippocampus, striatum, and hypothalamus, as well as from many other cortical areas.^34,35 The amygdala sends projections throughout OFC, but most densely to the caudal areas (e.g., area 13).³⁶ Similarly, OFC (mainly area 13, but also areas 12o, 14 and 11) sends projections to several nuclei of the amygdala, including the basal and lateral nuclei, which partially overlaps with input from temporal cortices.^13,14,37 OFC may therefore modulate the processing of sensory information in the amygdala.

The OFC, together with other parts of the PFC, the striatum and amygdala, is thought to assign values to stimuli, which in turn can contribute to emotional responses and decision making.^38,39 Human patients with lesions of the OFC have abnormal emotional and social behavior (including disinhibition), are impaired on decision-making tasks, and have an inability to generate normal physiological reactions to negatively CSs.^38,40,41 Lesion studies in non-human primates have been consistent with these findings; for example, monkeys with OFC lesions, or even an effective “disconnection” of OFC and amygdala, are impaired on a reinforcer-devaluation paradigm.^42,43 Functional imaging studies have revealed OFC involvement in anticipating and acting upon positive and negative outcomes.^44-46

Studies of OFC neurophysiology in rats and monkeys have suggested a role for the OFC in associative learning.^16,47-51 Nearly all of these studies have used instrumental tasks, that is, tasks in which the subject must perform (or actively refrain from performing) an action in order to obtain reward or avoid punishment. In these types of tasks, aversive stimuli do not occur after learning occurs; therefore, one cannot conclude that neural activity recorded during task performance represents the anticipation of aversive stimuli. Conclusions about the encoding of negative value are best supported by tasks in which aversive stimuli always occur, even after learning. However, the responses of OFC neurons have not been examined using classical aversive conditioning, in which the CS is consistently predictive of punishment.

Recent primate studies have largely focused on operant forms of appetitive conditioning. Many OFC neurons develop responses to visual cues that predict reward, as well as anticipatory responses to uncued predictable rewards, which are often modulated by the type and amount of reward anticipated.^52-56 These signals have often been interpreted as reflecting expectation of the value of the anticipated reward, although two new studies have called into question whether all aspects of value are integrated in the OFC.^57,58 Furthermore, one recent study suggests that OFC in rodents is involved in updating value representations during procedures containing Pavlovian features, but not during procedures containing only instrumental procedures.⁵⁹ Interestingly, several studies have suggested that reward signals in OFC are context dependent—that is, some OFC neurons respond differentially to a stimulus predicting a particular reward depending upon the available alternatives.^52,55,56 Overall, OFC carries information about the identity and value of rewards that are available, rewards that are expected, and rewards that are actually received. Considered together with the OFC’s anatomic connectivity, this makes OFC a prime candidate for the flexible modulation of value representations for stimuli associated with reward or punishment.

In the studies described here, we have employed both appetitive and aversive classical conditioning procedures to ask two fundamental questions: (1) What information is represented by neurons in the amygdala and OFC during reinforcement learning? and (2) How are these response properties related to behavioral learning?

HOW IS VALUE REPRESENTED IN THE AMYGDALA?

Our initial studies have been aimed at describing neural signals in the amygdala during the learning and reversal of affective associations. Monkeys performed a trace-conditioning task in which novel, abstract visual stimuli were followed by USs: either a liquid reward, nothing, or an aversive air-puff directed at the face (Fig. 1A). Trace conditioning is a version of classical conditioning in which a brief temporal gap is inserted between CS offset and US onset.^60,61 It is possible that differential responses to CSs associated with rewards and punishments could be attributed to the sensory properties of the CSs themselves, rather than to their reinforcement contingencies; therefore, after initial learning occurred, without warning, and at a variable trial number across experiments, we reversed the contingencies of the images initially associated with rewards and air-puffs, and the monkeys learned the new associations. The monkeys demonstrated their learning by licking a spout in anticipation of a liquid reward, or closing their eyes—a defensive behavior—in anticipation of an air-puff. Note that in our task, CSs predicted USs with 100% certainty, so aversive stimuli could not be avoided.

FIGURE 1.

The trace-conditioning task and localization of recording sites in the amygdala. (A) Sequence of events in the three trial types during trace-conditioning. Top and bottom squares: images that reverse values from positive to negative or vice-versa. Middle square: image does not reverse and was always non-reinforced. (B) Coronal MRI acquired with a two-dimensional (2D) spoiled gradient recalled acquisition (SPGR) sequence in monkey V. The susceptibility artifact from a tungsten microelectrode dorsal to the amygdala (circled) is visible. (C–F) Coronal MRI with 2D inversion recovery (IR) sequence (C, arrows point to the electrode artifact, which is less evident). Magnified images show the recording site locations (slice in F is immediately posterior to E). The arrow in D corresponds to a possible border of the lateral nucleus, which contains a fiber tract. Recording sites spanning 2 mm in the anterior–posterior dimension were collapsed onto each image slice. In many cases, this resulted in the superposition of multiple cells with different properties (the key above F gives the properties denoted by symbols: “+” denotes positive value-coding, “−” denotes negative value-coding, and “no” symbol indicates no value-coding). Recording sites from monkey P occurred in an overlapping region of the amygdala. (From Paton et al.³³ Reproduced by permission.) (In color in Annals online.)

While monkeys performed the trace-conditioning task, we recorded the activity of individual amygdala neurons. We hypothesized that neurons encoding value would change their response profile to the same CSs once the US associated with them switched. Furthermore, we hypothesized that separate populations of neurons would preferentially respond to positive and negative values, respectively, and that these neurons would rapidly update their responses to CSs during learning. We defined “value” operationally, with positive and negative “CS value-coding neurons” referring to neurons whose activity was higher in response to a CS paired with rewards or a CS paired with aversive air-puffs, respectively.

We found that some amygdala neurons encoded positive value, whereas other amygdala neurons encoded negative value. Figure 2 shows an experiment in which we recorded a neuron encoding positive value. Anticipatory licking and blinking behavior (Fig. 2A,B) demonstrated that the monkey learned about the value of the images. We scored every trial according to whether the monkey licked or blinked during the last 500 ms of the trace interval. For both images, licking response rates were greater, and blinking response rates lower, when an image was positive than when the same image was negative. In this experiment, activity in the neuron under study was higher during the trace interval when images had a positive value compared to when the same images had a negative value (Fig. 2C–F), typical of a positive value-coding neuron. Figure 3 shows the results from another experiment, in which we recorded the activity of a neuron encoding negative value, predominantly during the visual stimulus interval. Neurons encoding positive and negative value were dispersed throughout our recording sites in the amygdala, which largely spanned the lateral, basal, accessory basal, and central nuclei, as estimated by reconstructing recording sites with MRI (Fig. 1B–F). These recording sites overlapped those used in prior monkey neurophysiology studies of the amygdala.^26-32

FIGURE 2.

Neural activity from a single amygdalar neuron that encoded positive value during learning, in relation to behavioral learning. (A,B) Behavioral performance. Cumulative (curves) and trial-by-trial (tick marks) licking (red) and blinking (blue) responses, plotted as a function of trial number for images 1 and 2; black dots represent change points. Value reversals occurred at the vertical green lines. (C–F) Rasters and peri-stimulus time histograms (PSTHs) for the amygdala cell recorded during the same experiment. The plots are truncated at US delivery. Each dot represents one action potential, and each row of dots represents the timing of action potentials during one trial. PSTHs sum and average activity across trials, smoothing with a 10-ms moving average of activity. Blue ticks indicate fixation point onset; red ticks indicate visual stimulus onset/offset. (G,H) Spike count and cumulative spike count during the trace interval for the cell depicted in C–F, plotted as a function of trial number separately for images 1 and 2. Red dots show change points that demarcate the onset of a significant change in neural firing rate. (From Paton et al.³³ Reproduced by permission.) (In color in Annals online.)

FIGURE 3.

Neural activity from an amygdala neuron encoding negative value in relation to behavioral learning. (A,B) Behavioral performance. (C–F) Rasters and peri-stimulus time histograms (PSTHs) for the neuron recorded during the same experiment as the behavior shown in A,B. (Gs,H) Change-point analysis of neural responses during the visual stimulus epoch. All labeling conventions in this figure follow those shown in Figure 2. (In color in Annals online.)

The activity of neurons encoding positive and negative values was not related to motor responses. When neural activity was aligned on licking and blinking onset, no amygdala neurons had neural responses that were related to either motor action.³³ Thus value-related response properties are not well described as representing the link between a CS and a particular response. Moreover, many neurons responded to both rewards and air-puff, suggesting that these cells do not simply represent a link between CS and the sensory modality of the US it is associated with. Finally, it is difficult to account for the value-related signals as simply being related to autonomic reactivity. In general, arousing stimuli of both positive and negative valences can trigger autonomic responses.⁶²

To understand how neural activity in the amygdala might relate to activity in other brain areas, such as the OFC, it is critical to characterize the response dynamics of amygdala neurons during learning so that these dynamics can be compared across areas. In particular, it is important to know how rapidly amygdala neurons learn in relation to changes in reinforcement contingencies, and how rapidly amygdala neurons can encode value after the onset of a CS. Therefore, to examine how individual neurons changed their response level when a CS reversed its value, we applied a change-point test⁶³ to neural responses that reflected value during either the trace (Fig. 2) or visual stimulus (Fig. 3) time epochs. The change-point test identifies the onset of a significant change in response rate in relation to the reversal of image value (P<0.05), represented graphically by a change in slope of the cumulative record of responses.⁶³ Inspection of the data for the neuron depicted in Figure 2 reveals that neural activity changed over the course of a number of trials after the change point for Image 1, but activity changed in a single trial for Image 2. We used the same change-point test to detect the onset of changes in behavioral responses in this experiment (Fig. 2A,B). For the experiments depicted in Figures 2 and 3, changes in neural activity and in behavioral responses indicative of learning occurred at about the same time.

Across experiments, behavioral learning of image value reversals was correlated with changes in neural activity. Figure 4A,B shows the change points of neural activity plotted against the corresponding change points for the behavioral data. Each data point compares a neural activity change point with either a blinking or licking change point. The distributions of licking and blinking change points were not significantly different from one another, so they were combined here (P>0.25, t-test). For both monkeys tested, and in both the visual stimulus and trace intervals, behavioral learning was significantly correlated with changes in neural activity (visual stimulus interval: monkey V, P=0.02, r=0.24; monkey P, P<10⁻⁵, r=0.66; trace interval: monkey V, P<10⁻⁵, r =0.63; monkey P, P<10⁻⁵, r=0.57). Moreover, there was not a significant difference between neural and behavioral change points in all cases (paired t-test, P>0.05). Finally, change points during the visual stimulus and trace intervals were not significantly different from each other (P>0.1, t-test).

FIGURE 4.

Neural activity changes as fast as behavioral learning. (A,B) Change points from neural data in the visual stimulus and trace intervals plotted against the change points found for licking or blinking responses reflecting learning. Histograms show that on average there is no difference between neural and behavioral change points. Data and regression lines for monkey V are shown in blue; data and regression lines for monkey P are shown in red. (C) Average normalized neural activity and behavioral responses plotted as a function of trial number relative to the reversal in image value. Shaded regions indicate 95% prediction intervals for best-fit Weibull functions. Behavioral and neural learning curves overlap. (D) Neural responses do not change to the non-reinforced image, as shown by applying the same analysis done for C to the trials containing non-reinforced images. Shaded regions show SEM of data points. (From Paton et al.³³ Reproduced by permission.) (In color in Annals online.)

The tight correlation between the onset of changes in behavior and neural activity suggested that the time course of behavioral and neural learning was similar. We confirmed this by comparing, across neurons encoding image value, the time course of average neural responses with the time course of average behavioral responses. Figure 4C shows the normalized and then averaged neural activity and behavior from the 20 trials before and after the value reversal of each image. The data were fit with sigmoidal functions to construct “neural” and “behavioral” learning curves. The time courses of these curves were quite similar and statistically indistinguishable. Moreover, the changing activity and behavior was specific for the images that changed image value, as demonstrated by the same analysis applied to the same cells from the trials with non-reinforced images (Fig. 4D). Thus, the dynamics of behavioral learning could be accounted for if monkeys based their decisions to lick or blink on the evolving representation of value in the amygdala.

It is also worth emphasizing that both neural activity and behavior changed very rapidly after image value reversal, becoming asymptotic within 4–10 trials, on average. These data imply that amygdala neurons update their response profile on a short timescale, since changes in their activity occur within very few trials of a change in image value. Amygdala neurons may update their representation of value rapidly by virtue of receiving information about reinforcement that does not match expectations; indeed, amygdala responses to reinforcement are often stronger when rewards or aversive stimuli occur unexpectedly, such as immediately after a reversal.⁶⁴ Neural signals representing reinforcement in relation to expectation could come to the amygdala from a variety of subcortical and cortical pathways. In particular, OFC is a candidate for providing rapid, flexible regulation of amygdala neural activity, consistent with a proposed role for OFC in helping account for amygdala response flexibility during reversal learning.⁶⁵

In addition to rapidly updating its representation during learning of reversed CS-US contingencies, amygdala neurons provide a temporally extended representation of value that appears shortly after visual CS onset and spans until the time of reinforcement. To quantify this, we used a receiver operating characteristic (ROC) analysis⁶⁶ to estimate the extent to which activity was different before and after an identified change point (Fig. 5). By convention, ROC values > 0.5 indicated activity that was greater when an image was associated with reward, and values < 0.5 indicated activity that was greater when an image was associated with air-puff. Using this approach, we characterized how neurons represent value across time during a trial by repeating the ROC analysis in consecutive overlapping time-windows of 100 ms (advanced in steps of 20 ms). Figure 5 shows how each value-coding cell represented the value of images as a function of time within trials (each row corresponds to how a single cell represented the value of a single image, as quantified by the ROC analysis). On average, for positive-coding cells, the first bin significantly greater than 0.5 occurred 80–180 ms after visual stimulus onset; all subsequent bins were also significantly greater than 0.5 (downward white arrow, P<0.05, t-test). The first significant bin for neurons encoding negative value was 120–220 ms after visual stimulus onset (upward white arrow). This representation of value over time could correspond to the sort of representation posited by models of reinforcement learning to be required for computations of prediction error signals.⁶ However, the temporal dynamics of amygdala neurons have not been compared quantitatively during learning with neurons encoding prediction errors, such as dopamine neurons.⁶

FIGURE 5.

The temporally extended representation of value in the amygdala. Value signals in the amygdala plotted as a function of time, with positive and negative value-coding neurons represented by different colors. Each row in the color map shows how value was represented by a neuron during the presentation of a single image. Positive and negative cell rows are sorted in opposite order according to the latency of the first post-visual-stimulus data point significantly different from 0.5 (P < 0.05, permutation test). The white curves depict the mean ROC values across the populations of neurons encoding positive and negative value. Time 0 corresponds to the start of the bin spanning from 0–99 ms after visual stimulus onset. White arrows indicate the first bins significantly different from 0.5 for the mean ROC values (P < 0.05, t-test). (Adapted from Fig. 3D in Paton et al.³³) (In color in Annals online.)

HOW IS VALUE REPRESENTED IN PRIMATE OFC?

Given the anatomic interconnections between the amygdala and OFC, a significant challenge for neuroscientists is to understand how neural activity in the two brain areas is interrelated during appetitive and aversive reinforcement conditioning. We have therefore been interested in comparing neural signals in the OFC and amygdala during learning induced by classical conditioning. How and when might signals representing value develop during learning in each area, and what is the timing of these signals relative to each other? OFC is frequently thought of as being critical for reversal learning, but are the response dynamics of OFC neurons appropriate for regulating amygdala responses during our task?

To address these questions, we have been recording simultaneously from individual neurons in the OFC (primarily area 13) and amygdala while monkeys performed a trace-conditioning task similar to the one described above. We find that the responses of neurons in the OFC, like those in the amygdala, can be modulated by image value during one or more periods of the task.⁶⁷ Value-related signals in the amygdala and OFC develop with a range of overlapping latencies after CS onset. Furthermore, both OFC and amygdala neurons can rapidly change their activity during reversal learning, with the time course appearing to be similar in a small data set.⁶⁷ Both OFC and the amygdala therefore contain signals that could underlie appetitive and aversive learning during classical conditioning. On the basis of these preliminary data, amygdala and OFC may be seen to represent closely connected and interrelated components of a neural circuit that assigns affective significance to sensory stimuli, and the two areas may largely act in unison during learning.

SUMMARY AND REFLECTIONS ON THE REGULATION OF VALUE REPRESENTATIONS IN THE BRAIN

Both the amygdala and OFC have long been hypothesized to play a role in aspects of emotional learning and behavior. We are investigating the neurophysiology of the amygdala and OFC during learning induced by appetitive and aversive classical conditioning, and during the reversal of learned reinforcement contingencies. We have discovered that the amygdala contains different populations of neurons, some that respond more strongly to a CS associated with a reward, and others that respond more strongly to a CS associated with an aversive stimulus. This representation of CS “value” is rapidly updated during reversal of contingencies. In some cells, and for some CSs, it occurs within a single trial, and on average it occurs within 4–10 trials, at the same time as changes in behavior indicative of learning. In addition, the representation of value appears rapidly after CS onset. Similar signals appear to be present in OFC. We are in the process of characterizing the rates of learning in OFC neurons compared to amygdala neurons, and the relative latency and duration of value-related signaling in the two brain areas. If one brain area is driving the other, then learning rates should be faster, and latencies shorter, in that brain area. In general we are interested in determining the different roles of OFC and amygdala during learning induced by classical conditioning. Does the amygdala or OFC drive learning in the other brain area, or do the two brain areas function in a parallel fashion? Do distinct response properties in each area depend upon input from the other brain area? What mechanisms are responsible for the flexible representation of value in these brain areas?

The data presented here and in our recent paper,³³ characterize the neurophysiological properties of primate amygdala neurons during appetitive and aversive classical conditioning. Prior work provided conflicting data about whether primate amygdala neurons rapidly changed their response properties upon reversal of reinforcement contingencies using instrumental tasks,^28,29,68 but analogous instrumental tasks in rodents had shown that amygdala neurons were sensitive to reversals in contingencies.^16,48 Our use of classical conditioning procedures (rather than avoidance tasks, where the aversive stimuli do not occur after learning), our use of novel CSs in every experiment, and our tighter requirements for visual fixation all may have contributed to our identification of a flexible representation of value in primate amygdala.

Several lines of experiments have suggested that the OFC is itself a critical structure for reversal learning. In monkeys, lesions of OFC disrupt performance on an operant task involving stimulus-reward reversal learning.⁴² Similar findings have been reported in human clinical populations.⁶⁹ Moreover, selective lesions in primate amygdala, in contrast to OFC, do not appear to impair performance on a similar operant task.⁷⁰ Overall, these results mirror the effects of OFC lesions reported in rodents on a different operant task in which animals learned to use cues to acquire rewards and to avoid aversive stimuli.⁷¹ Interestingly, rat OFC lesions decreased the proportion of amygdala neurons that reversed cue-related responses during this task. The extent of homology between the parts of rat OFC that were lesioned in these studies, and those which we are studying physiologically in monkeys, needs to be determined. Nonetheless, these studies raise the possibility that the flexible representation of value in the amygdala may depend, directly or indirectly, on OFC input. If this is the case, however, the regulation must be sufficiently rapid to account for the largely overlapping temporal dynamics of amygdala and OFC neural activity during learning. Furthermore, it is worth noting that the tasks establishing that OFC is required for reversal learning have all been operant tasks, and it remains unclear how amygdala and OFC lesions would affect performance on a classical conditioning task that includes a reversal of contingencies, such as the one we have employed.

Our experiments have revealed that the primate amygdala provides a flexible representation of the positive and negative value of visual CSs. The rate of learning exhibited by cells, however, was variable. Even the same cell could learn at different rates for different CSs (see, for example, the cell in Fig. 2). This raises the possibility that combinations of different types of mechanisms may be employed in regulating the representation of value in the amygdala. At one extreme, synaptic weights may gradually change during learning through a Hebbian mechanism; this is possible because the amygdala is anatomically well-situated to receive the required coincident input from sensory stimuli and reinforcers. At the other extreme, gated inputs might dictate the response level of amygdala neurons, perhaps explaining how single-trial reversal learning could occur. For a model of this type of flexible learning, we might look to supervised response modulation.^72,73 In this scheme, a neural network is set up to perform a certain function (e.g., compute expected reinforcement); based on the result of the network’s performance, a supervising neural circuit will make direct adjustments to the response properties of the neurons in the network. By interacting with Hebbian plasticity in the synapses of the main network, this kind of supervising circuit can result in efficient learning, even when multiple outputs are required. Thus, gradually changing synaptic weights are not the only physiologically plausible mechanisms for modulating the output of amygdala neurons. OFC could constitute a part of the supervising circuit that may help control representations of value in the amygdala.