Linking dynamic patterns of neural activity in orbitofrontal cortex with decision making

Abstract

Humans and animals demonstrate extraordinary flexibility in choice behavior, particularly when deciding based on subjective preferences. We evaluate options on different scales, deliberate, and often change our minds. Little is known about the neural mechanisms that underlie these dynamic aspects of decision-making, although neural activity in orbitofrontal cortex (OFC) likely plays a central role. Recent evidence from studies in macaques shows that attention modulates value responses in OFC, and that ensembles of OFC neurons dynamically signal different options during choices. When contexts change, these ensembles flexibly remap to encode the new task. Determining how these dynamic patterns emerge and relate to choices will inform models of decision-making and OFC function.

INTRODUCTION

BRIDGE KEEPER: What is your favorite color?

SIR GALAHAD: Blue. No yellow– Auuuuuuuugh!

Monty Python and the Holy Gail[1].

Subjective preference is a fickle thing, as the fate of Sir Galahad demonstrates. In Monty Python and the Holy Grail, King Arthur and the Knights of the Round Table have to cross the Bridge of Death on their quest for the Holy Grail. A fearsome Bridge Keeper poses three questions to each knight, which he must answer successfully in order to pass. Sir Galahad bravely steps forward to face his turn. He has no problem answering fact-based questions, but when questioned about his subjective preference— “What is your favorite color?” — Sir Galahad wavers. First he answers confidently: “Blue”, but then hastily changes his mind; “No, yellow.” This lack of conviction causes Sir Galahad to be cast into the Gorge of Eternal Peril.

Unlike Sir Galahad, most choices in our daily lives do not need us to consider our doom in a fiery chasm as a possible outcome. However, like Sir Galahad both humans and animals often deliberate over different options when choosing based on subjective preference. Orbitofrontal cortex (OFC) is critical for decisions based on preferences, particularly when different sources of information need to be integrated [2,3]. Recent lesion and inactivation studies show that OFC is also required to integrate the sensory features of potential outcomes of a choice – its taste or size – with the moment-to-moment value of that outcome [4,5,6**,7]. Data from neurophysiology studies generally agree with this notion; neurons in OFC encode the attributes of potential choice outcomes that are essential for preference judgments including, but not limited to outcome magnitude, sensory attributes, probability, risk, delay, and effort [8]. Within primate OFC there even appear to be functionally distinct subregions contributing to this evaluative process. Walker’s area 13 is required for updating the subjective value of options, and Walker’s area 11 is required to utilize these updated values when making choices [9**].

While neuropsychology studies have identified behaviors that critically depend on OFC function, and neurophysiology studies have revealed what OFC neurons encode, little is known about how OFC contributes to dynamic aspects of decision-making. In some cases these dynamics happen rapidly, for example during deliberation, changes of mind, or even counterfactual considerations, whereas in others these dynamics involve longer time-scale changes across contexts or tasks. Investigation of rapidly changing aspects of neural activity in decision-making has been revealing in other areas of the prefrontal cortex (PFC) [10**,11]. Recent research has begun to address these questions in OFC, and here we review work that has considered how the activity patterns of single neurons change during individual trials or across different task settings. Across-trial dynamics within one context are not considered here, as they have already received a great deal of attention [e.g. 12,13,14]. Understanding what drives dynamic shifts in patterns of neural activity will be critical in determining how OFC parses information and functions in more naturalistic settings, which will help to constrain theories of OFC function.

Valuation and choice dynamics in OFC

As we make choices, we tend to evaluate each possibility in turn. When the options are visually presented, our eyes rapidly move from one option to the next [15]. These shifts of visual attention are observed during exploratory behavior [16] and are believed to coincide with a decision-maker serially assessing the current value of each option [15]. Since there are indications that attention can modulate OFC outcome encoding [17–19], McGinty and colleagues tested whether self-initiated shifts in visual attention in monkeys correlated with changes in OFC neurons on a moment-by-moment basis [20*]. Here, different images predicted different amounts of reward, and single images were presented randomly on different trials. A key feature of their design was to allow subjects to freely move their eyes after the stimulus was presented (Fig. 1A). Each trial imposed a 4s wait-time between the appearance of a cue and the delivery of reward, so the monkey’s eyes tended to drift around the screen, frequently returning to the reward predicting stimulus. Consistent with prior work, the firing rates of many neurons in OFC were modulated by the amount of reward associated with the image on the screen (Fig. 1B), but this reward encoding changed depending on where the monkeys were looking. When monkeys looked directly at the stimulus, OFC reward signals were strongest, and they diminished as monkeys looked further away (Fig. 1C). In such free viewing conditions, eye position likely reflects the monkey’s fluctuating attention. Indeed, salient cues tend to capture attention and monkeys will look at salient stimuli more frequently during natural viewing [21]. In addition, a recent study manipulated the salience of stimuli in peripheral vision to attract attention while requiring monkeys not to move their eyes, and found that OFC encoding was also modulated by covert attention [22]. Taken together, there is clear evidence that attention dynamically modulates value signals in OFC.

Figure 1. The locus of fixation modulates OFC reward responses.

A) After an initial fixation period, monkeys were shown images associated with no reward, a small reward or a large reward. Immediately after an image appeared, the monkeys were free to move their eyes. The image remained on screen for 4s before the corresponding outcome was delivered. B) Taking a standard analytic approach to the data in which firing is aligned to stimulus onset, McGinty et al. found that the firing rates of neurons in OFC were modulated by the amount of reward associated with the different stimuli, most prominently around the time of initial stimulus presentation. C) When the data were aligned to where the subjects were looking during the period of free viewing, a different pattern emerged. When monkeys’ gaze was directed close to the stimuli, this OFC neuron encoded the reward amount associated with the stimuli (left), whereas this encoding was suppressed when gaze was directed away from the stimuli (right). Adapted from [20].

McGinty and colleagues only presented single images to the monkeys on each trial, meaning it is unclear how the dynamics they observed might contribute to deliberation and choice when multiple options are available. One challenge in extending these findings to choice settings is how rapidly most decisions can be made. While McGinty et al. found dynamics in OFC over seconds, monkeys likely make simple preference decisions in a few hundred milliseconds. This makes it statistically difficult to determine if neural activity is related to one particular option when there are multiple alternatives. To overcome this problem, Rich and Wallis recorded from neurons in OFC while monkeys performed two different, but related tasks [23*]. In the first, monkeys were shown stimuli predicting four different amounts of reward in a design similar to that used by McGinty et al., but here monkeys had to fixate the images for a short duration. In the second, monkeys chose between two of the images that they had seen in the first task (Fig. 2A). When monkeys viewed single reward-predicting pictures, OFC neurons tended to encode the value of the reward predicted by each one. Using these trials, the authors trained a classification algorithm to recognize patterns of activity across ensembles of OFC neurons that corresponded to each outcome value. Similar analytic approaches have been employed to understand covert processes such as motor planning and execution [24]. In this case, they were used to reveal neural correlates of outcome evaluation, and identified unique patterns, or ‘states’, corresponding to each reward value. Using the trained classifiers, the authors asked whether OFC activity exhibited similar states when the monkeys made choices between two of these pictures.

Figure 2. OFC ensembles dynamically represent potential outcomes during decision making.

A) Monkeys viewed and chose between different pictures, each predicting one of 4 values of reward. On single picture trials, one image, chosen at random, was shown and monkeys had to fixate the picture for 450 ms to complete the trial and obtain the predicted reward. A linear discriminant analysis (LDA) was trained to classify the 4 picture values from ensemble activity collected on these trials, during the time when each picture was fixated. On interleaved trials, monkeys were presented with 2 pictures drawn randomly from the set of 8, and allowed to make a choice. The trained classifiers were then used to decode picture values from the same neural ensemble in short epochs of time on individual choice trials. B) One example trial, in which the monkey chose between pictures of value 3 and 4. The color bar at the top shows the categorical classification at each time point, and the plot below shows the posterior probabilities associated with each of the 4 values. The dotted line shows when a choice was made. C) Choice times were predicted by the posterior probabilities associated with the chosen values (red) and unchosen values (blue). In the first column, a regression model was used to predict choice times from chosen and unchosen probabilities at each time point. The top panel shows beta coefficients, the bottom quantifies the variance in choice times explained by each factor as coefficients of partial determination (CPDs). Higher chosen probabilities predicted faster choices, while higher unchosen probabilities predicted slower choices. In the right column, the same regression predicted choice times, but unchosen probabilities were replaced with probabilities associated with options that were not available (NA). D) During choices, single OFC neurons encoded both picture values similarly. The plot shows the beta coefficients from a regression for each neuron when it did not contribute to the ensemble used for decoding. The value of the picture presented on the left side of the screen was encoded at times when the left picture value was decoded from ensemble activity (x-axis), and the value of the picture on the right was encoded when the corresponding picture value was decoded (y-axis). Adapted from [23].

In individual choice trials, Rich and Wallis reported that neural activity rapidly alternated between states associated with the two images that were presented (Fig. 2B). Notably, the pattern of state changes was not random, but was related to monkey’s choice behavior. Over the course of each trial, OFC activity was most likely to be in the state corresponding to the image that was ultimately chosen, potentially because subjects attended to that stimulus more often during the trial. In addition, the more this “chosen state” was decoded in a trial, the faster the monkey made a decision (Fig. 2C). Of course it is hard to know whether these changing states in OFC reflect the evaluation of the different options, as opposed to the comparison between them, or both. The finding that options are evaluated by neurons in a largely independent frame of reference, however, suggests that the signals in OFC may be more related to option evaluation (Fig. 2D). If options were being actively compared, one might expect the states to interact with or inhibit each other, as has been reported in ventromedial PFC [25]. Regardless, these results, together with a rodent study that also found transient representations of different choices, outcomes, and forgone choices [26**], begin to reveal how dynamic changes in OFC activity are associated with deliberation between choice alternatives in real time. In light of the studies showing attentional modulation in OFC, one possibility is that these dynamics reflect rapid shifts of covert attention. While subjects were allowed to freely look at the options, most of the time they did not, instead relying on peripheral vision to make a choice. When they did view both options, the targets of gaze did not reliably drive OFC states, suggesting that OFC modulations, while potentially related, are not constitutively linked to eye movements. This raises an important caution in interpreting decision-making studies in OFC. Considering the influence of events that capture both covert and overt attention, even transiently, clearly needs to be central to experimental design [27].

Cross-task encoding dynamics in OFC: similar or different substrates for valuation and choice?

The aforementioned studies detail how neurons in OFC dynamically encode outcomes and choices within individual trials, but what about dynamics from one situation to another? Despite response variability within a trial, the average activity of most neurons can be explained by one or a few task variables, for example the amount of reward associated with a stimulus. When considering neuronal activity across task contexts separated by minutes, we could expect that this average response property is relatively fixed for any given neuron. For example, neurons may encode the value of chosen stimuli irrespective of the type of decision being made. Another possibility is that neurons may be differentially recruited during decision-making depending on the task at hand [28,29], so that response properties in one condition do not predict those in another [30]. Finally, there could be heterogeneity, such that some neurons have similar properties across tasks, potentially as a result of shared task features or training history, while others remap to uniquely encode different situations, perhaps playing a role in differentiating between rules, contexts, or conditions. Such distinct neuronal recruitment patterns across contexts have been reported previously in hippocampal place cells [31] and frontal eye fields [32].

Recently, two studies investigated these possibilities [33,34]. In one, Xie and Padoa-Schioppa tested monkeys in two contexts. In the first context, they chose between two fruit juice of varying amounts, for instance apple versus orange juice, then in the second they made choices between another two additional juices, for instance cranberry versus grape juice (Fig 3A). These researchers previously found that subgroups of neurons in OFC encode the value of one type of juice (offer value neurons), the value of the option that was selected regardless of juice identity (chosen value neurons), and the identity of the chosen juice (chosen juice neurons) [35]. In this experiment many OFC neurons continued to encode the same characteristic variable from one context to the next (numbers on the matrix diagonal, Fig 3B). For example, a neuron that encoded chosen value in the first block would continue to encode chosen value in the second, even though the options the animal was choosing from were different. That is, the authors showed that neurons in OFC were to some degree blind to the sensory attributes of the juices, and instead encoded the subjective preference of the monkeys. Indeed, when one juice was held constant across blocks (i.e. block 1: A versus B; block 2: B versus C, where preference order was A>B>C), chosen juice neurons predominantly encoded the monkey’s preferred juice even if this meant encoding the opposite juice identity (i.e. neuron signaled the amount of juice A, not B, offered in first block and then switched to signaling the juice B, not C, in second block). Thus, there were elements of the OFC circuit with persistent coding properties across contexts, notably the encoding of subjective preferences in OFC neurons.

Figure 3. Neural encoding in OFC across tasks.

A) To investigate how neurons in macaque OFC encoded decision factors across tasks, Xie and Padoa-Schioppa tested monkeys in two settings: context 1 where subjects chose between juices A and B (e.g. apple versus orange juice) and context 2 where they chose between juices C and D (e.g. cranberry versus grape). On each trial two stimuli were presented, each associated with different amounts of juice. Monkeys’ subjective preferences were computed by systematically varying the amounts of juice of each type over the course of a block. Then the relative values of each option could be derived. B) The matrix plots show the number of OFC neurons that either encoded offer value (positive or negative encoding association), chosen value (positive or negative encoding association), and chosen juice or were unselective for any decision variables across the two contexts. Red borders represent where proportions were statistically above chance based on an odds ratio test on the joint probability. C) Peri-event time histograms and raster plots from Rudebeck et al., showing an example neuron recorded across two tasks: the first where stimuli were well-learned (top panels) and a second where novel stimuli were presented and subjects had to learn how much reward was associated with each image (bottom panels). The neuron encoded the amount of reward received on each trial in the familiar and novel settings, albeit with different encoding schemes. Inset figure shows waveform shape in familiar and novel settings. D) Proportion of neurons encoding the amount of reward in the familiar setting (blue) and in both familiar and novel (red). Yellow numbers represent the percentage of neurons that had persistent encoding across tasks. Adapted from [33,34].

A less emphasized aspect of these results is that a substantial proportion of OFC neurons did, in fact, change their encoding across task contexts. For example, in Fig. 3B approximately a third of offer value neurons in OFC neurons encode this decision-making feature irrespective of the context, i.e. neurons that encoded offer value in the first context also encoded offer value in the second context. This proportion is much higher than expected based on the joint probability, meaning that OFC circuits maintain some consistencies across tasks. But it also means that roughly two thirds of the neurons changed their response properties, either switching to become chosen value, chosen juice or becoming non-selective. The proportion of chosen value neurons persistent across tasks was higher, around 50%, which implies that the other 50% of neurons changed what aspect of the task they encoded. Therefore, these results suggest that there are also flexible aspects of OFC encoding across contexts, which is consistent with recent computational accounts of prefrontal function [30,36].

In another recent study, Rudebeck and colleagues looked at how reward-encoding neurons in OFC remapped when monkeys switched from a setting where they made choices between well-learned, ‘familiar’ images that predicted different amounts of reward to ‘novel’ images, where they had to learn new stimulus-reward associations [34] (Fig 3C). Similar to Xie and Padoa-Schioppa, they reported that a sizable proportion, 25–30% of the neurons, signaled the reward-value of the images presented on each trial (similar to offer value neurons) irrespective of whether the stimuli were familiar or novel (Fig. 3D). Even higher proportions of neurons signaled the expected and received value of the reward across both familiar and novel tasks (50–60%, Fig. 3D). This higher correspondence across settings mirrors the Xie and Padoa-Schioppa finding that chosen value neurons are more likely to persist across tasks (Fig 3B). However, it was again the case that over half of the neurons remapped when images changed from familiar to novel. Thus, some OFC neurons are variably recruited into functional ensembles depending on the setting while others maintain their roles. This mirrors studies reporting variable encoding across tasks in the PFC in general and OFC in particular [28,32]. Such variability may improve adaptive decision making by allowing settings to be differentiated. Determining what causes neurons in OFC to be differentially recruited into ensembles between contexts, tasks, or settings is important to establish in future studies. In this endeavor, computational models that describe how different behavioral states are parsed will be particularly helpful [37].

Intrinsic and extrinsic influences on OFC dynamics during valuation and choice

Given these within- and across-task dynamics, we can return to the question of how such changes in OFC activity patterns arise. For instance, are they the product of intrinsic network computations, such as competition between different neural ensembles or the properties of individual neuronal types [38]? Or perhaps these dynamics are a consequence of the inputs to OFC from sensory, limbic or mnemonic areas [39,40], and different OFC responses reflect information about the potential outcomes being processed through a distributed network. Investigating these intrinsic and extrinsic drivers of OFC dynamics will be a critical task going forward. One approach is to simultaneously record in OFC and an input structure [41]. An alternative is to assess OFC activity when an input is removed. Neural activity related to reward in OFC is inextricably linked to the amygdala [41,42] and recent anatomy studies have shown that parts of OFC form distinct circuits with the amygdala, striatum, and thalamus [43**,44]. Rudebeck and colleagues found that excitotoxic amygdala lesions led to a decrease in the proportion of neurons in OFC encoding the reward value of visual stimuli during choices (Fig 4A)[17], in keeping with previous findings in rodents [42]. However, while these responses were altered, suggesting an extrinsic influence of the amygdala, some dynamic processes in OFC remained intact. Before and after amygdalectomy, OFC neurons encoded the reward value of two choice options presented sequentially in a largely independent manner, i.e. the value of one option did not appreciably influence the coding of the other option (Fig. 4B). This is similar to what is shown in Figure 2D. While the proportion of neurons encoding stimulus values decreased after amygdalectomy, this independent coding scheme was still present in the remaining neurons. While OFC receives many other inputs, it is possible that independent encoding of stimuli in OFC is a feature of the intrinsic circuitry in this region. Determining how other parts of the brain, such as the hippocampus, causally influence OFC encoding dynamics, both within trial and across tasks, will be important to determine, as recent findings suggest a role for hippocampus in OFC encoding of associative structure in rodents [45**].

Figure 4. Amygdala lesions do not affect independent value coding of choice options in OFC.

A) Neural activity was recorded in OFC while monkeys performed a two-option choice task where stimuli associated with different amounts of reward were sequentially presented [17]. After the stimuli were presented, monkeys were allowed to choose between them and receive the reward amount associated with the chosen stimulus. Many OFC neurons encoded the amount of reward associated with both the first stimulus (S1) and second stimulus (S2). B) Percent of neurons in OFC before (blue) and after (red) lesions of amygdala classified by a sliding hierarchical ANOVA as encoding the S2 reward value, S1 relative value or interaction between these two factors during the stimulus-2 period. A high proportion of neurons encoding the interaction between S2 reward and S1 relative value would indicate that OFC correlates of reward-value are relative. A low proportion would suggest that encoding is independent of other options. Inset figure shows across all neurons the maximum slope of encoding taken from a regression conducted on neurons that encoded the reward-value of S1 and S2 before (blue) and after lesions (red). Regression lines are fitted to the data. Non-significant neurons are in gray. Adapted from [17].

CONCLUSIONS

During preference decisions, the neural correlates of value and choice in OFC are not stable but are dynamically modulated, both within trials and across contexts. Dynamic aspects are rarely addressed by theoretical accounts of OFC function [7,46]. For example, the hypothesis that OFC tracks the current position in ‘task space’ [47] predicts across trial dynamics of neural activity, as well as some aspects of across context dynamics. From this view, the state space transitions occur when incoming information indicates that a new context has been encountered, and this should be reflected by altered neural responses. However, it’s not clear how some aspects of state spaces generalize across contexts, or how within-trial neural fluctuations fit in this theory. In the case of single trial dynamics, since the proposed cognitive maps are defined by the information carried in OFC, it could be that neural coding traverses different representations, ‘states’ [23], or attractors within this map of ‘task space’ [48] during single trials. From this view, features of the task constrain the state space of neural activity, but there is also moment-to-moment variability within this space. Shifts from one state to another could be driven by intrinsic or extrinsic processes, for example through modification of transition probabilities between states. Under different decision pressures, such as time, or in psychiatric disorders where decision biases are prominent [49,50] the transition probabilities may be altered. In the latter case a reduction in the transition probabilities away from one specific state could drive decisions in a stereotypical manner. Determining what shapes these influences in OFC is, we think, the holy grail of understanding the role of OFC in decisions based on subjective preference.

HIGHLIGHTS.

• Reward-value signals in orbitofrontal cortex (OFC) are modulated by eye position.

• During choices ensembles of neurons in OFC dynamically represent options.

• Reward encoding of OFC neurons heterogeneously remaps across tasks.

• Independent encoding of choice options in OFC is not dependent on amygdala input.