The orbitofrontal cortex and response selection

Abstract

Orbitofrontal cortex (OFC) function is often characterized in terms of stimulus–reward mapping; however, more recent evidence suggests that the OFC may play a role in selecting and representing extended actions. First, previously encoded reward associations in the OFC could be used to inform responding in novel but similar situations. Second, when evaluated in tasks requiring the animal to perform extended actions, response selective activity can be recorded in the OFC. Finally, the interaction between the OFC and hippocampus illustrates OFC’s role in response selection. The OFC may facilitate reward-guided memory retrieval by selecting the memories most relevant to achieve a goal. This model for OFC function places it within the hierarchy of increasingly complex action representations that support decision making.

Introduction

The function of the orbitofrontal cortex (OFC) has been described in many ways, including the inhibition of inappropriate responses,¹ the binding of emotional salience to particular outcomes,² the mapping of stimuli to their associated reward, particularly in the context of flexible responding,³ and the integration of reward history with respect to an arbitrary stimulus to generate a reward expectation.⁴ Current theories argue that while the OFC may contribute to decision making, its primary role is to represent the rewards associated with stimuli. In contrast, under current theories higher-order rules and action plans are represented in other parts of the brain, especially the lateral prefrontal cortex (LPFC). Recent findings suggest, however, that the OFC may play a more prominent role in the selection and representation of extended sequences of actions or plans than previously supposed.^5,6 Although neuronal activity in the OFC of rats and monkeys is reliably selective for stimulus–reward conjunctions, extended actions are also coded by OFC neurons. In light of this evidence, we propose that beyond mapping reward history of stimuli or responses, the OFC contributes proactively to action selection and planning, a natural consequence of its role in computing reward expectancy. The example of reward-guided memory retrieval illustrates how the OFC performs this function in action selection. We propose that reward expectancy signals by the OFC contribute to memory retrieval and more generally that the OFC contributes to the prospective function of the prefrontal cortex: the organization of plans and actions.

The OFC and projecting future outcomes

Most studies have focused on the OFC’s role in modifying previously learned responses or mapping new rewards to existing responses, that is, reversal learning. Reversal learning requires animals to withhold previously rewarded responses and produce previously unrewarded ones, and an OFC lesion or inactivation reliably produces deficits in reversal in rats,^7–12 monkeys,^13–15 and humans.^16,17 These results imply that animals with OFC lesions cannot update their representations of rewards associated with particular stimuli or responses when the task contingencies change. Thus, these experiments support the model that the OFC map stimuli or responses to their associated reward.

The evidence also suggests that the OFC contributes to a high-resolution mapping between stimuli and reward history, such that the expected values of stimuli are represented specifically. For example, Pavlovian unblocking is impaired in rats with OFC lesions when the reward is changed, even when the value of the reward is held constant.¹⁸ Similarly, monkeys with OFC lesions have difficulty attributing rewards to particular stimuli.¹⁹ Both studies suggest that the OFC contributes to associating reward with particular stimuli and responses rather than encoding of running tally of recent rewards from all stimuli. This discrete encoding extends to conjoint stimuli. In a Pavlovian overexpectation task, rats were trained to associate reward with each of two separate stimuli. Normal rats learn to reduce their reward expectations after the two stimuli are presented together for several sessions; rats in which the OFC is inactivated during this conjoint training do not.²⁰ Thus, the association of rewards with stimuli extends to the combination of stimuli and the ability to summate their predicted rewards as well. Finally, the OFC reward map includes the reward probability associated with stimuli. If the probability of reward is low enough, humans and monkeys with OFC lesions are impaired in matching stimulus selection with the probability of reward.^17,19 These studies confirm that the OFC is important for mapping highly specific stimuli or responses with reward probability and, thereby, expectancy.

Merely recording the history of reward associated with specific stimuli, however, does not imply a role in response selection. Yet accumulating evidence suggests that the OFC uses information derived from the outcomes of prior choices to guide decision making prospectively in novel contexts, as has been suggested elsewhere.²¹ Consider the following scenario. If you live in New York City and are late for a date downtown, you have two options for getting there quickly: either take the subway or a cab. To minimize the time required for the trip, you must estimate the likely total duration of both means of transit. We argue that this cost/benefit estimate requires the OFC. By drawing on a “library” of remembered facts, responses, and their associated outcomes, the OFC helps you to estimate the likely outcome when placed in this new, but undoubtedly familiar, situation. The purpose of generating a library of reward expectancies is not nostalgia alone, but rather to support future choices. To the extent that the OFC helps encode such a library of specific reward associations, it can participate in a prospective function: estimating the outcome of potential actions. Hence, evidence for a specific reward map in the OFC is consistent with a contribution to response selection, as long as the map can inform future decisions in similar contexts.

The experimental evidence linking stimulus–outcome information coding by the OFC to the prospective selection of action is just emerging. A recent study found that the OFC and the dorsolateral prefrontal cortex code the reward value of both actual and hypothetical outcomes.²² Further, this view may explain anomalous results we observed in rats given temporary inactivation of the OFC and trained to enter one arm (e.g., East) of a plus-maze to find food. Spatial reversal learning (e.g., learning to enter the West arm) was impaired by OFC inactivation,¹¹ but only in the first reversal; subsequent reversals were learned and retained normally (Fig. 1). Moreover, rats trained in the same maze to make serial strategy switches (e.g., from “go East” to “turn left”) did not require the OFC to learn their first spatial reversal, suggesting that something learned during strategy switching was applied to reversal learning. We hypothesized that during strategy switching the rats learned about rewarded paths and that this information was stored in the OFC. Later, when the rat was first given spatial reversal training, it “remembered” the paths that were rewarded previously, even if the path was taken in the context of a different strategy. This interpretation of the experimental evidence supports a role for the OFC in response selection because it explains what the OFC must encode in order to perform this function. What the OFC encodes is a library of paths—sequences of actions toward a goal. Using this knowledge as a base, the likely reward of transit to a known goal arm via a novel path can be estimated. Without an active OFC to retain this information, the animal cannot make a good estimate, leading to poor performance. We could further test this hypothesis by requiring animals to make novel discriminations using familiar responses. For example, an animal could be trained to go north rather than south on the plus-maze. Then the discrimination could be changed such that the animal would still have to go to the north part of the room, but by traversing different paths that previously trained. Normal animals should be able to apply prior training to perform this new contingency rapidly. We hypothesize that rats given OFC lesions would not show the benefit of prior training when tested on this new contingency, and the impairment would be due to a failure of the prospective function of the OFC.

Figure 1.

Knowledge of previously rewarded paths may help generate novel responses. (A) Rats trained on repeated spatial reversals no longer require the OFC. Further, animals trained on repeated strategy switches—an OFC-independent task—no longer require the OFC on subsequent reversals. (B) Because strategy switching in this task has a common path with spatial reversal, knowledge of the rewarded paths learned during strategy switching may be used to support responding during later reversals. (C) A sample experiment for testing the prospective function of the OFC. Animals would be trained to go north rather than south on the plus-maze. Later the animals would be tested on going north via a different path. If the OFC contributes to selecting novel responses, then the OFC lesions should impair this new discrimination.

OFC neurons encode temporally extended actions

OFC activity correlates with stimulus–reward pairings in both rats and monkeys. OFC neurons in rats fire in response to odor-reward pairings.^23–31 Selective firing to the odor–reward pair rises in correlation with learning,^23,24,28,29 and selective firing changes on reversal.^25,29,31 Similar results have been obtained in recordings of activity in the primate OFC.^32–34 This evidence forms an additional basis for the argument that the OFC dynamically tracks the reward values associated with specific stimuli.

OFC coding of responses, on the other hand, is controversial. OFC activity correlates solely with reward in some studies,^35–37 but with response direction in others.^38–40 We observed that OFC neuronal activity in rats was associated with extended sequences of actions on the plus-maze.⁴¹ The OFC neurons fired selectively as rats followed specific paths, and the firing rate was correlated with the likelihood that the path would be rewarded. The neuronal activity did not vary with overt behavior, which was kept constant, but rather with the history of reward associated with a given path. For example, OFC firing patterns on the start arm of the plus-maze anticipated the pending “decision” in the choice point. Thus, while OFC activity did not code behavioral responses per se, it discriminated the reward expectancies associated with these responses in advance of a choice. This pattern of activity suggests that OFC contributes prospectively to response selection.⁴²

The collection of studies that do or do not report response correlates in OFC activity differ along many dimensions, including task design and species, making direct comparisons difficult. Differences in observed OFC coding across experiments are not surprising given the wide range of task demands employed. The OFC is required for some decision-making tasks in rats¹¹ and monkeys^13,43 but not others, and unit activity can vary even during identical overt behaviors. In addition, most of the other studies recorded OFC activity in tasks that require associating explicit, temporally discrete cues such as odors or visual images with temporally limited responses such as a nose poke or a lever press. In these tasks, unit activity is compared during brief, discrete events. Changing the stimulus provides most information within and between trials. Responses are limited spatially and temporally to a visual saccade or motion in a small operant chamber. The limited duration and extent of these responses could obscure response selective activity even as it reveals strong correlations with stimulus presentation. In contrast, our experiment used the distal cues in a relatively large room as discriminative stimuli, which were kept constant throughout testing. The same goal was maintained across several trials. Intertrial intervals and response durations were relatively unrestricted. Because the same stimuli were always available, they could not predict reward reliably, allowing activity to become organized around response contingencies. These factors explain why our study observed activity selective for rule-guided responses in the form of path coding.

Other studies have also suggested that response-related activity in the OFC could reflect species differences or difficulty in specifying homologous neurons in monkeys and rats.⁴⁰ More recently, however, OFC activity was found to correlate with the presentation of the cue that signaled response strategy prior to the response, and the OFC signal preceded activity in the dorsolateral prefrontal cortex.⁵ As in our study, the selective coding was observed in a task that required the animal to combine knowledge about the prior trial with the cue that signaled different response rules. OFC neurons discriminated the strategy during the cue presentation and then the specific response before rewards were presented.⁵ Moreover, OFC activity discriminated responses after the decision and in anticipation of reward.⁴⁰ These patterns of activity are analogous to the “prospective” and “retrospective” activity patterns we observed in path coding neurons in the rat OFC⁴¹ (discussed below). Together, the results suggest that OFC activity in both rats and monkeys encodes information about responses in the context of temporally extended sequences of rewarded actions.^42,44

OFC coding, comparative neuropsychology, and delay discounting

Enormous differences between humans, monkeys, and rats are obvious in both the neuroanatomy of the frontal cortex and its associated behaviors. Differences in the ability to perform complex tasks may be related to the degree to which the species can delay gratification. Delay discounting quantifies the degree to which reward value interacts with its temporal immediacy. People and other animals typically prefer rewards now rather than later (e.g., would you rather I give you $100 now or $200 next year?). Delay discounting has been observed in nonhuman primates, rats, and pigeons.⁴⁵ Moreover, the steepness of decline in value, reflecting a declining ability to delay gratification, is highly variable between species. The persistence of a reward value can be measured by its half-life: the time required for a given reward to be valued at half of its subjective value in the present. For rats, this time is roughly 5 sec;⁴⁶ for monkeys, depending on species, it ranges between 30 sec and a minute.⁴⁷ In humans, the half-life of a reward is usually measured in months but can be highly variable depending on the individual, their age, and the reward used.⁴⁵ Further, delay discounting correlates with the persistence of reward representations in the OFC. OFC activity shows delay discounting in both rats⁴⁸ and primates⁴⁹ on a time scale similar to behavioral measures, in the sense that reward activity declines at the same rate as the animal’s responsiveness to a rewarded stimulus. Differences in delay discounting in both unit activity and behavior may contribute to different aspects of OFC coding relevant to responses, because it may constrain the interval over which response-related OFC activity can be detected. The behavioral complexity of problems and delay discounting may be mechanistically related, in that problem solving may be limited by the half-life of rewards and, thus, the number of cognitive steps that can support conjunctions between stimuli, responses, and rewards. From this view, the OFC in people, monkeys, and rats may perform identical computations (e.g., mapping reward expectancies), but the temporal domain and behavioral range of the reward expectancy mapping may lead to the obvious qualitative differences in behavior among species.

Reward history and guided memory retrieval

The interaction between the OFC and other structures provides a tractable experimental opportunity for investigating mechanisms by which the OFC influences decision making. For example, spatial reversal learning requires both the hippocampus and the OFC, and the two structures may interact in this and similar tasks. The hippocampus and the OFC are reciprocally connected. CA1 directly projects to the PFC in rats and primates,^50–52 and the PFC projects back to the hippocampus via the nucleus reunions of the thalamus,⁵³ which then projects to the entorhinal cortex.^54,55 Lesion studies in human, primates, and rats also support functional interaction between the two structures. OFC lesions in primates⁵⁶ and prefrontal-temporal disconnection⁵⁷ impair object-in-place learning, a hippocampus-dependent task.⁵⁸ OFC lesions in rats can also impair spatial navigation.^59,60 Further, other experiments have shown that the prefrontal cortex interacts with the hippocampus specifically in the context of temporally extended actions.⁴² These studies support the idea that the PFC and hippocampus participate in a functional network to support the memory and execution of spatial and other types of responses.

Simultaneous recording in the OFC and hippocampal circuits suggest that the two structures interact as rats learned in the plus-maze. Local field potentials (LFPs) reflect population oscillations in dendritic excitability within circuits. LFP coherence measures the correlation of these oscillations between structures and is believed to reflect mutual information.^61–63 Simultaneous recordings in the hippocampus and OFC showed substantial theta band (5–12 Hz) coherence, i.e., the two structures oscillated in phase and frequency.⁴¹ Coherence was high during baseline performance, declined during learning, and returned slowly as performance improved. In addition, while OFC neurons fired throughout significantly more of the maze than hippocampal neurons, OFC neurons fired in patterns that resembled prospective and retrospective coding in the hippocampus.^41,64 Prospective coding is defined operationally by observing firing rates on the start arm that differ significantly depending on the goal arm the animal will take. Retrospective coding is defined operationally by differential firing that depends on the start arm of the path taken. Path-selective neurons in the OFC met both of these operational definitions. Thus, in the OFC and the hippocampus, firing on a start or goal arm was dependent on the unfolding history of behavior—on the journey the animal would take or had taken. These observations both support a functional interaction between the OFC and hippocampus to support performance of these tasks.

These results are consistent with several types of interactions between the OFC and hippocampus that could facilitate expectancy guided memory retrieval. Clearly, the covariation in theta frequency across learning could reflect a common influence on otherwise independent brain structures. On the other hand, the interactions between the OFC and hippocampus may be important for memory-guided behavior (Fig. 2). During hippocampus-dependent learning and memory, OFC activity could signal reward from the “top down” and modify hippocampal encoding. From this view, OFC activity would modulate the activity of hippocampal place fields along rewarded paths, thereby facilitating the formation of prospective and retrospective coding activity in the hippocampus. Conversely, the hippocampus may modulate OFC path-selective activity from the “bottom up” and help integrate contextual information with reward expectancy. In this model, OFC path-selective activity could be generated by the combined activity of ensembles of hippocampal place fields to which the OFC adds reward information derived from other brain regions. Finally, the OFC–hippocampal interaction may be bidirectional. OFC activity may modulate hippocampal place fields along rewarded paths while simultaneously drawing contextual information from the hippocampus and associating it with reward. To illustrate these possible roles, consider again the scenario of deciding whether to take a cab or the subway downtown during rush hour. Facts and episodes can help inform the selection of one course over the other. The hippocampus is critical for episodic memories perhaps because its neurons encode the spatial and temporal context of events.⁶⁵ Memory for these past journeys could be used to estimate the duration of each possible route. Furthermore, only specific episodes related to subway and cab travel are particularly relevant in this scenario. Not all memories related to cab or subway travel are relevant; rather, subsets of memories that pertain to cab travel during similar circumstances are key. For example, finding a cab during rush hour is difficult, and traffic moves slowly. Both factual and episodic knowledge would affect the decision of whether to take a cab or the subway. In this sense, the OFC could be involved in selectively retrieving those memories that are most related to the chosen goal—an instance of reward-guided memory retrieval. The OFC may interact with the hippocampus to selectively retrieve the memories that are most relevant to securing the estimated reward. This selection among reward memories based on relevance is an instance of OFC’s role in response selection.

Figure 2.

The OFC and hippocampus may interact through different mechanisms to support reward-guided memory retrieval. (A) OFC path coding could be composed of the combined activity of hippocampal place fields to which the expected reward could be associated. (B) Alternatively, OFC coding could activate hippocampal place fields along rewarded paths, generating prospective and retrospective coding in the hippocampus.

The notions of a “top-down” and “bottom-up” interaction between the OFC and hippocampus are not mutually exclusive, and each model suggests testable predictions, for example, about the effects of OFC lesions on prospective and retrospective coding in the hippocampus. If the OFC modulates hippocampal activity from the top down, OFC lesions should eliminate prospective and retrospective coding in the hippocampus, but hippocampal lesions should not alter path coding by the OFC. In contrast, if the hippocampus influences the OFC from the bottom up, hippocampal lesions should reduce path coding by the OFC, but OFC lesions should not affect hippocampal prospective and retrospective coding. Finally, if the interactions between the OFC and hippocampus are bidirectional, lesions of either structure should impair coding by the other. Ongoing experiments in our lab will test these predictions and assess the degree to which OFC and hippocampal interaction contributes to response selection.

Conclusion

In this review, we propose a model for the role of the OFC in representing and selecting temporally extended sequences of actions. We argue that the OFC encodes a library of outcomes associated with particular stimuli, and this library can be used to estimate the reward expectancy associated with novel responses. In addition, path-selective cells in the OFC shows that it can encode extended responses. Finally, a specific example of the OFC in cognitive control is reward-based memory retrieval. The OFC and hippocampus appear to interact because of high LFP coherence and the presence of prospective and retrospective coding in both regions. The OFC may influence memory retrieval by selecting from the associations in the hippocampus that are most relevant to achieving the coded goal.

This perspective extends existing models of OFC function by suggesting a prospective role for value expectancy coding. Further, we emphasize that the OFC participates in a larger network that represents and supports extended action, which may help to bridge strategic encoding in PL-IL of rats⁶⁶ and the LPFC of primates^67,68 with memory coding by the hippocampus⁶⁵ and action sequence coding by the striatum.⁶⁹ In this model, actions are hierarchically encoded in increasing levels of abstraction by different brain regions, as suggested by Joaquin Fuster, who described these distributed action representations as cognits.⁷⁰ This model of distributed action representation is supported by recent lesion studies that have shown impairment in different aspects of the same task through lesions of different portions of the primate PFC.⁷¹ By describing how the OFC could contribute to response selection, the larger brain networks that support decision making can be better understood.