Prefrontal cortex and hybrid learning during iterative competitive games

Abstract

Behavioral changes driven by reinforcement and punishment are referred to as simple or model-free reinforcement learning. Animals can also change their behaviors by observing events that are neither appetitive nor aversive, when these events provide new information about payoffs available from alternative actions. This is an example of model-based reinforcement learning, and can be accomplished by incorporating hypothetical reward signals into the value functions for specific actions. Recent neuroimaging and single-neuron recording studies showed that the prefrontal cortex and the striatum are involved not only in reinforcement and punishment, but also in model-based reinforcement learning. We found evidence for both types of learning, and hence hybrid learning, in monkeys during simulated competitive games. In addition, in both the dorsolateral prefrontal cortex and orbitofrontal cortex, individual neurons heterogeneously encoded signals related to actual and hypothetical outcomes from specific actions, suggesting that both areas might contribute to hybrid learning.

Hybrid learning during iterative games

Difficulty of choosing an optimal action in a particular situation, namely, an action that produces the outcome most desirable to the decision maker, varies tremendously according to the uncertainty and stability of the decision-maker’s environment. Since the environment changes constantly for all animals, and since the outcomes of alternative actions are seldom completely known, animals must always monitor the outcomes of their actions and adjust their estimates appropriately in order to improve their action selection strategies. The reinforcement learning theory characterizes the computational properties of algorithms that can be used to choose optimal actions in a dynamic environment.¹ As in many other models of decision making, reinforcement learning algorithms are based on a set of quantities, referred to as value functions, that correspond to the desirabilities of outcomes expected from a particular action or a particular state of the environment. In this framework, the probability of choosing a particular action increases with the value function associated with that action.

In the framework of reinforcement learning, specific learning rules describe how the value functions are adjusted through the decision maker’s experience. Depending on the type of information utilized to update value functions, two different kinds of learning models can be distinguished. For example, simple or model-free reinforcement learning models utilize only the information about the value of reward or penalty directly received by the decision maker as a result of chosen actions. However, values of such direct reward or penalty correspond to a relatively small portion of information that can be utilized to predict the outcomes of future actions accurately. If a decision maker could learn from merely observing unexpected changes in his or her environment, then this information might be used to revise the value function for a particular action even before or without taking the corresponding action. This implies that the decision maker’s newly revised model of his or her environment was used to simulate the hypothetical outcomes of alternative actions. Consequently, these simulated hypothetical outcomes can then be used to update the value functions. Accordingly, this is referred to as model-based reinforcement learning.¹ In both types of reinforcement learning models, the value functions are adjusted according to the discrepancies between the reward received by the decision maker and the reward expected from the current value functions. The difference between real and expected reward is referred to as reward prediction error. In model-based reinforcement learning model, value functions can be updated by the reward prediction error computed using the value functions estimated by the decision maker’s model. Moreover, even for actions not chosen by the decision maker, their value functions can be revised according to the difference between hypothetical rewards and the rewards predicted by the current value functions. This is often referred to as fictive or counterfactual reward prediction error.^2,3

Given that model-based reinforcement learning algorithms can allow decision makers to revise their strategies much more rapidly and flexibly, it is perhaps not surprising that choices in both humans and animals can be better accounted for by model-based reinforcement learning.^4–9 In particular, in social settings, behaviors of other intelligent decision makers can change more frequently and unpredictably than inanimate objects. Therefore, simple learning algorithms that depend only on the actual outcomes of previous actions would be insufficient to utilize a variety of social cues available to infer the intentions and likely behaviors of other individuals. Thus, the ability to deploy model-based reinforcement learning algorithms would be especially advantageous when the outcomes of decisions are determined by the behaviors of multiple decision makers in social settings.⁷ During social interactions, decision makers might revise their strategies according to their beliefs about the likely choices of other decision makers, and this is referred to as belief learning in game theory.¹⁰ Therefore, model-based reinforcement learning is analogous to belief learning. In model-based reinforcement learning or belief learning models, the value function for a given action would be updated by the payoff expected for that action, regardless of whether it was actually chosen by the decision maker or not. By contrast, previous studies have demonstrated that during iterative competitive games, behavioral changes in decision makers or players can be accounted for best by a learning model that incorporates the features of both model-free and model-based reinforcement learning.^6,11–13 While hypothetical outcomes associated with unchosen actions still influence the decision maker’s future behaviors, actual payoffs from chosen actions tend to do so more strongly. For example, in the experience-weighted attraction (EWA) model of Camerer and Ho,¹¹ the value function for a given action is updated differently depending on whether that action was actually taken or not. For a chosen action, the value function is updated according to the actual payoff, whereas value functions of unchosen actions are updated according to hypothetical payoffs that would have been obtained from such actions. The learning rate, which controls how rapidly reward prediction errors are incorporated into value functions, can be set differently for actual and hypothetical outcomes. Thus, the EWA is a hybrid learning model combining the features of both simple reinforcement learning and belief learning.

Neural substrates of simple reinforcement learning

For optimal decision making, two different types of computations are necessary. Prior to committing to a particular action, the desirability of outcomes expected from each action must be estimated. This process is likely to be distributed in multiple brain areas, since single-neuron or metabolic activity related to the subjective values of outcomes expected from different actions and objects has been observed in the prefrontal cortex,^14–20 medial frontal cortex,^{21, 22} posterior parietal cortex,^23–26 basal ganglia,^27–35 and amygdala.³⁶ Once a chosen action is executed, then its outcomes must be monitored, and any discrepancies between the expected and actual outcomes must be taken into consideration to revise the decision-maker’s behavioral strategies. Such reward prediction errors are used to update the value functions.¹ Dopamine neurons in the substantia nigra and ventral tegmental area encode the reward prediction errors,^{37, 38} although some of them encode rectified reward prediction errors that might correspond to the saliency of sensory information.^39–41

Many different behavioral tasks have been used to investigate the neural signals related to the subjective values of expected outcomes and reward prediction errors, including Pavlovian conditioning^{36, 37} and dynamic foraging tasks.²⁴ Behavioral tasks simulating competitive games have been also used.^{16, 25, 26, 42–46} For many simple competitive games, the optimal strategy is to choose multiple options stochastically and independently across trials. In game theory, a set of strategies is referred to as a Nash equilibrium when no individual players can increase their payoffs by changing their strategies unilaterally. For example, during a matching pennies game, each of the two players chooses one of two options (e.g., heads and tails), and one of them wins when the two choices match and loses otherwise. For this game, there exists a single Nash equilibrium strategy, which is to choose each of the two options with a 0.5 probability. However, the actual choices observed in humans and non-human primates during competitive games often display systematic deviations from Nash equilibrium strategies. For example, during the matching pennies task, both people and monkeys tend to repeat the same choice after winning more frequently than after losing.^{11, 47–49} This so-called win-stay-lose-switch strategy is a hallmark of a model-free reinforcement learning algorithm, in which the value functions for actions leading to successes and failures are increased and decreased, respectively. Therefore, near-equilibrium behaviors observed in humans and animals during competitive games might result from a model-free reinforcement learning.

Consistent with these behavioral findings, single-neuron recording studies have found that neural signals related to the value functions for alternative actions estimated by a model-free reinforcement learning algoritihm are distributed in multiple brain areas, including the prefrontal cortex and posterior parietal cortex.^{16, 25, 26, 50, 51} In most of these studies, the possibility that the animals might have also utilized model-based reinforcement learning algorithms was not tested. Nevertheless, in order to represent and update value functions for different actions, signals related to the animal’s previous choices and their outcomes must be combined appropriately. Indeed, many neurons in the prefrontal cortex and posterior parietal cortex maintained the signals related to the animal’s choice and its outcome during several trials (Figure 1).^{26, 45, 50, 51} Moreover, it was found that the memory signals related to the animal’s previous choices and their outcomes decay with a range of time constants in a population of neurons in multiple cortical areas, suggesting that they might provide the reservoir of different time constants that might be necessary for optimizing the rate of learning as the stability of the animal’s environment changes.⁵² The signals related to the animal’s previous choices and their outcomes have been identified in other brain areas, including the orbitofrontal cortex and striatum, suggesting that the process of updating the value functions during model-free reinforcement learning is broadly distributed.^53–57

Figure 1.

A. Medial (top) and lateral (bottom) views of the rhesus monkeys’ brain, showing the locations of recorded areas in dorsolateral prefrontal cortex (DLPFC),^{16, 51} dorsal anterior cingulate cortex (ACCd),⁴⁵ and lateral intraparietal cortex (LIP).²⁶ B. Temporal changes in the fraction of neurons significantly modulating their activity according to the animal’s choice (top), choice of the computer opponent (equivalent to action-outcome conjunction; middle), and the outcome of the animal’s choice (bottom) in the current (trial lag=0) and 3 previous trials (trial lag=1 to 3) during a computer-simulated matching-pennies task. The results for each trial lag are shown in two sub-panels showing the proportion of neurons in each cortical area modulating their activity significantly according to the corresponding factor relative to the time of target onset (left panels) or feedback onset (right panels). Large symbols indicate that the proportion of neurons was significantly higher than the chance level (binomial test, p<0.05). Gray background corresponds to the delay period (left panels) or feedback period (right panels).

Prefrontal cortex and hybrid learning

Most animal behaviors can be understood as actions that are selected to maximize the overall desirabilities of expected outcomes. Accordingly, unexpected outcomes of the animal’s chosen actions have powerful influence on its subsequent behaviors. However, this does not mean that the animal’s behavioral strategies are modified only on the basis of actual outcomes of their actions. When humans and animals acquire a new piece of information about the possible changes in their environments, this can also lead to changes in their decision-making strategies even before they experience unexpected outcomes from their actions. Whereas these two different types of reinforcement learning can be clearly distinguished by their computational characteristics, they might be implemented by a common neural substrate. For example, during a multistep decision making used by Daw and his colleagues,⁸ subjects adjusted their behaviors not only according to whether a particular action was followed by reward or not, but also according to whether such reward was obtained following expected sequence of events in the environment or not. These results suggest that a part of the error signals used to modify the subject’s behavior was computed using a model-based reinforcement learning algorithm. Moreover, the blood-oxygen-level-dependent (BOLD) signals in the striatum were influenced by both model-free and model-based reward prediction errors.^{2, 8} Similarly, subjects performing a navigation task in a dynamic virtual maze largely made their choices using a model-based reinforcement learning algorithm, and the value of the chosen actions estimated by the same model was localized in the striatum.⁹

In our recent study, we tested whether the activity of individual neurons in different regions of the primate prefrontal cortex encoded signals that can be utilized to implement model-based reinforcement learning or hybrid learning.¹³ We focused on the dorsolateral prefrontal cortex (DLPFC) and orbitofrontal cortex (OFC, Figure 2A) for several reasons. First, the prefrontal cortex is often considered to play an important role in flexible, context-dependent action planning, which would be facilitated by model-based reinforcement learning,^{58, 59} whereas the striatum, especially the dorsolateral striatum, might support model-free reinforcement learning.⁶⁰ Second, a number of studies have demonstrated that the prefrontal cortex is involved in evaluating the outcomes expected from different actions and choices^{14–17, 61} as well as the actual outcomes resulting from chosen options.^{45, 62, 63} In addition, unexpected state transitions in the environment lead to increased BOLD activity in the human prefrontal cortex, even when they are not associated with reinforcement or punishment.⁶⁴ In hybrid learning and model-based reinforcement learning, the value functions are updated by actual outcomes from chosen actions as well as hypothetical outcomes associated with unchosen actions. Therefore, these results raise the possibility that neurons encoding both actual and hypothetical outcomes might be involved in updating the value functions according to hybrid learning or model-based reinforcement learning algorithms. Finally, neuroimaging studies have also revealed BOLD activity in DLPFC and OFC related to the experience of regret and relief, namely, the discrepancy between the realized outcome and the outcome that could have been obtained by choosing a different action.^{65, 66}

Figure 2.

A. Magnetic resonance image of a rhesus monkey used for neurophysiological recording experiments during a rock-paper-scissors task. Numbers indicate different cytoarchitectonic divisions of the orbitofrontal cortex.⁷⁴ A light blue arrow indicates an electrode track. B. Temporal sequence of a rock-paper-scissors task used to investigate neuronal signals related to hypothetical outcomes.¹³ The amount of reward delivered 0.5 s after feedback onset was determined by the payoff matrix of a biased rock-paper-scissors task (C). D. Feedback colors used to indicate different payoffs. N, Q, S refer to the three monkeys trained on this task.

We examined the activity of individual neurons in the DLPFC (n=308) and OFC (n=201, Figure 2A) while the monkeys played a computer-simulated rock-paper-scissors task (Figure 2B).¹³ During this task, the animal began each trial by fixating a small central target. After a brief (0.5 s) fore-period, three identical peripheral targets were presented, and the animal was required to shift its gaze towards one of these peripheral targets when the central target was extinguished 0.5 s later. Once the animal fixated its chosen peripheral target for 0.5 s, all three peripheral targets changed their colors and revealed the amount of juice reward available to the animal from each target (Figure 2D). This was determined by the payoff matrix of a biased rock-paper-scissors game (Figure 2C), and the reward was delivered to the animal 0.5 s after the targets changed their colors. For example, in trials in which the computer chose the target corresponding to rock, the payoff for the animal was 0, 1, or 3 drops of juice (0.2 ml/drop) for choosing the targets corresponding to scissors, rock, and paper, respectively. The positions of the targets for the animals that correspond to rock, paper, and scissors were counter-balanced across blocks, and the computer opponent simulated a rational opponent in a zero-sum game.^6,13

As described previously,^{6, 13} the animal’s choices during this rock-paper-scissors task are better accounted for by a hybrid learning model in which value functions for different choices are adjusted not only by the actual outcome of the animal’s choice in each trial but also by the hypothetical outcomes from targets not chosen by the animal, better than simple reinforcement learning or belief learning models. Also, neurons in DLPFC and OFC often encoded the magnitude of reward obtained by the animal (Figure 3). For some neurons, activity related to this actual outcome varied significantly across different target locations, suggesting that they might contribute to updating the value functions of different actions according to a model-free reinforcement learning algorithm.^{46, 50} The proportion of neurons encoding the actual outcomes was similar in the DLPFC and OFC. For example, 20.5% and 16.4% the neurons in the DLPFC and OFC, respectively, changed their activity significantly according to the actual outcomes differently depending on the position of the target chosen by the animal. In addition, neurons in both DLPFC and OFC modulated their activity according to the hypothetical payoff available from the winning target even during the trials in which the animal did not win (Figure 4). As in hybrid learning, neurons in both areas were more likely to encode the signals related to actual outcomes of the animal’s choices than those related to hypothetical outcomes. The overall proportion of neurons encoding the hypothetical outcomes from winning targets was not significantly different for the DLPFC (21.4%) and OFC (16.9%). In addition, the proportion of neurons encoding hypothetical outcomes was significantly higher among those also encoding actual outcomes (χ²-test, p<0.001) in both DLPFC (32.3%) and OFC (25.3%). Therefore, in both areas, individual neurons tended to process the information about both actual and hypothetical outcomes. Notably, DLPFC neurons were more likely to encode hypothetical outcomes differently according to the position of the winning target than OFC neurons (17.2% vs. 8.0%;χ²-test, p<0.005).

Figure 3.

An example OFC neuron that modulated its activity only according to the actual outcome of the animal’s choice. A. Average spike density function estimated separately according to the position of the winning target (columns), the position of the target chosen by the animal (rows), and the winning payoff (colors). Thus, the results shown in the main diagonal are from the winning trials. B. Average spike density functions shown as a function of actual payoffs. C. Average spike density function shown as a function of the animal’s choice.

Figure 4.

An example OFC neuron that modulated its activity according to the hypothetical outcome from the winning target. A. Same format as in Figure 3A. B. The average spike rate estimated separately according to the position of the winning target (columns) and the position of the target chosen by the animal (colors).

Signals related to hypothetical or fictive reward⁶⁷ as well as those related to actual reward^{45, 62, 68} had been also observed in the anterior cingulate cortex (ACC). Therefore, the information about both actual and hypothetical rewards might be widespread in multiple areas of the frontal cortex. Nevertheless, there is also some evidence that signals related to hypothetical outcomes in different cortical areas might contribute to different functions. For example, neurons in the ACC tended to encode both actual and hypothetical outcomes indiscriminately,⁶⁷ whereas neurons in the DLPFC and OFC often encoded actual and hypothetical outcomes that resulted or could have resulted from a particular action.¹³ The tendency to encode hypothetical outcomes from specific actions, namely, conjunctions of actions and hypothetical outcomes, was stronger in the DLPFC than in the OFC,¹³ suggesting that DLPFC might be particularly important in updating the value functions for various actions according to both actual and hypothetical outcomes, thereby subserving hybrid learning.

Neural basis of counterfactual thinking

Counterfactual thinking refers to the perception and imagination of alternative outcomes that could have been materialized by a course of actions different from the chosen one, and inferences related to such hypothetical outcomes.⁶⁹ The fact that neurons in the primate prefrontal cortex encode not only the actual outcomes resulting from the animal’s actions but also hypothetical outcomes suggests that the same neural machinery might also underlie counterfactual thinking.⁷⁰ Indeed, disruption of prefrontal functions might impair counterfactual thinking.⁷¹ Moreover, patients with schizophrenia also show reduced abilities to utilize counterfactual thinking to improve their behaviors.^{72, 73} During the rock-paper-scissors task used to examine the activity of prefrontal cortex, both actual and hypothetical outcomes were revealed to the animal explicitly by visual cues.¹³ Nevertheless, convergence of information about actual and hypothetical outcomes in the prefrontal cortex suggests that the prefrontal cortex might mediate counterfactual thinking by aligning and integrating the signals from heterogeneous sources and using them for updating the value functions for different actions and therefore planning future actions. Overall, neural signals related to actual and hypothetical outcomes were distributed similarly in the DLPFC and OFC. However, compared to the DLPFC neurons, OFC neurons were more likely to encode the hypothetical outcomes regardless of the nature of corresponding actions. Therefore, the DLPFC might be more involved in using the information about the hypothetical outcomes to guide the animal’s future behaviors, whereas the OFC might contribute more to the emotional aspect of counterfactual thinking.