Neural Basis of Learning and Preference during Social Decision Making

Abstract

Social decision making is arguably the most complex cognitive function performed by the human brain. This is due to two unique features of social decision making. First, predicting the behaviors of others is extremely difficult. Second, humans often take into consideration the well-beings of others during decision making, but this is influenced by many contextual factors. Despite such complexity, studies on the neural basis of social decision making have made substantial progress in the last several years. They demonstrated that the core brain areas involved in reinforcement learning and valuation, such as the ventral striatum and orbitofrontal cortex, make important contribution to social decision making. Furthermore, the contribution of brain systems implicated for theory of mind during decision making is being elucidated. Future studies are expected to provide additional details about the nature of information channeled through these brain areas.

Introduction

Decision making can be understood as the process of selecting an option that is expected to produce the most desirable outcome. In most cases, the predictions for the outcomes from alternative actions are based on the previous experience of the decision maker. In addition, decision making can be considered social, when its outcome depends jointly on the choices of multiple decision makers. For animals living in groups, including humans and other primates, purely individual decision making is rare, and most decisions are made in social settings. This review focuses on recent neurobiological findings that have begun to shed light on two important features of social decision making. First, predicting the outcomes of different actions is difficult in social settings, as the actions of other decision makers change more unpredictably than inanimate objects in the animal's environment. In this regard, the ability to infer about the intentions and knowledge of other animals, referred to as the theory of mind, is crucial. Second, social decision making in humans and other primates can be influenced by other-regarding or social preferences. A central tenet in the classical game theory is that decision makers, or players, choose their actions purely on the basis of self-interest. However, the predictions of such classic game theory often fail to predict actual human behaviors. Moreover, whether and how much the decision maker cares about the reward given to others is affected by a variety of neural and social factors [1-3].

Model-based reinforcement learning and social decision making

When humans and animals face an unfamiliar environment or their environment changes unpredictably, their decision-making strategies will be adjusted accordingly. The reinforcement learning theory provides a parsimonious account of this process for many types of decision making [4], including social decision making. In this framework, the likelihood of selecting each action is determined by a set of value functions that are adjusted according to the animal's experience. Algorithms in the reinforcement learning theory can be divided into two categories. In simple or model-free reinforcement learning, changes in the value functions are driven by the actual outcomes resulting from the actions chosen by the decision maker. In model-based reinforcement learning, value functions for multiple actions can be changed simultaneously and more flexibly according to the internal model of the decision maker about his or her environment without necessarily having to experience the outcome of each action (Figure 1).

Figure 1.

Components of social decision making and their possible neural substrates. The environment of a decision maker provides 3 different types of information useful for decision making. First, it delivers reward, and its deviation from the predicted reward (reward prediction error) modifies the value functions used to select actions. Second, information about various physical objects can be used to update the decision maker's model about his or her environment. Third, observed actions of other actors can be used to update the decision maker's model about their likely future behaviors. In addition, social preference of the decision makers can also influence their value functions. FEF, frontal eye field; Hipp, hippocampus; M1, primary motor cortex; PFC, prefrontal cortex; PPC, posterior parietal cortex; SC, superior colliculus; SNc, substantia nigra pars compacta; vmPFC, ventromedial prefrontal cortex; VS, ventral striatum; VTA, ventral tegmental area.

Behaviors of human decision makers in a dynamic but non-social environment are best accounted for by hybrid learning models that combine the features of model-free and model-based reinforcement learning models [5••,6]. In a social setting, decision makers might update their beliefs about the choices of other decision makers according to their newly observed behaviors, and utilize such information to guide their subsequent choices. This is an example of model-based reinforcement learning, and is referred to as belief learning in game theory [7]. As in individual decision making, human behaviors during iterative social interactions are also more consistent with hybrid learning models than with model-free reinforcement learning models or pure belief learning models [8, 9••]. Hybrid learning models also more accurately account for the choices of non-human primates during a computer-simulated rock-paper-scissors game [10, 11•].

Much of our knowledge about the neural mechanisms of reinforcement learning is based on the results from experiments in which the observed behaviors of the subjects could not distinguish between these two different types of algorithms. For example, neurons modulating their activity according to the rewards expected from a particular state or action are widespread in the brain [12-16, 17•]. In addition, signals related to the reward prediction error, namely, the difference between actual and expected rewards are also found in multiple brain areas, including the ventral tegmental area, substantia nigra pars compacta [18], striatum [19,20], anterior cingulate cortex (ACC) [21,22•], and prefrontal cortex [23]. However, whether such signals reflect the output of model-free or model-based reinforcement learning algorithm is still not well understood.

Recently, several studies have begun to elucidate how signals related to the reward values and prediction errors computed by model-based reinforcement learning algorithms are distributed in the brain. Signals related to reward prediction errors derived from model-free and model-based reinforcement learning algorithms might be co-localized in the striatum [5••, 6]. Interestingly, during strategic social decision making, activity in the rostral ACC was related only with reward prediction errors derived from a belief learning algorithm [9••]. These results raise the possibility that model-based reward prediction errors might be processed differently in the brain depending on the social features of the behavioral task. Additional brain areas, such as the hippocampus and prefrontal cortex, might be involved in predicting the outcomes of choices according to a model-based reinforcement learning algorithm in a non-social context [6].

During social decision making, inferences about the likely behaviors of other decision makers become recursive, as a group of decision makers try to figure out how others in the group expect each other to behave. A set of brain areas associated with theory of mind, such as the medial prefrontal cortex (mPFC) and temporoparietal junction (TPJ), might be critically involved in such recursive strategic reasoning [24-26]. For example, during the beauty contest game, in which the object is to pick a number as close to 2/3 times the average of all the numbers chosen by the participants, subjects who displayed high levels of strategic reasoning also showed higher activity in the mPFC [27••]. Another study found that uncertainty in the inferences about the other decision maker's strategy during a stag-hunt game recruited the rostral medial prefrontal cortex, while activity in the DLPFC increases with the depth or level of strategic reasoning [28••]. Activity in the DLPFC was also related to the level of strategic deception during a bargaining game [29••]. These findings suggest that DLPFC activity during strategic reasoning might reflect the higher demands for working memory and cognitive control.

Single-neuron recording studies in non-human primates have identified signals related to specific conjunctions of actions and their outcomes during computer-simulated competitive games [12, 30•]. Activity related to action-outcome conjunctions provides the information necessary for updating the value functions for specific actions according to model-free reinforcement learning algorithms. Neurons involved in updating the value functions for unchosen actions according to their hypothetical outcomes are co-localized in the same brain areas that process actual outcomes and the corresponding reward prediction errors, such as the ACC [31••] and prefrontal cortex [11•]. These areas might provide converging inputs to the brain circuits responsible for updating the value functions for different actions. In particular, during a simulated rock-paper-scissors game, neurons in the orbitofrontal cortex and lateral prefrontal cortex often encode hypothetical reward that could have been obtained from a particular action, in addition to specific conjunctions of chosen action and its actual outcome [11•] (Figure 2). In both humans and monkeys, neurons in the medial frontal cortex encode specific actions produced by others, and thereby might contribute to extracting the information about hypothetical outcomes associated with the same actions [32, 33••].

Figure 2.

Orbitofrontal cortex encodes hypothetical outcomes during a competitive game. (a) Simulated rock-paper-sciossors task performed by monkeys (adapted from 11•). The animal was required to shift its gaze towards one of the 3 peripheral targets when the white central target was extinguished. After a short delay, the peripheral targets all changed their colors and indicated the amount of reward available from each location (e.g., purple = 2 drops of fruit juice), which was determined by the payoff matrix of a biased rock-paper-scissors game shown on the right (R, rock; P, paper; S, scissors). (b) A single neuron in the orbitofrontal cortex that encoded how much reward could have been earned from an unchosen winning target (indicated by ‘W’ in the top panels). The spike density function of this neuron is shown separately according to the position of the winning target (columns), the position of the target chosen by the animal (rows), and the magnitude of the reward available from the winning target (colors).

Neural basis of social preference

The game theory, originally developed by von Neunman and Morgenstein [34], seeks to identify a set of strategies expected for a group of rational and selfish decision makers, and can provide useful approximations to human behaviors observed in a broad range of social interactions [7]. However, there are many counter-examples violating the assumption of purely self-interested homo economicus. Not only are people often willing to give up some of their incomes to benefit others, but they can also choose costly actions to punish others acting unfairly. The neural correlates of such prosocial preferences are increasingly better understood.

Charitable donations are common in human societies, and sharing valuable resources or donating them to others induces activity patterns in the brain that resemble those resulting from individual gains. For example, decisions to donate to charitable organizations increase the activity in the ventral striatum, whereas decisions to oppose such donations activate the lateral orbitofrontal cortex [35]. Furthermore, the ventral striatum and caudate nucleus showed greater activations when the monetary transfer to the charity was voluntary compared to when it was forced or taxed [36]. The level of activity in the ventral striatum and ventromedial prefrontal cortex (vmPFC) is also correlated with the subjective value of donation to others [37, 38]. Activity in these two areas is temporally correlated with the activity in other cortical areas implicated in empathy and agency perception, such as the ACC and posterior superior temporal cortex [37,38], suggesting that they might constitute a network of brain areas responsible for modulating social preference according to the nature of interpersonal relationship.

A number of studies have used ultimatum games to investigate the nature of neurobiological correlates of fairness norms. During an ultimatum game, a proposer receives a fixed amount of money and offers a proportion of it to the responder, who then chooses to accept or reject the offer. Although game theory predicts that a self-interested rational proposer would offer the smallest amount possible, such small offers are frequently rejected when humans play this game. Unfair offers leads to activation in the anterior insula, DLPFC, ACC, and amygdala of the responder [39, 40]. Among these areas, DLPFC is implicated for enforcing costly actions with the aim of achieving fair outcomes in the long run. For example, disrupting the DLPFC activity in the responder with repetitive transcranial stimulation (rTMS) during an ultimatum game makes it more likely for unfair offers to be accepted [41], and also reduces the activity related to unfair offers in the vmPFC [42••]. Similarly, the baseline activity of the lateral prefrontal cortex of the responder predicts the rate of acceptance [43]. Faces of proposers that are judged to be trustworthy also increase the acceptance rate, and this might be mediated by the lateral orbitofrontal cortex and its connections with the amygdala and insula [44].

Non-selfish behaviors, including altruistic donations or punishment, can be accounted for by a model of inequity aversion [45]. In this model, the utility for a particular distribution of wealth among the members in a group is diminished by both advantageous and disadvantageous inequity that is related to guilt and envy, respectively. This model has been corroborated and further elaborated by the results from neuroimaging studies. For example, inequitable monetary transfer activates the anterior insula [46], and decreases the activity in the ventral striatum and vmPFC [47••]. Envy and its dissolution (schadenfreude) also activate ACC and ventral striatum, respectively [48]. In addition, individual variability in the strength of social preference is correlated with the activation related to inequity in the amygdala [49]. These results illustrate how effects of multiple contextual factors on social preference might be mediated by a network of brain areas. Both pro-social and anti-social preference has been demonstrated in non-human primates [50, 51••], so this remains an important topic for further research. In addition, activity of neurons in the orbitofrontal cortex of monkeys is influenced by the reward given to another monkey, suggesting that the neural mechanisms underlying social preference in humans and other primates might overlap [51••].

Conclusions

The results from the studies summarized above suggest that the brain regions involved in the valuation of different options during individual decision making, such as the ventral striatum and vmPFC, might perform similar functions during social decision making. Other areas, such as the amygdala and insula, might also contribute to the emotional aspect of decision making in both social and non-social context. Furthermore, areas involved in specific aspects of social perception and cognition, such as the TPJ, might be additionally recruited during social decision making, as when decision makers are engaged in recursive reasoning to predict the actions of others. Recent studies showed that many of these areas thought to be important for social decision making are enlarged when the size of the social group increases [52••, 53], suggesting that additional processing power in these brain areas must be beneficial during complex social interaction. We expect that the future studies will provide more detailed insights into the features of our brain that make us socially competent.

Highlights.

1. Human social decision making is characterized by hybrid learning and non-selfish preference.

2. Striatum and prefrontal cortex underlie learning and inference during social decision making.

3. Orbitofrontal cortex might be a hub of brain areas that determines social preference.