Game theory and neural basis of social decision making

Abstract

Decision making in a social group displays two unique features. First, humans and other animals routinely alter their behaviors in response to changes in their physical and social environment. As a result, the outcomes of decisions that depend on the behaviors of multiple decision makers are difficult to predict, and this requires highly adaptive decision-making strategies. Second, decision makers may have other-regarding preferences and therefore choose their actions to improve or reduce the well-beings of others. Recently, many neurobiological studies have exploited game theory to probe the neural basis of decision making, and found that these unique features of social decision making might be reflected in the functions of brain areas involved in reward evaluation and reinforcement learning. Molecular genetic studies have also begun to identify genetic mechanisms for personal traits related to reinforcement learning and complex social decision making, further illuminating the biological basis of social behavior.

Introduction

The problem of decision making is challenging, because the future outcomes from a particular action are seldom fully predictable. Therefore, decision makers must always take uncertainty into consideration when they make choices¹. In addition, such action-outcome relationships can change frequently, requiring adaptive decision-making strategies that depend on the observed outcomes of their previous choices². Accordingly, neurobiological studies on decision making have focused on the brain mechanisms for mediating the effect of uncertainty and improving the decision-making strategies by trial and error. Such studies have found that signals related to reward magnitude and probability are widespread in the brain and often modulated by the active process of decision making^3–9 (see other papers in this issue). Some of these brain areas might be also involved in updating the preference and strategies of decision makers^10–14.

Compared to solitary animals, animals living in a large social group face many unique challenges and opportunities, as reflected in various cognitive abilities in social domain, such as communication and other prosocial behaviors¹⁵. This review focuses on the neural basis of socially interactive decision making in humans and other primates. The basic building blocks of decision making that underlie the process of learning and valuation also play important roles for decision making in social contexts. However, interactions among multiple decision makers in a social group display some new features. First, behaviors of humans and animals can change frequently, as they seek to maximize their self-interests according to the information available from their environment. This makes it difficult to predict the outcomes of a decision-maker’s actions and to choose optimal actions accordingly. As a result, more sophisticated learning algorithms might be required for social decision making^16,17. Second, social interactions open the possibility of competition and cooperation. Humans and animals indeed act not only to maximize their own self-interest, but sometimes also to increase or decrease the well-beings of others around them. These unique aspects of social decision making are reflected in the activity of brain areas involved in learning and valuation.

Game theory and social preference

A good starting point for studies of social decision making is game theory¹⁸. In its original formulation, game theory seeks to find the strategies that a group of decision makers will converge on, as they try to maximize their own payoffs. Nash equilibrium refers to a set of such strategies from which no individual players can increase their payoffs by changing their strategies unilaterally¹⁹. In a two-player competitive game known as the matching pennies (Fig 1a), for example, each player can choose between two alternative options, such as the head and tail of a coin. One of the players wins if both players choose the same option, and loses otherwise. For the matching pennies game with a symmetrical payoff matrix as shown in Figure 1a, the Nash equilibrium is to choose both options with the same probabilities. Any other strategy can be exploited by the opponent and therefore reduces the expected payoff. A large number of studies in both humans and non-human primates found, however, that for competitive games, such as matching pennies, the predictions based on Nash equilibrium are often systematically violated^17,20,21. As discussed below, this might be due to various learning algorithms used by the decision makers to improve the outcomes of their choices iteratively.

Figure 1.

Payoff matrix for the games of matching pennies (a) and prisoner’s dilemma (b,c). a. A pair of numbers within each parenthesis indicate the payoffs to the matcher and non-matcher, respectively, for the matching pennies game. b. A pair of numbers within the parenthesis indicate the payoffs to the players I and II, respectively, for the prisoner’s dilemma game. The yellow and green rectangles correspond to mutual cooperation and mutual defection, respectively. c. The player I’s utility function adjusted according to the model of inequality aversion. The values of α and β indicate the sensitivity to the disadvantageous and advantageous inequality. For β > .0.4, mutual cooperation becomes a Nash equilibrium.

How game theory can be used to investigate the question of cooperation and altruism can be illustrated by the well known game of the prisoner’s dilemma. Two players participate in this game, and each can choose between cooperation and defection. Each player receives a higher payoff by defecting, regardless of whether the other player chooses to cooperate or defect, but the payoff to each player is higher for mutual cooperation than for mutual defection, hence creating a dilemma (Fig 1b). If this game is played only once, and the players care only about their own payoffs, both players should defect, which corresponds to the Nash equilibrium for this game. In reality as well as in laboratory experiments, however, both of these assumptions are frequently violated. Games can be played repeatedly, often among the same set of players. This makes it possible for some players to train others and force them to deviate from the equilibrium predictions for one-shot games. In addition, humans often cooperate in the prisoner’s dilemma games, regardless of whether the game is one-shot or repeated²². Therefore, for humans, decision making in social contexts may not be entirely driven by their self-interest, but at least partially by their preferences regarding the well-beings of other individuals. In fact, cooperation and altruistic behaviors abound in human societies²³, and might also exist in other non-human primates^24–26. Theoretical studies show that multiple mechanisms, such as kin selection, direct and indirect reciprocity, and group selection, can increase the fitness of cooperators and thus sustain cooperation^23,27. Ironically, costly punishment of defectors or free-riders, often referred to as altruistic punishment, also provides an effective means to deter defection^{23, 28–30}.

In economics, the subjective desirability of a particular choice is quantified by its utility function. Although the notion of utility is often linked to the state of the decision-maker’s personal wealth, when people take into consideration the well-beings of other individuals, the utility function can be expanded to incorporate social preference. For example, Fehr and Schmidt³¹ proposed that the utility function can be modified by the decision maker’s aversion to inequality. For two-player games, the first player’s utility, U₁(x), for the payoff to the two players x=[x₁ x₂], can be defined as follows.

$${\text{U}}_1(\text{x})={\text{x}}_1-\alpha {\text{I}}_{\text{D}}-\beta {\text{I}}_{\text{A}},$$

where I_D = max{x₂ − x₁, 0} and I_A = max{x₁ − x₂, 0} refer to the inequalities that are disadvantageous and advantageous to the first player, respectively. The coefficients α and β indicate sensitivities to disadvantageous and advantageous inequalities, respectively, and it is assumed that β ≤ α and 0 ≥ β < 1. Therefore, for a given payoff to the first decision maker, x₁, U₁(x) is maximal when x₁=x₂, giving rise to the preference for equality. When the monetary payoff in the prisoner’s dilemma is replaced by this utility function with the value of β sufficiently large, mutual cooperation and mutual defection both become Nash equilibria³² (Fig 1c). When this occurs, a player will cooperate as long as he or she believes that the other player will cooperate as well.

Evidence for altruistic social preference and aversion to inequality has been also found in other experimental games, such as dictator game, ultimatum game, and trust game^17,32, and their possible neural substrates have been examined in several studies³³. In a dictator game, the dictator receives a fixed amount of money and donates a part of it to the recipient. This ends the game, so there is no opportunity for the recipient to retaliate. Any amount of donation reduces the payoff to the dictator, so the amount of money donated by the dictator provides a measure of altruism. During dictator games, people tend to donate on average about 25% of their money¹⁷. An ultimatum game is similar to the dictator game in that one of the players (proposer) offers a proportion of the money given to the recipient, who now has the opportunity to reject the offer. If the offer is rejected, neither player receives any money. The average offer in ultimatum games is about 40%, and since this is significantly higher than in the dictator game, it implies that the proposers are motivated to avoid the potential rejection¹⁷. Indeed, during the ultimatum game, the recipients reject the offers below 20% about half the time. Another important element in social interaction is captured by a trust game, in which one of the players (investor) invests a proportion of his or her money. This money then is multiplied, often tripled, and transferred to the other player (trustee). The trustee then decides how much of this transferred money would be returned to the investor. The amount of money invested by the investor measures the trust the investor has on the trustee, and the amount of repayment reflects the trustee’s trustworthiness. In other words, trust games quantify the effect of any moral obligations the trustee might feel towards the investor. Empirically, the investor tends to invest roughly a half of his or her money, and the trustee tends to repay the amount comparable to the original investment¹⁷.

Studies on experimental games in non-human primates can provide important insights into the evolutionary origins of social preference displayed by human decision makers. For example, when chimpanzees were tested in a reduced form of the ultimatum game in which the proposer’s choice was binary and between two different pre-set offers, they tended to choose the options that maximize their self-interest, both as proposers and recipients³⁴. Therefore, even though chimpanzees and other non-human primates display altruistic behaviors, fairness plays much more important role in social decision making for humans.

Learning in social decision making

When a group of players play the same game repeatedly, some players might try to train other players. For example, the recipient in an ultimatum game might reject some offers, not as a result of aversion to inequality, but in order to increase his or her long-term payoff by penalizing a greedy proposer. To better isolate the effect of social preference, therefore, many of the behavioral studies on experimental games do not allow their subjects to interact with the same partners repeatedly. In reality, however, learning plays an important role, since people and animals interact with the same individuals repeatedly.

Reinforcement learning theory² formalizes the problem faced by a decision maker trying to discover optimal strategies in an unfamiliar environment (Fig 2). This theory has been successfully applied to the environment that includes multiple decision-makers^{17,20,21,35,36}. In reinforcement learning theory, the sum of future rewards expected from a particular action in a particular state of the environment is referred to as value function. Future rewards are often exponentially discounted so that immediate rewards contribute more to the value functions. Similar to utility functions in economics, value functions determine the actions chosen by the decision makers. In addition, the difference between the reward predicted from the value functions and the actual reward is referred to as reward prediction error. In simple or direct reinforcement learning algorithms, value functions are updated only for the chosen actions and only when there is a reward prediction error².

Figure 2.

A model-based reinforcement learning model applied to social decision making. The decision maker receives reward according to his or her own action and those of other decision makers (DM) in the environment, and updates the value functions according to the reward prediction error. In addition, the decision maker updates his or her model of the environment, including the predicted actions of other decision makers. The fictive reward prediction errors (fRPE) resulting from such model simulations also influence the value functions.

Although reward has a powerful effect on choice behavior, decision makers receive many other signals from their environment. For example, they may discover, after their choices, how much reward they could have received had they chosen a different action. When such hypothetical payoffs or fictive rewards differ from the rewards expected from the current value functions, the resultant error signals, referred to as fictive reward prediction error³⁷ or regret³⁸, can be used to update the value functions of corresponding actions. Such fictive reward prediction errors can indeed influence the decision maker’s subsequent behaviors during financial decision making³⁷. In model-based reinforcement learning algorithms, fictive reward signals can be generated from various types of simulations or inferences based on the decision maker’s model or knowledge of the environment. These fictive reward signals might play a crucial role in social decision making, when the simulated environment includes other decision makers (DM in Fig 2).

In game theory, estimating the payoffs from alternative strategies based on the expected actions of other players is referred to as belief learning^16,17. For example, imagine that you have observed that a particular decision maker tended to apply the strategy of tit-for-tat during a repeated prisoner’s dilemma game. By simulating hypothetical interactions with such a player, you can update the value functions for cooperation and defection and might discover that in the long run, cooperation with this player would produce a higher average payoff than defection. Belief learning and other model-based reinforcement learning algorithms can also update the value functions for multiple actions simultaneously. So far, studies on competitive games in humans and other primates have failed to provide strong evidence for such model-based reinforcement learning or belief learning^20,36,39,40. In contrast, both theoretical and empirical studies have shown that the reputation and moral characters of individual players influence the likelihood and level of cooperation^41–43. For example, a player who has donated frequently in the past is more likely to receive donations when such information is publicly available⁴². Similarly, people tend to invest more money as investors in trust games when they face individuals with positive moral qualities⁴³. Therefore, belief learning models might account for how images of individual players are propagated.

Neural basis of reinforcement learning and valuation

During the last decade, reinforcement learning theory has become a dominant paradigm for studying the neural basis of decision making (see other articles in this issue)⁴⁴. In particular, single-neuron recording studies in non-human primates found that midbrain dopamine neurons encode reward prediction errors^10,45. Dopamine neurons also decrease their activity when the expected reward is delayed⁴⁶ or omitted^10,47. In addition, neurons modulating their activity according to rewards and value functions have been identified in many areas in the primate brain, including the amygdala⁴⁸, the basal ganglia^49–51, the posterior parietal cortex^3,52,53, the lateral prefrontal cortex^54–56, the medial frontal cortex^57–59, and the orbitofrontal cortex^60–62. Nevertheless, how these signals related to value functions in multiple areas are updated by real and fictive reward error signals and influence action selection is still largely unknown⁶³.

Neuroimaging studies in human subjects have also found signals related to expected reward in multiple brain areas, such as the amygdala, the striatum, the insula, and the orbitofrontal cortex^64–66. Non-invasive nature of neuroimaging makes it possible to investigate the neural mechanisms for complex financial and social decision making in humans. On the other hand, the signals measured in neuroimaging studies, such as blood-oxygen-level-dependent (BOLD) signals, reflect the activity of individual neurons only indirectly. In particular, BOLD signals in functional magnetic resonance imaging (fMRI) experiments might reflect inputs to a given brain area more closely than its outputs⁶⁷. Comparison of the results obtained from single-neuron recording and fMRI studies must take into consideration such methodological differences.

Neural correlates of social decision making

Socially interactive decision making tends to be dynamic and the process of discovering an optimal strategy can be further complicated by the fact that decision makers often act according to their other-regarding preferences. Nevertheless, the basic neural processes involved in outcome evaluation and reinforcement learning might be generally applicable, regardless of whether the outcome of choice is determined socially or not. For example, single-neuron recording studies have found that neurons in the dorsolateral prefrontal cortex of rhesus monkeys often encode the signals related to the animal’s previous choice and its outcome conjunctively not only during a memory saccade task⁶⁸ but also in a computer-simulated matching pennies task⁵⁶. Neurons in the posterior parietal cortex also modulate their activity according to the expected reward or its utility during both a foraging task⁵² and a computer-simulated competitive game⁵³. Similarly, imaging studies have found that many brain areas implicated in reward evaluation and reinforcement learning, such as the striatum, insula, and orbitofrontal cortex, are also recruited during social decision making (Fig 3). However, as described below, activity in these brain areas during social decision making is also influenced by factors that are unique to social interactions.

Figure 3.

Brain areas involved in social decision making. a and b. Coronal sections of the human brain showing the caudate nucleus (CD), the insula (Ins), and the orbitofrontal cortex (OFC). c. Sagittal section showing the anterior paracingulate cortex (APC).

One of the areas that play a key role in socially interactive decision making is the striatum. During decision making without any social interactions, activity in the striatum is influenced by both real and fictive reward prediction errors^11,12,37. Reward prediction errors during social decision making also lead to activity changes in the striatum. For example, during the prisoner’s dilemma game, cooperation results in a positive BOLD response in the ventral striatum, when this was reciprocated by the partner, but produces a negative BOLD response in the same areas when the cooperation was not reciprocated^69,70. In addition, the caudate nucleus of the trustee in a repeated trust game displays activity correlated with the reputation of the investor⁷¹. Interestingly, when the investors in trust games receive detailed descriptions of the trustees’ positive moral characters, they tend to invest money more frequently. Moreover, the activity in the caudate nucleus of the investor related to the decision of the trustee is attenuated or abolished when the investor relies on the information about the trustee’s moral character⁴³.

As described above, theoretical and behavioral studies have shown that altruistic punishment of unfair behaviors promotes cooperation. Neuroimaging studies have provided an important insight into the neural mechanisms for producing such costly punishing acts. For example, during the ultimatum game, unfair offers produce stronger activation in the recipient’s anterior insula, when they are rejected than when they are accepted⁷². Since the insula is involved in evaluation of various negative emotional states, such as disgust⁷³, its activation during the ultimatum game might reflect negative emotions associated with unfair offers. In addition, the investors who have the option of punishing unfair trustees during the trust game often apply costly punishment⁷⁴. Such punishment may have some hedonic value to the investors, since activity in the caudate nucleus of the investor was correlated with the magnitude of punishment and increased only when this punishment was effective. By comparing the proposer’s brain activity during the ultimatum game and dictator game, a recent study has also found that the dorsolateral prefrontal cortex, lateral orbitofrontal cortex, and caudate nucleus play an important role in evaluating the threat of potential punishment⁷⁵.

Inequality aversion can give rise to not only altruistic punishment of norm violators but also charitable donation. It has been found that the mesolimbic dopamine system, including the ventral tegmental area and the striatum, is activated by both personal monetary reward and the decisions to donate money to charity⁷⁶. In contrast, the activity of the lateral orbitofrontal cortex increased when the decision makers opposed the charitable organization by refusing donations. Activity in the caudate nucleus and ventral striatum increased with the amount of money donated to a charity, even when this was mandatory⁷⁷. However, the activity in both of these areas was higher when the donation was voluntary.

Although fairness norms have strong influence on social decision making, what is considered fair is likely to depend on various contextual factors, such as the sense of entitlement⁷⁸ and the need for competitive interactions with other players⁷⁹. Similarly, when two participants play the same task and receive the monetary reward for correct answers, the activity in the ventral striatum increased with the amount of money paid to the subject but decreased with the amount of money earned by its partner⁸⁰. In other words, when the subjects are evaluated and rewarded by the same criterion, the activity in the ventral striatum was more closely related to the subject’s relative payment compared to the partner’s payment than the absolute personal payment of the subject. This raises the possibility that the striatal response to the reward received by others might change depending on whether a particular social interaction is perceived as competition or cooperation. Indeed, during a board game in which the subjects were required to interact with each other competitively or cooperatively, a number of brain areas were activated differentially depending on the nature of interaction⁸¹. For example, compared to competition, cooperation resulted in stronger activation in the anterior frontal cortex and medial orbitofrontal cortex. However, whether and how these cortical areas influence the striatal activity related to social preference is currently not known.

Social decision making frequently requires theory of mind, namely, the ability to predict the actions of other players based on their knowledge and intentions^82,83. Many of neuroimaging studies on experimental games have found that social interactions with human players produce stronger activations in several brain areas, often in the anterior paracingulate cortex (Fig 3c), than similar interactions with computer players^84–86. Assuming that more sophisticated inferences are used to deal with human players than with computer players, such findings might provide some clues on the cortical areas specialized for theory of mind. Accordingly, it has been suggested that the anterior paracingulate cortex might play an important role in representing mental states of others^{82, 84–8}. Using the trust game, a recent study has also identified a unique role of the cingulate cortex in representing the information about the agent responsible for a particular outcome⁸⁹. The cortical network involved in theory of mind and perception of agency is, however, still not well characterized, and is likely to involve additional areas. For example, the posterior superior temporal cortex has been implicated in perception of agency^83,86,88, and its activity is correlated with the subject’s tendency for altruistic behavior⁹⁰.

Genetic variations in social decision making

The fitness value of many social behaviors, such as cooperation with genetically unrelated individuals, often depends on various environmental conditions, including the prevalence of individuals with the same behavioral traits. Thus, individual traits related to social decision making could remain heterogeneous in the population, because the selective forces favoring different traits might be balanced⁹¹. Indeed, studies on experimental games commonly reveal substantial individual variability in the behaviors of decision makers, and neuroimaging studies on social behaviors found that activity in several brain areas, such as the striatum and insula, is correlated with the decision maker’s tendency to display altruistic behaviors^69,73,74,75. Some of this variability might be due to genetic factors. For example, the minimum acceptable offer during an ultimatum game is more similar between monozygotic twins than between dizygotic twins⁹².

The genetic mechanisms regulating the synaptic transmission for dopamine and serotonin might underlie individual differences in behaviors and neural circuitries implicated in reinforcement learning and therefore also contribute to the individual variability in social decision making. Among the genes related to dopamine functions, dopamine receptor D2 (DRD2) gene has received much attention. For example, DRD2 polymorphism, such as Taq 1A and C957T, influences how efficiently the decision makers can modify their choice behaviors according to the negative consequences of their previous actions^93,94. Taq 1A polymorphism also influenced the magnitude of fMRI signals related to negative feedback⁹³. In contrast, polymorphism in the dopamine- and cAMP-regulated phosphoprotein of molecular weight 32 kDa (DARPP-32) influences the rate of learning based on positive outcomes, whereas Val/Met substitution in catechol-O-methyltransferase (COMT) might influence the ability to adjust the choice behavior rapidly on a trial-by-trial basis by modulating the dopamine level in the prefrontal cortex^56,94. Social decision making might be also influenced by the genes involved in the serotonin metabolism, such as serotonin (5-HT) transporter-linked polymorphism (5-HTTLPR)^95,96. For example, rhesus monkeys carrying only the short variant of rhesus 5-HTTLPR displayed reduced abilities to switch in object discrimination reversal learning and displayed a higher level of aggression⁹⁷. Little is known, however, about the neurophysiological changes associated with such genetic variability that might underlie behavioral changes in social decision making. It should be also emphasized that any effects of genetic variability on such complex behaviors as social decision making are likely to involve interactions among multiple genes and among genes and environment^96,98.

Conclusion

Social decision making represents one of the most complex animal behaviors, and it oftenrequires the animals to recognize the intentions of other animals correctly and to adjust their behavioral strategies rapidly. In addition, humans can cooperate or compete with one another, and institutional and other contextual factors influence the extent to which humans would sacrifice their personal gains in order to increase or decrease the well-beings of others. As demonstrated by a number of recent studies, the neural basis of such complex social decision making can be investigated quantitatively by applying game theory. These studies have found that the key brain areas involved reinforcement learning, such as the striatum and orbitofrontal cortex, also underlie choices made in social settings. In addition, although it was not focused in this article, the influence of specific hormones on social behavior has been demonstrated. For example, a high level of testosterone increases the likelihood that the recipient would reject relatively low offers during the ultimatum game⁹⁹, and oxytocin increases the amount of money transferred by the investor during the trust game¹⁰⁰. Nevertheless, our current knowledge of neural mechanisms for social decision making is still limited. This will improve as we understand better the genetic and neurophysiological basis of information processing in the brain’s reward system.