Neurocomputational mechanisms of prosocial learning and links to empathy

Significance

Prosocial behaviors are essential for social bonding and cohesion, but the mechanisms that underpin these behaviors are still poorly understood. Using computational modeling and neuroimaging, we show that people can learn to benefit others and that this learning is underpinned by reinforcement learning signals in the subgenual anterior cingulate cortex (sgACC). However, there is substantial individual variability in people’s ability for prosocial learning. More empathic people learn faster and have more selective responses in the sgACC when benefitting others. Our results thus reveal a computational mechanism driving prosocial learning in humans and why empathy and prosocial behavior may be linked. This framework could help to explain reduced empathy and prosocial behavior in people with disorders of social cognition.

Abstract

Reinforcement learning theory powerfully characterizes how we learn to benefit ourselves. In this theory, prediction errors—the difference between a predicted and actual outcome of a choice—drive learning. However, we do not operate in a social vacuum. To behave prosocially we must learn the consequences of our actions for other people. Empathy, the ability to vicariously experience and understand the affect of others, is hypothesized to be a critical facilitator of prosocial behaviors, but the link between empathy and prosocial behavior is still unclear. During functional magnetic resonance imaging (fMRI) participants chose between different stimuli that were probabilistically associated with rewards for themselves (self), another person (prosocial), or no one (control). Using computational modeling, we show that people can learn to obtain rewards for others but do so more slowly than when learning to obtain rewards for themselves. fMRI revealed that activity in a posterior portion of the subgenual anterior cingulate cortex/basal forebrain (sgACC) drives learning only when we are acting in a prosocial context and signals a prosocial prediction error conforming to classical principles of reinforcement learning theory. However, there is also substantial variability in the neural and behavioral efficiency of prosocial learning, which is predicted by trait empathy. More empathic people learn more quickly when benefitting others, and their sgACC response is the most selective for prosocial learning. We thus reveal a computational mechanism driving prosocial learning in humans. This framework could provide insights into atypical prosocial behavior in those with disorders of social cognition.

Prosocial behaviors, namely, social behaviors or actions intended to benefit others, are a fundamental but poorly understood aspect of social interaction (1). To behave prosocially, animals need to learn about the consequences that their actions can have for others. In reinforcement learning theory (RLT), prediction errors (PEs)—differences between expected and actual outcomes—drive learning (2). RLT provides a powerful framework for understanding how animals learn to obtain rewards for themselves (3). However, the processes by which animals learn to make choices that benefit others are unknown. Here we use RLT to characterize prosocial learning, combining functional magnetic resonance imaging (fMRI) and detailed computational modeling of behavior.

Studies using economic games, moral judgments, or charity donation tasks have consistently reported activity in the ventral striatum, posterior regions of the subgenual cingulate cortex/basal forebrain (hereinafter referred to as sgACC), dorsal anterior cingulate cortex (dACC), and dorsolateral prefrontal cortex (DLPFC) during prosocial behavior (4–7). Each of these regions receives input from midbrain dopaminergic neurons (8), and these cortical regions all project to the ventral striatum (9–11). There is substantive evidence that dopamine neurons projecting to this circuit code PEs for rewards delivered to animals (and humans) themselves (3).

The ventral striatum, sgACC, dACC, and DLPFC are also implicated in processing information about rewards others will receive (12–16), PEs when interacting with others (13, 17–19), and prosocial behavior (e.g., refs. 4–7 for reviews). Therefore, information processing in these regions may conform to RLT principles during social interactions. However, no prior work has examined how we learn to make choices that benefit others, a fundamental aspect of behaving prosocially. Do any of these areas signal a unique prosocial PE specifically when learning to benefit another? Or is learning to benefit another encoded in regions that signal PEs regardless of the beneficiary?

Moreover, although humans have a remarkable inclination to engage in prosocial behaviors, there are also substantial individual differences (1, 20–22). Empathy, the capacity to vicariously experience and understand the affect of others (23–27), has been hypothesized to be a critical motivator of prosocial behaviors (25–28). Previous studies have consistently shown that empathy can modulate neural responses to viewing others’ pain (29) and viewing desirable outcomes (rewards) that will be delivered to others (15, 30). Moreover, although empathy can be broken down into separable components associated with different social behaviors and traits (23, 31, 32), studies have suggested that both cognitive and affective aspects of empathic processing may motivate prosocial behaviors (33). Despite this body of research, the mechanistic link between empathy and prosocial learning remains unknown. If empathy is indeed linked to prosocial behavior, we might predict that empathy and prosocial learning would be associated, with those higher in empathy learning more quickly to benefit others.

Participants (n = 31) performed a reinforcement learning task during fMRI. On each trial participants were required to choose between one of two symbols. One symbol was associated with a high probability (75%) and one was associated with a low probability (25%) of a reward. These contingencies were not instructed but had to be learned through trial and error. Critically, participants performed this task for themselves (self), for another person (prosocial), or in a control condition with no beneficiary (no one) (Fig. 1A). The no one condition was crucial to account for previous studies showing fictive reward or fictive prediction error brain responses, which occur when rewards are not delivered to ourselves in entirely nonsocial situations (34–36). This condition also allowed us to test for regions that showed relative specificity for processing social information, that is, regions that did not respond to self or nonself information (see also ref. 37 for a recent discussion).

Fig. 1.

Behavioral task and data. (A) Participants performed a reinforcement learning task in which they had to learn the probability that abstract symbols were rewarded. At the beginning of each block, participants were told whom they were playing for, either themselves, for the other participant, or in a condition where no one received the outcome. (B) Group-level learning curves showing choice behavior in the three learning conditions. Trials are averaged over the three blocks (48 trials total per condition, 16 trials per block) for the self, prosocial, and no one conditions. Dotted line shows chance level. (C) Comparison of learning rates (α) from the computational model. Participants had a significantly higher learning rate when learning in the self compared with the prosocial and no one conditions. (D) Individual differences in empathy (online simulation) modulated the prosocial vs. self learning rate difference, with those higher in empathy having a more similar learning rate between the prosocial and self conditions. (E) Participants were less consistent (higher β) when choosing for no one compared with choosing in the self and prosocial conditions. Asterisks represent significant differences (P < 0.05).

Using an RLT framework, we then conducted detailed computational modeling of trial-by-trial variation of behavior, supported by Bayesian model comparison, to examine whether people were able to learn to benefit others at the same rate that they learned to benefit themselves. We examined whether activity in brain areas previously implicated in coding PEs for ourselves or in prosocial behavior signaled PEs regardless of the beneficiary that received the outcome or whether any of these regions exclusively reflected a prosocial PE when learning to benefit another [ventral striatum; sgACC (Brodmann Area (BA) 25/s24); dACC (BA24); and DLPFC (BA9/46d); Experimental Procedures and SI Experimental Procedures]. Moreover, we hypothesized that if empathy motivates prosocial behavior, then the rate at which people can learn to obtain rewards for others and the neural signatures that underpin prosocial learning should vary with trait levels of empathy.

Results

Behavioral Differences in Learning to Obtain Rewards for Self, Another Person, or No One.

Participants were able to learn to obtain rewards for themselves, the other person, and no one, performing significantly above chance in all conditions (all t values > 9.1, all p values < 0.001, all degrees of freedom = 30; Fig. 1B and Fig. S1). Bayesian model comparison revealed that participants’ choices were best characterized by a model with separate learning rates and choice variability parameters in each condition [winning model evidence (ΔBIC) > 600; see also SI Experimental Procedures and Fig. S2]. Comparing the learning rate parameters between conditions revealed a main effect of learning condition [F(2,60) = 11.47, P < 0.001]. Participants learned more slowly if they were obtaining rewards for another person (prosocial) (d = 0.87, P < 0.001) or no one (d = 0.53, P = 0.01) than if they were obtaining rewards for themselves (Fig. 1C). There was no difference in learning rate between the prosocial and no one conditions (d = 0.25, P = 0.18). Choice variability [main effect of condition: F(2,60) = 7.87, P < 0.001] could not explain these results because participants had similar consistency scores when choosing for themselves and the other person (d = 0.24, P = 0.20) but were more random when choosing for no one compared with themselves (d = 0.46, P = 0.017) and the other person (d = 0.58, P = 0.003) (Fig. 1E). Together, these findings suggest that people have a self-bias in their learning, learning more quickly about rewards for themselves compared with for another person or no one. However, people are similarly variable when choosing for themselves and others and most variable when no beneficiary will receive the reward.

Fig. S1.

Choice behavior in a representative subject in the self, prosocial, and no one conditions. Note that each point represents the average of three trials.

Fig. S2.

Bayesian model comparison results. (A) Results of Bayesian model comparison (natural log scale) using the Bayesian information criterion (BIC), showing decisive evidence in favor of the full model (delta BIC > 10), which includes separate α and β parameters for each condition. (B and C) The BIC score was evaluated for each model/condition and subtracted from the full RL model BIC score for each participant. Values greater than zero indicate that the full RL model was better at describing the behavior of the participant than the null model (B) or the model with fixed parameters across conditions (C) (score above 2 suggests positive evidence in favor of the full RL model, whereas a score above 10 indicates decisive evidence). Box plots (blue) represent the distribution of BIC scores for each condition (median is indicated by the red vertical line, whereas the mean is denoted by the large red cross). Upper and lower whiskers denote the highest and lowest datum within 1.5 IQR of the upper and lower quartile (data points beyond this range are marked by small red crosses). Violin plots (shaded gray area) represent the smoothed distribution of the data.

Identifying Common and Distinct Coding of Prosocial Prediction Errors Using Functional Imaging.

Using concurrently collected fMRI data, we next examined activity corresponding to the magnitude of PEs, time-locked to choice outcome and modeled independently of choice-related activity (Experimental Procedures). To identify common coding of self, prosocial, and no one PEs we first used a stringent conjunction-null analysis (38) (for main effects, see Tables S1 and S2; for categorical analyses of the outcome, see Table S3). Only responses bilaterally in ventral striatum, a region consistently shown to encode rewards delivered to self and others (39, 40), covaried with PEs in all three conditions (MNI coordinates [x = 10, y = 15, z = −9], Z = 4.09, k = 91, P = 0.006 voxel-level small-volume family-wise error corrected (SVC-FWE), and [x = −12, y = 10, z = −11], Z = 3.72, k = 78, P = 0.023 SVC-FWE; Fig. 2 A–C). Responses in each learning condition were significantly greater than 0 (all Z > 3.72, P < 0.05 SVC-FWE; Fig. S3). The ventral striatum therefore signaled PEs regardless of the beneficiary.

Table S1.

Main effects for chosen value and reward prediction error across conditions and individual condition responses to prediction errors

		Peak voxel
Brain region	L/R	x	y	z	k	t	z
Main effect: chosen value
Gyrus rectus	L	−2	45	−14	245	5.88	5.41
Posterior cingulate	R	9	−54	12	82	5.74	5.3
Main effect: PE
Caudate	R	10	15	−9	694	7.97	6.92
Putamen	R	20	9	−11		7.17	6.36
Putamen	L	−10	9	−11	605	7.58	6.65
Middle temporal gyrus	L	−40	−66	24	399	7.1	6.31
Middle frontal gyrus	L	−26	20	52	102	6.11	5.57
Posterior cingulate sulcus	L	−10	−42	37	854	6.09	5.56
Ext. Precuneus	L	−4	−57	22		5.89	5.40
	L	4	−31	39		5.82	5.35
Fusiform gyrus	L	−20	−82	−20	74	5.91	5.42
Amygdala	L	−26	−4	−21	67	5.89	5.40
Long insula gyrus	L	−34	−1	13	42	5.83	5.36
Fusiform gyrus	R	−34	−76	−20	34	5.61	5.18
Superior frontal gyrus	L	−10	62	6	45	5.54	5.12
Middle temporal gyrus	L	−48	−45	−2	10	5.47	5.07
Inferior parietal lobe	L	51	−27	28	22	5.43	5.03
Superior frontal gyrus	L	−16	62	22	24	5.34	4.96
Supramarginal gyrus	L	45	−55	21	35	5.32	4.95
Ext. Middle temporal gyrus	L	50	−64	21		5.25	4.89
Frontal pole	R	12	51	−3	5	5.27	4.91
Middle frontal gyrus	R	24	33	48	1	5.22	4.87
Cerebellar Lobule VIIB	R	18	−72	−50	2	5.19	4.84
Precentral gyrus	R	34	−16	28	1	5.17	4.82
Middle frontal gyrus	L	−21	26	43	2	5.15	4.81
Posterior cingulate	R	20	−24	45	1	5.14	4.80
Fusiform gyrus	L	−50	−45	−21	2	5.11	4.78
Cerebellar Lobule Crus II	R	42	−75	−36	3	5.1	4.76
Middle temporal gyrus	L	−57	2	−17	1	5.09	4.76
Middle temporal gyrus	R	54	−4	−20	1	5.07	4.75
Self PE
Putamen	R	18	6	−8	226	6.7	6.02
Caudate	R	12	12	−6		5.63	5.19
Putamen	L	−12	8	−11	104	5.66	5.22
Putamen	L	−26	3	−9	5	5.24	4.88
Amygdala	L	−27	−6	−17	1	5.08	4.75
Prosocial PE
Caudate	R	8	12	−11	30	5.41	5.02
Subgenual anterior cingulate cortex/basal forebrain	L	−6	15	−8	12	5.32	4.95
No one PE
Precentral gyrus	R	44	−12	51	11	5.27	4.91

For all regions, FWE P < 0.05 voxel-level whole-brain corrected. ext, extending into; k, cluster extent; L, left; PE, prediction error; R, right.

Table S2.

Comparison of reward prediction errors between conditions

		Peak voxel
Brain region	L/R	x	y	z	k	t	z
Prosocial PE > self PE + no one PE
Subgenual anterior cingulate cortex/basal forebrain	L/R	0	6	−15	182	4.05	3.87
	L	−3	18	−6		3.6	3.47
	R	2	12	−9		3.33	3.22
Self PE + no one PE > prosocial PE
Middle frontal gyrus	L	−36	18	43	347	4.75	4.47
	L	−32	23	49		3.81	3.66
	L/R	0	−33	9	212	4.39	4.17
Inferior frontal gyrus	L	−54	39	−3	190	4.27	4.06
Supramarginal gyrus	L	−58	−46	33	124	3.94	3.77
Superior temporal gyrus	L	−58	−28	−2	114	3.94	3.77
Inferior parietal lobule	L	−64	−36	39	136	3.93	3.77
	L	−57	−28	30		3.20	3.11
Middle frontal gyrus	L	−50	39	24	298	3.70	3.56
	L	−38	45	28		3.51	3.39
	L	−32	54	27		3.46	3.34
Superior frontal gyrus	R	42	39	36	184	3.65	3.51
	R	33	51	37		3.57	3.44
Self PE > prosocial PE + no one PE
No suprathreshold voxels
No one PE > self PE + prosocial PE
Parrahippocampul gyrus	L	−26	−55	−8	320	4.6	4.35
		−26	−64	−9		3.88	3.72
Self PE + prosocial PE > no one PE
No suprathreshold voxels
Prosocial PE + no one PE > self PE
No suprathreshold voxels

For all regions, P < 0.001 voxel-level uncorrected, extent threshold k = 100. L, left; R, right.

Table S3.

Categorical analysis of the outcome

		Peak voxel
Brain region	L/R	x	y	z	k	t	z
Prosocial outcome > self + no one outcome
Parahippocampul gyrus	L	−39	−51	−6	132	4.25	4.05
Fusiform gyrus	L	−46	−46	−6		3.42	3.3
Inferior temporal gyrus	L	−38	−60	−5		3.41	3.3
Posterior cingulate	R	15	−43	30	101	4.22	4.02
	R	15	−34	31		3.96	3.79
Inferior parietal lobe	R	45	−49	28	127	4.18	3.98
Midbrain	L	−4	−27	1	618	4.15	3.96
	L	−4	−16	12		3.95	3.78
	R	8	−19	12		3.83	3.68
Middle temporal gyrus	R	51	4	−30	108	4.11	3.92
Self outcome > prosocial + no one outcome
No voxels

Note that parametric analyses incorporating prediction errors allow a more precise estimation of how brain activity fluctuates during learning compared with examining outcome-associated activation alone. Parametric regressors are orthogonal to the events (in this case, outcomes) that they modulate. The relevant contrasts for the categorical analyses of outcome across conditions are presented below. Due to covariance between cues and outcomes in the categorical analysis these results should be interpreted with caution. For all regions, P < 0.001 voxel-level uncorrected, extent threshold k = 100. No regions survived voxel-level whole brain correction. L, left; R, right.

Fig. 2.

fMRI data, common coding of prediction errors. (A and C) Ventral striatum responses to PEs regardless of the agent the outcome was received by. (B) Overlay of ventral striatum response. All images displayed at P < 0.001 uncorrected. Peak voxels all survive P < 0.05 FWE-SVC (see Fig. S3).

Fig. S3.

Statistical parametric maps from the flexible factorial model showing that neural responses to prediction errors at the time of outcome occurred in the ventral striatum in each of the three learning conditions separately. Activation in ventral striatum parametrically varied with reward prediction errors for (A) self, (B) prosocial, and (C) no one conditions: (A) MNI coordinates [x = 16, y = 8, z = −9], Z = 5.8. k = 488, P = 0.001 voxel-level small-volume family-wise error corrected (SVC-FWE), and [x = −12, y = 8, z = −11], Z = 5.22, k = 516, P = 0.001 SVC-FWE. (B) [x = 8, y = 12, z = −11], Z = 5.02, k = 435, P = 0.001 (SVC-FWE), and [x = −9, y = 12, z = −11], Z = 4.66, k = 403, P = 0.001 SVC-FWE. (C) [x = 10, y = 15, z = −11], Z = 4.23, k = 151, P = 0.001 (SVC-FWE), and [x= −12, y = 10, z = −11], Z = 3.72, k = 97, P = 0.001 SVC-FWE. In the prosocial condition, this reward prediction error signal was also present in the subgenual cingulate cortex (medial component of the cluster in B, FWE P < 0.05, small-volume corrected). Images are thresholded at P < 0.001 uncorrected for display purposes and the color bar indicates t values.

We then identified regions that responded to prosocial PEs exclusively by contrasting the prosocial condition against the combined self and no one conditions. The sgACC was the only region to specifically respond to prosocial PEs ([x = −2, y = 4, z = −15], Z = 3.83, k = 148, P = 0.019 SVC-FWE; Fig. 3 A–C), and only parameter estimates for prosocial PEs were greater than 0 (Z = 4.95, P < 0.001, SVC-FWE). The sgACC therefore uniquely signaled PEs when learning to benefit another.

Fig. 3.

fMRI data, distinct coding of prosocial prediction errors. (A) Subgenual anterior cingulate cortex (sgACC) responses to PEs when learning to benefit another person. (B) Overlay of sgACC response (green) and VS response (yellow). Image displayed at P < 0.001 uncorrected, and all peak voxels survive P < 0.05 FWE-SVC. Crucially, the sgACC region in which activity covaried with prosocial PEs did not overlap with the ventral striatum clusters. (C) sgACC response was modulated by individual differences in empathy (online simulation).

We also tested for regions that showed greater responses to self/no one than prosocial PEs. Both left ([x = −36, y = 18, z = 43], Z = 4.47, k = 62, P = 0.006 SVC-FWE) and right ([x = 32, y = 15, z = 39], Z = 4.36, k = 27, P < 0.020 SVC-FWE) DLPFC showed this pattern. We did not observe significant responses in the dACC for any contrast. All significant results remained when comparing the self condition only to the prosocial condition (see Table S4).

Table S4.

Comparison of reward prediction errors between conditions without the control (no one) condition

		Peak voxel
Brain region	L/R	x	y	z	k	t	z
Prosocial PE > self PE
Subgenual anterior cingulate cortex/basal forebrain	R	2	6	−14	190	4.08	3.90
Self PE > prosocial PE
Thalamus	—	0	−33	9	209	4.83	4.54
Middle frontal gyrus	L	−36	18	43	349	4.75	4.47
Cerebellum	R	16	−79	−32	103	3.68	3.54
Conjunction
Caudate	R	12	14	−9	292	5.02	4.70
Middle temporal gyrus	L	−39	−66	24	222	4.71	4.44
Caudate	L	−9	11	−8	218	4.68	4.42

For all regions, P < 0.001 voxel-level uncorrected, extent threshold k = 100.

Mechanistic Links Between Empathy and Prosocial Learning.

Next we tested whether individuals higher in empathy learned at a similar rate to obtain rewards for others compared with themselves and whether variability in empathy modulated neural responses to prosocial PEs. Consistent with our predictions, we found that the online simulation subscale of the Questionnaire of Cognitive and Affective Empathy (41), a validated and psychometrically rigorous measure of trait empathy that probes the tendency to imagine how other people will feel (SI Experimental Procedures), was positively associated with the learning rate for the prosocial condition, relative to the self (to control for individual differences in learning per se) condition (r = 0.44, P = 0.01, 95% CI = 0.18, 0.66; Fig. 1D and Table S5). We then tested whether the online simulation subscale also predicted neural responses to prosocial PEs. Online simulation was also positively associated with prosocial (compared with self) PE responses in the sgACC (r = 0.39, P = 0.03, 95% CI = 0.13, 0.60; Fig. 3D), with those higher in online simulation showing greater sgACC specificity to prosocial relative to self PEs. Together these findings suggest that there are both behavioral and neural links between empathy and prosocial behavior.

Table S5.

Correlations between the prosocial–self learning rate difference score and the subscales of the Questionnaire of Cognitive and Affective Empathy

Description	Online simulation	Perspective taking	Emotion contagion	Peripheral responsivity	Proximal responsivity
Mean (SD)	27.45 (4.59)	30.55 (4.33)	10.74 (3.00)	10.39 (2.71)	11.19 (2.71)
Reliability (α)	0.81	0.84	0.82	0.70	0.77
Correlation with prosocial–self LR difference	r = 0.44	r = 0.19	r = −0.13	r = −0.07	r = 0.18
	P = 0.01	P = 0.29	P = 0.47	P = 0.70	P = 0.32

LR, learning rate.

SI Experimental Procedures

Trait Empathy Measure.

Participants completed a measure of empathy, the Questionnaire of Cognitive and Affective Empathy (QCAE), a standardized and validated measure of trait empathy with good construct validity and internal consistency (41). The QCAE is a multidimensional questionnaire devised to measure five key components of empathy. In the development of the QCAE, two raters selected items from other well-validated and commonly used empathy measures [e.g., Hogan Empathy Scale (HES (78)), Interpersonal Reactivity Index (IRI (79)), Balanced Emotional Empathy Scale (BEES (80)), and Empathy Quotient (EQ (81))] if they were deemed to measure empathy. Items from these scales deemed to measure other processes (e.g., sympathy) were not included. The five subscales composing the QCAE are online simulation (e.g., “I find it easy to put myself in somebody else’s shoes”), perspective-taking (e.g., “I can easily tell if someone else wants to enter a conversation”), emotion contagion (e.g., “I am happy when I am with a cheerful group and sad when the others are glum”), peripheral responsivity (e.g., “I often get deeply involved with the feelings of a character in a film, play, or novel”), and proximal responsivity (e.g., “I often get emotionally involved with my friends’ problems”). Items are rated on a four-point scale from “strongly disagree” to “strongly agree.”

Participant Instructions.

“In this experiment you will see a pair of symbols on each trial and you need to select one of them.

“You will receive points for some of your choices, which will be converted into money at the end of the experiment, so the more points you get the more extra money you will earn. The two symbols are not the same in terms of how often they give you points: some of the symbols will give you points more often than others. Each symbol has its own meaning, regardless of where it appears on the screen (so left and right are not important) or when it occurs in the task.

“You will play the task in three conditions, for yourself, for the other participant, and for no one. When you are playing for yourself, you will receive any money you win. When you are playing for the other participant they will receive any money you win for them. However, the other participant won’t know how much money you earn from them until they leave the experiment (you will leave the scanning center at different times, and the money you earn for the other participants will be given to them in a sealed envelope), and they do not know that you are performing a task where you could win extra money for them. When you play for no one, neither of you will receive any of the money.

“Respond using the left and right arrow keys to make your selection. Try to respond as quickly as possible, you have about 2 seconds to make your choice.”

Trial Structure.

Each block began with an instruction screen that indicated who would receive the outcomes (self, confederate, or no one) for 2,000 ms (Fig. 1A). This was followed by the presentation of two abstract stimuli for 3,000 ms during which participants were required to select one of these. These stimuli were letters from the Agathodaimon font. If no response was indicated during this time, the word “missed” appeared in red on the screen. The selected option was shown for 300 ms, followed by a delay (2,500 ms), then by the outcome of their choice (win 100 points/win 0 points). A randomly jittered fixation (2,000–4,000 ms) was shown after the outcome before the two symbols were presented again. The side of the screen on which the two symbols were presented was counterbalanced so that participants could not perform action-based learning. They were also instructed that the side of the stimulus did not matter, to encourage subjects not to learn based upon the action they had performed but to learn the contingencies between the stimuli regardless of position and outcome.

There were 144 trials in total, 48 self, 48 prosocial, and 48 no one, presented in 3 blocks of 16 trials. Each block began with a new pair of symbols to learn. Blocks were presented in pseudorandom order, with the same block type never presented twice in a row.

Debriefing Questions.

• 1.Are you surprised at what you have just read?
• 2.Did you believe that the other person outside of the scanner was another participant who would be receiving the money you saw?
• 3.We would be grateful if you did not inform anyone about the nature of the deception in the experiment.

Model Fitting.

To fit the RL model to participants’ behavior, we used the maximum a posteriori (MAP) approach (82). This method provides a better estimation than maximum likelihood estimation (MLE) alone of each individual’s true learning rate and temperature parameter and is less susceptible to the influence of outliers (82). MAP is a two-stage procedure which begins by estimating the model parameters (α and β) for each subject using MLE and then reestimating the model parameters, this time applying priors on the possible range of model parameters. The priors used during the MAP procedure are defined using a Gaussian kernel with mean and SD defined by the summary statistics of the distribution of the α and β parameters estimated during the MLE stage (82).

Model Comparison.

We compared the RL model to a null model in which it is assumed that participants exhibit no learning and choose options at random. In this model the value for α was set to 0 (equivalent to no learning), and the value for β was varied between 0 and ∞. This null model was compared with the RL model using Bayesian information criterion scores (BIC) to examine whether participants’ behavior was more parsimoniously explained by our full RL model. The BIC compares the parsimony of the two models based on the number of parameters and the likelihood of the model fits: more parsimonious models with similar fits but fewer parameters are favored. The full RL model had much better evidence at the group level [ΔBIC > 10, indicating decisive evidence (83)] than the null model (Fig. S2).

We additionally tested a less complex model in which data were fit using the same α and β parameters for each of the self, other, and no one conditions. Again, the full RL model accounted most parsimoniously for the participants’ behavior at the group level (ΔBIC > 10; Fig. S2).

In summary, the results of our modeling suggest that participants learned successfully during the task and that a model that represents different RL learning in each condition explains the data better than either a null model (which effectively represents random choices) or a model with fixed parameters across self, prosocial, and no one conditions.

fMRI Analysis and ROI Selection.

We selected the VS because this region has been consistently implicated in signaling PEs (3). The VS mask was the default mask in the Harvard–Oxford Atlas and was created by segmenting the striatum into caudate, putamen, and ventral striatum according to anatomical structure. The sgACC has previously been shown to signal prosocial decisions and PEs (13, 16, 45, 46, 52). This region was defined as Brodmann areas (BA) 25 and s24 because these were the regions within which coordinates in several previous studies of social decision-making and PEs were identified (13, 16, 45, 46, 52). Because the sgACC region is adjacent to the VS, there was a small amount of overlap in voxels between these masks, but the significant activations for the conjunction of all PEs and the contrast for prosocial PEs were nonoverlapping (Fig. 3B). Thus, if a peak coordinate fell in the VS for the sgACC mask, it was not interpreted. The dACC mask corresponded to BA 24a/b, dorsal to the corpus callosum, and corresponding to the gyral portion of the anterior cingulate cortex. This region was selected given previous evidence of response of this region to PEs for others (12, 13, 15, 16). The DLPFC mask covered much of the anterior middle frontal gyrus and corresponded to what has been referred to in previous studies as BA 9/46 (74). This region has also been consistently implicated in social decision-making and PE signaling (5, 7, 84).

To test for regions that responded to PEs regardless of learning condition, a conjunction-null analysis (38, 73) (logical AND) was performed (contrast: self PE ∩ prosocial PE ∩ no one PE) within the SPM flexible factorial framework. To test for regions that responded greater to prosocial PEs than to self and no one PEs, we contrasted the prosocial condition against the combined self and no one PEs (contrast: −0.5 1 –0.5). Finally, to identify regions that responded greater to self and no one PEs than prosocial PEs, we contrasted the combined self and no one PE conditions against the prosocial condition (contrast: 0.5 –1 0.5). Note that parametric analyses incorporating prediction errors allow a more precise estimation of how brain activity fluctuates during learning compared with examining outcome-associated activation alone. Note that parametric regressors are orthogonal to the events (in this case, outcomes) that they modulate; the relevant contrasts for the categorical analyses of outcome across conditions are presented in Table S4.

Discussion

RLT has provided important insights into how we learn about rewarding information for ourselves. Here we used the RLT framework to characterize prosocial learning and its underlying computational and neural basis. We found that the ventral striatum commonly coded PEs in all conditions, responding to PEs for self, another person, and no one. In contrast, the sgACC coded PE signals specifically when learning to benefit another. We also observed substantial variability in prosocial learning with differences in both neural and behavioral responses predicted by trait empathy.

Our findings advance theoretical accounts of the neural basis of social behavior by finding evidence for both common coding and socially specific regions of the brain underpinning prosocial behavior (4–7, 42). PE signals in the ventral striatum, which is extensively connected with the sgACC (9), were evident regardless of the context of learning. This finding cannot be easily accommodated within current theories of ventral striatum contributions to learning, which suggest that this region is engaged when learning to obtain beneficial outcomes for oneself (3). Moreover, this finding adds to and extends existing studies of the role of the ventral striatum in social behavior. Consistent with previous research, we find ventral striatum responses to rewards delivered to both self and other (39, 40). However, we also find that these signals are evident even when no one receives a rewarding outcome. The ventral striatum may therefore be important for learning in many contexts even when a reward is not obtained or consumed by anyone. The profile of response in the ventral striatum (VS) differs from the sgACC, which shows specificity for signaling PEs when learning to benefit another. Thus, although both regions may play a role in driving prosocial behavior, the function of ventral striatum may be more domain general than that of sgACC, which we speculate may compute a prediction error specifically for outcomes delivered to others.

The sgACC region we identified as responding exclusively to prosocial PEs overlapped with a septal-anterior hypothalamic area that is part of the basal forebrain (43). Recent studies of sgACC function have found signals in this region relevant for social cognition and behavior including credit assignment (44), prosocial and moral behavior (6, 16, 45, 46), the experience of positive affect (47), trust (48), social emotion (49), and vicarious reward (50, 51). Moreover, there is evidence that the sgACC may signal PEs for self-reward but only when learning occurs at a specific level of abstraction beyond basic stimulus–response association (52). One possible explanation for this convergence is that similar abstract learning mechanisms may drive how we learn to benefit others and that both are underpinned by RLT principles. Further studies could compare tasks that manipulate both the social and the hierarchical context of a learning environment to directly test the parallels between these types of learning. In addition, future research should aim to dissociate the functions of the heterogeneous cortical and subcortical structures of the sgACC complex, using high-resolution fMRI, to understand which of these subregions contribute to social cognition and prosocial learning.

We did not observe responses in anterior insula or dACC in any of our contrasts. This may seem surprising given that these regions have previously been implicated in empathy and/or social behavior (15, 29, 30, 37). Although it is difficult to interpret a null finding, because there can be a number of reasons that a particular neural response is not observed, we note that in this experiment, participants were performing their task in a different reference frame (see refs. 53 and 54 for recent discussions of the role of references frames in studies of social cognition) compared with other studies measuring empathic/vicarious processing. In our task, participants were making choices for another person, not only observing events that happened to others. In other words, in our task, decisions were made in a self-action reference frame. Studies that compare different reference frames in the same paradigm could help shed light on the functional roles of specific brain areas during empathy and prosocial behavior.

An important aim of our study was to explain what might drive variability in prosocial learning (1, 20–22). We identify evidence of a mechanism linking variability in empathy to variability in prosocial behavior, with highly empathic individuals having an increased learning rate, and stronger sgACC PE signals, for other people’s rewards. Many accounts of empathy have argued for a crucial role of empathy in the development of prosocial and moral behavior and the inhibition of aggression (25, 27, 55–58). Our demonstration that empathy is associated with a higher rate of learning about actions that result in beneficial outcomes for other people as well as the neural drivers of prosocial learning suggests a computational link by which empathy could influence the development of prosocial and moral behaviors. Enhanced PE signaling and faster learning when benefitting others also provide a potential explanation for reports of individuals higher in empathy being more motivated to behave in a prosocial manner compared with those lower in empathy (21). Moreover, our results support an emerging view that PE signals may be crucial for learning how to interact, and empathize, with others (59). Twin data indicate substantial heritability of prosociality across development (60, 61). Longitudinal developmental investigations, particularly ones that are able to tease apart genetic and environmental contributions to brain function (e.g., twin studies), would be helpful in determining the degree to which the specificity of sgACC activity during prosocial learning reflects an endophenotype for prosocial behavior.

An influential theory within the literature on empathy is that we empathize with others by a process of (embodied) simulation (62). This view is largely driven by studies that show a degree of overlap in the neural responses to self and other pain, particularly in anterior insula and dACC (reviewed in refs. 29 and 30), and for pleasant touch (63). Other studies have also supported a simulationist view of empathy by exploiting the placebo analgesia effect, showing that placebo analgesia changes self pain as well as vicarious pain (64, 65). Although these studies are consistent with a simulationist account, it remains possible that there exist additional processes that do not operate through self–other overlap that participate in the experience of empathy (15, 30). For example, studies have suggested that other neurocognitive processes, in addition to simulation, may be important when processing vicarious information, particularly in the domain of positive affect (see ref. 51 for a metaanalysis and ref. 30 for review). In the present study we observed that PE responses in the sgACC were present only when learning to benefit another person and not when learning to benefit oneself or no one (including the latter to control for fictive PE signals), suggesting that a socially specific signal is important for prosocial learning in this context. We also observed that those who self-report that they simulate most readily also have the most specific signals related to prosocial learning in the sgACC. This points to the intriguing possibility that simulation at the level of self-report may not necessarily be encoded in brain areas that respond to both self and other during learning. It should be noted that given potential gender differences in empathy and prosocial behavior (e.g., ref. 66) our sample in this study was composed only of males. Future studies would benefit from also examining prosocial learning in females.

Prosocial behaviors are fundamental for promoting social bonds and cohesion (1, 55, 67) and are disrupted in a number of psychiatric and neurological disorders (23, 68–70). Using the framework of RLT to understand how we learn to make decisions that benefit other people could offer insights into why these disorders are associated with atypical prosocial behavior and empathy. Taken together, our findings reveal a computational link between prosocial learning and empathy in humans and therefore pave the way to characterize atypical prosocial interactions in those with disorders of social cognition and behavior.

Experimental Procedures

Participants.

Thirty-four right-handed healthy males (age 19–32, M = 22.7, SD = 3.0) were recruited through university participant databases. Exclusion criteria included previous or current neurological or psychiatric disorder, nonnormal or noncorrected to normal vision, nonnative English language, and previous or current study of psychology. Three participants were excluded from the analysis [two due to performance at chance level (∼50%) in all learning conditions and one due to neurological abnormalities evident on the MRI scan], leaving a final sample of 31. With 31 subjects we had 80% power to detect a medium effect size of d = 0.52 at alpha = 0.05 (two-tailed), an effect size smaller than typically reported in this field, indicating sufficient power. All participants gave written informed consent, and the study was approved by the University College London Research Ethics Committee.

Experimental Task.

We examined BOLD signals that scaled parametrically with the size of a PE at the time of an outcome delivered to self, another person (here a confederate–prosocial condition), or no one. Participants performed a probabilistic reinforcement learning task where they were required to learn the probability that each of two symbols would be rewarded. One symbol of each pair was associated with a high probability (75%) and one was associated with a low probability (25%) of reward. Participants performed this task in three different learning contexts: self, prosocial, and no one. Participants were instructed that when they were playing for themselves they would receive any money they won. Crucially, when they were playing for the confederate, that participant would receive the money (see SI Experimental Procedures for full instructions given to participants). When they were playing for no one, the points they saw would not be converted into any additional payment, either for themselves or the other participant. Participants were informed that the other participant was not aware that they were performing a task where they could earn extra money and that any money they won would be given to the other participant anonymously (i.e., it would be placed in a sealed envelope and the two participants would leave the scanning center at different times).

Self blocks began with the instruction “play for you” and had the word “you” written above all choice symbols and outcomes. Prosocial blocks had the name of the confederate participant written above them. No one blocks had the words “no one” written above elements in a trial. This ensured that participants were explicitly aware whether the decisions they made resulted in outcomes for themselves, for the other participant, or for no one (for trial structure and order, see SI Experimental Procedures and Fig. 1A). Participants practiced one block (16 trials) of the task in a separate session ∼7 d before the scanning session to familiarize them with the experimental task. During this practice they were instructed that the outcomes would not be converted into any payment.

Procedure.

Participants were paired with one of two age- and gender-matched confederates whom they believed were naïve participants and had never met before the experiment. The confederates were trained in acting as naïve participants during a pilot experiment. Participants attended two sessions. The first session was attended only by the experimental participant and involved practicing the experimental task and completing questionnaires. This was done due to scheduling considerations and so that participants could practice the learning task on their own without the confederate present. The second session (<7 d later) was attended by both the experimental participant and the confederate. The participant and confederate were taken together to the MRI center and filled in consent forms together in the same room. The confederate was then led into a behavioral testing room and instructed to complete questionnaires, with the experimental participant within earshot of this interaction to increase belief in the deception. The experimental participant was taken to the scanning room and reminded of the instructions for the task, with the confederate participant unable to overhear this interaction to ensure that the experimental participants’ choices remained anonymous. Participants were told that they would view a pair of symbols on each trial and that they should select one of them. They would receive points for some of their choices that would be converted into money at the end of the experiment, such that the more points they received, the more extra money they would earn. They were instructed that the two symbols would not be the same in terms of how often they gave points, and with some symbols they were more likely to win points than with other symbols. Whether the symbols appeared on the left or right did not affect their meaning.

Participants were instructed that they would receive extra payment based on the outcomes they received during the experimental task, but in fact, all participants were paid the same amount due to ethical restrictions (total £30, representing an additional £7 to the standard participant payment for the required time commitment). They also believed that the confederate participant could earn an extra payment based on the choices the experimental participant made during the task. A set of standardized questions completed after the scan confirmed that no participant had become suspicious about the deception during the experiment (see SI Experimental Procedures).

Computational Modeling of Behavioral Data.

Learning behavior in the self, prosocial, and no one conditions was modeled using a reinforcement learning (RL) algorithm (2), which has been extensively used to examine the behavioral and neural basis of arbitrary visuomotor associations in both self and social contexts (13, 17–19). The RL model assumes that the associative value of an action (or stimulus) changes when new information reveals that the actual outcome of a decision is different from the expected outcome (2). Thus, on each trial t, an action a has an expected associative value Q_t(a) that is updated by the mismatch between experienced and expected outcome on the current trial (the PE). At their most simple, RL algorithms state that expectations of future reward for action a, Q_t+1(a), should be a function of current expectations Q_t(a) and the discrepancy between the actual reward that has just been experienced on this trial r_t (coded as 1 or 0 for reward or no reward, respectively) and the expected reward for this trial t, Q_t(a). The degree to which this discrepancy updates the expectation is scaled by the learning rate α (bounded between 0 and 1), such that

$$Q_{t+1}\left(a\right)=Q_t\left(a\right)+\alpha \times \underset{\text{Prediction}\hspace{0.17em}\hspace{0.17em}\text{Error}}{\underbrace{\left[r_t-Q_t\left(a\right)\right]}}.$$

The learning rate α controls the extent to which the current expected value is updated by new information. Consequently, a low learning rate will minimize the influence of the prediction error and the amount that the value is updated. The probability that a subject chooses action a on trial t, given the expected values of the available actions Q_t(a), is given by the softmax link function

$$p_t\left[a|Q_t\left(a\right)\right]=\frac{e^{\left[Q_t\left(a\right)/\beta \right]}}{{{\displaystyle \sum }}_{a\prime }{e}^{\left[Q_t\left(a\prime \right)/\beta \right]}}.$$

The temperature parameter β controls the amount of exploration or noisiness for that participant (i.e., extent to which the subject decides to choose the most rewarding option vs. exploring potentially more rewarding actions). The softmax link function estimates the trial-by-trial probability of each action by weighting the ratio of expected values by the temperature parameter. In this framework, a high temperature parameter β would lead to similar action probabilities irrespective of the expected value of each action (resulting in random behavior). A low β would lead to consistent behavior, where the action with the higher expected value is invariably selected on each trial. In the full model, separate α and β parameters were estimated for each of the self, prosocial, and no one conditions because this provided the most parsimonious explanation for the behavioral data (see SI Experimental Procedures for details of model fitting and model comparison and Fig. S2).

Statistical Analysis of Behavioral Data.

Analyses of behavioral data were performed in SPSS 22 (IBM Corp.). We examined differences between conditions in the learning rate and temperature parameters at the group level using separate repeated measures analyses of variance (ANOVAs), with three levels (self, prosocial, and no one). Separate ANOVAs were conducted because the learning rate and temperature parameters represent different units of measurement. We examined bivariate associations between the prosocial–self difference in learning rate and temperature and empathy questionnaire subscales using the Pearson correlation coefficient. Effect sizes (Cohen’s d) were calculated by dividing the mean difference between conditions by the SD of the difference (71). Confidence intervals for correlation coefficients were estimated using SPSS 22 Bootstrap procedure.

fMRI Acquisition and Analysis.

A Siemens Avanto 1.5-T MRI scanner was used to acquire a 5.5-min three-dimensional T1-weighted structural scan and 424 multislice T2*-weighted echo planar volumes with blood oxygenation-level–dependent (BOLD) contrast. The structural scan was acquired using a magnetization prepared rapid gradient echo (MPRAGE) sequence with 176 slices, slice thickness = 1 mm, gap between slices = 0.5 mm, TR = 2730 ms, TE = 3.57 ms, field of view = 256 mm × 256 mm², matrix size = 256 × 256, and voxel size = 1 × 1 × 1 mm resolution. The functional imaging sequence was acquired in an ascending manner, at an oblique angle (∼30°) to the AC–PC line to decrease the effect of susceptibility artifact in the orbitofrontal cortex (72), and had the following acquisition parameters: 424 volumes, 1 mm gap, echo time = 50 ms, repetition time = 2,975 ms, flip angle = 90°, field of view = 192 mm, and matrix size = 64 × 64.

Imaging data were analyzed using SPM8 (www.fil.ion.ucl.ac.uk/spm). Data preprocessing followed a standard sequence: the first four volumes and last volume were discarded. Images were then realigned and coregistered to the participant’s own anatomical image. The anatomical image was processed using a unified segmentation procedure combining segmentation, bias correction, and spatial normalization to the MNI template using the new segment procedure (73); the same normalization parameters were then used to normalize the EPI images. The voxel size was resampled to 1.5 × 1.5 × 1.5mm. Last, a Gaussian kernel of 8 mm FWHM was applied to spatially smooth the images. Before the study, example first-level design matrices were checked to ensure that estimable GLMs could be performed with independence between the parametric regressors (chosen value and PE in the three conditions), with correlations coefficients of r < 0.25. This allowed us to look at PE-related responses independent of chosen value.

Eight event types were used to construct regressors in which event timings were convolved with SPM’s canonical hemodynamic response function. The three conditions at the time of the cues and three conditions at the time of the outcome were modeled as separate regressors using stick functions. Each of these regressors was associated with a parametric modulator taken from the computational model. At the time of the cue this was the chosen value, and at the time of the outcome, the PE. The PEs were estimated using each subject’s own alpha and beta from each condition. The instruction cue at the beginning of each block was also modeled in a single regressor as a stick function. In some participants, an eighth regressor modeled all missed trials, on which participants did not select one of the two symbols in the response window. For those participants where there was visible head motion in a particular scan (scans with >1 mm or 1° movement relative to the next were examined visually) an extra regressor was included corresponding to each scan. These images were removed and replaced with an image created by interpolating the two adjacent images to prevent distortion of the between-subjects mask (four participants, less than 1% of total time series). Six head motion parameters modeled the residual effects of head motion as covariates of no interest. Data were high-pass filtered at 128 s to remove low-frequency drifts, and the statistical model included an AR (1) autoregressive function to account for autocorrelations intrinsic to the fMRI time series. Our primary analysis focused on the PEs at outcome (for response to chosen value, see Table S1).

Contrast images from the first level were input into a second-level flexible-factorial design with three levels (self PE, prosocial PE, and no one PE). Main effects are reported at P < 0.05, family-wise error (FWE) corrected at the voxel level across the whole brain, or P < 0.05 small volume corrected (SVC) at the voxel level in regions where we had a strong a priori hypothesis (see below).

ROI Selection and fMRI Contrasts.

The a priori regions of interest (ROIs) were defined anatomically using masks taken from an appropriate atlas [bilateral VS, sgACC, dACC, bilateral DLPFC; toolboxes: Harvard-Oxford Atlas, regions 46v and 9 (74), Anatomy Toolbox regions s24 and 25 (75), and region 24 from ref. 76 (corresponding to the gyral portion of the anterior cingulate cortex); see Fig. S4 and SI Experimental Procedures for further details of ROI selection). We additionally applied a false discovery rate correction (FDR) (77), rather than Bonferroni correction [which may be overly conservative given that our ROIs were not entirely independent from one another, because they are functionally and anatomically connected (9–11)], for the number of ROIs. All ROI comparisons remained significant (P < 0.05) when controlling for the number of comparisons using FDR.

Fig. S4.

Anatomically defined regions of interest. (A) Bilateral ventral striatum mask as defined by the Harvard–Oxford Atlas. (B) Subgenual ACC mask (regions 25 and 24s) and dorsal ACC mask (region 24) (75, 76). (C) Dorsolateral prefrontal cortex mask (regions 9/46d) (74).

SI Results

One previous study suggested that mentalizing-related activation in dorso-medial PFC (DMPFC) is related to prosocial behavior (85). Therefore, in an exploratory analysis requested by one of the anonymous reviewers we also examined whether there were significant responses in this region. An anatomical mask was created comprising areas 9 and 10 from Sallet et al. (74). We found responses to the contrast self PE + no one PE > prosocial PE in left ([x = −6, y = 32, z = 46], Z = 3.45, k = 10, P < 0.001 uncorrected) and right ([x = 16, y = 64, z = 16], Z = 3.21, k = 11, P = 0.001 uncorrected) DMPFC, but these did not survive correction for multiple comparisons.