The Modular Neuroarchitecture of Social Judgments on Faces

Abstract

Face-derived information on trustworthiness and attractiveness crucially influences social interaction. It is, however, unclear to what degree the functional neuroanatomy of these complex social judgments on faces reflects genuine social versus basic emotional and cognitive processing. To disentangle social from nonsocial contributions, we assessed commonalities and differences between the functional networks activated by judging social (trustworthiness, attractiveness), emotional (happiness), and cognitive (age) facial traits. Relative to happiness and age evaluations, both trustworthiness and attractiveness judgments selectively activated the dorsomedial prefrontal cortex and inferior frontal gyrus, forming a core social cognition network. Moreover, they also elicited a higher amygdalar response than even the emotional control condition. Both social judgments differed, however, in their top-down modulation of face-sensitive regions: trustworthiness judgments recruited the posterior superior temporal sulcus, whereas attractiveness judgments recruited the fusiform gyrus. Social and emotional judgments converged and, therefore, likely interact in the ventromedial prefrontal cortex. Social and age judgments, on the other hand, commonly engaged the anterior insula, inferior parietal cortex, and dorsolateral prefrontal cortex, which appear to subserve more cognitive aspects in social evaluation. These findings demonstrate the modularity of social judgments on human faces by separating the neural correlates of social, face-specific, emotional, and cognitive processing facets.

Introduction

For humans, faces probably convey the most precious environmental information. Accordingly, newborns develop an early predilection for attending faces rather than objects (Morton and Johnson 1991). Throughout life, the impact of social judgments of faces is pervasive in everyday social interaction. Trustworthiness and attractiveness in a face convey particularly pivotal social information. Assessing facial trustworthiness is decisive because trusting an untrustworthy individual could have severe negative consequences, whereas not trusting a trustworthy one may constitute a missed opportunity for cooperation (Cosmides and Tooby 1992, 2000). In fact, group-averaged ratings of facial trustworthiness better predict neural activation than do individual ratings (Engell et al. 2007). Trustworthiness evaluation might therefore reflect a conserved adaption rather than a mere result of enculturation. On the other hand, assessing attractiveness is relevant for estimating the reproductive fitness of a potential mating partner. Analogous to trustworthiness judgments, attractiveness judgments show more similarities than dissimilarities across different cultures, age groups, and sexual orientations (Langlois et al. 2000). Interestingly, both these social judgments of faces are influenced by similarity to self. That is, subjects rated new faces more trustworthy with increasing portions of the self-face (DeBruine 2005) and were faster at selecting self-morphs with increasing attractiveness of the other-face (Epley and Whitchurch 2008).

This efficiency of faces to transport socially relevant information, including emotions, has long been known from psychological investigation (Izard 1971; Ekman 1982). In contrast to the yet sparse literature on the neural correlates of social judgments, neuroimaging research on emotion recognition has yielded an extensive body of literature (Fusar-Poli et al. 2009). The neurobiology of both social and emotional judgments have largely been elucidated by comparison to age judgments, assumed to recruit neither social nor emotional but rather purely cognitive brain networks. Importantly, however, social appraisal may be argued to have a substantial cognitive component as well (Cunningham et al. 2004). Moreover, social traits derived from faces and emotional states in these may share an insufficiently understood common valence (Oosterhof and Todorov 2008). Consequently, this functional magnetic resonance imaging (fMRI) study set out to delineate the functional neuroanatomy of social, emotional, and cognitive facial judgments and characterize these appraisal processes that crucially govern human behavior.

The goal of the present fMRI study was to assess the neural correlates of 2 different social judgments of visually presented faces as well as examine the commonalities and differences between explicit social, emotional, and cognitive judgments of these stimuli. We addressed these goals by examining the social target conditions (trustworthiness [TR] and attractiveness [AT] judgments) in comparison to an emotional (happiness [HA] judgment) and a cognitive (age [AG] judgment) control condition. In this study, 44 healthy adults were scanned while prompted to overtly choose the more trustworthy, more attractive, happier, or older face from simultaneously presented pairs of faces (Fig. 1). By using identical stimulus material throughout all 4 conditions, we aimed at isolating top-down effects solely depending on the type of judgment subjects had to make.

Figure 1.

Timeline of screens within one block of the fMRI experiment. During the actual trials participants were required to compare the face pair based on 4 different questions: “(Who do you regard as) More trustworthy?,” “More attractive?,” “Happier?,” and “Older?.” The face pair was presented for 3 s, with an additional double arrow between the 2 faces during the last second of presentation to prompt a response if not already given (ISI, interstimulus interval).

Material and Methods

Participants

Forty-four right-handed adults (20 females, mean age = 32.8 ± 11.5 years, range = 21–60) were recruited via advertisements and paid for their participation in the study. Inclusion was based on the absence of any neurological or psychiatric disorder or contraindications for fMRI investigation. All participants had normal or corrected-to-normal vision. Written informed consent was obtained before entering the study, which had been approved by the local ethics committee of the RWTH Aachen University Hospital.

Stimuli

We selected 60 prerated color pictures from the FEBA facial stimuli set with either slightly happy or sad faces of 30 actresses and 30 actors (Gur, Sara, et al. 2002). The faces had forward head and gaze position. Hair, but no accessories (jewellery, clothing, glasses) were depicted in this material (Fig. 1). The experiment was run using the PRESENTATION 14.3 software package (Neurobehavioral Systems Inc., San Francisco, CA). All stimuli were projected through the bore of the magnet onto an angled mirror attached to the radio frequency coil above the participants' eyes.

Experimental Paradigm

In the experimental design, every block was preceded by a question frame displayed for 5 s, which indicated the attribute participants had to judge in the upcoming set of stimulus pairs based on the displayed faces. Participants were required to compare the stimuli based on 4 different questions: “(Who do you regard as) More trustworthy?,” “More attractive?,” “Happier?,” and “Older?.” This question frame was presented again before each individual stimulus pair for 1 s. In each trial, the face pair was presented for 3 s, with an additional double arrow between the 2 faces during the last second of presentation to prompt a response if not already given. The choice (left or right) was indicated by pressing the (left or right) button. Participants were told to use their “gut instinct” to answer intuitively and quickly. The task was presented in a blocked fashion (sequences of 6 trials followed by a short resting baseline), and the interval between trials within each block was randomly jittered, varying from 2 to 5 s. Thus, while we used a blocked presentation for different types of judgment, we modeled each individual trial separately in an event-related manner. Clustering trials in blocks reduced the potential impact of task-switching and sequence effects, and the event-related analysis increased specificity by enabling the exclusion of instruction frames and pauses between trials as well as a parametric analysis of judgment time on a trial-by-trial basis.

One block comprised 6 trials and lasted 1 min. Eight such blocks of each condition were presented consecutively, amounting to 48 trials per condition. In sum, 192 trials were presented in 32 blocks. Randomized blocks of the 4 conditions were therefore counterbalanced in every subject. The entire experimental session lasted 32 min. For each participant, 48 stimuli were preselected randomly from the pool of 30 slightly happy faces and another 48 stimuli from the pool of 30 slightly sad faces. Thus, some stimuli were selected twice, but no stimulus was selected more often than that. Individual face pairs of 1 specific condition consisted of 1 item from either of those 2 stimulus sets. Thus, all tasks were performed on the same facial stimuli, yet, stimulus order was randomized in every task. The position of the 2 faces on the screen (left/right) was completely randomized as well.

Behavioral Data Analysis

The behavioral measurements taken during the fMRI experiment were analyzed off-line using MATLAB (MathWorks, Natick, MA) and SPSS (SPSS Inc.). We wanted to test for correlations between the selection preferences of the face stimuli depending on the facial judgment being made. Therefore, we counted the selections of each stimulus per condition across participants. The absolute selection counts were ranked condition-wise and, finally, the Spearman correlation was calculated among these condition rankings.

fMRI Image Acquisition

Imaging data were acquired on a 3-T Siemens MRI whole-body system (Siemens Medical Solutions, Erlangen, Germany) with the vendor-supplied 12-channel phased-array head coil. The blood oxygenation level–dependent (BOLD) signal was measured using a 2D-echo-planar imaging (EPI) sequence with the following parameters: gradient-echo EPI pulse, echo time = 30 ms; repetition time = 2200 ms; field of view = 192 × 192 mm², 3 × 3 mm² within-slice pixel size; flip angle = 90°. Whole-brain coverage was achieved with 36 axial scans with 3.1 mm slice thickness (distance factor = 15%). In the scanning session, 888 volumes were acquired. The initial 4 of these images were dummy scans to allow for longitudinal equilibrium and were discarded before further analysis.

fMRI Image Processing

Using SPM5 (Welcome Department of Cognitive Neurology, London, UK; http://www.fil.ion.ucl.ac.uk/spm), the EPI images were corrected for head movement by affine registration using a 2-pass procedure, by which images were initially realigned to the first image and subsequently to the mean of the realigned images. After realignment, the mean EPI image for each subject was spatially normalized to the Montreal Neurological Institute (MNI) single-subject template (Holmes et al. 1998) using the “unified segmentation” approach (Ashburner and Friston 2004). The resulting parameters of a discrete cosine transform, which define the deformation field necessary to move the participant data into the space of the MNI tissue probability maps, were then combined with the deformation field transforming between the latter and the MNI single-subject template. The ensuing deformation was subsequently applied to the individual EPI volumes that were hereby transformed into the MNI single-subject space and resampled at 1.5-mm isotropic voxel size. The normalized images were spatially smoothed using an 8-mm full-width at half-maximum Gaussian kernel to meet the statistical requirements for corrected inference on the general linear model and to compensate for residual macroanatomical variations. Spatial smoothing is a necessary prerequisite for correcting the statistical inference using Gaussian random field (GRF) theory. Thresholds using GRF theory to control corrected P values assume that the residual field is a sufficiently smooth lattice approximation of an underlying smooth random field. Only if these requirements are met, the resel count, denoting the kernel that an independent noise field needs to be convolved with to yield the same smoothness as the residuals, becomes meaningful.

fMRI Image Analysis

The fMRI data were analyzed using a general linear model as implemented in SPM5. Each experimental condition was modeled using the stimulus onset and the time until a response was made convolved with a canonical hemodynamic response function and its first-order temporal derivative. Low-frequency signal drifts were filtered using a cutoff period of 128 s. Parameter estimates were subsequently calculated for each voxel using weighted least squares to provide maximum likelihood estimators based on the temporal autocorrelation of the data (Kiebel and Holmes 2004).

Four regressors were based on the stimulus onset and duration until response according to the 4 experimental tasks, whereas 2 other regressors were based on the response onset, that is, button pressure with left or right hand, to capture motor activity. In particular, the 2 motor regressors were especially introduced into the design to remove the motor-related variance from the fMRI time-series signal. In doing so, reducing variance in the error term enhanced the statistical power of the model. To preclude hypothetically conceivable collinearity issues between the task and motor regressors, we additionally examined design collinearity and estimability on the first and second level (Supplementary Fig. S6–S9). These analyses clearly showed that collinearities between task and motor regressors were only moderate in the first- and basically absent in the second-level design. The absence of any relevant collinearity and the very good design estimability indicated that adding the motor regressors was unlikely to have acted as a confound detrimental to design statistics. To consolidate this conclusion, we also computed all discussed contrasts based on a study design without motor regressors (Supplementary Fig. S10–S15). As the same pattern of brain activity emerged (albeit at a much more lenient uncorrected threshold of P < 0.01, indicating more unmodeled residual variance, i.e., noise), we deduced that task and motor regressors predicted 2 distinct aspects of the overall brain activity. Taken together, the introduction of motor regressors into the study design was highly beneficial since they removed the motor-related (i.e., task-irrelevant) variance from the fMRI time-series signal, thereby enhancing the model's statistical power without distorting the variance explained by the task-related regressors.

Additionally, individual trial-wise reaction times were incorporated as a parametric modulator into the 4 task regressors of the model to isolate BOLD signal variance related to decision uncertainty as reflected by long reaction times. In particular, we did neither transform nor truncate the reaction time distribution to remove outliers because this would have entailed the chance of reducing power in the parametric modulation, given that this analysis aims at finding intraindividual correlations between a behavioral measure and brain activity. Moreover, the subjects' mean reaction times in each of the 4 tasks were included as covariates into an analysis of covariance (ANCOVA) to separate task-related from performance-related activity. More precisely, the condition-wise averaged reaction times from each participant were modeled by their level-specific interaction with the main regressors reflecting the separate conditions. Lastly, we included 6 nuisance regressors as movement parameters to remove artificial motion-related signal changes. For each subject, simple main effects for each experimental condition and parametric modulator were computed by applying appropriate baseline contrasts.

The individual first-level contrasts of interest were then fed into a second-level random-effects analysis of variance (ANOVA) (factor: condition, blocking factor subject; Penny and Holmes 2004). In the modeling of variance components, we allowed for violations of sphericity by modeling nonindependence across images from the same subject and allowing unequal variances between conditions and subjects as implemented in SPM5. Simple main effects of each task (vs. resting baseline) as well as comparisons between experimental factors were tested by applying appropriate linear contrast to the ANOVA parameter estimates. Composite main effects (i.e., activations, which were present in each of 2 different conditions) were tested by means of a conjunction analysis using the minimum statistic (Nichols et al. 2005). The resulting SPM(t) and SPM(F) maps were then thresholded at voxel-level P < 0.001 and cluster-level P < 0.05 (corrected for multiple comparisons; Worsley et al. 1996).

Results

Behavioral Results

Mean reaction time (± standard deviation) for each type of judgment was trustworthiness (TR): 1543 ± 448 ms; attractiveness (AT): 1489 ± 422 ms; age (AG): 1448 ± 408 ms; happiness (HA): 1211 ± 457 ms. A repeated-measures ANOVA indicated that reaction times were significantly different between conditions. Pair-wise contrasts revealed that differences in reaction time were statistically significant between all conditions (all P < 0.01, all η_p² > 0.16) but attractiveness and age (AT–AG: F_3,129 = 2.49, P = 0.122, η_p² = 0.06).

We observed a positive correlation between choice preferences from judgments of trustworthiness, attractiveness, and happiness (TR–AT: r = 0.900, P < 0.001; AT–HA: r = 0.873, P < 0.001; TR–HA: r = 0.859, P < 0.001). In contrast, choice preferences in age judgments correlated negatively with those 3 categories (AG–AT: −0.590, P < 0.001; AG–HA: −0.485, P < 0.001; AG–TR: −0.371, P = 0.003). In other words, the more often a given face was selected as appearing older, the less often it was selected as more trustworthy, more attractive, or happier.

The button presses that indicated judgment outcomes were virtually identically distributed to either side across conditions: trustworthiness judgments: 1009/1031 (total left/right button presses across subjects); attractiveness judgments: 1027/1022; happiness judgments: 1025/1047; and age judgments: 1046/1015. We further found that mean number of missed responses was also very similar between the conditions: trustworthiness judgments: 1.64 missed responses on average in 48 trials; attractiveness judgments: 1.43; happiness judgments: 0.91; and age judgments: 1.16.

Imaging Results

Unless stated otherwise, all fMRI results were derived from whole-brain analyses, are reported in the MNI space and survived a cluster-level corrected significance threshold (cluster-forming threshold: P < 0.001 at voxel level). We computed second-level activation maps after modeling trials on the first level with a duration according to 1) the average response time of all 4 conditions; 2) the average response time of the respective condition; or 3) a constant value of 3 s according to the stimulus presentation time. All 3 approaches yielded essentially identical results for the reported contrasts at group level.

To delineate the neural substrates activated in all 4 facial judgments, a conjunction across all conditions relative to rest (TR ∩ AT ∩ HA ∩ AG) was performed. This revealed bilateral activation in the premotor cortex (PMC; BA 6), supplementary motor area (SMA; BA 6, extending into dorsal anterior cingulated cortex), pars opercularis of the inferior frontal gyrus (IFG; BA 44), intraparietal sulcus and superior parietal cortex (IPS/SPC) as well as pronounced bilateral activation of the ventral visual cortex extending from the inferior occipital gyrus into the middle occipital gyrus, fusiform gyrus (FG), and lingual gyrus (for details, see Supplementary Table S1 and Fig. S1, note that parts of the parietal activations were rendered into the sulci). Subcortical activation were observed in the putamen (PU; extending into head of the caudate), posterior thalamus (pTH), and midbrain tectum (Tec) (Supplementary Fig. S2A).

The core network specifically subserving trustworthiness and attractiveness, that is, social, judgments was unveiled by a conjunction analysis across both of these socially relevant decisions in comparison to happiness and age judgments ((TR–AG) ∩ (AT–AG) ∩ (TR–HA) ∩ (AT–HA)). This “social network” consisted of significantly increased activity in the bilateral dorsomedial prefrontal cortex (dmPFC), left IFG (pars orbitalis et triangularis, BA 44/45), and right cerebellum (CRB) (Table 1 and Fig. 2, Supplementary Fig. S2B).

Table 1.

Main effects across conditions and of social judgments

Macroanatomical location	x	y	z	k	Z
(TR–AG) ∩ (AT–AG) ∩ (TR–HA) ∩ (AT–HA)
L dmPFC (extending into right hemisphere)	−2	57	17	3,827	6.16
L dmPFC	−17	58	23	3,827	4.15
L IFG (pars orbitalis et triangulairs)	−54	24	6	1,505	5.41
R cerebellum [CRB]	35	−81	−38	1,619	6.47

Note: Table shows coordinates derived from respective cluster peaks (x, y, z), cluster size (k), and z-scores (Z). TR, trustworthiness; AT, attractiveness; HA, happiness; AG, age judgments.

Figure 2.

Main effects of social judgments. Medial and lateral views of T1 MNI single-subject template for common activation in social judgments, relative to emotional and cognitive judgments. The significance threshold is set at P < 0.001 with a cluster-forming threshold of P < 0.05 (for subcortical and cerebellar activation, see also Supplementary Fig. S2).

To allow further functional stratification, we assumed that explicit happiness judgments involve brain areas for basic emotion recognition (“emotional network”) but lack elaborate social reflection or cognitive appraisal. Likewise explicit age judgments (“cognitive network”) should involve cognitive brain areas but lack specific social or emotional processing. It should be stressed, that this distinction only pertains to the explicit task-based components, not implicit, stimulus-driven processing of, for example, emotional displays. Due to the balanced stimuli, however, the latter should be equally present in all conditions and hence not influence any comparison. As noted above, social judgments may also involve, besides specific processes, both emotional and cognitive components. Given the 2 control conditions, these could be separately parceled out from the activity during social judgments. To delineate activity related to (social and) emotional aspects in face-based judgments, we computed a conjunction of trustworthiness and attractiveness judgments relative to age (but not happiness) judgments ((TR–AG) ∩ (AT–AG)), thereby effectively removing neural activity related to cognitive aspects. In addition to the “social network” described above (IFG, dmPFC, CRB; cf. Tables 1 and 2), this analysis also yielded left-lateralized activity in the ventromedial prefrontal cortex (vmPFC), posterior cingulate cortex, amygdala (AM; LB), hippocampus (HC; SUB/CA), and temporal pole (TP), as well as bilateral activity in the rostral anterior cingulate cortex (rACC) and midcingulate cortex (MCC) (Table 2 and Fig. 3A, for amygdalar, hippocampal, and cerebellar activation, see Supplementary Fig. S3A). Conversely, factoring out variance only shared with the emotional but not the cognitive control condition ((TR–HA) ∩ (AT–HA)) allowed the additional assessment of the neural correlates of cognitive aspects in social judgments. This approach revealed additional (to the “social network” cf. above) activity in the bilateral anterior insula (AI), dorsolateral prefrontal cortex (dlPFC), ventral and posterior thalamus (vTH/pTH), as well as left TP, right IFG (pars orbitalis), left cerebellum, and right inferior parietal cortex (IPC) (Table 2 and Fig. 3B, Supplementary Fig. S3B).

Table 2.

Emotional and cognitive aspects in social judgments

Macroanatomical location	x	y	z	k	Z
(TR−AG) ∩ (AT−AG)
L dmPFC (extending into right hemisphere)	−3	60	15	8,149	7.10
L vmPFC	−3	39	−17	8,149	3.94
L IFG (pars orbitalis et triangularis)	−50	27	0	1,729	6.33
L TP	−45	12	−41	669	5.63
L rACC (extending into right hemisphere)	−3	38	11	8,149	4.27
L MCC (extending into right hemisphere)	−2	−20	38	335	4.53
L posterior cingulated cortex	−3	−48	29	281	4.10
L hippocampus [HC]	−24	−23	−18	564	4.47
L hippocampus [HC]	−33	−12	−23	564	4.11
L amygdala [AM]	−29	−6	−21	564	4.09
L amygdala [AM]	−33	−6	−29	564	4.05
L cerebellum [CRB]	−27	−81	−39	513	5.17
L cerebellum [CRB]	−35	−80	−35	513	4.73
R cerebellum [CRB]	33	−81	−36	2,193	6.94
(TR−HA) ∩ (AT−HA)
L dmPFC (extending into right hemisphere)	−2	57	17	6,338	6.16
L IFG (pars orbitalis et triangularis)	−48	20	6	5,214	5.92
R IFG (pars orbitalis)	45	26	−8	1,823	5.41
L AI	−38	24	−20	5,214	7.35
R AI	29	18	−20	1,823	6.33
L vTH (extending into right hemisphere)	−5	−2	3	768	4.61
R pTH (extending into left hemisphere)	2	−17	5	768	4.73
L dlPFC	−45	12	44	356	4.45
R dlPFC	42	23	23	758	4.50
L TP	−45	12	−39	5,214	4.16
R IPC	50	−62	38	894	4.94
R IPC	33	−75	50	894	4.51
L cerebellum [CRB]	−44	−69	−42	1,845	5.74
L cerebellum [CRB]	−29	−81	−36	1,845	4.39
R cerebellum [CRB]	35	−81	−38	3,189	6.47
(TR−AG) ∩ (AT−AG) ∩ (HA−AG)
L vmPFC	−11	51	−11	414	4.59
(TR−HA) ∩ (AT−HA) ∩ (AG−HA)
L AI	−41	24	−11	882	5.52
R AI	33	24	−6	1,166	4.82
R AI	41	21	−6	1,166	4.91
L dlPFC	−53	32	15	406	4.98
R dlPFC	42	23	23	755	4.50
R dlPFC	33	47	11	401	4.31
R IPC]	33	−75	50	579	4.42
L cerebellum [CRB]	−41	−65	−41	892	5.23

Note: Table shows coordinates derived from respective cluster peaks (x, y, z), cluster size (k), and z-scores (Z). TR, trustworthiness; AT, attractiveness; HA, happiness; AG = age judgments.

Figure 3.

Emotional and cognitive aspects in social judgments. Medial and lateral views of T1 MNI single-subject template for common activation in social judgments when not parceling out (A) emotion decision–related activation and (B) cognitive decision–related activation. Lateral and section views of T1 MNI single-subject template for (C) conjunction across trustworthiness [TR], attractiveness [AT], and happiness [HA] judgments, relative to age [AG] judgments, to specifically depict the convergence of social and emotional processes during social judgments (sagittal section at x = −10); (D) conjunction across trustworthiness, attractiveness, and age judgments, relative to happiness judgments, to specifically depict convergence of social and cognitive processes during social judgments (axial section at z = −9). The significance threshold is set at P < 0.001 with a cluster-forming threshold of P < 0.05 (for subcortical and cerebellar activation, see also Supplementary Fig. S3).

To further characterize the emotional network inherently active during social judgments, we tested for an overlap between explicit emotional and social judgments, relative to cognitive decisions ((TR–AG) ∩ (AT–AG) ∩ (HA–AG)). This revealed a convergence between emotional and social appraisal (relative to cognitive age assessment) only in the left vmPFC (Table 2 and Fig. 3C). Likewise, the conjunction between cognitive and social judgments, relative to emotional decisions ((TR–HA) ∩ (AT–HA) ∩ (AG–HA)), showed the cognitive network inherently active during social judgments. This revealed common activity in the AI and dlPFC bilaterally as well as right IPC and left cerebellum (Table 2 and Fig. 3D).

It is noteworthy that we reexamined all of the above contrasts based on a study design without motor regressors to refute the theoretical probability of those regressors exerting a confounding effect. Importantly, the same pattern of brain activity emerged from this analytical study design (Supplementary Fig. S10–S15), yet, only at a more lenient uncorrected threshold of P < 0.01. These supplementary analyses thus strongly support the obtained results and, moreover, indicate the utility of including the motor regressors to remove task-unrelated activity as well as to increase statistical power.

To isolate brain activity that is specific to either of the 2 examined social judgments (trustworthiness, attractiveness), we calculated differences between these 2 conditions. The AT–TR contrast yielded activation in the bilateral FG as well as in the left inferior and middle occipital gyrus (IOG/MOG) (Table 3 and Fig. 4A,B). The TR–AT contrast yielded activation in the bilateral posterior superior temporal sulcus (pSTS) and right middle temporal gyrus (MTG) (Table 3 and Fig. 4C,D, Supplementary Fig. S5). Taken together, the 2 social judgments diverged primarily in brain regions attributed to processing visually presented faces or, more general, human stimuli.

Table 3.

Neural differences between trustworthiness and attractiveness judgments

Macroanatomical location	x	y	z	k	Z
AT−TR
R FG	33	−39	−12	987	5.77
L FG	−24	−44	−11	1,301	4.27
L IOG	−12	−78	−9	1,301	4.51
L MOG	−41	−78	15	444	3.68
TR–AT
L pSTS	−66	−39	−5	310	3.97
R pSTS	57	−51	15	515	4.64
R MTG	57	−8	−17	525	5.59

Note: Table shows coordinates derived from respective cluster peaks (x, y, z), cluster size (k), and z-scores (Z). TR, trustworthiness; AT, attractiveness; HA, happiness; AG, age judgments.

Figure 4.

Neural differences between trustworthiness and attractiveness judgments. Activation for attractiveness versus trustworthiness judgments (A/B) and vice versa (C/D) superimposed on a T1 MNI single-subject template. (B) Horizontal section through cluster maximum in the right FG (z = −12) in the attractiveness–trustworthiness contrast, (D) sagittal section through cluster maximum in the left pSTS (x = −66) in the trustworthiness–attractiveness contrast. Coordinates in MNI space. The significance threshold is set at P < 0.001 with a cluster-forming threshold of P < 0.05. TR, trustworthiness; AT, attractiveness; HA, happiness; AG, age judgments.

While a recent activation likelihood estimation (ALE) meta-analysis revealed the amygdala as an important point of convergence in both trustworthiness and attractiveness judgments (Bzdok et al. 2011), this was not revealed in the conservative analyses reported above. To explicitly test the hypothesis of amygdala involvement in social judgments, we performed a small-volume correction (family-wise error corrected at P < 0.05) of the omnibus F-test for differences between conditions within the AM as defined by probabilistic cytoarchitectonic maps (Amunts et al. 2005; Eickhoff et al. 2006). This yielded a significant cluster in the left amygdala (peak at x = −29, y = −2, −23; 380 voxels; laterobasal and superficial nuclei group) as well as small locus of activation in the right one (peak at x = 27, y = −5, z = −20; 37 voxels; laterobasal and superficial nuclei group) (Fig. 5). On the left hemisphere, social judgments evoked higher amygdalar activity than both happiness and age conditions, on the right side, only age judgments differed significantly from all other conditions.

Figure 5.

Amygdalar response across social, emotional, and cognitive facial judgments. Small volume–corrected analysis of the difference F-test across conditions using an anatomical template for the (A) left amygdala and (B) right amygdala. Activity significant at voxel-level P < 0.05 (family-wise error corrected) is shown on 2 coronal sections through the T1 MNI single-subject template at (A) y = −2 and (B) y = −5. Coordinates in MNI space. Two histograms, added for purely illustrative purposes, depicting BOLD signal changes across conditions in the cluster peak voxel in the (A) left and (B) right amygdala. TR, trustworthiness; AT, attractiveness; HA, happiness; AG, age judgments. The asterisk indicates significant differences between conditions (P < 0.05).

Testing for differences between male (n = 24, 35.1 ± 11.7 years) and female (n = 20, 29.9 ± 10.7 years) participants in any of the 4 task conditions, revealed a significant effect only in the attractiveness judgments, where males showed higher ventral and posterior thalamic activity (Supplementary Table S2).

Finally, neural activity attributable to varying task difficulty across trials, that is, decisions on face pairs, was determined by condition-wise parametric modulators reflecting trial-by-trial variations in intraindividual reaction time. The conjunction across the parametric modulators associated with the 4 different conditions revealed task difficulty or indecisiveness effects in the dlPFC, AI, IPS, IPC, PMC, and SMA bilaterally as well as the right middle frontal gyrus (MFG) (Supplementary Fig. S4A). The activation pattern obtained in this experiment overlapped substantially with the cognitive network in social judgments ((TR–HA) ∩ (AT–HA) ∩ (AG–HA)). In particular, both contrasts showed significant activation in the AI and dlPFC bilaterally as well as the right IPC which further supports the role of these areas in the cognitive “decision” aspect of explicit facial judgments (Supplementary Fig. S4B).

Discussion

The present fMRI study aimed at delineating the neurobiology of explicit social judgments of visually presented faces. We therefore compared neuronal activations evoked by judgments of social traits of trustworthiness (TR) and attractiveness (AT) with those evoked by explicit emotion judgments (here: happiness, HA), as well as age judgments (AG), a cognitive control condition. In doing so, we unveiled the highly congruent neural signature of TR and AT. Notably, both social judgments were mainly divergent in face-sensitive regions. Further, we dissected subnetworks in social judgments reflecting emotional and cognitive processing by contrasting the social judgments with only one of the control conditions and performing a conjunction with the other, hereby effectively parceling out shared variance.

Social Correlates in Trustworthiness and Attractiveness Judgments

Explicit trustworthiness and attractiveness judgments, relative to judgments of emotionality or a cognitive assessment (age), converged in the dmPFC and IFG. The dmPFC has been implicated in a range of complex tasks, such as action monitoring, free thought, person perception, and self-knowledge (Amodio and Frith 2006). The global maximum, found in this brain area, is well in line with the localization of activity during mentalizing, that is, inferring others' beliefs (Gilbert et al. 2006). Put broadly, dmPFC activity has consistently been interpreted to serve inference and assessment of one's own and others' mind states (Gusnard et al. 2001; Gallagher and Frith 2003). This makes it a likely candidate region for the interaction of self- and other-oriented processes during social judgments.

IFG recruitment (BA 44/45) during social judgments might be interpreted in 3, potentially nested, ways: First, we cannot reject the assumption that Broca's area, a language-related region situated in the IFG (Price 2010), underlies these activation patterns due to inner speech or subvocalization. Second, this brain structure has been implicated in characterizing others by attribution of traits (Mitchell et al. 2004), affect (Shamay-Tsoory et al. 2009), mind states (Ochsner et al. 2004), or facial properties (Ishai et al. 2002). This interpretation, however, could easily be reducible to the more parsimonious first interpretation of covert speech. Third, the IFG, harboring mirror neurons in monkeys (Pellegrino et al. 1992) and presumably also humans (Caspers et al. 2010), is involved in representing others' actions and, thus perhaps, goals, as mediated by, for example, moving facial musculature (Carr et al. 2003; Gallese 2003). Considering that humans unconsciously synchronize their facial expressions even with people attending to a third individual (Schilbach et al. 2008), we deem it possible that similar social cohesion–promoting automaticities influence social judgments.

Face-Sensitive Correlates in Trustworthiness and Attractiveness Judgments

Despite the high correlation between the ratings of trustworthiness and attractiveness (Oosterhof and Todorov 2008), we show that these 2 judgments are partly dissociable in a network involved in visual face processing. Within this network, the pSTS is believed to predominantly process “variable” facial features (such as emotion and gaze), while the FG may predominantly process “stable” facial features (such as identity or scars). This functional segregation was initially proposed in cognitive psychology (Bruce and Young 1986) and later reframed in neuroanatomical terms (Kanwisher et al. 1997; Haxby et al. 2000). Our data corroborate a functional dichotomy by demonstrating selectively increased pSTS and FG activity during trustworthiness and attractiveness judgments, respectively. Further support for a distinction between pSTS and FG in social judgments comes from studies of neurological dysfunction on prosopagnosia, that is, the inability to recognize the identity of faces which is frequently caused by FG lesions (Barton et al. 2002). In 2 individual studies, prosopagnosic patients differed from healthy controls in attractiveness ratings (Iaria et al. 2008) but not trustworthiness ratings (Todorov and Duchaine 2008). Moreover, pSTS lesions were reported to debilitate gaze cognition in monkeys (Heywood and Cowey 1992) and humans (Akiyama et al. 2006). It may hence be speculated that decisions of trustworthiness might prioritize information on variable features, subserved by the pSTS, for example, mimics and gaze direction (cf. Emery 2000; Nummenmaa and Calder 2009). Conversely, attractiveness judgments may be more based on structural features, subserved by the FG, in line with psychological literature indicating that attractiveness hinges on facial symmetry and averageness (Rhodes 2006).

In summary, we observed a dissociation between the FG and pSTS presumably emerging from top-down modulation during explicit social judgments. This finding might be read as a selective facilitation of processing stable versus variable facial features in explicit social judgments, consistent with both neuropsychological and lesion studies. The observed dissociation in face-processing regions in conjunction with the convergent activity in high-level “social cognition” related brain areas (dmPFC and IFG) suggests that the latter may exert top-down influences on those areas processing the relevant stimulus attributes, depending on the goal of an inference.

Emotional Aspects of Trustworthiness and Attractiveness Judgments

Explicit trustworthiness, attractiveness, and happiness judgments overlapped in the vmPFC. We therefore propose this region as a hub interfacing high-level social and basic emotional cognitions. In fact, immunohistochemical and tracer studies are supportive of the vmPFC's connections with the dmPFC and with the limbic system, including amygdala (AM) as well as hippocampus (HC) (Amaral and Price 1984; Ongur et al. 2003). What structure–function relationship might lie behind this connectivity pattern? On the one hand, vmPFC damage does not impair conscious reflection of social phenomena (Saver and Damasio 1991). On the other hand, vmPFC damage disrupts affective but not cognitive theory of mind (Shamay-Tsoory et al. 2006), reduces emotional impact on moral judgments (Koenigs et al. 2007), and impairs empathy (Eslinger 1998). Importantly, vmPFC lesions also leave intellectual capacities (Eslinger and Damasio 1985) and recognition of faces (Shamay-Tsoory et al. 2005) intact. The anatomical connections and lesion data, in concert with the observed activity pattern in the present study, thus suggest that the vmPFC interweaves abstract social thought, probably relying on the dmPFC, and emotionality, probably relying on limbic activity.

By virtue of their anatomical interconnections, the vmPFC may receive input from the AM which has been conceptualized as an evolved significance detector constantly scrutinizing the environment for biologically relevant cues (Sander et al. 2003). Amygdalar bottom-up emotion processing is known to be rapid, mandatory, and prereflexive (Vuilleumier 2005). In this context, it is important to reiterate that implicit emotional processing (which was repeatedly shown to drive in particular the amygdala; Adolphs 2008) should be equally present in all conditions and hence cancel out in the subtractions. The top-down modulation–driven effects in this study, however, revealed a higher amygdalar response in social judgments, compared with both explicit emotional and cognitive processing. Importantly, the analysis of reaction time as a proxy for cognitive demand did not reveal any statistically significant effect of the covariate in the amygdala, indicating neither a statistically reliable increase nor decrease in activation. Most importantly, brain activation patterns in all 4 experimental conditions as well as the reported differences remained virtually identical after parceling out variance explained by reaction time. Furthermore, we did not find any significant effect of reaction time (i.e., cognitive demand) in the amygdala when assessing trial-by-trial variation in reaction time by the parametric modulator at the single-subject level. Taken together, there is thus no evidence, that the differences in signal strength between conditions may be explained by systematic differences in cognitive demand. This might provide another piece of evidence that the AM's function may not be restricted to a “gateway to the emotions” (Aggleton and Mishkin 1986). It is tempting at this point to conjecture that the vmPFC specifically heightens amygdalar activity in TR/AT as part of its integrator role with high-level social cognition. Such an additive AM response to social processing on top of baseline emotional processing concurs with an fMRI study focusing on the social–emotional interplay (Norris et al. 2004).

Cognitive Aspects of Trustworthiness and Attractiveness Judgments

The overlapping activity between social judgments and age assessment, relative to emotionality judgments, recruited the dlPFC and AI bilaterally as well as right IPC, and left cerebellum. Among the functions ascribed to the dlPFC are working memory processes and response selection (Rowe et al. 2000). In social cognition, a role for this region in response selection was suggested by a TMS study using the ultimatum game, in which transient dlPFC disruption led to inappropriate acceptance of offers actually recognized as unfair (Knoch et al. 2006). Following this hypothesis, the dlPFC activity in the present study might reflect the cognitive effort needed to formulate a decision following the presentation of each face pair. Evidently, a more difficult decision would also entail higher working memory load, as evidence for either choice has to be stored. Consistent with this interpretation, activity associated with varying task difficulty across individual trials, as indicated by parametric modulation with reaction time, was also found in this region. Likewise, the IPC has also been implicated in working memory (LoPresti et al. 2008), cognitive decision processes (Corbetta et al. 2000), and attention (Pessoa et al. 2002). Its activation thus appears to reflect, in tandem with the dlPFC, the cognitive load imposed by the social task (Bokde et al. 2006). In line with this view, the coupled activity of these 2 areas was reported as a marker for cognitive conflict in difficult, as compared with easy personal moral judgments (Greene et al. 2004).

The role of the insula is typically portrayed as pertaining to autonomic arousal regulation, interoceptive awareness, and subjective feeling (Craig 2002). A stimulus-driven autonomic arousal, however, would be common to all 4 conditions due to the identical stimulus material and, hence, is likely to be eliminated to a large extent by subtraction. In contrast, brain activation pertaining to differential autonomic–interoceptive top-down processes may result as a function of the type of judgment. For example, attractiveness judgments may entail increased insula activation related to arousal regulation, interoceptive awareness, and subjective feeling (Paulus and Frank 2003; Turk et al. 2004). On a different note, it is also important to remember the less widely recognized but very consistent activation of the (anterior) insula in various cognitive tasks, including memory, attention, and speech (Kurth et al. 2010). In light of the insula's role in both affective and cognitive functions, any interpretation of the observed activations must remain tentative. Insular recruitment might reflect arousal-related processes, cognitive processes, or a mixture of both.

Taking a wider perspective, prior fMRI studies of face processing that employed age judgments as control conditions may have biased results in some brain areas conceptualized as being involved in social cognition, in particular the AI (Ochsner 2008). Furthermore, the magnitude of dlPFC, IPC, and AI involvement, a likely index of cognitive load, might have been underestimated in face processing studies that used age judgments as a control condition. This is particularly true given that reaction times have frequently been reported to be longer for age versus emotional judgments (Habel et al. 2007), suggesting that cognitive load was indeed higher in the control condition. It is plausible to conjecture that this stronger cognitive effort in the “control” condition might bias results by evoking substantial activity in regions of interest by reallocation of attentional resources. The proposal to use a simpler “older than 30 years or not” style of question may better balance the cognitive demand across conditions (Gur, Schroeder, et al. 2002). Based on the previous results, we would, however, argue for the benefits of employing more than one control condition in face-processing research.

To assess the influence of potential differences in cognitive demand between trials and conditions, we ran 2 separate analyses. First, we performed an ANCOVA model that included a behavioral covariate reflecting task difficulty. In particular, we used the individual mean reaction time of the 4 conditions as indices of cognitive demand related to each type of judgment. In doing so, brain activation that could be explained by differences in difficulty between conditions was isolated and therefore excluded from the main task regressors. However, brain activation patterns in all 4 experimental conditions as well as the reported difference contrasts remained virtually identical after parceling out variance with reaction time. Second, we introduced a parametric modulator for each of the 4 conditions in the present ANOVA model, which reflected the trial-wise reaction time. That is, we used the intraindividual variance in reaction time across trials as indices of fluctuations in cognitive demand. The ensuing activation maps revealed a main effect in the SMA, precentral gyrus, dlPFC, MFG, AI, and IPC. This “cognitive load” network is thus divergent from the brain areas implicated in social and emotional aspects of facial judgments, as discussed above, including the vmPFC/dmPFC, amygdala, pSTS, FG, and IFG. Taken together, we provide evidence that the presented results are largely independent of task difficulty by means of 2 independent statistical analyses.

The Cerebellum in Facial Judgments

While the cerebellum is traditionally associated with motor functions (Holmes 1939), the observation that other motor areas such as the (pre)motor cortex and the SMA were not activated in the assessed contrasts renders a purely motor-based interpretation unlikely. In line with this interpretation, button presses were virtually identically distributed to either response side, and the mean number of missed responses was also distributed quite equally among subjects and conditions, making it unlikely that finger presses acted as a confound in our analyses. There is, however, a growing literature that points to an array of nonmotor functions of the cerebellum, including emotion regulation and social cognition. For example, increased cerebellar activity was found during pain perception in self and others (Singer et al. 2004), social conformity conflicts (Klucharev et al. 2009), appraisal of facial emotion (Fusar-Poli et al. 2009), emotional and cognitive perspective taking (Gallagher et al. 2002; Hooker et al. 2008) as well as encountering cooperative behavior (Decety et al. 2004). Moreover, studies on circumscribed cerebellar lesions consistently reported deficits in nonmotor aspects of behavior, including language, visual–spatial, affective, and executive functions (Stoodley and Schmahmann 2009b).

In the present study, the absence of activation in the cerebellum for the conjunction across all 4 conditions indicates that different facial judgments do not generally recruit common cerebellar regions. Notably, a recent quantitative meta-analysis (Stoodley and Schmahmann 2009a) revealed spatial processing to be largely left lateralized and language processing to be largely right lateralized in the cerebellum. The right cerebellar activation specific to social judgments might therefore be related to the left hemisphere's dominance for language representation (cf. Strick et al. 2009) and, together with the concomitant IFG activation, may indicate covered speech as part of the evaluation process. In contrast, we found a convergence between social judgments and the putative cognitive network activated by age judgments in the left cerebellum. That is, social and age judgments might necessitate intensified visual–spatial processing, perhaps jointly implemented by the cerebellum and the posterior parietal cortex (Strick et al. 2009).

Conclusion

We decomposed the neural correlates underlying trustworthiness and attractiveness judgments of faces into social, face-sensitive, emotional, and cognitive networks. This was achieved by a step-by-step conjunction approach with 2 functionally divergent control conditions, reflecting emotional and cognitive aspects, respectively, that should both be inherent to social judgments. In doing so, the vmPFC emerged as a potential social–emotional interface and likely contributor of the observed heightened amygdalar response during social judgments. Moreover, this study demonstrated that, except for a few face-sensitive regions, explicit trustworthiness and attractiveness judgments share a congruent neural signature. Ultimately, as a result of cultural indoctrination, most of us intuitively think of facial trustworthiness and attractiveness as distinct entities. Yet, this belies the possibility that trustworthiness and attractiveness judgments might constitute societal mechanisms rooted in a common neurobiological implementation.

Supplementary Material

Supplementary material can be found at: http://www.cercor.oxfordjournals.org/

Funding

German Research Council (DFG, IRTG 1328) to K.Z., S.B.E., and D.B.; Human Brain Project (R01-MH074457-01A1 to S.B.E.); Helmholtz Initiative on Systems Biology “The Human Brain Model” to K.Z. and S.B.E.