Neurons in the human amygdala encode face identity but not gaze direction

Abstract

The amygdala is a key structure in face processing, and direction of eye gaze is one of the most socially salient facial signals. Recording from over 200 neurons in the amygdala of neurosurgical patients, we here find robust encoding of the identity of neutral-expression faces, but not to their direction of gaze. Processing of gaze direction may rely on a predominantly cortical network rather than the amygdala.

Direction of gaze is one of the most potent social signals in primates¹, and is processed by a specialized network of brain regions². While considerable work has elucidated key cortical components of this network, notably sectors of parietal cortex and the posterior superior temporal sulcus (pSTS)², subcortical contributions remain debated. In particular, there is good evidence both for and against a role for the amygdala. Some studies³ but not others⁴ find impaired gaze perception following human amygdala lesions. Some neuroimaging studies find greater amygdala activation to direct than to averted gaze⁵, some greater activation to averted than direct gaze^6,7, as well as interactions with emotional expression⁷, and some greater activation just to the anticipation of direct gaze than direct gaze per se⁸. Taken together, a sizeable literature addresses the role of the amygdala in processing eye gaze from faces, but with no clear consensus at all.

The issue is also of clinical interest, since impairments in processing the social meaning of eye gaze are well documented in autism, and this impairment has been linked to amygdala dysfunction in neuroimaging studies^9,10. Perhaps most relevantly, direct neuronal recordings in the human amygdala have found reliable responses to the eye region of faces, whereas these responses are strikingly reduced in people with autism¹¹.

To directly address the amygdala’s role in processing eye gaze, we recorded from a total of 904 single- and multi-units (225 SU, 679 MU) in the medial temporal lobe of 14 neurosurgical patients (see Online Methods for details; 223 units from the basolateral amygdala, 343 from hippocampus, 150 from entorhinal cortex, and 188 from parahippocampal cortex, see also Supplementary Tables 1,2, and Supplementary Fig. 1). Given the possibly complex interactions between facial expressions of emotion and gaze^7,12, we here focused on only three dimensions of neutral-expression faces: their identity (5 different actors with similar low-level visual image properties, Supplementary Table 3), their direction of gaze with head direction frontal (8 directions in 45-degree steps), and their head direction (with congruent eye gaze, 8 directions) (cf. Fig. 1; Supplementary Fig. 2).

Figure 1.

Responses of a single neuron in the amygdala to different persons and head/gaze directions. (a–d) Responses separated by person identity for all presented, only vertical, only right, only left head/gaze directions, respectively. Note that the neuron selectively responds to person 5, regardless of gaze or head deviations. (e) Responses of the same neuron during the 'live' encounter with the experimenter (stimulus person 5) show no effect of gaze. (f) Action potentials recorded from this neuron. All analyses were conducted on the time epoch from stimulus onset to 1000ms.

In line with prior recordings from the human amygdala^11,13, we found reliable responses to one or more persons featured in the stimulus viewing task (Fig. 1; see Supplementary Figs. 3,4 for responses across the entire population of 223 amygdala neurons). Two-way ANOVA of response firing rates yielded a significant main effect for person identity in 14% of amygdala units (p < 10⁻⁶, one-sided binomial test, see Online Methods for details), but a main effect for gaze direction in only 6% (p = 0.23). We thus find no evidence for gaze-selectivity (above what would be expected by chance). Moreover, the cells selective for person identity (31 cells; Supplementary Fig. 5) were almost entirely non-overlapping with the cells selective for gaze direction (14 cells; only 3 cells overlapped), and the proportion of gaze-selective cells was significantly smaller than the proportion that were identity-selective (p=0.007; Fisher’s exact test).

A separate comparison experiment carried out in 13 of the same patients showed that the overall selectivity of amygdala neurons to faces, as compared to non-face stimuli, was typical, making it unlikely that we were recording from unusual locations, or an unusual group of patients. As this comparison experiment was carried out with different stimuli and on a different day, it is not possible to link it to exactly the same neurons as in our main study, but it suggests sparse responses across a range of stimulus categories in the amygdala that are entirely consistent with prior studies¹³ (response probabilities and response magnitudes were equivalent for faces and non-faces, see Online Methods, and Supplementary Fig. 6).

Main effects for either person identity or gaze direction in our primary experiment were not observed above chance level in the other three MTL regions, except in parahippocampal cortex where 11% of units (p = 0.001) also showed a main effect of identity. We confirmed that neuronal responses to direct and averted gaze did not differ above chance levels with direct post-hoc tests (two-sided Wilcoxon ranksum tests comparing all direct gaze trials to all averted gaze trials, followed by one-sided binomial tests, p>0.05 for all regions). These findings remained valid when we used a coarser distinction of gaze directions (left/middle/right or up/middle/down) or when we restricted our analysis to single units only (see Online Methods).

To test whether an influence of person identity or gaze direction is present at the population level, we analyzed how images are segregated by response patterns using a categorization technique, Representational Similarity Analysis¹⁴. Representational similarity matrices displaying the similarity in response between all pairs of the 45 stimuli (gaze and head directions pooled together) revealed a specific response pattern in the amygdala to person identity, reflected by similar responses to a given person that differed from responses to all other persons (p < 10⁻⁵, permutation test, see Fig. 2). No comparable effect was found for gaze direction in any of the four MTL regions (p > 0.05). A significant influence of person identity was also found in the hippocampus (p = 0.001) and parahippocampal cortex (p < 10⁻⁴), but not in entorhinal cortex. When comparing all effect sizes of person identity vs. gaze direction yielded by the two-way ANOVA mentioned above at the population level, we confirmed significantly stronger effect sizes (ω²) for person identity than gaze direction for the amygdala (p = 0.0001) and parahippocampal cortex (p = 0.008), but not for hippocampus and entorhinal cortex (see Online Methods). Thus, although we certainly do not rule out some degree of gaze encoding in the human amygdala, it is not detectable above chance levels in our study, and it is a significantly weaker effect than identity encoding.

Figure 2.

Human amygdala neurons encode stimulus identity rather than gaze direction. Representational Similarity Analysis shows the similarity of the population response from all units in a given region to every pair of stimuli, grouped by person identity across gaze directions (left column) and by gaze direction across persons (right column). Higher r-values along the main diagonal of the matrices, indicating higher similarity within groups than between groups, can be seen in the amygdala for face identity, but not for gaze directions.

One shortcoming of our stimuli may have been their artificial nature: possibly, amygdala neurons would respond to eye gaze, but only with dynamic, more naturalistic, and more socially engaging stimuli rather than static images. To address this possibility, we also conducted in the same 14 patients a “live encounter” task. In this task, one of the experimenters (F.M. or J.N.) sat physically in front of the patient (Supplementary Fig. 7). Subjects were instructed to look at the experimenter for approximately 2 minutes while the experimenter randomly switched between looking directly at the subject (direct gaze), looking down (averted gaze), and closing his eyes (approx. every 2 sec.; all transitions were time-stamped and neuronal responses analyzed with respect to them). This procedure was performed twice, resulting in 40 trials for each of the three conditions.

As with the first task, post-hoc tests between the different gaze conditions failed to find any significant responses modulated by gaze in this live encounter experiment (Fig. 1; all p>0.05, one-sided binomial test). In a second version of the "live encounter" task, recorded during four experimental sessions in two additional patients (see Online Methods), the experimenter kept his gaze fixed on the subject, while the subject switched between the three conditions. Among the 76 additional amygdala units recorded with this version, there was again no significant modulation by condition (all p>0.05, one-sided binomial test).

Surprisingly, comparison of mean response activity during the static picture viewing vs. live encounter task across units showed a significantly higher activity during the static picture viewing (p=0.013 in the amygdala; p>0.2 for all other brain regions; two-sided Wilcoxon signed-rank test), a finding that was unexpected given the more arousing, socially engaging, and ecologically valid nature of the live encounter task.

Taken together, our findings are consistent with prior work showing that neurons in the human amygdala process the identity of faces, but we found no evidence for a role in processing gaze direction, either from eye gaze or head direction, even with a live person as the stimulus. These findings are surprising for at least two reasons. First, they seem discrepant with data from monkeys, which have reported neuronal amygdala responses to gaze direction^15,16; however, the proportion of such neurons may be very low (ca. 5% in Ref. 16) and the stimuli differed in several details from ours. Secondly, there is a body of data from neuroimaging that documents amygdala responses to gaze direction in both monkeys¹⁷ and humans^5–8, although the conclusions of these varied studies, as noted earlier, do not cohere very well.

It is also important to note that a recent study in monkeys found that amygdala neurons respond to direct gaze-- but only when the monkey is fixating onto the eyes of the stimuli¹⁸. In our experiment, subjects were required to attend to the eyes of the stimuli by the task that they had to perform, and task performance levels as well as observation by the experimenter all suggest that they fixated the eyes of the stimuli (95% task accuracy, cf. Online Methods). As well, the additional four sessions of our “live encounter” task required subjects to fixate onto, or away from, the experimenter’s direct (and live) gaze-- even here, to our surprise, we found no evidence for amygdala responses. It may be that human and monkey amygdala neurons respond differently to a conspecific’s gaze, for reasons that could range from species differences in basic perceptual processing, to differences in social meaning of the stimuli (the humans, unlike the monkeys, always know that they are in an experiment, even with the live encounter condition).

Another plausible explanation for our failure to find gaze-responsive neurons may be that we did not sample those regions of the amygdala where such neurons might be concentrated: in the monkey, BOLD-fMRI responses to gaze were found primarily in the central nucleus and bed nucleus of the stria terminalis¹⁷, but not in more basal or lateral parts of the amygdala, from which we recorded (see Supplementary Fig. 1). A further possibility is that neurons within the amygdala may not spike in response to gaze, even though they receive cortical inputs that carry such information— inputs whose gaze-sensitivity might instead be reflected in BOLD responses, field potential changes¹⁹, or MEG responses. It was also surprising that we did not find any response to gaze even with a real person as the stimulus; indeed, we found that in general this stimulus resulted in weaker responses than did our static images. Explanations here may include the possibility of habituation or top-down modulation of the amygdala due to context effects. Equally surprising was the fact that amygdala neurons were not modulated even by whether the experimenter had open or closed eyes—fMRI studies have found amygdala responses to eyes²⁰, as have some single-unit studies¹¹. One factor that may contribute to these differences is that these other studies both used emotional faces (happy or fear) whereas ours only showed neutral faces.

In conclusion, our findings argue that the role of the human amygdala in processing information about eye gaze should be reconsidered. The human amygdala’s strong selectivity for information from the eye region of faces may instead be used for a host of other kinds of processing, including recognition of identity and emotional expression^11,13 whereas gaze processing may draw predominantly on a cortical network². Given that neurons in the monkey amygdala do appear to respond to the eye region of faces¹⁸, our findings suggest that the specific ability for further fine discrimination of gaze direction may require cortical processing. While speculative, this account would be consistent both with the more detailed resolution of visual representations in cortex, and with the uniquely human evolution of the white sclera of the eyes to signal gaze direction, a cue unavailable to monkeys¹.