Comparing social attention in autism and amygdala lesions: Effects of stimulus and task condition

Abstract

The amygdala plays a critical role in orienting gaze and attention to socially salient stimuli. Previous work has demonstrated that SM a patient with rare bilateral amygdala lesions, fails to fixate and make use of information from the eyes in faces. Amygdala dysfunction has also been implicated as a contributing factor in autism spectrum disorders (ASD), consistent with some reports of reduced eye fixations in ASD. Yet, detailed comparisons between ASD and patients with amygdala lesions have not been undertaken. Here we carried out such a comparison, using eye tracking to complex social scenes that contained faces. We presented participants with three task conditions. In the Neutral task, participants had to determine what kind of room the scene took place in. In the Describe task, participants described the scene. In the Social Attention task, participants inferred where people in the scene were directing their attention. SM spent less time looking at the eyes and much more time looking at the mouths than control subjects, consistent with earlier findings. There was also a trend for the ASD group to spend less time on the eyes, although this depended on the particular image and task. Whereas controls and SM looked more at the eyes when the task required social attention, the ASD group did not. This pattern of impairments suggests that SM looks less at the eyes because of a failure in stimulus-driven attention to social features, whereas individuals with ASD look less at the eyes because they are generally insensitive to socially relevant information and fail to modulate attention as a function of task demands. We conclude that the source of the social attention impairment in ASD may arise upstream from the amygdala, rather than in the amygdala itself.

We evaluate the social meaning of scenes that contain people by directing our attention and gaze to socially relevant features. We integrate the significance of these features in the context of the entire image, and link it to our stereotypes, memory representations, and reasoning about what we see. Despite the complexity of the psychological processes and neural substrates that underlie this ability, there has been much progress regarding specific components. Healthy individuals are remarkably reliable in how they direct their gaze to important parts of a social scene. In particular, there is a strong bias to look at people's faces (Yarbus, 1967), and particularly at the eyes within faces (Birmingham, Bischof, & Kingstone, 2008a, 2008b; Henderson, Williams, & Falk, 2005; Pelphrey et al., 2002; Walker-Smith, Gale, & Findlay, 1977).

Faces and eyes attract our attention from birth (Johnson, 2005), and attract the attention of many other highly visual species (Emery, 2000) including nonhuman primates (Keating & Keating, 1982). A wealth of information about other persons can be gleaned from looking at their eyes. For instance, the eyes can provide clues about people's emotional or mental state, the focus of their attention, and their communicative intent. Thus, it is not surprising that, when we look at faces, much of our gaze is directed to the eyes (Yarbus, 1967). Furthermore, face detection is speeded by the presence of eyes (Lewis & Edmonds, 2003). In fact, even 3-day-old babies seem to be tuned into the eye region of faces, preferring to look at faces with direct eye contact as opposed to faces with averted gaze (Farroni, Csibra, Simion, & Johnson, 2002). Consistent with this, it has been demonstrated that infants prefer to look at faces with eyes open rather than eyes closed (Batki, Baron-Cohen, Wheelwright, Connellan, & Ahluwalia, 2000). These findings and many others resonate with the proposal that, just as there is thought to be a module for face processing (Yovel & Kanwisher, 2004), there may be a module for detecting eyes in the social environment (Baron-Cohen, 1995). This strong interest in the eyes of faces is an important component of social attention (Birmingham & Kingstone, 2009).

In the following section, we provide a brief overview on what is known about the abnormalities of social attention associated with autism spectrum disorders (ASD). We then consider the role of the amygdala in directing attention to the eyes of faces, and discuss the proposal that amygdala dysfunction underpins social attention abnormalities in ASD. Finally, we note the absence of direct comparisons between patients with amygdala damage and individuals with ASD on identical tasks and stimuli—-comparisons that are essential to determining the link between amygdala dysfunction and ASD. The present study sets out to provide such a comparison.

ASD AND SOCIAL ATTENTION

Many developmental disorders exhibit prominently abnormal levels of eye contact, including fragile X syndrome (Farzin, Rivera, & Hessl, 2009), Williams syndrome (Riby & Hancock, 2008), and, most notably, ASD. Both clinical and anecdotal evidence suggests that reduced or absent eye contact is a pervasive feature of ASD (Kanner, 1943; Lord, Rutter, & Le Coutier, 1994; Lord et al., 2000). Due to the fundamental role of eye contact in social communication, it is perhaps not surprising that many behavioral interventions attempt to teach individuals with ASD to make eye contact (e.g., Hwang & Hughes, 2000). Several prominent studies have documented abnormal eye gaze to static images of faces or to films in people with ASD (Klin, Jones, Schultz, Volkmar, & Cohen, 2002; Pelphrey et al., 2002). It has also been shown that there is a failure to make use of information from the eye region of faces in order to make social judgments, in both ASD (Spezio, Adolphs, Hurley, & Piven, 2007a) and in parents of people with autism who have the broad autism phenotype (Adolphs, Spezio, Parlier, & Piven, 2008).

Despite these findings to date, two key questions remain. The first concerns the reliability of the social attention impairment in ASD. The second concerns the neural mechanisms underlying these social attention impairments.

Reliability of the social attention impairment in ASD

Although reduced eye contact in ASD is a clinically and anecdotally well-known phenomenon, and is a primary focus of early behavioral intervention, experimental studies aimed at quantifying this phenomenon have in fact yielded rather mixed results. Using eye-tracking methodology, some studies have found reduced eye contact in ASD (e.g., Klin et al., 2002; Pelphrey et al., 2002; Sterling et al., 2008), but other studies have failed to document impaired eye fixation (e.g., Neumann, Spezio, Piven, & Adolphs, 2006; van der Geest, Kemner, Camfferman, Verbaten, & van Engeland, 2002). Even more puzzling is the fact that some studies using the same subjects but different tasks or stimuli found a reduction in eye contact in one condition, but not in others (Neumann et al., 2006; Speer, Cook, McMahon, & Clark, 2007), suggesting that the deficit is only revealed in particular contexts. That is, the lack of reliability of social attention abnormalities in ASD may be attributable to variance due to individual differences in the subjects, differences in the stimuli used and their context, and differences in task demands. One major aim of our study was thus to elucidate these sources of variability, in particular the effect that different tasks would have on eye gaze when people are viewing the same stimuli.

Neural substrates of abnormal social attention in ASD

A second open question that has perhaps attracted even more attention is to understand the neural substrates that might be responsible for abnormal social attention in ASD. Two key brain regions have emerged as playing important roles: the superior temporal sulcus (Pelphrey, Morris, & McCarthy, 2005b), and the amygdala (Baron-Cohen et al., 2000; Howard et al., 2000). While these are only pieces of a larger network for social cognition that is likely to be impaired in ASD (Damasio & Maurer, 1978; Pelphrey, Adolphs, & Morris, 2005a), they have been the focus of most studies. The amygdala, in particular, has been closely tied to ASD and to abnormal eye gaze in a number of studies. For instance, morphometric studies of postmortem brains have shown that the amygdala is structurally abnormal in ASD (Kemper & Bauman, 1993; Schumann & Amaral, 2006). This structural abnormality of the amygdala is also evident from magnetic resonance imaging (MRI) quantification of living individuals (Howard et al., 2000; Schumann et al., 2004). Individuals with ASD have also been shown to have abnormal amygdala activation when fixating the eyes of faces (e.g., Dalton et al., 2005), buttressing the hypothesis that one of the key neural structures responsible for impaired eye gaze in ASD is the amygdala.

The amygdala in eye gaze

The amygdala has long been known to play critical roles in social behavior (Kluver & Bucy, 1939), emotional learning (LeDoux, 1993), aspects of face processing (Adolphs, Tranel, Damasio, & Damasio, 1994), attention (Holland & Gallagher, 1999), and reward processing (Murray, 2007). Pulling these diverse findings together into a single story has been challenging, but a key feature appears to be evaluating the social meaning of stimuli in a context-dependent way (Adolphs, 2010).

The amygdala plays a central role in processing information about eyes in faces. It is activated by eyes in functional (f)MRI studies (Kawashima et al., 1999; Whalen et al., 2004), and lesions of the amygdala impair eye gaze onto photographs of faces (Adolphs et al., 2005) as well as onto the faces of real people (Spezio, Huang, Castelli, & Adolphs, 2007b). Amygdala damage also impairs social judgments about people's faces within complex scenes (Adolphs & Tranel, 2003). Just as in autism (Baron-Cohen, Wheelwright, Hill, Raste, & Plumb, 2001), subjects with bilateral amygdala lesions are impaired in gleaning the social meaning of eyes alone (Adolphs, Tranel, & Baron-Cohen, 2002).

Most informative have been studies in a rare patient with complete bilateral lesions of the amygdala, known as SM (Adolphs, 2010; Buchanan, Tranel, & Adolphs, 2009). Recent research suggests that SM shows lack of interest in the eyes because her attention is attracted to other regions of the face, such as the nose (Kennedy & Adolphs, 2010) or mouth (Spezio et al., 2007b). In contrast, SM is able to gaze at the eyes in faces with no difficulty when instructed to do so (Adolphs et al., 2005). Kennedy and Adolphs (2010) examined the hypothesis that SM's endogenous interest in eyes is largely intact by restricting SM's visual input to the center of her gaze (“gaze-contingent” viewing), essentially eliminating competing visual information. When viewing faces with this restriction, SM looked at the eyes to the same extent as controls, suggesting that in unrestricted viewing conditions her fixations are captured away from the eyes by other regions of the face. This hypothesis is bolstered by the finding that SM's fixations to the eyes of whole faces are most abnormal in the first fixation, and become more normative in later fixations when endogenous control can take over (Kennedy & Adolphs, 2010). Taken together, these results suggest that the amygdala is normally involved in guiding fixations early on to regions of the face that are socially salient; that is, the eyes.

Comparing ASD and amygdala lesions

While there are thus considerable similarities in how people with ASD and with amygdala lesions fail to look at eyes in faces, there is also reason to suspect that they might do so for different reasons. For instance, Neumann et al. (2006) suggested that individuals with ASD show an abnormal endogenous bias away from the eyes and toward the mouth, rather than an exaggerated sensitivity to the bottom-up saliency of the features of the mouth. Specifically, individuals with ASD looked at the mouth region of “bubbles” faces even when the mouth was low contrast, suggesting an endogenous strategy for allocating visual attention to the mouth region. In contrast, as we reviewed above, it is thought that SM looks less at the eyes because she has abnormally increased exogenous attention capture by the mouth.

To elucidate the possible mechanistic role of amygdala dysfunction in the abnormal social attention of ASD, we undertook direct comparisons with SM. To test the roles of endogenously driven and exogenously driven attentional effects, we examined the effects of task (endogenous effects) and of stimuli (exogenous effects). To date, there have been no direct comparisons of social attention processes in ASD and patients with amygdala lesions. We expected to find a reduction in time spent looking at the eyes in both populations, but for different reasons. We hypothesized that SM would show normal endogenous, but abnormal exogenously driven attention to social features, whereas people with ASD would show the converse impairment. In particular, we expected that SM would fixate socially less salient regions (e.g., the mouth) in stimuli, especially early on, but show a relatively normal modulation of fixation to the eyes when driven by endogenous task demands. By contrast, we expected that individuals with ASD would show a reduction in fixations to the eyes that would be relatively stable across task conditions, due to impaired endogenous attention to social features. By this reasoning, we would expect to find the greatest abnormality in fixations to the eyes in ASD on those tasks for which the eyes are normally highly informative.

Present study

To compare SM to individuals with ASD, we chose to present participants with a variety of real-world social scenes under different task conditions, using a previously validated set of stimuli and task instructions (Birmingham et al., 2008b; Smilek, Birmingham, Cameron, Bischof, & Kingstone, 2006). In particular, we were interested in determining how fixation preferences change during a task that forces observers to process social attention information, relative to more neutral or general tasks. Previous research has validated such an approach with typically developing individuals, demonstrating that when asked to infer where other people in a scene are directing their attention, observers enhance their fixations to the eyes of those people, relative to when they are simply describing a scene (Birmingham et al., 2008b; Smilek et al., 2006). Importantly, this enhanced inspection of the eyes is driven by an understanding that eyes provide important social attention information, subjective reports revealing that observers mention “eye gaze” and “looking” more often when inferring attentional states (even though the instructions do not explicitly direct observers to report on the eye gaze of people in the scene; Smilek et al., 2006).

We chose this task manipulation to compare SM and individuals with ASD to two matched control groups, both because the task manipulation has been validated in previous work (Birmingham et al., 2008b; Smilek et al., 2006) and because it offers a rich interplay between competition for attention driven by the social saliency of stimuli shown in the scenes, and the demands made by the different tasks. Notably, fixations of typical observers to these scenes cannot be explained by visual saliency (Birmingham, Bischof, & Kingstone, 2009).

Finally, we expected a fair degree of variability in fixation patterns, as a function of both the stimulus and the participant. To explore these sources of variance, we conducted analyses exploring viewing patterns as a function of image, and performed exploratory correlational analysis on the ASD group between time spent looking at eyes and various neuropsychological variables.

METHOD

Participants

Subject SM

SM is a 43-year-old (at the time of test) woman with bilateral amygdala damage. She suffers from a rare genetic disorder, Urbach–Wiethe disease (Hofer, 1973), which has led to complete bilateral calcification and atrophy of her amygdalae (Adolphs et al., 1994; Tranel & Hyman, 1990).

SM controls

The controls were 10 neurologically and psychiatrically healthy females of comparable age (mean age: 43.7 years; range 38–51). There was no significant difference between SM and her respective controls in age, verbal performance, or full-scale IQ (p > .05 for each comparison, modified two-tailed t-test; Crawford & Garthwaite, 2002).

ASD subjects

There were nine ASD subjects with high- functioning autism (mean age: 31.6 years; range 20–55). Participants with ASD were recruited from our participant database. All ASD participants met DSM-IV diagnostic criteria for autism or Asperger syndrome, and met the cutoff scores for autism or Asperger syndrome on both the Autism Diagnostic Interview–Revised (ADI–R; Lord et al. 1994) and the Autism Diagnostic Observation Schedule (ADOS, Module 4; Lord et al., 2000). There were two participants in the ASD group for whom the AD I–R was not available; for these participants, ADOS scores alone were used.

ASD controls

The ASD controls were five neurologically and psychiatrically healthy individuals with no family history of ASD (mean age: 32.6 years; range 18–48). ASD controls were recruited through local advertisement, and were screened to be similar in demographic characteristics to the ASD group. There was no significant difference between the ASD group and ASD controls in age, verbal performance, or full-scale IQ (p > .1 for each comparison, Wilcoxon rank-sum test). See Table 1.

TABLE 1.

Participant Demographic and Clinical Data

	SM (n = 1)	SM Controls (n = 10)	ASD (n = 9)	ASD Controls (n = 5)
Age (years)	43.0	43.7 (3.8) [38–51]	31.6 (12.2) [20–55]	32.6 (12.6) [18–48]
Males, Females	0,1	0,10	8,1	5,0
Full Scale IQ	88.0	107.4 (10.8)	108.8 (14.0)	97.0 (17.0)
Performance IQ	95.0	105.9 (11.6)	105.7 (14.5)	93 (22.6)
Verbal IQ	86.0	103 (15.9)	112.7 (17.1)	103 (15.9)
ADI-R Social (Cut-off = 10)			20.0 (4.2) [12–25]
ADI-R Communication (Cut-off = 8)			15.0 (3.6) [10–20]
ADI-R Stereotypy (Cut-off =3)			5.1 (2.3) [2–9]
ADOS Communication (Cut-off = 3/2)			4.0 (1.2) [2.0–6.0]
ADOS Reciprocal Social Interaction (Cut-off = 6/4)			8.3 (3.5) [5.0–17.0]
ADOS Communication + Social (Cut-off = 10/7)			12.3 (4.4) [9.0–23.0]
ADOS Imagination/Creativity			0.6 (0.5) [0.0–1.0]
ADOS Stereotyped behaviors and restricted interests			1.0 (1.0) [0.0–3.0]

Data are given as mean (SD) and, for age, ADI-R and ADOS scores, ranges [in square brackets]. Higher scores on the ADOS and ADI-R are indicative of more severe impairment. For the ADOS, Cut-offs indicate cut-off values for Autism/ASD.

Participants had normal or corrected-to-normal vision, and all gave informed consent to participate in the studies under a protocol approved by the Institutional Review Board of the California Institute of Technology.

Apparatus

Eye movements were measured with an Eyelink 1000 eye tracker (SR Research, Ltd, Hamilton, Ontario, Canada), which has an accuracy of 0.5° of visual angle and sampling rate of 1,000 Hz. We tracked the right eye only.

Stimuli

Full color digital photos were used (Birmingham et al., 2008b). Four sets of five images (20 total) were presented. Example images can be found in Figure 1 Set 1 took place in a waiting area, in which one or three people sat on a couch or a chair; Set 2 took place in another waiting area, again with one or three people sitting on a couch and a chair. Sets 3 and 4 took place in a meeting room, in which one or three people sat at a table. Images were presented on a CRT monitor (120 Hz), using the Psychophysics Toolbox (Brainard, 1997; Pelli, 1997) and Eyelink Toolbox (Cornelissen, Peters, & Palmer, 2002) for Matlab (Mathworks, Inc., Natick, MA, USA). Image size was 28° × 21° at the viewing distance of 80 cm, and image resolution was 1024 × 768 pixels. Four of the images were “filler” images designed to maintain participants' interest, and all contained one person doing something unusual (e.g., wearing a frisbee on his head). These four images were not included in the analysis. Additionally, two images were excluded from analysis because the mouth regions were not visible (because the actors were drinking from a cup).

Figure 1.

Left column: Samples of the social stimuli used. Right column: the different regions of interest (eyes, mouths, heads, bodies, foreground objects, and background.). From top to bottom: sample images from Set 1, Set 2, Set 3, and Set 4.

Procedure

Participants were seated in a dark room, and were placed in a chin rest to ensure a fixed distance from the display computer screen. Participants were told that they would be shown several images, each one appearing for 15 s. They were informed that these images would be repeated and that they would have to complete three different tasks:

1. Neutral task: What kind of room is this? Explain.

2. Describe task: Describe the picture.

3. Social Attention task: Describe where people in the picture are directing their attention. How do you know?

Image and task order was randomized within task blocks of 10 images. Before the experiment, a calibration procedure was conducted. At the beginning of each trial, a fixation cross was displayed in the center of the computer screen in order to correct for drift in gaze position. Participants were instructed to fixate this cross to start the trial. Each trial required fixation calibration within 0.3° of the pre-trial fixation cross. After image presentation, participants gave an oral response to the task instruction (e.g., describe the picture), which was recorded onto a Sony digital MP3 recorder. Participants pressed a button to proceed to the next trial.

As an additional accuracy check for eye-tracker calibration, participants were presented with an image containing nine squares, numbered 1–9. Participants were asked to fixate each square, in order from 1 to 9, as accurately as possible. This image appeared at the beginning, halfway point, and end of the experiment. The fixation plots for this image were further used to verify that there was no drift in the calibration.

Analysis

A total of 14 images per task condition (14 images × 3 tasks=42 total trials) were analyzed. For each image, an outline was drawn around each region of interest (e.g., “eyes”), and each region's pixel coordinates were recorded. We defined the following regions in this manner: eyes, mouths, heads (excluding eyes and mouths), bodies (including arms, torso, and legs), foreground objects (e.g., tables, chairs, objects on the table), and background (e.g., walls, shelves, items on the walls) (Figure 1). Regions were pooled such that there was one composite “eye” region made up of all eye regions, one composite “head” region made up of all head regions, etc. (in cases where the image contained numerous entities).

1. Time spent looking at each region

To determine what regions were of most interest to observers, we computed the total time spent fixating each region, divided by total viewing time (proportion of time in region). To compare SM to her controls (SM controls), we used Crawford and Garthwaite's (2002) modified t-test for comparing an individual's score on a single test with the score of a normative or control sample. Using this approach, we compared the mean proportion of time in a region for SM to that for SM controls. Similar analyses were used to compare SM to the ASD group. To examine whether SM's preference for eyes changed between tasks, we performed paired tests on her fixation time for eyes, paired by image. Finally, to examine the influence of task on attention to eyes for SM controls, we performed an ANOVA on the proportional dwell time for eyes, using a within-subjects factor of task (Neutral, Describe, Social Attention), followed by post hoc pairwise comparisons (Tukey–Kramer multiple comparisons test; alpha=.05).

To compare the ASD group with their control group (ASD controls), the data were submitted to mixed 2 × 3 repeated measures ANOVAs on each region separately with Group (ASD vs. ASD control) as a between-subjects factor and Task (Neutral, Describe, Social Attention) as the within-subjects factor. Pairwise comparisons (Tukey–Kramer multiple comparisons test, alpha=.05) were also performed post hoc to confirm differences between means.

Initial fixation proportion

To determine where observers' initial saccades landed in the visual scene, we computed the number of first, second, third, fourth, and fifth fixations that landed in a region (first five fixations). As with the proportional dwell time data, SM was compared to SM controls and the ASD group by modified t-tests; ASD and ASD controls were compared by mixed ANOVAs, followed by Tukey–Kramer post hoc comparisons (alpha=.05).

RESULTS

SM vs. SM controls

1. Time spent looking at each region

SM spent a considerably greater proportion of viewing time looking at the mouth region than did SM controls (SM: .21, controls: .10, Crawford's modified t-test, t(9)=1.84; p < .05; see Figure 2a). There was also a clear trend for SM to spend a smaller proportion of time looking at the eyes than her controls (SM: .04, controls: .17), although this difference did not achieve significance due to the large variance in the SM controls (Crawford's modified t-test, p > .10).

Figure 2.

Fixations in patient SM. (A) Proportional dwell time for SM and her control group (means and SD) shown for each of the regions of interest. SM showed a trend to look less at the eyes and looked significantly more at the mouth than SM controls (p < .05). (B) Task effects: both SM and SM controls increased their dwell times on the eyes in the Social Attention task relative to the Describe and Neutral tasks (p < .05).

In contrast, SM did increase her dwell time on the eye region of faces when the task involved inferring social attention information (Figure 2b). Paired t-tests confirmed that SM spent more time looking at eyes in the Social Attention task (.10) than in the Neutral task (.02), t(14)=2.20, p < .05 (one-tailed), and the Describe task (.02), t(14)=1.93, p < .05 (one-tailed). For SM controls, an ANOVA on the time proportion for the eye region revealed a similar effect of task, F(2, 18)=16.74, p < .0001, with Tukey–Kramer post hoc comparisons revealing that the eyes were fixated for longer during the Social Attention task (.26) than in the Neutral (.09) and Describe (.16) tasks (p < .05).

2. Initial fixation proportion

SM differed most notably from SM controls in her initial fixations, which were predominantly onto the mouth rather than the eyes (see Figure 3). Analyzed individually (without correction for multiple comparisons), the second and third fixations showed a significantly higher proportion of mouth fixations for SM than for controls; second fixation: Crawford's modified t-test, t(9)=2.90, p < .01; third fixation: t(9)=2.356, p < .05, one-tailed, but this was not significant for subsequent fixations (fixations 4 and 5, p > .05). There was also a trend for SM to look less at the eyes in these initial fixations, although this did not reach significance (p > .10).

Figure 3.

Proportion of initial fixations on eyes and mouth, for SM and SM controls (CONTR). SM committed significantly more of her second and third fixations to the mouth than did SM controls (p < .05).

ASD vs. ASD controls

1. Time spent looking at each region

Our Group × Task ANOVAs of proportional dwell time on each region initially revealed no differences between ASD and controls, as reflected in nonsignificant effects of group, eyes: F(1, 12)=1.32; p > .10; heads: F(1, 12)=4.57, p > .05; mouth: F < 1; bodies: F < 1; foreground objects: F < 1; background: F(1, 12)=1.53, p > .10. This suggested that the ASD and ASD=control groups distributed their fixations across the scene in a broadly similar way. However, there were some hints that the ASD group was looking less at the eyes—but with a very large variance (see Figure 4a).

Figure 4.

Fixations in ASD. (A) Proportional dwell time for each region of interest, for ASD and ASD controls (means and SD). (B) Dwell time broken down by individual stimuli. Sets 3 and 4 produced larger group differences than Sets 1 and 2 (p < .05). (C) Group and task effects for proportion of dwell time on eyes for images in Sets 3 and 4. Whereas ASD controls enhanced their fixations to the eyes in the Social Attention task (p < .05), the ASD group did not.

We also noticed a high degree of variability among the images in terms of time spent looking at the eyes, not an unexpected finding given the complexity and heterogeneity of our stimuli. To investigate fixations for each stimulus separately, we submitted the data for the ASD and ASD controls to a mixed repeated-measures ANOVA with Group, Task, and Image (random factor) as factors. This analysis revealed a significant effect of Image, F(13, 156)=9.27, p < .0001, indicating that time spent looking at the eyes varied among the images (see Figure 4b). This effect alone was not surprising, as images varied on eye region size, and this alone could account for variations in time spent looking at the regions. However, a significant Group × Image interaction, F(13, 156)=2.28, p < .01, could not be explained by variations in region size. Rather, it suggested that certain images produced greater group differences in time spent looking at the eyes than others. Furthermore, a significant Group × Task × Image interaction, F(26, 312)=1.80, p < .05, suggested that this effect of Image depended on the task given to participants, as we explore below. Interestingly, these interactions with Image were absent in the comparison of SM to SM controls (F < 1).

In order to examine these stimulus and task effects more closely, we ranked the images according to how much time the ASD controls spent looking at the eyes (Figure 4b). We noticed that the images that tended to produce longer dwell times in the controls were from Sets 3 and 4. Interestingly, these images had in common one characteristic that differed from the images in Sets 1 and 2: The people in Sets 3 and 4 were seated around a table, whereas the people in Sets 1 and 2 were seated on couches or chairs with their full bodies visible. We constrained our analysis to images in Sets 3 and 4 in order to examine whether these images produced different gaze behavior for the ASD group. We found that for the images in Sets 3 and 4, there was a strong trend for the ASD group to look less at the eyes than ASD controls, F(1, 12)=3.65, p=.08. Furthermore, while controls spent proportionally more time looking at the eyes in the Social Attention task than the other tasks, the ASD group did not. This observation was confirmed by a Group × Task interaction, F(2, 24)=7.12, p < .01, and Tukey–Kramer pairwise comparisons between task conditions within each group (see Figure 4c). That is, for ASD controls, the proportional dwell time on eyes was significantly higher in the Social Attention task (.49) than in the Describe (.30) and Neutral tasks (.17), p < 0.05. In contrast, there was no difference between task conditions for the ASD group (Social Attention: .21; Describe: .18; Neutral: .12, p > .05). In other words, for this subset of images the ASD group failed to enhance their inspection of the eyes when inferring social attention information. This also resulted in a significant group difference for eyes in the Social Attention task (ASD: .21 vs. ASD controls: .49 Tukey–Kramer, p < .05), but no significant group difference for eyes in the Describe task (ASD: .17 vs. ASD controls: .30, Tukey–Kramer, p > .05) or the Neutral task (ASD: .12 vs. ASD controls: .17, Tukey–Kramer, p > .05). This effect was specific to the eyes; that is, there were no group differences in time spent looking at mouths (p > .10).

2. Initial fixation proportion

A Group × Task ANOVA on the initial fixations to eyes for images in Sets 3 and 4 revealed significant group differences in the proportion of second fixations that fell on eyes. The ASD group had fewer second fixations to the eyes than did the ASD controls, ASD: .26, controls: .48; F(1, 12)=4.84, p < .05 (see Figure 5). Further, there was a trend toward a Group × Task interaction, F(2, 24)=2.64, p=.09. This suggested that while the ASD controls tended to commit more of their second fixations to the eyes in the Social Attention task (Social Attention: .67, Describe: .40, Neutral: .37), the ASD group did not (Social Attention: .28, Describe: .22, Neutral: .28), although this was not significant, Tukey–Kramer pairwise comparisons, p > .05. There were no group differences for the mouth region, p > .10.

Figure 5.

Proportion of initial fixations on eyes and mouths, for ASD and ASD controls (CONTR), for images in Sets 3 and 4. The ASD group looked less at the eyes than ASD controls by the second fixation (p < .05).

Intersubject variability

Finally, having examined both effects of stimuli and of task, we explored a third and final source of variability: individual differences in the participants. We found considerable individual differences, both between the two control groups and at an individual level. For instance, the ASD control group, who was predominantly male and with a mean age of 31.6, spent slightly more time on the eyes (.25) than the SM control group (.17), who were all female and with a mean age of 43.7 (although the group difference for eyes was nonsignificant, p > .10). There was also large variation within each group—-for instance, there were subsets of participants in each group who consistently failed to look at the eyes, and others who quite consistently showed a preference to look at the eyes. In the ASD group, we also explored correlations between individual differences in eye gaze and neuropsychological variables. Although exploratory in nature due to our small sample size, we uncovered a highly significant positive correlation between the time spent looking at eyes and scores on the communication subscale of the ADI–R (Pearson's r=.91, t(6)=4.5, p < .025, two-tailed, Bonferroni corrected). That is, participants with ASD who were more impaired in communication (higher scores on ADI–R) spent more time looking at the eyes (see Supplementary Figure A). Conversely, there was a significant negative correlation in the ASD group between dwell time on the mouth and the communication subscale of the ADI–R (Pearson's r=−.83, t(7)=2.7, p < .025, two-tailed, Bonferroni corrected), suggesting that individuals who were more impaired in communication looked less at the mouth (see Supplementary Figure B). Although we interpret these findings with caution, they are broadly consistent with previous findings of correlations between eye and mouth gaze and communication skill (e.g., Norbury et al., 2009).

SM vs. ASD

Although SM was not matched to the ASD group in terms of gender (age and full-scale, verbal, and performance IQ did not differ statistically between SM and the ASD group: Crawford's modified two-tailed t-tests, p > .10), the comparison of fixation patterns between SM and ASD was of particular interest, given our hypotheses about the role of the amygdala in autism. Below, we compare SM to the ASD group on time spent looking at each region, and initial placement of fixations.

1. Time spent looking at each region

SM spent considerably more time looking at the mouth region than did the ASD group, SM: .21, ASD: .07, Crawford's modified t-test, t(8)=2.75, p < .05 (see Figure 6a). The increase in time on the mouth for SM was driven mostly by the Describe task, SM: .31, ASD: .07, t(8)=5.40, p < .001, and less so by the Social Attention task, SM: .21, ASD: .09, t(8)=1.25, p > .10, and the Neutral task, SM: .13, ASD: .05, t(8)=1.42, p=.10. Although there was also a trend for SM to look less at the eyes than the ASD group, this overall difference was not significant, SM: .04, ASD: .17, t(8)=t=−0.95, p > .10, nor was it significantly different in any of the tasks: Social Attention, SM: .10; ASD: .22, t(8)=−0.811, p > .10; Describe, SM: .02; ASD: .18, t(8)=−1.23, p > .10; and Neutral, SM: .02; ASD: .11, t(8)=−0.69, p > .10. SM did not differ from the ASD group on any other region.

Figure 6.

Comparing SM to ASD. Left panel: proportional dwell time for SM and the ASD group (means and SD) shown for each of the regions of interest. SM showed a trend to look less at the eyes and looked significantly more at the mouth than did the ASD group (p < .05). Right panel: proportion of initial fixations on eyes and mouth for SM and ASD. SM committed significantly more of her first, second, and third fixations to the mouth than did the ASD group (p < .05).

2. Initial fixation proportion

The ASD group was not as likely as SM to commit their initial fixations to the mouth (see Figure 6b). Analyzed individually, the first, second, and third fixations showed a significantly higher proportion of mouth fixations for SM than for the ASD group—respectively: Crawford's modified t-test, t(8)=2.08, p < .05; t(9)=4.15, p < .01; t(8)=2.64, p < .05, all one-tailed; and marginally more for subsequent fixations: fixation 4, p=.06; and fixation 5, p=.10. There was also a trend for SM to look less at the eyes than the ASD group in these initial fixations, although this did not reach significance (p > .10).

GENERAL DISCUSSION

Consistent with prior findings with the same stimuli and task (Birmingham et al., 2008b), the controls showed an expected high proportion of fixations onto the faces, and in particular the eyes, of people in the scenes. Three notable sources of variance were apparent. First, viewers looked more at the eyes in the stimuli when the task condition required more social attention. Second, there was variability due to the stimuli, with some drawing gaze more to the eyes than others. Third, there were substantial individual differences in the participants themselves.

SM looked noticeably less at the eyes and instead more at the mouth, findings consistent with prior studies in this individual (Adolphs et al., 2005). Moreover, this impairment was most noticeable on the first few fixations. By contrast, she did significantly increase her gaze to the eyes when the task required more social attention (Figure 2b). This supports our hypothesis that SM had impaired capture of attention by the social features shown in the stimuli, but intact endogenous modulation of eye gaze driven by task demands.

In the ASD group, we also found a nonsignificant, small trend towards looking less at the eye region of faces, but were struck by the very large variability. We therefore next teased apart the possible sources of this variability. First, we analyzed the results per individual stimulus, and found that some images resulted in much larger and more reliable group differences than others. Secondly, we found a significant group by stimulus by task interaction. Those stimuli showing the largest group differences in eye fixations did so because the ASD group failed to increase their eye fixations for those stimuli when the task required social attention. This finding supports our hypothesis that the ASD group had impaired endogenous modulation of eye gaze in the Social Attention task.

In addition to supporting general processing hypotheses about social attention deficits following amygdala lesions and in ASD, our findings urge caution in the interpretation of results that collapse across stimuli and/or tasks. While exploratory in nature, our analysis of interactions with image suggests that social attention deficits in ASD may be constrained to certain stimuli, a finding that warrants further investigation to establish the specific effect. We suggest that future studies pay particular attention to task demands, to differences among stimuli, and to individual differences in the participants. The large individual differences we observed were not entirely unexpected, given that we were presenting complex scenes instead of isolated faces. We were nonetheless surprised at how much variability there was in terms of time spent looking at eyes, even in our control groups, given previous reports in which the preference for eyes was striking and consistent across observers (e.g., Birmingham et al., 2008a, 2008b). It is possible that, as our control groups came from the general population (i.e., they did not come from a relatively homogeneous undergraduate subject pool), factors such as gender, age, IQ, and personality may have contributed to individual differences in preferences for looking at the eyes. Prior studies have shown that the use of eye-gaze information is influenced by the gender (Bayliss et al., 2005) and age of both participant and stimulus face (Slessor, Laird, Phillips, Bull, & Filippou, 2010), as well as aspects of social (e.g., Klin et al., 2002; Speer et al., 2007) and cognitive (Norbury et al., 2009) functioning.

Comparing ASD and amygdala lesions

Given the intense current interest in the amygdala's role in psychiatric illness, and in particular its contribution to the social impairments in autism (cf. the introduction), a particularly valuable aspect of our study was the opportunity for direct comparisons on the same task and stimuli. We found some global similarities in that both SM and the ASD group looked less at the eyes (although this did not reach statistical significance due to the large variance in the controls), but were more struck by the differences. These can be summarized as follows.

First, whereas SM showed a pronounced and significant increase in fixations to the mouth, especially at the earliest fixations, the ASD group showed no such effect. This difference is consistent with the idea that amygdala lesions impair the relative saliency of eyes compared to mouths in faces, whereas ASD has a less specific impairment in allocating social attention.

Second, whereas SM showed an intact modulation of eye gaze by the task, the ASD group showed a notable absence of such task-dependent modulation. Moreover, the effect in ASD was rather specific to certain images, resulting in a significant Group by Task by Image interaction for ASD vs. controls. This finding is consistent with the hypothesis that the amygdala is not essential for endogenous effects driven by task demands and the particular context of the stimuli, whereas ASD shows impairments that are sensitive to precisely these factors.

Third, the amygdala does not appear to be involved in general social orienting, since SM had no reduction in gaze to people and faces. Instead, we suggest that the amygdala is a critical component of a “social saliency map” that specifies the relative importance of social features for allocating attention. This finding would be difficult to determine from studies presenting isolated faces to participants (e.g., Adolphs et al., 2005) and hinged on the complex scenes we used in which many features can compete for attention.

Our conclusions are qualified by our relatively small sample size and large variance. In particular, we could not identify what drove the individual variation in dwell time on the eyes in our control groups; this could have been due to personality differences, age, gender, aspects of cognitive functioning, or any combination of these factors. Future research should examine these variables and their relationship to eye gaze in more detail. We also suggest that researchers explore the effects of stimulus context on social attention. While past research has examined some aspects of scene content on eye gaze (e.g., Birmingham et al., 2008a), it remains an open question what other image characteristics (e.g., social dimensions such as trustworthiness of the people in the image, or the emotional valence and intensity of the image) may enhance the selection of eye-gaze information in neurotypical individuals. This is sure to be an exciting avenue for future research on social attention (e.g., Birmingham & Kingstone, 2009).

Implications for the amygdala's role in autism

Although the data show superficial similarities between our amygdala patient and ASD, the differences are more striking than the similarities. The findings argue that the amygdala is more critical for stimulus-driven (exogenous) social attention but not task effects (endogenous attention modulation), whereas the converse pattern of impairment is featured in ASD. One interpretation of this pattern of impairments is that the amygdala is indeed dysfunctional in autism, but that it is not the source of the dysfunction. Instead, other regions that process context- and task-dependent allocation of attention (plausibly drawing on the prefrontal cortex) may be dysfunctional, and may then pass on abnormal signals to the amygdala, which contains a saliency map driven by both exogenous and endogenous factors. This explanation would predict abnormal amygdala activation in fMRI studies of ASD, but argue that amygdala lesions by themselves do not duplicate the social dysfunction of autism—-a conclusion consistent with the finding that amygdala lesions do not duplicate the social behavior of ASD either (Paul, Corsello, Tranel, & Adolphs, 2010). We thus suggest that future studies on the neural basis of the social impairments of ASD focus on the network of structures with which the amygdala is connected, rather than investigating the amygdala in isolation as the source of social cognitive impairments.