Skip to main content

Some NLM-NCBI services and products are experiencing heavy traffic, which may affect performance and availability. We apologize for the inconvenience and appreciate your patience. For assistance, please contact our Help Desk at info@ncbi.nlm.nih.gov.

NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2009 Jul 22.
Published in final edited form as: Curr Biol. 2008 Jul 22;18(14):1090–1093. doi: 10.1016/j.cub.2008.06.073

Distinct Face Processing Strategies in Parents of Autistic Children

Ralph Adolphs 1, Michael L Spezio 1,2, Morgan Parlier 3, Joseph Piven 3,*
PMCID: PMC2504759  NIHMSID: NIHMS60423  PMID: 18635351

Summary

In his original description of autism, Kanner [1] noted that the parents of autistic children often exhibited unusual social behavior themselves, consistent with what we now know about the high heritability of autism [2]. We investigated this so-called “Broad Autism Phenotype” in the parents of children with autism, who themselves did not have a diagnosis of any psychiatric illness. Building on recent quantifications of social cognition in autism [3], we investigated face processing using the “Bubbles” method [4] to measure how viewers make use of information from specific facial features in order to judge emotions. Parents of autistic children who were assessed as socially aloof (N=15), a key component of the phenotype [5], showed a remarkable reduction in processing the eye region in faces, together with enhanced processing of the mouth, compared to a control group of parents of neurotypical children (N=20), as well as to non-aloof parents of autistic children (N=27, whose pattern of face processing was intermediate). The pattern of face processing seen in the Broad Autism Phenotype showed striking similarities to that previously reported to occur in autism [3], and for the first time provides a window into the endophenotype that may result from a subset of the genes that contribute to social cognition.

Results and Discussion

We used a sensitive and data-driven technique, called “bubbles” [4], to assess how viewers make use of information from faces. The approach is conceptually related to reverse correlation and shows participants only small regions of a randomly sampled face space (Figure 1). Using this technique, it is possible to identify statistically the extent to which specific facial features contribute to judgments about the faces. In our prior studies with this method [3,6,7], we asked subjects to discriminate between two emotions, fear and happiness, because these emotions are distinguished by particular facial features [8], and because there is evidence that their recognition can be differentially impaired following brain pathology [6,9]. To provide comparisons with prior results, we used the identical task here.

Figure 1.

Figure 1

Construction of the stimuli. On the far left is one of the base images we began with (an expression of happiness or of fear from the Pictures of Facial Affect [24]). The base face was decomposed and randomly filtered so as to reveal only small parts of it. The amount of the face revealed was adjusted interactively so as to keep performance accuracy relatively stable throughout the experiment, except for the first few trials. A sample stimulus image is shown on the far lower right.

We contrasted three groups of participants (Table 1): parents who had a child with autism and met criteria for the aloof component of the Broad Autism Phenotype (“BAP+”); those who had a child with autism but were negative for the aloof component (“BAP−”); and those who had a neurotypical child that was not autistic (“Controls”). All three groups had similar mean performance accuracy on the Bubbles Task (82.2%, 82.2%, 83.5% for Controls, BAP−, BAP+, respectively) as well as reaction times (2.0, 1.7, 1.8 sec), and very similar mean number of bubbles (across all spatial frequency bands); moreover, since the bubble locations were randomly and homogeneously distributed, all subjects saw similar parts of the face revealed in the trials, on average. There were no significant differences on any of these measures between groups (all ps>0.1 from uncorrected t-tests).

TABLE 1.

Means and S.D. of the participant groups for age, gender, and full-scale IQ.

Age Gender IQ N
BAP+ 47±8 12M/3F 122±12 15
BAP− 48±7 7M/20F 120±7 27
Controls 44±7 12M/8F 117±9 20

Despite the essentially identical stimuli and overall performances, the three groups differed in their performance strategies. We calculated classification images that revealed the effective information a viewer uses to perform the task. Classification images from the Control group looked similar to those that have been published previously for psychiatrically and neurologically healthy participants, showing substantial use of the eye region of the face (Figure 2a). Comparisons of the classification images revealed statistically significant differences between the three groups (all images thresholded at P<0.001, corrected). Compared to the Control group, both the BAP+ group (Fig. 2b) and the BAP− group (Fig. 2c) made less use of the eyes, an effect most notable for the right eye region. When contrasting the two BAP groups, we found that the BAP+ group made more use of the mouth than the BAP− group (Fig. 3a) whereas the BAP− group made more use of the eye region than did the BAP+ group (Fig. 3b).

Figure 2.

Figure 2

Classification images showing the use of facial information. A: Controls (parents of a child without autism). B: Difference between Controls and BAP+ (the image shows the region of the face used more by controls). C: Difference between Controls and BAP−. All classification images are based on the accuracy with which subjects performed the task, and are statistically thresholded at P<0.001 (corrected). The regions of the face that are visible in the images thus correlate with performance accuracy at P<0.001.

Figure 3.

Figure 3

Classification difference image between BAP groups and between BAP and autism. A: Information used more by BAP+ than by BAP−. B: Information used more by BAP− than by BAP+. For comparison purposes, panels C and D reproduce prior published findings by us, using the identical task and analysis, in people with autism. C: Information used more by autism subjects than by controls. D: Information used more by controls than by autism subjects. Panels C and D from [3]. E: Information used more by BAP+ than autism subjects. F: Information used more by BAP− than autism subjects. The converse classification images of autism-BAP revealed no regions where autism subjects used more information than BAP subjects, for either of the two BAP groups.

Overall, these findings indicate that BAP+ participants did not show the normal pattern of dependence upon the eyes for judgements of emotions. Instead, they relied more heavily on the mouth. The pattern in use of facial information we found here bears a striking similarity to what we have reported previously in individuals with autism [3] (reproduced in Figure 3c,d). Direct difference classification images between the present data and our previously published data from autism subjects [3] showed that both BAP groups still used the eyes more than did autism subjects, although as expected the BAP+ less so than the BAP− group (Fig. 3e,f). Since our groups had different gender ratios (see Table 1 and Methods), we also repeated all the above analyses solely on male participants; the overall pattern of results in the classification images remained significant, verifying that the group differences in the use of facial information that we report were not driven by different gender ratios between the groups (see Supplementary Figure 1).

To provide additional quantification to the results from the classification images we calculated and contrasted SSIM scores, which focus specifically on the relative use of information from the eyes or the mouth in faces (see Methods). The use of information from either the right eye region or the left eye region was highest in the control group, somewhat lower in the BAP− group, and lowest in the BAP+ group (right eye region SSIM: 0.24, 0.22, 0.20; left eye region SSIM: 0.27, 0.26, 0.24), consistent with what would be expected from the classification images. Also consistent with the classification images was the SSIM from the mouth, which was lowest in the controls and higher in the two BAP groups (0.29, 0.33, 0.33) (Figure 4). By contrast, the SSIM values for the whole face or the nose did not differ between groups (0.50, 0.49, 0.49; and 0.46, 0.46, 0.46, respectively). It is intriguing to note the consistent rank-order of SSIM scores for the left and right eye regions seen in Figure 4 as one goes from Controls, to BAP− parents, to BAP+ parents. Such a pattern is consistent with the idea that there is a continuum of genetic liability for autism, expressed in non-autistic relatives in ways that are milder but qualitatively similar to those seen in autism.

Figure 4.

Figure 4

Numerical quantification of the use of facial information. The three bar graph categories plot the use of information from the right eye, left eye, and mouth region of the face. The y-axis plots the structural similarity metric (SSIM; see Methods), a measure of the degree to which information was used from a specific region of interest on the face. White bars: Controls. Light gray bars: BAP−. Dark Gray bars: BAP+. Means and SEM are shown for each subject group.

To produce a summary measure that captured the pattern apparent in the classification images, we took the maximum of the SSIM for the two eye regions, and divided it by the SSIM for the mouth region. We found statistically significant differences on this summary measure: the control group had the highest value, followed by the aloof-negative group, followed by the aloof-positive group. Non-parametric contrasts confirmed the reliability of this finding: Controls differed from aloof-positive participants (P<0.05, one-tailed Mann-Whitney U test, corrected for multiple comparisons) and from aloof-negative participants (P<0.05) (the contrast between aloof-negative and aloof-positive groups failed to achieve statistical significance).

The present study complements findings in siblings of autistic infants, which have indicated disproportionate gaze onto the mouth when interacting with people, even in the absence of any later development of autism [10] . Eye tracking studies of adolescents who had autistic siblings have found reduced eye fixation compared to controls, and both the autistic individuals and their non-autistic siblings had abnormally reduced amygdala volumes [11]. Possible pathology of the amygdala has been one among several hypotheses for social dysfunction in autism [12-14], although its precise role remains debated [15]. We have previously reported that neurological subjects with focal amygdala lesions fail to make use of the eye region of faces, showing classification images on the same Bubbles Task that are notable for an absence of the eyes [16]. However, while subjects with amygdala lesions fail to make normal use of the eyes, they do not appear to make increased used of the mouth. The present findings in parents of autistic children, as well as the sibling studies noted above [10,11] and findings in individuals with autism [3,7,14,17-21], are consistent with the hypothesis that there is active avoidance of the eyes in autism, but our findings also fit the hypothesis that there is reduced attraction to the eyes and/or increased attraction to the mouth. Of course, these hypotheses are not mutually exclusive; additional studies will be required to determine their relative importance.

This is the first study to quantify a specific cognitive endophenotype defining a distinct face processing style in the parents of individuals with autism. We found a pattern of face processing (increased use of the mouth and diminished use of the eyes) similar to that seen in autistic individuals. Our finding emerged from a difficult and sensitive task, and it will be important to extend this also to whole faces in the real world, an issue that could be probed further with methods such as eyetracking. The face processing style we found appears to segregate with a specific component of the Broad Autism Phenotype, aloof personality; further studies with larger samples will be needed to explore possible correlates with other dimensions of the BAP, such as rigid personality. Taken together, the findings provide further support for a Broad Autism Phenotype and suggest avenues for isolating the genes that influence social behavior in autism.

Experimental Procedures

Participants

All participants were neurologically and psychiatrically healthy adults. Specifically, none had any diagnosis of any autism spectrum disorder, including Asperger's Syndrome or PDD-NOS, as verified by detailed clinical assessment. We initially enrolled participants in two main groups: Fifty-three parents who had a child with a diagnosis of DSM-IV autistic disorder (and also meeting criteria on the ADI-R and the ADOS), and twenty parents who had a typically-developing child and no first-degree relatives who had autism (“Controls”). We note that none of our prior publications concerned the children of any of the parents in the present study (who were younger and lower functioning than autistic subjects we have tested previously [3,7,20]). The parents who had a child with autism were subsequently assessed in detail for the Broad Autism Phenotype [22,23] as described further in the supplementary materials, and classified on the basis of whether they had reliable evidence of ‘aloof personality’ (“BAP+” group) or whether they had no evidence of ‘aloof personality’ (“BAP−” group), using a consensus rating (2 independent raters) from videotaped interviews of both the subject and his/her spouse. Assessing the aloof personality feature of the Broad Autism Phenotype has previously been shown to be reliable [22,23], and in our study we had an inter-rater reliability of 85% for every component trait of the BAP.

On the basis of the above criteria, a total of sixty-two subjects entered the next phase of the study (42 parents of children with autism and 20 control parents) (see Table 1). There were no significant differences in age or IQ between any groups (Ps > 0.2, t-test for unequal variances). Gender ratios differed significantly between the three groups (Chi-square (2) = 10.9, P<0.005) and between the BAP+ and BAP− group (Fisher's exact test, P=0.0014), making gender an important factor to explore in future studies with larger sample sizes. Groups did not differ with respect to distribution of race, years of education, or socioeconomic status. All participants had normal or corrected-to-normal vision, and performed in the normal range on tasks of basic visual discrimination (the Benton Line Orientation and Benton Facial Recognition tasks; there were no group differences on these tasks). All subjects had given informed consent as approved by the Institutional Review Boards at the California Institute of Technology and the University of North Carolina.

Stimuli and Task

Participants were tested one at a time. Stimuli were 4 cropped (256 × 256 pixel) facial expressions of emotion selected from Paul Ekman's “Pictures of Facial Affect” [24], each of a different posing participant, and balanced for gender and facial expression (2 fearful, 2 happy), spatially normalized to align their features and with similar power across all spatial frequencies. The stimuli were randomly filtered using the “bubbles” method [4] (see Figure 1), and shown on a 19-inch flat-screen monitor with an eye-to-screen distance of ca. 24 inches. Participants were instructed to respond as quickly as possible with a button press to categorize the stimulus as “happy” or “afraid”, and to guess if they could not make up their mind. They took breaks as needed when a periodic “pause” screen came up, and indicated that they were ready to proceed with the experiment by pushing another “go” button. A given trial lasted the time it took participants to decide whether the face showed fear or happiness, for a maximal decision time of 10 seconds following image onset. All participants completed 512 trials.

“Bubbles” faces showed randomly revealed regions of an underlying whole face, as previously described [3,4]. Briefly, on each trial, a randomly selected Ekman face image was first decomposed into spatial frequencies and filtered with a number of Gaussian filters (“bubbles”) whose centers were randomly distributed across the image. The number of bubbles was adjusted for each participant on a trial-by-trial basis in order to maintain a relatively stable performance accuracy around 80% throughout the experiment; all participants required a similar mean number of bubbles to reach this performance criterion. We began the experiment with trials in which a relatively large number of bubbles was used, making the first few trials easy and encouraging normal face processing. Note that bubbles were allowed to overlap, increasing the amount of the face revealed beyond the size of a single bubble.

Analysis

Data from the Bubbles Task were analyzed as previously described [4,25] with some modification [3]. Analyses determined which regions of the face associated with correct emotion discrimination by summing the trial-specific bubbles masks across all correct trials and across all incorrect trials, yielding a “correct” and an “incorrect” bubbles mask. We then subtracted, for each spatial frequency level, the normalized incorrect from the normalized correct mask, resulting in a difference mask. In order to select regions of statistically significant difference for the difference mask, we converted all pixel values into Z-scores relative to that mean and standard deviation, and then subjected this Z-scored classification image to cluster tests, setting a threshold t = 2.5 and a significance P = 0.001. This resulted in a diagnostic image, showing which features of the face a participant used significantly more, on average, during the behavioral task.

Further quantification of individual participants' reliance on the eyes and mouth

In addition to the group analyses described above, we sought to quantify each individual's reliance on the eyes and mouth during emotion judgment. We estimated the effective strength of the appearance of each region of interest (i.e., eyes and mouth) in an individual's diagnostic image, using a metric known as the Structural SIMilarity (SSIM) index [26], a quantitative estimate of the similarity between two images that corresponds closely to similarity judgments by human observers. SSIM values were calculated between each individual's diagnostic image and the corresponding base image, for each specified region on the face.

Supplementary Material

01

Acknowledgements

The authors thank the families who participated in our study. The work was supported in part by grants from NIMH to J.P. (grants U54 MH66418 and R01MH077843), and from the Simons Foundation to R.A.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Supplemental Data

One Supplementary Figure and a Supplementary Methods description accompanies this paper.

References

  • 1.Kanner L. Autistic disturbances of affective contact. Nervous Child. 1943;2:217–250. [PubMed] [Google Scholar]
  • 2.Gupta AR, State MW. Recent advances in the genetics of autism. Biol Psychiatry. 2007;61:429–37. doi: 10.1016/j.biopsych.2006.06.020. [DOI] [PubMed] [Google Scholar]
  • 3.Spezio ML, et al. Abnormal use of facial information in high-functioning autism. J Autism Dev Disord. 2007;37:929–39. doi: 10.1007/s10803-006-0232-9. [DOI] [PubMed] [Google Scholar]
  • 4.Gosselin F, Schyns PG. Bubbles: a technique to reveal the use of information in recognition tasks. Vision Res. 2001;41:2261–71. doi: 10.1016/s0042-6989(01)00097-9. [DOI] [PubMed] [Google Scholar]
  • 5.Losh M, Piven J. Social-cognition and the broad autism phenotype: identifying genetically meaningful phenotypes. J Child Psychol Psychiatry. 2007;48:105–12. doi: 10.1111/j.1469-7610.2006.01594.x. [DOI] [PubMed] [Google Scholar]
  • 6.Adolphs R, et al. A mechanism for impaired fear recognition after amygdala damage. Nature. 2005;433:68–72. doi: 10.1038/nature03086. [DOI] [PubMed] [Google Scholar]
  • 7.Spezio ML, et al. Analysis of face gaze in autism using “Bubbles”. Neuropsychologia. 2007;45:144–51. doi: 10.1016/j.neuropsychologia.2006.04.027. [DOI] [PubMed] [Google Scholar]
  • 8.Smith ML, et al. Transmitting and decoding facial expressions. Psychological Science. 2005;16:184–189. doi: 10.1111/j.0956-7976.2005.00801.x. [DOI] [PubMed] [Google Scholar]
  • 9.Adolphs R. Recognizing emotion from facial expressions: psychological and neurological mechanisms. Behavioral and Cognitive Neuroscience Reviews. 2002;1:21–61. doi: 10.1177/1534582302001001003. [DOI] [PubMed] [Google Scholar]
  • 10.Merin N, et al. Visual fixation patterns during reciprocal social interaction distinguish a subgroup fo 6-month-old infants at-risk for autism from comparison infants. J Autism Dev Disord. 2007;37:108–121. doi: 10.1007/s10803-006-0342-4. [DOI] [PubMed] [Google Scholar]
  • 11.Dalton KM, et al. Gaze-fixation, brain activation, and amygdala volume in unaffected siblings of individuals with autism. Biological Psychiatry. 2007;61:512–520. doi: 10.1016/j.biopsych.2006.05.019. [DOI] [PubMed] [Google Scholar]
  • 12.Baron-Cohen S, et al. The amygdala theory of autism. Neuroscience and Biobehavioral Reviews. 2000;24:355–364. doi: 10.1016/s0149-7634(00)00011-7. [DOI] [PubMed] [Google Scholar]
  • 13.Bauman M, Kemper TL. Histoanatomic observations of the brain in early infantile autism. Neurology. 1985;35:866–874. doi: 10.1212/wnl.35.6.866. [DOI] [PubMed] [Google Scholar]
  • 14.Nacewicz BM, et al. Amygdala volume and nonverbal social impairment in adolescent and adult males with autism. Arch Gen Psychiatry. 2006;63:1417–1428. doi: 10.1001/archpsyc.63.12.1417. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Amaral DG, Bauman MD, Schumann CM. The amygdala and autism: implications from non-human primate studies. Genes, Brain and Behavior. 2003;2:295–302. doi: 10.1034/j.1601-183x.2003.00043.x. [DOI] [PubMed] [Google Scholar]
  • 16.Adolphs R, et al. A mechanism for impaired fear recognition after amygdala damage. Nature. 2005;433:68–72. doi: 10.1038/nature03086. [DOI] [PubMed] [Google Scholar]
  • 17.Pelphrey KA, et al. Visual scanning of faces in autism. Journal of Autism and Developmental Disorders. 2002;32:249–261. doi: 10.1023/a:1016374617369. [DOI] [PubMed] [Google Scholar]
  • 18.Klin A, et al. Visual fixation patterns during viewing of naturalistic social situations as predictors of social competence in individuals with autism. Arch Gen Psychiatry. 2002;59:809–16. doi: 10.1001/archpsyc.59.9.809. [DOI] [PubMed] [Google Scholar]
  • 19.Dalton KM, et al. Gaze fixation and the neural circuitry of face processing in autism. Nature Neuroscience. 2005;8:519–526. doi: 10.1038/nn1421. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Neuman D, et al. Looking you in the mouth: abnormal gaze in autism resulting from impaired top-down modulation of visual attention. Social Cognitive and Affective Neuroscience. 2006;1:194–202. doi: 10.1093/scan/nsl030. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Kylliainen A, Hietanen JK. Skin conductance responses to another person's gaze in children with autism. Journal of Autism and Developmental Disorders. 2006;36:517–525. doi: 10.1007/s10803-006-0091-4. [DOI] [PubMed] [Google Scholar]
  • 22.Piven J, et al. Broader autism phenotype: evidence from a family history study of multiple-incidence autism families. Am J Psychiatry. 1997;154:185–90. doi: 10.1176/ajp.154.2.185. [DOI] [PubMed] [Google Scholar]
  • 23.Piven J, Landa R, Santangelo S, Jacobi D, Childress D. Personality and language characteristics in parents from multiple-incidence autism families. American Journal of Medical Genetics. 1997;74:398–411. [PubMed] [Google Scholar]
  • 24.Ekman P, Friesen W. Pictures of facial affect. Consulting Psychologists Press; Palo Alto, CA: 1976. [Google Scholar]
  • 25.Schyns PG, Bonnar L, Gosselin F. Show me the features! Understanding recognition from the use of visual information. Psychological Science. 13:402–409. doi: 10.1111/1467-9280.00472. [DOI] [PubMed] [Google Scholar]
  • 26.Wang Z, et al. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing. 2004;13:600–612. doi: 10.1109/tip.2003.819861. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

01

RESOURCES