Perceiving gaze from head and eye rotations: An integrative challenge for children and adults

Abstract

Gaze is an emergent visual feature. A person’s gaze direction is perceived not just based on the rotation of their eyes, but also their head. At least among adults, this integrative process appears to be flexible such that one feature can be weighted more heavily than the other depending on the circumstances. Yet it is unclear how this weighting might vary across individuals or across development. When children engage emergent gaze, do they prioritize cues from the head and eyes similarly to adults? Is the perception of gaze among individuals with autism spectrum disorder (ASD) emergent, or is it reliant on a single feature? Sixty adults (M=29.86 years-of-age), thirty-seven typically-developing children and adolescents (M=9.3 years-of-age; range=7–15), and eighteen children with ASD (M=9.72 years-of-age; range=7–15) viewed faces with leftward, rightward, or direct head rotations in conjunction with leftward or rightward pupil rotations, and then indicated whether the face was looking leftward or rightward. All individuals, across development and ASD status, used head rotation to infer gaze direction, albeit with some individual differences. However, the use of pupil rotation was heavily dependent on age. Finally, children with ASD used pupil rotation significantly less than typically-developing (TD) children when inferring gaze direction, even after accounting for age. Our approach provides a novel framework for understanding individual and group differences in gaze as it is actually perceived—as an emergent feature. Furthermore, this study begins to address an important gap in ASD literature, taking the first look at emergent gaze perception in this population.

To discriminate another person’s gaze, the visual system utilizes information from the entire face, not just the eyes (Cline, 1967; Murayama & Endo, 1984; Otsuka, Mareschal, Calder, & Clifford, 2014; Wollaston, 1824). Perceived gaze is thus an emergent visual feature, and can vary depending on how the rotation of the eyes, the head, and other facial features are integrated. For example, different head rotations can make identical eye gazes appear to be directed at different points in space, or make two different eye gazes appear identical (Sweeny & Whitney, 2017). It is only by focusing on gaze at this emergent level that people may fully understand its mechanisms and development. Yet examinations of gaze in childhood rarely take this approach and instead focus on perception of the eyes alone. The current investigation aims to address this gap, asking for the first time how individuals balance and prioritize information from the head and eyes to determine the direction of a person’s attention, both in childhood and adulthood.

The perception of eye gaze has been studied extensively among typical adults and children. Adults utilize specialized mechanisms to precisely discriminate the rotation of a person’s eyes (Calder, Cassel, Jenkins, & Clifford, 2008). Direct gaze is detected faster than averted gaze (Senju, Kikuchi, Hasegawa, Tojo, & Osanai, 2008), it uniquely captures visual attention (Senju & Hasegawa, 2005), and people are biased to perceive direct gaze (Mareschal, Calder, & Clifford, 2013; Yumiko Otsuka et al., 2014). Infants look preferentially at faces with direct eye gaze (Farroni, Csibra, Simion, & Johnson, 2002), and even though core mechanisms of eye gaze perception are intact by around six to eight years-of-age (Vida & Maurer, 2012), they tend to be refined throughout childhood (Doherty, Anderson, & Howieson, 2009; Mareschal, Otsuka, Clifford, & Mareschal, 2016). Atypical development of these mechanisms has important consequences. Infrequent exposure to eye gaze early in life can disrupt deployment of spatial attention during communication (Senju et al., 2015), and in the case of autism spectrum disorder (ASD), may contribute to difficulties with mutual eye contact (Sigman, Mundy, Sherman, & Ungerer, 1986), and joint attention (Kasari, Freeman, & Paparella, 2006; Mundy, Sigman, & Kasari, 1994). Yet despite the clear importance of gaze, few research studies with adults, and especially typically-developing children and those with ASD, have evaluated it as it is commonly seen, as an emergent feature.

Emergent gaze results from two complementary visual analyses. First, the rotation of the head is gathered by evaluating the symmetry of features like the outline of the head (Wilson, Wilkinson, Li-Ming, & Castillo, 2000) or internal features, like the nose (Langton, Honeyman, & Tessler, 2004). Second, or in parallel, the rotation of the pupils within the apertures of the eyes is gathered. A unique metric of gaze is then calculated by combining these part-based indices, often producing a percept that differs from that seen in any feature alone. Figure 1A illustrates this process using a configuration in which perceived gaze is pulled in the direction of the head’s rotation (Wollaston, 1824).

Figure 1.

(A) Two faces may appear to have different gaze directions even when the rotations of the irises/pupils within the apertures of each pair of eyes are identical. (B) The twelve combinations of head and pupil rotations in our stimulus set. Each row depicts a unique pupil rotation (row 1: 25%; row 2: 5%; row 3: −5%; row 4: −25% rotation). Each column depicts a unique head rotation (column 1: −8°; column 2: 0°; column 3: 8°).

Although this integrative process can be quite linear (at least with adults; Otsuka, Mareschal, & Clifford, 2016), it is also known to be flexible at the group level, with non-linear weighting for particular combinations of head and eye rotations (Sweeny & Whitney, 2017) or according to each feature’s salience or visibility (Florey, Dakin, Clifford, & Mareschal, 2015; Florey, Clifford, Dakin, & Mareschal, 2016). Flexible weighting across changing circumstances makes sense, but how variable might this process be on an individual basis? A few researchers have speculated that weighting of head and eye cues may vary across individuals or among special populations (Mareschal et al., 2016; Otsuka et al., 2014). Yet to our knowledge this hypothesis has never been tested. Additionally, recent work has shown that this integrative process occurs even at 4–6 months of age (Nakato, Otsuka, Konuma, & Kanazawa, 2009; Otsuka, Ichikawa, Clifford, Kanazawa, & Yamaguchi, 2016). But even though emergent gaze is clearly operative during childhood, little is known about how it compares to integration among adults or how children, both typically- and atypically-developing, might vary in the way they prioritize information from the head or eyes.

Here we examined emergent gaze using a framework that not only allowed us to evaluate how children and adults combine head and eye rotations, at the group level, but also the extent to which individuals within these groups weight information from these features. We expected that, as groups, both adults and typically-developing children would engage emergent gaze. Our goal was to examine the possibility that not even adults would weight these cues perfectly equally. At the very least, we expected individual differences, with some adults prioritizing the head and others prioritizing the eyes. The few studies on emergent gaze with infants (Nakato, Otsuka, Konuma, & Kanazawa, 2009; Otsuka, Ichikawa, Clifford, Kanazawa, & Yamaguchi, 2016) simply indicate that they can perceive it; thus, we were uncertain as to whether children would weight gaze cues similarly to adults. We expected that typically-developing children would exhibit individual differences in prioritizing one cue over another, perhaps more strongly than adults.

We developed our framework for measuring individual differences in gaze integration with the hope that it would be useful for research among children known to struggle with gaze perception, such as in the case of ASD. Face perception deficits in ASD are particularly strong with regard to interpreting information from the eyes (Riby, Doherty-Sneddon, & Bruce, 2009; Rutherford & McIntosh, 2007; Wolf et al., 2008), especially when gaze is direct (Klin, Jones, Schultz, Volkmar, & Cohen, 2002; Pelphrey et al., 2002; Rice, Moriuchi, Jones, & Klin, 2012). Diminished neural response to gaze in infancy can even predict ASD diagnosis at 36 months-of-age (Elsabbagh et al., 2012), and atypical gaze processing has been demonstrated even among high-functioning adolescents with ASD (Freeth, Chapman, Ropar, & Mitchell, 2010). Research on gaze perception is thus essential to understanding social impairments in autism. Yet understanding of how gaze is perceived in autism is surprisingly incomplete since, up to now, research has focused exclusively on perception of gaze from the eyes. This investigation thus includes a preliminary evaluation of emergent gaze perception in children with ASD. We predicted that children with ASD would struggle to integrate information from the head and eyes at least into adolescence. Unlike typically-developing children, we expected children with ASD to judge gaze primarily based on head rotation. By taking this approach, our goal was to provide insight into the flexibility of this understudied process both among children and adults, and to pave the way for research among children known to struggle with interpreting eye gaze.

Method

Observers.

Sixty adults (M_age = 29.86; SD_age = 8.56, 31 females), thirty-seven typically-developing (TD) children (M_age = 9.3 years; SD_age = 2.48, 12 females), and eighteen children with ASD (M_age = 9.72; SD_age = 2.65, 3 females) gave informed consent to participate in the experiment. To be conservative, we obtained a sample size of TD children similar to one in a recent investigation of emergent gaze during infancy (Otsuka et al., 2016). We tested children between 7–15 years of age for several reasons. First, vision is coarse in early childhood, with temporal and spatial perception developing up to the age of seven (Ellemberg, Lewis, Liu, & Maurer, 1999; Farzin, Rivera, & Whitney, 2010, 2011). Second, although holistic perception of faces (presumably a related process; Nakato et al., 2009) is at least in place at the age of six (Markham & Adams, 1992), perception of global visual features develops gradually throughout childhood (Dukette & Stiles, 2001; Harrison & Stiles, 2009; Narasimhan & Giaschi, 2012; Parrish, Giaschi, Boden, & Dougherty, 2005), even into adulthood (Scherf, Berhmann, Kimchi, & Luna, 2009), and processing of eye gaze develops at least until the age of 11 (Mareschal et al., 2016). Testing ages 7–15 thus allowed us to measure emergent gaze perception during a potentially sensitive developmental window among children whose more basic visual mechanisms were presumably in place. All children in the ASD sample had a previous diagnosis of ASD, and 15 parents completed the Social Responsiveness Scale (SRS) for their child. The SRS provides a quantitative measure of traits associated with autism among children and adolescents between four and eighteen years-of-age (Bölte, Poustka, & Constantino, 2008). All SRS T-scores in our sample were above 65 (M = 79.4, SD = 4.9). All experimental protocols were approved by the University of Denver IRB.

Stimuli.

We used a set of 12 computer-generated faces from a previous investigation (Sweeny & Whitney, 2014). First, we created heads with leftward, direct, and rightward rotations (−8°, 0, and +8°). Next, we generated four horizontal pupil rotations (−25%, −5%, +5%, +25%) using a head with a direct rotation. These values reflect the percentage, and not the degrees, of a pupil’s rotation within the eye opening of a 3D head. A value of +25% indicates that, relative to its position when gaze is direct, the outside edge of the iris has been rotated 25% of the distance to the edge of the eye aperture. The 20% steps between pupil rotations reflected ~5.6° of angular rotation.

We then used Photoshop to extract each pair of rotated pupils (the iris and surrounding aperture of the eyeball, the sclera, but excluding surrounding eye contours), which we then superimposed onto each rotated head. Contours immediately outside the eye aperture varied with the rotation of the head. The face set included twelve faces with independently-varied head and pupil rotations, but the rotation, size, and shape of the eye apertures remained fixed (Figure 1B). This approach allowed us to isolate and measure the pure effect of attraction from head rotation on perceived gaze without an ongoing repulsive effect from changing eye apertures¹. We schematized the faces so that each was equated in terms of low-level visual information, thus only geometric information conveyed rotation.

Procedure.

Observers viewed faces on a 17” laptop screen, one at a time, and indicated whether each face appeared to be looking to the observer’s left or right. Each face (2.56° × 2.31°) appeared at the center of the screen against a white background until the observer responded by pointing, saying “left” or “right,” or using the keypad. A blank screen then appeared until the experimenter initiated the next trial. Children and adults were encouraged to sit 57-cm from the screen. Each observer viewed each of the twelve head-eye combinations twelve times (order was randomized for each observer), for a total of 144 trials across 3 blocks. Three TD observers completed only 120 trials, as did two ASD observers. One TD observer completed only 24 trials. The experimenter did not provide feedback.

Results

Our objective was to quantify the extent to which perception of gaze depended on the rotation of the head, the pupils, or both. We were especially interested in how weighting of information from the head and pupils varied across individuals, with age, and whether an observer had a diagnosis of autism spectrum disorder. We first visualized our data by calculating the average “rightward” judgment as a function of group (adults, TD, ASD), and the 12 combinations of head and pupil rotation. Figure 2 depicts this descriptive approach to visualizing our data (note that we analyzed our data with a multi-level logistic regression, described below). Each line corresponds to one pupil rotation, and indicates the proportion of rightward responses (y-axis) for faces with that pupil rotation across changes in head rotation (x-axis). Each line has a positive slope, suggesting that, regardless of the pupil rotation, gaze was attracted to the rotation of the head (e.g., gaze appeared more rightward when the head was rotated to the right). Spacing between the lines indicates that perceived gaze was also influenced by pupil rotation (e.g., regardless of the head’s rotation, gaze appeared more rightward when the pupils were rotated to the right).

Figure 2.

Data from adults (A), typically-developing children (B), and children with ASD (C). Each panel illustrates how the proportion of “rightward” judgments varied as a function of the rotation of the head (three values along the x-axis) and the rotation of the pupils (four shapes depicted in each legend). The purpose of this figure is to provide a direct look at our data, at the group level, prior to the multi-level logistic regression. Note that the spatial layout of the twelve head-pupil combinations in each panel maps directly to the layout of faces in Figure 1B.

Multi-level logistic regression:

We analyzed our data using a multi-level logistic model (Bates, Mächler, Bolker, & Walker, 2015). This allowed us to quantify individual differences and examine the effect of head and pupil rotation across our groups and ages, while pooling our data for maximum statistical power. We performed the regression in R (version 3.4.1) using the lme4 package (version 1.1–19). We fit two models. Model 1 predicted individuals’ change in “rightward” judgments with fixed effects for head rotation, pupil rotation, and their interaction, and random effects (i.e., individual differences from the group-level fixed-effects) for head rotation and pupil rotation (see Supplementary Materials for details of model-fitting)². We included no assumptions about group membership in Model 1 so that, if group-level differences did emerge, we would know that this was not the result of imposing these categories onto our dataset in advance. The regression weights (or β values) fit in this model indicate the extent to which head or pupils influence judgments of gaze direction. Values near zero indicate no influence on gaze perception, while increasingly positive values indicate a strong positive (and non-linear) relationship between these continuous variables and the binary rightward/leftward gaze judgment. Fitting Model 1 revealed significant effects for head rotation (β = 2.67, SE = 0.18, p < 2e-16), pupil rotation (β = 2.51, SE = 0.18, p < 2e-16), but not their interaction (p = 0.99). To examine whether the use of head rotation or pupil rotation varied across adults, typically-developing children, and children with ASD (despite the regression not specifying these groups), we first calculated each individual’s regression weights for the use of head rotation and for the use of pupil rotation.

Figure 3 shows each observer’s β values for the effect of head rotation and pupil rotation on gaze perception (adults, TD, and ASD are shown separately, though note that Model 1 included no regressors assuming that these groups were different). This figure illustrates the considerable individual differences in the way observers integrated head and pupil rotations in our task. Consider Figures 3A and 3B; if all adults and children prioritized information from the pupils (or head), data points would have been clustered in the upper left (or bottom right) corner of each scatterplot. Instead, some observers used the head but barely used the pupils, or vice versa, and most observers used some combination of the two. Figure 3C clearly illustrates our finding that, overall, individuals with ASD made judgments of gaze direction with little regard for the rotation of the pupils. The sizes of the dots in each panel correspond to the ages of observers. As is visually apparent, use of the pupils for gaze discrimination increases with age, independent of ASD status. But note, also, that even within the ASD group, the individuals who tended to use the pupils for their judgments were also the oldest in the sample.

Figure 3.

Beta weights from a multi-level logistic regression illustrate the extent to which each adult (A), typically-developing child (B), and child with ASD relied on both head rotation (x-axis) and pupil rotation (y-axis) to make binary judgments of gaze direction. Values were estimated using Model 1. Values near zero (in either dimension) indicate that a particular cue had little-to-no impact on gaze judgment. For example, the clustering of Beta values among children with ASD illustrates our finding that, as a group, they did not base their gaze judgments on pupil rotation, while they had normal variability in the usage of head rotation information. The size of each dot corresponds to each observer’s age, and the pattern of dot sizes across the panels illustrates that pupil use increased across development.

The average of individuals’ β weights for adults, TD children, and children with ASD for head rotation were 2.42, 2.71, and 2.71, respectively, and for pupil rotation were 3.12, 2.07, and 0.85, respectively. These values, and the visual appearance of the scatterplots in Figure 3 suggest significant differences across our groups as a function of age and ASD status, despite neither variable being included in Model 1. To properly account for, quantify, and test these differences, we fit a second multi-level logistic regression including these variables.

Model 2 predicted change in “rightward” judgments with fixed effects for head rotation, pupil rotation, and their interactions with age and ASD status. Again, we found significant effects of head rotation (β = 2.93, SE = 0.39, p = 7e-14) and pupil rotation (β = 1.42, SE = 0.32, p = 9e-6). This new analysis also indicated a significant interaction between pupil rotation and ASD status (β = −1.27, SE = 0.41, p = 0.002), as well as an interaction between pupil rotation and age (β = 0.063, SE = 0.013, p < 1e-6). These interactions indicate that usage of the pupils to discriminate gaze direction increased with age, regardless of ASD status, and that individuals with ASD used pupil rotations less than typically-developing individuals, even after accounting for age. While null results should be interpreted with caution, these findings also suggest that the use of head rotation in gaze judgment was not related to age (p = 0.40) or ASD status (p = 0.99). Figures 4 and 5 provide a window into the individual differences in the use of head and pupil rotation estimated in Model 2. These figures depict the logistic fits for each observer’s change in rightward judgments as a function of changes in head rotation (Figure 4) and pupil rotation (Figure 5). The slope of each observer’s fit is similar to his or her β value for head or pupil use in Figure 3 (e.g., the few observers with flat slopes are easy to find in the scatterplots; though note Figure 3 reflects the estimates of Model 1, and Figures 4 and 5 reflect the estimates from Model 2). Groups fits are shown as well (black, red, and blue lines). We calculated the just-noticeable-difference (JND) for each group fit in order to obtain an effect size estimate of each group’s threshold (i.e., peak sensitivity) for discriminating changes in head and pupil rotation. Adults, TD observers, and observers with ASD had similar JNDs for head rotation, requiring 3.46, 3.13, and 3.13 degrees of rotation for their perception of gaze to reliably change. In contrast, adults had a just-noticeable-difference of 8.29% for pupil rotations in our task, whereas TDs had a JND of 13.7% and observers with ASD had a JND of 35.7%.

Figure 4.

Logistic fits illustrate how changes in head rotation influenced the proportion of rightward judgments for individual observers (gray lines) and groups (bold black, red, and blue lines). Fits are from Model 2. Fits are shown separately for adults (A), typically-developing children (B), and children with ASD (C). Note that these fits reflect an inferred continuous relationship between perceived gaze and the three discrete head rotations in our stimulus set.

Figure 5.

Logistic fits illustrate how changes in pupil rotation influenced the proportion of rightward judgments for individual observers (gray lines) and groups (bold black, red, and blue lines). Fits are from Model 2. Fits are shown separately for adults (A), typically-developing children (B), and children with ASD (C). These fits reflect an inferred continuous relationship between perceived gaze and the four discrete pupil rotations in our stimulus set.

Finally, we evaluated the reliability of our measurement of individual differences by conducting a split-half reliability test. For each participant, we randomly selected half of their data, fit a logistic GLM with factors for head rotation and pupil rotation, and then calculated the mean choice likelihood of the other half of the data using the estimated betas for head and pupil rotation. For each participant, we randomly selected half of their data and fit a logistic GLM to those data with factors for head rotation and pupil rotation. The logistic model produces values for each trial that indicate the likelihood of the data on that trial given the estimated parameters. We used this quantity to assess reliability by quantifying the likelihood of the out-of-sample data, given the in-sample parameters, for example by calculating the mean likelihood of out-of-sample trials given the in-sample-estimated betas for head and pupil rotation. We repeated this procedure 200 times for each participant to produce a mean out-of-sample choice likelihood for each participant. Across all participants, the average mean out-of-sample choice likelihood was 0.78 (95% CI: 0.58, 0.90). In other words, the likelihood of having observed the pattern of “rightward” judgments that materialized in the out-of-sample half of the data, given model coefficients estimated from the in-sample half of the data, was on average 78%. In effect, this means that our model’s predictive accuracy could be expected to be 78% on a trial-by-trial basis, indicating high reliability of individual differences across trials in our study.

Discussion

We showed that typically-developing children engage emergent gaze perception like adults, pulling information from the rotation of the head and pupils to judge a person’s direction of attention. We demonstrated for the first time that adults and children show notable variability in the way they weight these cues, with some individuals prioritizing information from the head and others relying more on pupil rotation. On top of this variability, we found that use of the pupils to make gaze judgments increases across development, in contrast to the usage of head rotation. We also provide evidence that unlike typically-developing individuals, children and adolescents with ASD use information from the pupils much less to make gaze evaluations, even after accounting their age.

Our findings contribute to a growing literature on the perception of gaze as an emergent process. Previous work has pointed to the flexibility of this process, whereby the extent to which each cue is weighted may depend on contextual factors like visibility, distance, or salience (Florey et al., 2015; Florey et al., 2016). Our findings isolate an important source of this flexibility across individuals. Only a few recent studies indicate that emergent integration is operative during infancy (Nakato et al., 2009; Otsuka et al., 2016). Our findings add to this work and suggest that children, adolescents, and adults are quite similar not only in their ability to engage emergent gaze as groups, but also in the way individuals tend to prioritize one cue over another. We also found that weighting of the pupils tends to increase during development. This seems consistent with evidence that information from head rotations is most salient early in life (e.g., Corkum & Moore, 1998).

Children with ASD are well known for their general deficits in global processing and impairments in perceiving eye gaze. Yet surprisingly, it is unclear how children with ASD perceive gaze conveyed at the emergent level, via the combination of head and eye rotations. Our findings begin to bridge this gap and indicate that children with ASD rely on information from the eyes less than their typically-developing peers. This may be surprising because it suggests a hierarchy of facial-feature prioritization in ASD in the reverse order (head to eyes) from that proposed in earlier models of typical gaze detection (eyes to head) (Baron-Cohen, 1995; Perrett, Hietanen, Oram, & Benson, 1992). Our findings are preliminary and set up a framework for a more thorough evaluation of emergent gaze in children with ASD. Besides increasing sample size in order to compare younger children with adolescents, it will also be important to include standardized measures of autism spectrum characteristics, cognitive ability, and adaptive skills. These assessments will allow us to evaluate whether the effects we have shown here emerge from specific deficits in gaze perception or from more general developmental delays. We propose that the latter explanation is more likely given the trend for increasing pupil use in our sample, regardless of ASD.

Our demonstration of individual differences in gaze integration opens the door to new questions. For example, why do some individuals favor one cue over the other? Can differential weighting fluctuate on a trial-by-trial basis? If changes in prioritization of cues is tied to a developmental trajectory with younger children relying more heavily on head rotation (as our data suggest), why is this so? Gaze is a dynamic feature and emergent gaze has been proposed as a mechanism for maintaining gaze constancy across a variety of viewpoints (e.g., Otsuka et al., 2016). Our work clarifies how gaze constancy is achieved with static faces, but it is unclear how the integrative process may operate when faces are dynamic.

Perceiving gaze is the first step in a chain of processes that ends in more complex evaluations and inferences (Baron-Cohen, 1995). It is thus important to consider that children and adults may not make the same cognitive or social extrapolations despite arriving at similar perceptual interpretations. We also note that differential weighting of head and eyes may not necessarily be a persistent problem since head and pupil rotations are often redundant during free viewing (Nakashima et al., 2015). Future work should determine the kinds of circumstances in which variability in seeing emergent gaze may be especially problematic. It is also worth wondering how much (or little) attention the children in our sample with ASD paid to the eyes of the faces. This is not necessarily a shortcoming of our design. Rather, we have shown how children with ASD perform on our task during free viewing. It may be interesting to examine how children with ASD, or even adults and typical children, perform on our task with instructions to attend the eyes or head.

Seeing gaze is a complex process that involves integrating information from the eyes and the head. Our study provides new insight into this process, revealing individual differences in the way adults and children prioritize these cues. We also introduce a sensitive framework to evaluate emergent gaze among atypically-developing children, and we provide preliminary evidence that children with ASD make gaze judgements mostly using the rotation of the head.

Research Highlights.

• Typically-developing children and adults show individual differences in the way they prioritize information from the head and eyes when judging a person’s gaze direction.

• While all individuals use head rotation when making gaze judgments, as individuals develop, they increasingly incorporate information from the eyes as well.

• Children with autism spectrum disorder tend to prioritize head rotation over pupil rotation when making gaze judgments.

• Our research provides a novel approach for understanding individual differences in the emergent perception of gaze among adults as well as typically- and atypically-developing children.