Cracking the Code of Live Human Social Interactions in Autism: A Review of the Eye-Tracking Literature

Abstract

Human social interaction involves a complex, dynamic exchange of verbal and non-verbal information. Over the last decade, eye-tracking technology has afforded unique insight into the way eye gaze information, including both holding gaze and shifting gaze, organizes live human interactions. For example, while playing a social game together, speakers end their turn by directing gaze at the listener, who begins to speak with averted gaze (Ho et al., 2015). These findings reflect how eye gaze can be used to signal important turn-taking transitions in social interactions. Deficits in conversational turn-taking is a core feature of autism spectrum disorders. Individuals on the autism spectrum also have notable difficulties processing eye gaze information (Griffin & Scherf, 2020). A central hypothesis in the literature is that the difficulties in processing eye gaze information are foundational to the social communication deficits that make social interactions so challenging for individuals on the autism spectrum. Although eye-tracking technology has been used extensively to assess the way individuals on the spectrum attend to stimuli presented on computer screens (for review see Papagiannopoulou et al., 2014), it has rarely been used to evaluate the critical question regarding whether and how autistic individuals process non-verbal social cues from their partners during live social interactions. Here, we review this emerging literature with a focus on characterizing the experimental paradigms and eye-tracking procedures to understand the scope (and limitations) of research questions and findings. We discuss the theoretical implications of the findings from this review and provide recommendations for future work that will be essential to understand whether and how fundamental difficulties in perceiving and processing information about eye gaze cues interfere with social communication skills in autism.

Human social interaction involves a complex, dynamic exchange of verbal and non-verbal information. For example, eye gaze information, including both holding gaze and shifting gaze, provides powerful social communicative signals that organize social interactions. Among typically developing adults, sharing gaze (i.e., making eye contact) indicates liking and attraction, attentiveness, and a sense of competence and credibility (for review see Kleinke, 1986). Gaze also functions to regulate and organize social interactions. For example, in conversations, listeners tend to hold gaze more than speakers (see Kleinke, 1986) and speakers end their turn by shifting gaze to the listener, who begins to speak with averted gaze (Ho et al., 2015). These findings demonstrate the importance of gaze information for facilitating human face-to-face social interactions and signaling important turn-taking transitions in these interactions. An important implication of these findings is that deficits in the ability to perceive and/or interpret gaze information likely impacts multiple aspects of human face-to-face social interaction.

Autism spectrum disorder (ASD) is a lifelong neurodevelopmental disability that impacts social communication and the ability to process and understand eye gaze cues (American Psychiatric Association, 2013). A central hypothesis in the literature argues that the difficulties processing eye gaze information are foundational to the social communication deficits that make social interactions so challenging for individuals on the autism spectrum (Baron-Cohen et al., 1997; Senju et al., 2005). Specifically, the phenotypic deficits in processing eye gaze information may contribute to difficulties that people on the autism spectrum experience in developing and maintaining social relationships, adjusting their behavior to suit different social contexts, making friends, being interested in people, and demonstrating social-emotional reciprocity. For example, the difficulties in conversational turn-taking in ASD, including longer turn-taking gaps, more pause time, and fewer initiations and responses during conversations may be related to the notable difficulties in ASD gaze processing that organizes turn-taking transitions in social interactions (Ochi et al., 2019). Here, we argue that it is essential to understand the specific aspects of eye gaze processing that are challenging for individuals on the autism spectrum to perceive and/or engage during live face-to-face human social interactions. Understanding the mechanisms of these phenotypic deficits is likely to inform the development of targeted intervention strategies that may improve sensitivity to eye gaze information and ultimately social communication and social interactions in ASD.

The empirical work investigating gaze sharing, eye contact, and shifts in gaze during live social interactions dates to the early 1970’s and was originally conducted using manually operated recording devices (e.g., Levine & Sutton-Smith, 1973). Subsequently, live social interactions between participants were recorded on video, and gaze behavior was later coded on a frame-by-frame basis (e.g., Mirenda et al., 1983). For the last two decades, eye tracking technology has become the predominant methodology for measuring visual attention and gaze behavior. Eye tracking has evolved to become a remote, non-intrusive technology, which is particularly useful for working with special populations¹. The most common approach is video based and involves illuminating the eye with an infrared light source, which causes visible reflections, and capture the reflections with a camera. Eye position and gaze direction is then calculated using information about the cornea and pupil reflections. Eye tracking technology is very precise with reported accuracy less than 0.5° visual angle and a precision of less than 0.4° visual angle (e.g., Griffin & Scherf, 2020). Eye trackers are configured to be mounted on a table or on the participant’s head and allow for recording in real time (see Figure 1).

Figure 1.

Examples of table- and head-mounted eye-tracking systems.

A limitation of table-mounted eye-trackers is that it constrains the presentation of stimuli to a computer screen. Table-mounted eye-trackers can be used to assess social visual attention in live social interaction via video feed, but not via an in-person experimental set-up. The development of head-mounted eye tracking systems allowed researchers to measure gaze behavior in a much broader range of conditions, including face-to-face social interactions. The first head-mounted eye-tracker was developed in 1948 by Hartridge & Thomson, but the technology has substantially improved such that researchers can now map gaze locations to each frame of a recorded scene (Land, 1992). These head-mounted eye trackers are becoming incredibly unobtrusive; they are mounted on glasses that a participant wears allowing them to move their head and body freely. These newer head-mounted trackers also have high accuracy (~ 1.0° visual angle; MacDonald & Tatler, 2018) and afford researchers the ability to address important questions about the role of social visual attention during real world activities.

Although eye-tracking technology has been used extensively to assess the way individuals on the autism spectrum attend to faces presented on computer screens (for review see Papagiannopoulou et al., 2014), it has rarely been used to evaluate the critical question regarding whether and how autistic individuals process non-verbal cues, including gaze, from their partners during live social interactions. This is a critical distinction given that findings about social visual attention from laboratory studies do not always generalize to the real world. In other words, people attend differently to real people than to images of people (Risko et al., 2016). In this narrative review of this emerging literature, we focus on characterizing the experimental paradigms and eye-tracking procedures to understand the scope (and limitations) of the research questions and findings.

Why is live, dynamic, face-to-face social interaction important to assess in ASD?

To date, conclusions about the mechanisms underlying atypical gaze processing in ASD are largely derived from empirical studies that used image or video stimuli of human faces (for review see Papagiannopoulou et al., 2014). Although findings are mixed, those reporting reduced gaze to human faces in ASD are often highlighted (Dalton et al., 2007, 2005; Hosozawa et al., 2012; Noris et al., 2012), leading to the hypothesis that individuals on the autism spectrum will hold gaze less frequently in live social interactions, and as a result, miss out on the opportunity to perceive shifts in gaze as social signals that organize turn-taking in interactions (Chevallier et al., 2012).

However, results from neurotypical participants indicate that this hypothesis may not accurately reflect differences between the way gaze is used to both gather information and signal information in the presence of another person (i.e., dual function of gaze; Nasiopoulos et al., 2015; Rogers et al., 2018). Specifically, studies that include images and videos of people may only trigger the encoding function of gaze behavior and fail to capture the signaling function.

In an interactive social context, where gaze behavior simultaneously functions to gather and signal information, the patterns of gaze behavior often look different than during a non-interactive computer-based task. For example, image/video-based social attention paradigms routinely elicit a strong bias in gaze behavior that is focused on the social elements within the stimuli (e.g., faces; Birmingham & Kingstone, 2009; Birmingham et al., 2008). However, in similar paradigms with real people, participants are less inclined to focus gaze on a social partner’s face or shift gaze to look where a partner is looking (Laidlaw et al., 2012; Gallup et al., 2012). There is some evidence to support this task-related difference in gaze behavior in ASD as well (Grossman et al., 2019).

Therefore, our current understanding of potential mechanisms underlying difficulties processing eye gaze information in ASD, which is largely derived from studies without social partners, may be misleading. This is critical to sort out because of the way it has informed targeted intervention approaches to increase gaze to (images of) human faces and the development of theoretical models that attempt to characterize reduced gaze toward human faces in ASD. These models include the Social Motivation model, which hypothesizes that individuals on the autism spectrum are less intrinsically motivated to attend to social stimuli (Chevallier et al., 2012) and the Eye-Avoidance model, which hypothesis that people on the autism spectrum are affectively overwhelmed when gazing at faces, and specifically during eye contact (Hutt & Ounsted, 1966; Kliemann et al., 2012; Spezio et al., 2007a; Spezio et al., 2007b; Tanaka & Sung, 2016). Both models emphasize reduced social visual attention to faces as a core explanatory mechanism for deficient processing and understanding of eye gaze cues in autism.

The current study

Importantly, evaluating the range of paradigms and findings from studies investigating the perception and use of gaze during live social interactions in autism will help identify whether reduced gaze, does in fact, characterize face-to-face social interactions in ASD and is central to the social communicative deficits in autism. In what follows, we provide a comprehensive narrative review of the empirical studies evaluating gaze in autism and neurotypical individuals during live social interactions. The objective was to understand whether individuals on the autism spectrum exhibit: 1) reduced social visual attention to faces; 2) reduced gaze sharing (i.e., eye contact); and/or 3) atypical use of shifts in gaze to during live social interactions. We conclude with a discussion about the theoretical implications of the findings from this review and provide recommendations for future work that will be essential to understand whether and how fundamental difficulties in perceiving and processing information about eye gaze cues interfere with social communication skills in autism.

Method

Our methods were guided by the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines for preparing a transparent, complete, and accurate accounting of the review rationale, methodology, and findings (Page et al., 2021).

Study Search and Identification

The diagnostic criteria, and thus nomenclature, of autism changed considerably over the time period when this literature was published. Therefore, we included study search terms to capture the full range of terminology used to classify individuals on the autism spectrum (particularly by the multiple versions of the DSM) across this period.

We used two primary sources for identifying candidate articles. In October 2021, we conducted a literature search in the PubMed database. The search terms included: (autism OR autistic OR ASD OR Asperger) AND (Social Interaction OR live Interaction OR conversation OR interview) AND (eye-tracking OR eye gaze OR gaze OR looking time). We limited our search to experimental studies that compared ASD and typically developing (TD) participants in a measure of social visual attention during live social interaction with another person. Live social interactions could take place in the same physical space (i.e., face-to-face) or could be administered via live video feed on a computer screen. We searched for studies that included a measure of visual attention acquired by either eye tracking technology or frame-by-frame coding from video recordings.

Inclusion and Exclusion Criteria

We employed the following inclusion criteria: articles must (1) report using a task of live social interaction; (2) include an ASD group; (3) include a typically developing (TD) control group; and (4) include a measure of social visual attention (e.g., number of fixations, fixation duration). Autism groups were that were characterized as having a diagnosis of autism, ASD, Asperger syndrome, autistic disorder, or pervasive developmental disorder – not otherwise specified (PDD-NOS) were all included. Studies were excluded if they were not (1) published in English or (2) peer reviewed.

Study Selection

The full study search, identification, and selection process is shown in Figure 2. We applied our inclusion criteria in two steps. First, we screened the titles and abstracts of all articles returned from the initial database searches. Since this was the first level of screening, we emphasized overinclusion to maximize yield. For example, abstracts were only rejected based on exclusion criteria (i.e., not published in English, not an empirical paper). After this first set of articles were screened for inclusion, we assessed the remaining articles for inclusion by evaluating each full text article for group characteristics (i.e., ASD, TD), task paradigm (i.e., live social interaction), and measure of social visual attention (i.e., looking time, fixation duration).

Figure 2.

Flow chart of study selection process.

After title and abstract screening, all the studies identified from this initial review were evaluated for inclusion and exclusion criteria in a full text review. Any criterion discrepancies were discussed by S.L. and K.S.S until consensus was reached. The study selection process is illustrated in Figure 2 and the full set of studies included in the analysis is reported in Table 1.

Table 1.

Demographic information and design features of studies investigating gaze patterns of individuals on the autism spectrum during live social interaction paradigms

Paper	Participant groups	Social Partner	Example Questions	Gaze Recording	AOIs	Group Differences	Total Sample N	Age Range (yrs)
Auyeung et al. (2015)	TD, ASD	Adult researcher	“How was your journey to the research center?”	Table mounted eye-tracker	Eyes, Mouth	Face: NR Eyes: ASD < TD	74	NR
Barzy et al. (2020)	TD, ASD	Adult researcher	“Tell me somewhere you would like to go over Christmas and why you would like to go there?”	Video Recording	Eyes, Face	Face: ASD < TD Eyes: ASD = TD	53	NR
Cañigueral et al. (2021)	TD, ASD	Adult researcher	“You are going to the cinema this evening would you rather; option A. Watch a fantasy film or option B. watch a comedy film?”	Head mounted eye-tracker	Eyes, Mouth	Face: NR Eyes ASD = TD	52	NR
Freeth & Bugembe (2019)	TD, ASD	Adult researcher	“Can you tell me about things that you like and don’t like about living in your town”	Head mounted eye-tracker	Eyes, Mouth, Outer Face	Face: ASD = TD Eyes: ASD < TD	26	18–57
Hanley et al. (2014)	TD, ASD, SLI	Adult researcher posing as magician	“What is your name” “What age are you”	Head mounted eye-tracker	Eyes, Mouth, Nose, Outer Face	Face: ASD = TD Eyes: ASD < TD	50	7–12
Hutchins & Brien (2016)	TD, ASD	Adult researcher	“What kinds of things make people scared?”	Table mounted eye-tracker	Eyes, Mouth	Face: NR Eyes: ASD < TD	37	6–12
Jones et al. (2017)	TD, ASD	Adult researcher	“I went to the park this weekend”	Head mounted eye-tracker	Eyes	Face: NR Eyes: ASD = TD	40	4–13
Mirenda et al. (1983)	TD, ASD	Adult researcher	NR	Video recording	Face	Face: ASD = TD Eyes: NR	8	6–15
Nadig et al. (2010)	TD, ASD	Adult researcher	“Tell me about (pre-identified interest)”	Head mounted eye-tracker	Face	Face: ASD = TD Eyes: NR	37	NR
Wang et al. (2015)	TD, ASD	Participant dyads	Conversing freely about interests	NR	Eyes, Mouth, Face	Face: ASD = TD Eyes: ASD < TD	15	NR
Zhao (2021)	TD, ASD	Adult researcher	“What is your favorite thing to do and why”	Head mounted eye-tracker	Eyes, Mouth, Face	Face: ASD < TD Eyes: ASD = TD	43	6–13

Note: AOI = Area of interest; TD = typically developing; ASD = autism spectrum disorder; NR = not reported;

^*indicates that the study was peer reviewed for a conference submission but not a manuscript publication

Results

Study Selection

In total, we included 11 studies that employed live social interaction paradigms to measure and compare social visual attention in participants on the autism spectrum and TD individuals. In total, these studies include 206 ASD individuals, 214 TD individuals, and 17 individuals from other disability groups. Only 55% of the studies reported the age of the participants, which ranged from 4 – 57 years. The sample sizes, study characteristics, participant demographics, and social interaction paradigm details for each study are reported in Table 1.

Are there Group Differences in Social Visual Attention During Live Social Interaction?

A central question of this review was to understand whether and how social visual attention is altered and/or impaired in ASD during live social interactions. Three predictions from the literature using image/video stimuli are that among ASD participants there would be 1) reduced social visual attention to the face of social partners, 2) reduced gaze sharing (i.e., eye contact), and 3) fewer gaze shifts to organize conversational turn taking during live social interactions. We report that findings from this literature of social visual attention during live social interactions are very heterogenous and inconsistent with these predictions.

Social visual attention to faces.

Seven of the 11 studies measured social visual attention to faces during live social interactions. Nearly 60% of these studies (4/7) reported comparable social visual attention to faces in the TD and ASD groups during live social interactions (Cañigueral et al., 2021; Jones et al., 2017; Mirenda et al., 1983, Nadig et al., 2010).

Shared gaze.

Nine of the 11 studies measured attention to eyes and shared gaze during live interactions. Five of these studies (55%) reported reduced shared gaze during live social interaction (i.e., reduced eye contact; Auyeung et al., 2015; Freeth & Bugembe, 2019; Hanley et al., 2014; Hutchins & Brien, 2016; Wang et al., 2015) and four (45%) reported comparable shared gaze (Barzy et al., 2020; Cañigueral et al., 2021; Jones et al., 2017; Zhao et al., 2021) in ASD participants.

Using gaze shifts.

No studies assessed group differences in the use of gaze shifts to organize conversational turn taking in this literature.

In sum, the findings are mixed regarding the three hypotheses. First, a majority of studies reported no differences in visual attention to the faces of social partners during live social interactions between TD and ASD groups. These data do not support hypothesis one. Second, there also does not seem to be support for the second hypothesis of reduced gaze sharing during live social interactions among individuals on the autism spectrum. Nine studies evaluated shared gaze and there was a near 50/50 split in findings regarding whether there is comparable shared gaze or reduced shared gaze among the ASD participants. Third, there were no studies that investigated differences in the use of gaze shifts to organize live social interactions to be able to evaluate hypothesis three.

Discussion

We found that the results from the emerging literature investigating social visual attention during live social interactions in autism do not clearly support predictions from the literature employing images/videos of social interactions to study social visual attention in ASD. These findings are consistent with the notion that conclusions from the study of group differences in social visual attention from laboratory studies do not always generalize to the real world (e.g., Risko et al., 2016).

In what follows, we discuss these findings in terms of four key aspects of the methodology, including the structure of the social interaction, type of social partner (i.e., familiar vs unfamiliar), content of the interaction (i.e., personal vs impersonal), and measure of social visual attention. In so doing, we highlight strengths and limitations of the current approaches for addressing the essential questions regarding social visual attention and use of gaze sharing and gaze shifts to organize social interactions in ASD. We conclude by making recommendations about how to help move the field forward to address these most pressing questions.

Structure of the Social Interaction

In most of the existing studies (10/11), researchers used a structured interview paradigm to measure social visual attention during live social interaction in ASD individuals. In this paradigm, one social partner, most often an unfamiliar adult who is a trained member of the research staff, sits in front of the participant and presents the participant with a series of questions. Participants are instructed to answer the questions one at a time. Critically, although the participant talks to the researcher when answering the questions, there is limited to no conversational exchange between the participant and the researcher during the answer period. In other words, the nature of the social interaction is constrained with very limited conversational turn taking. This is important to note given the range of paradigms that have been used to discover the dynamic role of shared gaze and gaze shifts to organize conversational turn taking in social interactions among TD individuals. These paradigms are more naturalistic and specifically designed to be highly interactive and require conversational turn taking. They include playing games (Ho et al., 2015); performing collaborative tasks (Sandgren et al., 2012); and, interacting via dynamic social conversations (Kendon, 1967).

In the ASD literature we reviewed here, the social partner (i.e., trained researcher) asked the participant a set of prespecified questions one at a time (Mirenda et al., 1983; Nadig et al., 2010; Barzy et al., 2020; Cañigueral et al., 2021; Zhao et al., 2021; Hutchins & Brien, 2016; Freeth & Bugembe, 2019; Auyeung et al., 2015). In some studies, participants simply provided yes/no answers (Zhao et al., 2021). In other studies, participants were encouraged to give multiple responses to a question (Hutchins & Brien, 2016) or were instructed to respond for a specified amount of time (i.e., 30 seconds; Barzy et al., 2020). This structured interview format does not provide much opportunity for conversational interaction, and therefore, only affords a limited window into the dynamic nature of social interactions. In particular, the formalized structure of the interview (e.g., talk for 30 seconds) might minimize the use shifts in gaze to help organize the social interaction (e.g., to signal that speaker is finished talking). Therefore, findings from these studies may provide limited information about how shifts in gaze and shared gaze help organize social interactions.

Mindful of these limitations, two studies used a similar structured interview format, but also allowed the interviewer to ask unscripted follow-up questions to facilitate more natural social conversation (Freeth & Bugembe, 2019; Nadig et al., 2010). In both studies, the researchers reported no reduced visual attention to faces of social partners during the live social interaction in the ASD group. Only one of these studies investigated shared gaze (i.e., eye contact) and reported that it was reduced in the ASD group (Nadig et al., 2010).

Several studies included a secondary condition in which they specifically investigated whether the nature of the social interaction influenced social visual attention in either ASD or TD groups. For example, Jones et al. (2017) employed an interactive play segment and a conversation segment of a social interaction, administered as modified version of the Brief Observation and Social Communication Change (BOSCC; Grzadzinski et al., 2016). During the play segment, the child chose a toy to play with and the researcher joined in without guiding the interaction. The researcher then transitioned into conversation by making an open-ended statement such as, “I went to the park this weekend.” This introduction to conversation more closely mirrors real-world interactions. Findings indicated that both TD and ASD participants exhibited reduced eye contact during the interactive play segment compared to the conversation segment. Importantly, the conversation segment was restricted to a 2-minute period, which is relatively short in comparison to the paradigms that have been used to study live social interactions among TD dyads that last for 7–10 minutes (e.g., Ho et al., 2015; Jarick & Kingstone, 2015). Therefore, this short social interaction may not have captured differences in patterns of social interactions (and thus gaze patterns) between TD and ASD individuals that develop over time in a social exchange. Also, gaze patterns were not measured as a function of contingent behavior, which is critical for understanding how gaze behavior is used to organize social interactions.

Hanley et al. (2014) designed a “magic show” in which an adult researcher dressed up and pretended to be a magician. The magician initiated a short introductory conversation (~35 seconds) with the participant, asking questions like, “What is your name?” and “What age are you?” Following this initial conversation, the magician recited a poem using a hand puppet, while the participant listened. Gaze was measured throughout the entire social interaction. The authors reported that the ASD and TD groups exhibited similar gaze to the face of the magician throughout the social interaction, but that the ASD group exhibited less shared gaze (eye contact) with the magician than did the TD group.

Finally, there is only a single study in the autism literature that evaluated naturalistic conversations between two participants (Wang et al., 2015). In this work, participants were instructed to converse freely about their interests. This work was presented at the annual conference of the Vision Sciences Society but has not yet been published in a peer reviewed journal.

In sum, most studies investigating social visual attention during live social interactions in ASD employ a structured interview paradigm with a research trained confederate as the social partner. Using this kind of paradigm is an understandable first step to build on and extend the work measuring social visual attention in ASD in response to images/movie stimuli of human faces. It provides experimental control over the nature and structure of the social interaction to facilitate measurement of social visual attention. Importantly, this work seems to indicate that individuals on the autism spectrum gaze at faces during live social interactions, but that they may not share gaze (i.e., eye contact) in the same way that TD individuals do.

However, these findings need to be contextualized with an understanding of the important limitations of this paradigm regarding dynamic social interaction and conversational turn taking. This is especially important given the wealth of findings about how gaze sharing and shifts of gaze organize dynamic social interactions among TD individuals. Therefore, these findings provide a limited understanding about how gaze sharing and shifts of gaze function to organize social interactions in ASD.

Social Partner

All the published studies investigating social visual attention in ASD during live social interactions included an unfamiliar adult as the social partner (see Table 1). This is true regardless of whether the study involved child or adult participants. This ubiquitous strategy to ask participants to engage with an unfamiliar adult in live social interactions is notable for several important reasons.

First, work with TD adults indicates that social visual attention during live social interactions is influenced by the familiarity of social partners (Broz et al., 2012; Cordell & McGahan, 2004). For example, unfamiliar social partners share more mutual gaze as the duration of their interaction increases (Cordell & McGahan, 2004). Also, familiarity between social partners influences the likelihood that initiated shared gaze is returned (Broz et al., 2012). It is important to remember that gaze sharing and shifts of gaze provide multiple affective signals of social communication (e.g., threat, dominance, attentiveness, liking), which may vary as a function of the familiarity of the social partner. Therefore, it is important to consider that the unfamiliar nature of the social partner may impact the generalizability of findings from these studies.

Second, in the literature investigating dynamic social interactions among typically developing adults, the interactions are often measured in the context of participant dyads. This is important methodologically for several reasons. First, the relative familiarity of the participant dyads can be manipulated and measured. Regardless of personal familiarity, participant dyads are equally unfamiliar with the research context and environment. This contrasts with paradigms in which the research confederate is highly trained to conduct the research in a familiar environment. Also, the participant dyads are typically organized into age-matched same-sex peer dyads, which also often contrasts with the research interviewer approach. This may be especially influential in the case of developmental studies in which the participants are much younger than the research staff member. The differential age and relative experience in the lab environment of the social partners in these studies may influence gaze in systematic ways. For example, there is some evidence to support the notion that fear of failure in a testing environment increases the tendency to avert gaze among TD individuals (Doherty-Sneddon & Phelps, 2005). Therefore, employing a paradigm with a participant/researcher structured dyad may have critical implications for assessing gaze behavior.

Third, in many of the dyadic interaction paradigms between unfamiliar social partners that are reported in the typical literature, researchers include a warm-up or habituation period prior to the start of the social interaction. This warm-up period is designed to reduce social anxiety and increase familiarity. It typically ranges from 2–25 minutes and the members of the dyad are given time to introduce themselves and occurs before gaze is measured (see Ho et al., 2015; Hessels et al., 2017; Kendon, 1967; Levine & Sutton-Smith, 1973). This is an important methodological step because feelings of social anxiety reportedly impact gaze between unfamiliar partners (Kim et al., 2018; Kleberg et al., 2021). Post-study questionnaires indicate that this warm-up period reduces the “initial awkwardness” between themselves participants (Kendon, 1967). The warm-up period may also help participants adjust to eye-tracking equipment or to being video-taped/recorded while interacting.

In contrast, most studies in the autism literature, that include an unfamiliar adult social partner, do not report including a warm-up period prior to data collection (Auyeung et al., 2015; Barzy et al., 2020; Cañigueral et al., 2021; Mirenda et al., 1983; Nadig et al., 2010; Wang et al., 2015). Only two studies mentioned that the researcher introduced the task or provided brief instructions such as “Hi, I want to have a conversation with you” (Hutchins & Brien, 2016) or “I will be asking you some questions, there are no right or wrong answers” (Freeth & Bugembe, 2019). A warm-up period may be essential for children and adolescent participants, particularly those on the autism spectrum, to feel safe interacting with unfamiliar adults. It is possible that reported group differences in gaze behavior (e.g., sharing gaze) between ASD and TD participants could be related to differential social anxiety in the lab environment with an unfamiliar adult.

Finally, although many studies did not explicitly report the duration of the social interaction, the studies that did so varied greatly, ranging from less than a minute (Hanley et al., 2014) to 15 minutes (Jones et al., 2017; see Table 1). It is important to consider findings in the context of understanding the duration of the social interaction given reports that gaze behavior changes over time, particularly between unfamiliar social partners (e.g., Cordell & McGahan, 2004).

Conversational Content

The nature of the conversational prompts varied more than any other methodological feature across the studies in this autism literature. Some studies used prompts that only required a single word answer (e.g., yes/no), while others used open-ended prompts that elicited more conversational responses. The prompts also varied in terms of whether they asked participants to respond about personally relevant or more general information or information about other people.

In the studies that used questions requiring only single word answers, results indicated no differences in eye-directed gaze to the social partner among ASD individuals. The studies asked participants to respond to questions such as “do you like apples” (Zhao et al., 2021), or, “You are going to the cinema this evening, would you rather; option A, watch a fantasy film, or option B, watch a comedy film” (Cañigueral et al., 2021). In contrast, two studies used questions that were more open-ended and personal in nature, such as “Tell me some things you like and don’t like about living in your town”, and “Tell me some things you did last weekend and your plans for next weekend,” (Freeth & Bugembe, 2019; Jones et al., 2017). In both studies, researchers reported that participants on the autism spectrum showed reduced face-directed gaze to their social partner compared to TD participants.

Three studies used prompts that were personal and directly relevant to the social context, such as “What is your name?”, “How old are you?” (Hanley et al., 2014), “How was your journey to the research center,” (Auyeung et al., 2015). These prompts are realistic introductory questions for a first meeting social interaction. Both studies that employed these personally relevant prompts reported that participants on the autism spectrum showed reduced gaze to the eyes of their conversational partner, compared to TD participants. To test the notion that the personal relevance of the questions influences gaze patterns more directly, Barzy et al. (2020) included three types of prompts including questions about the participant. (e.g., tell me somewhere you would like to go over Christmas and why), questions about a familiar person (e.g., tell me somewhere your mother would like to go over Christmas and why), and questions about an unfamiliar character from a short vignette (e.g., tell me somewhere Jack would like to go over Christmas and why). The researchers hypothesized that differences in gaze between TD and ASD participants would be less pronounced when answering prompts about themselves and their family rather than about strangers, because these questions would be less cognitively demanding. They reported that both participant groups (ASD and TD) looked more at the face of their conversational partner during the self and familiar prompt conditions. In contrast to predictions, there were no group differences in gaze behavior across conditions.

Finally, several of the studies hypothesized that a conversation involving a participant’s specific interest may either (1) increase social motivation for participants on the autism spectrum, and therefore increase gaze towards the conversational partner, or (2) cause participants with autism to become hyper-focused on the topic, thereby creating a monologue-like interaction where gaze to the conversational partner is reduced (Nadig et al., 2010). To test these hypotheses, researchers included questions specific to restricted interests (e.g., What is your favorite thing to do and why; Zhao et al., 2021) and contrasted them with general personal interest questions (e.g., Tell me about your friends/siblings/pets; Nadig et al., 2010). Interestingly, neither study found that discussing restricted interests differentially influenced gaze behavior in ASD participants.

In sum, the conversational prompts that are used to elicit social interaction in this literature differ on multiple dimensions (single word vs open-ended answers; personally relevant vs generic; self- vs other-oriented; restricted interest- vs general personal interest). Perhaps it is not surprising then that there is no consistency about how variation in these dimensions influences gaze behavior in ASD.

Measuring gaze during live social interactions

Collecting gaze data.

Researchers typically use eye-tracking technology to measure the position of a participant’s gaze during live social interactions. However, the nature of the eye-tracking (table vs head-mounted) and the strategy for extracting the gaze data varies greatly across studies (see Figure 2 for example of table- and head-mounted eye-tracking systems). For example, 2 of the 11 studies reviewed here coded gaze behavior from video recordings, 2 reported using table-mounted eye-trackers, and 6 reported using head-mounted eye-trackers. This methodological difference impacts the experimental paradigm that can be employed, accuracy of data collection and nature of the analytic approach for measuring gaze behavior. Studies employing table-mounted eye-trackers have more constraints on the visual field of view, because it is limited to a computer screen, which limits the nature of the experimental paradigms that can be used particularly for social interactions. Data are much easier to process. Studies using head-mounted eye-trackers will have the most flexibility in the experimental paradigm since participants can freely move their head and there are minimal constrains on the field of view under study; however, strategies for extracting gaze behavior are most complicated with this technology.

Extracting gaze data.

Most of the reviewed studies link eye-movement measures to parts of the stimuli by defining areas of interest (AOIs). The AOIs reflect regions of the social and nonsocial aspects of the environment where researchers expect to test for within- and between-subject differences. Researchers derive dwell time or fixation duration, proportion of total fixation duration, number of fixations, and proportion of total fixations. Importantly, although there is often conceptual similarity in the motivation for the AOI definition across studies (define the face or eyes in the stimuli), there is heterogeneity in the methodological construction of these AOIs that influences the shape, size, and location of these regions. These differences have implications for computing the gaze metrics (see Hessels et al., 2016).

All the studies reviewed here used the researcher-defined, hand-drawn method of defining AOIs. This involves using a pre-specified shape to select an area around a part of the stimulus, like the face or the eyes, from which gaze data will be extracted. The location, shape, and size of the AOIs are all subjectively determined by the researcher, The problem of defining AOIs is amplified when using head-mounted eye-tracking because the AOI locations change for each frame of the captured scene video. Methodological differences that characterize the construction of static AOIs are compounded across frames in mobile eye-tracking.

For example, 4/11 studies included the whole face as an AOI during conversation (Freeth & Bugembe, 2019; Hanley et al., 2014; Nadig et al., 2010; Wang et al., 2015). However, the strategy for constructing the face AOIs varied greatly. Although these studies include the hair, ears, and chin within the whole face AOI, they differ in the shape used to capture the AOI. For example, one study used a square to roughly enclose the entire face (Nadig et al., 2010), and two studies used a freeform polygon to roughly (Barzy et al., 2020) or tightly (Zhao et al., 2021) enclose the face. For studies that assessed visual attention to face components like the eyes, nose, and mouth, a variety of AOI construction techniques were used. For example, rectangles (Zhao et al., 2021; Hutchins & Brien, 2016), ellipsoids (Freeth & Bugembe, 2019), half circles (Cañigueral et al., 2021), and free form polygons (Auyeung et al., 2015) enclosing the eyes and mouth have all been used to assess visual attention to eyes during conversation.

These differences in AOI construction, particularly in terms of AOI size, across studies are likely to have influenced the likelihood of observing group differences in gaze behavior. Hessels and colleagues empirically determined that AOI-production methods can disproportionately influence estimations of group differences in gaze behavior, particularly for ASD participants (2016). Furthermore, they determined that adopting large AOIs solves the problem of statistical differences between methods, particularly when making cross-group comparisons.

This “area-of-interest problem” remains a considerable limitation in eye-tracking research broadly, especially in mobile eye-tracking (Hessels et al., 2016). Methods for objective and systematic ways of defining AOIs are slow to develop (e.g., Caldara & Miellet, 2011). Importantly, there are recommendations for considering both the data quality and the type of stimulus when constructing hand drawn AOIs (see Holmqvist et al., 2011). There are also machine learning approaches to generating emergent AOIs from the fixation data; however, these approaches are only designed for static image stimuli (Chuk et al., 2014; Hsiao et al., 2021). The field of eye-tracking research needs computational strategies for processing real-time video-based gaze data from head-mounted trackers and analyzing the data from a priori specified or data-driven AOIs. This area is ripe for collaboration between computer and vision scientists.

Developmental changes in gaze behavior

Studies in this literature often include participants from a broad range of ages in a single study and do not have the sample size to address questions of age-related change. This is important to note given previous findings of age-related changes among TD individuals in gaze behavior during live social interactions. For example, gaze towards a social partner and mutual gaze between partners increases from young childhood (i.e., ages 4–5 years) through late childhood (i.e., 9 years), but then decreases in early adolescence (i.e., ages 10–12-years) before increasing again in late adolescence and early adulthood (Levine & Sutton-Smith, 1973). Therefore, it is important to consider age-related changes, particularly during adolescence, when evaluating gaze behavior in live social interactions. Although more than half of the extant autism studies (6/11) including child and adolescent participants in the same sample (see Table 1), none of them evaluate age-related changes in gaze behavior. This is a missed opportunity and findings of group differences might reflect differential age-related effects in ASD and TD groups that are important to understand (e.g., developmental plateau vs delay).

The age of the participant relative to the age of the social partner is also critical to consider. When adult researchers are the primary social partner, there may be confounds in the social dynamics between the participant and the researcher as a function of age. For example, a young child participant may view the researcher like an authority figure, which could affect synchrony of gaze during conversation (Anaya et al., 2021). In contrast, a participant near the same age as the researcher, may view the researcher as a peer and less like an authority figure. These are critical considerations to evaluate and understand potential developmental changes in the way typically developing individuals and those on the autism spectrum learn to use gaze behavior to gather and signal information in social interactions.

Moving Forward

This review provides a clear path forward for future research using live social interaction paradigms to understand social attention in individuals on the autism spectrum. For example, most studies utilized some form of an interview with a question-and-response format that we argue does not reflect the dynamic exchange of social cues in real life social conversations. As a result, these research findings are limited in scope and likely do not generalize to real social interactions experienced on a daily basis in individuals on the autism spectrum. To move the field forward, we have identified a series of recommendations for future research that include (1) assessing contingent gaze patterns to compare directly with findings in the TD literature; (2) developing task paradigms that incorporate naturalistic turn taking and familiar social partners such as family or friends, and (3) employing age-appropriate conversational partners and allowing participants to habituate to the environment.

In order to capture when and where social attention is deployed, dyadic interaction paradigms must present the opportunity for naturalistic turn-taking behavior. For example, paradigms used in TD literature incorporate creative platforms to encourage naturalistic interaction, such as playing games like 20 questions, Heads Up!, and story-telling games, which require a constant dynamic exchange between partners (Ho et al., 2015; Levine & Sutton-Smith, 1973). Some studies even measure gaze while participants work together on a common goal, such as building a structure out of blocks (Hessels et al., 2017; Levine & Sutton-Smith, 1973). These types of tasks allow us to understand contingent patterns of social visual attention during naturalistic conversation. Indeed, many studies in the TD literature report a specific gaze pattern, where the speaker directs a prolonged gaze to the listener, in order to signal that they are done talking and want to transition into a listening role (Bavelas et al., 2002; Ho et al., 2015; Kendon, 1967). This cue is critical for smooth turn-taking behavior, which is maybe difficult to process in ASD. These kinds of paradigms also elicit skills such as joint attention and social referencing, which may be especially accurate relevant for assessing difficulties in social interactions for those on the autism spectrum (Butterworth & Jarrett, 1991; Mundy, 2003; Mundy & Newell, 2007; Tomasello et al., 2005).

However, a majority of paradigms that measure gaze during live dyadic interaction in ASD do not encourage dynamic conversational exchanges. To address whether social visual attention in autism diverges from typical patterns in live interaction, experimental paradigms should provide the same opportunity for dynamic conversational turn-taking. Rather than investigating the broad macrostructure of gaze behavior during conversation (e.g., frequency and duration of gaze throughout the entire interaction), we suggest that future research incorporate a free-flow conversation phase or a game-playing phase, which allows for gaze patterns to be measured while the speaker and listener transition roles. Also, the duration of these tasks should be of similar duration to those used in the TD literature.

Alternative research suggests that event-based skills such as joint attention or social referencing are important indicators of social literacy (Butterworth & Jarrett, 1991; Mundy, 2003; Tomasello et al., 2005; Mundy & Newell, 2007). There has been some research in the autism literature using live face-to-face joint attention and social referencing paradigms, where a pair of participants are instructed to cue each other on shared targets using shift in eye gaze (Dravida et al., 2020), however, this research is still limited. We suggest that in future research, participants should engage in a goal-oriented task with a partner, such as a game or building activity that incorporates objects or toys. Researchers may also consider adding a component of social motivation, for example, the interaction may be structured as a game where a social partner talks about their “favorite” object among a group of similar objects set out on a table. The partner then bids for joint attention by cueing the location of their “favorite” object with shifts in eye gaze, and since there is an emotional component of the object being a “favorite” or having some meaning to the partner, this may motivate naturalistic joint attentional skills in those on the autism spectrum.

In most studies included in this review, participants interacted with an unfamiliar adult researcher during the live interaction, rather than an age-matched peer. Also, very few studies included a habituation period prior to data collection in which participants are given the chance to warm up to the social partner and acclimate to the experience of having their gaze recorded. This may be especially challenging for individuals on the autism spectrum who struggle with social communication and often, social anxiety. As a result, evaluating patterns of social attention during a social conversation with a strange adult may not reflect the way that individuals on the autism spectrum can deploy social attention during social conversations, particularly when they are more comfortable. We suggest that researchers design social conversational paradigms that include participants and a familiar social partner (e.g., friend, sibling, significant other) as is often done in the TD literature. This approach may reveal more about the capacity to modulate social visual attention versus the competency to do so in a manner similar to TD individuals. To maximize experimental control during conversation with a familiar partner, researchers could offer prompts such as “talk about a favorite memory that you share together” or “talk about your favorite places to go together”. Posing such prompts that focus on shared experiences may help to encourage dynamic exchange. Additionally, posing prompts that are personal and require a high level of emotion-reasoning may help to capture differences in gaze patterns between individuals on the autism spectrum and their TD peers (Hutchins & Brien, 2016).

Lastly, there are methodological inconsistencies in the assessment of visual attention across studies; specifically, the way that data is extracted as a function of AOI into particularly facial features. Within the autism literature, some researchers have proposed that individuals on the autism spectrum may have an aversion to looking at the eyes of a conversational partner (Tanaka & Sung, 2016); others have proposed that individuals on the spectrum may gain more social information by attending the mouth of a conversation partner, rather than the eyes (Falck-Ytter & von Hofsten, 2011). As such, it is important to measure gaze patterns specific to facial features in order to gain a more comprehensive understanding of syndrome-specific socio-communicative deficits.

Methodological inconsistencies between studies and lack of ecological validity may be the cause of persistently mixed findings in studies that investigate autism-related deficits in social attention. However, it is possible that inconsistencies in the data support a different hypothesis entirely; one where autistic people attend equally to social stimuli and therefore encode the same amount of social information as do TD people, but simply do not interpret the social information or signal social information to the same extent (Griffin & Scherf, 2020; Suri et al., 2021). With such inconsistent findings in social visual attention, it seems that raw looking time to the face/eyes alone may not be a reliable predictor of the diagnostic social deficits described in ASD. However, it is difficult to form a clear conclusion when almost no reliable social behavioral assessments are included in the literature. In the studies described above, there are few reports of supplemental social behavioral tasks to assess how gaze behavior may be associated with real-world social performance, for example, daily social behavior questionnaires, behavioral facial recognition tasks, or emotion recognition tasks.