Abstract
Social robots can offer tremendous possibilities for autism spectrum disorder (ASD) interventions. To date, most studies with this population have employed short, isolated encounters in controlled laboratory settings. Our study focused on a 1 month, home-based intervention for increasing social communication skills of 12 children with ASD between 6–12 years old using an autonomous social robot. The children engaged in a triadic interaction with a caregiver and the robot for 30 minutes every day to complete activities on emotional storytelling, perspective taking, and sequencing. The robot encouraged engagement, adapted the difficulty of the activities to the child’s past performance, and modeled positive social skills. The system maintained engagement over the 1-month deployment, and children showed improvement on joint attention skills with adults when not in the presence of the robot. These results were also consistent with caregiver questionnaires. Caregivers reported less prompting over time and overall increased communication.
Summary
Children with ASD show improved joint attention following 1-month of in-home social skills training with an autonomous robot.
Introduction
Autism spectrum disorder (ASD) is a neurodevelopmental condition characterized by social interaction and communication deficits and the presence of restricted, repetitive patterns of behavior (1). Children and adults with ASD often have difficulty in responding to social overtures, recognizing the emotional states of others from visual or auditory cues, and understanding the importance of gaze as a social cue (2). Therapies are diverse, but are typically time-, resource- and labor-intensive, and can put significant strain on families and caregivers (3).
Technology-based interventions for ASD, and robotics in particular, have been seen as a potential approach for augmenting the efforts of families and clinicians to provide on-demand, personalized, social skills training (4). The robots envisioned by these efforts are part of a new field called socially assistive robotics (SAR), which aims to construct systems that support social and cognitive growth using social rather than physical means (5,6,7). These robots share characteristics of educational robots, which attempt to convey information typically via a tutor-student relationship (8), and rehabilitation robots, which provide structured physical therapy for deficits such as stroke and paralysis (9).
Exploratory studies from dozens of research groups have shown that many individuals with ASD enjoy interacting with robots and in many cases even demonstrate more appropriate social behaviors with robots than they do with peers or caregivers (10,11). These initial exploratory studies focused on short interactions, spanning tens of minutes or less, under controlled laboratory or clinic conditions, often involving sample sizes of five children or fewer, and focused exclusively on robot-directed behavior (7). While these studies generated considerable excitement, they held little clinical value. Results tended to fade with repeated exposures and may have been the result of novelty, appropriate control conditions were rarely considered, and experiments failed to demonstrate learning that generalized to human-directed actions (12). While a few studies did examine longer-term interactions (13), or demonstrated improved adult-directed social behavior (14), none were able to demonstrate skill acquisition that could be considered clinically meaningful that generalized beyond the specific robot encounter.
We report here the first demonstration of directly-assessed improvements in social skills in children with ASD following an in-home, 1-month intervention in which daily social skills games were conducted by an autonomous, socially-assistive robot (Fig. 1). This study differs from prior work in this domain in four important aspects. First, this study uses a fully autonomous robot system that operates for a 1-month deployment duration with no adjustments made by clinical or research staff. Many socially assistive robots still operate under tele-operative control, as autonomous operation for this duration is a substantial challenge in the robotics community even when static program requirements are used throughout the deployment (6). Second, unlike previous work where predefined protocols are followed explicitly (15), the system used here must adapt to the strengths and weaknesses of the individual child by changing the difficulty of individual tasks based on the preferences and performance of the child. As individuals with ASD have substantial individual differences in the type and severity of their social skill deficits, the need to adapt to an individual child is essential to enabling a positive learning outcome. Further, the interaction between the need for autonomy and the need for adaptation creates additional technical challenges. Third, this is the first study to provide therapy directly in homes with a fully autonomous robot. While clinical and laboratory spaces represent known environmental conditions that can be controlled or explicitly planned for, the unconstrained home environment requires more complex sensing and behavioral routines to deal with greater variation in environmental conditions. Finally, this study focuses primarily on demonstrations of clinically meaningful measures of performance using standard evaluation metrics that are conducted by an independent assessor away from the robot. This represents a challenging evaluation standard as a child must not only learn a skill while practicing with the robot, but then be capable of generalizing that skill to interactions with an adult in an environment that differs from the practice games used by the robot.
Fig. 1. Robot-assisted intervention system.

Our system consists of a social robot, touchscreen monitor, and two RGB cameras. The system supports triadic interactions between the robot, the child, and the caregiver. Software running on the perception computer uses an elevated camera to track both the child’s and caregiver’s attentional foci, while the other camera records the intervention session (Fig. 2). The main computer controls the flow of the intervention as well as the robot’s behavior to ensure presentation of coherent, meaningful intervention.
Results
Our robot-assisted intervention included a 30-minute session everyday for 30 days and involved triadic interactions among the social robot, the child, and the caregiver, providing opportunities for the child to interact and share experience with the caregiver (Fig. 2). The robot modeled social gaze behaviors such as making eye contact (Fig. 3) and sharing attention throughout the sessions, and provided feedback to and guided the participants in six interactive games. The six games targeted different social skills including social and emotional understanding, perspective taking, and ordering and sequencing (Fig. 4). Each session began with the robot telling a daily story to engage the participants. The session continued with three games, which varied from day to day, and concluded with a caregiver survey, where the caregivers rated their observations of the child’s social communication skills.
Fig. 2. A typical interaction between the robot, the child, and the caregiver during our deployment.

Our robot system was design to engage and facilitate interactions between the child and the caregiver, therefore providing opportunities for the child to practice social skills in a fun, natural way.
Fig. 3. Robot-initiated joint attention.

The robot models appropriate social gaze behavior by demonstrating context-contingent gaze and facilitates mutual gaze and experience sharing between the child and the caregiver. When the child is engaged with the robot (A), the robot directs the child’s attention to relevant task content on the screen (B). As the child’s attention shifts to the robot-directed focus on the screen, the robot then attempts to redirect gaze to the caregiver (C) in the hope of redirecting the child’s visual attention to the caregiver (D). (These demonstration images were recreated in the laboratory to show both robot and child behavior, as this perspective was not recorded by the deployed system.)
Fig. 4. Screenshots of social skills games.

A set of interactive games were developed to allow children with ASD to practice social skills through play. The games were designed to support interactions between the caregiver and the child as well as between the robot and the child. The games targeted three social skills, including social and emotional understanding (A) (“Story”), perspective taking (B) (“Rocket”), and ordering and sequencing (C) (“Train”).
Participants Information
Fourteen families with a child with ASD enrolled in this study. Two families withdrew, one due to unrelated health problems of a caregiver and one due to technical difficulties with the robot installation. Among the 12 families who finished the study, 5 of the children with ASD were females and 7 were males. Caregivers reported that all 12 children identified racially as white, and 4 of the 12 as Hispanic or Latino. These participants’ age ranged from 6 years to 12 years old (M = 9.02, SD = 1.41). All had nonverbal IQ scores ≥ 70 as determined by the Differential Ability Scales (16) (M = 94.17, SD = 20.06). Diagnosis of ASD was based on standard-in-field clinical best-estimate (CBE) diagnosis by licensed clinical psychologists and/or speech-language pathologists with extensive experience in autism diagnosis. Measures used in the diagnostic process included the Autism Diagnostic Interview - Revised (ADI–R; 17) caregiver interview and the Autism Diagnostic Observation Schedule (ADOS; 18) semi-structured play observation. Scores on the ADI-R and the ADOS reflect the presence of autism symptoms, with higher scores reflecting greater autism severity. The ADI-R is broken down into 4 domains: reciprocal social interactions (M = 17.64, SD = 6.98; cutoff for ASD = 10); communication (M = 16.36, SD = 4.74; cutoff for ASD = 8); restricted, repetitive, and stereotyped behaviors (M = 6.00, SD = 1.41; cutoff for ASD = 3); and history of early abnormal development (M = 3.44, SD = .73; cutoff =1). The ADOS yields outputs including a calibrated severity score (M = 7.08, SD = 2.02; scale from 1 to 10, cutoff for ASD = 4). All participants, in addition to receiving a CBE of ASD, scored above the ASD-cutoff on either the ADOS or the ADI-R.
All participants were recruited from a large database of children with ASD who have either participated in prior research studies with our laboratory or expressed interest in participation. Eligible families were contacted via email to inquire about their interest in participating. Given the scope of the project, the first 12 eligible families were enrolled. Inclusionary criteria were (1) age between 4 and 12 years old, (2) good medical health, (3) cooperative with testing, (4) English is a language spoken in the family, and (5) having been diagnosed with ASD and meet the characterization cutoffs described above. Exclusionary criteria were (1) a fragile health status and (2) suspected or diagnosed hearing loss or visual impairment, or diagnosed neurological abnormality significantly impacting on visual or auditory acuity.
All children in the study were enrolled in school programming full-time and received intensive special education services as consistent with the state standards for educating children with ASD. As the scope and form of these services and therapies varied substantially across participants based not only on their individual needs but also on family preferences and local resource availability, we utilize a single-subject withdrawal design (ABA) that allows each child to serve as their own control (see the Methods and Materials section for details). Caregivers were instructed to maintain consistent intervention services during their participation in the study.
Engagement and skills performance
A total of 127 hours of data was collected from the interaction between the 12 children, the robot, and their caregivers. This data included video and audio data, head orientation of both child and caregiver, interaction logs containing the robot utterances and actions, game logs for the tablet-based games, and caregiver survey responses. As our primary study design is focused on showing the efficacy of this intervention, we focus in this paper on the analysis of child social performance as measured by game performance, caregiver reports, and clinical measures.
The children combined initiated a total of 653 games with the robot, which resulted in 540 complete games for analysis. (Games that were shortened due to the end of time in the session were not considered for analysis.) On average, each child performed 23.25 sessions with the robot across the month, and each session lasted on average for 27 minutes and 42 seconds. After a month of interacting with the robot on a daily basis, the robot was able to maintain engagement with the child during the interactions: children played with the robot for an average of 27 minutes during the first five sessions and an average of 25 minutes during the last five sessions.
The robot adapted the difficulty of each individual game based on the child’s history of performance in each skill set. On the emotion understanding game “Story”, 86% of children reached the most difficult level of the game by the last session. On the perspective taking games, 58% and 100% of children reached the highest level on “Rocket” and “House” respectively. On the sequencing and ordering game “Train”, 67% of the children reached the highest level. The “Spaceship” and “Traveler” games used only a single difficulty level and are excluded from this analysis.
Binomial generalized linear mixed models (Fig. 5) were used to model the level attained by children as a proportion of the maximum possible level as a function of the specific game and session number. Game and session number were included as both fixed and random effects. Likelihood ratio tests on the resultant model indicated significant main effects of game, session, and their interaction (all ps < .001). In terms of overall performance (i.e. intercept) and gains over sessions (i.e. slope), the “House” game was easier than other games (intercept, slope: vs. “Story”, p=.001, p=.030; vs. “Rocket”, p<.001, p<.001; vs. “Train” p=.014, p=.030).
Fig 5. Proportion of maximum level achieved as a function of game session.

Curves were modeled in a binomial generalized linear mixed model with session and game as fixed and random effects. 95% confidence intervals are shown. Children advanced in the level of each game when they achieved over 75% of correct answers, and regressed a level when giving less than 25% correct answers. When achieving between 25% and 75% of the correct answers, the children would remain at the same level.
Joint Attention
Performance on the joint attention probe was measured and recorded at four time points: (a) T0: 30 days before intervention began, (b) T1: on the first day of robot intervention, (c) T2: on the last day of intervention and, (d) T3: 30 days after the end of the intervention. The difference between time points T0 and T1 was computed to measure change in joint attention during a period of time with no robot intervention and is denoted as the pretest. The delta between time points T1 and T2 was calculated to measure joint attention changes resulting from the robot-administered intervention and is denoted as the test phase. Finally, the delta between time points T1 and T3 was evaluated to measure the stability of any changes recorded during the robot-administered intervention and is denoted as the posttest.
Two participants were excluded for lack of data at one or more time points. Another participant was excluded for being out of the age range in which the task was normed (7–12 years of age). Group means were: T0: M=16.89, SD=4.46; T1: M=15.67, SD=3.81; T2: M=20.89, SD=3.79; T3: M=18.22, SD=5.02. A linear mixed model with compound symmetry repeated effects covariance indicated a significant time point effect, F(3,24) = 5.03, p = .008. Planned comparisons showed that while no pretest or posttest effect was observed (T1-T0, p = .395; T3-T1, p = .083), joint attention improvements occurred in the test phase (T2-T1, p = .001) (Fig. 6). Test phase changes were negatively associated with nonverbal reasoning performance on the DAS, r(9) = −.750, p = .020. These results are consistent with greater joint attention gains made by children with lower nonverbal ability. Exploration of relationships between baseline nonverbal ability and average baseline (T0 and T1) joint attention performance indicated a strong positive relationship, r(9) = .831, p = .005, suggesting children with lower nonverbal reasoning skills had more to grow in terms of joint attention skills. Joint attention performance at T1 was also positively Pearson’s correlated with modeled participant overall performance on the “House” (r(9) = .702, p = .035) and “Story” (r(9) = .705, p = .034) games, suggesting shared variance in performance.
Fig. 6. Result of joint attention assessment.

Probe Scores for the child at four different time points: 30 days before the robot intervention started, on the start day of the robot intervention, on the last day of the robot intervention, and 30 days after the end of the robot intervention. There was a significant increase in joint attention scores when comparing before the robot intervention and after it.
Caregivers Survey
Caregivers completed an on-screen survey immediately after each day’s intervention session during the test phase. In all but one family, these interactions were conducted with the same caregiver (1 father, 1 grandmother, and 9 mothers).
The survey consisted of 5-point Likert scale ratings. The questions were grouped into two categories: questions on how children interacted with caregivers during the past 24 hours, and parallel questions about interactions with other people, and one final question regarding engagement. We compared the ratings scored by the caregivers on the first day and last day of interventions with paired sample t-tests. All 12 caregivers’ responses are included in the analysis.
Caregivers reported increased social skill performance between their child and themselves, including more eye contact (t(11) = −2.462, p = .032) with them on the last day of the intervention (M = 3.75, SD = 1.06) compared to the first day (M = 3.00, SD = 0.00), more attempts to initiate communication (t(11) = −2.930, p = .014) with them on the last day (M =4.08, SD = 1.00) than on the first day (M = 3.17, SD = 0.39), and more frequent responses to communication bids from the caregiver (t(11) = −3.000, p = .012) on the last day (M = 3.83, SD = 0.94) than on the first day (M = 3.08, SD = 0.29). (Fig 7.A.)
Fig. 7. Result of caregiver survey.

Caregivers reported increased eye contact, increased initiation of communication, and increased response to communication bids with them (A) and with other people (B). Based on comparisons of ratings from the last day of the robot intervention (T2) to the first day of the intervention (T1), these results showed that caregivers were able to observe improved communication abilities of the children beyond our robot-assisted intervention sessions over the period of 30 days.
Caregivers also reported increased social skill performance between their child and other people, including more eye contact (t(11) = −3.447, p = .005) with other people on the last day of the intervention (M = 3.83, SD = 0.83) when compared to the first day (M = 3.08, SD = 0.29), more attempts to initiate communication (t(11) = −3.527, p = .005) with other people on the last day (M = 3.91, SD = 0.90) than on the first day (M = 3.00, SD = 0.00), and more frequent responses to communication bids from other people (t(11) = −3.458, p = .005) on the last day (M = 3.75, SD = 0.75) than on the first day (M = 2.91, SD = 0.29). (Fig 7.B.)
Finally, caregivers were asked daily to rate how easy it was to engage their child with the robot therapy session. To confirm that the continued length of engagement was not solely a result of compliance to the protocol instruction, we modeled the engagement rating with a cumulative link mixed model fitted with an adaptive Gauss-Hermite quadrature approximation as a function of day with random participant effects. This model revealed no significant effect of day on engagement (p = .822). This suggests that participant engagement did not change in a systematic fashion throughout the study.
Discussion
The potential benefit of a socially assistive robot lies in the ability to provide personalized, on-demand, and structured cognitive or social support to augment the efforts of clinicians, teachers, and families. In the ideal case, robots could provide personalized support, whenever and wherever needed, and could be capable of producing lasting enhancements in social and communicative skills not only in human-robot interactions but also in human-human interactions (4). The system presented here takes steps in this direction beyond the current state of the art, but also does not yet live up to all of these grand visions. We focus our discussion around the points in which the current work makes substantial improvements and also describe the limitations and areas requiring continued focus as this field progresses.
Autonomous interaction.
Our deployed robots operated autonomously without any experimenter intervention for a total of 127 hours over 279 sessions. Caregivers contacted our 24/7 help line a total of 8 times: 6 times for confirmation that they were using the system correctly and which required no action from our team, and 2 times for a technical issue that was prompted by the sudden disappearance of an on-line software library which required a software update and was resolved quickly. Robot-assisted autism intervention in previous studies were mostly short episodic interactions that rarely lasted more than 30 minutes (19, 14) and usually required experimenters to supervise robot-directed actions (though, see 20 and 21 for exceptions). Moving from tele-operated to autonomous interactions presents significant challenges in computational perception and robot control in order to create meaningful therapeutic training. While challenging, increasing robot autonomy in assisted therapy has potential to reduce therapists’ cognitive load and ensures consistent therapy for the children with autism (21). Our system demonstrated the possibility and potential of autonomous robot interventions for autism, which would enable the implementation and application of robot-assisted intervention at a large scale in various environments, accelerating us towards the goal of achieving clinical significance.
Adaptive intervention.
Sustaining engagement with participants is key to effective interventions. Repetitive and unchallenging tasks are likely to bore participants, who then would disengage from the intervention and miss opportunities to practice and improve on targeted skills. As informed by the challenge point theory (22), optimal learning occurs when the task is neither too easy nor too difficult. Our system sought to keep the participating children challenged and adapted the difficulty level of practice games to the children’s skill performances as measured in the games. This adaptation allowed the children to practice and improve the target skills at their own pace. Our results confirmed that the children continued to engage with our system throughout the test phase. We speculate that such engagement with our robot-assisted intervention was crucial to the observed improvements in the children’s social skills.
Deployment in uncontrolled environments.
Deployment of robotic systems outside controlled, laboratory settings is challenging. Our deployment needed to address various environmental constraints and meet different human considerations. For example, the setup location of our system was constrained by electrical power, network connectivity, and family preferences. For instance, one child was particularly sensitive to light; therefore, our system had to be set up in a dimmed room, which created additional challenges for our perception system. Furthermore, our deployment needed to accommodate other family members’ needs, especially the participating child’s siblings. We provided robotic toys to the siblings, so that they would not interrupt daily intervention sessions. We also made the operation of the system user-friendly by automating startup procedures and by providing a simple checklist to ensure that caregivers would feel comfortable operating the systems on their own on a daily basis. These challenges, constraints, and considerations are unique to field deployment of robotic systems aiming to interact with non-specialist users every day over a long period of time. Yet, meeting these requirements are a practical necessity for the integration of robots into our environments to provide daily support.
Contributions of the social robot.
Although the focus of this study is not to understand the specific influence that any of the individual components of our system (including the robot, tablet-based games, perception system, etc.) have on our robot-mediated intervention, we believe that the social robot contributed positively to the observed behavioral based on three converging lines of research. First, in triadic interactions between a child, an adult, and a third interaction partner, children with ASD demonstrate more social behavior toward the adult when the third interaction partner is a robot rather than a tablet-based game or another adult (14). Second, the embodiment of the robot provides necessary affordance to convey gaze cues that are central to our behavioral intervention. Known as the Mona Lisa gaze effect (23), agents on a flat screen are limited in accurate communication of gaze directionality. Third, in tutoring interactions with both adults and typically-developing children, physically embodied robots increase learning outcomes (24, 25), increase compliance to instructions (25), and increase user engagement during the interaction (26, 27) over screen-based agents. Nonetheless, we acknowledge that, in this study, the impact of the robot (or any other system component) cannot be measured independently. We present this as a limitation of this study, and an area for future work.
Improvements in caregiver-reported social behavior.
Over the month of the robot-based intervention, caregivers reported improved social behavior directed both toward themselves and toward others in areas including eye contact, initiation of communication, and responses to communication. The change in reported behavior on the caregiver survey could be, in part, related to the caregiver attending more to the child’s social communication skills. It is unlikely the change is due solely to this increased focus given the nature of the sample; caregivers of children with ASD provide ongoing support to their children in this area and generally monitor social communication development.
Improvements in clinical measures.
Our results also showed improvements of children’s joint attention in the absence of the robot, indicating that the children were able to demonstrate greater skill in the context of human interactions. These results are independent of the caregiver reports and are not subject to the same limitations. These results advanced and differed significantly from prior research in robot-assisted autism therapy, where behavioral improvements in children with ASD were mostly documented in the context of robot-directed interactions (20). Successful demonstration of improvement in human interactions is the ultimate goal of robot-assisted intervention, evolving beyond the mediation and scaffolds that assistive robots offer during interactions with other people. Our results provide evidence illustrating possible transferable social skills beyond robot-mediated interactions in naturalistic human interactions.
The present results have multiple clinical implications. Joint attention is the critical foundation for many higher-level social communication skills, including reciprocal exchanges and perspective-taking. Therefore, with improvements in joint attention following this intervention, in time, we may see downstream effects on other higher-level skills. In fact, we did see broader gains in the context of the current study, even in this 1-month intervention. Future work with larger and longer trials will clarify this promising, yet preliminary, result. These results give promise to the potential for robot intervention studies in group treatment to facilitate interactions between peers and improve both foundational and high-level social skills in this context.
The specific developmental growth seen in the children during their participation in this study is likely due to our system, as opposed to other treatments they received, as the children did not show the same magnitude of gains during the pretest phase, just the test phase, and the children’s concomitant treatments remained stable throughout their participation in the study. The lack of significant effects between the first day of the test phase and the end of the posttest phase further emphasizes that the effects may be more likely due to the effects of the intervention components of this study, deteriorating over time in the absence of continued support. That said, from a clinical intervention perspective, our study is an open-label pilot. Future studies extending the duration of the study and with randomization with appropriate control groups are necessary to verify the gains we have observed and attributed to our intervention. In addition, the factors associated with long-term preservation of joint attention improvements in ASD remain to be elucidated. Finally, future randomized controlled studies will better control for practice effects of the tasks.
While our results provide evidence showing the benefits and possibility of using robot-assisted autism therapy for clinical intervention, limitations of our system motivate future research on the development of effective robot-based interventions. In particular, our system relied on pre-specified interaction content, which included daily opening stories and a fixed set of behavioral responses. While this approach was appropriate for our target scenarios, it would not scale well for interventions that span a longer period of time (e.g., over 30 minutes per day and over 30 days). How to generate coherent, engaging interaction content automatically is a core challenge for realizing a long-term, autonomous robot-assisted intervention, or human-robot interaction in general. Second, our intervention personalization was focused on adjusting difficulty levels of the practice games, analogous to personalization of educational contents in intelligent tutoring systems (ITS). Our personalization algorithm was simplistic, though it matched the complexity of personalization algorithms used successfully to demonstrate learning gains in other ITS systems (e.g. 28). More complex and detailed modeling of a child’s capabilities would likely provide more substantial impact. Furthermore, to support the wide variety of behavioral characteristics of individuals with ASD effectively, adaptive models that prioritize and personalize needs and preferences in addition to skill performance are necessary to maximize the potential of robot-assisted interventions. Third, our system was designed to provide targeted interventions, involving interactions between the robot, child, and caregiver for approximately 30 minutes each day. Although this design provided structures for targeted intervention, it missed naturalistic intervention opportunities outside of the intended sessions. These three limitations necessitate smart, adaptive systems that can provide personalized, engaging interventions to children with ASD in a variety of situations over long periods of time.
Materials and Methods
Objectives and Study Design
The objectives of this study were to investigate how a social robot may deliver behavioral intervention to children with ASD both autonomously and effectively outside clinical settings, as well as how such robot-assisted intervention can improve these children’s social-communicative abilities. This study was modeled after single-subject withdrawal (ABA) designs (29,30). This design included pretest (A), test (B), and posttest (A) phases, each phase lasting for about 30 days. The pretest phase served as a comparison baseline, capturing possible maturation of social communication abilities and the effectiveness of any other therapies or interventions that the family may have been using. The test phase involved the in-home deployment of a socially assistive robot system that engaged the participating child in our intervention program, which was based on intervention activities commonly used in clinical settings. The posttest phase sought to explore whether the benefits provided by our robot-assisted intervention would be sustained after the removal of the system. This study design is suitable for investigating the effects of a single intervention and for when there are wide variances in participants’ characteristics and responses to the intervention. Informed consent from families and assent from minors was obtained in all cases, as approved by the Yale University Institutional Review Board.
Assessment
To assess a child’s ability to respond to joint attention bids in their familiar environments, we employed the validated, naturalistic joint attention assessment of Bean and Eigsti (31). This assessment includes six naturalistic prompts that can be delivered at any point during an interaction with the child and is designed particularly for school-age children and adolescents. The six prompts examine different aspects of joint attention, including gaze following, response to name and a greeting opportunity, and recognition of the other person’s current interest. This assessment of joint attention was administered four times throughout the study while a researcher was interacting with a child in play-based activities.
To understand whether or not a child’s behaviors of social communication changed over time outside intervention sessions, we asked the child’s caregiver to fill out a survey regarding his/her own observations of the child’s communicative behaviors at the end of each daily session. The survey questions sought to measure the broader influence of our robot-assisted intervention outside of intervention sessions, focusing on the child’s ability to make eye contact with, initiate communication with, and respond to communication bids from the caregiver and others.
Robot-Assisted Intervention System
Our intervention system consisted of a social robot, a 24-inch touch screen, two external RGB cameras, and two computers. The social robot used was an early prototype of the Jibo robot (32). The Jibo robot is a 12-inch table-top robot with three degrees of freedom, capable of turning its head and body around 360 degrees. The robot can exhibit expressive behaviors through body movements, a ring of color-changing LED lights, and a pair of animated eyes (e.g., blinking and dilation). These capabilities allow the robot to make eye contact with the participants and signal shared attention. Additionally, the robot can deliver information verbally to the participants through its internal speakers. The 24-inch touch screen presented educational content and served as a shared medium where the robot and the participants could all interact with and reference to. One of the cameras tracked both the child’s and the caregiver’s attentional foci as approximated by head orientations, while the other camera recorded the intervention session.
Our software system (Fig. 8), which involved attention tracking and intervention presentation, was implemented in the Robot Operating System (ROS) framework (33). The attention tracking subsystem, running on one of the computers, continuously approximated users’ attentional targets in the environment. Using RGB camera stream input, the system estimated and tracked head poses and orientations using Constrained-Local-Model (CLM) face tracking and landmark detection algorithms (34), and approximated attentional targets according to the estimated head poses and orientations. The intervention presentation subsystem, running on the other computer, ensured smooth delivery of curricular content. It controlled the robot’s behaviors, scheduled intervention content, and adjusted difficulty levels of the social skills games.
Fig. 8. Diagram of Software Components.

Our software system consists of several components responsible for attention tracking of the participants, robot behavior control, and intervention presentation. These components together create rich, engaging interactions for our robot-assisted autism therapy. These components operate within the ROS framework.
In our implementation, we manually prepared interaction scripts that specified predefined behavioral animations for the robot, daily opening stories, and verbal encouragement and feedback to the participants. When the robot was not displaying pre-specified behaviors in a prepared interaction script, it maintained eye contact with the child to show engagement. The robot also shared attention with the child by looking toward the visual content on the screen from time to time throughout each session. These behaviors of making eye contact and sharing visual attention were meant to model social gaze behaviors for the child.
In addition to targeting the core social skill of joint attention, we designed and developed six interactive games that provide opportunities for the child to practice social and emotional understanding, perspective taking, and ordering and sequencing while interacting with the robot and the caregiver. The six games are an emotional understanding game (“Story”), two barrier games that facilitate perspective taking (“House” and “Rocket”), and three ordering and sequencing games (“Train,” “Spaceship,” and “Traveler”). Each game involves multiple levels of difficulty, ranging from 1–4 to 1–8, except for “Spaceship” and “Traveler” which had only one difficulty setting. Depending on the child’s performance in the game, the system adjusted the difficulty level accordingly. As inspired by the challenge point theory (22), our personalization module was focused on delivering learning contents with appropriate levels of difficulty to increase learning gains while reducing frustration. The personalization module kept track of the child’s performance in game activities, providing approximate measures of their abilities of social and emotional understanding, perspective taking, and ordering and sequencing. Using these performance measures, the module followed a simple decision-tree mechanism to decide the difficulty level of the game for the next round of interaction. In our implementation, we used 25% and 75% as criteria for decreasing and increasing a difficulty level, respectively. Similar approaches of discrete adaptation have been used in robot-child tutoring applications (25, 35, 36).
Social and emotional understanding.
The “Story” game targets the skills of social and emotional understanding. A typical example of this game is as follows. The robot provides a social situation, displayed as cartoon-like images on the touchscreen, and asks the child to choose what he/she thinks the story character is feeling at different points in the story by selecting one of multiple options displayed on the screen. As the child progresses, the social stories become longer and more complex. To succeed in this game, the child needs to understand the social situations and emotional states of the characters.
Perspective taking.
Two virtual barrier games, “Rocket” and “House”, target the ability of taking the other person’s perspective on a joint task. Modeled after physical barrier games commonly used in clinical interventions, these games provide spatial information to one of either the child or caregiver and ask them to relay that information to the other verbally. In both games, the robot facilitates interactions between the child and their caregiver and acts as a game moderator by keeping time and providing motivational support. In “Rocket”, the child and the caregiver take turns building a rocket ship. The first player builds a rocket by dragging modular component parts onto a rocket template while the second player looks away. The screen is then reset to hide this design and the first player must explain to the second player how to recreate the design. If the two designs are identical, the players have succeeded and win the game. Similarly, in “House”, the child and the robot take turns in the roles of builder and guesser. The builder builds a virtual house that comprises of various designs and materials while the guesser looks away. The game then shows six possible designs, one of which was constructed by the builder. The guesser then asks questions about whether the builder’s house has a particular design or material in order to guess which of the shown designs is the builder’s. These games provide opportunities for the child not only to understand that the caregiver or the robot has a different perspective but also to practice turn taking and verbal communication.
Ordering and sequencing.
The “Train” sequencing game targets the skills of ordering and sequencing. In this game, the robot instructs the child to build a train by dragging parts onto a template. To succeed in this game, the child needs to follow the robot’s instructions carefully in sequence. Two additional games, “Spaceship” and “Traveler”, involve various tasks such as sorting objects in order. In an ordering task, the child needs to place objects in the right order to complete the task successfully.
Substantial effort was placed into making the system robust and easy to use. Prior to the deployment described in this paper, we conducted multiple pilot tests of the system and the installation process in the homes of the research team. We attempted to make the system easy for families to use by limiting thstartup required to 4 button presses, providing in-home training on the first day, and continuous system state logging to allow for most troubleshooting to require only powering the system off and then on again with no loss of data. Significant efforts were made to minimize disruptions to normal operations after installation: backup power supplies in the system base guarded against short power failures, hardware components (including the cameras, robot, and tablet) were secured in place to the table, a troubleshooting sheet and a 24/7 technical support line (via email and phone) was provided to participating families. Finally, the system limited play use to conform to the study design; the robot would play games only for one session each day and only for a maximum of 30 minutes.
Supplementary Material
Gameplay dataset. Maximum gameplay level achieved by each child for each instance of game played, organized by game and play session, as shown in Figure 5.
Joint attention dataset. Probe scores for each child at four different time points: 30 days before the robot intervention started, on the start day of the robot intervention, on the last day of the robot intervention, and 30 days after the end of the robot intervention. Summarized in Figure 6.
Cargiver survey dataset. Survey results provided by the caregiver to a set of Likert-scale questions regarding the child’s social skills, organized by participant and question for the first and last day of the robot deployment, as shown in Figure 7.
Acknowledgments:
We are indebted to the other members of the Expedition in Computing on Socially Assistive Robotics for their contribution of social skills game software and the early development of other socially assistive systems that led to the deployment reported here, especially M. Mataric, C. Breazeal, C. Nass, F. Volkmar, M. Jung, A. Ramachandran, S. Sebo, D. Becerra, C. Claybaugh, J. Kory, S. Shen, M. Baranov and A. Waugh. We also thank L. Hall for scheduling and administrative support, L. Scassellati for photo assistance, and all of the families that welcomed a robot into their homes.
Funding: Support was provided by an NSF Expedition in Computing, PI Scassellati, #1139078 (Socially Assistive Robotics). F.S. was supported by funding by NIH grant #K01MH104739.
Footnotes
Competing interests: L.B. is now supported by Van Robotics, which constructs tutoring robots though not for children with ASD, and F.S. and B.S. serve as advisor board members.
Data and materials availability: All data needed to evaluate the conclusions in the paper are present in the paper or the Supplementary Materials.
References and Notes
- 1.American Psychiatric Association. Diagnostic and statistical manual of mental disorders (5th ed.). Arlington, VA: American Psychiatric Association; (2013). [Google Scholar]
- 2.Kanai C, Toth G, Kuroda M, Miyake A & Itahashi T. Social Skills in Autism Spectrum Disorders. In Handbook of Social Behavior and Skills in Children (pp. 217–248). Springer, Cham. (2017). [Google Scholar]
- 3.Mackintosh VH, Goin-Kochel RP & Myers BJ. “What do you like/dislike about the treatments you’re currently using?” A qualitative study of parents of children with autism spectrum disorders. Focus on Autism and Other Developmental Disabilities, 27(1), 51–60 (2012). [Google Scholar]
- 4.Scassellati B. How social robots will help us to diagnose, treat, and understand autism. Robotics research, 552–563: Springer, Berlin, Heidelberg: (2007). [Google Scholar]
- 5.Feil-Seifer D & Matarić MJ. Defining socially assistive robotics. In Rehabilitation Robotics, 2005. ICORR 2005. 9th International Conference on (pp. 465–468). IEEE. (2005). [Google Scholar]
- 6.Matarić MJ & Scassellati B. Socially assistive robotics. In Springer Handbook of Robotics (pp. 1973–1994). Springer, Cham. (2016). [Google Scholar]
- 7.Tapus A, Matarić MJ & Scassellati B. Socially assistive robotics [grand challenges of robotics]. IEEE Robotics & Automation Magazine, 14(1), 35–42. (2007). [Google Scholar]
- 8.Belpaeme T, Kennedy J, Ramachandran A, Scassellati B & Tanaka F. Social Robots for Education: A Review. In submission, Science Robotics. (2018) . [DOI] [PubMed] [Google Scholar]
- 9.Tejima N. Rehabilitation robotics: a review. Advanced Robotics, 14(7), 551–564. (2001). [Google Scholar]
- 10.Scassellati B, Admoni H & Matarić MJ. Robots for use in autism research. Annual review of biomedical engineering, 14, 275–294. (2012). [DOI] [PubMed] [Google Scholar]
- 11.Kim ES, Paul R, Shic F & Scassellati B. Bridging the research gap: Making HRI useful to individuals with autism. Journal of Human-robot interaction, 1(1). (2012). [Google Scholar]
- 12.Diehl JJ, Schmitt LM, Villano M & Crowell CR. The clinical use of robots for individuals with autism spectrum disorders: A critical review. Research in autism spectrum disorders, 6(1), 249–262. (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Wainer J, Robins B, Amirabdollahian F & Dautenhahn K. Using the humanoid robot KASPAR to autonomously play triadic games and facilitate collaborative play among children with autism. IEEE Transactions on Autonomous Mental Development, 6(3), 183–199. (2014). [Google Scholar]
- 14.Kim ES, Berkovits LD, Bernier EP, Leyzberg D, Shic F, Paul R & Scassellati B. Social robots as embedded reinforcers of social behavior in children with autism. Journal of autism and developmental disorders, 43(5), 1038–1049. (2013). [DOI] [PubMed] [Google Scholar]
- 15.Zheng Z, Bekele E, Swanson A, Weitlauf A, Warren Z & Sarkar N. The impact of robots on children with autism spectrum disorder. Autism Imaging and Devices, 397. (2017). [Google Scholar]
- 16.Elliott CD. DAS administration and scoring manual. San Antonio, TX: Psychological Corporation. (1990). [Google Scholar]
- 17.Lord C, Rutter M & Le Couteur A. Autism Diagnostic Interview-Revised: a revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders. Journal of autism and developmental disorders, 24(5), 659–685 (1994). [DOI] [PubMed] [Google Scholar]
- 18.Lord C, Rutter M, DiLavore PC, Risi S, Gotham K & Bishop SL. Autism diagnostic observation schedule (ADOS) Los Angeles. CA: Western Psychological Services; (1999). [Google Scholar]
- 19.Boccanfuso L, Kim ES, Snider JC, Wang Q, Wall CA, DiNicola L, ... & Chawarska K. Autonomously detecting interaction with an affective robot to explore connection to developmental ability. In Affective Computing and Intelligent Interaction (ACII), 2015 International Conference on (pp. 1–7). IEEE; (2015). [Google Scholar]
- 20.Kozima H, Nakagawa C & Yasuda Y. Children–robot interaction: a pilot study in autism therapy. Progress in Brain Research, 164, 385–400 (2007). [DOI] [PubMed] [Google Scholar]
- 21.Robins B, Dautenhahn K & Dickerson P. From isolation to communication: a case study evaluation of robot assisted play for children with autism with a minimally expressive humanoid robot. In Advances in Computer-Human Interactions, 2009. ACHI’09. Second International Conferences on (pp. 205–211). IEEE; (2009). [Google Scholar]
- 22.Guadagnoli MA, Timothy DL, “Challenge Point: A Framework for Conceptualizing the Effects of Various Practice Conditions in Motor Learning”, Journal of Motor Behavior; 36(2): 212–24, 2004. [DOI] [PubMed] [Google Scholar]
- 23.Al Moubayed S, Edlund J & Beskow J. Taming Mona Lisa: communicating gaze faithfully in 2D and 3D facial projections. ACM Transactions on Interactive Intelligent Systems, 1(2), 25 (2012). [Google Scholar]
- 24.Leyzberg D, Spaulding S, Toneva M & Scassellati B. The physical presence of a robot tutor increases cognitive learning gains. In Proceedings of the Annual Meeting of the Cognitive Science Society (Vol. 34, No. 34).(2012). [Google Scholar]
- 25.Ramachandran A, Huang CM, Gartland E & Scassellati B. Thinking Aloud with a Tutoring Robot to Enhance Learning. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction (pp. 59–68). ACM; (2018). [Google Scholar]
- 26.Powers A, Kiesler S, Fussell S & Torrey C. Comparing a computer agent with a humanoid robot. In Human-Robot Interaction (HRI), 2007 2nd ACM/IEEE International Conference on (pp. 145–152). IEEE; (2007). [Google Scholar]
- 27.Pereira A, Martinho C, Leite I & Paiva A. iCat, the chess player: the influence of embodiment in the enjoyment of a game. In Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems-Volume 3 (pp. 1253–1256). International Foundation for Autonomous Agents and Multiagent Systems; (2008). [Google Scholar]
- 28.Leyzberg D, Spaulding S & Scassellati B. Personalizing robot tutors to individuals’ learning differences. In Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction (pp. 423–430). ACM; (2014). [Google Scholar]
- 29.Byiers BJ, Reichle J & Symons FJ. Single-subject experimental design for evidence-based practice. American Journal of Speech-Language Pathology, 21(4), 397–414 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Wagner A. Designing research studies on psychosocial interventions in autism. Journal of autism and developmental disorders, 37(2), 354–366 (2007). [DOI] [PubMed] [Google Scholar]
- 31.Bean JL & Eigsti IM. Assessment of joint attention in school-age children and adolescents. Research in Autism Spectrum Disorders, 6(4), 1304–1310 (2012). [Google Scholar]
- 32.Jibo, Inc. Jibo. Retrieved from http://www.jibo.com. (2018).
- 33.Quigley M, Conley K, Gerkey B, Faust J, Foote T, Leibs J, ... & Ng AY. ROS: an open-source Robot Operating System. In ICRA workshop on open source software. 3(3.2), 5 (2009). [Google Scholar]
- 34.Baltrusaitis T, Robinson P & Morency LP. Constrained local neural fields for robust facial landmark detection in the wild. In Computer Vision Workshops (ICCVW), 2013 IEEE International Conference on (pp. 354–361). IEEE; (2013). [Google Scholar]
- 35.Ramachandran A, Huang CM & Scassellati B. Give me a break!: Personalized timing strategies to promote learning in robot-child tutoring. In Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction (pp. 146–155). ACM; (2017). [Google Scholar]
- 36.Ramachandran A, Litoiu A & Scassellati B. Shaping productive help-seeking behavior during robot-child tutoring interactions. In The Eleventh ACM/IEEE International Conference on Human Robot Interaction (pp. 247–254). IEEE; (2016). [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Gameplay dataset. Maximum gameplay level achieved by each child for each instance of game played, organized by game and play session, as shown in Figure 5.
Joint attention dataset. Probe scores for each child at four different time points: 30 days before the robot intervention started, on the start day of the robot intervention, on the last day of the robot intervention, and 30 days after the end of the robot intervention. Summarized in Figure 6.
Cargiver survey dataset. Survey results provided by the caregiver to a set of Likert-scale questions regarding the child’s social skills, organized by participant and question for the first and last day of the robot deployment, as shown in Figure 7.
