Can Robotic Interaction Improve Joint Attention Skills?

Abstract

Although it has often been argued that clinical applications of advanced technology may hold promise for addressing impairments associated with autism spectrum disorder (ASD), relatively few investigations have indexed the impact of intervention and feedback approaches. This pilot study investigated the application of a novel robotic interaction system capable of administering and adjusting joint attention prompts to a small group (n = 6) of children with ASD. Across a series of four sessions, children improved in their ability to orient to prompts administered by the robotic system and continued to display strong attention toward the humanoid robot over time. The results highlight both potential benefits of robotic systems for directed intervention approaches as well as potent limitations of existing humanoid robotic platforms.

Introduction

According to the Centers for Disease Control and Prevention (2012) an estimated 1 in 88 children and an estimated 1 out of 54 boys in the United States have an autism spectrum disorder (ASD). ASD is associated with enormous individual, familial, and social cost across the lifespan (Amendah et al. 2011; Ganz 2007). The cumulative ASD literature suggests early intensive behavioral interventions are efficacious for many children (Dawson et al. 2010). However, many families and service systems struggle to provide intensive and comprehensive evidence-based early intervention due to extreme resource limitations (Al-Qabandi et al. 2011; Warren et al. 2012). Further, even when such services are provided many children continue to display potent impairments across many domains of functioning (Warren et al. 2011). As such, there is an urgent need for more efficacious treatments whose realistic application will yield more substantial impact on the neurodevelopmental trajectories of young children with ASD within resource strained environments. Given recent rapid technological advances, it has been argued that specific computer and robotic applications could be effectively harnessed to provide innovative clinical treatments for individuals with ASD (Goodwin 2008; Bekele et al. 2013). The current pilot project examined the use of a novel robotic technology as part of an interactive intervention environment for improving early joint attention skills in children with ASD.

Work toward more impactful treatments has often focused on improving early joint attention skills since these skills are thought to be fundamental social communication skills of the disorder (Kasari et al. 2008, 2010). Joint attention refers to a social exchange in which a child coordinates attention between a social partner and an aspect of the environment. Fundamental differences in early joint attention skills likely underlie the deleterious neurodevelopmental cascade of effects associated with the disorder (Dawson et al. 2010). The joint attention intervention literature to date suggests that early intervention can systematically improve these skills and such improvements partially mediate improvements in other critical developmental areas, including social and language outcomes (Kasari et al. 2010; Poon et al. 2011).

Across interventions, which vary widely in terms of scope and methodology, transactional approaches that attempt to combine the advantages of developmental and discrete trial approaches via intensive graduated systems of prompts in game-like, interactional frameworks hold substantial promise for improving these core skills (Yoder and McDuffie 2006). Further, the accumulated sum of the early intervention literature to date suggests that social communication intervention approaches are most effective when children show sustained engagement with a variety of objects, can be utilized within intrinsically motivating settings, and when careful adaptation to small gains and shifts can be incorporated and utilized over time (Poon et al. 2011; Yoder and McDuffie 2006). Given these factors, as well as purported relative strengths and differences in understanding physical and visual worlds relative to social worlds, responding to technologically cued feedback, and intrinsic interests in technology for many, but not all, young children with ASD (Annaz et al. 2012; Diehl et al. 2012; Klin et al. 2009), it is logical to hypothesize that robotic technology could be used as a tool for the development of enhanced joint attention interventions.

A number of research groups have studied the response of children with ASD to both humanoid robots and non-humanoid toy-like robots. Data from these groups have demonstrated that many individuals with ASD show a preference for robot-like characteristics over non-robotic toys, and in some circumstances even respond faster when cued by robotic movement than human movement (see Diehl et al. 2012 for review). Although this research has primarily been accomplished with school aged children and adults, research noting the preference for very young children with ASD to orient to nonsocial contingencies rather than biological motion suggests that downward extension of this preference may be particularly promising (Annaz et al. 2012; Klin et al. 2009). In this regard, recent works have documented that brief interactions with robotic systems may result in concurrent increases in certain aspects of social behavior like language production (Kim et al. 2012) or enhanced social interactions (Duquette et al. 2008; Feil-Seifer and Mataric 2011). While these approaches have certainly suggested the potential and value of robots for potential intervention applications, such approaches have not yet systematically examined how directed robotic intervention and feedback approaches may impact core symptoms of impairment over time. Ultimately, questions of impact and generalization of skills are critical for understanding the true value of adaptive robotic interactions to ASD related intervention.

In the current project, we tested over the course of several sessions a novel adaptive robot-mediated architecture capable of administering joint attention prompts via humanoid robot and contingently activating aspects of the intervention environment to enhance performance. This study built upon an initial feasibility study wherein we developed a prototype system capable of administering joint attention tasks to young children with ASD (Bekele et al. 2012, 2013). In this prior work, we developed a test-bed that consisted of a humanoid robot NAO, a series of 23 inch networked computer monitors capable of displaying relevant recorded task stimuli, and an infrared camera system capable of inferring gaze based on a LED instrument baseball cap worn by the participant. We then compared performance and gaze detection for a sample of six typically developing children and six children with clinically confirmed ASD diagnosis (ages 3–5; IQ range = 49–102) as well as variable baseline skills regarding response to joint attention.

Within this pilot system, a series of joint attention prompts were administered via either a human administrator or the humanoid robot with randomized presentation to control order effect. The child sat in a chair across from the robot or interventionist for the trial block and was instructed through a hierarchy of prompts (i.e., head/gaze shifts, pointing, target activation) to look to a target. The system registered gaze across all trials and provided reinforcement for looking through a simple reinforcement protocol (e.g., praise and target activation). Available data suggested that children with ASD spent approximately 27 % more time looking toward the robot administrator than the human administrator, that they did not fixate on either robot or target, and ultimately directed gaze correctly to the target for 95.83 % of the total 48 trials, a rate equal to TD success. Further, children successfully oriented to robotic prompting, meaning they responded to robot prompts prior to target activation, at very high levels (i.e., ASD = 77.08 % success; TD = 93.75 %).

Collectively, these findings provide promising support for the capabilities and capacity of the current system to engage preschoolers with ASD. Preschool children with ASD directed their gaze more frequently toward the humanoid-robot administrator, accurately responded to robot administered joint attention prompts at high rates, and looked away from target stimuli at rates comparable to typically developing peers. This suggests that robotic systems endowed with enhancements for successfully pushing toward correct orientation to target, either with systematically faded prompting or by embedding coordinated action with human-partners, might be capable of taking advantage of baseline enhancements in non-social attention preference (Klin et al. 2009; Annaz et al. 2012) to meaningfully enhance coordinated attention skills. While this pilot data provided preliminary evidence that robotic stimuli and systems may have some utility in preferentially capturing and shifting attention, at the same time such work did not provide evidence that attentional preferences were either sustained over time or that such preferences could actually improve performance with repeated exposure. In this present work, we had young children participate in a series of four interaction sessions with our robot-mediated joint attention prompting system. We specifically hypothesized that children would demonstrate improved within-system performance on response to joint attention tasks and that they would not demonstrate substantially diminished attention to the humanoid robot over this time frame.

Methods

Participants

Six children with ASD (age m = 3.46, SD = 0.73; see Table 1) were recruited through an existing university based clinical research registry. All children had received a clinical diagnosis of ASD based on DSM-IV-TR (APA 2000) criteria from a licensed psychologist, met the spectrum cut-off on the autism diagnostic observation schedule (ADOS; Gotham et al. 2007, 2009; Lord et al. 1999, 2000) administered by a research reliable clinician, and had existing data regarding cognitive abilities in the registry (Mullen Scales of Early Learning; Mullen 1995). Although not selected a priori based on specific joint attention skills, varying levels of baseline abilities on the ADOS regarding formal assessments of joint attention (i.e., varied abilities on Responding to Joint Attention item of the diagnostic instrument) were present in the sample. The most recent assessments available in the registry for each child were utilized for descriptive purposes (time between assessment and enrollment, m = 1.13 years, SD = 0.65). Given the lag between original assessment and study participation all parents were asked to complete both the Social Communication Questionnaire (SCQ) (Rutter et al. 2003) and the Social Responsiveness Scale (SRS) (Constantino and Gruber 2002) to index current ASD symptoms (see Table 1).

Table 1.

Participant characteristics

Participant	Age	MSEL	ADOS CS	ADOS RJA	SRS- 2	SCQ
1	4.38	81	9	1	85	24
2	2.52	69	5	2	59	11
3	2.75	107	9	3	75	16
4	3.48	78	10	0	69	18
5	3.48	58	9	2	75	24
6	4.13	49	10	1	63	9
M	3.46	73.67	8.67	1.50	71.00	17.00
SD	0.73	20.29	1.86	1.05	9.38	6.32

MSEL Mullen scales of early learning, ADOS CS autism diagnostic observation schedule comparison score, ADOS RJA autism diagnostic observation schedule response to joint attention, SRS-2 social responsiveness scale-second edition, T-score, SCQ social communication questionnaire lifetime total score

Apparatus

The system was designed and implemented as a component-based distributed architecture capable of interacting via network in real-time. System components included (1) a humanoid robot that provided joint attention prompts, (2) two target monitors that could be contingently activated when children looked toward them in a time synched response to a joint attention prompt, (3) an eye tracker and linked camera system to monitor time spent looking at the robot facilitator and judge correct performance, and 4) a Wizard-of-Oz style human control system to mark correct performance. The term Wizard-of-Oz is commonly used within the field of human–computer interaction to describe systems that appear to operate autonomously to the participant, but are actually at least partially operated by unseen human administrators.

Humanoid Robot

The robot utilized, NAO (see Fig. 1), is a commercially available (Aldebaran Robotics Company) child-sized plastic bodied humanoid robot (58 cm tall, 4.3 kg) utilized in other recent applications for children with ASD (Bekele et al. 2012; Gillesen et al. 2011). In this work, a new rule-based supervisory controller was designed within NAO with the capacity to provide joint attention prompts in the form of recorded verbal scripts, head and gross orientation of gaze shifts, and coordinated arm and finger points. Prompts were activated based on real-time data provided back to the robot by a human facilitator.

Fig. 1.

Humanoid robot

Eye Tracker

We utilized a remote desktop Tobii120 eye tracker to index participant gaze toward the robot during the task. It controls a calibrated camera that records the participant’s view of the robot, which is streamed to the video feed shown at the monitoring station. This allows the technician to monitor each participant’s eye gaze in real time. To calibrate the eye tracker, the participant sits in the center of the room and views eye gaze calibration slides projected on to a screen. The calibration slides contain a small cartoon on the calibration point as well as music to catch the participant’s attention. After calibration, the screen was removed and the robot was positioned at the calibration point. The “robot attention gaze region” was defined as a box of 76 cm × 58 cm which covered the body and movement of NAO. Given the distance from the participant to the calibration screen/robot, the accuracy of gaze detection if the participant moved his or her head was about 5 cm in both the horizontal and vertical directions.

Target Monitors

Two 24 inch computer monitors hung at identical positions on the left and right sides of the experimental room. The flat screen monitors displayed static pictures of interest at baseline, but also played brief audio files and video clips based on study protocol. The target monitors were 58 cm × 36 cm (width × height). They were placed at locations approximately perpendicular to the participants such that target orientation would often require substantial head movement in addition to gaze shifts to aid in classification of successful orientation (see Fig. 2 for a diagram of the room arrangement).

Fig. 2.

Apparatus and room arrangement

Wizard-of-Oz Human Control System

A live video feed of the participant was streamed to a monitoring station where a technician continually monitored the participant performance. If the participant followed the robot’s instruction by looking toward the target, the technician hit a button to trigger correct looking. This marker would cue the system to provide reinforcement in accordance with the defined protocol. If the participant did not follow the robot’s instruction within 7 s of the prompt, the system registered the lack of a successful response and proceeded to the next level of prompting until all six prompts were administered. Timing of prompts and the time window for correct response was embedded within the system architecture (i.e., the technician was not responsible for gauging the 7 s window). The prototype system developed in our original work (Bekele et al. 2012, 2013) was capable of automatic inference of gaze via head-tracker and as such realized closed-loop adaptation (i.e., system capable of adjusting itself without human facilitation). However, in terms of tolerability, 40 % of our ASD sample was not able to tolerate wearing the instrumented cap. As such, we utilized a Wizard-of-Oz paradigm to test change over time as an interim step to determine the relevance of future movement toward a non-invasive computer vision detection methodology with potential for closed–looped interaction.

Design and Procedures

Participants came to the lab for four lab visits over the course 2 weeks on average (average days = 14; SD = 9.6; range = 4–30). Informed consent was obtained from all participating parents. At the initiation of each session participants were introduced to the experiment room and given time to explore the robot. The child was then seated in a Rifton chair at a table across from the robot with the parent was seated behind the child. Parents were instructed to avoid providing assistance to the child during the study. After initial eye tracker calibration, participants then participated in a series of joint attention trials. Each session included eight trials (see Table 2), for a total of 32 trials across all sessions.

Table 2.

Prompt content for each level within trials

Prompt Level	Robot speech	Robot motion	Target display
1	“Jim, look!”	Turn head	Static picture
2	“Jim, look!”	Turn head	Static picture
3	“Jim, look over there!”	Turn head and point	Static picture
4	“Jim, look over there!”	Turn head and point	Static picture
5	“Jim, look over there!”	Turn head and point	Audio display (3 s)
6	“Jim, look over there!”	Turn head and point	Video display (10 s)

At the beginning of each session, participants were told that they were going to play a game. The robot then greeted the participant (“Hi Jim. My name is Nao. I want you to find some things. Okay, ready?”) and provided the first prompt (“Jim, look!”).

Trial Format

Each trial included up to six potential prompt levels. For each trial, the system randomly put the target on the left or right monitor for the trial’s duration. The robot turned its head or turned while pointing to the corresponding target. After the start of each prompt, a 7 s response time window was set. “Target hit” was defined as the participant responding to (i.e., turning to look at) the correct target within this 7 s window. Regardless of the participant response, the robot turned back to a neutral position (standing straight and facing the participant) after each prompt.

The technician continually monitored the participant’s performance using direct observation and the calibrated eye tracking system. If the participant followed the robot’s instruction by looking at the target, the technician hit a button to trigger a reward (a clip from a children’s cartoon) and start the next trial. If the participant did not follow the robot’s instruction within 7 s of the prompt, the system registered the lack of a successful response and proceeded to the next level of prompting until all six prompts were administered.

The hierarchy moved children from simple name and gaze prompts, to prompts also combining pointing, to prompts combining all those plus audio and/or visual activation. In each trial, a 10-s video clip was turned on contingent to the registration of child success by the system, or at the conclusion of the prompts. These video clips were short musical video segments of common preschool television programs (e.g., Bob the Builder, Dora the Explorer, Sesame Street etc.) that were randomized across trial blocks and participants (see Bekele et al. 2013 for details of video selection). Although trial time varied as a function of performance within system, and sessions including warm-up and introduction took substantially more time, the trials themselves with system were accomplished over a fairly brief time window (m = 4.93 min, SD = 1.05).

Results

The primary objective of this study was to empirically test child performance in response to within system joint attention prompts over a series of sessions. The secondary objective was to assess attention to the humanoid robot over time. We hypothesized (1) children would be responding to the humanoid robot at lower levels of prompting within the hierarchy from baseline to outcome, and (2) children would not demonstrate diminished attention to the robot over time. As such, we analyzed target hit rates to assess change from baseline to final performance as well as time spent looking toward the robot across a similar time frame.

Target Hit Rate

Across all sessions and participants, 99.48 % of the 32 trials ended with a target hit. The average prompt level before participants hit the target is shown in Fig. 3, which displays how participant performance, as measured by the number of prompts needed before a successful target hit, improved across sessions. In Session 1, the average target hit prompt level was 2.17 (SD = 1.49) with the average target hit prompt level falling to 1.44 (SD = 1.05) by Session 4. A two-sided Wilcoxon rank-sum test indicated that the median difference between Session 1 and Session 4 was statistically significant (p = .003).

Fig. 3.

Average prompt level of target hits across sessions

In examining individual performance of children over time, five of the six participating children exhibited lower average levels of prompt level target hit across session (see Fig. 4). We next examined specific performance by prompt level. Specifically knowing that prompts 5 and 6 involved target activation in the form of audio and/or video activation, we wanted to determine if children showed an increased ability to respond to gaze and point shifts delivered by the robot prior to such activation (see Fig. 5). On average, participants responded to the first press of the robot more frequently across sessions and showed high levels of response prior to prompts that used an element of target activation. Specifically, in Session 1, 52.8 % of trials ended with a target hit on prompt 1; by Session 4, that number was 81.25 % (p < .05). Participants hit the target within the first four prompts 87.5 % of the time in Session 1 and 95.83 % of the time in Session 4. We also examined within session performance for individual children by indexing the total number of sessions where there was clear reduction or increase in prompt levels defined by ≥1 level of change in prompt level during an individual session. Within session prompt performance was present in only a relative minority of trials (25 %) with a majority of sessions demonstrating either unclear within session change (46 %) and the remaining session actually demonstrating increases in within session prompting. These results suggest that there while there was clearer improvement over time and across sessions for this group, specific improvement within individual sessions was not as clearly evident nor a reliable predictor of change over time.

Fig. 4.

Average participant prompt level for target hit

Fig. 5.

Target hits on initial prompt (prompt 1) and prior to target activation (prompts 1–4)

Attention Toward Robot

We analyzed eye gaze patterns in two ways: (1) Across the whole session (from the start of the first prompt to the end of the session), and (2) Within the 7 s response time window across all prompts within a trial. Movement restrictions related to eye-tracker calibration resulted in some data loss very much in line with other work regarding eye-tracker use and young children (Sasson and Elison 2012). There was a trend for lower levels of data loss over time, with estimates of 30 % data loss for Session 1 and less than 10 % for subsequent sessions.

The average time that participants looked at the robot across all sessions was 14.75 % of the total experiment time. Within the 7 s window, the average time that the participants looked at the robot across all sessions was 24.80 %. From Session 1 to Session 4, participants’ average times looking at the robot region were 14.88, 15.17, 17.94, and 11.02 % for the whole session, and 22.15, 26.52, 28.14, and 22.41 % for the 7 s response window. Two-sided Wilcoxon rank-sum tests showed that these differences in looking time across sessions were not statistically different.

Discussion

In the current pilot study, we studied the development and application of an innovative adaptive robotic system with potential relevance to core areas of deficit in young children with ASD. The ultimate objective of this study was to test children’s performance over time across interactions with a humanoid robot-based system capable of administering and altering a joint attention hierarchy based on performance. Within our small sample, children with ASD demonstrated improved performance within system across sessions and documented sustained interest with the humanoid robot over the course of interactions. These findings together are promising in both supporting system capabilities and potential relevance of application. Despite promise, available pilot data are not yet sufficient for suggesting that such short-term changes may translate into broader changes beyond the experimental paradigm itself.

In line with previous findings, children with ASD in our sample were quite often able to respond accurately to prompts delivered by a humanoid robot within the standardized protocol (Bekele et al. 2013). Further, participants also spent a significant portion of the experimental sessions looking at the humanoid robot, replicating other work suggesting that young children with ASD show attentional preferences for robotic interactions over brief intervals of time (Bekele et al. 2013; Dautenhahn et al. 2002; Duquette et al. 2008; Kozima et al. 2005; Michaud and The´berge-Turmel 2002; Robins et al. 2009). In addition, within the current work we also documented that children could demonstrate improved performance over time in a basic core social communication skill and area of deficit (i.e., response to joint attention) and that over the course of sessions, children maintained interest in the humanoid robot central to platform.

Although children in this sample demonstrated variable baseline joint attention skills both within system and as coarsely indexed by ADOS response to joint attention item scores, all but one of our participants (83 % of total sample) documented improved joint attention response over time. Further, children successfully followed the human-oid’s gestures and movements to accurately orient to targets, orienting prior to target activation in 95.83 % of trials in the final session. Collectively, these findings suggest that robotic systems endowed with mechanisms for successfully pushing participants toward correct orientation to target, via a behaviorally sophisticated prompting and reinforcement system, might be able to capitalize on non-social attention preferences for many children with ASD in order to meaningfully enhance skills related to coordinated attention over time.

Despite this potential, the current system only provides a preliminary structure for examining ideal instruction and prompting patterns for a humanoid robotic system. Future work examining prompt levels, the number of prompts, cumulative prompting, or a refined and condensed prompt structure would likely enhance future applications of any such system. Although our data provides preliminary evidence that robotic stimuli and systems may have some utility in preferentially capturing, shifting, and attention, it is unclear how such performance would compare to instruction provided by a human administrator in the current study. In many of their current forms, humanoid robots are not as capable of performing sophisticated actions, eliciting responses from individuals, and adapting their behavior within social environments as their human counterparts. Though NAO is a state-of-the-art commercial humanoid robot, its interaction capacities have numerous limits. Its limb motions are not as fluid as human limb motions, it creates noise while moving its hand that is not present in the human limb motion, flexibility and degrees of freedom limitations produce less precise gestural motions, and its embedded vocalizations have inflection and production limits related to its basic text-to-speech capabilities. As such, it is extremely unlikely that the mere introduction of a humanoid robot that performs a simple comparable action of a human in isolation will drive behavioral change of meaning and relevance to ASD populations. Robotic systems will likely necessitate much more sophisticated paradigms and approaches that specifically target, enhance, and accelerate skills for meaningful impact on this population.

There are several significant methodological limitations of the current study that are crucial to highlight. The small sample size examined and the limited time frame of interaction, although significantly expanded from previous work, are the most powerful limits of the current study. Further, although we had standardized assessments of those children who participated, there was a substantial lag between assessment and enrollment which somewhat limits our ability to understand fully the sample participating in this study. While we are left with data suggesting the potential of the application, the utilized methodology potently restricts our ability to realistically comment on the value and ultimate clinical utility of this system as applied to a variety of young children with ASD. Eventual success and clinical utility of robot-mediated systems hinges upon their ability to accelerate and promote meaningful change in core skills that are tied to dynamic neurodevelopmentally appropriate learning across environments. Although we did assess brief learning within-system to positive effect, we did not systematically compare such improvements to learning in other methods nor did we see if such learning generalized to other interactions. As such, questions regarding whether such a system could constitute a viable intervention paradigm remain open.

Another important technical limitation was the utilization of a human confederate within the robotic system loop. While this modification from our original closed-loop system resulted in dramatic improvement in terms of tolerability (i.e., all children completed the protocol), such Wizard-of-Oz paradigms carry additional human resource burdens. Closed-loop technologies (Liu et al. 2008; Bekele et al. 2013) that harness powerful differences in attention to technological stimuli may hold great promise in this regard. Developing non-invasive technologies for capturing gaze such as adaptation of computer vision systems (Sasson and Elison 2012) would be critical for more realistic deployment of the current system without encountering large fail rates associated with a worn head tracking device or necessitating extremely expensive available technological solutions (e.g., numerous integrated systems of eye-trackers). Further, such systems could help us understand and track visual attention patterns in a more complete form with potential application toward more robust systems.

Despite limitations, this work is the first to our knowledge to design and empirically evaluate the usability, feasibility, and preliminary efficacy of an adaptive interactive robotic technology capable of modifying performance regarding joint attention skills for young children with ASD. Only a few other existing robotic systems developed for other aspects of autism intervention have specifically addressed how to detect and flexibly respond to individually derived social and disorder relevant behavioral cues within an intelligent adaptive robotic paradigm for young children with ASD (Feil-Seifer and Mataric 2011; Liu et al. 2008). Progress in this direction may introduce the possibility of technological intervention tools that are not simply response systems, but systems that are capable of necessary and more complex adaptations. Systems capable of such adaptation may ultimately be utilized to promote meaningful change related to the complex and important social communication impairments of the disorder itself. Questions of skills generalization remain perhaps some of the most important ones to answer for the expanding field of robotic applications for ASD (Bekele et al. 2013; Diehl 2012). It is both unrealistic and unlikely that in the immediate future robotic technology will constitute a sufficient intervention paradigm addressing all areas of impairment for all individuals with the disorder. However, if we are able to discern measurable and modifiable aspects of adaptive robotic intervention with meaningful effects on skills seen as tremendously important to neurodevelopment, we may realize transformative accelerant robotic technologies with an important place and pragmatic role regarding real-world intervention.