Using technology-based learning tool to train facial expression recognition and emotion understanding skills of Chinese pre-schoolers with autism spectrum disorder

Abstract

Objectives: Given the pervasiveness of emotional and behavioral deficits of individuals with Autism Spectrum Disorders (ASD), there is a pressing need for effective interventions to address their difficulties on Facial Expression Recognition and Emotion Understanding (FER/EU). Qunatiandi, a structured, app-based intervention program that is designed for Chinese children with ASD was utilized in this study. Three young children (two girls and one boy; age M = 4.94 years) completed an 8-week one-on-one intervention in a rehabilitation center setting. It was hypothesized that the three children would show greater progress in their FER/EU.

Methods: In this study, a modified multiple probe across the program phases design was used; the dependent variable was the percentage of unprompted correct receptive identification responses for FER/EU tests during instruction and probes. Data were taken during baseline, the endpoint of instruction sessions, and a maintenance stage followed by intervention termination.

Results: Findings of the study revealed that all participants demonstrated significant improvements in social acuity, gains were mostly achieved on emotion distinguishing and understanding (above 80% of accuracy). A similar scoring pattern was also found in the maintenance probe phase.

Conclusion: The study is one of only a few early intervention projects to improve FER/EU skills for children with ASD using an app-based intervention. The results demonstrated that children with ASD can experience increases in emotion distinguishing and understanding skills. Maintenance probe results showed that the interventional effect can be maintained for a period of time after intervention.

Introduction

Children with Autism Spectrum Disorder (ASD) experience delayed functioning in facial-emotional recognition and social-emotional reciprocity, which may cause the inability to develop and maintain age-appropriate social communication (American Psychiatric Association 2013, Bellini et al. 2007). In addition to the presence of repetitive behaviors, difficulties in recognizing facial expressions, identifying, and interpreting subtler emotions and nonliteral language may also impact their social opportunities and development (Daniels et al. 2018, Elartseva et al. 2015, Harms et al. 2010, Krasny et al. 2003); as youngsters become more aware of these difficulties in adolescence, their likelihood of employment and independent living in the future may also largely decrease (Flores et al. 2012, Reed et al. 2011). Therefore, there is a great need to identify treatment approaches that are efficient, accessible, and flexible to improve students’ learning outcomes.

Previous research has implemented various trials on children with ASD to improve their FER/EU abilities. The evidence-based practices most frequently used in interventional cases are Discrete Trial Teaching (DTT), Pivotal Response Treatment (PRT), and Picture Exchange Communication System (PECS) (Genc and Vuran 2013, Masiello 2007, Zhou et al. 2010). However, most of the aforementioned examinations failed to design a follow-up stage to gather maintenance data; moreover, these studies only involved three basic types of emotions (joy, anger, and sadness), which are not enough to reflect the full complexity of human emotions. Children with ASD are usually delayed in developing Theory of Mind (ToM), the ability to interpret other individuals’ beliefs, desires, and emotions (Brewer et al. 2017). Sun (2006) adopted a series of ToM-based intervention approaches to train a student with ASD to read joy, sadness, anger, and fear. The participant made great progress in facial expression recognition and emotion recognition. Baron-Cohen (2009) proposed Empathizing-Systemizing Theory (E-S Theory), which assumed that children with ASD were below average in terms of empathy (both cognitive and affective), and were either average, or even above average in systemizing. Based on E-S theory, Golan and Baron-Cohen (2006) had developed Mindreading DVD, an electronic encyclopedia of emotional expressions. As a result, the intervention groups significantly improved their recognition ability of complex emotions. Golan et al. (2010) designed Transporters, an animation that embedded human facial expressions on cartoon vehicle characters, to enhance emotion recognition of children with ASD. Other intervention types, such as music therapy (Tan and Khetrapal 2016) and Lego therapy (Owens 2008), are also based on E-S Theory. However, most of these studies were not evidence-based practices and need future research to assess their validity.

There is increasing evidence that Computer Assisted Instruction (CAI) are helpful for children with ASD (Glaser et al. 2012, Yamamoto and Miya 1999). Among those, handheld electronic devices are especially considered user-friendly due to their accessibility, portability, and flexibility; they enabled parents to implement systematic intervention instructions in a comparatively standard way toward their children (Egger et al. 2018). Researchers have begun incorporating technologies into interventions to teach youth with ASD a variety of skills to supplement traditional forms of instruction (Kagohara et al. 2013). Among them, FER/EU training is especially promising. By incorporating visual, dynamic stimuli, and user-centric application context, it facilitated children’ attention to different types of facial cues, and reinforced social response attempts (Daniels et al. 2018, Lamm et al. 2016). Such an interactive and supportive learning atmosphere is essential given that children with ASD often have a history of negative real-life social interaction experiences (White et al. 2010). Gay and Leijdekkers (2014) used Capture My Emotion, a series of adaptive application software for mobile phones and iPad, showed that high-functioning ASD children responded well to apps and were less prone to be overwhelmed by complex multimedia material. Furthermore, mobile-app tools are much friendlier to teachers and parents as they are much more accessible, with less limitation in time and place, and easy to operate, such that even parents can easily carry out intervention instructions in a standardized way. Egger et al. (2018) adopted a smartphone-based tool, Autism & Beyond, and suggested that the acceptance attitude of the app-based method to parents is high, in terms of parents’ willingness, and the ease of using the method. It is important to examine and design interventional tools to teach emotion-reading strategies, and app-based intervention has created new options and opportunities for a diverse group of learners and learning styles.

The field of special education in China has made significant improvements over the past decades. However, constant challenges regarding the unsystematic ASD teacher training programs, shortage of specialists, and lack of adequate training and preparation among current Chinese special education teachers are still substantial (Wang and Mu 2014). As a result, a study to identify treatment approaches that are grounded in the principles of applied behavior analysis, and to provide investigation on how such technology can be used to benefit FER/EU training outcomes is warranted for the Chinese special education field. An interactive tablet application, Qunatiandi (Li and Wang 2018, Lin and Wang 2018), was developed for this study by the Pac Rim Research Center, University of California Santa Barbara.

The Qunatiandi intervention program has seven major training domains, 19 subdomains, and is organized into more than 160 targeted skills that are each based on an essential skill under a learning theme. The aforementioned FER/EU skill is under one of the seven major domains and offer students a comprehensive curriculum to learn various emotional recognition and management skills. Each targeted FER/EU skill features its own videos and illustrated facial expressions in short animated episodes. The program utilized evidence-based practices, incorporated teacher-mediated instruction, visual support, and an embedded adaptive training system where students could practice through interactive activities. In this study, Qunatiandi was incorporated in the intervention process for all participants. This study was undertaken to examine the outcomes of the development of an application-based intervention targeting FER/EU improvement in Chinese young children with ASD. The purpose was not only to evaluate the efficacy and feasibility of the training program, but also to gather preliminary data on efficacy through reliable experimental measurements.

Method

Participants and settings

All participants were diagnosed with ASD according to diagnostic criteria outlined in the Diagnostics and Statistical Manual – 5th Edition Text Revision (DSM-5, American Psychiatric Association 2013) by authoritative local hospitals in China. Once a formal agreement form from the rehabilitation center was obtained, the school administrator sent a letter of consent to study participants’ parents to gain written consent. Verbal assents were deemed satisfactory for the student participants. Pseudonyms were used to protect the confidentiality of all participants.

Child 1. Xi Xi was a 6-year-old child diagnosed with speech and language impairment by an independent physician. She would follow limited verbal requests, which indicated that she had basic receptive language abilities, but would not express feelings (such as being uncomfortable, thirsty, and hungry) in complete sentences. She experienced articulation disorder and had difficulty with the mouth and tongue movements needed to make certain speech sounds. She was able to complete simple visual matching tasks and responded to visual cues presented by her teacher. According to her Psychoeducational Profile, Third Edition (PEP-3) assessment result, Xi Xi presented moderate levels of disorder in aspects of verbal expression, language understanding, and emotional expression.

Child 2. Deng Deng was a 5-year-old child who demonstrated relatively impaired delays in communication. He could make distinct pronunciations but at a very low speaking volume; he could initiate expressions of physiological needs, such as the need or desire for food, drink, and toys, through incomplete sentences. His PEP-3 assessment result showed moderate disorders in emotional expression and verbal expression, and mild disorder in language understanding.

Child 3. Xiao Jun was aged 3 years and 10 months at the start of the intervention. Her teacher reported concerns about her lack of conversational skills; her active speech was mainly about her personal needs and desires, such as ‘I want to play with a Barbie doll’, ‘I want to go to the bathroom’, and ‘I want to go back home’. She could occasionally speak in sentences of five to seven words for tangible objects or activities, but there was inaccuracy of words and tones. In her PEP-3 assessment, Xiao Jun had mild disorder in verbal expression, moderate disorder in emotional expression, and normal development in language understanding.

The study was conducted in a child rehabilitation center in northern China every day during regular teaching time. The participant, therapist, and a paraprofessional who was responsible for video recording were present for all sessions. The therapist had more than 10 years of teaching experience in special education. All sessions were conducted in a self-contained therapy room at a rehabilitation center. The pieces of training were regularly implemented in the morning when the children were in an excellent physical condition.

Material

Qunatiandi is an Android or iOS application-based intervention program that incorporates a number of evidence-based treatment approaches that are recognized by the National Professional Development Center on Autism Spectrum Disorder (NPDC) in the United States. A few examples of the practices incorporated in the program are Discrete Trial Training (DTT), Pivotal Response Treatment (PRT), SCERTS, Positive Behavior Support (PBS), Treatment and Education of Autistic and related Communication handicapped CHildren (TEACCH). Qunatiandi covered seven key training domains: (1) cognition and academic skills; (2) adaptability and regulation skills; (3) social interaction skills; (4) facial expression recognition and emotion understanding; (5) communication skills; (6) behavior assessment and support; and (7) play skills.

This study mainly explored Qunatiandi’s efficacy on improving participants’ facial expression recognition and emotion understanding abilities. Intervention approaches designed for this topic combined PRT, DTT, ToM, and SCERTS models. The curricula included recognition and understanding skills of 13 common emotions, including four basic emotions (joy, anger, sadness, and fear), and nine complex emotions (curiosity, boredom, exhaustion, hilariousness, anxiety, pride, sickness, surprise, and disgust). After a complete series of intervention sessions, students were expected to be able to recognize these facial emotions and understand situations where a particular facial expression may appear, as well as deliver an appropriate facial expression in accordance with the social context.

Instructional design

Emotional discrimination and understanding are two phases that were implemented in turn by the teacher. Intervention will not advance to the next phase unless the child receives at least 80% of accuracy in the tests of the current phase.

Teaching

The teaching process has eight sub-sections, which were later further divided into several small video clips that contained one or two descriptive sentences – taking one of the basic emotions – ‘happy’ as an example. First, a short animation with emoticons would be presented with voice-over narration and explanation of why someone would have a smiley face when he/she feels happy. The animation also described related facial cues of happiness (e.g. eyebrows, eyes, mouth, and teeth). Also, voice-over narration would give examples regarding what situations people may feel happy (e.g. when Ming’s mother brings him his favorite toy car, Ming feels happy, and he smiles). The student would be presented with pictures depicting different facial expressions and asked to identify a target emotion given in an instruction, coupled with the corresponding emoticon. When the voice-over narration became triggered, the student would hear ‘this is a happy face of a boy’. Each emoticon is presented repeatedly and alternately with a random order. Then the student will be taught in what contexts someone would feel happy. He or she would first be presented with three sets of cartoon pictures, respectively, describing three contexts in which people usually feel happy with narration. For example, Ming is playing on the slide with his friend, and he is happy on the playground. Then the other three contexts would be shown with real-life photos with narration. For a specific topic, although the same content would appear repetitively, slightly different phrasings and sentence structure were used in the narration to promote generalization to real-life settings.

Distinguishing

In this section, the student would be first requested to select a single picture of a smiley face, as part of a review practice of the previous study. Then the student needs to judge whether a given facial expression is a happy one or not. The next step is to choose the facial expression matching an existing picture (of a happy facial expression), but the picture for choosing is of different characters to ensure the student indeed recognized and distinguished the facial expression rather than finding the same picture. Lastly, the student is required to choose the correct facial expression from a gradually increasing number of pictures of different facial expressions.

Understanding

In this section, the student would be requested to place a smiley face onto a blank cartoon face. There are five practice scenarios: (1) eyes, eyebrows, and mouth appear as a whole, and the student will only be asked to match these facial features on an empty cartoon face; (2) the student will be asked to place the ‘mouth’ onto the cartoon face with only eyebrows and eyes; (3) the student will be asked to put ‘eyes’ onto the carton face with only eyebrows and a mouth; (4) the student will be asked to put ‘eyebrows’ onto the face with only eyes and a mouth; (5) the student will be directed to put ‘eyes’, ‘eyebrows’, and ‘mouth’ separately. In order to teach children with ASD to understand the importance of looking at people’s eyes, separating them from the eyebrows and mouth of a happy face would be helpful for them to learn facial expression recognition. This sequence is intended to be more natural for the student to start with, and the student can later move on to more unfamiliar scenarios when he or she becomes more practiced. In this phase, there are also sections where various contexts are given, and the student needs to choose the most appropriate facial expressions accordingly. For example, ‘Ming’s mother praised him for washing dishes after dinner, what facial expression he would probably have? Please choose from the faces in the right column’. The difficulty level increases in terms of the amount of pictures and the variation of the situations.

When it comes to assessing more complicated emotions, most processes are the same as for those of basic emotions. Additionally, some new sections are added, such as integrating jigsaw puzzles of faces with the target facial expressions and asking students to draw the faces of certain facial expressions on the screen of the mobile device.

Procedure

This study used a modified multiple probe across the program phases design (A-B-C design) with three pre-schoolers diagnosed with ASD. Baseline (A), intervention (B), and maintenance (C) data were collected over a 3-month period.

Baseline

In baseline, no direct instruction or prompting was provided to the three participants. According to Curriculum-based Assessment Theory, the investigator randomly drew a certain amount of questions from Qunatiandi’s test question bank. During each baseline session, the investigator drew 10 questions to do the FER/EU assessment. Types of questions included multiple choice, blank-filling, and puzzle. Each student’s test score were recorded as he/her baseline data. Baseline tests of Distinguishing and Understanding phases took place separately. Also, in each phase, intervention began until the student obtained stable baseline data, which means the scores of the tests would not display sharp changes for approximately five consecutive tests. Deng Deng and Xiao Jun both took five tests in the baseline phase, and Xi Xi took three more baseline tests as she did not complete the 5th test therefore received zero scores for that trial.

Intervention

Intervention sessions were scheduled to be implemented every day, 15 min for each teaching segment, three courses each day. However, there were some time intervals with no training due to holidays or students’ absence from the class.

All intervention sessions started with a Teaching phase with no tests. During intervention, students mostly learned independently according to instructional content of Qunatiandi, during which time the teacher would only play a facilitator role in assisting the students, and students would do the practices in the app accordingly. During the practice, teachers would give some instruction and explanation based on the feedback of the training system. Scores of the practice sessions would only be used as a learning reference for the intervention effect and thus would not be recorded. Lastly, Qunatiandi’s question bank would be used in the baseline sessions would to evaluate baseline student levels, and this data would be recorded as intervention data in which no instruction from the teachers or the app would be given. Intervention phases would not advance to the next phase until the student received at least an 80% accuracy score in three consecutive tests. Thus, intervention timelines may vary from student to student depending on individual needs, abilities, and learning progressions.

In Distinguishing and Understanding phases, a student would be continuously tested after every day’s training to see the trend and speed of the effect of the intervention. Tests would always be presented with the same method as in the baseline session, in the same settings and conditions to avoid unexpected influence from uncontrolled variables. The tests were all of the same difficulty level, but the contents were differently arranged in random order to avoid a participant practice effect as much as possible.

Maintenance

After the intervention, maintenance probes would be carried out three to four times, 2 weeks after the intervention. Students would be tested in the same way as during the baseline phase, in order to examine whether the effect of the intervention could be maintained after some time without further intervention.

Results

Figures 1 and 2 showed the results of the tests including baseline, intervention, and maintenance for three children with ASD.

Figure 1.

Scores of emotion distinguishing of three children at baseline, intervention and maintenance periods.

Figure 2.

Scores of emotion understanding of three children at baseline, intervention and maintenance periods.

As shown in Figure 1, in Distinguishing phase, the mean scores in the baseline periods of the three participants were: 50 for Xi Xi, 42 for Deng Deng, and 60 for Xiao Jun. Mean scores were all below 80, showing that the expression distinguishing ability of the subjects was weak and without a clear upward trend. Xi Xi did not finish the 5th baseline test, therefore she scored 0 in that section, and this score was not included into the data analysis.

During Distinguishing interventional periods, the average scores of the tests of the three participants were: 64 for Xi Xi, 75 for Deng Deng, and 92.5 for Xiao Jun. The scores increased by 21.9%, 44%, and 35.1%, compared to the baseline data, respectively. The three participants had different performances in the intervention periods; therefore, the time taken to reach a stable stage was not the same. For Xi Xi, the score was above 80 and became stable after 15 days of intervention. Deng Deng achieved significant progress in only 8 days. A noticeable and instant decrease in the test score happened in the second trial. We suppose that the reduction was attributed to intervention resistance and the performance recovered soon. For Xiao Jun, the intervention effect was the most significant, and she received full scores and stability after only two trials.

During Distinguishing maintenance periods, the scores of the three participants were all above 80 and became stable after three probes, which indicated that the intervention had achieved a positive effect.

As shown in Figure 2, in the Understanding phase, the mean scores in the baseline periods of the three participants were 92 for Xi Xi, 43.33 for Deng Deng, and 42 for Xiao Jun. For Xi Xi, her scores were all above 80 after five probes and achieved stability during the baseline periods. Although it took a much longer time for Xi Xi to complete the distinguishing phase compared with the other two participants, she performed excellently in the baseline phase of Understanding phase, therefore there was no further need for her to take any Understanding intervention sessions. After 2 weeks, we took four probes for the maintenance period and the scores were all above 80. The results showed that emotional discrimination learning had been generalized so that emotional understanding had a higher score.

During the Understanding intervention phase, the average score of Deng Deng was 88, an increased of 51%. The average score of Xiao Jun was 100, an increase of 58%. Comparing these results with their baseline, all participants achieved considerable improvements. The time spent in the Understanding intervention was shorter than in the Distinguishing intervention suggests that learning of emotional discrimination promoted emotional understanding. During the Understanding maintenance periods, the scores of Xiao Jun and Deng Deng were all above 80 and remained stable after three probes.

Social validity information was collected and measured in the form of semi-structured student-parent interviews and teacher interviews. Interview questions addressed parents and teacher perceptions of FER/EU training, student learning outcomes, and allowed for personal comments and/or suggestions. As to the research outcomes, the teacher reported all three participants’ improvement on noticing changes in other people’s emotions; when seeing their classmates, students were able to make comment sentences, for example, ‘he smiles, he is happy’. More specifically, Xi Xi could now tell the teacher, ‘I have a Peggy toy, I am happy’; Deng Deng was able to say ‘I am angry’ to her teacher when experiencing frustrations. All participants’ parents also expressed their observations on their child’s improvements. More impressively, Xiao Jun’s mother mentioned that her daughter was now able to show compassion through sentences when seeing her in a state of sickness. Overall, with the introduction of the app-based intervention, all participants showed an increased rate in FER/EU, and consistently made gains through the intervention.

Discussion

Qunatiandi, a structured, app-based training program for Chinese children with ASD was utilized and examined in this study. Three young child participants completed an 8-week one-on-one intervention in a rehabilitation center setting. A modified multiple probe across the program phases design was used; findings revealed that all participants demonstrated significant improvements in social acuity, gains were mostly achieved on emotion distinguishing and understanding.

For Xi Xi, analyzing the span of time of each of the phases, we noticed that her Distinguishing phase took an extended period of time, and her learning progress advanced much more smoothly and quickly in Understanding phase. In the pre-test phase, Xi Xi’s ability in emotional and verbal expression was lower than the other two students and needed more intervention sessions in order to meet the criteria. A long period of intervention in emotion recognition resulted in a good generalization in her emotional understanding; her baseline data of emotional understanding reached 80% accuracy, and no intervention was needed for this session. As for Deng Deng and Xiao Jun, their emotional understanding interventions were shorter than that of emotional recognition. It is possible that because the participants’ facial expression recognition and emotion understanding abilities had already obtained improvement through trainings in the distinguishing phase, that students became more skilled and thus their progress accelerated in later phase.

The intervention can be effective due to its adaptive and user-friendly nature, specifically for children with ASD. For example, the highly systematic instructional structure of this curriculum has already been shown to be the most effective method for teaching children with autism. In the curriculum, teaching contents were broken down into discrete, but linked segments (teaching units), and had been taught rapidly, frequently, and repeatedly. New and old contents were arranged alternately, with a smooth transition from easy to difficult learning modes.

Qunatiandi’s directions and voice-over narration were clear and straightforward. Moreover, the directions would be given automatically and repeatedly if the student (1) did not give any response, (2) gave the wrong answer, or (3) needed to redo the clip. When advanced into a new learning stage, the instructional contents were arranged and delivered in a cartoon form first, and then in a real-life situation. This is because children with ASD have been shown to be able to perform better in emotion recognition when using cartoons rather than pictures of real faces (Lorusso et al. 2018). Moreover, the ‘forward’ and ‘backward’ buttons could be clicked in every clip for the student to control the pace of their learning progression. If a student ever feels that he/she needs to learn a clip again, or he/she does not need to study the learned content of a particular clip, he/she can click the buttons accordingly to review/practice repeatedly or skip any clips. This enabled teaching to be individualized for each student in accordance with his or her ability and actual learning status, which is necessary for teaching children with ASD who display heterogeneous characteristics. Furthermore, this also empowered the student with a sense of agency and control, which a powerful conditioned reinforcement for children with ASD. There is also a progress bar on the top of the screen to clearly show, with different colors, how many tasks have been completed and how many are to be learned. This is a friendly design for children with ASD who emphatically prefer highly predictable and systematic routines, based on E-S theory (Baron-Cohen 2009).

Overall, Qunatiandi made the best use of the behavioral principle of reinforcement. The student was allowed to choose one or more animation patterns (e.g. colorful pinwheels, donuts, or cars) as rewards for giving the correct answer in each clip, along with verbal praise ‘you did a great job!’ The student would have a much higher chance to present more correct answers after being reinforced by the animation. The students were reinforced immediately after emitting correct responses in each elemental instructional unit (clip) so that children could gradually notice that this behavior would result in rewards, thus encouraging additional effort to produce more correct responses. Immediate reinforcing is also a fundamental interventional principle for children with ASD. Moreover, as the reinforcer is chosen by the students themselves, they will be much more effective than the reinforcers chosen by the teacher.

Two limitations of this study bear noting. First, as single-subject designs typically contain a small pool of participants, without replication of studies across a large sample size, caution should be taken in generalizing the research finding across students, settings, and severity of disabilities; another limitation of the study is related to the lack of social contextualization assessments. Additional research endeavors should explore participants’ FER/EU performances in real-life situations.

Nevertheless, intervention efficacy remains to be investigated further in order to gain empirical evidence of gains for maintenance and generalization. The results from this study have contributed a preliminary understanding of student learning patterns through single-subject experimental design. This study has also assessed the changes in FER/EU of children with ASD, which is an important step before undertaking randomized controlled group designs to determine the efficacy of this intervention approach and to compare the effects of different treatment approaches. Additionally, as an intervention platform, Qunatiandi enables teachers to conduct professional FER/EU teaching activities to a large number of students with disabilities in China. Problems arising from solely picture-based instruction that prevent children from receiving effective learning are also alleviated by technology-based intervention program, as it presents structured, systematic instruction with animation and video clips. The aforementioned features of the technology-based intervention program in this study therefore address some of the key challenges in the field of special education in China.

Funding Statement

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Beijing Education and Science Project 2013 (ABA13014); the key project of Ministry of Education (DBA130218); 2018 Comprehensive Discipline Construction Fund of Faculty of Education, Beijing Normal University; the Fundamental Research Funds for the Central Universities.

Disclosure statement

No potential conflict of interest was reported by the author.