Virtual voice assistant applications improved expressive verbal abilities and social interactions in children with autism spectrum disorder: a Single-Subject experimental study

Abstract

The use of interactive technologies has been demonstrated to enhance verbal and non-verbal communication, as well as the social interaction tendencies of children with Autism Spectrum Disorder (ASD). We examined effects of using Virtual Voice Assistant (VVAs) in children with ASD with respect to two outcomes: speech skills and social interaction skills. A single-case study included three children with ASD (4–11 years old) that utilized VVAs for three months. Pre- and post-intervention questionnaires and semi-structured interviews were used to measure the communication and social interaction skills of the participating children. Participant One, Two and Three showed improvement in the number of correct words produced the VVA intervention. All participants showed increases in social interactions in the intervention phase. Overall, the results showed that the VVAs had positive effects on the speech and social interaction skills of autistic children. The findings demonstrate that children with ASD may benefit from VVAs to improve their communication skills.

Introduction

Autism Spectrum Disorder (ASD) is a developmental disorder that affects the way individuals interact with their environment and how they communicate with others (Copeland 2018) as well as induce difficulties with initiating or taking turns in conversations (Cholemkery et al. 2016) . ASD is considered a spectrum of disorders because the degree of impairment differs from one individual to another. Consequently, ASD management strategies should be individually tailored (Patel et al. 2015). One way to achieve this, is through the use of interactive technologies. Several interventions, both clinical and non-clinical, using interactive technologies have been tested on children with ASD, with varying degrees of success (Boucenna et al. 2014, Esteban et al. 2017, Mata et al. 2018, Porayska-Pomsta et al. 2018, Thabtah 2019). Further, recent advances in information communication technology continue to provide novel methods that attract renewed attention from practitioners to utilize artificial intelligence to stimulate and facilitate human-like interactions (Jaliaawala and Khan 2020). These research avenues show much promise, and research into the widespread application of such technology for managing ASD clearly represents the next frontier for providing support to autistic children.

Scholars noted that several strategic therapies as well as different approaches and methods have been developed to address verbal abilities and social interactions in autistic children. These approaches may include Augmentative and Alternative Communication (AAC) (Ganz 2015); Picture Exchange Communication System (Flippin et al. 2010); social stories to increase social skills (Safi et al. 2021); as well as trained dogs and meditation (Solomon and Bagatell 2010) to lessen the symptoms of ASD. Further, other approaches were developed to address autistic characteristics in the classroom (Reynhout and Carter 2006, Safi et al. 2021). Since many children with ASD have limited imitation skills and are unable to pick up on social cues, social interactions literally need to be taught to them (Jacklin and Farr 2005). Therefore, the use of advances in technologies such as virtual reality (VR) (Beaumont and Sofronoff 2008, Parsons and Cobb 2011, Parsons and Mitchell 2002, Schmidt and Schmidt 2008, Strickland et al. 2007) as well as virtual voice assistants (VVA) (Hoy 2018) have been used to improve social and language skills in children with special needs as well as children with ASD.

Virtual Reality has been considered as a useful tool for individuals with ASD because it can support the generalization of social interaction learning to the real-world situations while providing a controlled and safe learning environment (Beaumont and Sofronoff 2008, Parsons and Cobb 2011, Parsons and Mitchell 2002, Schmidt and Schmidt 2008, Strickland et al. 2007). Particularly, Parsons and Mitchell (2002) argued that VR allows role play with flexible scenarios and can increase cognitive flexibility of individuals with ASD. Moreover, individuals with ASD have opportunities to practice diverse responses in simulated real-world scenarios with reduced anxiety toward the social interaction (Moore et al. 2000). Additionally, several research studies showed that children with ASD could transfer newly learned knowledge, such as fire safety skills, tornado safety skills, and road safety skills from VR-based training programs to real-world situations (Josman et al. 2008, Self et al. 2007, Strickland et al. 2007). Another empirical study also revealed the possibility of utilizing VR as a training platform to teach children with ASD to recognize others' thoughts and feelings (Kandalaft et al. 2012) as well as the effects of augmented reality on their interpretations of non-verbal cues (Sahin et al. 2018).

Virtual Voice Assistant is another available interactive technology that can interpret people’s speech and respond in a synthesized or human-like voice. VVAs can interact with humans through such actions as answering basic questions, telling jokes and stories, singing, and performing simple math calculations (Hoy 2018). At present, there is inadequate body of research exploring the role of VVAs in helping autistic children learn effective communication skills, despite the significant promise that such an approach holds. Although the fact that language difficulties in children with ASD is widely known, the application of artificial intelligence and VVAs in teaching language is limited (Hoy 2018). Most research has focused primarily on the roles of augmented reality, virtual reality, and games in relation to social and communication skills (Ireland et al. 2018, Parsons et al. 2006, Shin et al. 2017).

The wide availability of VVA offers potential benefits for improving social and communication skills in children with ASD, as VVA technology can enhance their quality of life by helping them to integrate into society more effectively. However, no studies have been found to date that investigate this topic. With improvements in VVA voice quality and the availability on most smart device platforms, such as smartphones, tablets, and computers—which are accessible and easy to use at home or school (Hoy 2018, Ireland et al. 2018), VVAs could be a daily asset for children with ASD who have difficulties with social interactions and communication (Hoy 2018).

In many areas worldwide, the resources necessary to improve communication and speech skills, such available speech therapists and special education teachers, often are inaccessible. Consequently, to improve these skills in children with ASD, there is a need for innovative methods using tools and devices that are readily available in their home environment. However, there is currently a lack of research, due to the recency of technological developments. Thus, this study aimed to address this imbalance and explore the use of artificial intelligence—particularly the recent developments in VVAs technology—to facilitate the development of language and social skills in children with ASD.

The study was guided by a conceptual framework, which proposed that the interactive nature of VVAs has a positive direct impact on speech development, interpretation of non-verbal cues, and social interaction skills in children with ASD. VVAs can be used as tools that provide children with a chance to listen and respond verbally, thus improving their ability to produce intelligible verbal responses. Furthermore, interaction with virtual characters can also serve to expose children to non-verbal cues, which can improve their ability to interpret non-verbal cues through repetition.

Additionally, it has been argued that VVAs can act as partners with whom children with ASD can interact, which may boost their social skills when interacting with adults and other children. In this framework, the strength of the correlation between the independent variable (VVAs) and dependent variables (expressive verbal abilities and social interaction skills) will be influenced by each child’s age and degree of impairment, which serve as the moderating variables.

The following research questions were addressed in the present study:

1. What are the effects of expressive language stimulation activities using VVAs on expressive vocabulary skills in children with ASD?

2. What are the effects of expressive language stimulation activities using VVAs on the social interaction skills in children with ASD?

3. What are the effects of expressive language stimulation activities using VVAs on the intelligibility of children with ASD, as rated by their mothers?

Material and methods

Research design

The study applied a single-case research A-B-A design, which involves the manipulation of the independent variable(s) over time to determine how such variables impact the dependent variable(s), thus allowing for hypothesis testing (Zirpoli 2016). This research was interventional and aimed to determine the effectiveness of an expressive language intervention using a VVA by assessing participants before the intervention and comparing those assessments with measurements taken during various study phases (Lobo et al. 2017). The data obtained prior to the intervention acted as a control (baseline). This research design allows for vigorous experimental evaluation of an intervention’s effectiveness while relying on one or only a few participants as opposed to obtaining data from a large pool of samples. Although single-subject experimental studies use a small sample size, such a design can be preferable as a source of rich information on each participant (Shaughnessy et al. 2015), can be used in making causal inferences (Lobo et al. 2017), and seeks to obtain data from children with ASD before, during, and after implementing an intervention using an A-B-A design. Further, this design would be adequate to explore whether (a) the intervention produced a performance improvement and (b) intervention withdrawal caused a performance decline.

Participants

This study targeted children who were diagnosed with ASD at around preschool age (4 years). The United Arab Emirates University Ethics Review Board approval number ERS_2021_7331 was obtained for this study, and informed consent was obtained from each participant’s parents. Initially, the study took place at students’ schools for the purpose of skill demonstration on using VVA for both children and their parents, followed by application at children home environments. To confirm participants’ ASD diagnoses, the GARS-3 (Gilliam 2013), was administered by a qualified speech language pathologist. The selection criteria for participants were a) a valid diagnosis of ASD, b) the presence of an expressive language delay, and c) difficulties with social interactions.

Participant 1 (P1): A four-year-old male enrolled in kindergarten at the time of the study that did not provide any special education support. He did not receive any speech language therapy while participating in this study. His autism index was 84, as measured by the GARS-3 (Gilliam 2013), equivalent to Level Two in the DSM IV. Regarding receptive language abilities, he was able to follow instructions if they were broken down into one- or two-step commands.

Participant 2 (P2): A six-year-old male enrolled in the first grade at the time of the study. He had received special education support, as well as speech therapy and behavioral modification sessions in the past; however, he did not receive any therapy while participating in this study. His autism index was 87, as measured by the GARS-3 (Gilliam 2013), equivalent to Level Two in the DSM IV. Similar to P1, P2 was able to follow instructions if they were broken down into short steps.

Participant 3 (P3): An 11-year-old male enrolled in grade six at the time of the study. He had received special education support, as well as speech therapy and behavioral modification sessions in the past; however, he did not receive any therapy while participating in this study. His autism index was 87, as measured by the GARS-3 (Gilliam 2013), equivalent to Level Two in the DSM IV. His receptive language abilities were at a level where he is able to follow instructions if they were broken down into short steps.

Procedure

Phases

A. Baseline phase

The baseline phase involved observing and assessing each child for speech and social interactions over four weeks. During this phase, participants took part in traditional language stimulation sessions with their mothers, using only traditional toys, cards, and storybooks. No VVAs (i.e. Apple’s Siri) were used.

To decrease threats to internal validity, a subsequent baseline (A) assessment period that lasted four weeks was also implemented following the intervention phase. This assisted in confirming the intervention’s effectiveness and whether the changes occurred due to the intervention or another extraneous variable (Norbury et al. 2010). In this second baseline, the intervention withdrew, and each student was introduced to new words with only his mother’s assistance, and not with a VVA.

B. VVA intervention phase

Training parents on the VVA intervention

Each family was provided with an iPad. The three participants were assessed for their ability to use Apple smartphone and tablet devices prior to the intervention. Each of the participants and their mothers were taken through a training session on using VVAs (i.e. Apple’s Siri). The first three sessions of the intervention phase (Phase B) were conducted with the researcher, while the remaining sessions were conducted with the children’s mothers at their homes. The first session was an introductory training session to provide education on the study’s purpose and procedure, as well as to establish a rapport with the child in an informal familiar setting (i.e. home). Then, the mothers were asked to sign the consent form. Unlike the mother of P3, the P1 and P2 mothers were familiar with the usage of the iPad device. However, neither the children nor their mothers were familiar with Siri Therefore, the researcher introduced Siri to them and provided training on how to interact with Siri application via the iPad. It is worth noting that only P3 and his mother required extra session on how to use the iPad. Therefore, more repetitions were provided during the sessions compared to others. During the next two consecutive sessions, the researcher modeled a typical session using VVA for the mothers. The mothers were asked to repeat the demonstration to ensure that they understood the concept and are able to implement the intervention adequately. Further, mothers were asked to praise their children using encouraging words when noted a positive change during interaction with Siri.

VVA application intervention sessions

Each participant used a VVA (Siri) for two months. Participants took part in two 10-minute interventional sessions per day. Most sessions were conducted after school and during weekends. To reduce threats to the validity of the results, the intervention was administered in the children’s natural environment (home) with the same researcher and parent throughout the program (Porcheron et al. 2017).

The mothers were required to initiate each intervention session by using Siri before the child (for instance saying, ‘hey Siri’). Then, the child was given the chance to do the same. If the child did not follow the stimulus, the mother was required to repeat the modeling behavior. If the child did not show interest, the mother was asked to terminate the session and reattempt later that day. If the child was successful in engaging with the stimulus, the mother was required to use the next word or phrase from an assigned list (Appendix 1 presents a sample conversation during an intervention session). Three sets of stimuli were developed to be used in this study; Set 1 consisted of eight social words such as ‘hi’ or ‘hello,’ Set 2 consisted of nine two-word phrases and Set 3 consisted of seven three-word phrases. In this study, only Sets 1 and 2 were used.

Response measurement

A. Daily measurements

Following each session, the parents counted how many times their child verbally produced the assigned words/phrases (from the stimuli sets noted above) correctly. This was based on Siri’s response. The mothers were required to video record at least one session per day and send it to the researcher.

B. Weekly progress measurements

To monitor each mother’s consistency in conducting the sessions, as well as each child’s progress, two weekly checklists were used. The Parent Assessment Checklist (PACL) is a weekly checklist that was developed by the researcher and consists of a questionnaire to collect data on parents’ consistency in following the intervention procedure as well as the child’s ability to use Siri on his own (see Appendix 2 for details). The Social Assessment Checklist (SACL) was developed by the researcher to monitor progress related to social interactions before the initiation of Phase A and following each phase. The SACL consists of 10 questions to evaluate social aspects of communication (see Appendix 3 for details). The two checklists were subjected to content validation following the Delphi approach which is sharing the draft scale to experts to review it (Mengual-Andrés et al. 2016). The two checklists were given to three academics and three practitioners to determine whether the items on the scale was appropriate.

Data analysis

In this study, data analysis involved visual analysis of graphical data, as suggested by Lobo et al. (2017). The data for the three participants were presented graphically, with spaghetti plots for each participant’s data. This served to aid in visualization of the data and making valid comparisons. Visual analysis is generally used to determine whether there is a functional relationship between the independent and dependent variables. Each case will be compared to itself. The visual analysis focused on (a) slope/trend and (b) the total ratio of responses to opportunities.

Results

This study was conducted over three months, from September to December 2019. The total number of sessions per child ranged from 22 to 29.

Improvements in expressive verbal skills

The first research question was ‘What are the effects of expressive language stimulation activities using VVAs on expressive vocabulary skills in children with ASD?’

P1 showed a notable improvement in the total number of correct words produced with fewer attempts during the VVA intervention (Phase B) compared with the traditional ones (Phase A; see Table 1). Additionally, a visual inspection of Figures 1 and 2 reveals that the number of new words produced was higher in the VVA intervention phase, indicating that VVA contributed to improvements in P1’s expressive vocabulary.

Table 1.

Total number of attempts and correct words produced for P1

	Set 1		Set 2
Phase	Total Attempts	Total Correct Words	Total Attempts	Total Correct Words
A	47	28	35	14
B	30	51	28	57
A	40	48	23	29

Figure 1.

Change in number of correct words produced in relation to attempts per sessions for P1 (1st set of words).

Figure 2.

Change in number of correct words produced in relation to attempts per sessions for P1 (2nd set of words).

Additionally, P1 demonstrated improvements in developing new expressive words during the VVA intervention (Phase B), compared to the traditional intervention (Phase A; see Figure 3).

Figure 3.

Comparing the ratio of new words produced to number of sessions between the traditional intervention and VVA intervention for P1.

P2 also showed notable improvement the total number of correct words produced with fewer attempts during the VVA intervention (Phase B), compared with the traditional intervention (Phase A; see Table 2). Visual inspection of Figures 4 and 5 reveals that the number of new words produced was higher during the VVA intervention phase, indicating that using the VVA contributed to improvements in P2’s expressive verbal vocabulary.

Table 2.

Total attempts and correct words produced for P2

	Set 1		Set 2
Phase	Total Attempts	Total Correct Words	Total Attempts	Total Correct Words
A	48	37	56	48
B	30	70	31	76
A	40	68	34	37

Figure 4.

Change in number of correct words produced in relation to attempts per sessions for P2 (1st set of words).

Figure 5.

Change in number of correct words produced in relation to attempts per sessions for P2 (2nd set of words).

Like P1, P2 demonstrated an improvement in developing new expressive words during the VVA intervention (Phase B), compared to the traditional intervention (Phase A; see Figure 6). The number of new words produced was significant in the VVA intervention phase; P2 produced five new words, compared to no new words in the first traditional phase and two words in the second traditional phase.

Figure 6.

Comparing the ratio of new words produced to number of sessions between the traditional intervention and VVA intervention for P2.

P3 demonstrated similar results. He showed a notable improvement in the total number of correct words produced with fewer attempts during the VVA intervention (Phase B) compared with the traditional ones (Phase A; see Table 3). Visual inspection of Figures 7 and 8 shows that the number of new words produced was higher during the VVA intervention phase.

Table 3.

Total attempts and correct words produced for P3

	Set 1		Set 2
Phase	Total Attempts	Total Correct Words	Total Attempts	Total Correct Words
A	47	35	65	51
B	19	51	32	63
A	43	49	29	44

Figure 7.

Change in number of correct words produced in relation to attempts per sessions for P3 (1st set of words).

Figure 8.

Change in number of correct words produced in relation to attempts per sessions for P3 (2nd set of words).

Similar to P1 and P2, P3 demonstrated an improvement in developing new expressive words during the VVA intervention (Phase B), compared to the traditional intervention (phases A; see Figure 9).

Figure 9.

Comparing the ratio of new words produced to number of sessions between the traditional intervention and VVA intervention for P3.

Changes in social interaction skills

The second question was ‘What are the effects of expressive language stimulation activities using VVAs on the social interaction skills in children with ASD?’

To answer this question, the results from the SACL were used. Visual analysis was also used to investigate changes in social interactions during VVA use. The bar charts for P1, P2, and P3 in Figures 10–12, respectively, show increases in the social interactions of all participants in the intervention phase, compared with the baseline. For example, all children allowed mothers to take turn in talking with Siri. In addition, P1 & P3 allowed their siblings to interact with Siri, mothers reported frequent enjoyment and excitement facial expressions during sessions. These results indicated that the use of artificial interactive technological programs, such as VVAs, could produce a small yet positive influence on participants’ social interaction skills.

Figure 10.

Social Assessment Checklist (SACL) scores per phase for P1.

Figure 11.

Social Assessment Checklist (SACL) scores per phase for P2.

Figure 12.

Social Assessment Checklist (SACL) scores per phase for P3.

Mothers’ perceptions of changes in speech intelligibility and social interactions

The third research question was ‘What are the effects of expressive language stimulation activities using VVAs on the intelligibility of children with ASD, as rated by their mothers?’

To answer this question, the results from the PACL was used. Before engaging in the study, as well as after each phase was completed, mothers were asked to complete the PACL. Figures 13–15 present the PACL results for P1, P2, and P3, respectively. As shown in the figures, all the mothers noted improvement in their children’s speech intelligibility and social interactions. Additionally, they indicated that they were satisfied with the VVA. Overall, the results suggested that using VVAs had a positive effect on the intelligibility of children with ASD, with mild to minimal parental support.

Figure 13.

PACL Scores per Phase for P1.

Figure 14.

PACL Scores per Phase for P2.

Figure 15.

PACL Scores per Phase for P3.

Discussion

Children with ASD tend to have communication and social interaction deficits. Their deficiencies in communication mainly come from the difficulties they face regarding language acquisition. Although researchers have demonstrated the significant role artificial intelligence can play in interventions for children with ASD, the role and effectiveness of VVAs has not been satisfactorily examined. Therefore, this study aimed to explore the effectiveness of VVAs for enhancing speech and social interaction skills in children with ASD.

Overall, this study’s results showed improvements in participants’ expressive verbal vocabulary, production of short phrases, and social interactions during the intervention phases compared with a traditional language stimulation activities such as the use of toys, picture cards and story books. The participating children were able to interact effectively with the VVA platform and produce new expressive vocabulary. The intervention phase data showed improvement in expressive verbal output and social interactions compared with the traditional baseline data. The children were able to correctly pronounce more words in fewer attempts. Additionally, the children’s interactions with their siblings were also reported to increase. The mothers reported being satisfied with using this program. These findings suggest that VVA programs might be useful as home-based intervention for children with speech and social difficulties. Sahin et al. (2018), reported very similar findings in their study that investigated the use of digital augmented reality in an intervention for social communication, motivation, and cognition in autistic individuals in a school environment. Further, this study is notable, as it explores the use of a technology that is readily available in home settings. Furthermore, it strengthens suggestions that utilizing VVAs’ ‘humanlike’ conversational skills could provide support for developing speech and social interaction skills at home (Hoy 2018, Porcheron et al. 2017).

Children with ASD generally have significant challenges in social interactions. Thus, utilizing a tool that could be readily available in their homes and that could potentially improve their social interaction skills could be very beneficial to their families. In studies performed by Ireland et al. (2018) and Parsons et al. (2006), participants showed improvements in their social interactions with the use of avatars and virtual reality applications in a simulated environment. Likewise, the findings of this study supported the effectiveness of using VVAs to enhance the participants’ speech and social interaction skills.

Qualitatively, the mothers, who participated in this study, noted multiple advantages of using VVAs. They asserted that Siri captured the children’s attention, as they asked their siblings to speak to and play with Siri. Further, the mothers noted that the time the children spent interacting with their siblings increased dramatically while using Siri. They indicated that prior to this; they experienced difficulties in finding activities that would be of interest to their children and could also potentially be beneficial for developing social and language skills. Indeed, these advantages could indirectly benefit children who have difficulties using expressive language, as well as with social interaction. It is also worth noting that one of the participants developed an interest in writing whatever he asked Siri, since the child observed Siri typing whatever was said to it.

Although the results of this study showed that the VVA intervention had positive effects on expressive language and social interaction skills, it is still unclear whether this was entirely due to the VVA. These improvements could be attributable to the mothers’ compliance to the intervention and follow-up. However, undoubtedly, the VVAs played some role, because the mothers also noted that their children were more interested in using the VVAs than they were in the traditional (baseline) method of learning new vocabulary and social interaction skills. The participating mothers reported that their children started to imitate the new words they heard from Siri. For example, the mother of P1 described her surprise over her child’s improvement and excitingly sent a video to the researcher showing the child saying, ‘I am not sure I understand,’ which was identical to a phrase commonly used by Siri. Similarly, the mother of P2 said that her child kept asking Siri questions, such as ‘Who are you?’ and ‘How old are you?’ P3 showed less interest in saying new words and more interest in asking Siri to open his favorite game. Nevertheless, this can be considered a positive interaction. Previous research suggested that distraction-free, non-judgmental conversation could be why autistic children find it appealing to interact with smart interactive platforms similar to VVAs, and the present study also supports such a supposition (Robins et al. 2017, Wood et al. 2017).

Study limitations and implications

Several limitations can be identified in this study. First, it was a single-subject design study with a low number of participants, thus limiting the generalizability of the findings. The second is the use of the A-B-A single-subject design because of time constraints. To further develop the findings on the effectiveness of VVA use for children with ASD, an A-B-A-B design might be stronger, because it would provide another opportunity to evaluate the effects of the intervention phase on the target skills or behaviors for the second withdrawal phase. The third limitation is the small number of sessions per phase. Increasing the number of sessions for each phase could also help more accurately determine the intervention’s effectiveness.

Directions for future research

In addition to considering how to overcome the aforementioned limitations in future studies, the following constructive suggestions can be made. The first is to extend the range of the target population to include not only children with ASD but also all children with speech and social difficulties. The second is to measure the impact of mothers’ intervention as compared to regular therapy sessions provided by therapists or special education teachers using validated objective and subjective assessment tools.

Despite its limitations, the findings of this study could contribute to providing assistance to children with speech and social difficulties, such as ASD, by using artificial intelligence platforms, such as VVAs, in homes and schools.

Conclusion

In conclusion, this study’s results demonstrated that using VVA had positive influence on expressive verbal vocabulary, production of short phrases, and social interactions in 3 children who have expressive language difficulty as well as social interaction difficulty due to ASD. The participated children were able to interact effectively with the VVA platform and produce new expressive vocabulary. The children were able to correctly pronounce more words in fewer attempts. Additionally, this study strengthens the idea that utilizing VVAs’ ‘humanlike’ conversational skills may provide support for the development of speech and social interaction skills at home for children with ASD.

Appendix 1.

A sample interaction with siri

Appendix 2.

Parent’s assessment checklist (PACL)

Appendix 3.

Social assessment checklist (SACL)

Funding Statement

No funding was obtained for this study.

Disclosure statement

No potential conflict of interest was reported by the authors.

Ethical approval

This study was approved by the United Arab Emirates University Ethics Review Board number ERS_2021_7331.

Informed consent

Informed consent was obtained from parents or guardians of all children.

Data availability statement

The datasets used during the current study are available from the corresponding author on reasonable request.