In-Home Speech and Language Screening for Young Children: A Proof-of-Concept Study Using Interactive Mobile Storytime

Abstract

Early identification and intervention of speech and language delays in children contribute to better communication and literacy skills for school readiness and are protective against behavioral and mental health problems. Through collaboration between the data science and clinical teams at Cognoa, we designed Storytime, an interactive storytelling experience on a mobile device using a virtual avatar to mediate speech and language screening for children ages 4 to 6 years old. Our proof-of-concept study collects Storytime session footage from 71 pairs of parents and children including 57 typically developing children and 14 children with a current or prior history of communication impairments. Initial findings suggest that participating children verbally engaged with the video avatar without significant differences in performance across age, gender, and experimental location, leading to promising implications for using Storytime as a future tracking tool with automated feature analyses to detect speech and language delays.

Introduction

Background

Speech and language disorders affect approximately 5 to 12% of U.S. children between two and five years old and are considered the “most common and least diagnosed disability of childhood” by primary care pediatricians²⁴. These speech and language disorders lead to deficits in communication, narrative, and social skills, and may also occur with complex neurodevelopmental disorders such as autism spectrum disorders and attention deficit/hyperactivity disorder^15,17. Speech and language impairments affect different modalities of children’s communication skills (e.g., listening, speaking, reading, and writing), leading to subsequent behavioral and mental health problems that also hinder children’s development of literacy and academic skills as well as school readiness¹⁹. Due to various sociocultural and economic factors (e.g., misconceptions of the impairments, lack of insurance coverage, long waitlists for therapy), many children often do not receive timely initial assessment and treatment, which can result in delays in intervention during a critical developmental period and fail to achieve maximal therapy outcomes.

Using mobile health (mHealth) technology offers unique opportunities to identify and monitor delays in speech and language development among young children. Today, many young children grow up using mobile devices on a regular basis. It is estimated that 38% of U.S. children under 2 years old have used mobile devices for media consumption, and 80% of children between 2 and 4 years old spend at least 20 minutes a day using a tablet or a smartphone^11,23. Adoption of mobile technology among speech language pathologists (SLPs) is also on the rise. Since 2012, it is estimated that at least 50% of SLPs who work with children with communication impairments have implemented mobile devices (e.g., the iPad) during speech therapy⁸, as these mobile devices offer numerous benefits (e.g., portability, customizability, multimodal interactions for intervention) to motivate children to learn¹⁸. With the widespread use of mobile technology among modern youth, concerns about excessive “screen time” spent for entertainment purposes and its potential detrimental effects on children’s communication development, have drawn public debates and research interests from different stakeholders such as health professionals, parents, and technology inventors. Many researchers have attempted to leverage the ubiquitous mobile technology through designing mobile games using automatic speech recognition to increase children’s motivation to participate in speech therapy tasks^7,9, however, existing progress on these applications are still under development and have not yet reached clinical success. Commercialized mHealth products such as Cognoa⁵ have succeeded in using machine learning to detect early signs of autism through parental questionnaires and home video screening¹. Yet, conventional speech and language screening still rely on skilled SLPs, and due to issues such as access to care and cost for services, children who are at risk for communication impairments have not benefited from technology-mediated means for early screening and timely recommendation for intervention.

Exploring children’s use of speech and voice technologies (e.g., Amazon Echo^TM) has also been an emerging area of research that draws attention from multiple research disciplines, such as educational technology²⁶, tele-rehabilitation²², and human computer interaction^4,25. It is estimated that 30 million US families have a smart speaker with a conversational agent (CA)⁴, a voice-only dialog system “embedded in personal technologies and devices”¹⁶. Existing efforts regarding CA primarily target adult users, and are limited to basic built-in conversational functions that are not tailored to the learning objectives, communication patterns, and individualized needs for children with disabilities. Even among children without communication impairments, commercial voice technologies have also led to communication breakdowns, frustration, and disengagement^4,25. These research suggested that in order to utilize CA as a clinical tool, researchers must acknowledge the fact that given its current accuracy in recognizing children’s speech and conversational intent, CA cannot function solely as a “speaker-independent speech recognition system” to offer clinical interpretations of children’s speech³. Therefore, it is imperative to investigate how to design a child-friendly voice-based interaction experience that augments the sophisticated human mediation that is typically provided by a medical professional.

In this digital era, more electronic books (e-books) and mobile storybooks are also being created in addition to traditional paper storybooks. Shared storybook reading is a naturalistic daily routine for families across different cultures and countries, and is also an evidence-based intervention technique used by SLPs to assess children’s receptive and expressive language skills as well as literacy skills¹². Many researchers have evaluated the use of mobile storybooks to increase children’s motivation and participation in storybook reading^2,20,26. They also have found that e-books with audio narration enable preliterate children to read independently⁶, and children who speak different languages can benefit from reading e-books¹⁴. While these digital books have become more ubiquitous, research that has attempted to use digital storybooks to mediate screening and monitoring of speech and language skills remains limited.

Research Questions

Informed by multiple strands of aforementioned research (e.g., mHealth, CA, and digital storybooks), we designed Storytime, an iPad-based interactive story-reading experience that uses a virtual avatar to mediate speech language assessment experiences in a home setting. Our study seeks to solve the current clinical dilemma of delayed speech and language screening by investigating the feasibility of designing an interactive mobile story-reading experience that augments a clinician-administered speech and language screening. Our study aims to address the following research questions:

1. How feasible is it for children with and without communication impairments to verbally engage with the interactive storytelling session?

2. Do factors such as age, gender, experiment location, and presence of impairments impact their performance, and does parent-rated performance score correlate with their actual performance?

3. Does the initial manual analysis of speech and language data collected from this interaction discriminate the children with and without impairments?

Methods

Participant & Settings

Participants in this study were recruited by emailing a select group of current Cognoa registered users, posting flyers at local clinics, schools, and public bulletin boards, and advertisement on multiple social media platforms (Facebook, LinkedIn, Twitter). Parents who were interested in the study were asked to complete an online survey and a phone screening to determine eligibility criteria, prior to being scheduled for the study. Inclusion criteria for the participating children included: (a) between the ages of 4 years, 0 months and 6 years, 11 months; (b) speak English with the participating parent in the home environment; (c) have familiarity with tablet computers; (d) able to follow instructions in English; (d) have at least 50 spoken words and can use 3-word sentences; (e) live in Northern California. These criteria ensured that children we recruited have adequate verbal communication skills and are capable of participating in this mobile Storytime interaction. During the study period from June to September 2017, we received 263 completed online surveys from parents, and the pilot study was conducted between August and September 2017 with 76 eligible pairs of children and parents (male = 36, female = 40; children without impairments = 65, children with reported communication impairments = 11; mean age = 60.6 months). During the study, video recordings for participants 5, 9, 43, 56, and 76 from the healthy group were lost or incomplete, therefore, the present study included video and audios from a total of 71 children and parents. In order to increase participation of families who cannot travel far to the primary study location, the pilot study was conducted by the lead researcher at one of the participants’ preferred locations: a user testing room at the Cognoa office (n = 40), a local library near the participant’s family (n =16), or the participant’s home (n =15). Specifically, children in the impaired group have the following primary diagnoses per parent report: apraxia of speech, articulation disorders, expressive language delays, mixed receptive and expressive language disorder, stuttering, autism, and sensory processing disorder. The study was approved by the Western Institutional Review Board (#1706380).

Materials

The total length of the Storytime video with both the story and the screening questions is 5 minute 22 seconds long. Prior to designing Storytime, we conducted extensive literature review with existing language assessment procedures and obtained feedback from the clinical team, which includes a licensed SLP, a developmental psychologist, a clinical research director, and a pediatric neurologist. We first adapted a story based on a poem from Valerie Cox “The Cookie Thief” and designed child-friendly animation for the video. We then created a female avatar to imitate the storybook reading experience, and also elicit children’s responses using 20 questions from four sections in the video: answering questions during Storytime, answering listening comprehension after Storytime, retelling a sequence of pictures from Storytime (Figure 1), and creating a new story using a colored version of the Cookie Theft Picture¹⁰.

Figure 1.

The visual and scripts of the avatar and individual screenshots from Storytime. The question in text was shown to children as audio only during the actual interaction.

After the initial development of the video, the lead researcher conducted a usability study with six children (five typically developing children, one child with autism) in their home. We then developed three different persona types for children: inattentive, shy, and disinterested, and added more prompts for the avatar to augment a more naturalistic interaction. Additionally, the child’s background information and parent feedback are recorded by the lead researcher by filling out an adapted version of the Kaderavek- Sulzby Book-Reading Observational Protocol (KSBOP)¹³. In the adapted version of KSBOP, we collect parent feedback and parent rating of their children’s performance during Storytime using a rating scale of 0-10, with 0 being least close and 10 being the closest to their typical performance for paper-based storybook reading at home. The rationale behind this measure is that since parents are familiar with their children’s personality and communication patterns, it is likely that parents’ perceptual report about their children’s performance during the Storytime interaction may offer insights to how their children engage in storybook reading at home.

Experiment Procedure & Measures

During the study, a child-parent pair was invited to one of the experiment settings to watch the Storytime video on an iPad, covered in a protected screen. The parent and the child were asked to sit side-by-side, as how they typically engage in story time together. All interactions were video-recorded from an external camera installed on a Macbook Air laptop computer, placed across from the child (Figure 2). The lead researcher used a remote control to pause the video after each question was presented and waited for the child to verbally give an answer to the question before proceeding to the next story line or question. In order to obtain a non-interrupted and naturalistic speech sample from the child, the parent was asked not to verbally or non-verbally prompt the child throughout the study. During incidents when the child is unable to answer specific questions, the lead researcher will either repeat the question or verbally prompt the child to respond.

Figure 2.

Setup of the iPad and screen protector, and the recording laptop and camera

After data collection, the lead researcher extracted the audio data from all video recordings using Adobe Audition, then segmented and transcribed each child’s responses for the 20 assessment questions from in the video. After all audios were segmented and transcribed to their corresponding assessment questions, a scoring system was developed using two response variables: “Total Score” and “Listening Comprehension Accuracy Score”. “Total Score” calculates how many times a child responded to all 20 assessment questions (response = 1 point; no response = 0 point). “Listening Comprehension Accuracy Score” calculates how many correct answers based on children’s response to eight specific assessment questions out of the 20 total assessment questions that are specifically asked after the story to test comprehension of the story. These eight questions were specifically designed to assess children’s understanding of the story and can be scored using a binary scoring system (correct = 1 point, incorrect = 0 point). It is important to note that children may encounter difficulties answering questions during the study; therefore, responses prompted by either the parent or the lead researcher were not considered as correct responses. Unintelligible responses and grammatically incorrect responses were counted as incorrect.

Results

Children’s Total Verbal Response

We first analyzed the Total Score of all 71 participating children to determine how many times they produced an intelligible verbal response during the Storytime interaction (Table 1). A regression analysis of Total Score using age and parent rating measures suggest that children’s age and the perceptual rating from their parents are significantly to predict their performance. This finding is reasonable because we anticipate that the Storytime interaction offers adequate opportunities for children to respond verbally, despite differences in age. Parent rated their children’s response after observing the study, so their rating should be reliably predictive of their children’s actual response. This finding suggests that in addition to children’s interaction with Storytime, parent-reported can be utilized to provide an additional measure to evaluate children’s performance during Storytime compared to their typical storybook interaction.

Table 1.

Participant Performance.

Measure	Mean	SD	t	p
Age (in Months)	60.62	9.03	56.58	< 0.0001
Experiment Location (Office/ Library/ Home)	40/ 16/ 15
Gender (Female/ Male)	38/ 33
Group (Healthy/ Impaired)	57/ 14
Parent Rating	6.264	0.12	23.81	< 0.0001

Influential Factors: Age, Location, Gender, and Parent Rating

In order to evaluate all children’s performance during Storytime across different settings, we first used Total Score for all 71 children and conducted group comparison for several measures, including experiment locations (Cognoa office, library, and home), gender (female vs. male), and presence of impairment (impaired vs. health). The results from our statistical analysis ( p < 0.001) found no interaction across three experimental settings, genders, or presence of the impairment (Table 2). This suggests that although our pilot study was conducted across different settings, children were able to respond similarly to the interactive video across all three settings without significant differences in gender and presence of impairments.

Table 2.

Children’s Total Score and Listening Comprehension Accuracy Score.

	Total Score		Listening Comprehension Accuracy Score
Measure	Estimate	p	Estimate	p
Age (in Months)	0.17	< 0.001	0.13	< 0.0001
Location
Office	15.68	< 0.001	4.50	< 0.0001
Library	2.33	0.11	0.44	0.52
Home	-0.61	0.68	0.03333	0.96
Gender
Female	0.87	0.46	0.62	0.24
Male	15.61	< 0.0001	4.27	< 0.0001
Group
Healthy	1.24	0.40	0.75	0.25
Impaired	15.07	< 0.0001	4.00	< 0.0001
Parent Rating	1.31	< 0.0001	0.55	< 0.0001

Group Differences Between Children with and Without Impairments

We then analyzed children’s performance on the Listening Comprehension Accuracy Score using categorical factors of interest (e.g., gender, experiment location, and presence of impairments) as well as other numerical factors (e.g., age, parents’ ratings). These measures were taken from background information recorded on the KSBOP-Adapted Protocol. Independent t-tests were also used to compare the impaired group and the healthy group (Table 2), and we found these groups were not statistically significantly different between measures such as age, gender, and experiment location, suggesting that accuracy score is not affected by these measures. However, we also found no statistical significance between the healthy and impaired groups. This means that the eight questions in the listening comprehension section were not sensitive to differentiate children who have impairments and therefore, future design iteration may need additional assessment questions.

In addition, as illustrated in Figure 3, a positive correlation was found when comparing the age of participants and their performance on the Listening Comprehension Accuracy Score section. This upward trend suggests that as children’s age increases, older children achieve more accuracy in their Listening Comprehension Accuracy Score comparing to younger children who are still acquiring language skills. Although we cannot conclude that this graph alone suggest that our test items are created to be developmentally appropriate, it shows a potential age effect.

Figure 3.

Age effect by children’s age and their Listening Comprehension Accuracy Score.

Discussion

In this paper, we presented the background research and design of Storytime and described initial findings from manual analysis based on the video and audio recordings collected during the Storytime interaction from 71 participants. Preliminary data analysis showed promising results, indicating that the Storytime video has the potential to become a more robust tool to engage children with communication impairments in order to obtain verbal speech and language production for screening. Our Total Score demonstrated that children with and without impairments across all ages, genders, and experiment locations were all able to engage with the interactive experience without significant variability. Analysis for Listening Comprehension Accuracy Score” demonstrated an age effect, however, a significant difference to discriminate between the impaired and healthy group was not found through the eight questions.

Parent rating was an effective predictor for both the Total Score and Listening Comprehension Accuracy Score, revealing the value of using reports from parents’ perceptual rating. Parent feedback collected during the pilot study also highlighted several areas of improvement for our study. Many parents reported that they prefer child-friendly visual designs (e.g., using animals and children to replace human characters) with a simpler story plot in children’s familiar environments (e.g., park, home). Parents with younger children expressed interests in having more verbal prompts in the video to facilitate both the story and assessment questions, so that they can avoid interrupting the Storytime experience and providing prompts to children. Some parents also wanted to see more playful elements with silliness and humor so that children can interact with them during Storytime, such as moving screenshots around as puzzles to complete the story retell task. Additionally, many parents have different values for teaching social concepts (e.g., sharing), therefore, when designing assessment questions such as “tell me a time when you shared with someone,” we may impose additional cultural-linguistic challenges for children to respond, difficulties for children to recall past experiences, as well as ambiguity in scoring due to children’s diverse levels of response. Also, it is more appropriate to avoid asking questions related to subjective and personal experience or preferences (e.g., “What type of cookies do you like?”), because these types of questions introduce new challenges in scoring. On the other hand, we must acknowledge that, comparing to fact- based questions in the story, personal experience questions are also relatively easier to answer and allow children to generate a relatively larger linguistic corpus for data analysis later. These findings have both clinical and design implications on the importance of balancing assessment validity with most optimal user experiences during the conversational interaction with an avatar in the video.

There are several study limitations that need to be addressed. Our experiment location was not randomly assigned, but was determined purely based on convenience reasons from participating families. This resulted in unbalanced groups for three different settings, but also is applicable to existing practices of speech language evaluation which typically take place across multiple settings. During data analysis, prompted answers were not taken into consideration. Because we did not specify the type of prompts, it is likely that children were underscored for their true language ability if prompts were to encourage children to continue talking (e.g., parent encourage children with verbal praises) rather than giving cues for correct answers (e.g., repetition of questions by the parents or the lead researcher). Also, all data were coded by the lead researcher, therefore, the transcribed verbal response may have implicit biases especially when speech is unintelligible to the lead researcher.

Analyzing parent-child interaction during media use and investigating ubiquitous computing opportunities using mobile technology allow researchers to redesign traditional clinical assessments; however, additional research is needed for developing best practices when designing for young children. Despite the rapid development in mobile and interactive media for language learning, creating a fun and engaging language assessment application remains a challenge for child language researchers and interaction designers for children. The scholarly contribution of our study is multifold. In terms of clinical implication, we have learned the challenges in terms of creating a developmentally sensitive speech and language screening tool using technology-mediated mobile interaction for young children between 4 to 6 years old. Future studies should seek to control several variables (e.g., experimenter effect, parental influence during the interaction) or capture multiple interactive videos to ensure a more representative performance from the child. Responses from participating children during Storytime revealed that they are more likely to respond to questions when seeing an avatar than only hearing an audio question in a scene in the story. As prior literature suggested, having a human-like avatar facilitates the conversation2 and reduces the opportunity for mislabeling children’s lack of response as potential impairments. Additionally, despite being told by the lead researchers that there are no right or wrong answers, children are hesitant to answer if they are unsure of the correct answer. Future studies should investigate the feasibility of designing an intelligent virtual assistant that provides different levels and amounts of built-in prompts that are more adaptive to children’s age, ability, and personality.

In terms of design implication, implementing child-centered design and inclusive design to accommodate diverse abilities and interests from young children across 4 to 6 years old is more difficult than we anticipated. Additionally, offering children ample time to think, pause, and reformulate a response is crucially important. Therefore, for future studies that intend to create an automated Storytime experience, researchers must conduct extensive research to design to record response time and evaluate the length of response time prior to automation. Since the ultimate goal of Storytime is to utilize the existing parent- child reading experience at home to collect data from children’s verbal outputs, designers need to create an interaction that ensures children are proving their most naturalistic responses rather than overperforming or underperforming in a testing situation. This can be achieved by designing Storytime using child-centered approach with visuals and content of children’s interests to improve engagement with Storytime experience, prior to creating multiple interactions to track and monitor children’s ongoing speech and language development.

In terms of implication for ongoing data analysis, our clinical team will continue the manual analysis of children’s responses by analyzing their vocabulary use, speech sound errors, sentence structure, and story elements, then apply natural language processing techniques to develop salient linguistic features that can distinguish children with and without impairments. Our data science team will transform raw audios into spectrogram and build machine learning algorithms to investigate salient characteristics (e.g., length of speech, rate of speech, number of speech sound produced, prosody, and pitch) that supports future classification of speech and language delay for children using Storytime. We anticipate that features from speech and language signals extracted can offer valuable clinical information to classify various pediatric communication disorders, such as articulation and speech sound disorders and expressive language disorders.

Conclusion

Through analyses of children’s response and parent feedback collected from this experiment, our study suggests that using the interactive mobile Storytime has great potential to capture speech and language production from 4 to 6 year old children, and over time, may offer opportunities for clinical and data science researchers to aggregate multiple video interactions of children’s communication development in a trajectory to screen and track potential delays and impairments. Our study also brings both clinical and design implications for conducting studies to investigate voice-based interfaces in the context of evaluating children’s speech and language development. By combining automatic speech recognition and machine learning analysis of acoustic and linguistic features for an automated assessment at home, tools such as Storytime have the potential to alleviate burden of care for primary care providers, support parents with reliable information about their children’s development, and provide access to early intervention during critical neurodevelopmental periods. Although modern youth live in a digital era and are highly exposed to mobile and interactive media for entertainment and learning, the role and value of new technology in traditional clinical assessment remain unexplored to designers and researchers. We anticipate a successful tool that detects speech and language delays and disorders has the potential to alleviate the burden of care for primary care providers, support parents with reliable information about their children’s development, and provide access to early intervention during critical developmental periods.