A Web-Based Tutorial for Parents of Young Children with Autism: Results from a Pilot Study

Abstract

Objective: Early intervention can significantly improve long-term outcomes for children with autism. Unfortunately, many children do not receive early intervention services due to a critical shortage of trained professionals in this area. To bridge this gap, we evaluated a Web-based parent training tutorial (Enhancing Interactions), based on evidence-based practices and utilizing the Web-based platform to maximize learning. Methods: Twenty-three parents with a child between 18 months and 6 years with an autism spectrum disorder participated. Pre- and posttest scores of parents' knowledge were used to evaluate tutorial effectiveness. The system usability scale (SUS) evaluated technical user-friendliness and the user satisfaction questionnaire (USQ), gauged satisfaction with content. Results: The mean number of correct items on the posttest significantly increased, from 12.6 to 20.4, p<0.001. The mean SUS score was 85 (standard deviation=17), corresponding to a score of “excellent.” All participants found the tutorial user friendly, well integrated, and 96% (all but one participant) thought it was easy to use, felt confident using the technical features, and would use a tutorial like this again. On the USQ, all participants found that the tutorial was well organized, clearly presented, and easy to understand; that it increased their knowledge about communicating with their child; and that they felt capable of applying these techniques with their child. Conclusions: The tutorial appears effective in increasing parents' knowledge with high user satisfaction.

Introduction

Autism is a neurodevelopmental disorder characterized by deficits in social reciprocity, impairments in verbal and nonverbal communication, and a restricted behavioral repertoire that can include repetitive movements, sensory abnormalities, and a need for sameness.^1,2 A growing body of scientific literature has established that autism spectrum disorders (ASDs) have an estimated prevalence of 1 in 110 children, making ASD among one of the most prevalent of all developmental disabilities.³

While there is no known cure for autism, there is a growing consensus that participation in intensive and specialized early intervention can significantly improve long-term outcomes for children and their families.^4–6 Early intervention is critical for preventing a cascade of effects resulting from early deficits that interfere with later functioning.⁷ Early specialized interventions for preschoolers have been shown to improve language functioning,⁸ cognitive/developmental skills,⁹ and social and adaptive behavior.¹⁰ We also know from developmental neuroscience that because of greater brain plasticity in very young children, early interventions can play a significant role in shaping neural connections in the brain.¹¹ Such interventions can attenuate the development of negative secondary sequelae.⁷

Unfortunately, there is a gap between the growing knowledge regarding effective intervention strategies and the availability of these services, due to a critical shortage of trained professionals in this area.¹² Recent surveys show that many children who are eligible for early intervention services do not receive them,¹³ as these federally mandated systems are often not equipped with adequate financial or human resources to provide individualized intervention.^14,15 This situation often places the burden on parents for identifying, locating, and paying for children's therapies, as well as serving as primary interventionists themselves.

One way to bridge the gap between the need for services and their availability is to teach parents how to incorporate evidence-based intervention techniques in their daily interactions with their children. By empowering parents with the knowledge they need to teach their young child with ASD, we can provide resources directly to the people who have the most vested interest and the longest lasting influence on a child's long-term growth and development. One such successful parenting program called Enhancing Interactions provides one-on-one training to caregivers of young children with autism that focuses on promoting social communication skills and managing challenging behaviors across a variety of everyday interactions and routines. This program, developed at the Treatment and Research Institute for Autism Spectrum Disorders (TRIAD) at Vanderbilt University, was created in part to address the shortage of available community intervention services. Based on empirically validated techniques that integrate developmental and behavior analytic approaches, this program addresses challenging behaviors and fosters skill development by modeling techniques as well as coaching parents as they implement the strategies themselves. The program has the dual goals of enhancing interactions and parenting effectiveness while at the same time improving the core deficits and long-term outcomes in the child.

While the Enhancing Interactions program has been highly successful and well received by participating parents, the availability of this training is limited by geographic constraints as well as the shortage of clinicians trained to provide such services. One powerful way to increase the availability of this program is through the use of Web-based technologies. In addition to enabling wider access and dissemination, the Web affords interactive and multimedia learning, which when coupled with principles of learning and instructional design has been shown to increase compliance and knowledge retention.^16,17 Web-based training is also an effective way to reduce treatment costs. Recent reports have shown an increased interest in utilizing Web-based techniques and telemedicine applications for the identification and treatment of autism.^18,19

The goal of the current study was to develop and evaluate the efficacy and user satisfaction of a Web-based version of the Enhancing Interactions program.

Methods

Participants were 23 parents with a child between 18 months and 6 years of age with an ASD. Parents were recruited from TRIAD-affiliated clinical, research, and training programs at the Vanderbilt Kennedy Center. The mean age of parents was 33.7 years (standard deviation [SD]=6.4; range 24–51). Participants were 74% Caucasian, 22% African American, and 4% from other racial/ethnic backgrounds. Sixty-one percent had received the diagnosis of ASD for their child within 6 months prior to participating. None had participated previously in the one-on-one Enhancing Interactions parenting skills training.

In order to evaluate the efficacy and user satisfaction with a Web-based version of Enhancing Interactions, we developed a prototype of the tutorial containing three of the Enhancing Interactions modules: an educational overview, describing the core characteristics of autism and how they impact communication and behavior; a description of teaching strategies, which reviews the concepts of reinforcement, modeling, and prompting; and specific strategies for improving communication, which focus on teaching functional communication skills to improve a child's ability to request desired objects or activities. The tutorial contains videotaped illustrations of both the target behaviors and a parent using the intervention techniques correctly. Learning is interactive and employs graphics and multimedia techniques to illustrate concepts and enhance retention.

The study was reviewed and approved by the Allendale Institutional Review Board. After obtaining informed consent, parents were given a username and password, and completed the tutorial at their own pace.

Measures

Pre- and posttest scores of parents' knowledge of the concepts presented in the tutorial were used to evaluate teaching effectiveness. A pool of items was developed using a separate development sample and 24 items were selected based on their sensitivity in discriminating between parents who had and had not been exposed to the Enhancing Interactions training.

Two measures were used to assess user satisfaction: the system usability scale (SUS)^20,21 and the user satisfaction questionnaire (USQ).²² The SUS is a reliable, well-validated 10-item scale designed to evaluate satisfaction with the technical aspects of Web-based applications and other technologies. It obtains quantitative feedback on a 0–100 scale regarding the effectiveness, efficiency, and user satisfaction experience while interacting with engineered systems, which evaluates parents' satisfaction with the technical aspects of the tutorial. The USQ has been used to evaluate how well subjects like the clinical content of computerized or Web-based applications.^22–28 It contains 15 statements, each of which is rated on a 4-point scale (strongly agree, agree, disagree, strongly disagree), and covers several dimensions of the user experience (Table 1). Good internal consistency has been reported (Cronbach's alpha=0.92). An overall mean score of 45 (i.e., an average score of 3 [“agree”] on all items) is considered to reflect acceptable user satisfaction with the clinical content.²³

Table 1.

User Satisfaction Questionnaire (USQ) Results

	STRONGLY DISAGREE	DISAGREE	AGREE	STRONGLY AGREE
1. The objectives of the tutorial were clear	-	-	5 (21.7%)	18 (78.3%)
2. The tutorial was well-organized	-	-	7 (30.4%)	16 (69.6%)
3. The material was presented in an interesting manner	-	1 (4.3%)	9 (39.1%)	13 (56.5%)
4. There were sufficient examples and illustrations	-	1 (4.3%)	5 (21.7%)	17 (73.9%)
5. The concepts were clearly presented and easy to understand	-	-	10 (43.5%)	13 (56.5%)
6. The video examples were helpful in illustrating the concepts	-	-	5 (21.7%)	18 (78.3%)
7. The animations were helpful in illustrating the concepts	-	1 (4.3%)	8 (34.8%)	14 (60.9%)
8. The photos were helpful in illustrating the concepts	-	-	6 (26.1%)	17 (73.9%)
9. The tutorial increased my knowledge about communication and improving my child's requesting	-	-	6 (26.1%)	17 (73.9%)
10. I feel capable of administering these techniques with my child	-	-	7 (30.4%)	16 (69.6%)
11. The length of the tutorial was appropriate	1 (4.3%)	4 (17.4%)	7 (30.4%)	11 (47.8%)
12. This technology was as effective as traditional teaching methods in helping me learn the material	-	-	9 (39.1%)	14 (60.9%)
13. I would recommend this course to others	-	1 (4.3%)	6 (26.1%)	16 (69.6%)
14. I enjoyed taking the tutorial	1 (4.3%)	1 (4.3%)	6 (26.1%)	15 (65.2%)
15. Overall, I was satisfied with the tutorial	-	2 (8.7%)	3 (13.0%)	18 (78.3%)

The course was developed using Articulate, a rapid-development e-learning tool that publishes into HTML/Flash, and delivered on a Moodle platform (for Modular Object-Oriented Dynamic Learning Environment), a popular open-source learning management system. Moodle provides full featured functionality including user authentication, course delivery, and user reports and statistics. Moodle is developed in hypertext preprocessor (PHP), and because it is open source, can be modified to meet custom requirements.

Results

To examine whether the tutorial was effective in increasing parent's knowledge of the concepts presented in the tutorial, we examined changes in the number of correct answers on the pretest (prior to taking the tutorial) versus the posttest (Table 2). We found a significant increase in the mean number of correct items, from 12.6 to 20.4, t(23)=10.72, p<0.001. Seventy-nine percent scored 80% correct or better after taking the tutorial, compared to only 8% prior to taking the tutorial.

Table 2.

Pretest and Posttest Scores on Tutorial Concepts

	MEAN	N	SD	STANDARD ERROR MEAN
Pretest	12.6667	24	4.15636	0.84841
Posttest	20.4167	24	2.39414	0.48870
Difference	7.75000		3.54168
T	10.72
P	.000

The mean score on the SUS was 85 (SD=17), which corresponds to a score of excellent (Table 3). Fifty-two percent of parents had a mean score above 90 (the highest rating possible), and 44% rated it good or excellent. No parent rated it lower than OK (scale midpoint). On individual SUS items, all parents found the tutorial user friendly, and thought the technical features in the tutorial were well integrated. Ninety-six percent (all but one parent) said they would use a tutorial like this again, thought the tutorial was easy to use, thought other people would find the tutorial easy to use, and felt confident using the technical features of the tutorial. Only one parent (4%) thought that she would need help from someone else to use the tutorial.

Table 3.

Interpreting System Usability (SUS Score)

ADJECTIVE	SUS CUTOFF SCORE	PERCENTAGE OF PHASE I SUBJECTS ABOVE CUTOFF
Worst imaginable	12.5	0%
Awful	20.3	0%
Poor	35.7	0%
OK	50.9	4%
Good	71.4	13%
Excellent	85.5	31%
Best imaginable	90.0	52%

From Bangor et al. (2009).²⁰

SUS, system usability scale.

The mean score on USQ was 54.5 (SD=5.9). This score was significantly higher than the minimum score of 45 defined as an acceptable level of user satisfaction with the course content, t(22)=7.73, p<0.001. On individual items, all parents agreed or strongly agreed with the following statements: “The objectives of the tutorial were clear,” “The tutorial was well organized,” “The concepts were clearly presented and easy to understand,” “The tutorial increased my knowledge about communication and improving my child's requesting,” “I feel capable of applying these techniques with my child,” “The video examples were helpful in illustrating the concepts,” and “This technology was as effective as traditional teaching methods in helping me learn the material.” All but one parent (96%) agreed or strongly agreed with the statements: “The material was presented in an interesting manner,” “There were sufficient examples and illustrations,” and “I would recommend this course to others.” Eighteen parents (78%) agreed or strongly agreed with the statement that “the length of the tutorial was appropriate.”

Open-ended comments included are “I have already started using some of the tips in the tutorial with my child. The content helped me understand why she reacts certain ways to new and different situations when it comes to communication,” “The information was practical and personal. I especially liked seeing the families engaged in the actual suggested materials,” “I believe I have learned more than I have with any other program so far. It was easier to follow and remember,” “This helped me to understand my son better and to work with him with his communication,” “This tutorial gave me lots of great ideas on improving requests and using pictures,” “I enjoyed this tutorial and will definitely use these techniques and strategies with my child,” “I would recommend this course to other parents and caregivers of ASD children,” “I look forward to using other tutorials like this one in the future,” and “Wish I had exposure to a tutorial like this when we started speech therapy 2½ years ago, well before we even had a diagnosis.”

Comments that included constructive criticism are “I prefer to learn by reading on my own and at my own pace, and there wasn't much of an option for that. I found it annoying to have to listen to someone speak to me on the slides,” “Maybe the videos for examples could be longer or put more videos for the examples,” My only frustration with this material was the downgrading of sign language,” “I noticed there were a lot of boys in the examples and few girls,” and “Photos were helpful, would have liked them to be bigger.”

Discussion

Results provide evidence for the efficacy and user satisfaction of the Web-based version of Enhancing Interactions. The course was successful in improving parents' knowledge of the course concepts and parents rated the experience as positive both on content and usability. Further research will be conducted on the full version containing all the content from the live Enhancing Interactions training. Autism is a widely prevalent disorder with significant impact on the quality of life for individuals with the disorder and their families. An effective Web-based tutorial will help address the significant gap between the large unfulfilled need for effective intervention services and their availability.

Autism also has a significant financial impact on both families and society as a whole. Direct costs (i.e., direct medical costs, prescription medication, and behavioral therapies) are estimated at $29,000 per person per year and indirect, nonmedical costs (i.e., special education and child care) are estimated at more than $39,000 per person per year.²⁹ A Web-based tutorial is an effective way to reduce treatment costs to families, as well as to state agencies and educational systems that could use the tutorial to provide immediate intervention opportunities for eligible children without delay.

Future research will examine the extent to which parents are able to take the knowledge gained from the tutorial and successfully apply it with their child at home. There is a difference between a skill set and a knowledge set. One can have a didactic understanding of the concepts and techniques presented in the tutorial and not necessarily be able to implement these techniques with their child. As a pilot study, examining whether the tutorial increased conceptual knowledge and was technologically feasible and acceptable from the users' perspective were appropriate initial goals. In our follow-up study we will evaluate the extent to which the tutorial actually changes parenting behaviors, and whether these changes in parenting behaviors result in improvements in their child's functioning.

In addition, the combination of a Web-based tutorial and subsequent remote observation of parents at home will be explored. Advances in videoconferencing technology have made its use both inexpensive and widely available. Through the use of simple Webcams and free videoconferencing services such as Skype, interventionists can observe parents applying these skills in real time with their child, providing feedback, encouragement, and support through the use of a wireless Bluetooth headset. A case report of such an intervention was recently reported, where weekly face-to-face office sessions were supplemented with at-home sessions supervised and coached remotely by the clinician using videoconferencing.¹⁹ A two-stage training model involving the use of Web-based tutorials for didactic training combined with live, remote observation via videoconferencing has been successfully implemented in training on assessing depression,²⁷ and schizophrenia.²⁴ Such a model will be piloted in development of the full Enhancing Interaction tutorial.

Rapidly evolving advances in Web technology, combined with the increasing use of the Web, provide a unique opportunity to deliver effective interventions directly to parents of children with autism. The number of mothers using the Web regularly increased from 11% to 63% between 2006 and 2009.³⁰ Thirty results from the current study show that effective Web-based services can be developed and delivered directly to parents, and have the potential to solve the dilemma of increasing demand coupled with dwindling resources. Such an intervention may help remediate the core deficits associated with autism, improve both short- and long-term functioning, reduce caregiver stress, and improve the quality of life for both parent and child. Telemedicine applications administered directly to children with autism may also have great potential,¹⁸ since it overcomes the problems children with autism have with social interaction and social behavioral processing. The use of such an approach in improving social skills in children with autism has recently been reported,³¹ and further research in this area is ongoing and promising.

Acknowledgments

This project has been funded in whole or in part with Federal funds from the National Institute of Mental Health, National Institutes of Health, Department of Health and Human Services, under Grant No. 5R43MH086936-02.

Disclosure Statement

Kenneth A. Kobak, P.I. Drs. Stone, Warren, Kobak, and Ms. Wallace have a commercial interest in the Enhancing Interactions tutorial. No other conflicts of interest exist.