Web-Based Training in Early Autism Screening: Results from a Pilot Study

Abstract

Background

Lack of familiarity with early signs of autism by community service providers has resulted in significant delays in children receiving early intervention services necessary to improve long-term outcomes. The Screening Tool for Autism in Toddlers and Young Children (STAT) was specifically developed to identify early behavioral features of autism. Although STAT training has been available for years, access is limited because of few STAT trainers and geographic concerns. This study evaluated the efficacy and acceptability of Web-based training of the STAT as a means of increasing accessibility to this training.

Materials and Methods

Thirty professionals from three geographic areas participated. Roughly 1 of 3 had little or no training on autism assessment. The tutorial contains a general overview, administration and scoring conventions, and item-specific content and concepts. Participants completed a pretest and then completed the STAT tutorial at their own pace, followed by a post-test and a user satisfaction questionnaire.

Results

Mean scores on STAT concepts significantly improved after taking the tutorial (p < 0.001). At pretest, only 1 person (3%) obtained correct scores on at least 80% of the items (a priori cutoff for a “pass”), compared with 22 (73%) at post-test (p < 0.001). The majority of trainees enjoyed taking the tutorial, thought it was well organized, relevant, interesting, and useful, and felt it was easy to understand and operate.

Discussion

Results support Web-based training as a promising method for promoting early identification of autism and may help overcome problems associated with the critical shortage of autism-screening professionals.

Introduction

In recent years, the early identification of children with autism has become an important priority for research as well as clinical endeavors. A significant impetus for this movement has been the recognition that autism-specialized intervention can lead to significant gains in social, communication, behavioral, and cognitive functioning when implemented at young ages.^1–3 Unfortunately, there are often long delays between the time a child is suspected of autism and the time he or she receives a definitive diagnosis. Research over the years has consistently found that parents begin to have concerns about their child's development at an average age of 16–20 months^4–8; however, many children fail to receive a definitive diagnosis of autism until the age of 4 or 4½, or even older.^9–12 This delay can prevent children from receiving appropriately specialized early intervention and may ultimately increase the lifetime cost and service system demands associated with providing care and support to individuals with autism and their families.

Because early identification is the first step toward providing children and families with specialized intervention, there has been a dual focus on identifying early behavioral signs of autism as well as developing effective methods and measures for early screening. Several professional groups, including the American Academy of Neurology¹³ and the American Academy of Pediatrics (AAP),¹⁴ have developed practice guidelines to promote early identification and intervention for children with autism. Reports of increasing prevalence of autism spectrum disorders, currently estimated to be 1 in 110 children,¹⁵ led to the AAP's recommendation that all children receive routine autism-specific screening twice before the age of 30 months.¹⁴ However, many obstacles to conducting autism screening in primary care settings have been identified, including time constraints and a lack of familiarity or training in the early behavioral manifestations of autism, and previous studies have indicated that autism screening by primary healthcare providers is not a routine practice.^16,17

An additional, and complementary, approach for increasing early detection is the use of second-stage autism-specific screeners in referral settings.^17,18 After developmental concerns have been identified, second-stage autism screeners can be used by Birth to Three personnel and other community providers to direct high-risk children toward autism-specialized assessments or interventions. The Screening Tool for Autism in Toddlers and Young Children (STAT)^19,20 is an interactive, play-based screening measure that was developed for this purpose. It consists of 12 activities assessing imitation, play, and communication that take about 20 min to administer. Originally designed for children between 24 and 36 months, its potential utility for children as young as 14 months has been demonstrated.²¹ The STAT has good sensitivity and specificity, high concurrent validity with both the Autism Diagnostic Observation Schedule (ADOS) and Diagnostic and Statistic Manual of Mental Disorders, Fourth Edition (DSM-IV) diagnoses, and good inter-rater reliability when used by professionals trained in its administration and scoring.^19,20

Because of the interactive nature of the STAT items and the subtlety of the early social-communication markers of autism (i.e., negative symptoms), training on the STAT is required to ensure appropriate administration and behavioral interpretation. For over 10 years, community service providers have received STAT training through participation in formal face-to-face training workshops. More recently, a Web-based training tutorial for service providers was developed to increase access to STAT training. The purpose of this study was to provide a preliminary assessment of the acceptability and usability of the Web-based STAT tutorial as well as its effectiveness in increasing the user's knowledge of STAT administration procedures and scoring conventions.

Methods

Participants

Thirty community healthcare professionals (29 female and 1 male) from three geographic areas (Madison, WI; Nashville, TN; and Atlanta, GA) participated in the evaluation of the Web-based STAT tutorial. The sample included a range of professionals in community and academic settings who self-identified as psychologists (n=6), researchers or professors (n=5), research staff (n=5), speech-language pathologists (n=4), educators (n=3), consultants (n=3), pediatricians (n=2), a nurse, and an occupational therapist. Twelve participants (40%) had an M.D. or Ph.D., 14 (47%) had a master's degree, and 4 (13%) had a bachelor's degree. Their experience working with children with autism ranged from <1 year to 35 years (mean=12.0 years; SD=10.7; mode=4 years). About 25% of the sample (n=8) reported having little or no training in autism assessment. Twenty-five participants (83%) were Caucasian, 2 (6%) were African American, 2 (6%) were Asian, and 1 (3%) was Hispanic. Inclusion criteria were broad to enroll professional from a wide variety of educational backgrounds and professions and consisted of having a college degree and be currently working with children with autism spectrum disorder (ASD).

For the purpose of obtaining a benchmark, results obtained by participants completing the Web-based tutorial were compared with those obtained by individuals (n=27) participating in two face-to-face STAT training workshops before the Web-based tutorial was developed. STAT training workshops are conducted in small groups (usually 12–18 participants) in 1 day. The face-to-face training parallels the online training in terms of content and topics covered. Participants begin with a general overview of the STAT, a review of general administration and scoring guidelines, and video illustrations of scores on the STAT items, followed by ratings of videos and discussion. Although lack of randomization to the different training formats limits the inferences that can be made, this comparison sample provides a reference point for expected gains in knowledge. Participant demographics for the workshop sample were similar to those of the Web-based training cohort. The workshop sample was 70% Caucasian and 85% female. Participants included four clinical psychologists, one pediatrician, two social workers, nine speech-language or behavioral analysts/interventionists, and 11 from a variety of other related professions. Six participants (22%) had an M.D. or Ph.D., 17 (63%) had a master's degree, and 4 (15%) had a bachelor's degree.

STAT Training Tutorial

The Web-based STAT training tutorial covers the same content areas that are provided in the workshop training format and was designed using principles of adult learning and instructional design, such as use of multimodal input,²² high levels of interactivity,²³ appropriate “chunking” and sequencing of information to increase retention,²⁴ and use of the “test-enhanced learning” to improve long-range retention.²⁵ Interactivity was extensively utilized to engage learners and improve retention. The tutorial contains three content sections: a general overview on the structure of the STAT; a description of general administration procedures; and a description of item-specific content, concepts, and scoring conventions. The Overview describes the 12 STAT items that comprise the domains of play (2 items), motor imitation (4 items), requesting (2 items), and directing attention (4 items). The General Administration section reviews rules such as presenting the materials exactly as specified, scoring each item before proceeding to the next item, praising the child's efforts as well as his or her successes, and remaining flexible in terms of the location and order of presenting the material. An interactive, informal self-test is incorporated at the end of the section to reinforce learning. The Individual Item section describes the content and scoring for each of the 12 STAT items. Information for each item includes a description of what the item measures, the specific administration procedures, video examples of a child who passes and fails the item, and a final video that the trainee scores. The correct answer is presented after the trainee's response as a means of reinforcing learning.

In addition, two forms of evaluation were used to assess the training uptake and experience: a pre- and post-test assessment of knowledge; and a post-test assessment of feasibility of use. The Knowledge Questionnaire comprises 25 items (10 multiple-choice questions and 15 video vignettes that are scored pass or fail). The multiple-choice questions assess the participant's knowledge of administration procedures (e.g., the use of verbal instructions and the order and presentation of items) as well as scoring conventions (e.g., coding refusals and behaviors directed to a parent instead of the examiner). The video vignettes consist of segments that the participant scores as “pass,” “fail,” or “emerge.” The Feasibility Questionnaire consists of 13 items assessing different dimensions of the training experience, such as organization of the tutorial, utility of video examples, and time allotment. Each item consists of a statement that is rated on a four-point scale (4, strongly agree; 3, agree; 2, disagree; 1, strongly disagree); scores can range from 13 to 52, with higher scores indicating greater satisfaction.

Procedures

The Web-based training sample received instructions about how to access the Web site and then all subsequent training procedures occurred through the Web interface. Participants first completed the Knowledge Questionnaire, then completed the STAT Tutorial at their own pace, and then completed the Knowledge Questionnaire again. Following completion of the post-test Knowledge Questionnaire, the correct answers and their justifications were provided to trainees, so that the assessment would serve a teaching function as well as an evaluative one. Knowledge Questionnaire results were automatically scored, stored in a database, and sent to the first author via e-mail. A score of 80% correct or better was decided a priori to be a passing score. Following completion of the post-test Knowledge Questionnaire, trainees completed the paper-and-pencil Feasibility Questionnaire and mailed it back to the first author. The workshop training sample completed a hard copy of the Knowledge Questionnaire before and after the STAT workshop. This study was reviewed by the Allendale Institutional Review Board, and participants in both the Web-based and workshop training formats were aware that their responses would be used for the purpose of program evaluation.

Results

The average time to complete the tutorial was 3.17 h (SD=0.89; range: 2–5 h). About 70% of the participants completed the tutorial in 3 h or less. Scores on the Knowledge Questionnaire showed significant improvement after completing the STAT tutorial, from a mean of 15.7 (SD=2.2) at pretest to a mean of 21.4 (SD=1.8) at post-test (t(29)=11.5, p<0.001). The average improvement was 5.7 points (SD=2.7). At pretest, only one person (3%) scored at or above 80% (our a priori cutoff for a “pass”), compared with 22 (73%) at post-test (χ²(1)=31.09, p<0.001). This improvement is comparable to that demonstrated by the reference sample of STAT workshop attendees, whose scores increased from a pretest mean of 17.1 (SD=3.2) to a post-test mean of 20.3 (SD=2.6). The average improvement from pre- to post-test was 3.2 points (SD=2.4). At pretest, 6 trainees (22%) scored at or above 80%, compared with 18 (66%) at post-test.

The mean score on the Feasibility Questionnaire completed only by the tutorial participants was 46.2 (SD=3.8), indicating that participants enjoyed taking the tutorial, thought it was well organized, relevant, interesting, and useful, thought the time allotted to each section was about right, and felt it was easy to understand and navigate (Table 1).

Table 1.

Results of Feasibility Questionnaire

	STRONGLY DISAGREE	DISAGREE	AGREE	STRONGLY AGREE
The objectives of the tutorial were stated clearly			6 (20%)	24 (80%)
The tutorial was well organized and easy to follow			1 (3%)	29 (97%)
The tutorial was comprehensive and covered all the important information		1 (3%)	7 (23%)	22 (73%)
Information was presented in an interesting manner			11 (37%)	19 (63%)
The video examples were helpful in illustrating the scoring of the STAT			6 (20%)	24 (80%)
The time allotted to cover each STAT item was about the right length		1 (3%)	15 (52%)	13 (45%)
The time allotted for the general guidelines was about the right length		1 (3%)	12 (41%)	16 (55%)
I feel capable administering the STAT		5 (17%)	12 (40%)	13 (43%)
I feel capable of scoring each STAT item		3 (10%)	18 (60%)	9 (30%)
I feel capable of interpreting and explaining STAT results to a child's parents		4 (13%)	14 (47%)	12 (40%)
It was easy to understand how to start and navigate through the CD tutorial		2 (7%)	8 (27%)	20 (67%)
The CD tutorial is an effective tool for learning the STAT across ethnic groups		1 (3%)	15 (50%)	14 (47%)

STAT, Screening Tool for Autism in Toddlers and Young Children.

Discussion

Results from this pilot study demonstrate that a Web-based training format can be effective for increasing knowledge about early autism screening among community providers with diverse levels of education and experience. Because the STAT involves observation of key behavioral features of young children with autism, this training tutorial may serve not only to identify children's risk status, but also to enhance recognition of early behavioral features of autism and lead to earlier referral and treatment. The dearth of trained autism experts, and the long waits for diagnostic evaluations, makes the identification of signs and symptoms by community professionals who are not necessarily autism experts a critical factor in facilitating early identification of this disorder.

Lack of randomization to the different training formats limits inferences that can be made between the training formats. With that caveat, however, exploratory comparisons can be made in terms of the types of gains to be expected in an online format, compared with those found in face-to-face training. After training, both groups had similar scores on the post-test and similar rates of passing. Mean improvement was slightly higher in the online training group, largely a function of slightly lower pretest scores. Time to complete the online tutorial was about 3 h, substantially less than the face-to-face training, which is typically a full day. These positive results suggest that further research beyond this pilot study is warranted, using a fully randomized design to more effectively compare online to face-to-face training.

Web-based e-learning is rapidly becoming an effective tool for mental health education and training. Similar online programs have been shown to be effective in training clinicians on the identification and assessment of the symptoms of other disorders, such as depression²⁶ and schizophrenia.²⁷ However, the extent to which increased knowledge resulting from this training tutorial will translate into clinical competence in administering and interpreting the STAT in clinical or community settings is not yet known. Several studies have shown that didactic and applied skills are independent skill sets and that a didactic knowledge of how to administer a measure does not necessarily imply clinical competence in its administration.^28,29 Thus, future work will be needed to determine the reliability with which the STAT is administered, scored, and interpreted within community settings.

One promising new approach in telemedicine is augmenting Web-based didactic training with videoconferencing, so that the trainee can be observed in real time applying the conceptual knowledge learned in the Web-based training into actual clinical skills. This two-stage training approach of combining Web-based tutorials with live remote observation via videoconference has been successfully used in training clinicians on the assessment of other psychiatric disorders,^26,27 and combining face-to-face didactic training with remote observation via videoconference has been recently reported in parent training for autism.³⁰ This combined approach has the potential to augment future Web-based training with the STAT and will help ensure that the training results in clinical competence in the use of the scale in community settings.

Acknowledgments

This project has been funded in whole or in part by Federal Funds from the National Institute of Mental Health, National Institutes of Health, Department of Health and Human Services, under Contract No. N44MH54080 (PI: K.A. Kobak).

Disclosure Statement

Drs. W.L. Stone and O. Ousley have a commercial interest in the STAT. No other conflicts of interest exist.