Recent Developments in Treatment Outcome Measures for Young Children with Autism Spectrum Disorder (ASD)

Abstract

Significant advancements have been made in early intervention programs for children with ASD. However, measuring treatment response for children with ASD is difficult due to the heterogeneity of changes in symptoms, which can be subtle, especially over a short period of time. Here we outline the challenge of evaluating treatment response with currently available measures as well as newly developed or refined measures that may be useful in clinical trials for young children with ASD. Continued development of treatment outcome measures will help the field identify and compare efficacious interventions and tailor treatments for children with ASD.

Background

Over the last decade, researchers have focused significant efforts on the development of novel interventions for young children with Autism Spectrum Disorder (ASD).^1–3 While advancements continue for a variety of behavioral and pharmacological treatments for ASD,^4,5 the field is still in a great need of treatment outcome measures that adequately quantify improvements in child behavior that could assist with determining the efficacy of short-term interventions.^6,7 The aim of this work is to highlight the challenges associated with measuring treatment response in ASD and discuss how the field is identifying treatment response measures appropriate for young children with, or at elevated likelihood for, ASD.

The Challenge of Measuring Treatment Response in ASD

Treatments for young children with ASD, or who are at elevated likelihood of developing ASD, often focus on ameliorating deficits in the social communication domain, a key symptom in ASD.⁸ Naturalistic Developmental Behavioral Interventions (NDBIs)⁹ often use a caregiver-mediated model with an emphasis on child engagement and facilitation of child social communication (e.g., creating situations in which the child is motivated to communicate or engage and teaching the child how to do so). Examples of NDBIs include the Early Start Denver Model (ESDM),¹⁰ Joint Attention, Symbolic Play, Engagement and Regulation (JASPER),¹¹ Pivotal Response Treatment (PRT),¹² Early Social Interaction Project (ESI),¹³ and the Preschool Autism Communication Trial (PACT).¹⁴ Applied behavior analysis (ABA) techniques, which have a strong evidence base,^3,8 are often embedded into NDBI teaching strategies. Despite the strong evidence for these intervention practices, children with ASD often vary in their response to intervention, highlighting the need for measures that can identify children who are responders to treatment and those who are not.^15–18

While ameliorating the challenges associated with limited social communication skills is an important pursuit, the social communication behaviors that interventionists aim to improve are often subtle, making them difficult to quantify, particularly over a relatively short period of time.^6,7,19,20 Measuring social communication change is especially challenging due to the variability of changes seen across children and interventions.¹⁷ For example, some changes in social communication behaviors may relate to the frequency with which a particular behavior is engaged in while other changes may relate more to the quality of a behavior.^14,18,21 Similarly, interventions vary with regards to their foci, with some interventions attempting to impact behaviors across a range of ASD-like symptoms while others focus more specifically on a particular social communication skill (e.g., joint attention).^10,22 Developing measures that can quantify subtle, though clinically meaningful, changes across a full range of social communication skills, including frequency and quality of a behavior, is essential, yet challenging.⁶

The scale of this challenge was discussed in a review article that highlighted the lack of consistency in measures used across studies to quantify changes in social communication.²³ Many measures used do not have the flexibility or standardization for use across different types of interventions. Specifically, Bolte and Diehl (2013) found that of the 195 intervention trials reviewed, over 200 different measures were used in an attempt to quantify treatment-associated changes in child behavior. The majority of measures used were study-specific, while only three measures were used in more than 2% of intervention studies²³: the Aberrant Behavior Checklist,²⁴ the Clinical Global Impression Scale²⁵ and the Vineland Adaptive Behavior Scales.²⁶ With the large number of measures being used, comparing the efficacy of different interventions is a difficult, yet essential first step toward identifying effective treatments and ultimately tailoring interventions.

The Limitations of Currently Available Outcome Measures

1.). Limited Validity as Outcome Measures.

Many measures used have been criticized as having little validity as treatment response measures.⁷ For example, in a thorough review of the treatment response measures used in ASD, none of the three “most commonly” used measures (the ABC, CGI, and VABS)²³ nor any other measures, were recommended for use to track treatment related changes because many have not been effective at capturing change in response to treatment, were not developed for that purpose, and have significant implementation limitations (e.g., very limited age range or extensive training required to use).⁷ In contrast, in a separate expert panel review, six measures, including the ABC and the VABS, were deemed satisfactory as treatment response measures, although some limitations were noted.⁶ For instance, the ABC does not assess non-verbal communication skills or the quality of a child’s social interactions (see below for more details). The VABS has not always been responsive to changes over time, particularly if an intervention is shorter than six months.²⁷ Additional measures that were deemed appropriate for use were the Behavior Assessment System for Children (BASC-2),²⁸ the Communication and Symbolic Behavior Scales (CSBS),²⁹ the Early Social Communication Scales (ESCS),³⁰ and the Social Skills Improvement System (SSIS).³¹

2.). Few Measures Quantify Changes in ASD-specific Symptoms.

Many commonly used treatment outcome measures, such as the ABC and the VABS, are not intended to quantify ASD symptoms, though most ASD interventions target ASD specific symptoms.^10,11,13,14 For example, the ABC was developed to measure general child psychopathology in five areas: irritability/agitation, lethargy/social withdrawal, stereotypic behavior, hyperactivity, and inappropriate speech.²⁴ The VABS is a measure of global adaptive functioning across four domains: communication, daily living, socialization, and motor skills.³² While many children with ASD will display challenges in the areas measured by the ABC and VABS, and improvements in these areas in response to treatment are clinically relevant, understanding whether core symptoms of ASD, such as social communication deficits and repetitive behaviors, are effectively impacted by treatments requires measurement of ASD-specific symptoms.

In addition to the ABC and VABS, researchers also use measures of general cognitive ability that, in some cases, are not the core target of intervention.³³ For example, use of a pre and post-treatment cognitive assessment, such as the Mullen Scales of Early Learning (MSEL),³⁴ is common.^21,35–37 Several studies have noted significant improvements in general cognitive abilities or specific cognitive domains (e.g., receptive or expressive language),³⁵ suggesting that ASD-specific treatment targets may have downstream impacts on more global cognitive skills. ^10,38,39 In fact, cognitive measures like the MSEL may help with identifying groups of children who are more likely to show improvements in response to intervention.^18,40 In a recent trial employing ESDM, children who began intervention at a younger age and who had less severe language delay demonstrated greater gains in verbal cognitive functioning.¹⁸ While gains in cognitive skills may not always be present based on standard scores, clinical improvements may be noted based on raw scores or age equivalents, demonstrating specific skill acquisition over the course of intervention.

However, because the MSEL is useful in examining age-appropriate development in children, in the case of early intervention, improvements on the MSEL may be an indicator of developmental maturity rather than specific treatment effect. Though it may be difficult to clearly separate ASD symptom change from general development in language and cognitive abilities, given the inherent overlap among these developmental domains and social communication, understanding a treatment’s ability to improve core ASD symptoms that are targeted in intervention is necessary.

To quantify change in ASD specific symptoms, the Autism Diagnostic Observation Schedule (ADOS)⁴¹ has been used in many studies, though the ADOS was developed as a diagnostic measure and is not designed for treatment response. Researchers have used both the ADOS raw scores as well as the Calibrated Severity Scores (CSS)⁴², which were intended to be more sensitive to changes over time. However, raw scores were usually not useful in identifying treatment related changes, and when changes were observed, these changes were difficult to interpret since changes were observed across all children and in control groups.^14,43,44 The ADOS CSS does seem to be useful in quantifying changes over the course of years, but most intervention trials are six months or less-- too brief for the ADOS CSS to capture changes.^45–47 Given these limitations, a recent review recommended against using ASD diagnostic measures, like the ADOS, as outcome measures, though use of the ADOS was previously encouraged for this purpose.^6,19

3.). Potential Bias with Caregiver, Teacher, or Clinician Report.

Many measures that are currently available, such as caregiver reports of adaptive functioning (e.g., ABAS, VABS) as well as clinician ratings of treatment response (e.g., Clinical Global Impression; CGI),⁴⁸ can accentuate placebo effects, above and beyond more subtle changes that might happen in response to intervention.^5,43 The CGI is one of the most commonly used outcome measures,²³ particularly in medication trials.^5,49 The CGI, which relies on the report of the treating clinician, is similar to the ABC and VABS, which rely on the report of the caregiver. Since clinicians and caregivers are often actively involved in the treatment being provided or are at least aware of intervention group, biased results may be particularly strong. For example, when using caregiver-rated measures, changes observed in a child over time may be more related to the caregiver’s perception of whether the child is receiving treatment than to the treatment itself, greatly limiting one’s ability to understand the true utility of an intervention.⁵

Given this limitation, one currently available observation-based measure was deemed by the expert panel to be adequate for use as a treatment response measure: the Early Social Communication Scale (ESCS).^6,30 The ESCS measures verbal and non-verbal social skills in toddlers during a structured play setting involving a variety of toys and several opportunities for the toddlers to initiate and reciprocate interaction with the examiner.³⁰ However, the use of the ESCS is limited to children between 8 and 30 months of age and requires substantial training to be able to use reliably.

While NDBIs often focus on improving deficits in social communication, researchers also believe that these social communication changes may be related to changes in other ASD-related behaviors, such as RRBs. To examine changes in RRBs, particularly in medication trials, researchers have used the Repetitive Behavior Scales, Revised (RBS-R)⁵⁰ as an outcome measure.^51–53The RBS-R, a caregiver-report measure, captures behaviors in six domains: stereotyped behavior, self-injurious behavior, compulsive behavior, routine behavior, sameness behavior, and restricted behavior. The RBS-R is appropriate for six to seventeen-year-old children, but not validated for use with younger children. Though the aforementioned limitations associated with caregiver-report measures exist, measuring RRBs based on direct observation may be challenging given the limited frequency, timing, and context-specificity of many of these behaviors.^54–56 A recent study found that the RBS-R was sensitive to changes in response to Pivotal Response Training intervention in school-age children with ASD.⁵⁷ Recently, the RBS-Early Childhood (RBS-EC)⁵⁸ has been developed specifically for children younger than 5 years.⁵⁸ However, no studies have been published examining changes in RRBs based on the RBS-EC. Since RRBs are a core component of ASD, understanding the utility of NDBIs for reducing the impairment associated with RRBs is important, especially for younger children, though objective treatment response measures are limited.⁵⁹

Developments in Quantifying Treatment Response

When choosing an appropriate treatment outcome measure, researchers will need to consider several factors: 1) not all children will respond to intervention in the same way;¹⁷ 2) no measurement tool is comprehensive enough to capture all possible changes; 3) some children may display stable or worsening behaviors over time despite receiving intervention; and 4) measurement tools may miss some children who truly change and identify some children who truly don’t, and the distinction between these two groups may not be readily apparent. Here we highlight some recommended, newly developed, or revised measures that may be useful in some studies to quantify treatment response in toddlers and preschoolers with ASD.

Adaptive Behavior.

ABAS-3.

The Adaptive Behavior Assessment System, Third Edition (ABAS-3)⁶⁰ is a caregiver report assessing three different domains of adaptive skills: conceptual, practical and social skills. Responses are recorded on a Likert scale ranging from 0 (skill not attained) to 4 (consistently displays skill). A summary of adaptive functioning across these domains is represented by the General Adaptive Composite standard score (GAC). The second edition of the ABAS (ABAS-2)⁶¹ has been used to assess adaptive functioning among young children and adolescents with ASD.^62–65 The ABAS-2 has also been used as an outcome measure in children diagnosed with ASD, finding that children without intellectual disability displayed the largest improvements in adaptive behavior over the course of intervention.⁶⁶ The ABAS-2 has also been used to assess the concurrent validity of different play measures for preschoolers with ASD and to assess socialization skill improvement after a theatre-based intervention for adolescents with ASD.^67,68 Given its recent availability, the third edition of the ABAS (ABAS-3) has yet to be used in ASD intervention trials. Future research should focus on whether the ABAS-3 may be a useful measure of adaptive skills that can capture changes in these behaviors over relatively short periods of time in young children with ASD. Its utility as a treatment response measure, particularly in comparison to the VABS (see below), will help researchers to select appropriate measures for use in intervention trials.

VABS-3.

The Vineland Adaptive Behavior Scale, Third Edition, is a measure used to assess a child’s level of adaptive functioning (VABS-3),³² similar to the ABAS-3. It contains four different subdomains: socialization, daily living skills, communication, and motor skills. The Adaptive Behavior Composite (ABC) provides a picture of a child’s general level of adaptive functioning. Relative strengths and weaknesses in a child’s skill profile are assessed via V-scores within subdomains. The VABS-3 is available as a caregiver or teacher interview as well as caregiver or teacher report forms. Across forms, caregivers or teachers rate behaviors that a child is able to complete with little help or prompting, with “Often”, “Sometimes”, or “Never” answer choices, allowing for reports on emerging skills (“Sometimes”). The VABS-3 has included a comparative ASD sample in validity studies, unlike the second edition (VABS-II), a strength of this new edition.³²

While studies using the third edition of the VABS as an outcome measure have yet to be published, the VABS-II has been used as an outcome measure in a number of research studies examining early intervention programs with adaptive skills being a primary or secondary outcome.^18,35–39 Trials that have used the VABS-II varied in length, ranging from 12 weeks to one year within varied group-based and caregiver-mediated settings highlighting the VABS’s utility across a variety of early intervention models.^18,35–39 Some studies have found measuring changes in raw scores and age equivalents useful, which provides information about children who are showing progress in skills but who may not be at their target for chronological age.³⁵ In contrast, some researchers note its lack of sensitivity in comparison to more proximal measures, such as the direct assessments of video coded variables such as child vocalizations during interactions with a caregiver.³⁷ Given the global nature of the behaviors measured by the VABS, longer intervention periods may be necessary to be able to quantify meaningful changes.²⁷ While these studies highlight both the potential utility as well as the limitations of the VABS, more research examining the VABS-3 is necessary to truly evaluate its validity as an outcome measure, particularly given the new ASD comparison sample used for development of the measure’s updated edition.

ASD Symptoms.

AIM.

The Autism Impact Measure (AIM)⁶⁹ is a newly-developed treatment outcome measure specific to ASD symptoms that is appropriate for children 2 to 17 years old. The AIM is a 41-item caregiver questionnaire that considers behavior over the last two weeks and is rated on a 5-point Likert scale capturing both the frequency of a behavior (“never” to “always”) as well as the behavior’s functional impact (“not at all” to “severely”).⁶⁹ The AIM items were developed based on expert review of items from several well-validated measures of ASD symptoms, including the Autism Diagnostic Interview (ADI-R),⁷⁰ the Gilliam Autism Rating Scale (GARS)⁷¹, the Social Communication Questionnaire (SCQ),⁷² and the Social Responsiveness Scale (SRS).^69,73 Caregivers of children with ASD were also consulted during the item development phase.

The initial psychometric properties of the AIM were tested in a sample of over 400 children with ASD ranging from 2 to 17 years old across a multisite study in the United States. A four-factor model was well-supported consisting of domains in the areas of repetitive behavior, atypical behavior, communication/language, and social/emotional reciprocity.⁶⁹ Test-retest reliability (within eight days) ranged from 0.53 to 0.85 and cross-informant reliability ranged from 0.46 to 0.73.⁶⁹ The AIM was significantly correlated with the SCQ and several domains of the VABS suggesting good convergent validity with other standard measures of ASD symptoms and adaptive behaviors.⁶⁹ A follow-up study suggested that the factor structure of the AIM is best represented by five factors, not four: repetitive behavior, atypical behavior, communication, social reciprocity, and peer interaction.⁷⁴ This second study also added evidence for convergent validity; the AIM showed significant correlations with measures of ASD symptoms including the RBS-R, ABC, and ADOS-2.

In combination with the strong initial psychometric properties, the AIM has promise as an effective treatment response measure for children with ASD. So far, the AIM has been used as an outcome measure in two pharmacological trials with mixed results.^75,76 One trial did not find a significant amount of change on the AIM⁷⁵ while the other found significant changes on the atypical behavior domain of the AIM⁷⁶. Yet, whether the AIM can detect changes in ASD symptoms over time or in response to intervention, and whether these changes are clinically meaningful, remains to be explored beyond pharmacological trials.^75,76 In addition, the reliance on caregiver report, while having significant advantages, may also yield less objective results.

BOSCC.

The Brief Observation of Social Communication Change (BOSCC)⁵⁵ is a newly-developed treatment response measure of ASD symptoms. In order to address many of the concerns regarding currently used outcome measures, the BOSCC was developed to be observation-based, specific to core ASD symptoms (originally developed based on codes from the ADOS), and able to capture subtle changes in ASD symptoms over a relatively short period of time (e.g., 6 months or less).⁵⁵ A recent commentary highlights the benefits and limitations of the BOSCC.⁷⁷ The BOSCC is a coding scheme that is appropriate for minimally-verbal toddlers or preschoolers and can be flexibly applied to videos of caregiver-child interactions,⁵⁷ clinician or researcher-child interactions, segments of ADOS videos,⁷⁸ or daily routines,⁵⁶ so long as the context is consistent over the course of treatment.

The coding scheme consists of 15–16 items (depending on the context in which it is applied) and yields sub-domain totals in social communication and restricted/repetitive behaviors (RRBs). Summing these sub-domains provides a Core Total. The BOSCC is able to be coded by relatively naïve trainees, such as baccalaureate-level research assistants who have established reliability with a lead coder at a site who is familiar with ASD and preferably the ADOS. The BOSCC typically takes 30–45 minutes to code a 10 to 12-minute video and coders typically reach reliability standards within 12–15 videos. Another benefit of the BOSCC is that it can be both administered and coded by researchers who are unaware of the child’s treatment status or timepoint in treatment, minimizing the bias associated with caregiver or clinician report.

The initial psychometric properties of the BOSCC have been established, confirming the factor structure of the measure and indicating that the BOSCC has strong inter-rater reliability (Intraclass Correlation Coefficients; ICCs ranging from 0.88 to 0.98 across domains) and test-retest reliability (ICCs ranging from 0.79 to 0.90 across domains).^55,78 Initial validity analyses have confirmed that the BOSCC can capture significant amounts of change over the course of high-intensity intervention trials that range from about 6–8 months.^55,78 Of importance, while the BOSCC contains a separate RRB domain, the items in this domain were skewed (often not observed) so the chances of capturing meaningful change in this domain may be limited though applying the BOSCC to the ADOS context may assist with this limitation.^55,78 In addition, the test-retest of the RRB domain, though still adequate (ICC=0.79), was lower than in the other domains, suggesting that these behaviors may not be stably measured by the BOSCC over time.⁵⁵

Since the initial publication of the BOSCC, several additional studies have been published from researchers trialing its use across a range of interventions.^{22,56,79–82} The results have been mixed, highlighting that the BOSCC may not capture changes in every intervention. For example, in intervention trials that track change over the course of one year, the BOSCC has been successful at capturing a significant amount of change with a medium effect size.^56,80 In contrast, in interventions that have been low in intensity and/or short in duration (8–10 weeks), the BOSCC has not been successful in capturing a significant amount of change over time.^22,81 These studies, in combination with the initial psychometric studies, highlight that the BOSCC may be a feasible and useful measure to capture treatment response in intervention trials of young children with ASD, though more research is needed to truly evaluate its feasibility and utility in capturing intervention-related changes.

CSBS-DP-BS.

The Communication and Symbolic Behavior Scales Developmental Profile Behavior Sample (CSBS-DP-BS),²⁹ recommended for use by an expert panel,⁶ measures social communication behaviors in the context of a coded video of a structured play session. During the session, the clinician presents different play materials and provides opportunities to elicit social communication behaviors. It typically takes 50–75 minutes to complete and is widely used by speech pathologists and psychologists to assess levels of expressive language and social communication skills.^6,83 It can be used in children between the ages of 6 months and 2 years, and in children up to 6 years old with a developmental level of 24 months. The CSBS-DP-BS has three domains measuring social affect, communication, and symbolic behavior, as well as seven cluster scores measuring a range of social and nonverbal communication behaviors.

The CSBS-DP-BS has been used as a screener for ASD and has shown utility in tracking changes in social communication behaviors for children at elevated risk for developing ASD.^35,84–86 The CSBS-DP-BS has proven to be a useful outcome measure in a caregiver-mediated, low-intensity, 9 month intervention,³⁵ though it was less sensitive to changes in lower intensity interventions over the course of 6 months.^87–89 The CSBS-DP-BS has also been used in short-term pharmacological studies. Though the CSBS-DP-BS did not capture significant amounts of change in these studies, neither did other standard measures, leaving the true efficacy the CSBS-DP-BS unanswered^90,91 Though it has been recommended as a potentially effective outcome measure, the CSBS-DP-BS is less frequently used than other measures,^7,92 potentially due to the length of time required to administer as well as the extensive training required to use reliably.²⁹ Regardless, a tool such as the CSBS-DP-BS can serve as a useful, objective measure of social communication skills across some intervention contexts and thus merits more research into its utility as an intervention outcome measure.

ESCS.

Also recommended for use by an expert panel,⁶ the Early Social Communication Scales (ESCS)³⁰ is a video-taped observational measure assessing the frequency of social initiations and responses across different play activities in children between 8 and 30 months of age. Coders detect the frequency of communication behaviors, such as joint attention (e.g., directing the examiner’s attention to a toy), social interaction (e.g., eye contact and turn taking), and requests (e.g., using gestures to request for objects).³⁰ The ESCS typically takes 15–25 minutes to complete and involves the examiner sitting across from the child and presenting different objects with a hierarchy of probes in order to assess behaviors of interest.

Early intervention studies have demonstrated mixed effects of the ESCS’s sensitivity to change.^11,93–98 This inconsistency indicates that there may be certain types of interventions that are best suited to use the ESCS as an outcome measure. For example, minimal changes were identified in the context of trainee-mediated or teacher-mediated interventions that are of low intensity.^97–99 While this contradicts the effects seen in other studies, it suggests that the ESCS may be most effective in more intensive interventions conducted by highly-trained clinicians.^11,93,96 In the context of community-implemented interventions administered by less trained staff, some researchers have suggested that an adaptation of the ESCS, the Short Play and Communication Evaluation (SPACE), be used.¹⁰⁰ The SPACE also draws from the CSBS-DP-BS and the Structured Play Assessment (SPA)¹⁰¹ to measure discrete social communication behaviors that may improve over the course of an intervention.¹⁰⁰ The SPACE may be more feasible as it requires less training and has stronger ecological validity given its ability to be conducted in non-experimental settings (e.g., the classroom where intervention occurs). The SPACE’s utility as a treatment response measure, though perhaps more feasible than the ESCS, remains unknown.

In addition, the ESCS may be particularly helpful in capturing changes in specific skills such as joint attention.^81,94,102 This suggests that the use of the Joint Attention Measure from the ESCS (ESCS-JAMES)³⁰ may be most advantageous for many intervention trials for ASD.⁶ While the ESCS-JAMES has had minimal use in ASD intervention trials, it has been used successfully in one pharmacological study and in a study examining longitudinal trajectories of joint attention skills following a joint-attention focused early intervention.^103,104

Social Skills and Problem Behaviors.

BASC-3.

The Behavior Assessment for Children, Third Edition, as with previous editions, was designed to identify symptoms of behavioral disorders rather than specific to treatment-related changes (BASC-3).¹⁰⁵ The newly revised BASC-3 includes a number of improvements, including a Clinical Index and Executive Functioning Index, and can be administered and scored electronically.¹⁰⁵ However, while outcome studies using the BASC-3 have yet to be published, the Behavior Assessment System for Children, Second Edition (BASC-2) has been recommended by an expert panel review to assess intervention outcome.^6,28 The BASC-2 is an assessment scale that examines clinical and adaptive skills in individuals aged 2–25 years of age, based on caregiver, teacher and self-report. Responses are recorded on a 0–4 Likert scale, examining the frequency of different behaviors. The test-retest reliability of the BASC-2 ranges from ICC of 0.74 to 0.88, and its concurrent validity with other measures addressing ADHD, adaptive skills, emotional difficulties and executive function is also excellent.²⁸ For pre-school aged children, the BASC-2 produces four different composite scores from the caregiver and teacher report scale: internalizing behavior, externalizing behavior, adaptive skills, and behavioral skills index. An additional subscale, the developmental social disorders subscale, is designed to be sensitive to symptoms of developmental disorders and has been shown to serve as a comparable screening tool to detect preschoolers with ASD.¹⁰⁶ However, recent studies find specificity of the subscale was limited when distinguishing between samples of preschoolers with ASD versus ID.¹⁰⁷ BASC-2 scores have also been compared between young children with and without ASD with significant differences in scores observed across several subscales.^108,109 In regards to its utility as an outcome measure, intervention studies using the BASC-2 have typically included older children and adolescents rather than preschoolers,^110–113 highlighting the need for additional studies examining its utility in intervention trials for young children. Nevertheless, with its high level of reliability and validity and routine use among ASD populations, the BASC-2 may be a favorable outcome measure.

SSiS.

The Social Skills Improvement System (SSiS)³¹ is a caregiver, teacher and self-report questionnaire assessing the presence and frequency of social skills and deficits in children across a variety of contexts, from 3 to 18 years of age.¹¹⁴ Caregiver and teacher report begin at 3 years of age, and self-report measures can be used from 8 years old.³¹ Originally named the Social Skills Rating System (SSRS),¹¹⁵ research in populations of children with ASD and other developmental disorders led to a revision and renaming to the SSiS. The SSiS has two major scales: social skills and problem behaviors. Respondents report the frequency of behaviors on a Likert Scale from “Almost Never” to “Always” (ie., 0–4) as well as on the importance of these behaviors to the respondent in regards to the child’s social and academic functioning also on a Likert Scale from “Not Important” to “Critical” (i.e., 0–3).³¹

Though recommended for use by an expert panel,⁶ the SSiS has mainly been used as an outcome measure in populations of school-age children and adolescents with ASD, rather than toddlers and preschoolers.^116,117 In non-ASD populations, the SSiS has mostly been used with preschoolers with other neurodevelopmental disorders, such as ADHD.¹¹⁸ For preschoolers with ASD, the SSiS has been used to assess the outcome of a pre-school intervention and to assess changes in social skills of preschoolers as they transition to kindergarten.^119,120 As a measure designed to be sensitive across a broad range of contexts and behaviors, the SSIS merits more research in its utility as an outcome measure for young children with ASD.

Cognitive Functioning.

MSEL.

The Mullen Scales of Early Learning (MSEL)³⁴ is a clinician-administered measure of cognitive functioning across five domains: gross motor skills, receptive and expressive language skills, fine motor skills and visual reception skills.³⁴ Scoring is standardized and based on whether a child can perform the skill required to complete tasks.³⁴ The MSEL informs treatment planning and early intervention eligibility as it examines both strengths and weaknesses across different domains and allows for comparison of a child’s abilities to typically developing children of the same age.³⁴ A recent analysis of its concurrent validity in a sample of toddlers with ASD has determined that the MSEL is a valid measurement of relative constructs in the assessed domains for this population.¹²¹ Additionally, several recent studies have shown the MSEL’s convergent validity in determining cognitive ability with the Differential Ability Scales (DAS)^122,123, and also its validity in determining cognitive ability in different clinical populations including ASD.^124,125 Studies have found that even though toddlers with ASD display less time actively engaged in the MSEL, which may affect its validity, the Receptive and Expressive Language scores on the MSEL scores correlated positively with related domains on the VABS.¹²⁶ Extensive research into the use of the MSEL further supports the use of its age-equivalence scores, rather than T-scores, to better understand IQ estimates in young children with ASD.^124,126

For children with ASD, the MSEL has been used as a primary and secondary outcome measure to assess gains in skill acquisition via interventions lasting from 12 weeks to more than 12 months with mixed results.^{18,27,35–39,127–130} The MSEL may be useful in capturing change even in shorter interventions of a lower intensity,^27,39 though this result is not consistent. Across studies, results suggest that children at younger ages show greater amounts of change in their MSEL scores, pointing to the need to implement intervention as early as possible in toddlers diagnosed with, or with suspected diagnoses of, ASD.^18,27,130 Differences in MSEL baseline scores also predict specific gains across different domains measured by the MSEL or in other measures, such as adaptive skills, though these results are also variable between studies.^129,130 Recently, Paynter et al. (2018) reported that children in community samples with higher MSEL scores at baseline were among the group of children who displayed the most change in response to intervention, regardless of age.¹²⁸ These studies indicate that both baseline cognitive abilities as well as age may provide information about whether and how a child will respond to intervention. Researchers should also consider that any changes observed on the MSEL may be due to a general development over the course of treatment rather than effects of an intervention. Utilization of control groups that have MSEL comparisons over the same time may help to separate the effects of maturation from the effects of intervention.

Bayley-III.

The Bayley Scales of Infant Development, Third Edition, (Bayley-III)¹³¹ were originally formulated as a neurodevelopmental assessment similar to the MSEL for infants and toddlers between 1 and 42 months, including scales to assess development across Motor, Socio-Emotional, Adaptive, Cognitive and Language domains.

Unlike the MSEL, the Bayley-III does not compute an overall composite score, but rather scaled scores, and percentile ranks for each individual scale, resulting in a descriptive category of the child’s development compared to population norms of a large U.S. sample.^131,132 Reliability coefficients of the Cognitive, Language and Motor subscales range from 0.86–0.91.¹³¹ Test-retest reliability studies conducted in these same subscales in infants aged 2–4 months ranged from 0.67–0.84, and in children aged 33–42 months coefficients ranged from 0.83 to 0.94.^131,132 Convergent Validity studies of the Bayley-III have included the Wechsler Preschool and Primary Scale of Intelligence, Third Edition,¹³³ The Preschool Language Scale, Fourth Edition,¹³⁴ and the Peabody Developmental Motor Skills, Second Edition with high to moderate correlations.^131,132,135 Special groups, such as those with developmental delays, have also been included as comparative standardization samples during the development of the Bayley-III. A sample of 70 children diagnosed with pervasive developmental disorder (PDD) were included as a group, with lower cognitive scores and language scores in comparison to matched controls.¹³¹

The Bayley is routinely used in preterm infant prospective studies to examine its ability to predict neurodevelopmental delays and is used within neonatal intensive care unit (NICU) follow up programs to assess for developmental delays.¹³⁶ In the context of toddlers with ASD, the Bayley has been used as a predictive measure of cognitive and language outcomes in toddlers diagnosed with ASD in Spain and has also been used to study cognitive and language skill profiles among children with and without ASD.^137,138 The Socio-Emotional subscale, adapted from the Greenspan Social-Emotional Growth Chart,¹³⁹ was used as a pre-treatment and follow up measure in an imitation intervention study, showing some promise as sensitive to changes in intervention.⁹³

Some criticisms of the Bayley include potentially inflated norm scores due to the inclusion of children with various clinical difficulties included in the standardization sample.¹⁴⁰ The upcoming Bayley-IV will feature updated norm samples.^141,142 Other important updates include the Adaptive Behavior domain adapted from the Vineland-3 instead of the older ABAS-II, and updated questions and content.¹⁴² The Bayley’s use among infants and toddlers with, or at elevated risk for, ASD and as a treatment response measure merits more research.

Discussion

In order for ASD intervention research to maximally progress, the field must identify feasible, reliable, and accurate measures that quantify meaningful changes in response to treatment. The heterogeneous nature of ASD as well as the variable outcomes observed, even in evidence-based interventions, highlight this need.^16–18 While it is unlikely that any one measure will adequately capture meaningful changes across all interventions in all children, some measures reviewed here have shown potential as effective treatment outcome measures across various clinical trials.

Most intervention studies for ASD have used outcome measures that have limited validity, do not focus on ASD symptoms, and rely on potentially biased reports from caregivers, teachers, or treating clinicians. Though evidence is still limited with regards to their utility, a few measures can help us address some of these limitations. For instance, the BOSCC was specifically developed as an outcome measure of ASD symptoms and is administered and coded by researchers who are blind to treatment status and time point in order to maximize objectivity. So far, studies using the BOSCC have shown mixed results, suggesting that it may be most effective at capturing changes in interventions that have a broad focus across a range of ASD symptoms and that occur intensively over about six months.^55,78 In addition, the newly-developed AIM may also prove to be a useful measure of treatment response that is specific to ASD symptoms, though its sensitivity to change has yet to be tested. In addition, since the BOSCC and the AIM are new to the field, additional work is needed to determine their ability to quantify clinically meaningful changes. In the meantime, developers of the BOSCC are focusing on expanding the BOSCC to children with a range of language abilities (from phrase to fluent speakers) and various contexts (e.g., clinic vs. home). In addition, to minimize the burden on researchers to get reliable coders of the BOSCC, the developers are working to develop an online coding system. This would allow researchers to upload videos that can be coded by blind, reliable coders.

Though the BOSCC addresses many limitations of previously used measures, it continues to have limitations of its own. The BOSCC was developed to be flexible enough to be used across different social contexts, but the initial study⁵⁵ focused on caregiver-child interactions. In caregiver-mediated interventions in particular, any changes observed in the child cannot be separated from changes in the caregiver or the dyad. Future research should focus on whether there is an optimal context in which the BOSCC should be applied and ways in which to examine how changes in the child might be affected by the changes in caregiver and the caregiver’s implementation of treatment strategies.

In addition, the RRB domain of the BOSCC may not be particularly useful in quantifying changes in RRBs. Overall, few RRBs were observed during the brief BOSCC observations,⁵⁵ suggesting that this may not be the most objective measure of RRB symptoms. It may be best that researchers using the BOSCC who are interested in changes in RRBs, apply the BOSCC coding scheme to the ADOS context as more RRB behaviors seem to be elicited during the ADOS allowing for more opportunity to identify improvements.⁷⁸ While the BOSCC may not be ideal for measuring changes in RRBs, few other observation-based measures that are sensitive to changes in RRBs currently exist, suggesting an area of great need in treatment outcome measures. Alternative observation coding strategies of RRBs have been used by some researchers to provide diagnostic information in young children.^143,144 This method may be useful to quantify changes over time, though has not been tested for this purpose.

Besides the AIM and the BOSCC, the ABAS-3 and the VABS-3 may be useful in capturing global changes in adaptive functioning in response to intervention. The VABS-II has demonstrated its utility as an outcome measure in some studies and, with the new comparative sample of ASD children, the VABS-3 may be particularly useful.^18,32,38,39 However, both the ABAS-3 and the VABS-3 rely on caregiver or teacher report, suggesting that changes observed may be the result of caregiver perception of the child’s participation in treatment rather than the treatment itself. In addition, because the ABAS-3 and VABS-3 are not specific to ASD symptoms, they may be more advantageous in capturing outcomes for interventions that target a range of functional outcomes, particularly those that occur over a longer period of time. Because the ABAS-3 and VABS-3 are new editions, they have not been used in many studies, and thus future research is required to yield important information about the use of these measures to capture treatment response.

In addition to the ABAS-3 and the VABS-3, several other measures have been suggested as useful outcome measures, including the ABC, the BASC-2, the CSBS-DP-BS, the ESCS, and the SSiS.⁶ Quick questionnaires, like the ABC and BASC-2, may not add substantial burden to researchers or families, making them more universally feasible across a range of studies. While the field continues to identify useful outcome measures, use of the ABC and/or the BASC-3 may contribute to our understanding of these measures with few drawbacks. With the exception of the CSBS-DP-BS and the ESCS, other measures, like the ABAS-3 and VABS-3, are still limited by their reliance on caregiver or teacher report. In addition, given that the ABC and the BASC-2 are not appropriate for children under 6 and 3 years of age, respectively, the utility of these measures in early intervention trials for ASD is still limited.

The CSBS-DP-BS and the ESCS are both appropriate for children as young as eight months, and thus can be potentially useful in measuring treatment outcomes for very early intervention trials. However, since both of these measures were not developed specifically to quantify treatment-related changes, further research is needed to determine whether they are sensitive to subtle changes in ASD symptoms. In addition, both of these measures require substantial training in order to administer and score reliably, potentially limiting their utility across studies. While the ESCS takes less than 30 minutes to administer, the CSBS-DP-BS can take up to 75 minutes, a potential limitation for some studies. The SSiS may prove useful as an outcome measure, particularly in school-based interventions, though few studies have used it so far with young children with ASD. For the CSBS-DP-BS, ESCS, and the SSiS, more research is needed to examine their ability to capture changes in early intervention trials for ASD and to determine the feasibility of use across different studies.

In order to advance the field of ASD intervention research, a thorough review on the efficacy of different outcomes measures as well as their limitations is necessary. Though intervention research should continue while novel treatment response measures are developed, researchers should be cognizant of the many benefits and limitations when using these different measures. Likely, there is not a “one size fits all” approach to determining which outcome measure is most likely to capture meaningful change across the wide variety of ASD interventions. Therefore, when a researcher is considering an outcome measure for a study, the researcher should evaluate what he or she believes will change as a result of the intervention (e.g., ASD symptoms, a specific behavior, or global development), the intensity and duration of the intervention (e.g., 8 weeks versus 6 months), and the potential bias and placebo effects associated with different measures (e.g., caregiver report). Care should be taken to acknowledge these limitations and, whenever possible, to minimize the limitations of these measures. In addition, research would benefit from a priori justifications for choosing a certain outcome measure rather than another in order to minimize the possibility of Type 1 errors resulting from multiple comparisons across different measures. Ongoing examinations of different outcome measures in combination with continued advances in interventions will assist with the identification of effective and efficacious treatment programs for children with ASD.

Table 1.

Summary of Outcome Measures.

Treatment Outcome	Measure	Measured Constructs	Reporters	Target Age Range	Composite Score	Test-Retest Reliability	Norm group with ASD	Utility as an Outcome Measure
Adaptive Behavior	Vineland Adaptive Behavior Scales (VABS-3)	Socialization, Communication, Daily Living, and Motor Domains	Interview, Caregiver, Teacher Form	0–90 years (Interview, Caregiver); 3–21 years (Teacher)	Adaptive Behavior Composite	Interview Form: 0.82; Caregiver Form:0.87; Teacher Form: 0.88	Yes	(VABS-2): 18, 35–39
	Adaptive Behavior Assessment System (ABAS-3)	Conceptual, Practical, and Social domains	Caregiver, Teacher, Adult Forms	0–21 years (caregiver); 2–21 years (teacher); 16–89 years (adult)	General Adaptive Composite	General Adaptive Composite: 0.90	No	(ABAS-2): 66, 68
ASD Symptoms	Autism Impact Measure (AIM)	Repetitive Behavior, Atypical Behavior, Communication, Social Reciprocity, and Peer Interaction	Caregiver Questionnaire	2–17 years	Frequency and Impact Scores	0.53–0.85	Yes	75–76
	Brief Observation of Social Communication Change (BOSCC)	ASD-specific symptoms in Social Communication, Restricted and Repetitive Behavior	Blind Raters, Flexible social context	Toddler/Preschoolers using single words or less	BOSCC Core Total	0.78–0.90	N/A	56, 78–80
	Communication and Symbolic Behavior Scales Developmental Profile -Behavior Sample (CSBS-DP-BS)	Communication, Social-Affective Functioning, Symbolic Abilities	Direct Clinical Observation	8–24 months	Communication Composite Score	0.91	No	35
Social Skills and Problem Behaviors	Behavioral Assessment Scales for Children (BASC-3)	Adaptive Skills, Behavioral Symptoms Index, Externalizing Problems, Internalizing Problems, and School Problems	Caregiver, Teacher Report	2–21 years	Behavioral Symptoms Index	Caregiver Rating Scales (Preschool): 0.92	No	(BASC-2): 110–113
	Social Skills Improvement System (SSiS)	Social Skills, Problem Behaviors, Academic Competence	Caregiver, Teacher Report	3–18 years	Rating Scales, ASD Rating Subscale	Caregiver: 0.70–0.80 Teacher: 0.90–0.80	Yes	116–117;119–120
CognitiveFunctioning	Mullen Scales of Early Learning (MSEL)	Verbal and Non-verbal domains	Clinician Observation, standard administration	0–68 months	Early Learning Composite (ETC)	0.82–0.85 for ages 1–24 mos; 0.71–0.79 for 25–56 mos	No	18, 35–39, 121,127–130
	Bayley Scales of Infant and Toddler Development, Third Edition (Bayley-III)	Verbal and Non-verbal domains	Clinician Observation, standard administration	1–42 monthscf	Composite Sc ore for each domain	0.83–0.94 (across Motor, Cognitive, Language;33–42 months)	Yes (PDD)	(Socioemotiona l Subscale): 93