Direct observation in a large-scale randomized trial of parent training in children with autism spectrum disorder and disruptive behavior

Abstract

A direct observation strategy (Standardized Observation Analogue Procedure, SOAP) was used in a large-scale randomized trial of parent training versus parent education in young children with autism spectrum disorder (ASD) and disruptive behavior. The 16-minute SOAP, modified from an earlier version of this same measure, included parentchild interaction to assess child behavior in a clinical laboratory setting. Despite study entry criteria for all child participants requiring moderate levels of disruptive behavior in this project, 126 of 168 children with complete SOAP data at baseline showed no disruptive behavior on this measure. Although the primary purpose of the study was to determine whether the SOAP could detect differences between the two conditions (i.e.,parent training (PT) and parent education (PE)), baseline observation data was not consistent with parent ratings at baseline or subsequent follow up visits, leaving little room to demonstrate improvement with this observation measure. This and the challenging, time-consuming and resource intensive effort involved in using such a measure in a large randomized scale trial, raises fundamental questions about the validity of the SOAP as an outcome measure in such a study. Further consideration related to the feasibility and practicality of using direct observation as a primary measure in larger scale efforts overall are also discussed.

Introduction

Autism spectrum disorder (ASD), defined by impaired social communication, repetitive behavior and restricted interests, affects as many as 18.5 per 1000 school age children (Maenner et al., 2020). Several reports estimate that 50% of children with ASD exhibit disruptive behaviors, such as tantrums, aggression, self-injury and noncompliance (Hartley et al., 2008; Mazurek et al., 2013; Maskey et al., 2013). In addition to added stress for parents, these behaviors can impede the child’s educational progress, interfere with regular performance and acquisition of daily living skills, and increase social isolation (Postorino et al., 2019; Maskey et al., 2013; Scahill et al., 2016).

Parents, who play a central role in promoting skill development as alternatives to problem behavior in children with ASD (Matson, Mahan, & Matson, 2009), report decreased parenting stress and increased self-efficacy following successful treatment with parent training (Ingersoll & Dvortcsak, 2006; Iadarola et al. 2018). Careful and intentional study on the mechanisms of change that promote such reduced disruptive behavior in the child and improved parent-child interaction is warranted.

Historically, studies conducted in the tradition of applied behavior analysis (ABA) and designed to reduce disruptive behavior in children with ASD have relied on the use of single subject methodology and direct observation to evaluate outcome (Campbell, 2003; Didden et al., 2006; Wong et al., 2015). The use of direct observation, like used in ABA-based studies, is seen as advantageous given strengths of this option. Namely, direct observation is preferred because it provides counts of child-specific behaviors under reproducible clinical laboratory conditions and does not rely on subjective parent reports or clinician inference (Kahng et al., 2008; Pellegrini, 2001).

However, to promote replication and wider application, investigators have accepted the challenge to assemble empirically supported techniques into parent training manuals and to conduct randomized controlled trials (RCTs). Although direct observation is a mainstay in single subject research, these RCT studies have traditionally used caregiver ratings as the primary outcome (Postorino et al., 2017). Though parent-rated outcomes, such as the Irritability subscale of the Aberrant Behavior Checklist, have shown sensitivity to change, caregivers are not blind to treatment assignment (Aman et al., 2009; Bearss et al., 2015). Thus, a potential advantage of direct observation, video recording and subsequent coding, is that it allows for the assessment of child behavior that is masked to treatment assignment. Moreover, even if laboratory conditions are standardized, direct observation methods can be child-specific (e.g., using a highly desired toy or activity). Caregiver ratings, by contrast, include a pre-selected set of items that may not always capture the behaviors of greatest concern to parents in interactions with their children.

In their review of parent training studies that included 9 RCTs, Tarver and colleagues (2019) noted that direct observation has rarely been used to measure outcome. Further prior work by our own group has shown that, the use of direct observation as an outcome measure in parent training RCTs faces multiple challenges (Johnson et al., 2009; Handen et al., 2013). First, in a multi-site study RCT, internal validity requires a standardized direct observation sequence that also satisfies ecological validity. The standardized conditions may not capture the situations that precede disruptive behavior in all children with ASD. Second, recording and coding video recordings from direct observations require training that are time-consuming. These constraints may impose limits on the length of the direct observation sequence and the number of repeated recordings during the trial. Finally, when collecting video recordings across multiple time points in an efficacy study, inevitable technical failures will result in missing data (Handen et al., 2013). Provided these considerations of direct observation as an outcome measure in studies related to behavioral interventions in ASD, the current study seeks to clarify and highlight considerations for use.

Though recent research has emerged involving the use of group-based parent training to reduce problem behaviors in children with ASD (e.g., O’Donovan et al., 2019; Parlade et al., 2020), these studies have been without the size and/or methodological rigor comparable to the Research Units in Behavioral Intervention (RUBI) Autism Network study serving as the base for the current report. The RUBI network study included a standardized direct observation sequence as a secondary outcome measure in our primary trial comparing parent training versus parent education (i.e., psychoeducation) interventions (Bearrs et al., 2015). For this multi-site trial of 180 children (age 3 to 7 years) with ASD and disruptive behavior problems (Bearss et al. 2015; Scahill et al. 2016), our interdisciplinary team of behavior analysts, psychologists, and clinical trialists modified the Standardized Observation Analogue Procedure (SOAP; Johnson et al., 2009; Handen et al., 2013) utilized in our previous network studies to improve upon the direct observation and to obtain a measure of parent and child behavior over time.

The SOAP was originally developed to complement caregiver and blinded clinician ratings and to provide a more multimodal and objective assessment that would build upon the respective strengths of tools used in measuring changes in child behaviors as a result of changes in parent skills. To develop the SOAP procedures for this study, we adapted the SOAP conditions from our network’s previous trials (Johnson et al., 2009; Handen et al. 2013) and also incorporated traditional procedures used in functional analysis (Hanley et al., 2003; Iwata et al., 1982) and procedures used in Parent Child Interaction Therapy (i.e., PCIT; Brinkmeyer & Eyberg, 2003).

In this report, we present the results of the direct observation using the SOAP in modified form and informed by experience with the measure in previous trials. We test whether, 1) parent training (PT) is superior to parent education program (PEP) in decreasing child inappropriate behavior (e.g., aggression, tantrums, self-injury or property destruction) and increasing compliant behavior and 2) parents in PT, compared to PEP, show greater use of appropriate reinforcement strategies (e.g., nonverbal positive attention, verbal praise) and use of appropriate target commands. We also compared SOAP findings with the primary outcomes reported from the original trial.

Methods

Design

Children were randomly assigned to a structured Parent Training (PT) program (n=89) or a structured Parent Education program PEP; n=91) for six months (Bearss et al., 2015; Scahill et al., 2016). PT provided parents with behavioral principles and techniques designed to reduce disruptive behavior. PEP offered general information on ASD, but no instruction on behavioral management strategies. Both PT and PEP sessions were 75–90 minutes in length. The number of sessions was similar: 11 core sessions, 2 home visits and 2 booster sessions for PT; 13 sessions, 1 home visit and 1 booster session for PEP. Both treatments were individually delivered by a trained therapist (i.e., one therapist to each parent/child dyad over six months). There were a total of 23 therapists across 6 network sites). The manuals included session-specific scripts for therapists, activity sheets and homework assignments. PT also included brief video clips to illustrate specific behavioral techniques. The general session topics for both PT and PEP are outlined in Table 1.

Table 1.

General Session Topics (PT and PEP)

Parent Training (PT)*	Parent Education Program (PEP)**
Description of Behavioral Model	Autism Diagnosis
Functional Behavioral Assessment	Understanding and Interpreting Clinical Evaluations
Antecedent Management Preventive Strategies	Understanding Behavioral Measures
Use of Schedules	Developmental Issues
Reinforcement	Medical and Genetic Issues
Teaching Compliance	Choosing Effective Treatments
Planned Ignoring	Advocacy and Support Services
Teaching Skills	Review of Treatment Options
Generalizing Skills	Educational Planning
Functional Communication Skills	Other Services
	Play Activities

To be eligible, children had to be between age 3:0–6:11 years, have a Diagnostic and Statistical Manual-Fourth Edition (DSM-IV-TR; American Psychiatric Association, 2000) diagnosis of Autistic Disorder, Asperger’s Syndrome, or Pervasive Developmental Disorder-Not Otherwise Specified and at least moderate disruptive behavior as evidenced by a raw score ≥15 on the parent-rated Aberrant Behavior Checklist-Irritability (ABC-I; Aman et al., 1985) subscale. Children on stable behavioral and medical treatments for at least 6 weeks with no planned changes during the trial were allowed to enroll.

Procedures

Children were evaluated by an experienced diagnostic team at each study site (Emory University, Indiana University, Ohio State University, University of Pittsburgh, University of Rochester, Yale University). ASD diagnoses were supported by the Autism Diagnostic Interview-Revised (Lord et al., 1994) and the Autism Diagnostic Observation Schedule (Lord et al., 2000). Primary and key secondary outcomes are reported elsewhere (Bearss et al., 2015; Iadarola et al., 2018; Lecavalier et al., 2017; Scahill et al., 2016). Informed consent was obtained from all individual participants included in the study. All procedures performed were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments (2000) or comparable ethical standards.

Measures

The Aberrant Behavior Checklist-Irritability Subscale (ABC-I; Aman and Singh, 2018) is a reliable and valid 58-item, parent-rating of behavior that includes five subscales. The primary outcome variable in this study was the ABC-Irritability subscale (Aman et al., 2009; Bearss et al., 2015). This 15-item subscale includes disruptive behavior, tantrums, and aggression. Each item is rated 0–3 with higher scores indicating greater severity

The Home Situations Questionnaire-Autism Spectrum Disorder (HSQ-ASD; Chowdhury et al., 2015) is a 24-item parent rating of noncompliant behavior in real life situations. The measure has two subscales (Socially Inflexible and Demand-Specific), each with 12 items. Questions answered “yes” are rated on a 1 to 9 Likert scale with higher scores indicating greater noncompliance. The severity ratings of the “yes” responses are summed (range 0–2) and divided by 24 for a per item mean score. To derive the per-item mean for the subscales, the severity items for that subscale are summed and divided by 12.

Clinical Global Impression-Improvement Scale

(CGI-I; Guy, 1976) is a clinician-rated, 7-point scale designed to measure overall improvement from baseline. Scores range from 1 (Very Much Improved) to 4 (Unchanged) to 7 (Very Much Worse). The IE (independent evaluator) at each site, who was masked to group assignment, used all available information to judge treatment response. By convention, CGI-I ratings of Much Improved (score of 2) or Very Much Improved (score of 1) classify subjects as having a positive response to treatment.

Standardized Observation Analogue Procedure (SOAP).

The SOAP is a 16-minute semi-structured observation of parent-child interaction designed to assess child behavior and parenting skills in a clinical laboratory setting. The SOAP was completed by PT and PEP participants at three time points (baseline, Week 12, Week 24) and included four conditions: 1) Free Play (3 minutes) - children were allowed to familiarize themselves to the clinic setting and choose activities; 2) Parent-Led Play (5 minutes) – parents were asked to direct the child’s play and to encourage the child to participate; 3) Parent Demand (5 minutes) - parents were prompted to provide 10 child-specific commands (pertinent to individual families and identified before the session began) and respond as they would at home to gain child compliance; 4) Clean Up (3 minutes) - parents were asked to instruct the child to put away toys. This standardized sequence and timeframe for each condition and each participant resembled procedures used in prior RCTs of children with disruptive behavior problems (Brinkmeyer & Eyberg, 2003; Handen et al., 2013). To be included in the current analysis study, participants had to complete a SOAP session at baseline and at least one post baseline session.

Coding Process

Coders used a customized interval recording method (Observer® XT program, YR). The coding scheme was applied to parent and child behaviors in the “parent led play” (PLP), “parent demand” (PD), and “clean up” (CU) conditions. The 3-minute acclimation, free play period was not coded. Operationally defined child behaviors (i.e., inappropriate behavior, initial compliance) and parent skills (i.e., reinforcement, commands) are detailed in Table 2. Instances in which the parent or child’s upper body could not be viewed on the video were coded as “not observed parent” or “not observed child,” respectively.

Table 2.

Outline of Systematic Observation Analogue Procedure (SOAP)

CHILD BEHAVIOR
Inappropriate Behavior	Aggression	Behaviors directed towards another person (e.g., biting, hitting, kicking, pinching, spitting, pushing, grabbing hair pulling, scratching or using objects to hit or poke).
	Property destruction	Behaviors such as throwing, hitting or kicking object. Actual “destruction” or damage to the object was not required.
	Self-injury	Behaviors such as head banging, self-biting, hair pulling, eye gouging, slapping, skin picking, body slamming, inserting objects into body cavities, scratching, or using objects to hit, scratch, or stab.
	Tantrums	Behaviors such as stomping, yelling, flailing arms and legs, screaming, crying, jumping up and down, and/or falling to the floor.
Initial Compliance		Begin to comply with parent prompt or command. Compliance was not coded as present if the child complied after the parent used physical prompts to gain task completion.
PARENT BEHAVIOR
Reinforcement		Verbal praise or nonverbal attention (e.g., smiles, pats on back, high five).
Commands	Appropriate target command	Single step demand that included an observable action required for completion.
	Inappropriate target command	Instruction which included multiple questions or directions or required multiple steps in order to achieve task completion.
	Repeat command	Parent repeated the same or similar verbal command after an initial command

Inter-Rater Reliability

An experienced clinician trained a lead coder (staff member) to 80% reliability using live and recorded practice with coaching to clarify coding decisions. Two subsequent coders (an undergraduate and a graduate student) were trained to reliability with the lead coder (≥80% agreement on 3 consecutive practice video recordings). Study videos were assigned ID numbers to ensure that coders were masked to treatment condition and assessment time point. To check reliability of study videos a 10% randomly selected sample was coded by a second coder, who was also trained to reliability on the coding procedure. Agreement on each coded variable was divided by the number of agreements plus disagreements times 100 (Handen et al., 2013).

Analytic Plan

Participants with useable SOAP data within each subgroup (PT and PEP) and across the full sample of participants were compared (total = 168; 82 in PT, 86 in PEP). To assess SOAP measurement of treatment effects on child disruptive behavior, compliance as well as parental use of appropriate reinforcement strategies and effective target commands, we used generalized estimation equation models (GEE) to compare across PT and PEP. GEE models are appropriate to control for the correlation on repeated measurement of counts on the same subject (Wang, 2014). Preliminary analyses revealed a leftward skewed distribution on the coded behaviors. Therefore, we used the negative binomial distribution in the analyses. The models were adjusted for the baseline on each outcome, time (i.e., weeks of intervention), treatment group (PT vs PEP), condition (i.e., SOAP condition) and the interaction terms between time, treatment group and condition. Linear contrasts were used to estimate group difference at each time point and outcomes are presented as least square means with 95% confidence interval. To contrast the capacity of child SOAP measures to detect change from baseline to Week 24, we calculated change on the least squares means of Child SOAP measures against the change on the Irritability subscale of the Aberrant Behavior Checklist and Home Situations Questionnaire-ASD. Two-sided p value less than 0.05 was considered statistically significant. All analyses were performed with SAS, version 9.4 (SAS Institute).

Results

Of the 180 study participants in the RCT, the SOAP dataset included 168 participants (82 in PT and 86 from PEP groups). Missing baseline data on 12 participants were due to technical failures with video recordings or attrition. Demographics and clinical characteristics for the study sample are provided in Table 3. Compared to the sample in the original RCT (Bearss et al., 2015), there were no differences on any baseline demographic and clinical characteristics.

Table 3.

Baseline clinical and demographic characteristics by treatment group (n=168)

Characteristics	Parent Training (n= 82)		Parent Education (n=86)
Mean (SD) Age in Years	4.8	1.15	4.7	1.00
	N	%	N	%
Males	74	90.24	75	87.21
Race
White	72	87.8	73	84.88
Black	9	10.98	6	6.98
Asian	1	1.22	4	4.65
Pacific Islander	0	0	2	2.33
Other	0	0	1	1.16
Ethnicity
Hispanic	13	15.85	13	15.12
Non-Hispanic	69	84.15	73	84.88
Two parent family	71	86.59	77	89.53
Maternal Education
Advanced/Profession Degree	28	34.15	22	25.58
College Degree	21	25.61	35	40.70
Some College	25	30.49	24	27.91
High School Graduate	7	8.54	5	5.81
Some High School	1	1.22	0	0
CGI-Severity
Moderately Ill	38	46.34	48	55.81
Markedly Ill	29	35.37	28	32.56
Severely Ill	15	18.29	10	11.63
Aberrant Behavior Checklist	Mean	SD	Mean	SD
Irritability	23.8	6.41	24.1	6.21
Social Withdrawal	12.9	8.42	12.5	8.12
Stereotypy	6.1	4.49	6.4	4.99
Hyperactivity	29.7	9.60	31.5	8.73
Inappropriate Speech	5.3	3.15	5.9	3.09
Home Situations Questionnaire-ASD
Demand-Specific	3.6	1.71	3.26	1.69
Socially-Inflexible	4.4	1.60	4.4	1.71
Total	4.0	1.52	3.8	1.50
Vineland II Adaptive Scales
Communication	81	15.18	81.5	15.56
Daily Living Skills	77.1	12.92	79.3	14.49
Socialization	70.8	11.46	73.2	10.57

Child Outcomes on SOAP

Although study participation required at least moderate disruptive behavior at baseline, 62 of 82 (76%) participants in PT and 64 of 86 (74%) in PEP displayed no disruptive behavior at baseline on at least one SOAP condition. The PT group had a lower rate of inappropriate behavior and greater improvement in compliance at Week 24 than PEP, but these differences were not statistically significant.

Parent Outcomes

Parental use of Effective Commands (Figure 3) showed no change from baseline for either group. Parental use of Reinforcement (Figure 4) showed slight improvement from baseline for PT compared to a slight reduction in use of reinforcement for PEP. However, there were no group differences in the assessment of parental behavior on effective commands or reinforcement.

Figure 3.

Shows the mean occurrence appropriate parental commands by group at baseline, Weeks 12 and 24 (N=168)

Figure 4.

Shows the mean occurrence appropriate parental reinforcement by group at baseline, Weeks 12 and 24 (N=168)

Table 4 shows contrasts of PT and PEP from baseline to Week 24 on the ABC Irritability and HSQ-ASD scores (co-primary outcomes) and the change in Child SOAP measures. These results suggest that the SOAP measures of child behavior were not able to detect group differences despite clear superiority for PT compared to PEP on the co-primary outcomes (Bearss et al., 2015).

Table 4.

Comparison of SOAP with primary outcome measures (n=168)

	PT (n=82)		PE (n=86)		Difference at WK 24	P
	BL Mean (SD)	WK 24 Mean (SD)	BL Mean (SD)	WK 24 Mean (SD)	LS mean (95% CI)
ABC-I	23.57 (6.28)	11.88 (6.54)	24.02 (6.13)	16.47 (7.63)	−3.42 (−5.34, −1.49)	0.0005
HSQ Demand-Specific	3.65 (1.71)	1.58 (1.39)	3.23 (1.64)	1.81 (1.53)	−0.63 (−1.02, −0.25)	0.0014
HSQ Soc Inflexible	4.4 (1.61)	2.11 (1.68)	4.42 (1.67)	2.91 (1.63)	−0.77 (−1.24, −0.29)	0.0015
HSQ Total	4.02 (1.53)	1.84 (1.47)	3.82 (1.44)	2.36 (1.46)	−0.71 (−1.11, −0.30)	0.0007
Child SOAP measure 1 (inappropriate behavior)	0.17 (0.58)	0.07 (0.16)	0.22 (0.49)	0.17 (0.47)	−0.08 (−0.20, 0.03)	0.1331
Child SOAP measure 2 (child compliance)	37.35 (17.18)	44.51 (19.68)	34.37 (19.05)	39.9 (17.35)	3.04 (−2.05, 8.13)	0.2407

Discussion & Implications

To our knowledge, this is the largest randomized trial of a behavior intervention in ASD to date that used a direct observation strategy to evaluate child and parent behavior. The 16-minute SOAP was designed to measure treatment effects of PT on child disruptive behavior (i.e., tantrums, aggression, and self-injury), child compliance and parental use of positive reinforcement and effective commands. This version of the SOAP was revised from our previous network study in order to address shortcomings noted in previous trials (Handen et al., 2013). Despite criteria requiring at least moderate levels of disruptive behavior as assessed on multiple study measures to enter the study, 126 of 168 children with complete SOAP data at baseline showed zero disruptive behavior on this particular measure.

The lack of agreement between SOAP measures and parent-ratings of behavior at baseline raises fundamental questions about the validity of the SOAP as an outcome measure in a large-scale multi-site randomized trial. Other studies have reported similar discrepancies between informant reports and direct observation (Aman et al., 2009; Handen et al. 2013). It may be that the novelty of the clinical laboratory situation inhibited baseline behavior in study participants and contributed to the low frequency of observed disruptive behavior. In fact, the challenge of reliably eliciting target behavior using standardized protocols is not unusual in research nor clinical settings (Handen et al., 2013; Palmer et al., 2021). It is manifestly difficult to replicate the natural setting within the clinical and research laboratory or to provide the contextual cues necessary to trigger behavior in these novel environments. This suggests a potentially systematic bias in baseline SOAP measures. The low frequency of disruptive behavior at baseline also left little room for improvement. In addition, the application of SOAP required substantial time, space and technical support to collect the video recordings and substantial effort to code the videos. Indeed, the financial expenditure for collecting the videos, training coders and the actual coding of the SOAP data was the single most expensive outcome measure in this trial.

Limitations

The SOAP conditions used in this the trial were designed to follow a specific sequence for all children. Although child-specific components such as selecting preferred toys and targeted child commands were incorporated, failure to elicit disruptive behaviors at baseline suggests that SOAP conditions in the laboratory setting did not resemble everyday living contexts for child participants. By contrast, traditional single subject design studies use a more idiographic approach to direct observation in which conditions are more specifically tailored for each individual child. Despite some level of individualization (e.g., demands issued to elicit behavior) in this study, the large number of children engaged required a more standardized approach (e.g., toys, order of sessions). A more child-specific approach in which more individualized choice of toys and order of observational contexts as well as sessions more specifically aligned with specific triggers known to elicit disruptive behavior consistent with the child’s usual behavioral patterns may have had bearing on the results. Another possible limitation is the relatively brief length of SOAP conditions (3–5 minutes per condition). For example, we included a 3-minute acclimation period as the first SOAP condition. A longer acclimation period may have reduced the novelty of the clinical setting and induced a more typical behavioral pattern at baseline and in subsequent recording sessions. Another limitation, though not included in this project, it would have been interesting for parents to view and rate sessions as to how well they correlate to their day-to-day experiences with their child. In that way, even if the behavioral coding methods did not pick up on the particular areas of concern, some indication as to the validity of the SOAP coding results could have been obtained.

The proportion of videos coded for reliability by a second coder was 10%. This 10% sample of videos selected for reliability coding was driven by the large study sample size and is the same proportion used in our previous study (Handen et al. 2013). Although the behaviors included in this report were our primary targets, the second coder also reviewed other behaviors that were not included in this report. Therefore, we could not calculate the reliability of the behaviors included in this report separately.

Conclusions

Results obtained from direct observation across SOAP conditions were not consistent with the primary outcomes (ABC Irritability subscale and HSQ) nor the key secondary outcome (blinded clinician rating of the CGI for Improvement or the Vineland Daily Living domain) in this study (Bearss et al. 2015; Scahill et al. 2016). The incongruence of the direct observation with other study measures was particularly evident at baseline. It may be that the direct observation procedures used successfully in single-case research are not easily applied to a more large-scale effort. Direct observations of single-case designs are traditionally highly individualized and used to identify functions of behavior, guide behavioral intervention strategies and evaluate outcome. By contrast, the standardized SOAP sessions used in this large-scale multi-site study were developed to serve as outcome measures rather than a functional analysis of behavior. Although direct observation may not be practical in large-scale RCTs, it may be a useful tool for assessing behavior change in smaller RCTs (e.g., Lindgren et al., 2020). The application of direct observation in this large-scale trial of PT versus PEP proved challenging, time-consuming, and not informative. The study results of direct observation offer a cautionary note on the feasibility of direct observation in large-scale randomized trials. In the future, innovative technology-based strategies may be used in large-scale trials to collect ecologically valid observations of child behavior and parent-child interaction in home and community settings.

Figure 1.

Shows the mean rates of child inappropriate behavior by group at baseline, Weeks 12 and 24 (N= 168)

Figure 2.

Shows the mean occurrence of child compliance by group at baseline, Weeks 12 and 24 (N=168)

Highlights.

• Lessons learned from using a direct observation strategy (i.e., SOAP) in the evaluation of child and parent behavior as part of the largest randomized trial of behavior intervention in ASD to date

• Despite meeting criteria for the study, child participants did not demonstrate appreciable levels of behavior at baseline on the SOAP

• Significant effort and resources were subsumed in training, coding and otherwise utilizing the measure across sites

• Study results give rise to considering the suitability (i.e., feasibility, practicality, risk-benefit) of using direct observation as a primary outcome measure for larger scale randomized control trials