The Pediatric Autism Spectrum Therapy Observation System: Development, Psychometric Properties, and Sensitivity to Treatment

Abstract

An observational coding system was developed to track clinical change in children with autism spectrum disorder (ASD) during psychotherapy. The Pediatric Autism Spectrum Therapy Observation System (PASTOS) consists of 23 items divided into 5 subscales and is used to rate child behaviors in individual psychotherapy sessions. Manual-based cognitive behavioral therapy session transcripts of 22 children diagnosed with ASD (IQ > 70) and a concurrent anxiety disorder (M = 9.41 years, SD = 1.56 years) enrolled in a randomized, controlled trial were coded. Results suggested that the PASTOS exhibited promising interrater reliability, internal consistency, convergent validity at post-treatment, and treatment sensitivity. The PASTOS may be a useful tool for studying process and outcome in psychotherapy research on children with ASD.

Autism spectrum disorder (ASD) is a chronic neurodevelopmental disorder of impaired social communication and restricted and repetitive behaviors (Angarita & Kolevzon, 2017). It is currently estimated to affect 1 in every 59 children in the United States (Baio et al., 2018). Anxiety disorders are a common and impairing co-occurring condition for children with ASD (de Bruin, Ferdinand, Meester, de Nijs & Verheij, 2007; Kerns, Renno, Storch, Kendall, & Wood, 2017; Leyfer, Folstein, Bacalman, Davis & Dinh, 2006; Simonoff et al., 2008; van Steensel, Bogels, & de Bruin, 2013) and may add additional pathways to impairment (Bellini, 2004; Sukhodolsky et al., 2008; Ung et al., 2013; Wood & Gadow, 2010). Recently, cognitive behavioral therapy (CBT) approaches have been tested and appear promising as a support for children and youth with ASD (cf. Danial & Wood, 2013). In particular, CBT has emerged as a viable treatment for impairing anxiety in the context of ASD, with some evidence suggesting that related social and adaptive impairment can be addressed through cognitive and behavioral practices as well (e.g., White et al., 2009; Wood, Kendall, Wood, Kerns, Seltzer, Small, Lewin, & Storch, 2019). However, most significant findings in this line of research have been based upon parent-report instruments; child-report measures have often not exhibited significant treatment effects. There are inevitable challenges in interpreting treatment effects when informant reports do not agree with one another (McMahon & Solomon, 2015), leading to the need for complementary forms of outcome assessment in clinical trials for children with ASD. Relatedly, measuring anxiety and other outcomes in clinical trials for children with ASD has been hindered by a lack of specific measure development for assessing clinical change in ASD: many measures for ASD have been developed with diagnosis as a focus, rather than sensitivity to treatment (e.g., Lecavalier et al., 2014; Anagnostou et al., 2015).

Parents of children with ASD typically perceive their children as underperforming compared to typically developing peers on clinical measures, while children with ASD often rate themselves as performing comparably to typically developing peers on many measures (e.g., Lerner, Calhoun, Mikami, & De Los Reyes, 2012). Possible explanations for this discrepancy are that parental frustration, awareness of child diagnosis, the ability to accurately recall retrospective information, or ASD severity may influence parent and child reports differentially (Davis & Carter, 2008; McMahon & Solomon, 2015; Ozonoff et al., 2011). In the case of children with ASD and anxiety, it has been proposed that parents may struggle to understand and report the child’s internal symptoms or could confuse symptoms of ASD with anxiety in their children (Storch, Ehrenreich-May et al., 2012). Children with ASD may not always be accurate reporters of their own emotions (Hill, Bertoz, & Frith, 2004) and may have varying levels of social motivation and relatedness as well as difficulty with abstract concepts, all possibly contributing to differences in response patterns from parents (Storch, Larson et al., 2012). Whatever their cause, informant discrepancies have implications for evaluating outcomes of clinical trials. Inconsistent reports lead to ambiguity about outcome results (De Los Reyes, Alfano & Beidel, 2010) and the true impact of treatment.

There have been recent calls for the development of clinical instruments that have greater objectivity and sensitivity to treatment for children with ASD (e.g., Lecavalier et al., 2014; Anagnostou et al., 2015). One fruitful approach to the development of such instruments may be extrapolated from the treatment process research tradition (e.g., McLeod & Weisz, 2010). In treatment process research, methods focus on interactional sequences between clients and therapists, recognizing that the mechanisms of change in psychotherapy are partially embedded within client-therapist interactions (e.g., Woolley, Butler, & Wampler, 2000). Studies examining treatment process have often utilized observational methods because of the complexity of the content and the relative difficulty for individuals involved in the sessions to reflect and decompose the dynamics of their interactions objectively. Typically, trained raters code video or audio-taped therapy sessions using measures of therapeutic strategies, the therapeutic relationship, competence, or adherence. Although untested for assessing outcomes in CBT for children with ASD, the treatment process research approach offers a level of objectivity that informant-based measures lack (i.e., trained raters rather than participants assess behaviors). Hypothetically, a process measure focusing on client outcomes might add a certain degree of objectivity, or at least, a different lens on outcome, in comparison with informant-based measures.

In a separate line of research, personalized session-by-session clinical symptom tracking has been an increasingly commonly metric for assessing psychotherapy treatment outcomes (e.g., Weisz et al., 2011). These measures have become more central in psychotherapy research in part because of the sensitivity to treatment that personalized outcome measurement offers and in part because of the increased statistical power yielded from acquiring multiple repeated measurements of outcome.

In this study, a treatment process tool was developed to assess clinical change in children with ASD and clinical anxiety participating in CBT in an effort to expand options for assessing response to treatment for this population. The traditions of treatment process research and session-by-session treatment outcome tracking were fused in the development of the Pediatric Autism Spectrum Therapy Observation System (PASTOS). The PASTOS utilizes independent evaluators who rate entire therapy sessions for indices of manifestations of emotion dysregulation (e.g., anxiety), core ASD symptoms, self-help skills, coping skills, and positive parenting. Although this set of treatment foci is not a comprehensive accounting of all possible clinical needs among children with ASD, it does address many of the most common clinical target areas in psychotherapy programs for children with ASD (cf. Danial & Wood, 2013). Although positive parenting has not been assessed as a mediator of psychotherapy effects in children with ASD, research has shown that changes in positive parenting mediate clinical improvements in children with disruptive behavior disorders (e.g., Gardner, Burton, & Klimes, 2006).

This study utilized treatment session data from the first randomized controlled trial (RCT) of a CBT program, Behavioral Interventions for Anxiety in Children with Autism (BIACA), for children with ASD aged 7 to 11 years old (Wood, Drahota, Sze, Har, Chiu, & Langer, 2009). BIACA entails modular CBT primarily addressing impairing anxiety as well as clinical phenomena that may exacerbate anxiety or impact the effectiveness of common CBT practices, including social difficulties and repetitive behaviors, challenges with self-help skills, poor self-perceptions, and impacted parent-child communication. BIACA is comprised of 16 weekly 90-minute CBT sessions split evenly between children and their parents. In the first study of BIACA (Wood, Drahota, Sze, Har et al., 2009), independent evaluator ratings of anxiety disorder remission and positive treatment response on the Clinical Global Impressions-Improvement scale (CGI-I; Guy, 1976), as well as parent ratings of impairing anxiety, improved more in the BIACA condition than in a waitlist comparison condition. In secondary outcome studies of the same trial, parent ratings of social responsiveness and children’s self-help skills also improved more in the BIACA condition than in the waitlist condition (Drahota, Wood, Sze, & Van Dyke, 2011; Wood, Drahota, Sze, Van Dyke et al., 2009). Additional randomized controlled trials have confirmed and extended these findings in larger samples with more stringent comparison groups (e.g., Wood et al., 2019). In the present study, the PASTOS is cross-validated against several informant-report outcome measures utilized in the first BIACA clinical trial (Wood, Drahota, Sze, Har et al., 2009) to preliminarily examine construct validity.

The broad goal of this study was to develop a treatment process coding system that could be used to track clinical outcomes in CBT for children with ASD and anxiety, and possibly other variants of psychotherapy for school-aged children with ASD, as a complement to informant-based measures. This study assessed initial indices of interrater reliability, internal consistency, convergent and discriminant validity, and sensitivity to treatment for the PASTOS measure. It was hypothesized that (a) scores on the PASTOS subscales would exhibit adequate interrater reliability and internal consistency based upon conventional standards; (b) PASTOS scores based on an early treatment session (session 2) would correspond most to scores on informant-based measures within the same construct (i.e., the PASTOS subscale assessing anxiety and emotion dysregulation would correspond more to informant-based measures of anxiety than to informant-based measures of social skills and other constructs) measured at pretreatment, and that a similar pattern would hold for PASTOS scores from a late-treatment therapy session (session 15) compared to scores on informant-based measures of anxiety and emotion dysregulation collected at posttreatment; (c) that PASTOS scores would decline from early treatment to late treatment; and (d) that children rated by the independent evaluator as having had a positive response to treatment on the CGI-I would have lower PASTOS scores than children rated as not having had a positive response to treatment. Although in this study the PASTOS was tested with a sample of children with ASD and anxiety, its five subscales were designed with the broader population of children with ASD in outpatient psychotherapy in mind, such that a subset of one or more subscales might be used in a given trial depending on its main emphasis (e.g., social communication as opposed to emotion dysregulation).

Method

The sample includes 22 school-aged children with ASD who were in the immediate treatment group of a waitlist-controlled RCT of BIACA (Wood, Drahota, Sze, Har et al., 2009) (age range = 7 to 11 years; M = 9.41 years; SD = 1.56 years). Participants were recruited from local hospitals and diagnostic centers for autism, parent support groups, state regional centers, community-based clinics and private practices in the western United States. Participants’ demographic information can be found in Table 1. See Wood, Drahota, Sze, Har et al. (2009) for a detailed description of the participants. Informed consent was obtained from all individual participants included in the study. Participant inclusion was determined by the administration of standardized diagnostic interviews with both the child and the parent separately to verify the presence of a DSM-IV anxiety disorder using the Anxiety Disorders Interview Schedule (ADIS): Children and Parent Versions (Silverman & Albano, 1996). ASD assessment was conducted using a published diagnostic algorithm (Klin, Pauls, Schultz, & Volkmar, 2005) incorporating scores from both the Autism Diagnosis Interview-Revised (ADI-R; Rutter, Le Counteur, & Lord, 2003), a semi-structured interview administered to parents, and the Autism Diagnostic Observation Schedule-Module 3 (ADOS; Lord, Rutter, DiLavore & Risi, 1999). In the Klin et al. system, children need to meet the cut-score for the ADI-R Social symptom area at minimum. Slightly below half of the children (17/40) in the first BIACA RCT (Wood, Drahota, Sze, Har et al., 2009) met the minimum criteria and were classified with Pervasive Developmental Disorder Not Otherwise Specified; the others met additional ADOS and ADI-R criteria for ASD and thus met Klin et al. criteria for autism or Asperger syndrome. Hence, to have participated in the RCT (i.e., Wood, Drahota, Sze, Har et al., 2009) the children needed to surpass, at minimum, the clinical cut-off score for the Social area on the ADI-R, and to meet criteria for an anxiety disorder based on the ADIS interview. Potential participants whose estimated IQs were below 70 were excluded from the first BIACA RCT (Wood, Drahota, Sze, Har et al., 2009). This criterion was determined by initial telephone screenings in which parents were asked if their child was able to speak in complete sentences. This determination was further informed through the review of previous assessments at intake. When needed, the Wechsler Intelligence Scale for Children-IV was administered by qualified doctoral students.

Table 1.

Participant Characteristics

Characteristic	n = 22
Child Sex (male)	17 (77%)
Child Age	M = 9.41 (SD = 1.56)
Parent Sex (female)	18 (82%)
Child Ethnic Background
Caucasian	12 (55%)
Hispanic/Latino	2 (9%)
Asian/Pacific Islander	4 (18%)
Multiracial	4 (18%)
Family Gross Annual Income ($)
Under 40,000	4 (18%)
40,001 – 50,000	1 (5%)
60,001 – 70,000	1 (5%)
70,001 – 80,000	1 (5%)
80,001 – 90,000	1 (5%)
90,000+	14 (62%)

The data for the current study is comprised of 88 audiotapes of treatment sessions from the Wood and colleagues (2009) RCT, which were transcribed. Four sessions from each of the 22 CBT treatment group participants across four time points (sessions 2, 4, 10, and 15) were sampled (note: BIACA is a 16-session treatment, but because the last session includes a celebration which is generally not representative of treatment process, session 15 was selected as representative of late treatment instead). The four time points were selected to sample the early phase of treatment (sessions 2 and 4) and the middle to later phases of treatment (sessions 10 and 15) (e.g., McLeod & Weisz, 2010).

Measures

The Pediatric Autism Spectrum Therapy Observation System (PASTOS).

The PASTOS is 23-item behavioral observation instrument with five subscales developed by the authors to use for coding child and parent verbalizations and behaviors during treatment sessions. It was developed in reference to previous treatment process instruments (e.g. McLeod & Weisz, 2010; Patterson & Chamberlain, 1994) but with a focus on items pertaining to five child and family outcome domains: (a) Social Communication, (b) Anxiety and Related Emotional States, (c) Self-Help Skill Mastery, (d) Child Self-Perception, and (e) Parent-Child Interactions.

The items developed for the PASTOS were based on extensive literature reviews conducted by the lead author. Prominent measures used in outcome studies for children with ASD were reviewed (cf. Lecavalier et al., 2014; Anagnostou et al., 2015), particularly those that corresponded with treatment targets in CBT and parent training programs for children with autism such as BIACA. The instruments consulted included the Multidimensional Anxiety Scale for Children (MASC; March, 1998); the Social Responsiveness Scale (SRS; Constantino & Gruber, 2005); the Social Skills Rating System (SSRS; Gresham & Elliott, 1990); the Children’s Yale-Brown Obsessive-Compulsive Scale (CY-BOCS; Scahill et al., 1997); and the Vineland Adaptive Behavior Scales (VABS; Sparrow, Balla & Cicchetti, 1984). Because children’s self-perceptions have been found to be a mediator of CBT outcome in the treatment of child anxiety disorders, items relevant to self-efficacy and coping were developed in reference to this body of literature (e.g., Muris, 2001; Treadwell & Kendall, 1996). In the case of parenting targets, parenting constructs associated with children’s treatment outcome in clinical trials of cognitive behavioral interventions for general child mental health outcomes were reviewed (e.g., intrusiveness and warmth; Gardner et al., 2006; Wood, McLeod, Piacentini, & Sigman, 2009). Following the initial review of the literature and review of the session transcripts, the first author drafted 29 items relevant to each target construct in terms that were relevant to the content of CBT sessions with children with ASD and their parents. Feedback was provided on inclusion of the items on the PASTOS by five experts in ASD and closely related areas of research (tenured faculty at UCLA and California State University, Los Angeles). Items were refined iteratively based on initial feedback, revision, a second round of feedback, and final revisions. The items were conceptually grouped together into subscales by the authors and checked with tests of internal consistency (see Results). Of the 29 original items, 23 were ultimately retained: 4 items were removed because they focused on the inverse of positive parenting and were essentially duplicative, and 2 items were discarded due to poor interrater reliability (see Results). Abbreviated item titles are presented in Table 2.

Table 2.

Means and Standard Deviations for PASTOS Individual Items and Scales

PASTOS Scale / Item	Session 2 M (SD)	Session 4 M(SD)	Session 10 M (SD)	Session 15 M (SD)
PASTOS Total	77.27 (10.92)	86.55 (12.87)	94.77 (16.30)	103.41 (18.03)
Social Communication	38.61 (6.16)	43.43 (6.82)	45.81 (8.44)	50.07 (8.30)
Friendships	1.86 (1.18)	1.93 (1.28)	3.00 (1.79)	4.05 (1.99)
Positive Affect	3.39 (0.89)	3.84 (1.22)	4.02 (1.38)	4.39 (1.24)
Speech Dysfluencies	2.20 (1.70)	1.85 (1.10)	1.31 (1.00)	1.26 (0.73)
Child Use of Humor	1.98 (1.23)	2.34 (1.56)	1.95 (1.60)	2.50 (1.86)
Emotion	4.02 (1.06)	4.70 (0.84)	4.64 (0.97)	4.64 (1.26)
Awareness of others	3.50 (1.00)	4.14 (1.49)	4.07 (1.26)	4.25 (1.33)
Initiations	3.48 (0.99)	4.57 (1.21)	4.50 (1.44)	5.16 (1.36)
Expected responses	4.18 (1.23)	4.75 (1.15)	4.98 (1.30)	5.59 (1.02)
Unexpected responses	3.00 (1.84)	2.45 (1.34)	1.66 (0.86)	1.30 (0.57)
Repetitive topics	2.57 (1.96)	2.43 (1.78)	2.10 (1.46)	1.71 (1.28)
Anxiety and Related Emotional States	24.91 (10.46)	25.95 (10.82)	19.64 (9.54)	15.63 (7.24)
Obsessive-Compulsive	1.71 (1.65)	2.14 (1.59)	2.45 (2.01)	2.23 (1.97)
Fear of Harm	5.14 (1.39)	5.04 (1.91)	3.47 (1.74)	2.60 (1.93)
Fear of Negative Evaluation	4.31 (2.02)	4.45 (2.11)	2.71 (1.65)	2.04 (1.36)
Self –Deprecation	2.65 (2.25)	2.38 (2.10)	2.33 (2.10)	2.32 (1.81)
Academic Worries	3.35 (2.21)	2.80 (2.29)	2.00 (1.67)	1.71 (1.55)
Perfectionism	3.38 (2.29)	3.32 (2.21)	2.71 (1.90)	2.14 (1.62)
Physical Symptoms of Anxiety	4.42 (1.50)	4.09 (1.77)	2.60 (2.23)	1.14 (0.35)
Child Awareness of Stigma	2.47 (1.65)	2.76 (2.09)	2.32 (2.08)	1.95 (1.70)
Self-Help Skills Mastery	1.61 (0.75)	2.89 (1.41)	2.75 (1.70)	3.39 (1.89)
Child Self-Perception	4.20 (1.44)	7.41 (2.44)	8.27 (2.78)	9.20 (3.64)
Child Coping Skills	2.23 (0.92)	4.25 (1.16)	4.25 (1.38)	4.59 (1.94)
Child Confidence	1.98 (0.88)	3.16 (1.49)	4.02 (1.53)	4.61 (1.88)
Parent-Child Interactions	13.39 (5.67)	17.18 (5.90)	25.50 (9.35)	13.27 (4.21)
Parental Sensitivity	1.86 (0.79)	2.50 (1.46)	2.86 (1.54)	4.02 (1.80)
Parental Warmth	1.91 (0.83)	2.59 (1.52)	3.23 (1.70)	4.27 (1.78)

The 23 PASTOS items were defined with a brief title followed by a concise item description and one or more examples. For example, for the item Repetitive Topics, the PASTOS manual describes the target behavior as “Child steers conversation back to favorite topic by jumping to it with little regard for the listener and minimal transition words.” The example given for Repetitive Topics in the manual is: “Therapist asks, ‘So what might this girl [in the cartoon] be thinking?’ Child jumps in with a comment like, ‘You will do my homework or Biff will steal Marty’s car!’ (from the movie Back to the Future, which was a repetitive topic for the child throughout the session).”

The 23 items were grouped into subscales corresponding to the five target constructs assessed by the PASTOS. The subscales (and corresponding items) are: (a) Social Communication (items: Friendships, Positive Affect, Speech Dysfluencies, Child Use of Humor, Emotion, Awareness of Others, Initiations, Expected Responses, Unexpected Responses, and Repetitive Topics), (b) Anxiety and Related Emotional States (items: Obsessive Compulsive, Fear of Harm, Fear of Negative Evaluation, Self-Deprecation, Academic Worries, Perfectionism, Physical Symptoms of Anxiety, Child Awareness of Stigma), (c) Self-Help Skill Mastery (item: Mastery of Self-Help Skills), (d) Child Self-Perceptions (items: Child Confidence, Child Coping Skills), and (e) Parent-Child Interactions (items: Parental Sensitivity, Parental Warmth).

The scoring strategy for the PASTOS was adapted from treatment integrity research (Elkin, Pilkonis, Docherty & Sotsky, 1988; Hill, O’Grady & Elkin, 1992; Hogue, Liddle & Rowe, 1996; McLeod & Weisz, 2010; Shapiro & Startup, 2010; Waltz, Addis, Koerner & Jacobson, 1993), in which therapists’ behaviors during treatment sessions are rated systematically. The PASTOS rating system uses the metric developed for the Therapy Process Observational Coding System for Child Psychotherapy Strategies scale (TPOCS-S; McLeod & Weisz, 2010), which employs extensiveness ratings for each item using a 1 to 7 scale, with 1= not at all, 3= some, 5= considerably, and 7= extensively. In the case of the PASTOS, the extensiveness rating scale was used to measure the extent to which a particular child behavior or parent behavior described in each of the 23 items was exhibited during a treatment session. For example, if a child was adept at recognizing emotions in others throughout a treatment session or extensively in a portion of the session, then an extensiveness rating of 6 or 7 would be given on the relevant PASTOS item. PASTOS items are defined in a coding manual developed by the authors, which is available upon request.

Five undergraduate coders were trained over a 9-month period to reach adequate pre-study reliability (ICC>.59; Cicchetti, 1994). Training consisted of reading the PASTOS training manual, listening to audiotapes, reviewing items, and scoring exemplar sessions. Once adequate reliability on the PASTOS had been achieved, scoring for the study began, with tapes being randomly assigned to the five coders with regular reliability assessments being performed every week. Percent agreement was maintained at 70% or greater throughout coding. The results were discussed in weekly meetings to prevent rater drift (Margolin et al., 1998). Coders listened to session audiotapes while concurrently reading a written transcript of the session. Coders rated all portions of the treatment sessions in which the child was present, which ranged in length from 30 to 65 minutes (M = 45, SD = 8.45). Coders took notes on the transcripts as they listened. At the conclusion of an audiotaped session, the coders assigned an extensiveness rating (ranging from 1 to 7) for each of the 23 PASTOS items. The coders were unaware of treatment session numbers.

The Social Communication, Self-Help Skill Mastery, Child Self-Perception, and Parent-Child Interactions subscales were scored so that higher values reflect a greater level of the specific construct (e.g., higher Social Communication subscale scores reflect greater use of appropriate social communication skills by the child during a treatment session, such as increased talk of friendships, more use of humor, an increase in a child’s prosodic variation as well as increased enthusiasm in their voice- increased affect). The Anxiety and Related Emotional States subscale was scored so that higher values reflected greater symptomatology or impairment in its specific indicators (e.g., greater obsessive-compulsive behaviors displayed, greater expression of academic worries). The Total PASTOS score was an aggregate of all the child-behavior related subscales (note: parenting subscales were not included in this total score). Higher Total PASTOS values reflect better overall functioning (i.e., items in the Anxiety and Related Emotional States subscale were reverse scored before being added to the Total PASTOS score).

Other measures.

The Multidimensional Anxiety Scale for Children (MASC; March, 1998) is a 39-item, 4-point Likert self-report scale that demonstrates robust psychometric properties in typically developing children (March, Parker, Sullivan, Stallings, & Connors, 1997). The parent version was administered in this investigation. Alphas were .88 for the parent MASC total score at pre-treatment and .82 at post-treatment.

The Social Skills Rating System (SSRS; Gresham & Elliott, 1990) is an informant-report measure that was originally standardized on over four thousand typically developing children and their parents. In this study, the parent-report form was used. Social skills were rated on a 3-point Likert scale. There are five subdomains that make up the SSRS: (1) Cooperation, (2) Assertion, (3) Responsibility, (4) Empathy, and (5) Self-Control. The SSRS has established reliability and validity in typically developing samples (Gresham, Elliott, Cook, Vance & Kettler, 2010; Pedersen, Worrell, & French, 2001). Alphas at pre-treatment were: SSRS-Total, .69, Cooperation, .68, Assertion, .54, Responsibility, .47, and Self-Control, .72. The alphas at post-treatment were: SSRS-Total, .78, Cooperation, .76, Assertion, .79, Responsibility, .56, and Self-Control .75.

The Self-Efficacy Questionnaire for Children (SEQ-C; Muris, 2001) contains 24 items that represent multiple domains of child self-efficacy (e.g., social, academic, and emotional self-efficacy). Each item is scored on a five-point scale. The SEQ-C has demonstrated good psychometric properties in typically developing children (Muris, 2001). In the current study, the parent-report version was used. Alphas were .74 for the total score at pre-treatment and .85 at post-treatment.

The Clinical Global Impressions-Improvement Scale (CGI-I; Guy, 1976) is an independent evaluator-rated scale used to assess the severity of clinical symptoms after treatment. In the Wood, Drahota, Sze, Har and colleagues (2009) study, the CGI-I was rated by an independent evaluator blind to randomization status following the administration of the Anxiety Diagnostic Interview Scale: Child and Parent Versions (ADIS; Silverman & Albano, 1996) at the end of treatment (session 16; see Wood, Drahota, Sze, Har et al., 2009). It provides a global rating of clinical improvement ranging from a score of 1 (completely recovered) to 5 (no change) to 8 (very much worse). The CGI was a primary outcome measure in Wood, Drahota, Sze, Har, et al. (2009).

Data Analysis

To evaluate the four study hypotheses, a set of planned analyses was conducted, followed by several exploratory analyses conducted to further probe the primary findings. First, interrater reliability was calculated for each of the 23 PASTOS items using the intraclass correlation coefficient (ICC; Shrout & Fleiss, 1979) using a two-way mixed effects model where person effects are random and measure effects are fixed. The ICC provides an estimate of the ratio of the true score variance to total variance. Based upon Cicchetti (1994), ICC values that fall below .40 reflect “poor” agreement, ICCs that equal .40 to .59 reflect “fair” agreement, ICCs that equal .60 to .74 reflect “good” agreement, and ICCs that equal .75 and above are considered “excellent” agreement. Internal consistency of the PASTOS subscales and Total score was calculated using Cronbach’s alpha. A minimum threshold of alpha=.60 was used to retain a scale.

Second, the degree of overlap between the PASTOS subscales and the informant-report scales—the MASC, SSRS, and the SEQ-C—was calculated to examine convergent and discriminant validity using Pearson correlation matrices. PASTOS scores from an early treatment session (session 2) were correlated with MASC, SSRS, and SEQ-C scores from the pretreatment assessment, and PASTOS scores from a late treatment session (session 15) were correlated with MASC, SSRS, and SEQ-C scores from the posttreatment assessment for this set of analyses. The correlations were interpreted following Rosenthal and Rosnow’s (1984) guidelines: r is “small” if 0.10–0.23, “medium” if 0.24–0.36, and “large” if > 0.36.

Third, to assess whether PASTOS scores declined from early treatment to late treatment, repeated measures ANOVAs were used to assess the effect of time (modeled as session number) on the five PASTOS subscales.

Fourth, to examine the predictive power of PASTOS scores in reference to treatment response on the CGI-I, simple regressions were used in which change in each PASTOS score from session 2 to session 15 was used to predict CGI-I scores at post-treatment.

Results

Descriptive statistics for the four treatment session points on all PASTOS items and subscales, as well as the pre and post-treatment data for the other study variables (e.g., MASC), are presented in Tables 2 and 3.

Table 3.

Means and Standard Deviations for MASC-P, SSRS, and SEQ-C

Scale	Pre-treatment M (SD)	Post-treatment M (SD)
MASC-P Total	69.74 (16.31)	54.90 (13.38)
SEQ-C Total	53.05 (7.61)	62.39 (11.44)
SSRS Total	49.44 (7.70)	53.23 (10.05)
SSRS-Cooperation	6.34 (2.90)	8.41 (3.36)
SSRS-Assertiveness	7.90 (2.65)	9.90 (3.77)
SSRS-Self-control	8.08 (3.16)	9.41 (3.70)
SSRS-Responsibility	8.35 (2.604)	10.43 (3.60)

Means and Standard Deviations for MASC-P, SSRS, and SEQ-C

Interrater Reliability and Internal Consistency

Two measures of reliability were computed for the PASTOS: (a) interrater reliability amongst coders for each PASTOS item and subscale, and (b) internal consistency for each PASTOS subscale and Total score. For 20% of the 88 sessions (18 sessions total), two coders rated the same session to test for interrater reliability on the PASTOS items. The ICCs for the 23 PASTOS items were computed for each individual item and each of the five subscales. ICCs ranged from .56 to .96 (see Table 4).

Table 4.

Interrater Reliability for the Individual PASTOS Items

PASTOS Items	ICC	Agreement
PASTOS Total	.79	EXCELLENT
Social Communication	.77	EXCELLENT
Friendships	.93	EXCELLENT
Positive Affect	.82	EXCELLENT
Speech Dysfluencies	.61	GOOD
Child Use of Humor	.69	GOOD
Emotion	.62	GOOD
Awareness of Others	.76	EXCELLENT
Initiations	.56	FAIR
Expected Responses	.75	EXCELLENT
Unexpected Responses	.90	EXCELLENT
Repetitive Topics	.60	GOOD
Anxiety and Related Emotional States	.70	GOOD
Obsessive Compulsive	.58	FAIR
Fear of Harm	.90	EXCELLENT
Fear of Negative Evaluation	.75	EXCELLENT
Self-Depreciation	.91	EXCELLENT
Academic Worries	.79	EXCELLENT
Perfectionism	.74	GOOD
Physical Symptoms of Anxiety	.86	EXCELLENT
Child awareness of Stigma	.89	EXCELLENT
Self-Help Skills Mastery	.96	EXCELLENT
Child Self –Perception	.78	EXCELLENT
Child Coping Skills	.91	EXCELLENT
Child Confidence	.62	GOOD
Parent-Child Interactions	.89	EXCELLENT
Parental Sensitivity	.91	EXCELLENT
Parental Warmth	.93	EXCELLENT

Internal consistency of the four PASTOS subscales with multiple items was found to be acceptable: (a) Social Communication (α = .84); (b) Anxiety and Related Emotional States (α = .65); (c) Child Self-Perception (α = .78); (d) Parent-Child Interactions (α = .80); and (e) Total score (α = .78). Note that the Self-Help Skills Mastery domain consisted of a single item.

Next, the degree of overlap among the PASTOS subscales was assessed (n = 22). Pearson intercorrelations were calculated for the five PASTOS subscales. Correlation coefficients are presented in Table 5. Statistically significant correlations were found between the five subscales; correlation magnitudes ranged from medium to large.

Table 5.

Intercorrelations for PASTOS Subscales

PASTOS Subscales	1	2	3	4
1. Self-Help skill mastery	___	−.22	.44*	.44*
2. Anxiety and related emotional states	−.22	___	−.64**	−.51*
3. Social Communication	.44*	−.64*	___	.85**
4. Child Self-Perception	.44*	−.51*	.85*	___
5. Parent-Child interactions	.50*	−.53*	.79**	.80**

Note.

^*Correlation is significant at the 0.05 level (2-tailed).

^**Correlation is significant at the 0.01 level (2-tailed).

Convergent and Discriminant Validity

The Total PASTOS score at session 2 negatively correlated with total SSRS scores (r = −.48, p = .02), indicating lower parental ratings of children’s social skills were associated with better overall functioning according to PASTOS ratings at pre-treatment (i.e., contrary to expectations). Non-significant correlations were obtained between Total PASTOS scores at session 2 and the total MASC-P and SEQ-C parent-report scores from the pre-treatment assessment. Hence, there was no evidence of convergent validity for the PASTOS Total score at pre-treatment.

At post-treatment, PASTOS Total scores at session 15 significantly correlated with total MASC-parent report scores (r = −.53, p = .01) and total SEQ-C parent-report scores (r = .56, p = .01), in the expected direction. When parents reported less child anxiety and more child self-efficacy, PASTOS Total scores were higher at post-treatment, as would be expected.

Exploratory subscale correlational analyses were also conducted, although few significant findings were obtained at pre-treatment (corresponding to PASTOS ratings at treatment session 2). At the subscale level at post-treatment, the SSRS Self-Control subscale significantly correlated with the following PASTOS session 15 scores: Total score (r = .53, p = .01), Social Communication subscale (r =.50, p = .02), Self-Help Skill Mastery subscale (r = .50, p = .02), and Child Self-Perception subscale (r = .51, p =.02). This reflects a moderate positive relationship between parent-reported ability to display appropriate behaviors in conflict (e.g., responding to teasing) and cooperative play (e.g., taking turns), and children’s overall functioning observed by PASTOS coders at post-treatment. Similarly, post-treatment SEQ-C scores significantly correlated with the following PASTOS session 15 scores: Child Self-Perception subscale (r = .57, p = .01), Social Communication subscale (r = .61, p = .01), and Parent-Child Interactions subscale (r = .52, p = .02), all large effects. Higher ratings of children’s self-efficacy were related to higher ratings of children’s social communication skills, self-concept, and parent-child communication as assessed on the PASTOS at session 15. Hence at the Total score and subscale level, there was evidence of convergent validity for the PASTOS scales, measured late in treatment, and the informant-based measures from the posttreatment assessment. Evidence of discriminant validity among the PASTOS subscales was limited.

Sensitivity to Treatment

To examine clinical change over the course of treatment based on PASTOS scores, repeated measures ANOVAs were estimated using the five PASTOS subscales. Results show that over the course of 16 sessions of CBT (sampled from 4 sessions per child), all scores improved in the expected direction: Social Communication, F = 18.40, p < .01; Anxiety and Related Emotional States, F = 9.10, p < .01; Child Self-Perception, F = 26.89, p < .01; Self-Help Skill Mastery, F = 6.89, p < .01; and Parent-Child Interactions, F = 18.32, p < .01. Hence, these analyses were consistent with linear improvement in skills and decline in symptoms over the course of 16 sessions of treatment.

Lastly, OLS regressions were conducted in which change in each PASTOS subscale was used to predict CGI-I scores at post-treatment. The change in PASTOS Social Communication subscale scores from session 2 to session 15 significantly predicted the post-treatment CGI-I score (n = 22, β = −.57, p < .01). Parallel analyses for the other PASTOS subscales yielded similar, significant findings: post-treatment CGI-I scores were predicted by improvements in the PASTOS Child Self-Perception subscale (n = 22, β = −.54, p < .01), PASTOS Parent-Child Interactions subscale (n = 22, β = −.48, p < .02) and PASTOS Self-Help Skill Mastery subscale (n = 22, β = −.54, p < .01). The change in Total PASTOS yielded similar results (n = 22, β = −.53, p < .01). Hence, improvement over the course of treatment in four of the PASTOS subscales and the Total PASTOS score were predictive of better CGI-I scores at posttreatment (in the expected direction).

Discussion

In this study, a quantitative coding system was developed to assess clinical improvement in skills and symptomatology during psychotherapy for children with ASD. Because informant report measures can be biased, the PASTOS endeavored to leverage observational methodology (in this case, using session audiotapes) to attain independent evaluator ratings of a wide range of clinical outcomes in children with ASD. The interrater reliability and internal consistency for PASTOS scores was promising. There was evidence supporting the convergent validity of PASTOS scores at the late-treatment period when using total scores for the PASTOS from session 15 and comparison scales obtained at the post-treatment assessment. However, subscale analyses revealed little evidence of discriminant validity, and convergence was also negligible at the early treatment (session 2) time point. Two different types of analyses suggested that the PASTOS may be sensitive to clinical change and improvement. Overall, a promising psychometric pattern emerged although further refinement and exploration may help address the limitations noted in convergent and discriminant validity, particularly early in treatment.

In terms of interrater reliability and internal consistency, PASTOS subscale scores and the Total score were all in the acceptable range. All PASTOS individual items had at least “fair” interrater reliability, with most in the good range or better, whilst all PASTOS subscales and the Total score yielded interrater reliability estimates in the acceptable range. Hence, trained undergraduate raters were able to learn and maintain adequate consistency of ratings amongst themselves on the PASTOS observational system, and their ratings achieved internal consistency comparable to other treatment observational process instruments (e.g., McLeod & Weisz, 2010).

The analysis of convergent and discriminant validity yielded a more complex pattern, with strong evidence of convergent validity at the Total score level, and modest evidence of convergent validity at the subscale level, but at the late-treatment period (session 15) only. Discriminant validity, as would be demonstrated by stronger correlations within domain than between domains (e.g., PASTOS Social Communication subscale with SSRS scores as compared to MASC scores) was not clearly in evidence for the PASTOS scores in this study. With regard to the lack of convergence of early treatment (session 2) PASTOS scores with informant-report measures obtained at pre-treatment, this may be related to the high level of structure that typifies early treatment sessions. Early in treatment in CBT, therapists focus on coping with anxious states before the graduated exposure component of CBT begins. Because children are asked to disclose anxieties and other concerns at this time, it is possible that there is less meaningful variability among children at this time point based on PASTOS-type ratings of symptomatology than there is during the less-structured and more individualized second half of treatment which focuses on exposure therapy, social skills, and contingency management to address specific areas of clinical concern. It is possible that as an outcome measure, PASTOS scores from near the end of treatment (e.g., session 15) may be more accurate than those from early in treatment in terms of between-person variability. The lack of discriminant validity attained in these analyses also suggests that, pending further research, the PASTOS total score may be the most meaningful score to use, reflecting a global improvement (or lack thereof) in skills and symptomatology.

In terms of sensitivity to treatment, change in PASTOS scores over the course of treatment were found to predict CGI-I ratings made by independent evaluators. This indicates that children whose skills and symptomatology as broadly assessed by the PASTOS improved over the course of treatment had better treatment outcomes on a primary outcome measured used in the Wood, Drahota, Sze, Har and colleagues (2009) trial as well as other pediatric clinical trials. In conjunction with the linear improvement in skills and decline in symptoms seen on PASTOS scores over the course of CBT, these findings suggest that the PASTOS system may be capable of tracing intra-individual improvement during psychotherapy among children with ASD, potentially offering a sensitive tool for assessing treatment response for future clinical trials research on psychotherapy models in this population.

Interestingly, those parents who increased their use of a positive style of communication had children with the best CGI-I scores at post-treatment. The current study is one of the first to link changes in parental communication style to gains in treatment for children with ASD. Research has shown that changes in positive parenting skills significantly mediate change in problem child behavior in children with conduct disorder (Gardner et al., 2006). The program used in Gardner and colleagues’ study (2006) was based on a collaborative model between parent and therapist that built on the parent’s strengths and expertise. The components that made up the program included parent-child play, praise, incentives and rewards, problem-solving with child, limit setting, and discipline. Many of these components of parenting skills were also used in the BIACA CBT program implemented in the present study. This finding may illustrate the value of treatments that include parental psychoeducation and training.

A number of limitations are now addressed. One potential limitation was that PASTOS scores were based on audiotape recordings rather than videotape recordings, potentially raising concerns about how to code items such as “parental warmth”, “positive affect”, and “emotion”. Audiotaping was used because the children who came into the Wood, Drahota, Sze, Har et al. (2009) RCT were too anxious to tolerate videotaping. For the coding of emotion, parental warmth, and affect, scores were based on the participants’ vocal quality such as range in prosody and the speech content. For example, coders were able to distinguish between happy/positive voices versus the inverse. Nonetheless, even more refined assessments of these categories could be made using video recordings.

The repetitive topics code that was used to address restrictive interests talked about by a child during conversational exchanges, but not other repetitive behaviors, such as repetitive body movements or facial tics, would be better measured using a video recorder and use of two coders.

Another important caveat in interpreting these findings is that the sample size was small and may not be representative of other children with ASD seeking treatment. Although raters were not told about study hypotheses or the session numbers they were rating, it is possible that this was discerned by coders based on children and therapist discourse during treatment sessions (e.g., “this is one of the last times I’ll see you”), adding an unmeasured possibility of bias into the findings. One area of future research that may prove fruitful would be to investigate parental psychopathology and its effects on PASTOS scores. Since BIACA and other treatment models for children rely heavily on parent involvement, parental behavior and attitudes are likely to play a role in how well they are able to learn and implement support strategies that may mediate treatment outcome for children with ASD. Additionally, the coded PASTOS sessions (e.g., at sessions 2 and 15) did not correspond exactly to the timing of the pre- and post-treatment assessments such as the MASC and CGI-I, making the validity analyses somewhat more imprecise than if the sessions and informant-based measures had corresponded precisely in timing.

Another limitation of the study is that training coders to reliability took 9 months, and coding entire sessions of treatment is time-consuming. While efficiencies in training are likely to be attainable once a research group gains coding mastery, the PASTOS assessment procedure is likely only feasible for research purposes and not as a practical tool for use in community settings. The PASTOS was developed as a complement to other assessments of treatment process and outcome for children with autism, not as a primary outcome measure. Relatedly, while no measure may lay claim to complete objectivity, the PASTOS has the potential advantage of being rated by independent evaluators who may be less biased in their ratings than the intervention participants themselves.

The PASTOS is the first known observational coding system to assess clinical progress during psychotherapy for school-aged children with ASD and to offer an alternative to typical pre- and post-treatment symptom comparison based on informant reports. Assessing clinical changes throughout treatment based on direct observations and ratings of therapy sessions appears to be an achievable goal for this population, although there is clearly a need for continuation of research and refinement of the coding instrument to assess whether divergent validity of the subscales might be improved through further revisions and whether the measure may be redesigned moderately to achieve convergent validity if used at pretreatment. For the present, it is advised that the PASTOS Total score derived primarily from sessions closer to post-treatment is the most conservative and empirically supported use of this instrument for assessing treatment outcome in children with ASD who have participated in psychotherapy.

Summary

In this study, the psychometric properties and practical uses of an observational coding system to track clinical progress in children with ASD during psychotherapy were evaluated. The Pediatric Autism Spectrum Therapy Observation System (PASTOS) was used to rate clinically-relevant child behaviors in individual psychotherapy sessions. Four sessions per participant from a 16-session CBT intervention were rated with the PASTOS. Coders were able to achieve good interrater reliability and the items underlying the PASTOS subscales showed acceptable internal consistency. Convergent validity for the PASTOS scales was evident in relation to post-treatment outcome measures, though not in relation to pre-treatment scores on these measures. Additionally, the PASTOS exhibited treatment sensitivity insofar as scores improved on average over the course of the intervention and change in PASTOS scores over treatment predicted positive treatment response on the CGI-I. The PASTOS may be a useful tool for studying process and outcome in psychotherapy research with children diagnosed with ASD.