Informant (Dis)Agreement on Ratings of Challenging Behaviors and Social Communication in Preschool Children With Autism Spectrum Disorder

Abstract

Purpose

Cross-informant ratings are considered best practice for assessing children with autism spectrum disorder (ASD). However, informant disagreement often occurs, which can pose significant challenges to various aspects of clinical services. This study explored the degree of parent and speech-language pathologist (SLP) agreement on ratings of challenging behaviors and social communication skills in preschool children with ASD.

Method

Fifty-eight informant ratings of challenging behaviors and social communication skills were collected from parents and SLPs on the same 29 preschool children with ASD (M = 49.93 months, SD = 11.67 months) using the Pervasive Developmental Disorder Behavior Inventory. Parent versus SLP group rating comparisons were assessed with paired t tests and Cohen's d effect sizes. Intraclass correlation coefficients were calculated to examine interrater reliability between individual parent and SLP ratings. Bland–Altman plots were generated to evaluate informant agreement across the entire range of Pervasive Developmental Disorder Behavior Inventory composite scores.

Results

Group comparisons indicated that parents rated arousal regulation problems as more severe than SLPs, with no other group differences observed. Parents and SLPs exhibited poor agreement on ratings of challenging behaviors; however, moderate to good agreement was observed for social communication ratings.

Conclusions

These results highlight the importance of including parents in the assessment and treatment planning process for preschool children with ASD, as parents may report key behavioral concerns that clinicians may not otherwise observe. Understanding behaviors that may be more prone to informant disagreement has implications for promoting a shared understanding of behavioral concerns and treatment targets between parents and clinicians.

Children with autism spectrum disorder (ASD) exhibit substantial heterogeneity in their behavioral and social communication profiles, often demonstrating behaviors differently across contexts (Hartley et al., 2011; Kanne et al., 2009). To effectively capture variability in the ASD phenotype, it is best practice to collect and integrate behavioral observations from multiple informants during the assessment of young children with ASD (Ozonoff et al., 2005). However, informant disagreement often occurs, which can pose significant challenges to various aspects of clinical services for young children with ASD (De Los Reyes & Kazdin, 2005; Hartley et al., 2011).

Informant-based rating scales and checklists are administered to collect observations from multiple informants to obtain a comprehensive understanding of child symptom presentation. These tools often afford quick, cost-effective, and valid estimates of child behavioral functioning (De Los Reyes et al., 2015; Norris & Lecavalier, 2010). However, studies examining informant agreement on challenging behaviors in school-age children with ASD (i.e., behavioral and emotional difficulties such as aggression, anxiety, depression, oppositionality, self-regulation, and attention difficulties) have generally found low to moderate agreement, with parents rating behaviors as more problematic than teachers (Jordan et al., 2019; Lane et al., 2013; Stadnick et al., 2017; Stratis & Lecavalier, 2015). In contrast, informant discrepancies on ratings of social communication behaviors in youth with ASD (i.e., use of verbal and nonverbal communication to initiate and respond to others for social purposes) are somewhat more variable. For instance, Donnelly et al. (2018) found low to moderate informant agreement in a sample of children with ASD (6–12 years), with parents rating social skills as more severely impaired than teachers. However, in one of the only studies on informant discrepancies in preschoolers with ASD, moderate parent–teacher correspondence was found on ratings of social skills (Thompson & Winsler, 2018), with parents endorsing higher social skills compared to teachers.

While general patterns of informant discrepancies are beginning to emerge in the literature, most studies have focused on school-aged children with ASD (Donnelly et al., 2018; Lopata et al., 2016; Ung et al., 2017) or wide age ranges (e.g., Levinson et al., 2020). In turn, results cannot be assumed to directly apply to preschoolers with ASD. Given prior reports of substantial informant disagreement among young children without ASD (e.g., Winsler & Wallace, 2002), it is particularly critical to address this gap in the literature. From a developmental perspective, poor informant disagreement can have adverse cascading effects on diagnostic consensus, access to early intervention, treatment planning, and progress monitoring for young children with ASD (Hawley & Weisz, 2003). For instance, a recent study found that higher parent–teacher disagreement on behavior problems in children with ASD was associated with lower parent–school involvement in the following school semester (Levinson et al., 2020).

The Attribution Bias Context Model (ABC Model; De Los Reyes & Kazdin, 2005) is a theoretical framework developed to facilitate a better understanding of informant discrepancies among different types of informant pairs (parent–teacher, teacher–teacher, parent–parent, parent–child, etc.). The ABC Model suggests that informant discrepancies occur because informants may differ in (a) what they attribute to being the cause of the child's behavior (i.e., disposition of the child versus contextual factors), (b) perspectives on what behaviors require treatment (e.g., an informant may overestimate challenging behaviors because this is the behavior the informant wishes to be targeted in treatment), and (c) the context where they observe or report behaviors (e.g., home vs. clinical setting). Based on the ABC Model, parent–teacher informant discrepancies may reflect differences in informant perspectives due to different “decision thresholds” for determining whether a behavior is problematic or not based on the background and previous experience of the informant (De Los Reyes, 2013). For example, parents may rate child behavior with reference to other children in the family. In contrast, teachers and clinicians may rate the child's behavior with reference to other children they have taught, evaluated, or treated. To date, informant discrepancies for ratings of children with ASD have mainly been examined between parents, teachers, and clinicians (i.e., developmental pediatricians, child psychologists, and child psychiatrists; e.g., Levinson et al., 2020; Sacrey et al., 2018). However, there is limited understanding of informant discrepancies that may arise between parents and other clinical providers, such as speech-language pathologists (SLPs). This is a significant gap in the literature as SLPs are core members of the ASD assessment and treatment team, providing services that cut across both the clinic and the classroom (Philofsky, 2008; Prizant, 2017). Given the unique clinical training and perspective of SLPs, examining informant discrepancies between parents and SLPs may provide insight into clinically informative types of discrepancies that may arise within this specific informant pairing.

One reliable and valid informant-based rating scale is the Pervasive Developmental Disorder Behavior Inventory (PDDBI). The PDDBI measures challenging behaviors and social communication specific to the ASD phenotype (Cohen & Sudhalter, 2005) and challenging behaviors that are not specific to ASD (e.g., aggressiveness, specific fears, and arousal regulation problems). Several studies have validated the clinical utility of the PDDBI for determining behavioral profiles of children with ASD to guide treatment decisions (Cohen & Flory, 2019; Schutte et al., 2019). The PDDBI has also been used as an outcome measure in quasi-experimental and randomized controlled intervention trials (Holzinger et al., 2019; Mankad et al., 2015) and has been included in an extensive phenotyping battery for an ongoing biomarker qualification study (McPartland et al., 2020). Given the strong psychometric properties of the PDDBI and its widespread use in the field, it is important to understand the agreement between raters. To date, informant agreement on the PDDBI has only been reported in the manual, indicating poor-to-moderate parent–teacher interrater reliability for challenging behaviors (rs = .24–.57; Approach/Withdrawal domain), with stronger interrater reliability observed for social communication abilities (rs = .51–.82; Receptive/Expressive Social Communication Abilities [REXSCA] domain; Cohen & Sudhalter, 2005). However, no data have been presented beyond the manual on parent–teacher interrater reliability for the PDDBI. Understanding which behaviors are potentially more prone to informant disagreement on the PDDBI has implications for promoting a shared understanding of child strengths and behavioral concerns, which can lead to collaborative intervention goal setting between parents and providers using this measure.

The Current Study

The current study examines the degree of parent and SLP agreement on ratings of challenging behaviors and social communication skills, as measured by the PDDBI, in children with ASD enrolled in an ongoing group-based naturalistic developmental behavioral intervention (NDBI; Landa et al., 2011). This study extends previous literature on informant ratings of young children with ASD by (a) focusing on preschool-age children with ASD; (b) examining informant discrepancies between caregivers and certified and licensed SLPs, an informant pair not previously empirically examined; and (c) using a rating scale that measures behaviors both specific and nonspecific to ASD (e.g., aggressiveness, specific fears, and arousal regulation problems).

To our knowledge, this is the first study to examine parent–SLP agreement on the PDDBI in preschool children with ASD. As a result, the hypotheses of this exploratory study are based on previous literature concerning parent–teacher agreement on the PDDBI (from the manual) and similar rating scales of social communication and behavioral concerns in children with ASD, as reviewed above. Specific research questions and exploratory hypotheses are as follows:

1. How will informants (parents or SLPs) differ on mean ratings of challenging behaviors and social communication skills? We hypothesized that parents would report higher mean levels of challenging behaviors (Jordan et al., 2019; Lane et al., 2013; Lopata et al., 2016) and higher mean levels of social communication skills (Thompson & Winsler, 2018) compared to SLPs.

2. Is parent–SLP agreement stronger for challenging behaviors or social communication skills? Based on initial PDDBI parent–teacher interrater reliability findings (Cohen & Sudhalter, 2005) and a recent investigation on parent–teacher agreement in a diverse sample of preschoolers with ASD (Thompson & Winsler, 2018), we hypothesized that parents and SLPs would show poor agreement on ratings of challenging behaviors, with stronger agreement on ratings of child receptive and expressive social communication abilities.

Method

Participants

Data for this cross-sectional, retrospective study were obtained from a sample of children with ASD receiving ongoing group-based NDBI services in a simulated preschool classroom setting led by certified and licensed SLPs. These services were provided through an urban, outpatient ASD specialty clinic located in the Mid-Atlantic region of the United States between the years 2016 and 2020. The research was approved by the Johns Hopkins Medicine Institutional Review Board.

Inclusion criteria for our analytic sample consisted of children: (a) whose parents agreed to join the institutional review board–approved research registry; (b) who had completed PDDBI parent and teacher forms; and (c) who had received an ASD diagnosis by a licensed, doctoral-level provider based on the Autism Diagnostic Observation Schedule–Second Edition (ADOS-2) and team clinical judgment, aligned with the Diagnostic and Statistical Manual of Mental Disorders (5th ed.). Based on these inclusion criteria, the final analytic sample consisted of 29 children with ASD, ranging in age from 29.85 to 71.24 months (M = 49.93, SD = 11.67). Children in this sample were predominantly male (83%) and White (52%).

A total of 58 informant PDDBI ratings were collected from SLPs and parents on the same 29 children. The PDDBI teacher forms were completed by 15 female SLPs (age range = 23–39 years [M = 29.98, SD = 4.99]; race/ethnicity: Asian = 1 [7%]; Hispanic/Latino = 1 [7%]; Middle Eastern = 1 [7%]; White = 12 [80%]; years of practice = 1–15 years [M = 5.50, SD = 5.03]). Eleven of the SLPs completed master's degrees at American Speech-Language-Hearing Association (ASHA)–accredited universities, were state licensed, and ASHA certified. Four of the SLPs were state-licensed SLP assistants (bachelor's degree = 3; master's degree = 1) who completed the forms under the supervision of a state-licensed and ASHA-certified SLP. All SLPs were trained to fidelity to implement the NDBI group-based intervention for young children with ASD and were all Practitioner 1–certified in Professional Crisis Management by the Professional Crisis Management Association. Almost all parent forms were completed by mothers (90%). Educational levels were available for 21 of the parents, all reporting at least a high school education. Table 1 presents additional child and parent demographic characteristics.

Table 1.

Descriptive statistics for sample (N = 29).

Variable		M (SD) or n (%)
Age at PDDBI		49.9 (11.7)
Male sex		24 (82.8%)
Race	Asian	6 (20.7%)
	Black	3 (10.3%)
	Hispanic/Latino	2 (6.9%)
	Multiracial/other	3 (10.3%)
	White	15 (51.7%)
Diagnosing provider	MD	20 (69%)
	Psychologist	9 (31%)
Language level	1 - emerging language	17 (58.6%)
	2 - phrase speech	9 (31%)
	3 - fluent sentences	3 (10.3%)
Age at ADOS-2		31.5 (9.5)
ADOS-2 module	Toddler Module	16 (55.2%)
	Module 1	10 (34.5%)
	Module 2	3 (10.3%)
ADOS-2 SA		14.4 (4.5)
ADOS-2 RRB		4.1 (2.1)
ADOS-2 CSS		7.6 (1.8)
Parent informant	Biological father	3 (10.3%)
	Biological mother	26 (89.7%)
Family insurance	Private	22 (75.9%)
	Public	4 (13.8%)
	Other	3 (10.3%)
Parent education a	Bachelors	4 (13.8%)
	Graduate	9 (31%)
	High school	3 (10.3%)
	Trade school or associates	5 (17.2%)

Note. Age = age in months; PDDBI = Pervasive Developmental Disorder Behavioral Inventory; MD = doctor of medicine; ADOS-2 = Autism Diagnostic Observation Schedule– Second Edition; SA = Social Affect; RRB = Restricted and Repetitive Behavior; CSS = Calibrated Severity Score.

^a Eight parents did not report education level.

Measures

Background and History Form

Parents completed this clinic-specific questionnaire upon initiating their child's intake process at the specialty clinic from which they received NDBI group-based services. This form captured parental education information (measured as high school or below, trade school/associates degree, bachelor's degree, or graduate degree). Two additional demographic variables were gathered from the electronic medical record. This included the child's race (Asian, Black/African American, Hispanic, multiracial, White, unknown) and insurance type (private vs. public/Medicaid vs. other). Ethnicity information was gathered under the same question as race and, as a result, could not be identified independently. This form was used for parent and child sample characterization.

ADOS-2

The ADOS-2 is a standardized, semistructured, play-based, clinician-administered measure designed to assess the presence or absence of ASD symptomatology related to communication, social interaction, play, and restricted, repetitive behaviors (Lord et al., 2012). The ADOS-2 consists of different modules, with module selection based on chronological age and language ability at the time of testing. The children in the current investigation were administered the ADOS-2 during their diagnostic evaluation, which occurred before the commencement of group-based NDBI services. The ADOS-2 was administered by a certified and licensed SLP or a licensed psychologist, clinically trained to administer the ADOS-2. Specifically, all clinicians completed a required ADOS-2 clinical training workshop with a certified ADOS-2 trainer prior to the administration of the ADOS-2 in the clinic. Clinicians had access to quarterly booster trainings and research-reliable ADOS-2 clinicians for consultation, if needed. Across all modules, an ADOS-2 Calibrated Severity Score (scores 1–10) was derived, reflecting the relative severity of autism-specific symptoms and allowing comparisons across modules (Esler et al., 2015). Higher Calibrated Severity Scores reflect greater ASD symptom severity. The ADOS-2 was used for child sample characterization.

Natural Language Sample

Prior to the commencement of group-based NDBI services, a 30-min, play-based, natural language sample (NLS) was conducted in a simulated preschool classroom by a certified and licensed SLP. The SLP who administered the NLS at intake was not the same SLP who completed the PDDBI. The NLS was used to measure and estimate child language abilities and consisted of social communicative presses within child–clinician, play-based interactions. The SLP then categorized children into one of three language levels, in line with a proposed developmental language framework (Tager-Flusberg et al., 2009). Level 1 consisted of children with minimal or emerging spoken language, including children who used augmentative alternative communication strategies, word approximations, and some single words to communicate. Children in this group had limited communicative functions, often using their language mainly to request. Level 2 included children who consistently used two-word phrases and had a wide variety of communicative functions, but did not use flexible, conversational speech. These children could answer simple questions but could not comprehend abstract or social questions. Level 3 included children who presented with near typical linguistic abilities (i.e., could speak in sentences using correct grammatical markers) but exhibited challenges in pragmatic and social cognitive skills. Children in the current sample were predominantly rated as having a Level 1 language level (59%), with 31% showing Level 2 skills, and 10% Level 3 skills (see Table 1).

Pervasive Developmental Disorder Behavioral Inventory–Parent and Teacher Extended Forms

The Pervasive Developmental Disorder Behavioral Inventory–Parent (PDDBI-PX) and the Pervasive Developmental Disorder Behavioral Inventory–Teacher (PDDBI-TX; Cohen & Sudhalter, 2005) both measure the same 10 domains of challenging behaviors and social communication skills across two behavioral dimensions: (a) Approach-Withdrawal Problems (AWP) and (b) REXSCA. Table 2 contains brief descriptions of the domain and composite scores for each behavioral dimension.

Table 2.

Pervasive Developmental Disorder Behavioral Inventory (PDDBI) domain and composite descriptions (Cohen & Sudhalter, 2005).

PDDBI domain/composite	Description
Approach/withdrawal problems
Sensory/perceptual approach behaviors	Sensory seeking behaviors, as well as repetitive play and motor movements.
Ritualisms/resistance to change	Ritualistic behavior or difficulty with changes in the environment or routines.
Social pragmatic problems	Challenges understanding social cues, or in social response/initiations.
Semantic/pragmatic problems	Difficulty using spoken language to understand/communicate meaning and engage in social interactions.
Arousal regulation problems	Reduced emotional response, hyperactivity, or problems with sleep.
Specific fears	Fears and wariness associated with withdrawal from social or nonsocial stimuli.
Aggressiveness	Aggressiveness toward self or others and associated negative mood.
AWP/C	Sum of all AWP behavioral domains. Higher scores denote greater behavioral severity.
Receptive/expressive social communication abilities
Social approach behaviors	Nonverbal social communication behaviors, including use of gestures, joint-attention, imitation, and social play.
Expressive language	Use of speech sounds, words, and sentences.
learning, memory, and receptive language	Memory skills and understanding of language.
REXSCA/C	Sum of all REXSCA behavioral domains. Higher scores denote greater social communication ability.

Note. AWP/C = Approach/Withdrawal Problems Composite; REXSCA/C = Receptive/Expressive Social Communication Abilities Composite.

The PDDBI-PX contains 188 items, while the PDDBI-TX contains 180 items. Many items are identical across the two forms. However, the PDDBI-PX items are written to capture behaviors observed in the child's home/daily environment, while the PDDBI-TX items are written to capture behaviors observed in the child's classroom environment. As a result, the PDDBI-PX form includes some items that are not included on the PDDBI-TX and vice versa. For example, within the Arousal Regulation Problems domain, the PDDBI-PX form consists of items related to sleep regulation problems that are not included on the PDDBI-TX form; and within the Learning, Memory, and Receptive Language domain, the PDDBI-TX form consists of items that capture associative learning skills (e.g., matching objects/pictures) that are not included on the PDDBI-PX form. The PDDBI manual specifies that observed discrepancies between the PDDBI-PX and PDDBI-TX forms can offer information regarding the degree that a child's behaviors may be specific to certain settings (Cohen & Sudhalter, 2005). The PDDBI was standardized using a sample of children with ASD (age range: 1.5–12.5 years) and provides age-normed T scores (M = 50, SD = 10) for domain and composite scores within each of the two behavioral dimensions. Test–retest reliability for the PDDBI is acceptable for both the PDDBI-PX (12-month test–retest interval; .64) and the PDDBI-TX (6-month test–retest interval; .87). Both forms have moderate-to-high internal consistency ranging from .80 to .98 for the PDDBI-PX and .81 to .98 for the PDDBI-TX (Cohen & Sudhalter, 2005).

In the current study, parents completed the PDDBI-PX prior to the commencement of group-based NDBI services. The PDDBI manual specifies that other educational professionals with whom the child has had at least 4 weeks of frequent and continuing contact are eligible to complete the teacher form (Cohen & Sudhalter, 2005). Therefore, the SLPs completed the PDDBI-TX after providing ongoing group-based NDBI services to children for approximately 4 weeks, a sufficient amount of time to become familiar with the child's behavioral presentation, meeting administration guidelines. Children participated in the NDBI group-based program for at least 10 months. Therefore, the 4 weeks of therapy did not cover the child's full treatment period.

Statistical Analyses

To examine informant agreement, we implemented a comprehensive statistical analysis plan in line with prior investigations (Donnelly et al., 2018; Jordan et al., 2019; Lopata et al., 2016). First, we examined parent versus SLP group mean differences for the two behavioral composite scores (AWP and REXSCA) and the 10 domain scores (see Table 2) using paired t tests and Cohen's d effect sizes (with 0.2, 0.5 and 0.8 corresponding to small, medium, and large effect sizes, respectively). We then employed intraclass correlation coefficients (ICC) to calculate interrater reliability estimates based on a single rater (k = 2), absolute-agreement, two-way, random-effects model. Given the small sample size, power analyses were carried out for the ICC calculations. Power analyses indicated that the sample size of 29 pairs of raters was sufficient to detect an ICC of .50 at an alpha of .05 with a power of .83 (Zou, 2012). The ICC of .50 was selected as the lower limit based on ICC guidelines suggesting that values less than .50 indicate poor reliability (Koo & Li, 2016). Finally, Bland–Altman plots were created to visualize the level of agreement between parent and SLP ratings and identify systematic differences between parent and SLP ratings (Giavarina, 2015). The difference between each parent–SLP rating pair was plotted as a function of their mean for the two behavioral PDDBI composites. Each plot included one difference value for each participant, a reference line (equal to zero, indicating perfect agreement), and the mean difference between informants (i.e., the mean bias). Limits of agreement (LoA; the mean difference ± 1.96 SD) were drawn to indicate the extent of agreement between informants for approximately 95% of the sample (Hanneman, 2008). In the current study, a positive mean difference indicated more elevated parent-rated composites and vice versa. Good agreement is reflected by a reduced dispersion of values, falling relatively close to the reference line, whereas poor agreement is assumed when difference values are more nonsystematically dispersed away from the reference line (Giavarina, 2015; Hanneman, 2008).

For all statistical analyses, an alpha of p < .05 was considered statistically significant and 95% confidence intervals were estimated. All analyses were carried out in R Studio (Version 1.1.463; R Version 3.6.2, RStudio Team, 2020) using the “stats” (Version 3.6.2), “DescTools” (Version 0.99.38), “ICC.Sample.Size” (Version 1.0), and “blandr” (Version 0.5.1) packages.

Results

Parent and SLP Group Comparisons on PDDBI Domain and Composite Scores

We examined differences on the 58 total parent- and teacher-rated PDDBI domain and composite scores. As shown in Table 3, a significant informant group difference was observed on the Arousal Regulation Problems domain score, with higher parent-informant ratings of child arousal and regulation problems, relative to the SLP informants (p = .04, d = 0.54). No other domain or composite scores were significantly different between the two informant groups.

Table 3.

Differences between parent- and speech-language pathologist (SLP)–rated Pervasive Developmental Disorder Behavioral Inventory (PDDBI) domain and composite scores.

PDDBI domain/composite	Parent report M (SD)	SLP report M (SD)	t(28)	p	d	95% CI
Approach/withdrawal problems
Sensory/perceptual approach behaviors	47.00 (11.70)	44.21 (7.26)	1.15	.26	0.30	−2.17, 7.76
Ritualisms/resistance to change	52.21 (12.68)	46.59 (11.49)	1.66	.11	0.48	−1.32, 12.56
Social pragmatic Problems	49.17 (11.46)	47.62 (8.43)	0.68	.50	0.16	−3.12, 6.23
Semantic/pragmatic problems	50.97 (11.87)	51.79 (10.17)	−0.43	.67	0.08	−4.76, 3.10
Arousal regulation problems	50.66 (10.09)	45.34 (10.22)	2.12	.04	0.54	0.18, 10.44
Specific fears	52.97 (11.89)	53.45 (12.13)	−0.16	.87	0.04	−6.59, 5.62
Aggressiveness	49.48 (10.88)	46.10 (7.97)	1.34	.19	0.37	−1.79, 8.54
AWP/C	50.38 (12.73)	46.03 (10.54)	1.36	.18	0.38	−2.19, 10.88
Receptive/expressive social communication abilities
Social approach behaviors	54.21 (11.48)	54.34 (12.00)	−0.08	.94	0.01	−3.81, 3.53
Expressive language	53.48 (12.71)	54.55 (11.86)	−0.69	.49	0.09	−4.22, 2.09
learning, memory, and receptive language	56.07 (10.88)	54.48 (8.36)	0.94	.36	0.17	−1.87, 5.04
REXSCA/C	54.72 (12.14)	56.45 (13.81)	−0.918	.37	0.14	−5.57, 2.12

Note. AWP/C = Approach/Withdrawal Problems Composite; REXSCA/C = Receptive/Expressive Social Communication Abilities Composite; CI = confidence interval.

Parent and SLP Reliability Estimates

ICC correlation coefficients were calculated for PDDBI domain and composite scores to examine interrater reliability between parent and SLP individual ratings. Bland–Altman plots were generated to evaluate informant agreement across the entire range of each PDDBI composite score. As shown in Table 4, the AWP domain and composite coefficients showed mostly poor interrater reliability between parent and SLP ratings of challenging behaviors. However, semantic/pragmatic problems informant ratings indicated moderate interrater reliability (ICC = .54, 95% confidence interval = [0.21, 0.75]). For the AWP informant composite scores, the Bland–Altman results indicated poor informant agreement (mean difference = 4.34; 95% LoA = −29.34 to 38.03), with the positive mean difference value indicating more elevated parent ratings (i.e., parents rated behaviors as more severe compared to SLPs). Figure 1 shows a nonsystematic dispersion of difference values observed (i.e., values scattered away from the reference line), which is further evidence of poor informant agreement between parent and SLP ratings of challenging behaviors.

Table 4.

Intraclass correlations between parent- and speech-language pathologist–rated Pervasive Developmental Disorder Behavioral Inventory (PDDBI) domain and composite scores.

	Intraclass correlations
PDDBI domain/composite	Estimate	p	95% CI
Approach/withdrawal problems
Sensory/perceptual approach behaviors	.04	.42	−0.32, 0.39
Ritualisms/resistance to change	−.20	.87	−0.51, 0.16
Social pragmatic problems	.20	.14	−0.18, 0.53
Semantic/pragmatic problems	.54	.001	0.21, 0.75
Arousal regulation problems	.05	.38	−0.27, 0.38
Specific fears	.04	.41	−0.34, 0.41
Aggressiveness	−.08	.68	−0.42, 0.28
AWP/C	−.15	.80	−0.48, 0.21
Receptive/expressive social communication abilities
Social approach behaviors	.65	< .001	0.37, 0.82
Expressive language	.76	< .001	0.55, 0.88
Learning, memory, and receptive language	.53	.001	0.22, 0.75
REXSCA/C	.68	< .001	0.42, 0.83

Note. AWP/C = Approach/Withdrawal Problems Composite; REXSCA/C = Receptive/Expressive Social Communication Abilities Composite; CI = confidence interval.

Figure 1.

Correlation and agreement between parent and speech-language pathologist (SLP) Approach/Withdrawal Problems Composite (AWP/C) and the Receptive/Expressive Social Communication Abilities Composite (REXSCA/C). Top row: Scatter plots show the relationship between parent and SLP composites. The solid black line denotes the regression line, with the shaded area depicting the 95% confidence interval. Bottom row: Bland–Altman scatter plots show the level of agreement between parent and SLP composites. The middle dotted line denotes the mean difference, with the 95% limits of agreement represented by the dotted lines above and below the mean difference score.

In contrast, all REXSCA dimension domain and composite coefficients indicated moderate to good interrater reliability (combined ICC range: 0.53–0.76). For the REXSCA informant composite scores, the Bland–Altman plot indicated good informant agreement (mean difference = −1.72; 95% LoA = −21.55 to 18.09), with a small mean difference close to zero, and randomly distributed difference values close to the reference line.

Discussion

To the best of our knowledge, this is the first investigation of informant agreement between parents and SLPs in preschool children with ASD using the PDDBI, a well-known and widely administered ASD rating scale. In partial support of our hypotheses, group comparisons indicated higher parent ratings of a specific domain of challenging behaviors (i.e., arousal regulation problems) compared to SLP ratings. However, no group-level differences in parent and SLP ratings of social communication problems or abilities were found. Parents and SLPs exhibited poor agreement on most ratings of challenging behaviors, with the exception of semantic/pragmatic problems. Moderate to good agreement was observed for all ratings of social communication skills.

These findings confirmed our hypothesis that parents and SLPs would show poor agreement on ratings of challenging behaviors with stronger agreement on ratings of child receptive and expressive social-communication abilities. These results are aligned with a large body of literature focused on mainly school-age children with ASD that has reported generally poor informant agreement on ratings of challenging behaviors (Jordan et al., 2019; Lane et al., 2013; Lopata et al., 2016; Stadnick et al., 2017; Thompson & Winsler, 2018; Ung et al., 2017). In contrast, we found moderate reliability between parent and SLP ratings of semantic/pragmatic problems, as well as moderate to good reliability and good overall agreement for ratings of receptive and expressive social communication abilities. Taken together, our findings are aligned with an emerging body of literature suggesting that informants may be better aligned on ratings of social communication, compared to other behavioral concerns in young children with ASD (Neuhaus et al., 2018; Thompson & Winsler, 2018). It is important to note that the preschool children in the current study had a generally low level of language functioning. As these children develop language, parent and SLP discrepancies may be more likely to arise, as indicated in older, verbal children with ASD (Donnelly et al., 2018). Indeed, it is possible that with more advanced language ability, children with ASD may show more significant context-dependent variability in their use of receptive and expressive social communication. Longitudinal studies are needed to further examine how age and level of language functioning may differentially moderate parent–SLP informant discrepancies in young children with ASD.

Given the specific pattern of parent and SLP informant agreement findings observed in the current study, social communication abilities, as measured by the PDDBI-PX and PDDBI-TX, may be relatively stable across home and classroom settings in preschool children with ASD who also have generally low language skills. In contrast, challenging behaviors, such as arousal regulation problems, may be more likely to differ across home and classroom settings. Indeed, previous literature indicates certain elements of the home environment (e.g., the distracting sensory landscape) may exacerbate challenging behaviors in individuals with ASD (Nagib & Williams, 2017). In contrast to the home setting, SLPs interacting with participants in the current study provided therapy in a potentially more predictable setting using child-initiated teaching opportunities paired with other naturalistic developmental behavioral therapeutic strategies (Schreibman et al., 2015). As a result, the display of challenging behaviors may have been less prominent for SLP informants in this particular setting using these specific strategies compared to parents interacting with their children across less structured home/daily environments.

In addition, as suggested (in part) by the ABC Model (De Los Reyes & Kazdin, 2005), it is possible that differences in parent and SLP perspectives on child behaviors may have also contributed to patterns of informant disagreement in the current study. For example, parents may have rated child behavior with reference to other typically developing children in the family, whereas SLPs may have rated behavior with reference to other children with ASD and related communication disorders for whom they have evaluated or treated. Although not examined in the current study, the ABC Model further suggests that child, parent, and family factors can also lead to differences in informant perspectives (De Los Reyes & Kazdin, 2005), which may moderate discrepancies among informants (De Los Reyes, 2013; Neuhaus et al., 2018). For example, child African American racial status and low household income have been found to lead to greater caregiver–clinician disagreement on reports of autism symptoms (Neuhaus et al., 2018). To our knowledge, the examination of clinician characteristics (e.g., years of training, racial status, gender) as moderators of parent–clinician agreement has not been explored. Future investigations should aim to identify child, parent, clinician, and family characteristics that may moderate informant discrepancies on ratings of social communication and challenging behaviors in young children with ASD to improve clinicians' awareness of these factors and to minimize potential biases that may arise during assessment and treatment.

Limitations and Future Directions

Our study has several strengths. We fill an important gap in ASD informant agreement/discrepancy literature by using a high-quality measure (the PDDBI) to examine preschoolers (an underresearched population in this area). We also focus on informants who are certified and licensed SLPs, an informant group that has not previously been examined. However, our study is not without limitations. Our sample size was small, possibly increasing the likelihood of Type II error. Second, our sample consisted of predominantly White, male preschool children with ASD. Given that child and family characteristics, such as racial status and household income, have been found to moderate informant agreement on the reporting of ASD symptoms (Neuhaus et al., 2018), it is possible that the findings of the current study may not generalize across different racial groups and socioeconomic contexts.

Conclusions

In conclusion, while parents and SLPs exhibited poor agreement on ratings of challenging behaviors, moderate to good agreement was observed for ratings of social communication skills in preschool children with ASD with limited language skills. Future research is needed to replicate these findings in a larger, more heterogeneous sample of preschool-age children with ASD and extend these findings to examine potential moderators of informant discrepancies. Taken together, these results highlight the critical importance of following evidence-based practice guidelines and including parents in the ASD assessment, treatment planning, and implementation process, as they may report key behavioral concerns that may not otherwise be observed by the teacher or clinician.

Funding Statement

This study was supported, in part, by a National Institutes of Health Eunice Kennedy Shriver National Institute of Child Health & Human Development Intellectual Developmental Disabilities Research Center grant (1P50HD103538-01).