In-Session Caregiver Behaviors during Evidence-Based Intervention Delivery for Children with ASD in Community Mental Health Services

Abstract

An Individualized Mental Health Intervention for ASD, (“AIM HI”), is a collaborative, caregiver-mediated and child-directed intervention for reducing challenging behaviors in children with autism spectrum disorder developed for delivery in community mental health programs. Using observational data from AIM HI sessions, the present study characterized the occurrence of two types of in-session caregiver behaviors: expressed concerns (i.e., expressing concerns about treatment strategies; expressing difficulty using skills; expressing difficulty completing homework) and participation engagement (i.e., asking questions; participating in session activities; showing commitment to therapy). Further analyses examined cultural differences in caregiver behaviors and associations between caregiver behaviors and clinician adherence. Participants included 39 caregiver-clinician dyads enrolled in a community effectiveness trial of AIM HI. Video recordings from 107 sessions during the first two months of treatment were coded for in-session caregiver behaviors and clinician adherence. Results indicated that expressed concerns were observed in 47% of sessions. When controlling for household income, Latinx caregivers were rated lower for expressing concerns about treatment strategies and demonstrated lower participation engagement behaviors in session compared with non-Latinx White caregivers, suggesting that cultural factors may impact verbal engagement in sessions. Finally, expressing concerns about treatment strategies, expressing difficulty using skills, and participation engagement were positively associated with clinician adherence. Findings suggest that some expressed concerns and participation engagement behaviors may be indicators of positive caregiver engagement in the context of a collaborative intervention, and lower levels of such caregiver engagement may actually impede clinicians’ delivery of intensive evidence-based intervention in routine care.

The past few decades have seen substantial advances in the development of efficacious mental health interventions for children and adolescents (Chorpita et al., 2011). To further the impact of these advances, implementation research has increasingly focused on identifying challenges to delivering evidence-based interventions (EBIs) within complex, routine care service settings, with the ultimate goal of addressing these challenges to optimize the effectiveness of community care. Specifically, research has identified challenges related to therapy process in youth psychotherapy, including difficulties with ongoing engagement of caregivers in services (McKay & Bannon, 2004). In order to identify and address early signs of disengagement, it is important to examine in-session indicators of challenges in caregiver engagement that negatively impact treatment delivery.

In-Session Caregiver Behaviors

Given the prevalence of youth EBIs requiring a significant level of caregiver involvement (e.g., parent training for youth conduct problems; Kaminski & Claussen, 2017), it is important to understand how caregiver engagement is manifested in sessions and how such in-session behaviors affect treatment delivery. Caregivers have the potential to play a considerable role in shaping the course of treatment for youth, yet little is known about how they participate in EBI sessions (Haine-Schlagel & Walsh, 2015). In particular, we were interested in two types of in-session challenges that may arise in the treatment of diverse, disadvantaged families often served in community settings: caregiver-expressed concerns about treatment acceptability and mastery and low levels of in-session caregiver participation. Thus, the present study focused on describing caregivers’ expressed concerns and participation engagement in sessions, which will be referred to collectively as in-session caregiver behaviors.

Expressed Concerns.

Early research on parent training interventions identified that “resistant” caregiver behavior, such as unwillingness to cooperate with the clinician’s suggestions, was associated with clinicians spending less time teaching social learning skills (Patterson & Chamberlain, 1994). Yet, beyond active refusal to comply with treatment, there may be a subtler range of caregiver behaviors and attitudes that complicate EBI delivery. Research on parent training with ethnic minority families has identified two broad categories of expressed concerns: (1) caregivers’ negative perceptions about the acceptability of EBI strategies, which may limit their willingness to participate in treatment, and (2) caregivers’ difficulty with mastery of skills taught in EBIs, which may decrease perceived self-efficacy to carry out treatment activities and contribute to more passive forms of disengagement (Lau, Fung, Ho, Liu, & Gudiño, 2011; Lau, Fung, & Yung, 2010). The present study examined three specific caregiver behaviors related to those categories: expressing concerns about treatment strategies, expressing difficulty using skills, and expressing difficulty completing homework.

While this range of challenges in therapeutic process may act as a barrier to delivering EBIs in routine care settings, expression of these concerns may also reflect processes inherent in collaborative parent training intervention approaches (Brookman-Frazee, 2004; Webster-Stratton, 1998). For example, caregiver concerns may be elicited by the clinician in a collaborative manner to promote discussion to ultimately improve treatment. Additionally, caregivers who ask questions and express concerns may elicit more individualized attention and higher quality care (Alegría et al., 2008). To date, there is limited information about the role of expressed concerns in the context of collaborative, caregiver-mediated interventions.

Participation Engagement.

While expressed concerns represent caregivers describing difficulties with treatment, participation engagement is defined as a caregiver’s active, independent, and responsive contribution to treatment (Haine-Schlagel & Walsh, 2015). Participation engagement can include both in-session (e.g., asking questions, sharing one’s opinion, participating in session activities) and between-session behaviors (e.g., homework completion). In contrast to caregiver session attendance and levels of attitudinal engagement (e.g., cognitive preparation), participation engagement is a relatively understudied aspect of treatment engagement in youth and family treatments (Becker et al., 2015). However, it is essential to better understand these behaviors as they are likely to impact treatment effectiveness (Hoagwood, 2005; Nock & Ferriter, 2005). The current study focused on three specific indicators of participation engagement: asking questions, participating in session activities, and showing commitment to therapy.

Although both expressed concerns and participation engagement are conceptualized as indicators of caregivers’ in-session engagement in treatment, it remains unknown how these two constructs relate to one another. Caregivers who express concerns about treatment may be less likely to participate in sessions due to their difficulties understanding or implementing treatment strategies. Conversely, being willing to speak up about concerns may indicate a positive working alliance and a higher level of comfort with engaging in the treatment process.

Cultural Influences on In-Session Caregiver Behaviors

Racial/ethnic minority clients, some of whom do not speak English as a primary language, constitute a majority of the population served within many publicly-funded mental health settings (e.g., Zima et al., 2005). Given the growing efforts to implement EBIs in routine care settings, it is critical to examine cultural factors as they relate to treatment processes and delivery. Previous literature suggests that ethnic minority caregivers may experience greater barriers to engaging in mental health treatment for youth. For example, ethnic minority caregivers demonstrated lower levels of attendance and provider-rated participation in a parent management training than their non-Latinx White (NLW) counterparts (Nix, Bierman, McMahon, & The Conduct Problems Prevention Research Group, 2009). Additionally, Latinx caregivers in particular demonstrated lower levels of observer-rated participation engagement across usual care sessions as well as usual care sessions that included use of caregiver engagement tools (Dickson, Zeedyk, Martinez, & Haine-Schlagel, 2017). Latinx Spanish-speaking caregivers were also less likely to report participating in usual care treatment (e.g., meeting with clinician, carrying out recommendations at home) than NLW caregivers (Fawley-King, Haine-Schlagel, Trask, Zhang, & Garland, 2013).

Cultural, language, and stress factors may contribute to these disparities in caregiver engagement. For instance, many EBIs were developed with majority populations (Miranda, Nakamura, & Bernal, 2003), such that the skills targeted may not be culturally typical (e.g., Chinese caregivers’ unfamiliarity with positive reinforcement strategies Lau et al., 2011). The active participation of Latinx Spanish-speaking caregivers may also be constrained by language and nuanced meanings that are difficult to translate (Yeh, McCabe, Hough, Dupuis, & Hazen, 2003). Furthermore, a variety of stressors may make it more difficult for some Latinx families to participate in treatment. Latinx families are likely to experience higher levels of environmental stress than NLW families (Crouch, Hanson, Saunders, Kilpatrick, & Resnick, 2000), and immigrant families may experience increased stressors such as family separation and difficulty acculturating (Gudiño, Nadeem, Kataoka, & Lau, 2011). Additionally, some Latinx families may engage less in services due to fears of authority figures (e.g., child protective services) and/or reporting of their undocumented status. While these and other factors may contribute to findings of lower engagement in Latinx caregivers, the more subtle, behavioral forms of in-session engagement and disengagement have rarely been assessed (Haine-Schlagel & Walsh, 2015). Thus, it remains unknown whether expressed concerns and participation engagement differ across racial/ethnic groups receiving caregiver-mediated interventions.

In-Session Caregiver Behaviors and Their Impact on Clinician Adherence to EBI

In addition to cultural differences, little is known about how in-session caregiver behaviors impact clinicians’ adherence to treatment - that is, the extent to which clinicians deliver prescribed EBI procedures as intended (Perepletchikova & Kazdin, 2006). Clinician adherence is an important ingredient in the success of a given EBI and has been linked to improving client outcomes. For instance, clinicians’ adherence to Multisystemic Therapy (MST), an EBI for at-risk youth implemented in community settings, predicted improved youth clinical outcomes and functioning over time (Schoenwald, Carter, Chapman, & Sheidow, 2008; Schoenwald, Chapman, Sheidow, & Carter, 2009). With regard to engagement, it is possible that clinicians who encounter disengaged client behaviors (including expressed concerns as well as less active participation from caregivers) may be inclined to adapt EBIs in ways that are inconsistent with core treatment principles (e.g., omitting elements perceived as less acceptable to clients), thereby reducing their protocol adherence. Indeed, in a study of implementation of six EBIs in a large community mental health system, clinicians’ report of limited client engagement in therapy (e.g., disinterested or avoidance behavior in session) was significantly associated with lower self-assessed ability to carry out the EBI as intended (Lau et al., 2018). However, it remains to be seen whether these associations hold when using observational measures of session activity rather than clinician report.

The study of caregiver engagement and clinician adherence is particularly important in early phases of treatment. Past research suggests that early sessions are essential to interventions because symptom change is most likely to occur in this stage of treatment (Ilardi & Craighead, 1994), and improvements in this phase are less likely to be attributable to other non-treatment related variables introduced over time. Client engagement in the first two months of treatment has been shown to significantly predict long-term treatment retention (Simpson & Joe, 2004), suggesting that early engagement is vital. In addition, research on cognitive therapy in depressed individuals has identified clinician adherence in early sessions as a strong predictor of symptom improvement (Strunk, Brotman, & DeRubeis, 2010); similarly, adherence to therapy-specific techniques early in treatment predicts subsequent positive outcomes (Feeley, DeRubeis, & Gelfand, 1999). Thus, it is important to investigate both engagement and clinician adherence in the context of early treatment, particularly among ethnic minority families who are more likely than non-minority families to drop out of treatment early on (McKay & Bannon, 2004).

An Individualized Mental Health Intervention for Children with Autism Spectrum Disorder (AIM HI)

As an intervention that relies on active caregiver participation, particularly in the early phases of treatment, An Individualized Mental Health Intervention for Children with Autism Spectrum Disorder (AIM HI; Brookman-Frazee & Drahota, 2010) is a fitting treatment in which to examine in-session caregiver behaviors. Developed in collaboration with community clinicians and caregivers and designed specifically for delivery in publicly-funded mental health service settings, AIM HI is a package of evidence-based behavioral strategies designed to reduce challenging behaviors in children ages 5 to 13 years with autism spectrum disorder (ASD). In general, the rationale for caregiver involvement in treatment protocols for children with ASD is to increase the intensity of intervention (i.e., caregivers use treatment strategies throughout naturally occurring interactions with their children) and improve the generalization of child skills across settings and people, which is a significant deficit associated with ASD (Lauren Brookman-Frazee, Stahmer, Baker-Ericzén, & Tsai, 2006; Owen-DeSchryver, Carr, Cale, & Blakeley-Smith, 2008). Thus, AIM HI uses a collaborative, caregiver partnership approach (Brookman-Frazee, 2004) to teach caregivers to understand patterns in their children’s behaviors and to use strategies to reduce challenging behaviors and increase acquisition of child skills. This model uses active caregiver engagement strategies (e.g., proactively seeking caregiver input; considering caregiver as partner in teaching child skills). The caregiver-clinician partnership approach has been shown to improve observed caregiver involvement during treatment sessions, including reduced stress, increased confidence, and improved parent-child interactions in session (Brookman-Frazee, 2004). Data from the present study are drawn from a recently completed, large-scale randomized controlled effectiveness trial demonstrating the positive impact of training community mental health providers to deliver AIM HI [CITATION MASKED FOR REVIEW].

The Present Study

The present study involved supplemental data coding and analysis to address three major research questions using early treatment sessions from clinicians being trained to deliver AIM HI. The first aim was to characterize in-session caregiver behaviors during the early treatment phase of AIM HI by examining the occurrence and intensity of expressed concerns and participation engagement in session. Secondly, we examined potential cultural differences in in-session caregiver behaviors. In particular, we were interested in whether expressed concerns and participation engagement differed by caregiver ethnicity and preferred language. Based on previous literature suggesting lower engagement in Latinx caregivers, we hypothesized that English- and Spanish-speaking Latinx caregivers would express greater concerns as well as participate less actively in sessions than NLW caregivers. Thirdly, we investigated associations between in-session caregiver behaviors (expressed concerns and participation engagement) and in-session clinician adherence to AIM HI strategies used with caregivers (i.e., structuring the sessions for skill-building, engaging/collaborating with participants, and teaching specific skills). We hypothesized that within a given session, greater expressed concerns and reduced participation engagement would be associated with reduced clinician adherence.

Method

Data for this supplemental study were drawn from a large-scale, randomized community effectiveness trial of AIM HI conducted in Southern California to examine the impact of training mental health therapists to deliver AIM HI on child, family, and clinician outcomes. As part of the trial, publicly-funded outpatient and school-based mental health programs were randomized to immediate AIM HI training and delivery or to a wait-list control/routine care observation condition. Given our interest in clinician adherence to a particular treatment, for the purposes of this study, only data from the AIM HI training condition were used.

Participants

All families in the study were current clients in one of the participating programs. Families were referred to the study for eligibility assessment by participating clinicians. Of the 91 families who agreed to be contacted by the research team and were referred within the first cohort of AIM HI training, 70 participated in an eligibility assessment and 68 of those 70 participated in the study (two were deemed ineligible after the assessment). Families were considered eligible for the effectiveness trial if they fit the following inclusion criteria for children: (a) 5–13 years old at the time of recruitment, (2) had a current ASD diagnosis on record, (3) exhibited clinically significant ASD symptoms on a standardized ASD diagnostic measure and (4) were English or Spanish speaking. In addition, children were required to present with at least one challenging behavior that interfered with their functioning (e.g., externalizing behaviors, noncompliance, socially inappropriate behaviors).

A subset of clinician-caregiver dyads from the AIM HI training condition of the effectiveness trial was included in the current analyses. The inclusion criteria for dyads was as follows: (a) participated in Cohort 1 of the effectiveness trial, (b) caregiver identified as Latinx or non-Latinx White (we excluded ten [15%] caregivers from Cohort 1 who identified as another race/ethnicity, given that it would be difficult to draw conclusions about other ethnic groups with such a small number), and (c) caregiver attended at least one video-recorded session during the first two months of therapy following baseline assessment. Note that caregivers were encouraged to attend all sessions of AIM HI, but there were no set number of sessions or specific AIM HI sessions that were intended only for caregivers. Thus, out of 58 dyads meeting criteria for (a) and (b), the final sample included in the current analyses consisted of 39 caregiver-clinician dyads drawn from six publicly-funded mental health programs. This final included sample did not significantly differ from the excluded sample with regard to proportion of Latinx to NLW caregivers, χ²(1, N = 58) = 0.43, p = .51. Informed consent was obtained from all participants, and all study procedures were approved by the Institutional Review Board at [MASKED FOR REVIEW].

Clinician participants.

Clinicians were considered eligible for the effectiveness trial if they: (a) were employed as trainee or staff at a participating program and were anticipated to be at the program for at least the next seven months, and (b) had a child on their caseload meeting the study inclusion criteria. Four of the clinicians were enrolled as part of two caregiver-clinician dyads (i.e., had two clients participate in the study).

Thus, the present study included 35 clinicians who were 83% female, with a mean age of 33.51 years (SD = 7.90, range = 23–57). Clinicians’ reported race/ethnicity was 51% NLW, 37% Latinx (including 3% who also identified as Native Hawaiian/Other Pacific Islander and 3% as multiracial), 6% Asian American, 3% non-Latinx multiracial, and 3% not reported. Regarding professional discipline, 34% of clinicians endorsed a marriage and family therapy discipline, 29% psychology, 20% social work, 9% psychiatry, 3% school psychology, and 6% other.

Caregiver and youth participants.

The 39 participating caregivers were primarily female (95%) with a preferred language of English (82%), with the remaining 18% preferring Spanish. Caregivers’ race/ethnicity was reported as 49% Latinx and 51% NLW; two Latinx caregivers (5%) additionally identified as American Indian/Alaskan Native. Thus, when combining across caregiver preferred language and race/ethnicity, 51% of caregivers identified as English-speaking NLW, 31% as English-speaking Latinx, and 18% as Spanish-speaking Latinx. Families’ reported household income was $25,000 or below for 36% of families, $25,001 - $50,000 for 28% of families, $50,001 - $75,000 for 18% of families, and $75,001 or above for 18% of families.

The 39 participating children were ages 5–14 years at baseline (M = 8.44, SD = 2.42; two participants turned 14 between recruitment and baseline data collection in the study). Of these participants, 85% were male. Caregiver-reported youth race/ethnicity was 62% Latinx (including two who additionally identified as American Indian/Alaskan Native and two who identified as multiracial), 33% NLW, 3% non-Latinx American Indian/Alaskan Native, and 3% Asian.

The severity of children’s ASD was assessed at baseline by the Autism Diagnostic Observation Schedule-2 (Lord et al., 2012), a standardized observational assessment of ASD behaviors. In the current sample, average ASD severity using the comparison score was moderate (M = 6.90, SD = 2.15, range = 1–10). The frequency of child disruptive behaviors was assessed at baseline based on caregiver report on Eyberg Child Behavior Inventory (ECBI; Eyberg & Pincus, 1999). In the current study, the 39 participating children had an average baseline ECBI Intensity t-score of 65.72 (SD = 10.32, range = 48–94), indicating clinically elevated disruptive behavior problems.

The majority of the 39 families (82%) received services from programs providing care in the outpatient clinic setting, 8% received care from a program providing school-based services and 10% received care from a program providing services in multiple settings. The primary funding source for services was Medi-Cal (California’s Medicaid) for 82% of clients and school special education funds for the remaining 18% of clients.

Intervention

An Individualized Mental Health Intervention for Children with Autism Spectrum Disorder (AIM HI).

AIM HI (Brookman-Frazee and Drahota, 2010) is based primarily on the principles of applied behavior analysis and was designed to be delivered in public mental health settings by clinicians with limited experience with ASD or behavioral interventions. The intervention was developed in collaboration with community stakeholders and ASD intervention experts based on a comprehensive needs assessment of the targeted service setting (Brookman-Frazee, Drahota, Stadnick, & Palinkas, 2012). It includes a series of protocol steps and within-session strategies to reduce challenging behaviors and increase positive alternative skills (e.g., social skills, self-regulation) in children with ASD ages 5–13. All individual components of the intervention are considered “well established” for this population (National Autism Center, 2009; Wong et al., 2015). While there is no set number or frequency of sessions, a minimum of 13 sessions is required, and completion of the protocol typically takes approximately six months. Early sessions of intervention focus on working with caregivers to conduct a functional behavior assessment to develop a structured behavior plan, including antecedent- and consequence-based strategies (i.e., caregiver strategies) and alternative behaviors for children (i.e., child skills) (see Brookman-Frazee et al., 2012 for more details about the protocol). While caregivers are not required to be present for every session of AIM HI, caregiver involvement is required to deliver AIM HI. The need for caregivers to be involved is emphasized during the clinician training, and clinicians individually tailor the type of involvement according to what is feasible for the family and the setting (e.g., in-person sessions with caregiver and youth, phone sessions, caregiver-only sessions, etc.). AIM HI intervention materials are provided in both English and Spanish and were available to clinicians to use depending on the family’s preferred language.

Clinician Training.

Clinicians in the AIM HI condition received initial training and didactic and structured case consultation for six months using a variety of components. First, clinicians attended an 8-hour workshop introducing them to the underlying concepts, required session elements, and protocol steps of AIM HI (e.g., determining the underlying function of behaviors; developing behavior plans; skill building using active teaching strategies). The workshop consisted of didactic lecture, video exemplars, and interactive practice opportunities. Following the workshop, clinicians delivered AIM HI for six months with participating families while participating in structured consultation meetings and receiving case-specific performance feedback approximately twice a month from an AIM HI trainer. Eleven consultation meetings (9 group, 2 individual) were provided that included dyadic instruction, case-specific feedback, and performance feedback based on review of session videos.

Sampling of Sessions for Caregiver Behaviors Coding

In total, a subset of 107 video recordings was coded for this study based on meeting the following criteria: (a) clinicians completed training as part of the first of four AIM HI training cohorts (i.e., had completed the post-training data collection), (b) session came from the first two months of treatment following study intake (given our focus on caregiver engagement and clinician adherence in the early phases of treatment), and (c) caregiver was present in session. Coded sessions represented 82% of all sessions meeting criteria (a) and (b). Within this sample, the mean number of recordings per family was 2.72 (SD = 1.12, range = 1–4). To assess for inter-rater reliability, 25% of recordings were randomly selected for double coding.

Measures

Expressed Concerns.

To examine caregivers’ expressed concerns about treatment, an observational coding system for therapy sessions was developed for the current study. The measure consists of three items: (1) expressing concerns about treatment strategies (e.g., “That strategy won’t work for my child,” “We don’t have time to give him choices for everything”), (2) expressing difficulty using skills (e.g., “I don’t understand how to fill out the Behavior Tracking sheet,” “It’s really hard for me to use the timer”), and (3) expressing difficulty completing homework (e.g., “I didn’t have time to do the Behavior Tracking form”).

Expressed concerns were assigned a global extensiveness rating for the entire session on a 5-point Likert scale for (1 [absent] to 5 [strongly expressed]. Extensiveness ratings incorporated both the presence/frequency and the intensity of the expressed concern. For expressing difficulty completing homework, a “not applicable” code was given for sessions in which no homework was assigned and no between-session instructions were given, as demonstrated by neither the clinician nor the caregiver mentioning homework/instructions from the previous session. Inter-rater reliability of expressed concerns’ extensiveness fell in the good to excellent ranges (ICC = .74 for expressing concerns about treatment strategies, ICC = .85 for expressing difficulty using skills, and ICC = .95 for expressing difficulty completing homework; Cicchetti, 1994). Due to limited internal consistency of the extensiveness ratings of the three items (α = .19; Cicchetti, 1994) and poor fit to a latent expressed concerns factor (reported in the results below), analyses were conducted separately for each item.

Participation Engagement.

To capture caregivers’ participation engagement behaviors, several codes from the Parent Participation Engagement Measure were employed (Haine-Schlagel & Martinez, 2014; Haine-Schlagel, Martinez, Roesch, Bustos, & Janicki, 2016). In the current study, global extensiveness ratings on a Likert scale from 1 (no participation) to 5 (high participation) were assigned for each of three types of caregiver behaviors observed in session: (1) asking questions (e.g., administrative or clinically relevant questions), (2) participating in session activities (e.g., engaging in activity, making comments and spontaneous contributions), and (3) showing commitment to therapy (e.g., commenting on helpfulness, demonstrating enthusiasm for session topics, encouraging client or other family members to participate). If no explicit activities (i.e., role-plays, in-session practice, therapeutic games, and psychoeducation) were observed within a session, coders rated the participating in session activities item as “not applicable.” Extensiveness incorporated both presence/frequency and intensity, with the exception of participating in session activities, which relied on intensity only. Inter-rater reliability for these three participation engagement behaviors was within acceptable ranges (ICC = .85 for asking questions, ICC = .52 for participating in session activities, ICC = .77 for showing commitment to therapy; Cicchetti, 1994). Construct validity of the three participation engagement extensiveness ratings was supported by confirmatory factor analyses reported below; thus, an average participation engagement score was used for analyses.

Observed In-Session Clinician Adherence to AIM HI.

These data were extracted from the larger effectiveness study, for which a larger sample of sessions was coded for clinician adherence. Clinicians’ AIM HI adherence was measured via observer ratings of the extensiveness of delivery of caregiver-directed strategies required in the AIM HI protocol. Given acceptable internal consistency (Cronbach’s α = .73; Cicchetti, 1994), a composite of seven strategies related to session structure, caregiver engagement, and active teaching of caregiver skills was calculated for each session. Adherence ratings were comprised of an extensiveness rating reflecting both the presence/frequency and intensity of use. Strategy extensiveness was rated on a scale of 0 (strategy not observed) to 6 (strategy used with high extensiveness). The seven strategies included: (1) using an agenda/session schedule, (2) using materials with caregiver, (3) providing psychoeducation/ information to caregiver, (4) modeling/demonstrating skills to caregiver, (5) providing opportunity for in-session practice to caregiver, (6) providing feedback to caregiver, and (7) assigning/reviewing between-session practice to caregiver. Inter-rater reliability for each of the seven strategies was in the good to excellent ranges within the larger effectiveness trial (ICC = .65 for using an agenda/session schedule, ICC = .82 for using materials with caregiver, ICC = .74 for providing psychoeducation/ information to caregiver, ICC = .81 for modeling/demonstrating skills to caregiver, ICC = .67 for providing opportunity for in-session practice to caregiver, ICC = .66 for providing feedback to caregiver, ICC = .77 for assigning/reviewing between-session practice to caregiver).

Coding Procedure for Supplemental Caregiver Behavior Coding

Coder Training.

A team of nine psychology undergraduate, post-baccalaureate, graduate students, and postdoctoral coders was trained to reliably code expressed concerns and participation engagement behaviors according to the coding manual. Three coders were fluent in Spanish and coded sessions conducted in Spanish. Training was conducted by two of the authors and included group didactics, individual practice sessions, and meetings with the trainers to discuss discrepancies with gold standard ratings. After coders achieved 80% reliability on at least six “gold standard” training sessions, they began coding sessions independently. Coding meetings were conducted every 2–3 weeks thereafter to clarify portions of the manual and prevent drift. The coding team was masked to study hypotheses.

Analysis Plan

Prior to primary analyses, a confirmatory factor analysis (CFA) was conducted using Mplus Version 7.4 to analyze the factor structure of expressed concerns and participation engagement. Similar to previous research on client concerns and engagement [MASKED FOR REVIEW], the expressed concerns factor included extensiveness ratings for the three items of expressing concerns about treatment strategies, expressing difficulty using skills, and expressing difficulty completing homework, while the participation engagement factor included extensiveness ratings for the three items of asking questions, participating in session activities, and showing commitment to therapy. The two-factor CFA for expressed concerns and participation engagement demonstrated inadequate model fit across indices (Root Mean Square Error of Approximation = .05, Comparative Fit Index = .95, Tucker Lewis Index = .90, Standardized Root Mean Square Residual = .07; Hu & Bentler, 1999). An examination of standardized factor loadings suggested that nonsignificant loading of expressed concerns onto a common factor (range: −.30-.16) accounted for the inadequate fit. However, a single-factor model for participation engagement revealed excellent fit (Root Mean Square Error of Approximation = .00, Comparative Fit Index = 1.00, Tucker Lewis Index = 1.00, Standardized Root Mean Square Residual = .00), with significant standardized factor loadings of .46, .53, and .67 for the three participation engagement behaviors. Thus, separate analyses were run for each of the three expressed concerns while a composite participation engagement score was used for the three participation engagement behaviors.

For Aim 1, descriptive analyses were conducted to characterize expressed client concerns and participation engagement in Latinx and NLW groups and in the overall sample. In addition, Pearson correlations between all constructs were examined.

For Aims 2 and 3, due to the nested nature of the data (sessions within client-clinician dyads within site), we first determined whether there was significant variance attributable to the client-clinician and site levels by running unconditional models with the outcome variables of interest (extensiveness of the three expressed concerns, the participation engagement composite, and clinician adherence). While a significant proportion of variance across outcomes was attributable to the caregiver-clinician dyad level (ICCs ranging from .02 to .32), the proportion of variance attributable to site was less than .01 for each outcome, demonstrating that modeling a third level was not necessary. Although four clinicians in the sample had two clients, this number was too low to necessitate another level of nesting, and therefore client-clinician dyads were used as a single level. Thus, analyses employed a two-level model with session observations (Level 1; n = 107) nested within unique client-clinician dyads (Level 2; n = 39). However, given that homework was not mentioned as being previously assigned in many sessions, analyses containing the CEC variable of expressing difficulty completing homework had 48 sessions nested within 24 client-clinician dyads. For all models, the Level 1 intercept was modeled as random. Level 1 slopes were initially modeled as random followed by removal of nonsignificant slopes to achieve the most parsimonious model. All multilevel analyses were run using HLM 7.01.

Aim 2 sought to examine caregiver cultural characteristics (preferred language, ethnicity) as predictors of in-session caregiver behaviors. To determine whether severity of child ASD (ADOS comparison score), behavior problems (ECBI Intensity score), and household income (3 categories as described in Participants) should be included in the model as control variables, we first ran multilevel regression models with each of these variables as independent predictors of in-session caregiver behaviors. Only household income emerged as a significant predictor of any expressed concerns or participation engagement. Therefore, the final Aim 2 analyses employed a combined language/ethnicity variable (using two dummy codes for English-speaking Latinx and Spanish-speaking Latinx, with NLW as the reference group) and a household income variable (using two dummy codes for $25,001-$75,000 and $75,001 and above, with $0-$25,000 as the reference group) as Level 2 predictors of extensiveness of the three expressed concerns and the participation engagement composite, respectively (Level 1 variables).

Aim 3 sought to examine in-session caregiver behaviors as predictors of in-session clinician adherence. Multilevel regression models were employed to investigate expressed concerns and participation engagement (all Level 1 variables) as predictors of adherence ratings (also Level 1) in separate analyses, followed by a single model containing all significant predictors to compare their effects.

Results

Aim 1: Characterize In-Session Caregiver Behaviors in AIM HI Sessions

Table 1 presents full descriptive statistics for expressed concerns, participation engagement, and in-session clinician adherence. Caregivers expressed at least one concern in 47% of sessions, with each individual type identified in 18–21% of sessions. Of the expressed concerns, expressing difficulty completing homework had the highest frequency and average extensiveness rating. Average extensiveness of participation engagement behaviors varied by behavior, with average extensiveness of participating in session activities being the highest.

Table 1.

Descriptive Statistics for Constructs across Sampled Sessions

	Latinx (n = 52)		NLW (n = 55)		Total (n = 107 sessions)
Item Description	Frequency [95% CI]	Mean (SD)	Frequency [95% CI]	Mean (SD)	Frequency [95% CI]	Mean (SD)
Expressed Client Concerns (1-5 Extensiveness)
% Sessions with at Least One Expressed Concern	29% [16-42%]		64% [51-77%]		47% [37-56%]
Number of Expressed Concerns per Session		0.40 (0.72)		0.84 (0.81)		0.63 (0.80)
Expressed Concern Types
Expressing Difficulty Completing Homework	14% [4-23%]	2.30 (1.87)	31% [18-44%]	2.89 (1.64)	22% [14-30%]	2.65 (1.74)
Expressing Concerns about Treatment Strategies	12% [3-20%]	1.19 (0.60)	31% [18-44%]	1.67 (1.56)	22% [14-29%]	1.44 (0.95)
Concerns Elicited by Clinician’s Question					52%
Expressing Difficulty Using Skills	15% [5-26%]	1.27 (0.72)	22% [11-33%]	1.42 (0.85)	19% [11-26%]	1.35 (0.79)
Participation Engagement Behaviors Composite (1-5)		2.45 (0.84)		2.95 (0.89)		2.71 (0.90)
Participating in Session Activities		3.30 (1.08)		4.09 (0.93)		3.73 (1.07)
Showing Commitment to Therapy		2.23 (1.20)		2.78 (1.24)		2.51 (1.25)
Asking Questions		2.19 (0.99)		2.36 (1.10)		2.28 (1.04)
In-Session Clinician Adherence to AIM HI:
Session Adherence Composite (0-6)		3.04 (1.30)		3.03 (0.98)		3.03 (1.14)

Note: NLW = non-Latinx White caregiver. CI = confidence interval, SD = standard deviation.

Table 2 presents correlations among all primary variables. Notably, while two expressed concerns (expressing concerns about treatment strategies and expressing difficulty using skills) were positively associated with the participation engagement behaviors, expressing difficulty completing homework was negatively associated with participation engagement.

Table 2.

Correlations between Extensiveness of Expressed Concerns, Participation Engagement, and Clinician Adherence (n = 107 sessions)

	Concern 1	Concern 2	Concern 3	Participation 1	Participation 2	Participation 3	Clinician Adherence
Concern 1	--	.20*	−.04	.20*	.16	.06	.14
Concern 2	--	--	.06	.16	.08	.10	.28*
Concern 3	--	--	--	-.34*	−.05	−.36*	.02
Participation 1	--	--	--	--	.32*	.35*	.29*
Participation 2	--	--	--	--	--	.24*	.15
Participation 3	--	--	--	--	--	--	.33*

Note:

^*p < .05.

Concern 1: expressing concerns about treatment strategies, Concern 2: expressing difficulty using skills, Concern 3: expressing difficulty completing homework, Participation 1: asking questions, Participation 2: participating in session activities, Participation 3: showing commitment to therapy. All constructs are measured at the session level.

Aim 2: Examine Cultural Differences in In-Session Caregiver Behaviors

We next investigated whether there were cultural differences in expressed concerns and participation engagement, when controlling for families’ household income. Results of the four multilevel regression analyses are presented in Table 3. As seen in the table, significant differences were found based on both caregiver ethnicity/language and family income. In general, Latinx English- and Spanish-speaking caregivers expressed fewer concerns and demonstrated lesser participation engagement than NLW caregivers when controlling for income level, and caregivers in the middle income level ($25,001-$75,000) expressed greater concerns and demonstrated greater participation engagement than caregivers in the lowest income level ($0-$25,000) when controlling for ethnicity/language.

Table 3.

Caregiver Ethnicity/Language and Household Income as Predictors of In-Session Caregiver Behaviors

Outcome:	Concern 1			Concern 2			Concern 3			Participation Engagement
Predictor	b	SE	t	b	SE	t	b	SE	t	b	SE	t
Intercept	1.34	0.14	9.56***	1.20	0.11	10.67***	2.61	0.74	3.50**	2.57	0.21	12.49***
English-speaking Hispanic^	−0.48	0.21	−2.26*	−0.08	0.19	−0.43	−0.32	0.57	−0.55	−0.33	0.24	−1.36
Spanish-speaking Hispanic^	−0.26	0.19	−1.39	−0.23	0.12	−1.95*	−1.61	1.27	−1.27	−0.56	0.21	−2.66*
$25,001-$75,000#	0.5	0.19	2.69*	0.39	0.16	2.52*	0.24	0.80	0.29	0.52	0.23	2.24*
$75,001 and above#	0.27	0.23	1.19	−0.04	0.16	−0.24	0.44	0.88	0.51	0.34	0.23	1.45

Note.

^{^}Dummy codes with English-speaking Non-Hispanic White caregivers as reference group.

^#Dummy codes with household income $0-$25,000 as reference group.

Concern 1: expressing concerns about treatment strategies, Concern 2: expressing difficulty using skills, Concern 3: expressing difficulty completing homework, Participation Engagement: composite of asking questions, participating in session activities, and showing commitment to therapy.

^*p < .05,

^**p < .01,

^***p < .001.

Aim 3: Examine Associations between In-Session Caregiver Behaviors and Clinician Adherence

Finally, we examined in-session caregiver behaviors in relation to clinician adherence to AIM HI within-session strategies directed to caregivers. With regard to expressed concerns, results of the first multilevel regression indicated that higher ratings of caregivers expressing concerns about treatment strategies was a marginally significant predictor of higher observed clinician adherence (b = 0.17, SE b = 0.09, p = .056). In addition, higher ratings of expressing difficulty using skills significantly predicted higher clinician adherence (b = 0.31, SE b = 0.09, p = .002). Expressing difficulty completing homework was not significantly associated with clinician adherence (b = 0.05, SE b = 0.07, p = .457). With regard to participation engagement, results revealed that higher participation engagement significantly predicted higher clinician adherence (b = 0.58, SE b = 0.09, p < .001).

Results of the final analysis comparing all significant expressed concerns and participation engagement as predictors in a single model are presented in Table 4. As shown in the table, expressing difficulty using skills and the participation engagement composite remained significant predictors of clinician adherence, whereas expressing concerns about treatment strategies did not significantly predict adherence.

Table 4.

Expressed Concerns and Participation Engagement as Predictors of Clinician Adherence

	b	SE	t
Predictor
Intercept	1.12	0.32	3.52**
Expressing Concerns about Treatment Strategies	0.07	0.09	0.72
Expressing Difficulty Using Skills	0.23	0.09	2.42*
Participation Engagement Composite	0.55	0.10	5.77***

Note. Analysis contains only expressed concerns and participation engagement that were found to be marginally significant or significant predictors of clinician adherence in separate analyses.

^*p < .05,

^**p < .01,

^***p < .001.

Discussion

The current study examined two types of in-session caregiver behaviors - expressed concerns and participation engagement - during publicly-funded delivery of AIM HI, a collaborative, caregiver-mediated package of evidence-based strategies for children with ASD. Findings from our first aim indicated that in this context, expressed concerns were identified in nearly half of the selected early treatment sessions, and participation engagement intensity varied across behaviors. Furthermore, expressed concerns and participation engagement behaviors were inconsistent in the direction of their associations with one another. For our second aim, we found that Latinx and low-income caregivers were less likely than NLW and middle-income caregivers to express concerns regarding treatment and participate in session. Results from our third aim revealed that when caregivers expressed greater concerns about treatment strategies, greater difficulty using skills, and demonstrated more active participatory engagement, clinicians had higher adherence to AIM HI in-session strategies directed to caregivers.

Findings from our confirmatory factor analyses and the first study aim suggest that the in-session caregiver behaviors of expressed concerns and participation engagement have a complex relationship to one another in this AIM HI sample. While the participation engagement behaviors fit well onto a common construct, the expressed concerns did not. In particular, while two expressed concerns (expressing concerns about treatment strategies and expressing difficulty using skills) were significantly positively associated with one another, difficulty completing homework was uncorrelated with them. Additionally, difficulty completing homework was significantly negatively associated with two participation engagement behaviors, whereas the other two expressed concerns were positively or not significantly associated with participation engagement. It is worth noting that analyses using the expressing difficulty with homework code contained only 48 out of the full 107 sessions due to homework not being references in many sessions; thus, findings should be interpreted with caution. It is possible that caregiver expressions of difficulty completing homework may represent a qualitatively different indicator of engagement than expression of concerns or participatory engagement in session, perhaps because homework completion relies on between-session follow-through by the caregiver.

Furthermore, observational coding of expressed concerns and participation engagement focused almost entirely on caregiver verbalizations. It is therefore possible that, based on their positive correlations, the expressed concerns and participation engagement measured in this study may have reflected verbal engagement in session. Overall verbal engagement (including comments unrelated to expressed concerns or participation engagement) was not measured in the current study. Of note, although the protocol does not specifically require clinicians to seek client perspectives by encouraging verbal input, caregiver input on therapy process of all kinds, including caregiver expression of concerns, is encouraged within the collaborative framework of AIM HI.

Contrary to our hypotheses were the findings that when controlling for socio-economic status in the form of household income, Latinx caregivers expressed fewer concerns about treatment strategies than NLW caregivers. From previous literature, we had expected that Latinx families would experience greater cultural barriers to engaging in treatment (Fawley-King et al., 2013). Our findings do not necessarily contradict past findings, as it is possible that Latinx families experienced greater barriers, but were less likely to speak up about them in session. This interpretation is consistent with the finding that in addition to reporting fewer concerns, Latinx families demonstrated lower levels of active participation (e.g., expressing commitment to therapy) – a finding that was also observed in previous research on Latinx caregivers’ participatory engagement (Dickson et al., 2017).

Latinx cultural values may have contributed to caregivers being less likely to verbally engage in therapy. Respeto, for instance, is a traditional Latinx value of respect and obedience to authority figures (such as clinicians), whereas simpatia is the desire to avoid conflict to maintain kindness (Kim, Lau, & Chorpita, 2016). These factors were identified in focus groups with clinicians who delivered AIM HI to Latinx caregivers and were highlighted as being particularly relevant with monolingual Spanish-speaking caregivers (Chlebowski, Magana, Wright, & Brookman-Frazee, in press). These values may have influenced Latinx caregivers to be hesitant to speak up during therapy for fear of appearing disrespectful or creating conflict by raising a concern (e.g., Añez, Paris Jr., Bedregal, Davidson, & Grilo, 2005). Semi-structured interviews with a subset of Latinx caregivers who participated in AIM HI factors identified themes of personalismo (development of a personal connection with their child’s clinician) and confianza (developmental of trust and a mutual respect) as necessary precursors to comfortable participation in their child’s therapy, suggesting that initial reticence from caregivers may be due the absence of these elements (Chlebowski, Magana, Wright, & Brookman-Frazee, in press). Furthermore, it is notable that across expressed concern and participation engagement analyses, findings demonstrated greater differences between Spanish-speaking Latinx and NLW caregivers as opposed to English-speaking Latinx and NLW caregivers. These results suggest that caregivers with limited English proficiency, who may be more tied to traditional values, are especially unlikely to express concerns or actively contribute to treatment. Thus, clinicians working with Latinx caregivers, particularly caregivers who primarily speak Spanish, may benefit from enhanced training regarding consideration of cultural values regarding parental involvement in children’s mental health services.

Along with ethnicity and language, family income was included in the models as a covariate and was found to be significantly associated with in-session caregiver behaviors. Specifically, caregivers in the $0-$25,000 income range expressed significantly fewer concerns and participatory engagement behaviors than those in the middle $25,000-$75,000 range. This income effect emerged even when controlling for caregiver ethnicity and language, suggesting that there may be unique factors, such as increased financial stressors, that lead the lowest-income caregivers to engage less in sessions. These findings have implications for providers working with families in the lowest income range, who may find it beneficial to prioritize caregiver engagement through strategies such as assessing and addressing barriers to participation in treatment.

It is essential to situate our findings within the context of the AIM HI intervention, which utilizes a collaborative approach between clinicians and caregivers to individualize caregiver and child skill-building For instance, in our sample, over half of caregiver-expressed concerns about treatment strategies were elicited directly by the clinician’s questions (e.g., “How do you think this strategy will work with your son?”). Thus, caregiver behaviors that have been traditionally viewed in the literature as client resistance or disengagement may actually be indicative of positive treatment engagement in the context of a collaborative intervention that encourages caregivers to think critically about the intervention and bring up concerns. This interpretation is supported by our findings that greater expressed concerns, as well as higher levels of participation engagement, were associated with higher adherence to AIM HI in-session strategies. Caregivers who shared their concerns likely provided clinicians with an opportunity to address those concerns using AIM HI strategies, such as providing psychoeducation or feedback about caregivers’ use of a skill. Thus, it seems as if caregivers who actively participated in session, whether through expressing concerns or participation, were likely to receive more intensive delivery of AIM HI strategies. It is also possible that caregivers who received more intensive delivery of AIM HI strategies likely had greater comfort with the intervention and increased opportunities to engage deeply with the concepts, which may have led to increased expression of concerns and participation.

In comparison, a separate study of therapy process challenges reported by clinicians delivering six EBIs in a public mental health system (Lau et al., 2018) found that although clinician reports of limited client engagement in the form of passive client behaviors (e.g., disinterested behavior) predicted lower clinician-reported treatment adherence, clinician-reported expressed concerns were not associated with clinician adherence in the past two months. This difference in results may be attributable to the method of data collection (observational coding versus clinician perceptions of expressed concerns and adherence). Alternatively, it may be that the six EBIs employed that study contained less explicit focus than AIM HI on collaborating with caregivers to improve caregiver engagement and treatment delivery, such that expressed concerns did not promote higher clinician adherence. Additionally, Lau and colleagues did not differentiate between concerns expressed by caregivers as opposed to child clients.

Taken together, findings from the present study indicate the importance of studying in-session caregiver behaviors in the context of youth EBIs, as these behaviors are likely to influence how clinicians deliver treatment. In particular, Latinx caregivers were less likely to verbally engage in session (whether through sharing concerns about treatment or demonstrating positive participation behaviors). While this study was not powered to examine whether these cultural differences ultimately impacted clinician adherence to treatment, it is possible that Latinx caregivers’ lower levels of speaking up resulted in fewer opportunities for clinicians to intensively deliver AIM HI strategies. Future studies should examine this possibility directly. Results of the current study point to potential implications for building clinicians’ cultural understanding and competence in working with Latinx caregivers. For instance, a greater focus on establishing rapport and trust may facilitate caregivers’ comfort in sharing their concerns, as may normalization of concerns and increased clinician-led conversations to elicit concerns with these populations. Motivational interviewing may provide a framework for some of these strategies to increase engagement (Chaffin et al., 2009).

Several future directions for research can further our understanding of in-session caregiver behaviors and their implications for community implementation of EBIs. Firstly, given AIM HI’s collaborative, caregiver-focused approach, it will be important to see if our findings hold for other youth interventions that are less focused on caregiver engagement. This research will shed light onto the types of treatment approaches and strategies that are most effective to elicit and respond to caregiver concerns. It may be that enhancements are needed to ensure positive outcomes with less acculturated families or those with limited income to fit a family’s needs. Secondly, it would be helpful to compare observed in-session caregiver behaviors to caregiver report of expressed concerns and participation engagement. As previously discussed, there may be a discrepancy between what caregivers experience (e.g., doubts about being able to master a skill) and what they report in session (e.g., agreeing to try a skill), particularly for Latinx caregivers. If such a discrepancy exists, it will be important to investigate its influence on treatment process and outcome. Thirdly, examination of cultural differences in caregiver engagement should be broadened to look at other cultural groups and factors, as their in-session behaviors are likely influenced by different contexts and values. Fourthly, it will be important to investigate the impact of in-session caregiver behaviors on longer-term therapy outcomes, such as client attrition and progress in treatment.

Limitations

Several limitations should be noted regarding the present study. We employed a relatively small sample of clinicians and caregivers in this supplemental study, such that we were not able to examine an overall, indirect effect between language/ethnicity, in-session caregiver behaviors, and clinician adherence. Such a model would have provided greater insight into how cultural factors influence implementation quality. In addition, the sample size used in this study may have had limited power to detect effects. This study was also conducted as part of an effectiveness trial; as such, clinician behaviors such as adherence to in-session strategies were likely affected by the ongoing feedback that they received from AIM HI trainers as they delivered AIM HI. Therefore, results may be more relevant to community implementation of EBI with an ongoing consultation component. Findings from the present study should also be interpreted within the context of caregivers of children with ASD, who may differ from other types of caregivers in ways that could affect their treatment engagement (e.g., decreased parenting self-efficacy, increased stress and health concerns; Karst & Van Hecke, 2012). Finally, the focus of the current study was on characterizing in-session caregiver behaviors, which required that caregivers were in attendance for at least one video-recorded session during a two-month period. This represents a subset of families in usual care who may be more engaged than other families. However, we expect that the current sample is generalizable given that a previous study conducted in some of the same clinics with a broader group of children presenting with behavior problems indicated that caregivers were present in 70% of sessions (Garland et al., 2010).

Conclusion

This study offered a unique view into the behaviors that caregivers demonstrate in evidence-based psychotherapy sessions for youth with ASD. In the context of a collaborative intervention focused on caregiver engagement being delivered in routine mental health settings, we found that in-session caregiver engagement behaviors (both negative and positive) were common, but they were less likely to be demonstrated by Latinx than NLW caregivers. Furthermore, increased expression of caregiver concerns and positive participation in treatment predicted higher in-session clinician adherence to treatment. Findings indicate that in-session caregiver behaviors are an important mechanism through which to examine challenges and positive predictors of EBI delivery, such that treatment effectiveness can be enhanced in community care settings.