Caregiver Attendance as a Quality Indicator in the Implementation of Multiple Evidence-Based Practices for Children

Abstract

Objective:

This study investigated a quality indicator for children’s mental health, caregiver attendance in youth psychotherapy sessions, within a system-driven implementation of multiple evidence-based practices (EBPs) in children’s community mental health services.

Method:

Administrative claims from nine fiscal years were analyzed to characterize and predict caregiver attendance. Data included characteristics of therapists (n = 8,626), youth clients (n = 134,368), sessions (e.g., individual, family), and the EBP delivered. Clients were primarily Latinx (63%), male (54%) and mean age was 11; they presented with a range of mental health problems. Three-level mixed models were conducted to examine the association between therapist, youth, service, EBP characteristics and caregiver attendance.

Results:

Caregivers attended, on average, 46.0% of sessions per client for the full sample and 59.6% of sessions for clients who were clinically indicated, based on age and presenting problem, to receive caregiver-focused treatment. Following initial EBP implementation, the proportion of caregiver attendance in sessions increased over time. Caregivers attended a higher proportion of youth psychotherapy sessions when clients were younger, had an externalizing disorder, were non-Hispanic White, and were male. Further, higher proportions of caregiver attendance occurred when services were delivered in a clinic setting (compared with school and other settings), by bilingual therapists, and the EBP prescribed caregiver attendance in all sessions.

Conclusions:

Overall, the patterns of caregiver attendance appear consistent with evidence-informed practice parameters of client presenting problem and age. Yet, several improvement targets emerged such as client racial/ethnic background and service setting. Potential reasons for these disparities are discussed.

In an effort to maximize the benefit of mental health care for children, recent large-scale efforts have focused on implementing evidence-based practices (EBPs) in publicly-funded mental health systems nationwide (Brookman-Frazee et al., 2016; Garland et al., 2013; Hoagwood et al., 2014; Rubin et al., 2016). These system-driven reforms have been identified as a strategy to improve the overall quality of children’s mental health services (Park, Tsai, Guan, & Chorpita, 2018). Quality indicators are need to evaluate how these EBP implementation efforts impact services (Schoenbaum & Holmgren, 2006). A quality indicator is defined as an component of patient care that is clinically meaningful, evidence-based, tractable and quantifiable (American Psychological Association, 2008). One key quality indicator for children’s mental healthcare is caregiver engagement in their child’s treatment (Garland et al., 2013; Nock & Kazdin, 2005; Wright, Lau, & Brookman-Frazee, 2019).Caregivers are critical for managing treatment participation for everyone involved in child therapy, as children cannot attend treatment without assistance (Nock & Ferriter, 2005). However, for many EBPs, caregivers themselves need to actively participate in treatment for successful outcomes. Therefore, in this study, we specifically examined caregiver attendance, the most frequently measured form of treatment engagement (Staudt, 2007), as a quality indicator of mental health care provided within a system-driven implementation of multiple EBPs.

The Importance of Caregiver Attendance in Child Mental Health Treatment

Meta-analyses of children’s mental health treatments that involve caregivers have larger effect sizes than individual child treatment (Dowell & Ogles, 2010; Sun, Rith-Najarian, Williamson, & Chorpita, 2018). In particular, research consistently shows that treatments that focus on teaching caregivers strategies to manage their child’s behaviors are the most effective for children with externalizing problems (Kaminski & Claussen, 2017). Systematic reviews have established behavioral parent training to be the front-line treatment for disruptive behavior disorders and attention deficit/hyperactivity disorder (ADHD) for children 12 years of age and younger, with limited evidence for child focused interventions (Evans, Owens, Wymbs, & Ray, 2017; Kaminski & Claussen, 2017). Similarly, the American Academy of Child and Adolescent Psychiatry’s clinical guidelines also recommend caregiver-focused treatments for oppositional defiant disorder (Steiner & Remsing, 2007), conduct disorder (Steiner & Dunne, 1997) and ADHD (Pliska, 2007). Therefore, high quality care for children 12 and under with externalizing disorders should include caregivers in every session, with the treatment content delivered individually to caregivers, to groups of caregivers, or with the caregiver interacting directly with their child in session (Evans et al., 2017; Kaminski & Claussen, 2017).

Apart from externalizing disorders, select treatments for other disorders have also been explicitly designed to include caregivers as critical agents in driving therapeutic change. Examples include attachment-based treatments for trauma-exposure (e.g., Lieberman, 2004; Lieberman, Horn, & Ippen, 2005; Yasinski et al., 2016) as well as family interventions for depression and autism-spectrum disorder (Brookman-Frazee, Drahota, & Stadnick, 2012; Kaslow, Broth, Smith, & Collins, 2012; Stahmer, Collings, & Palinkas, 2005). In contrast to systematic reviews for externalizing disorders, there is less research on the impact of caregiver involvement for children with internalizing disorders and trauma exposure (Buchanan-Pascall, Gray, Gordon, & Melvin, 2018; Dorsey et al., 2017). The existing research does suggest that caregiver involvement enhances outcomes. Specifically, a recent meta-analysis on cognitive behavioral therapy for youth with internalizing disorders, including anxiety, depression, posttraumatic stress disorder, and obsessive-compulsive disorder, found that treatments had larger effect sizes at long-term follow-up when caregivers were involved (Sun et al., 2018). Therefore, caregiver treatment inclusion is an established quality indicator for youth with externalizing disorders, and may be for youth with internalizing disorders as well.

Challenges to Caregiver Attendance

Despite the demonstrated benefits of their participation, engaging caregivers in children’s mental health therapy remains challenging, which could threaten the quality of community-implementation of EBPs (McKay & Bannon, 2004; National Institute of Mental Health, 2001; Staudt, 2007). Caregivers of color are especially vulnerable to encounter barriers to treatment engagement. A plethora of studies have outlined that Latinx (e.g., Dickson, Zeedyk, Martinez, & Haine-Schlagel, 2017; Stadnick, Haine-Schlagel, & Martinez, 2016; Young & Rabiner, 2015), African-American (e.g., Harrison, McKay, & Bannon, 2004) and Asian American/Pacific Islander (e.g., Ho, Yeh, McCabe, & Hough, 2007) caregivers often face disproportionate difficulties to treatment participation or engagement. Provider-level and structural barriers such as poor therapeutic alliance, clinic location in low-resource neighborhoods, and inadequate numbers of treatment providers to provide linguistically appropriate care can all contribute to poorer engagement for low-income, underserved communities (Barnett, Lau, & Miranda, 2018; Gopalan et al., 2010; Ingoldsby, 2010).

Apart from cultural and socioeconomic backgrounds, other client-level factors may affect levels of caregiver engagement. For instance, caregiver participation is especially challenging for children involved in the child welfare system, as foster parents and court-mandated biological parents both have multiple logistical and motivational barriers to engage in their children’s care (Dorsey, Conover, & Revillion Cox, 2014; Dorsey, Pullmann, et al., 2014). Additionally, service delivery setting may inhibit caregiver treatment involvement (e.g., Lindhiem & Kolko, 2010). Though school services have been deemed the de facto mental health system for youth (Burns et al., 1995; Whitaker et al., 2018), the evidence for school settings as facilitators for increased caregiver treatment involvement is mixed (Reardon et al., 2017). In fact, school-based mental health typically provides child-focused individual treatment as opposed to involving caregivers, which may compromise the effectiveness of care (Atkins, Hoagwood, Kutash, & Seidman, 2010; Atkins et al., 2015).

Challenges with caregiver engagement not only impact the effectiveness of treatment for children and their families, but also EBP implementation. As many EBPs require caregiver participation, recruitment and retention of caregivers into services is necessary for therapists to implement the practices with fidelity and meet EBP certification requirements (Scudder & Herschell, 2015). Poor caregiver engagement can result in therapist attrition from training initiatives, which limits the sustainability of the interventions, and ultimately the return on investment needed to implement caregiver-focused EBPs into publicly-funded systems of care (Timmer et al., 2016; Beveridge et al., 2015).

Measuring Caregiver Attendance

Caregiver engagement has been studied using chart reviews and behavioral observations (Dickson et al., 2017; Garland et al., 2010; Zima et al., 2005). Though these methods can provide rich data on therapist and caregiver behaviors, they can be time and resource intensive, limiting their utility as a routine measure of a quality care, which can evaluate caregiver attendance at the population level. On the other hand, administrative claims data are traditionally used and preferred for quality monitoring purposes within the public service sector (Institute of Medicine, 2006). Given that different claiming codes are used based on who is present in a treatment session, administrative claims could provide a pragmatic measure of caregiver attendance across diverse service settings, which is consistent with the definition of a quality indicator.

Quality Indicators

Quality indicators or performance measures exist to, ideally, guide quality improvement efforts. Nationally recognized quality measures include indicators from the National Committee for Quality Assurance’s Healthcare Effectiveness Data and Information Set (HEDIS); exemplars include follow-up care for medication treatment for youth with attention deficit hyperactivity disorder and continuity of care following hospitalization (Zima et al. 2019). Yet, major critiques of existing indicators include challenges with feasibility, insufficient data supporting predictive validity for outcome improvement, and concerns that they set too low a threshold for quality care (Hayward, 2007; Pincus et al., 2016). Although leading organizations including the Institute of Medicine (2006) have encouraged use of indicators to close the “quality chasm,” quality measures are seldom used in the mental health field especially for youth treatment (Pincus et al., 2016; Zima et al., 2019). Additionally, quality indicators for mental healthcare are few compared to other area of health services, which may explain sluggish progress in improvements in the quality of mental health care (Pincus et al., 2016).

Notably, nationally endorsed indicators in mental health, such as those required by the Centers for Medicare and Medicaid Services, predominantly index care related to management of psychiatric medications (National Quality Forum, 2019; Zima et al., 2019; Heuer et al., 2019). Within youth mental healthcare, this focus is misaligned with consumer demand for psychosocial interventions and trepidation about overreliance on psychotropic medication for children (e.g., Corcoran et al., 2017). Moreover, there has been inattention to the reality that children depend on their caregivers to access and facilitate maximal benefit from mental healthcare. Caregiver attendance is a quality indicator that has the potential to overcome some major critiques; in particular, it has strengths in clinical validity due to evidence demonstrating its positive relationship with clinical outcomes (e.g., Kaminski et al, 2017). Furthermore, if caregiver attendance can be accurately captured using standard procedure codes in administrative claims data, it can be feasibly measured.

The Current Study

In response to calls by the National Committee for Quality Assurance (Schoenbaum & Holmgren, 2006) and the American Psychiatric Association Task Force on Quality Indicators (American Psychological Association, 2008) to identify and assess quality indicators, the current study investigates caregiver attendance as a quality indicator within the context of a system driven implementation of multiple EBPs within the nation’s largest county mental health department, the Los Angeles County Department of Mental Health (LACDMH). Specifically, with the LACDMH Prevention and Early Intervention (PEI) Transformation, contracted mental health agencies were offered the opportunity for reimbursement for delivery of a number of EBPs, which targeted a range of presenting problems including trauma exposure, externalizing disorders, depression, anxiety, and substance use.

EBPs included caregiver-focused interventions to target externalizing disorders and promote healthy attachment and parent-child relationships including, Child-Parent Psychotherapy (CPP; Lieberman et al., 2005), Parent-Child Interaction Therapy (PCIT; Eyberg & Funderburk, 2011), and Positive Parenting Program (Triple P; Sanders, Turner, & Markie-Dadds, 2002). Other EBPs included caregiver and child-focused components, including Trauma Focused Cognitive Behavior Therapy (TF-CBT; Cohen, Mannarino, & Deblinger, 2006), Incredible Years (IY; Webster-Stratton, 2011), and Managing and Adapting Practices (MAP; Chorpita, Daleiden, & Collins, 2014). MAP is an evidence-based system of resources and models; therapists use the system to identify, select and track the delivery of intervention strategies to caregivers and children (Chorpita et al., 2014; Southam-Gerow et al., 2014). MAP includes intervention strategies or “practice guides” that are directed to caregivers or children based on the client’s characteristics, including presenting problems and age. Notably, IY includes groups for caregivers of children ages 0-12 years and a child-focused social-skills group (i.e., Dina Dinosaur School) for children ages 4 to 8 years as part of its training series. LACDMH and developer materials recommend but do not require that children receive both the caregiver and child focused components (LACDMH, 2016; (Webster-Stratton, 2011). Finally, some EBPs were predominately child-focused, including Cognitive Behavioral Interventions for Trauma in Schools (CBITS; Jaycox, 2003), Seeking Safety (SS; Najavits, 2002), and Interpersonal Psychotherapy (IPT; Mufson & Moreau, 1999).

Expanding on past research, this study uses administrative claims data to examine patterns of caregiver attendance within the unique context of a large-scale implementation of multiple EBPs serving racially and ethnically diverse children and families. In line with guidelines set by the Society for Clinical Child and Adolescent Psychology Evidence Base Updates (Southam-Gerow & Prinstein, 2014), the current study uses “treatment families,” or categories of interventions (e.g., caregiver-focused interventions, individual child therapy), rather than brand-name treatments to understand how EBP implementation impacts caregiver attendance. The aims of the current analyses are:

1) Characterize rates of caregiver attendance in publicly funded children’s mental health system for: (a) all clients served and, (b) for clients who are clinically indicated to receive caregiver-focused treatment (i.e., youth with externalizing disorders who were 12 years old and younger).

2) Identify session, client and provider characteristics that predict the proportion of sessions that caregivers attended for all clients and the clinically indicated subsample.

Method

Procedure

This study used LACDMH PEI administrative claims data from May 2010 through December 2017 (9 fiscal years) for children and adolescents. For this study, only claims billed as “psychotherapy” (individual, family, group) were included. Claims related to medication management, case management, evaluation and assessment, and crisis services were excluded. A total of 4,860,518 psychotherapy claims for children ages 0 to 21 years old at the time of their unit of service were available for analysis. Therapists submitted a practice-specific claim for each unit of service. Each claim was uniquely associated with client demographic and clinical characteristics, setting in which the service was delivered, Current Procedural Terminology (CPT) code and therapist demographic and professional background (discipline) characteristics.

Participants

Clients were eligible for PEI services based on age, presenting problem and diagnostic criteria for each practice. To be eligible for inclusion in this study, a child needed to have at least two psychotherapy claims in the dataset, have received at least one session of the nine PEI practices, and be 21 years or younger at the time of their first claim. The total number of youth in our sample was 134,368 who received psychotherapy services from 8,626 unique therapists within 95 unique agencies. See Table 1 for information regarding the demographics, presenting problems, and care setting for children. A subsample of these children was created to examine predictors of caregiver attendance for the indicated population, defined as children aged 12 and younger with an externalizing disorder. This subsample included a total of 29,603 children at their first unit of service and had received an externalizing disorder diagnosis at the time of entrance into the service system. See Table 1 for demographic information regarding these children.

Table 1.

Descriptives of Child and Therapist Variables

Child Variables	All Children (n = 134,368)	Clinically-Indicated Subgroup (n = 29,603)
Child age, M (SD)	11.0 (4.6)	7.6 (2.6)
Child gender: Male, No. (%)	72,905 (54.3)	21,440 (72.4)
Child race/ethnicity, No. (%)
Non-Latinx White	9,502 (7.1)	2,001 (6.8)
Latinx	85,021 (63.3)	18,193 (61.5)
Asian American & Pacific Islander	1,944 (1.4)	356 (1.2)
African American	17,187 (12.8)	4,632 (15.6)
Other	20,714 (15.4)	4,421 (14.9)
Child diagnosis
Internalizing	55,722 (41.5)	---
Externalizing	37,960 (28.3)	29,603 (100.0)
Trauma	12,408 (9.2)	---
Adjustment	16,342 (12.2)	---
Other	11,936 (8.9)	---
Child CPS/JJ involved	27,080 (20.2)	4,556 (15.4)

Notes: CPS = Child Protective Services; JJ = Juvenile Justice.

Regarding therapist characteristics, a total of 8,626 therapists were included in the sample. On average, therapists submitted a claim for 2.9 PEI practices (SD = 1.6; range = 1-9). Approximately 37.5% (n = 3,235) of therapists delivered at least one psychotherapy service in Spanish or a non-English language. In terms of therapist discipline, 41.4% were counselors (n = 3,575), 27.2% marriage and family therapists (n = 2,348), 9.4% social workers (n = 814), 2.7% psychologists (n = 231), and 19.2% other disciplines (n = 1,656).

Measures

All measures were generated from the administrative claims data. All variables were aggregated to the child-level.

Outcome variable: Proportion of claims with caregiver present per child.

The outcome variable was the proportion of a child’s psychotherapy claims in which a caregiver was present. Caregiver attendance for each session claim was determined based on whether one of the following CPT codes was used: Family Psychotherapy with Patient Present (90847) and Collateral (90887). These CPT codes were chosen because they are traditionally used by community therapists to bill for sessions with caregiver presence. The proportion of claims with caregiver present per child was calculated by aggregating the CPT codes to the child-level and determining the proportion of claims per child in which the caregiver was present.

Predictor variables.

Child demographics.

We included the following demographic variables that were all based on the child’s first unit of service: age (in years), gender (male was the reference group) and race/ethnicity (Non-Latinx White [reference category], Latinx, Asian/Pacific Islander, African American, and Other).

Child diagnosis.

Each child was assigned to a diagnostic category based on the diagnosis the child was assigned when they entered the system. This method was chosen because therapist-reported diagnostic assignment was largely stable across claims for a given child. The mutually diagnostic categories were: Externalizing (reference category), Internalizing, Trauma, Adjustment, and Other. As therapists could include a primary and secondary diagnosis, a dichotomous variable was also included, which indicated if the child had one or two diagnoses at admission.

Child service system involvement.

To index service system involvement outside of LACDMH, one dichotomous variable was included that indicated if the child was involved in the juvenile justice system and/or the child welfare system.

Setting.

A categorical variable was created to indicate the primary service setting in which a child was served. Each child was assigned one setting based on where the majority of their claims took place. The setting categories included: clinic (reference category), school, and other (which included home and community).

Evidence-based practice.

The nine PEI practices were categorized into three mutually exclusive variables based on the requirements for caregiver involvement in the EBP. Each child was assigned to one of these mutually exclusive categories based on the majority of the claims they received. Materials from the EBP developers and LACDMH PEI Practice Guidelines were used to classify if the child received a: (1) caregiver-focused practice (PCIT, Triple P, CPP) (2) caregiver- and youth components practice (TFCBT, MAP, IY), or a (3) youth-focused practice (IPT, SS, CBITS). See Table 2 for information and categories of each EBP. One categorical variable was created with the caregiver-focused practices as the reference category.

Table 2.

Description of EBPs

Practice	Age Range	Caregiver Participation	Format	General Description
CBITS	10-15 years	Youth-Focused	Group, school based, youth sessions	CBT skills related to trauma
CPP	0-6 years	Caregiver-Focused	Caregiver or caregiver-youth sessions	Improve parent-child interactions through play
IY	0-12 years	Caregiver & Youth Components	Caregiver or youth group sessions	Programs for children and caregivers, including caregiver groups and a children’s social-skills group
IPTD-A	12-25 years	Youth-Focused	Individual youth sessions	Reduce symptoms of depression and improve interpersonal functioning by focusing on one more interpersonal problem areas
MAP	0-21 years	Caregiver & Youth Components	Youth, caregiver, or caregiver-youth sessions	A suite of decision support tools to identify, select, and monitor evidence-based practice
PCIT	2-7 years	Caregiver-Focused	Conjoint caregiver-youth sessions	Caregiver receives live-coaching as they play with their child to practice behavior management skills and improve caregiver-child interaction patterns
Seeking Safety	13-20 years	Youth-Focused	Group or individual youth sessions	CBT, present-focused, problem oriented focused on coping skills
TF-CBT	3-18 years	Caregiver & Youth Components	Youth, caregiver, and conjoint sessions	CBT skills related to trauma
Triple P	0-18 years	Caregiver-Focused	Group or individual sessions with caregivers	Behavioral parenting skills

Notes: CBITS = Cognitive behavioral intervention in schools. CPP = Child Parent Psychotherapy. IY =. The Incredible Years. IPT-A = Interpersonal Psychotherapy for Depressed Adolescents. MAP = Managing and adapting practice. PCIT = Parent-Child Interaction Therapy.. TF-CBT Trauma-Focused Cognitive Behavioral Therapy

Time.

Because we were interested in changes in reach of caregiver attendance over time, we included a continuous variable (1-9) that indicated the fiscal year of the child’s first claim. For example, a time variable of “1” indicated that the child’s first claim was in the first fiscal year of the claims dataset, and a time variable of “9” indicated that the child’s first claim was in the last fiscal year of the claims dataset.

Therapist language.

One categorical variable was created to indicate the therapist’s primary language in which they deliver services. The categories were: English only (reference) and non-English language.

Analytic Plan

Data analyses were conducted for: 1) all of the psychotherapy claims for children, and 2) psychotherapy claims for children in which it would be clinically indicated for caregivers to be present in sessions. As described above, the proportion of claims with caregiver attendance per child was calculated by aggregating to the child-level from the claims-level dichotomous variable of caregiver session attendance in each session claim. For Aim1, descriptive statistics were run for the proportion of claims with caregiver session attendance per child for both samples.

For Aim 2, due to the nested nature of the data (children within therapists within agencies), we determined whether there was significant variance attributable to the therapist and agency levels by running an unconditional models for each sample with the outcome variable. A significant proportion of variance was attributable to the therapist level (ICCs = 0.49-0.51), and to the agency level (ICCs = 0.23-0.31). Thus, analyses employed a three-level mixed model with children (Level 1; n = 134,368) nested within therapists (Level 2; n = 8,626), nested within agencies (Level 3; n = 95). All multilevel analyses were run using Stata/SE 15.1.

Results

Aim 1: Characterize Caregiver Attendance

Aim 1a. Characterize caregiver attendance for claims for all clients.

Descriptive analyses were run to examine the proportion of claims in which a caregiver was present for all child psychotherapy claims. From a total of 4,860,518 child psychotherapy claims, 1,919,693 (or 39.5%) had a caregiver present. A total of 134,368 unique children were identified who had received one of the nine PEI practices of interest, and the percentage of claims with a caregiver present per child was calculated. On average, children had 46.0% (SD = 30.8; range = 0-100) of their claims with a caregiver present.

Aim 1b. Characterize caregiver attendance for claims for subset of clinically-indicated clients.

At the claims level, 946,214 claims were identified for children 12 years old and under with a diagnosis of an externalizing disorder, with 469,565 (or 49.6%) having a caregiver present. A total of 29,603 unique children were in the clinically indicated sample and had received at least one of the nine PEI practices. On average, caregivers were present in 59.6% (SD = 30.4; range = 0-100) of their child’s claims. The clinically indicated children had a significantly higher proportion of caregiver session attendance compared with the not clinically indicated children (B = 10.12, p < 0.01).

Aim 2: Identify Predictors of Caregiver Attendance

Predictors of the proportion of claims with a caregiver present per child were explored using a three-level mixed model for both samples. Significant predictors of caregiver attendance were similar across both samples (see Table 3). An R², or the amount of variance in percentage of caregiver attendance per child accounted for by the entire model was approximated for both of the samples (Edwards, Muller, Wolfinger, Qaqish, & Schabenberger, 2008; Xu, 2003). For the entire sample, 38.0% of the variance in the percentage of caregiver attendance per child was accounted for by the entire model. For the clinically indicated sample, 36.6% of the variance in the percentage of caregiver attendance per child was accounted for by the model.

Table 3.

Predictors of Proportion of Claims with Caregiver Present Per Child

	All Children (n = 134,368)			Clinically-Indicated Children (n = 29,603)
Predictors	M (SD) % CG involvement	B (SE)	95% CI	M (SD) % CG involvement	B (SE)	95% CI
Setting (Ref = Clinic) a	52.2 (31.6)	---	---	67.8 (28.7)	---	---
School	26.6 (19.3)	−19.49 (.18)**	−19.85, −19.13	32.3 (20.1)	−24.20 (.38)**	−24.95, −23.46
Other (Home/Community)	47.0 (30.3)	−2.65 (.17)**	−2.99, −2.31	60.5 (28.6)	−2.69 (.38)**	−3.43, −1.95
Primary EBP Type (Ref = CG-focused) b	84.8 (21.8)	---	---	84.6 (21.6)	---	---
CG & Youth Components	43.1 (26.3)	−25.25 (.22)**	−25.67, −24.82	49.5 (27.4)	−23.60 (.37)**	−24.33, −22.87
Youth-focused	23.4 (23.9)	−29.01 (.30)**	−29.59, −28.42	34.8 (28.2)	−25.37 (1.31)**	−27.94, −22.79
Child gender (Ref = Male)	48.5 (31.4)	---	---	59.6 (30.3)	---	---
Female	43.2 (30.0)	−2.09 (.12)**	−2.32, −1.85	59.5 (30.7)	−.96 (.27)**	−1.48, −.44
Child age at first claim	---	−1.77 (.02)**	−1.81, −1.74	---	−1.55 (.05)**	−1.65, −1.45
Child race/ethnicity (Ref = Non-Latinx White)	47.7 (31.1)	---	---	61.4 (29.4)	---	---
Latinx	45.4 (30.2)	−1.62 (.23)**	−2.08, −1.16	59.6 (30.2)	−1.79 (.51)**	−2.79, −.80
Asian/Pacific Islander	41.6 (30.4)	−2.08 (.55)**	−3.15, −1.00	54.6 (30.4)	−1.33 (1.26)	−3.71, 1.15
African American	44.9 (31.7)	−1.55 (.27)**	−2.09, −1.01	57.1 (30.3)	−2.55 (.57)**	−3.67, −1.42
Other	49.2 (32.6)	−2.42 (.29)**	−2.98, −1.86	61.6 (31.5)	−2.19 (.61)**	−3.39, −.99
Child primary diagnosis (Ref = Externalizing)	51.1 (31.6)	---	---	59.6 (30.4)	---	---
Internalizing	38.6 (28.0)	−2.96 (.15)**	−3.25, −2.67	---	---	---
Trauma	44.5 (28.3)	−4.07 (.22)**	−4.51, −3.63	---	---	---
Adjustment	49.9 (31.1)	−2.71 (.20)**	−3.11, −2.32	---	---	---
Other	61.0 (33.8)	.05 (.23)	−.41, .50	---	---	---
Child co-morbid diagnoses (Ref = one diagnosis)	47.0 (31.3)	---	---	59.8 (30.8)	---	---
More than one diagnosis	43.6 (29.7)	.42 (.14)**	.14, .70	58.8 (29.4)	−.36 (.29)	−.94, .21
Child CPS/JJ involvement (Ref = Not involved)	46.2 (30.7)	---	---	60.0 (30.4)	---	---
Child CPS/JJ involved	45.3 (31.7)	−1.15 (.16)**	−1.47, −.84	57.1 (30.4)	−1.33 (.36)**	−2.04, −.63
Therapist language (Ref = Other than English)	48.3 (30.5)	---	---	62.2 (30.1)	---	---
Only English	44.0 (31.0)	−2.04 (.25)**	−2.54, −1.55	57.1 (30.5)	−2.04 (.42)**	−2.87, −1.21
FY of child’s first claim	---	.83 (.04)**	.75, .90	---	.98 (.08)**	.82, 1.13

Note: ^aSetting of the majority of the claims per child, reference is clinic

^bType of EBP for majority of claims per child, reference is caregiver-focused EBP (CPP, PCIT, Triple P); caregiver and youth component EBP (IY, MAP, TF-CBT); youth-focused EBP (CBITS, IPT, SS)

^*p < 0.05,

^**p < 0.01.

Child demographics.

In both samples, girls had a significantly lower proportion of claims with caregiver present, compared with boys (all children: B = −2.09, p < 0.01; clinically-indicated children: B = −0.96, p < 0.01). Similarly, child age was significantly associated with caregiver attendance, with younger child age linked to a higher proportion of claims with a caregiver present (all children: B = −1.77, p < 0.01; clinically-indicated children: B = −1.55, p < 0.01). Compared with Non-Latinx White children, the proportion of claims with a caregiver present was significantly lower for Latinx children (all children: B = −1.62, p < 0.01; clinically-indicated children: B = −1.79, p < 0.01) and African American children (all children: B = −1.55, p < 0.01; clinically-indicated children: B = −2.55, p < 0.01). Asian American/Pacific Islander children had the lowest percentage of caregiver involvement (all children: M = 41.6%, SD = 30.4; clinically-indicated children: 54.6%, SD = 30.4) compared to Non-Latinx White children (all children: M = 47.7%, SD = 31.1; clinically-indicated children: 61.4%, SD = 29.4), though this was only significant for the full sample (all children: B = −2.08, p < 0.01; clinically-indicated children: B = −1.33, p > 0.05).

Child diagnosis.

In the full sample of all child psychotherapy claims, child diagnosis was significantly associated with the proportion of claims with caregiver session attendance. Compared to children with an externalizing diagnosis, the proportion of claims with a caregiver present was significantly lower for children with an internalizing diagnosis (all children: B = −2.96, p < 0.01), a trauma diagnosis, (all children: B = −4.07, p < 0.01), and an adjustment disorder diagnosis (all children: B = −2.71, p < 0.01). There was no significant difference between children with an externalizing diagnosis and children with an “other” diagnosis with regard to proportion of claims with a caregiver present. For the full sample, children with only one diagnosis were more likely to have caregiver presence, but this was not significant for the clinically indicated sample (all children: B = .42, p < 0.01; clinically-indicated children: B = −.36, p > 0.05).

Child service system involvement.

Child involvement with service systems such as Child Protective Services (CPS) and juvenile court was found to be significantly linked to the proportion of their claims with a caregiver present. Children with service system involvement had a significantly lower proportion of claims with a caregiver present compared with children who were not involved with CPS or juvenile court (all children: B = −1.15, p < 0.01; clinically-indicated children: B = −1.33, p < 0.01).

Evidence-based practice.

The type of EBP received was also significantly associated with the proportion of child claims with a caregiver present for both samples. Compared with children who primarily received a caregiver-focused EBP, children who primarily received an EBP with caregiver and youth components had a significantly lower proportion of claims with a caregiver present (all children: B = −25.25, p < 0.01; clinically-indicated children: B = −23.60, p < 0.01), as did children who primarily received a youth-focused EBP (all children: B = −29.01, p < 0.01; clinically-indicated children: B = −25.37, p < 0.01). For clinically-indicated children, caregivers were involved in 84.6% of sessions when the child received a caregiver-focused EBP as compared to 49.5% of sessions when they received and EBP with components that focused on both caregivers and youth.

Setting.

Results showed that, compared with children who primarily received services in the clinic, children who received services at school had a significantly lower proportion of claims with a caregiver present (all children: B = −19.49, p < 0.01; clinically-indicated children: B = −24.20, p < 0.01). Children who primarily received services in other settings (such as home or the community) also had a significantly lower proportion of claims with a caregiver present compared with children who primarily received services in the clinic (all children: B = −2.65, p < 0.01; clinically-indicated children: B = −2.69, p < 0.01).

Therapist language.

A lower proportion per child of claims with a caregiver present was found for therapists who delivered services in English only compared with therapists who were able to deliver services in a language other than English (all children: B = −2.04, p < 0.01; clinically-indicated children: B = −2.04, p < 0.01).

Time.

Time, entered as the fiscal year of the child’s first claim coded as 1 through 9, was a significant predictor of the proportion of claims per child with a caregiver present (all children: B = 0.83, p < 0.01; clinically-indicated children: B = .98, p < 0.01).

Discussion

Caregiver attendance in their children’s psychotherapy sessions can be considered an indicator of quality care, especially for children 12 and younger presenting with externalizing disorders. The vast majority of research on caregiver engagement has focused on usual care services without systematic EBP implementation (Garland et al., 2010; Haine-Schlagel, Brookman-Frazee, Fettes, Baker-Ericzen, & Garland, 2012; Zima et al., 2005). Given that many EBPs require caregiver participation, research is needed to examine the impact of system-driven implementation efforts on this quality indicator for children’s mental health services. This study used administrative claims data to characterize rates and predictors of caregiver treatment attendance between 2010 and 2017 within the PEI Transformation, a large-scale children’s mental health reform that implemented multiple EBPs in Los Angeles County. The current sample captured information about caregiver attendance across the course of treatment for over 130,000 ethnically and racially diverse children, who were primarily Latinx, and served in a variety of settings. Specifically, we investigated caregiver attendance for the total population of clients who received EBPs and a subsample of clients who were clinically indicated to have caregivers attend all treatment sessions.

Children’s age and diagnoses predicted caregiver attendance in expected ways. In the full sample, caregivers of younger children and those with externalizing disorders attended greater proportions of psychotherapy sessions. These findings are consistent with evidence-based recommendations to provide caregiver-focused interventions to children presenting with externalizing disorders, especially if they are young (Evans et al., 2017; Kaminski & Claussen, 2017). Children with co-morbid diagnoses in the full sample had lower caregiver attendance than children with only a single primary diagnosis, whereas for the clinically-indicated sample a secondary diagnosis did not significantly impact the proportion of caregiver involvement. For the younger clinically-indicated population, it is possible that therapists did not change the treatment format delivery as many caregiver-mediated interventions have been extended to address internalizing and externalizing symptoms (Comer et al., 2012; Lube, Lenze, & Tillman, 2012). On the other hand, for the full sample, which included older children and adolescents, it is possible that co-morbidity increased the focus on treatment strategies delivered to youth, as the emphasis may be placed on treating internalizing or trauma related symptoms.

When examining treatment families, children who received caregiver-focused EBPs (e.g., PCIT, Triple P) had significantly more caregiver attendance on average, compared to children who received interventions that recommend, but do not exclusively require, caregiver components (e.g., MAP). Clearly, the structure of the caregiver-focused EBPs facilitates greater caregiver attendance, as it is required in the interventions. Our results suggest two important implementation considerations, especially for children who are clinically indicated to have caregiver participation in treatment. First, implementing caregiver-focused EBPs may lead to an enhanced dose of caregiver involvement compared to implementing EBPs that allow for greater flexibility in determining treatment participants. One reason for this may relate to the specialized training therapists receive in these interventions regarding how to work with caregivers, as community-based child therapists have reported feeling overwhelmed and underprepared on how to engage caregivers in treatment (Baker-Ericzén, Jenkins, & Haine-Schlagel, 2013; Brookman-Frazee, Drahota, Stadnick, & Palinkas, 2012). In fact, therapists have been shown to have more positive perceptions of caregiver-focused interventions than agency leaders, which may be related to therapists preferring EBPs that provide them with direct skills to work with caregivers (Stadnick et al., 2017). However, past research has demonstrated that caregiver-focused interventions, including Triple P and CPP, had lower overall reach and sustainment within the PEI Transformation (Brookman-Frazee et al., 2016, 2018). Based on the challenges of engaging caregivers in care, it is possible that key decision makers such as agency leaders may perceive that the implementation of caregiver-focused EBPs has limited return on investment. Indeed, agency leaders have reported having poorer perceptions of these EBPs (Stadnick et al., 2017), which may relate to how they can be costly to implement and limited in the age range and presenting problems that they treat (Okamura et al., 2018). This relates to the second implementation consideration based on our findings, which is that additional implementation strategies may be needed to help increase caregiver involvement in interventions that recommend their attendance, but allow for flexibility in treatment delivery. These strategies likely need to be multi-level, with strategies that focus on preparing caregivers to be involved in treatment and training therapists on how to better promote caregiver engagement (Haine-Schlagel, Martinez, Roesch, Bustos, & Janicki, 2016).

Overall, caregiver attendance in the PEI Transformation occurred in 59.6% of sessions for children 12 and younger with externalizing problems. Though it is ideal for caregivers to attend every treatment session for children with these presenting problems, this rate compares favorably to one study that examined caregiver session attendance in usual mental health care using chart review, which found caregiver attendance in approximately 49% of sessions (Zima et al., 2005). However, rates of caregiver attendance were lower than another community study which found caregiver attendance in 70% of sessions aged 4-13 presenting with externalizing problems in usual care treatment (Garland et al., 2010). That study used a behavioral observation of outpatient mental health services, which might have shown a higher level of involvement because of the clinic-based setting. Another reason could stem from the vast majority of current sample being from a racial/ethnic background given the documented disparities showing that caregivers of color have lower treatment engagement compared to non-Hispanic White families (e.g., Alegría, Green, McLaughlin, & Loder, 2015; Dickson et al., 2017; Stadnick et al., 2016; Zimmerman, 2005).

Indeed, this study identified disparities in caregiver attendance between non-Hispanic White children, and children of color, with Asian American/Pacific Islander children having the lowest percentage of sessions with caregivers present. These disparities are consistent with past findings on mental health service utilization, with lowest access rates for Asian American/Pacific Islander children (Garland et al., 2005; Gudiño, Lau, Yeh, McCabe, & Hough, 2009). Racial/ethnic disparities in caregiver attendance likely relate to caregiver, workforce, and systemic barriers that impact caregiver willingness and ability to participate in their child’s treatment (Barnett et al., 2018). Caregivers have reported numerous barriers to participating in their children’s mental health treatment, including logistical constraints (e.g., work schedules), stigma related to mental illness and help seeking, and negative past experiences in systems of care (Gopalan et al., 2010; Harrison et al., 2004; McKay & Bannon, 2004). Policies and the sociopolitical climate can further exacerbate disparities. Undocumented immigrants are especially unlikely to seek mental health services due to fear of being reported to authorities (Philbin, Flake, Hatzenbuehler, & Hirsch, 2018). These concerns have now also extended to documented immigrants in the wake of ‘public charge’ policies (Perreira & Pedroza, 2019).

Having bilingual therapists may help ameliorate disparities for immigrant caregivers, as we found that therapists who were able to deliver services in languages other than English had more caregiver attendance. However, challenges exist in recruiting and retaining bilingual therapists, as there may not be enough providers who can provide treatment in another language and those who can are often burdened with the additional workload associated with translating materials and conducting outreach for non-English speaking clients (Regan et al., 2017). Additional workforce solutions may be needed to help engage caregivers in treatment. For example, lay health workers, who are more likely to share similar linguistic and cultural backgrounds to the caregivers they serve, may be well positioned to support EBP implementation by focusing on increasing caregiver engagement in care (Barnett et al., 2018; Barnett et al., 2019; Lakind & Atkins, 2018). Additionally, it is important to recognize that the population in this service system was predominately Latinx and findings on therapist language may not be generalizable to other populations. Though having a provider who speaks the same language as the caregiver is a critical first step for engagement, caregiver participation may be influenced by additional factors including mental health literacy, stigma concerns, attitudes and beliefs about treatment.

One strategy to increase access and decrease disparities has been providing school-based mental health services (Burns et al., 1995; Whitaker et al., 2018). However, we found that caregivers attended significantly fewer sessions when children received services in the school setting. Though the school setting increases access for children by providing care without the usual barriers to treatment, treatment typically has not included caregivers (Atkins et al., 2010; 2015). This may account for the findings from a meta-analysis of school-based mental health services for urban, low-income youth, which found negative effects on externalizing behaviors (Farahmand, Grant, Polo, & Duffy, 2011). Additionally, one study found that parents marked delayed communication from school-based clinicians as a barrier to treatment participation (George, Mcdaniel, Michael, & Weist, 2014). Innovative service delivery models may be needed to increase caregiver participation in school-based services and promote the effectiveness of services. Indeed, one model, which involved teachers and parent advocates in service delivery, not only was effective at reducing behavior challenges in children, but also showed higher rates of caregiver involvement than those found in clinic based studies with low-income, urban youth (Atkins et al., 2015).

One promising finding was that the year that the child entered into care through the system-driven PEI context was a significant predictor of the percentage of sessions with a caregiver present for the total sample and the indicated subsample; with caregiver attendance increasing over time. Though rates of caregiver attendance was suboptimal when examined by several factors, the improvements in attendance over times suggests that EBP implementation and delivery is improving the quality of care (Park et al., 2018). Due to the many barriers that families served in the community often face, it may be unrealistic to have caregiver participation in all child sessions despite the strong evidence for it. Overall, the positive trend in caregiver attendance over time signals support for policy-based implementation strategies to support the uptake, utilization, and sustainment of caregiver-involved EBPs to improve the quality of children’s mental health care. Caregiver attendance should be considered as a novel quality indicator to be assessed in conjunction with extant aforementioned indictors. Extending beyond measures related to medication management and post-hospitalization care, caregiver attendance provides data points on quality of care for youth receiving less intensive services at earlier stages of illness and particularly supports quality monitoring for youth with externalizing behavior problems, which are the most common referral reason in child mental healthcare (Garland et al., 2001). Caregiver attendance is a practical indicator that can be feasibly assessed through billing while also conveying information on when caregiver-directed evidence-based interventions are being omitted from care, thereby providing one potential explanation for lack of clinical outcome progress.

Limitations and Future Directions

This study had a few primary limitations that need to be noted. Though administrative claims data allowed for an evaluation of caregiver attendance at the system level, these data share the same limitations inherent to therapist self-report measures, in that inaccuracies in coding may not reflect the actual level of caregiver participation in treatment. Further, the CPT codes serve as a proxy for caregiver attendance, and it is not certain exactly who was present in treatment sessions or what activities occurred during those sessions. This is especially challenging for the collateral code, as collateral contacts could include other individuals in a child’s life, such as pediatricians or teachers. As defined in the LACDHM PEI Claiming Guide, collateral contacts could include any “significant support persons in relation to mental health needs of the client,” (LACDMH, 2013, p. 7), which may include individuals other than caregivers. Although we examined the data to understand distributions of procedure codes and the percentage of caregiver attendance was comparable to studies using chart reviews and behavior observations (Garland et al., 2010; Zima et al., 2005), reliance on claims data represents a limitation of our study. If systems plan to use administrative claims data to index caregiver attendance as a quality indicator in mental health services, it would be important to the precision in codes related to session participants. Furthermore, administrative claims data cannot assess aspects of engagement beyond attendance. The extent to which caregivers actively participated in session activities and the types of strategies used by the therapist are unknown. Within caregiver sessions therapists should deliver evidence-based treatment strategies (e.g., assigning/reviewing homework, role-play/behavioral rehearsal), which have been infrequently observed within usual care for children with externalizing disorders (Garland et al., 2010). Further, caregiver engagement needs to extend beyond attendance and include active participation in the treatment session as well as home practice between sessions to experience positive clinical outcomes (Nock & Ferriter, 2005). Future research needs to understand how the implementation of multiple EBPs in children’s mental health impacts the treatment strategies that therapists use with caregivers, to identify if therapists deliver these active learning strategies with increased frequency and intensity than they do in usual care. It would be especially beneficial to investigate the interaction of in-session therapist and caregiver behaviors to understand how to promote engagement within EBP implementation. Furthermore, reasons underlying absence of caregivers are unknown and future studies should address whether it is family or provider directed.

Even with these limitations, this study illuminates how EBP implementation may impact caregiver engagement in children’s mental health services. Using administrative claims data to identify caregiver engagement is an innovative and pragmatic strategy to monitor quality of care at a system level. Even with improvements in caregiver attendance over time in the PEI Transformation, a number of challenges were identified that suggest areas for future research. Specifically, challenges with caregiver engagement may compromise the quality of care that children receive, and potentially could impede the successful implementation and sustainment of caregiver-focused EBPs. As such, implementation strategies may be needed that specifically focus on how to engage caregivers in care and support sustainment of caregiver-focused EBPs (Barnett et al., 2019). Further, given evidence that agency leaders may have worse perceptions of caregiver-focused interventions, implementation strategies are needed that target leadership buy-in to support delivery of these practices and considering fiscal supports for ongoing implementation (Stadnick et al., 2017). Future research needs to focus on how these strategies impact clinical and implementation outcomes, to maximize the return on investment on implementing EBPs that engage caregivers in treatment.