Community Therapist Reports of Client Engagement Challenges During the Implementation of Multiple EBPs in Children’s Mental Health

Abstract

This study investigated therapist reports of client engagement challenges in delivering evidence-based practices (EBPs), within the context of a large-scale implementation of multiple practices in children’s mental health services. Data were drawn from an online survey of 668 therapists (88% female, 35.0% Non-Hispanic White). The majority of therapists (75.4%) endorsed at least one client engagement challenge during their implementation of an EBP with an identified client over the previous two months. Two types of EBP client engagement challenges with distinct correlates but similar overall frequencies could be readily differentiated – Expressed Client Concerns and Limited Client Engagement in therapy activities. Limited Client Engagement were more commonly reported for clients with externalizing problems and by therapists with higher emotional exhaustion and negative perceptions of the specific EBP being delivered, whereas Hispanic/Latino therapists were less likely to report Limited Client Engagement. In contrast, Expressed Client Concerns were more commonly reported by therapists with positive general attitudes towards EBPs, and among therapists delivering a parent training EBP. Limited Client Engagement but not Expressed Concerns were linked with therapists’ self-reported ability to carry out the EBP with the target client. Findings suggest that client engagement challenges are frequent during the delivery of EBPs, but may impact implementation differently based on whether they relate to challenges in engaging clients in therapy activities versus addressing concerns raised by clients

Efforts to implement evidence-based practices (EBPs) in public sector mental health service systems have been promoted as a strategy to improve the quality of care for the most disadvantaged consumers (e.g., Aarons, Hurlburt, & Horwitz, 2011; Hoagwood, Burns, Kiser, Ringeisen, & Schoenwald, 2001). Yet, data suggest that both client outcomes and provider implementation outcomes (e.g., fidelity) are diminished when EBPs are delivered in community contexts rather than clinical research contexts (Southam-Gerow et al., 2010; Torrey, Bond, McHugo, & Swain, 2012; Weisz, Krumholz, Santucci, Thomassin, & Ng, 2015; Weisz et al., 2009;). This so-called implementation cliff marked by poorer client clinical outcomes and provider implementation outcomes has been attributed to multiple factors. First, community therapists tend to receive less intensive training, supervision, and performance feedback and tend to work in settings with fewer organizational supports for EBP delivery than are provided in randomized trials (e.g., Beidas, & Kendall, 2010; Drake et al., 2001; Langley, Nadeem, Kataoka, Stein, & Jaycox, 2010; Rapp et al., 2005). These factors have been related, in turn, to lower therapist integrity to EBP models in community implementation efforts (Brunette et al., 2008; Rapp, Goscha, & Carlson, 2010; Torrey et al., 2012).

Second, explanations for poorer clinical outcomes of EBP delivery in community settings have focused on differences between participants in effectiveness trials and consumers in public mental health service settings. Compared to children in effectiveness trials, children presenting in usual care tend to have greater diagnostic complexity and comorbidity and greater levels of chronic and acute stress (Ehrenreich-May et a., 2011; Southam-Gerow, Weisz, & Kendall, 2003). In addition, families served in community settings are more likely to be ethnic minorities who experience poverty and socioeconomic disadvantage (Baker-Ericzén, Hurlburt, Brookman-Frazee, Jenkins, & Hough, 2010; Garland et al., 2010; Southam-Gerow et al., 2003).

Given the differences in populations served, it is plausible that community therapists may encounter more difficulty in engaging clients in general comparison with study therapists in controlled trials. Indeed, rates of nonattendance and attrition are high in community mental health treatment (Ingoldsby, 2010; Snell-Johns, Mendez, & Smith, 2004; Stevens, Kelleher, Ward-Estes, & Hayes, 2006; Whittaker & Cowley, 2012). Over and above general client engagement challenges in usual care settings, EBP implementation may introduce additional concerns. Many critics have opined that EBPs that have been largely developed and validated with middle-class and Non-Hispanic White participants may be viewed as less acceptable or feasible among ethnic minority groups or disadvantaged families (Bernal & Scharró-del-Río, 2001; Lau, 2006). Resulting engagement challenges may be manifest when clients (i.e., youth and/or their caregivers) express concerns about the relevance appropriateness or feasibility of skills being taught, or when clients behaviorally display limited participation in therapy activities (Dickson, Zeedyk, Martinez, & Haine-Schlagel, 2017; Orrell-Valente et al., 1999; Stadnick, Haine-Schlagel, & Martinez, 2016).

Related to client expressing concerns about EBPs, an early observational study of parent training with low-income disadvantaged families documented instances of parents voicing difficulties enacting EBP target skills (i.e., “I can’t” behaviors) and instances of parents resisting therapist instruction (i.e., “I won’t” behaviors; Patterson &Chamberlain, 1994). Sequential analyses suggested that “I won’t” behaviors denoting disengagement were more likely to lead to negative therapist response, such as confrontation and expressing dislike for the caregivers, than “I can’t” behaviors, which encompassed the expression about low self-efficacy in carrying out skills. Over the course of treatment, “I won’t” engagement challenges predicted a lower dose of parent training being delivered. Other studies have shown that client engagement challenges may be more likely to manifest as low levels of active participation rather than overt resistance behaviors, particularly among low-income, ethnic minority caregivers (Carr et al., 2016; Dickson et al., 2017; Nix, Bierman, & McMahon, 2009; Pereira et al., 2016; Stadnick et al., 2016).

Indeed, in addition to limiting client treatment gains, challenges with client engagement are likely to impact therapist delivery of EBPs. Limited client engagement in treatment has been shown to result in decreased therapist productivity and lower return on investments in EBP implementation (Weisz, Ugueto, Cheron, & Herren, 2013). Yet the relationship between therapists perceptions of EBPs and their experiences in delivery of EBPs may be complex. As previously mentioned, community therapists commonly raise the concern that EBPs may not fit the needs of heterogeneous clients seen in usual care, and this may be related to the concern that EBPs may not be effective for ethnic minority families who are inadequately represented in the controlled trials that compose the evidence base (Aarons et al., 2010; Aisenberg, 2008). A key issue appears to be whether community therapists perceive EBPs to be flexible enough to tailor treatment to client characteristics, preferences, and/or culture (DiMeo, Moore, & Lichtenstein, 2012). Community therapists who perceive EBPs as being difficult to use, having fewer advantages over other approaches, or being a poor fit for their diverse clients were more likely to report omitting EBP components in their delivery (Lau et al., 2017).

Although data suggest that some community therapists hold negative attitudes of EBPs, there are few studies that describe the extent to which therapists encounter client engagement challenges during the real-time delivery of EBPs with their clients. Much of the literature on community therapists’ perceptions of EBPs has assessed attitudes toward unspecified interventions (Aarons et al., 2010; Borntrager, Chorpita, Higa-McMillan, & Weisz, 2009), examined global perceptions of practices (Jensen-Doss, Hawley, Lopez, & Osterberg, 2009), and has been conducted during preimplementation phases (e.g., Stirman et al., 2013). For example, Lewis and Simons (2011) found that therapists trained in cognitive-behavioral therapy (CBT) reported lower adoption when they felt that effectiveness was limited for clients with severe or comorbid problems and low receptivity to CBT. Therapists’ attitudes toward EBPs may in fact shape their subsequent experiences with EBP delivery. Providers’ attitudes toward EBPs influence how and the extent to which they implement EBPs as intended (Beidas et al., 2012; Southam-Gerow, Rodriguez, Chorpita, & Daleiden, 2012), which may in turn impact clients buy-in (Aarons & Palinkas, 2007). Therapist skepticism toward EBPs may (a) be communicated (albeit subtly) to clients, (b) shape implementation behaviors, and (c) increase their perceptions of encountering barriers to EBP delivery. Indeed, therapists who had positive attitudes toward EBPs in general were less likely to perceive client engagement challenges to implementation of CBT for depression (Lewis & Simons, 2011). Beyond general attitudes toward EBPs, therapists’ perceptions of specific EBPs vary and appear shaped by intervention characteristics (Barnett et al., 2017). If therapists find fault with the specific EBP that they are expected to deliver, this may hinder their ability or motivation to engage clients in that intervention. Beyond attitudes, aspects of the therapist’s background, such as discipline, workload, and job stress, may impact perceptions of challenges in implementing EBPs when they are mandated (Barnett et al., 2017; Reding, Chorpita, Lau, & Innis-Gomberg, 2014). In fact, therapist who reported feeling more competent with and more positive toward the EBPs they delivered reported lower rates of emotional exhaustion, suggesting an interplay between experience delivering EBPs, attitudes, and overall burnout with the job (Kim et al., in press). Therefore, burnout and attitudes could be related to perceived barriers to implementing EBPs.

The likelihood of therapists encountering and reporting client engagement challenges in EBP delivery may be also associated with client factors. In child mental health treatment, it is unclear whether expression of concerns with interventions and limited engagement are more likely in youth-versus caregiver-directed interventions. It is plausible that client engagement challenges may arise differentially in youth-directed treatment given that treatment is typically sought out by caregivers, and there is often caregiver–youth disagreement about presenting problems and even the need for treatment (Hawley & Weisz, 2003). Presenting problems may also be a factor in client engagement challenges. For example, youth from well-functioning, dual-parent families were more likely to show strong engagement in CBT for anxiety, whereas youth with comorbid externalizing problems showed lower engagement (Lee et al., 2017).

The current study examined therapist reports of client engagement challenges arising in the course of implementing multiple EBPs with youth and care-givers in community mental health settings. There were three aims of the study, with the first two exploratory aims investigating frequency and predictors of different types of client engagement challenges. First, we examined the frequency of therapist reports of two types of client engagement challenges: expressed client concerns about therapy strategies and limited client engagement in therapy activities. Therapist reported on the occurrence of these client engagement challenges in their delivery of one of six EBPs with a given client over the previous 2 months.

Second, we examined therapist, client, and service characteristics that predicted therapist-reported occurrence of client engagement challenges. Based on previous findings regarding challenges delivering EBPs with culturally diverse and clinically complex clients (Southam-Gerow et al., 2003), we sought to identify if the client’s race/ethnicity and presenting problem related to different types of client engagement challenges. Related to therapist characteristics, we investigated the role of therapist attitudes towards EBPs and feelings of burnout, as these constructs have been shown to impact EBP implementation (Kim et al., in press; Lewis & Simons, 2011). Third, we investigated the extent to which therapists’ reports of these two types of challenges were associated with their self-assessed ability to carry out delivery of the EBP. We hypothesized that therapist perceptions of both types of client engagement challenges in a given case would be associated with their assessments of success in delivering the EBP as intended. Specifically, we hypothesized that when therapists reported encountering more client engagement challenges, they would also report being less successful in delivering the EBP as intended. As such, these client engagement challenges may signify perceived barriers to EBP implementation in community settings.

Method

Context of the current study

The current study was conducted in the context of a system-driven implementation of multiple EBP and evidence-informed practices in children’s community mental health services in Los Angeles County’s Department of Mental Health (LACDMH), the nation’s largest county mental health department, serving more than 250,000 county residents on average each year (LACDMH, 2017). The Prevention and Early Intervention (PEI) Transformation offered agencies reimbursement for the delivery of select EBP and evidence-informed practices for children and transition-age youth to intervene earlier in the course of mental illness. LACDMH initially offered provider training and implementation support for six of the selected EBP and evidence-informed practices (hereafter referred to as practices) to address an array of youth mental health problems, including Cognitive Behavioral Intervention for Trauma in Schools, Child–Parent Psychotherapy, Managing and Adapting Practice (MAP), Seeking Safety (SS), Trauma-Focused Cognitive-Behavior Therapy (TFCBT), and Triple P–Positive Parenting Program (Triple P).

Recruitment and procedures

Data for the current study were collected as part of the Knowledge Exchange on Evidence-based Practice Sustainment (4KEEPS) study (Lau & Brookman-Frazee, 2016), examining sustainment of the six PEI practices that received initial implementation support. Data were extracted from an online survey, the 4KEEPS Therapist Survey, that included 98 agencies directly operated or contracted by LACDMH to deliver at least one of the six practices of interest to children or transition age youth in Fiscal Year 2013–2014. Eligible therapists included any therapist who billed for core psychotherapy services to at least one of the six practices of interest to children or transition age youth. Because there is no centralized directory of therapists providing PEI services in LACDMH directly operated and contracted agencies, we enumerated therapists into the survey through management at individual agencies. Of the 98 agencies in the sampling frame, 69 agencies(70.4%) were recruited into the study.

A total of 777 therapists completed the 4KEEPS Therapist Survey between March 2015 and July 2015. Most therapists were recruited through a direct e-mail campaign; agency management provided contact information for eligible therapists employed at agencies directly operated or contracted by LACDMH. Direct e-mail invitations were sent with personalized survey links to 1,656 community-based mental health therapists in 63 agencies. A total of 688 of these therapists completed the survey, for a response rate of 41.5% for the direct e-mail campaign. Management in an additional six agencies elected not to release staff contact information but instead agreed to forward an e-mail to therapists that would allow them to provide their contact information to the research team to opt in to the survey; 89 community therapists completed the survey through this opt-in process. Participants received a $20 gift incentive for survey completion. Institutional Review Boards at LACDMH and the University of California, Los Angeles approved all study procedures.

Participants

Out of the 777 participants who completed the 4KEEPS Therapist Survey, 668 (86%) were eligible for analysis in the current study because they reported that they had delivered one of the six practices of interest (Cognitive Behavioral Intervention for Trauma in Schools, Child–Parent Psychotherapy, MAP, SS, TF-CBT, Triple P) within the past 2 months. Participants were community-based mental health therapists employed in 64 agencies directly operated or contracted by LACDMH. Therapists were primarily female (n = 588, 88%) and an average age of 38.60 years (SD = 9.16). Therapists were racially/ethnically diverse (n = 286, 43.0% Hispanic/Latino; n = 234, 35.0% non-Hispanic White; n = 42, 6.3% African American; n = 79, 11.8% Asian/Pacific Islander; n = 27, 4% Multiracial/Other). The majority of therapists reported that they were able to deliver services in another language, with many reporting that they could deliver services in Spanish (n = 318, 47.6%), some who could deliver services in a language other than Spanish or English (e.g., Mandarin, Cantonese, Farsi; n = 61, 9.1%), and few who could deliver services in both Spanish and another language (n = 6, 0.9%). Notably, 93% of Hispanic/Latino therapists reported that they were able to deliver services in Spanish, indicating a high proportion of bilingual/bicultural therapists in this sample. Therapists were predominately master’s-level therapists (n = 577, 86.4%) and unlicensed (n = 379, 56.7%). Only a small number of therapists were trainees (i.e., students still matriculated in graduate programs; n = 46, 6.89%). The most common therapist discipline was marriage and family therapy (n = 369, 55.2%), followed by social work (n = 202, 30.2%), and psychology (n = 87, 13%). Therapists reported their primary theoretical orientation to be cognitive behavioral/behavioral (n = 355,53.1%), followed by family systems (n = 98, 14.7%), eclectic (n = 81, 12.1%), psychodynamic (n = 59,8.8%), humanistic (n = 35, 5.2%), and other (n = 10,1.5%). On average, therapists had been practicing for6.46 years (SD = 5.77) and had been employed at their current agency for 4.14 years (SD = 4.21). Therapists reported an average caseload size of 15.54 clients (SD = 9.41) that predominately comprised Hispanic/Latino clients (M = 71.44%, SD = 28.48). Of the six practices of interest, on average, therapists reported that they had been trained in 2.29 (SD = .90) practices.

Therapists were instructed to identify a client with whom they had recently delivered a specified EBP. For this identified case, therapists reported on client engagement challenges that arose over the past 2 months. The majority of child cases were in the 12 to 18 age range (n = 335, 50.5%) followed by the 6 to 11 age range (n = 239, 36.0%). Child cases were predominately female (n = 356, 53.4%) and Hispanic/Latino (n = 464, 69.6%). The most common presenting problems endorsed by therapists for their child cases were internalizing disorders(i.e., anxiety or depression; n = 423, 63.3%) followed by externalizing (i.e., disruptive behavior disorders; n = 331, 49.6%), trauma (n = 316, 47.3%), and other (e.g., autism spectrum disorders, substance use) (n = 93, 13.9%). Given high rates of comorbidity that are common in community mental health samples, the presenting problem of child cases was not mutually exclusive. Therapists most commonly reported on their implementation of MAP (n = 221, 33.1%) followed by TF-CBT (n = 193,28.9%) and SS (n = 125, 18.7%). A full list of client case and practice descriptives can be found in Table 1.

Table 1.

Descriptive statistics of child cases and practice.

	Frequency	%
Client Characteristics
Age
0–2	8	1.2
3–5	81	12.2
6–11	239	36.0
12–18	335	50.5
Female	356	53.4
Race/Ethnicity
Hispanic/Latino	464	69.6
Other Ethnic Minority	126	18.9
Non-Hispanic White	77	11.5
Presenting Problema
Internalizing	423	63.3
Externalizing	331	49.6
Trauma	316	47.3
Other	93	13.9
Practice
MAP	221	33.1
TF-CBT	193	28.9
SS	125	18.7
CPP	66	9.9
Triple P	57	8.5
CBITS	6	0.9

Note. MAP = Managing and Adapting Practice; TF-CBT = Trauma-Focused Cognitive-Behavior Therapy; SS = Seeking Safety; CPP = Child–Parent Psychotherapy; Triple P = Triple P–Positive Parenting Program; CBITS = Cognitive Behavioral Intervention for Trauma in Schools.

^aPresenting problems were not mutually exclusive.

Measures

Therapist characteristics

Questions about therapist characteristics were derived from the Therapist Background Questionnaire (Brookman-Frazee, Drahota, & Stadnick, 2012). Therapists completed questions about their sociodemographic and professional characteristics, including age, gender, race/ethnicity, licensure status, education, and current caseload.

Burnout

Therapists were presented with five items regarding their perceptions of job burnout using selected items from the Emotional Exhaustion subscale of the Organizational Social Context Questionnaire (Glisson, Green, & Williams, 2012; Glisson et al., 2008). Therapist participants rated their agreement with each item on a 7-point Likert scale from 0 (strongly disagree) to 6 (strongly agree). A mean composite score (possible range = 0–6) was calculated. Higher mean composite scores indicated stronger feelings of emotional exhaustion. In our therapist sample, the measure had excellent internal consistency (α = .89).

General attitudes toward EBPs

Two subscales from the self-report Evidence-Based Practice Attitudes Scale (EBPAS; Aarons, 2004) were used to assess general therapist perceptions towards adopting EBPs. Therapists completed the Openness and Divergence subscales, each of which consisted of four items and were rated on a 5-point Likert scale from 0 (not at all) to 4 (a very great extent). The Openness scale measures therapists’ willingness to try or use new interventions and EBPs and includes items such as “I like to use new types of therapy/interventions to help my clients.” The Divergence scale measures therapist views of EBPs as not clinically useful and less valuable than professional experience and includes items such as “Clinical experience is more important than using manualized therapy/interventions.” In the current sample, the Cronbach’s alpha demonstrated acceptable internal consistency for the Openness scale (α = .79) and the Divergence scale (α = .70).

Practice-specific perceptions

Therapist perceptions toward each of the six practice in which they had received training were assessed using the Perceived Characteristics of Intervention Scale (PCIS; Cook, Thompson, & Schnurr, 2015). Psychometric analysis of the original PCIS indicated a unidimensional construct of EBP-specific perceptions (Cook et al., 2015). For our study, eight of the original 20 items were administered to therapists in the current sample. Examples of items included “[The practice] is more effective than other therapies I have used,” “[The practice] is aligned with my clinical judgment,” “[The practice] is easy to use,” and “[The practice] can be adapted to meet the needs of my patients.” Therapists rated their agreement with each item on a 5-point Likert scale from 1 (not at all) to 5 (a very great extent). A mean composite score of the eight items was calculated (possible range = 1–5). Higher scores represented more favorable perceptions toward a practice. The total scale demonstrated excellent internal consistency for all practices in the current sample, with Cronbach’s alphas ranging from .92 to .96.

Therapist-reported client engagement challenges

Therapists were asked to identify a client to whom they have delivered one of the EBPs over the course of the past 2 months. Therapists provided information of therapeutic strategies delivered in the past 2 months with the identified client (Brookman-Frazee et al., 2018), then were asked to indicate whether they encountered any of seven client engagement challenges in EBP implementation with the identified client or their caregiver in same time frame: “Over the past two months, did the client or caregiver ….” The seven items were based on client engagement challenges that had been discussed in the literature (Becker et al., 2015; Haine-Schlagel & Walsh, 2015; Patterson & Chamberlain, 1994; Stadnick et al., 2016). The Expressed Client Concerns subscale included the following four items about the client: “Expressed concerns about the relevance/acceptability/helpfulness of an intervention strategy,” “Described practical barriers to using an intervention strategy (e.g., resources, materials, time),” “Verbalized a lack of familiarity of concepts presented in therapy,” and “Expressed difficulty mastering skills presented in therapy.” The Limited Client Engagement in Therapy Activities sub-scale included the following three items about the client: “Demonstrated apathetic or disinterested behavior,” “Avoided participating in therapy activities,” and “Consistently veered off topic from material presented.” Construct validity and internal consistency of items within the two scales are supported by confirmatory factor analyses reported in the following results.

Data analytic plan

Confirmatory factor analysis

A conceptually driven, confirmatory factor analysis (CFA) was conducted using Mplus statistical software to analyze the factor structure of therapist-reported client engagement challenges (Muthén & Muthén, 2012). The Expressed Client Concerns subscale included four items that involved clients voicing a concern about therapy related to (a) low relevance/acceptability/helpfulness of the intervention, (b) describing practical barriers related to the intervention, (c) describing difficulties mastering skills, and (d) stating a lack of familiarity of concepts. The Limited Client Engagement in Therapy Activities subscale included the following three items: (e) apathetic or disinterested behavior in session, (f) avoiding participation in therapy activities, and (g) consistently veering off topic. The CFA was specified using a weighted least squares estimator, a robust approach to the analysis of categorical variables (Flora & Curran, 2004; Muthén, Du Toit, & Spisic, 1997; Rhemtulla, Brosseau-Liard, & Savalei, 2012).

Predictors of therapist reported client engagement challenges

Since therapists were recruited from 69 agencies, with some therapists working within the same agency, assuming that each therapist is an independent observation may lead to underestimation of standard errors. Thus, we accounted for agency-level clustering in each of the following analyses through multilevel modeling (Hox, 2010). Two multilevel Poisson regression models were used to examine how client, therapist, and service characteristics were associated with counts of either (a) expressed client concerns or (b) limited client engagement in therapy activities. Poisson regression models were used to model the two count variables that displayed zero inflation within the sample, as many therapists endorsed no client engagement challenges in implementation with their client in the past 2 months (n = 164, 24.6%). The same predictors were entered into both models and included client, therapist, and service characteristics. Client predictors included child age, gender, race/ethnicity, and presenting problem. Presenting problem was effect coded with four categories: internalizing problem (anxiety and mood disorders), externalizing problem (attention/hyperactivity problems, disruptive behavior, and conduct problems), trauma, and other presenting problem (e.g., autism spectrum disorders, substance use). Therapist predictors included gender, race/ethnicity, licensure, attitudes toward EBPs (both general and practice specific), caseload, and burnout. Service predictors included practice (effect coded), treatment modality (group vs. individual), and caregiver involvement in sessions.

Associations with therapist-reported ability to carry out the EBP as intended

A multilevel regression model was used to examine whether therapist-reported client engagement challenges were associated with therapists’ own report of their ability to carry out the EBP with their identified client over the past 2 months. As with the previous analyses predicting client engagement challenges, we exampled the contribution of each predictor within a multilevel structure with therapists nested within agencies (i.e., two-level model). The model examined the effects of clients’ limited engagement challenges and expressed concerns over and above other client, therapist, and service characteristics that may relate to perceived ability to implement the EBP.

Results

The CFA of therapist-reported client engagement challenges

The CFA demonstrated good model fit for a two-factor solution measuring expressed client concerns and limited client engagement (root mean square error of approximation = .04, comparative fit index = .98, Tucker–Lewis index = .93, weighted root mean square residual = .59; Hu & Bentler, 1998; Yu & Muthen, 2002). Standardized factor loadings ranged from .46 to .75 (see Table 2). The conceptualized two-factor model was compared with a single factor model using the Mplus DIFFTEST function for comparing CFAs derived using mean- and variance-adjusted weighted least squares estimators (Li, 2016; Muthén & Muthén, 2012). The chi-square test for difference testing revealed that the theoretically driven two-factor model best fit the data, χ²(1) = 23.53, p < .001. There was a small correlation between the two scales (r = .19, p < .001).

Table 2.

Factor loadings from two-factor model.

Client Barriers	Factor Loading
Client Expressed Concerns
Expressed concerns about the relevance/acceptability/helpfulness of an intervention strategy	.77**
Described practical barriers to using an intervention strategy (e.g., resources, materials, time)	.46**
Verbalized a lack of familiarity of concepts presented in therapy	.65***
Expressed difficulty mastering skills presented in therapy	.65***
Limited Client Engagement
Demonstrated apathetic or disinterested behavior	.75***
Avoided participating in therapy activities	.83***
Consistently veered off topic from the material presented	.57***

Note. Standardized factor loadings depicted.

^**p ≤ .01.

^***p ≤ .001.

Descriptive analyses

The majority of therapists (n = 504, 75.4%) endorsed at least one EBP client engagement challenge. Exactly one half of the sample reported at least one expressed client concern over the past 2 months (n = 334,50.0%). Similarly, roughly one half (n = 318, 47.6%) of the sample endorsed at least one instance of limited client engagement over the past 2 months. Figure 1 displays the percentage of therapists who endorsed each EBP client engagement challenge within the index case in the past 2 months. The client engagement challenge with the highest rate of endorsement was clients expressing practical barriers (n = 197,29.5%), followed by clients veering off topic (n = 182,27.2%). The challenge that was endorsed by the lowest percentage of the sample was client’s lack of familiarity of concepts covered in therapy (n = 63, 9.4%). The average number of expressed client concerns endorsed across the whole sample was 0.75 (SD = 0.90), whereas the average number of limited engagement challenges endorsed was 0.74 (SD = 0.93).

Figure 1.

Therapist endorsement of evidence-based practice client engagement challenges.

Predictors of therapist reported client engagement challenges

Expressed client concerns

There were no client characteristics that significantly predicted the number of expressed client concerns reported by therapists. However, therapist attitudes toward EBPs predicted variance in this client engagement challenge. Therapists with higher scores on the EBPAS Openness subscale reported significantly more expressed client concerns (Incidence Rate Ratio [IRR] = 1.23, b = .21, p < .01). That is, for a 1-point increase on the Openness subscale, therapists are 1.23 times as likely to report client-expressed concerns. Alternatively, higher scores on the Divergence subscale therapists were associated with fewer reported expressed client concerns (IRR = .84, b = –.17, p < .01). There were no other therapist factors that significantly predicted the number of expressed client concerns. Therapist use of Triple P with the identified client was associated with 1.55 times greater likelihood of reporting a client-expressed concern (IRR = 1.55, b = .44, p < .01). There were no other significant effects of service characteristics on expressed client concerns (Table 3).

Table 3.

Poisson regression models predicting therapist-reported client engagement challenges.

	Expressed Concerns		Limited Engagement
	b	SE	b	SE
Intercept	−.56	.43	−.02	.46
Client Characteristics
Age (12–18)
0–2	.33	.45	−.97	.77
3–5	.03	.23	−.42	.28
6–11	.16	.12	−.07	.12
Female	−.02	.10	−.12	.10
Race/Ethnicity (Non-Hispanic White)
Hispanic/Latino	.13	.16	.02	.15
Other	.17	.18	.17	.17
Presenting Problem
Internalizing	−.01	.11	.11	.11
Externalizing	.13	.11	.24*	.11
Trauma	−.001	.13	.22†	.13
Other	.14	.16	−.07	.16
Therapist Characteristics
Female	−.01	.14	−.19	.14
Other Ethnic Minority (Non-Hispanic White)
Hispanic/Latino	−.20†	.12	−.35**	.12
Other	−.11	.13	−.15	.12
EBPAS Subscales
Openness	.21**	.08	.04	.08
Divergence	−.17**	.06	.01	.07
PCIS Mean	−.07	.06	−.19**	.06
Caseload	.01	.005	−.003	.01
Licensed	−.10	.10	−.11	.10
Emotional Exhaustion	.003	.03	.08**	.03
Service Characteristics
Practicea
CPP	−.27	.22	−.002	.27
CBITS	.31	.39	−.15	.49
MAP	−.15	.13	−.03	.14
SS	−.16	.19	.05	.17
TF-CBT	−.17	.13	.18	.15
Triple P	.44**	.15	−.05	.20
Caregiver Involvement	.30†	.16	.08	.14
Treatment Modality (Individual Only)
Group	.02	.27	−.17	.34
Individual and Group	−.17	.20	−.17	.21

Note. Reference group is in parentheses. EBPAS = Evidence-Based Practice Attitudes Scale; PCIS = Perceived Characteristics of Intervention Scale; CPP = Child–Parent Psychotherapy; CBITS = Cognitive Behavioral Intervention for Trauma in Schools; MAP = Managing and Adapting Practice; SS = Seeking Safety; TF-CBT = Trauma-Focused Cognitive-Behavior Therapy; Triple P = Triple P–Positive Parenting Program.

^aPractices were effect coded, and each is compared to the grand mean.

Limited engagement in therapy activities

Externalizing child presenting problem was the only client characteristic that significantly predicted more limited engagement such that in cases targeting a child externalizing problem, therapists were 1.27 times as likely to report limited engagement in therapy activities (IRR = 1.27, b = .24, p < .05). Hispanic/Latino therapists were less likely to report challenges associated with limited client engagement (IRR = .70, b = −.35, p < .01) compared with their non-Hispanic White counterparts. More favorable therapist perceptions of the specific EBP being used was significantly associated with fewer reports of limited client engagement (IRR = .83, b = −.19, p < .01). A 1-point increase in therapists’ ratings of emotional exhaustion was associated with 1.09 increased likelihood of reporting limited client engagement (IRR = 1.09, b = .08, p < .01). There were no significant effects of practice or any other service characteristic on therapist reports of limited client engagement (Table 3).

Associations with therapist-reported ability to carry out the EBP as intended

A multilevel regression model was used to examine whether therapist-reported client engagement challenges were associated with therapists’ own report of their ability to carry out the EBP with their identified client over the past 2 months. As with the previous analyses predicting client engagement challenges, we examined the contribution of each predictor within a multilevel model with therapists nested within agencies (i.e., two-level model). The model examined the effects of therapist-reported client limited engagement and expressed concerns over and above other client, therapist, and service characteristics that may relate to perceived ability to implement the EBP.

As shown in Table 4, therapists who reported more instances of limited client engagement in activities reported a lower ability to carry out the EBP (b = −.20, p < .001). However, client engagement challenges related to client-expressed concerns were unrelated to self-assessed ability to carry out the EBP as intended (b = −.02, p = .59). No client characteristics were predictive of a therapist’s reported ability to carry out the EBP as intended. As for therapist characteristics, more favorable attitudes toward the EBP as measured by the PCIS were related to a greater ability to carry out the EBP (b = .46, p < .001). No other therapist characteristics were related to the therapist’s ability to carry out the EBP. The only service characteristic associated with self-assessed implementation was SS practice use, which was associated with lower reported ability to carry out activities as intended (b = −.09, p < .05).

Table 4.

Regression model predicting therapist-reported ability to implement the evidence-based practice.

	b	SE
Client Characteristics
Age (12–18)
0–2	−.28	.39
3–5	−.30	.19
6–11	−.06	.10
Female	−.13	.08
Race/Ethnicity (Non-Hispanic White)
Hispanic/Latino	−.16	.12
Other	−.12	.14
Presenting Problem
Internalizing	−.003	.09
Externalizing	−.16	.09
Trauma	.15	.10
Other	−.13	.13
Therapist Characteristics
Female	.04	.12
Other Ethnic Minority (Non-Hispanic White)
Hispanic/Latino	−.09	.09
Other	−.08	.10
EBPAS Subscales
Openness	.12	.06
Divergence	−.08	.05
PCIS Mean	.46***	.05
Caseload	−.01†	.004
Licensed	−.08	.08
Emotional Exhaustion	−.05	.12
Service Characteristics
Practicea
CPP	.03	.06
CBITS	.02	.11
MAP	.03	.03
SS	−.09*	.04
TF-CBT	.02	.04
Triple P	−.01	.05
Caregiver Involvement	.09	.12
Treatment Modality (Individual Only)
Group	.49	.23
Both Individual & Group	−.16	.15
Client Engagement Challenges
Client-Expressed Concerns	−.02	.04
Limited Engagement in Activities	−.20***	.04

Note. Reference group is in parentheses. EBPAS = Evidence-Based Practice Attitudes Scale; PCIS = Perceived Characteristics of Intervention Scale; CPP = Child–Parent Psychotherapy; CBITS = Cognitive Behavioral Intervention for Trauma in Schools; MAP = Managing and Adapting Practice; SS = Seeking Safety; TF-CBT = Trauma-Focused Cognitive-Behavior Therapy; Triple P = Triple P–Positive Parenting Program.

^aVariables were effect coded, and each is compared to the grand mean.

^†p < .10.

^*p ≤ .05.

^***p ≤ .001.

Discussion

The purpose of the present study was to examine community therapist perspectives on client engagement challenges encountered in the delivery of multiple EBPs with diverse children and families within the context of a system-driven implementation effort in public mental health services. These challenges were common; three out of four therapists reported encountering at least one client engagement challenge in a given client case in the previous 2 months of care. Consistent with observations from previous community-based effectiveness trials, client engagement challenges aligned with two qualitatively distinctive factors: client-expressed concerns about EBP skills and limited client engagement in session activities (Dickson et al., 2017; Lau, Fung, Ho, Liu, & Gudiño, 2011; Stadnick et al., 2016). About half of the sample reported at least one of each type of client engagement challenge in the past 2 months. However, the correlation between these factors was low, suggesting that these client engagement considerations were qualitatively different and may arise amid different client profiles. Indeed, there was little overlap in the correlates of these two types of client engagement challenges with client, therapist, and service characteristics.

In terms of limited client engagement in therapy activities, about one in four therapists reported the occurrence of each of the three challenges, including clients veering off topic, appearing disinterested, or avoiding participation. This type of client engagement challenge was not associated with service characteristics such as the EBP being delivered or whether the caregiver was involved in treatment in the past 2 months. In addition, challenges related to low engagement were not related to child age, gender, or race/ethnicity but were more likely to be reported in the treatment of externalizing child behavior problems relative to internalizing, trauma-related or other presenting problems. These findings were consistent with findings from one study, which found that youth with internalizing disorders reported higher levels of treatment engagement, whereas youth with externalizing problems reported lower levels of engagement (Van Damme et al., 2015). Because most previous studies of EBP implementation have been limited to a single intervention for a single target problem, little is known about how presenting problems impact therapists’ experience with implementing multiple EBPs.

In terms of therapist characteristics, Hispanic/Latino therapists, therapists who liked the EBP being delivered and who were less burned out reported encountering fewer challenges related to limited client engagement. As the vast majority of Hispanic/Latino therapists reported that they were able to deliver therapy in Spanish, it is plausible that they encountered fewer challenges related to limited client engagement due to their ability to engage with Spanish-speaking clients in their native language. The fact that the therapists who were delivering preferred interventions were less likely to report trouble engaging clients in session activities likely reflects bidirectional links. As seen in preimplementation studies (e.g., Lewis & Simons, 2011), therapists who are unenthusiastic about EBPs are more likely to anticipate difficulties with client fit and engagement. Therapists’ negative assessments of specific EBPs may impact their implementation of clinical strategies to promote client participation. On the other hand, if clients demonstrate limited engagement in an EBP, this may negatively impact therapists’ attitudes toward that practice. Prospective studies using observational methods controlling for therapist adherence and competence may tease these explanations apart. With respect to burnout, therapists who are emotionally exhausted on the job may likewise have limited energy to support client engagement through adherent and competent EBP delivery. Conversely, routinely caring for challenging clients with low motivation is likely to take a toll on therapist vitality (Alves de Oliveira & Vandenberghe, 2009; Dynes, 2016). Finally, our findings noted that, along with negative perceptions of the EBP being delivered, therapists’ reports of client engagement challenges related to limited client engagement were associated with negative self-assessed delivery of the EBP with the client in the last 2 months. As such, therapist perceptions that clients were avoidant, nonparticipatory, or tangential in therapy sessions appeared to present a barrier to the implementation of EBPs in our sample.

The second class of client engagement challenges—client-expressed concerns about EBPs—was associated only with therapists’ general attitudes toward EBPs. There was no link between client-expressed concerns and therapist perceptions of the specific intervention being delivered. Furthermore, the directions of associations with EBPAS scales were unexpected. Therapists reporting higher openness to new treatments, such as EBPs, reported more challenges related to clients expressing concerns about EBP skill content and mastery, whereas therapists reporting that EBPs diverged from their own approach to therapy reported encountering fewer client-expressed concerns. This pattern coupled with the finding that client-expressed concerns were unrelated to therapist assessments of their ability to deliver the EBP may indicate that client-expressed concerns do not necessarily exemplify a barrier to implementation. Rather, it may be that therapists who are open to new treatment innovations, including EBPs, also approach therapy with an openness to engaging clients in dialogue about their concerns and questions about treatment and their progress in skill development. In fact, soliciting and addressing these types of concerns have been identified as helpful ways to promote engagement in care and could indicate a stronger relationship with the therapist, which facilitates a comfort expressing concerns (Haine-Schlagel, Martinez, Roesch, Bustos, & Janicki, 2016). Alternatively, regarding the finding from the Divergence subscale, it is plausible that therapists who report that EBPs diverge from their own approach might work harder to deliver the EBP and therefore encounter fewer client-expressed concerns.

Also of note, therapists delivering Triple P, a parent training program, reported more client-expressed concerns, and this would be consistent with the collaborative approach of many parent training EBPs that encourage discussion of how skills fit with the family context and troubleshooting difficulties putting new parenting skills in place (e.g., Webster-Stratton & Herbert, 1994). Observational data from an effectiveness trial of AIM HI, a collaborative caregiver-directed intervention for children with autism spectrum disorder, suggest that client-expressed concerns are positively associated with indicators of parent participation and therapist adherence to the intervention (Guan et al., 2018). It is possible that clients who speak up about their worries or misgivings in treatment evoke high-quality care in which therapists must work harder to explain and teach skills. Alternately, community therapists who adhere to EBP protocols that outline collaborative processes may be effective in generating critical discussions in therapy. Taken together, our findings suggest that these client engagement challenge categories may represent two sides of a coin with regard to client engagement in the delivery of EBPs by community therapists in usual care.

Limitations

This study represents an initial effort to characterize the types and predictors of client engagement challenges encountered by community therapists delivering multiple EBPs in children’s mental health. However, interpretations of the findings must be tempered by multiple study limitations. First, the cross-sectional survey design precludes conclusions about causality and directionality of the observed associations. With regard to multiple findings, we have noted that the links between the reported client engagement challenges and predictor variables could be plausibly bidirectional. Further, the composition of the study sample could limit generalizability of the findings. The proportion of unlicensed therapists in our sample(i.e., 57%) is larger than that found in a nationally representative sample of therapists working in community mental health settings, which was closer to 40% (Schoenwald et al., 2008). Therefore, it is possible that the reported client engagement challenges in this study were related to limited experience providing therapy and not related to EBP implementation specifically. However, it is important to note that licensure status was not significantly associated with therapist report of either type of client engagement challenge. Also, as noted previously, the study relied on therapists’ self-reported perceptions of implementation processes and outcomes. It is unclear whether therapist reports of client engagement challenges reflect actual in session client behavior. A next step in this line of investigation will be to utilize observational methods to understand therapist EBP implementation experiences and predictors of these client engagement challenges. In addition, it would be valuable to study how client engagement challenges impact EBP delivery in a variety of settings to inform implementation efforts across different contexts. Another limitation of the current study is that it is unknown whether therapists reported on the occurrence of engagement challenges in the context of cases representative of their full caseload. To offset potential demand characteristics that could influence therapists to select cases with “successful implementation,” study recruitment efforts (e.g., presentations at meetings, consent documents, and recruitment emails) emphasized the goal of understanding a range of barriers to therapists making EBPs work for children and families in real-world community settings. Nonetheless, it is possible that some therapists may have focused on cases in which they felt that their EBP implementation went well, rather than focusing on cases that were more challenging. To the extent that such bias affected case selection, the rates of occurrence of engagement challenges may not be generalizable. However, findings concerning the agency, therapist, and client characteristics associated with client engagement challenges may still shed light on conditions that complicate therapists’ delivery of EBPs. Finally, it is important to note that this study investigated only client engagement challenges within the context of a fiscal mandate to use multiple EBPs, and it is not clear if these challenges were specific to EBP implementation or would similarly arise if therapists were provided treatment as usual.

Implications

Notwithstanding these limitations, the findings of the current study suggest value in distinguishing between at least two types of client engagement challenges in the implementation of EBPs in community settings. This could have important implications for therapists delivering EBPs or supervisors overseeing implementation. First, it is valuable to understand the potential differences in the impact of the two types of client engagement challenges. Although we had conceived of both types of client engagement challenges as negatively impacting EBP implementation, clients verbalizing their concerns or doubts about an intervention may not necessarily portend problems in implementation. In fact, it is possible that client-expressed concerns may at times denote a positive working relationship where input is being productively elicited and shared. Whereas client withdrawal, avoidance or diversion of therapy activities may be more likely to derail therapists from EBP implementation. Therefore, community EBP implementation efforts may be aided by implementation support strategies that help therapists promote client engagement and participation in therapy activities, potentially including the active encouragement of clients to express their concerns with treatment rather than passively disengaging from the process. These implementation supports may include training in a range of available youth and caregiver engagement approaches and protocols detailed in the literature for children’s mental health treatments (e.g., Becker, Boustani, Gellatly, & Chorpita, 2017; Haine-Schlagel et al., 2016; Nock & Kazdin, 2005; Sterrett, Jones, Zalo, & Shook, 2010). These protocols have been shown to improve client-level outcomes, and there may be reason to suspect that they could also enhance provider implementation outcomes including EBP adherence, fidelity, and competence.