Behavioral Rehearsal for Analogue Fidelity: Feasibility in a State-funded Children’s Mental Health Initiative

Abstract

A substantial number of evidence-based treatments (EBTs) are available, but are delivered infrequently in public mental health. To improve the quality of care, some states and systems have focused on EBT training; however, these efforts have rarely included objective measurement of clinician fidelity because of feasibility issues. The primary goal of the current study was evaluating the feasibility of the Behavioral Rehearsal (BR) method to assess “analogue fidelity” in a children’s mental health quality improvement initiative. Results indicated low—but representative—clinician participation. Participants demonstrated greatest improvement at post-training with maintenance or decreases at 6-months post-consultation. Implications for future use of BR are discussed.

One in five youth in the United States experience a diagnosable and clinically impairing psychiatric disorder (Merikangas et al., 2010). Less than 20 percent of those with need will receive services, with only a small proportion receiving evidence-based treatments (EBTs; President’s New Freedom Commission on Mental Health, 2003). One of the biggest challenges currently facing the mental health field is the dissemination and implementation of EBTs to community settings (McHugh & Barlow, 2010). A range of implementation strategies have been developed to improve implementation efforts (Powell et al., 2015). Among these strategies, training community clinicians in EBTs is one of the most frequently used (Addis & Krasnow, 2000; Herschell, McNeil, & McNeil, 2004; Rakovshik & McManus, 2010; Williams, Martinez, Dafters, Ronald, & Garland, 2011), with research suggesting that active, experiential training, followed by additional supports (e.g., EBT consultation or supervision; other organizational supports) may be required to achieve clinician behavior change and EBT fidelity (Beidas & Kendall, 2010; Herschell, Kolko, Baumann, & Davis, 2010).

Common Elements Approaches

One barrier to clinician training in EBTs is that most EBTs are disorder-specific. This singular focus is problematic given that clinician caseloads in community mental health are typically comprised of youth with a wide range of disorders and high rates of comorbidity (Weisz & Gray, 2008). Recent efforts to increase the implementation of effective practices have sought to shift the focus from training in individual EBTs to training in the common practice elements (e.g., exposure, cognitive processing) across EBTs to improve usual care (Beidas, Koerner, Weingardt, & Kendall, 2011; Chorpita, Daleiden, & Weisz, 2005; Garland, Bickman, & Chorpita, 2010). Such approaches also include training in cross-cutting Cognitive Behavioral Therapy (CBT) techniques (e.g., using standardized assessment, CBT model explanation, assigning/reviewing homework; Sburlati, Schiering, Lyneham, & Rapee, 2011) that support treatment delivery (e.g., Weisz et al., 2012). Common elements approaches allow for the aggregation and dissemination of research knowledge, facilitate more flexible training and implementation, offer greater coverage for youth, and seem to have greater appeal among community providers (Bernstein, Chorpita, Daleiden, Ebesutani, & Rosenblatt, 2015; Chorpita & Daleiden, 2009; Chorpita, et al., 2005; Chorpita et al., 2015).

In Washington State, a common elements training approach informed by the work of Weisz and Chorpita (Chorpita et al., 2005; Weisz et al., 2012), the CBT+ Initiative (pronounced “CBT Plus”), has been funded by the state since 2009 (see Dorsey, Berliner, Lyon, Pullmann, & Murray, 2014). In CBT+, clinicians learn CBT for the most common clinical problems in children (i.e., behavior problems, depression, anxiety, and trauma-related anxiety). Due to limited Initiative funding (i.e., $80,000–100,000 per year), evaluation efforts have predominantly relied on clinician self-report measures (see Dorsey et al., 2014; Lyon, Dorsey, Pullmann, Silbaugh-Cowdin, & Berliner, 2014), which may be problematic in light of findings that self-report ratings can be higher than objective ratings (e.g., Hurlburt, Garland, Nguyen, & Brookman-Frazee, 2010; Miller & Mount, 2001; Nakamura et al., 2014) (for exceptions, see Chapman, McCart, Letourneau, & Sheidow, 2013; Ward et al., 2013). Given that the ultimate goal of EBT implementation is improved outcomes for clients, EBT initiatives must demonstrate that trainees can deliver interventions with fidelity to ensure that clients served by trainees will have positive outcomes. Research suggests that high-fidelity service delivery is associated with more positive outcomes (e.g., Huey, Henggeler, Brondino, & Pickrel, 2000; Washington State Institute for Public Policy, 2004). Fidelity refers to the “extent to which the intervention was implemented as intended” following training and includes—at a minimum—adherence and competence (Perepletchikova, Treat, & Kazdin, 2007, p. 829).

As in CBT+, few other state or system-funded quality improvement efforts have included rigorous assessment of clinician fidelity following training. In contrast, externally funded research studies evaluating training effectiveness typically include objective, observational ratings of fidelity (Proctor et al., 2011), which can be both time-consuming and costly. There is growing recognition, however, that traditional research methods—direct observation and coding of therapist sessions with clients—may not be feasible for large scale implementation research and practice without substantial resources, yet, limited alternatives currently exist (Schoenwald et al., 2011; Schoenwald & Garland, 2013).

Potential Promise of Behavioral Rehearsal Methodology

One innovative methodology to measure “analogue” fidelity is Behavioral Rehearsal (BR), which is defined as “a simulated interaction between a trainee and another individual” (Beidas, Cross, & Dorsey, 2014, p. 2). In BR, the trainee engages in a role play in which he or she is asked to demonstrate the technique being evaluated. Role plays may be performed with an actor trained to respond in a standardized manner, which facilitates comparison of performance across individuals and time. BR may offer a more efficient assessment of fidelity than direct observational ratings (i.e., interactions with a client) and a more effective assessment than self-report (see Beidas et al., 2014 for more information on BR methodology in mental health). BR is commonly used in the medical field to assess physician competency (referred to as standardized patient methodology; see Shah et al., 2012). In children’s mental health, BR was used to assess fidelity in two externally-funded research studies (see Beidas, Edmunds, Marcus, & Kendall, 2012; Cross et al., 2011). In addition, Nakamura et al., (2014) employed BR to test the effectiveness of a pilot train-the-trainer program for CBT in a state-funded implementation study, without the resources of external-funding (i.e., suggesting feasibility). The study included BR and self-report measures of fidelity, with findings that although both suggested skill improvement for trainers and their clinician trainees, self-report ratings were comparatively inflated. Taken together, these studies support the potential promise of BR methodology, but more research is needed on feasibility.

Study Goals

The primary goal of the current study was to assess feasibility of BR methodology in a state-funded quality improvement initiative designed to evaluate training effectiveness. Specifically, we were interested in whether or not trainees would volunteer to do BRs, retention over multiple assessment points, and representativeness of participating clinicians. Prior to the current study, trainee fidelity was evaluated using self-report measures and voluntary evaluation participation rates were approximately 60%, with no significant differences found between trainees who did or did not participate in the evaluation (e.g., Lyon et al., 2014). For the current study, we hypothesized that including BR would result in a decline in evaluation enrollment rates, due to increased effort required (e.g., preparation, scheduling, time to do the BR), in addition to potential trainee anxiety or discomfort associated with engaging in BR (Beidas et al., 2013). To our knowledge, BR has not been used to evaluate clinician fidelity in large state-funded implementation initiatives. Our goal was to examine the feasibility and utility of a relatively brief BR (less than 10 minutes each) that might be used in place of, or to supplement, therapist self-report.

The secondary goal was to assess training effectiveness by using BR to evaluate clinician analogue fidelity for two cross-cutting, core CBT techniques: CBT Model Explanation and Homework Planning. We hypothesized that CBT model explanation is a foundational technique, as clinicians need to understand and be able to explain CBT for varying presenting problems. Homework planning was selected because homework—defined as between session exercises in which clients practice skills learned in session (Kazantzis, Deane, Ronan, & L’Abate, 2005)—is both central to CBT and associated with enhanced therapy outcomes (see Kazantzis, Deane, & Ronan, 2000; Kazantzis, Whittington, & Datillio, 2010 for meta-analytic reviews). However, usual care research in children’s mental health indicates that homework assignment and review are substantially underutilized (Garland et al., 2010). Our evaluation focused on assessing analogue fidelity for these two techniques across the three internalizing presenting problems targeted in CBT+ (i.e., anxiety, depression, and posttraumatic stress [PTS]).

Method

Procedure

CBT+ training and consultation

CBT+ includes 3 days of active and experiential training (including multiple, in-training BRs with peer and trainer feedback) on the common practice elements that comprise the four CBT approaches (e.g., psychoeducation, cognitive restructuring, exposure) and core CBT techniques (e.g., agenda setting, homework assignment) (see Dorsey et al., 2014 for more information on the CBT+ approach and development). Approximately 100 clinicians are trained each year. Post-training, participants receive 6 months of biweekly telephone consultation (12 calls) led by a CBT expert and predominantly focused on case presentations. During each call, one to two clinicians present cases, and receive expert and peer feedback. Calls also include some combination of follow up on previously presented cases (e.g., clinician implementation of recommendations, client response), time for clinician questions, consultant modeling, opportunity for clinician role play, and linkage to case-specific CBT resources (http://depts.washington.edu/hcsats/PDF/TF%20CBT/CBT_Plus_NB.html). However, microanalysis of consultation has not been conducted (see Edmunds et al., 2013 for an example). CBT+ also offers yearly advanced trainings, a listserv, and supervisor-specific supports.

Recruitment

All procedures were reviewed and exempted by the Washington State IRB. Participants were recruited in October and November of 2011, when they registered for the mid-November CBT+ training. An email invitation described the voluntary (i.e., lack of participation would not impact to training or consultation) CBT+ evaluation (i.e., self-report measures and BRs at three time points—pre-training, immediately post-training, after 6-months of consultation) and incentive for participation ($10 per time point). There were no exclusionary criteria.

Behavioral rehearsals

Two structured BRs, each with versions for the three presenting problems, were developed by the authors (CBT experts) to assess participant adherence and competency in simulated clinical interactions with a child (see Beidas et al., 2014 for procedural details). All BR versions were evaluated by outside CBT experts to ensure that each was a realistic portrayal and to equate for level of difficulty. Research assistants were trained to portray youth experiencing symptoms of anxiety, depression, or post-traumatic stress (PTS). Clinicians were asked to participate in two 8-minute BRs by telephone at each time point (total of 16 minutes). Twenty-four hours before the scheduled BR, clinicians received an email that included a client vignette with detailed instructions for what clinicians should do for each of the two BRs. The first BR involved explaining the CBT model to the child. The second involved collaboratively planning for a homework task, either exposure (anxiety and PTS) or pleasant activity scheduling (depression) (see Appendices for vignettes, clinician instructions, and scoring). The presenting problem focus (i.e., depression, anxiety, PTS) for the BRs at each of the three time points was counterbalanced across participants to control for potential presenting problem order effects (e.g., Clinician A: Pre-training – Depression; Post-training – Anxiety; Post-consultation –PTS; Clinician B: Pre-training – PTS; Post-training – Depression; Post-Consultation – Anxiety). BRs were digitally audio-recorded, transcribed, and later independently coded using two checklists developed by the study team to code adherence and skill (see subsequent descriptions in measures).

Participants

Participants were 41 clinicians and supervisors employed at public mental health clinics in Washington State, out of the 100 who registered for the fall 2011 CBT+ training. Participants were predominantly female (70.7%) and Caucasian (78%), with fewer African American (7.3%), Latino/a (4.9%), and Asian (4.9%) participants (two were missing data on race/ethnicity). Most were between 25–29 (24.4%) or 30–39 (36.6%) years old, with 14.6% between 40–49 and 17% over 50. In terms of role, 71.8% were clinicians, 12.8% were supervisors, and 7.7% reported both roles. Self-reported experience with CBT varied, 9.8% rarely (1–5% of clients) used CBT, 4.9% occasionally (6–25% of clients), 17.1% sometimes (26–50% of clients), 31.7% often (51–75% of clients) and 22% almost always (76% or more of clients) used CBT.

Measures

Clinician demographics and background

An existing Washington State CBT+ Initiative self-report measure assessed demographic and background characteristics including race/ethnicity, age, gender, agency role, orientation, and CBT experience.

CBT model explanation: Adherence and skill checklist

Therapist adherence and skill for the first BR (BR1; explaining the CBT model) was assessed by coding three elements: 1) introducing the CBT triangle diagram; 2) teaching that thoughts, feelings, and behavior are linked (e.g., a specific thought leads to related feelings and behaviors); and 3) demonstrating that changing either thoughts or behavior changes the other related points on the triangle, ideally using a child-appropriate example and the triangle diagram. Each element was coded on a 7-point Likert scale from 0 (not present/non-adherent) and 1 (present/adherent; low skill) to 6 (present/adherent; high skill). An overall mean score representing both adherence and skill was calculated in line with frequently used coding measures in children’s mental health research (e.g., McLeod & Weisz, 2010). To aid interpretation, we considered scores 1–2 to be “low”; 3–4 to be “medium”; and 5–6 to be “high” analogue fidelity. Using a 70% criterion (Beidas et al., 2012), scores 4.2 and higher (i.e., ≥ 70% of a perfect score of 6) were considered to represent acceptable analogue fidelity. Coders were masked to time point and participant characteristics.

Homework planning: Adherence and skill checklist

Therapist adherence and skill for the second BR (Homework Planning) were informed by elements captured on the homework item of the Cognitive Therapy Scale (Young & Beck, 1980). The goal was to test clinician ability to help the client plan for use of a key behavioral change strategy between sessions. Five elements were coded: 1) Introducing the technique (exposure/pleasant activity scheduling); 2) providing a rationale for expected clinical benefit; 3) identifying activities for the technique (e.g., for exposure: identifying possible tasks to try at home); 4) role play/practicing the technique with the child (e.g., for pleasant activity scheduling: a brief mood-boosting activity); and 5) collaborative facilitators/barriers discussion (e.g., determining the best time, identifying specific facilitators of completion [e.g., parental reminder], addressing barriers to completion). The scoring procedure (i.e., 0–6) and interpretation of “medium” and “high” scores were identical to the first BR.

Coder training and reliability

The initial codebook for both BRs was developed by the first, second, fourth, and last authors (all CBT experts, spanning the three target presenting problems. All coders (N = 5) were post-baccalaureate or undergraduates. Training took approximately 17 hours and included readings, lecture, and demonstration of techniques by the codebook developers, followed by coding meetings with the first author (group coding, discussion, manual refinement). Between meetings, BRs were coded independently and then discussed in the following meeting (see Beidas et al., 2014 for more information on the protocol). Once the codebook was finalized, prior to coding for the study, all coders met a single-rater, random effects, absolute agreement Intraclass Correlation Coefficient [ICC_(2,1)] threshold of .80 compared to the first author (2 additional hours of coding). To ensure continued high reliability throughout the process, at least 20% of BR1 and BR2 audio files were double coded. Reliability was acceptable for BR1, and very good for BR2 (Cicchetti, 1994), with absolute agreement ICC_(2,1) of .68 and .82, respectively. Once trained, coders were able to code each 8-minute BR in 15–30 minutes. Variation in coding time was typically due to variation in audio recording quality which affected audibility.

Data Analyses Plan

Descriptive analyses, chi squares, and t-tests were used to assess BR feasibility. To assess analogue fidelity, Longitudinal Multilevel Models (MLM; time point score nested within clinician) using Full Maximum Likelihood estimates, robust standard errors, and autoregressive covariance structures with heterogeneous variances (Singer & Willett, 2003), were run for overall mean sum scores and individual items for each BR, including a linear and a quadratic time trend. These tested for changes over time from pre- to post-training, and post-training to 6-months post-consultation. The intercept term was allowed to randomly vary, with time trends fixed due to a small sample size (N = 41), which led to nonconvergence and unstable estimates in alternative models that included varying time trends. MLM results were replicated using Repeated-Measure Analyses of Variance (RM-ANOVAs) for the subsample with both BRs at all three time points (n = 18; 41%) to perform a check on the MLM results.

Results

BR Feasibility

The 41 participants who agreed to participate in the evaluation represent 41% of the 100 registrants for the CBT+ training. Of 41 participants, 35 (85%) completed BRs for at least two of the three time points. Retention of BR participants from pre-training to post-training (about a 2-week period) was 82%. Retention from pre-training to post-consultation (about a 6.5-month period) was 68% for clinicians still employed at their organizations at the post-consultation time point (seven participants left their organizations during consultation). Only 18 participants (44%) completed BRs at all three time points; however, one-third of this attrition was due to clinicians leaving their organizations during consultation and, therefore, no longer eligible for initiative or study participation (state-funding requirement).

BR Sample Representativeness

We explored potential differences in trainees who volunteered to participate (n = 41) and those who did not (n = 59) (e.g., generalizability of the sample). Crosstabulations with χ2 tests and t-tests were run to examine baseline differences, with no statistically significant differences found between the two groups. Potential areas of difference examined were demographic (i.e., age, race/ethnicity, gender) and background characteristics (i.e., role [supervisor; therapist], level of education, number of years providing therapy), CBT experience, and receipt of CBT-specific supervision.

Missing Longitudinal Data for the BR Sample

Given the high rates of attrition in the BR sample, we explored attrition bias by comparing participants who completed BRs at all three time points (n = 18) to those who missed at least one time point (n = 35). Independent samples t-tests and crosstabulations with χ² tests revealed no statistically significant differences in participant demographics, background characteristics, or in pre-training BR item or total scores.

Model Explanation Adherence and Competency

For the first BR, Model Explanation, both the overall mean score and the Introducing the CBT Triangle item score significantly increased from pre- to post-training and then significantly flattened from post-training to post-consultation (see Table 1; Figure 1). For the Demonstrating Change item, scores significantly increased from pre- to post-training, and also to post-consultation. Scores did not significantly change for the Linking Thoughts, Feelings, and Behaviors item, although they trended toward improvement (p = .059). From pre- to post-training to 6-months post-consultation, the percentage of participants who scored in the acceptable or higher analogue fidelity range on overall mean score (≥ 4.2) changed from 20.5% to 55.2% to 52.4%. While only one participant (2.6%) scored in the high range of 5 or above at pre-, about one-fifth to a quarter of participants scored in the high range at post-training (22.6%) and at 6-months post-consultation (23.8%).

Table 1.

Analogue Fidelity: Multilevel Model Estimated Overall and Item Scores, and Slope Significance Values

Model Introduction	Pre-training	Post-training	6-month post-consultation	Pre-training variance	Linear slope p	Quadratic slope p
Overall mean	2.83	3.88	4.15	0.67*	< 0.001	0.033*
Introducing CBT triangle	2.86	4.21	4.20	0.98	0.004	0.048*
Linking thoughts, feelings, and behaviors	3.56	4.31	4.62	0.73*	0.059	0.400
Demonstrating change	2.27	3.28	3.53	1.34*	0.006	0.141
Homework Assignment	Pre-training	Post-training	6-month post-consultation	Pre-training variance	Linear p	Quadratic p
Homework overall mean	2.29	3.12	2.51	0.05	< 0.001	< 0.001*
Introduce specific technique (BA/exposure)	1.36	2.39	1.56	0.26	0.011	0.018
Provide Rationale for BA/Exposure	1.84	2.93	1.86	0.23	0.003	< 0.001*
ID Activities for BA/exposure	3.82	4.7	4.22	0.12	< 0.001	< 0.001*
Role Play or Practice BA/Exposure	1.53	1.97	1.37	0.64	0.354	0.297
Homework or practice plans for BA/exposure	2.95	3.43	3.43	0.40	0.302	0.437

^*p < .05

Figure 1.

Behavioral rehearsal mean scores at all time points. Scores on Model Introduction significantly increased from pre-training to post-training, then significantly flattened from post-training to 6-months post-consultation (p < .05). Scores on Homework significantly increased from pre-training to post-training, and then significantly decreased to near pre-training levels (p < .05).

Homework Planning Adherence and Competency

For the second BR, Homework Planning, scores for the overall mean and two items, Introduce Specific Technique and Provide Rationale, significantly increased from pre- to post-training and then significantly decreased to near pre-training levels at post-consultation. One item, Identify Activities for the Technique, increased from pre- to post-training and then showed a slight, but statistically significant decrease at post-consultation. There were no significant changes for two items: Role Play/Practice and Facilitators/Barriers Discussion. From pre- to post-training to 6-months post-consultation, the percentage of participants who scored in the acceptable range or higher on overall mean score (≥ 4.2) changed from 5.3% to 9.1% to 4.5%. None scored in the high range at pre-, post-training, or 6-months post-consultation.

RM-ANOVA replication

RM-ANOVAs repeating these analyses with the subsample that completed BRs at all three time points (n = 18; 41%) resulted in nearly identical findings to the full sample analyses with one exception: the item Introducing the CBT Triangle significantly increased at each follow-up time point in the RM-ANOVA (pre- M(SD) = 3.12 (2.18), post-training M(SD) = 3.71 (1.53), post-consultation M(SD) = 4.53 (1.51), F_(2,32) = 3.59, p < .039) instead of flattening as was found in the MLM analysis with all 41 participants. All RM-ANOVA results are available upon request.

Discussion

The BR methodology holds promise for measuring analogue fidelity in an effective and efficient way (Beidas et al., 2014), but to date has predominantly only been used in externally-funded research studies. This study was one of the first to test the feasibility of the BR methodology for assessing training effectiveness in a quality improvement initiative through the measurement of analogue fidelity. Our experience suggests that including BR to more rigorously assess analogue fidelity was feasible; however, fewer trainees participated and study attrition was higher in comparison to previous evaluation efforts (that did not include BR), suggesting that future efforts should potentially allocate greater resources to enrollment and retention. Using BR methodology, we determined that only half the clinicians achieved acceptable levels of analogue fidelity, and for only one of the two techniques assessed (i.e., model explanation). Feasibility and training effectiveness are discussed further below.

BR Feasibility

Fewer trainees overall (41%) were willing to participate in an evaluation that included BR compared to past evaluations that included only self-report (e.g., 60% enrollment), resulting in about a 30% difference between enrollment rates. In addition, study attrition at the two follow-up time points was about 20% higher when BR was included (see Lyon et al., 2014; Dorsey et al., 2014). Although we cannot definitively link enrollment and retention to BR inclusion, both variables were relatively stable across three prior evaluations (Dorsey et al., 2014). BR participants; however, seemed representative of trainees in that there were no significant differences between participants and non-participants across a range of characteristics (e.g., demographics, background, CBT experience). In comparison to some externally-funded training studies in which training and research participation were linked, per state mandate, trainees in the current study had access to the training and consultation regardless of evaluation participation and received only a small incentive ($10 per time point). Retention over time was relatively low, although a substantial proportion of the attrition was due to participants leaving their organizations, reflective of high turnover rates in public mental health (e.g., Beidas et al., 2015b). Although we did not assess factors associated with initial and ongoing BR participation, other research suggests that clinicians who engage in BR may feel some anxiety or discomfort (Beidas et al., 2013). Additionally, productivity requirements likely played a role, in that clinicians were already out of the office for the 3-day training, so participating in the evaluation added more lost time just prior to and after the training. We estimated that the two BRs took approximately 20 minutes each (preparation; BR participation), representing about 40 minutes for each time point. Scheduling the assessment further from the training (2–3 weeks before/after) might improve enrollment. Alternatively, randomly or purposively sampling a subset of training registrants (e.g., based on demographics or background characteristics) with greater incentives for participation, could provide a general estimate of analogue fidelity using a smaller sample size that could then be generalized.

Given our somewhat low rates of enrollment and retention, our team identified strategies that might assist other implementation researchers in using BR even when resources are limited. These include: 1) requiring participation in BR to register for training (as long as this strategy itself does not negatively impact training participation); 2) embedding BR participation into the training using technology (see Schulman et al., 1999 for an example from medicine); 3) enlisting organizational leaders and clinical team managers to encourage and support participation; 4) providing additional incentives to clinicians and organizations who participate, such as access to webinars that are of value to participants (e.g., ethics training); and 5) finding ways to give clinicians and/or organizations feedback on performance so that BR is viewed as having professional development value.

One remaining challenge for using BR outside of externally funded research projects—when using standard research procedures—is the time involved in training actors and coders and the coding process. Without a state-university partnership, which provided access to volunteer student research assistants, faculty willing to devote additional effort, and a state funder that valued evaluation (e.g., contract included a 25% paid research assistant position), time for these activities alone may prohibit BR use. Although other state-university partnerships exist (e.g., New York and New York University: www.ctacny.com), reliance on such partnerships would make BR an unlikely candidate for assessing training effectiveness. One option is to use BR as more of a quality control tool, without using objective, masked coders and a formal coding scheme. Trainers, consultants, or clinician’s own supervisors could engage clinicians in BR and make general ratings (e.g., unacceptable, acceptable, very good), with ratings tied to specific follow-up activities (e.g., focus supervision time on challenging techniques; additional training/reading). One study with external, expert consultants demonstrated that BR inclusion in consultation predicted greater EBT delivery in the next session (Bearman et al., 2013).

Ongoing and soon-to-be-initiated research by our team will provide some answers to guide decisions about BR potential, effectiveness, cost, and particularly, how BR can be used as a quality control mechanism in everyday work settings. The first author is currently testing BR as a supervision tool in community mental health, as it can serve as both an assessment method and a mechanism for building competency if the clinician receives feedback on his/her effort (REMOVED FOR BLIND REVIEW). Anecdotally, supervisors and clinicians have reported BR as a worthwhile skill-building activity. This study will provide guidance on the effectiveness of BR under more feasible conditions—when used as part of supervision by community-based supervisors—in improving in clinician fidelity. Additional research on optimizing BR feasibility, testing BR validity, and developing other efficient methods of fidelity assessment are needed. Findings from the Nakamura et al. (2014) pilot study suggested that participants overestimate their own performance, in comparison to results from coded BR, which parallels findings from research comparing objectively coded therapy sessions to self-report (e.g., Hurlburt et al., 2010). Two of the authors (RB and SD) will be conducting a study that should offer some information on both accuracy and cost-effectiveness the BR methodology.

Training Effectiveness

Inclusion of BR allowed for objective evaluation of skill in two core CBT techniques for a small, but representative group of trainees. Results indicate that for both techniques, clinicians gained skills with training, moving from mean “minimal/low” to “medium” levels of analogue fidelity. After 6 months of consultation, improvement in Model Explanation was maintained, but fidelity for Homework Planning declined to near pre-training levels (i.e., to mean “low” levels). A little over 50% achieved acceptable levels of analogue fidelity for Model Explanation at post-training and post-consultation, which is mostly in line with other studies using a similar training and supervision approach (e.g., Beidas et al., 2012; 61%), particularly given the more complex focus on CBT for multiple presenting problems. Although disappointing, these rates may be the best that can be achieved with current approaches to implementation, which may not pay enough attention to other contextual factors that can explain as much or more of the variance in clinician practice (e.g., organizational climate (see Beidas et al., 2015a)). In contrast to Model Explanation, very few clinicians achieved acceptable levels of analogue fidelity at either time point for Homework Planning. These rates are particularly disappointing, but are not entirely out of line with other studies of technique use among EBT-trained clinicians in public mental health settings (McLeod, Smith, Southam-Gerow, Weisz, & Kendall, 2015).

Differences in Analogue Fidelity between Model Explanation and Homework Planning

One question these findings raise is why analogue fidelity was higher for Model Explanation than for Homework Planning. We viewed Model Explanation, which requires understanding the CBT theoretical model of change and how to explain it to a client, as a foundational skill that underpins CBT treatment delivery. BR results suggest that, at least for this small sample, the majority of trainees acquired and maintained acceptable analogue fidelity. Future research should investigate factors associated with trainees who do not attain acceptable levels during the training and consultation period (e.g., type of supervision received, implementation climate). In contrast to maintained improvement for Model Explanation, clinician analogue fidelity for Homework Planning did not operate as expected, based on findings from prior reviews of clinician training (Beidas & Kendall, 2010; Herschell et al., 2010). We see four potential reasons for lower rates of improvement specifically for Homework Planning. First, we conceptualized this technique as more complex in that it required greater tailoring both to the treatment target (depression, anxiety, PTS) and to the individual child (e.g., identification strategies to facilitate completion). Our findings of low scores at pre-training and limited improvements over time may partially explain prior research demonstrating that homework planning infrequently occurs in public mental health by both objective (Garland et al., 2010) and clinician report (Dorsey et al., 2014). Potentially, if clinicians struggle with this technique (as our results suggest), they may be less likely to use it in their clinical practice.

Second, homework planning may also be less consistent with other clinical orientations (e.g., client-centered) and therefore less likely to occur (Brookman-Frazee, Garland, Taylor, & Zoffness, 2009; Brookman-Frazee, Haine, Baker-Ericzén, Zoffness, & Garland, 2010). In contrast to CBT, other orientations view what happens in session over what happens outside of session (e.g., homework) as the mechanism of change. Third, some research suggests that clinicians view homework as unrealistic for some families with high levels of outside stressors (Ringle et al., 2015). Finally, our speculation is that homework planning and review typically receive too little attention in training and consultation and need more targeted initial training and ongoing support to increase effective use. In our study, improvement with training but regression over the consultation period suggests that consultants need to focus more on building homework planning skills and providing opportunities for expert modeling and trainee behavioral rehearsal on the calls. Other work suggests homework is overlooked during supervision in community mental health (Accurso, Taylor, & Garland, 2011). As a result, some studies are attempting to better integrate coverage of core CBT techniques, like homework, into supervision (Dorsey et al., 2013). Regardless, these findings suggest that a measure of quality control (like BR) is essential to assess clinician fidelity over time.

Limitations

The primary limitations are related to assessing training effectiveness using BR given low enrollment that resulted in fewer than half of the trainees participating in the evaluation and high attrition. Although the evaluation with BR was more rigorous than prior evaluations, we may have traded greater internal validity for decreased external validity, despite the fact that participants and non-participants did not have significant differences. Second, given resources required in BR, we only examined analogue fidelity to two CBT techniques. Many other techniques and practice elements were taught and not assessed. Third, we did not document or analyze training and consultation time focused on the two techniques assessed or on BR itself (see Edmunds et al., 2013 for examples of this kind of microanalysis). Fourth, BR does not include an evaluation of clinicians’ actual fidelity with clients or client outcomes. Fifth, our evaluation did not test BR as a quality assurance or competency development tactic in the work setting, which are important next steps for research on the feasibility of BR to improve clinician fidelity. Finally, our assessment of feasibility did not include an estimate of cost, time, or reasons trainees chose not to participate (time, incentive, anxiety, etc.).

Conclusion

In general, our experience and findings suggest that BR has the potential to add value as part of an evaluation of quality improvement initiatives (e.g., identification of techniques/elements that are more challenging; training effectiveness). However, future efforts should make use of lessons learned by our team to strengthen external validity by increasing enrollment and retention. BR offers an intermediary outcome option between self-report and assessment of actual fidelity with clients; however, to make the time investment worthwhile—on both the part of the research team and participants—improving enrollment and retention is necessary.