Show Me, Don’t Tell Me: Behavioral Rehearsal as a Training and Analogue Fidelity Tool

Abstract

Behavioral rehearsal, when a trainee engages in a simulated interaction with another individual, is an underutilized but potentially cost-effective and feasible solution for two difficult questions in implementation science: how to improve training, a commonly used implementation strategy, and how to feasibly measure fidelity using analogue methods in community settings. This paper provides practical information on how to develop and use behavioral rehearsal for both of these purposes to implementation researchers. Therefore, we focus on development and use of behavioral rehearsal as a training and analogue fidelity tool in the context of three illustrative studies.

One pressing challenge facing the mental health field is the dissemination and implementation (DI) of evidence-based practices (EBPs) from research to community settings (McHugh & Barlow, 2010, p. 829). A barrier to implementation of EBPs in community settings is ensuring that trainees deliver treatments with fidelity (McHugh & Barlow), the “extent to which the intervention was implemented as intended” (Perepletchikova, Treat, & Kazdin, 2007, p. 829). To date, the impact of training, one of the most frequently used implementation strategies (Powell et al., 2011), has been largely disappointing (Beidas & Kendall, 2010). Two questions have emerged around training and fidelity in EBPs: (a) What are the most effective training strategies (Beidas & Kendall; Herschell, Kolko, Baumann, & Davis, 2010; Rakovshik & McManus, 2010)? and, (b) How can fidelity be feasibly measured in community settings, given that few reliable, valid, and efficient fidelity measurement systems exist (Schoenwald, 2011; Schoenwald & Garland, in press)? Behavioral rehearsal (BR), a simulated interaction between a trainee and another individual (Cross, Matthieu, Cerel, & Knox, 2007), is an underutilized but potential answer to these two thorny questions.

BR Is Critical for Implementation Science

BR is a methodology that has important implications for implementation science (IS) given its potential to improve training and reduce the resource intensiveness of fidelity measurement, when an analogue method is acceptable. A robust literature suggests that traditional passive training practices are ineffective at changing provider behavior (Beidas & Kendall, 2010; Farmer et al., 2008; Herschell et al., 2010; Rakovshik & McManus, 2010). Active learning may be the most effective way to change behavior, particularly for new or complex skills (Milne, Aylott, Fitzpatrick, & Ellis, 2008) and can improve trainee fidelity (Cross, et al., 2011). When used in training, BR initiates active learning processes, meaning that the trainee experiences and reflects through practice opportunities (Kolb, 1984). However, little guidance exists for researchers and trainers when desigining trainings that incorporate BR (Rakovshik & McManus).

BR methodology can also address a major challenge in the IS literature (Schoenwald, 2011) because it may allow a feasible analague for capturing fidelity, a frequently measured outcome in implementation trials (Proctor et al., 2011). The primary methods of measuring fidelity are direct (i.e., viewing sessions) or indirect (i.e., self-report; Perepletchikova et al., 2007). We conceptualize BR methodology as a rapprochement between direct methods, which can be expensive and not feasible, and indirect methods, which typically are inaccurate (Beidas & Kendall, 2010). BR offers a potentially “effective and efficient” analogue method of measuring fidelity¹ (Schoenwald et al., 2011) akin to the standardized patient methodology in the medical literature (Shah et al., 2012).

Illustrative Studies Using BR

To illustrate how to use BR as a training and/or analogue fidelity tool, we provide examples from three studies. Each study has a unique mental health intervention, trainee population, and context, demonstrating the utility for BR across heterogeneous settings. Our goal is to focus on BR methodology; therefore, only a brief description of each study is provided (see Beidas, Edmunds, Marcus, & Kendall, 2012; Cross et al., 2011; Dorsey, et al., in preparation). One limitation of the published manuscripts is that they provide only limited detail about the BR methodology, given the focus on presenting outcomes. Therefore, here we focus on the BR methodological detail, and refer readers to the published manuscripts for study procedural and outcome details.

Example 1: CBT for Child Anxiety

BR was used as a training and analogue fidelity tool in a randomized controlled trial (RCT) of three training conditions for cognitive-behavioral therapy (CBT) for child anxiety (Beidas et al., 2012). CBT has the most empirical support for the psychosocial treatment of child anxiety (Silverman, Pina, & Viswesvaran, 2008), yet access to CBT in the community is limited (Shafran et al., 2009). The RCT training conditions were as follows: (a) a 1-day routine training (training-as-usual); (b) computer training (computerized version of training-as-usual); and (c) augmented training (BR-focused training). Outcomes included trainee analogue fidelity, knowledge, and satisfaction. Participants were 115 trainees in the northeastern United States who were predominantly female (90%), Caucasian (71%), master’s-level (37%), and middle-aged (M = 35.93; SD = 11.36). All procedures were Institutional Review Board (IRB) approved and participants provided written consent.

Example 2: Common Elements CBT

BR was used as an analogue fidelity tool in a state-funded common elements initative for child-serving Washington State therapists employed in public mental health (Dorsey et al., in preparation). The common elements approach was selected given recommendations to improve usual care for children (Garland, Bickman, & Chorpita, 2010) and findings that common elements (Chorpita & Weisz, 2009) resulted in better client outcomes than traditional EBP or usual care (Weisz et al., 2012). BR was used to assess analogue fidelity for CBT competencies across depression, anxiety, and posttraumatic stress disorder (PTSD; e.g., CBT case formulation; homework assignment; Sburlati, Schniering, Lyneham, & Rapee, 2011). Participants were 38 trainees (of 100), predominantly Caucasian (81.6%), females (76.3%), with master’s-level degrees (92.1%), ages 25 to 39 (63.1%). Evaluation procedures were deemed research exempt by the Washington State IRB.

Example 3: Suicide Prevention

BR was used in an RCT training study of a suicide prevention program as both a training and analogue fidelity tool (Cross et al., 2011). Suicide is the third leading cause of death in individuals of 10 to 24 years of age (Centers for Disease Control and Prevention, 2007). One evidence-supported prevention strategy involves “gatekeeper” training to teach community members informal surveillance, detection, and identification of suicide risk (Wyman, et al., 2008). This RCT tested gatekeeper training (Quinnett, 1995) in a school setting, and examined training as usual compared to BR training on trainee knowledge, attitudes, and analogue fidelity (measured via BR). Participants were community members including school personnel (e.g., teachers, coaches; n = 91), mental health professionals (n = 22) and parents of students (n = 56). School personnel were predominantly female (76.9%), Caucasian (97.8%), ages 24 to 70 (M = 42.07 years). Mental health professionals were predominantly female (90.9%), Caucasian (86.4%), and between 25 to 59 years of age (M = 40.64). Parents also were predominantly female (94.6%), Caucasian (89.3%), and 30 to 54 years in age (M = 43.49). All procedures were IRB approved and participants provided written consent.

BR How-to

This paper provides practical support to implementation researchers aiming to use BR methodology to improve training or for analogue fidelity measurement purposes. Therefore, we focus on the development and use of BR as a training tool and as a fidelity tool in the context of three illustrative studies. The purpose of this paper is not to provide empirical findings from these three trials; rather, our goal is to show researchers how BR might be used in three different contexts as a training and/or analogue fidelity tool. We will accomplish this goal by presenting the steps necessary to use BR for each of these two purposes.

BR as a Training Tool

Utilizing BR as a training tool requires two steps: (a) developing BR materials and (b) planning for BR in training.

Developing BR Materials

Typically, trainees are asked to take on one of three roles: interventionist, client, or observer. To assist trainees in engaging in the roles effectively (e.g., realistic client portrayal, providing feedback), written materials are developed to structure BR use in the training.

CBT for Child Anxiety

BR training materials were developed in collaboration with a clinical psychologist who specialized in child anxiety. Five sets of written materials were developed for five BRs representing each of the five CBT techniques. The written materials included a backstory about a 9-year-old anxious female, her symptoms (e.g., anxious cognitions), and interference (e.g., trouble going to school). Three sets of instructions were created for each of the five BRs specific to the three roles: interventionist, youth, or observer, who would provide brief feedback on the interventionist’s performance guided by a checklist of key steps. The written materials built upon their predecessors so that by the fifth BR, the trainee was practicing all of the techniques together (i.e., scaffolding; Rakovshik & McManus, 2010; Sun, Merrill, & Peterson, 2001; Appendix A – “Child Anxiety: BR scripts used in training”).

Suicide Prevention

An interdisciplinary team developed BR training materials. The written materials portrayed middle or high school students and included symptoms (e.g., unable to get out of bed), a stressor (e.g., parents’ divorce), and a series of disclosed risk factors for suicide (e.g., hopelessness). Three sets of instructions were created for each BR: youth, interventionist, and the observer who provided brief feedback guided by a checklist of expected behaviors. Two versions of the instructions were created—one for school personnel and the other for parents (Appendix A – “Suicide prevention: BR scripts used in training”).

Planning for BR Training

When planning BR training, the core techniques necessary to carry out the intervention must be delineated. Typically, techniques are introduced didactically and BR is used for practice. An important consideration is the balance between having enough time dedicated to BR without engendering BR fatigue. In both of our examples, approximately one-third of the training was spent in BR, allowing time for other learning strategies including didactics and modeling. Another important consideration is how BR will be used in the actual training. Decision points include the number of BR, trainee group size, timing, and strategies to ensure that each trainee practices different roles.

CBT for Child Anxiety

In our training RCT, the 6-hour workshop included BR trainings for five core CBT techniques for child anxiety (Sburlati et al., 2011). Each of the five techniques was first introduced didactically, modeled by the trainer, and followed by small-group BR. This allowed for distributed learning and the opportunity for trainees to engage in five BRs over the course of the day. Trainees divided into groups of three by numbering off based on their seating to ensure practice with unknown individuals. Each group received color-coded written materials for the three roles. Trainees spent approximately 20 minutes for each BR; at 15 minutes, observers were asked to provide feedback to the trainee playing the role of interventionist. Over the course of the five BRs, participants were asked to take each role at least once.

Suicide Prevention

Similarly, an RCT of training for a suicide prevention program compared standard training (i.e., 1-hour lecture, short video) to standard training plus BR. For the BR condition, immediately following the standard training, participants were divided into groups of three to engage in three sequential BRs. Trainees practiced questioning about suicide, persuading, and referring to a professional for three different youth scenarios. Each group received three envelopes with three color-coded roles. Trainees spent approximately 7 minutes for each BR and rotated roles until each experienced all three, over 25 minutes.

BR as an Analogue Fidelity Tool

Fidelity is a multicomponent construct and includes treatment adherence (i.e., use of specified procedures) and competence (i.e., skill; Perepletchikova et al., 2007, p. 829). Typically, fidelity is conceptualized as a construct that is measured in direct practice. However, given the challenges with observation of direct practice, we suggest that BR provides a promising analogue method to assess trainee adherence and skill. In comparison to using BR as a training tool, employing BR as an analogue fidelity tool requires more planning, time, and resources. Despite the intensiveness of this methodology, it still demands fewer resources than direct observational methods. Specifically, there are four steps outlined below: (a) developing BR materials, (b) training actors, (c) collection of fidelity data, and (c) creation and use of the BR coding instrument.

Development of BR Materials

When using BR as an analogue fidelity tool, actors are trained to play the role of client. To ensure standardization, more detailed written materials, compared to those used in BR for training, are created to train the actors. In creating written materials for the trainee to prepare for the interaction, it is important to develop clear instructions, create multiple versions for repeat assessment, and ensure review by experts.

CBT for Child Anxiety

We developed standardized written materials describing an 11-year-old anxious male in collaboration with a clinical psychologist who specialized in child anxiety. Two alternate versions were created for repeat administration. All versions were rated by experts to ensure comparability. Written materials included a backstory describing an anxious youth with symptoms and interference and instructions on the specific task (i.e., prepare the youth for exposure). This task was selected because it integrates all five techniques from the training (Appendix B – “Child Anxiety: BR standardized script for trainees”).

Common Elements CBT

Three sets of standardized BR written materials were developed for a youth struggling with anxiety, depression, or PTSD by a team of three clinical psychologists. The youth backstories were written to be representative of the public mental health population (e.g., trauma exposure, foster care). Initial drafts were sent to experts to ensure comparability. Trainees received a backstory and instructions to demonstrate two core CBT competencies in two separate interactions. Instructions for the first BR prompted trainees to present the CBT model and apply it to the problem area (e.g., depression, anxiety, PTSD). Instructions for the second BR prompted trainees to plan for and assign CBT homework (i.e., exposure for anxiety and PTSD; behavioral activation for depression). Although the problem areas differed, trainees were asked to demonstrate the same skills in the two BRs and across problem areas (i.e., present CBT model; plan for/assign homework; Appendix B – “Common elements: BR standardized scripts for trainees”).

Suicide Prevention

We developed two sets of standardized written materials in consultation with the same expert group that created the training scripts. Actors were trained to deliver specific lines in order. Written materials were matched for difficulty and tailored for interactions with school personnel and parents. As part of the written materials, a backstory about a student was presented. For example, in one scenario, the student is the target of “Instant Message” rumors. Specific symptoms were detailed (e.g., irritability, missing school) in addition to suicidal ideation and means. Trainees were expected to demonstrate three suicide-specific techniques (i.e., ask directly about suicide, persuade youth to accept help, provide referral to a professional) and one general technique from the training (i.e., active listening; Appendix B – “Suicide prevention: Backstory for trainees”).

Training Actors

After written materials are developed, actors must be trained to criterion (Association of Standardized Patient Educators, 2012) to ensure standardization. Who should be trained to play the client? Options include staff therapists, child actors, community actors, or research assistants. We know of no empirical study that has directly compared the use of different actors on trainee outcomes. Until then, the decision about who to train may be driven by pragmatic considerations, including cost and availability. Using a staff therapist for BR can be effective (Ducharme & Feldman, 1992); however, they may portray the most difficult clients (Miller, Yahne, Moyers, Martinez, & Pirritano, 2004). Child actors are another feasible option (Rowe, Onikpo, Lama, & Deming, 2012); however, questions remain regarding ethical concerns. Community theater actors or trained “standardized patients” in medical school programs are highly skilled but potentially expensive. Research assistants (RAs) are a practical option, but may require more training. Regardless of who is trained, a method for determining when an actor meets criterion must be established.

CBT for Child Anxiety

We selected to use five RAs based on resource constraints. Training involved reading articles and watching tapes of anxious youth. During the BR, RAs were instructed to cooperate with the trainee by following their lead and were given standardized responses in an actor guide. For example, if asked about physical symptoms when anxious, they were to respond: “My heart races, my stomach hurts, and I turn red.” However, if the trainee did not ask this question, they were told not to provide these symptoms. To be trained to criterion it was necessary for the RA to provide at least 80% of the desired prompts (see Appendix B – “Child Anxiety: BR standardized script for actors”).

Common Elements CBT

Due to resource constraints, four RAs were used as actors. All RAs participated in initial BR training, which entailed three 60- to 90-minute calls that provided an overview of typical symptom presentation for each problem area. Trainers modeled introducing CBT, planning for and assigning CBT homework, and typical child responses. Following training, RAs practiced with each other using an actor guide. The guide included standardized responses to questions for a trainee who was “on” task (e.g., following BR instructions) and who was “off” task (e.g., not following BR instructions). For example, trainees were asked to present the CBT model, using the cognitive triangle, with an example situation from the child. If trainees asked about a situation, each guide included a specific situation to provide (e.g., “drawing”) as well as related thoughts (e.g., “I’m good at this”), feelings, and behaviors. RAs were trained to criterion of providing 80% or more of the specified responses (Appendix B – “Common elements: BR standardized scripts for actors”).

Suicide Prevention

We employed 11 undergraduate theater students on an hourly basis ($20 an hour) to play the suicidal youth role for five reasons: (a) professionalism, given the sensitivity of the topic; (b) need for script memorization, as BR would be conducted in-person; (c) need for physical and emotional authenticity; (d) need for ongoing availability (i.e., part time employment versus short-term course credit for RAs); and (e) availability of funds. A preliminary interview was conducted to explore current mental health concerns and previous exposure to suicide. All actors received 5 to 6 hours of one-to-one training to deliver the standardized scripts. During training, actors practiced being adherent to the scripts, integrating nonsequitors, consistent pacing of the interaction, responding to challenges, and smoothly ending the interaction. Actor guides delineated specific responses to trainee statements or probes. For example, we anticipated that some trainees would not ask directly about suicide. Therefore, actors were instructed to respond in a standardized way (“I don’t want to cut myself or anything… I just want it all to be over”) in response to a vague inquiry. Once confident with the scripts and procedures, actors were assessed by participating in a BR with a research team member who presented the actor with challenges. At least one team member observed and coded the interaction using the Actor Adherence Checklist (Cross et al., 2011). Actors needed to reach 100% criteria before study participation; all reached criterion. (Appendix B – “Suicide prevention: standardized scripts for actors”).

Collection of Analogue Fidelity Data

When using BR methodology to assess analogue fidelity, there are a variety of methods to conduct the BR and obtain a recording for later coding. These include Skype (video or audio), telephone, and in-person interactions. Decisions about when BR will be used (i.e., before and/or after training) must also be considered.

CBT for Child Anxiety

We conducted BR to assess for analogue fidelity before, after, and 3 months following training. At each assessment, participants were provided with one of three versions of the written materials over email the night before the BR and were asked to spend approximately 5 minutes preparing. The day of the BR, an RA called participants using Skype (audio only). The conversations were recorded, with permission, using Pamela, a web application. The RA read the instructions with the participant, answered questions, and engaged the trainee in an 8-minute BR. We conducted BRs using Skype because we were concerned about high participant attrition if inperson visits were required, and because of logistics (i.e., we could not conduct 20 BRs pre-post training in 1 day).

Common Elements CBT

All trainees registered for the 2011 training were invited to participate in the BR evaluation via email. In addition to completing web-based surveys, trainees were asked to complete BR before, after, and 6 months following training. Once trainees completed the pretraining survey, they were randomized to one of three BRs (e.g., depression, anxiety, PTSD), counterbalanced by trainee across the three time points. Trainees received the BR 24 hours before to allow for review. Trainees were given 8 minutes for each BR (16 minutes total). BRs were conducted by speakerphone and were audio-recorded digitally.

Suicide Prevention

The suicide prevention BRs were conducted in person and videotaped immediately after training and at a 3-month follow-up. Trainees were provided with a backstory about the awaiting youth. Three months later, the participant returned for a second videotaped BR. We counterbalanced written materials for scenario and gender of the youth. At follow-up, the research staff member offered the trainee feedback on their skills. With permission, we include an example of a videotaped interaction between a school-based trainee and an actor in Appendix B (Video 1: “Suicide prevention video example”).

Creation and Use of the Coding Instrument

One of the strengths of using the BR methodology as an analogue fidelity tool is the opportunity to objectively code therapist behavior. An iterative process of development is needed to operationalize the target techniques to be coded. Target techniques should be linked to the competencies necessary to administer the intervention, rules for coding must be developed and utilized consistently, and the coding measure needs to be easily scored, reliable, and meaningful.

CBT for Child Anxiety

An investigator-created coding instrument was developed to measure therapist analogue fidelity (Appendix B – “Child anxiety fidelity coding instrument”). Adherence was assessed by coding the presence or absence of six CBT techniques. Independent coders scored each category for the presence of the technique and each participant received a total score (0–6). When rating skill, coders were asked to rate skill in preparing a child for exposure using the CBT framework on a 7-point Likert scale. Coders were one doctoral-level psychology graduate student and three RAs. Training included a review of CBT for child anxiety, observation of taped sessions, reading materials, didactics, and supervised practice with feedback. Subsequently, the coding scheme (Beidas et al., 2012) was introduced and explained in detail to the raters. Following initial training, the group met three times to code sample BRs (N = 18). Interrater reliability for adherence and skill ratings was established between the primary investigator and the four coders. Coders were blind to hypotheses, condition, and time-point of the assessment. All coders met an intra-class correlation (ICC) or kappa coefficient criterion of ≥.75 at the outset of the study on a different sample of 20 to 25 BRs. All BRs used in training to criterion were from the study sample, but were later coded again.

Common Elements CBT

Similar to the prior example, study investigators created a coding instrument to measure analogue fidelity (i.e., adherence and skill) for the two selected CBT techniques. Adherence for the first BR, introducing the CBT model, was assessed by coding the presence or absence of four elements whereas adherence for the second BR, planning for an assigning homework, was assessed by coding the presence or absence of seven elements. Coding for both BR was developed to apply to all three problem areas. We have just begun training our coders who will code each BR separately and be blind to time point. Each element will be scored 0–6, for a combined score of 0–24 for the first BR (CBT model) and a combined score of 0–42 for the second BR (assigning/planning homework). Coders will be four undergraduate RAs trained through didactics, reading materials, and practice with investigator feedback. Inter-reliability procedures also will follow the child anxiety example (Beidas et al., 2012). We have not included the instrument in the Appendix due to ongoing refinement.

Suicide Prevention

Analogue fidelity was rated using a modified Observation of Gatekeeper Skills Rating Scale (OGSRS; Cross et al., 2007), which has five items reflecting four primary training domains (Appendix B – “Suicide prevention fidelity coding instrument”). Each item was rated on a 4-point scale (0–3) using behavioral descriptors for each item and score, for a total possible score of 15. Each videotaped BR interaction was coded twice to assess the trainee’s fidelity and to assess the actor’s adherence to the script. Thirty percent were double coded to assess interrater reliability. A randomly selected 10% of already coded tapes were included into later coding assignments to check for coder team drift. Coders were Ph.D. and master’s-level psychologists trained to criterion. ICCs were adequate to excellent (range 0.65 – 0.92) except for the “Persuade” item (0.40). Coder ratings did not drift over time (range across 4 domains: r =.80 – .97, between rating time one and two).

Lessons Learned

BR is a heterogeneous methodology that can be used as both a training and analogue fidelity tool across a variety of interventions and trainees. These examples illustrate that BR is a highly generalizable methodology that can be flexibly used in training, as well as a tool that can be used as an analogue for assessing fidelity when observation of actual practice is not possible. Based on our varied experiences, we have identified a number of recommendations for implementation scientists adopting this methodology.

BR as a Training Tool

BR is best used as a training tool for behaviors that require practice. For example, BR can be used to enhance learning of complex treatment skills (e.g., CBT for child anxiety) and a focused but highly sensitive intervention skill (e.g., asking about suicide). It can be used in a distributed learning format, where participants engage in multiple BRs over the course of a daylong training session, or it can be used for consolidated in-depth practice at the end of a 1-hour workshop. Additionally, BR can be used with various kinds of interventionists, from clinical providers to community members. An interesting observation about BR duration in training sessions emerged from the examples: approximately one-third of the time was dedicated to BRs. Although the optimal BR “dosage” is an empirical question, our studies planned for a ratio of 2:1 passive to active learning activities. This is an important consideration because in our experience, trainers frequently omit the BR portion of training due to time constraints, discomfort facilitating BR practice activities, or participant anxiety. However, the BR component is arguably the most important and effective part of training for skill development from the perspective of both trainee outcomes and reported beliefs about which methods are most effective (Bennett-Levy, McManus, Westling, & Fennell, 2009; Cross et al., 2011). We therefore encourage trainers to spend adequate time on active learning methodologies, such as BR, if skill development is an objective of the training. When faced with unexpected time constraints, we recommend eliminating a training topic to avoid cutting the practice opportunities, given that practice is required to facilitate learning (Beidas & Kendall, 2010; Herschell et al., 2010).

We have encountered a number of challenges in using BR as a training tool. First, some participants were anxious about participating in BR, perhaps in part due to evaluation by their peers and trainers and discomfort with sensitive topics. In the case of CBT for child anxiety, we found that initially it was difficult to engage participants in conducting BR, especially in the role of the therapist. Qualitative semistructured interviews provide some insight into concerns about peer evaluation, thus reducing connectedness, and feeling inauthentic (Beidas et al., in press). Once participants acclimated to BR, the hesitancy decreased, consistent with the principles of exposure. Similarly, anxiety was noted in participants in the suicide prevention study. Individuals had difficulty broaching the youth’s suicidal thoughts or feelings despite clear instructions to ask about suicidal thoughts. Heightened anxiety during BR among participants is not necessarily an obstacle given that there is an optimal level of anxiety that improves performance (Yerkes & Dodson, 1908). However, trainers may require their own training to skillfully create an emotionally safe learning environment for BR.

A second challenge emerged regarding feedback. Feedback has repeatedly demonstrated an effect in changing provider behavior (Ivers et al., 2012). In trainings, we have provided clear instructions to training participants on the process of BR feedback, and not just on the content (i.e., observer checklist). We have instructed participants to first give feedback on what the interventionist did well and to then give constructive feedback. The analogy of the “feedback sandwich” has been used, so that constructive feedback is bookended between positive comments. Although not formally evaluated, this structured approach to the process of BR feedback seems to reduce anxiety. However, despite these efforts, in both the child anxiety and suicide prevention studies, observer feedback was not used as effectively as intended. First, observers did not necessarily use the checklist to guide their feedback. Second, observers were generally complementary rather than constructive. This may be due to a desire to please others or due to observers themselves being novice in the techniques. Interestingly, in follow-up qualitative interviews, participants indicated that they desired more individualized feedback to enhance their performance (Beidas, et al., in preparation). These findings set the stage for future research around how to best organize small group feedback as part of the BR methodology as well as the role and timing of expert feedback for skill development.

BR as an Analogue Fidelity Tool

BR can effectively be used to measure fidelity as an analogue tool and is a more feasible option than other direct observational methods. Quality assurance in RCTs typically involves licensed psychologists observing and coding a percentage of sessions from treatment cases (e.g., Kendall, Hudson, Gosch, Flannery-Schroeder, & Suveg, 2008), which is more time intensive and costly, and unlikely to commonly occur in community settings (Schoenwald, 2011). Direct observational methods require a number of additional steps, which make them cumbersome and less likely to be used in community settings (Schoenwald, 2011). To obtain actual practice samples in audio- or videotape format, it requires the consent and assent of clients who are receiving services. This adds an additional layer of complexity and may be one of the reasons that implementation studies that have endeavored to obtain practice samples have often had low rates of collection (e.g., Moyers et al., 2008). Second, to obtain examples of all of the potential skills that an individual has been trained in, it would be necessary to watch and code multiple sessions over the course of treatment, which would greatly increase the amount of time needed for observation and coding. Third, coding training is greatly simplified in BR and can be conducted by undergraduate research assistance when only target skills are being measured (compared to an entire treatment) and standardized client responses are provided.

Although BR as an analogue fidelity tool requires significant resources, it is certainly less intensive than the procedures outlined above for direct observational methods. Currently, BR methods are more time intensive than they will be in the future when a standardized set of materials are available. Procedures include the need to create BR written materials and coding instruments for each specific intervention and treatment population. One reason for developing BRs that assess core CBT techniques common to multiple interventions (e.g., common elements example) was an attempt to address this limitation. Further, a primary purpose of this paper is to provide our materials to other implementation scientists with the hope that as others use BR methods, the instrument base will grow. Another reason for the resource-intensive nature of BR is the need to select and train actors. The use of actors can be costly, whereas RAs are more easily obtained and less costly but may require more training. Finally, the resource-intensive nature of BR as an analogue fidelity tool stems from multiple and potentially cumbersome steps of development: development of written materials that need expert review, revision subsequent to expert feedback, scheduling, conducting, transcribing, creating the coding instrument, training independent raters, and coding the BR interactions.

There are other important considerations when adopting BR methods. The first has to do with the development of the coding instrument with regard to reliability and feasibility. Decisions must be made about what aspects of the interaction between the trainee and actor should be coded. We recommend that only the most germane behaviors be coded rather than rating every aspect of the interaction. It is important to balance both rigor and relevance and to code relevant behaviors (Proctor, et al., 2011; Schoenwald et al., 2011). It is necessary to consider which techniques to code in multi-step, complex interventions conducted over multiple sessions. In the child anxiety example, we included techniques identified via literature review (e.g., preparing for exposure) and did not include others that were interesting but perhaps not critical. Second, feedback is critical as part of the BR process. In both the child anxiety and suicide prevention examples, trainees reported wanting more feedback during BRs (Beidas et al., in preparation). Training or analogue fidelity ratings that emerge from the BR could be shared with trainees to help shape skills and to aid in self-assessments.

A number of logistical considerations are warranted. Will BR be used in person or through technology (e.g., video, computer)? In two of our examples (e.g., child anxiety and common elements), we used audio only (Skype or telephone) to conduct the BR, whereas in the suicide prevention example, BR was conducted face-to-face and video recorded. Audio-only interactions may allow the participant the opportunity to imagine they are interacting with a child portrayed by a young adult. However, the absence of nonverbal communication and potential for awkward feelings interacting virtually are limitations. Interestingly, in the common elements example, clinicians did engage in interactions as if they were face-to-face with the actor (e.g., “Okay, so choose a color. Red? Okay. Draw a triangle here and write…”). A final logistical consideration has to do with the decision to transcribe BR after completion to assist in coding. If not professionally completed, transcription can be time intensive (i.e., typically twice the interaction time). One potential tool to speed this process includes the use of digital transcription foot pedal devices, which allow for easy control of audio (e.g., pause, rewind), allowing the transcriber to continue typing uninterrupted, speeding transcription by approximately 50% (e.g., http://www.martelelectronics.com/616.html).

A challenge in the use of BR is the potential for negatively impacting participant enrollment rates in evaluation due to the increased time requirements and potential for discomfort. For example, BR was feasible within the context of a state-funded training and evaluation program, but resulted in a one-third reduction in evaluation enrollment rates (from 60% to 38%). As a result, although the method was more rigorous than prior self-report evaluations, the sample had less generalizability. These tradeoffs are important to track and consider. Potentially, with greater resources to better compensate clinicians for increased time requirements, or if BR completion is a requirement for acquiring training, the impact would be lessened.

Despite the exciting potential for the BR methodology as both a training and analogue fidelity tool, it is still in its infancy and there are a number of limitations. First, the most important limitation is that currently there is no empirical information on the comparability between performance on BR and behavior during actual practice. Until this information is provided, there is no way to determine the effectiveness of this analogue methodology. However, the standardized patient literature suggests its promise (Shah et al., 2012). Second, an aspect of EBP is the ability to identify and choose which treatment components are necessary in unexpected and complex situations. The ways that we have used BR in our three examples do not tap into this construct. However, it would not be impossible to construct a BR where trainees were not given instructions a priori and were asked to implement treatment methods and skills they deemed appropriate. This would be an important future direction for consideration. Third, there may be a practice effect in which repeated exposure to BR increases comfort with BR independent of the techniques being trained and/or assessed. Fourth, we are currently limited by a lack of existing coding schemes other than the ones we have presented here, and those that colleagues have graciously shared for review during our development. For individuals interested in exploring BR for other types of interventions, the resource-intensive work must be conducted from scratch. One consideration is specificity versus generalizability. There is a tradeoff between creating coding schemes that cut across projects because they would be less sensitive to specific interventions. Creating coding schemes for common elements of EBP (Chorpita, Daleiden, & Weisz, 2005) or for cross-cutting competencies is one way to address this limitation (Sburlati et al., 2011). Another option would be to use existing coding systems and to adapt them for use in BR, such as the Therapy Process Observational Coding System for Child Psychotherapy-Strategies Scale (TPOCS-S; McLeod & Weisz, 2010). A fifth limitation has to do with the manner in which instructions are provided when using BR as an analogue fidelity tool. Trainees are provided clear instructions on how to behave in the BR. The structured guidance provided to the trainee may enhance learning early in training but may interfere with learning later in training when trainees need to learn to make choices about which methods to use with a client at various points of time. Further, structured guidance might spoil the meaning of fidelity ratings since trainees are explicitly guided in which skills to apply. Interestingly, despite the explicit instructions, it has been our experience that trainees still omit critical parts of the treatment, suggesting that this concern may not be warranted.

There are a number of exciting future directions for the BR methodology. As a training tool, it can augment and improve passive learning models of training. Further, it has the potential to be used with new technology to provide cost-effective training. For example, a recent study used BR methodology for training Veterans Administration clinicians virtually (Ruzek et al., 2012), and BR is currently being used to scale up training for community clinicians providing Problem Solving Therapy (PST; Cross & Arean, 2012). The BR method can be conducted virtually through meeting platforms such as Google Plus Hangout (free for up to 9 people) and WebEx or Adobe Connect (monthly fee, allows larger groups), to provide face-to-face interactions, even if the trainee and actor cannot be in the same physical place. Additionally, BR “boosters” using the aforementioned technology can be delivered by stationing actors at computers for trainees to “call in” and practice. BR can also be used to augment supervision of EBPs. In a recently funded NIH study, supervisors complete BR with supervisees based on an upcoming treatment component (e.g., analogue fidelity assessment) and then, based on their rating of the BR, supervisors coach them to improvement with feedback (e.g., training tool; MH095749, Principal investigator; Dorsey). BR can also be combined with other active learning methods (e.g., self-practice models; Bennett-Levy & Lee, 2012) to capitalize on learning. Future research understanding how the use of multiple active learning methods impact skill is necessary.

Summary

We have presented BR as a methodology that can be used as both a training and analogue fidelity tool across multiple contexts and interventions. We believe that it is an important methodology that enhances learning for trainees and can also enhance researcher understanding of what trainees are able to learn. Through our examples, we hope we have illustrated the feasibility of this methodology and some of the important decision points that must be considered when adopting. Given that developing BR is time and resource intensive, we encourage others to share their materials to further improve upon the feasibility of the methodology.

Highlights.

• Behavioral rehearsal (BR) is when a trainee engages in a simulated interaction with another individual

• This methodology is important for implementation science

• BR can be used to improve training in evidence-based practices

• BR can be used as an analogue method to measure fidelity in community settings

Appendix A

Training Tools

1. Child Anxiety: Behavioral rehearsal (BR) scripts used in training

   These written materials include the five BRs used in training in CBT for child anxiety with three roles for each BR (therapist, youth, observer).

2. Suicide prevention: BR scripts used in training

   These written materials include an example of one of the three BRs used in the training. Materials for three roles are included: a) school personnel or parent; b) youth and c) observer (For others, please contact Wendi Cross, PhD). The included observer checklist was used across all three training BRs.

Appendix B

Fidelity Tools

1. Child Anxiety: BR standardized script for trainees

   Written materials included a back-story describing an anxious youth with symptoms and interference and instructions on the specific task (i.e., prepare the youth for exposure).

2. Common elements: BR standardized scripts for trainees

   Written materials included a backstory describing a youth with anxiety, depression, or trauma, and instructions to demonstrate two core CBT techniques in two separate interactions.

3. Suicide prevention: Backstory for trainees

   Written materials included a backstory about the student with whom the trainee interacts during the BR.

4. Child Anxiety: BR standardized script for actors

   Written materials included a script for actors to guide their responses to trainee statements.

5. Common elements: BR standardized scripts for actors

   Written materials included a script for actors to guide their responses to trainee statements.

6. Suicide prevention: BR standardized scripts for actors

   Written materials included a script for actors to guide their responses to trainee statements.

7. Suicide prevention video example

   Video provides an opportunity to see an interaction between a trainee and actor.

8. Child anxiety fidelity coding instrument (Adherence and Skill Checklist)

   The coding instrument used in this example included the Adherence and Skill Checklist.

9. Suicide prevention fidelity coding instrument (Observation of Gatekeeper Skills Rating Scale; Actor Adherence Fidelity Rating)

   The coding instrument used in this example included the Observation of Gatekeeper Skills Rating Skill (to rate trainees) and the Actor Adherence Fidelity Rating (to rate actors).