Implementation of the Wolverine Mental Health Program, Part 3: Sustainment Phase

Abstract

Sustaining the implementation of an evidence-based practice (EBP) is the ultimate goal of often years of significant personnel and financial investment. Some conceptualize sustainment as a distinct phase following an active implementation period where the contextual factors, processes, and supports are bolstered to ensure continued EBP delivery. This study provides an overview of the sustainment strategies deployed to embed cognitive-behavioral therapy (CBT) in a Midwestern residential treatment facility serving youth with complex mental health needs. Seven key strategies and their outcomes are described: use of CBT teams, new hire orientation plans, monthly campaigns, change in job descriptions and performance evaluations, development of a behavioral reinforcement system for youth, and a pathway to CBT certification. This study provides a window into how one might sustain an EBP by addressing barriers unique to this phase of work.

Implementation science was developed to bridge the science–practice gap by garnering empirical support for methods to integrate evidence-based practices (EBPs) into new settings. Since the birth of implementation science, numerous phase-based models have been articulated for delineating the work needed for successful implementation (Mendel et al., 2008; Moullin et al., 2019). Across models, there are differences in the number of phases, the levels of influence, the processes prioritized, the barriers to address, the strategies needed, the stakeholders involved, and the variables for evaluation, for example. Despite their many differences, one thing is constant: the importance of the sustainment phase. Moullin and colleagues define the sustainment phase as the period of time following active implementation where the contextual factors, processes, and supports are ongoing to ensure continued EBP delivery, with adaptations as needed, to maximize public health impact (Moullin et al., 2019). Some argue that sustainment is perhaps the most critical phase of the work because without it the investment in the earlier phases may be undermined and the positive impacts of earlier phases have been shown to diminish over time (Stirman et al., 2012).

Despite the importance of this phase, it is difficult to study. Federal funding can be challenging to obtain so that the timeline aligns with the natural move into the sustainment phase. However, a recent systematic review of EBP sustainability that largely featured program evaluations revealed that fewer than half sustained the EBP at a high level of skill, intensity, or fidelity. Four broad categories of influences on EBP sustainability emerged: (a) influences related to the EBP itself, (b) organizational context, (c) capacity to deliver the EBP, and (d) processes used to sustain the EBP. These factors make clear that the work of sustainment must begin before entering this phase (e.g., ensuring a favorable implementation climate, securing resources and delivering training to grow EBP capacity; Weiner et al., 2011), but that there are unique barriers to be addressed (e.g., maintaining EBP priority in the face of new clinical programs) and strategies (e.g., ongoing fidelity monitoring) to be deployed in the sustainment phase.

This paper delineates the strategies deployed in the sustainment phase of a three-phased implementation of cognitive-behavioral therapy (CBT) in residential treatment facilities (RTFs) as the third paper (i.e., Part 3) in a special series. This CBT implementation effort was a partnership between the RTF, Wolverine Human Services (WHS), the Beck Institute (the intermediary), and an implementation research team. The collaboratively developed blueprints for these three phases are published elsewhere (Lewis et al., 2018). The current paper provides a detailed account of seven sustainment strategies: sustainment teams (formerly referred to as implementation teams), creating a new hire orientation plan, hosting monthly CBT campaigns, changing job descriptions and evaluations to align with CBT competence, modifying the point system, monitoring and incentivizing CBT fidelity through an endorsement system, and establishing a pathway to CBT certification for all therapists.

Initiating the Sustainment Phase

Setting Overview

WHS is a youth RTF located in Michigan, with both secure and nonsecure facilities offering services for over 250 male and female youth ages 12–21. Although the youth are placed in WHS through the juvenile justice system (due to truancy, violence) or failed foster care placements, the majority struggle with comorbid mental health disorders (e.g., major depressive disorder, anxiety disorders). WHS offers group and individual services, as well as family therapy, life skills training, and full-time on-site schooling. Staff at WHS represent two types: clinical and operations. Clinical staff consist of therapists, team managers, permanency treatment leaders, permanency planning specialists, clinical managers, and case managers. Therapists generally have a master’s degree in social work and provide therapeutic support to youth. Team managers typically have a bachelor’s degree and manage the operations within youth units or programs. Permanency treatment leaders have a bachelor’s degree and are responsible for delivery of CBT to youth and their families in support of their transition home. Clinical managers are master’s-level staff who manage operations for units and programs. Case managers are bachelor’s-level staff who ensure that youth are receiving personalized clinical care. Operation staff consist of youth care workers (YCWs) and safety and support team (SST) coordinators. YCWs have high school diplomas or the equivalent and help ensure youth safety and security by maintaining line of sight. SSTs are bachelor-level staff that aid the YCW by providing guidance and direction, typically during times of crisis.

CBT Overview

As described in detail in the implementation phase (Part 2; Rodriguez-Quintana et al., this issue) of this series, WHS needed a component-based version of CBT that would be accessible to paraprofessionals (e.g., YCWs) and relevant to complex youth with comorbid internalizing and externalizing symptoms. CBT was adapted via a collaborative, iterative process between WHS, the intermediary, and the implementation research team, and resulted in a three-level system that together comprised a comprehensive treatment approach for a milieu-based setting like RTFs. Level 1 consisted of the CBT model and behaviorism as the foundational lens for conceptualizing youth experiences. Level 2 consisted of active listening for relational building (Payton et al., 2000) and crisis diffusion (Linehan et al., 2015); mood monitoring and intervention mapping for a common language between youth and staff, and individualization of coping skills (self-perception of excess energy and distress [SPEED] maps); and problem solving for empowering youth to be in control of difficult situations (Creed et al., 2011). Level 3 consisted of activity scheduling for building resilience in the face of daily challenges; distress tolerance for coping with high emotion situations (Linehan et al., 2015); and CBT chat forms (i.e., cognitive restructuring) for engendering the ability to catch, check, and change thoughts (Creed et al., 2011). For more details of the CBT for residential environments (CBT-RE), see Part 2 of this series (Rodriguez-Quintana et al., this issue).

Implementation Overview

The Framework for Dissemination in Healthcare Intervention Research guided the implementation and evaluation approach. It was selected as it succinctly delineated the contextual factors that might influence implementation and was derived from an academic–community partnership like ours (Mendel et al., 2008). This model separates EBP diffusion into three stages—(a) adoption (preimplementation), (b) implementation, and (c) sustainment—which aligns with the three papers in this series, the latter of which is the focus of this study.

The sustainment phase began after CBT had been adapted, and the partnership engaged in 2.5 years of training that leveraged early adopters and included a train-the-trainers approach (see Part 2 of this series; Rodriguez-Quintana et al., this issue). This 1-year sustainment phase (of the 5-year partnership) was guided by its own blueprint (see Table 1; Lewis et al., 2018) and included two weeklong site visits by the CBT intermediary and implementation research team (Lewis et al., 2018). A total of 12 strategies were articulated across six categories for this phase of work: train and educate stakeholders (n = 4, 33.0%), use evaluative and iterative strategies (n = 3, 25.0%), develop stakeholder relationships (n = 2, 16.7%), provide interactive assistance (n = 2, 8.3%), engage consumers (n = 1, 8.3%), and utilize financial strategies (n = 1, 8.3%; Lewis et al., 2018). In addition, four strategies from the implementation phase were carried over into sustainment: change performance evaluations, change professional roles, develop and institute a self-assessment of competency, and alter incentives (e.g., certification, vacation, salary; Rodriguez-Quintana et al., this issue). This paper elaborates on seven key strategies, several of which are blended: engage sustainment teams (formerly referred to as implementation teams), create new hire orientation plan, host monthly campaigns, change performance evaluations and professional roles, change the point system, deploy a CBT endorsement system, and build a pathway to certification.

Table 1.

Sustainment Blueprint (Reproduced From Lewis et al., 2018)

Strategy category	Sustainment strategy
Develop stakeholder interrelationships	Engage implementation team
Develop stakeholder interrelationships	Hold cross-staff clinical meetings
Use evaluative and iterative strategies	Develop and implement for quality monitoring––must monitor fidelity through observation regularly and randomly
Train and educate stakeholders	Conduct educational meetings––hold regularly for new staff and as refreshers
Train and educate stakeholders	Use train-the-trainer strategies––only those certified in CBT
Provide interactive assistance	Centralize technical assistance––create standard operating procedure for training and use of CBT at each staff level
Utilize financial strategies	Alter incentives––provide raise earlier based on competency
Use evaluative and iterative strategies	Obtain and use consumer feedback with PQI data collection
Train and educate stakeholders	Shadow other experts––elongate period for new staff
Train and educate stakeholders	Develop learning collaborative
Use evaluative and iterative strategies	Stage implementation scale-up to generate plan across site
Engage consumers	Use mass media––get press release out with data from implementation

Note. CBT = cognitive-behavioral therapy; PQI = Prevention Quality Indicators.

Sustainment Strategies

Sustainment Teams

Implementation teams were created in the preimplementation phase (see Part 1; Scott et al., this issue), consisting of clinician and operations staff that were identified as champions and opinion leaders (Valente & Pumpuang, 2007). Their tasks differed across the phases of diffusion, starting with barrier identification in the preimplementation phase, moving to early adopters building buy-in in the implementation phase, for example. The shift in focus was so profound in the sustainment phase that they elected to change their name to “sustainment teams.” Their sustainment phase tasks focused on maintaining the organizational context that supported, rewarded, and expected CBT; adapting the point system; overseeing the monthly campaigns serving as on-site experts; and ensuring that all staff get endorsed across all three levels of CBT-RE. Between the biannual site visits, the sustainment teams continued to have biweekly meetings, alternating between meetings with the intermediary to maintain their CBT competency, and the lead implementation research team member to discuss strategy development and deployment.

New Hire Orientation Plan

Turnover is one of the most widely cited barriers to sustainment (Beidas et al., 2016; Glisson et al., 2008; Woltmann et al., 2008). A handful of studies show that EBP implementation can actually lead to turnover after organizations invest in staff training, making it possible for them to secure higher-paying positions elsewhere. Other studies show EBP implementation can have a protective effect on turnover, although rates remain high in behavioral health contexts (Herschell et al., 2020). Regardless of the reason and the ultimate value to the organization, turnover most certainly undermines EBP sustainment. To limit turnover’s impact on CBT sustainment, WHS prioritized establishing a robust new hire orientation plan. The plan consisted of a standardized slide deck to be used in a full day of foundational training in the staff member’s first week of the job, before he or she goes on the unit with the youth. Staff are presented with a test at the end of their orientation week that assesses their declarative and procedural knowledge about the CBT model and behaviorism (see Appendix A for the full test)—that is, staff have the opportunity to enter the unit with Level 1 endorsement of CBT-RE (see below for endorsement details). The idea is that staff will begin their new role with the core principles for understanding youth, which should allow for an easier transition, and an appreciation for CBT as a critical element of the care structure.

Monthly Campaigns

Given the high rates of turnover (e.g., 50% of YCWs turn over each year at WHS) and the constant cycling among the youth (i.e., new youth come to WHS each day), sustainment teams hosted monthly campaigns that were devoted to each core skill (e.g., January: sctive listening, February: mood monitoring and intervention mapping), which was a multifaceted blended implementation strategy designed to offer learning opportunities, including practice and feedback, as well as to build buy-in for CBT on a continuous basis. The components of this strategy included distribution of psychoeducational materials to staff and youth, training by sustainment team members (demonstration in team meetings, practice, and feedback), modeling by therapists delivering group therapy to youth (including YCWs as participants), and a poster competition among the youth to interact with the skill and bring awareness to it on the unit. Given the six core skills of CBT-RE and typical timelines for booster sessions, each of the six skills was the focus of a campaign twice annually. Sustainment team members rotated through and signed up for leading elements of the monthly campaigns.

Professional Roles and Performance Evaluations

To ensure that new hires appreciated the importance of CBT’s role in WHS’s care, professional role descriptions were modified to include CBT language throughout. It was made clear that all roles, both operations and clinical, were intended to learn and deliver CBT with youth. For instance, it was stated that YCWs held the responsibility of prompting youth to monitor their moods, cueing youth to engage in a CBT skill in response to mood shifts, and ensuring the activity scheduling was completed daily. As another example, SST members were expected to use distress tolerance skills with youth to help them diffuse high SPEEDs in crisis situations (see Paper 2; Rodriguez-Quintana et al., this issue). Team managers were responsible for ensuring that CBT skills were elevated in the treatment plan. Therapists were expected to be on the pathway toward CBT certification (see below). In addition, all staff performance evaluations were revised to align with the new professional role documents, emphasized the importance of CBT, and included specific expectations that certain roles achieve CBT endorsement (levels expectations varied between roles) within specified time windows, and most certainly before going up for promotion. The changes to staff performance evaluations served as an accountability mechanism and also an incentive to deliver CBT with competency.

Reinforcement System

As noted in Parts 1 and 2 of this series (Rodriguez-Quintana et al., this issue; Scott et al., this issue), WHS’s culture rewarded youth compliance, prioritized physical safety, and favored discipline and accountability as consequences to youth behavior. WHS’s existing point system, in theory, was set up to acknowledge desirable behaviors and reward youth with tangible items (e.g., small items, like snacks, to large items, like clothes). However, remnants of WHS’s culture to hold youth accountable led to misuse of the point system in many cases. The use of “negative points” were deployed daily by staff, which led to youth falling behind in their progress toward graduation (i.e., being able to complete the program and return to the community). The misalignment between the points system and CBT-RE became apparent only in the sustainment phase, and it served as a threat to CBT-RE’s impact on youth.

WHS contemplated removing the point system altogether, but decided instead to ground it in behaviorism, infuse it with CBT, and refine its structure to remove any possible misuse by staff. The point system was renamed the “reinforcement system.” Staff awarded points to youth with the expectation that all youth should receive points each hour, and that staff would immediately award points upon observing desirable and skillful behavior. Youth earned points for accepting corrective feedback (25), accepting guidance (50), requesting help (75), and/or independently engaging in skillful behavior (100). Each staff member had a specified role in executing and monitoring the administration of the reinforcement system to ensure integrity to the spirit and principles of its intended use.

To complement the reinforcement system, which rewarded desirable behavior broadly speaking, therapists developed a CBT skills tracker that youth brought to the unit for daily tracking (morning, afternoon, and evening). The skills tracker was based on diary cards developed by Linehan (2013) for dialectical behavior therapy, but included SPEED tracking (see Appendix B). The skills tracker was intended to facilitate automatic SPEED monitoring, identify effective skills for different SPEEDs, support independent CBT skill use by youth, and, as a by-product, help youth earn points while at WHS and be successful upon their return to the community. The reinforcement system and skills tracker integration in WHS’s daily schedule served as an implementation strategy that drove a significant shift in WHS’s culture. Staff evolved to look for desirable and skillful behaviors instead of negative, rule-breaking behaviors.

CBT Endorsement System

A strategy was also developed to support and reward staff for implementing CBT-RE with fidelity. CBT certification from a body like the Academy of Cognitive Therapy was neither appropriate nor approachable for staff, whereas this was an expectation for therapists (see the “Pathway to Certification” section below). This strategy involved generating a three-level CBT endorsement system to foster staff’s knowledge, use, and competency with each of the core CBT-RE skills. As noted above, the first level included the principles of behaviorism and the CBT model, which combined offer a complementary, foundational view to conceptualizing youth behavior. To become Level 1 endorsed, staff were required to score 100% on a 10-item, multiple-choice vignette-based test on principles of behaviorism (e.g., reinforcement, punishment, extinction, shaping), plus one open-ended question in which staff were asked to articulate the CBT model as if they was introducing it to a youth for the first time.

The second endorsement level included three core skills: active listening, problem solving, and mood monitoring and intervention mapping. These skills were chosen as the second level due to the importance of fostering adequate communication, problem identification, and mood monitoring with appropriate intervention selection. The third endorsement level consisted of three more core skills: activity scheduling, distress tolerance, and cognitive restructuring, as more advanced skills that rely on skills from Levels 1 (e.g., CBT model: thoughts lead to emotional and behavioral reactions instead of the situation itself) and 2 (e.g., ability to recognize distress quickly in order to apply distress tolerance appropriately). To become Level 2 and 3 endorsed, staff first needed to pass an oral examination of the critical elements of each core skill (see Table 2), and then demonstrate competent delivery of the skill in a behavioral rehearsal role play (i.e., standardized role play; Beidas et al., 2014). The competency scale used to assess the role play was an adapted version of Item 10, “Application of CBT Techniques,” of the Cognitive Therapy Rating Scale (CTRS; Young & Beck, 1980). This item was rated for each skill on a 0–6 scale that ranged from poor to excellent; staff had to receive a 4 or higher to become endorsed in each core skill. Staff could approach endorsement for one skill at a time, after practicing first with a sustainment team member who was also trained in competency assessment and providing feedback through a CBT lens. Important to note, only CBT-certified coaches were allowed to formally assess competency and endorse staff (see below). Following endorsement at each level, staff were recognized by having their names listed in the WHS newsletter and receiving certificates. Staff also received endorsement incentives, including monetary bonuses and eligibility for promotion.

Table 2.

CBT Endorsement by Staff Role

Time	Role	Active listening	ITCH	SPEED	CAPES	TIP	CBT chat
		N (%)	N (%)	N (%)	N (%)	N (%)	N (%)
March 17	Operations	17 (2.2)	17 (2.2)	15 (1.9)	15 (1.9)	10 (1.3)	5 (0.5)
	Clinical	12 (19.7)	11 (18.0)	10 (16.4)	10 (16.4)	6 (10.2)	5 (6.4)
April 17	Operations	18 (2.3)	18 (2.3)	16 (2.0)	15 (1.9)	10 (1.3)	5 (0.5)
	Clinical	12 (20.0)	11 (18.3)	10 (16.7)	10 (16.7)	6 (10.3)	7 (9.0)
May 17	Operations	19 (2.5)	18 (2.3)	16 (2.1)	15 (1.9)	10 (1.3)	5 (0.5)
	Clinical	13 (21.7)	11 (18.3)	10 (16.7)	10 (16.7)	6 (10.3)	7 (9.0)
June 17	Operations	18 (2.3)	18 (2.3)	15 (1.9)	14 (1.8)	9 (1.2)	6 (0.6)
	Clinical	13 (21.3)	11 (18.3)	10 (16.4)	10 (16.4)	6 (10.2)	7 (9.0)
July 17	Operations	20 (2.5)	19 (2.4)	15 (1.9)	14 (1.8)	9 (1.1)	6 (0.6)
	Clinical	13 (21.3)	11 (18.0)	10 (16.4)	10 (16.4)	6 (10.2)	7 (9.0)
August 17	Operations	22 (2.9)	19 (2.5)	19 (2.5)	14 (1.8)	11 (1.5)	9 (0.8)
	Clinical	14 (23.0)	11 (18.0)	11 (17.7)	10 (16.1)	7 (11.7)	7 (9.0)
September 17	Operations	29 (3.8)	21 (2.7)	20 (2.6)	14 (1.8)	11 (1.4)	9 (0.8)
	Clinical	17 (27.9)	11 (18.0)	12 (19.4)	10 (16.1)	7 (11.7)	7 (9.0)
October 17	Operations	44 (5.7)	29 (3.8)	27 (3.5)	19 (2.5)	16 (2.1)	12 (1.1)
	Clinical	20 (33.3)	15 (25.0)	15 (24.6)	14 (23.0)	9 (15.3)	7 (9.0)
November 17	Operations	46 (6.0)	33 (4.3)	29 (3.8)	20 (2.6)	17 (2.2)	14 (1.3)
	Clinical	22 (36.1)	17 (27.9)	19 (30.6)	14 (22.6)	10 (16.7)	8 (10.3)
December 17	Operations	54 (6.9)	36 (4.6)	30 (3.9)	21 (2.7)	18 (2.3)	14 (1.3)
	Clinical	26 (43.3)	22 (36.7)	23 (37.7)	13 (21.3)	11 (18.6)	8 (10.3)
January 18	Operations	72 (9.3)	54 (7.0)	47 (6.1)	20 (2.6)	18 (2.3)	14 (1.3)
	Clinical	33 (54.1)	30 (49.2)	31 (50.0)	13 (21.0)	11 (18.3)	9 (11.5)
February 18	Operations	73 (9.5)	55 (7.1)	52 (6.7)	19 (2.5)	17 (2.2)	13 (1.2)
	Clinical	35 (57.4)	32 (52.5)	33 (53.2)	13 (21.0)	11 (18.3)	9 (11.5)
March 18	Operations	74 (9.5)	56 (7.2)	55 (7.1)	19 (2.4)	17 (2.2)	13 (1.2)
	Clinical	35 (55.6)	32 (50.8)	32 (50.0)	14 (21.9)	13 (21.0)	10 (12.8)
April 18	Operations	73 (9.4)	56 (7.2)	55 (7.1)	18 (2.3)	16 (2.1)	12 (1.1)
	Clinical	35 (55.6)	32 (50.8)	32 (50.0)	15 (23.4)	13 (21.0)	10 (12.8)
May 18	Operations	73 (9.3)	59 (7.6)	56 (7.2)	18 (2.3)	16 (2.0)	12 (1.1)
	Clinical	38 (60.3)	34 (54.0)	32 (50.0)	15 (23.4)	13 (21.0)	11 (14.1)
June 18	Operations	69 (8.9)	56 (7.2)	54 (7.0)	17 (2.2)	16 (2.1)	12 (1.5)
	Clinical	38 (60.3)	34 (54.0)	35 (54.7)	15 (23.4)	14 (22.6)	11 (17.2)

Note. ITCH = identify a problem, think of solutions, choose a solution, and how did it work?; SPEED = self-perceived excess energy and distress; CAPES = closeness, accomplishment, physical activity, enjoyment, sleep; TIP = toward engaging the senses, intense physical activity, and paced breathing; CBT = cognitive-behavioral therapy.

Staff were eligible for CBT endorsement at the start of the sustainment year, and by the end of that year approximately 59.3% of staff (both clinical and operations) had achieved Level 1 endorsement; about one-fifth achieved endorsement for each of the Level 2 skills; and closeness, accomplishment, physical activity, enjoyment, sleep (CAPES) from Level 3; whereas less than 7.0% had achieved endorsement for the two most complex Level 3 skills (i.e., CBT chat forms and toward engaging the senses, intense physical activity, and paced breathing [TIP] skills; see Figure 1). Rates of endorsement differed by staff role. Therapists showed much higher rates of CBT endorsement across all skills and all levels, as compared to operations staff (see Table 2).

FIGURE 1.

Fidelity rates (%) over time.

Note. AL = active listening; ITCH = Identify problem, Think about solutions, Choose an option, How did it go? (i.e., problem solving); SPEED Maps= Self-Perceived Excess Energy and Distress (i.e., mood identification and intervention mapping); CAPES = Closeness, Accomplishment, Pleasure, Exercise, Sleep (i.e., behavioral activation); TIP = Temperature, Intensive Exercise, Paced breathing (i.e., distress tolerance); CBT Model = Cognitive Behavioral Therapy conceptualization and restructuring.

Pathway to Certification

As noted in Part 2 (Rodriguez-Quintana et al., this issue), we used a train-the-trainers approach to identify and grow on-site experts to ensure CBT sustainment (Martino et al., 2011). These experts preferred to be called “coaches” and an initial cohort of four coaches (two from each site) was formed. It was critical for therapists to become certified in CBT through the Academy of Cognitive Therapy, as this is perhaps the most stringent test of CBT competence that would optimize therapy effectiveness. But, it was also important for therapists to become certified so that they could be considered for serving as a CBT coach. Not all certified therapists became coaches given that this unique role required a host of leadership responsibilities that may not have been desirable to all. The pathway to certification included group-based CBT readings and discussion followed by review of recorded therapy sessions by certified coaches who applied the full CTRS and offered feedback in group supervision sessions. WHS was also committed to paying the fees associated with certification, as they are not insignificant. This sustainment strategy was designed to optimize CBT fidelity among clinical staff, prevent drift, and increase the number of eligible coaches for overseeing the CBT endorsement system. Importantly, therapists who became coaches had reduced youth caseload expectations to ensure sufficient time to engage in their coach-specific duties, such as meeting with staff as part of the endorsement process.

Discussion

This study is the third in a three-part series with the first articulating the activities central to preimplementation (e.g., identifying and prioritizing barriers, matching and deploying strategies to prioritized barriers; Scott et al., this issue) and the second articulating six critical strategies deployed in the implementation phase (e.g., adapting CBT, training the trainers; Rodriguez-Quintana et al., this issue). This study offered a detailed account of the strategies deployed in the sustainment phase of a CBT implementation effort in a youth residential setting. Seven strategies addressed critical barriers, such as turnover, buy-in, and CBT drift; sustainment teams; new hire orientation plan; hosting monthly CBT campaigns; professional roles and performance evaluation; CBT endorsement system; and pathway to certification. Taken together, across phases, WHS had the knowledge, expertise, materials, and strategies to deliver a component-based model of CBT to youth with comorbid conditions in a milieu-based treatment setting.

Despite the extra work demanded of staff on the sustainment teams, WHS prioritized their indefinite continuation. This was, in part, because there remained a significant number of staff who had not yet achieved CBT skill endorsement. The sustainment teams identified staff who appeared ready for endorsement and offered encouragement to submit an application to an on-site coach for formal assessment. The sustainment teams also identified staff who appeared disinterested in or who struggled with CBT skills and tried to motivate or provide practice and feedback opportunities. The other reason the sustainment teams continued was to ensure that CBT remained a priority for youth and staff alike. They maintained monthly campaigns and biannual in-house Core Skills Refresh daylong trainings. This kind of investment may prove critical for the livelihood of CBT in RTFs where resources are slim, competing demands are high, and turnover is abundant.

The levels of CBT endorsement showed promise but were particularly suboptimal for Levels 2 and 3 at the end of the 5-year partnership. Recall, Level 1 endorsement (CBT model and behaviorism) could be obtained via a written test that was intended to elicit declarative and procedural knowledge (Bennett-Levy et al., 2009; Scott et al., 2016). Moreover, the Level 1 content was taught in the new hire orientation week and time was protected at the end of the week to take the test. This, and because approaching endorsement was inherently linear (staff could approach Level 1, then Level 2, then Level 3) explains why so many staff had achieved Level 1 endorsement by the end of the sustainment phase. Levels 2 and 3 consisted of the six core CBT skills (three at each level). Staff could approach endorsement for each skill individually within their current level. Staff were required to first practice articulating the critical elements of the core skill with a sustainment team member, and then engage in a standardized role play where the sustainment team member offered feedback consistent with a rating from the CTRS (Young & Beck, 1980). If staff received a score of 4 or higher in their practice session, then they could formally apply (via a one-page application) for an endorsement session with a coach. During the implementation phase, a cohort of four coaches was established who were the only staff able to officially endorse staff in CBT. This created a severe bottleneck given that WHS employed over 400 staff across the two sites and each skill required at least 15 minutes (average 30) for endorsement testing with a single coach. In our sustainment team meetings, we engaged in creative problem solving to open up new opportunities for the endorsement process. For instance, we considered allowing staff to do the endorsement testing with a sustainment team member and audio record it for coach review. However, this idea was rejected because (a) coaches were certified to deliver CBT, whereas sustainment team members were not; and (b) coaches felt strongly that the training they received in CBT supervision allowed them to offer actionable feedback to staff. We also considered dropping the role-play component of the testing, but dismissed that option given the robust evidence that self-reported competence does not align with objective observations or even supervisor report (Rozek et al., 2018). WHS may eventually achieve its goal of at least 80% endorsement across staff and CBT levels simply as a result of time, but this goal will be made reachable once additional therapists are certified through the pathway created in the sustainment phase.

Finally, of the strategies deployed, WHS did prioritize a couple from the category “utilize financial strategies” (Waltz et al., 2015). There are some pilot data that community mental health therapists do implement EBPs when monetary rewards are provided directly to them as opposed to the agency as a whole (Beidas et al., 2017). WHS was limited in the amount of, and how frequently they could offer, monetary incentives. They decided to provide a one-time bonus for achieving endorsement at each of the levels. They were also able to offer promotion with a pay raise that could only be achieved if endorsed at certain levels. Although staff reported an appreciation for these incentives, it seemed that they were too distal or insignificant in their amount to entice staff to move toward endorsement. It is important for implementation researchers to develop and test other incentive strategies that do not require money given the limited resources of these types of settings. Examples that we have observed in other settings include team competitions to achieve a threshold of penetration and staff recognition in team-based meetings (Boyd et al., 2018).

Two strategies were not prioritized from the original blueprint that was codeveloped in the preimplementation phase: obtain and use consumer feedback and use mass media (e.g., get a press release out with data from CBT implementation). Similar types of strategies were also not deployed in the implementation phase. To be clear, youth codeveloped many of the CBT materials and processes through daily input and informal interaction with the implementation teams. However, it seems as though this replaced any formal opportunity to obtain and use consumer feedback. This may be because the process and products were truly codesigned, but also could be a missed opportunity to engage in formal program evaluation (Bach-Mortensen & Montgomery, 2018; Posavac, 2015). Given this lack of formal evaluation, WHS was not in a position to engage the mass media during the sustainment phase. WHS continues to seek funding opportunities to engage in a formal and robust evaluation given their incredibly positive experience with CBT implementation and how they believe it has transformed their care and youth experience.

Limitations

Despite the many successes observed within the sustainment phase, there are several noteworthy limitations. First, we are unable to disentangle the impact of individual strategies given they were deployed contemporaneously. Given this, we cannot say whether the observed impacts require such a comprehensive and potentially costly implementation plan. Second, we were unable to follow WHS beyond the 1 year of active sustainment support, which limits our ability to report on the trend of increasing CBT endorsements across staff. It could be that once active sustainment support was removed that CBT also decreased. Third, the lack of a formal program evaluation limits our ability to form conclusions about staff and youth views about the CBT-RE skills and subsequent changes across time. Fourth, the full partnership and study lasted 5 years, which might not be feasible for other RTFs seeking to implement CBT. Having said that, given the dependencies and protracted timeline for (a) certifying therapists, (b) training coaches, and then (c) endorsing staff, the implementation phase could have benefited from a full additional year to get to target levels of endorsement that WHS would seek to sustain. Finally, although WHS experienced growth in their programs, offerings, and capacity across our partnership, which suggested a positive fiscal and community benefit related to CBT, we cannot confirm this link due to the lack of a comparison group.

Beyond these evaluation-related limitations, issues did arise in the final year of our partnership that threatened the sustainment of CBT-RE at WHS. In particular, the director of clinical training, who had partnered with the director of operations to initiate this entire effort, was recruited to another position outside of the organization. This person was also one of the only four CBT certified coaches, which would further shrink the bottleneck for endorsement. Despite this threat, WHS nimbly shuffled the work of existing coaches, promoted other therapists to fill gaps that emerged, and carried on in her absence. Fortuitously, contemporaneously, the most influential operational opinion leader who was once a “nay-sayer” came out in our final site visit and explicitly declared his champion status. He shared with us his reflections across the life of the partnership that is offered below. “When the CBT implementation was first introduced I was very hesitant that instituting CBT in the culture would have productive outcomes. It took me awhile to get [in]vested because I wasn’t a fan of the positive reinforcement aspect. Once I took the time to learn more about CBT and became fully endorsed I began to speak about CBT in team meetings and just general conversation with staff which helped them to buy in. I began to see improvements on the way staff communicated to clients and also a reduction in the numbers of physical management that the site was having. CBT has definitely had a positive shift at Wolverine Secure Treatment Center. It has changed the way we think and to better understand how to help clients.

“There was a situation where a client was upset during a basketball game and grabbed the ball and threw it trying to hit another client with it and actually hit a staff with it. Before the CBT Implementation, just last year this time, the client would’ve been restrained, no questions asked. But this time staff attempted to do CBT with the client. The client then ran over to the basketball rim and jumped and pulled himself up on the rim standing on it 10 feet off the ground. Staff continued to actively listen with the client for 30 minutes before the client safely came down.

“There have been multiple clients that left WSTC that call us to give us updates on their progress after leaving [the] program. They all discuss how they’ve incorporated CBT into their personal life and how positive it has been for them. The core skill they discussed most is SPEED and how to regulate it to help them to problem solve to avoid them getting to the crisis stage.

“Overall, the CBT Implementation Project was a great direction for the agency to move in. It has helped our staff and clients to better understand themselves and to self-regulate.”

Conclusions

This study described the sustainment phase of a 5-year collaborative effort to integrate CBT into youth RTFs. This phase was characterized by seven key strategies for which details of their deployment are offered for ease of replication. This process offers an exemplar for others exploring how best to support community settings in the final phase of integrating a new, complex psychosocial intervention.

Highlights.

• Planning for sustainment must begin early in an implementation effort, but there remain unique sustainment phase activities to address barriers like turnover and drift

• Seven strategies were key to sustaining cognitive-behavioral therapy (CBT) in this academic–community partnership

• A system for monitoring fidelity to CBT can be administered by on-site experts and motivate staff to learn CBT

Appendix

Appendices

Appendix A

CBT Level 1 Test: Behaviorism and the CBT Model

Select the best option.

1. One of your clients who doesn’t like attention showed up to group for the first time in weeks. What would be an effective way to increase her behavior?

   1. Say nothing but award her bonus points.
   2. Praise her as she steps into the room.
   3. Ask her why she finally came.
   4. Call her out during group to contribute.
2. One of the youth who reports hating all staff recently started also arguing with his peers during lunchtime. What would be an example of negatively reinforcing the behavior of NOT arguing with peers?

   1. Provide him with an extra dessert every time he does not argue with his peers.
   2. Remove television time from his schedule every time he argues with his peers.
   3. Assign a youth care worker to sit with him every day at lunch. He stops arguing with peers so the youth care worker doesn’t accompany him to lunch.
   4. Allot extra time for him to play sports every time he does not argue with his peers.
3. Which type of reinforcement schedule would be more effective for increasing cooperation between youth if it seems like it is a new behavior for them?

   1. Praise them every time.
   2. Praise them every other time.
   3. Praise them after a variable number of times (like after 2 times, then 5 times, then 3 times, with no stable number).
   4. Praise them every 5 minutes.
4. A client has broken a window. Which of the following is an example of a positive punishment for the behavior of breaking the window?

   1. Client is required to clean up the glass from the window.
   2. Remove television time.
   3. Client isn’t allowed to have dessert.
   4. Client can’t participate in music time.
5. One of the clients spits on a staff member for not having enough time to shower. What would be an effective way to decrease the spitting behavior?

   1. Apologize to the client for delaying their shower.
   2. Move the teen to their room and have them wait quietly for 5 minutes before their shower.
   3. Ignore the spitting and turn your back on the client.
   4. Give them extra shower time.
6. A sociable client rejects going back to the unit after mealtime. What is an example of negative punishment?

   1. Do a room clear (so the teen is the only one left in the lunchroom).
   2. Clean up table at next meal.
   3. Go to bedtime early.
   4. Do nothing.
7. Your team has tried to get rid of an undesired set of behaviors, like attempts at property destruction (e.g., flipping over table, kicking walls), by first approaching the teen using active listening and then trying to identify a problem, think of solutions, choose a solution, and how did it work? (ITCH), but the teen seems to be getting worse. Your team considers extinction, which would mean

   1. The teen would be punished with increasingly worse consequences until the behavior is extinguished.
   2. The teen is reinforced for desirable behaviors until the acting out is extinguished.
   3. The staff do not engage the teen during the aggressive acts to avoid accidentally reinforcing it, continue this strategy even though it initially increases, and the behavior eventually stops.
   4. The teen is restrained every time and eventually the behavior will be extinguished.
8. A youth has been cursing out the teacher. The staff have been variably reinforcing him by giving him attention. Recall that the variable schedule is the most resistant to extinction. Why might extinction of this behavior be really hard for staff to do given that it’s been variably reinforced?

   1. Because the staff have to resist the urge to accidentally reinforce the teen when an extinction burst of cursing worse than usual emerges.
   2. Because the staff have to resist the urge to punish the teen when an extinction burst of cursing worse than usual emerges.
   3. Because the staff has to count the number of times they reinforce the teen to make sure they don’t do it too often.
   4. Because the teen is in control.
9. A client yells at a staff member for “making them fail their day.” What skills can you use to redirect the client?

   1. Assertiveness and redirection.
   2. Active listening and redirection.
   3. Command and redirection.
   4. Punishment and redirection.
10. A client just witnessed their best friend flipping over a table during dinner and their SPEEDis in the orange zone. What can you do to redirect their attention?

    1. Ignore their distress.
    2. Ask them to be assertive.
    3. Mention that their friend will be punished.
    4. Distract the teen by talking about the movie they will play on the unit tonight.

11. Describe the cognitive-behavioral therapy (CBT) model using one of the following examples: (a) roller coaster story, (b) texting a person who does not text you back, or (c) a person not saying hello back to you when you pass them in the hall. (Use the page below.)

Appendix B

Self-Perceived Excess Energy and Distress and CBT Skill Tracking

Skill	Time/Date	Sun.	Mon.	Tues.	Wed.	Thurs.	Fri.	Sat.
SPEED check	Morning
	Afternoon
	Evening
Active listening	Morning
	Afternoon
	Evening
ITCH	Morning
	Afternoon
	Evening
SPEED maps	Morning
	Afternoon
	Evening
CAPES	Morning
	Afternoon
	Evening
CBT chat forms	Morning
	Afternoon
	Evening
TIP	Morning
	Afternoon
	Evening

Note. SPEED = self-perceived excess energy and distress; ITCH = identify a problem, think of solutions, choose a solution, and how did it work?; CAPES = closeness, accomplishment, physical activity, enjoyment, sleep; CBT = cognitive-behavioral therapy; TIP = toward engaging the senses, intense physical activity, and paced breathing.

Use this scale to rate how you used the core skills:
0	Didn’t think about it
1	Thought about using but didn’t
2	Tried using but it didn’t help
3	Tried using and it helped