Stakeholder Perspectives on Implementing Cognitive Behavioral Social Skills Training on Assertive Community Treatment Teams

Abstract

This study examined stakeholder perceptions of the “fit” between cognitive-behavioral social skills training (CBSST) and assertive community treatment (ACT) when implementing CBSST into existing community-based ACT teams. Focus group feedback was collected from a diverse set of stakeholders (i.e., clients, providers, supervisors, agency administrators, public sector representatives, and intervention developers). Results identified perceived client and provider benefits for integrating CBSST into ACT while highlighting the importance of purposeful adaptations, training, and implementation tools to facilitate structural and values fit between CBSST and ACT. Study findings will inform future endeavors to implement CBSST and other relevant EBPs into ACT.

Introduction

Research has identified implementation “context” as a critical component affecting evidence-based practice (EBP) implementation and sustainment (Greenhalgh, Robert, Macfarlane, Bate, & Kyriakidou, 2004; Aarons, Hurlburt & Horwitz, 2011; Klein & Sorra, 1996). This awareness has contributed to the development of initiatives to improve EBP-contextual fit via strategically assessing and changing aspects of the implementation setting (Ehrhart, Aarons, & Farahnak, 2014; Shea, Jacobs, Esserman, Bruce, & Weiner, 2014) and/or adapting non-essential aspects of the EBP to better fit local conditions (Aarons et al 2012). While the importance of understanding and enhancing EBP-contextual fit is widely accepted, much of this research focuses on situations where a new EBP was implemented into a “services as usual” setting that was not predominately EBP-based. Less is known about how the fit between multiple interventions affects the integration a new EBP into a service delivery context that was already providing one or more specific EBPs. As the number of available EBPs has proliferated across many different service areas and organizations have shifted towards greater emphasis on the provision of EBPs, it is becoming increasingly likely that new EBPs are implemented within the context of other existing EBPs. Since a key feature of EBPs is that they focus attention to intervention fidelity (i.e., the extent to which to the actual services provided adhere to the research-informed practice protocols), when multiple EBPs are expected to be provided the fidelity requirements of each must be taken into consideration to ensure high fidelity implementation and utilization. This circumstance can create new challenges as organizations and service systems attempt to identify, adopt, implement, and sustain multiple, potentially overlapping, EBPs relevant to desired outcomes (Sedlar, Bruns, Walker, Kerns, & Negrete, 2017). In this present research we assess how perceptions of fit between an existing EBP (Assertive Community Treatment; ACT) and a new EBP (Cognitive Behavioral Social Skills Training; CBSST) affected the implementation of the new EBP. This current study was part of a larger Hybrid Type 1 effectiveness and implementation study (Curran, Bauer, Mittman, Pyne, & Stetler, 2012), that focused on the assessment of CBSST effectiveness when added to ACT teams (as compared to ACT services as usual), while also systematically collecting information regarding implementation processes.

Assertive Community Treatment (ACT)

The ACT model is a team treatment approach with shared, low caseloads and community-based service delivery focused on reducing hospitalizations, maintaining housing, and improving daily living skills (Stein & Santos, 1998). ACT services are available 24 hours a day, 7 days a week to respond to and support ACT clients if they are in crisis situations. Caseloads on ACT teams are shared across different team members; thus, clients have contact with multiple different clinicians within and across weeks. ACT teams typically consist of case managers, substance abuse specialists, vocational/supported employment specialists, peers, nurse specialists, and psychiatrists. The education level of ACT staff is varied, but staff typically have a bachelor’s or master’s level education, with very few doctoral level staff or licensed mental health providers. ACT team members provide a combination of services, including case management, 24/7 crisis intervention, acting as a payee, interacting with collaterals, monitoring clients, responding to crises, and assistance with housing and other basic living needs. ACT treatment, including CBSST-enhanced ACT, is usually provided in the community (i.e., not in a clinic office). ACT visits are expected to occur at least weekly over an extended period of time (i.e., typically more than a year). The length of each ACT visit varies considerably based on the current needs of the client. ACT services are typically reserved for persons with multiple psychiatric hospitalizations over a one year period or frequent episodes of homelessness or incarceration.

Cognitive Behavioral Social Skills Training (CBSST)

CBT and SST are two well-validated EBPs that have been shown to improve functioning and are recommended in several treatment guidelines for schizophrenia (Gaebel et al., 2005; Dixon et al., 2010). SST targets functional skills capacity, and CBT focuses primarily on cognitions (e.g., defeatist performance attitudes) that interfere with effective skill performance and taking steps towards personal recovery goals. Prior research has demonstrated that CBSST, a bundling of these two potent interventions, is effective at improving functioning in schizophrenia (Granholm et al., 2005; 2007; 2015; Granholm, Holden, Link, McQuaid, & Jeste, 2013; Granholm, McQuaid, & Holden, 2016). A key element of the CBSST intervention is a treatment manual with a client workbook that describes the skills and homework assignments for each session (Granholm et al., 2016).

The CBSST manual consists of three 6-session modules for a total of 18 weekly therapy sessions. Within sessions, cognitive therapy is combined with role-play practice of communication skills and problem-solving training. Participants usually repeat the 18 sessions to further practice the new skills and reinforce their understanding of the concepts; thus, CBSST is frequently administered over 18-36 weeks.

Implementation of CBSST into ACT

Best practice guidelines and nationalized healthcare systems recommend or mandate provision of several psychosocial evidence-based practices (EBPs) for schizophrenia, such as cognitive-behavioral therapy for psychosis (CBT) and social skills training (SST; Gaebel, Weinmann, Sartorius, Rutz, & McIntyre, 2005; Dixon et al., 2010). Psychosocial EBPs like CBT and SST improve social functioning (Wykes, Steel, Everitt, & Tarrier, 2008; Kurtz & Mueser, 2008); however, they are frequently unavailable to people with schizophrenia (Lehman et al., 1998; Mojtabai et al., 2009). To increase the availability of CBT and SST for persons with schizophrenia, we sought to implement cognitive-behavioral social skills training (CBSST), an intervention that combines CBT and SST to improve functioning in schizophrenia (Granholm, McQuaid, & Holden, 2016), into ACT. Since ACT is widely implemented in community mental health systems throughout the United States (Drake, Bond, & Essock, 2009; Mueser, Bond, Drake, and Resnick, 1998); implementing CBSST on ACT teams could broadly increase the availability of CBSST. In addition to being widely disseminated, ACT provides a promising platform for CBSST implementation because the rehabilitation focus of ACT and some components of the service delivery structure of ACT (e.g., low client-to-staff ratios and weekly client contact) are consistent with the focus and structure of psychosocial rehabilitation interventions like CBSST. The community-based service model of ACT should also facilitate many opportunities for in vivo practice of CBSST.

Assessing Fit between CBSST and ACT

Acknowledging that fit is typically comprised of both structural and value-based/ideological aspects (Aarons et al, 2011), we adapt the definition of intervention contextual fit advanced by Horner, Blitz, and Ross (2014), to define intervention integration fit as the perceived match between the components, procedures, and values of the EBP to be implemented and the existing EBP(s). Some characteristics of ACT and CBSST were expected to fit reasonably well with each other due to their mutual focus on psychosocial rehabilitation and recovery goals and at minimum weekly contact with clients. However, other characteristics of the two interventions appeared less compatible and suggested the need for adaptations to one or both of the models, such as the team-based delivery of ACT vs. single clinician delivery of CBSST, the relative brevity of many ACT visits vs. 1-hour (or longer) sessions for CBSST, and the delivery of most ACT services in the community vs. typical delivery of CBSST in a clinic setting.

To summarize how ACT and CBSST were expected to fit together and identify specific areas with which to assess ACT-CBSST fit, we developed the Tool for Integrating Multiple Interventions (TIMI). The six intervention domains included in the TIMI (target population, intervention content, frequency/duration, setting, service delivery format, and primary outcomes), were informed by prior initiatives implementing other EBPs into ACT teams (e.g., Borroughs and Somerville, 2013; Williams, 2008), a systematic review of EBP modifications that identified intervention content and context (e.g., format, setting, population), as primary features adapted to improve EBP fit (Stirman, Miller et al, 2013), and an understanding of the importance of good “innovation-values” fit for successful implementation (Aarons et al, 2011), which highlights the need for the compatibility of anticipated goals and desired outcomes across the multiple EPBs in addition to demonstrating reasonable structural fit together.

Figure 1 depicts the characteristics of ACT, CBSST, and the projected integration of CBSST into ACT across the following key intervention domains: 1) target population, 2) content and structure, 3) frequency/duration, 4) context/setting, 5) service delivery approach, and 6) outcomes. These six domains essentially represent, respectively, the “who”, “what”, “when”, “where”, “how”, and “why” for the service delivery of each intervention and for the integrated intervention as in more detail in the following sections.

Figure 1.

Application of the Tool for Integrating Multiple Interventions (TIMI) to the Implementation of CBSST into ACT

Target Population

As shown in Figure 1, the population targeted to receive the CBSST intervention as part of their ACT team services was expected to consist of persons with serious mental illness (SMI), particularly schizophrenia, which is largely consistent with the traditional target populations for both the CBSST and ACT EBPs.

Intervention Content and Structure

The CBSST intervention content consisted of structured/manualized lessons that addressed both cognitive and social skills development. This CBSST content was expected to be delivered during standard ACT case management visits in order to teach clients the skills to better meet their specific needs. The manuals provided both the content (i.e., specific cognitive or social skills) as well as structure (i.e., components to include in each session such as review of prior homework and overall sequence of sessions that build upon each other to teach the skills). The core CBSST skills, including the patient workbook and therapist manual, all skills content and training procedures were identical to prior trials and not altered for delivery on ACT teams.

Frequency/Duration

To further facilitate the combination of case management with CBSST within a single session, and to accommodate for the higher levels of symptoms exhibited in many ACT clients, the usual length of CBSST sessions was reduced from 60-90 minutes to 30-45 minutes, and even briefer when needed. It was also expected that the CBSST-related content would be delivered in at least one of the weekly visits for up to 36 weeks. Although all of the CBSST content could be delivered in 18 sessions, additional practice of the skills for up to a total of 36 sessions was predicted to lead to greater improvements. While the briefer sessions would likely result in less time for skills training, it was expected that increased opportunities to support in vivo use of participants’ skills in ACT would compensate for less in-session practice.

Context/Setting

Adapted CBSST sessions were delivered at sites in the community that were mutually agreed upon by the client and ACT team member (e.g., residential settings, clubhouses, coffee shops, parks) instead of a typical office setting. Clinician’s strategic utilization of selected community settings was expected to create opportunities to provide CBSST training to clients in comfortable and familiar settings (e.g., doing repeated role plays of a skill alone with a client at his/her home), as well as for in vivo practice of targeted CBSST skills in community settings relevant to the needed skills (e.g., prompting an interaction with a store clerk). Both types of community practice were expected to be common and contribute to improvements in real-world functioning.

Service Delivery Format

Traditionally, all CBSST sessions were delivered as a group therapy by a single clinician or the same pair of leader clinicians. However, to fit into the structure of ACT services (i.e., delivered in individual one-on-one meetings with shared caseloads across clinicians), the service delivery format of CBSST was adapted such that different clinicians would deliver different individual sessions from one week to the next.

Outcomes

Both CBSST and ACT are recovery-oriented interventions with a goal to improve functional outcomes in clients with SMI. Allness and Knoedler (1998) identified the primary goals of ACT treatment as being: “to lessen or eliminate the debilitating symptoms of mental illness…, to meet basic needs and enhance quality of life, to improve functioning in adult social and employment roles and activities…” (p. 2). ACT often focuses on daily living needs and maintaining stable housing, close monitoring of the psychiatric illness and treatment, avoiding crises, and reducing psychiatric hospitalizations. CBSST similarly has goals to improve living, learning, working and socializing, but attempts to do so by teaching cognitive, social, and problem-solving skills. In this manner, CBSST and ACT have similar and complementary overall goals, but often focus on achieving different specific outcomes.

Summary of Anticipated Fit between CBSST and ACT

Overall, CBSST and ACT were expected to fit relatively well together, either through their similarity, complementarity, or purposeful adaptations to facilitate integration of CBSST into ACT. The degree of change experienced by ACT team members due to the integration of CBSST was expected to be relatively minor and primarily confined to the “Content” domain with the additional requirements to provide structured/manualized sessions with ACT clients. The implementation of CBSST into ACT was purposefully designed such that there would be no reduction in capacity to achieve high fidelity for either EBP (i.e., no model components related to delivering either EBP with high fidelity were removed or negatively impacted as part of their integration).

Method

Participants

The delivery of CBSST within ACT teams was handled by two private large multi-service behavioral health agencies located in a southwestern metropolitan area. Because each agency was funded by local governments to provide ACT services, the ACT teams were considered part of the public sector behavioral health system. As part of the overall study assessing the implementation of CBSST and ACT, representatives from six different stakeholder groups were recruited to participate in a total of 14 structured focus groups: two ACT client groups (n=8), six ACT team service provider groups (n=54), three ACT team supervisor groups (n=11), one agency administrator group (n=5), one public sector administrator group (n=5), and one group for the integrated CBSST in ACT team developers/trainers (n=4). The parent clinical trial (Granholm et al., 2015) targeted functional outcome in schizophrenia, so all clients with schizophrenia or schizoaffective disorder on the ACT teams were informed of the study and invited to participate through mailings, flyers and individual interactions with ACT providers. A subsample of these clients who enrolled in the study who received a meaningful exposure to the CBSST intervention and expressed interest in sharing their experiences were invited to participate in this study. All staff involved in providing ACT services at each site were encouraged to participate in the focus groups so the feedback included the perspectives of those recently exposed to the intervention (e.g., new hires) as well as those involved throughout the entire CBSST implementation process. Agency and county administrators who were involved in overseeing the ACT programs were contacted and asked to participate in their respective focus groups. To encourage frank discussion among participants, each focus group included only representatives from that stakeholder group. All stakeholders had direct experience either delivering or receiving CBSST or were in supervision or leadership positions during the implementation process.

Fidelity to the ACT model was evaluated for each ACT using the Dartmouth Assertive Community Treatment Scale (DACTS; Teague, Bond, & Drake, 1998). The DACTS assesses fidelity to 28 items/elements of ACT with 5-point anchored scales (1 = not implemented, 5 = fully implemented). These items are grouped according to three broad categories or subscales: Human Resources, Organizational Boundaries and Nature of Service. All teams had very similar fidelity and the mean total DACTS fidelity rating was 3.76 (range 3.61-3.86), suggesting fair to moderate level of quality and adherence. We also rated fidelity from over 600 sessions using standard fidelity measures and ACT providers delivered CBSST with adequate fidelity. The mean fidelity rating on the Cognitive Therapy Rating Scale for Psychosis (CTS-Psy total score; Haddock et al. 2001) was 36.2 (SD=7.1); 30 is considered adequate fidelity (85% of providers achieved a total score > 30).

ACT provider and supervisor focus groups were conducted after each ACT site had implemented CBSST for at least 7 months (range=7-17 months). The mean number of months of experience with CBSST for each stakeholder group at the time of the focus group was as follows: 8.9 (SD=4.8) for providers; 12.9 (SD=8.8) for clients; 11.1 (SD=5.8) for team supervisors; 28.0 (SD=0.0) for agency administrators; 24.8 (SD=7.2) for public sector administrators; and 24.0 (SD=0.0) for intervention developers/trainers.

Of the 87 total focus group participants, 73.6% (n=64) were female. Almost half were between 25 and 34 years of age (n= 39; 44.8%). Most participants had a bachelor’s (n=21; 24.1%) or master’s degree (n=41, 47.1%), and most were ACT providers or ACT team supervisors (n=64, 73.6%). Of the 64 ACT provider and supervisor focus group participants 8 (12.5%) were licensed providers and 10 (15.6%) were interns/trainees. These participants primarily had a background in psychology or social work (n=37; 57.8%). Most were employed full-time (n=60, 93.8%) and had been with their respective agencies for less than 5 years (n=50, 78.1%). Most had been providing mental health services for at least 5 years (n=44, 68.8%).

The study protocol was reviewed and approved for the ethical treatment of human subjects by the Institutional Review Board (IRB) of the Veterans Affairs San Diego Healthcare System. Ethical review and approval was also provided by the Research Committee of the local public sector behavioral health system.

Data Collection

Focus groups typically lasted between 45-60 minutes, with at least two study representatives conducting each focus group. Participants were asked to respond to one overall question: “What are the factors that influenced the implementation and use of CBSST on the ACT teams?” This prompted a wide-ranging and open discussion among the participants, in which many subtopics were articulated and expounded upon. One of the study representatives facilitated the discussion by asking clarifying questions or confirming understanding of thoughts expressed by participants, aiming to keep discussion focused on the subtopic at hand to capture all related ideas. Once a subtopic was exhausted, the study representative provided another open prompt, such as, “What else influenced the implementation and use…?” Each factor communicated by a participant was paraphrased and documented by another study representative in “real time” and projected on a screen for all participants to see during the focus group. This allowed for refining and clarifying responses as needed and encouraging others to contribute factors not yet identified by the group. At the conclusion of the focus group, the running list of factors was reviewed to see if any items should be added or amended. The focus group was audio recorded to ensure that no implementation factors were missed during the “real time” generation of the list.

Data Preparation and Analysis

The audio recording for each focus group session was reviewed in conjunction with the associated list of implementation factors generated during the session to establish list accuracy and completeness. Each individual focus group list was also reviewed to identify and remove any clear duplicate factors within that focus group. The final statement lists from the 14 focus groups (a total of 934 implementation factor statements) were then entered into the NVivo 10 qualitative data analysis software package (QSR International, 2012).

The first data analysis step involved open coding the focus group implementation factor statement lists to identify major issues and themes among the implementation factors identified by participants (Corbin and Strauss, 2008). The statement list was reviewed for salient categories of statements, such as EBP content-related statements regarding agendas, workbooks, and homework, or EBP delivery-related statements regarding session time or session location. A sample of focus group implementation factor lists was independently coded and then compared by two investigators (DS and JN). The intervention developers (EG and JH) did not participate in the coding. Where discrepancies existed, the investigators discussed the rationale for each code and arrived at a consensus code by creating a new code or agreeing to use an existing one. The investigators reiterated this process of coding, reviewing, discussing, and refining the scheme before arriving at a stable set of primary codes, which was then applied to each factor statement from the focus group lists. The two investigators reviewed any factor statements that were identified as potentially difficult to categorize or spanning two different categories and collaboratively determined the most appropriate code via consensus.

While the coding of all factor statements resulted in the identification of a wide range of implementation factors, this study focused on themes directly related to the fit between CBSST and ACT across the six intervention domains presented in Figure 1 (i.e., target population, content & structure, frequency/duration, context/setting, service delivery format, and outcomes).

Results

Key findings related to stakeholder perceptions of how the fit between CBSST and ACT affected its implementation and use are presented for each of the intervention domains listed in Figure 1. Some findings relate to the intersection of multiple invention domains—for example, characteristics of the target population were perceived by some participants as creating challenges for achieving the intended frequency/duration of delivering CBSST. In these instances, the authors located the discussion of the findings in the domain determined to be the focal point of the stakeholder comments.

Target Population (Who)

Overall consistency between CBSST and ACT target populations.

Based on stakeholder feedback from all 14 focus groups, CBSST was generally perceived to be appropriate for the typical client populations already receiving ACT services. Clients who participated in the focus groups indicated that that the lessons and workbooks appealed to them. Providers and clients also commented that they thought CBSST helped achieve positive client outcomes, especially those related to functioning and goal attainment. These statements indicated that CBSST was generally perceived to meet important needs of clients who were receiving ACT services.

Difficulty implementing CBSST with some ACT clients.

While CBSST was generally thought to be appropriate for ACT clients, comments from providers and supervisors in nine focus groups indicated that CBSST was thought to be more difficult to deliver to some clients, particularly those with “complex needs.” Team supervisors and providers noted that co-occurring substance use problems and impaired cognitive functioning in some clients made delivery of CBSST more difficult (e.g., some clients were unable to keep track of homework and maintain their CBSST workbook). Providers also indicated that it was more difficult to deliver CBSST to clients who were clinically unstable or who frequently become involved in crisis situations. However, one provider acknowledged that these characteristics were often not a challenge specific to implementing CBSST, but a challenge to provide any services, including ACT.

Content and Structure (What)

Intervention Content

CBSST content was appropriate.

Stakeholders from the two client focus groups expressed positive comments regarding the appeal and relevance of CBSST lessons and workbooks. Some clients expressed having a positive experience with the workbooks, and others stated that they still referenced it after completing CBSST. Clients and providers from six focus groups described certain CBSST concepts as “great,” “helpful,” and “fun,” and agency administrators stated they heard from staff that the clients typically liked doing the different modules. Overall, clients and providers seemed to think that the concepts within CBSST were relevant to the types of topics and issues they deemed important.

CBSST content was challenging in both positive and potentially negative ways.

Though clients and providers generally found CBSST to be appropriate, comments from six focus groups indicated that the content could be challenging to get through. One ACT client said: “the workbook challenges us [the client] to think, which is good, but can also be frustrating.” Other clients stated that some of the workbooks were too difficult to use, causing them to not want to try other workbooks. The perceived difficulty of the material could play a role for certain clients in terms of how much they are willing to engage in the intervention.

Intervention Structure

Providing CBSST competed with other ACT responsibilities.

A common concern expressed by providers and team supervisors in six focus groups was that delivering CBSST was a lower priority or altogether not possible in the midst providing crisis management or case management services. Providers commented about having “other things besides CBSST to accomplish during visits” and that at times “CBSST interferes with delivery of certain services to clients.” This was evident to them particularly “when clients [were] sick/in crises” or when “client safety” was perceived to be at stake. These statements indicated that there was a belief among providers that delivering a CBSST session and fulfilling their primary ACT responsibilities were competing demands, often at odds with each other. In these instances, providers found it challenging to successfully integrate the structured, skills-based teaching approach of CBSST into the ACT visit.

Resistance to manualized treatment practices.

One factor that inhibited utilization of CBSST within ACT services was the perception that CBSST was overly rigid or inflexible (identified in 10 focus groups). For example, one provider stated that when clients were motivated to work on or discuss a particular issue, “the CBSST curriculum may not match the client’s situation at that moment.” Some providers perceived “client resistance to structured material” delivered during their ACT visits. Provider, team supervisor, and intervention developer/trainer comments frequently highlighted the need to make the intervention feel more “organic” within the ACT model.

In addition to providing ongoing supervision and feedback to increase provider familiarity with CBSST and facilitate more seamless or natural inclusion into ACT visits, intervention developers/trainers continued to adapt materials and training techniques during the initial implementation of CBSST to reduce perceptions of intervention rigidity and facilitate flexible integration of CBSST. A simple, but important, training change identified by providers was the explicit recommendation by the intervention developers/trainers to teach whichever CBSST session in the treatment manual best matched the client’s current circumstances, rather than needing to maintain a specific session order. Providers and team supervisors also emphasized that encouragement from intervention developers/trainers to not use the CBSST manual at all or to apply the intervention flexibly “on the fly” during regular ACT encounters facilitated implementation and use of CBSST by ACT staff in a more “naturalistic” manner. Comments from the four provider and supervisor focus groups at the agency which first implemented CBSST into ACT indicated that the developer/training responsive to feedback and the resulting adaptations to initial CBSST session materials were perceived as making CBSST more flexible and easier to use.

Beneficial effects of structured CBSST sessions.

While challenging to implement, the additional structure CBSST provided to ACT client visits was also identified as a positive change. For example, some clients thought that CBSST provided a “powerful structure” or made the ACT visits more “purposeful.” As one client stated, “CBSST creates structure for staff to listen, which helps staff better understand our needs.” Similarly, some providers stated that CBSST provided beneficial structure to otherwise unstructured visits. Providers indicated that the “organized sessions gave providers confidence” in doing their jobs. This provider confidence seemed to result from the idea that as long as they were delivering the intervention based on the guidelines, they felt that they would be doing their job correctly and contributing to client improvements. This aspect of CBSST was believed to be particularly salient for new staff.

Frequency/Duration (When)

Challenge of completing CBSST sessions within typical ACT visits.

In contrast to the fixed weekly schedules and session duration of traditional CBSST, the frequency and duration of CBSST sessions when integrated within ACT visits was determined by the frequency and length of ACT team visits, which could vary significantly between clients and over time. Participants from five focus groups highlighted the challenge of integrating an intervention like CBSST that relies on dedicated time specifically focused on intervention content delivery into ACT. One provider stated that “providers lack the time during normal sessions with clients to delve as ‘deep’ as trainers want.” Other providers and clients also mentioned that the even the abbreviated length of the CBSST sessions initially designed for utilization within ACT was perceived as too long for some clients. As discussed in the Intervention Content section above, some providers also saw CBSST as competing for time with other priority tasks to be completed during ACT visits, such as crisis management.

Based on provider feedback, the intervention developers/trainers further revised the CBSST workbook during the initial CBSST implementation to highlight key CBSST elements essential to delivering each session effectively in a more compressed time span. One provider stated that “the revised, shortened CBSST manuals work much better” than the previous workbooks, and an intervention developer acknowledged that the “abbreviated workbook [was] more manageable,” indicating that adapting CBSST’s length to the typical duration of an ACT visit helped providers with their delivery of CBSST within ACT. This was another example where CBSST implementation benefited from intervention developers listening to staff feedback and responding through intervention adaptation.

Concerns about the intended frequency of CBSST session delivery.

In addition to concerns about the duration of ACT visits needed to adequately provide CBSST lessons, participants from six different focus groups indicated that the frequency of client contacts in which CBSST was discussed was not conducive to delivering high quality CBSST. As one provider stated, “consistent, weekly, face-to-face sessions [are] ideal but not feasible for ACT teams.” Weekly face-to-face sessions were hindered not only by “provider workload,” which was cited by providers and intervention developers as a substantial barrier, but also by needing to respond to client crises during sessions or when client crises prevented clients from meeting with ACT providers altogether. One provider stated that “when clients don’t participate for a while (e.g., sick/client crises) [it is] easy to lose CBSST momentum,” indicating that client circumstances can contribute to infrequent CBSST sessions.

Context/Setting (Where)

Interference with CBSST session delivery in some community settings.

While the “in vivo” context and setting of ACT visits was anticipated by the research team to provide a good opportunity for practicing CBSST skills in real-world situations, participants in seven focus group noted some difficulties delivering CBSST in their typical ACT meeting locations. Specific challenges with the location of service delivery cited by providers and team supervisors involved locations that had distractions or a lack of privacy. Providers indicated that these types of locations (e.g., “in the car,” areas with “people passing by”) were typical for ACT visits and could interfere with delivering CBSST sessions. Having a private, comfortable space for client sessions was considered ideal for delivering the CBSST lessons, but this was often not feasible when delivering CBSST within ACT.

Service Delivery Approach (How)

Importance of logistical supports for a team-based approach to CBSST delivery.

Using a team-based approach for delivering CBSST meant that one provider might deliver one session, and a different provider might deliver the next session. Stakeholder comments from 10 different focus groups highlighted the adaptation of CBSST to a team-based service delivery model and the additional logistics associated with providing a structured, multi-week curriculum as a team, particularly related to managing information exchange among clinicians. Providers and team supervisors noted an initial lack of systematic methods or documentation tools to help guide the sharing of information between staff members regarding CBSST progress (e.g., tracking which session should be delivered next or what homework had been assigned by using a team tracking sheet). This contributed to information getting “lost in the shuffle.” Comments also highlighted the need for agency and team leadership to facilitate logistics of delivering CBSST (e.g., team supervisors assigning the next CBSST session to providers at their daily morning staff meeting).

However, stakeholders also indicated that providers and intervention developers/trainers designed additional methods and tools that helped to facilitate the logistics of team-based CBSST delivery during the initial implementation process. These activities included creating and maintaining “an extra ‘provider’ manual to keep track of different providers’ progress” that was kept in the ACT office. The additional tools were seen as essential supports for delivering CBSST as a team.

Need for accountability and responsibility with team-based delivery of CBSST sessions.

In addition to logistical supports, comments from six focus groups regarding the adaptation of CBSST to a team-based service delivery model also mentioned a need to establish responsibility for CBSST client progress among the team members. Providers indicated that team delivery of CBSST had the potential effect of reducing the sense of “ownership” or “responsibility” that any one individual provider might feel to ensure a client’s progress through the CBSST sessions. In other words, since all team members were theoretically responsible for providing CBSST services, a diffusion of responsibility could occur in which no one felt responsible or was accountable for ensuring that CBSST sessions were delivered. Providers and team supervisors found it useful to create a “lead CBSST clinician” for each client to help address the issue of diffused responsibility. Having this lead clinician ensured that there was one person who made sure to monitor client progress through the CBSST sessions.

Reduced effectiveness of team-based teaching of CBSST.

Concerns about a lack of consistency caused by the team delivery approach were also raised, particularly by clients in the two client focus groups. Some clients indicated that not having the same provider deliver CBSST at every visit could be a “challenge to client progress” or make it “difficult to get into the flow of the material,” especially because providers may not be familiar with what was done in previous meeting or what strategies “worked” for teaching CBSST lessons to a specific client. One provider noted a potential problem with consistency as well, stating that “working as a team to serve one client can make it difficult to be consistent.” Concerns about team-based teaching highlighted the importance of establishing procedures to share information among team members about CBSST session progress (e.g., CBSST lead clinician and provider workbook) and possibly developing new systems to further support and adapt CBSST using a team approach.

Outcomes (Why)

Positive client outcomes attributed to CBSST.

Stakeholders in all 14 of the focus groups identified positive client outcomes from implementing CBSST and signaled its usefulness as an intervention with the ACT population. Providers stated that “CBSST is an effective tool” and “they have seen it work with clients,” especially when helping “facilitate achieving client goals.” Clients provided positive feedback about the role of CBSST in improving goal attainment as well, saying that it helped “client[s] reach their goals” and “break down a goal into smaller steps.” Clients also mentioned other positive CBSST outcomes, such as helping to “focus and organize thoughts,” “think about things in a new way,” “seek out information and tools to solve a problem,” and provide “proof that they can solve a problem.” Based on both provider and client comments, overall attitudes seemed to cast CBSST as beneficial in helping clients achieve positive outcomes.

Client improvements attributed to CBSST encouraged further provider use of CBSST.

ACT team members from four focus groups noted a positive relationship between seeing client successes due to CBSST and their own motivation to deliver CBSST. Team supervisors also identified a potential social learning aspect, in that “seeing provider success with CBSST increases buy-in” for other providers. Overall, stakeholders believed that seeing favorable client outcomes encouraged further efforts to implement and use CBSST.

Discussion

Utilizing the TIMI to assess stakeholder feedback regarding their perspectives on implementing CBSST on ACT teams in community mental health agencies provided insight into how well ACT and CBSST were perceived to “fit” together across key intervention domains and how this perceived fit affected implementation and delivery of CBSST. Stakeholder comments identified areas of good fit, but also lack of fit that required adaptations primarily to the format of delivery. While the CBSST skills training content, focus on recovery goals, and training procedures were unchanged, adaptations involved a change in format from a group intervention delivered in a clinic to an individual, team-delivered intervention in the community. Implementation was facilitated by the provision of needed implementation tools, organizational supports, and trainer/developer ongoing attention to adaptation and coaching during the implementation process.

Feedback from stakeholders suggested that it is vital to address the structural fit of CBSST with the existing ACT model and ensure that needed modifications and intervention tools are in place and supported by the organization. Examples of this feedback included provider comments that the frequency and length of sessions needed to deliver CBSST did not fit the typical frequency or length of ACT visits. Adaptations made after initial implementation to further shorten CBSST workbook sessions were reported to help providers quickly focus on key content areas and increased utilization of CBSST within ACT visits. Similarly, changing the service delivery format of CBSST from a single clinician-based method to a team-based method created logistical challenges related to tracking client progress, communicating client progress among team members, and a diffusion of responsibility among team members. However, these challenges were mitigated by developing tools and systems like assigning a lead CBSST clinician, who took responsibility for tracking client sessions and assigning providers to deliver CBSST to specific clients each week, and using a shared provider workbook for each client, which helped communicate client goals and homework assignments among providers. Ensuring that ACT teams have the time, tools, and organizational supports to deliver EBPs like CBSST that involve progressive skills training over multiple sessions is crucial for promoting a good structural fit with an existing ACT program.

Stakeholder feedback also highlighted the importance of good ideological or value-based fit between CBSST and the existing ACT program. Both interventions were designed to provide services to seriously mentally ill persons with schizophrenia. However, while both interventions were seen as beneficial for clients and complementary to each other in accomplishing the overall goal of improving client functioning within the community, each intervention was viewed as targeting different outcomes with varying levels of importance. ACT was perceived to emphasize basic needs, case management, and crisis prevention or intervention to a greater extent than CBSST and the need for these ACT services was often determined to be of higher priority than delivering CBSST. Stakeholder comments seemed to indicate that these value differences inhibited provision of CBSST within the ACT teams.

Despite the fact that these two EBPs seemed to be viewed by providers as competing demands, CBSST might provide to ACT teams a connection to a lost identity from the past, when ACT was more actively involved in efforts to systematically improve psychosocial functioning, independence, and other functioning outcomes, in addition to being crisis and case management focused (Allness & Knoedler, 1998). In many ways, CBSST provides an important ingredient in terms of what has been missing from the ACT of recent years: a stronger focus on recovery-oriented psychosocial rehabilitation that clients certainly need. This challenge of returning to a focus on rehabilitation while maintaining service orientation values was also highlighted as an implementation barrier when Illness Management and Recovery was implemented on ACT teams (Salyers et al., 2010). Future efforts to implement CBSST (or other EBPs) on ACT teams will need to ensure that providers believe that the EBP being implemented is consistent with their current roles and responsibilities as ACT providers. Successfully addressing this issue through training and leadership support will be increasingly important with the new emphasis on providing additional EBPs as part of high-fidelity ACT service delivery (Monroe-DeVita et al., 2011).

An important finding from the stakeholder comments was the value of purposeful adaptations made during the initial implementation process, which greatly facilitated and improved the structural fit between CBSST and ACT. Examples of such adaptations included those made by intervention developers in response to provider feedback, such as the adjustments to workbook materials and training that encouraged flexibility in service delivery, as well as adaptations made by the providers and team supervisors themselves, such as utilizing a shared provider workbook and developing other CBSST tracking and monitoring tools. The stakeholder comments affirmed that strategic and purposeful adaptations are crucial components of the implementation process that facilitate intervention fit and ongoing intervention sustainment (Aarons et al., 2012). While many of the tools and strategies created or refined during this current study will be available to facilitate future CBSST into ACT implementation endeavors, it will be important to continue to maintain feedback mechanisms that systematically identify and inform any other needed organizational adaptations.

The findings suggest that use of the TIMI would be beneficial in other settings where a new intervention is being introduced into the context of an existing intervention a new EBP. By utilizing the TIMI to review the characteristics of the new EBP and existing intervention(s) side-by-side, researchers may be able to: 1) gain a clearer picture into how best to integrate the EBPs; 2) identify potential areas of difference or tension between the new and existing practices; and 3) create or plan adaptations and training prior to and during initial implementation that are intended to facilitate good structural and values fit with the existing ACT service delivery model and beliefs of the ACT service providers.

Certain limitations to the study should be noted. First, given that clients in the clinical trial were recruited to have schizophrenia or schizoaffective disorder and only clients with meaningful exposure to CBSST were invited to participate in the focus groups, the opinions expressed by the clients may not generalize to other types of ACT clients. Second, in regards to the generalizability of the findings, the study collected data from nine ACT teams in two different agencies in a large metropolitan area; however, some studies have shown that ACT programs are prone to drift or experience variations over time (Monroe-DeVita, Morse, & Bond, 2012). The potential for local variations across ACT programs reinforces the importance of and need for local adaptations before and during the implementation process to improve the perceived fit of CBSST (or other EBPs) and ACT within the local context. The seven ACT teams showed very similar fidelity in the fair to moderate range, so finding should generalize to teams that meet at least basic fidelity.

Returning to the domains where the structural or values fit seemed to be strained during initial implementation, stakeholder comments highlighted specific areas that are crucial for guiding future efforts to integrate CBSST into ACT teams. Overall, the most important areas to address appeared to be: 1) the logistical challenges integrating CBSST session delivery into ACT delivery (e.g., managing team-based delivery of a multi-week structured EBP); and 2) provider perceptions about CBSST and ACT as competing for limited time during visits, with ACT providers emphasizing management to a greater extent than skills training for rehabilitation. Both of these issues can be addressed in future implementation endeavors by utilizing the implementation support tools and practices developed during this study (e.g., requiring a lead CBSST clinician for each client), as well as providing refined trainings and leadership coaching that highlight the importance of a rehabilitation focus and demonstrates its contribution to meeting overall ACT objectives.

Team supervisors and providers also commented that some client characteristics, such as substance use, homelessness, and greater severity of cognitive impairments, made CBSST delivery more difficult. We previously found that severity of cognitive impairment did not influence the effectiveness of CBSST when delivered in groups in a more structured and controlled research setting (Granholm et al., 2008), but clients on ACT teams in the community are typically lower functioning than clients in other outpatient settings. Additional research on the potential moderators of CBSST outcomes is needed to determine whether specific clients will benefit more from receiving CBSST on ACT teams. It is important to note that these client characteristics may not be specific moderators of outcome in CBSST. As suggested by providers in our focus group, factors such as homelessness, substance abuse, and cognitive impairments are likely to make delivery of all treatments more difficult, including ACT.

In future studies, we plan to develop and test the utilization of technological tools, such as smartphone applications and web-based resources, which will provide real-time coaching, fidelity feedback, and progress-tracking capabilities in a way that is more accessible to providers. Utilizing this type of technology will allow providers to reinforce their skills, share information about sessions with other providers, monitor recovery outcomes, and track logistics more easily while out in the field. Monitoring and rapid feedback of outcomes may also help providers identify and highlight clients who are benefitting from CBSST, which, according to the current study, may contribute to improved provider satisfaction and greater utilization of CBSST. These enhanced tools have the potential to strengthen the perceived structural and values fit between CBSST and ACT.