What Influences Evidence-Based Treatment Sustainment after Implementation Support Ends? A Mixed Method Study of the Adolescent-Community Reinforcement Approach

Abstract

Background:

Little is known about clinician perspectives regarding the factors that support or hinder the long-term delivery (i.e., sustainment) of evidence-based treatments in community-based treatment settings.

Methods:

Clinical staff from 82 community-based treatment organizations that received federal grant funding to support the delivery of the Adolescent Community Reinforcement Approach (A-CRA), an evidence-based treatment for adolescent substance use, were asked to participate in interviews focused on understanding their perspectives about the sustainment of A-CRA. Qualitative themes were identified using inductive and deductive approaches. Then the themes were dichotomized (present/absent) so that quantitative comparisons could be made between staff from organizations that sustained and did not sustain delivery of A-CRA. Administrative data about each organization in relation to federal funding support and their primary focus was also examined to explore whether these characteristics were associated with A-CRA sustainment.

Results:

Staff (n= 134) representing 78 organizations participated in the interviews. Staff from organizations that had received multiple federal grants to support the delivery of A-CRA and whose primary focus was substance use rather than other conditions (mental health or primary care) were more likely to report sustaining A-CRA. Staff from sustaining organizations were more likely to report positive grant experiences and success with maintaining both organizational and external support in comparison to staff from non-sustaining organizations. Staff from non-sustaining organizations were more likely to report barriers to sustaining A-CRA, including more challenges with intervention delivery, and lack of internal support and external funding.

Conclusions:

Our findings lend empirical support for implementation theories in that multiple factors appear to be associated with long-term delivery of an evidence-based treatment. Although A-CRA was generally perceived positively by staff from both organizations that sustained

A-CRA and organizations that did not sustain A-CRA, inner setting factors (e.g., structural policies, leadership support and staff retention) along with outer setting factors (e.g., external funding support) were reported as key to A-CRA sustainment.

1. Introduction

Despite the recent development of several evidence-based treatments (EBTs) for the nearly two million adolescents with substance use disorders (SUDs), a mere 10% receive necessary treatment (Substance Abuse and Mental Health Services Administration, 2017). A host of stakeholders, including government agencies and community-based providers, have invested in closing this treatment gap. The Adolescent Community Reinforcement Approach (A-CRA) is one EBT for SUDs among adolescents (Dennis et al., 2004; S. H. Godley, Smith, Meyers, & Godley, 2016) that has received considerable community-based implementation. A-CRA seeks to supplant factors in an adolescent’s environment that encourage substance use, replacing them with pro-social activities and behaviors that foster recovery. To date, five randomized controlled trials testing A-CRA have demonstrated positive outcomes for alcohol use, mental health, and social functioning (Dennis et al., 2004; M. D. Godley, Godley, Dennis, Funk, & Passetti, 2007; M. D. Godley et al., 2014; Henderson et al., 2016; Slesnick, Prestopnik, Meyers, & Glassman, 2007). A recent review identified A-CRA as one of only five well-established treatments for adolescent SUDs (Hogue, Henderson, Becker, & Knight, 2018).

1.1. Strategies to improve EBT delivery

Government agencies and policy-makers who seek to improve the delivery of high-quality treatment, and ultimately to advance health outcomes, may use a variety of implementation strategies to encourage and support the adoption of EBTs, such as A-CRA, in private practice and community-based settings (B. D. Hunter, Godley, Hesson-McInnis, & Roozen, 2014; Miller & Muñoz, 2013; Powell et al., 2015). For example, EBT developers have provided training and consultations, which have improved implementation quality in community-based settings (Liddle et al., 2006; Morganstern, Morgan, McCrady, Keller, & Carroll, 2001) and government agencies have provided funding or financial incentives to encourage EBT adoption across state and county service systems that include community-based care (Jaramillo et al., 2019; Scudder et al., 2017; Sigel, Benton, Lynch, & Kramer, 2013). Yet there is a lack of knowledge of how implementation strategies contribute to sustainment, or the continued use of an innovation in practice to achieve program and population outcomes (Aarons, Hurlburt, & Horwitz, 2011; Scheirer, Hartling, & Hagerman, 2008). In other words, if an EBT can be successfully implemented, why are organizations not able to sustain it? This question is key to maximizing return on investments in SUD treatment delivery, which in the United States is largely done through public sources (S. H. Godley, Garner, Smith, Meyers, & Godley, 2011). Furthermore, sustained EBTs are essential to adequately reduce the public health burdens of substance use (Aarons et al., 2014; Hodge, Turner, Sanders, & Filus, 2017).

In one of the largest federally funded EBT implementation efforts to date, the Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Substance Abuse Treatment (CSAT) provided funds to community-based programs across 27 states from 2006 to 2012 to support the adoption of A-CRA (B. D. Hunter et al., 2014). Community-based SUD treatment organizations were given approximately three years of support, during which they were provided with funds for implementation, technical training, and performance feedback on clinician and supervisor certification processes. Despite that A-CRA has been shown to improve health outcomes when successfully implemented, its sustainment has been inconsistent—in terms of the number of elements sustained (e.g., delivery, model-specific supervision, training) and their quality (e.g., level of fidelity, comprehensiveness of training)—in community-based organizations following completion of CSAT funding (S. B. Hunter, Han, Slaughter, Godley, & Garner, 2015, 2017).

1.2. Sustainment within implementation science

Within the burgeoning field of implementation science, a host of conceptual frameworks have been put forth to understand the factors that influence implementation and sustainment (see Tabak, Chambers, Hook, & Brownson, 2018). At the time that we initiated our research on factors related to the successful sustainment of EBTs in substance use contexts, the Consolidated Framework for Implementation Research (CFIR; Damschroder and colleagues, 2009) was the most comprehensive of such frameworks available—it was based on systematic review of the literature that synthesized information from several theories and approaches. The CFIR remains one of the most widely used and cited frameworks in implementation science to date (Kirk et al., 2016). More specifically, we organized our data collection and analysis approach around the major domains that CFIR identifies as impacting EBT implementation. First, intervention characteristics are influential aspects of the EBT’s implementation, such as its relative advantage, design quality, adaptability, and complexity. Individual characteristics are those of the individuals who carry out implementation, such as their motivation and values, readiness to change, and knowledge and attitudes about the EBT. The inner setting refers to factors within the organizational setting where the EBT will be implemented (e.g., climate for change, leadership, available resources, structural characteristics such as size and stability), where the outer setting refers to the context within which that organization resides (e.g., external policies and regulations, characteristics of the patient population served, relationships with other organizations). Finally, implementation process refers to the activities through which implementation of the EBT proceeds, including the stages (e.g., planning, executing, evaluating) and people (e.g., opinion leaders, EBT champions) involved in managing the process.

Despite these insights, few studies have empirically tested how conceptual implementation frameworks like CFIR help to explain EBT sustainment. Studies of EBT implementation in youth mental health have identified various factors related to sustainment across CFIR domains, such as ongoing training and support, availability of stable and predictable funding mechanisms, positive provider and agency attitudes toward the EBT, provider retention, and match between the EBT and the patient population (Brookman-Frazee et al., 2018; Jaramillo et al., 2019; Rodriguez, Lau, Wright, Regan, & Brookman-Frazee, 2018; Scudder et al., 2017). Similar keys to sustainment have been identified for youth SUD treatments (S. B. Hunter et al., 2015, 2017), but that research is less extensive and primarily quantitative (whereas studies in youth mental health incorporated considerable qualitative data to better understand context). More work incorporating in-depth qualitative data is needed to understand how factors identified in the CFIR influence EBT sustainment within the unique contexts of SUD treatment clients, practitioners, and practice settings. Studies such as this can also help to build empirical evidence to support, refine, or refute conceptual frameworks as they currently stand.

1.3. Study aims

This study aims to contribute to the burgeoning research on EBT sustainment by identifying key factors that facilitated or hindered delivery of A-CRA following the end of the three-year, federally funded implementation support to community-based organizations. We used qualitative data collected from interviews conducted with clinicians responsible for delivering adolescent substance use treatment to help address our research questions. We aimed to examine whether these factors differed depending on the sustainment status of the organization. More specifically, we hypothesized that staff from organizations that sustained A-CRA (henceforth sustaining organizations, or SOs) would report more facilitators and fewer barriers to delivering A-CRA than staff from non–sustaining organizations (non-SOs). We also set out to examine whether the reported facilitators and barriers differed across staff from SOs and non-SOs and to provide illustrative examples from a clinical perspective. Finally, we examined how these reported factors aligned with the CFIR.

2. Methods

2.1. Sample of organizations

All participating organizations received funding from the SAMHSA CSAT to support delivery of A-CRA. On average, organizations received $300,000 per year for up to three years. Sixty-nine organizations started to receive funding sometime between 2006 and 2010 via the Assertive Adolescent and Family Treatment (AAFT) initiative. SAMHSA also offered additional funding opportunities during this period, including: the Juvenile Drug Court, Juvenile Drug Treatment Court, Offender Reentry Project, and Targeted Capacity Expansion initiatives, where organizations could select an EBT. Many of those organizations opted to deliver A-CRA (n = 22) and thus were also included in our study. Overall, 15 organizations received more than one grant during the study period, such that a total of 82 unique organizations received at least one grant that had ended at the time of our study and thus were targeted for inclusion in the sample.

Organizations were located across 27 U.S. states or territories. To receive funding, organizations had to demonstrate operation in the same geographical location(s) for at least two years prior to the proposed project period, and demonstrate compliance with local, state/tribal licensing and certification requirements. Given these requirements, the organizations that were funded had a history of substance use treatment provision and the projects were not demonstrations. Agencies used grant funds to cover expenses such as personnel salary and benefits, materials, and infrastructure support. In addition to the funds provided directly to agencies, CSAT also funded the implementation support (training and coaching) for A-CRA provided through Chestnut Health Systems. The implementation support provided to grantees is described in greater detail in Godley et al. (2011).

2.2. Recruitment

In collaboration with Chestnut Health Systems, we identified key stakeholders at each treatment organization that would be best acquainted with the implementation of A-CRA. These stakeholders were largely counselors and clinical supervisors who had previously received training or certification from Chestnut Health Systems to deliver the treatment and/or to serve as the point of contact for the grant.

Potential study participants were contacted via e-mail with a description of the purpose of the study and the details of participation, and subsequently by telephone to request and/or arrange for an interview. Following the interview, respondents were asked to complete an online demographic survey. The study team used an administrative dataset that contained the contact information of staff at the organizations who were grant-supported during the implementation period. Participants were notified that they would be compensated for their time with $50.

2.3. Measures and data collection

2.3.1. Administrative data.

Information about the CSAT grants (e.g., amounts, start and end dates, points of contact) were retrieved from records kept by the implementation support team (Chestnut Health Systems).

2.3.2. Demographic data.

We collected demographic information about the respondents (i.e., age, gender, race/ethnicity, educational background) from each participating organization using an online survey that was completed after the interviews.

2.3.3. Qualitative data.

The primary data collection component consisted of telephonically conducted semistructured interviews with the aforementioned staff members at previously funded organizations. We collected the data over three time periods, approximately nine months apart starting in fall 2013 and ending in spring 2015. We attempted to contact staff from each site at each wave per our data collection protocol. However, if staff from a site reported that they were no longer sustaining A-CRA, we did not complete any subsequent interviews with that site (note that no site began using A-CRA again once they had discontinued it). We utilized data only from a respondent’s first interview so that all datapoints were unique. The interviews lasted for approximately 60 minutes, were conducted by trained field staff, and were audio recorded.

The study team developed a semistructured interview guide based on existing literature regarding program sustainability and implementation (e.g., Fixsen, Naoom, Blasé, Friedman, & Wallace, 2005; Scheirer et al., 2008; Simpson, 2002). The topics identified for inclusion in the guide were highly consistent with the CFIR, which was published soon after we conceptualized our study. Sustainment was assessed using a self-reported binary variable whereby the interviewee specified whether their organization had continued to offer A-CRA since completion of the implementation funding support phase. After interviewers confirmed the sustainment status of the organization, respondents were asked about their perceptions of the A-CRA treatment (e.g., “What do you think of A-CRA as a treatment for youth?”), their experiences with the grant (e.g., If you had a chance to participate in a SAMHSA CSAT project again, would you consider it? Why or why not?), and about the factors that facilitated or impeded in sustaining the A-CRA treatment model (e.g., “What helped your efforts to sustain your project after the CSAT funding ended?” and “What barriers have you encountered in attempting to sustain the project activities or services after the end of its CSAT funding?”). Data were also gathered on characteristics of respondents and their organizations; for the current study, we used an item that asked about the focus of services at their organization, which we transformed into a yes/no indicator of whether an organization focused on SUD treatment.

Trained interviewers administered a standard interview protocol across both SOs and non-SOs while allowing participants to elaborate on responses and offer additional details. Probes were employed to elicit responses to specific items that were not voluntarily offered by participants to ensure that responses were exhausted (e.g., “Is there anything else you think would have helped to sustain the use of A-CRA at your agency?”). The interview protocols for SOs and non-SOs differed only insofar as questions on implementation were asked in past tense for non-SOs. Following each interview, the interviewers developed field notes—a well-established qualitative research technique (Bernard, 2006; Lincoln & Guba, 1985; Sanjek, 1990). Field notes are detailed summaries of the interview that document themes based on interview questions that are targeted to generate insights about specific topics. Field notes best met the purposes of our study and were less time consuming and expensive than complete transcription. Any quotes that are used throughout this manuscript were confirmed for accuracy by reviewing the original audio recording. Additional details on the interview protocol can be found in Hunter et al. (2014).

2.4. Data analysis

2.4.1. Administrative and demographic data.

Information from the administrative dataset (i.e., describing organizational characteristics) and from the online survey data (i.e., describing respondent characteristics) were cleaned and descriptive statistics were performed using SAS/STAT (2012) to help describe the study sample.

2.4.2. Qualitative data.

Qualitative data from the semistructured interviews were imported into Dedoose, a software program designed for qualitative data analyses (SocioCultural Research Consultants, 2016). Thematic analysis was performed through coding of interview notes. Codes were determined both inductively through a repeated team-based approach and deductively by applying a priori codes derived from the CFIR (Fereday & Muir-Cochrane, 2006; Ryan & Bernard, 2003; Tuckett, 2005). This enabled the building of a summative and interpretive structure out of complex, semistructured data.

Two research assistants (Felician and Pham) co-coded the interviews under the guidance of the Principal Investigator (Hunter), who met initially with the research assistants to describe the purpose of the study and then weekly to discuss themes and any discrepancies or questions regarding interpretation. Themes were generated and organized in the form of a hierarchical code tree (Kairuz, Crump, & O'Brien, 2007) that organized individual codes within four key areas of the interview protocol: 1) perceptions of A-CRA, 2) grant experiences, and 3) factors that either facilitated sustainment of A-CRA or 4) impeded sustainment of A-CRA. After inductively coding the responses to the facilitator and barrier items, we then organized the codes and associated data using the CFIR codebook template (available from at the CFIR Guide website; [Consolidated Framework for Implementation Research, 2020]) into characteristics of individual, intervention, inner- and outer-setting factors. We did not use the process domain of the CFIR because the implementation process during the federal-funded grant period was the same across all sites. While the CFIR codebook constructs aligned well with most of the themes identified through the inductive process, we also expanded by coding subthemes under one CFIR construct (complexity) and added one construct that was not found in the CFIR (workforce retention as an inner-setting factor).

At the end of the coding process, 42 excerpts from the dataset (covering the range of the code-tree) were selected by another researcher for an inter-rater reliability test by the two coders. Observed kappas from these coded excerpts ranged from 0.67 to 0.76, demonstrating moderate to substantial agreement levels (Landis & Koch, 1977).

Finally, we took a mixed-method approach to our analysis by integrating quantitative analyses into our primarily qualitative approach. The quantitative data came from two sources: (1) the administrative data and (2) the qualitative codes themselves, which we transformed into binary variables indicating the presence or absence of a theme by respondents at each type of organization (SOs and non-SOs). In other words, in cases where we had more than one respondent per organization, we aggregated data across respondents so that themes were explored at the site level (respondents from SOs, M = 1.89; respondents from non-SOs, M= 1.36). We then conducted chi-square and Fisher’s exact tests to evaluate differences in staff responses from SOs and non-SOs regarding organizational characteristics, perceptions of the treatment, grant experiences, and facilitators and barriers to sustaining A-CRA. We did not have any instances in which staff from same organization reported different sustainment status. Under the taxonomy from Palinkas and colleagues (2011), this was a QUAL + Quant (simultaneous, primarily qualitative) approach that provided a higher level of understanding of our thematic findings by transforming them into a format that allowed for additional quantitative analysis.

3. Results

3.1. Participating organizations and respondents.

Staff from a total of 78 of the 82 funded organizations agreed to participate in an interview, yielding a 93% response rate at the site level. Of those, staff from 53 organizations (68%) reported sustaining A-CRA at the time of the interview (i.e., SOs), and staff from 25 organizations (32%) reported no longer delivering A-CRA (i.e., non-SOs). We interviewed a total of 134 staff, of whom 41% classified themselves as counselors, 33% as supervisors, and 26% as both. Demographic characteristics of respondents from SOs and non-SOs are presented in Table 1. Although there appeared to be significant differences in age, gender, and race among respondents from the different types of organizations, there was also some missingness as not all respondents completed the online survey after the qualitative interview, so those differences should be interpreted with caution.

Table 1.

Demographic characteristics of respondents from sustaining (SOs, n = 53) and non-sustaining organizations (non-SOs, n =25).

Characteristics	Sustaining Organization Respondents (n=100; mean 1.89 per site)	Non- Sustaining Organization Respondents (n=34; mean 1.36 per site)	(p-value)
Age (Mean/SD)	39 (11.21)	46 (12.35)	0.015
Not Reported (%)	15	29
Gender (%)
Female	66	44	0.046
Not Reported	10	23
Ethnicity-Hispanic (%)	29	27	0.147
Race (%)
White	72	26	<0.001
Other	18	44
Not Reported	10	29
Education (%)
Some College, AA or Bachelor’s Degree	25	24	0.130
Graduate Degree (Master’s or Doctoral)	65	53
Not Reported	10	23

General characteristics of the organizations, stratified by SO and non-SO status, are included in Table 2. A larger proportion of non-SOs (56%) compared to SOs (8%) were funded by the earlier AAFT grant mechanisms in 2006 and 2007 as compared to the later cohorts funded in 2009 and 2010. Also, organizations that sustained A-CRA were more likely to have received multiple CSAT grants (38%) compared to those that did not continue A-CRA (4%). We did not find that organizations that received funding through one of the non-AAFT initiatives were more or less likely to sustain A-CRA. Staff from organizations that mainly focused on substance use (as compared to mental health, general health, or some combination of issues) were also more likely to report sustaining A-CRA (66%) compared to organizations without a primary substance use treatment focus (24%).

Table 2.

Percentage of sustaining and non-sustaining organizations by organizational characteristics.

Variable	% Sustaining Organizations (n=53)	% Non-Sustaining Organizations (n=25)	χ2 Fisher's Exact Test (p-value)
Funding Mechanism(s)
AAFT 1 or 2 (2006, 2007)	8	56	.000
AAFT 3 or 4 (2009, 2010)	38	16	.032
More than 1 AAFT grant or AAFT grant + other grant	38	4	.001
Other grant only (JDC, JDTC, ORP, or TCE)	17	24	.542
Organizational Focus on Substance Use	66	24	.001

Notes: AAFT = Assertive Adolescent and Family Treatment; JDC = Juvenile Drug Court; JDTC = Juvenile Drug Treatment Court; ORP = Offender Reentry Project; TCE = Targeted Capacity Expansion.

3.2. Perceptions of A-CRA.

Next, we examined perceptions of A-CRA by the respondents from the different types of organizations (i.e., SOs and non-SOs). Staff from both types of organizations expressed enthusiasm for the A-CRA treatment approach; however, staff from SOs were significantly more likely to report positive perceptions (staff from 98% of SOs compared to 80% of staff from non-SOs, respectively; χ²₁ = .012). For example, one participant found the A-CRA emphasis on cognitive behavior change to be a positive aspect of the treatment intervention:

“I think it's a really great form of therapy because it really puts a lot of emphasis on the adolescent making positive changes for themselves so it's a cognitive behavioral type of approach where they have freedom. We're non-judgmental and meet them where they are so it takes the pressure off. We really get the family involved and try to meet their needs and we use the community so it's like a village.” (Staff from SO)

Another participant highlighted the individual nature of A-CRA:

“I think it was great. I think it allowed for a lot of one-on-one time. Actually it's been one of the best programs I've worked with when it comes to youth, and the reason for that is because I felt that there was a lot of one-on-one time with the individual specifically, and it allowed us to identify their strengths and weaknesses and try to work with that specifically.” (Staff from non-SO)

Additionally, participants noted their satisfaction with the ease of implementation of the model and caregiver involvement:

“I liked it! I think it's useful and very simple to understand for the clinician and the clients. Not too complicated to follow. It gave me the opportunity to connect with the parents, the youth, and also planning their goals into something long term for example the job skills procedures that they had. That was very helpful to plan for future goals after they were done with treatment.” (staff from non-SO)

3.3. Perceptions of the grant funding mechanism(s)

While participants were overall enthusiastic about A-CRA, responses varied in terms of experiences and interactions with the grant funding mechanism. Intuitively, staff from SOs were significantly more likely to express positive experiences (64% from SOs, or respondents from 34 out of 53 sustaining sites) than staff from non-SOs (28% of non-SOs, or staff from 7 out of 25 non-sustaining sites; χ²₁ = .004). For example, one participant from a sustaining site noted:

“I liked the network of people around the country. I'm an accountability person, so if I have consultation calls then I have to upload things, I am so much more likely to do it, so I really liked that. I like the energy of A-CRA. I liked the trainers a lot, and the support”. (staff from SO)

Although many participants found the support provided by the grant to be of good quality, but for some staff and organizations, it was perceived as overwhelming; for example:

“I think that SAMHSA provided a lot of good training with the agencies involved, with A-CRA, GAIN [Global Assessment of Individual Needs], and all of the tools. It was a lot of information which, at first it was like, ‘Oh my gosh, I'm never doing this again,’ but there was a lot of support with why the treatment works and why what we were trained on was important, to do it correctly. It helped us get through what we needed to do, but because of the amount of training you had to go through to get to it, the three years really went by too quick. By the time we actually had staff appropriately trained and good in understanding it was like the grant was all over.” (staff from non-SO)

Even sites that sustained A-CRA highlighted the challenges in sustaining the level of support for quality delivery after the grant ended:

“I think that it's really high quality good stuff, and if you are going to implement an EBT into fidelity that's what you need, and I think they do an excellent job at it. I really question whether that is sustainable without grant money. I think the tradeoff then is if you cut back on that and do your own monitoring and your own reviewing of tapes, do it through peers as opposed to through an outside-trained person, do you lose quality?” (staff from SO)

3.4. Factors associated with sustainment

Next, we examined staff perceptions of factors that helped (i.e., facilitators) or hindered (i.e., barriers) the continuation of A-CRA delivery after the federal funding had ended. The degree to which specific issues were raised by staff from SOs and non-SOs is estimated as the percentage of SO and non-SO sites that identified a particular factor as a facilitator or barrier to sustaining A-CRA. These are organized by the four CFIR constructs (characteristics of individuals, intervention characteristics, inner and outer setting factors) in Tables 3 and 4, respectively. We provide further descriptive details of the context of the respondent statements and illustrative quotes in the following sections.

Table 3.

Percentage of organizations by sustainment status reporting the following domains as facilitators to A-CRA delivery continuation.

Facilitators	% Sustaining Organizations (n=53)	% Non- Sustaining Organizations (n=25)	χ2 Fisher's Exact Test (p-value)
Individual characteristics
Staff knowledge and attitudes	83	56	.024
Innovation characteristics
Adaptability	09	04	.658
Compatibility	38	16	.068
Design quality and package	58	24	.007
Evidence strength and quality	36	20	.195
Inner setting
Available resources	70	16	<.001
Leadership engagement	74	28	<.001
Readiness	58	44	.331
Outer setting
Fit with client population	38	28	.453

Table 4.

Percentage of organizations by sustainment status reporting the following domains as barriers to A-CRA delivery continuation.

Barriers	% Sustaining Organizations (n=53)	% Non- Sustaining Organizations (n=25)	χ2 Fisher's Exact Test (p-value)
Individual characteristics
Staff knowledge and attitudes about A-CRA	17	04	.155
Innovation characteristics
Compatibility	11	28	.101
Complexity	49	72	.086
Assessment tool	06	36	.001
Certification process	06	24	.027
Delivery of offsite services	26	16	.395
Recording protocol	11	24	.184
Reporting/paperwork requirements	13	20	.031
Training or technical support issues	04	20	.507
Inner setting
Available resources	32	16	.176
Leadership engagement	13	24	.329
Readiness	06	12	.379
Workforce turnover	30	68	.003
Outer setting
External policy and incentives	51	80	.025
Fit with client population	13	28	.126

3.4.1. Facilitators of sustainment

3.4.1.1. Characteristics of individuals.

We found that clinical knowledge and beliefs about A-CRA were often stated in response to our query about factors that helped with A-CRA sustainment. More specifically, staff from 83% of the SOs (i.e., 44 of the 53 SOs) and staff from 56% of non-SOs (i.e., 14 of 25 non-SOs) stated that having staff who supported the model was a key facilitator for continuing A-CRA at their site. For example:

“Every staff person that we had using the model over the years was a strong supporter of the model. They were believers.” (staff from SO)

3.4.1.2. Intervention characteristics.

Consistent with the generally positive perceptions of A-CRA, clinical staff identified various aspects of the intervention that they perceived facilitated sustainment, including its design quality and packaging. More specifically, staff from 58% of SOs (i.e., 31 out of 53 sites) and 24% of non-SOs (i.e., 6 out of 25 sites) mentioned A-CRA training and technical support as a facilitator to sustaining A-CRA (p = 0.007). For example, a respondent noted:

“The fact that the training and support on the model was so excellent. Right now, we have two certified supervisors that do the training here when a new employee comes in.” (staff from SO)

Staff from both types of sites also mentioned that the strong evidence base for A-CRA was important for sustaining it (i.e., 19 out of 53 SOs and 5 out of 25 non-SOs; p > 0.05). For example, a respondent mentioned:

“The effectiveness of it and that it was evidence-based. We were able to present that to jobs and family services and the courts and get those contracts.” (staff from SO)

Respondents from 38% of SOs (i.e., 20 of the 53 organizations) and respondents from 16% of the non-SOs (i.e., 4 of the 25 non-SOs; p > 0.05) also mentioned that the compatibility of the A-CRA treatment approach with their organization facilitated sustainment. For example, a respondent reported:

“The way the program description is, it is built completely around A-CRA so we definitely don’t need to work against the machine to incorporate it.” (staff from SO)

3.4.1.3. Inner setting.

Overall, respondents from SOs reported that more organizational characteristics facilitated A-CRA sustainment than respondents from non-SOs. Internal resources to support A-CRA delivery, such as staff time allocated to provide clinical supervision, was more likely to be mentioned by staff from SOs compared to non-SOs (i.e., staff from 37 of 53 SOs and staff from 4 of 25 non-SOs; p < 0.001). Leadership engagement was also viewed as especially key to A-CRA sustainment among staff from SOs (i.e., mentioned by staff from 39 of 53 SOs) relative to staff from non-SOs (7 out of 25 non-SOs; p < 0.001). The sentiments from staff at SOs included general aspects of support, “[We] had lot of support from upper management; they were trouble shooting how to sustain the model even before funding ended”; and specific forms of support, “The leaders made sure that staff was well equipped and trained. Another commonly cited inner-setting factor identified as a facilitator was readiness.

Respondents from around half of SO and non-SO sites identified readiness as a facilitator (i.e., staff from 31 of the 53 SOs and staff from 11 of 25 non-SOs; p > 0.05). Readiness was characterized as planning for long-term delivery, such as proactively taking advantage of the training supports available during the federal funding period and looking for alternative financial supports once the federal funding ended:

“We did get all of our staff certified to the highest degree possible, we tried to take advantage of that while the grant was paying for it. We talked to insurance companies about covering A-CRA, we talked to them about it and they were in favor of it, but it never really materialized into dollar figures we could utilize” (staff from non-SO)

3.4.1.4. Outer setting.

Staff from about one-third of SOs and non-SOs also mentioned that A-CRA was a good fit for the clientele that they served, which helped facilitate sustainment (i.e., 20 of 53 SOs and 7 of 25 non-SOs; p > 0.05). For example:

“It really related to the kids. It took where kids are at and their cognitive abilities, and it was a perfect fit for them.” (staff from non-SO)

3.4.2. Barriers to sustainment

3.4.2.1. Characteristics of individuals.

In general, only a small proportion of SO staff (17%) and non-SO staff (4%) reported staff knowledge and attitudes about A-CRA as a barrier to sustainment. The difference between SOs and non-SOs was not statistically significant (p > 0.05). For example:

“A-CRA was very top heavy. . . I wasn't quite sure in the long run whether that was a good use of time for a clinician.” (staff from SO)

3.4.2.2. Intervention characteristics.

Clinical staff from both SOs and non-SOs identified various characteristics of the intervention as barriers to continued implementation. The intervention-specific barriers mentioned had to do with its complexity, which we coded under the specific subthemes: the assessment tool, certification process, offsite service delivery, recording protocol for clinical supervision, reporting/paperwork requirements, and training and technical support issues. As shown in the Table 4, significant differences were found between respondents from the two types of organizations for the following subthemes: assessment tool, certification process, and reporting/paperwork requirements. Staff from non-SOs were more likely to mention that the certification process was “overwhelming”, “hard” or “huge” than staff from SOs. Regarding the assessment tool, one respondent reported:

“It's (GAIN) too long and it doesn't create a good atmosphere for the youth coming into the program.” (staff from non-SO)

We found that staff from non-SOs were more likely to identify reporting/paperwork as a barrier, noting, for example, that “it was a lot of work with the data entry and data collection”. We should note that the assessment tool and reporting/paperwork requirements mentioned by respondents were related to the federal grant requirements and were not specific components of the A-CRA treatment model. However, respondents likely considered these as part of A-CRA because CSAT required grantees to use the GAIN and Chestnut Health Systems provided training and support for both A-CRA and GAIN. It may help to explain why some organizations discontinued A-CRA—they may have misunderstood what was required to deliver A-CRA. Regarding the clinical supervision recording protocol, one respondent highlighted:

“The interns had a hard time doing the digital session recordings which would delay my time in reviewing the fidelity to the model. This was time consuming and it is hard to collect the data. If my sole job was just to do the A-CRA responsibilities to review the DSRs [digital session recordings] and counsel the clinicians then it would have been possible. As the supervisor I did not have this time.” (staff from non-SO)

3.4.2.3. Inner setting.

Clinical staff identified various factors related to the organization as barriers. These included lack of resources to support A-CRA, lack of readiness/planning for sustainment, and lack of leadership engagement. These themes were not commonly reported or different between the two types of organizations. For example, while clinicians from SOs often cited lack of available resources to support A-CRA, they often demonstrated that they found internal solutions to the problem; for example,

“For a while we were using the goals of counseling until we were told we have to use these other individual treatment plan forms that the agency uses…we used to butt heads, but now we just do both forms now. It's just added work, not a big deal.”

Workforce retention, on the other hand, was more often noted as a barrier to A-CRA sustainment by respondents from non-SOs (i.e., staff from 17 out of 25 sites) than respondents from SOs (i.e., staff from 16 out of 53 sites). Respondents reported that due to staff attrition, they no longer had employed staff that were certified to deliver the A-CRA and did not have the funding to support training them in the model or hiring staff who were already certified; for example,

“It just seems like no matter what we did and got in place, our staff would leave and we'd retrain new staff and those staff would leave and the training for A-CRA was pretty intense and long, and we had some people that weren't completely trained and they were gone, and we had to start all over with new people.” (staff from non-SO)

3.4.2.4. Outer setting.

A common concern of both types of organizations was finding external funding, however staff from non-SOs (i.e., 20 out of 25 sites) were more likely to mention it as a key barrier to sustainment compared to staff from SOs (i.e., 27 out of 53 sites; p < 0.025). For example, one respondent from non-SO mentioned:

“We were working with our state to get some Medicaid services, coverage for the services, and at the time the grant ended we did not have that.”

Both types of organizations also mentioned fit with the client population (7 of 53 SOs and 7 of 25 non-SOs, p > 0.05). A respondent from a non-SO noted:

“I do think there is this subset of kids that we get referred who are really deeply embedded in the legal system and it's really hard to make A-CRA match up to the legal system that's in place at times. For example, it is a harm reduction vs, an abstinence model, the court doesn't care if the levels go down, they just care if it was positive or negative. Things like that. So sometimes that comes up where the two don't really interface very well. I think that's the only real drawback at this point.”

4. Discussion

The current study explored the organizational and clinical characteristics associated with the sustainment of an EBT for adolescent SUD (i.e., A-CRA), after three years of federal grant support ended. We found organizations that had received multiple discretionary federal grants to support A-CRA as well as organizations whose primary focus was substance use, rather than other health conditions (mental health or general health care) were more likely to report sustaining A-CRA. We also collected clinical perspectives about sustaining A-CRA, including perceptions about the EBT and federal funding grant mechanism, along with factors that clinical staff reported helped or hindered its delivery following the grant’s end. Based on these data, we found that adolescent SUD treatment program staff generally had positive perceptions of the A-CRA, regardless of sustainment status, but other factors hindered an organization’s ability to continue delivering the treatment after initial support had ended.

More specifically, staff from SOs reported more organizational (i.e., inner setting) support to continue A-CRA delivery. For example, when asked what facilitated A-CRA delivery, staff from SOs were more likely to report available resources within their organization and leadership support to deliver A-CRA than staff from non-SOs. While staff from both SOs and non-SOs perceived A-CRA favorably, staff from SOs were more likely to mention the positive staff perceptions of A-CRA as a factor that helped to sustain it. Also, staff from SOs were more likely to mention characteristics of the intervention itself, such as the training and certification infrastructure, as well as the positive therapeutic approach of the A-CRA model as factors that led to continuation at their site. Staff from both SOs and non-SOs mentioned that the A-CRA was a good fit for their clientele and respondents from both types of organizations were just as likely to report that sustainability planning helped to facilitate A-CRA at their site. These findings suggest that inner-setting characteristics are often critical for sustainment EBT delivery.

Staff from non-SOs generally felt the A-CRA model was more burdensome than staff from SOs and noted a number of other challenges beyond the treatment model, such as the assessments required by SAMHSA (e.g., GAIN), administrative paperwork, staff turnover, and, most notably, the lack of external funding to support continued delivery after the grant ended. It is important to note that although the assessment requirements and reporting/administrative paperwork were not specific to A-CRA but were part of the federal grant mechanism, staff from many of the non-SOs seemed to conflate the two initiatives. On the other hand, staff from SOs were less likely to mention the grant-specific issues. Both sites also mentioned outer-setting factors but these seemed to be the main barrier to A-CRA continuation at the non-SOs rather than SOs. We did find that when asked, staff from SOs described a number of barriers to continuing A-CRA, but oftentimes they were able to overcome the barriers, perhaps due to more inner-setting support, for example, from leadership. These findings are consistent with those observed in mixed-method studies of EBT sustainment in youth mental health services (Brookman-Frazee et al., 2018; Jaramillo et al., 2019; Rodriguez et al., 2018; Scudder et al., 2017), but the in-depth qualitative data provided by those studies, and ours, illustrate how similar factors play out within specific organizational and service contexts. They also are consistent with studies of the implementation of EBT for adolescent substance use more broadly that report that quality assurance mechanisms, like clinical supervision, are often reported as intransigent barriers to EBT delivery in real world settings (Hogue et al., 2018). In general, these findings are also consistent with implementation frameworks that suggest that implementation and sustainment are related to provider perceptions of the intervention, characteristics of the individuals charged with delivering it, and the inner and outer settings in which the intervention is being delivered (e.g., see Aarons et al., 2011; Damschroder et al., 2009).

Even though many providers cited that the intervention was effective with clients and that they enjoyed using it, their organizations were still not able to sustain A-CRA due to other factors, including leadership support and external funding—this key finding has ramifications for policy and practice. While positive perceptions about an intervention, including perceived fit for clients; effectiveness of the approach; and provider support for the intervention have been theorized to support implementation, these factors were not enough for A-CRA to be sustained in many organizations. Other factors, such as external funding for the intervention and organizational factors such as leadership support may be critical to sustain and EBT in substance use treatment settings.

These findings highlight the complex interplay between the numerous factors that support EBT delivery in routine substance use treatment settings. Even with clinical support for a particular model and an implementation approach designed to equip supervisors to train and certify clinicians within their own organization, factors such as financial support to cover the training and supervision costs are needed to maintain the intervention. These findings suggest that to sustain an EBT, comprehensive support, especially for program leadership and external factors, such as reimbursement for clinical supervision, is required. Rare qualitative analyses of sustainment in other substance use treatments, which have focused on adult treatment services (e.g., Ford et al.’s (2015) study of mobile applications; Stumbo, Ford, & Green’s [2017] examination of quality improvement practices), also had similar, albeit context- and intervention-specific findings.

4.1. Limitations

One limitation of our approach was that we used a self-reported, dichotomous (yes/no) measure of A-CRA sustainment. Although this is not uncommon (Stirman et al., 2012), we later constructed a more objective measure of A-CRA sustainment that is a composite of 10 elements (e.g., A-CRA certification, usage, training, etc.; Huang et al., 2017), which validated our self-reported dichotomous measure as the two were largely consistent (>90% congruence). Additionally, the interviews summarized by interviewers and the interviewers’ notes, rather than transcripts, were used in coding and analysis. We retained digital audio-files of all the interviews to help with the provision of illustrative examples of the themes generated in the qualitative analyses. Finally, we used the CFIR to guide our analyses of staff responses of general barriers and facilitators of active implementation and sustainment, whereas some newer frameworks distinguish specific sustainment-related factors (e.g., Exploration, Practice, Implementation and Sustainment [EPIS] framework by Aarons et al., 2011; Dynamic Sustainability Framework by Chambers, Glasgow, & Stange, 2013; Integrated Sustainability Framework by Shelton, Cooper, & Wiltsey Stirman, 2018). It will be useful for future qualitative research on sustainment to use these frameworks for guidance as well as to compare the utility of those findings versus those using general (e.g., CFIR) frameworks. We also did not examine responses from sites longitudinally, as we had a limited sample to do so (i.e., staff from 16 sites reported sustaining A-CRA at initial wave(s) and discontinuing it at following waves), and it would have changed the analyses, giving them a different purpose and scope. Future studies may want to examine respondent perceptions over time. Our findings regarding the demographic characteristics of respondents suggest that staff from SOs may have been younger, female and white. While the amount of missingness for these items suggests that the age and gender effect may be spurious, the race finding cannot be explained entirely by the nonresponse rate. We are not aware of any studies that have found that EBT sustainment is related to the race of practitioners, but it may be a proxy for the organization or environment in which the provider is based. More research is needed to determine if provider characteristics are related to EBT sustainment.

5. Conclusions

Understanding the factors that influence the sustainment of EBTs, such as A-CRA, could lead to more effective dissemination strategies and improvements in the quality of care that is available in community-based settings. This is particularly important in the case of adolescent substance use treatment since there has been a tendency to rely heavily on adult models and peer-based practices that are not backed by strong theoretical or empirical foundations. Our findings suggest that implementation strategies should emphasize staff training, leadership engagement, and efforts to reduce the perceptions of treatment complexity to help improve EBT sustainment. By improving access to high quality care for youth substance use treatment, successful and sustained implementation of A-CRA and similar EBTs can mitigate the adverse consequences of substance use, including short- and long-term violence, accidents, disease, and criminal behavior (Merline, Jager, & Schulenberg, 2008; Substance Abuse and Mental Health Services Administration, 2017) and lead to improved public health outcomes.

Highlights.

• We interviewed 134 clinicians trained in an evidence-based treatment model

• Both sustaining and non-sustaining organizations had positive views of the model

• Contextual factors within and outside organizations were related to sustainment

• Sustained delivery of effective substance use treatments is challenging, but possible

Abbrevations:

A-CRA

Adolescent Community Reinforcement Approach

AAFT

Assertive Adolescent and Family Treatment

CSAT

Center for Substance Abuse Treatment

CFIR

Consolidated Framework for Implementation Research

EPIS

Exploration, Practice, Implementation and Sustainment

EBT

evidence-based treatment

GAIN

Global Appraisal of Individual Needs

SAMHSA

Substance Abuse and Mental Health Services Administration

SUD

substance use disorder

SO

sustaining organization