Sustainment of Integrated Care in Addiction Treatment Settings: Primary Outcomes from a Cluster Randomized Controlled Trial

Abstract

Objective:

Integrated treatment services are the gold standard for addressing co-occurring mental health and substance use disorders, yet they are not readily available. The Network for the Improvement of Addiction Treatment (NIATx) was hypothesized to be an effective implementation strategy to implement and sustain integrated mental health and substance use care in addiction treatment programs. We report here on the sustainment of integrated services for up to two years post-implementation support.

Methods:

The effectiveness of NIATx strategies to implement and sustain integrated services was evaluated using a cluster-randomized waitlist control group design. Forty-nine organizations were randomized to either NIATx1 (active implementation strategy) or NIATx2 (waitlist control). The Dual Diagnosis Capability in Addiction Treatment (DDCAT) Index was used to evaluate organizations' capability for integrated care. The NIATx Stages of Implementation Completion scale was used to assess participation and adherence to the NIATx implementation process. Mixed-effects modeling was used to evaluate changes from baseline to end of the sustainment period.

Results:

Both cohorts sustained their capability to provide integrated treatment services. Both groups achieved successful implementation and sustained integrated services to a similar degree regardless of sustainment year. Sustainment did not vary as a function of NIATx adherence.

Conclusions:

The delivery of integrated treatment services was sustained for two years post-active implementation support. Future research should consider how contextual factors may predict, mediate, and moderate sustainment outcomes.

Implementing evidence-based practices is an investment requiring time, effort, and expenditures. These investments are worthwhile insomuch as the implemented evidence-based practices (EBP) are sustained. Sustainment, or the state of continued evidence-based intervention, service, or program delivery, is the final stage of the implementation process and is ongoing.^1-3 It is a stage during which most organizations fail to continue to deliver an EBP as originally planned or intended.^4,5 One could therefore argue that the true measure of a successful implementation process and effectiveness of any implementation endeavor lies in the sustainment phase. Maintenance or sustainment of outcomes of interest can serve to demonstrate and validate the value of investments and expenditures, ultimately translating to better care for patients. In this report, we present the sustainment outcomes of a study designed to evaluate the effectiveness of the Network for the Improvement of Addiction Treatment (NIATx) – a multifaceted implementation strategy – to implement integrated mental health and addiction treatment services for persons with co-occurring substance use and mental health disorders.^6-8

In 2018, in the United States, an estimated 9.5 million adults suffered from co-occurring disorders.⁹ This is a significant public health concern given the associated societal burden.¹⁰ Often under- or misdiagnosed due to their inherent complexity, co-occurring disorders are also often not treated effectively. ¹¹ Integrated care, where treatment for mental health and substance use problems is delivered conjointly, at the same time, and by the same provider(s), is the most effective form of treatment, yet it is rarely provided.^12-18 Data from the 2018 National Survey on Drug Use and Health confirms that less than 8% of individuals with co-occurring disorders reported receiving integrated services.⁹

Although persistent and significant efforts have been made to integrate substance use and mental health treatment services, barriers to delivery persist. Deterrents such as financial cost, administrative obstacles, or provider resistance obstructing integrated care delivery are situated at multiple contextual levels (system, organizational, individual).^19-21 It is therefore imperative to utilize a multilevel, multifaceted approach, such as the Network for the Improvement of Addiction Treatment, that targets barriers at every level to enable and sustain delivery of integrated care. Additionally, because NIATx is effective for making practice change in behavioral health settings, promotes sustainment of change, and combines process improvement tools and techniques with quality improvement interventions, we hypothesized it to be potentially effective to install and sustain integrated services.^6,22-29

Outcomes from our longitudinal study confirmed NIATx as an effective implementation approach to conjointly deliver mental health and substance use treatment.^7,8 Previous results indicate that the majority of participating agencies, regardless of study arm, successfully transitioned from offering addiction-only treatment services at baseline to providing integrated services by the end of the implementation period. ^7,8 Furthermore, we demonstrated that organizations' capability to provide integrated treatment services varied by level of NIATx adherence.^7,8

Here, we present sustainment outcomes associated with the implementation of integrated care for co-occurring mental health and substance use disorders using NIATx and investigate their durability. Sustainment was considered when gains in capability for integrated services were maintained up to two years beyond the active implementation phase.

Methods

Design

This cluster-randomized waitlist control group trial was designed to evaluate the effectiveness of NIATx strategies for implementing and sustaining integrated services for co-occurring disorders. Following baseline assessments, organizations enrolled in the study were randomized to one of two cohorts: NIATx1 (n=25) or NIATx2 (n=24). NIATx1 received active NIATx for the first 12 months (Year 1), followed by 24 months of sustainment (Years 2 and 3). NIATx2 organizations were delayed in starting NIATx for 12 months (Year 1), followed by active NIATx for 12 months (Year 2), and sustainment for 12 months (Year 3) (online supplement). In this manuscript, we report on results from the sustainment period. Given the study design, the length of the sustainment period varied between groups (2 years for NIATx1 and 1 year for NIATx2) therefore providing the opportunity to examine whether gains in integrated service capability are differentially durable. The study protocol has been described in-depth previously by Ford et al. and Assefa et al.^6,7

Participants

To participate, organizations had to: be state-licensed to provide addiction treatment services at outpatient, intensive outpatient, or residential levels of care; be tax-exempt; have government status or meet a 50% minimum of public funding; and have no previous NIATx training or participation in a NIATx project. Participating organizations were recruited via letter by state behavioral health authorities. Forty-nine out of the initially enrolled 53 (92%) community-based addiction treatment programs from the State of Washington were randomized at baseline to either the NIATx1 (n=25) or NIATx2 (n=24) study arms.

Measures

Dual Diagnosis Capability in Addiction Treatment Index (DDCAT).

The Dual Diagnosis Capability in Addiction Treatment (DDCAT) Index is a quantitative organizational measure used to evaluate the current capability of addiction treatment programs to deliver integrated treatment services for persons with co-occurring disorders.³⁰ It comprises 35 items across seven dimensions, including Program Structure; Program Milieu; Clinical Process: Assessment; Clinical Process: Treatment; Continuity of Care; Staffing; and Training. The 35 items are rated on a five-point Likert scale, which includes anchor scores of 1 – Addiction Only Services (AOS), 3 – Dual Diagnosis Capable (DDC), and 5 – Dual Diagnosis Enhanced (DDE). Ratings of items within a dimension are averaged to generate a subscale score. A DDCAT Total score is derived by averaging the 35 items.

The DDCAT also produces a categorization of the organization's integrated service capacity using the American Society of Addiction Medicine taxonomy.^31,32 If less than 80% of scores are rated a three or higher an organization is categorized as AOS. If at least 80% of scores are 3 or higher, an organization is categorized as DDC, which is the adequate and acceptable standard for all specialty addiction treatment organizations.^31,33 If at least 80% of scores are a 5, then an organization is categorized as DDE. The reliability and validity of the DDCAT have been demonstrated in numerous psychometric studies, and public and private health care systems have adopted it as a measure of behavioral health service integration.^30,33-36 The DDCAT Index (Version 4.1) and Toolkit (Version 4.0) are in the public domain, free and available from the authors.³²

Evaluators, blind to the study arm, were rigorously trained in data collection, specifically pertaining to the DDCAT for this study. Evaluators conducted site visits in pairs to the addiction treatment organizations every 12 months for data collection. Data were collected from key informant interviews with staff and patients, site observations, and chart and/or program manual review over a 2 to 3 hour period. The information collected was then synthesized and summarized across all data sources by the evaluators to complete the DDCAT Index measure.

NIATx Stages of Implementation Completion (NIATx-SIC).

The Stages of Implementation Completion (SIC) was adapted to track each organization’s participation and adherence to the NIATx implementation process.^37,38 Similar to the original SIC, the NIATx-SIC measure captures the: 1) proportion of completed activities, 2) the duration of activities, and 3) total timeframe from first to last activity.^7,37-39 Organizations with values above the average across all three components are deemed to have full NIATx adherence. NIATx activities completed during the pre-implementation phase and active implementation phase were recorded by our research team and entered into an online data collection and reporting system.

Ethics

The Institutional Review Boards at Stanford University, the University of Wisconsin – Madison, and the Washington State Health Care Authority reviewed the study and determined it to be exempt.

Statistical Analysis

McNemar’s test was used to compare the proportion of agencies categorized as AOS versus DDC/DDE between baseline and study end. Longitudinal outcomes were analyzed both with and without consideration for NIATx adherence. Standard linear mixed-effects modeling was employed to estimate changes from baseline to end of sustainment in the organization's capability for dual diagnosis in addiction treatment.^40,41 For all model estimation, we used maximum likelihood embedded in the Mplus program.⁴² Specifically, we used a random intercept piecewise model assuming three segments of linear change over time, from baseline to year 1, year 1 to year 2, and year 2 to year 3.

Per the intention to treat principle, all randomized agencies were included in the analyses as long as their data are available from at least one assessment. Unavailable data due to attrition or missed assessments were treated as missing at random conditional on available (observed) information, a standard strategy of handling missing data in mixed-effects modeling. In the per-protocol analysis, we examined the two randomized arms after excluding organizations that were not fully adherent to NIATx, as determined by the NIATx-SIC. In both the intent to treat and per-protocol analyses, Cohen’s d, a measure of effect size, was calculated based on observed standard deviation pooled across the two conditions at each time point.

Results

Baseline characteristics

The Extended CONSORT Diagram is available as an online supplement.⁴³ Forty-nine of the 53 organizations enrolled in the study were randomized to either NIATx1 (n=25) or NIATx2 (n=24). By study end, 40 organizations (82%) remained enrolled. Seventy-one percent (n=5) of organizations that discontinued participation did so due to their facility closing or being sold.

Baseline characteristics of organizations are presented extensively in previous papers by Assefa et al. and Chokron Garneau et al.^7,8 Participants were community addiction treatment organizations located in Washington State. Participating organizations were primarily from the public sector, located in regions with shortages in healthcare services.⁴⁴ Approximately 60% of programs offered outpatient/intensive outpatient services, and 40% were residential care facilities. Organizations in the NIATx1 group had more annual admissions on average (487.9±777.6) compared to those in the NIATx2 group (276.5±243.1). NIATx2 programs had longer lengths of stay on average (137.1±109.6) than those in the NIATx1 (113.2±109.4).

Primary Outcome: Changes in Dual Diagnosis Capability

Across both study arms, dual diagnosis capability improved and was maintained throughout the study period. Both groups started with a substantial number of organizations categorized as AOS (78%). At study end, in a complete reversal, the preponderance of organizations (78%) was at least Dual Diagnosis Capable across both groups (p<0.05). Specifically, 35% of NIATx1 organizations were at least DDC at baseline compared to 80% at year 3 (p<0.001). For NIATx2, 10% were DDC/DDE at baseline compared to 75% at year 3 (p<0.001). Although most gains were made between baseline and the end of the active implementation period, both cohorts continued to experience growth in DDC capable organizations between the end of the intervention and the end of their first sustainment period (NIATx1: 50% to 65%; NIATx2: 60% to 70%). NIATx1 saw further accrual in DDC organizations in the second sustainment year (from 65% to 75% in Year 3).

Evidence for Sustained Evidence-Based Practice

Intent to Treat

Intent to treat (ITT) analyses, including all randomized agencies (n=49), were conducted using mixed-effects modeling (Table 1, online supplement). No significant changes occurred in the NIATx1 group in its first-year post-implementation support. During its second sustainment year, NIATx1 showed significant improvement in Clinical Process: Assessment, Clinical Process: Treatment, and Continuity of Care. No significant changes occurred for the NIATx2 group in its sustainment year (Table 1). Effect sizes for changes from baseline to study end ranged from small to large (Cohen’s d = 0.25 to 0.88). The only significant between-group difference when looking at the change from baseline to study end was for the Training subscale where NIATx2 outperformed NIATx1 (Cohen’s d=0.88, p=0.02).

Table 1.

Estimated Intention to Treat Effects on Changes in DDCAT based on Mixed Effects Modeling

DDCAT Dimensions	Change from Year 1 to Year 2		Change from Year 2 to Year 3		Change from Baseline to Year 3
	NIATx1 (S Y1)	NIATx2 (active)	NIATx1 (S Y2)	NIATx2 (S Y1)	NIATx1	NIATx2	NIATx1 vs. NIATx2
Overall	−0.14	0.38*	0.21	0.05	0.83**	1.07**	−0.24
Program Structure	−0.08	0.49*	0.19	0.09	0.87**	1.19**	−0.33
Program Milieu	−0.25	0.33	0.31	0.04	1.10**	0.86**	0.24
Clinical Process: Assessment	−0.10	0.28	0.36*	0.11	0.68**	0.87**	−0.20
Clinical Process: Treatment	−0.17	0.28	0.26*	0.04	0.83**	1.05**	−0.21
Continuity of Care	−0.22	0.45**	0.30*	0.03	0.85**	1.08**	−0.23
Staffing	−0.23	0.35*	0.27	−0.03	0.79**	1.05**	−0.26
Training	0.03	0.44	−0.20	0.03	0.68**	1.35**	−0.68*

^**p ≤ 0.001

^*p ≤ 0.05

Footnote: S Y1: Sustainment Year 1; S Y2: Sustainment Year 2; active: active implementation phase

Per-Protocol

Twenty-three agencies were fully adherent to NIATx per their SIC scores (NIATx1=11, NIATx2=12) and included in Per-Protocol analyses (Table 2, online supplement). Neither NIATx1 or NIATx2 agencies show any significant changes in their first year post-implementation support. There were also no significant changes for NIATx1 in its second sustainment year. NIATx1 did however slightly improve on the Clinical Process: Assessment subscale, whereas NIATx2 slightly deteriorated on that same subscale. Effect sizes for changes from baseline to study end were small and ranged from 0.04 to 0.34. There were no significant between-group differences at study end when looking at the change from baseline.

Table 2.

Estimated Per Protocol Effects on Changes in DDCAT based on Mixed Effects Modeling

DDCAT Dimensions	Change from Year 1 to Year 2		Change from Year 2 to Year 3		Change from Baseline to Year 3
	NIATx1 (S Y1)	NIATx2 (active)	NIATx1 (S Y2)	NIATx2 (S Y1)	NIATx1	NIATx2	NIATx1 vs. NIATx2
Overall	−0.12	0.57**	0.06	−0.09	0.98**	1.06**	−0.08
Program Structure	−0.14	0.75*	−0.03	0.04	0.98**	1.20**	−0.23
Program Milieu	−0.35	0.63*	0.15	−0.19	1.16**	0.90**	0.26
Clinical Process: Assessment	−0.14	0.38	0.37	−0.23	0.93**	0.85**	0.08
Clinical Process: Treatment	−0.18	0.33	0.21	−0.19	1.00**	0.94**	0.06
Continuity of Care	−0.12	0.65**	0.22	−0.16	1.12**	1.16**	−0.04
Staffing	−0.05	0.73**	−0.14	−0.02	0.77*	1.11**	−0.34
Training	0.15	0.54**	−0.35	0.08	0.89*	1.16**	−0.27

^**p ≤ 0.001

^*p ≤ 0.05

Footnote: S Y1: Sustainment Year 1; S Y2: Sustainment Year 2; active: active implementation phase

Discussion

Gains in capability for integrated treatment services were sustained post-implementation support. Both groups achieved successful implementation and sustained the implementation to a similar degree regardless of the sustainment year. At baseline, 78% of participating organizations were categorized as Addiction Only Services. By the end of the trial, the ratio was reversed: 78% of participating agencies across both groups were at least Dual Diagnosis Capable. These thirty-one agencies significantly improved and maintained care and services for their patients with dual diagnoses of substance use and mental health disorders. Attainment of the Dual Diagnosis Capable or Dual Diagnosis Enhanced status mainly was achieved during the active implementation phase, and these changes were sustained until study end. Additionally, NIATx1 continued to improve on DDCAT subscales of Clinical Process: Assessment, Clinical Process: Treatment, and Continuity of Care throughout the sustainment period.

There are some limitations to this work. There is a volunteer bias for organizations that agreed to participate in the study, those who completed the implementation strategy and maintained participation through study end. It is also important to note baseline differences between cohorts. Although statistical analyses did account for these differences, it would have been preferable to balance groups on key variables. Further, an additional measure of integrated treatment delivery would validate our results and add richness to the interpretation of findings, including patient-level or service-level outcomes. Nonetheless, previous studies with the DDCAT have found this measure to have good validity, be associated with integrated service delivery and receipt, and patient outcomes.^{6,7,30,35,36,45,46}

These results build on, solidify and extend our previous findings. We previously established the importance of 1) using a proven manualized but adaptive strategy, such as NIATx, to enable change, and 2) documenting participation during the active implementation stage and measuring adherence to participation when examining outcomes.^7,8 Here, we demonstrate that increased capability to deliver integrated treatment services for individuals with co-occurring mental health and substance use disorders is sustained. This finding is significant as individuals with comorbid disorders in Washington State now have increased access to integrated care, the gold standard for such diagnosis.

Results from both intent-to-treat and per-protocol analyses converged, and minor differences in sustainment outcomes emerged between both sets of analyses. Thus, our results indicate sustained changes in behavioral health settings both when evaluated under actual and ideal circumstances.⁴⁷

It is, however, challenging to determine what accounted for the lack of difference between both sets of analyses. Following our year 1 and year 2 results, we would expect adherent organizations to outperform non-adherent organizations during the sustainment period. We suggest that this lack of differences is attributable to enhanced monitoring and feedback. The use of performance data is compiled and reflected to participants for their respective agencies and relative to all participating agencies combined. All organizations received this enhanced monitoring and feedback from baseline to study end regardless of adherence and participation status. Our study design did not account for enhanced monitoring and feedback only condition, and thus, it cannot be statistically confirmed. However, the results from the intent to treat analyses in sustainment make a convincing argument that enhanced monitoring and feedback is a viable and effective implementation strategy.

Conclusions

This paper presents a study of the sustainment of integrated services post-active implementation support. We observed wide variation in adherence to the study implementation strategy (NIATx) yet good overall sustainment of the target outcome. To further confirm these findings and advance the field of addiction treatment health services dissemination and implementation science, a better understanding of what and how an organization’s contextual factors may predict, moderate, and mediate sustainment outcomes is needed.⁴⁸ This need is further strengthened by findings from Swain et al. who report differences in sustainability of mental health services for people with serious mental illness based on agency characteristics such as financing, staffing, training, fidelity, and agency leadership.⁴⁹ Future studies will also be needed to determine whether NIATx is effective over time compared to groups that did not receive the intervention.

In implementation practice and research, it is typical to offer a fixed protocol for expected strategy participation. A one-size-fits-all implementation strategy is, however, neither effective or efficient. It creates a high demand and burden on organizations to engage in all activities, whether they need to or not, to increase access to an EBP. Much like a stepped care approach in a health care situation, we wonder if a similar approach to implementation might apply whereby low-intensity strategies, such as feedback and monitoring, might be tried first, then evaluated for desired outcomes. If not effective, then increasingly resource intensive strategies might be deployed. This stepped measurement-based implementation approach lends itself to an adaptive trial design and may inform future implementation endeavors with greater specificity and cost-efficiency. In addition, it would, decrease program burden and make better use of limited resources to support program implementation. Intensive supports that would have been spent on programs needing only a light touch could be re-directed toward reaching other programs.

Highlights:

• Access to integrated treatment services for individuals with co-occurring mental health and substance use disorders is a persistent health service delivery gap.

• Prior results indicate that the Network for the Improvement of Addiction Treatment (NIATx) implementation strategies can successfully be used to integrate mental health and addiction treatment services

• Delivery of integrated treatment services was sustained for two years post-active implementation support.