Applying Implementation Drivers to Scale-up Evidence-Based Practices in New York State

Abstract

Mental health authorities in several states, often working with academic partners, have played important roles in disseminating evidence-based practices (EBPs) for adults diagnosed with serious mental illness. This work has been facilitated by intermediary organizations that work directly with providers to implement EBPs. This report uses two case studies to describe how the Center for Practice Innovations (CPI), an intermediary organization, has used the Active Implementation Research Network’s nine implementation drivers to successfully implement EBPs across the large state of New York. One case study focuses on supported employment and the second on integrated treatment for co-occurring mental health and substance use conditions. We provide these case studies to illustrate how intermediary organizations can use implementation science to organize and select effective support strategies to disseminate and implement a range of EBPs within a state system.

Dissemination and Implementation of Evidence-Based Practices

Although evidence-based practices (EBPs) for adults diagnosed with serious mental illness have existed for some time, to this day few individuals receive them on a consistent basis (McKance-Katz 2018; Bond and Drake 2017). For example, only 2% of adults diagnosed with serious mental illness receiving public mental health services have access to supported employment, a well-documented recovery oriented EBP (Bazelon Center 2018). Similarly, in the 2017 National Survey on Drug Use and Health, only 12% of adults with co-occurring serious mental illness (8% with any mental illness) and substance use disorder received both mental health and specialty substance use treatment (Substance Abuse and Mental Health Services Administration, SAMHSA 2018).

Mental health authorities in a number of states, often working with academic partners, have played important roles in the dissemination of these practices (Drake et al. 2009). In New York State, the Center for Practice Innovations at Columbia Psychiatry, New York State Psychiatric Institute (CPI) has taken the lead with this effort (Covell et al. 2014). CPI is funded by the New York State Office of Mental Health (OMH) to bring to the field EBPs for adults diagnosed with serious mental illness, including those with co-occurring substance use. CPI is an intermediary organization (Franks and Bory 2015) that works with providers to implement EBPs and with OMH to disseminate these practices across NYS. This requires planners, trainers, and intermediary organizations to have knowledge and expertise in EBPs and the process of dissemination and implementation. With this knowledge, funders, policy makers, trainers and, most importantly, providers work together to realize this goal.

New York State has one of the largest behavioral healthcare workforces with about 100,000 providers working in over 6000 programs across the state. For the past decade, CPI has incorporated practical approaches in implementation science to increase the knowledge and change the practice of behavioral health providers in NYS, focusing on EBPs for adults diagnosed with serious mental illness (Nossel et al. 2018; Covell et al. 2011, 2014, 2015, 2016; Margolies et al. 2015; Dixon and Patel 2020). These practices include supported employment, integrated treatment for people with co-occurring mental health and substance use conditions, assertive community treatment, suicide prevention, early intervention for first episode psychosis, and wellness self-management. CPI promotes the use of EBPs by providing targeted implementation support, in addition to training. Complex clinical practices require new competencies, processes and procedures to insure their adoption and sustainment over time. This requires behavior change throughout the organization, not simply increasing knowledge in a subset of organization staff. Thus, implementation supports are key to the effective use of EBPs.

Recent literature (Franks and Bory 2015; Proctor et al. 2019) has pointed to the gap in understanding how intermediary organizations like CPI use implementation science in their approach to implementation. Since its inception, CPI has used a blended, practical and science-informed approach to their work including the Active Implementation Research Network’s (AIRN) Frameworks (Fixsen et al. 2005, 2019; Bertram et al. 2015; Pollastri et al. 2020). In 2007, we sought consultation from Dean Fixsen to understand and develop strategies based on implementation drivers to disseminate and implement EBPs across NYS. The synthesis of the implementation literature at that time (Fixsen et al. 2005) offered a practical view of implementation that defines stages of implementation and core implementation components or drivers. These have been subsequently refined (Bertram et al. 2015; Fixsen et al. 2019) so that there are now 4 implementation stages, 3 domains, and 9 implementation drivers. The 9 AIRN Frameworks implementation drivers are (Bertram et al. 2015; Fixsen et al. 2019): Competency Drivers (i.e., Staff Selection, Training, Coaching, Fidelity Including Performance Assessment), Leadership Drivers (i.e., Technical, Adaptive), and Organizational Drivers (i.e., Facilitative Administration, Systems Level Interventions, Decision Support Data Systems). Descriptions of each driver can be found in Table 1.

Table 1.

Initiative drivers and descriptions

Driver	Description
Competency Driver: Staff Selection	Ideally, a program can select staff with specific skills for each evidence-based practice (EBP) and an intermediary organization, like CPI, can help support that selection
Competency Driver: Training	Once staff have been selected, training and coaching are the primary competency drivers through which behavior change that supports the EBP occurs
Competency Driver: Coaching	Coaching supports staff’s development of competency by helping them to move from increasing knowledge to developing and appropriately using skills
Competency Driver: Fidelity Including Performance Assessment	Performance assessment includes both practitioner performance with consumers and organizational performance toward fidelity (e.g., have staff been carefully selected? Is training provided to staff? Is coaching occurring as scheduled and informed observations of practice?)
Leadership Driver: Technical	CPI staff use consultation calls and on-site technical assistance visits to work with program leaders and supervisors to address challenges identified through data (i.e., fidelity, performance indicators), observation, and program staff. Following these calls and visits, CPI offers solutions to leadership to address these challenges
Leadership Driver: Adaptive	Adaptive leadership is important when both the definition of the problems and their solutions are less clear, which is often the case with system-wide interventions
Organizational Driver: Facilitative Administration	Facilitative administration drives the organizational culture and climate to focus on and actively support implementation and sustainability of EBPs
Organizational Driver: Systems Level Interventions	Practice fidelity and sustainability are often influenced by the alignment of federal, state, organization, and community systems
Organizational Driver: Decision Support Data Systems	Timely data related to model fidelity to guide administrative decisions about organizational change and fidelity of staff performance are essential for quality improvement and program sustainability

In this paper, we will describe two case studies focusing on the application of the AIRN Frameworks’ nine implementation drivers in two initiatives at CPI. Individual Placement and Support (IPS) addressed employment, and Focus on Integrated Treatment (FIT) addressed mental health and substance use conditions. These case studies will highlight how we have applied strategies derived from these implementation drivers within these two initiatives. We provide this information to illustrate how intermediary organizations can guide implementation support for programs using an implementation framework.

Case Study: Supported Employment Initiative

Background

A decade ago, supported employment was not typically available as a service for individuals diagnosed with serious mental illness in NYS and disseminating supported employment became an early focus of CPI. A review of the literature at the time made it clear that IPS is the gold-standard supported employment EBP, helping adults with serious mental illness find and keep meaningful jobs (Bond et al. 2001; Drake et al. 2012). CPI’s IPS initiative (Margolies et al. 2015) began a decade ago and currently is training and providing implementation support to over 80 community rehabilitation programs and clinics across NYS. Planning for and implementation of this initiative included use of all nine AIRN implementation drivers (Table 2). Training and support is tailored to the needs and implementation status of each site, based upon fidelity and performance indicator data. Sites that have recently joined the initiative and are beginning implementation are provided with foundational activities such as online training modules that introduce the principles and practices of IPS, use of an implementation guide consisting of concrete activities informed by AIRN drivers, and on-site technical assistance focusing on practitioner competency development and implementation guidance and support for managers and supervisors. Sites that have been participating for a number of years (some from the beginning of the initiative a decade ago) and are sustaining good implementation receive less frequent technical assistance visits that are driven by that site’s data reports. This initiative has consistently produced comparable or better outcomes of employment than that of national benchmarks (described in greater detail in “Outcomes and Challenges” section) for a large number of implementation sites (over 80) in a single state. Next, we describe how the AIRN drivers were applied to IPS.

Table 2.

Initiative activities mapped onto the active implementation drivers

Driver	Individual Placement and Support (IPS)	Focus on Integrated Treatment (FIT)
Competency Driver: Staff Selection	• Consultation provided to agency leadership about qualities and competencies to look for in candidates • Distribution of job descriptions • Sitting in on interviews of candidates for employment specialist positions	• Consultation provided to program leadership and implementation support call participants about qualities and competencies to look for in candidates
Competency Driver: Training	• Online modules  • Introduction to IPS  • Job development  • Use of the Employment Resource Book • Online library • Webinars • In-person training workshops • Site-specific technical assistance visits	• Online modules (39), 29 of which Integrated Mental Health/Addictions Treatment Training • Online library • Webinars • Face-to-face training workshops • Site-specific technical assistance calls
Competency Driver: Coaching	• Training supervisors in field mentoring and use of data • Webinars • Face-to-face training workshops • Site-specific technical assistance visits • Site-specific consultation calls • Learning collaborative meetings	• Two online modules designed for supervisors • Webinars • Face-to-face training workshops • Twice monthly program implementation support calls • Site-specific consultation calls • Continuous quality improvement projects through Learning Collaboratives • Training supervisors about how to coach/supervise MI, stage-wise treatments, and other components of integrated treatment (modules, webinars, LC) • LC activities that support and encourage sharing and collaboration across programs being trained
Competency Driver: Fidelity	• Use of Individual Placement and Support (IPS) fidelity scale • Use of performance indicator data • Observation of staff competency during site-specific technical assistance visits	• Use of Dual Diagnosis Capability in Mental Health Treatment capability scale to target areas for improvement • Use of Video Assessment of Simulated Encounters • Online training in the form of modules and webinars devoted to performance assessment
Leadership Driver: Technical (resolving procedural problems)	• Address identified challenges • Consultation calls • On-site technical assistance visits	• Changing specific procedures (e.g., including a specific screening form in an electronic medical record) and policies (e.g. smoke free workplace) to support implementation • Online modules designed for leadership
Leadership Driver: Adaptive (resolving complex problems)	• Implementation issues identified and addressed with program leaders • Technical assistance provided to workgroups	• Implementation workgroups including key leadership and champions to identify and respond to challenges • Technical assistance provided to workgroups
Organizational Driver: Facilitative Administration	• Implementation guide • Training about IPS fidelity and performance indicators • Data provided to implementation sites—fidelity and performance indicators to facilitate continuous quality improvement Projects • Tailored interventions for struggling programs • Guidance re: sustainability	• Data provided to LC participants—fidelity and performance indicators to facilitate CQI Projects • Training about Dual Diagnosis Capability in Mental Health Treatment capability scale • Guidance re: sustainability • Program-specific consultation calls (upon program request and/or OMH/other leadership)
Organizational Driver: Systems level intervention	• Discussions with OMH about expectations and incentives set by OMH • Ongoing review of data with OMH and discussion of implementation challenges and solutions • Liaison role with OMH to communicate system-wide issues • Identifying champion programs and increasing their visibility	• Discussions with OMH and OASAS about expectations and incentives for programs • Ongoing review of data with OMH and discussion of implementation challenges and solutions • Liaison role with OMH to communicate system-wide issues
Organizational Driver: Decision Support Data System	• Use of fidelity scale • Monthly performance indicators • CPI analyzes this data, maintains data bases, and provides data to OMH and the implementation sites on a regular basis	• Use of fidelity scale • Performance indicators submitted as part of LC participation • Program-specific consultation on existing data systems that can inform fidelity and implementation success

Competency Driver: Staff Selection

CPI educated implementation sites about the specific knowledge and skills employment specialists need for IPS. Employment specialists combine an understanding of IPS, behavioral health, and business needs and practices to help individuals find and maintain employment. For IPS, some programs were able to hire new staff with skills specific to supported employment; in these cases, CPI has helped these programs develop and distribute job descriptions and also attended some candidate interviews.

At other times, however, programs are unable to hire additional staff and must find a way to use existing staff. Additionally, even if programs can post a new position, some are bound by union or internal policies that limit who they can hire. In these cases, CPI has helped programs identify staff who might have existing skills and qualities that, with some additional training, coaching, and feedback (examples described in “Training, Coaching and Performance Assessment Drivers” section), could develop the full complement of skills needed for IPS.

Competency Driver: Training

Training employment specialists begins with online training modules (Patel et al. 2018), available at no cost through CPI’s learning management system. The initiative also includes an online library of resources including implementation guides, presentation slides, fidelity scales, and other helpful training tools. For IPS, three online modules focus on an introduction to IPS, the important skill of job development, and use of a CPI-developed tool called the Employment Resource Book.

CPI identifies ongoing training needs for employment staff several different ways. First, existing fidelity and performance indicator data point to both system-wide and site-specific needs. Second, in interactions with programs, staff often raise specific challenges and training needs. These have included supporting individuals with forensic histories, helping individuals with online job applications, learning how to access statewide resources, and adapting IPS services to remote platforms during the COVID-19 pandemic. Third, licensing staff refer programs to CPI for training and supports when programs are newly developing or when a visit to a more developed program identifies areas needing support.

Once additional training needs are identified, CPI develops and offers this training through interactive live webinars that are archived for future viewing, regional face-to-face workshops, and site-specific technical assistance visits and calls.

Competency Driver: Coaching

CPI staff train program supervisors about an approach to coaching that includes field mentoring and use of data using modules, webinars, in-person workshops and site-specific technical assistance visits.

CPI staff are also able to provide direct in-vivo coaching to employment staff. For example, during site-specific technical assistance visits, IPS staff go into the field with employment specialists to model and observe first hand job development and other IPS skills. IPS also uses program-specific consultation calls to help program staff develop and refine their skills and to provide supervisors with tips and tools for coaching those skills.

The IPS initiative offers learning collaboratives for groups of programs where program staff learn from one another as they discuss their implementation efforts and supervisors receive tools to support their coaching practices. At times, these collaboratives have focused specifically on enhancing supervisors’ coaching skills.

Competency Driver: Fidelity Including Performance Assessment

IPS fidelity self-assessments using an established tool (Bond et al. 2012) are completed annually by each implementation site. Performance indicator data is submitted by sites monthly and includes information such as IPS caseload size, time spent by staff in the community, and employment status of those receiving IPS services. Fidelity and performance indicator data are used to focus subsequent training and support activities.

Learning communities (Friedman et al. 2017) collect data, interpret them, and feed the information obtained back into their own system to continuously improve their processes.

Strategies used in the IPS initiative included training managers, supervisors and practitioners in continuous quality improvement methods and applying these methods to opportunities for improvement as identified through fidelity and performance indicator data. Stakeholders learn how to apply Deming’s Plan-Do-Study-Act (PDSA) approach (2000) to identified opportunities for improvement. Data collection is designed to be user-friendly through the use of clearly constructed online data collection surveys. Examples of continuous quality improvement projects have included increasing employment staff time in the community, inspiring individuals not interested in employment to consider the possibility, and incorporating a focus on work into the intake process.

CPI’s training staff assess program employment staff’s skills directly during site-specific technical assistance visits.

Leadership Driver: Technical

CPI staff use consultation calls and on-site technical assistance visits to work with program leaders and supervisors to address challenges identified through data (i.e., fidelity, performance indicators), observation, and program staff. Following these calls and visits, CPI offers solutions to leadership to address these challenges. For example, IPS has provided technical assistance to leaders around changes in procedures, staff roles, and tasks.

Leadership Driver: Adaptive

CPI staff work with program leaders to identify larger systems issues needing resolution. For IPS, these have included promoting a culture of employment within each implementation site that highlights the importance of employment within the program’s mission and implements procedures that integrate employment with treatment services, prioritizing IPS implementation, and incentivizing its use and employment outcomes.

The IPS initiative encourages programs to develop workgroups, including key leadership and champions, to identify and respond creatively to challenges in implementation as they arise. CPI offers technical assistance to these workgroups both through established learning collaboratives and upon request.

Organizational Driver: Facilitative Administration

As programs begin to implement new practices, CPI provides leaders with training and guidance to facilitate this process. For IPS, program leaders and supervisors are provided with an IPS implementation guide that maps out the important leadership tasks to promote proper IPS implementation. Examples include establishing a steering committee/quality improvement team, informing and engaging stakeholders, and collecting and using data. In addition, they are taught about IPS fidelity and performance indicator measures.

Once implementation is underway, CPI uses data to provide leadership with results from fidelity measures and performance indicators to inform quality improvement efforts. CPI also works with program leaders of struggling implementation sites to tailor our efforts to meet their needs.

The idea of sustainability is embedded throughout the process of implementation, where CPI encourages programs to adopt policies, procedures, and workflow that embed the EBP as a routine practice in the program. Once programs have maintained some level of fidelity to the practice, the IPS initiative works with leaders of successful sites to develop strategies for sustaining the gains. Through time, CPI has learned that it is important to maintain contact with sustaining programs in order to help them identify and address new challenges that arise as a result of organizational and systemic changes.

Organizational Driver: Systems Level Interventions

CPI initiatives, including IPS, work with the mental health authorities (OMH) to discuss expectations and incentives related to each initiative and provide data on a regular basis to inform decision making. At the same time, CPI is able to inform OMH about system-wide challenges that can best be addressed by the mental health authority through guidance, clarification, incentives and/or licensing. CPI also identifies “champion” implementation sites and promotes their visibility and availability to other sites. These sites, who have demonstrated clear and consistent commitment to the initiative, and high levels of fidelity and outcomes, are featured in learning collaborative meetings providing guidance and concrete examples of their work.

Organizational Driver: Decision Support Data Systems

As noted previously, CPI conducts fidelity assessments with participating programs, and provides data back to programs to drive quality improvement processes following the plan-do-study-act process (Deming 2000). CPI also collects performance indicator data from participating sites, providing summary reports that are actionable to OMH and the implementation sites on a regular basis. For example, IPS collects monthly performance indicator data related to program implementation.

Outcomes and Challenges

Sites that implement IPS report competitive employment outcomes consistent with national benchmarks (i.e., 40–50% of those receiving IPS are competitively employed each month) (Becker et al. 2011, 2012). For example, in August 2019, 70 participating sites provided employment data, with an average employment rate of 52.4% for those receiving IPS. This means that 927 IPS participants were working that month.

An important challenge for IPS implementation is changing the cultures of behavioral health settings to embrace employment as an important aspect of their missions. Strategies employed included working with leadership to communicate the importance of employment within the organization’s priorities, addressing the needs and concerns of individuals at all levels of the organization, involving individuals at all levels of the organization to develop implementation plans, identifying “champions” (i.e., staff who enthusiastically endorse the IPS practice, model its use, and speak to their positive observations and experiences with its implementation) who provide formal or often informal leadership and inspiration for the practice, institutionalizing the practice through record keeping and policies/procedures, and celebrating accomplishments.

A second challenge for IPS implementation is convincing clinicians and program leaders that employment is a reasonable recovery goal and not an inevitable contributor to relapse. Strategies employed included online training about IPS, in-person training and technical assistance that focuses on competency development, scheduling discussions with recipients who are successfully working to show that this is positive and possible, and tailoring in-person training to meet learners where they are. A third challenge is ensuring that the important focus on employment isn’t eclipsed by urgent matters such as clinical crises, which can result in staff time being reallocated away from providing IPS services. This requires CPI’s implementation specialists to work closely with program supervisors and leaders to develop strategies for handling this possibility. Employment staff and supervisors also learn strategies from one another during learning collaborative meetings.

A more recent challenge has emerged due to the COVID-19 pandemic. Many of our implementation sites are now working remotely and this impacts the way in which we provide training and implementation support and the manner in which IPS services are being provided. For now, our training is entirely through online platforms and we have worked with implementation sites to develop training methods to meet their needs including statewide webinars designed so that sites share their innovative approaches with one another and site-specific calls and online meetings to help with problem solving and creative thinking. Fortunately, since an important aspect of our pre-COVID-19 approach has included distance learning methods, adapting to current realities had not been difficult for our Center. Despite the impact of COVID-19, a recent survey of our implementation sites as well as performance indicator data indicate that 64 of 88 (73%) reporting sites are still providing IPS services, 49 of these 64 sites (77%) report job losses of 25% or less, and a small number of individuals are starting new jobs that have been created during the pandemic including warehouse workers, grocery shoppers and COVID-19 remote tracers.

Case Study: Integrated Treatment Initiative

Background

For more than a decade, the Focus on Integrated Treatment (FIT) initiative has supported evidence-based integrated treatment for co-occurring severe mental health and substance use disorders, where a single practitioner or team helps the consumer address both mental health and substance use disorders (Substance Abuse and Mental Health Services Administration, SAMHSA 2009). Integrated treatment mitigates hospitalizations and other risks, reduces treatment costs, and promotes individuals’ recovery, independent living, and employment (Drake et al. 2004, 2006; Grella and Stein 2006). FIT is charged with making training and technical assistance available to nearly 6000 OMH and Office of Addiction Services and Supports (OASAS) programs statewide. Given this, FIT offers an array of supports ranging from light touch (all programs are able to access online self-paced training and tools for free) to more intensive efforts, such as those described here. Table 2 displays the various ways each of the nine implementation drivers have been applied throughout those efforts. The drivers that are selected and emphasized for each site are tailored to that site, depending upon their needs and what they plan to work on first. Where possible, these decisions are based upon fidelity and other data. Typically, this support is through a learning collaborative, usually including monthly online meetings and direct technical support (e.g., reviewing and providing guidance on implementation plans) between meetings for 1 year with some continued contact following that year. Where resources have allowed, more intensive support has continued as long as three years with new programs joining the effort along the way. Depending on a program’s goals and needs, the process of implementation can follow a series of continuous quality improvement efforts lasting several years; hence, the longer FIT is able to continue this support, the better. We include this initiative as a contrast, because the mandate to provide training and implementation support is on a larger scale across two systems of behavioral health care (OMH and OASAS) and the outcomes are more challenging to define and measure compared to IPS.

Competency Driver: Staff Selection

Staff who provide integrated treatment need to be skilled at identifying and treating both mental health and substance use disorders, including strategies both for people who are ready to make a change and those who are not yet ready or ambivalent about addressing mental health issues and/or substance use. FIT encourages programs to include these skills in job descriptions for new staff and list, as “preferred”, people who have already completed an Integrated Mental Health and Addictions Treatment Training certificate (see “Training” section). Additionally, FIT encourages programs to identify and mentor existing staff who are passionate about integrated treatment.

Competency Driver: Training

FIT offers online training modules and access to an online library of resources available at no cost. For staff providing services, online training includes the following: screening and assessment of mental health and substance use disorders, treatment planning, stage-wise treatment, stage-wise treatment groups, motivational interviewing, cognitive–behavioral therapy, social skills training, recovery and community support, integration of physical health, medications to treat behavioral health disorders, treatment of tobacco use disorders, and working with adolescents who have co-occurring disorders. Staff who complete all of this training earn the Integrated Mental Health and Addictions Treatment Training certificate which is signed by both the commissioners of OMH and OASAS. In many cases, programs have adopted this online training and certificate completion as part of their regular process of onboarding new staff.

FIT identifies ongoing training needs for staff in programs several different ways. First, existing measures such as the Dual Diagnosis Capability in Mental Health Treatment scale (DDCMHT; Gotham et al. 2009) can identify areas where programs can benefit from additional training. Second, in interactions with programs, staff often raise specific challenges or even identify specific training needs. In twice monthly implementation support calls for all programs implementing integrated treatment across NYS, challenges raised around implementation (e.g., attendance at stage-wise treatment groups) and requests for specific staff training led to FIT developing additional training and tools. Third, licensing staff refer programs to CPI for training and supports, either as programs are newly developing or as a visit identifies areas needing support. Finally, as service needs change, state leadership may identify specific areas where programs require additional training and support. For example, to address the significant opioid use problem, OMH and OASAS expanded the use of medication assisted treatment for opioid use into programs primarily licensed to treat mental health concerns. FIT is integrally involved with the team providing training and supports to provide integrated treatment for this specific population of people with co-occurring mental health and opioid use disorders.

Once additional training needs are identified, CPI offers ongoing training, often with continuing education, several different ways. First, training can be delivered using interactive live webinars that are archived for future viewing. For FIT, webinar topics have ranged from basic information about substance use to more advanced skills such as modifying motivational interviewing for people with psychotic disorders and webinar series devoted to a deeper exploration of specific skills (e.g., cognitive–behavioral therapy). FIT has also offered in-person training for specific competencies (e.g., motivational interviewing), as well as site-specific technical assistance calls.

Competency Driver: Coaching

FIT offers two online modules for supervisors on the topics of group supervision, skills training, field mentoring, modeling, coaching with feedback, and data-based supervision. FIT also offers live and archived webinars specifically for supervisors to enhance their capability to coach staff in specific skills related to integrated treatment (e.g., the Video Assessment of Simulated Encounters Rosengren et al. 2008, a tool for providing coaching and feedback around Motivational Interviewing).

FIT also trains supervisors how to use either established or newly created observation tools to rate staff performance and inform coaching sessions. For example, FIT has developed several checklists supervisors can use to help to assess staff performance (e.g., facilitating stage-wise treatment groups) across various skills. Twice monthly program implementation support calls and site-specific consultation calls often focus on coaching practitioners as they develop and refine needed skills. FIT also offers learning collaboratives for groups of programs where program staff learn from one another as they discuss their implementation efforts and supervisors receive tools to support their coaching practices. At times, these collaboratives have focused specifically on enhancing supervisors’ coaching skills. For example, FIT offered a learning collaborative aimed at enhancing supervisors’ ability to coach staff in providing stage-wise treatment groups.

Competency Driver: Fidelity Including Performance Assessment

FIT offered a quality improvement learning collaborative where programs used results from the DDCMHT (Gotham et al. 2009) to target areas for improvement. FIT also works with programs around fidelity to particular components of integrated treatment. For example, using the Video Assessment of Simulated Encounters (Rosengren et al. 2008), implementation specialists work directly with program staff to increase their proficiency with Motivational Interviewing. FIT has also trained supervisors (see “Coaching”) to use this tool with staff.

FIT also offers online training in the form of modules and webinars devoted to performance assessment. For example, FIT includes a module describing two established scales for assessing a program’s ability to treat co-occurring mental health and substance use disorders—Dual Diagnosis Capability in Mental Health Treatment (Gotham et al. 2009) and Dual Diagnosis Capability in Addiction Treatment (McGovern et al. 2007) and discusses data-based supervision in one of the aforementioned modules for supervisors.

Leadership Driver: Technical

FIT offers consultation calls to work with program leaders and supervisors to address challenges identified through fidelity and performance indicator data, in CPI observation, or by program staff. Following these calls and visits, FIT offers solutions to leadership to address these challenges. For example, FIT has offered solutions to technical leadership around adopting specific policies (e.g., tobacco free grounds) and change in workflow (e.g., adding a routine screening form for both mental health and substance use to their intake process) that embeds the practice of integrated treatment.

FIT also offer offers online training to inform technical leadership. For example, FIT offers modules for clinical leadership that detail processes and procedures to facilitate implementation of integrated treatment broadly and treating tobacco use, specifically.

Leadership Driver: Adaptive

FIT encourages programs to identify champions and develop leadership buy-in early in the implementation process. Champions, along with leadership, form workgroups, including staff representing various roles and other key stakeholders, such as service recipients, where possible. FIT supports the workgroups in developing an implementation plan that includes clear steps with measurable outcomes. These workgroups meet regularly to discuss progress and to identify and respond creatively to challenges in implementation as they arise. FIT offers ongoing technical assistance to these workgroups through established learning collaboratives, attendance at workgroup meetings, and separately as requested by the champion and/or leadership.

Organizational Driver: Facilitative Administration

As programs begin to implement new practices, FIT offers online training modules and documents in the resource library to provide guidance around implementing integrated treatment. Additionally, as noted above, FIT offers training around measures to assess fidelity to integrated treatment.

Once implementation is underway, FIT continues to support implementation and sustainability through use of data and upon request from programs, OMH or OASAS. Programs that participate with FIT in learning collaboratives submit regular performance indicators that are summarized as reports that leadership and supervisors can use to drive quality improvement projects. FIT also offers program-specific consultation, upon request, to help troubleshoot challenges and identify quality improvement processes to target those challenges.

Sustainability is embedded throughout the process of implementation in a number of ways. For example, FIT encourages programs to adopt policies, procedures, and workflow that embed the EBP as a routine practice in the program. Once programs have maintained some level of fidelity to the practice, FIT works with leaders of successful sites to develop strategies for sustaining the gains. For example, FIT encourages programs to continue monitoring relevant outcomes and to use these measures as part of a continuous quality improvement process. Programs are also encouraged to use the DDCMHT scale to track progress through time and address any new needs that emerge (e.g., ensuring that an adequate number of practitioners are trained to deliver integrated treatment and ensuring the availability of ongoing supervision and feedback to improve staff skills, ensuring the continued availability of evidence-based treatment components). Similarly, FIT encourages programs to include training as part of new staff orientation and to make competencies in integrated treatment a regular part of staff evaluations. Through time, FIT has learned that it is important to maintain contact with sustaining programs in order to help them identify and address new challenges that arise as a result of organizational and systemic changes.

Organizational Driver: Systems Level Interventions

FIT works with program leaders and leaders of OMH and OASAS to identify larger systems issues needing resolution. Examples include incentivizing training through the Integrated Mental Health and Addictions Treatment Training certificate program, prioritizing integrated treatment, providing exemplary status on licensing reviews for programs meeting certain benchmarks, strengthening guidance and clarification documents to support the practice, and aligning integrated treatment with other system priorities and initiatives. FIT also informs OMH and OASAS of any system-wide challenges that can best be addressed by through guidance, clarification, incentives and/or licensing.

Organizational Driver: Decision Support Data Systems

As noted previously, FIT conducts fidelity assessments with participating programs, and provides those data back to programs to drive quality improvement processes. FIT also collects performance indicator data from participating sites, providing summary reports that are actionable to OMH and the implementation sites on a regular basis. For example, FIT collects performance indicator data from programs that participate in learning collaboratives and offers detailed reports with suggested next steps for improvement. FIT also offers online modules around tracking process and outcome improvements.

Outcomes and Challenges

Unlike IPS which has a clearly defined outcome (employment), documenting outcomes among service recipients is a challenge for FIT. Specifically, people with co-occurring mental health and substance use disorders may experience positive improvements in a number of areas that are related to their personal recovery goals (e.g., reduced mental health symptoms or substance use, return to work or school, improved relationships with significant others, increased health and well-being). Capturing these personalized outcomes is challenging in a large state where providers operate across two behavioral healthcare systems (OMH and OASAS) and use their own, often home-grown, electronic medical records.

However, FIT is able to measure impacts of training and technical support. For example, as of July 31, 2020, 34,098 people completed a total of nearly 400,000 FIT online training modules (395,894), and 5290 had completed the Integrated Mental Health and Addictions Treatment Training certificate. Similarly, performance measures collected within learning collaboratives have demonstrated significant improvements in implementation of integrated treatment (Covell et al. 2015, 2019).

When providing technical assistance, a common challenge for FIT is demonstrating to clinicians that existing skills can be used to treat the co-occurring disorder that they are least familiar with (e.g., clinicians who are comfortable treating mental health conditions already have many of the skills needed to address substance use). FIT strives to increase buy-in and self-efficacy among staff a number of ways. First, we find that stories of people who achieved dual recovery (for both mental health and substance use) inspire staff that dual recovery is possible and motivate them to want to learn more. Second, online training, webinars, and technical assistance illustrate existing skills they can apply to help people working toward dual recovery. Finally, FIT works with supervisors to help them coach staff in ways that increases their confidence to treat both mental health symptoms and substance misuse.

As with IPS, another challenge is maintaining focus on ensuring that the important focus on integrated treatment amidst competing demands. This requires implementation specialists to work closely with program supervisors and leaders to develop strategies to ensure that integrated treatment is embedded as routine practice. Additionally, FIT continues to work with OMH to ensure that messaging, incentives, policies, and program standards reflect the importance of continuing to provide integrated treatment.

A more recent challenge has emerged due to the COVID-19 pandemic. The stress from the pandemic and physical distancing is expected to increase both mental health symptoms and substance use, particularly among those already vulnerable to these conditions (Du et al. 2020; McKay and Asmundson 2020). Emerging data (Czeisler et al. 2020; McKay and Asmundson 2020) and anecdotal reports on implementation support calls support this concern. Additionally, those with substance use disorders have higher rates of the comorbid health conditions that increase vulnerability to the impacts of COVID-19 (Farhoudian et al. 2020; Volkow 2020). At the same time, people who use substances are at increased risk of infection as they venture into the community to obtain substances, and because they experience housing instability and other situations that further increase their risk (Farhoudian et al. 2020; Volkow 2020). Further, some of the critical components of integrated treatment, such as connection to peers in groups, have been compromised with physical distancing (Volkow 2020). In short, the need for integrated treatment has never been greater. With input from OMH and OASAS, FIT developed two guidance documents for telehealth with people who have co-occurring mental health and substance use conditions in a way that is consistent with integrated treatment (one for any co-occurring substance use and one specifically for co-occurring tobacco use). These materials were presented in a live webinar which is now available as an archive. We also continue to address how to implement and sustain integrated treatment through telehealth during our twice monthly implementation support calls.

Summary and Limitations

In its role as an intermediary organization, CPI is funded by OMH to bring EBPs to the behavioral healthcare system across NYS. Using case studies, we describe how strategies based on the AIRN Frameworks’ implementation drivers have guided CPI’s work in two initiatives—IPS supported employment and integrated treatment (Focus on Integrated Treatment; FIT). Importantly, even though the treatment components and expected outcomes differ considerably between IPS and FIT, it was easy to apply and tailor the AIRN Frameworks’ implementation drivers to each EBP. The nine drivers provide easy-to-use, concrete, actionable guidance concerning the development of implementation activities. They also provide a relatable common language (Moullin et al. 2020) that could be shared by the intermediary organization and implementation sites as they plan, execute and evaluate implementation efforts. That said, it is essential that these implementation activities be tailored to the specific context of the implementation effort, taking into account the nature of the specific EBP, regulatory and funding environment, and specific aspects of the implementation sites including receptivity to innovation, level of buy-in from leaders, managers and practitioners, and ability/commitment to prioritize the importance of the innovation in the midst of other urgent matters. As noted in both case studies, the nine drivers helped implementers with these important tasks.

This report includes a number of important limitations. First, both case studies reflect the work of a single intermediary organization and may not be generalizable to other intermediary organizations. Even so, that these drivers were helpful for two very different evidence-based practices suggests that they may be applicable more broadly. Similarly, both case studies were implemented in NYS, and therefore may not be generalizable to other states with differing levels of support from their mental health authorities. It would be beneficial if future studies replicated the use of the nine AIRN Frameworks’ implementation drivers with other evidence-based practices, in other states, in other settings, and with other populations. Second both case studies used all nine implementation drivers and thus we are unable to determine the relative contributions of each of the drivers separately. Future studies might include models examining these contributions. It would also be useful to receive feedback from implementation sites concerning support received from the intermediary organization, including which drivers were most helpful, and ways in which these efforts can be improved. Third, given the challenges of measuring a wide array of personalized outcomes for people receiving integrated treatment across a vast number of programs, FIT was not able to track intervention outcomes. It would be useful for FIT to conduct smaller longitudinal studies of implementation efforts to change practice in targeted areas (e.g., implementation of treatment for co-occurring mental health and tobacco use) with more specific, measurable outcomes (e.g., prescriptions of medication to treat tobacco use, reduction of tobacco use). Finally, much of the work presented was conducted prior to the COVID-19 pandemic. As noted in the paper, COVID-19 has presented challenges—including the necessity of remote training and service provision—that all implementation efforts need to address. It will be important to study how the nine drivers will be adjusted to COVID-19 realities. It is likely that all nine drivers will be impacted. For example, remote service provision, training, and implementation support will clearly affect the Fidelity, Training and Coaching drivers since changing realities result in changing methods. Even Staff Selection will be adjusted to accommodate remote interviews as well as placing value on experience using online platforms. Leaders have quickly adapted to these changing realities and no doubt will continue to make adjustments as the situation evolves – highlighting the importance of and resulting in new areas of focus for the two Leadership drivers. The foci of the three organizational drivers will likely evolve due to COVID-19-driven changes in how organizations and systems function and what data will be prioritized.

This paper may be useful as a roadmap for other implementation practitioners or intermediary organizations tasked with not only disseminating but also implementing a range of EBPs within a state system. The two case examples illustrate how implementation science can provide practical and actionable strategies to guide organizations throughout the implementation process. Additional guidance and resources are available from a number of sources including the National Implementation Resource Network (https://nirn.fpg.unc.edu) and the Society for Implementation Research Collaboration (https://societyforimplementationresearchcollaboration.org).

Funding

The New York State Office of Mental Health provided funding for the activities described in this report.

Data Availability

Not applicable.

Compliance with Ethical Standards

Conflict of interest

The authors report no financial relationships with commercial interests.

Ethical Approval

The New York State Psychiatric Institute Institutional Review Board determined that this evaluation did not meet the definition of human subjects research.

Informed Consent

Not applicable.