The implementation and sustainment facilitation strategy improved implementation effectiveness and intervention effectiveness: Results from a cluster-randomized, type 2 hybrid trial

Abstract

Background:

Substance use disorders (SUDs) among people with HIV are both prevalent and problematic. The Substance Abuse Treatment to HIV care project was funded to test the Implementation and Sustainment Facilitation (ISF) strategy as an adjunct to the Addiction Technology Transfer Center (ATTC) strategy for integrating a motivational interviewing-based brief intervention (MIBI) for SUDs within HIV community-based organizations.

Methods:

Using a cluster-randomized, type 2 hybrid trial design, 39 HIV organizations were randomized to either (1) ATTC (n = 19) or (2) ATTC + ISF (n = 20). Each HIV organization identified two staff members to be prepared to implement the MIBI (N = 78). Subsequently, during the implementation phase, HIV organizations in each condition randomized client participants (N = 824) to one of the two intervention conditions: usual care (UC; n = 415) or UC + MIBI (n = 409). Both staff-level outcomes and client-level outcomes were examined.

Results:

The ISF strategy had a significant impact on the implementation effectiveness (i.e., the consistency and the quality of implementation; β = .65, p = .01) but not on time-to-proficiency (β = −.02) or level-of-sustainment (β = .09). In addition, the ISF strategy was found to have a significant impact on the intervention effectiveness (the effectiveness of the MIBI), at least in terms of significantly decreasing the odds (odds ratio = 0.11, p = .02) of clients using their primary substance daily during follow-up.

Conclusion:

The ISF strategy was found to be an effective adjunct to the ATTC strategy in terms of implementation effectiveness and intervention effectiveness. It is recommended that future efforts to integrate the project’s MIBI for SUD within HIV organizations use the ATTC + ISF strategy. However, given the ISF strategy did not have a significant impact on level-of-sustainment, implementation research testing the extent to which the ATTC + ISF strategy can be significantly enhanced through effective sustainment strategies is warranted.

Plain language abstract

Substance use among people living with HIV is associated with increased mental health problems, worse medication adherence, and worse HIV viral suppression. Increasing substance use-related services in HIV community-based organizations is an important public health need. The Substance Abuse Treatment to HIV care project tested two strategies for helping HIV organizations implement a brief intervention (BI) designed to motivate clients to decrease their substance use. The project also tested if receiving a BI improved clients’ outcome. Two staff from each of the 39 participating organizations were taught how to deliver the BI using the Addiction Technology Transfer Center (ATTC) training strategy (online and in-person training, monthly feedback, and coaching). Half of the organizations also received the Implementation and Sustainment Facilitation (ISF) strategy, which included monthly meetings with an ISF coach for the two BI staff and one or more leadership staff from the organization. Organizations that received both the ATTC and ISF strategies delivered more BIs and higher quality BIs than organizations that only received the ATTC strategy. In addition, clients receiving BIs at organizations that received both strategies were more likely to decrease their substance use. However, receiving both strategies did not improve how quickly staff learned to deliver the BI or improve the number of BIs delivered during the project’s 6-month sustainment phase. Future research focused on implementing BIs within HIV organizations should consider using the ATTC and ISF strategies while also seeking to enhance the strategies to improve sustainment.

Substance use among people with HIV is a significant public health issue given it has been found to be associated with increased psychiatric problems (Gaynes et al., 2008), poorer HIV viral suppression (Arnsten et al., 2002; King et al., 2009; Lucas et al., 2001), poorer HIV medication adherence (Azar et al., 2010; Friedman et al., 2009; Hendershot et al., 2009; Malta et al., 2008), and increased likelihood of engaging in risk behaviors that result in infection transmission to others (Hutton et al., 2019; Palepu et al., 2003; Satre et al., 2020). Increasing its public health significance further, research suggests approximately half of the people with HIV have a substance use disorder (SUD) (Hartzler et al., 2017).

Complementing HIV primary care, HIV community-based organizations (hereafter HIV organizations) provide medical and non-medical case management services and are a major source of care for people with HIV (U.S. Department of Health and Human Services, Health Resources and Services Administration, Bureau of Primary Health Care, 2017). Thus, in 2014, as part of its effort to help improve the integration of substance use services within HIV service settings, the National Institute on Drug Abuse funded the Substance Abuse Treatment to HIV care project. The project’s primary aim was to test the Implementation and Sustainment Facilitation (ISF) strategy as an adjunct to the Addiction Technology Transfer Center (ATTC) strategy for helping HIV organizations and their staff integrate a motivational interviewing-based brief intervention (MIBI) for SUDs. Given the importance of context (Aarons et al., 2011; Damschroder et al., 2009) and given research on the effectiveness of MIBIs for SUDs in HIV settings was limited (Aharonovich et al., 2006, 2012; Hasin et al., 2013), the project also examined the impact of the ISF strategy on intervention effectiveness (the effectiveness of the MIBI on improving client-level outcomes) (Garner, Gotham, et al., 2017; Helfrich et al., 2007; Klein et al., 2001; Klein & Sorra, 1996; Weiner et al., 2009). Written in accordance with both the Consolidated Standards of Reporting Trials (CONSORT) guidelines for cluster-randomized trials (Campbell et al., 2012) (see Supplemental File 1 for checklist) and the Standards for Reporting Implementation Studies (StaRI) guidelines (Pinnock et al., 2017) (see Supplemental File 2 for checklist), this article presents the main findings from the project.

Rationale for trial design

We used a cluster-randomized design (HIV organizations were the unit of randomization) to minimize the likelihood of contamination across the project’s two implementation conditions and because cluster-randomized designs had been noted as being preferred over other designs, including stepped-wedge designs (Kotz et al., 2012a, 2012b; Mdege et al., 2012). However, beyond the use of a cluster-randomized design, we used a type 2 hybrid trial design given Curran et al.’s (2012) recommendation of it as an innovative design “in support of more rapid translation” and to “provide more valid estimates of potential clinical effectiveness.”

Rationale for testing a MIBI for SUDs as an adjunct to usual care within HIV organizations

As highlighted by DiClemente et al. (2017), multiple reviews have supported the efficacy and effectiveness of MIBIs for reducing alcohol use (Kaner et al., 2009; Lundahl et al., 2010; McQueen et al., 2011; Stewart, 2012) and cannabis use (Baker et al., 2009, 2010; Dennhardt & Murphy, 2013; Lundahl et al., 2010). However, supporting our rationale for integrating a MIBI for SUDs within HIV service settings was research conducted within HIV service settings and found MIBIs can be effective for reducing alcohol use (Aharonovich et al., 2006; Hasin et al., 2013) and the use of other substances (Aharonovich et al., 2012).

To help maximize the external validity of the project and its findings, we aimed to examine the effectiveness of the project’s MIBI for SUD as an adjunct to usual care (UC) within HIV organizations (UC + MIBI compared with UC only). Regarding UC within HIV organizations, we found standardized substance use screening was rare, with it being even rarer for HIV organizations to employ staff adequately trained to address substance SUDs. Rather, we found UC for SUDs within HIV organizations was primarily referral to treatment.

Rationale for testing the ISF strategy as an adjunct to the ATTC strategy

The combination of staff training, performance feedback, and coaching has been found to be one of the most effective strategies for helping prepare individuals to implement MIBIs with proficiency (Barwick et al., 2012; Darnell et al., 2016; de Roten et al., 2013; Madson et al., 2009, 2019; Martino, 2010; Miller et al., 2004). The ATTC Network has long used this multifaceted strategy to help addiction treatment organizations improve the integration of motivational interviewing for SUDs. As such, the ATTC strategy was identified as one of the most promising strategies for helping HIV organizations and their staff integrate the project’s MIBI. However, given implementation and sustainment are acknowledged as multilevel processes (Aarons et al., 2011; Proctor et al., 2009), the ATTC strategy, which mostly focuses on individual staff training (i.e., staff-focused), was hypothesized to be necessary but not sufficient. Thus, building upon research that identified facilitation as a promising strategy (Baskerville et al., 2012; Cully et al., 2012; Gustafson et al., 2013; Harvey et al., 2002; Kauth et al., 2010; Kitson et al., 2008; Liddy et al., 2011; Owen et al., 2013; Parchman et al., 2013; Seers et al., 2012; Stetler et al., 2006), we aimed to test the ISF strategy, which focuses on training the staff in MIBI and the organization’s leadership (i.e., team-focused) to support MIBI implementation.

As detailed by Garner, Zehner, et al. (2017), the ISF strategy is a multifaceted strategy with facilitation as the overarching approach, encompassing six additional discrete strategies. Grounded in the theory of implementation effectiveness (Helfrich et al., 2007; Klein et al., 2001; Klein & Sorra, 1996; Weiner et al., 2009), the ISF strategy seeks to improve implementation effectiveness (the consistency and the quality of implementation of the clinical intervention) and intervention effectiveness (the clinical intervention’s effectiveness in terms of improving client outcomes) through improving implementation climate (the extent to which implementation is expected, supported, and rewarded). Guided by the Exploration-Preparation-Implementation-Sustainment (EPIS) framework (Aarons et al., 2011), we sought to expand the theory of implementation effectiveness in two ways. Specifically, by examining the extent to which the ISF strategy would help decrease staff time-to-proficiency and increase staff level-of-sustainment. In addition, the ISF strategy was grounded in motivational interviewing principles (Wagner & Ingersoll, 2012), which is similar to how Kauth et al. (2010) employed motivational interviewing techniques as part of their multifaceted facilitation strategy for improving the implementation of cognitive behavioral therapy. Thus, as part of each ISF strategy meeting, the ISF facilitator attempted to (1) engage the implementation team, (2) help focus the implementation team on the project’s key goal(s), (3) help evoke from the implementation team’s pros and cons related to the project’s key goal(s), and (4) help the implementation team plan how best to achieve the project’s key goals and sustain those achievements over time.

Primary aims and hypotheses

The primary aim of the project was to test the ISF strategy as an adjunct to the ATTC strategy for helping HIV organizations and their staff integrate a MIBI for SUDs. Guided by the theory of implementation effectiveness (Helfrich et al., 2007; Klein et al., 2001; Klein & Sorra, 1996; Weiner et al., 2009) and the EPIS framework (Aarons et al., 2011), we hypothesized that the ISF strategy would have significant impacts on three staff-level outcome measures (see Figure 1). In addition, as detailed by MacKinnon (2011), integrating moderators into research design is important to understand the generalizability by first examining the extent to which there are any differential effects that would impede interpretation of a main effect. Thus, consistent with the decomposed-first strategy (Preacher et al., 2016), we started with moderation-focused hypotheses to avoid biases associated with conflated effects. We hypothesized that the impact of the ISF strategy on these staff-level outcomes would be moderated by the components of the Consolidated Framework for Implementation Research’s (CFIR) (Damschroder et al., 2009) characteristics of individuals domain (prior experience with motivational interviewing, personal recovery status) and inner setting domain (implementation climate, implementation readiness, and leadership engagement). Finally, we hypothesized that the ISF strategy would moderate the effect of MIBI on client outcomes.

Figure 1.

Aims and hypotheses.

Note. ATTC = Addiction Technology Transfer Center; ISF = implementation and sustainment facilitation; MIBI = motivational interviewing-based brief intervention; UC = usual care. Bolded lines indicate primary aim and hypotheses; thin line indicates other aim; and dashed lines indicate hypothesized moderators.

Methods

Trial design

The trial design was a cluster-randomized, type 2 hybrid trial. Following an exploration phase in which HIV organizations were recruited, HIV organizations (and their staff) were randomized to one of the two strategies: (1) the ATTC strategy or (2) the ATTC + ISF strategy. Following randomization, the trial was deployed using a multiphase design that included three 6-month phases corresponding to the preparation, implementation, and sustainment phases of the EPIS framework (Aarons et al., 2011). During the implementation phase, HIV organizations recruited and randomized client participants to one of the two clinical intervention conditions: UC or UC + MIBI. Institutional Review Board (IRB) approval and oversight of all research activities were provided by RTI International’s IRB.

Context

HIV organizations, located in 23 states and the District of Columbia within the United States, provided the context for the project.

Participants

Staff participants

To be eligible to participate, an HIV organization was required to serve a minimum of 100 individuals living with HIV per year; have at least two staff members willing to be prepared to implement a MIBI for SUDs; and have at least one leadership staff member (e.g., supervisor, manager, director) willing to help ensure that MIBI staff were given sufficient time to participate. There were no exclusion criteria. Each collaborating HIV organization identified two staff to be prepared to implement the MIBI as part of the project’s implementation phase and to be recruited for participation in staff surveys. Each HIV organization also identified one to three leadership staff to be recruited for participation in staff surveys. After staff provided informed consent, which was obtained electronically, staff completed surveys prior to randomization (before the preparation phase), after the implementation phase (Month 13), and after the sustainment phase (Month 19), and received a US$25 e-gift card per survey. For more details, see the study protocol paper (Garner, Zehner, et al., 2017).

Client participants

Client eligibility was assessed by HIV organization staff through the project’s standardized screener. Eligibility criteria included having been diagnosed with HIV; being above 18 years of age; and acknowledging the use of at least one substance in the past 28 days with self-reported endorsement of two or more of the 11 Diagnostic and Statistical Manual of Mental Disorders (DSM-5) criteria (American Psychiatric Association, 2013) for SUD for that substance during the past 12 months. An exclusion criterion was not being able to speak English, which was due to the project’s research staff and MIBI proficiency raters being monolingual. Clients who met eligibility criteria were recruited for study participation by one of the several trained HIV organization staff. It was one of the two MIBI staff members from each HIV organization who was trained to obtain written informed consent, administer the baseline assessment, follow-up locator form, and open the randomization envelope with client participants. Each HIV organization was provided compensation to cover staff time to complete these research-related activities. Clients randomized to the UC + MIBI condition received the MIBI at no cost. Clients received a US$20 gift card for completing the baseline assessment and US$20 for completing a 4-week follow-up assessment administered by research staff blinded to all condition assignments.

Implementation strategies

Complementing the comprehensive descriptions provided as part of the open-access study protocol paper (Garner, Zehner, et al., 2017) and information provided at www.ISFstrategy.org, Table 1 defines and specifies the 10 discrete strategies in the ATTC strategy, as well as the 7 discrete strategies in the ISF strategy. For the ATTC strategy, the HIV organization’s two recipient MIBI staff were given the opportunity to receive 12 months of MIBI training and technical assistance: training (5-hr online didactics and 2-day in-person workshop), performance feedback (standardized feedback on one to three MIBIs during the preparation phase and standardized feedback on all MIBIs during the implementation phase), and consultation (up to three 1-hr individual consultation calls during the preparation phase and monthly 1-hr group consultation calls during the implementation phase). Thus, the maximum possible dose of the ATTC strategy was 30 hr per MIBI staff. For the ISF strategy, the HIV organization’s recipient MIBI staff and leadership staff were given the opportunity to receive 18 months of external facilitation led by one of the project’s ISF facilitators (monthly virtual meetings lasting up to 60 min, up to two in-person facilitation meetings lasting up to 6 hr each). Thus, the maximum possible dose of the ISF strategy was 30 hr for each of the HIV organization’s staff working on the project. To maximize the extent to which the ISF strategy was implemented with consistency and quality, the project’s lead developer of the ISF strategy trained each ISF facilitator, reviewed randomly selected ISF session recordings (each virtual ISF facilitation meeting was video-recorded for the quality assurance purposes), and regularly supervised the ISF facilitators (no less than monthly, usually weekly).

Table 1.

Discrete strategies included within the ATTC strategy and the ISF strategy.

Blended strategy Specification regarding dose	Discrete strategy (definition) Specification regarding actor, action, action target, and temporality
ATTC strategy  Dose—up to 30 hr (5 hr for the online training course; 16 hr for the 2-day in-person training workshop; 3 hr for up to three 1-hr individual consultation calls; and 6 hr for up to six 1-hr group consultation calls).	Centralized technical assistance (develop and use a system to deliver technical assistance focused on implementation issues).  Actor—regional ATTC (e.g., Mid-America, Northwest, Northeast).  Action—the overarching discrete implementation strategy that encompasses the other discrete implementation strategies listed below.  Action target—two MIBI staff per HIV organization.  Temporality—the initial kickoff meeting should be within 1 month of completing the exploration phase.
	Develop educational materials (develop and format guidelines, manuals, toolkits, and other supporting materials in ways that make it easier for stakeholders to learn about the innovation and for clinicians to learn how to deliver the clinical innovation).  Actor—regional ATTC (e.g., Mid-America, Northwest, Northeast).  Action—the MIBI protocol manual, which provides information and knowledge about how the MIBI is intended to be implemented.  Action target—two MIBI staff per HIV organization.  Temporality—finalization of educational materials (e.g., MIBI protocol manual) should be prior to the initial kickoff meeting.
	Develop and organize quality monitoring system (develop and organize systems and procedures that monitor clinical processes and/or outcomes for quality assurance and improvement).  Actor—regional ATTC (e.g., Mid-America, Northwest, Northeast).  Action—a web-based system (sat2hivproject.org) that enables secure and efficient sharing of data relevant to the evidence-based practice preparation and implementation process.  Action target—two MIBI staff per HIV organization.  Temporality—finalization of quality monitoring systems (i.e., sat2hivproject.org) should be prior to the initial kickoff meeting.
	Develop tools for quality monitoring (develop, test, and introduce quality monitoring tools with inputs [e.g., measures] specific to the innovation being implemented).  Actor—regional ATTC (e.g., Mid-America, Northwest, Northeast).  Action—the Independent Tape Rater Scale, which enables reliable and valid rating of the extent to which staff deliver the evidence-based practice with fidelity.  Action target—two MIBI staff per HIV organization.  Temporality—finalization of tools for quality monitoring (i.e., Independent Tape Rater Scale) should be prior to the initial kickoff meeting.
	Distribute educational materials (distribute educational materials [e.g., manuals] in-person, by mail, and/or electronically).  Actor—regional ATTC (e.g., Mid-America, Northwest, Northeast).  Action—distribute professionally printed copies of the MIBI protocol manual to each MIBI staff.  Action target—two MIBI staff per HIV organization.  Temporality—distribute at the workshop training.
	Conduct educational meetings (hold meetings targeted toward providers, administrators, other organizational stakeholders, and community, patient or consumer, and family stakeholders to teach them about the clinical innovation).  Actor—regional ATTC (e.g., Mid-America, Northwest, Northeast).  Action—in-person and web-based meetings that enable direct interaction between the actors (ATTC) and targeted users of the evidence-based practice (MIBI staff).  Action target—two MIBI staff per HIV organization.  Temporality—educational meetings should begin at least 3 months before the implementation phase begins.
	Make training dynamic (vary the information delivery methods to cater to different learning styles and work contexts and shape the training in the innovation to be interactive).  Actor—regional ATTC (e.g., Mid-America, Northwest, Northeast).  Action—incorporate standardized role plays that enable MIBI staff to practice with each other and that facilitate understanding of the evidence-based practice from both staff and client perspectives.  Action target—two MIBI staff per HIV organization.  Temporality—should begin during the first contact.
	Audit and provide feedback (collect and summarize clinical performance data over a specified period and give data to clinicians and administrators in the hopes of changing provider behavior).  Actor—regional ATTC (e.g., Mid-America, Northwest, Northeast).  Action—generate and email standardized feedback reports to MIBI staff using the standardized quality monitoring tool (i.e., Independent Tape Rater Scale).  Action target—two MIBI staff per HIV organization.  Temporality—should begin approximately 1–2 weeks following the end of the in-person educational training workshop.
	Provide ongoing consultation (provide clinicians with continued consultation with an expert in the clinical innovation).  Actor—regional ATTC (e.g., Mid-America, Northwest, Northeast).  Action—phone-based individualized meetings that enable direct contact between the ATTC trainer and one MIBI staff.  Action target—two MIBI staff per HIV organization.  Temporality—should begin approximately 1–2 weeks following the end of the in-person educational training workshop.
	Create a learning collaborative (develop and use groups of providers or provider organizations that will implement the clinical innovation and develop ways to learn from one another to foster better implementation).  Actor—regional ATTC (e.g., Mid-America, Northwest, Northeast).  Action—web-based group meetings that enable direct contact between the ATTC trainer and a group of 10–14 MIBI staff, who can share lessons learned.  Action target—two MIBI staff per HIV organization.  Temporality—should begin approximately 3–4 weeks after the implementation phase begins.
ISF strategy  Dose—up to 30 hr (18 hr for up to 18 monthly virtual facilitation meetings lasting up to 1 hr each; 12 hr for up to two in-person facilitation meetings lasting up to 6 hr each).	External Facilitation (seek guidance from experts in implementation, including consultation with outside experts [e.g., university-affiliated faculty members, quality improvement experts, and implementation professionals]).  Actor—an individual with training and experience in assisting organizations with practice improvement and implementation efforts.  Action—the overarching implementation strategy that encompasses the other discrete implementation strategies listed below.  Action target—the HIV organization’s designated team of staff working on the project (two MIBI staff and two to four leadership staff).  Temporality—the initial kickoff meeting should be held within 1 month of completing the exploration phase.
	Develop tools for quality improvement (develop, test, and introduce quality improvement tools with inputs [e.g., measures] specific to the innovation being implemented).  Actor—an individual with training and experience in assisting organizations with practice improvement and implementation efforts.  Action—Decisional Balance Exercise; Performance Review, Evaluation, and Planning Exercise; Implementation Climate Evaluation Exercise.  Action Target—the HIV organization’s designated team of staff working on the project (two MIBI staff and two to four leadership staff).  Temporality—finalization of tools for quality improvement should be prior to the initial kickoff meeting.
	Organize implementation team meetings (develop and support teams of clinicians who are implementing the innovation and give them protected time to reflect on the implementation effort, share lessons learned, and support one another’s learning).  Actor—an individual with training and experience in assisting organizations with practice improvement and implementation efforts.  Action—meetings that enable direct interaction between ISF staff and the HIV organization’s staff working on the project.  Action target—the HIV organization’s designated team of staff working on the project (two MIBI staff and two to four leadership staff).  Temporality—first implementation team meeting should be held within 1 month of completing the exploration phase.
	Identify and prepare champions (cultivate relationships with people who will champion the clinical innovation and spread the word of the need for it).  Actor—an individual with training and experience in assisting organizations with practice improvement and implementation efforts.  Action—learning about and engaging with the HIV organization’s staff working on the project.  Action target—the HIV organization’s designated team of staff working on the project (two MIBI staff and two to four leadership staff).  Temporality—identification and preparation of champions should begin during the process of organizing the initial implementation team meeting.
	Assess for readiness and identify barriers (assess various aspects of an organization to determine its degree of readiness to implement, barriers that may impede implementation, and strengths that can be used in the implementation effort).  Actor—an individual with training and experience in assisting organizations with practice improvement and implementation efforts.  Action—utilization of the ISF exercises described above.  Action target—the HIV organization’s designated team of staff working on the project (two MIBI staff and two to four leadership staff).  Temporality—assessments of readiness and identification of barriers should begin during the process of organizing the initial implementation team meeting.
	Conduct local consensus discussions (include providers and other stakeholders in discussions that address whether the chosen problem is important and whether the clinical innovation to address it is appropriate).  Actor—an individual with training and experience in assisting organizations with practice improvement and implementation efforts.  Action—completion of an in-person, stakeholder engagement and sustainment-planning meeting.  Action target—the HIV organization’s designated team of staff working on the project (two MIBI staff and two to four leadership staff).  Temporality—should be held as soon as possible after the first implementation month has been completed.
	Conduct cyclical small tests of change (implement changes in a cyclical fashion using small tests of change).  Actor—an individual with training and experience in assisting organizations with practice improvement and implementation efforts.  Action—completion of study-act-plan-do cycles.  Action target—the HIV organization’s designated team of staff working on the project (two MIBI staff and two to four leadership staff).  Temporality—should begin as soon as necessary.

Note. See the study protocol paper (Garner, Zehner, et al., 2017) for justification of each discrete strategy. ATTC = Addiction Technology Transfer Center; ISF = implementation and sustainment facilitation; MIBI = motivational interviewing-based brief intervention.

Clinical interventions

UC consisted of referral to formal addiction treatment, mutual help services, or both. Clients randomized to UC + MIBI received the project’s 20- to 30-min MIBI for SUD provided by one of the HIV organization’s prepared/trained MIBI staff. The MIBI was designed to motivate individuals living with HIV who have an SUD to change their substance use by examining their reasons for change, receiving feedback about common negative interactions of substance use and HIV-related health issues, further developing the importance or confidence to reduce or stop their primary substance use, and making a plan for change. For more details, see the study protocol paper (Garner, Gotham, et al., 2017).

Outcome measures

Organized by phase (preparation phase, implementation phase, and sustainment phase), Table 2 details the staff-level outcome measures (time-to-proficiency, implementation effectiveness, and level-of-sustainment) and client-level outcome measures (days of primary substance use, number of substance-related problems, times engaging in risky behaviors, days of substance use treatment, and days of medication non-adherence) collected.

Table 2.

Outcome measures.

Measure name (phase)	Measure description
1. Time-to-proficiency (preparation phase)	A staff-level measure of the number of weeks between MIBI staff completing the in-person training and demonstrating proficiency in the project’s MIBI for SUDs. Developed for this project based on research by Saldana (Saldana et al., 2012). Proficiency was determined by one of the project’s MI experts, who rated audio recordings of MIBI practice sessions using the Independent Tape Rater Scale (Martino et al., 2008). The Independent Tape Rater Scale is used to rate 10 MI-consistent items for adherence and competence along seven-point scales. MIBI proficiency was demonstrated when a single MIBI session had half the items rated at the mid-point or higher for both adherence and competence items.
2. Implementation effectiveness (implementation phase)	A staff-level measure of the overall consistency and the quality of MIBI implementation during the project’s implementation phase. Developed for this project based on research by Klein et al. (2001). First, the cumulative number of MIBIs implemented was summed and standardized for each MIBI staff (MIBI consistency). Second, the MIBI proficiency scores were summed and standardized for each MIBI staff (MIBI quality). Proficiency of each MIBI session was assessed by the project’s cadre of raters who were trained, calibrated, and supervised by one of the project’s MI experts. A quality score was calculated for each MIBI session by multiplying the corresponding adherence rating (ranged from 1 to 7) and competence rating (ranged from 1 to 7) and summing for all 10 of the MI-consistent items (ranged from 10 to 490). Finally, MIBI consistency and MIBI quality scores were summed and standardized.
3. Intervention effectiveness (implementation phase) regarding:
3.1. Days of primary substance use	A client-level measure of the number of days client participants self-reported using their primary substance during the past 28 days and measured using a modified version of the Addiction Severity Index (McLellan et al., 1992). Client’s primary substance was identified at baseline by asking “Of the substances that you have used in the past 4 weeks (not including tobacco), which one has been the biggest problem for you OR caused you the most problems?” At follow-up, client participants were reminded of the substance they had indicated was their primary substance.
3.2. Number of substance-related problems	A client-level measure of the number of the 11 DSM-5 (American Psychiatric Association, 2013) SUD symptoms client participants self-reported recognition of regarding their primary substance and during the past 28 days. At follow-up, client participants were reminded of the substance they had indicated was their primary substance.
3.3. Times engaging in risky behaviors	A client-level measure of the number of times client participants self-reported engaged in unprotected sex, injection drug use, or needle sharing during the past 28 days, which was developed for this project based on the Addiction Severity Index (McLellan et al., 1992).
3.4. Days of substance use treatment	A client-level measure of the number of days client participants self-reported having attended residential treatment, outpatient treatment, or self-help group meetings during the past 28 days, which was developed for this project based on the Addiction Severity Index.
3.5. Days of HIV medication non-adherence	A client-level measure of the number of days client participants self-reported having missed at least one dose of their HIV medications during the past 28 days, which was developed for this project based on the on the Addiction Severity Index (McLellan et al., 1992).
4. Level-of-sustainment (sustainment phase)	A staff-level measure of the number of MIBIs delivered during the project’s 6-month sustainment phase and measured through MIBI staff self-report as part of the project’s sustainment phase survey.

Note. Intervention effectiveness was assessed in terms of the impact of the ISF strategy on each of the client outcome measures. MIBI = motivational interviewing-based brief intervention; SUD = substance use disorder; MI = motivational interviewing; DSM-5 = Diagnostic and Statistical Manual of Mental Disorders, 5th ed.

Moderator measures

Table 3 details the staff-level measures (implementation readiness, implementation climate, leadership engagement, tension-for-change, motivational interviewing experience, and personal recovery status) hypothesized to moderate the impact of the ISF strategy on the staff-level outcomes.

Table 3.

Staff-level moderator measures.

Measure name	Measure description
Motivational interviewing experience	A baseline measure of MIBI staff perception of their motivational interviewing experience. Measured by having MIBI staff, select one of the five response options (none, beginner, intermediate, advanced, or expert).
Personal recovery status	A baseline measure indicating if MIBI staff considered themselves to be in recovery from alcohol/drugs (0 = no; 1 = yes).
Implementation readiness	A baseline measure representing MIBI staff perception regarding their organization’s readiness for implementing a brief intervention for SUDs. Measured using the average of 12 items developed by Shea et al. (2014). Each item (e.g., staff working on this project want to implement this change; staff working on this project are committed to implementing this change; staff working on this project will do whatever it takes to implement this change) was measured on a five-point scale (1 = Disagree, 2 = Somewhat Disagree, 3 = Neither Agree nor Disagree, 4 = Somewhat Agree, 5 = Agree). Coefficient alpha = .95.
Implementation climate	A baseline measure representing MIBI staff perception regarding the extent to which implementing brief intervention for SUDs is expected, supported, and rewarded within their organization. Measured using the average of the six items developed by Jacobs (Jacobs et al., 2014). Each item (e.g., staff working on this project are expected to use brief intervention for substance use with a certain number of clients; staff working on this project get the support they needed to use brief intervention for substance use with eligible clients; staff working on this project receive recognition for using brief intervention for substance use with eligible clients) was measured on a five-point scale (1 = Disagree, 2 = Somewhat Disagree, 3 = Neither Agree nor Disagree, 4 = Somewhat Agree, 5 = Agree). Coefficient alpha = .91.
Leadership engagement	A baseline measure representing MIBI staff perception regarding the extent to which their HIV organization leadership is committed, involved, engaged, and accountable for implementing brief intervention for SUDs. Measured using the four items developed for this project based on the leadership engagement construct described by Damschroder (Damschroder et al., 2009). Each item (to what extent was the leadership of this organization committed to the implementation of brief intervention for substance use; to what extent was the leadership of this organization involved in the implementation of brief intervention for substance use; to what extent was the leadership of this organization engaged in the implementation of brief intervention for substance use; to what extent was the leadership of this organization accountable for the implementation of brief intervention for substance use) was measured on seven-point scale (0 = not at all to 6 = highest extent possible). Coefficient alpha = .94.
Tension-for-change	A baseline measure representing MIBI staff perception regarding the extent to which implementing a brief intervention for SUDS is important, needed, and desired. Measured using the three items developed for this project based on the tension-for-change construct described by Damschroder (Damschroder et al., 2009). Each item (to what extent do staff working on this project believe implementation of brief intervention for substance use is important; to what extent do staff working on this project believe implementation of brief intervention for substance use is needed; to what extent do staff working on this project believe implementation of brief intervention for substance use is desired) was measured on seven-point scale (0 = not at all to 6 = highest extent possible). Coefficient alpha = .92.

Note. MIBI = motivational interviewing-based brief intervention; SUD = substance use disorder.

Targeted sample size

The targeted sample size was estimated through power analyses with Optimal Design Software (Raudenbush et al., 2011). For analyses of staff-level outcomes, it was estimated that 78 MIBI staff nested within 39 HIV organizations would provide 80% power to detect a statistically significant (p < .05) difference for effect sizes .67 or greater (Garner, Zehner, et al., 2017). For analyses of client-level outcomes, it was estimated that 1,872 clients, nested within 78 MIBI staff members, nested within 39 HIV organizations would provide 80% power to detect a statistically significant difference for effect sizes .20 or greater (Garner, Gotham, et al., 2017).

Randomization sequence generation

For randomization of HIV organizations (the clusters), each HIV organization was allocated to one of the two implementation strategy conditions (ATTC; ATTC + ISF) through an urn randomization process (Stout et al., 1994). Specifically, using survey data collected during the exploration phase from HIV organization staff, seven organizational-level factors (importance of substance use screening, importance of brief intervention for substance use, innovation-value fit, implementation strategy-value fit, implementation climate for MIBI, implementation readiness for MIBI, and implementation effectiveness for MIBI) were entered into an urn randomization program (Charpentier, 2003) that optimized the balance of the two implementation strategy conditions based on these factors.

During the project’s implementation phase, HIV organizations randomized client participants to one of the two intervention conditions (UC; UC + MIBI) through a blocked randomization sequence (blocking size of 6) generated through a blocked randomization program (Sealed Envelope, n.d.). Within each participating HIV organization, each MIBI staff had a lock box containing 36 sequentially numbered tamper-evident security envelopes containing a randomization slip indicating condition assignment. The randomization envelope was opened in front of the client participant. Staff updated a centralized recruitment tracking log monitored multiple times per week by research staff.

Blinding

It was not possible to blind HIV organizations and their staff to the assigned implementation strategy condition, but the project’s ATTC strategy staff and quality raters were blinded to implementation strategy condition assignment. In addition, it was not possible to blind HIV organizations, their staff, or client participants to clinical intervention condition assignment, but the project’s research staff who conducted the follow-up assessments were blinded to all condition assignments.

Statistical methods

Statistical analyses were conducted using an intention-to-treat approach. Staff-level outcomes were approximately normal, and within-site variation was close to zero. A series of multilevel adjusted analyses was conducted, each of which controlled for project cohort and was weighted through a propensity score weight derived by regressing implementation strategy condition assignment on staff characteristics. The interaction between the implementation strategy condition assignment and each hypothesized moderator was examined first, with main effects examined as appropriate.

Client-level outcomes had strong floor effects (0 of 28 days) and/or strong ceiling effects (28 of 28 days), which led to bimodal u-shaped, j-shaped, or inverted j-shaped distributions. Given these non-normal distributions, linear regression analyses were not appropriate. Rather, these types of distributions are appropriately addressed using zero-and-one inflated beta regression after data are transformed to a proportion scale (0 to 1). This model is a mixture model with three parts: a prediction of the probability of the ceiling effect vs. other values (the ceiling effect), a prediction of the mean for values in between, but not including, the floor and ceiling effect (non-ceiling/non-floor effect), and a prediction of the probability of the floor effect vs. other values (the floor effect). We fit three-level multilevel models to account for the nesting of client participants within MIBI staff and MIBI staff within HIV organizations using the R package (Liu & Kong, 2015). Each model was adjusted for the baseline value of the respective outcome measure, client characteristics (i.e., age, White, male, heterosexual, transgender, married, high school or higher, alcohol as primary substance, and engagement in HIV care), project cohort, randomization to ATTC + ISF condition, randomization to UC + MIBI condition, and the cross-level interaction between ATTC + ISF condition and UC + MIBI condition.

Results

Participant flow and recruitment

HIV organizations were recruited in three cohorts, each lasting 20 months from the randomization of HIV organizations to the final data collection. The first cohort, in the central United States, occurred from January 2015 through August 2016 and resulted in the recruitment of 14 HIV organizations, 28 MIBI staff, and 191 client participants. The second cohort, in the western United States, occurred from January 2016 through August 2017 and resulted in the recruitment of 11 HIV organizations, 22 MIBI staff, and 300 client participants. The third cohort, in the eastern United States, occurred from January 2017 through August 2018 and resulted in the recruitment of 14 HIV organizations, 28 MIBI staff, and 333 clients. Figure 2 details the flow of HIV organizations, MIBI staff, and client participants through the project’s preparation, implementation, and sustainment phases. Although similar numbers of staff were recruited across conditions, about twice as many clients were screened, enrolled, and randomized within the ATTC + ISF condition.

Figure 2.

Participant flow.

Note. ATTC = Addiction Technology Transfer Center; IQR = interquartile range; ISF = implementation and sustainment facilitation; MIBI = motivational interviewing-based brief intervention; UC = usual care.

Baseline characteristics

Table 4 presents baseline characteristics for MIBI staff participants for the overall sample (N = 78) and each condition (ATTC = 38; ATTC + ISF = 40). Overall, MIBI staff participants were 25–34 years of age (46%), female (71%), Caucasian/White (62%), a graduate degree or higher (50%), 12 months or less tenure with current HIV organization (35%), and intermediate motivational interviewing experience or higher (53%).

Table 4.

Staff characteristics at baseline.

	Overall (n = 78)		ATTC (n = 38)		ATTC + ISF (n = 40)
	n	%	n	%	n	%
Age (years)
18–24	3	3.8	2	5.3	1	2.5
25–34	36	46.2	12	31.6	24	60.0
35–44	16	20.5	9	23.7	7	17.5
45–54	14	17.9	9	23.7	5	12.5
55–64	9	11.5	6	15.8	3	7.5
Biological sex
Female	55	70.5	26	68.4	29	72.5
Male	23	29.5	12	31.6	11	27.5
Hispanic or Latino	16	20.5	11	28.9	5	12.5
Race
African American/Black	27	34.6	14	36.8	13	32.5
Asian	3	3.8	1	2.6	2	5.0
Caucasian/White	48	61.5	23	60.5	25	62.5
Graduate degree or higher	39	50.0	16	42.1	23	57.5
Experience in current profession (months)
<12	14	17.9	6	15.8	8	20.0
13–24	10	12.8	2	5.3	8	20.0
25–60	15	19.2	7	18.4	8	20.0
81–120	18	23.1	10	26.3	8	20.0
>121	21	26.9	13	34.2	8	20.0
Tenure at current organization (months)
<12	27	34.6	17	44.7	10	25.0
13–24	20	25.6	6	15.8	14	35.0
25–60	15	19.2	6	15.8	9	22.5
81–120	9	11.5	4	10.5	5	12.5
>121	7	9.0	5	13.2	2	5.0
Moderator measures
Intermediate MI experience or higher	41	52.6	22	57.9	19	47.5
In recovery for alcohol or drugs	11	14.1	8	21.1	3	7.5
Readiness for implementing change, M (SD)	78	2.9 (1.4)	38	3.3 (1.4)	40	2.6 (1.4)
Implementation climate, M (SD)	78	2.8 (1.1)	38	3.0 (1.1)	40	2.7 (1.1)
Leadership engagement, M (SD)	78	3.7 (1.7)	38	3.8 (1.6)	40	3.5 (1.7)
Tension-for-change, M (SD)	78	4.4 (1.4)	38	4.5 (1.3)	40	4.4 (1.4)

Note. ATTC = Addiction Technology Transfer Center; ISF = implementation and sustainment facilitation; MI = motivational interviewing; SD = standard deviation.

Table 5 presents the baseline characteristics for client participants for the overall sample (N = 824) and each condition (ATTC and UC = 134; ATTC and UC + MIBI = 130; ATTC + ISF and UC = 281; ATTC + ISF and UC + MIBI = 279). Overall, clients were male (76%), African American/Black (54%), heterosexual (42%), a high school graduate or higher (70%), and engaged in HIV care (95%). Primary substance use for the overall sample was alcohol (37%), cannabis (23%), cocaine/crack (18%), methamphetamine (17%), heroin (2%), and other (3%). On average, client participants reported using their primary substance 16 days during the past 28 days (57% of days). However, 222 (27%) of client participants reported using their primary substance daily during the past 28 days (see Figure 3).

Table 5.

Client characteristics at baseline.

	Overall (n = 824)		ATTC and UC (n = 134)		ATTC and UC + MIBI (n = 130)		ATTC + ISF and UC (n = 281)		ATTC + ISF and UC + MIBI (n = 279)
	N	%	n	%	n	%	n	%	n	%
Age (years)
18–24	46	5.6	5	3.8	3	2.3	16	5.7	22	7.9
25–34	138	16.8	27	20.3	25	19.2	46	16.4	40	14.3
35–44	179	21.7	24	18.0	24	18.5	69	24.6	62	22.2
45–54	287	34.9	51	38.3	43	33.1	99	35.2	94	33.7
55–64	156	19.0	24	18.0	31	23.8	49	17.4	52	18.6
>65	17	2.1	2	1.5	4	3.1	2	0.7	9	3.2
Biological sex
Male	627	76.1	99	73.9	91	70.0	220	78.3	217	77.8
Female	197	23.9	35	26.1	39	30.0	61	21.7	62	22.2
Gender identity
Male	575	69.9	91	67.9	87	66.9	203	72.5	194	69.5
Female	203	24.7	35	26.1	38	29.2	65	23.2	65	23.3
Transgender	45	5.5	8	6.0	5	3.8	12	4.3	20	7.2
Hispanic or Latino	109	13.2	22	16.4	17	13.1	37	13.2	33	11.8
Race
African American/Black	447	54.2	77	57.5	72	55.4	161	57.3	137	49.1
American Indian/Alaska Native	23	2.8	1	0.75	2	1.5	6	2.1	14	5.0
Asian	4	0.5	0	0.0	0	0	3	1.1	1	0.4
Caucasian/White	298	36.2	48	35.8	50	38.5	96	34.5	104	37.6
Native Hawaiian/Other Pacific Islander	10	1.2	0	0.0	1	0.8	2	0.7	7	2.5
More than one	20	2.4	2	1.5	2	1.5	7	2.5	9	3.2
Sexual orientation
Heterosexual	347	42.1	60	44.8	70	53.8	97	34.6	120	43.2
Homosexual, gay, or lesbian	315	38.3	51	38.1	40	30.8	124	44.3	100	36.0
Other	160	19.5	23	17.2	20	15.4	59	21.0	58	20.8
Married	109	13.2	16	11.9	26	20.0	36	12.8	31	11.1
High school graduate or higher	576	70.4	94	71.8	86	66.7	203	72.2	193	69.7
Engaged in HIV care	778	95.2	127	95.5	122	94.6	260	93.5	269	97.1
Primary substance
Alcohol	304	36.9	56	41.8	54	41.5	99	35.2	95	34.0
Cannabis	186	22.6	35	26.1	37	28.5	57	20.3	57	20.4
Cocaine/crack	145	17.6	23	17.2	15	11.5	52	18.5	55	19.7
Methamphetamine	143	17.4	12	9.0	14	10.8	59	21.0	58	20.8
Heroin	20	2.4	2	1.5	6	4.6	7	2.5	5	1.8
Other	26	3.2	6	4.5	4	3.1	7	2.5	9	3.2
Outcome measures	N	M (SD)	n	M (SD)	n	M (SD)	n	M (SD)	n	M (SD)
Days of use	823	15.9 (9.7)	134	16.3 (9.8)	130	16.9 (9.9)	280	15.6 (9.7)	279	15.4 (9.5)
Problem recognition	824	7.3 (3.1)	134	6.9 (3.1)	130	7.0 (3.2)	281	7.4 (3.2)	279	7.4 (3.1)
Risky behaviors	818	3.8 (9.8)	132	2.8 (10.7)	130	3.4 (7.5)	279	4.2 (11.2)	277	4.1 (8.7)
Engagement in SUD treatment	822	2.4 (7.1)	133	3.2 (8.5)	129	2.0 (6.5)	281	2.1 (7.5)	279	2.5 (6.2)
Days of missed HIV medication	740	3.8 (6.3)	126	3.6 (6.2)	120	3.5 (5.8)	240	4.1 (6.4)	254	3.8 (6.4)

Note. ATTC = Addiction Technology Transfer Center; UC = usual care; MIBI = motivational interviewing-based brief intervention; ISF = implementation and sustainment facilitation; SD = standard deviation; SUD = substance use disorder.

Figure 3.

Baseline distribution for client’s days of primary substance use.

Implementation strategy dose

Regarding the ATTC strategy, the average dose (measured in hours) per MIBI staff was 23.85 hr (SD = 1.62) for the ATTC condition and 25.02 hr (SD = 1.63) for the ATTC + ISF condition. Regarding the ISF strategy, the average dose (also measured in hours) was 9.29 hr (SD = 2.83) for MIBI staff and 4.83 hr (SD = 3.09) for leadership staff.

Outcomes

Table 6 summarizes results of analyses focused on testing the extent to which the ISF strategy had an impact on the staff-level outcomes. Consistent with our hypotheses, results of each moderator analysis are presented first, with a main effect analysis reported as appropriate.

Table 6.

Moderator-first analyses of the impact of the ISF strategy on time-to-proficiency, implementation effectiveness, and level-of-sustainment.

	Time-to-proficiency (ICC = .22)			Implementation effectiveness (ICC = .06)			Level-of-sustainment (ICC = .001)
	Estimate [95% CI]	SE	p	Estimate [95% CI]	SE	p	Estimate [95% CI]	SE	p
MI experience
ATTC + ISF	−0.35 [−0.93, 0.23]	0.30	.24	0.72 [0.12, 1.32]	0.30	.02	0.34 [−0.42, 1.10]	0.39	.39
MI experience (intermediate plus)	−0.02 [−0.58, 0.54]	0.29	.95	0.21 [–0.37, 0.79]	0.30	.48	0.18 [−0.56, 0.92]	0.38	.64
ATTC + ISF × MI experience	0.63 [−0.14, 1.40]	0.39	.11	–0.08 [–0.88, 0.72]	0.41	.85	−0.47 [−1.49, 0.55]	0.52	.37
Personal recovery status
ATTC + ISF	0.09 [−0.33, 0.51]	0.21	.67	0.72 [0.29, 1.15]	0.22	.00	0.26 [−0.28, 0.80]	0.28	.34
In recovery	0.35 [−0.39, 1.09]	0.38	.35	–0.01 [–0.77, 0.75]	0.39	.99	0.64 [−0.33, 1.61]	0.49	.20
ATTC + ISF × in recovery	−0.90 [−2.04, 0.24]	0.58	.13	–0.57 [–1.74, 0.60]	0.60	.35	−1.36 [−2.84, 0.12]	0.76	.08
Implementation readiness
ATTC + ISF	0.43 [−0.53, 1.39]	0.49	.38	–0.05 [–1.02, 0.92]	0.50	.92	−0.40 [−1.67, 0.87]	0.65	.54
Implementation readiness	0.15 [−0.06, 0.36]	0.11	.16	–0.20 [–0.41, 0.01]	0.11	.06	−0.08 [−0.35, 0.19]	0.14	.58
ATTC + ISF × implementation readiness	−0.13 [−0.41, 0.15]	0.15	.36	0.21 [–0.08, 0.50]	0.15	.15	0.16 [−0.22, 0.54]	0.19	.41
Implementation climate
ATTC + ISF	0.20 [−0.93, 1.33]	0.58	.73	0.67 [–0.48, 1.82]	0.59	.26	−0.56 [−2.04, 0.92]	0.75	.46
Implementation climate	0.02 [−0.25, 0.29]	0.14	.86	–0.05 [–0.32, 0.22]	0.14	.73	−0.15 [−0.50, 0.20]	0.18	.41
ATTC + ISF × implementation climate	−0.08 [−0.43, 0.27]	0.18	.67	–0.01 [–0.37, 0.35]	0.18	.96	0.21 [−0.25, 0.67]	0.24	.37
Leadership Engagement
ATTC + ISF	0.18 [−0.82, 1.18]	0.51	.72	0.04 [–0.98, 1.06]	0.52	.94	−0.53 [−1.84, 0.78]	0.67	.43
Leadership engagement	0.02 [−0.16, 0.20]	0.09	.84	–0.12 [–0.30, 0.06]	0.09	.21	−0.03 [−0.26, 0.20]	0.12	.82
ATTC + ISF × leadership engagement	−0.06 [−0.31, 0.19]	0.13	.66	0.16 [–0.09, 0.41]	0.13	.20	0.17 [−0.15, 0.49]	0.16	.31
Tension-for-change
ATTC + ISF	0.20 [−1.25, 1.65]	0.74	.79	0.07 [–1.41, 1.55]	0.75	.93	−0.30 [−2.21, 1.61]	0.97	.76
Tension-for-change	−0.01 [−0.24, 0.22]	0.12	.93	–0.01 [–0.25, 0.23]	0.12	.91	−0.10 [−0.40, 0.20]	0.16	.54
ATTC + ISF × Tension-for-change	−0.05 [−0.36, 0.26]	0.16	.76	0.13 [–0.18, 0.44]	0.16	.41	0.08 [−0.32, 0.48]	0.21	.69
Main effect
ATTC + ISF	−0.02 [−0.41, 0.37]	0.20	.91	0.66 [0.26, 1.06]	0.20	.00	0.09 [−0.42, 0.60]	0.26	.74

Note. ISF = implementation and sustainment facilitation; ICC = intracluster correlation; CI = confidence interval; SE = standard error; MI = motivational interviewing; ATTC = Addiction Technology Transfer Center.

For time-to-proficiency, none of the hypothesized moderators were found to be significant. Furthermore, the ISF strategy was not found to have a significant main effect on decreasing time-to-proficiency, β = −.02, 95% confidence interval (CI) = [−0.41, 0.37]. On average, time-to-proficiency was 12.35 days (SD = 3.18) for MIBI staff in the ATTC condition and 11.44 days (SD = 4.87) for MIBI staff in the ATTC + ISF condition.

For implementation effectiveness, none of the hypothesized moderators were found to be significant. However, the ISF strategy was found to have a significant main effect on increasing implementation effectiveness, β = .65, 95% CI = [0.25, 1.05], p < .01. On average, the sum number of MIBIs implemented during the implementation phase (the consistency dimension of staff-level implementation effectiveness) was 3.32 (SD = 4.13) for MIBI staff in the ATTC condition and 6.93 (SD = 5.49) for MIBI staff in the ATTC + ISF condition. On average, the sum quality score of MIBIs (the quality dimension of staff-level implementation effectiveness) was 560 (SD = 780) for MIBI staff in the ATTC condition and 1,324 (SD = 1,054) for MIBI staff in the ATTC + ISF condition.

For level-of-sustainment, none of the six hypothesized moderators were found to be significant, and there was not a significant main effect for the ISF strategy, β = .09, 95% CI = [−0.42, 0.60]. On average, the number of MIBIs implemented during the sustainment phase was 3.42 (SD = 6.31) for MIBI staff in the ATTC condition and 3.18 (SD = 8.33) for MIBI staff in the ATTC + ISF condition.

Table 7 summarizes results of analyses focused on testing the extent to which the ISF strategy had an impact on the effectiveness of the MIBI to improve client outcomes. Consistent with our hypotheses, the cross-level interactions between implementation condition and clinical intervention condition are presented first, with the other key terms presented below.

Table 7.

Cross-level interactions examining the impact of the ISF strategy on the intervention effectiveness of the MIBI.

	Days of primary substance use (ICC = .07)		Number of substance-related problems (ICC = .06)		Times engaging in risky behaviors (ICC = .03)		Days of substance use treatment (ICC = .03)		Days of HIV medication non-adherence (ICC = .01)
	OR [95% CI]	p	OR [95% CI]	p	OR [95% CI]	p	OR [95% CI]	p	OR [95% CI]	p
Ceiling effect
UC + MIBI × ATTC + ISF	0.11 [0.08, 0.15]	.02	0.26 [0.20, 0.36]	.16					•
ATTC + ISF	4.11 [3.25, 5.19]	.04	6.68 [5.39, 8.28]	.01					2.50 [2.07, 3.02]	0.50
UC + MIBI	1.87 [1.44, 2.42]	.39	1.50 [1.18, 1.91]	.62					0.36 [0.29, 0.45]	0.13
Non-ceiling/ Non-floor effect
UC + MIBI × ATTC + ISF	1.03 [0.74, 1.43]	.87	0.86 [0.64, 1.16]	.33	0.62 [0.36, 1.08]	.10	1.39 [0.79, 2.44]	.25	1.18 [0.90, 1.55]	0.23
ATTC + ISF	1.02 [0.81, 1.29]	.86	1.36 [1.09, 1.68]	.01	1.02 [0.66, 1.56]	.93	0.95 [0.65, 1.40]	.81	1.09 [0.90, 1.32]	0.36
UC + MIBI	1.09 [0.84, 1.41]	.53	1.05 [0.82, 1.33]	.72	1.26 [0.78, 2.05]	.35	0.72 [0.45, 1.15]	.17	0.88 [0.70, 1.10]	0.25
Floor effect
UC + MIBI × ATTC + ISF	1.51 [1.09, 2.10]	.58	1.51 [1.12, 2.03]	.73	0.91 [0.52, 1.59]	.86	0.82 [0.47, 1.44]	.62	0.89 [0.68, 1.17]	0.75
ATTC + ISF	2.00 [1.58, 2.53]	.20	0.85 [0.68, 1.05]	.85	0.59 [0.39, 0.91]	.15	1.40 [0.95, 2.05]	.26	1.23 [1.01, 1.48]	0.45
UC + MIBI	1.05 [0.81, 1.37]	.94	0.93 [0.74, 1.19]	.94	1.08 [0.66, 1.74]	.87	1.10 [0.69, 1.74]	.78	1.14 [0.91, 1.43]	0.68

Note. • indicates variable removed to allow model to converge. Ceiling effects were excluded for outcomes without a ceiling effect. Times engaging in risk behavior have no theoretical ceiling. Past 28 days substance use treatment does have an actual ceiling at 28, but it was so infrequently observed in the data that a ceiling effect did not result. ISF = implementation and sustainment facilitation; MIBI = motivational interviewing-based brief intervention; ICC = intracluster correlation; OR = odds ratio; CI = confidence interval; UC = usual care; ATTC = Addiction Technology Transfer Center.

For days of primary substance use (see Figure 4 for the distribution at follow-up), the ISF strategy had a significant impact on the effectiveness of the MIBI, at least in terms of significantly decreasing the odds, (odds ratio [OR] = 0.11, 95% CI = [0.08, 0.15], p = .01) of clients using their primary substance daily during the 28-day follow-up period. To help interpret the size of this effect, an OR of 0.11 is equivalent to an OR of 9.09 (1/0.11 = 9.09), which is considered a large effect (Chen et al., 2010). The ISF strategy increased the odds (OR = 1.51) of clients being completely abstinent from their primary substance at follow-up, but this small effect was not statistically significant. Complementing the results shown in Table 7, Figure 5 helps visualize the cross-level interaction between implementation condition and intervention condition on days of primary substance use.

Figure 4.

Follow-up distribution for client’s days of primary substance use.

Figure 5.

The impact of the ISF strategy on intervention effectiveness.

Note. ISF = implementation and sustainment facilitation.

The ISF strategy was not found to have a significant impact on the effectiveness of the MIBI for the other client outcome measures. However, there were significant effects for the ATTC + ISF strategy on clients’ endorsement of problems related to their primary substance (problem recognition), which is important, yet distinct from the effectiveness of the MIBI (intervention effectiveness). Specifically, the ATTC + ISF strategy increased the odds of client participants endorsing that their primary substance was associated with each of the 11 DSM-5 symptoms (ceiling effect; OR = 6.68, 95% CI = [5.39, 8.28], p = .01) and the number of the 11 DSM-5 symptoms endorsed for client participants without a ceiling/floor effect (OR = 1.36, 95% CI = [1.09, 1.68], p = .01).

Discussion

We used a cluster-randomized, type 2 hybrid trial to simultaneously test the impact of the ISF strategy (as an adjunct to the ATTC strategy) on (1) the integration of a MIBI for SUDs within HIV organizations across the United States and (2) the effectiveness of the MIBI (as an adjunct to UC within HIV organizations). Contributing to the growing literature on the effectiveness of facilitation-based strategies (Chinman et al., 2015, 2017; Jones et al., 2015; Kilbourne et al., 2014, 2015; Kirchner et al., 2014; Lessard et al., 2015; Liddy et al., 2015; Seers et al., 2018) and the effectiveness of MIBIs for SUD within HIV service settings (Aharonovich et al., 2012, 2017; Haldane et al., 2018; Hasin et al., 2014; Kahler et al., 2018; Scott-Sheldon et al., 2017; Wray et al., 2016), we found at least two findings of significance. First, we found evidence that the ISF strategy had a significant impact on improving the consistency and quality of MIBI implementation during the implementation phase (implementation effectiveness). Second, we found evidence that the ISF strategy had a significant impact on improving the effectiveness of the MIBI (intervention effectiveness). However, our main findings also included null results. Indeed, we did not find support for our hypotheses that staff-level measures of the inner setting domain (implementation readiness, implementation climate, leadership engagement, and tension-for-change) and characteristics of individuals domain (motivational interviewing experience and personal recovery status), two of the key CFIR domains (Damschroder et al., 2009), moderated the impact of the ISF strategy. Although these measures were not found to moderate the impact of the ISF strategy, we posit it remains possible one or more of these measures may mediate (i.e., help explain) the impact of the ISF strategy on implementation effectiveness, which has been explicitly hypothesized. As such, subsequent mediational analyses are warranted. In addition, we did not find support for our hypotheses that the ISF strategy would significantly decrease time-to-proficiency and significantly increase the level-of-sustainment. Organized in chronological order along the EPIS continuum (Aarons et al., 2011), below we discuss the limitations, generalizability, and implications of our findings (Campbell et al., 2012; Pinnock et al., 2017).

In terms of time-to-proficiency, we did not find evidence supporting our hypotheses. However, we believe that the potential for the ISF strategy to decrease time-to-proficiency was limited by requiring MIBI staff to demonstrate proficiency before they were allowed to help implement/test the project’s MIBI for SUDs and/or instructing MIBI staff to demonstrate MIBI proficiency sometime before the beginning of the project’s implementation phase, rather than as soon as possible. As such, our findings may or may not generalize to contexts in which there is a stronger justification for staff demonstrating MIBI proficiency as soon as possible (e.g., fee-for-service contexts). In terms of implications, this finding advances knowledge regarding the preparation of staff to implement a MIBI for SUDs and highlights the need for research experimentally testing the extent to which strategies minimize the time to complete key activities (Saldana et al., 2012) and the extent to which the impact of these strategies is significantly moderated by constructs hypothesized to be important (Damschroder et al., 2009).

Consistent with our time-to-proficiency finding, we did not find evidence to support our moderation-focused hypotheses regarding the impact of the ISF strategy on implementation effectiveness. We did, however, find evidence that the ISF strategy significantly improved the average level of implementation effectiveness achieved by MIBI staff. An early indicator of the impact of the ISF strategy on this outcome was the finding that about twice as many clients were screened, enrolled, and randomized within the ATTC + ISF condition (see Figure 1). A potential limitation of this finding is that MIBI staff were asked to limit the number of MIBIs implemented to three per month. This was done to help increase the likelihood that the monthly performance feedback and group consultation provided as part of the ATTC strategy could have an impact on MIBI quality, which is important given implementation effectiveness is defined as the both the consistency (i.e., the number of MIBIs implemented) and quality (i.e., the adherence and competence to the MIBI protocol) of implementation (Garner, Zehner, et al., 2017; Klein et al., 2001; Klein & Sorra, 1996). Although we believe this approach was well-justified, our findings may not generalize to less controlled contexts or contexts in which the quality component of implementation effectiveness is not using the Independent Tape Rater Scale to measure MIBI quality. Nonetheless, a key implication of this finding is that the ISF strategy was a promising adjunct to the ATTC strategy, at least for improving the implementation of our project’s MIBI for SUDs within HIV organizations. Thus, it is recommended that intermediary/purveyor organizations seeking to improve the integration of a MIBI for SUD within HIV organizations, such as the AIDS Education and Training Center network, consider use of the ATTC + ISF strategy for such efforts.

Consistent with prior research highlighting that variation in implementation influences program outcomes (Durlak & DuPre, 2008; Garner et al., 2016), we found that in addition to significantly improving implementation effectiveness, the ISF strategy significantly improved the intervention effectiveness. Notwithstanding the importance of this finding, it is important to note that the ISF strategy’s impact on the effectiveness of the project’s MIBI for SUDs was limited to a single client outcome, days of primary substance use. Unfortunately, less than optimal recruitment and randomization of client participants, which has recently been highlighted as a key potential drawback of type 2 hybrid trials (Landes et al., 2019), limited our power to detect statistically significantly differences in the study’s other client outcome measures. Other noteworthy limitations are that the days of primary substance use and the other client outcome measures were based on client self-report and limited to a 4-week follow-up period. To the best of our knowledge, we are the first to have experimentally tested the impact of an implementation strategy on intervention effectiveness (i.e., the cross-level impact of an implementation strategy on the relative effectiveness of the experimental clinical intervention compared to the control clinical intervention), which is distinct from our prior implementation research that tested the impact of an implementation strategy on client outcomes (i.e., the direct impact of the implementation strategy to improve client outcomes relative to the control implementation strategy; Garner et al., 2012). Thus, the generalizability of our findings may need to be limited to contexts similar to our current study. The key implication of this finding is that future efforts to improve the integration of MIBIs for SUDs within HIV organizations, both implementation research and implementation practice, should consider the use of the ATTC + ISF strategy. Finally, Foy et al. (2015) noted, “If studies evaluating the effects of implementation intervention are to be of relevance to policy and practice, they should have endpoints related to evidence-based processes of care, patient outcomes, or population outcomes.” Thus, another implication of our finding is the need for more type 2 hybrid trials that enable tests of impact on intervention effectiveness, which is arguably one of the most relevant endpoints of all.

Although the level-of-sustainment is not possible without some level of implementation effectiveness occurring first, the level-of-sustainment is another endpoint of significant relevance (Chambers et al., 2013; Proctor et al., 2015; Stirman et al., 2012). Thus, it is significant to note that we did not find support for our hypotheses related to the level-of-sustainment, and the average level-of-sustainment was not only similar between conditions but was also relatively low (only three MIBIs during the 6-month sustainment period). The key limitation associated with this finding is that level-of-sustainment was based on self-report from MIBI staff. It does not appear that self-report led to MIBI staff overestimating their level-of-sustainment. Our level-of-sustainment finding was also limited by not being able to measure the extent to which MIBIs were implemented with quality. Conservatively, the generalizability of our findings should be limited to efforts to testing the impact of the ISF strategy as an adjunct to the ATTC strategy or the level-of-sustainment of a MIBI for SUDs within HIV organizations. However, we believe our findings generalize more broadly to research that has advanced knowledge regarding sustainment (Hunter et al., 2015, 2017). Arguably, sustainment is one of the most important outcomes to identify the effective strategies for, especially given that the lack of sustainment minimizes the return-on-investments for resources expended during prior phases along the EPIS continuum (exploration phase, preparation phase, and implementation phase) (Aarons et al., 2011). Thus, a key implication is that future research is needed to test strategies that can significantly improve both the relative effectiveness of the ATTC + ISF strategy on the level-of-sustainment and the extent to which the ATTC + ISF strategy improves the absolute level-of-sustainment.

In conclusion, although the ATTC strategy was found to be sufficient for the preparation of HIV organization staff to implement a MIBI for SUDs, the ISF strategy was found to be an effective adjunct to the ATTC strategy in terms of both implementation effectiveness and intervention effectiveness. Based on these findings, future implementation research and practice focused on integrating a MIBI for SUD within HIV organizations should consider using the ATTC + ISF strategy. However, given the ISF strategy did not have a significant impact on the level-of-sustainment, which was similarly low in both implementation conditions, we also conclude that future efforts should seek to enhance the ATTC + ISF strategy through strategies focused on improving the level-of-sustainment during the sustainment phase.