Skip to main content
NCBI home page
Implementation Research and Practice logoLink to Implementation Research and Practice
. 2021 May 19;2:26334895211005809. doi: 10.1177/26334895211005809

Opening the “black box”: Four common implementation strategies for expanding the use of medications for opioid use disorder in primary care

Lauren Caton 1, Hanyang Shen 1, Gloria M Miele 2, Kendall Darfler 2, Jose R Sandoval 2, Darren Urada 2, Mark P McGovern 1,3,✉
PMCID: PMC9978648  PMID: 37090014

Abstract

Background:

Despite the persistent increase in overdose deaths, access to medications for opioid use disorders remains limited. Recent federal funding aimed at increasing access prompts a need to understand if implementation strategies improve access.

Methods:

This is an analysis of data from 174 primary care clinics enrolled in a state-wide medications for opioid use disorders (MOUD) implementation effort in California. We examined clinic use of one of four implementation strategies: learning collaboratives, Project Extension for Community Health care Outcomes (ECHO), didactic webinars, and clinical skills trainings. The primary implementation outcome was categorical change in new patients prescribed buprenorphine. Univariate and multivariate logistic regressions were used to determine the impact of clinic attendance in all or individual implementation strategies, respectively, on patient growth.

Results:

Clinics attending learning collaboratives, Project ECHO, and clinical skills trainings had significantly higher odds of patient growth (odds ratio [OR] = 3.56; 95% confidence interval [CI] = 1.78, 7.10, p < .001), (OR = 3.39; 95% CI = 1.59, 7.24, p < .01), (OR = 3.90, 95% CI = 1.64, 9.23, p < .01) than non-attending clinics. The impact of attendance at learning collaboratives (OR = 5.81, 95% CI = 1.89, 17.85; p < .01), didactic webinars (OR = 3.59; 95% CI = 1.04, 12.35; p < .05), and clinical skills trainings (OR = 3.53, 95% CI = 1.06, 11.78, p < .05) on patient growth was greater for Federally Qualified Health Centers. When comparing strategies in multivariate models, only the relationship between learning collaborative attendance and new patients prescribed buprenorphine remained significant (OR = 2.57; 95% CI = 1.12, 5.88; p < .05).

Conclusions:

This study reported on a large, statewide, implementation-as-usual project offering four typical implementation strategies. Clinic attendance at learning collaboratives, a multi-component strategy, had the most consistent impact on new patients prescribed buprenorphine. These results suggest that while a broad array of strategies was initially reasonable, optimizing the selection of implementation strategies could be more effective.

Plain Language Summary

Access to life-saving medications for opioid use disorder, such as buprenorphine, remains limited despite strong evidence of effectiveness. Systems and organizations often select from a variety of implementation strategies aimed at expanding access to these medications. However, scant research exists to enable these organizations to select the most effective and efficient strategies. Our study—within a large state-wide system of care—examined the impact of primary care clinic attendance in four common implementation strategies on new patients prescribed buprenorphine. Learning collaboratives were the strategy that most consistently improved outcomes. These results highlight the challenges to strategy selection inherent in implementation-as-usual systems-level approaches. The field needs evidence-based information on which implementation strategies are most likely to yield desired implementation outcomes.

Keywords: Implementation research, implementation strategies, medications for opioid use disorders (MOUD), buprenorphine, opioid use disorders, learning collaboratives, Project ECHO

The number of Americans with opioid use disorder (OUD) continues to rise and contribute to significant morbidity and mortality in the United States (Centers for Disease Control and Prevention, 2017; Jones et al., 2015). Even with this growing prevalence, on average, less than one-third of individuals with OUD receive any method of treatment (Saloner & Karthikeyan, 2015; Jones et al., 2015). The most effective form of treatment, the use of Food and Drug Administration (FDA)-approved medications for opioid use disorders (MOUD), remains limited in reach despite strong evidence in preventing overdose death and reducing opioid use (Jones et al., 2015; M. Krause et al., 2011; Murthy, 2016; Volkow et al., 2014). To address this treatment gap, the 21st Century Cures Act provided states and territories with federal funding for MOUD expansion initiatives (21st Century Cures Act of 2016). Capitalizing on this opportunity enables systems and organization leaders to understand what implementation strategies have targeted impact on the implementation outcome of increased MOUD prescribing (Korthius et al., 2016).

Similar to many states receiving this federal funding, California offered a range of reasonable implementation strategies for clinics to participate in: learning collaboratives, Project Extension for Community Health care Outcomes (ECHO), didactic webinars, and clinical skills trainings (Miele et al., 2020). These were the only strategies employed by California through the federal grant. Clinics could choose to participate in some or all of the implementation strategies, selecting those most relevant to them, in a “buffet-style” approach. In the absence of actual evidence for any one or all of these strategies, the options were selected based on what seemed feasible and had been historically deployed. This is an unfortunately common practice in expansion initiatives. As if in a “black box” chosen strategies are often neither specified in their selection, monitored for fidelity or adaptation, tracked for attendance, nor examined for cost and burden (Kirchner et al., 2017; Powell et al., 2019).

Within implementation science research, these strategies are often organized into single, or “discrete,” and “combined” strategies (Powell et al., 2015, 2020; Proctor et al., 2013). California employed several discrete and combined strategies over the study period. Learning collaboratives, a combined and multi-component implementation strategy, has some support in mental health evidence-based practice (EBP) implementation (Becker et al., 2011; Institute for Health care Improvement [IHI], 2003; Nadeem et al., 2014). Only one study explored the impact of learning collaboratives on MOUD, discovering a positive outcome in reduced practice variation among 60 primary care clinics, but not an increase in number of patients prescribed buprenorphine (Nordstrom et al., 2016). Project ECHO, another multi-component strategy, has demonstrated impact on managing pain, decreasing opioid prescribing, and addressing substance use disorders (Carey et al., 2016; Katzman, 2020; Katzman et al., 2014; Komaromy et al., 2016; Project ECHO, 2019). However, studies that have investigated opioid prescribing, only evaluated provider self-efficacy and not patient outcomes (Holmes et al., 2020; Komaromy et al., 2016; Tofighi et al., 2019). Didactic webinars and clinical skills trainings are discrete implementation strategies and are the most common in organizational initiatives to improve implementation outcomes. The present study, in alignment with a call to investigate Project ECHO’s potential for MOUD expansion (Holmes et al., 2020), is among the first to investigate the impact of Project ECHO or learning collaboratives on new patients prescribed buprenorphine. It is also among the first to evaluate the specific effects of a set of implementation strategies on number of new MOUD patients.

The strategies explored in this MOUD expansion project are exceedingly common and pervasive in systems- and state-wide initiatives. It is therefore important to analyze if these prevalent strategies have an impact on the desired outcome, with immediate relevance to ongoing projects and the implementation science literature. We analyze clinic-level attendance in four common implementation support strategies on increases in new patients prescribed buprenorphine. Federally Qualified Health Centers (FQHCs) serve as safety nets for the uninsured and medically disadvantaged, these organizations have significant potential to treat patients with OUD. In fact, FQHCs have been a major focus for MOUD expansion (Arfken et al., 2010; Watson et al., 2020). Therefore, we also examine the impact of implementation strategy attendance on patient growth in FQHC clinics. We hypothesize that clinics attending these implementation strategies will have better implementation outcomes on growth in new patients prescribed buprenorphine.

Methods

Clinics and sampling

Data were gathered from 174 primary care clinics from August 2017 to September 2019 (Darfler et al., 2018). The clinics were voluntarily enrolled in a Substance Abuse and Mental Health Services Administration grant to the State of California, funded through the State Targeted Response and State Opioid Response mechanisms. These clinics, located across the state, were recruited by and networked with 18 regional opioid treatment programs, in alignment with the Hub and Spoke model (Simpatico, 2015; Darfler et al., 2019). The clinics were eligible to receive implementation support through a state-funded training center via four types of activities described below. In exchange, primary care practices were to report simple data on clinical activity including number of new patients prescribed buprenorphine. Because data were aggregated at the clinic-level, the study was deemed exempt and approved by The University of California, Los Angeles and Stanford University’s institutional review boards, and also approved by the California Office of Statewide Health Planning and Development (P 2018 -047). There were no adverse events in this study.

Implementation strategies

Four types of implementation strategies were offered: (1) quarterly in-person learning collaboratives, (2) quarterly virtual didactic webinars, (3) monthly Project ECHO sessions, and (4) twice-yearly clinical skills trainings. Learning collaboratives and didactic webinars were available to all clinic and administrative staff. Project ECHO (1-hr and virtual) and clinical skills trainings (multi-hour and in-person) targeted clinical staff. Activities were free of charge and eligible for continuing medical education credit through Stanford University School of Medicine. The technical assistance team at The University of California, Los Angeles coordinated all 27 events during the 25-month project period. These included 7 learning collaboratives, 13 virtual Project ECHO sessions, 4 didactic webinars, and 3 clinical skills trainings. Attendance data for all activities included individual attendance, clinic type, and clinic role for each attendee through sign-in documentation. Portal login information captured the same information for virtual trainings.

Learning collaboratives

Learning collaboratives aim to reduce practice variation through evidence-based information and quality improvement data sharing across systems (IHI, 2003; Schouten et al., 2008). They are typically comprised of three components over a 3-hr session: didactic lecture, practice presentation, and quality improvement or common data sharing and reporting (IHI, 2003). The practice presentations were either clinical cases or discussions of MOUD best practices. The seven learning collaboratives began in September 2017, conducted in five geographic regions, and occurred quarterly. Five sessions were in-person and the remaining two were virtual. Topics included building a system of care for persons with OUD, talking to patients about MOUD teams, interdisciplinary care, co-occurring substance use, treatment retention, and network sustainability.

Project ECHO

Project ECHO is an hour-long distal-learning model consisting of brief didactic presentation, then case discussion by a treatment provider (Project ECHO, 2019). The virtual trainings began in December 2017 and continued monthly. Didactic topics covered treatment of OUD, risk reduction strategies, Hepatitis C/infectious disease, MOUD and pregnancy, pain management, polysubstance use, nurse-led models of care, patient communication styles, trauma screening, suicide prevention, naloxone rescue, and aging and the opioid epidemic, and screening, brief intervention, and referral to treatment (SBIRT).

Didactic webinars

The virtual didactic webinars began in September 2018, were held quarterly, and open to non-clinicians and staff as well as the broader community. The 2-hr sessions included a didactic presentation. Topics covered: OUD stigma, MOUD team integration and care coordination, and co-occurring stimulant use.

Clinical skills trainings

The clinical skills trainings began in May 2018 and were held twice a year. The half-day sessions were in-person didactic and interactive presentations geared toward already-waivered and non-waivered providers. These differed from the generalized audience of the didactic webinars, but were similar to the practitioner-focused Project ECHO. Topics included managing pain in patients with OUD, motivational interviewing, and treating pregnant women with MOUD.

Independent and dependent variable measures

Implementation strategy activity attendance

The main independent variable in this study is clinic-level attendance at an implementation strategy during the project period (August 2017 to September 2019). Since clinics were enrolled throughout the implementation project, clinic-level attendance was counted only for strategies available to them during their tenure (all clinics had access to at least one event for each strategy). As there was a designated grant coordinator for each clinic, attendance was operationalized as presence of at least one representative (staff coordinator, member, or prescriber) at a session. Clinic attendance was coded binary as attending or not attending an implementation activity based on the presence or absence of a representative at the session. While data were collected on staff type and number of staff attending each event, this analysis did not explore the additive effects of these variables.

New patients prescribed buprenorphine

The primary outcome is the dependent variable: categorical increase in new patients prescribed buprenorphine. Patient growth was the focus of this paper, as opposed to growth in providers with a buprenorphine-waiver. Recent literature points to limitations of adoption as an evaluation metric for MOUD expansion, as many buprenorphine-waivered physicians do not utilize their waiver or under-prescribe to their federally-mandated limits (Knudsen, 2015; Knudsen et al., 2020). The present analysis centers on clinic-level data from staff-submitted monthly reports from baseline (August 2017 or later) thru September 2019. Not all 174 primary care clinics started in August 2017 and accordingly there was “rolling” engagement in the project. Thus, each clinic’s baseline data began the month they started.

Procedure

The primary outcome was collected from online monthly reports submitted by each clinic. These reports contain aggregate information on nine data elements from the clinic, but only two—new patients prescribed buprenorphine and number of waivered providers—were used for this study (Darfler et al., 2019). The reports do not collect data directly from patients or their health records. A data reporting handbook was developed for participating clinics and three informational webinars described data reporting guidelines to standardize submissions across months. The same, trained, assigned staff member submitted these reports monthly. Approximately 16% of clinics would fail to submit a report in a given month and data were adjusted using mean imputation (Engels & Diehr, 2003). However, this most often—in 76% of cases—occurred when the clinic had no data to report (no new patients). Therefore, only 6 reports per month, on average, were adjusted with a non-zero value. When a clinic enrolled, they provided additional demographic information on urbanicity, clinic type, and baseline number of MOUD patients served.

Statistical analysis

This is a naturalistic, retrospective cohort analysis of monthly aggregate data from 174 clinics across a state system of care. Typical in large-scale implementation projects, clinics were not randomized to implementation strategy conditions, were invited to attend all activities, and had rolling start dates. Neither participants nor administrators were blinded to the study. The current analysis focused on the relationship between implementation strategy attendance and change in new patients prescribed buprenorphine by clinic.

Categorization of patient growth

Linear regression was used to estimate the growth pattern for each clinic during the study period. Clinics, with both a positive linear regression coefficient and a p-value < 0.05, indicating significant increase in new patients prescribed buprenorphine, were assigned to the significant “growth” group. The remaining clinics were categorized into the non-significant growth group. We then investigated the impact of attendance on this two-category grouping (i.e., growth vs. no growth), which is our primary outcome. To investigate the sensitive effects of enrollment date, a sensitivity analysis on duration enrolled was conducted to detect significant growth in clinics with fewer data points. The sensitivity analysis on enrollment year confirmed the groupings, as enrollment date did not disproportionately interfere with growth outcomes for clinics of differing tenures.

Univariate and multivariate analyses

Mann–Whitney unpaired and chi-square tests evaluated significant continuous and categorical differences, respectively, in clinic characteristics between growth groups. Univariate logistic regressions examined the relationship between clinic-level attendance in all strategies and increases in new patients prescribed buprenorphine (growth vs. no growth). We employed multivariate logistic regressions, adjusting for attendance at any one or combinations of the other strategies, to explore whether attendance by strategy predicted the primary outcome. This was repeated in subanalyses stratifications by FQHC status and number of providers. All tests were completed using R version 3.6.1.

Results

Of the 174 clinics in the study, 51 (29.3%) experienced a significant increase in the new patients prescribed buprenorphine. Over the 25-month program period, there was an average increase of 7.45 new patients per month. The remaining 123 clinics did not experience a significant increase in new patients. These clinics had an average increase of 0.54 new patients per month. Table 1 presents clinic characteristics. The majority of clinics were enrolled in 2018 (61.5%), were located in census-designated urban areas (75.7%), and the largest proportion were designated FQHCs (41.4%) (U.S. Census Bureau, 2010). Specialty addiction treatment programs, community health centers, hospital-based practices, community mental health centers, telehealth, private practices, tribal health centers, and specialty pain clinics comprised the remaining heterogeneous types of organizations (58.6%). Enrolled clinics started with an average of 1.70 patients prescribed MOUD at baseline (standard deviation [SD] = 4.20). This average was slightly higher in clinics in the no-growth group (M = 2.00, SD = 4.64) than clinics in the growth group (M = 0.98, SD = 2.78; p = .043). Across all time points, 74 clinics attended learning collaboratives, 43 attended Project ECHO sessions, 37 attended didactic trainings, and 29 attended clinical skills trainings. Figure 1 depicts the differential growth trajectories in new patients prescribed buprenorphine across implementation strategies.

Table 1.

Clinic characteristics and implementation strategy attendance.

Clinic characteristic Total (N = 174) Clinics with no increase in new patients (N = 123) Clinics with increase in new patients (N = 51) p-value
n/Ma (%)/SDa n/M (%)/SD n/M (%)/SD
Baseline patients prescribed buprenorphine 1.70 4.20 2.00 4.64 0.98 2.78 .043*
Urbanicity
 Rural 42 (24.3%) 34 (27.9%) 8 (15.7%) .131
 Urban 131 (75.7%) 88 (72.1%) 43 (84.3%)
Clinic type
 Non-FQHC 102 (58.6%) 74 (60.2%) 28 (54.9%) .637
 FQHC 72 (41.4%) 49 (39.8%) 23 (45.1%)
Year enrolled in project
 2017 36 (20.7%) 15 (12.2%) 21 (41.2%) <.001**
 2018 107 (61.5%) 79 (64.2%) 28 (54.9%)
 2019 31 (17.8%) 29 (23.6%) 2 (3.9%)
Prescriber status
 No increase in prescribers 106 (60.9%) 93 (75.6%) 13 (25.5%) .001*
 Increase in prescribers 68 (39.1%) 30 (24.4%) 38 (74.5%)
Learning collaborative attendance
 No attendance 100 (57.5%) 81 (65.9%) 19 (37.3%) .001*
 Attendance 74 (42.5%) 42 (34.1%) 32 (62.7%)
Project ECHO attendance
 No attendance 131 (75.3%) 100 (81.3%) 31 (60.8%) .008*
 Attendance 43 (24.7%) 23 (18.7%) 20 (39.2%)
Didactic webinar attendance
 No Attendance 137 (78.7%) 100 (81.3%) 37 (72.5%) .280
 Attendance 37 (21.3%) 23 (18.7%) 14 (27.5%)
Clinical skills training attendance
 No attendance 145 (83.3%) 109 (88.6%) 36 (70.6%) .007*
 Attendance 29 (16.7%) 14 (11.4%) 15 (29.4%)
Open in a new tab

FQHC: Federally Qualified Health Centers; ECHO: Extension for Community Health care Outcomes.

Note: For continuous variables, -values were calculated using Mann–Whitney Unpaired test; for categorical variables, p-values were calculated using Chi-square test.

a

Mean and standard deviation (SD) apply to the continuous variable: baseline patients prescribed buprenorphine.

*

Significant at p-value < .05.

**

Significant at p-value < .001.

Figure 1.

Figure 1.

Open in a new tab

Comparative growth in new patients prescribed buprenorphine by attendance at implementation strategy type.

The number of baseline patients prescribed buprenorphine and time of enrollment in the program differed significantly between the two categories of growth groups. Given the possible sensitivity of our results to baseline differences, we conducted our analyses with (Model 2) and without (Model 1) adjusting for the number of baseline patients prescribed buprenorphine. Table 2 shows that Model 1, unadjusted, revealed higher odds of new patients prescribed buprenorphine among clinics that attended learning collaboratives (odds ratio [OR] = 3.25; 95% confidence interval [CI] = 1.65, 6.41, p = .001), Project ECHO (OR = 2.81; 95% CI = 1.36, 5.77, p = .005), and clinical skills trainings (OR = 3.24, 95% CI = 1.43, 7.37, p = .005) compared to clinics that did not attend these activities. There was no significant difference in outcome by participating in didactic webinars. From the unadjusted multivariate analysis of all implementation strategies considered together, only attendance at learning collaboratives significantly increased the odds of patient growth compared to non-attending clinics (OR = 2.57, 95% CI = 1.12, 5.88, p = .026). Attendance at the other three implementation strategies had no significant effect on the outcome.

Table 2.

Implementation activity attendance effect on new patients prescribed buprenorphine.

Implementation activity attendance Model 1a Model 2b
Univariate logistic regression Multivariate logistic regression Univariate logistic regression Multivariate logistic regression
OR (95% CI) p-value OR (95% CI) p-value OR (95% CI) p-value OR (95% CI) p-value
Learning collaboratives
 No attendance 1.00 1.00 1.00 1.00
 Attendance 3.25 (1.65, 6.41) .001* 2.57 (1.12, 5.88) .026* 3.56 (1.78, 7.10) <.001** 2.49 (1.08, 5.76) .032*
Project ECHO
 No attendance 1.00 1.00 1 1
 Attendance 2.81 (1.36, 5.77) .005* 1.89 (0.78, 4.56) .158 3.39 (1.59, 7.24) .002* 2.05 (0.83, 5.06) .122
Didactic webinars
 No attendance 1.00 1.00 1 1
 Attendance 1.65 (0.77, 3.53) .202 0.54 (0.20, 1.44) .22 2.04 (0.91, 4.55) .083 0.67 (0.25, 1.85) .444
Clinical skills training
 No attendance 1.00 1.00 1 1
 Attendance 3.24 (1.43, 7.37) .005* 2.16 (0.85, 5.47) .105 3.9 (1.64, 9.23) .002* 2.36 (0.91, 6.16) .079
Open in a new tab

OR: odds ratio; CI: confidence interval; ECHO: Extension for Community Health care Outcomes.

a

Unadjusted model not controlling for baseline new buprenorphine patients.

b

Adjusted model controlling for baseline new buprenorphine patients.

*

Significant at p-value < .05.

**

Significant at p-value < .001.

Table 2 also shows that this pattern was replicated in adjusted Model 2, controlling for baseline differences in clinic capacity. In the univariate analysis, the odds of new patients prescribed buprenorphine increased by attending learning collaboratives (OR = 3.56; 95% CI = 1.78, 7.10, p < .001), Project ECHO (OR = 3.39; 95% CI = 1.59, 7.24, p = .002), and clinical skills trainings (OR = 3.90, 95% CI = 1.64, 9.23, p = .002). When comparing strategies in the multivariate model, however, only one significant association remained: learning collaborative attendance and the outcome (OR = 2.49, 95% CI = 1.08, 5.76, p = .032).

Sub-analyses

Because of the emphasis on increasing MOUD in FQHCs across the US, we explored the comparative outcomes for these 102 clinics (Table 3). In the univariate analysis, FQHC clinics that attended learning collaboratives (OR = 5.81, 95% CI = 1.89, 17.85; p = .002), didactic webinars (OR = 3.59; 95% CI = 1.04, 12.35; p = .043), and clinical skills trainings (OR = 3.53, 95% CI = 1.06, 11.78, p = .040) had higher odds of new patients prescribed buprenorphine than non-attending FQHC clinics. This stronger impact of clinic attendance at learning collaboratives on in patients initiating buprenorphine for FQHCs remained in the multivariate analysis (OR = 5.21; 95% CI = 1.07, 25.43, p = .041). As in Models 1 and 2, learning collaboratives were consistently impactful on increases in patients initiating buprenorphine. As provider growth may impact a clinic’s ability for patient growth, we also conducted a univariate and multivariate sub-analysis of clinics with and without an increase in providers (Table 4). This mirrored the FQHC analysis. Among clinics with a growth in providers, there was significantly higher odds of increased new patients prescribed buprenorphine for clinics attending all implementation strategy types. However, this did not hold for the multivariate analysis.

Table 3.

Implementation activity attendance effect on new patients prescribed buprenorphine, sub-analysis by FQHC status.

Implementation activity attendance FQHC clinics (n = 72) Non-FQHC clinics (n = 102)
Univariate logistic regression Multivariate logistic regression Univariate logistic regression Multivariate logistic regression
OR (95% CI) p-value OR (95% CI) p-value OR (95% CI) p-value OR (95% CI) p-value
Learning collaboratives
 No attendance 1.00 1.00 1.00 1.00
 Attendance 5.81 (1.89, 17.85) .002* 5.21 (1.07, 25.43) .041* 2.81 (1.13, 6.99) 0.026* 1.7 (0.56, 5.13) .348
Project ECHO
 No attendance 1.00 1.00 1.00 1.00
 Attendance 3.07 (0.90, 10.54) .074 1.61 (0.35, 7.43) .544 3.78 (1.42, 10.08) .008* 3.32 (0.95, 11.55) .060
Didactic webinars
 No attendance 1.00 1.00 1.00 1.00
 Attendance 3.59 (1.04, 12.35) .043* 0.78 (0.14, 4.42) .778 1.29 (0.43, 3.88) .655 0.39 (0.09, 1.64) .200
Clinical skills training
 No attendance 1.00 1.00 1.00 1.00
 Attendance 3.53 (1.06, 11.78) .040* 1.28 (0.29, 5.62) .747 4.26 (1.22, 14.95) .024* 3.52 (0.84, 14.70) .084
Open in a new tab

FQHC: Federally Qualified Health Centers; OR: odds ratio; CI: confidence interval.

*

Significant at p-value < .05.

Table 4.

Implementation activity attendance effect on new patients prescribed buprenorphine, sub-analysis by provider growth.

Implementation activity attendance Clinics with no increase in providers (n = 106) Clinics with increase in providers (n = 68)
Univariate logistic regression Multivariate logistic regression Univariate logistic regression Multivariate logistic regression
OR (95% CI) p-value OR (95% CI) p-value OR (95% CI) p-value OR (95% CI) p-value
Learning collaboratives
 No attendance 1 1 1 1
 Attendance 6.57 (1.84, 23.43) .004* 3.89 (0.83, 18.25) .085 1.23 (0.45, 3.38) .686 1.36 (0.38, 4.94) .637
Project ECHO
 No attendance 1 1 1 1
 Attendance 7.63 (2.15, 27.03) .002* 2.7 (0.57, 12.90) .213 1.39 (0.47, 4.15) .552 1.84 (0.51, 6.63) .352
Didactic webinars
 No attendance 1 1 1 1
 Attendance 4.59 (1.06, 19.92) .042* 0.98 (0.15, 6.27) .985 0.54 (0.19, 1.55) .253 0.31 (0.08, 1.17) .084
Clinical skills training
 No attendance 1 1 1 1
 Attendance 7.2 (1.60, 32.30) .010* 3.96 (0.73, 21.41) .113 1.36 (0.44, 4.24) .594 1.59 (0.40, 6.35) .511
Open in a new tab

OR: odds ratio; CI: confidence interval; ECHO: Extension for Community Health care Outcomes.

*

Significant at p-value < .05.

Conclusion

Summary of findings

This study evaluated whether clinic attendance at four implementation strategies increased patient access to MOUD, specifically buprenorphine. Our models controlled for baseline patient characteristics and adjusted for attendance at other strategy events. In these models, clinics that attended learning collaboratives were significantly more likely to experience an increase in the outcome—new patients prescribed buprenorphine. This effect was even stronger for Federally Qualified Health Centers (FQHCs). Didactic webinars were the least effective of all four strategy types.

Unlike the only existing study on learning collaboratives and MOUD implementation outcomes (Nordstrom et al., 2016), we found an impact on new patients prescribed buprenorphine. In alignment with the literature on Project ECHO, and recent studies exploring its impact on opioid prescriptions, activity attendance increased odds of MOUD prescribing (Komaromy et al., 2016; Tofighi et al., 2019). The most successful strategies, learning collaboratives and Project ECHO, target known MOUD expansion barriers in the field—low provider confidence in delivering specialty care and lack of institutional support (Huhn & Dunn, 2017; Hutchinson et al., 2014). However, over a 25-month period, only 29.3% (51/ 174) of clinics increased patients prescribed buprenorphine. This modest impact highlights the importance of examining the details within the “black box” of implementation strategies. Integrating contextual factors that prompt decision-making on strategy selection is imperative.

Limitations

Because this was not a controlled implementation research study, limitations of causal interpretations are inherent in the naturalistic, retrospective cohort study design. Limitations to generalizability include: unaccounted for outside support from other initiatives to participating clinics during the project period; lack of a benchmark for “success” of MOUD expansion projects; self-report bias; and no measures of contextual determinants at organizational (e.g., leadership) or individual (e.g., staff pattern) levels. Furthermore, we did not account for the role or number of the clinic staff member(s) who participated in these events. Monthly patient growth was low across many clinics in the program. Categorizing the outcome variable based on the results of the clinic-specific regression models may have introduced estimation uncertainty in the analysis; and this was not accounted for in the final hypothesis tests and confidence intervals presented. This chosen method is exploratory. More rigorous analyses are needed. In addition, our focus is on the effect of attendance. It was not feasible, in this study, to collect other interesting process measures, such as the Stages of Implementation Completion (SIC) or Costs of Implementing New Strategies (COINS) (Saldana, 2014; Saldana et al., 2014). Such measures would add an important level of rigor and enable us to answer additional questions about the effects of engagement in strategies. The four implementation strategies, although typical, were equivalent in neither dose, duration, nor temporality. Heterogeneous characteristics between clinics such as motivation to engage with the project and preference for certain strategies may have enhanced the effect of attendance at learning collaboratives on the outcome of interest. Nonetheless, this “buffet-style” approach to a set of reasonable implementation strategies is typical in evidence-based practice (EBP) start-up or scale-up efforts in state systems.

Implications and next steps

Despite the aforementioned limitations, these analyses advance our understanding of typical implementation support strategies designed to expand MOUD in routine practice settings. Currently, systems-level projects select and employ multiple strategies, either based on organizational preference or prior precedence in the field (Grimshaw et al., 2012). In many cases, a ‘buffet” of every possible strategy is deployed with hope that at least one will drive the desired outcome (Kirchner et al., 2017). The scale-up of these initiatives would benefit from more precision in tailoring to the contextual determinants of organization types (J. Krause et al., 2014; Powell et al., 2017). Implementation science and practice is moving toward this more rigorous approach to select candidate strategies. An emerging focus reinforces selecting strategies to identified barriers, followed by ongoing adaptation based on contextual factors and performance toward outcomes (Bosch et al., 2007; J. Krause et al., 2014; Powell et al., 2017; Squires et al., 2014). In fact, much like a stepped-care approach to managing a medical problem, a stepped-implementation strategy, could be based on ongoing process measurement of key outcomes. Furthermore, even the most potent implementation strategy will involve variation in participant engagement and retention (e.g., Stages of Implementation Completion) and fidelity in its delivery (e.g., Implementation Facilitation) (Collins et al., 2008; Ritchie et al., 2017; Saldana et al., 2012).

In expanding access to MOUD, multiple implementation strategies afforded options—buffet style—that participants could choose to attend or not attend. Attendance is commonly tracked in funded expansion initiatives (Nordstrom et al., 2016; Watson et al., 2020). It is therefore important for studies—like this one—to evaluate whether attendance veritably impacts the outcome of interest. The need to evaluate expanded access to evidence-based treatment remains, even as the opioid epidemic rapidly transitions to stimulants such as methamphetamine and cocaine (Ellis et al., 2018). Enhanced understanding of how to optimize the selection and delivery of implementation strategies will help us better respond to substance-related public health crises even with dynamic substance use shifts.

Footnotes

Author Note: This article was derived from significant project input completed by Hélène Chokron Garneau. The authors would like to thank Ida Chen and Mina Yuan for their contributions to earlier renditions of the data analysis used for the manuscript. In addition, to Kendall Darfler for their role in data procurement. The staff at the Center for Behavioral Health Services and Implementation Research contributed significantly to manuscript review. The staff, clinicians, and patients of the clinics involved in the study played an instrumental part in the success of the program.

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding: The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was funded through the California Department of Health Care Services (DHCS) “MAT Expansion Project” (DHCS Contract #17-94653) by the SAMHSA State Targeted Response to the Opioid Crisis Grants (TI-17-014). The contents do not represent the official views of the State of California.

ORCID iD: Jose R Sandoval Inline graphic https://orcid.org/0000-0002-8378-0350

References

  1. Arfken C., Johanson C. E., di Menza S., Schuster C. R. (2010). Expanding treatment capacity for opioid dependence with office-based treatment with buprenorphine: National surveys of physicians. Journal of Substance Abuse Treatment, 39(2), 96–104. 10.1016/j.jsat.2010.05.004 [DOI] [PubMed] [Google Scholar]
  2. Becker D., Drake R., Bond G., Nawaz S., Haslett W., Martinez R. (2011). Best practices: A national mental health learning collaborative on supported employment. Psychiatric Services, 62(7), 704–706. 10.1176/ps.62.7.pss6207_0704 [DOI] [PubMed] [Google Scholar]
  3. Bosch M., Van Der Weijden T., Wensing M., Grol R. (2007). Tailoring quality improvement interventions to identified barriers: A multiple case analysis. Journal of Evaluation in Clinical Practice, 13(2), 161–168. 10.1111/j.1365-2753.2006.00660.x [DOI] [PubMed] [Google Scholar]
  4. Carey E., Frank J., Kerns R., Ho P., Kirsch S. (2016). Implementation of telemonitoring for pain management in the Veterans Health Administration: Spatial analysis. Journal of Rehabilitation Research and Development, 53(1), 147–156. 10.1682/JRRD.2014.10.0247 [DOI] [PubMed] [Google Scholar]
  5. Center for Disease Control and Prevention. (2017). National vital statistics system, mortality. https://wonder.cdc.gov
  6. Collins L. M., Murphy S. A., Strecher V. (2008). The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): New methods for more potent eHealth interventions. American Journal of Preventative Medicine, 32(5), 112–118. 10.1016/j.amepre.2007.01.022 [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Darfler K., Sandoval J., Pearce Antonini V., Urada D. (2019). Preliminary results of the evaluation of the California hub and spoke program. Journal of Substance Abuse Treatment, 108(1), 26–32. 10.1016/j.jsat.2019.07.014 [DOI] [PubMed] [Google Scholar]
  8. Darfler K., Urada D., Antonini V., Padwa H., Joshi V., Sandoval J. (2018). California state targeted response to the opioid crisis: Year 1 evaluation report. http://www.uclaisap.org/ca-hubandspoke/docs/reports/CA%20HSS-Evaluation-Annual%20Report-October%202018.pdf
  9. Ellis M. S., Kasper Z. A., Cicero T. J. (2018). Twin epidemics: The surging rise of methamphetamine use in chronic opioid users. Drug and Alcohol Dependence, 193, 14–20. 10.1016/j.drugalcdep.2018.08.029 [DOI] [PubMed] [Google Scholar]
  10. Engels J., Diehr P. (2003). Imputation of missing longitudinal data: A comparison of methods. Journal of Clinical Epidemiology, 56(10), 968–976. 10.1016/S0895-4356(03)00170-7 [DOI] [PubMed] [Google Scholar]
  11. Grimshaw J. M., Eccles M. P., Lavis J. N., Hill S. J., Squires J. E. (2012). Knowledge translation of research findings. Implementation Science, 7(1), 50. 10.1186/1748-5908-7-50 [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Holmes C. M., Keyser-Marcus L., Dave B., Mishra V. (2020). Project ECHO and opioid education: A systematic review. Current Treatment Options in Psychiatry, 7, 9–22. 10.1007/s40501-020-00199-8 [DOI] [Google Scholar]
  13. Huhn A., Dunn K. (2017). Why aren’t physicians prescribing more buprenorphine? Journal of Substance Abuse Treatment, 78, 1–7. 10.1016/j.jsat.2017.04.005 [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Hutchinson E., Catlin M., Andrilla C., Baldwin L., Rosenblatt R. (2014). Barriers to primary care physicians prescribing buprenorphine. Annals of Family Medicine, 12(2), 128–133. 10.1370/afm.1595 [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Institute for Healthcare Improvement. (2003). The breakthrough series: IHI’s collaborative model for achieving break-through improvement [IHI innovation series white paper]. http://www.ihi.org/resources/Pages/IHIWhitePapers/TheBreakthroughSeriesIHIsCollaborativeModelforAchievingBreakthroughImprovement.aspx
  16. Jones C. M., Campopiano M., Baldwin G., McCance-Katz E. F. (2015). National and state treatment need and capacity for opioid agonist medication-assisted treatment. American Journal of Public Health, 105(8), e55–e63. 10.2105/AJPH.2015.302664 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Katzman J. G. (2020). ECHO telementoring for pain, palliative care, and opioid management: Progress, challenges, and future goals. Pain Medicine, 21(2), 220–225. 10.1093/pm/pnz337 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Katzman J. G., Comerci G., Boyle J., Duhigg D., Shelley B., Olivas C. (2014). Innovative telementoring for pain management: Project ECHO pain. Journal of Continuing Education in Health Professions, 34(1), 68–75. 10.1002/chp.21210 [DOI] [PubMed] [Google Scholar]
  19. Kirchner J. E., Waltz T. J., Powell B. J., Smith J. L., Proctor E. K. (2017). Implementation strategies. In Brownson R. C., Colditz G. A., Proctor E. K. (Eds.), Dissemination and implementation research in health (2nd ed.). (pp. 245–266). Oxford University Press. [Google Scholar]
  20. Knudsen H. K. (2015). The supply of physicians waivered to prescribe buprenorphine for opioid use disorders in the United States: A state-level analysis. Journal of Studies on Alcohol and Drugs, 76(4), 644–654. 10.15288/jsad.2015.76.644 [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Knudsen H. K., Lin L. A., Lofwall M. R. (2020). Adoption of the 275-patient buprenorphine treatment waiver for treating opioid use disorder: A state-level longitudinal analysis. Substance Abuse, 41(2), 259–268. 10.1080/08897077.2019.1635959 [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Komaromy M., Duhigg D., Metcalf A., Carlson C., Kalishman S., Hayes L. (2016). Project ECHO (Extension for Community Healthcare Outcomes): A new model for educating primary care providers about treatment of substance use disorders. Substance Abuse, 37(1), 20–24. 10.1080/08897077.2015.1129388 [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Korthius P., McCarty D., Weimer M., Bougatsos C., Blazina I., Chou R. (2016). Primary care–based models for the treatment of opioid use disorder. Annals of Internal Medicine, 166(4), 268–278. 10.7326/m16-2149 [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Krause J., Van Lieshout J., Klomp R., Huntink E., Aakhus E., Flottorp S., . . .Baker R. (2014). Identifying determinants of care for tailoring implementation in chronic diseases: An evaluation of different methods. Implementation Science, 9(1), 102. 10.1186/s13012-014-0102-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Krause M., Alford D., Kotz M. (2011). Statement of the American Society of Addiction Medicine consensus panel on the use of buprenorphine in office-based treatment of opioid addiction. Journal of Addiction Medicine, 5(4), 254–263. 10.1097/ADM.0b013e3182312983 [DOI] [PubMed] [Google Scholar]
  26. Miele G. M., Caton L., Freese T. E., McGovern M., Darfler K., Antonini V. P., Perez M., Rawson R. (2020). Implementation of the hub and spoke model for opioid use disorders in California: Rationale, design and anticipated impact. Journal of Substance Abuse Treatment, 108, 20–25. 10.1016/j.jsat.2019.07.013 [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Murthy V. (2016). Ending the opioid epidemic—A call to action. New England Journal of Medicine, 375(8), 2413–2415. 10.1056/NEJMp1612578 [DOI] [PubMed] [Google Scholar]
  28. Nadeem E., Olin S., Hill L., Hoagwood K. (2014). A literature review of learning collaboratives in mental health care: Used but untested. Psychiatric Services, 65(9), 1088–1099. 10.1176/appi.ps.201300229 [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Nordstrom B., Saunders E., McLeman B., Meier A., Xie H., McGovern M. (2016). Using a learning collaborative strategy with office-based practices to increase access and improve quality of care for patients with opioid use disorders. Journal of Addiction Medicine, 10(2), 117–123. 10.1097/ADM.0000000000000200 [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Powell B. J., Waltz T. J., Chinman M. J., Damschroder L. J., Smith J. L., Matthieu M. M., Proctor E. K., Kirchner J. E. (2015). A refined compilation of implementation strategies: Results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science : IS, 10, 21, 10.1186/s13012-015-0209-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Powell B. J., Beidas R. S., Lewis C. C., Aarons G. A., McMillen J. C., Proctor E. K., Mandell D. S. (2017). Methods to improve the selection and tailoring of implementation strategies. Journal of Behavioral Health Services Research, 44(2), 177–194. 10.1007/s11414-015-9475-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Powell B. J., Fernandez M. E., Williams N. J., Aarons G. A., Beidas R. S., Lewis C. C., . . . Weiner B. J. (2019). Enhancing the impact of implementation strategies in healthcare: A research agenda. Frontiers in Public Health, 7, Article 3. 10.3389/fpubh.2019.00003 [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Powell B. J., Haley A. D., Patel S. V., Amaya-Jackson L., Glienke B., Blythe M., . . . Weinberger M. (2020). Improving the implementation and sustainment of evidence-based practices in community mental health organizations: A study protocol for a matched-pair cluster randomized pilot study of the Collaborative Organizational Approach to Selecting and Tailoring Implementation Strategies (COAST-IS). Implementation Science Communications, 1(1), 9. 10.1186/s43058-020-00009-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Proctor E. K., Powell B. J., McMillen J. C. (2013). Implementation strategies: Recommendations for specifying and reporting. Implementation Science, 8, 139. 10.1186/1748-5908-8-139 [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Project ECHO. (2019). Project ECHO—changing the world. University of New Mexico. https://echo.unm.edu/ [Google Scholar]
  36. Ritchie M., Dollar K., Miller C., Oliver K., Smith J., Lindsay J., Kirchner J. (2017). Using implementation facilitation to improve care in the Veterans Health Administration (Version 2). Veterans Health Administration, Quality Enhancement Research Initiative (QUERI) for Team-Based Behavioral Health. https://www.queri.research.va.gov/tools/implementation.cfm [Google Scholar]
  37. Saldana L. (2014). The Stages of Implementation Completion for evidence-based practice: Protocol for a mixed methods study. Implementation Science, 9(1), 43. 10.1186/1748-5908-9-43 [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Saldana L., Chamberlain P., Bradford W. D., Campbell M., Landsverk J. (2014). The Cost of Implementing New Strategies (COINS): A method for mapping implementation resources using the stages of implementation completion. Children and Youth Services Review, 39, 177–182. 10.1016/j.childyouth.2013.10.006 [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Saldana L., Chamberlain P., Wang W., Brown C. H. (2012). Predicting program start-up using the stages of implementation measure. Administration and Policy in Mental Health, 39(6), 419–425. 10.1007/s10488-011-0363-y [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Saloner B., Karthikeyan S. (2015). Changes in substance abuse treatment use among individuals with opioid use disorders in the United States, 2004-2013. Journal of the American Medical Association, 314(14), 1515–1517. 10.1001/jama.2015.10345 [DOI] [PubMed] [Google Scholar]
  41. Schouten L. M., Hulscher M. E., van Everdingen J. J., Huijsman R., Grol R. P. (2008). Evidence for the impact of quality improvement collaboratives: Systematic review. British Journal of Medicine, 336(7659), 1491–1494. 10.1136/bmj.39570.749884.BE [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Simpatico T. A. (2015). Vermont responds to its opioid crisis. Preventive Medicine, 80, 10–11. 10.1016/j.ypmed.2015.04.002 [DOI] [PubMed] [Google Scholar]
  43. Squires J. E., Sullivan K., Eccles M., Worswick J., Grimshaw J. (2014). Are multifaceted interventions more effective than single-component interventions in changing health-care professionals’ behaviours? An overview of systematic reviews. Implementation Science, 9, 152. 10.1186/s13012-014-0152-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Tofighi B., Isaacs N., Byrnes-Enoch H., Lakew R., Lee J. D., Berry C., Schatz D. (2019). Expanding treatment for opioid use disorder in publicly funded primary care clinics: Exploratory evaluation of the NYC health + hospitals buprenorphine ECHO program. Journal of Substance Abuse Treatment, 106, 1–3. 10.1016/j.jsat.2019.08.003 [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. 21st Century Cures Act, H. R. 34 C.F.R. (2016). [Google Scholar]
  46. U.S. Census Bureau. (2010). Geographic terms and concepts—Core based statistical areas and related statistical areas. https://www.census.gov/topics/housing/housing-patterns/about/core-based-statistical-areas.html
  47. Volkow N. D., Frieden T. R., Hyde P. S., Cha S. S. (2014). Medication-assisted therapies—Tackling the opioid-overdose epidemic. New England Journal of Medicine, 370(22), 2063–2066. 10.1056/NEJMp1402780 [DOI] [PubMed] [Google Scholar]
  48. Watson D. P., Andraka-Christou B., Clarke T., Wiegandt J. (2020). Introduction to the special issue on innovative interventions and approaches to expand medication assisted treatment: Seizing research opportunities made available by the opioid STR program. Journal of Substance Abuse Treatment, 108, 1–3. 10.1016/j.jsat.2019.10.004 [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from Implementation Research and Practice are provided here courtesy of SAGE Publications

RESOURCES