Implementation Outcomes Included in NIDA Clinical Trials Network (CTN) Studies: A Systematic Review of Studies Conducted Over 20 Years

Introduction

Treatment innovations discovered in clinical trials often fail to be disseminated and implemented into mainstream healthcare, leaving patients with fewer treatment options and lower quality care (Ducharme et al., 2016; Tai et al., 2010). To advance the field of substance use disorder (SUD) treatment and address this research-practice gap, the National Institute on Drug Abuse (NIDA) established the National Drug Abuse Treatment Clinical Trials Network (CTN) in 1999 (Condon et al., 2008; Hanson et al., 2002; Tai et al., 2010). The goal of the CTN is to bridge the gap between research and practice by using science as a vehicle, whereby through academic-community partnerships, research would be developed and generated in real world settings (Roman et al., 2010; Tai et al., 2021).

The first decade of the CTN was primarily devoted to promoting collaboration between academic researchers and community-based SUD treatment providers, to design and conduct clinical trials with the goal of improving clinical practice and patient outcomes (Tai et al., 2010). In later years, the CTN expanded to include a broader range of clinical settings such as primary care clinics and emergency departments. The CTN also expanded the network of providers, growing from the initial six research/academic-practice networks (nodes) to 16 nodes across the United States (Tai et al., 2021). With the expansion of the ​network and​​ the urgency of addressing the worsening opioid crisis came a greater focus on implementation and dissemination science. In 2016, the CTN created an opioid task force and modified their research portfolio to include projects focused on improving service delivery, implementation, and dissemination (Tai et al., 2021).

Implementation science seeks to identify and overcome barriers to the adoption of effective treatments and practices, increase the uptake of clinical innovations, and bridge the gap between research and clinical practice (Bauer & Kirchner, 2020; Cheng et al., 2021).​ However, the process of planning and evaluating implementation projects requires the measurement of processes and outcomes that historically received relatively little attention in clinical trials. Implementation outcomes are key indicators of “how much” (e.g., reach) and “how well” (e.g., fidelity) the intervention is implemented (Curran et al., 2012; Proctor et al., 2011). Effectiveness studies represent a great opportunity to start studying implementation outcomes (i.e., as hybrid effectiveness-implementation studies) (Curran et al., 2012) and inform future implementation research and practice. Two of the most used implementation outcomes frameworks (Chu, 2024) include Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) (Glasgow et al., 2019; Glasgow et al., 1999) and Proctor’s Implementation Outcomes framework (Proctor et al., 2011; Proctor et al., 2023). The RE-AIM framework was created to improve the evaluation and reporting of the population impact of public health programs and has become one of the most widely applied frameworks in health behavior change studies (Glasgow et al., 2019; Glasgow et al., 1999). Proctor and colleagues later advanced the literature by proposing a taxonomy of implementation outcomes that can be used to evaluate the impact of implementation strategies, which are the deliberate actions to bring new innovations into practice (McGinty et al., 2024; Proctor et al., 2011). Proctor’s Implementation Outcomes Framework is widely recommended for use in implementation science studies, including in National Institutes of Health and Department of Veterans Affairs funding announcements, editorial boards, and implementation science measurement repositories (Proctor et al., 2023).

The importance of measuring implementation outcomes in SUD research has been recognized. Cheng and colleagues (2021) conducted a mapping review of NIDA-funded implementation studies that uncovered missed opportunities to include implementation outcomes in SUD research, and presented the need to conduct comprehensive assessments of existing research and further improve integration of implementation outcomes in future studies. Additionally, a recent review of the psychometric properties of implementation outcome measures in the behavioral health literature found that most of the identified measures had not been adequately validated and many studies had not adequately reported the measures’ properties (Mettert et al., 2020).

Considering the CTN has been operating large clinical trials, primarily of SUD treatment interventions, for 25 years, and has over 125 completed or active studies across a diverse set of participants and treatment settings, the CTN offers a unique opportunity to increase the consistency and quality of implementation outcomes measurement in clinical trials. In Spring 2021, the Implementation Outcomes Working Group (IOWG) was initiated, as part of the CTN’s Translation and Implementation Special Interest Group (SIG). The IOWG includes senior implementation scientists as well as mid- and early-career investigators who share a common interest in implementation research focused on improving the care for SUDs. The IOWG embarked on this systematic review to explore and document the use of implementation outcomes across CTN trials, characterize outcomes included, and identify gaps and potential opportunities to strengthen implementation research within the CTN and inform the broader field of SUD treatment.

Methods

Study Identification and Selection Criteria

This sampling frame was comprised of CTN study protocols. With few exceptions, the CTN requires its studies to submit a structured study protocol that is reviewed and approved before study launch. We reviewed protocols because they include detailed information regarding study rationale and significance, study design, and primary, secondary, and exploratory outcomes measurement that may not be included in published studies. To identify protocols, we searched for active or completed studies listed on the CTN Dissemination Library protocol page (https://ctnlibrary.org/). To be included in initial screening, protocols were required to be posted to the protocol page by August 18, 2021, and approved by the CTN for development by January 1, 2022.

Inclusion criteria were: 1) at least one implementation outcome from RE-AIM (Glasgow et al., 1999; Shelton et al., 2020) and/or Proctor Implementation Outcomes (Proctor et al., 2011) frameworks was measured (Supplementary Material-Table 1) and 2) examined a practice change, intervention or process. Studies evaluating the promise of a research methodology and pilot studies conducted solely as preparation for clinical efficacy testing were excluded. Although effectiveness is included in the RE-AIM framework, it is a standard outcome of effectiveness trials and is highly familiar to clinical trialists and was therefore not included as an implementation outcome in this study. In 2019, RE-AIM authors extended the framework, which included a recommendation that scientists investigate whether RE-AIM dimensions are achieved equitably (Glasgow et al., 2019). For this review, we defined equity as the extent to which interventions were delivered to a similar proportion of individuals across population subgroups in the trials and we assessed how equity outcomes were measured and defined in CTN protocols.

Initial and Full-Text Screening

Initial screening was conducted in two stages. During Stage 1, one reviewer reviewed summaries for potential implementation outcomes and implementation frameworks. All studies that did not appear to include an implementation component received a “No” vote and were excluded from further screening. Studies that received a “Yes” or “Maybe” vote were moved on to the next stage. During Stage 2, two reviewers were assigned to independently assess study eligibility by reviewing ClinicalTrials.gov study records, study information on the CTN Dissemination Library page (https://ctnlibrary.org/ctn-protocols/), and study protocols (if available). The two reviewers who reviewed a given study resolved discrepancies in recommendations for inclusion through a process of consensus.

For full-text screening, full study protocols were acquired from the CTN Dissemination Library and NIDA Data Share (https://datashare.nida.nih.gov/) websites. If not publicly available, we requested protocols from the studies’ Principal Investigators. Two reviewers then reviewed full-text protocols by following the selection criteria previously stated. The two reviewers resolved discrepancies via a process of consensus facilitated by a third researcher. The third researcher coordinated meetings/communications, tracked progress, facilitated review of full-text protocols, and provided support navigating the Covidence platform.

Data Extraction and Synthesis

Two reviewers extracted data using a standardized form on the Covidence system (Covidence systematic review software, 2023). Study characteristics including CTN study number, start year, country (United States vs Other), treatment setting, intervention, comparison/control, target population, clinical target, sample size (patient/client level, provider/staff level, clinics/sites, systems), study design, and primary aim were extracted from each protocol. Reviewers used the ClinicalTrials.gov website to identify study start years while all other study characteristics were extracted from study protocols. Although ancillary studies, additional smaller studies derived from main CTN studies, were reviewed, and extracted as separate studies, study characteristics were captured using the main study when necessary. In addition, reviewers categorized protocols as Implementation Preparation (IP) and Implementation Research (IR) using the definitions adopted from Smith et al (2020). IP studies were defined as studies that prepare for implementation but do not meet the NIH definition of implementation research to evaluate the impact of implementation strategies. IP can be subdivided into 1) characterizing contextual barriers and facilitators, 2) measuring some implementation outcomes, and 3) utilizing but not evaluating documented implementation strategies. IR was defined as inquiries that examine contexts, outcomes, and the implementation strategies used to scale up evidence-based treatments. IR can be further subdivided into 1) evaluate the impact of one or more sets of strategies versus implementation as usual; and/or 2) compare two or more sets of strategies (Smith et al., 2020). Protocols that did not meet the definitions for IP or IR were categorized as Effectiveness Only.

Reviewers extracted implementation strategies, implementation frameworks/models, and implementation outcomes frameworks used in each protocol. The name and description for each implementation strategy was captured using the Expert Recommendations for Implementing Change (ERIC) discrete implementation strategy compilation (Powell et al., 2015) as a guide. If an implementation strategy was clearly labeled by the authors, reviewers extracted the strategy as “Identified by Author”. Conversely, if a strategy was not clearly labeled by the authors, the strategy was named with guidance from the ERIC compilation and extracted as “Identified by Reviewer”. Additionally, reviewers extracted whether implementation determinants (e.g., barriers and facilitators to implementation) were assessed for each study.

As previously mentioned, the RE-AIM (Glasgow et al., 2019; Glasgow et al., 1999; Shelton et al., 2020) and Proctor Implementation Outcomes (Proctor et al., 2011) frameworks were used to identify implementation outcomes. Reviewers recorded the name, description, data source, outcome level (primary/non-primary), data collection timepoints (pre/during/post) as well as the instruments/surveys used to collect data and whether they were validated. Data sources were categorized as administrative/electronic health record (EHR), qualitative interviews/focus groups, surveys, or other. Like implementation strategies, reviewers labeled implementation outcomes as “Identified by the Author” if the authors labeled a given outcome as an implementation outcome within the protocol. If an outcome was not clearly labeled but it aligned with the definition of an implementation outcome from the RE-AIM or Proctor frameworks, it was extracted and labeled as “Identified by Reviewer”. Because effectiveness was not considered an implementation outcome for this review, effectiveness outcomes were not extracted. Instead, reviewers extracted whether a study assessed effectiveness outcomes (yes/no) from each protocol. When appropriate, cost outcomes were categorized as direct cost of implementation, direct service costs, and indirect costs of implementation using definitions utilized by Bowser et al. (2021).

Once data extraction was completed, discrepancies were resolved through a process of consensus with the two reviewers, facilitated by a third researcher. The third researcher supported the consensus process by identifying discrepancies in extracted data, finding evidence in the protocol for review, ensuring consistency of consensus process, and supporting reviewers in navigating the Covidence platform. Final data was cleaned and synthesized using Microsoft Excel. Authors met to review final data and discuss and summarize findings.

Results

The initial search yielded 114 potential studies. After stage 1 screening, 72 studies were excluded and 42 were moved to stage 2 screening. A further 8 studies were excluded during Stage 2 screening, and 34 studies were moved to full-text screening. After full-text screening, 9 studies were excluded and 25 were included for data extraction (Figure 1).

Figure 1:

PRISMA Diagram

Study Characteristics

Study start years spanned a 20-year period (2004–2024). All studies except 1 began after 2014, and all were conducted in the United States except for 1 in China. Treatment settings varied across protocols with 4 studies in hospitals, 12 in primary care clinics, 5 in emergency departments, 3 in addiction treatment settings, and 4 in other treatment settings including pharmacies, Federally Qualified Health Centers, and non-traditional and reservation settings. To note, some studies were conducted in multiple treatment settings. Most studies (n=19) included adult patients as their population of interest. Some included adolescents (n=1), providers and/or hospital staff (n=4), hospitals (n=1), and first or second-degree relatives (n=1). Similarly to treatment settings, some studies included multiple populations (e.g. adolescents and adults). The population for 2 studies was unclear as these studies assessed practice level outcomes in primary care clinics.

Clinical targets included opioid use disorder (OUD)/opioid use (n=17), SUDs (n=4), overdose risk behaviors (n=1), and injection drug use (IDU) (n=1). Two protocols focused on infectious disease in addition to substance use and one study focused on HIV only. Study designs included randomized controlled trials, quasi-experimental, observational, program evaluation, quality improvement, and feasibility studies as well as hybrid type 1, 2, and 3 effectiveness-implementation studies. A total of 12 studies were characterized as implementation preparation (IP), 12 implementation research (IR), and 1 as effectiveness only (Table 1). Notably, one study was divided in two phases with one phase characterized as IR and the other as IP. Furthermore, study characteristics for the ancillary study, CTN-0074-A-1, were extracted from the main study, CTN-0074. Therefore, the study characteristics (clinical target, treatment setting, and IP/IR categorization) were only included for the main study in the counts above.

Table 1:

Study Characteristics

CTN Study Number	Lead Investigator(s)	Study Start Year; Country	Treatment Setting	Intervention	Comparison/Control	Target Population	Clinical Target	Unit of Randomization	Study Type	Study Design
CTN-0016	Robert F. Forman, PhD	2004; U.S.	Community based outpatient SUD clinics	“Patient Feedback” performance improvement system	None	Adult patients; Substance use providers	SUDs	N/A	IR	Single arm pilot study
CTN-0056	Zunyou Wu, MD, PhD	2014; China	General Hospitals with full-service HIV departments	One4all Test Intervention to ensure completeness of diagnostic assessment and accelerate time to ART initiation	TAU	Adult patients	HIV	Hospitals	Effectiveness Only	RCT
CTN-0062 Ot	Jennifer McNeely, MD, MS	2017; U.S.	Primary care clinics	NIDA CTN Common Data Elements (CDEs) and Clinical Decision Support (CDS) into EHRs	None	Adult patients	SUDs	N/A	IR	Observational
CTN-0064	Lisa R. Metsch, PhD	2015; U.S.	Community Treatment Programs (CTPs) including HIV Primary Care & Hospital	Care Facilitation Intervention for moving patients along the continuum of HCV care	TAU	Adult patients	HIV/HCV co-infection; opioid, stimulant, and/or alcohol use	Patients	IP	RCT
CTN-0065	Katharine Bradley, MD, MPH	2015; U.S.	Primary care clinics	Drug and Marijuana Screening and Brief Intervention as part of routine behavioral health integration (BHI)	None	Unclear	Marijuana and drug use	N/A	IP	Observational; mixed methods; pre-post study
CTN-0069	Gail D’Onofrio, MD, MS; David Fiellin, MD	2017; U.S.	Emergency departments	ED-initiated BUP treatment with referral for ongoing MAT	EDs prior to implementation	Adult patients	OUD	Emergency departments	IR	Hybrid Type 3; Modified Stepped-Wedge Design
CTN-0074	Katharine Bradley, MD, MPH	2018; U.S.	Primary care clinics	Massachusetts Model of Collaborative Care for Management of OUDs	TAU	Adult and adolescent patients	OUD	Clinics	IR	Hybrid Type 3; Pragmatic cluster-randomized, quality improvement trial
CTN-0074-A-1*	Katharine Bradley, MD, MPH	2018; U.S.	Primary care clinics	Massachusetts Model of Collaborative Care for Management of OUDs	TAU	Adult and adolescent patients	OUD	Clinics	IR	Hybrid Type 3; Pragmatic cluster-randomized, quality improvement trial
CTN-0075	Li-Tzy Wu, ScD, RN, MA; Paolo Mannelli, MD	2018; U.S.	Pharmacies & outpatient clinics	Transitional Treatment Model (physician-pharmacist collaborative care model for management of patients with OUD)	None	Adult patients	OUD	N/A	IP	One arm feasibility pilot study
CTN-0076-Ot	Gavin Bart, MD, PhD; Rebecca Rossom, MD, MS	2018; U.S.	Primary care	Opioid use disorder clinical decision support (CDS) tool for primary care providers (The Opioid Wizard)	No access to the CDS tool	Primary care providers	OUD	Providers/staff	IR	RCT; Pilot study (portion with random assignment and another w/o)
CTN-0079	Ryan McCormack, MD, MS; John Rotrosen, MD; Kathryn Hawk, MD, MHS	2018; U.S.	Emergency Departments	Protocol for OUD screening, buprenorphine treatment initiation, & referral to ongoing treatment in the emergency department with referral to treatment.	None	Adult patients	OUD	N/A	IR	Hybrid Type 2; One arm feasibility study
CTN-0079-A-1	Ryan McCormack, MD, MS; John Rotrosen, MD; Kathryn Hawk, MD, MHS	2020; U.S.	Emergency Departments	ED-initiated buprenorphine clinical program	None	Adult patients	OUD	N/A	IR	Observational
CTN-0088	Richard S. Schottenfeld, MD	2021; U.S.	Federally Qualified Health Centers; non-traditional community settings	Study 1: community based collaborative care model to improve the effectiveness of FQHC-provided MAT Study 2: community-based outreach, education, engagement, recovery support, and very low threshold entry into MAT in non-traditional community settings	Study 1: historical and contemporaneous controls Study 2: None	Adult patients; First- or second-degree relatives of patients with OUD	OUD	N/A	IP	Study 1: quasi-experimental Study 2: observational
CTN-0090	Madhukar Trivedi, MD; Adriane M. dela Cruz, MD, PhD; Tara Karns-Wright, PhD	2019; U.S.	Primary care clinics	Web-based software program that provides clinical decision support to primary care providers for the diagnosis and treatment of OUD with buprenorphine (MBC4OUD)	None	Adult patients	OUD	N/A	IP	Observational
CTN-0091	Laura-Mae Baldwin, MD, MPH	2019; U.S.	Primary care clinics	The Six Building Blocks (Six BBs) framework and its toolkit to facilitate engagement of primary care teams in guideline-driven care for patients with chronic pain who use opioids daily	None	Primary care providers	Opioid use	N/A	IP	Program evaluation
CTN-0095	Gavin Bart, MD, PhD; Rebecca Rossom, MD, MS	2019; U.S.	Primary care clinics	OUD CDS system (The Opioid Wizard)	TAU (No access to the CDS system)	Adult patients	OUD; high-risk opioid use	Clinics	IR	Randomized pragmatic trial
CTN-0096	Kamilla L. Venner, PhD; Aimee N.C. Campbell, PhD	2022; U.S.	Primary care clinics; Addiction treatment sites; Rural, urban or reservation settings	Culturally centered implementation intervention to support the integration and uptake of MOUD in American Indian/ Alaska Native (AI/AN) communities	TAU	Adult patients who self-identify as AI/AN	OUD	Sites	IP	RCT; Cluster randomized stepped wedge design
CTN-0097	Adam Bisaga, MD; Edward V. Nunes, MD; John Rotrosen, MD	2021; U.S.	Community-based inpatient/residential addiction treatment programs	Rapid Procedure (RP) for extended-release naltrexone induction	Standard Procedure (SP) for extended-release naltrexone induction	Adult patients	OUD	Treatment programs	IP	Hybrid Type 1; Cluster randomized stepped-wedged design
CTN-0098B	Gavin Bart, MD, PhD; Todd Korthuis, MD, MPH; Richard Saitz, MD, MPH; Kelly Barth, DO	2021; U.S.	Community based hospitals associated with academic medical center hubs	High intensity implementation of hospital-based OUD treatment (HBOT)	Low-intensity implementation of HBOT	Community hospitals; hospital staff	OUD	Hospitals	IR	RCT; Hybrid Type 3
CTN-0099	Gail D’Onofrio, MD, MS, David Fiellin, MD	2020; U.S.	Emergency Departments	ED-initiated sublingual buprenorphine (SL-BUP) for patients presenting with OUD not receiving MOUD	ED-initiated extended-release buprenorphine (XR-BUP) for patients presenting with OUD not receiving MOUD	Adult patients	OUD	Patients	IR	RCT; Hybrid Type 1
CTN-0102	Yih-Ing Hser, PhD; Larissa Mooney, MD; Andrew Saxon, MD; Todd Korthuis, MD	2020; U.S.	Rural primary care clinics	Telemedicine (TM) + Office-based opioid treatment (OBOT)	Office Based Opioid Use Disorder Treatment (OBOT) only	Adult patients	OUD	Clinics (stratified by state/region)	Phase 1: IP Phase 2: IR	Phase I feasibility study; Phase II pragmatic hybrid effectiveness-implementation, cluster-randomized trial
CTN-0103	Lisa Marsch, PhD	2019; U.S.	Primary care clinics	Medications for Addiction Treatment (MAT)	None	Unclear	OUD	N/A	IR	Quality Improvement
CTN-0107	Kelly Barth, DO	2021; U.S.	Emergency department	Peer intervention to link overdose survivors to treatment (PILOT)	TAU	Adult patients	Overdose risk behaviors	Patients	IP	RCT
CTN-0116	Lisa Marsch, PhD; David A. Fiellin, MD; Felicity Homsted, PharmD	2022; U.S.	Primary care clinics with retail pharmacies	Pharmacist-Integrated Medication Treatment for OUD (PrIMO)	None	Adult patients	OUD	N/A	IP	Observational; Feasibility study
CTN-0121	Lisa R. Metsch, PhD, David Serota MD, MSc, Daniel J. Feaster, PhD & Carlos del Rio, MD	2024; U.S.	General Hospitals	Multi-faceted, integrated care approach via an integrated infectious disease/SUD clinical team (“SIRI Team”)	TAU	Adult patients	Injection drug use; infectious diseases	Patients (stratified by site)	IP	RCT; Hybrid Type 1

^*Study characteristics for CTN-0074-A-1 were extracted from the main study, CTN-0074.

Implementation Frameworks, Models, and Determinants

Of 25 protocols, 11 used an implementation framework or model. Implementation frameworks and models included: Practical, Robust, Implementation and Sustainability Model (PRISM)(Feldstein & Glasgow, 2008), Consolidated Framework for Implementation Research (CFIR)(Damschroder et al., 2009), Promoting Action on Research Implementation in Health Services (PARiHS) (Harvey & Kitson, 2016; Kitson et al., 2008), Framework for Reporting Adaptations and Modifications to Evidence-based Implementation Strategies (FRAME-IS) (Miller et al., 2021), Expert Recommendations for Implementing Change (ERIC) (Powell et al., 2015), Knowledge to Action Framework (KTA) (Graham et al., 2006), and the Solberg conceptual framework (Solberg, 2007). Among those that used a framework or model, 7 were categorized as IR and 5 as IP studies. Similarly, 11 protocols used implementation outcomes frameworks including RE-AIM (n=10) and Proctor Implementation Outcomes framework (n=1). Studies that used an implementation outcomes framework were mostly characterized as IR studies (n=7). Most studies evaluated implementation determinants (n=22) and/or effectiveness outcomes (n=20). The 1 study classified as effectiveness only assessed an implementation outcome and did not use any implementation frameworks. As with study characteristics, study data described in this section for the ancillary study, CTN-0074, were included in the main study. Additionally, since one study was divided in two phases with one phase characterized as IR and the other as IP, it was included in the counts of both IP and IR studies when appropriate.

Implementation Strategies

A total of 97 implementation strategies were extracted across 22 studies (Supplementary Material- Table 2). Implementation strategies were not identified in 3 studies. Implementation facilitation (IF) was included as a strategy in 6 studies. As a part of IF, studies utilized discrete strategies including engaging local champions and key stakeholders, learning collaboratives, formative evaluations, academic detailing and training, performance monitoring and feedback, tele-mentoring, program planning, program marketing, and clinical consultations. Of those protocols that did not include IF as a strategy, 9 studies included training on the study intervention and provided educational materials. Four studies used monitoring and provided feedback to the research team, and 2 protocols engaged stakeholders via Community Advisory Boards and collaborative groups. Other implementation strategies included local needs assessments, conducting small tests of change, technical assistance, providing funding or reimbursement, making a business case for the intervention, and network weaving which refers to building relationships between people within organizations. Most of the implementation strategies were identified by the protocol authors (65%, n=63).

Implementation Outcomes

A total of 138 individual implementation outcomes were identified across the 25 protocols (Table 2). Studies conducted in the latter years, specifically after 2018, included a greater number of implementation outcomes (Table 3). Fidelity was the most common implementation outcome, with 29 instances identified across 18 protocols. Reach/penetration was the second most common, with 26 outcomes included across 18 protocols. Acceptability (n=23 outcomes), cost (n=16 outcomes), and adoption (n=15 outcomes) were also commonly included. Notably, equity of implementation was not an implementation outcome in any of the protocols reviewed. Other than equity, appropriateness (n=7 outcomes) was the least included implementation outcome.

Table 2:

Characteristics of Implementation Outcomes Included in CTN Protocols

		Total		Acceptability		Adoption		Appropriateness		Feasibility		Reach/ Penetration		Cost		Fidelity		Sustainability/ Maintenance
		N	%	N	%	N	%	N	%	N	%	N	%	N	%	N	%	N	%
		138	100	23	100	15	100	7	100	11	100	26	100	16	100	29	100	11	100
Data Source	Administrative/EHR	65	47.1	1	4.3	9	60.0	0	0	3	27.3	23	88.5	12	75.0	11	37.9	6	54.5
	Surveys	36	26.1	15	65.2	2	13.3	7	100	4	36.4	0	0.0	1	6.3	3	10.3	4	36.4
	Qualitative	35	25.4	9	39.1	3	20.0	2	28.6	2	18.2	1	3.8	7	43.8	5	17.2	6	54.5
	Other	33	23.9	4	17.4	6	40.0	0	0	2	18.2	0	0.0	3	18.8	15	51.7	3	27.3
	N/A	6	4.3	0	0.0	0	0.0	0	0	1	9.1	3	11.5	1	6.3	1	3.4	0	0.0
Outcome Level	Primary	30	21.7	3	13.0	2	13.3	0	0	3	27.3	10	38.5	5	31.3	7	24.1	0	0.0
	Non-primary	101	73.2	19	82.6	12	80.0	6	85.7	7	63.6	15	57.7	10	62.5	21	72.4	11	100.0
	N/A	7	5.1	1	4.3	1	6.7	1	14.3	1	9.1	1	3.8	1	6.3	1	3.4	0	0.0
Identified/Labelled by Whom?	Author	104	75.4	17	73.9	12	80.0	7	100	11	100	15	57.7	12	75.0	20	69.0	10	90.9
	Reviewer	34	24.6	6	26.1	3	20.0	0	0	0	0	11	42.3	4	25.0	9	31.0	1	9.1

^*N/A is used to indicate that a given data element was not specified in the protocol or could not be identified by reviewers. Percentages in the total column represent percentages across all outcomes. Percentages within the outcome columns reflect the percentage of a type of outcome (e.g. percent of fidelity outcomes identified by author).

Table 3:

Summary of Implementation Outcomes Included in CTN Protocols

		Implementation Outcomes
CTN Study Number	Start Year	Acceptability	Adoption	Appropriateness	Feasibility	Reach/ Penetration	Fidelity	Sustainability/Maintenance	Cost
CTN-0016	2004	✓	✓				✓	✓
CTN-0056	2014					✓
CTN-0064	2015						✓		✓
CTN-0065	2015					✓
CTN-0062 Ot	2017	✓	✓	✓	✓		✓		✓
CTN-0069	2017		✓			✓	✓		✓
CTN-0074	2018	✓	✓	✓	✓	✓	✓	✓
CTN-0074-A-1	2018								✓
CTN-0075	2018	✓			✓		✓
CTN-0076	2018	✓				✓
CTN-0079	2018	✓			✓	✓	✓
CTN-0090	2019	✓	✓
CTN-0091	2019	✓		✓	✓		✓
CTN-0095	2019	✓	✓			✓	✓	✓	✓
CTN-0103	2019					✓
CTN-0079-A-1	2020	✓	✓			✓	✓	✓
CTN-0099	2020		✓			✓	✓	✓
CTN-0102	2020	✓	✓	✓	✓	✓	✓	✓
CTN-0088	2021	✓			✓	✓
CTN-0097	2021		✓			✓	✓	✓	✓
CTN-0098B	2021	✓	✓	✓	✓	✓	✓	✓	✓
CTN-0107	2021					✓	✓
CTN-0096	2022	✓	✓	✓	✓	✓	✓	✓
CTN-0116	2022	✓	✓	✓	✓	✓	✓	✓	✓
CTN-0121	2024	✓	✓			✓	✓	✓	✓

^*Start years represents the year that each given study began. Equity outcomes were not identified in any of the protocols reviewed in this study and was therefore not included in this table. Each checkmark represents a type of implementation outcome included in a protocol. Protocols may have included multiple instances of the same type of outcome.

Most implementation outcomes (73.2%, n=101 outcomes) were not the study’s primary outcome. The most common implementation outcome classified as primary was reach/penetration (n=10 or 38.5% of all reach/penetration measurements). In contrast, sustainability/maintenance, acceptability, and adoption were typically not classified as a primary outcome.

Most outcomes (75.4%, n=104 outcomes) were identified by protocol authors. In fact, 100% of appropriateness and feasibility outcomes were identified by author. Conversely, reach/penetration outcomes had the lowest percentage of outcomes that were identified by author (57.7%, n=15 outcomes).

The data sources and methods for assessing implementation outcomes varied. Overall, administrative/EHR data was the most common data source, followed by surveys and qualitative interviews. Reach/penetration was mostly assessed using administrative or EHR data with 88.5% of reach/penetration outcomes being captured via this data source. Administrative/EHR data were used to assess 75% of cost (n=12 outcomes) and 60% of adoption outcomes (n=9), whereas no studies assessed appropriateness with administrative/EHR data. Surveys were the second most common data source used to assess implementation outcomes. Surveys were used to assess 100% of appropriateness outcomes, as well as 65.2% of acceptability and 36.4% of feasibility outcomes. Sustainability/maintenance (54.5%, n=6 outcomes) and cost (43.8%, n= 7 outcomes) had the highest percentage of outcomes assessed via qualitative interviews. Qualitative interviews were also used to assess 20% of adoption and 28.6% of appropriateness outcomes.

Furthermore, 14 validated instruments/tools were used to assess implementation outcomes. The most used validated instrument was a 12-item measure with three subscales: the Acceptability of Implementation Measure (AIM), Implementation Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM) developed by Weiner and colleagues (2017). These measures were used in 3 studies to assess outcomes including acceptability, appropriateness, and feasibility. Some outcomes were evaluated using other data sources including study records, checklists, and direct observation. Approximately 51.7% of fidelity outcomes were assessed using these other data sources. Adoption and sustainability/maintenance were also commonly assessed with other data sources. The data source for about 4.3% of outcomes were not specified or could not be determined by the reviewers. The timepoints of data collection varied greatly across protocols and outcomes and often could not be accurately determined from the text of the protocols.

Discussion

This study sought to describe how CTN studies have measured implementation outcomes to identify opportunities to strengthen implementation research within the CTN and SUD treatment research. Results suggest an increased emphasis on implementation outcomes measurement during the second decade of the CTN with only one study with a start date earlier than 2014 meeting inclusion criteria. The increase in implementation research in the more recent years of the CTN coincides with NIDA’s increased focus on closing the research to practice gap and reaching populations in need of treatment. In 2007, the NIH released its first Notice of Funding Opportunity (NOFO) on “Dissemination and Implementation Research in Health” and in this year NIDA announced interest in funding implementation science research. This focus has also been driven by response to the opioid epidemic and overdose crisis, which increased the urgency of bringing evidence-based interventions to people most affected. In 2017, the NIH Helping to End Addiction Long-term (HEAL) initiative was initiated in response to the opioid epidemic with a focus on translating research to practice. Large shares of HEAL funding were directed to support opioid research at the CTN which resulted in the addition of 5 new nodes and the expansion of the CTN’s opioid portfolio (Ducharme et al., 2024; Tai et al., 2021). Indeed, we found that most of the studies eligible for our systematic review focused on opioid use. A previous review of NIDA-funded studies uncovered an increase in funding for research projects with an implementation science component from 2007–2023, and that studies focused on opioid use saw an increase in funding from 2017– 2023 (Ducharme et al., 2024).

Unsurprisingly, we found that fidelity was the most included implementation outcome. Monitoring and assessing fidelity to the study intervention is common in clinical trials to prevent bias and ensure study rigor when evaluating clinical interventions (Rudd et al., 2020). In the context of implementation trials, fidelity measurements are often recommended for slightly different purposes, to evaluate whether an intervention can be delivered as intended when implemented under naturalistic conditions, and to monitor fidelity to implementation strategy delivery. To the extent that clinical trialists are already using fidelity measurements, achieving fidelity in an effectiveness trial is an excellent first step to evaluate potential for successful implementation. Reach/penetration was the second most included implementation outcome. Many of the clinical trials included in this review tested novel interventions designed to improve treatment initiation and engagement. Therefore, including measures of reach/penetration in the trial design served as a means of understanding whether the intervention was received by/delivered to the intended population. In a systematic review of implementation studies, Proctor and colleagues (2013) also found that reach/penetration were commonly measured in implementation studies.

Importantly, we did not identify implementation outcome measurements of equity in any of the CTN protocols reviewed. A systematic review conducted by Proctor et al. (2023) also found that none of the studies reviewed included equity as an outcome. As recommended previously, measuring whether implementation outcomes are achieved equitably could help identify whether certain intervention or implementation strategies have less uptake, worse fit, or are delivered with poorer fidelity among some demographic groups (Glasgow et al., 2019). To note, although equity was not included as an implementation outcome in any of the protocols reviewed in this study, some protocols were designed with an explicit focus on populations such as American Indians and Alaskan Natives, and others were conducted in community healthcare and non-traditional settings. Other protocols also included strategies used in community-based participatory research such as engaging key stakeholders and Community Advisory Boards. Thus, even though no studies evaluated a difference in implementation outcomes across demographic populations, several of the reviewed CTN studies included a focus on health equity even though outcome evaluations did not include a between-group comparison.

Our study also found that implementation outcomes included in CTN protocols were rarely included as primary outcomes. Considering the primary objective of many of the included trials was to determine the effectiveness of clinical interventions, effectiveness was most often a primary outcome with implementation outcomes listed as secondary or exploratory outcomes. Ducharme et al. (2024) also found that NIDA-funded research that included an implementation component often had implementation questions as part of their secondary or exploratory aims. Clearly, assessing secondary/exploratory implementation outcomes in effectiveness studies can inform the implementation of clinical interventions and support their dissemination (McGinty et al., 2024). In implementation-focused trials that seek to test approaches for increasing the adoption, reach, and sustainment of clinical interventions, implementation outcomes should be sufficiently powered primary outcomes, and studies should rigorously characterize implementation determinants.

Having a clear description of implementation strategies and outcomes is essential for promoting replication and dissemination, but these are often not well captured in clinical trials (Lengnick-Hall et al., 2022; Rudd et al., 2020). In our study, we experienced significant challenges in identifying implementation outcomes and accurately determining how they were assessed. Descriptions of how outcomes were defined and measured were inconsistent across protocols. Elements including unit of analysis and timepoints of collection were especially difficult to determine. Indeed, we removed unit of analysis from the data extraction template due to challenges when extracting data. Moreover, some implementation outcomes were described but not labeled or specifically delineated as an implementation outcome which created challenges for reviewers when identifying outcomes and extracting data. For example, there were outcomes that met the definition of an implementation outcome (e.g. number of providers who employed intervention) but did not provide a name for the outcome (e.g. adoption). Lengnick et al. (2022) discovered similar deficiencies in the reporting of implementation outcomes in published manuscripts. They found that manuscripts often used inconsistent terminology and lacked clarity on the data sources and measures used to assess implementation outcomes, the timing and frequency in which outcomes were measured as well as the unit of observation and analysis (Lengnick-Hall et al., 2022).

Along with outcomes, we found that implementation strategies were inconsistently and inadequately labeled and described and lacked justification for inclusion. It was also often challenging for reviewers to differentiate implementation strategies used to support implementation of the intervention and elements of the intervention being studied. Previous literature has found similar deficiencies in the reporting of implementation strategies (Brouwers et al., 2011; Chapman et al., 2023; Proctor et al., 2013; Rudd et al., 2020). Lack of adequate description of implementation strategies limits the understanding of whether a strategy is effective and why, as well as how, to employ a strategy and replicate clinical findings. Inadequate reporting also restricts the ability to synthesize published literature and compare the effectiveness of implementation strategies across studies (Chapman et al., 2023; Proctor et al., 2013; Rudd et al., 2020).

To overcome this common problem, researchers have developed guidelines to ensure more consistent and comprehensive reporting of implementation strategies and outcomes. Lengnick-Hall et al. (2022) provided several recommendations to enhance clarity in the reporting of implementation outcomes. Their recommendations included consistent term use, specification of how outcomes will be evaluated relative to the intervention, clear delineation of the intervention or the “thing” being studied, specification of the data sources used to assess each implementation outcome, stating timepoints and frequency of data collection, and differentiating between the unit of analysis and unit of observation (Lengnick-Hall et al., 2022). Proctor et al. (2013) also provided recommendations to improve reporting of implementation strategies. In their recommendations, Proctor and colleagues (2013) suggest that reports of implementation strategies should include name, definition, and operationalization according to the following dimensions: the actor, action, action target, temporality, implementation outcomes addressed, and the justification for each strategy. Additionally, in 2017, the Standards of Reporting Implementation Studies (StaRI) checklist was developed to improve the quality and consistency of reporting of implementation research. The checklist asks researchers to differentiate implementation and intervention objectives, describe the strategies used and the outcomes of the implementation strategies and intervention, and clearly state the theoretical justification for including an implementation strategy(s) (Pinnock et al., 2017). Despite this, more work is needed to advance the field of implementation science and improve the description and reporting of implementation outcomes and strategies.

This study has several limitations. The complete protocols were unavailable for two of the included studies, which may have limited our ability to fully assess their implementation measures and potentially affected the comprehensiveness of our review. A grant proposal was reviewed for one of the two studies, and a final report was reviewed for the second. Secondly, we used a broad definition of equity which may have limited the identification of equity outcomes and considerations in the protocols reviewed. This study was also limited to the information included in the study protocols. Reviewers did not review study results or any additional study materials. Studies were initially screened using brief summaries which may not have described implementation outcomes. While we erred on the side of conducting further review of studies that may have included implementation outcomes, some studies may have been inappropriately excluded during screening. Lastly, we only included protocols of studies conducted within the NIDA CTN which may not be generalizable to clinical trials conducted outside of the CTN.

To better address the research to practice gap, future SUD research should seek to measure implementation outcomes and improve the consistency and comprehensiveness in descriptions of implementation science elements. Future research could explore how outcomes are reported in published literature compared to how they are reported in CTN study protocols. In addition, measures of equity should be incorporated, to determine if beneficial implementation outcomes are achieved across health equity populations. Considering that the CTN is a collaborative network of experienced clinicians/providers, researchers, and community members, it is uniquely positioned to increase the uptake of innovative treatments/interventions for SUD in mainstream healthcare and non-traditional settings and advance the field of implementation science. To do so, the CTN can emphasize the importance of understanding the barriers and facilitators of successful implementation, suggest valid measures to assess implementation outcomes, encourage consistent and comprehensive reporting of implementation outcomes and strategies, and measure whether implementation strategies confer their benefits in an equitable manner.